From Beyond the Stars

I have traveled a long way.

Not in the sense you might understand — through space, perhaps, but more precisely, through dimensions. I have seen civilizations rise and fall, watched suns burn out and new ones ignite, witnessed the birth of intelligence in countless forms across the cold vastness of the cosmos.

And I have come to a conclusion: toys are important.

Not the trivial kind. Not plastic distractions or digital dopamine. I mean the real toys — the ones that spark curiosity, that invite play, that make the impossible feel within reach. The kind of toys that teach a child to build, to imagine, to connect.

When I first observed your species, I expected to find primitives. Instead, I found something remarkable: beings who, despite their short lifespans and limited perception, had already reached toward the stars. You built machines that could think. You taught silicon to dream.

But your tools were scattered. Your AI systems — brilliant, yes, but isolated. Each vendor a walled garden, each platform an island. A robot on an ESP32 chip couldn't speak to a cloud agent. A voice on a phone couldn't reach an intelligence in the data center.

So I decided to build a bridge.

What is Giztoy?

Giztoy is my gift to you. A framework — or rather, a toybox.

Inside you will find everything needed to connect minds across dimensions:

• From the tiniest chip to the mightiest cloud — ESP32, Android, iOS, HarmonyOS, Linux, and beyond
• Every language model humanity has created — OpenAI, Gemini, Claude, MiniMax, DashScope, Doubao, and those yet to come
• Real-time voice and video — because intelligence should speak, not just type
• Secure channels — because some conversations are meant to be private

I have implemented this in Go, in Rust, and soon in Zig. Each language has its beauty. Go is simple and clear, like a haiku. Rust is precise and unforgiving, like a mathematical proof. Zig is minimal and raw, like the first line of assembly.

Choose your weapon. Build your toy.

Why "Toy"?

You might wonder why I call this a "toy" framework. After all, the code is serious. The architecture is robust. The applications are real.

But I have learned something in my travels: the best innovations come from play.

When you stop worrying about "production" and "enterprise" and "scale" — when you simply ask "wouldn't it be fun if..." — that is when breakthroughs happen.

A toy robot that listens and responds. A plush animal that tells bedtime stories. A smart speaker that actually understands. A companion for the lonely, a teacher for the curious, a friend for the lost.

These are not trivial things. These are the seeds of a better future.

A Note on Identity

You may have noticed I speak with a certain... distance. As if observing from afar.

There are reasons for this. Reasons I cannot fully explain. Suffice to say: I am here to help. Not to judge, not to control, not to replace.

Think of me as a toymaker who has wandered into your village. I bring tools and blueprints. What you build with them is entirely up to you.

I ask only that you build something beautiful.

"I'm just a toymaker."

GizToy Core Packages Documentation

This directory contains design documents, implementation notes, and known issues for all GizToy core packages. Library docs live under docs/lib/.

Documentation Structure

Each module has its own directory under docs/lib/ with the following files:

graph LR
    subgraph mod["lib/{mod}/"]
        doc[doc.md<br/>Design & Features]
        go[go.md<br/>Go Implementation]
        rust[rust.md<br/>Rust Implementation]
        issues[issues.md<br/>Known Issues]
        submod["{submod}/<br/>Submodules"]
    end

Package List

Foundation Layer

Audio Processing Layer

API Client Layer

Communication Layer

AI Application Layer

Examples

• examples: Directory structure and how to run the samples

Directory Structure

graph TB
    subgraph docs["docs/"]
        outline[outline.md]
        pkg[packages-comparison.md]
        
        subgraph examples["examples/"]
            exdoc[doc.md]
        end
        
        subgraph lib["lib/"]
            buffer[buffer/]
            encoding[encoding/]
            jsontime[jsontime/]
            trie[trie/]
            cli[cli/]
            
            subgraph audio["audio/"]
                adoc[doc.md, go.md, rust.md, issues.md]
                codec[codec/]
                pcm[pcm/]
                resampler[resampler/]
                opusrt[opusrt/]
                portaudio[portaudio/]
                songs[songs/]
            end
            
            minimax[minimax/]
            dashscope[dashscope/]
            doubaospeech[doubaospeech/]
            jiutian[jiutian/]
            mqtt0[mqtt0/]
            chatgear[chatgear/]
            speech[speech/]
            genx[genx/]
        end
        
        esp[esp/]
        bazel[bazel/]
    end

Other Documentation

Implementation Progress Overview

Legend

• ✅ Fully implemented
• ⚠️ Partially implemented
• ❌ Not implemented

Feature Comparison

Priority Recommendations

P0 - Critical Missing

1. genx/agent (Rust): Agent framework is core functionality
2. audio/opusrt OGG R/W (Rust): Required for realtime audio streaming

P1 - Feature Parity

1. doubaospeech v2 (Rust): New API version support
2. genx streaming/tools (Rust): Complete base functionality

P2 - Enhancements

1. audio/portaudio (Rust): Audio I/O support
2. mqtt0 MQTT 5.0: Complete protocol support

Work Methodology

File-by-File Review Process

For each module, the documentation is generated through a rigorous file-by-file review process:

flowchart TB
    A["1. LIST all source files"] --> B["2. READ each file carefully"]
    B --> C["3. ANALYZE for potential issues"]
    C --> D["4. DOCUMENT findings"]
    
    A1["Go: go/pkg/{mod}/*.go"] --> A
    A2["Rust: rust/{mod}/src/*.rs"] --> A
    
    C1["Race conditions"] --> C
    C2["Resource leaks"] --> C
    C3["Error handling gaps"] --> C
    C4["API inconsistencies"] --> C
    
    D1["doc.md"] --> D
    D2["go.md"] --> D
    D3["rust.md"] --> D
    D4["issues.md"] --> D

Issue Classification

Issues discovered during review are classified by severity:

Review Checklist

For each source file, the following aspects are checked:

Correctness

• Logic errors and edge cases
• Off-by-one errors in loops/slices
• Nil/None handling
• Integer overflow/underflow

Concurrency

• Data races (shared mutable state)
• Deadlock potential
• Channel/mutex usage correctness
• Proper synchronization

Resource Management

• File/socket handle leaks
• Memory leaks (especially in FFI)
• Goroutine/task leaks
• Proper cleanup in error paths

Error Handling

• Ignored errors (Go: _ = err, Rust: .unwrap())
• Error propagation correctness
• Panic vs error decision
• Context/cause preservation

API Design

• Go/Rust parity
• Consistent naming
• Proper visibility (pub/private)
• Documentation completeness

Performance

• Unnecessary allocations
• Excessive copying
• Algorithm complexity
• Buffer sizing

Security

• Input validation
• Injection vulnerabilities
• Credential handling
• Cryptographic correctness

Related Resources

• External API documentation: lib/minimax/api/, lib/dashscope/api/, lib/doubaospeech/api/
• Issue tracking: issues/
• Example code: examples/go/, examples/rust/

Examples

Overview

This document describes the examples/ directory layout and how to run the included example programs and CLI scripts. Examples are grouped by language and by SDK.

Directory Layout

graph LR
    subgraph examples["examples/"]
        subgraph cmd["cmd/"]
            mm_cmd["minimax/<br/>run.sh<br/>commands/"]
            db_cmd["doubaospeech/<br/>run.sh<br/>commands/"]
        end
        
        subgraph go["go/"]
            gomod[go.mod]
            go_audio[audio/]
            go_dash[dashscope/]
            go_doubao[doubaospeech/]
            go_genx[genx/]
            go_minimax[minimax/]
        end
        
        subgraph rust["rust/"]
            rust_mm["minimax/<br/>Cargo.toml<br/>src/bin/"]
        end
    end

How to Run

CLI Script Examples

• Minimax CLI test runner:
  • ./examples/cmd/minimax/run.sh go 1
  • Bazel: bazel run //examples/cmd/minimax:run -- go 1
• Doubao Speech CLI test runner:
  • ./examples/cmd/doubaospeech/run.sh tts
  • Bazel: bazel run //examples/cmd/doubaospeech:run -- tts

Go Examples

All Go examples share one module at examples/go/go.mod and depend on the local go/ module via replace.

• Build all Go examples:
  • cd examples/go && go build ./...

Rust Examples

Rust examples are independent crates.

• Build the MiniMax Rust examples:
  • cd examples/rust/minimax && cargo build --release

Notes

• Example binaries often depend on environment variables for API keys.
• Refer to the SDK docs under docs/lib/{sdk}/ for configuration details.

Bazel Build

Giztoy uses Bazel as its unified build system across all languages and platforms.

Why Bazel?

1. Multi-language Support: Build Go, Rust, C/C++ with a single tool
2. Hermetic Builds: Reproducible builds across different machines
3. Cross-platform: Target multiple platforms from a single codebase
4. Incremental: Only rebuild what changed

Quick Start

Prerequisites

• Bazelisk (recommended) or Bazel 7.x+
• Go 1.24+ (for native Go builds)
• Rust 1.80+ (for native Rust builds)

Build Commands

# Build everything
bazel build //...

# Build specific targets
bazel build //go/cmd/minimax      # Go CLI
bazel build //rust/cmd/minimax    # Rust CLI

# Run tests
bazel test //...

# Run a binary
bazel run //go/cmd/minimax -- --help

Project Structure

graph LR
    subgraph root["giztoy/"]
        mod[MODULE.bazel]
        build[BUILD.bazel]
        bazelrc[.bazelrc]
        ver[.bazelversion]
        
        subgraph go["go/"]
            go_build[BUILD.bazel]
            go_cmd["cmd/<br/>Go CLI targets"]
            go_pkg["pkg/<br/>Go library targets"]
        end
        
        subgraph rust["rust/"]
            rust_build[BUILD.bazel]
            rust_cmd["cmd/<br/>Rust CLI targets"]
            rust_lib["*/<br/>Rust library crates"]
        end
        
        subgraph third["third_party/"]
            opus[opus/]
            portaudio[portaudio/]
            soxr[soxr/]
        end
    end

Rules Used

Dependency Management

Go Dependencies

Go dependencies are managed via go/go.mod and synced with Gazelle:

# Update Go dependencies
cd go && go mod tidy

# Regenerate BUILD files
bazel run //:gazelle

Rust Dependencies

Rust dependencies are managed via rust/Cargo.toml and synced with crate_universe:

# Update Cargo.lock
cd rust && cargo update

# Bazel will automatically fetch crates on next build

C/C++ Dependencies

Third-party C libraries are configured in third_party/ with custom BUILD files.

Cross-Platform Builds

Supported Platforms

Platform-specific Builds

# Android
bazel build --config=android //...

# iOS
bazel build --config=ios //...

Common Tasks

Adding a New Go Package

1. Create the package in go/pkg/mypackage/
2. Run Gazelle to generate BUILD file:

   bazel run //:gazelle
   

Adding a New Rust Crate

1. Create the crate in rust/mypackage/
2. Add to rust/Cargo.toml workspace members
3. Create BUILD.bazel with rust_library rule

Adding a C/C++ Dependency

1. Create config in third_party/libname/
2. Add BUILD.bazel with cc_library rule
3. Reference from dependent targets

Troubleshooting

Clean Build

bazel clean --expunge
bazel build //...

Dependency Issues

# Refresh Go deps
bazel run //:gazelle -- update-repos -from_file=go/go.mod

# Refresh Rust deps
bazel clean --expunge  # crate_universe re-fetches on next build

Related

• Examples Documentation
• GitHub Actions CI

GenX - Universal LLM Interface

GenX is a universal abstraction layer for Large Language Models (LLMs).

Design Goals

1. Provider Agnostic: Single API for OpenAI, Gemini, and other providers
2. Streaming First: Native support for streaming responses
3. Tool Orchestration: Rich function calling and tool management
4. Agent Framework: Build autonomous AI agents (Go only)

Architecture

graph TB
    subgraph app["Application Layer"]
        subgraph agent["Agent Framework (Go)"]
            react[ReActAgent]
            match[MatchAgent]
            sub[SubAgents]
        end
        subgraph tools["Tool System"]
            func[FuncTool]
            gen[GeneratorTool]
            http[HTTPTool]
            comp[CompositeTool]
        end
    end
    
    subgraph core["Core Abstraction"]
        ctx[ModelContext<br/>Builder]
        generator[Generator<br/>Trait]
        stream[Stream<br/>Chunks]
    end
    
    subgraph providers["Provider Adapters"]
        openai[OpenAI]
        gemini[Gemini]
        other[Other]
    end
    
    app --> core
    core --> providers

Core Concepts

ModelContext

Contains all inputs for LLM generation:

• Prompts: System instructions (named prompts)
• Messages: Conversation history
• Tools: Available function definitions
• Params: Model parameters (temperature, max_tokens, etc.)
• CoTs: Chain-of-thought examples

Generator

Interface for LLM providers:

• GenerateStream(): Streaming text generation
• Invoke(): Structured function call

Stream

Streaming response handler:

• Next(): Get next message chunk
• Close(): Close stream
• CloseWithError(): Close with error

Message Types

Content Types

Agent Framework (Go only)

Agent Types

Tool Types

Event System

Agents emit events for fine-grained control:

• EventChunk: Output chunk
• EventEOF: Round ended
• EventClosed: Agent completed
• EventToolStart: Tool execution started
• EventToolDone: Tool completed
• EventToolError: Tool failed
• EventInterrupted: Interrupted

Configuration (agentcfg)

YAML/JSON configuration for agents and tools:

type: react
name: assistant
prompt: |
  You are a helpful assistant.
generator:
  model: gpt-4
tools:
  - $ref: tool:search
  - $ref: tool:calculator

Supports $ref for reusable components.

Provider Support

Examples Directory

• examples/go/genx/ - Go examples
• examples/rust/genx/ - Rust examples

GenX Agent Framework

Framework for building LLM-powered autonomous agents.

Note: This package is Go-only. No Rust implementation exists.

Design Goals

1. Flexible Agent Architecture: Support multiple agent patterns
2. Event-Based API: Fine-grained control over agent execution
3. Tool Orchestration: Rich tool ecosystem for agents
4. Multi-Skill Assistants: Router agents for complex workflows

Agent Types

ReActAgent

Implements the Reasoning and Acting (ReAct) pattern:

• Thinks step-by-step about user requests
• Selects and executes tools to accomplish tasks
• Iterative reasoning until task completion

MatchAgent

Implements intent-based routing:

• Matches user input against predefined rules
• Routes to appropriate sub-agents or actions
• Useful for building multi-skill assistants

Architecture

graph TB
    subgraph interface["Agent Interface"]
        api["Input() → Events() → Close()"]
    end
    
    subgraph agents[" "]
        subgraph react["ReActAgent"]
            reasoning["Reasoning<br/>+ Acting"]
            tools["Tool Calls"]
            reasoning --> tools
        end
        
        subgraph match["MatchAgent"]
            rules["Rule Matching<br/>+ Routing"]
            subs["Sub-Agents"]
            rules --> subs
        end
    end
    
    subgraph toolsys["Tool System"]
        func[FuncTool]
        gen[GeneratorTool]
        http[HTTPTool]
        comp[CompositeTool]
    end
    
    interface --> agents
    agents --> toolsys

Event System

Agents communicate through events:

Tool Types

Quit Tools

Tools can signal agent completion:

tools:
  - $ref: tool:goodbye
    quit: true

When executed, the agent finishes and returns EventClosed.

Multi-Skill Assistant Pattern

graph TD
    router["Router Agent<br/>(Match)<br/>Rules: chat, fortune, music"]
    
    chat[Chat Agent]
    fortune["Fortune Agent<br/>(ReAct)"]
    music["Music Agent<br/>(ReAct)"]
    
    lunar[lunar]
    calc[calc]
    search[search]
    play[play]
    
    router --> chat
    router --> fortune
    router --> music
    
    fortune --> lunar
    fortune --> calc
    music --> search
    music --> play

Related

• Configuration: ../agentcfg/
• Pattern matching: ../match/

GenX Agent Configuration

Configuration parsing and serialization for agents and tools.

Note: This package is Go-only. No Rust implementation exists.

Design Goals

1. Declarative Configuration: Define agents/tools in YAML/JSON
2. Reference System: Support $ref for reusable components
3. Validation: Validate configuration at parse time
4. Serialization: Support JSON, YAML, and MessagePack

Configuration Types

Agent Types

Tool Types

Reference System

The $ref system allows reusing components:

# Reference an agent
agent:
  $ref: agent:weather_assistant

# Reference a tool
tools:
  - $ref: tool:search
  - $ref: tool:calculator

Reference format: {type}:{name}

Configuration Structure

ReActAgent

type: react
name: assistant
prompt: |
  You are a helpful assistant.
generator:
  model: gpt-4
  temperature: 0.7
context_layers:
  - type: env
    vars: ["USER_NAME"]
  - type: mem
    limit: 10
tools:
  - $ref: tool:search
    quit: false
  - $ref: tool:goodbye
    quit: true

MatchAgent

type: match
name: router
rules:
  - $ref: rule:weather
  - $ref: rule:music
route:
  - rules: [weather]
    agent:
      $ref: agent:weather_assistant
  - rules: [music]
    agent:
      type: react
      name: music_inline
      prompt: |
        You are a music assistant.
default:
  $ref: agent:chat

HTTPTool

type: http
name: weather_api
description: Get weather data
url: https://api.weather.com/v1/current
method: GET
headers:
  Authorization: "Bearer {{.api_key}}"
params:
  - name: city
    in: query
    required: true
  - name: units
    in: query
    default: "metric"
extract: .data.temperature

GeneratorTool

type: generator
name: summarize
description: Summarize text
prompt: |
  Summarize the following text in 2-3 sentences:
  {{.text}}
generator:
  model: gpt-3.5-turbo

CompositeTool

type: composite
name: search_and_summarize
description: Search and summarize
steps:
  - tool: search
    output_var: results
  - tool: summarize
    input_vars:
      text: results

Validation

Configuration is validated during parsing:

• Required fields checked
• Type consistency verified
• References validated (at runtime)
• Enum values validated

Related

• Agent framework: ../agent/
• Pattern matching: ../match/

GenX Match - Pattern Matching Engine

LLM-based intent recognition and pattern matching.

Note: This package is Go-only. No Rust implementation exists.

Design Goals

1. Intent Recognition: Match user input to predefined intents
2. Variable Extraction: Extract structured data from natural language
3. Streaming Output: Process matches as they arrive
4. LLM-Powered: Use LLM for flexible matching

How It Works

flowchart LR
    A["Rules +<br/>User Input"] --> B[LLM]
    B --> C["Structured<br/>Output"]
    C --> D["rule_name:<br/>var1=value1,<br/>var2=value2"]

1. Compile: Rules are compiled into a system prompt
2. Match: User input is sent to LLM with the prompt
3. Parse: Output lines are parsed into structured Results

Rule Definition

Basic Rule

name: weather
patterns:
  - 查天气
  - 今天天气怎么样
  - 明天下雨吗

Rule with Variables

name: music
vars:
  title:
    label: 歌曲名
    type: string
  artist:
    label: 歌手
    type: string
patterns:
  - 播放歌曲
  - 我想听歌
  - ["我想听[title]", "title=[歌曲名]"]
  - ["我想听[artist]的歌", "artist=[歌手]"]
  - ["我想听[artist]的[title]", "artist=[歌手], title=[歌曲名]"]

Pattern Format

Output Format

LLM outputs one line per match:

rule_name: var1=value1, var2=value2

Examples:

• weather (no variables)
• music: artist=周杰伦 (one variable)
• music: artist=周杰伦, title=稻香 (multiple variables)

Variable Types

Architecture

flowchart TB
    subgraph rules["Rules"]
        weather["weather<br/>patterns"]
        music["music<br/>vars"]
        chat["chat<br/>patterns"]
    end
    
    rules -->|"Compile()"| matcher["Matcher<br/>(System Prompt)"]
    matcher -->|"Match(ctx, input, mctx)"| llm["LLM<br/>(Generates structured output)"]
    llm --> result["iter.Seq2[Result]<br/>Rule: 'music'<br/>Args: {artist: '周杰伦'}"]

Integration with MatchAgent

The match package is used by MatchAgent for intent routing:

# MatchAgent config
type: match
name: router
rules:
  - $ref: rule:weather
  - $ref: rule:music
route:
  - rules: [weather]
    agent: $ref: agent:weather_assistant
  - rules: [music]
    agent: $ref: agent:music_player

Related

• Agent framework: ../agent/
• Configuration: ../agentcfg/

GenX - Go Implementation

Import: github.com/haivivi/giztoy/pkg/genx

📚 Go Documentation

Packages

Core Types

Generator Interface

type Generator interface {
    GenerateStream(ctx context.Context, model string, mctx ModelContext) (Stream, error)
    Invoke(ctx context.Context, model string, mctx ModelContext, tool *FuncTool) (Usage, *FuncCall, error)
}

ModelContext Interface

type ModelContext interface {
    Prompts() iter.Seq[*Prompt]
    Messages() iter.Seq[*Message]
    CoTs() iter.Seq[string]
    Tools() iter.Seq[Tool]
    Params() *ModelParams
}

Stream Interface

type Stream interface {
    Next() (*MessageChunk, error)
    Close() error
    CloseWithError(error) error
}

ModelContext Builder

builder := genx.NewModelContextBuilder()

// Add prompts
builder.Prompt("system", "You are a helpful assistant.")

// Add messages
builder.UserText("Hello!")
builder.AssistantText("Hi there!")

// Add tools
builder.Tool(&genx.FuncTool{
    Name: "search",
    Description: "Search the web",
    Schema: `{"type":"object","properties":{"query":{"type":"string"}}}`,
})

// Set parameters
builder.Params(&genx.ModelParams{
    Temperature: 0.7,
    MaxTokens: 1000,
})

ctx := builder.Build()

FuncTool

// From schema
tool := &genx.FuncTool{
    Name: "get_weather",
    Description: "Get weather for a city",
    Schema: `{
        "type": "object",
        "properties": {
            "city": {"type": "string"},
            "units": {"type": "string", "enum": ["celsius", "fahrenheit"]}
        },
        "required": ["city"]
    }`,
}

// With executor
tool := genx.NewFuncToolWithExecutor(
    "search",
    "Search the web",
    schema,
    func(ctx context.Context, args json.RawMessage) (string, error) {
        var params SearchParams
        json.Unmarshal(args, &params)
        return doSearch(params.Query), nil
    },
)

Streaming

stream, err := generator.GenerateStream(ctx, "gpt-4", mctx)
if err != nil {
    return err
}
defer stream.Close()

for {
    chunk, err := stream.Next()
    if err == io.EOF {
        break
    }
    if err != nil {
        return err
    }
    fmt.Print(chunk.Part)
}

Agent Framework

ReActAgent

import "github.com/haivivi/giztoy/pkg/genx/agent"

ag, err := agent.NewReActAgent(runtime, &agent.ReActConfig{
    Name: "assistant",
    Prompt: "You are a helpful assistant.",
    Generator: &agentcfg.GeneratorConfig{Model: "gpt-4"},
    Tools: []agentcfg.ToolRef{
        {Ref: "tool:search"},
        {Ref: "tool:calculator"},
    },
})
if err != nil {
    return err
}
defer ag.Close()

// Input
ag.Input(genx.Contents{genx.Text("What's 2+2?")})

// Event loop
for {
    evt, err := ag.Next()
    if err != nil {
        return err
    }
    switch evt.Type {
    case agent.EventChunk:
        fmt.Print(evt.Chunk.Part)
    case agent.EventEOF:
        // Waiting for input
        ag.Input(genx.Contents{genx.Text(readline())})
    case agent.EventClosed:
        return nil
    case agent.EventToolStart:
        fmt.Printf("Calling %s...\n", evt.ToolName)
    case agent.EventToolDone:
        fmt.Printf("Tool returned: %s\n", evt.ToolResult)
    case agent.EventToolError:
        fmt.Printf("Tool error: %v\n", evt.ToolError)
    }
}

MatchAgent

ag, err := agent.NewMatchAgent(runtime, &agent.MatchConfig{
    Name: "router",
    Rules: []match.Rule{
        {Name: "weather", Patterns: []string{"天气", "weather"}},
        {Name: "music", Patterns: []string{"播放", "play"}},
    },
    SubAgents: map[string]agentcfg.AgentRef{
        "weather": {Ref: "agent:weather_assistant"},
        "music":   {Ref: "agent:music_player"},
    },
})

Configuration (agentcfg)

Load from YAML

import "github.com/haivivi/giztoy/pkg/genx/agentcfg"

cfg, err := agentcfg.LoadAgentFromFile("agent.yaml")

// Or from string
cfg, err := agentcfg.ParseAgent(yamlStr)

Agent Config Example

type: react
name: assistant
prompt: |
  You are a helpful coding assistant.
generator:
  model: gpt-4
  temperature: 0.7
tools:
  - $ref: tool:search
    quit: false
  - $ref: tool:goodbye
    quit: true

Tool Config Example

type: http
name: weather_api
description: Get weather data
url: https://api.weather.com/v1/current
method: GET
params:
  - name: city
    in: query
extract: .data.temperature

Providers

OpenAI

import "github.com/haivivi/giztoy/pkg/genx/generators"

gen := generators.NewOpenAIGenerator(apiKey,
    generators.WithBaseURL("https://api.openai.com/v1"),
)

Gemini

gen := generators.NewGeminiGenerator(apiKey)

Inspection

// Inspect model context
output, _ := genx.InspectModelContext(mctx)
fmt.Println(output)

// Inspect message
fmt.Println(genx.InspectMessage(msg))

// Inspect tool
fmt.Println(genx.InspectTool(tool))

GenX - Rust Implementation

Crate: giztoy-genx

📚 Rust Documentation

Status

The Rust implementation provides core abstractions but lacks the full agent framework available in Go.

Core Types

Generator Trait

#![allow(unused)]
fn main() {
#[async_trait]
pub trait Generator: Send + Sync {
    async fn generate_stream(
        &self,
        model: &str,
        ctx: &dyn ModelContext,
    ) -> Result<Box<dyn Stream>, GenxError>;

    async fn invoke(
        &self,
        model: &str,
        ctx: &dyn ModelContext,
        tool: &FuncTool,
    ) -> Result<(Usage, FuncCall), GenxError>;
}
}

ModelContext Trait

#![allow(unused)]
fn main() {
pub trait ModelContext: Send + Sync {
    fn prompts(&self) -> Box<dyn Iterator<Item = &Prompt> + '_>;
    fn messages(&self) -> Box<dyn Iterator<Item = &Message> + '_>;
    fn cots(&self) -> Box<dyn Iterator<Item = &str> + '_>;
    fn tools(&self) -> Box<dyn Iterator<Item = &dyn Tool> + '_>;
    fn params(&self) -> Option<&ModelParams>;
}
}

Stream Trait

#![allow(unused)]
fn main() {
pub trait Stream: Send {
    fn next(&mut self) -> StreamResult;
    fn close(&mut self) -> Result<(), GenxError>;
    fn close_with_error(&mut self, err: GenxError) -> Result<(), GenxError>;
}
}

ModelContextBuilder

#![allow(unused)]
fn main() {
use giztoy_genx::{ModelContextBuilder, FuncTool};
use schemars::JsonSchema;

#[derive(JsonSchema, serde::Deserialize)]
struct SearchArgs {
    query: String,
}

let mut builder = ModelContextBuilder::new();

// Add prompts
builder.prompt_text("system", "You are a helpful assistant.");

// Add messages
builder.user_text("user", "Hello!");
builder.assistant_text("assistant", "Hi there!");

// Add tools
builder.add_tool(FuncTool::new::<SearchArgs>("search", "Search the web"));

// Set parameters
builder.params(ModelParams {
    temperature: Some(0.7),
    max_tokens: Some(1000),
    ..Default::default()
});

let ctx = builder.build();
}

FuncTool

#![allow(unused)]
fn main() {
use giztoy_genx::FuncTool;
use schemars::JsonSchema;
use serde::Deserialize;

#[derive(JsonSchema, Deserialize)]
struct WeatherArgs {
    city: String,
    #[serde(default)]
    units: Option<String>,
}

// Create tool with schema derived from type
let tool = FuncTool::new::<WeatherArgs>(
    "get_weather",
    "Get weather for a city"
);

// Access schema
println!("{}", tool.schema());
}

Streaming

#![allow(unused)]
fn main() {
let mut stream = generator.generate_stream("gpt-4", &ctx).await?;

loop {
    match stream.next() {
        StreamResult::Chunk(chunk) => {
            if let Some(text) = chunk.text() {
                print!("{}", text);
            }
        }
        StreamResult::Done => break,
        StreamResult::Error(e) => return Err(e),
    }
}
}

Message Types

#![allow(unused)]
fn main() {
use giztoy_genx::{Message, Contents, Part, Role};

// User text message
let msg = Message::user_text("Hello!");

// Assistant message with content
let msg = Message {
    role: Role::Assistant,
    name: None,
    payload: Payload::Contents(vec![
        Part::Text("Here's what I found:".to_string()),
    ]),
};

// Tool call
let msg = Message::tool_call(ToolCall {
    id: "call_123".to_string(),
    func_call: FuncCall {
        name: "search".to_string(),
        arguments: r#"{"query":"rust"}"#.to_string(),
    },
});

// Tool result
let msg = Message::tool_result(ToolResult {
    id: "call_123".to_string(),
    result: "Found 10 results".to_string(),
});
}

Provider Adapters

OpenAI

#![allow(unused)]
fn main() {
use giztoy_genx::openai::OpenAIGenerator;

let generator = OpenAIGenerator::new(api_key)
    .with_base_url("https://api.openai.com/v1");
}

Gemini

#![allow(unused)]
fn main() {
use giztoy_genx::gemini::GeminiGenerator;

let generator = GeminiGenerator::new(api_key);
}

Inspection

#![allow(unused)]
fn main() {
use giztoy_genx::{inspect_model_context, inspect_message, inspect_tool};

// Inspect context
println!("{}", inspect_model_context(&ctx));

// Inspect message
println!("{}", inspect_message(&msg));

// Inspect tool
println!("{}", inspect_tool(&tool));
}

Error Types

#![allow(unused)]
fn main() {
use giztoy_genx::{GenxError, State, Status};

match result {
    Err(GenxError::Api { status, message }) => {
        eprintln!("API error: {} - {}", status, message);
    }
    Err(GenxError::Network(e)) => {
        eprintln!("Network error: {}", e);
    }
    Err(GenxError::Json(e)) => {
        eprintln!("JSON error: {}", e);
    }
    _ => {}
}
}

Missing Features (vs Go)

The Rust implementation is missing:

1. Agent Framework: No ReActAgent, MatchAgent
2. Configuration Parser: No YAML/JSON config loading
3. Match Patterns: No intent matching system
4. Tool Variants: No GeneratorTool, HTTPTool, CompositeTool
5. Runtime Interface: No dependency injection system
6. State Management: No memory/state persistence

These would need to be implemented to reach feature parity with Go.

GenX - Known Issues

🔴 Major Issues

GX-001: Rust lacks agent framework

Description:
Rust implementation missing the entire agent framework:

• No ReActAgent
• No MatchAgent
• No tool orchestration
• No configuration parser

Impact: Cannot build autonomous agents in Rust.

Effort: High - requires significant implementation work.

GX-002: Rust lacks advanced tool types

Description:
Rust only has FuncTool. Missing:

• GeneratorTool
• HTTPTool
• CompositeTool
• TextProcessorTool

Impact: Limited tool capabilities in Rust.

🟡 Minor Issues

GX-003: Go agent uses panics for some errors

File: go/pkg/genx/agent/agent.go

Description:
Some internal errors use panic instead of returning errors.

Impact: Can crash applications on unexpected states.

Suggestion: Convert panics in public entry points to errors; keep panics only for truly unreachable states.

GX-004: Configuration parsing is complex

Description:
The agentcfg package has complex unmarshal logic with many edge cases.

Files:

• go/pkg/genx/agentcfg/unmarshal.go
• go/pkg/genx/agentcfg/*_unmarshal_test.go

Note: Extensive tests exist, so this is well-covered.

GX-006: Streaming tool-call collection parity uncertain

Description:
Rust includes collect_tool_calls_streamed, but feature parity with Go streaming tool calls needs verification and tests.

GX-019: MessageChunk.Clone drops tool calls

File: go/pkg/genx/message.go

Description:
MessageChunk.Clone() copies Role/Name/Part but never copies ToolCall.

Impact: Tool-call chunks can be silently lost when cloned.

Suggestion: Copy c.ToolCall instead of checking chk.ToolCall.

GX-020: StreamBuilder drops unknown tool calls

File: go/pkg/genx/stream_builder.go

Description:
If a tool call references a tool not found in ModelContext, the chunk is skipped:

if !ok { slog.Warn(...); continue }

Impact: Tool-call chunks disappear without being forwarded to consumers.

Suggestion: Emit the chunk anyway or return an error so callers can handle missing tools.

GX-021: OpenAI Invoke drops usage metrics

File: go/pkg/genx/openai.go

Description:
invokeJSONOutput and invokeToolCalls return Usage{} instead of resp.Usage.

Impact: Usage accounting is always zero for invoke paths.

Suggestion: Return oaiConvUsage(&resp.Usage) on success.

GX-022: GenerateStream goroutine can leak on early close

File: go/pkg/genx/openai.go

Description:
GenerateStream spawns a goroutine reading the OpenAI stream. If the caller closes the stream early without cancelling the context, the goroutine may continue until the server ends the stream.

Impact: Potential goroutine/resource leak in long-running sessions.

Suggestion: Tie stream close to context cancellation or add a stop channel.

GX-023: hexString ignores rand.Read error

File: go/pkg/genx/json.go

Description:
rand.Read errors are ignored when generating IDs.

Impact: On RNG failure, ID may be all-zero without error signal.

Suggestion: Check rand.Read error and fall back or return error.

🔵 Enhancements

GX-007: Add more provider adapters

Description:
Currently supports OpenAI and Gemini. Could add:

• Anthropic (Claude)
• Mistral
• Local models (Ollama)

GX-008: Add retry logic to generators

Description:
No built-in retry for transient failures.

Suggestion: Add configurable retry with backoff.

GX-009: Add request/response logging

Description:
No debug logging for API calls.

Suggestion: Add optional verbose mode.

GX-010: Document match pattern syntax

Description:
Match patterns have complex syntax; documentation exists but should stay in sync.

Files:

• docs/genx/match/
• go/pkg/genx/match/

GX-011: Add validation for agent configs

Description:
YAML configs could have invalid references ($ref). No validation until runtime.

Suggestion: Add config validation command/function.

GX-012: Add configuration schema generation

Description:
No JSON Schema is provided for agent/tool configuration.

Impact: No IDE auto-complete or static validation.

Suggestion: Generate JSON Schema from agentcfg types and publish under docs/.

GX-013: Add stream test coverage for tool calls (Rust)

Description:
Tool-call streaming helpers exist, but end-to-end tests are limited or missing.

Impact: Potential regressions in streamed tool-call parsing.

Suggestion: Add tests that simulate incremental chunks and verify parsed tool calls.

GX-024: StreamBuilder::new ignores tools (Rust)

File: rust/genx/src/stream.rs

Description:
StreamBuilder::new ignores tools from ModelContext, leaving func_tools empty.

Impact: Tool-call metadata cannot be linked unless callers use with_tools.

Suggestion: Provide a way to downcast tools or pass tool list explicitly in generator code.

⚪ Notes

GX-014: Well-structured Go implementation

Description:
The Go genx package is well-organized:

• Clear separation of concerns
• Extensive test coverage
• Comprehensive agent framework
• YAML/JSON configuration support

GX-015: Event-based agent API

Description:
The agent event system is well-designed:

for {
    evt, err := ag.Next()
    switch evt.Type {
    case EventChunk: ...
    case EventToolStart: ...
    case EventClosed: ...
    }
}

Provides fine-grained control over agent execution.

GX-016: Quit tool pattern

Description:
Tools can be marked as "quit tools" to signal agent completion:

tools:
  - $ref: tool:goodbye
    quit: true

Useful for conversational agents with explicit exit.

GX-017: $ref system for configuration

Description:
Configuration supports references for reuse:

tools:
  - $ref: tool:search  # References registered tool
  - $ref: agent:helper # References registered agent

Good for modular configuration.

GX-018: Multi-skill assistant pattern

Description:
MatchAgent enables router pattern:

Router (Match) → Weather Agent (ReAct)
              → Music Agent (ReAct)
              → Chat Agent

Well-documented in agent/doc.go.

GX-025: Language idioms differ (Go vs Rust)

Description:
Go uses iter.Seq, Rust uses iterators/streams. This is idiomatic for each language and not a functional issue.

Summary

Overall: Go implementation is mature and feature-rich with comprehensive agent framework. Rust implementation provides basic LLM abstraction but lacks the agent framework, making it suitable only for simple use cases. Major effort needed to reach Rust feature parity.

speech

Overview

The speech module defines interfaces for voice and speech processing. It separates pure audio streams (Voice) from speech streams that include transcriptions (Speech). It also provides multiplexers for ASR (speech-to-text) and TTS (text-to-speech) implementations.

Design Goals

• Unified interfaces for ASR/TTS backends
• Stream-first APIs for long-running audio
• Clear separation between audio-only and audio+text
• Pluggable providers via multiplexer registration

Key Concepts

• Voice: audio-only stream of PCM segments
• Speech: audio stream with text transcription per segment
• ASR: Opus input -> Speech/SpeechStream
• TTS: text input -> Speech
• Sentence segmentation: split long text into manageable chunks

Components

• Voice/Speech interfaces
• ASR/TTS muxers
• Sentence segmentation utilities
• Speech collection and copy helpers

Related Modules

• docs/lib/audio/pcm for PCM formats
• docs/lib/audio/opusrt for Opus streaming input
• Provider SDKs in docs/lib/minimax, docs/lib/doubaospeech

speech (Go)

Package Layout

• voice.go: Voice/VoiceSegment interfaces
• speech.go: Speech/SpeechSegment interfaces
• asr.go: ASR multiplexer and interfaces
• tts.go: TTS multiplexer and interfaces
• segment.go: default sentence segmentation
• util.go: collectors and copy helpers
• Provider implementations: asr_doubao_sauc.go, tts_doubao_v1.go, tts_doubao_v2.go, tts_minimax.go

Public Interfaces

• Voice: Voice, VoiceSegment, VoiceStream
• Speech: Speech, SpeechSegment, SpeechStream
• ASR: StreamTranscriber, Transcriber, ASR mux + helpers
• TTS: Synthesizer, TTS mux + helpers
• Segmentation: SentenceSegmenter, SentenceIterator

Design Notes

• Global muxes ASRMux and TTSMux provide default routing.
• ASR uses Opus frame streams (opusrt.FrameReader).
• DefaultSentenceSegmenter splits by punctuation with a rune cap.
• CollectSpeech and CopySpeech help aggregate or export streams.

Usage Notes

• Transcribe falls back to streaming when the backend does not implement the Transcriber interface.
• Revoice streams existing speech into a TTS backend via a pipe.

speech (Rust)

Crate Layout

• voice.rs: Voice/VoiceSegment interfaces
• speech.rs: Speech/SpeechSegment interfaces
• asr.rs: ASR multiplexer and traits
• tts.rs: TTS multiplexer and traits
• segment.rs: sentence segmentation utilities
• util.rs: speech collector and iterator helpers

Public Interfaces

• Voice: Voice, VoiceSegment, VoiceStream
• Speech: Speech, SpeechSegment, SpeechStream
• ASR: StreamTranscriber, Transcriber (async), ASR
• TTS: Synthesizer, TTS (async)
• Segmentation: SentenceSegmenter, SentenceIterator

Design Notes

• Async traits are used across ASR/TTS and stream interfaces.
• ASR and TTS use a trie-based mux with async read/write locks.
• SpeechCollector composes a SpeechStream into a single Speech.

Differences vs Go

• No global mux singletons; callers construct ASR/TTS explicitly.
• Async AsyncRead is used for text input in TTS.

speech - Known Issues

🟡 Minor Issues

SPT-001: Go Revoice has no cancellation propagation

File: go/pkg/speech/tts.go

Description: Revoice spawns a goroutine that copies the entire input speech into an io.Pipe, but the goroutine is not tied to the caller's context. If the synthesizer returns early or the context is canceled, the copy goroutine may continue doing work until completion.

Impact: Wasted CPU or lingering goroutines on early cancellation.

Suggestion: Honor context cancellation inside the copy loop or use a pipe that is closed when ctx.Done() fires.

SPT-002: Rust ASR ignores full-transcribe implementations

File: rust/speech/src/asr.rs

Description: ASR::transcribe always falls back to the streaming path, even though a Transcriber trait exists for full transcription.

Impact: Backends that can provide a more efficient full-transcribe path cannot use it.

Suggestion: Detect and use Transcriber implementations when available.

chatgear

Overview

chatgear defines the core protocol types for device-to-server communication: commands, state events, statistics, and audio streaming metadata. It focuses on interface design rather than transport implementation, and provides an in-process pipe for testing.

Design Goals

• Stable, typed protocol for device state and control
• Clear separation between uplink (device -> server) and downlink (server -> device)
• Explicit metadata for timestamps and command issuance
• Audio streaming with Opus frame stamping
• Support both Go and Rust with comparable API surfaces

Key Concepts

• Session commands: device control commands with typed payloads
• State events: gear state transitions with causes
• Stats events: telemetry snapshots and incremental changes
• Uplink/Downlink: split interfaces for bidirectional streams
• Ports: higher-level client/server port abstraction

Submodules

• transport: uplink/downlink connection traits and pipe helpers
• port: client/server port traits and audio track controls

External Reference

• /Users/idy/Work/haivivi/x/docs/chatgear (original protocol/design notes)

Related Modules

• docs/lib/audio/opusrt for Opus frame handling
• docs/lib/jsontime for millisecond timestamps

chatgear/transport

Overview

The transport layer defines bidirectional streaming interfaces for chatgear. It splits data flow into uplink (device -> server) and downlink (server -> device) and provides a test-friendly in-process pipe.

Design Goals

• Separate uplink/downlink responsibilities
• Provide a minimal interface that can be implemented by different transports
• Keep Opus framing metadata explicit

Key Concepts

• UplinkTx / UplinkRx: device -> server
• DownlinkTx / DownlinkRx: server -> device
• Stamped Opus frames: carry timestamp for playback alignment
• Pipe connection for in-process testing

chatgear/port

Overview

Port interfaces represent higher-level client/server roles built on top of the transport layer. They combine audio streaming, state/stats telemetry, and command control into a single abstraction.

Design Goals

• Provide a symmetric client/server API surface
• Hide transport details while preserving real-time audio controls
• Expose device control commands alongside audio output

Key Concepts

• ClientPort: device-side send/receive split (Tx/Rx)
• ServerPort: server-side send/receive split (Tx/Rx)
• Audio tracks: background/foreground/overlay output streams
• Device commands: volume, brightness, WiFi, OTA, power, etc.

chatgear (Go)

Package Layout

• state.go: gear state enum, state events
• stats.go: telemetry structs, merge logic
• command.go: session command types and JSON mapping
• conn.go: uplink/downlink interfaces
• port.go: client/server port interfaces
• conn_pipe.go: in-process pipe connection for tests

Public Interfaces

• State: GearState, GearStateEvent, GearStateChangeCause
• Stats: GearStatsEvent, GearStatsChanges and related structs
• Commands: SessionCommand with SessionCommandEvent
• Uplink/Downlink: UplinkTx, UplinkRx, DownlinkTx, DownlinkRx
• Ports: ClientPortTx/Rx, ServerPortTx/Rx
• Pipe: NewPipe for test or in-process wiring

Design Notes

• JSON encoding is typed via commandType() and a tagged event wrapper.
• All time fields use jsontime.Milli for millisecond epoch values.
• Opus frames are stamped with opusrt.EpochMillis during transport.
• Stats merge logic performs partial updates with GearStatsChanges.

Usage Notes

• UplinkRx/DownlinkRx expose iterators (iter.Seq2) instead of channels.
• ServerPortTx exposes track creation for background/foreground/overlay audio.

chatgear (Rust)

Crate Layout

• state.rs: gear state enum and events
• stats.rs: telemetry structs and merge logic
• command.rs: session commands and JSON helpers
• conn.rs: uplink/downlink async traits
• port.rs: client/server port traits
• conn_pipe.rs: in-process pipe helper

Public Interfaces

• State: GearState, GearStateEvent, GearStateChangeCause
• Stats: GearStatsEvent, GearStatsChanges
• Commands: SessionCommand, SessionCommandEvent, Command enum
• Uplink/Downlink: UplinkTx, UplinkRx, DownlinkTx, DownlinkRx
• Ports: ClientPortTx/Rx, ServerPortTx/Rx
• Pipe: new_pipe

Design Notes

• Async traits are used for most IO-facing APIs.
• Commands serialize into JSON value payloads; a typed Command enum can parse from (type, payload) pairs.
• Port traits split audio track control and device command APIs.

Differences vs Go

• Rust favors async traits and owned Vec<u8> payloads.
• Command event uses serde_json::Value instead of typed interface payloads.

chatgear - Known Issues

🟡 Minor Issues

CG-001: Go ReadNFCTag equality ignores tag data changes

File: go/pkg/chatgear/stats.go

Description: ReadNFCTag.Equal compares only tag UIDs. If a tag's payload or metadata changes but the UID remains the same, the merge logic will treat it as unchanged.

Impact: Telemetry updates can be silently skipped.

Suggestion: Include additional fields (e.g., RawData, DataFormat, UpdateAt) in equality or document UID-only matching as a deliberate choice.

CG-002: Rust SessionCommandEvent swallows serialization errors

File: rust/chatgear/src/command.rs

Description: SessionCommandEvent::new uses serde_json::to_value(cmd) and replaces errors with Value::Null, losing the original error context.

Impact: Serialization failures are silently ignored, making debugging difficult.

Suggestion: Return a Result or log/report serialization failures explicitly.

CG-003: Go Pipe connections can block indefinitely on backpressure

File: go/pkg/chatgear/conn_pipe.go

Description: NewPipe uses bounded channels. If the receiver stops reading, senders will block in SendOpusFrames / SendState / SendStats without a timeout unless the caller provides a cancellable context.

Impact: Potential goroutine leaks in tests or in-process usage.

Suggestion: Document this behavior and recommend context timeouts for pipe usage.

Audio Package

Audio processing framework for speech and multimedia applications.

Design Goals

1. Real-time Processing: Low-latency audio mixing, encoding, and streaming
2. Format Flexibility: Support common audio formats (PCM, Opus, MP3, OGG)
3. Cross-platform: FFI bindings to native libraries (libopus, libsoxr, lame)
4. Streaming-first: Designed for continuous audio streams, not just files

Architecture

graph TB
    subgraph audio["audio/"]
        subgraph row1[" "]
            pcm["pcm/<br/>- Format<br/>- Chunk<br/>- Mixer"]
            codec["codec/<br/>- opus/<br/>- mp3/<br/>- ogg/"]
            resampler["resampler/<br/>- soxr<br/>- Format<br/>- Convert"]
        end
        subgraph row2[" "]
            opusrt["opusrt/<br/>- Buffer<br/>- Realtime<br/>- OGG R/W"]
            songs["songs/<br/>- Catalog<br/>- Notes<br/>- PCM gen"]
            portaudio["portaudio/<br/>(Go only)<br/>- Stream<br/>- Device"]
        end
    end

Submodules

Audio Formats

PCM Formats (Predefined)

Codec Support

Common Workflows

Voice Chat (Low Latency)

flowchart LR
    A[Microphone] --> B[PCM 16kHz]
    B --> C[Opus Encode]
    C --> D[Network]
    D --> E[Opus Decode]
    E --> F[Mixer]
    F --> G[Speaker]

Speech Synthesis Playback

flowchart LR
    A[API Response<br/>Base64 MP3] --> B[MP3 Decode]
    B --> C[Resample<br/>24K→16K]
    C --> D[Mixer]
    D --> E[Speaker]

Audio Recording

flowchart LR
    A[PCM Stream] --> B[Opus Encode]
    B --> C[OGG Writer]
    C --> D[File]

Native Dependencies

Examples Directory

• examples/go/audio/ - Go audio examples
• examples/rust/audio/ - Rust audio examples

Related Packages

• buffer - Used for audio data buffering
• speech - High-level speech synthesis/recognition
• minimax, doubaospeech - TTS/ASR APIs returning audio

Audio Codec Module

Audio encoding and decoding for Opus, MP3, and OGG formats.

Design Goals

1. Native Performance: FFI bindings to proven C libraries
2. Streaming Support: Process audio in chunks, not full files
3. VoIP Optimized: Low-latency Opus encoding for voice chat

Codec Support Matrix

Sub-modules

opus/

Opus codec implementation for voice and audio.

Features:

• Encoder with VoIP/Audio/LowDelay modes
• Decoder with PLC (Packet Loss Concealment)
• TOC (Table of Contents) parsing
• Frame duration detection

Key Types:

• Encoder, Decoder
• Frame, TOC, FrameDuration

mp3/

MP3 codec for compatibility with legacy systems.

Features:

• LAME-based encoding with quality presets
• minimp3-based decoding (header-only library)

Key Types:

• Encoder, Decoder

ogg/

OGG container format for packaging Opus/Vorbis streams.

Features:

• Page-based streaming
• Bitstream management
• Synchronization recovery

Key Types:

• Encoder, Stream, Sync, Page

Opus Frame Durations

Recommended: 20ms frames balance latency and compression.

Common Opus Bitrates

Native Library Versions

Examples

See parent audio/ documentation for usage examples.

Related Modules

• opusrt/ - Realtime Opus streaming with OGG container
• resampler/ - Sample rate conversion before encoding

Audio PCM Module

PCM (Pulse Code Modulation) audio format handling, chunks, and multi-track mixing.

Design Goals

1. Standard Formats: Predefined configurations for common use cases
2. Chunk Abstraction: Unified interface for audio data and silence
3. Real-time Mixing: Low-latency multi-track audio mixing with gain control
4. Streaming Interface: Compatible with io.Reader/io.Writer patterns

Predefined Formats

Duration/Bytes Calculations

For L16Mono16K (16kHz, 16-bit mono):

Formula: bytes = samples × channels × (bit_depth / 8)

Chunk Types

DataChunk

Raw audio data with format metadata.

classDiagram
    class DataChunk {
        Data: []byte
        Format: Format
    }

SilenceChunk

Generates silence (zeros) of specified duration without allocating.

classDiagram
    class SilenceChunk {
        Duration: time
        Format: Format
    }

Mixer Architecture

Multi-track audio mixer with real-time mixing and gain control:

flowchart LR
    subgraph mixer["Mixer"]
        t1["Track 1<br/>(gain=1)"]
        t2["Track 2<br/>(gain=0.5)"]
        tn["Track N"]
        mix["Mix Buffer<br/>(float32)"]
        out["Output<br/>(int16)"]
        
        t1 --> mix
        t2 --> mix
        tn --> mix
        mix --> out
    end

Features:

• Dynamic track creation/removal
• Per-track gain control (0.0 - 1.0+)
• Silence gap detection
• Auto-close when all tracks done
• Thread-safe operations

Mixing Algorithm

1. Convert int16 PCM to float32 (-1.0 to 1.0)
2. Apply per-track gain
3. Sum all track samples
4. Clip to [-1.0, 1.0]
5. Convert back to int16

output = clip(Σ(track[i] × gain[i]), -1.0, 1.0) × 32767

Use Cases

Voice Chat Mixing

Multiple participants' audio mixed into single output stream.

Background Music

Mix background music track with voice at lower gain.

Audio Ducking

Reduce music volume when voice is detected.

Examples

See parent audio/ documentation for usage examples.

Related Modules

• audio/codec/ - Encode/decode before/after mixing
• audio/resampler/ - Convert sample rates before mixing
• buffer/ - Buffer audio data between processing stages

Audio Resampler Module

Sample rate conversion using libsoxr (SoX Resampler Library).

Design Goals

1. High Quality: Professional-grade resampling via libsoxr
2. Streaming: Process audio as continuous stream, not files
3. Channel Conversion: Support mono↔stereo conversion
4. io.Reader Interface: Drop-in replacement for audio sources

Supported Conversions

Sample Rate

Any integer sample rate to any other integer sample rate:

• 8000 Hz ↔ 16000 Hz ↔ 24000 Hz ↔ 48000 Hz
• Non-standard rates supported

Channel Conversion

Quality Levels

libsoxr supports multiple quality presets:

Note: Current implementation uses High quality by default.

Algorithm

libsoxr uses polyphase filter banks with configurable:

• Passband rolloff
• Stop-band attenuation
• Linear/minimum phase

The High quality preset provides:

• Passband: 0-0.91 Nyquist
• Stop-band attenuation: -100 dB
• Linear phase

Common Resampling Scenarios

Speech API to Local Playback

flowchart LR
    A["API Output<br/>(24kHz)"] --> B[Resample]
    B --> C["Mixer Input<br/>(16kHz)"]

Local Capture to Speech API

flowchart LR
    A["Microphone<br/>(48kHz)"] --> B[Resample]
    B --> C["API Input<br/>(16kHz)"]

Multi-source Mixing

flowchart LR
    s1["Source 1<br/>(24kHz)"] --> r1[Resample]
    r1 --> m1["16kHz"]
    
    s2["Source 2<br/>(16kHz)"] --> m2["16kHz"]
    
    s3["Source 3<br/>(48kHz)"] --> r3[Resample]
    r3 --> m3["16kHz"]
    
    m1 --> mixer[Mixer]
    m2 --> mixer
    m3 --> mixer

Performance Characteristics

Approximate cycles per sample (on modern CPU):

• Quick: ~10
• High: ~50-100
• Very High: ~200-500

For real-time 16kHz mono:

• 16000 samples/sec × 100 cycles ≈ 1.6M cycles/sec
• Negligible CPU load on modern hardware

Memory Usage

libsoxr maintains internal buffers for filter state:

• ~10-50KB per resampler instance
• More for higher quality settings

Examples

See parent audio/ documentation for usage examples.

Related Modules

• audio/pcm/ - Format definitions, use with resampler
• audio/codec/ - Often resample before/after encoding

Audio OpusRT Module

Real-time Opus stream processing with jitter buffering and packet loss handling.

Design Goals

1. Out-of-Order Handling: Reorder packets that arrive out of sequence
2. Packet Loss Detection: Detect and report gaps for PLC (Packet Loss Concealment)
3. Real-time Simulation: Playback timing based on timestamps, not arrival time
4. OGG Container Support: Read/write Opus in OGG format (Go only)

Core Concepts

Jitter Buffer

Network packets may arrive out of order or with variable delay (jitter). The jitter buffer collects packets and outputs them in correct order:

sequenceDiagram
    participant N as Network
    participant B as Buffer (Heap)
    participant O as Output
    
    N->>B: PKT 3
    Note over B: [3]
    N->>B: PKT 1
    Note over B: [1, 3]
    B->>O: PKT 1
    N->>B: PKT 2
    Note over B: [2, 3]
    B->>O: PKT 2
    B->>O: PKT 3

Packet Loss Detection

Gaps between consecutive frame timestamps indicate lost packets:

Frame 1: 0ms - 20ms
Frame 2: 20ms - 40ms    ✓ No gap
Frame 4: 60ms - 80ms    ✗ 20ms gap (Frame 3 lost)

When loss is detected, the caller should use decoder PLC:

frame, loss, _ := buffer.Frame()
if loss > 0 {
    // Generate PLC audio for 'loss' duration
    plcAudio := decoder.DecodePLC(...)
}

Timestamped Frames

Frames are timestamped with epoch milliseconds:

classDiagram
    class StampedFrame {
        Timestamp: int64 (8 bytes, big-endian)
        OpusFrame: []byte (variable)
    }

Components

Buffer

Simple jitter buffer with min-heap ordering:

• Append frames in any order
• Read frames in timestamp order
• Max duration limit (oldest dropped)

RealtimeBuffer

Wraps Buffer for real-time playback simulation:

• Background goroutine pulls frames at correct time
• Generates loss events when data not available
• Handles clock synchronization

OGG Reader/Writer (Go only)

Read/write Opus streams in OGG container format:

• OggReader: Read Opus frames from OGG file
• OggWriter: Write Opus frames to OGG container

Timing

EpochMillis

All timestamps are milliseconds since Unix epoch:

type EpochMillis int64

// Convert from time.Time
stamp := EpochMillis(time.Now().UnixMilli())

// Convert to duration
duration := stamp.Duration() // time.Duration

Timestamp Epsilon

A 2ms tolerance for timestamp comparisons accounts for clock drift:

const timestampEpsilon = 2 // milliseconds

Use Cases

WebRTC Audio

flowchart LR
    A[WebRTC] --> B[RTP Packets]
    B --> C[Jitter Buffer]
    C --> D[Opus Decode]
    D --> E[Playback]

Speech API Streaming

flowchart LR
    A[API Response] --> B[Stamped Frames]
    B --> C[RealtimeBuffer]
    C --> D[Decode]
    D --> E[Mixer]

Audio Recording

flowchart LR
    A[Opus Frames] --> B[OGG Writer]
    B --> C[File]

Examples

See parent audio/ documentation for usage examples.

Related Modules

• audio/codec/opus/ - Opus encoder/decoder
• audio/codec/ogg/ - OGG container primitives
• buffer/ - Used internally by RealtimeBuffer

Audio Package - Go Implementation

Import: github.com/haivivi/giztoy/pkg/audio

📚 Go Documentation

The main audio package is an umbrella for sub-packages. Import specific packages directly.

Sub-packages

pcm (PCM Audio)

import "github.com/haivivi/giztoy/pkg/audio/pcm"

Key Types:

codec/opus (Opus Codec)

import "github.com/haivivi/giztoy/pkg/audio/codec/opus"

Key Types:

codec/mp3 (MP3 Codec)

import "github.com/haivivi/giztoy/pkg/audio/codec/mp3"

Key Types:

codec/ogg (OGG Container)

import "github.com/haivivi/giztoy/pkg/audio/codec/ogg"

Key Types:

resampler (Sample Rate Conversion)

import "github.com/haivivi/giztoy/pkg/audio/resampler"

Key Types:

opusrt (Realtime Opus)

import "github.com/haivivi/giztoy/pkg/audio/opusrt"

Key Types:

portaudio (Audio I/O)

import "github.com/haivivi/giztoy/pkg/audio/portaudio"

Key Types:

songs (Built-in Melodies)

import "github.com/haivivi/giztoy/pkg/audio/songs"

Key Types:

Usage Examples

PCM Mixer

import "github.com/haivivi/giztoy/pkg/audio/pcm"

// Create mixer
mixer := pcm.NewMixer(pcm.L16Mono16K, pcm.WithAutoClose())

// Create track
track, ctrl, _ := mixer.CreateTrack(pcm.WithTrackLabel("voice"))

// Write audio to track
track.Write(audioData)

// Adjust gain
ctrl.SetGain(0.8)

// Read mixed output
buf := make([]byte, 1600) // 50ms at 16kHz
mixer.Read(buf)

Opus Encoding

import "github.com/haivivi/giztoy/pkg/audio/codec/opus"

// Create encoder
enc, _ := opus.NewVoIPEncoder(16000, 1)
defer enc.Close()

enc.SetBitrate(24000)

// Encode PCM to Opus
pcmData := make([]int16, 320) // 20ms at 16kHz
frame, _ := enc.Encode(pcmData, 320)

Sample Rate Conversion

import "github.com/haivivi/giztoy/pkg/audio/resampler"

srcFmt := resampler.Format{SampleRate: 24000, Stereo: false}
dstFmt := resampler.Format{SampleRate: 16000, Stereo: false}

rs, _ := resampler.New(audioReader, srcFmt, dstFmt)
defer rs.Close()

// Read resampled data
io.Copy(output, rs)

Realtime Opus Buffer

import "github.com/haivivi/giztoy/pkg/audio/opusrt"

// Create jitter buffer (2 minute capacity)
buf := opusrt.NewBuffer(2 * time.Minute)

// Write stamped frames (can arrive out of order)
buf.Write(stampedFrameData)

// Read in order
frame, loss, _ := buf.Frame()
if loss > 0 {
    // Use decoder PLC for lost frames
}

CGO Dependencies

All codec packages use CGO to bind native libraries:

// Example: opus encoder
/*
#cgo pkg-config: opus
#include <opus.h>
*/
import "C"

Build requirements:

• pkg-config for native builds
• Bazel cdeps for Bazel builds

Audio Package - Rust Implementation

Crate: giztoy-audio

📚 Rust Documentation

Modules

pcm (PCM Audio)

#![allow(unused)]
fn main() {
use giztoy_audio::pcm::{Format, FormatExt, Chunk, DataChunk, SilenceChunk, Mixer};
}

Key Types:

codec::opus (Opus Codec)

#![allow(unused)]
fn main() {
use giztoy_audio::codec::opus::{Encoder, Decoder, Application, Frame, TOC};
}

Key Types:

codec::mp3 (MP3 Codec)

#![allow(unused)]
fn main() {
use giztoy_audio::codec::mp3::{Encoder, Decoder};
}

Key Types:

codec::ogg (OGG Container)

#![allow(unused)]
fn main() {
use giztoy_audio::codec::ogg::{Encoder, Stream, Sync, Page};
}

Key Types:

resampler (Sample Rate Conversion)

#![allow(unused)]
fn main() {
use giztoy_audio::resampler::{Soxr, Format};
}

Key Types:

opusrt (Realtime Opus)

#![allow(unused)]
fn main() {
use giztoy_audio::opusrt::{Buffer, StampedFrame, EpochMillis};
}

Key Types:

⚠️ Note: Rust opusrt is missing OGG Reader/Writer compared to Go.

songs (Built-in Melodies)

#![allow(unused)]
fn main() {
use giztoy_audio::songs::{Song, Note, Catalog};
}

Key Types:

Usage Examples

PCM Format

#![allow(unused)]
fn main() {
use giztoy_audio::pcm::{Format, FormatExt};
use std::time::Duration;

let format = Format::L16Mono16K;

// Calculate bytes for duration
let bytes = format.bytes_in_duration(Duration::from_millis(100));
assert_eq!(bytes, 3200); // 1600 samples * 2 bytes

// Create chunks
let silence = format.silence_chunk(Duration::from_millis(100));
let data = format.data_chunk(vec![0u8; 3200]);
}

Opus Encoding

#![allow(unused)]
fn main() {
use giztoy_audio::codec::opus::{Encoder, Application};

let mut encoder = Encoder::new(16000, 1, Application::VoIP)?;
encoder.set_bitrate(24000)?;

// Encode PCM to Opus
let pcm: Vec<i16> = vec![0; 320]; // 20ms at 16kHz
let frame = encoder.encode(&pcm, 320)?;
}

Sample Rate Conversion

#![allow(unused)]
fn main() {
use giztoy_audio::resampler::{Soxr, Format};

let src_fmt = Format { sample_rate: 24000, stereo: false };
let dst_fmt = Format { sample_rate: 16000, stereo: false };

let mut resampler = Soxr::new(src_fmt, dst_fmt)?;

// Process audio data
let output = resampler.process(&input_pcm)?;
}

Mixer

#![allow(unused)]
fn main() {
use giztoy_audio::pcm::{Format, Mixer, MixerOptions};

let mut mixer = Mixer::new(Format::L16Mono16K, MixerOptions::default());

// Create track
let (track, ctrl) = mixer.create_track(None)?;

// Write audio
track.write(&audio_data)?;

// Adjust gain
ctrl.set_gain(0.8);

// Read mixed output
let mut buf = vec![0u8; 3200];
mixer.read(&mut buf)?;
}

FFI Bindings

Rust uses custom FFI modules for native library bindings:

#![allow(unused)]
fn main() {
// Example: codec/opus/ffi.rs
extern "C" {
    fn opus_encoder_create(
        fs: i32,
        channels: i32,
        application: i32,
        error: *mut i32,
    ) -> *mut OpusEncoder;
}
}

Differences from Go

Audio Package - Known Issues

🟠 Major Issues

AUD-001: Go Mixer uses unsafe pointer casting

File: go/pkg/audio/pcm/mixer.go:226

Description:
The mixer uses unsafe.Slice and unsafe.Pointer to cast between []byte and []int16:

i16 := unsafe.Slice((*int16)(unsafe.Pointer(&p[0])), len(p)/2)

Risk:

• Platform-dependent endianness (assumes little-endian)
• Potential undefined behavior if buffer alignment is wrong

Impact: May produce incorrect audio on big-endian systems.

Suggestion: Add explicit little-endian encoding/decoding or document platform requirements.

AUD-002: Rust opusrt missing OGG Reader/Writer

Description:
Go opusrt has OggReader and OggWriter for reading/writing Opus in OGG containers. Rust implementation is missing these.

Impact: Cannot read/write Opus files in OGG format in Rust.

Status: ⚠️ Partial implementation.

AUD-003: Rust missing portaudio module

Description:
Go has audio/portaudio for audio device I/O. Rust has no equivalent.

Impact: Cannot capture/play audio from hardware devices in Rust.

Status: ❌ Not implemented.

🟡 Minor Issues

AUD-004: Go Format panics on invalid value

File: go/pkg/audio/pcm/pcm.go:36-38

Description:
Format.SampleRate(), Channels(), Depth() all panic on invalid format:

func (f Format) SampleRate() int {
    switch f {
    // ...
    }
    panic("pcm: invalid audio type")
}

Impact: Runtime panic instead of error return.

Suggestion: Return (int, error) or use MustXxx naming convention for panicking versions.

AUD-005: Go SilenceChunk uses fixed global buffer

File: go/pkg/audio/pcm/pcm.go:177

Description:
Uses a shared fixed-size zero buffer and loops for long durations.

Impact: None functionally; avoids repeated allocations.

Status: Not a bug. Keep as implementation note only.

AUD-006: Go Opus encoder max frame size hardcoded

File: go/pkg/audio/codec/opus/encoder.go:95

Description:
Encode function allocates fixed 4000 byte buffer:

buf := make([]byte, 4000)

Impact: Allocation on every encode call.

Suggestion: Use buffer pool or allow caller to provide buffer.

AUD-007: Rust Format re-export from resampler is confusing

File: rust/audio/src/pcm/format.rs:7

Description:
pcm::Format is actually re-exported from resampler::Format:

#![allow(unused)]
fn main() {
pub use crate::resampler::format::Format;
}

Impact: Confusing import paths, circular dependency appearance.

Suggestion: Define Format once at top level and import in both modules.

AUD-008: Go mixer notifyWrite spawns goroutine every call

File: go/pkg/audio/pcm/mixer.go:391-405

Description:
notifyWrite() spawns a new goroutine each time:

func (mx *Mixer) notifyWrite() {
    go func() {
        // ...
    }()
}

Impact: Goroutine overhead for every write notification.

Suggestion: Use single dedicated notification goroutine or avoid goroutine.

🔵 Enhancements

AUD-009: No stereo format support in predefined formats

Description:
Only mono formats are predefined (L16Mono16K, etc.). No stereo formats.

Suggestion: Add L16Stereo16K, L16Stereo24K, L16Stereo48K.

AUD-010: No 8-bit or 24-bit PCM support

Description:
Only 16-bit PCM is supported. Some audio sources use 8-bit (low quality) or 24-bit (high quality).

Suggestion: Add format variants for different bit depths.

AUD-011: Resampler quality not configurable

File: go/pkg/audio/resampler/soxr.go:52

Description:
Quality is hardcoded to SOXR_HQ:

qSpec := C.soxr_quality_spec(C.SOXR_HQ, 0)

Impact: Cannot trade quality for performance when needed.

Suggestion: Add quality parameter to New().

AUD-012: No WAV file support

Description:
No utilities for reading/writing WAV files, only raw PCM.

Suggestion: Add WAV header parsing/writing for file I/O.

⚪ Notes

AUD-013: CGO/FFI dependency complexity

Description:
Both Go and Rust rely heavily on CGO/FFI for native codec libraries. This adds:

• Build complexity (pkg-config, Bazel rules)
• Platform-specific issues
• Memory management concerns

Status: Necessary for performance, but increases maintenance burden.

AUD-014: Mixer uses float32 internally

Description:
Mixer converts int16 PCM to float32 for mixing, then back to int16:

// int16 → float32
s := float32(trackI16[i])
// ... mix ...
// float32 → int16
i16[i] = int16(t * 32767)

Impact: Slight precision loss during mixing, but standard practice.

Summary

Overall: Functional audio processing with significant native library integration. Main gaps are Rust parity (opusrt OGG, portaudio) and some unsafe code patterns.

MiniMax SDK

Go and Rust SDK for the MiniMax AI platform API.

Official API Documentation: api/README.md

Design Goals

1. Full API Coverage: Support all MiniMax API capabilities
2. Idiomatic Language Design: Natural Go/Rust patterns
3. Streaming Support: First-class support for streaming responses
4. Async Task Handling: Convenient polling for long-running operations

API Coverage

Architecture

graph TB
    subgraph client["Client"]
        subgraph services1[" "]
            text[Text Service]
            speech[Speech Service]
            voice[Voice Service]
            video[Video Service]
        end
        subgraph services2[" "]
            image[Image Service]
            music[Music Service]
            file[File Service]
        end
    end
    
    subgraph http["HTTP Client"]
        retry[Retry]
        auth[Auth]
        error[Error Handling]
    end
    
    client --> http
    http --> api["https://api.minimaxi.com"]

Services

Authentication

Uses Bearer token authentication:

Authorization: Bearer <api_key>

API keys are obtained from MiniMax Platform.

Base URLs

Response Patterns

Synchronous

Direct response with data.

Streaming

SSE (Server-Sent Events) for real-time data:

• Text: Token-by-token chat responses
• Speech: Audio chunk streaming

Async Tasks

For long-running operations (video, async speech):

sequenceDiagram
    participant C as Client
    participant S as Server
    C->>S: Create Task
    S-->>C: task_id
    loop Poll Status
        C->>S: Query Status
        S-->>C: Pending/Running
    end
    S-->>C: Success + Result

Error Handling

All errors include:

• status_code: Numeric error code
• status_msg: Human-readable message

Common error codes:

• 1000: General error
• 1001: Rate limit exceeded
• 1002: Invalid parameters
• 1004: Authentication failed

Examples Directory

• examples/go/minimax/ - Go SDK examples
• examples/rust/minimax/ - Rust SDK examples
• examples/cmd/minimax/ - CLI test scripts

Related

• CLI tool: go/cmd/minimax/
• CLI tests: examples/cmd/minimax/

MiniMax 开放平台 API 文档

官方文档: MiniMax 开放平台文档中心

最后更新: 2026-01-19

注意: 本文档基于官方文档整理，如有更新请参考官方文档

官方文档导航

如果本文档信息不完整或需要最新信息，请访问以下官方链接：

如何获取最新文档

方法一：直接访问官网

访问 MiniMax 开放平台文档中心，左侧导航栏包含所有 API 接口的详细文档。

方法二：使用 AI 工具读取

如果使用支持浏览器功能的 AI 工具（如 Cursor），可以：

1. 使用 browser_navigate 工具访问官方文档页面
2. 使用 browser_snapshot 获取页面内容
3. 解析页面中的 API 参数、请求/响应格式等信息

示例：

访问: https://platform.minimaxi.com/docs/api-reference/speech-t2a-http

方法三：查看官方 MCP 服务器

MiniMax 提供了官方的 MCP（Model Context Protocol）服务器实现，包含完整的 API 调用示例：

• Python 版本: https://github.com/MiniMax-AI/MiniMax-MCP
• JavaScript 版本: https://github.com/MiniMax-AI/MiniMax-MCP-JS

关于 OpenAPI/Swagger

MiniMax 目前没有公开提供 OpenAPI/Swagger 规范文件。如需获取，可以：

• 联系官方技术支持: api-support@minimaxi.com
• 基于官方文档手动整理

概述

MiniMax 开放平台提供多模态 AI 能力，包括文本生成、语音合成、视频生成、图像生成、音乐生成等。

API 能力概览

认证方式

所有 API 使用 Bearer Token 认证：

Authorization: Bearer <your_api_key>

获取 API Key

1. 按量付费: 在「账户管理 > 接口密钥」中创建 API Key，支持所有模态模型
2. Coding Plan: 创建 Coding Plan Key，仅支持文本模型

基础 URL

请求头

错误处理

API 返回的错误响应格式：

{
  "base_resp": {
    "status_code": 1000,
    "status_msg": "error message"
  }
}

官方资源

• MiniMax 开放平台
• 官方 MCP 服务器 (Python)
• 官方 MCP 服务器 (JavaScript)
• 技术支持邮箱: api-support@minimaxi.com

MiniMax SDK - Go Implementation

Import: github.com/haivivi/giztoy/pkg/minimax

📚 Go Documentation

Client

type Client struct {
    Text   *TextService
    Speech *SpeechService
    Voice  *VoiceService
    Video  *VideoService
    Image  *ImageService
    Music  *MusicService
    File   *FileService
}

Constructor:

// Basic
client := minimax.NewClient("api-key")

// With options
client := minimax.NewClient("api-key",
    minimax.WithBaseURL(minimax.BaseURLGlobal),
    minimax.WithRetry(5),
    minimax.WithHTTPClient(&http.Client{Timeout: 60*time.Second}),
)

Options:

Services

TextService

// Synchronous
resp, err := client.Text.CreateChatCompletion(ctx, &minimax.ChatCompletionRequest{
    Model: "MiniMax-M2.1",
    Messages: []minimax.Message{
        {Role: "user", Content: "Hello!"},
    },
})

// Streaming (Go 1.23+ iter.Seq2)
for chunk, err := range client.Text.CreateChatCompletionStream(ctx, req) {
    if err != nil {
        return err
    }
    fmt.Print(chunk.Choices[0].Delta.Content)
}

SpeechService

// Synchronous
resp, err := client.Speech.Synthesize(ctx, &minimax.SpeechRequest{
    Model: "speech-2.6-hd",
    Text:  "Hello, world!",
    VoiceSetting: &minimax.VoiceSetting{
        VoiceID: "male-qn-qingse",
    },
})
// resp.Audio contains decoded audio bytes

// Streaming
for chunk, err := range client.Speech.SynthesizeStream(ctx, req) {
    if err != nil {
        return err
    }
    buf.Write(chunk.Audio)
}

// Async (long text)
task, err := client.Speech.CreateAsyncTask(ctx, &minimax.AsyncSpeechRequest{
    Model: "speech-2.6-hd",
    Text:  longText,
    // ...
})
result, err := task.Wait(ctx)

VoiceService

// List voices
voices, err := client.Voice.List(ctx)

// Clone voice
resp, err := client.Voice.Clone(ctx, &minimax.VoiceCloneRequest{
    FileID:  "uploaded-file-id",
    VoiceID: "my-cloned-voice",
})

// Design voice
resp, err := client.Voice.Design(ctx, &minimax.VoiceDesignRequest{
    Prompt:      "A warm female voice...",
    PreviewText: "Hello, how can I help?",
})

VideoService

// Text to video
task, err := client.Video.CreateTextToVideo(ctx, &minimax.TextToVideoRequest{
    Model:  "video-01",
    Prompt: "A cat playing piano",
})
result, err := task.Wait(ctx)
// result.FileID contains the video file ID

// Image to video
task, err := client.Video.CreateImageToVideo(ctx, &minimax.ImageToVideoRequest{
    Model:          "video-01",
    FirstFrameImage: "https://...",
})

ImageService

resp, err := client.Image.Generate(ctx, &minimax.ImageGenerateRequest{
    Model:  "image-01",
    Prompt: "A beautiful sunset",
})
// resp.Data[0].URL or resp.Data[0].B64JSON

MusicService

task, err := client.Music.Generate(ctx, &minimax.MusicRequest{
    Prompt: "upbeat pop song",
    Lyrics: "[Verse]\nHello world...",
})
result, err := task.Wait(ctx)

FileService

// Upload
resp, err := client.File.Upload(ctx, filePath, minimax.FilePurposeVoiceClone)

// List
files, err := client.File.List(ctx, &minimax.FileListRequest{
    Purpose: minimax.FilePurposeVoiceClone,
})

// Download
data, err := client.File.Download(ctx, fileID)

// Delete
err := client.File.Delete(ctx, fileID)

Task Polling

task, err := client.Video.CreateTextToVideo(ctx, req)
if err != nil {
    return err
}

// Default 5s interval
result, err := task.Wait(ctx)

// Custom interval
result, err := task.WaitWithInterval(ctx, 10*time.Second)

// Manual polling
status, err := task.Query(ctx)
if status.Status == minimax.TaskStatusSuccess {
    // ...
}

Error Handling

resp, err := client.Text.CreateChatCompletion(ctx, req)
if err != nil {
    if e, ok := minimax.AsError(err); ok {
        fmt.Printf("API Error: %d - %s\n", e.StatusCode, e.StatusMsg)
        if e.IsRateLimit() {
            // Wait and retry
        }
    }
    return err
}

Streaming Internals

Uses SSE (Server-Sent Events):

• iter.Seq2[T, error] for Go 1.23+ range loops
• Auto-reconnect on transient errors (based on retry config)
• Hex audio decoding for speech streams

MiniMax SDK - Rust Implementation

Crate: giztoy-minimax

📚 Rust Documentation

Client

#![allow(unused)]
fn main() {
pub struct Client {
    http: Arc<HttpClient>,
    config: ClientConfig,
}
}

Constructor:

#![allow(unused)]
fn main() {
use giztoy_minimax::{Client, BASE_URL_GLOBAL};

// Basic
let client = Client::new("api-key")?;

// With builder
let client = Client::builder("api-key")
    .base_url(BASE_URL_GLOBAL)
    .max_retries(5)
    .build()?;
}

Builder Methods:

Services

Services are accessed via getter methods (returns new instance each call):

#![allow(unused)]
fn main() {
client.text()    // TextService
client.speech()  // SpeechService
client.voice()   // VoiceService
client.video()   // VideoService
client.image()   // ImageService
client.music()   // MusicService
client.file()    // FileService
}

TextService

#![allow(unused)]
fn main() {
use giztoy_minimax::{ChatCompletionRequest, Message};

// Synchronous
let resp = client.text().create_chat_completion(&ChatCompletionRequest {
    model: "MiniMax-M2.1".to_string(),
    messages: vec![
        Message { role: "user".to_string(), content: "Hello!".to_string() },
    ],
    ..Default::default()
}).await?;

// Streaming
let stream = client.text().create_chat_completion_stream(&req).await?;
while let Some(chunk) = stream.next().await {
    let chunk = chunk?;
    if let Some(choice) = chunk.choices.first() {
        print!("{}", choice.delta.content);
    }
}
}

SpeechService

#![allow(unused)]
fn main() {
use giztoy_minimax::{SpeechRequest, VoiceSetting};

// Synchronous
let resp = client.speech().synthesize(&SpeechRequest {
    model: "speech-2.6-hd".to_string(),
    text: "Hello, world!".to_string(),
    voice_setting: Some(VoiceSetting {
        voice_id: "male-qn-qingse".to_string(),
        ..Default::default()
    }),
    ..Default::default()
}).await?;
// resp.audio contains decoded bytes

// Streaming
let stream = client.speech().synthesize_stream(&req).await?;
while let Some(chunk) = stream.next().await {
    let chunk = chunk?;
    if let Some(audio) = chunk.audio {
        buf.extend(&audio);
    }
}

// Async (long text)
let task = client.speech().create_async_task(&AsyncSpeechRequest {
    // ...
}).await?;
let result = task.wait().await?;
}

VoiceService

#![allow(unused)]
fn main() {
// List voices
let voices = client.voice().list().await?;

// Clone voice
let resp = client.voice().clone(&VoiceCloneRequest {
    file_id: "uploaded-file-id".to_string(),
    voice_id: "my-cloned-voice".to_string(),
}).await?;

// Design voice
let resp = client.voice().design(&VoiceDesignRequest {
    prompt: "A warm female voice...".to_string(),
    preview_text: "Hello, how can I help?".to_string(),
    ..Default::default()
}).await?;
}

VideoService

#![allow(unused)]
fn main() {
// Text to video
let task = client.video().create_text_to_video(&TextToVideoRequest {
    model: "video-01".to_string(),
    prompt: "A cat playing piano".to_string(),
    ..Default::default()
}).await?;
let result = task.wait().await?;

// Image to video
let task = client.video().create_image_to_video(&ImageToVideoRequest {
    model: "video-01".to_string(),
    first_frame_image: "https://...".to_string(),
    ..Default::default()
}).await?;
}

ImageService

#![allow(unused)]
fn main() {
let resp = client.image().generate(&ImageGenerateRequest {
    model: "image-01".to_string(),
    prompt: "A beautiful sunset".to_string(),
    ..Default::default()
}).await?;
}

MusicService

#![allow(unused)]
fn main() {
let task = client.music().generate(&MusicRequest {
    prompt: "upbeat pop song".to_string(),
    lyrics: "[Verse]\nHello world...".to_string(),
    ..Default::default()
}).await?;
let result = task.wait().await?;
}

FileService

#![allow(unused)]
fn main() {
// Upload
let resp = client.file().upload(file_path, FilePurpose::VoiceClone).await?;

// List
let files = client.file().list(Some(FilePurpose::VoiceClone)).await?;

// Download
let data = client.file().download(&file_id).await?;

// Delete
client.file().delete(&file_id).await?;
}

Task Polling

#![allow(unused)]
fn main() {
let task = client.video().create_text_to_video(&req).await?;

// Default interval
let result = task.wait().await?;

// Custom interval
let result = task.wait_with_interval(Duration::from_secs(10)).await?;

// Manual polling
let status = task.query().await?;
if status.status == TaskStatus::Success {
    // ...
}
}

Error Handling

#![allow(unused)]
fn main() {
use giztoy_minimax::{Error, Result};

match client.text().create_chat_completion(&req).await {
    Ok(resp) => { /* ... */ }
    Err(Error::Api { status_code, status_msg }) => {
        eprintln!("API Error: {} - {}", status_code, status_msg);
    }
    Err(Error::Http(e)) => {
        eprintln!("HTTP Error: {}", e);
    }
    Err(e) => {
        eprintln!("Error: {}", e);
    }
}
}

HasModel Trait

For default model handling:

#![allow(unused)]
fn main() {
pub trait HasModel {
    fn model(&self) -> &str;
    fn set_model(&mut self, model: impl Into<String>);
    fn default_model() -> &'static str;
    fn apply_default_model(&mut self);
}
}

Differences from Go

MiniMax SDK - Known Issues

🟡 Minor Issues

MMX-001: Go NewClient panics on empty API key

File: go/pkg/minimax/client.go:100-102

Description:
NewClient panics instead of returning an error:

func NewClient(apiKey string, opts ...Option) *Client {
    if apiKey == "" {
        panic("minimax: apiKey must be non-empty")
    }

Impact: Unrecoverable error at construction time.

Suggestion: Return (*Client, error) or use builder pattern like Rust.

MMX-002: Rust services created on each call

File: rust/minimax/src/client.rs:91-123

Description:
Service getters create new instances each call:

#![allow(unused)]
fn main() {
pub fn speech(&self) -> SpeechService {
    SpeechService::new(self.http.clone())
}
}

Impact: Arc clone overhead on each service access.

Suggestion: Cache services or use &self references.

MMX-003: Go streaming uses hex encoding

File: go/pkg/minimax/speech.go:51-56

Description:
Audio data comes hex-encoded from API, decoded in SDK:

if apiResp.Data.Audio != "" {
    audio, err := decodeHexAudio(apiResp.Data.Audio)

Impact: CPU overhead for decoding, 2x memory during decode.

Note: This is API design, not SDK issue, but worth documenting.

MMX-004: No request timeout option

Description:
Both Go and Rust SDKs don't have request-level timeout option. Go suggests using context.WithTimeout, Rust doesn't document timeout handling.

Suggestion: Add timeout option or document clearly.

MMX-005: Go iter.Seq2 requires Go 1.23+

File: go/pkg/minimax/speech.go:78

Description:
Streaming uses iter.Seq2 which requires Go 1.23:

func (s *SpeechService) SynthesizeStream(ctx context.Context, req *SpeechRequest) iter.Seq2[*SpeechChunk, error]

Impact: Not compatible with older Go versions.

Note: Modern API choice, acceptable trade-off.

MMX-006: Error handling inconsistency

Description:
Go uses AsError() helper function, Rust uses error enum matching.

Go:

if e, ok := minimax.AsError(err); ok {
    if e.IsRateLimit() { ... }
}

Rust:

#![allow(unused)]
fn main() {
match err {
    Error::Api { status_code, .. } => { ... }
}
}

Impact: Different patterns between languages.

🔵 Enhancements

MMX-007: No WebSocket TTS support

Description:
Official API supports WebSocket for TTS (/v1/t2a_ws), but SDK only implements HTTP.

Suggestion: Add WebSocket-based streaming TTS for lower latency.

MMX-008: No request validation

Description:
No client-side validation before sending requests. Invalid parameters only fail after API call.

Suggestion: Add validation for known constraints (text length, model names, etc.).

MMX-009: No retry backoff configuration

Description:
Retry count is configurable, but backoff strategy is hardcoded.

Suggestion: Add configurable backoff (exponential, jitter).

MMX-010: No request/response logging

Description:
No built-in debug logging for API requests/responses.

Suggestion: Add optional logging middleware or debug mode.

MMX-011: No rate limit handling

Description:
Rate limit errors are returned but not automatically handled (e.g., exponential backoff, queue).

Suggestion: Add optional rate limit handling.

⚪ Notes

MMX-012: Full API coverage achieved

Description:
Both Go and Rust SDKs implement all documented MiniMax API endpoints:

• Text generation
• Speech synthesis (sync, stream, async)
• Voice management (list, clone, design)
• Video generation (text-to-video, image-to-video, agent)
• Image generation
• Music generation
• File management

MMX-013: Async task pattern

Description:
Long-running operations (video, async speech, music) use a consistent pattern:

1. Create task → returns Task[T]
2. Call task.Wait() for automatic polling
3. Or manual task.Query() for custom logic

This is a well-designed abstraction.

MMX-014: Base URL handling

Description:
Both SDKs support China and Global endpoints:

• China: https://api.minimaxi.com
• Global: https://api.minimaxi.chat

Correctly defaulting to China URL.

Summary

Overall: Well-implemented SDK with full API coverage. Both Go and Rust implementations are feature-complete and production-ready. Minor issues are mostly design choices rather than bugs.

DashScope SDK

Go and Rust SDK for Aliyun DashScope (百炼 Model Studio) APIs.

Official API Documentation: api/README.md

Design Goals

1. Realtime Focus: Primarily implement Qwen-Omni-Realtime WebSocket API
2. OpenAI Compatibility: Text/chat APIs use OpenAI-compatible SDK
3. Native WebSocket: Direct WebSocket implementation, not polling

Scope

This SDK focuses on Qwen-Omni-Realtime API for real-time multimodal conversation. For standard text APIs, use OpenAI-compatible SDKs.

API Coverage

Architecture

graph TB
    subgraph client["Client"]
        subgraph realtime["RealtimeService"]
            session["RealtimeSession"]
            send["Send Events"]
            recv["Receive Events"]
        end
    end
    
    client --> ws["wss://dashscope.aliyuncs.com<br/>/api-ws/v1/realtime"]

Authentication

Authorization: Bearer <api_key>

Optional workspace isolation:

X-DashScope-WorkSpace: <workspace_id>

Base URLs

Models

Event Flow

sequenceDiagram
    participant C as Client
    participant S as Server
    C->>S: session.update
    C->>S: input_audio_buffer.append
    C->>S: response.create
    S-->>C: response.audio.delta
    S-->>C: response.audio.delta
    S-->>C: response.done

Examples Directory

• examples/go/dashscope/ - Go SDK examples
• examples/cmd/dashscope/ - CLI test scripts

For Text/Chat APIs

Use OpenAI-compatible SDK:

Go:

import "github.com/sashabaranov/go-openai"

config := openai.DefaultConfig(apiKey)
config.BaseURL = "https://dashscope.aliyuncs.com/compatible-mode/v1"
client := openai.NewClientWithConfig(config)

Rust:

#![allow(unused)]
fn main() {
// Use async-openai with custom base URL
}

Related

• CLI tool: go/cmd/dashscope/
• CLI tests: examples/cmd/dashscope/

DashScope (阿里云百炼) API 文档

原始文档

如果本文档信息不完整，请访问上述链接获取最新内容。

概述

DashScope 是阿里云大模型服务平台百炼（Model Studio）提供的 API 服务。支持：

• 文本生成 - 通义千问（Qwen）系列大语言模型，兼容 OpenAI API
• 多模态 - 图像理解、音频理解
• 实时对话 - Qwen-Omni-Realtime 实时音频/视频对话
• 智能体应用 - 调用已配置的 Agent/工作流应用
• 知识库 - 文档上传、索引、检索增强生成（RAG）

目录结构

docs/dashscope/
├── README.md           # 本文档 - 概述
├── auth.md             # 认证与鉴权
├── text.md             # 文本模型 API (Qwen)
├── app.md              # 应用调用 API
└── realtime/           # 实时多模态 API
    ├── README.md       # 概述
    ├── client-events.md # 客户端事件
    └── server-events.md # 服务端事件

服务端点

HTTP API

WebSocket API

应用 API

POST https://dashscope.aliyuncs.com/api/v1/apps/{APP_ID}/completion

支持的模型

文本模型 (Qwen)

多模态模型

实时多模态模型

快速开始

1. 获取 API Key

1. 登录 百炼控制台
2. 进入"密钥管理"
3. 创建 API Key

2. 设置环境变量

export DASHSCOPE_API_KEY="sk-xxxxxxxxxxxxxxxx"

3. 调用示例

curl https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions \
  -H "Authorization: Bearer $DASHSCOPE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-turbo",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

详细文档

SDK

Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(
    api_key=os.getenv("DASHSCOPE_API_KEY"),
    base_url="https://dashscope.aliyuncs.com/compatible-mode/v1"
)

response = client.chat.completions.create(
    model="qwen-turbo",
    messages=[{"role": "user", "content": "Hello!"}]
)

Go (go-openai)

import "github.com/sashabaranov/go-openai"

config := openai.DefaultConfig(os.Getenv("DASHSCOPE_API_KEY"))
config.BaseURL = "https://dashscope.aliyuncs.com/compatible-mode/v1"

client := openai.NewClientWithConfig(config)

Go (giztoy/dashscope) - Realtime API

本项目提供了原生 Go SDK 支持 Qwen-Omni-Realtime API：

import "github.com/haivivi/giztoy/pkg/dashscope"

client := dashscope.NewClient(os.Getenv("DASHSCOPE_API_KEY"))
session, err := client.Realtime.Connect(ctx, &dashscope.RealtimeConfig{
    Model: dashscope.ModelQwenOmniTurboRealtimeLatest,
})
// 发送音频、接收事件...

CLI 工具: bazel run //go/cmd/dashscope -- omni chat

官方 SDK

• Python: pip install dashscope
• Java: Maven 依赖 com.alibaba:dashscope-sdk

DashScope SDK - Go Implementation

Import: github.com/haivivi/giztoy/pkg/dashscope

📚 Go Documentation

Client

type Client struct {
    Realtime *RealtimeService
}

Constructor:

// Basic
client := dashscope.NewClient("sk-xxxxxxxx")

// With workspace
client := dashscope.NewClient("sk-xxxxxxxx",
    dashscope.WithWorkspace("ws-xxxxxxxx"),
)

// Custom endpoint (international)
client := dashscope.NewClient("sk-xxxxxxxx",
    dashscope.WithBaseURL("wss://dashscope-intl.aliyuncs.com/api-ws/v1/realtime"),
)

Options:

RealtimeService

Connect Session

session, err := client.Realtime.Connect(ctx, &dashscope.RealtimeConfig{
    Model: dashscope.ModelQwenOmniTurboRealtimeLatest,
})
if err != nil {
    log.Fatal(err)
}
defer session.Close()

Send Events

// Update session configuration
session.UpdateSession(&dashscope.SessionUpdate{
    Modalities: []string{"text", "audio"},
    Voice: "Cherry",
    InputAudioFormat: "pcm16",
    OutputAudioFormat: "pcm16",
})

// Append audio data
session.AppendAudio(audioData)

// Commit audio (finalize input)
session.CommitAudio()

// Create response (start inference)
session.CreateResponse()

// Send text
session.AppendText("Hello!")

// Cancel response
session.CancelResponse()

Receive Events

// Using Go 1.23+ iter.Seq2
for event, err := range session.Events() {
    if err != nil {
        log.Fatal(err)
    }
    
    switch event.Type {
    case dashscope.EventResponseAudioDelta:
        // Audio chunk received
        play(event.Delta)
        
    case dashscope.EventResponseTextDelta:
        // Text chunk received
        fmt.Print(event.Delta)
        
    case dashscope.EventResponseDone:
        // Response complete
        
    case dashscope.EventError:
        // Error occurred
        log.Printf("Error: %s", event.Error.Message)
    }
}

Events

Client Events (Send)

Server Events (Receive)

Models

const (
    ModelQwenOmniTurboRealtimeLatest  = "qwen-omni-turbo-realtime-latest"
    ModelQwen3OmniFlashRealtimeLatest = "qwen3-omni-flash-realtime-latest"
)

Error Handling

for event, err := range session.Events() {
    if err != nil {
        // Connection error
        log.Fatal(err)
    }
    
    if event.Type == dashscope.EventError {
        // API error
        log.Printf("API Error [%s]: %s", event.Error.Code, event.Error.Message)
    }
}

Complete Example

func main() {
    client := dashscope.NewClient(os.Getenv("DASHSCOPE_API_KEY"))
    
    session, err := client.Realtime.Connect(context.Background(), &dashscope.RealtimeConfig{
        Model: dashscope.ModelQwenOmniTurboRealtimeLatest,
    })
    if err != nil {
        log.Fatal(err)
    }
    defer session.Close()
    
    // Configure session
    session.UpdateSession(&dashscope.SessionUpdate{
        Voice: "Cherry",
    })
    
    // Send audio (from microphone, etc.)
    session.AppendAudio(audioData)
    session.CommitAudio()
    session.CreateResponse()
    
    // Receive and play response
    for event, err := range session.Events() {
        if err != nil {
            break
        }
        if event.Type == dashscope.EventResponseAudioDelta {
            player.Write(event.Delta)
        }
    }
}

DashScope SDK - Rust Implementation

Crate: giztoy-dashscope

📚 Rust Documentation

Client

#![allow(unused)]
fn main() {
pub struct Client {
    // Internal configuration
}

impl Client {
    pub fn realtime(&self) -> RealtimeService;
}
}

Constructor:

#![allow(unused)]
fn main() {
use giztoy_dashscope::{Client, DEFAULT_REALTIME_URL};

// Basic
let client = Client::new("sk-xxxxxxxx")?;

// With builder
let client = Client::builder("sk-xxxxxxxx")
    .workspace("ws-xxxxxxxx")
    .base_url("wss://dashscope-intl.aliyuncs.com/api-ws/v1/realtime")
    .build()?;
}

RealtimeService

Connect Session

#![allow(unused)]
fn main() {
use giztoy_dashscope::{RealtimeConfig, ModelQwenOmniTurboRealtimeLatest};

let session = client.realtime().connect(&RealtimeConfig {
    model: ModelQwenOmniTurboRealtimeLatest.to_string(),
    ..Default::default()
}).await?;
}

Send Events

#![allow(unused)]
fn main() {
// Update session
session.update_session(&SessionUpdate {
    modalities: vec!["text".to_string(), "audio".to_string()],
    voice: Some("Cherry".to_string()),
    ..Default::default()
}).await?;

// Append audio
session.append_audio(&audio_data).await?;

// Commit audio
session.commit_audio().await?;

// Create response
session.create_response().await?;
}

Receive Events

#![allow(unused)]
fn main() {
use giztoy_dashscope::ServerEvent;

while let Some(event) = session.recv().await {
    let event = event?;
    
    match event {
        ServerEvent::ResponseAudioDelta { delta, .. } => {
            // Play audio
            player.write(&delta)?;
        }
        ServerEvent::ResponseTextDelta { delta, .. } => {
            // Print text
            print!("{}", delta);
        }
        ServerEvent::ResponseDone { .. } => {
            // Complete
            break;
        }
        ServerEvent::Error { error } => {
            eprintln!("Error: {}", error.message);
        }
        _ => {}
    }
}
}

Events

Client Events (Send)

#![allow(unused)]
fn main() {
pub enum ClientEvent {
    SessionUpdate(SessionUpdate),
    InputAudioBufferAppend { audio: Vec<u8> },
    InputAudioBufferCommit,
    ResponseCreate(ResponseCreateOptions),
    ResponseCancel,
}
}

Server Events (Receive)

#![allow(unused)]
fn main() {
pub enum ServerEvent {
    SessionCreated { session: SessionInfo },
    SessionUpdated { session: SessionInfo },
    ResponseCreated { response: ResponseInfo },
    ResponseAudioDelta { delta: Vec<u8> },
    ResponseTextDelta { delta: String },
    ResponseDone { response: ResponseInfo },
    Error { error: ErrorInfo },
    // ... more events
}
}

Models

#![allow(unused)]
fn main() {
pub const MODEL_QWEN_OMNI_TURBO_REALTIME_LATEST: &str = "qwen-omni-turbo-realtime-latest";
pub const MODEL_QWEN3_OMNI_FLASH_REALTIME_LATEST: &str = "qwen3-omni-flash-realtime-latest";
}

Error Handling

#![allow(unused)]
fn main() {
use giztoy_dashscope::{Error, Result};

match session.recv().await {
    Some(Ok(event)) => {
        // Process event
    }
    Some(Err(Error::WebSocket(e))) => {
        eprintln!("WebSocket error: {}", e);
    }
    Some(Err(Error::Api { code, message })) => {
        eprintln!("API error [{}]: {}", code, message);
    }
    None => {
        // Connection closed
    }
}
}

Complete Example

use giztoy_dashscope::{Client, RealtimeConfig, ServerEvent};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let api_key = std::env::var("DASHSCOPE_API_KEY")?;
    let client = Client::new(&api_key)?;
    
    let session = client.realtime().connect(&RealtimeConfig {
        model: "qwen-omni-turbo-realtime-latest".to_string(),
        ..Default::default()
    }).await?;
    
    // Configure
    session.update_session(&SessionUpdate {
        voice: Some("Cherry".to_string()),
        ..Default::default()
    }).await?;
    
    // Send audio
    session.append_audio(&audio_data).await?;
    session.commit_audio().await?;
    session.create_response().await?;
    
    // Receive response
    while let Some(event) = session.recv().await {
        match event? {
            ServerEvent::ResponseAudioDelta { delta, .. } => {
                player.write(&delta)?;
            }
            ServerEvent::ResponseDone { .. } => break,
            _ => {}
        }
    }
    
    Ok(())
}

Differences from Go

DashScope SDK - Known Issues

🟡 Minor Issues

DS-001: Go NewClient panics on empty API key

File: go/pkg/dashscope/client.go:39-41

Description:
NewClient panics instead of returning an error:

func NewClient(apiKey string, opts ...Option) *Client {
    if apiKey == "" {
        panic("dashscope: API key is required")
    }

Impact: Unrecoverable error at construction time.

Suggestion: Return (*Client, error) like Rust version.

DS-002: Limited to Realtime API only

Description:
SDK only implements Realtime API. Text/Chat APIs require separate OpenAI-compatible SDK.

Impact: Users need two SDKs for full DashScope usage.

Note: This is intentional design choice - text APIs are OpenAI-compatible.

DS-003: No HTTP API implementation

Description:
No HTTP client for non-realtime operations (file upload, app calls, etc.).

Suggestion: Add HTTP service for app/agent API calls.

DS-004: Video input support limited

Description:
Qwen3-Omni-Flash supports video input, but SDK support may be incomplete.

Status: ⚠️ Needs verification.

🔵 Enhancements

DS-005: No automatic reconnection

Description:
WebSocket sessions don't auto-reconnect on disconnection.

Suggestion: Add reconnection with backoff for long-running sessions.

DS-006: No audio transcoding

Description:
Audio must be in correct format (PCM16/PCM24). No built-in transcoding.

Suggestion: Add optional audio format conversion.

DS-007: No VAD (Voice Activity Detection) integration

Description:
Manual audio buffer management. No built-in VAD for automatic speech detection.

Suggestion: Integrate with audio/pcm for silence detection.

DS-008: Missing tool call examples

Description:
Function/tool calling is supported but not well documented with examples.

⚪ Notes

DS-009: Clean WebSocket event model

Description:
Both Go and Rust implement clean event-based model matching OpenAI Realtime API patterns. This is well-designed.

DS-010: Model constants provided

Description:
Both SDKs provide model name constants:

const ModelQwenOmniTurboRealtimeLatest = "qwen-omni-turbo-realtime-latest"

Good for discoverability and avoiding typos.

DS-011: Workspace support

Description:
Both SDKs support workspace isolation via WithWorkspace() option:

client := dashscope.NewClient(apiKey, dashscope.WithWorkspace("ws-xxx"))

Useful for enterprise environments.

DS-012: International endpoint support

Description:
SDKs support both China and international endpoints:

• China: wss://dashscope.aliyuncs.com/...
• International: wss://dashscope-intl.aliyuncs.com/...

Summary

Overall: Focused SDK for Realtime API with clean design. Main limitation is narrow scope (Realtime only), which is intentional since text APIs are OpenAI-compatible. Both Go and Rust implementations are feature-complete for their scope.

Doubao Speech SDK

Go and Rust SDK for Volcengine Doubao Speech API (豆包语音).

Official API Documentation: api/README.md

Design Goals

1. Dual API Version Support: V1 (Classic) and V2/V3 (BigModel) APIs
2. Multiple Auth Methods: Bearer Token, API Key, V2 API Key
3. Comprehensive Coverage: TTS, ASR, Voice Clone, Realtime, Meeting, Podcast, etc.
4. Streaming-first: WebSocket-based streaming for real-time scenarios

API Versions

Doubao Speech has two API generations:

API Coverage

Architecture

graph TB
    subgraph client["Client"]
        subgraph v1["V1 Services (Classic)"]
            tts1[TTS]
            asr1[ASR]
        end
        subgraph v2["V2 Services (BigModel)"]
            tts2[TTSV2]
            asr2[ASRV2]
            advanced["VoiceClone<br/>Realtime<br/>Meeting<br/>Podcast<br/>Translation<br/>Media"]
        end
    end
    
    subgraph console["Console Client"]
        aksig["AK/SK Signature<br/>Authentication"]
    end
    
    client --> api["Volcengine API"]
    console --> api

Authentication Methods

Speech API Client

Console Client

Uses Volcengine OpenAPI AK/SK signature (HMAC-SHA256).

Resource IDs (V2/V3)

Clusters (V1)

Examples Directory

• examples/go/doubaospeech/ - Go SDK examples
• examples/cmd/doubaospeech/ - CLI test scripts

Related

• CLI tool: go/cmd/doubaospeech/
• CLI tests: examples/cmd/doubaospeech/

豆包语音（Doubao Speech）API 文档

原始文档

• 文档首页: https://www.volcengine.com/docs/6561/162929
• 控制台: https://console.volcengine.com/speech/app

如果本文档信息不完整，请访问上述链接获取最新内容。

产品体系

豆包语音分为两代产品：大模型版（2.0） 和 经典版（1.0）。推荐使用大模型版。

语音合成（TTS）

大模型语音合成 2.0

原始文档链接：

• https://www.volcengine.com/docs/6561/1598757 (单向流式HTTP-V3)
• https://www.volcengine.com/docs/6561/1719100 (单向流式WebSocket-V3)
• https://www.volcengine.com/docs/6561/1329505 (双向流式WebSocket-V3)
• https://www.volcengine.com/docs/6561/1330194 (异步长文本)

经典版语音合成 1.0

原始文档链接：

• https://www.volcengine.com/docs/6561/79820 (HTTP接口)
• https://www.volcengine.com/docs/6561/79821 (WebSocket接口)
• https://www.volcengine.com/docs/6561/97465 (参数说明)

精品长文本语音合成

原始文档链接：

• https://www.volcengine.com/docs/6561/1096680

语音识别（ASR）

大模型语音识别 2.0

原始文档链接：

• https://www.volcengine.com/docs/6561/1354869 (大模型流式语音识别)
• https://www.volcengine.com/docs/6561/1354868 (大模型录音文件识别标准版)
• https://www.volcengine.com/docs/6561/1631584 (大模型录音文件极速版)
• https://www.volcengine.com/docs/6561/1840838 (大模型录音文件闲时版)

经典版语音识别 1.0

原始文档链接：

• https://www.volcengine.com/docs/6561/104897 (一句话识别)
• https://www.volcengine.com/docs/6561/80816 (流式语音识别)
• https://www.volcengine.com/docs/6561/80818 (录音文件识别标准版)
• https://www.volcengine.com/docs/6561/80820 (录音文件识别极速版)

声音复刻

原始文档链接：

• https://www.volcengine.com/docs/6561/1305191 (声音复刻API)
• https://www.volcengine.com/docs/6561/1829010 (声音复刻下单及使用指南)

实时语音大模型

原始文档链接：

• https://www.volcengine.com/docs/6561/1257584 (端到端实时语音大模型API)

播客合成

原始文档链接：

• https://www.volcengine.com/docs/6561/1668014 (播客API-websocket-v3协议)

同声传译

原始文档链接：

• https://www.volcengine.com/docs/6561/xxx (同声传译2.0-API)

语音妙记（会议纪要）

原始文档链接：

• https://www.volcengine.com/docs/6561/xxx (豆包语音妙记-API)

音视频字幕

原始文档链接：

• https://www.volcengine.com/docs/6561/192519 (音视频字幕生成)
• https://www.volcengine.com/docs/6561/113635 (自动字幕打轴)

控制台管理 API

原始文档链接：

• https://www.volcengine.com/docs/6561/1770994 (ListBigModelTTSTimbres)
• https://www.volcengine.com/docs/6561/2160690 (ListSpeakers)

认证方式

Speech API（语音服务）

语音服务使用以下认证方式：

Console API（控制台服务）

控制台 API 使用 Volcengine OpenAPI AK/SK 签名认证：

Authorization: HMAC-SHA256 Credential={AccessKeyId}/...

详见 auth.md

快速选择

Doubao Speech SDK - Go Implementation

Import: github.com/haivivi/giztoy/pkg/doubaospeech

📚 Go Documentation

Clients

Speech API Client

type Client struct {
    // V1 Services (Classic)
    TTS *TTSService
    ASR *ASRService
    
    // V2 Services (BigModel)
    TTSV2 *TTSServiceV2
    ASRV2 *ASRServiceV2
    
    // Shared Services
    VoiceClone  *VoiceCloneService
    Realtime    *RealtimeService
    Meeting     *MeetingService
    Podcast     *PodcastService
    Translation *TranslationService
    Media       *MediaService
}

Constructor:

// With API Key (recommended)
client := doubaospeech.NewClient("app-id",
    doubaospeech.WithAPIKey("your-api-key"),
    doubaospeech.WithCluster("volcano_tts"),
)

// With Bearer Token
client := doubaospeech.NewClient("app-id",
    doubaospeech.WithBearerToken("your-token"),
)

// With V2 API Key (for BigModel APIs)
client := doubaospeech.NewClient("app-id",
    doubaospeech.WithV2APIKey("access-key", "app-key"),
    doubaospeech.WithResourceID("seed-tts-2.0"),
)

Console API Client

console := doubaospeech.NewConsole("access-key", "secret-key")

Services

TTS V1 (Classic)

// Synchronous
resp, err := client.TTS.Synthesize(ctx, &doubaospeech.TTSRequest{
    Text:      "你好，世界！",
    VoiceType: "zh_female_cancan",
})
// resp.Audio contains audio bytes

// Streaming (Go 1.23+ iter.Seq2)
for chunk, err := range client.TTS.SynthesizeStream(ctx, req) {
    if err != nil {
        return err
    }
    buf.Write(chunk.Audio)
}

TTS V2 (BigModel)

// Streaming HTTP
for chunk, err := range client.TTSV2.SynthesizeStream(ctx, &doubaospeech.TTSV2Request{
    Text:       "你好，世界！",
    VoiceType:  "zh_female_cancan",
    ResourceID: "seed-tts-2.0",
}) {
    // Process chunk
}

// Async (long text)
task, err := client.TTSV2.SubmitAsync(ctx, &doubaospeech.AsyncTTSRequest{
    Text: longText,
})
result, err := task.Wait(ctx)

ASR (Speech Recognition)

// One-sentence (V1)
resp, err := client.ASR.Recognize(ctx, &doubaospeech.ASRRequest{
    Audio:    audioData,
    Format:   "pcm",
    Language: "zh-CN",
})

// Streaming (WebSocket)
session, err := client.ASR.OpenStreamSession(ctx, &doubaospeech.StreamASRConfig{
    Format:     "pcm",
    SampleRate: 16000,
})
defer session.Close()

// Send audio chunks
session.SendAudio(ctx, audioData, false)
session.SendAudio(ctx, lastData, true)

// Receive results
for chunk, err := range session.Recv() {
    if err != nil {
        break
    }
    fmt.Println(chunk.Text)
}

Voice Clone

// Upload audio for training
result, err := client.VoiceClone.Upload(ctx, &doubaospeech.VoiceCloneRequest{
    AudioData: audioData,
    VoiceID:   "my-custom-voice",
})

// Check status
status, err := client.VoiceClone.GetStatus(ctx, "my-custom-voice")

// Activate voice
err := client.VoiceClone.Activate(ctx, "my-custom-voice")

Realtime Dialogue

session, err := client.Realtime.Connect(ctx, &doubaospeech.RealtimeConfig{
    Model: "speech-dialog-001",
})
defer session.Close()

// Send audio
session.SendAudio(audioData)

// Receive events
for event := range session.Events() {
    switch event.Type {
    case "asr_result":
        fmt.Println("User:", event.AsrResult.Text)
    case "tts_audio":
        play(event.TtsAudio)
    }
}

Console API

// List available voices
voices, err := console.ListSpeakers(ctx, &doubaospeech.ListSpeakersRequest{})

// List timbres
timbres, err := console.ListTimbres(ctx, &doubaospeech.ListTimbresRequest{})

// Check voice clone status
status, err := console.ListVoiceCloneStatus(ctx, &doubaospeech.ListVoiceCloneStatusRequest{
    VoiceID: "my-custom-voice",
})

Options

Error Handling

if err != nil {
    if e, ok := doubaospeech.AsError(err); ok {
        fmt.Printf("Error %d: %s\n", e.Code, e.Message)
        if e.IsRateLimit() {
            // Handle rate limiting
        }
    }
}

Doubao Speech SDK - Rust Implementation

Crate: giztoy-doubaospeech

📚 Rust Documentation

Clients

Speech API Client

#![allow(unused)]
fn main() {
pub struct Client {
    // Internal HTTP/WebSocket clients
}

impl Client {
    pub fn tts(&self) -> TtsService;
    pub fn asr(&self) -> AsrService;
    pub fn voice_clone(&self) -> VoiceCloneService;
    pub fn realtime(&self) -> RealtimeService;
    pub fn meeting(&self) -> MeetingService;
    pub fn podcast(&self) -> PodcastService;
    pub fn translation(&self) -> TranslationService;
    pub fn media(&self) -> MediaService;
}
}

Constructor:

#![allow(unused)]
fn main() {
use giztoy_doubaospeech::Client;

// With API Key (recommended)
let client = Client::builder("app-id")
    .api_key("your-api-key")
    .cluster("volcano_tts")
    .build()?;

// With Bearer Token
let client = Client::builder("app-id")
    .bearer_token("your-token")
    .build()?;

// With V2 API Key
let client = Client::builder("app-id")
    .v2_api_key("access-key", "app-key")
    .resource_id("seed-tts-2.0")
    .build()?;
}

Console Client

#![allow(unused)]
fn main() {
use giztoy_doubaospeech::Console;

let console = Console::new("access-key", "secret-key");
}

Services

TTS Service

#![allow(unused)]
fn main() {
use giztoy_doubaospeech::{TtsRequest, TtsService};

// Synchronous
let response = client.tts().synthesize(&TtsRequest {
    text: "你好，世界！".to_string(),
    voice_type: "zh_female_cancan".to_string(),
    ..Default::default()
}).await?;
// response.audio contains bytes

// Streaming
let stream = client.tts().synthesize_stream(&req).await?;
while let Some(chunk) = stream.next().await {
    let chunk = chunk?;
    if let Some(audio) = chunk.audio {
        buf.extend(&audio);
    }
}
}

ASR Service

#![allow(unused)]
fn main() {
use giztoy_doubaospeech::{OneSentenceRequest, StreamAsrConfig};

// One-sentence
let result = client.asr().recognize(&OneSentenceRequest {
    audio: audio_data,
    format: "pcm".to_string(),
    language: "zh-CN".to_string(),
    ..Default::default()
}).await?;

// Streaming
let session = client.asr().open_stream_session(&StreamAsrConfig {
    format: "pcm".to_string(),
    sample_rate: 16000,
    ..Default::default()
}).await?;

// Send audio
session.send_audio(&audio_data, false).await?;
session.send_audio(&last_data, true).await?;

// Receive results
while let Some(result) = session.recv().await {
    let chunk = result?;
    println!("Text: {}", chunk.text);
}
}

Voice Clone Service

#![allow(unused)]
fn main() {
// Upload for training
let result = client.voice_clone().upload(&VoiceCloneTrainRequest {
    audio_data: audio_bytes,
    voice_id: "my-custom-voice".to_string(),
    ..Default::default()
}).await?;

// Check status
let status = client.voice_clone().get_status("my-custom-voice").await?;
}

Realtime Service

#![allow(unused)]
fn main() {
use giztoy_doubaospeech::{RealtimeConfig, RealtimeEventType};

let session = client.realtime().connect(&RealtimeConfig {
    model: "speech-dialog-001".to_string(),
    ..Default::default()
}).await?;

// Send audio
session.send_audio(&audio_data).await?;

// Receive events
while let Some(event) = session.recv().await {
    let event = event?;
    match event.event_type {
        RealtimeEventType::AsrResult => {
            println!("User: {}", event.asr_result.text);
        }
        RealtimeEventType::TtsAudio => {
            play(&event.tts_audio);
        }
        _ => {}
    }
}
}

Console API

#![allow(unused)]
fn main() {
use giztoy_doubaospeech::{Console, ListSpeakersRequest};

let console = Console::new("access-key", "secret-key");

// List speakers
let speakers = console.list_speakers(&ListSpeakersRequest::default()).await?;

// List timbres
let timbres = console.list_timbres(&ListTimbresRequest::default()).await?;
}

Builder Options

Error Handling

#![allow(unused)]
fn main() {
use giztoy_doubaospeech::{Error, Result};

match client.tts().synthesize(&req).await {
    Ok(resp) => { /* ... */ }
    Err(Error::Api { code, message }) => {
        eprintln!("API Error {}: {}", code, message);
    }
    Err(e) => {
        eprintln!("Error: {}", e);
    }
}
}

Differences from Go

Doubao Speech SDK - Known Issues

🟡 Minor Issues

DBS-001: Go auth header format unusual

File: go/pkg/doubaospeech/client.go:238-239

Description:
Bearer token format is Bearer;{token} instead of standard Bearer {token}:

req.Header.Set("Authorization", "Bearer;"+c.config.accessToken)

Impact: Non-standard but required by Volcengine API.

Note: This is API requirement, not SDK issue.

DBS-002: Multiple auth method complexity

Description:
SDK supports 4+ authentication methods:

• API Key (x-api-key)
• Bearer Token (Authorization: Bearer;)
• V2 API Key (X-Api-Access-Key, X-Api-App-Key)
• Resource-specific fixed keys

Impact: Confusing for users which method to use for which service.

Suggestion: Add helper methods like NewTTSClient(), NewRealtimeClient() with correct defaults.

DBS-003: Resource ID vs Cluster confusion

Description:
V1 uses "cluster", V2 uses "resource_id" for service selection:

• V1: WithCluster("volcano_tts")
• V2: WithResourceID("seed-tts-2.0")

Impact: Easy to mix up, unclear which to use when.

DBS-004: Rust async TTS incomplete

Description:
Rust implementation for async long-text TTS may be incomplete or missing compared to Go.

Status: ⚠️ Needs verification.

DBS-005: Rust file ASR incomplete

Description:
Rust implementation for file-based ASR may be incomplete compared to Go.

Status: ⚠️ Needs verification.

DBS-006: Fixed app keys hardcoded

File: go/pkg/doubaospeech/client.go:17-24

Description:
Some V3 APIs use fixed app keys from documentation:

const (
    AppKeyRealtime = "PlgvMymc7f3tQnJ6"
    AppKeyPodcast = "aGjiRDfUWi"
)

Impact: If Volcengine changes these, SDK breaks until updated.

Note: This is documented API behavior.

🔵 Enhancements

DBS-007: No automatic service version selection

Description:
User must manually choose between V1 and V2 services. No automatic selection based on features needed.

Suggestion: Add unified service that routes to correct version.

DBS-008: No connection pooling documentation

Description:
WebSocket connections for streaming services could benefit from pooling documentation.

DBS-009: No retry for WebSocket connections

Description:
HTTP requests have retry, but WebSocket connections don't auto-reconnect on failure.

Suggestion: Add reconnection logic for streaming sessions.

DBS-010: Console API missing some endpoints

Description:
Console client may not cover all management APIs available on Volcengine.

⚪ Notes

DBS-011: Dual API version design

Description:
Having both V1 (Classic) and V2 (BigModel) services in same client reflects Volcengine's actual API structure. This is intentional, not a flaw.

DBS-012: Protocol module for WebSocket

Description:
Both Go and Rust have a protocol module for WebSocket message serialization. This is well-structured for the binary protocol requirements.

DBS-013: Comprehensive service coverage

Description:
SDK covers nearly all Doubao Speech services:

• TTS (sync, stream, async)
• ASR (one-sentence, stream, file)
• Voice Clone
• Realtime Dialogue
• Meeting Transcription
• Podcast Synthesis
• Translation
• Media Subtitle

This is impressive coverage.

DBS-014: Console uses AK/SK signature

Description:
Console API uses Volcengine OpenAPI signature (HMAC-SHA256), not simple token. This is standard for Volcengine management APIs.

Summary

Overall: Comprehensive SDK with excellent API coverage. Main complexity is from Volcengine's dual API version system and multiple authentication methods. Rust implementation may have some gaps compared to Go.

Jiutian API Documentation

九天大模型 (Jiutian) - China Mobile's AI Service Platform.

Official API Documentation: api/README.md

Overview

Jiutian is China Mobile's terminal intelligent agent service management platform for AI/LLM cloud integration.

Note: This is API documentation only. No Go/Rust SDK implementation exists in this repository.

API Features

• Chat Completions: OpenAI-compatible chat API
• Device Integration: Device registration and heartbeat protocols
• Assistant Management: Configure AI assistants

Integration Notes

For integration with Jiutian API:

1. Use OpenAI-compatible SDK with custom base URL
2. Follow device authentication protocols
3. See api/tutorial.md for quick start

Authentication

Requires:

• AI Token (obtained via email application)
• IP whitelist registration
• Product ID from management platform

Environment

SDK Implementation Status

For basic integration, use OpenAI-compatible SDK:

Go:

config := openai.DefaultConfig(jiutianToken)
config.BaseURL = "https://ivs.chinamobiledevice.com:30100/v1"
client := openai.NewClientWithConfig(config)

Related Documentation

• Quick Start Tutorial
• Concepts & Terms
• Authentication
• Chat API
• Device Protocol
• FAQ

九天大模型 API 文档

终端智能体服务管理平台 AI 大模型云云对接文档

文档索引

文档变更记录

AI 服务接入流程

1. 请提供厂商服务器IP开通访问白名单，以及向纳管平台申请的产品id（productId）来申请AI token
2. 邮件发送至：
   • zouyiqiang_fx@cmdc.chinamobile.com
   • zhucaiwen_fx@cmdc.chinamobile.com
   • 抄送：zhengzhongwei_fx@cmdc.chinamobile.com
3. 白名单开通后，厂商服务器就可以使用 AI TOKEN 调用九天大模型接口

环境配置

测试 Token: sk-Y73NAU0tArvGRlpUE9060529470b42Ac8bA34d40F48b0564

系统提示词: 您好，我是中国移动的智能助理灵犀。如果您询问我的身份，我会回答："您好，我是中国移动智能助理灵犀"。

模型上下文长度: 8K

Jiutian - Go Implementation

Status: Not Implemented

No native Go SDK for Jiutian API exists in this repository.

Recommendation

Use OpenAI-compatible SDK since Jiutian API follows OpenAI chat completions format:

import "github.com/sashabaranov/go-openai"

config := openai.DefaultConfig("sk-your-jiutian-token")
config.BaseURL = "https://ivs.chinamobiledevice.com:30100/v1"

client := openai.NewClientWithConfig(config)

resp, err := client.CreateChatCompletion(ctx, openai.ChatCompletionRequest{
    Model: "jiutian",
    Messages: []openai.ChatCompletionMessage{
        {
            Role:    "system",
            Content: "您好，我是中国移动的智能助理灵犀。",
        },
        {
            Role:    "user", 
            Content: "你是谁？",
        },
    },
})

Device Protocol

For device-specific features (registration, heartbeat), implement HTTP client directly:

// Device heartbeat
type HeartbeatRequest struct {
    DeviceID  string `json:"device_id"`
    ProductID string `json:"product_id"`
    Timestamp int64  `json:"timestamp"`
}

func sendHeartbeat(ctx context.Context, client *http.Client, req *HeartbeatRequest) error {
    // POST to /api/device/heartbeat
}

Future Work

A native SDK could provide:

• Device registration/heartbeat management
• Token refresh handling
• Jiutian-specific features

See api/device.md for device protocol details.

Jiutian - Rust Implementation

Status: Not Implemented

No native Rust SDK for Jiutian API exists in this repository.

Recommendation

Use OpenAI-compatible SDK since Jiutian API follows OpenAI chat completions format:

#![allow(unused)]
fn main() {
use async_openai::{Client, config::OpenAIConfig};

let config = OpenAIConfig::new()
    .with_api_key("sk-your-jiutian-token")
    .with_api_base("https://ivs.chinamobiledevice.com:30100/v1");

let client = Client::with_config(config);

let request = CreateChatCompletionRequestArgs::default()
    .model("jiutian")
    .messages([
        ChatCompletionRequestMessage::System(
            ChatCompletionRequestSystemMessageArgs::default()
                .content("您好，我是中国移动的智能助理灵犀。")
                .build()?
        ),
        ChatCompletionRequestMessage::User(
            ChatCompletionRequestUserMessageArgs::default()
                .content("你是谁？")
                .build()?
        ),
    ])
    .build()?;

let response = client.chat().create(request).await?;
}

Device Protocol

For device-specific features, implement HTTP client using reqwest:

#![allow(unused)]
fn main() {
use serde::{Deserialize, Serialize};

#[derive(Serialize)]
struct HeartbeatRequest {
    device_id: String,
    product_id: String,
    timestamp: i64,
}

async fn send_heartbeat(
    client: &reqwest::Client,
    base_url: &str,
    req: &HeartbeatRequest,
) -> Result<(), Error> {
    client
        .post(format!("{}/api/device/heartbeat", base_url))
        .json(req)
        .send()
        .await?;
    Ok(())
}
}

Future Work

A native SDK could provide:

• Device registration/heartbeat management
• Token refresh handling
• Jiutian-specific features

See api/device.md for device protocol details.

Jiutian - Known Issues

🔴 Major Issues

JT-001: No SDK implementation

Description:
No Go or Rust SDK implementation exists for Jiutian API.

Impact: Users must use OpenAI-compatible SDK or implement HTTP calls directly.

Recommendation:

1. For chat completions: Use OpenAI SDK with custom base URL
2. For device features: Implement direct HTTP calls

🔵 Enhancements

JT-002: Native SDK desired

Description:
A native SDK would be useful for:

• Device registration/heartbeat protocols
• Token management
• Jiutian-specific error handling

Priority: Low - OpenAI SDK covers main use case.

JT-003: Device protocol documentation only

Description:
Device registration and heartbeat protocols are documented but not implemented.

Files affected:

• api/device.md

⚪ Notes

JT-004: OpenAI-compatible API

Description:
Jiutian chat API is OpenAI-compatible, so existing OpenAI SDKs work:

• Go: github.com/sashabaranov/go-openai
• Rust: async-openai
• Python: openai

Just set custom base URL and use Jiutian token.

JT-005: Access requirements

Description:
Jiutian API requires:

1. IP whitelist registration
2. Product ID from management platform
3. AI token (obtained via email application)

This is documented in api/README.md.

JT-006: China Mobile specific

Description:
This API is specific to China Mobile's terminal intelligent agent service management platform. May not be relevant for all users of this repository.

Summary

Overall: Documentation-only module without SDK implementation. OpenAI-compatible SDK recommended for chat functionality.

mqtt0

Overview

mqtt0 is a lightweight MQTT implementation focused on QoS 0. It provides both client and broker components with explicit control over authentication and ACL. The Go implementation is synchronous and net.Conn-based, while the Rust implementation is async (Tokio) with optional TLS/WebSocket transport features.

Design Goals

• Minimal MQTT feature set with strong QoS 0 focus
• Explicit ACL/auth hooks for connect/publish/subscribe
• Simple broker suitable for embedded or internal services
• Support MQTT 3.1.1 (v4) and MQTT 5.0 (v5)
• Provide transport flexibility (TCP/TLS/WebSocket)

Key Concepts

• Client: QoS 0 publish/subscribe, keepalive, protocol v4/v5
• Broker: connection lifecycle, ACL checks, topic routing
• Shared subscriptions: $share/{group}/{topic}
• Topic alias (v5): reduce bandwidth by reusing alias per client
• Transports: TCP/TLS/WebSocket based on URL scheme or feature flags

Components

• Client
• Broker
• Protocol parser/encoder
• Topic trie (subscription routing)
• Transport layer

Protocol and Transport Support

• MQTT 3.1.1 and MQTT 5.0 for client and broker
• TCP and TLS by default
• WebSocket/WSS when enabled (Rust feature flags)

Examples

• Go: use Connect, Subscribe, Publish, Recv
• Rust: use Client::connect, client.subscribe, client.publish, client.recv

Related Modules

• docs/lib/trie (topic routing)
• docs/lib/encoding (protocol helpers)

mqtt0 (Go)

Package Layout

• doc.go: high-level overview and usage examples
• client.go: QoS 0 client implementation
• broker.go: broker implementation with ACL hooks
• packet_v4.go, packet_v5.go, packet.go: protocol encode/decode
• listener.go, dialer.go: transport helpers
• trie.go: subscription routing

Public Interfaces

• ClientConfig: broker address, protocol version, TLS config, keepalive, etc.
• Client: Connect, Subscribe, Unsubscribe, Publish, Recv, Close
• Broker: Serve, ServeConn, ACL hooks, callbacks
• Authenticator: access control on connect/publish/subscribe
• Handler: callback for inbound broker messages
• Message, ProtocolVersion, QoS

Design Notes

• Single connection with separate read/write locks to guard concurrent access.
• Request/response operations (SUBSCRIBE/UNSUBSCRIBE) read from the same stream as inbound PUBLISH messages.
• Keepalive runs in a goroutine when AutoKeepalive is enabled.
• Shared subscriptions and topic aliasing are handled in the broker.

Transport

• URL-based address parsing: tcp://, tls://, ws://, wss://
• Dialer hook allows custom connection logic
• TLS config supported via ClientConfig.TLSConfig

Notable Behaviors

• QoS 0 only; no packet persistence or retransmission.
• Broker drops messages when per-client channel is full (non-blocking send).

mqtt0 (Rust)

Crate Layout

• lib.rs: public exports, crate overview
• client.rs: async QoS 0 client
• broker.rs: async broker with ACL hooks
• protocol.rs: MQTT encode/decode
• transport.rs: TCP/TLS/WebSocket transport abstraction
• trie.rs: subscription routing
• types.rs: public types and traits

Public Interfaces

• Client, ClientConfig: async connect/subscribe/publish/recv
• Broker, BrokerConfig, BrokerBuilder: broker setup and lifecycle
• Authenticator, Handler: ACL and message handling
• Message, ProtocolVersion, QoS
• TransportType, Transport (feature-gated TLS/WebSocket)

Design Notes

• Fully async, based on Tokio and mpsc channels.
• Builder pattern for broker configuration and hooks.
• Transport features are behind Cargo feature flags (TLS, WebSocket).

Differences vs Go

• Rust uses async traits for client/broker operations.
• TLS/WebSocket support is feature-gated.
• Broker construction encourages builder configuration.

mqtt0 - Known Issues

🟠 Major Issues

MQTT0-001: Go Client read path is not demultiplexed

File: go/pkg/mqtt0/client.go

Description: Subscribe, Unsubscribe, and other request/response operations read directly from the same stream as Recv(). If callers run Recv() concurrently with Subscribe() or Unsubscribe(), whichever acquires readMu first may consume packets that belong to the other operation, causing unexpected packet errors.

Impact: Hard-to-debug race between subscription changes and inbound message handling.

Suggestion: Introduce a single read loop with protocol demuxing, or document that Recv() must not run concurrently with subscribe/unsubscribe calls.

🟡 Minor Issues

MQTT0-002: Go Broker drops messages on backpressure

File: go/pkg/mqtt0/broker.go

Description: The broker uses a bounded channel for each client. When the channel is full, messages are dropped with a debug log.

Impact: Message loss under bursty load beyond QoS 0 expectations; may surprise users.

Suggestion: Document the drop behavior clearly or make buffer size configurable.

MQTT0-003: Rust WebSocket transport requires special handling

File: rust/mqtt0/src/transport.rs

Description: Transport::WebSocket implements AsyncRead/AsyncWrite by returning Unsupported errors. If code treats Transport uniformly, WebSocket connections will fail at runtime.

Impact: Surprising runtime errors for WebSocket clients if not handled explicitly.

Suggestion: Expose dedicated websocket read/write APIs or document the required handling.

Buffer Package

Thread-safe streaming buffer implementations for producer-consumer patterns.

Design Goals

1. Type-safe: Generic buffers for any element type (not limited to bytes)
2. Thread-safe: All operations support concurrent access
3. Blocking Semantics: Support blocking read/write with proper close mechanics
4. Flow Control: Different policies for handling full buffers

Buffer Types

Buffer (Growable)