Last updated 2026-05-17

Audio: TTS + STT

llmrouter models speech in two directions with two separate interfaces: Speaker for text-to-speech (TTS) and Transcriber for speech-to-text (STT). Each one is independent of Provider (chat) and of Embedder, so a provider package can implement any subset — OpenAI implements all three, ElevenLabs implements both audio interfaces but not chat, Deepgram implements only Transcriber, Cartesia implements only Speaker.

The two interfaces

type Speaker interface {
    Speak(ctx context.Context, req SpeechRequest) (*AudioStream, error)
}

type Transcriber interface {
    Transcribe(ctx context.Context, req TranscribeRequest) (*TranscriptStream, error)
}

Both return a stream handle with the same shape as Stream: a buffered channel, a terminal Err() error, a Cancel() hook, and full context.Context cancellation propagation. The lifecycle rules below are identical to chat streaming — if you've used CompletionStream, you already know how to use these.

Why streams for non-streaming providers. Some vendors return the whole audio file or the whole transcript as a single HTTP response. The library still wraps that response in an AudioStream / TranscriptStream so the consumer API is uniform — non-streaming responses simply arrive as a single chunk or a single Final=true segment.

Text-to-speech (Speaker)

SpeechRequest

type SpeechRequest struct {
    Model  string          // "tts-1", "tts-1-hd", "eleven_turbo_v2_5", "sonic-2"
    Input  string          // text to synthesise; required
    Voice  string          // "alloy", "echo", or a vendor voice id
    Format string          // "mp3" (default), "opus", "aac", "flac", "wav", "pcm", "ulaw"
    Speed  *float64        // 0.25-4.0; nil for default
    Stream bool            // request chunked audio
    Raw    json.RawMessage // optional byte passthrough; Model + Input are overlaid
}

Model: Provider-specific TTS model id. OpenAI: tts-1 / tts-1-hd. ElevenLabs: eleven_turbo_v2_5, eleven_multilingual_v2. Cartesia: sonic-2.
Voice: Voice identifier. OpenAI's seven canonical voices are alloy, echo, fable, onyx, nova, shimmer, coral. ElevenLabs and Cartesia use opaque voice IDs (often UUID-shaped) — see those providers' docs.
Format: Library-canonical audio format. The provider maps this to the upstream's native enum (OpenAI uses response_format directly; ElevenLabs uses its output_format codes such as mp3_44100_128). Empty defaults to mp3.
Speed: Playback-rate multiplier. 1.0 is normal, 0.5 is half speed, 2.0 is double. Providers clamp to their supported range.
Stream: When true, the upstream produces chunked audio (HTTP chunked transfer or SSE depending on vendor) and AudioStream.Chunks() emits many AudioChunks. When false, the entire audio arrives as a single chunk.
Raw: Byte passthrough escape hatch for fields the typed surface doesn't model — voice-stability parameters, style controls, language hints.

AudioStream

type AudioStream struct {
    ContentType string // "audio/mpeg", "audio/opus", "audio/wav", "audio/pcm", ...
    // unexported: chunks chan, cancel func, errMu chan, err error
}

func (s *AudioStream) Chunks() <-chan AudioChunk
func (s *AudioStream) Err() error
func (s *AudioStream) Cancel()

ContentType is populated by the producer before any chunks are emitted, so it is safe to read after the first chunk has arrived. Chunks() is single-consumer; Err() blocks until the producer finishes; Cancel() is idempotent.

AudioChunk

type AudioChunk struct {
    Data []byte // decoded audio bytes for this frame
    Raw  []byte // original wire bytes (often == Data)
}

Full TTS example

package main

import (
    "context"
    "io"
    "log"
    "os"

    "github.com/elloloop/llmrouter"
    "github.com/elloloop/llmrouter/providers/openai"
)

func main() {
    p, err := openai.New(llmrouter.WithAPIKey(os.Getenv("OPENAI_API_KEY")))
    if err != nil {
        log.Fatal(err)
    }

    stream, err := p.Speak(context.Background(), llmrouter.SpeechRequest{
        Model:  "tts-1",
        Input:  "The quick brown fox jumps over the lazy dog.",
        Voice:  "nova",
        Format: "mp3",
        Stream: true,
    })
    if err != nil {
        log.Fatal(err)
    }

    out, err := os.Create("out.mp3")
    if err != nil {
        log.Fatal(err)
    }
    defer out.Close()

    for chunk := range stream.Chunks() {
        if _, err := out.Write(chunk.Data); err != nil {
            stream.Cancel()
            log.Fatal(err)
        }
    }
    if err := stream.Err(); err != nil {
        log.Fatal(err)
    }
    log.Printf("wrote out.mp3 (%s)", stream.ContentType)
    _ = io.EOF // for the import if unused
}

Speech-to-text (Transcriber)

TranscribeRequest

type TranscribeRequest struct {
    Model          string          // "whisper-1", "nova-3", "scribe_v1"
    Audio          io.Reader       // binary audio source; required
    AudioFormat    string          // "audio/mpeg", "audio/wav", "audio/webm", ...
    Filename       string          // optional metadata for multipart-upload providers
    Language       string          // ISO-639-1 hint ("en", "fr", ...)
    Prompt         string          // context hint (Whisper, ElevenLabs)
    ResponseFormat string          // "json" (default), "text", "srt", "vtt", "verbose_json"
    Temperature    *float64        // Whisper sampling temperature
    Stream         bool            // request live transcription
    Raw            json.RawMessage // optional byte passthrough; Model is overlaid
}

Audio: Any io.Reader. The library reads until EOF. For retries, you must rewind yourself — the library does not buffer the input.
AudioFormat: Source MIME type. Some providers sniff (Whisper); others require an explicit format. When in doubt, set it.
Filename: Optional. Used by multipart-upload providers (OpenAI Whisper, Azure). Defaults to "audio" + an extension derived from AudioFormat when empty.
Language: ISO-639-1 hint. Speeds up detection and improves accuracy on Whisper-class models. Leave empty for automatic detection.
Prompt: Optional priming context. Whisper uses this as a glossary of proper nouns or domain terms.
ResponseFormat: json (default) → segments with text only. text → plain string in a single segment. srt / vtt → subtitle wire format, surfaced as one segment with the captions string in Text. verbose_json → per-word timing and confidence in Words. The library always emits TranscriptSegments regardless.
Stream: Live transcription. Providers that don't support streaming send a single final segment regardless of this field.

TranscriptStream

type TranscriptStream struct {
    // unexported: segments chan, cancel func, errMu chan, err error
}

func (s *TranscriptStream) Segments() <-chan TranscriptSegment
func (s *TranscriptStream) Err() error
func (s *TranscriptStream) Cancel()

TranscriptSegment

type TranscriptSegment struct {
    Type        string             // upstream event name; "" for transcript-content
    Text        string             // transcribed text for this segment
    Final       bool               // true on the terminal segment for an utterance
    SpeechFinal bool               // true when the provider detected end-of-speech
    Start, End  time.Duration      // timestamps relative to audio start
    Words       []TranscriptWord   // per-word timing (verbose_json, Deepgram, ElevenLabs)
    Confidence  float32            // per-segment confidence in [0,1]
    Raw         json.RawMessage    // original wire-format JSON
}

Streaming providers emit interim segments (Final=false) followed by exactly one final segment. Non-streaming providers emit one Final=true segment and close the channel.

Event-type discriminator (v0.6)

Streaming STT providers like Deepgram emit more than just transcript text on the wire — they also send endpointing, diarization, and session-metadata events. v0.6 surfaces these as TranscriptSegments with Type populated verbatim from the upstream event name.

Type == "" — transcript content. Use Text, Final, Words. This is the common case and the default zero value, so existing consumers keep working.
Type == "Results" — Deepgram's transcript event name when verbatim event types are enabled (some providers emit this even for transcript-content segments).
Type == "SpeechStarted" — Deepgram VAD: the user began speaking. Useful for barge-in detection in voice agents.
Type == "UtteranceEnd" — Deepgram VAD: the provider detected end-of-utterance based on silence padding.
Type == "Metadata" — session-level metadata (request id, model info, channel count). Usually ignored.

Consumers that don't care about non-transcript events should skip segments where Type != "":

for seg := range stream.Segments() {
    if seg.Type != "" && seg.Type != "Results" {
        continue // skip endpointing / metadata events
    }
    // ...handle transcript content
}

SpeechFinal vs Final (v0.6)

These look identical but they're not. The distinction matters for voice-agent turn-taking — pick the wrong one and your agent interrupts the user mid-sentence or sits waiting after they're clearly done.

Final — the upstream marks the current transcript window as final. Deepgram's is_final. Set when the provider has frozen the text it has so far; the next transcript window starts a new sliding buffer. Several Final=true segments can arrive within a single user utterance.
SpeechFinal — the provider detected end-of-speech for the current utterance. Deepgram's speech_final, surfaced as a separate bool. Set exactly once per utterance, at the natural turn boundary.

Voice agents should dispatch turn-taking off SpeechFinal, not Final — the latter fires too often and interrupts the user mid-thought.

for seg := range stream.Segments() {
    if seg.Type != "" && seg.Type != "Results" {
        continue
    }
    if seg.SpeechFinal {
        // Utterance is done — kick off the LLM turn.
        go runLLM(seg.Text)
    }
}

Migration note (v0.5 → v0.6). Deepgram callers that previously dispatched turn-taking off Final should switch to SpeechFinal. The Type field is new and defaults to "" — existing consumers that switch on Type get the same behaviour as before. v0.8 fixed a separate Deepgram decode bug (Issue #9) that caused some interim segments to surface with empty Text; bump to v0.8 or later.

Full STT example

package main

import (
    "context"
    "fmt"
    "log"
    "os"

    "github.com/elloloop/llmrouter"
    "github.com/elloloop/llmrouter/providers/openai"
)

func main() {
    p, err := openai.New(llmrouter.WithAPIKey(os.Getenv("OPENAI_API_KEY")))
    if err != nil {
        log.Fatal(err)
    }

    f, err := os.Open("meeting.mp3")
    if err != nil {
        log.Fatal(err)
    }
    defer f.Close()

    stream, err := p.Transcribe(context.Background(), llmrouter.TranscribeRequest{
        Model:          "whisper-1",
        Audio:          f,
        AudioFormat:    "audio/mpeg",
        Filename:       "meeting.mp3",
        Language:       "en",
        ResponseFormat: "verbose_json",
    })
    if err != nil {
        log.Fatal(err)
    }

    for seg := range stream.Segments() {
        fmt.Printf("[%s -> %s] %s\n", seg.Start, seg.End, seg.Text)
        for _, w := range seg.Words {
            fmt.Printf("  %-20s @ %s (%.2f)\n", w.Word, w.Start, w.Confidence)
        }
    }
    if err := stream.Err(); err != nil {
        log.Fatal(err)
    }
}

Cross-vendor format normalisation

Vendors disagree on how to spell audio formats. The library uses short library-canonical strings and translates per-provider on the way out, and surfaces canonical MIME types on AudioStream.ContentType on the way in.

Library `Format`	MIME type	OpenAI	ElevenLabs	Cartesia
`mp3`	`audio/mpeg`	`mp3`	`mp3_44100_128`	`mp3`
`opus`	`audio/opus`	`opus`	(unsupported)	`opus`
`flac`	`audio/flac`	`flac`	(unsupported)	(unsupported)
`wav`	`audio/wav`	`wav`	`pcm_16000`	`wav`
`pcm`	`audio/pcm`	`pcm`	`pcm_22050`	`raw`

Cancellation and timeouts

Cancelling the ctx passed to Speak or Transcribe closes the upstream HTTP request and unwinds the producer. stream.Cancel() is equivalent and idempotent. For long audio files, set a WithTimeout on the provider (default is 120 seconds) or wrap the ctx with context.WithTimeout.

Audio: TTS + STT

The two interfaces

Text-to-speech (Speaker)

SpeechRequest

AudioStream

AudioChunk

Full TTS example

Speech-to-text (Transcriber)

TranscribeRequest

TranscriptStream

TranscriptSegment

Event-type discriminator (v0.6)

SpeechFinal vs Final (v0.6)

Full STT example

Cross-vendor format normalisation

Cancellation and timeouts

See also