Last updated 2026-05-17

Realtime sessions

Realtime is a different mode of interaction from chat, TTS, or STT in isolation. The conversation runs over a long-lived bidirectional connection, both sides emit events asynchronously, and the user expectation is sub-second latency — the latency budget for a voice-agent turn (user stops speaking → assistant starts speaking) is around 800 ms end-to-end. v0.4 models this with three distinct patterns, each suited to a different vendor surface.

Why realtime is its own thing

Compare the three interaction shapes:

• Chat — request in, stream of token deltas out, channel closes. One-shot.
• TTS — text in, stream of audio bytes out. One-shot, even when "streaming."
• STT — audio in, transcript segments out. One-shot for batch, but the live variant is genuinely duplex.
• Realtime — open a session, send and receive audio and text continuously, change configuration mid-flight, close when done. The conversation is the unit, not a single request.

The first three fit cleanly into a request → stream-of-results shape (CompletionStream, Speak, Transcribe). Realtime does not — it needs a long-lived handle with multiple send methods, a single receive channel, and explicit lifecycle. That's what Session on openairealtime is.

The provider landscape in v0.4

Realtime is fragmented across vendors. Each does a different subset of full-duplex:

The library models this as three patterns, not one. A single "Realtime" interface that tried to cover all of these would be so loose it would be useless — the semantics of "stream me a transcript" are nothing like "stream me audio while I append text" which are nothing like "let me talk to a model."

The three patterns

1. STT streaming — already in Transcribe

Transcriber.Transcribe with TranscribeRequest.Stream = true opens a WebSocket to the provider, pumps your audio in, and emits interim and final TranscriptSegments as the vendor returns them. The interface is the same one that handles batch transcription — only the wire transport changes.

p, _ := deepgram.New(llmrouter.WithAPIKey(os.Getenv("DEEPGRAM_API_KEY")))

stream, err := p.Transcribe(ctx, llmrouter.TranscribeRequest{
    Model:       "nova-3",
    Audio:       micReader,         // io.Reader fed by your mic capture
    AudioFormat: "audio/wav",
    Language:    "en-US",
    Stream:      true,              // <-- WebSocket path
})
if err != nil {
    log.Fatal(err)
}

for seg := range stream.Segments() {
    if seg.Final {
        log.Printf("FINAL: %s", seg.Text)
    } else {
        log.Printf("interim: %s", seg.Text)
    }
}

This pattern works because STT is naturally one-directional: audio in, text out. The library only needs a single receive channel.

2. TTS realtime context — SpeakRealtime

Cartesia and ElevenLabs both expose a WebSocket TTS surface where you can incrementally append text to a running synthesis without restarting it. This matters for voice agents: once the LLM emits its first sentence, you can start speaking it while the rest is still being generated. The audio stream stays coherent across appends — prosody and timing carry over.

The library returns two handles from SpeakRealtime: an *AudioStream for the bytes coming back, and a *RealtimeContext for the text going out.

p, _ := cartesia.New(llmrouter.WithAPIKey(os.Getenv("CARTESIA_API_KEY")))

stream, rtc, err := p.SpeakRealtime(ctx, llmrouter.SpeechRequest{
    Model:  "sonic-2",
    Voice:  "d46abd1d-2d02-43e8-819f-51fb652c1c61",
    Format: "pcm",
})
if err != nil {
    log.Fatal(err)
}
defer rtc.Close()

// Producer: as the LLM emits sentence-shaped chunks, append them.
go func() {
    for sentence := range llmSentences {
        if err := rtc.Append(sentence); err != nil {
            log.Printf("append: %v", err)
            return
        }
    }
    rtc.Finalize() // tell the server "no more text coming"
}()

// Consumer: play audio as it arrives.
for chunk := range stream.Chunks() {
    speaker.Play(chunk.Data)
}
if err := stream.Err(); err != nil {
    log.Fatal(err)
}

The signature is provider-specific because the semantics differ per vendor — Cartesia and ElevenLabs both support text-append, but their flush, finalize, and reconnect semantics are not identical. The library does not try to hide those — it gives each provider the same method name and the same handle shape, and documents the vendor-specific bits on each provider page.

3. Full-duplex Realtime — openairealtime.Session

OpenAI's Realtime API is its own thing: a model that takes audio (or text), generates audio (or text), and runs as a long-lived session with mid-flight config changes. It does not fit any of the other interfaces, so it lives in its own package with its own surface.

p, _ := openairealtime.New(llmrouter.WithAPIKey(os.Getenv("OPENAI_API_KEY")))

sess, err := p.Connect(ctx, openairealtime.SessionConfig{
    Model:             "gpt-4o-realtime-preview",
    Voice:             "alloy",
    Instructions:      "You are a friendly assistant.",
    InputAudioFormat:  "pcm16",
    OutputAudioFormat: "pcm16",
    Modalities:        []string{"audio", "text"},
})
if err != nil {
    log.Fatal(err)
}
defer sess.Close()

// Pump mic in.
go func() {
    for frame := range mic.Frames() {
        sess.SendAudio(frame)
    }
}()

// Consume server events.
for ev := range sess.Events() {
    switch ev.Type {
    case "response.audio.delta":
        speaker.Play(ev.AudioDelta)
    case "response.text.delta":
        log.Printf("text: %s", ev.Text)
    case "error":
        log.Fatalf("server: %v", ev.Error)
    }
}

See the OpenAI Realtime provider page for the full method reference and worked examples.

When to use which

Pick by what your application is doing, not by what's newest:

• Live captioning, dictation, meeting transcripts → STT streaming. Deepgram has the best latency and diarization story; Whisper is fine for batch but not for live.
• Voice agents where the LLM is the brain → pair a chat provider with SpeakRealtime on Cartesia or ElevenLabs. The LLM generates text incrementally, you append each chunk, the audio plays as it synthesises. Best when you want full control over the LLM choice (Claude, GPT-4, your own model).
• Voice agents where speed beats LLM choice → OpenAI Realtime. Lower end-to-end latency than the chat-plus-TTS pipeline because the model and the synthesiser are co-located, at the cost of being locked into gpt-4o-realtime.
• Phone-call use cases (8 kHz μ-law) → OpenAI Realtime supports g711_ulaw natively; Cartesia and ElevenLabs both support ulaw output. Deepgram handles telephony audio on the input side.

Lifecycle and cancellation

All three patterns share the same cancellation story as the rest of the library:

• Cancelling the ctx passed to Transcribe / SpeakRealtime / Connect closes the WebSocket cleanly.
• The receive channel (Segments() / Chunks() / Events()) closes after the producer finishes.
• Err() returns the terminal error after the channel closes. It returns nil on clean shutdown.
• Explicit Cancel() / Close() is idempotent and safe to call from any goroutine.

The same single-consumer rule applies — only one goroutine should range over the receive channel. Producer-side methods (SendAudio, SendText, Append) are safe to call concurrently from a separate goroutine, which is the entire point of full-duplex.

See also

• OpenAI Realtime provider — full-duplex session API.
• Deepgram provider — WebSocket live transcription.
• Cartesia provider — SpeakRealtime for low-latency TTS.
• ElevenLabs provider — SpeakRealtime for high-quality TTS.
• Audio concept — non-realtime TTS / STT.
• Streaming model — the underlying lifecycle.