Last updated 2026-05-17

OpenAI Realtime API

The openairealtime provider wraps OpenAI's gpt-4o-realtime WebSocket API — a full-duplex, session-based surface for low-latency voice agents. It is deliberately separate from the OpenAI provider because the shape is fundamentally different: a long-lived session with bidirectional audio and text events, not a request-response exchange.

Read the Realtime sessions concept page first if you have not seen the three realtime patterns llmrouter models (STT streaming, TTS realtime context, full-duplex Realtime). This page documents the third.

Import path

import (
    "github.com/elloloop/llmrouter"
    "github.com/elloloop/llmrouter/providers/openairealtime"
)

Endpoint and authentication

The provider targets wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview by default. The model id is part of the URL, not the request body — it is taken from SessionConfig.Model at Connect time. Override the host with llmrouter.WithBaseURL if you are proxying or targeting an Azure deployment.

Authentication uses two headers on the WebSocket upgrade:

Authorization: Bearer {api_key}
OpenAI-Beta: realtime=v1

Both are set automatically from the API key. There is no per-message auth — the bearer token authorises the entire session.

Construction

p, err := openairealtime.New(
    llmrouter.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
)
if err != nil {
    log.Fatal(err)
}

The same options that work on other providers work here: WithAPIKey, WithBaseURL, WithHTTPClient (used for the WebSocket upgrade handshake), and WithTimeout (applied to the upgrade, not the session lifetime — once the socket is open, the session runs until Close or ctx cancellation).

Connect: opening a session

Provider.Connect is the entry point. It performs the WebSocket upgrade, sends an initial session.update with the supplied SessionConfig, and returns a live *Session handle.

func (p *Provider) Connect(ctx context.Context, cfg SessionConfig) (*Session, error)

SessionConfig

type SessionConfig struct {
    Model             string          // "gpt-4o-realtime-preview"
    Voice             string          // "alloy", "echo", "shimmer", ...
    Instructions      string          // system-style guidance for the session
    InputAudioFormat  string          // "pcm16" (default), "g711_ulaw", "g711_alaw"
    OutputAudioFormat string          // "pcm16" (default), "g711_ulaw", "g711_alaw"
    Modalities        []string        // {"text", "audio"} by default
    Temperature       *float64        // 0.6-1.2 typical
    Raw               json.RawMessage // byte passthrough; typed fields overlay
}

Model: Realtime model id. Required. Goes into the WebSocket URL query, not the message body.
Voice: One of OpenAI's realtime voices (alloy, echo, shimmer, ash, ballad, coral, sage, verse). Voice is a session property — changing it mid-session requires UpdateSession.
Instructions: Free-form system-style guidance. Plays the role of the system message in the chat API but lives on the session, not on each request.
InputAudioFormat / OutputAudioFormat: Supported values: pcm16 (16-bit signed PCM at 24 kHz mono — the default and the highest quality), g711_ulaw and g711_alaw (8 kHz telephony codecs). Mismatched formats are a server error, not a client-side validation failure.
Modalities: Subset of {"text", "audio"}. Leave empty for both. Audio-only sessions still receive response.text.delta events for transcribed audio.
Temperature: Sampling temperature, in the same range as chat. The realtime model is noticeably more sensitive to temperature than the synchronous models.
Raw: Byte passthrough for fields the typed surface does not model (turn detection config, tool definitions in v0.5, etc.). The typed fields overlay Raw after marshalling, so setting both is safe — the typed values win.

Session methods

type Session struct {
    // unexported: conn, events chan, errMu chan, err error, ...
}

func (s *Session) SendText(text string) error
func (s *Session) SendAudio(pcm []byte) error
func (s *Session) Commit() error
func (s *Session) CreateResponse() error
func (s *Session) UpdateSession(cfg SessionConfig) error
func (s *Session) Close() error
func (s *Session) Events() <-chan SessionEvent
func (s *Session) Err() error

SendText: Append a text item to the conversation buffer. Does not trigger generation by itself — call CreateResponse after the last SendText / SendAudio / Commit.
SendAudio: Append a PCM audio buffer to the input audio stream. The bytes must match SessionConfig.InputAudioFormat. Stream microphone capture by calling SendAudio on each frame as it arrives — typical frame sizes are 20-40 ms of 24 kHz PCM (960-1920 samples = 1920-3840 bytes).
Commit: Signal end-of-utterance on the input audio buffer. If server VAD is enabled (the default), commits happen automatically and this is a no-op; with VAD disabled you must commit explicitly before CreateResponse.
CreateResponse: Tell the server to generate a response from the current conversation state. The response streams back as SessionEvents on Events().
UpdateSession: Replace the session configuration mid-flight. Typically used to change Voice or Instructions; changing Model is not supported.
Close: Close the WebSocket cleanly. Idempotent. Cancelling the ctx passed to Connect is equivalent.
Events: Receive channel of SessionEvent values. Single consumer. Closed when the session ends, after which Err() returns the terminal error (nil on clean close).
Err: Terminal error. Blocks until the session ends. Returns nil on a clean close, *llmrouter.ErrUpstream for a server-reported error event, or a wrapped net / context error for transport failures.

SessionEvent

type SessionEvent struct {
    Type       string          // OpenAI event type, e.g. "response.text.delta"
    Text       string          // populated for text deltas
    AudioDelta []byte          // populated for audio deltas (decoded PCM bytes)
    Error      *llmrouter.ErrUpstream // populated for "error" events
    Raw        json.RawMessage // original wire JSON
}

The Type field is the OpenAI event name verbatim. The library populates the typed convenience fields for the handful of events most callers actually consume; everything else is available on Raw.

Event `Type`	Typed field populated	Meaning
`response.text.delta`	`Text`	Incremental text from the assistant.
`response.audio.delta`	`AudioDelta`	Incremental audio bytes (PCM in `OutputAudioFormat`).
`response.audio_transcript.delta`	`Text`	Server-side transcript of the assistant's audio output.
`response.done`	—	End of one response; the session continues.
`error`	`Error`	Server-side error; session usually ends after this.
`session.created`, `session.updated`, etc.	—	Lifecycle / acknowledgement; inspect via `Raw`.

Example: text in, audio out

package main

import (
    "context"
    "log"
    "os"

    "github.com/elloloop/llmrouter"
    "github.com/elloloop/llmrouter/providers/openairealtime"
)

func main() {
    p, err := openairealtime.New(llmrouter.WithAPIKey(os.Getenv("OPENAI_API_KEY")))
    if err != nil {
        log.Fatal(err)
    }

    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    sess, err := p.Connect(ctx, openairealtime.SessionConfig{
        Model:             "gpt-4o-realtime-preview",
        Voice:             "alloy",
        Instructions:      "You are a concise assistant.",
        OutputAudioFormat: "pcm16",
        Modalities:        []string{"text", "audio"},
    })
    if err != nil {
        log.Fatal(err)
    }
    defer sess.Close()

    if err := sess.SendText("Say hello in five words and play it back."); err != nil {
        log.Fatal(err)
    }
    if err := sess.CreateResponse(); err != nil {
        log.Fatal(err)
    }

    out, _ := os.Create("hello.pcm")
    defer out.Close()

    for ev := range sess.Events() {
        switch ev.Type {
        case "response.text.delta":
            log.Printf("text: %s", ev.Text)
        case "response.audio.delta":
            out.Write(ev.AudioDelta)
        case "response.done":
            return
        case "error":
            log.Fatalf("server error: %v", ev.Error)
        }
    }
    if err := sess.Err(); err != nil {
        log.Fatal(err)
    }
}

The resulting hello.pcm file is 24 kHz signed 16-bit mono PCM — play it with ffplay -f s16le -ar 24000 -ac 1 hello.pcm or wrap it in a WAV header.

Example: audio in, audio out

For voice-agent use cases you want mic-in / speaker-out. The llmrouter side is identical to the text example above plus a producer goroutine that calls SendAudio for each microphone frame. The mic capture is platform-specific — the sketch below shows the library calls; substitute your audio driver of choice (malgo, portaudio, etc.) for the pseudocode parts.

package main

import (
    "context"
    "io"
    "log"
    "os"

    "github.com/elloloop/llmrouter"
    "github.com/elloloop/llmrouter/providers/openairealtime"
)

// Pseudocode: replace with your audio capture / playback of choice.
type mic interface {
    Frames() <-chan []byte // 20ms of 24kHz s16le PCM per frame
    io.Closer
}
type speaker interface {
    Play(pcm []byte) error
    io.Closer
}

func openMic() (mic, error)         { /* malgo / portaudio / ... */ return nil, nil }
func openSpeaker() (speaker, error) { /* malgo / portaudio / ... */ return nil, nil }

func main() {
    p, err := openairealtime.New(llmrouter.WithAPIKey(os.Getenv("OPENAI_API_KEY")))
    if err != nil {
        log.Fatal(err)
    }

    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    sess, err := p.Connect(ctx, openairealtime.SessionConfig{
        Model:             "gpt-4o-realtime-preview",
        Voice:             "alloy",
        Instructions:      "You are a friendly voice assistant. Reply in short sentences.",
        InputAudioFormat:  "pcm16",
        OutputAudioFormat: "pcm16",
        Modalities:        []string{"audio", "text"},
    })
    if err != nil {
        log.Fatal(err)
    }
    defer sess.Close()

    m, err := openMic()
    if err != nil {
        log.Fatal(err)
    }
    defer m.Close()

    sp, err := openSpeaker()
    if err != nil {
        log.Fatal(err)
    }
    defer sp.Close()

    // Pump microphone frames into the session.
    go func() {
        for frame := range m.Frames() {
            if err := sess.SendAudio(frame); err != nil {
                log.Printf("send audio: %v", err)
                return
            }
        }
    }()

    // Consume server events and play audio deltas as they arrive.
    for ev := range sess.Events() {
        switch ev.Type {
        case "response.audio.delta":
            if err := sp.Play(ev.AudioDelta); err != nil {
                log.Printf("play: %v", err)
            }
        case "response.audio_transcript.delta":
            log.Printf("assistant: %s", ev.Text)
        case "error":
            log.Fatalf("server error: %v", ev.Error)
        }
    }
    if err := sess.Err(); err != nil {
        log.Fatal(err)
    }
}

With server-side voice activity detection enabled (the default), the model picks up end-of-utterance automatically — you do not need to call Commit or CreateResponse after each user turn. To disable server VAD and drive turn-taking yourself, set {"turn_detection": null} on SessionConfig.Raw.

Error handling

Three distinct error paths:

Connect-time errors — HTTP-upgrade failures (bad API key, network) return from Connect as *llmrouter.ErrUpstream (for non-2xx upgrade responses) or a wrapped net error.
In-session server errors — surface as a SessionEvent with Type == "error" and a populated Error field. The session typically ends after an error event; check Err() for the terminal status.
Transport errors — abrupt socket close, context cancellation, deadline exceeded. The events channel closes and Err() returns the underlying error.

Caveats and current limits

No tool use in v0.4. Function-calling events flow through Raw only. Typed tool surfaces on Realtime are on the v0.5 roadmap.
One session per connection. Each Connect opens a new socket. There is no multiplexing — for parallel sessions, call Connect multiple times.
Instructions live on the session, not on each turn. To change the system prompt mid-conversation, call UpdateSession with new Instructions.
Audio formats are limited to pcm16, g711_ulaw, and g711_alaw by OpenAI. Opus and MP3 are not supported on the input or output side.
No reconnect. If the socket dies, the session is dead — your application is responsible for reconnecting and replaying conversation state.

Roadmap

v0.5 — typed tool use on SessionEvent, function-calling end-to-end with the same ToolDef / ToolCall shapes used by chat.
v0.5 — streaming UpdateSession deltas so config changes don't require a full re-marshal.
v0.5 — Gemini Live wrapped through the same openairealtime-shaped surface for cross-vendor parity.