Last updated 2026-05-17

Gemini Live API

The geminilive provider wraps Google's Gemini Live WebSocket API — a full-duplex, session-based surface for low-latency voice agents on the Gemini family of models. The shape is deliberately a mirror of OpenAI Realtime so application code can pick a vendor by latency, voice, and price without restructuring around a different API.

Read the Realtime sessions concept page first if you have not seen the three realtime patterns llmrouter models (STT streaming, TTS realtime context, full-duplex Realtime). This page documents Google's take on the third.

Import path

import (
    "github.com/elloloop/llmrouter"
    "github.com/elloloop/llmrouter/providers/geminilive"
)

Endpoint and authentication

The provider targets the Gemini Live WebSocket endpoint: 

wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key={api_key}

Authentication is via API key in the query string — the same scheme as the synchronous Gemini (AI Studio) provider, not the OAuth/ADC flow used by Vertex. There is no Authorization header on the WebSocket upgrade; the key in the URL authorises the entire session.

Override the host with llmrouter.WithBaseURL if you are proxying or pointing at a regional endpoint. The query parameter is appended automatically from the API key supplied via llmrouter.WithAPIKey.

Construction

p, err := geminilive.New(
    llmrouter.WithAPIKey(os.Getenv("GEMINI_API_KEY")),
)
if err != nil {
    log.Fatal(err)
}

The standard options apply: WithAPIKey, WithBaseURL, WithHTTPClient (used for the WebSocket upgrade handshake), and WithTimeout (applied to the upgrade, not the session lifetime — once the socket is open, the session runs until Close or ctx cancellation).

Connect: opening a session

Provider.Connect is the entry point. It performs the WebSocket upgrade, sends an initial setup frame with the supplied SessionConfig, waits for the server's setup.complete acknowledgement, and returns a live *Session handle. 

func (p *Provider) Connect(ctx context.Context, cfg SessionConfig) (*Session, error)

SessionConfig

type SessionConfig struct {
    Model             string          // e.g. "models/gemini-2.0-flash-exp"
    Voice             string          // "Aoede", "Charon", "Fenrir", "Kore", "Puck"
    Instructions      string          // system-style guidance for the session
    OutputAudioFormat string          // "pcm16" (default — 24 kHz)
    Modalities        []string        // {"TEXT", "AUDIO"} by default
    Temperature       *float64        // 0.0-2.0
    TopP              *float64        // nucleus sampling
    Tools             []llmrouter.ToolDef // function-calling definitions
    Raw               json.RawMessage     // byte passthrough; typed fields overlay
}
Model
Gemini Live model id. Required. Goes into the setup.model field on the initial frame. As of writing the live-capable models live under the models/gemini-2.0-flash-exp name; check the Google docs for the current GA model id.
Voice
One of the five Live voices: Aoede, Charon, Fenrir, Kore, Puck. Voice is a session property — changing it requires a fresh Connect; Gemini Live does not yet support mid-session voice swaps.
Instructions
Free-form system-style guidance. Translated to system_instruction on the setup frame. Plays the same role as the system message in chat but lives on the session.
OutputAudioFormat
Currently fixed to pcm16 (16-bit signed PCM at 24 kHz mono). Listed as a field for forward compatibility — the only accepted value today is the default.
Modalities
Subset of {"TEXT", "AUDIO"}. Leave empty for both. Note the spellings are uppercase on the wire (unlike OpenAI Realtime). The library accepts either case from the caller and normalises before sending.
Temperature / TopP
Sampling controls passed through to generation_config. Pointers so that the zero value means “use the model default” rather than “0.0”.
Tools
Optional function-calling definitions. Uses the same llmrouter.ToolDef shape as chat, translated to Gemini's tools.function_declarations schema. See the tool use section below.
Raw
Byte passthrough for fields the typed surface does not model (e.g. realtime_input_config, speech_config.language_code). Typed fields overlay Raw after marshalling, so setting both is safe.

Session methods

type Session struct {
    // unexported: conn, events chan, err error, ...
}


func (s *Session) SendText(text string) error
func (s *Session) SendAudio(pcm []byte) error
func (s *Session) SendToolResult(toolCallID string, output any) error
func (s *Session) Close() error
func (s *Session) Events() <-chan SessionEvent
func (s *Session) Err() error
SendText
Send a text turn as client_content with turn_complete: true. Unlike OpenAI Realtime, there is no separate “create response” step — the turn_complete flag tells the server to start generating.
SendAudio
Append a PCM audio chunk to the realtime_input stream. The bytes must be 16-bit signed PCM at 16 kHz mono — note this is different from the output sample rate (24 kHz). Stream microphone capture by calling SendAudio on each frame as it arrives.
SendToolResult
Reply to a tool call from the server with the function's output. The output is marshalled to JSON and sent as tool_response.function_responses[].response keyed by toolCallID.
Close
Close the WebSocket cleanly. Idempotent. Cancelling the ctx passed to Connect is equivalent.
Events
Receive channel of SessionEvent values. Single consumer. Closed when the session ends, after which Err() returns the terminal error (nil on clean close).
Err
Terminal error. Blocks until the session ends. Returns nil on clean close, *llmrouter.ErrUpstream for a server-reported error, or a wrapped net / context error for transport failures.

SessionEvent

type SessionEvent struct {
    Type               string          // "setup.complete", "server.text", "server.audio", "server.tool_call", "server.turn_complete", "error"
    Text               string          // populated for server.text
    AudioDelta         []byte          // populated for server.audio (raw PCM bytes)
    AudioMime          string          // e.g. "audio/pcm;rate=24000"
    ToolCallID         string          // populated for server.tool_call
    ToolName           string          // populated for server.tool_call
    ToolArgs           json.RawMessage // populated for server.tool_call — full args object
    Error              *llmrouter.ErrUpstream // populated for "error"
    Raw                json.RawMessage // original wire JSON
}

Gemini Live events do not have stable string names on the wire the way OpenAI Realtime does — the server sends one of a handful of typed message shapes (setupComplete, serverContent, toolCall, ...) and the library normalises them to the Type values listed below.

Event Type Typed fields populated Meaning
setup.complete — Server acknowledged the initial setup frame; safe to start sending input.
server.text Text Incremental text from the model.
server.audio AudioDelta, AudioMime Incremental audio bytes (24 kHz PCM by default).
server.tool_call ToolCallID, ToolName, ToolArgs Server is asking the client to execute a registered tool. Reply with SendToolResult.
server.turn_complete — End of one server turn; the session continues for the next user turn.
error Error Server-side error; session usually ends after this.

Example: text in, audio out

package main


import (
    "context"
    "log"
    "os"


    "github.com/elloloop/llmrouter"
    "github.com/elloloop/llmrouter/providers/geminilive"
)


func main() {
    p, err := geminilive.New(llmrouter.WithAPIKey(os.Getenv("GEMINI_API_KEY")))
    if err != nil {
        log.Fatal(err)
    }


    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()


    sess, err := p.Connect(ctx, geminilive.SessionConfig{
        Model:        "models/gemini-2.0-flash-exp",
        Voice:        "Aoede",
        Instructions: "You are a concise assistant. Reply in one short sentence.",
        Modalities:   []string{"AUDIO", "TEXT"},
    })
    if err != nil {
        log.Fatal(err)
    }
    defer sess.Close()


    if err := sess.SendText("Say hello in five words and play it back."); err != nil {
        log.Fatal(err)
    }


    out, _ := os.Create("hello.pcm")
    defer out.Close()


    for ev := range sess.Events() {
        switch ev.Type {
        case "setup.complete":
            // ready
        case "server.text":
            log.Printf("text: %s", ev.Text)
        case "server.audio":
            out.Write(ev.AudioDelta)
        case "server.turn_complete":
            return
        case "error":
            log.Fatalf("server error: %v", ev.Error)
        }
    }
    if err := sess.Err(); err != nil {
        log.Fatal(err)
    }
}

The resulting hello.pcm file is 24 kHz signed 16-bit mono PCM — play it with ffplay -f s16le -ar 24000 -ac 1 hello.pcm or wrap it in a WAV header.

Example: audio in, audio out

Voice agents pump microphone frames in via SendAudio while the consumer goroutine drains server events and plays audio deltas. Mic capture is platform-specific — the sketch below shows the library calls; substitute your audio driver of choice (malgo, portaudio, etc.) for the pseudocode parts.

Pay attention to the sample-rate asymmetry: input must be 16 kHz PCM, output arrives as 24 kHz PCM. Resample on the way in (most mic libraries default to 48 kHz) and on the way out for your playback device. 

package main


import (
    "context"
    "io"
    "log"
    "os"


    "github.com/elloloop/llmrouter"
    "github.com/elloloop/llmrouter/providers/geminilive"
)


// Pseudocode: replace with your audio capture / playback of choice.
type mic interface {
    Frames() <-chan []byte // 20ms of 16kHz s16le PCM per frame
    io.Closer
}
type speaker interface {
    Play(pcm []byte) error // expects 24kHz s16le PCM
    io.Closer
}


func openMic16k() (mic, error)         { /* malgo / portaudio / ... */ return nil, nil }
func openSpeaker24k() (speaker, error) { /* malgo / portaudio / ... */ return nil, nil }


func main() {
    p, err := geminilive.New(llmrouter.WithAPIKey(os.Getenv("GEMINI_API_KEY")))
    if err != nil {
        log.Fatal(err)
    }


    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()


    sess, err := p.Connect(ctx, geminilive.SessionConfig{
        Model:        "models/gemini-2.0-flash-exp",
        Voice:        "Puck",
        Instructions: "You are a friendly voice assistant. Reply in short sentences.",
        Modalities:   []string{"AUDIO", "TEXT"},
    })
    if err != nil {
        log.Fatal(err)
    }
    defer sess.Close()


    m, err := openMic16k()
    if err != nil {
        log.Fatal(err)
    }
    defer m.Close()


    sp, err := openSpeaker24k()
    if err != nil {
        log.Fatal(err)
    }
    defer sp.Close()


    // Pump microphone frames into the session.
    go func() {
        for frame := range m.Frames() {
            if err := sess.SendAudio(frame); err != nil {
                log.Printf("send audio: %v", err)
                return
            }
        }
    }()


    // Consume server events and play audio deltas as they arrive.
    for ev := range sess.Events() {
        switch ev.Type {
        case "server.audio":
            if err := sp.Play(ev.AudioDelta); err != nil {
                log.Printf("play: %v", err)
            }
        case "server.text":
            log.Printf("assistant: %s", ev.Text)
        case "error":
            log.Fatalf("server error: %v", ev.Error)
        }
    }
    if err := sess.Err(); err != nil {
        log.Fatal(err)
    }
}

Gemini Live uses server-side voice activity detection by default — the model picks up end-of-utterance automatically from the audio stream, so you do not need a manual “commit” step the way OpenAI Realtime has when VAD is disabled.

Example: tool use

Declare tools on the session, watch for server.tool_call events, execute the function, and reply with SendToolResult. The same llmrouter.ToolDef type used by chat works here — no Gemini-specific tool schema in your application code. 

sess, err := p.Connect(ctx, geminilive.SessionConfig{
    Model:        "models/gemini-2.0-flash-exp",
    Voice:        "Kore",
    Instructions: "You are a weather assistant. Use the get_weather tool when asked.",
    Modalities:   []string{"AUDIO", "TEXT"},
    Tools: []llmrouter.ToolDef{{
        Type: "function",
        Function: llmrouter.FunctionDef{
            Name:        "get_weather",
            Description: "Look up current weather for a city.",
            Parameters: json.RawMessage(`{
                "type": "object",
                "properties": {
                    "city": {"type": "string"}
                },
                "required": ["city"]
            }`),
        },
    }},
})
if err != nil { log.Fatal(err) }
defer sess.Close()


_ = sess.SendText("What is the weather in Berlin?")


for ev := range sess.Events() {
    switch ev.Type {
    case "server.tool_call":
        // Execute the tool — args is whatever the schema declared.
        var args struct{ City string `json:"city"` }
        _ = json.Unmarshal(ev.ToolArgs, &args)


        result := map[string]any{
            "city":        args.City,
            "temperature": "18C",
            "conditions":  "cloudy",
        }
        if err := sess.SendToolResult(ev.ToolCallID, result); err != nil {
            log.Fatal(err)
        }
    case "server.text":
        log.Printf("assistant: %s", ev.Text)
    case "server.turn_complete":
        return
    case "error":
        log.Fatalf("server error: %v", ev.Error)
    }
}

Error handling

Three distinct error paths, identical in shape to OpenAI Realtime:

  • Connect-time errors — HTTP-upgrade failures (bad API key, network) return from Connect as *llmrouter.ErrUpstream (for non-2xx upgrade responses) or a wrapped net error.
  • In-session server errors — surface as a SessionEvent with Type == "error" and a populated Error field. Gemini Live typically closes the socket after an error; check Err() for the terminal status.
  • Transport errors — abrupt socket close, context cancellation, deadline exceeded. The events channel closes and Err() returns the underlying error.

Caveats and current limits

  • Beta API on the Google side. Gemini Live is still labelled experimental — the wire shape can shift between Google SDK releases. The geminilive wrapper pins to a snapshot and may need updates when Google iterates.
  • Audio sample rates are asymmetric. Input is 16 kHz PCM, output is 24 kHz PCM. Get your resampler right or you will hear chipmunk / molasses voices.
  • One session per connection. Each Connect opens a new socket. There is no multiplexing — for parallel sessions, call Connect multiple times.
  • Voice cannot change mid-session. To switch voices, close and reconnect with a new SessionConfig.
  • No reconnect. If the socket dies, the session is dead — your application is responsible for reconnecting and replaying conversation state.
  • Input audio format is fixed to 16-bit signed PCM at 16 kHz. Opus, MP3, and other codecs are not supported on the input side.

Gemini Live vs OpenAI Realtime

Both providers expose the same Connect / SendText / SendAudio / SendToolResult / Events / Close shape, so the choice is about voice quality, latency, model behaviour, and price — not API ergonomics. Some rules of thumb:

  • Latency profile. OpenAI Realtime tends to have lower first-audio latency in benchmarks; Gemini Live is catching up and is often cheaper per minute.
  • Voices. OpenAI has 8 voices today (alloy, echo, ...); Gemini has 5 (Aoede, Charon, Fenrir, Kore, Puck). Pick by ear, not spec sheet.
  • Multimodal input. Gemini's strength is vision + audio — if the agent needs to look at a screenshot or a frame from a video stream, Gemini Live's realtime_input accepts inline image bytes alongside audio in a way OpenAI Realtime does not.
  • Stability. OpenAI Realtime is closer to GA; Gemini Live is still flagged experimental and gets shape changes from Google more often.

See also