Last updated 2026-05-17

Roadmap

llmrouter is pre-1.0. This page is the public plan for what has shipped, what is being built next, and — equally important — what is intentionally not part of the library. The roadmap is best-effort, not a contract; dates are omitted on purpose because most of the work is driven by a small team.

Versions follow semantic versioning. While we are pre-1.0, expect minor breakages between minor versions. Patch versions never break the API.

Shipped

v0.8.0 — The Router, Vertex AI Anthropic, Azure Serverless, Deepgram fix

The headline release for the "magic model × platform" story. A new router/ package decouples the model vendor from the hosting platform — callers ask for "Claude on Bedrock" or "Llama on Groq" and the router resolves to the right underlying provider. Two new provider packages (vertexanthropic, azureserverless) complete the Foundry / Vertex Claude story and unlock the Llama / Mistral / Cohere / Phi / DeepSeek catalogue on Azure.

  • New router/ package — router.Resolve(Request{Model, Platform, Credentials}) returns a working llmrouter.Provider. Twelve platforms (Direct / Bedrock / Vertex / Azure / OpenRouter / Together / Groq / Fireworks / Cerebras / DeepSeek / Perplexity / xAI) × nine model families (OpenAI / Anthropic / Llama / Mistral / Cohere / Gemini / Grok / DeepSeek / Other). PlatformAuto picks the first platform with credentials present; ResolveFromEnv is the one-line shortcut. See the Router concept and API reference.
  • New providers/vertexanthropic — Claude on Google Vertex AI Model Garden. Pure HTTP, no google.golang.org/genai dependency. Vertex-specific quirk handled: the anthropic_version field goes in the body (not a header), and the model id uses @-versioning. See the provider page.
  • New providers/azureserverless — catch-all for non-OpenAI / non-Anthropic models on Azure AI Foundry's Serverless API (MaaS) deployment mode. Llama, Mistral, Cohere, Phi, Jais, Nemotron, DeepSeek. Two URL shapes (deployment-scoped vs hub-scoped), api-key XOR AAD bearer auth, no api-version query. See the provider page.
  • Deepgram decode fix — Issue #9: some interim segments were surfacing with empty Text on certain Nova-3 streams. The decoder now handles the affected event shape.
  • Test suite — grew by ~127 new regression tests covering the router resolution matrix, the Vertex Anthropic body / SSE translation, the Azure Serverless URL composition and auth XOR, and the Deepgram decode regression.

v0.7.0 — Azure Foundry Anthropic, README polish, 10 examples, hardened CI

The "make it usable" release. A new providers/azureanthropic package fills the Claude gap on Azure AI Foundry, the README is rebuilt as the canonical install + capability matrix surface, ten end-to-end examples land under examples/, a CONTRIBUTING guide gets added, and CI is hardened with race-clean test runs and static-analysis gates.

  • New providers/azureanthropic — Claude on Azure AI Foundry. Two URL variants (deployment-scoped vs /openai/v1/messages), api-key XOR AAD bearer auth, mandatory WithAPIVersion. The body and SSE event stream are byte-identical to direct Anthropic — every capability of the direct provider works (typed tool use, thinking deltas, prompt cache, multipart, structured outputs, mid-stream errors). See the provider page.
  • Polished README — the canonical capability matrix moves to the README so pkg.go.dev visitors see it without clicking through. The 22-provider matrix is the single source of truth shared with the landing page.
  • 10 examples under examples/ — end-to-end runnable programs covering chat (OpenAI, Anthropic), embeddings, TTS, STT, realtime, rerank, structured outputs, and the new Azure Anthropic + Vertex Anthropic flows. Drop-in for new users; run with go run against the right env vars.
  • CONTRIBUTING.md — explicit guidance on writing new providers, the byte-passthrough contract, and the test conventions.
  • Hardened CI — race-clean test runs on every PR, go vet + staticcheck gates, and coverage tracking surfaced on the README badge.

v0.6.0 — STT event-type discriminator + SpeechFinal

A focused STT release. Streaming transcribers now distinguish transcript content from endpointing / metadata events, and voice-agent callers get a clean turn-taking signal that's independent of the per-window is_final marker.

  • New TranscriptSegment.Type field — populated verbatim from the upstream event name. Empty string is the default and means "transcript content"; non-empty values surface the upstream's discriminator (Deepgram: "Results", "SpeechStarted", "UtteranceEnd", "Metadata"). Existing consumers that don't switch on Type keep working — they receive transcript content as before.
  • New TranscriptSegment.SpeechFinal field — distinct from Final. Final is the upstream's per-window is_final; SpeechFinal is the end-of-utterance signal from VAD. Voice agents should dispatch turn-taking off SpeechFinal, not Final. See the audio concept.
  • Migration note — Deepgram callers that previously dispatched turn-taking off Final should switch to SpeechFinal; the previous behaviour was effectively wrong (it fired mid-utterance on long sentences).

v0.5.0 — Gemini Live, Realtime tool use, structured outputs, rerank

The function-calling and RAG-refinement release. Tool use lands on Realtime sessions, structured outputs become a first-class cross-vendor field, Google's full-duplex Gemini Live joins the realtime tier, and a brand-new Reranker interface closes the second-stage RAG gap.

  • New providers/geminilive package — wraps Google's Gemini Live WebSocket API. Mirrors the openairealtime shape exactly: Provider.Connect returning a *Session with SendText, SendAudio, SendToolResult, Close, Events(), and Err(). Application code that already drives OpenAI Realtime ports across with an import swap. See the Gemini Live provider page.
  • Tool use over OpenAI Realtime — SessionConfig.Tools and SessionConfig.ToolChoice accept the same llmrouter.ToolDef shapes as chat. New SessionEvent fields (ToolCallID, ToolName, ToolArgumentsDelta, ToolArguments) surface response.function_call_arguments.{delta,done} events as typed values. New Session.SendToolResult(ctx, toolCallID, output) method closes the function-calling loop without dropping to Raw.
  • Structured outputs (JSON Schema) — new root llmrouter.ResponseSchema type and ChatRequest.ResponseSchema *ResponseSchema field. OpenAI translates to response_format.json_schema. Anthropic translates to forced tool-use with a synthetic tool whose input_schema matches the requested schema. Vertex and Gemini translate to GenerateContentConfig.ResponseMIMEType + ResponseSchema via the genai SDK. See the Structured outputs concept.
  • Rerank — new root llmrouter.Reranker interface plus RerankRequest, RerankResponse, and RerankResult types. Implemented on Cohere (rerank-v3.5), Voyage (rerank-2), and Together (Salesforce/Llama-Rank-V1). Best-in- class second-stage filter for RAG pipelines. See the Rerank concept and API reference.
  • Embeddings extended to Fireworks and DeepSeek — thin delegation through the OpenAI-compatible provider, completing the OpenAI-compat embedding coverage across the ten verified vendors.
  • Provider capability matrix — Gemini Live row added; Rerank column added (Cohere, Voyage, Together); Embed column gains Fireworks and DeepSeek; Structured outputs column added (OpenAI, Anthropic via tool-use coercion, Vertex, Gemini).
  • Test suite — grew from 1,742 to ~2,200 subtests covering the Gemini Live state machine, Realtime tool-call event translation, structured-output schema translation on three providers, and rerank request/response shapes on three providers.

v0.4.0 — Realtime, WebSocket streaming, tool-result messages

llmrouter moves into the realtime tier. Existing audio providers gain WebSocket transports, a brand-new top-level surface wraps OpenAI's full-duplex Realtime API, and the typed chat surface grows a proper tool-result message type.

  • WebSocket live transcription for Deepgram — Transcribe with Stream=true now activates the WebSocket path against wss://api.deepgram.com/v1/listen. Interim and final segments stream back with per-word timing.
  • Cartesia SpeakRealtime — new method returning an *AudioStream plus a *RealtimeContext for multi-turn TTS: append text incrementally, finalize, close.
  • ElevenLabs SpeakRealtime — same signature, WebSocket TTS via /stream-input with the same multi-turn context surface as Cartesia.
  • New providers/openairealtime package — wraps OpenAI's gpt-4o-realtime WebSocket API. Bidirectional audio and text with a session-based API (Connect, SendText, SendAudio, Commit, CreateResponse, UpdateSession, Close) that sits alongside Provider, Speaker, and Transcriber. See the OpenAI Realtime provider page.
  • First-class tool-result message type — llmrouter.ToolResultMessage(toolCallID, content) helper plus new Message.ToolCallID and Message.Name fields. OpenAI carries the typed shape directly; Anthropic translates Role: "tool" to a user-role message containing a tool_result block — same caller code, two wire formats.
  • Anthropic recommended embeddings shim — providers/anthropic.NewRecommendedEmbedder(voyageAPIKey, opts...) returns a Voyage-backed Embedder. Codifies Anthropic's documented recommendation that Claude users embed with Voyage.
  • Concept: realtime sessions — new Realtime sessions page explains the three patterns (STT streaming, TTS realtime context, full-duplex Realtime) and when to reach for which.
  • Test suite — grew from 1,592 to 1,742 subtests covering the new WebSocket transports and the Realtime session state machine.
  • New dependency — github.com/coder/websocket v1.8.14 (single pure-Go WebSocket library, no transitive churn).

v0.3.0 — Audio, embeddings, four new specialist providers

llmrouter grows beyond chat. Three new root capabilities (Embeddings, TTS, STT) and four new specialist provider packages dedicated to audio and embeddings.

  • Embedder interface on the root package — single method Embed(ctx, EmbedRequest) (*EmbedResponse, error). Implemented by OpenAI, Azure OpenAI, Bedrock (Titan + Cohere), Vertex, Gemini, Cohere, Mistral, Together, and Voyage AI.
  • Speaker interface for TTS — single method Speak(ctx, SpeechRequest) (*AudioStream, error). Implemented by OpenAI, Azure, Gemini (partial via Vertex path), ElevenLabs, Cartesia.
  • Transcriber interface for STT — single method Transcribe(ctx, TranscribeRequest) (*TranscriptStream, error). Implemented by OpenAI Whisper, Azure Whisper, Groq Whisper, Gemini (audio understanding), ElevenLabs Scribe, Deepgram Nova.
  • ElevenLabs provider — TTS + STT, no chat. Voice ID, model selection, output_format mapping, language detection, Scribe diarization via Raw.
  • Deepgram provider — STT only. Nova-3 / Nova-2 / enhanced, per-word timing, confidence scores, diarization via Raw. WebSocket streaming is on the v0.4 roadmap; v0.3 supports pre-recorded transcription.
  • Cartesia provider — TTS only. Sonic-2, sub-100 ms first-token latency, SSE streaming, real-time voice agent use cases. WebSocket streaming on v0.4 roadmap.
  • Voyage AI provider — Embeddings only. voyage-3 / voyage-3-large / voyage-code-3 / voyage-finance-2 / voyage-multilingual-2. Recommended pairing for Anthropic Claude.
  • Cross-vendor format normalisation for audio (mp3 / opus / wav / pcm / ulaw) and for embedding task types (Vertex canonical → Cohere / Voyage / OpenAI translations).
  • AudioStream / TranscriptStream with the same single-consumer, context-cancellation, terminal-error lifecycle as Stream — if you've used CompletionStream, you already know how.

v0.2.0 — Cloud providers, tool use, multimodal

The cloud triple plus the typed surface for the features that previously rode on Raw.

  • Azure OpenAI Service provider — deployment URL, api-key header, api-version query parameter. Supports chat, embeddings, TTS, Whisper STT.
  • AWS Bedrock provider — full SigV4 signing, per-model-family body translation (Claude / Llama / Titan / Nova / Mistral / Cohere). Chat + Titan/Cohere embeddings.
  • Google Vertex AI provider — ADC auth, project/region scoping, chat (Gemini + Claude), embeddings, partial TTS via Gemini.
  • Google Gemini (AI Studio) provider — API-key based, chat + embeddings + TTS + audio-understanding STT.
  • Cohere provider — Command family chat + Embed v3 embeddings with mandatory input_type.
  • Mistral provider — chat + mistral-embed.
  • Ten OpenAI-compatible providers verified — OpenRouter, Together, Groq, DeepSeek, Fireworks, xAI (Grok), Perplexity, Cerebras, vLLM, Ollama. All share the OpenAI provider machinery via WithBaseURL.
  • Typed tool-call passthrough — ChatRequest.Tools []ToolDef on the way out and Choice.Delta.ToolCalls on the way back, with full translation to Anthropic tool_use / tool_result blocks. Closes #1.
  • Extended thinking blocks for Claude 3.7 and onwards, surfaced via Choice.Delta.Thinking.
  • Prompt caching — typed Message.CacheControl field for Anthropic ephemeral cache; OpenAI auto-caching surfaces via Usage.CachedTokens.
  • Multimodal content helpers — llmrouter.ImageURLContent and ImageBytesContent builders for image inputs. Closes #2.
  • Mid-stream error surfacing — errors that arrive as SSE events after the channel opened are now delivered to Stream.Err() with the right status, not as silent EOF.

v0.1.1 — Test hardening

A pure quality release. 330-subtest table-driven test suite, 100% statement coverage on the root package, 90.4% on each provider. No public-API changes.

v0.1.0 — Initial public release

  • Core API: Provider interface, ChatRequest / Message / Chunk, Stream handle, option pattern, ErrUpstream.
  • OpenAI provider with byte passthrough + WithBaseURL.
  • Anthropic provider with full request and SSE translation.

Planned for v0.9 — Vertex-native Llama / Mistral / Cohere, batch APIs

The router landed in v0.8 with placeholder errors for the Vertex routes that don't yet have a backing package. v0.9 fills those in and adds the operational surface that's still vendor-SDK-only today.

  • New providers/vertexllama — Llama family models on Vertex AI Model Garden. Same pure-HTTP shape as vertexanthropic; ADC bearer token, regional :streamRawPredict endpoint.
  • New providers/vertexmistral — Mistral on Vertex Model Garden.
  • New providers/vertexcohere — Cohere on Vertex Model Garden.
  • Batch APIs — OpenAI and Anthropic both expose 24-hour batch endpoints at ~50% cost. A cross-vendor Batcher interface that takes a slice of ChatRequest and returns a job handle, mirroring the existing Stream lifecycle for poll-based jobs.
  • OpenAI Files API + Assistants v2 — typed file-upload surface for documents, plus a thin wrapper around Assistants v2 for the cases that genuinely need server-side thread state.
  • Audio in chat (gpt-4o audio) — typed audio-input message content for the synchronous gpt-4o-audio-preview family, complementing the Realtime path for cases that don't need full duplex.

Planned for v0.10 — Prompt management, semantic caching

  • Prompt management / versioning — a typed PromptTemplate surface with named versions, so application code can refer to prompts by id and pin to a version separately from the model id.
  • Semantic caching — an opt-in cache layer that hashes the request body and short-circuits identical (or embedding-similar) requests. Strictly opt-in; defaults off.
  • Embeddings for Anthropic when (and if) Anthropic ships a first-party embedding endpoint. Until then, use anthropic.NewRecommendedEmbedder or pair with Voyage AI directly.

Planned for v1.0 — API freeze

Once v0.10 ships and the surface has been stable for a few months, v1.0 freezes the public API. The contract from v1.0 onwards:

  • No breaking changes to exported types, functions, or interfaces within v1.x.
  • Backward-compatible additions (new options, new methods with sensible defaults, new fields) are minor bumps.
  • Bug fixes are patch bumps.
  • Breaking changes — if they ever happen — bump major.

Until v1.0, we do not guarantee backward compatibility between minor versions. Pin a minor in your go.mod and read the changelog before bumping.

Out of scope — and staying that way

The most important part of any roadmap is what it’s not. The following features are intentionally not in llmrouter, will not be added, and would be rejected in a PR. They are not bad ideas — they are the wrong layer. They belong in the gateway you build on top of a library like llmrouter, not inside the library itself.

  • Rate limiting. Token-bucket rate limiting across vendors is a property of your deployment, not of the library.
  • Budget caps. Per-tenant or per-key cost ceilings need persistent counters, an authorisation model, and a billing source of truth. None of that belongs in a library.
  • Response caching. Caching needs a cache key strategy, an eviction policy, and a backing store — all deployment decisions.
  • Vector store / RAG. Vector databases, retrievers, chunkers, and prompt assembly are an entire product category. llmrouter gives you embeddings and (since v0.5) rerank; the storage and retrieval pipeline is yours to build.
  • Persistent conversation state. Storing turns across requests, summarising old context, paging history out to disk — these are application concerns.
  • A bundled HTTP server. llmrouter is a library, not a binary. Run it inside your own server.

If you want all of these in one box, the answer in the Python ecosystem is LiteLLM. There is no Go equivalent today, and llmrouter is deliberately not trying to become one.

How to influence the roadmap

The plan above is a plan, not a promise. File an issue at github.com/elloloop/llmrouter/issues with:

  • The type:feature label.
  • A concrete use case — not just “please add more providers.”
  • The shape of the API you’d want, if you’ve thought about it.
  • Whether you’d be willing to send the PR yourself.