Last updated 2026-05-17

llmrouter

A polyglot Go client for LLM providers. One OpenAI-shaped API surface fans out to multiple vendors; pluggable backends translate the request body, the response stream, and the auth scheme to whatever the upstream speaks. Streaming-first, byte-passthrough where possible, and built around context.Context cancellation that propagates all the way to the upstream HTTP request.

Today the library ships with two providers: OpenAI (and any OpenAI-compatible endpoint via a base-URL override) and Anthropic (with full request and SSE event translation). Azure OpenAI, AWS Bedrock, and Google Vertex AI are on the v0.2 roadmap.

What is llmrouter?

llmrouter is a single Go module that exposes one Provider interface backed by pluggable provider implementations. The public type surface is shaped after OpenAI's /v1/chat/completions endpoint: you build a ChatRequest, hand it to a Provider.CompletionStream, and read normalized Chunk values off a channel until the upstream finishes.

The library has four design pillars:

  • One API surface. Application code does not branch on provider. The same ChatRequest works for OpenAI, Anthropic, OpenRouter, Together, Groq, a self-hosted vLLM, or whatever you point a base URL at.
  • Pluggable backends. Each provider lives in its own subpackage under providers/, depends on llmrouter for the shared types, and decides how to translate. OpenAI is a passthrough; Anthropic is a full translator. Future Azure / Bedrock / Vertex implementations follow the same contract.
  • Per-provider configuration. Every provider's New(opts ...Option) accepts the same shared options: WithAPIKey, WithBaseURL, WithHTTPClient, WithTimeout, WithExtra. Provider-specific knobs (Azure api-version, Bedrock region, Vertex project) flow through WithExtra so the public surface never grows a special case.
  • Streaming-first. The library does not have a blocking "wait for the full completion" helper. Everything is a Stream; use a context with a deadline if you want a bound on wall time.

Why llmrouter?

Go already has excellent per-vendor SDKs — openai-go, anthropic-sdk-go, google.golang.org/genai — and one notable unified library, mozilla-ai/any-llm-go. llmrouter exists for a specific shape of project that none of those serve well:

  • Per-vendor SDKs branch your application code. Every new provider you adopt means a new import, a new request type, a new streaming idiom, a new error type, and a new place to put the model id. If you support three vendors, your call site either has a three-way switch or a hand-rolled adapter you have to maintain forever.
  • The Python equivalents have a Go shaped hole. LiteLLM and friends are well loved in Python; any-llm-go is the closest Go analogue, supports ten providers, and is a good pick if its provider matrix matches your needs. It does not currently cover Azure Foundry (deployment URL + api-key header + api-version query param), AWS Bedrock (SigV4 + per-model-family body shapes), or Google Vertex AI (ADC auth, project/region) — the three cloud-vendor surfaces enterprise deployments almost always need.
  • Existing libraries parse the wire format. They decode the upstream stream into typed Go shapes, drop fields they don't model, and re-marshal on the way out. That is fine for application code; it is a problem if you are building a proxy or gateway, because anything you don't model (tool calls, vision content, response_format, structured outputs, vendor-specific extensions) is silently lost.
  • llmrouter exposes byte passthrough. Every Chunk carries the original wire-format JSON in a Raw field, and ChatRequest carries a Raw request body. Passthrough providers (OpenAI and OpenAI-compatibles) forward the bytes unchanged. Translating providers (Anthropic now, Bedrock/Vertex later) still expose Raw on each chunk so a proxy can forward the translated bytes without re-marshaling.

In short: llmrouter targets the proxy/gateway use case and the multi-vendor application use case in the same library. If you only call one vendor, use that vendor's official SDK. If your needs match LiteLLM-in-Python and every provider you care about ships in any-llm-go, use that. Otherwise, llmrouter.

What ships in v0.1

The 0.1 line is small on purpose. It contains:

  • OpenAI provider — a thin passthrough adapter for /v1/chat/completions. The request body is serialized from ChatRequest (or forwarded verbatim if Raw is set), the SSE event stream is parsed enough to surface Chunk values, and every chunk's Raw is the original wire bytes. Pointing this provider at https://openrouter.ai/api/v1 or https://api.together.xyz/v1 or http://localhost:11434/v1 (Ollama) Just Works.
  • Anthropic provider — a full translator. The request body is rewritten from OpenAI messages + max_tokens + temperature + sampling knobs into Anthropic's /v1/messages shape, with the system role lifted into Anthropic's top-level system field. The SSE event stream (message_start, content_block_delta, message_delta, message_stop) is translated into OpenAI delta chunks on the fly. x-api-key + anthropic-version headers are set automatically.
  • Stream type with single-consumer semantics, a buffered chunk channel, context cancellation that propagates to the in-flight HTTP request, and a terminal Err() for the final error.
  • Shared options — WithAPIKey, WithBaseURL, WithHTTPClient, WithTimeout, WithExtra. Every provider reads from the same Config; only the upstream URL and translation logic differ.
  • ErrUpstream — non-2xx responses surface as a typed error with the provider name, status code, and response body, so application code can branch on the wire response without parsing strings.

On the v0.2 roadmap

The next minor version brings the three cloud-vendor providers that distinguish llmrouter from the existing Go ecosystem:

  • Azure OpenAI — deployment URL of the form https://{resource}.openai.azure.com/openai/deployments/{deployment}/chat/completions, api-key header (not Authorization: Bearer), and a required api-version query parameter passed via WithExtra("api-version", "2024-10-21"). Request and response bodies are OpenAI-shaped, so the translation cost is zero — only auth and URL construction change.
  • AWS Bedrock — SigV4-signed requests to https://bedrock-runtime.{region}.amazonaws.com/model/{model-id}/invoke-with-response-stream with per-model-family body shapes (Anthropic on Bedrock uses Anthropic's body; Cohere uses Cohere's; Meta Llama uses its own). The provider negotiates the body shape from the model id and streams Bedrock's event-stream framing into OpenAI-shaped chunks. Region travels via WithExtra("region", "us-east-1").
  • Google Vertex AI — Application Default Credentials (ADC) for auth, with WithExtra("project", "my-gcp-proj") and WithExtra("region", "us-central1"). The provider targets Vertex's generateContent / streamGenerateContent endpoints for Gemini, and Vertex's Anthropic / Meta passthrough endpoints for the partner models.

Beyond v0.2, the v0.3 plan is to surface tool-call passthrough as a first-class concept (today, tool calls reach providers via ChatRequest.Raw but aren't part of the typed surface) and to add multimodal content helpers. v1.0 is the API freeze.

Use cases

The library is designed for a handful of concrete shapes:

Building an OpenAI-compatible gateway

You run an HTTP service that exposes /v1/chat/completions to your applications and routes each request to a different upstream — OpenAI for general traffic, Anthropic for long-context, a self-hosted vLLM for cheap development calls. llmrouter is your forwarder: the gateway deserializes the incoming OpenAI request, picks a provider, calls CompletionStream, and forwards Chunk.Raw bytes directly back to the caller as SSE. Anything the gateway doesn't model passes through.

Multi-provider fallback

Your application calls OpenAI for everything, but if OpenAI returns a 429 or a 5xx you fall back to Anthropic without changing the request shape. With llmrouter the fallback is a few lines: construct two providers at startup, attempt the first, and on ErrUpstream with a retryable status code, retry the same ChatRequest against the second.

Swapping providers without touching application code

You picked OpenAI for your prototype and now you want to switch to Anthropic because pricing or context window changed. With llmrouter, the only change is the provider constructor and the model id — the call site, the request shape, the streaming consumer, and the error handling are all unchanged.

Self-hosted model proxy

You run a vLLM or Ollama instance internally and you want to expose it via the same code paths your team already uses for OpenAI. Point the OpenAI provider at the self-hosted URL with WithBaseURL("http://localhost:11434/v1"); everything else is identical.