Last updated 2026-05-17

Architecture Overview

llmrouter is built around one interface and three shared types. The interface is Provider; the types are ChatRequest (in), Stream (out), and Chunk (the unit a stream yields). Everything else — auth, base URLs, body translation, SSE parsing, error wrapping — is a per-provider concern hidden behind that interface.

This page covers the big idea, the type wiring, the two translation strategies (passthrough vs full translator), the streaming model, and the byte-passthrough escape hatch. For per-provider behavior see the OpenAI and Anthropic reference pages.

The big idea

One Provider interface, multiple backends, OpenAI-shaped types as the lingua franca. Application code holds a llmrouter.Provider and never knows which upstream it's talking to. Provider subpackages translate between the shared types and whatever the upstream API actually wants.

The interface itself is two methods:

// Provider is the upstream-LLM contract. Each implementation
// translates to/from the OpenAI /v1/chat/completions wire format
// internally.
type Provider interface {
  // Name returns the provider's stable id (e.g. "openai", "anthropic").
  Name() string


  // CompletionStream issues a streaming chat completion request. The
  // returned Stream yields normalized chunks until upstream finishes,
  // the context cancels, or an error occurs. Callers must drain the
  // stream (or cancel ctx) to release resources.
  CompletionStream(ctx context.Context, req ChatRequest) (*Stream, error)
}

That is the entire public surface a provider exposes. The implications:

  • One call shape for every provider. Application code that takes llmrouter.Provider is provider-agnostic by construction.
  • Streaming is the only mode. There is no Complete() that returns a string. If you want the full response as a single value, drain the stream and concatenate the deltas — but doing so explicitly is the point: the library doesn't pretend the upstream isn't streaming.
  • Construction is per-provider. Each provider subpackage exposes New(opts ...llmrouter.Option) which returns a value implementing the interface. The constructor is where API keys, base URLs, and timeouts get baked in.

How the pieces fit

The flow from your application code down to the upstream HTTPS request looks like this: 

┌──────────────────────────────────────────────────────────┐
│                  Your application code                   │
│                                                          │
│   p := /* one of the providers below */                  │
│   stream, _ := p.CompletionStream(ctx, llmrouter.ChatRequest{...})
│   for chunk := range stream.Chunks() { ... }             │
│   if err := stream.Err(); err != nil { ... }             │
└────────────────────────┬─────────────────────────────────┘
                         │
                         │  llmrouter.Provider interface
                         │  (Name, CompletionStream)
                         │
   ┌─────────────────────┼─────────────────────────────────┐
   │                     │                                 │
   ▼                     ▼                                 ▼
┌────────────┐    ┌────────────────┐         ┌──────────────────────────┐
│  OpenAI    │    │   Anthropic    │         │  v0.2 roadmap:           │
│ passthrough│    │ full translator│         │  Azure / Bedrock / Vertex│
│            │    │                │         │  (full translators +     │
│ - body:    │    │ - body:        │         │   SigV4 / ADC auth)      │
│   verbatim │    │   OpenAI →     │         └──────────────────────────┘
│   if Raw,  │    │   Anthropic    │
│   else     │    │ - SSE:         │
│   marshal  │    │   Anthropic →  │
│ - SSE:     │    │   OpenAI       │
│   parse    │    │ - headers:     │
│   OpenAI   │    │   x-api-key,   │
│   chunks   │    │   anthropic-   │
│ - headers: │    │   version      │
│   Bearer   │    │                │
└─────┬──────┘    └────────┬───────┘
      │                    │
      ▼                    ▼
┌───────────────────────────────────────────────────────────┐
│              upstream HTTPS (api.openai.com,              │
│              api.anthropic.com, your-vllm-host, …)        │
└───────────────────────────────────────────────────────────┘

The shape generalises cleanly. Every new provider is one more box in the middle row; the interface above and the HTTP below are invariant.

Three core types

Provider

Already shown above. Two methods, both pure: Name() is a stable string id used in error messages and metrics; CompletionStream is the workhorse. The Stream it returns is owned by the caller — the provider's only obligation after returning is to keep the producer goroutine fed until the upstream finishes or the context cancels.

See API reference: Provider.

ChatRequest

OpenAI-shaped request. Required fields are Model and Messages; everything else is optional. The Raw json.RawMessage field is the byte-passthrough hook discussed below. 

type ChatRequest struct {
  Model       string          `json:"model"`
  Messages    []Message       `json:"messages"`
  MaxTokens   int             `json:"max_tokens,omitempty"`
  Temperature *float64        `json:"temperature,omitempty"`
  TopP        *float64        `json:"top_p,omitempty"`
  Stop        []string        `json:"stop,omitempty"`
  User        string          `json:"user,omitempty"`
  Stream      bool            `json:"stream,omitempty"`
  Raw         json.RawMessage `json:"-"`
}


type Message struct {
  Role    string          `json:"role"`
  Content json.RawMessage `json:"content"`
}

Message.Content is json.RawMessage, not string, on purpose: OpenAI's multimodal content is an array of typed parts, not a plain string. By leaving it as raw JSON, the library lets passthrough providers forward multimodal requests without modeling every content-part variant. For the plain-text case use the llmrouter.TextMessage(role, text) convenience.

See API reference: ChatRequest.

Stream

The streaming-completion handle. Three operations: Chunks() returns a receive-only channel that closes when the upstream finishes or the context cancels; Err() blocks until the producer finishes, then returns the terminal error (nil on success); Cancel() asks the producer to stop and is equivalent to cancelling the parent context. 

type Stream struct {
  chunks chan Chunk
  cancel context.CancelFunc
  errMu  chan struct{}
  err    error
}


func (s *Stream) Chunks() <-chan Chunk
func (s *Stream) Err() error
func (s *Stream) Cancel()

Single-consumer by contract. The internal channel is buffered at 16 chunks so a slow consumer can absorb short bursts without backpressuring the upstream HTTP read.

See API reference: Stream and Streaming model.

Translation strategy

Each provider picks one of two strategies for the request body and the response stream: byte passthrough or full translator. The choice is determined by how similar the upstream is to OpenAI's wire format.

Passthrough (OpenAI and OpenAI-compatibles)

The OpenAI upstream speaks /v1/chat/completions natively. Any provider talking to that endpoint — OpenAI itself, OpenRouter, Together, Groq, a self-hosted vLLM, Ollama — uses the passthrough path:

  • Request body. If ChatRequest.Raw is non-nil, the provider forwards those bytes verbatim. Otherwise it serializes the typed fields directly. There is no intermediate Go struct that drops unknown fields — fields the typed surface doesn't model (tools, vision, response_format, structured outputs, vendor-specific extensions) survive when sent via Raw.
  • Response stream. The SSE event stream is already OpenAI-shaped, so each event is parsed exactly once into a Chunk, with the original wire JSON preserved on Chunk.Raw. A gateway forwarding to a downstream caller emits Chunk.Raw bytes directly with no re-marshaling.
  • Headers. Authorization: Bearer <apiKey>, Content-Type: application/json, Accept: text/event-stream.

Full translator (Anthropic, future Bedrock/Vertex)

When the upstream wire format diverges meaningfully from OpenAI's — different body shape, different SSE event types, different auth — the provider runs a full translator on both sides.

Anthropic specifically:

  • Request body. OpenAI messages → Anthropic /v1/messages shape. The system role is lifted out of the messages array into Anthropic's top-level system field. max_tokens is required by Anthropic and is always included.
  • Response stream. Anthropic emits a richer SSE event vocabulary (message_start, content_block_start, content_block_delta, content_block_stop, message_delta, message_stop). The provider folds these into OpenAI delta chunks: text-delta events become Delta.Content, the first chunk carries Delta.Role = "assistant", the message_stop event becomes a final chunk with a FinishReason.
  • Headers. x-api-key: <apiKey>, anthropic-version: 2023-06-01, Content-Type: application/json, Accept: text/event-stream.

The future Bedrock and Vertex providers will follow the same pattern: SigV4 / ADC for auth, per-model-family body shapes, event-stream-framing → OpenAI chunk translation. The interface above doesn't change.

Streaming model

The streaming machinery is owned by the root package, not by each provider. llmrouter.NewStream(parent context.Context) returns a *Stream, a derived context.Context (with cancellation chained to the consumer's Cancel()), and a ProducerHooks struct with Send and Finish callbacks. 

type ProducerHooks struct {
  // Send returns false if the consumer cancelled — the producer
  // should then stop and call Finish(ctx.Err()).
  Send   func(Chunk) bool


  // Finish closes the stream and finalises the terminal error.
  // Pass nil on a clean stream.
  Finish func(error)
}

The provider's CompletionStream implementation looks roughly like:

func (p *Provider) CompletionStream(parent context.Context, req llmrouter.ChatRequest) (*llmrouter.Stream, error) {
  httpReq, err := p.buildRequest(parent, req)
  if err != nil {
    return nil, err
  }


  stream, ctx, hooks := llmrouter.NewStream(parent)


  go func() {
    // HTTP request uses ctx so cancellation propagates to the wire.
    httpReq = httpReq.WithContext(ctx)
    resp, err := p.config.HTTP().Do(httpReq)
    if err != nil {
      hooks.Finish(err)
      return
    }
    defer resp.Body.Close()


    if resp.StatusCode >= 400 {
      body, _ := io.ReadAll(resp.Body)
      hooks.Finish(&llmrouter.ErrUpstream{
        Provider:   p.Name(),
        StatusCode: resp.StatusCode,
        Body:       body,
      })
      return
    }


    // Parse SSE events; translate to Chunks; deliver via hooks.Send.
    for evt := range parseSSE(resp.Body) {
      chunk := translate(evt)
      if !hooks.Send(chunk) {
        return // consumer cancelled
      }
    }
    hooks.Finish(nil)
  }()


  return stream, nil
}

Key invariants:

  • The producer is a single goroutine. No shared mutable state between the consumer and the producer except the channel. The buffered channel (capacity 16) absorbs short bursts.
  • Cancellation propagates downward. NewStream derives ctx from the caller's parent using context.WithCancel. The producer attaches that ctx to the HTTP request via WithContext, so the standard library cancels the in-flight connection when the consumer calls Stream.Cancel() or the parent context expires.
  • Send is bounded. hooks.Send uses select with ctx.Done(), so a consumer that bails mid-stream does not pin the producer. If Send returns false, the producer returns immediately.
  • Finish is terminal. The producer must call Finish(nil) on success or Finish(err) on failure exactly once. After Finish, the consumer's Chunks() channel closes and Err() unblocks.

The Raw byte field

Chunk.Raw is a deliberate design choice: every chunk carries the original wire-format JSON, exactly as it arrived from the upstream (or, for translating providers, exactly as the translated event was serialized into the OpenAI shape). This makes the proxy/gateway use case a one-liner.

Concretely, a passthrough proxy looks like:

func proxyHandler(w http.ResponseWriter, r *http.Request) {
  var req llmrouter.ChatRequest
  // Preserve the original request body verbatim so the upstream
  // sees fields we don't model (tools, vision, response_format).
  body, _ := io.ReadAll(r.Body)
  _ = json.Unmarshal(body, &req)
  req.Raw = body


  stream, err := provider.CompletionStream(r.Context(), req)
  if err != nil {
    http.Error(w, err.Error(), 502)
    return
  }


  w.Header().Set("Content-Type", "text/event-stream")
  flusher := w.(http.Flusher)


  for chunk := range stream.Chunks() {
    fmt.Fprintf(w, "data: %s\n\n", chunk.Raw)
    flusher.Flush()
  }
  if err := stream.Err(); err == nil {
    fmt.Fprint(w, "data: [DONE]\n\n")
    flusher.Flush()
  }
}

Note what is not happening: no re-marshaling, no field enumeration, no risk of losing fields llmrouter doesn't model. The upstream's bytes go out to the downstream caller verbatim.

Even with a translating provider (Anthropic), Chunk.Raw still carries the translated-into-OpenAI-shape bytes, so the same proxy code works against Anthropic with zero changes — the downstream caller sees OpenAI SSE no matter which upstream you chose.

See Byte passthrough for the deeper treatment.

Dependency hygiene

llmrouter has exactly one third-party dependency: github.com/google/uuid (used by the Anthropic provider to mint request ids). The OpenAI provider uses nothing beyond the standard library.

The reasoning:

  • No vendor SDKs. A library that aggregates four official SDKs inherits four module trees, four release cadences, and four sets of breaking changes. llmrouter implements every provider's HTTP transport directly so we own the surface end to end.
  • v0.2 still keeps the bar low. Azure OpenAI is pure standard library (HTTPS + headers). AWS Bedrock will add github.com/aws/aws-sdk-go-v2 only for SigV4 signing — the request and response bodies are read/written directly. Google Vertex will add golang.org/x/oauth2 + ADC for auth; the generative endpoints are HTTPS + JSON.
  • You don't pay for providers you don't import. Provider subpackages are independent. Importing providers/openai does not pull providers/anthropic; importing providers/anthropic does not pull the future providers/bedrock with the AWS SDK.

Next steps