Last updated 2026-05-17

Quick Start

From an empty directory to a streaming completion in about two minutes. We'll go through the OpenAI example first, then run the same program against Anthropic, then walk through the anatomy of the call and how to cancel a stream cleanly.

This page assumes you have already run go get github.com/elloloop/llmrouter. If not, do that first.

1. Set up environment variables

Neither the OpenAI nor the Anthropic provider reads environment variables on its own — you pass the key in via llmrouter.WithAPIKey. We use the standard env vars below for symmetry with the official SDKs.

export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."

Pick whichever provider you have a key for. The OpenAI example is the canonical entry point; the Anthropic example is the same program with one import and one model id changed.

2. Hello, OpenAI

Save the following as main.go in a fresh directory. It opens a streaming chat completion, prints each delta to stdout as it arrives, and exits when the upstream closes the stream.

package main

import (
  "context"
  "fmt"
  "log"
  "os"

  "github.com/elloloop/llmrouter"
  "github.com/elloloop/llmrouter/providers/openai"
)

func main() {
  apiKey := os.Getenv("OPENAI_API_KEY")
  if apiKey == "" {
    log.Fatal("OPENAI_API_KEY is not set")
  }

  provider, err := openai.New(llmrouter.WithAPIKey(apiKey))
  if err != nil {
    log.Fatalf("provider init: %v", err)
  }

  ctx := context.Background()
  stream, err := provider.CompletionStream(ctx, llmrouter.ChatRequest{
    Model: "gpt-4o-mini",
    Messages: []llmrouter.Message{
      llmrouter.TextMessage("system", "You are concise."),
      llmrouter.TextMessage("user", "Say hi in five words."),
    },
  })
  if err != nil {
    log.Fatalf("completion: %v", err)
  }

  for chunk := range stream.Chunks() {
    for _, choice := range chunk.Choices {
      fmt.Print(choice.Delta.Content)
    }
  }
  fmt.Println()

  if err := stream.Err(); err != nil {
    log.Fatalf("stream finished with error: %v", err)
  }
}

Run it:

go run .

You should see five words print, one delta at a time, then a newline. For example:

Hi there, friend, hello world.

No buffering, no full-response wait — each token reaches your terminal as soon as OpenAI emits it. The for chunk := range stream.Chunks() loop terminates when the upstream closes the SSE connection or when the context is cancelled.

3. Hello, Anthropic

The exact same program, with three changes: the import, the constructor, and the model id. Note the explicit MaxTokens — the Anthropic API requires max_tokens on every request, so set it on ChatRequest whenever you talk to Anthropic.

package main

import (
  "context"
  "fmt"
  "log"
  "os"

  "github.com/elloloop/llmrouter"
  "github.com/elloloop/llmrouter/providers/anthropic"
)

func main() {
  apiKey := os.Getenv("ANTHROPIC_API_KEY")
  if apiKey == "" {
    log.Fatal("ANTHROPIC_API_KEY is not set")
  }

  provider, err := anthropic.New(llmrouter.WithAPIKey(apiKey))
  if err != nil {
    log.Fatalf("provider init: %v", err)
  }

  ctx := context.Background()
  stream, err := provider.CompletionStream(ctx, llmrouter.ChatRequest{
    Model: "claude-3-5-sonnet-latest",
    Messages: []llmrouter.Message{
      llmrouter.TextMessage("system", "You are concise."),
      llmrouter.TextMessage("user", "Say hi in five words."),
    },
    MaxTokens: 256,
  })
  if err != nil {
    log.Fatalf("completion: %v", err)
  }

  for chunk := range stream.Chunks() {
    for _, choice := range chunk.Choices {
      fmt.Print(choice.Delta.Content)
    }
  }
  fmt.Println()

  if err := stream.Err(); err != nil {
    log.Fatalf("stream finished with error: %v", err)
  }
}

Notice what didn't change: the loop, the ChatRequest shape, the Stream consumer, the error handling. Anthropic's request body and SSE event format are completely different from OpenAI's, but the Anthropic provider translates both directions inside CompletionStream so your call site never sees the difference.

Tip. The system role in Messages is special-cased by the Anthropic provider — it gets lifted out of the messages array and into Anthropic's top-level system field on the wire. You don't need to do anything; just write system as if you were calling OpenAI.

4. Anatomy of the call

Three types do the work: ChatRequest, Stream, and Chunk. Understanding them is most of the library.

ChatRequest

ChatRequest mirrors OpenAI's /v1/chat/completions request body. The fields most callers set are:

type ChatRequest struct {
  Model       string          // e.g. "gpt-4o-mini", "claude-3-5-sonnet-latest"
  Messages    []Message       // role + content for each turn
  MaxTokens   int             // required by Anthropic; optional for OpenAI
  Temperature *float64        // *float64 so 0 is distinguishable from "unset"
  TopP        *float64
  Stop        []string
  User        string
  Stream      bool            // ignored — streaming is implicit in CompletionStream
  Raw         json.RawMessage // see below
}

Raw is the byte-passthrough escape hatch. When you set Raw on a request handed to a passthrough provider (OpenAI), the provider forwards those bytes verbatim instead of serializing the typed fields. Anything the typed surface doesn't model — tool calls, vision, response_format, structured outputs, vendor-specific extensions — reaches the upstream unchanged. For translating providers (Anthropic), Raw is ignored; the provider always serializes its own body shape from the typed fields.

Stream

The Stream handle has three operations:

type Stream struct { /* ... */ }

// Chunks returns a receive-only channel. Range over it until it closes.
func (s *Stream) Chunks() <-chan Chunk

// Err returns the terminal error. Blocks until the producer finishes;
// returns nil on a clean stream.
func (s *Stream) Err() error

// Cancel asks the producer goroutine to stop. The chunks channel will
// close once the cancellation is observed. Safe to call multiple times.
func (s *Stream) Cancel()

The canonical consumer loop is exactly:

for chunk := range stream.Chunks() {
  for _, choice := range chunk.Choices {
    fmt.Print(choice.Delta.Content)
  }
}
if err := stream.Err(); err != nil {
  // handle: network failure, ErrUpstream non-2xx, context cancelled, parse error
}

The order matters. stream.Err() blocks until the producer goroutine has finished, so calling it before the Chunks() channel closes would deadlock. The library enforces single-consumer semantics: only one goroutine should read from Chunks(); if you need to fan out, do it on your side of the channel.

Chunk

Each Chunk is one streaming delta, normalized to OpenAI shape:

type Chunk struct {
  ID      string
  Object  string
  Created int64
  Model   string
  Choices []Choice
  Usage   *Usage          // populated only on the final chunk when upstream reports it
  Raw     json.RawMessage // original wire-format JSON for this chunk
}

type Choice struct {
  Index        int
  Delta        Delta
  FinishReason string
}

type Delta struct {
  Role    string
  Content string
}

Most chunks have one Choice with a non-empty Delta.Content. The first chunk in a stream usually carries Delta.Role = "assistant" and empty content; the last chunk carries a FinishReason ("stop", "length", "content_filter", etc.). Don't assume one-content-per-chunk if you're building a gateway — always range over Choices.

Chunk.Raw is the original wire-format JSON for this event. If you are building a proxy and want to forward bytes unchanged to a downstream caller, forward Raw instead of re-marshaling the typed fields. See Byte passthrough for the details.

5. Cancellation

Cancelling the context.Context you pass to CompletionStream propagates all the way to the in-flight HTTP request. The producer goroutine notices, closes the upstream connection, closes the Chunks() channel, and sets stream.Err() to the context error.

The most common pattern is context.WithTimeout for a wall-clock bound on the whole call:

package main

import (
  "context"
  "errors"
  "fmt"
  "log"
  "os"
  "time"

  "github.com/elloloop/llmrouter"
  "github.com/elloloop/llmrouter/providers/openai"
)

func main() {
  provider, err := openai.New(llmrouter.WithAPIKey(os.Getenv("OPENAI_API_KEY")))
  if err != nil {
    log.Fatal(err)
  }

  // Bound the entire call to 10 seconds.
  ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
  defer cancel()

  stream, err := provider.CompletionStream(ctx, llmrouter.ChatRequest{
    Model: "gpt-4o-mini",
    Messages: []llmrouter.Message{
      llmrouter.TextMessage("user", "Write a short haiku about Go."),
    },
  })
  if err != nil {
    log.Fatalf("completion: %v", err)
  }

  for chunk := range stream.Chunks() {
    for _, choice := range chunk.Choices {
      fmt.Print(choice.Delta.Content)
    }
  }
  fmt.Println()

  switch err := stream.Err(); {
  case err == nil:
    // success
  case errors.Is(err, context.DeadlineExceeded):
    log.Println("upstream took too long; partial output above")
  case errors.Is(err, context.Canceled):
    log.Println("call cancelled; partial output above")
  default:
    log.Fatalf("stream error: %v", err)
  }
}

stream.Cancel() is equivalent to calling the cancel returned by context.WithCancel — it's a convenience for cases where you didn't keep a reference to the cancel func. Cancelling part-way through is safe: any chunks already buffered in the channel are still delivered, but no new ones are read from the upstream.

Always bound the context. If you call CompletionStream with context.Background() and the upstream stalls, the goroutine pinned to that request will sit forever. Use context.WithTimeout or context.WithDeadline in any code path that runs in a long-lived server.

6. Common pitfalls

You must drain Chunks() or call Cancel(). The producer goroutine writes into a 16-deep buffered channel. If you stop reading mid-stream without cancelling, the producer blocks on the next send and the goroutine leaks. Either consume to the end of the channel or call stream.Cancel() when you bail.
Stream is single-consumer. Don't read from Chunks() from two goroutines. The channel is unsynchronized — concurrent reads will interleave deltas unpredictably. If you need fan-out, build it on your side: one goroutine drains Chunks() and rebroadcasts to N subscribers.
Err() blocks until the producer finishes. Calling stream.Err() before the Chunks() channel closes is not a bug per se — it just deadlocks until something drains the channel. Always range over Chunks() first, then call Err().
Anthropic requires MaxTokens. Anthropic's API rejects requests without max_tokens. Set MaxTokens on every ChatRequest you hand to the Anthropic provider. OpenAI accepts the request without it.
Don't set Stream = true. ChatRequest.Stream exists for serialization symmetry; the providers always stream regardless. Setting it has no effect and is not surfaced in any non-streaming call (there is no non-streaming call).
Non-2xx responses surface as ErrUpstream. A 401, a 429, or a 500 reaches you as llmrouter.ErrUpstream through CompletionStream (immediate failure) or stream.Err() (failure after the connection opened). Use errors.As to read StatusCode and Body; see error handling.

Next steps

Architecture overview — how the provider interface and the streaming producer fit together.
Streaming model — producer goroutine, channel buffer, context cancellation in detail.
Byte passthrough — Chunk.Raw and ChatRequest.Raw for proxy/gateway use cases.
OpenAI provider — base-URL override patterns for OpenRouter, Together, Groq, vLLM, Ollama.
Anthropic provider — request and SSE translation in detail.
API reference — every exported type and option, every method, every error.