Last updated 2026-05-17

The Provider Interface

Every upstream LLM in llmrouter — OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Vertex (planned), and any OpenAI-compatible endpoint reached via WithBaseURL — implements a single two-method interface. That interface is the seam the rest of the library is built on: requests in, normalized chunks out, with the OpenAI Chat Completions wire format as the lingua franca.

The interface

The contract lives in llmrouter.go and is intentionally tiny: 

type Provider interface {
    // Name returns the provider's stable id (e.g. "openai", "anthropic").
    Name() string


    // CompletionStream issues a streaming chat completion request. The
    // returned Stream yields normalized chunks until upstream finishes,
    // the context cancels, or an error occurs. Callers must drain the
    // stream (or cancel ctx) to release resources.
    CompletionStream(ctx context.Context, req ChatRequest) (*Stream, error)
}

Two methods. No options bag on the call site, no synchronous one-shot path, no model registry, no token counter. Everything else — retries, fallback, fan-out, budgeting — is a wrapper you compose around this surface. If you can satisfy these two methods, you can plug into any code that takes a Provider.

Why streaming-only

The most common question people ask about this API is: "where's the non-streaming version?" There isn't one, on purpose.

  • Gateways always stream. If you're building a proxy, an OpenAI-compatible aggregator, or a chat UI, you're streaming already. A non-streaming path would just be a buffer-and-drop on top of the streaming one.
  • A one-shot request is the degenerate case of streaming — open the stream, drain it into a string, return the string. Two extra lines of code, no extra concept to learn, no second code path in the library to keep in sync.
  • Streaming is the surface that exposes cancellation, backpressure, partial-result handling, and incremental UI rendering. If we offered a synchronous API we'd quietly encourage callers to give those things up.

The shape of the buffer-and-collect helper you'd write looks like this. It's small enough that it doesn't need to live in the library: 

package main


import (
    "context"
    "strings"


    "github.com/elloloop/llmrouter"
)


// collect drains a Provider into a single string. Use this when you
// don't actually need streaming and want a synchronous-style call.
func collect(ctx context.Context, p llmrouter.Provider, req llmrouter.ChatRequest) (string, error) {
    stream, err := p.CompletionStream(ctx, req)
    if err != nil {
        return "", err
    }
    var out strings.Builder
    for chunk := range stream.Chunks() {
        for _, c := range chunk.Choices {
            out.WriteString(c.Delta.Content)
        }
    }
    if err := stream.Err(); err != nil {
        return "", err
    }
    return out.String(), nil
}

Polymorphism: code that takes any provider

Because Provider is an interface, any code that operates on it works across every upstream. Here's a function that doesn't care whether it's talking to OpenAI directly, OpenRouter, a self-hosted vLLM instance, or Anthropic: 

package main


import (
    "context"
    "fmt"
    "strings"


    "github.com/elloloop/llmrouter"
)


// chat asks the given provider one question and returns its answer.
// Works with any Provider implementation — OpenAI, Anthropic, etc.
func chat(ctx context.Context, p llmrouter.Provider, prompt string) (string, error) {
    req := llmrouter.ChatRequest{
        Model: defaultModel(p.Name()),
        Messages: []llmrouter.Message{
            llmrouter.TextMessage("user", prompt),
        },
    }
    stream, err := p.CompletionStream(ctx, req)
    if err != nil {
        return "", fmt.Errorf("%s: open stream: %w", p.Name(), err)
    }
    var out strings.Builder
    for chunk := range stream.Chunks() {
        for _, c := range chunk.Choices {
            out.WriteString(c.Delta.Content)
        }
    }
    if err := stream.Err(); err != nil {
        return "", fmt.Errorf("%s: stream: %w", p.Name(), err)
    }
    return out.String(), nil
}


// defaultModel picks a sensible default per provider. Real code would
// take this from config; this is just illustration.
func defaultModel(name string) string {
    switch name {
    case "anthropic":
        return "claude-sonnet-4-5"
    default:
        return "gpt-4o-mini"
    }
}

The same chat function plugs into both providers from providers/openai and providers/anthropic: 

package main


import (
    "context"
    "fmt"
    "log"
    "os"


    "github.com/elloloop/llmrouter"
    "github.com/elloloop/llmrouter/providers/anthropic"
    "github.com/elloloop/llmrouter/providers/openai"
)


func main() {
    ctx := context.Background()


    oa, err := openai.New(llmrouter.WithAPIKey(os.Getenv("OPENAI_API_KEY")))
    if err != nil {
        log.Fatal(err)
    }
    an, err := anthropic.New(llmrouter.WithAPIKey(os.Getenv("ANTHROPIC_API_KEY")))
    if err != nil {
        log.Fatal(err)
    }


    for _, p := range []llmrouter.Provider{oa, an} {
        ans, err := chat(ctx, p, "in one sentence, what is a goroutine?")
        if err != nil {
            log.Printf("%s: %v", p.Name(), err)
            continue
        }
        fmt.Printf("%-10s %s
", p.Name(), ans)
    }
}

The ChatRequest shape

ChatRequest is the OpenAI Chat Completions payload, transposed into Go: 

type ChatRequest struct {
    Model       string          `json:"model"`
    Messages    []Message       `json:"messages"`
    MaxTokens   int             `json:"max_tokens,omitempty"`
    Temperature *float64        `json:"temperature,omitempty"`
    TopP        *float64        `json:"top_p,omitempty"`
    Stop        []string        `json:"stop,omitempty"`
    User        string          `json:"user,omitempty"`
    Stream      bool            `json:"stream,omitempty"`
    Raw         json.RawMessage `json:"-"`
}
  • Model — the model id, passed through to the upstream. Anthropic ids (claude-sonnet-4-5) work with the Anthropic provider; OpenAI ids (gpt-4o-mini) work with the OpenAI provider and with OpenAI-compatible endpoints behind WithBaseURL.
  • Messages — the conversation. See below for the message shape.
  • MaxTokens — upper bound on completion length. Anthropic requires this (the provider defaults to 4096 if you leave it zero); OpenAI treats it as optional.
  • Temperature, TopP — sampling knobs. Pointer types so zero values are distinguishable from "not set."
  • Stop — stop sequences. The Anthropic provider maps these onto stop_sequences.
  • User — opaque end-user id forwarded to OpenAI for abuse-detection.
  • Stream — ignored. Providers always force stream: true on the wire.
  • Raw — original request bytes for passthrough mode. See Byte Passthrough.

The Message shape

type Message struct {
    Role    string          `json:"role"`
    Content json.RawMessage `json:"content"`
}

The unusual part is Content json.RawMessage. OpenAI's content field is a union — it's either a plain string (text message) or an array of typed parts (multimodal: text + image_url + audio + …). Modelling that as a Go union would force you into a sum-type pattern with constructors and visitors. Using json.RawMessage instead means the bytes pass through untouched: plain strings stay strings, vision arrays stay arrays, and any future content type is forward-compatible without a library change.

For the common case of plain text, TextMessage hides the JSON entirely: 

// TextMessage is a convenience constructor for plain-text messages.
func TextMessage(role, text string) Message {
    b, _ := json.Marshal(text)
    return Message{Role: role, Content: b}
}

And to read text out, PlainText handles both shapes:

// PlainText extracts text from a Message. For multimodal content arrays
// it concatenates the "text" parts and ignores image bytes.
func (m Message) PlainText() string { /* ... */ }

A complete plain-text example: 

req := llmrouter.ChatRequest{
    Model: "gpt-4o-mini",
    Messages: []llmrouter.Message{
        llmrouter.TextMessage("system", "you are concise"),
        llmrouter.TextMessage("user", "what is the capital of france?"),
    },
}

And a multimodal example — passed in as raw bytes:

vision := json.RawMessage(`[
    {"type":"text","text":"what is in this image?"},
    {"type":"image_url","image_url":{"url":"https://example.com/cat.jpg"}}
]`)
req := llmrouter.ChatRequest{
    Model: "gpt-4o",
    Messages: []llmrouter.Message{
        {Role: "user", Content: vision},
    },
}

Full example: provider-agnostic chat

Putting it all together — a small program that takes a provider name on the command line and asks it a question: 

package main


import (
    "context"
    "flag"
    "fmt"
    "log"
    "os"
    "strings"
    "time"


    "github.com/elloloop/llmrouter"
    "github.com/elloloop/llmrouter/providers/anthropic"
    "github.com/elloloop/llmrouter/providers/openai"
)


func main() {
    var (
        provider = flag.String("provider", "openai", "openai|anthropic")
        model    = flag.String("model", "", "model id (defaults per provider)")
        prompt   = flag.String("prompt", "say hi in five words", "user prompt")
    )
    flag.Parse()


    p, err := buildProvider(*provider)
    if err != nil {
        log.Fatal(err)
    }
    if *model == "" {
        *model = defaultModel(p.Name())
    }


    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()


    answer, err := chat(ctx, p, *model, *prompt)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(answer)
}


func buildProvider(name string) (llmrouter.Provider, error) {
    switch name {
    case "openai":
        return openai.New(llmrouter.WithAPIKey(os.Getenv("OPENAI_API_KEY")))
    case "anthropic":
        return anthropic.New(llmrouter.WithAPIKey(os.Getenv("ANTHROPIC_API_KEY")))
    default:
        return nil, fmt.Errorf("unknown provider %q", name)
    }
}


func defaultModel(name string) string {
    switch name {
    case "anthropic":
        return "claude-sonnet-4-5"
    default:
        return "gpt-4o-mini"
    }
}


// chat is provider-agnostic — it only depends on the Provider interface.
func chat(ctx context.Context, p llmrouter.Provider, model, prompt string) (string, error) {
    req := llmrouter.ChatRequest{
        Model: model,
        Messages: []llmrouter.Message{
            llmrouter.TextMessage("user", prompt),
        },
    }
    stream, err := p.CompletionStream(ctx, req)
    if err != nil {
        return "", err
    }
    var out strings.Builder
    for chunk := range stream.Chunks() {
        for _, c := range chunk.Choices {
            out.WriteString(c.Delta.Content)
        }
    }
    return out.String(), stream.Err()
}

Next steps