Last updated 2026-05-17

Adding a New Provider

Every backend in llmrouter — OpenAI, Anthropic, the planned Azure / Bedrock / Vertex implementations — is a small Go package that implements one interface. If you need to target an upstream the library doesn't ship with yet (or you want to fork the Anthropic provider to fix one of its caveats), this page is the recipe.

The contract

A provider is anything that satisfies llmrouter.Provider:

type Provider interface {
    Name() string
    CompletionStream(ctx context.Context, req ChatRequest) (*Stream, error)
}

Name() returns a stable string id (the same value used in ErrUpstream.Provider) and CompletionStream opens a streaming chat completion that yields normalized Chunk values until the upstream finishes, the context cancels, or an error occurs.

That's the whole surface. Everything else is convention.

Skeleton

Start with this layout under providers/myprovider/:

// Package myprovider implements llmrouter.Provider for MyProvider.
package myprovider

import (
    "context"
    "fmt"

    "github.com/elloloop/llmrouter"
)

const defaultBaseURL = "https://api.myprovider.example/v1"

// Provider implements llmrouter.Provider against MyProvider's wire format.
type Provider struct {
    cfg *llmrouter.Config
}

// New constructs a MyProvider client.
func New(opts ...llmrouter.Option) (*Provider, error) {
    cfg, err := llmrouter.NewConfig(opts...)
    if err != nil {
        return nil, err
    }
    if cfg.APIKey == "" {
        return nil, fmt.Errorf("%w: myprovider requires an api key",
            llmrouter.ErrInvalidConfig)
    }
    if cfg.BaseURL == "" {
        cfg.BaseURL = defaultBaseURL
    }
    return &Provider{cfg: cfg}, nil
}

// Name returns the provider id.
func (p *Provider) Name() string { return "myprovider" }

// CompletionStream issues a streaming chat completion request.
func (p *Provider) CompletionStream(
    ctx context.Context, req llmrouter.ChatRequest,
) (*llmrouter.Stream, error) {
    // ... see "HTTP + streaming" below
    return nil, nil
}

Validation in New

Two rules every provider follows:

• Build the Config via llmrouter.NewConfig(opts...). Don't construct it by hand — the options validate things like WithAPIKey("") being rejected, and the Timeout default is set there.
• Wrap config errors in llmrouter.ErrInvalidConfig. Use fmt.Errorf("%w: ...", llmrouter.ErrInvalidConfig) so callers can use errors.Is(err, llmrouter.ErrInvalidConfig).

Default the BaseURL only after validating the API key, so the error from "missing API key" arrives even when no WithBaseURL was passed.

HTTP client

Use p.cfg.HTTP() to get the HTTP client. It lazily constructs an http.Client with the configured Timeout if the caller didn't supply one via WithHTTPClient. Don't construct your own — WithHTTPClient is how callers inject retry middleware, tracing, and custom transports, and bypassing p.cfg.HTTP() breaks that.

Stream construction

A streaming completion fans out into two goroutines: the caller's consumer drains the chunks channel, and your provider's producer pushes chunks into it. The llmrouter package provides the plumbing — you just need to drive it.

stream, sctx, hooks := llmrouter.NewStream(ctx)
go p.pump(sctx, resp, hooks)
return stream, nil

llmrouter.NewStream returns three values: the *Stream to hand back to the caller, a child context.Context the producer must check for cancellation, and a ProducerHooks struct with two callbacks:

• hooks.Send(chunk) — delivers one chunk. Returns false if the consumer cancelled; on false the producer should stop and call hooks.Finish(sctx.Err()).
• hooks.Finish(err) — terminates the stream. Pass nil on clean completion or an error on failure. Must be called exactly once. The producer goroutine should defer a single call (or use a sentinel) to guarantee this.

Inside the pump goroutine, check sctx.Done() in your read loop:

func (p *Provider) pump(ctx context.Context, resp *http.Response, hooks llmrouter.ProducerHooks) {
    defer resp.Body.Close()

    scanner := bufio.NewScanner(resp.Body)
    scanner.Buffer(make([]byte, 0, 64*1024), 1024*1024) // 1 MiB max line

    var dataLines []string
    for scanner.Scan() {
        select {
        case <-ctx.Done():
            hooks.Finish(ctx.Err())
            return
        default:
        }

        line := scanner.Text()
        if line == "" {
            // event boundary: parse and emit
            if len(dataLines) == 0 {
                continue
            }
            payload := strings.Join(dataLines, "\n")
            dataLines = dataLines[:0]

            if payload == "[DONE]" {
                hooks.Finish(nil)
                return
            }
            chunk, ok := decodeChunk(payload)
            if !ok {
                continue // tolerate malformed payloads
            }
            if !hooks.Send(chunk) {
                hooks.Finish(ctx.Err())
                return
            }
            continue
        }
        switch {
        case strings.HasPrefix(line, "data: "):
            dataLines = append(dataLines, strings.TrimPrefix(line, "data: "))
        case strings.HasPrefix(line, "data:"):
            dataLines = append(dataLines, strings.TrimPrefix(line, "data:"))
        }
    }

    if err := scanner.Err(); err != nil && err != io.EOF {
        hooks.Finish(fmt.Errorf("myprovider: read stream: %w", err))
        return
    }
    hooks.Finish(nil)
}

The 1 MiB scanner buffer is the convention both shipping providers use; an SSE payload over 1 MiB is almost always pathological.

SSE parsing rules

The Server-Sent Events grammar the providers care about:

• Lines starting with data: are payload. Leading space after the colon is optional in the spec; strip both "data: " and "data:".
• Multi-line data: each data: line is one payload line, joined with "\n" when the event terminates.
• An empty line terminates an event — that's when you parse and emit.
• Lines starting with event: set the event type (Anthropic uses this; OpenAI does not).
• Lines starting with : are comments — heartbeats from the server. Ignore them.
• data: [DONE] terminates OpenAI-shaped streams. Translating providers (like Anthropic) use a typed event (message_stop) instead.

Chunk normalization

Each emitted llmrouter.Chunk should have:

• ID — a stable id for the whole stream. OpenAI passes the upstream's id through; Anthropic generates one (chatcmpl-<uuid>) per stream and reuses it.
• Object — always "chat.completion.chunk" for streaming. Hard-code it.
• Created — Unix timestamp captured once at stream start (Anthropic) or copied from the upstream chunk (OpenAI).
• Model — the model id, ideally from the upstream's reply since some upstreams normalize it (e.g. claude-3-5-sonnet-latest → a concrete dated id).
• Choices — at least one Choice for any chunk carrying a delta. The first chunk of a stream should set Delta.Role = "assistant"; subsequent chunks set Delta.Content; the final chunk sets FinishReason.
• Usage — populated on the final chunk if the upstream reports it. Leave nil on intermediate chunks.
• Raw — always populate. Set Chunk.Raw to the original wire bytes of the chunk (or, for translating providers, the bytes of the OpenAI-shaped chunk after re-marshaling). Gateway / proxy use cases depend on this. Treat the Raw contract as part of the provider's API.

Error handling

Non-2xx HTTP responses become *llmrouter.ErrUpstream with the provider's name, the status code, and a snippet of the response body:

if resp.StatusCode >= 400 {
    defer resp.Body.Close()
    body, _ := io.ReadAll(io.LimitReader(resp.Body, 8*1024))
    return nil, &llmrouter.ErrUpstream{
        Provider:   p.Name(),
        StatusCode: resp.StatusCode,
        Body:       strings.TrimSpace(string(body)),
    }
}

Cap the body read so a misbehaving upstream can't OOM your process. The OpenAI provider caps at 1 KiB; Anthropic at 8 KiB. Pick what matches your upstream's typical error envelope size.

Errors that occur during streaming (IO error, unparseable framing, malformed JSON the consumer needs to know about) flow through hooks.Finish(err), surfacing on Stream.Err() after the chunks channel closes. Errors that occur before the stream starts (HTTP 4xx/5xx, network refused) return directly from CompletionStream.

Putting it together

The full CompletionStream:

func (p *Provider) CompletionStream(
    ctx context.Context, req llmrouter.ChatRequest,
) (*llmrouter.Stream, error) {
    body, err := buildRequestBody(req)
    if err != nil {
        return nil, fmt.Errorf("myprovider: build request: %w", err)
    }

    hreq, err := http.NewRequestWithContext(ctx, http.MethodPost,
        p.cfg.BaseURL+"/chat/completions", bytes.NewReader(body))
    if err != nil {
        return nil, err
    }
    hreq.Header.Set("Content-Type", "application/json")
    hreq.Header.Set("Accept", "text/event-stream")
    hreq.Header.Set("Authorization", "Bearer "+p.cfg.APIKey)

    resp, err := p.cfg.HTTP().Do(hreq)
    if err != nil {
        return nil, fmt.Errorf("myprovider: http: %w", err)
    }
    if resp.StatusCode >= 400 {
        defer resp.Body.Close()
        body, _ := io.ReadAll(io.LimitReader(resp.Body, 8*1024))
        return nil, &llmrouter.ErrUpstream{
            Provider:   p.Name(),
            StatusCode: resp.StatusCode,
            Body:       strings.TrimSpace(string(body)),
        }
    }

    stream, sctx, hooks := llmrouter.NewStream(ctx)
    go p.pump(sctx, resp, hooks)
    return stream, nil
}

Testing

The pattern both shipping providers use: httptest.NewServer with a handler that emits a fixed SSE script, point the provider at the test server's URL via WithBaseURL, drain the stream, assert on chunk.Choices[0].Delta.Content, the final FinishReason, and Stream.Err() == nil.

package myprovider_test

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "net/http/httptest"
    "strings"
    "testing"

    "github.com/elloloop/llmrouter"
    "github.com/elloloop/llmrouter/providers/myprovider"
)

const sseScript = `data: {"id":"x","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant"}}]}

data: {"id":"x","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hello"}}]}

data: {"id":"x","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" world"}}]}

data: {"id":"x","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":4,"completion_tokens":2,"total_tokens":6}}

data: [DONE]

`

func TestCompletionStream(t *testing.T) {
    srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "text/event-stream")
        io.WriteString(w, sseScript)
    }))
    t.Cleanup(srv.Close)

    p, err := myprovider.New(
        llmrouter.WithAPIKey("test-key"),
        llmrouter.WithBaseURL(srv.URL),
    )
    if err != nil {
        t.Fatalf("New: %v", err)
    }

    stream, err := p.CompletionStream(context.Background(), llmrouter.ChatRequest{
        Model: "test-model",
        Messages: []llmrouter.Message{
            llmrouter.TextMessage("user", "hi"),
        },
    })
    if err != nil {
        t.Fatalf("CompletionStream: %v", err)
    }

    var (
        body   strings.Builder
        finish string
        usage  *llmrouter.Usage
        count  int
    )
    for chunk := range stream.Chunks() {
        count++
        for _, c := range chunk.Choices {
            body.WriteString(c.Delta.Content)
            if c.FinishReason != "" {
                finish = c.FinishReason
            }
        }
        if chunk.Usage != nil {
            usage = chunk.Usage
        }
    }
    if err := stream.Err(); err != nil {
        t.Fatalf("Stream.Err: %v", err)
    }

    if got, want := body.String(), "Hello world"; got != want {
        t.Errorf("body = %q, want %q", got, want)
    }
    if finish != "stop" {
        t.Errorf("finish_reason = %q, want %q", finish, "stop")
    }
    if usage == nil || usage.TotalTokens != 6 {
        t.Errorf("usage = %+v, want total=6", usage)
    }
    if count < 4 {
        t.Errorf("got %d chunks, want >= 4", count)
    }
    fmt.Println("ok")
}

Add cases for: upstream 4xx returning ErrUpstream; context cancellation closing the chunks channel cleanly; malformed-payload tolerance (a bad chunk in the middle of a good stream shouldn't abort the whole stream).

The v0.2 candidates

Three cloud-vendor providers are on the v0.2 roadmap:

• Azure OpenAI Service — deployment-scoped URL of the form https://{resource}.openai.azure.com/openai/deployments/{deployment}/chat/completions with an api-key header (not bearer) and an api-version query parameter passed via WithExtra("api-version", "2024-10-21"). Request and response shapes are OpenAI's, so this is closer to a fork of the OpenAI provider than a fresh implementation. Upstream docs: Azure OpenAI reference.
• AWS Bedrock — SigV4-signed requests to https://bedrock-runtime.{region}.amazonaws.com/model/{model-id}/invoke-with-response-stream with per-model-family body shapes (Anthropic on Bedrock uses Anthropic's body; Cohere uses Cohere's; Meta Llama uses its own). The provider negotiates the body shape from the model id and streams Bedrock's event-stream framing (binary, not SSE) into OpenAI-shaped chunks. Upstream docs: InvokeModelWithResponseStream.
• Google Vertex AI — Application Default Credentials for auth (google.golang.org/api/option) with WithExtra("project", ...) and WithExtra("region", ...). Targets the streamGenerateContent endpoint for Gemini and the partner-model passthrough endpoints for Anthropic / Meta on Vertex. Upstream docs: Vertex AI inference.

If you're interested in contributing one of these, file an issue on elloloop/llmrouter first so we can align on the shape before you spend a weekend on it.

PR checklist

A new provider PR should land with the following (CONTRIBUTING.md expands on each — that file is in flight, this list is the summary):

• Provider lives at providers/<name>/ with the same file layout as the existing two (<name>.go + <name>_test.go).
• New calls llmrouter.NewConfig, requires an API key (or documents why it doesn't), defaults the base URL, and wraps config errors in llmrouter.ErrInvalidConfig.
• Name() returns a stable lowercase id used in ErrUpstream.Provider.
• CompletionStream returns *llmrouter.ErrUpstream for non-2xx, drives a goroutine via llmrouter.NewStream, calls hooks.Finish exactly once, and respects sctx.Done() in its read loop.
• Every emitted Chunk populates Raw. Either the original upstream bytes (passthrough providers) or the re-marshaled OpenAI-shape bytes (translating providers).
• Tests cover: happy path, upstream 4xx → ErrUpstream, context cancellation, malformed payload tolerance, final Usage when the upstream reports it.
• go vet ./... clean, golangci-lint run clean, go test ./... passing.
• A docs page under docs-site/src/pages/docs/providers/<name>.astro following the structure of the OpenAI and Anthropic pages.

See also

• OpenAI provider — reference implementation of a passthrough provider.
• Anthropic provider — reference implementation of a translating provider.
• Streaming model — the Stream + ProducerHooks contract.
• API reference — every exported type and option.