Last updated 2026-05-17

OpenAI Provider

The openai provider speaks the OpenAI Chat Completions wire format natively. It is the simplest provider in the library and also the most permissive: incoming requests are forwarded as native OpenAI shape, and the upstream SSE event stream is decoded into llmrouter.Chunk values while the original wire bytes are preserved in Chunk.Raw. This is what makes it the right backend for proxy and gateway use cases.

The same provider implementation backs every OpenAI-compatible endpoint: OpenRouter, Together, Groq, DeepSeek, Fireworks, Perplexity, vLLM, Ollama, LM Studio, LocalAI. Use llmrouter.WithBaseURL to point at the host you want.

Import path

import (
    "github.com/elloloop/llmrouter"
    "github.com/elloloop/llmrouter/providers/openai"
)

Default endpoint

With no WithBaseURL override, the provider targets https://api.openai.com/v1. The request URL is {BaseURL}/chat/completions.

Construction

openai.New takes a variadic list of shared llmrouter.Option values and returns (*openai.Provider, error). The constructor validates that an API key is present and falls back to the default base URL if none was supplied. 

provider, err := openai.New(
    llmrouter.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
)
if err != nil {
    log.Fatal(err)
}

WithAPIKey rejects an empty string. If you need to talk to a local endpoint that doesn't require auth (Ollama, LM Studio, LocalAI), pass a non-empty placeholder — the upstream will ignore it.

What's wire-level

Unlike the Anthropic provider, this provider does not translate the request body or the response stream. The outgoing JSON is the OpenAI Chat Completions request, and the incoming SSE events are OpenAI chat.completion.chunk events. Three concrete consequences follow:

  • Request passthrough. If ChatRequest.Raw is set, that JSON object is the request body (with model, stream, and stream_options overlaid). Any field that the typed ChatRequest doesn't model — tools, response_format, logprobs, seed, multimodal content arrays — survives because the bytes are forwarded directly.
  • Response passthrough. Every chunk's Raw field is the original SSE data payload. A gateway can forward those bytes back to its caller as SSE without re-marshaling. The typed fields (ID, Choices, Usage) are populated for application code that doesn't care about wire format.
  • Forced streaming. The provider rewrites the request to set "stream": true and (if the caller didn't supply one) "stream_options": {"include_usage": true} so the upstream emits a final usage block. Non-streaming calls are not supported in v0.1.

Authentication

OpenAI uses bearer-token auth. The provider sets the Authorization: Bearer <key> header on every request. There are no other headers required for the public OpenAI API.

For organisation / project scoping, see Custom organization / project headers below.

Full example

A complete program that prints the streamed completion to stdout:

package main


import (
    "context"
    "fmt"
    "log"
    "os"


    "github.com/elloloop/llmrouter"
    "github.com/elloloop/llmrouter/providers/openai"
)


func main() {
    p, err := openai.New(
        llmrouter.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
    )
    if err != nil {
        log.Fatal(err)
    }


    ctx := context.Background()
    stream, err := p.CompletionStream(ctx, llmrouter.ChatRequest{
        Model: "gpt-4o-mini",
        Messages: []llmrouter.Message{
            llmrouter.TextMessage("system", "You are a concise assistant."),
            llmrouter.TextMessage("user", "Explain the speed of light in one sentence."),
        },
    })
    if err != nil {
        log.Fatal(err)
    }


    for chunk := range stream.Chunks() {
        for _, choice := range chunk.Choices {
            fmt.Print(choice.Delta.Content)
        }
    }
    if err := stream.Err(); err != nil {
        log.Fatal(err)
    }
    fmt.Println()
}

Multiple choices (n > 1)

When the upstream returns multiple choices for a single request, each chunk's Choices slice has one entry per choice with a distinct Index. The provider passes the upstream's indices through unchanged; consumers should bucket by Choice.Index: 

bodies := map[int]*strings.Builder{}


stream, err := p.CompletionStream(ctx, llmrouter.ChatRequest{
    Model:    "gpt-4o-mini",
    Messages: []llmrouter.Message{llmrouter.TextMessage("user", "Give me three taglines.")},
    Raw:      json.RawMessage(`{"n":3}`),
})
if err != nil {
    log.Fatal(err)
}


for chunk := range stream.Chunks() {
    for _, c := range chunk.Choices {
        b, ok := bodies[c.Index]
        if !ok {
            b = &strings.Builder{}
            bodies[c.Index] = b
        }
        b.WriteString(c.Delta.Content)
    }
}


for idx, b := range bodies {
    fmt.Printf("choice %d: %s\n", idx, b.String())
}

Note that n > 1 is not a typed field on ChatRequest in v0.1 — pass it via Raw.

Tool use / function calling

The typed ChatRequest does not yet model OpenAI's tools, tool_choice, or assistant-side tool_calls messages. To use function calling, build the request body yourself and pass it via ChatRequest.Raw. The byte-passthrough behavior ensures every field reaches OpenAI unchanged: 

reqBody := []byte(`{
  "model": "gpt-4o-mini",
  "messages": [
    {"role": "user", "content": "What is the weather in Paris?"}
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "parameters": {
          "type": "object",
          "properties": {
            "city": {"type": "string"}
          },
          "required": ["city"]
        }
      }
    }
  ],
  "tool_choice": "auto"
}`)


stream, err := p.CompletionStream(ctx, llmrouter.ChatRequest{
    Model: "gpt-4o-mini",
    Raw:   reqBody,
})
if err != nil {
    log.Fatal(err)
}


for chunk := range stream.Chunks() {
    // chunk.Raw contains the full SSE payload, including any
    // tool_calls field that the typed Choice/Delta does not model.
    fmt.Println(string(chunk.Raw))
}

Tool-call deltas live under choices[].delta.tool_calls in the wire JSON; read them off Chunk.Raw until v0.3 surfaces tool calls as a typed concept.

Vision / multimodal

Message.Content is a json.RawMessage, so a multimodal content array passes through unchanged. llmrouter.TextMessage is only useful for plain text; for image inputs, build the Content bytes yourself: 

content, _ := json.Marshal([]map[string]any{
    {"type": "text", "text": "What's in this image?"},
    {
        "type": "image_url",
        "image_url": map[string]string{
            "url": "https://example.com/cat.jpg",
        },
    },
})


stream, err := p.CompletionStream(ctx, llmrouter.ChatRequest{
    Model: "gpt-4o",
    Messages: []llmrouter.Message{{
        Role:    "user",
        Content: content,
    }},
})

Base64-encoded data URLs work the same way — the bytes are not inspected by the library.

Response format (JSON mode)

response_format is another field that isn't on the typed request. Pass it via Raw: 

reqBody := []byte(`{
  "model": "gpt-4o-mini",
  "messages": [
    {"role": "system", "content": "Return JSON only."},
    {"role": "user", "content": "List three primary colors."}
  ],
  "response_format": {"type": "json_object"}
}`)


stream, err := p.CompletionStream(ctx, llmrouter.ChatRequest{
    Model: "gpt-4o-mini",
    Raw:   reqBody,
})

Structured outputs ({"type": "json_schema", "json_schema": ...}) use the same pattern.

Custom organization / project headers

OpenAI's OpenAI-Organization and OpenAI-Project headers are not yet exposed via options. Use WithHTTPClient with a wrapping http.RoundTripper that adds them: 

type orgTransport struct {
    base    http.RoundTripper
    org     string
    project string
}


func (t *orgTransport) RoundTrip(req *http.Request) (*http.Response, error) {
    if t.org != "" {
        req.Header.Set("OpenAI-Organization", t.org)
    }
    if t.project != "" {
        req.Header.Set("OpenAI-Project", t.project)
    }
    return t.base.RoundTrip(req)
}


httpClient := &http.Client{
    Timeout: 120 * time.Second,
    Transport: &orgTransport{
        base:    http.DefaultTransport,
        org:     os.Getenv("OPENAI_ORG"),
        project: os.Getenv("OPENAI_PROJECT"),
    },
}


p, err := openai.New(
    llmrouter.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
    llmrouter.WithHTTPClient(httpClient),
)

The same pattern works for any custom request-scoped header (tracing, tenant id, audit context).

Token counting

The provider forces stream_options.include_usage: true so the upstream sends a final chunk with a usage object. That chunk surfaces as Chunk.Usage: 

var usage *llmrouter.Usage
for chunk := range stream.Chunks() {
    for _, c := range chunk.Choices {
        fmt.Print(c.Delta.Content)
    }
    if chunk.Usage != nil {
        usage = chunk.Usage
    }
}
if usage != nil {
    fmt.Printf("\nprompt=%d completion=%d total=%d\n",
        usage.PromptTokens,
        usage.CompletionTokens,
        usage.TotalTokens,
    )
}

The final usage chunk has an empty Choices slice, so the loop above is safe — the for _, c := range chunk.Choices inner loop simply doesn't execute on that chunk.

Error handling

A 4xx or 5xx response from the upstream surfaces as a typed *llmrouter.ErrUpstream with the provider name, the HTTP status code, and up to 1 KiB of the response body: 

stream, err := p.CompletionStream(ctx, req)
if err != nil {
    var upstream *llmrouter.ErrUpstream
    if errors.As(err, &upstream) {
        log.Printf("openai error: status=%d body=%s",
            upstream.StatusCode, upstream.Body)
        if upstream.StatusCode == 429 {
            // back off and retry
        }
        return
    }
    log.Fatal(err)
}

Errors that surface during streaming (a malformed SSE payload, an IO error mid-stream) come out of Stream.Err() after the chunks channel closes — not from CompletionStream itself.

Rate-limit retry example

OpenAI rate limits use HTTP 429. The simplest retry strategy wraps the HTTP client with a transport that retries on 429 with exponential backoff: 

type retryTransport struct {
    base     http.RoundTripper
    attempts int
}


func (t *retryTransport) RoundTrip(req *http.Request) (*http.Response, error) {
    var body []byte
    if req.Body != nil {
        body, _ = io.ReadAll(req.Body)
        req.Body = io.NopCloser(bytes.NewReader(body))
    }


    backoff := 500 * time.Millisecond
    for attempt := 0; attempt < t.attempts; attempt++ {
        if attempt > 0 && body != nil {
            req.Body = io.NopCloser(bytes.NewReader(body))
        }
        resp, err := t.base.RoundTrip(req)
        if err != nil {
            return nil, err
        }
        if resp.StatusCode != http.StatusTooManyRequests || attempt == t.attempts-1 {
            return resp, nil
        }
        resp.Body.Close()
        select {
        case <-time.After(backoff):
        case <-req.Context().Done():
            return nil, req.Context().Err()
        }
        backoff *= 2
    }
    return nil, errors.New("unreachable")
}


httpClient := &http.Client{
    Timeout: 120 * time.Second,
    Transport: &retryTransport{
        base:     http.DefaultTransport,
        attempts: 4,
    },
}


p, err := openai.New(
    llmrouter.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
    llmrouter.WithHTTPClient(httpClient),
)

Production code should honour the Retry-After header rather than fixed backoff, and should bail out early on non-429 errors. This snippet is the minimal shape.

When to use this provider vs Anthropic

The two providers serve different shapes of project:

  • OpenAI provider — byte passthrough. Choose this when you need every field of every request and every chunk to reach the upstream and return to your caller unchanged. Gateways, proxies, audit shims, and "I just want to use a newer OpenAI feature the typed surface doesn't model yet" all want this behavior. It also backs every OpenAI-compatible endpoint.
  • Anthropic provider — full translation. Choose this when you want a single application code path that targets both OpenAI and Anthropic without branching. The provider translates the request from OpenAI shape to Anthropic's /v1/messages shape and translates the SSE event stream back. The translation is lossy by design — fields the typed surface doesn't model (tool messages, vision content) need to ride on Raw.

Many applications use both: OpenAI provider for the default request, Anthropic provider as a fallback (or vice versa). See Streaming model for the cancellation semantics that make fallback safe.

See also