Last updated 2026-05-17

Byte Passthrough

If you're building an OpenAI-compatible gateway — a proxy in front of OpenAI, a multi-tenant key vault, an audit layer, a routing layer — there's a specific failure mode that bites you the moment you put real client SDKs in front of it: the bytes you emit have to be exactly the bytes the upstream emitted. Not equivalent JSON, not the same fields in a different order, not a re-marshaled struct. The same bytes.

llmrouter is built to make that easy. Every Chunk carries the original wire-format JSON in a Raw field, and every ChatRequest carries an original-bytes Raw field for the request side. The rest of this page covers when that matters, when it doesn't, and how to use it.

The problem

Suppose you write the obvious proxy: decode the OpenAI SSE chunk into a struct, look at it, re-encode it, write it back to the client. Functionally identical, right?

Three things break:

Field order. Go's encoding/json emits fields in the order they appear in the struct. OpenAI emits them in a different order. Client SDKs that diff against captured fixtures will see every chunk as "changed."
Unknown fields. OpenAI ships new fields all the time: logprobs, system_fingerprint, service_tier, model-specific extras. If your struct doesn't model them, your re-encoded chunk drops them. Some client SDKs use strict-mode JSON parsing that requires these fields, and break with a confusing schema error.
Whitespace and number formatting. A re-encoded 3.14 might become 3.140000. A pretty-printed input becomes minified output. Hash-based caches downstream invalidate on every chunk.

None of these are bugs in your code. They're bugs in the design. The fix is to never re-encode at all.

The solution

Every chunk on the response side carries its original bytes:

type Chunk struct {
    ID      string    `json:"id"`
    Object  string    `json:"object"`
    // ... typed fields ...

    // Raw is the original wire-format JSON for this chunk. Passthrough
    // providers populate this; consumers that want to forward bytes
    // unmodified should prefer Raw over re-marshaling the typed fields.
    Raw json.RawMessage `json:"-"`
}

The OpenAI provider populates Raw with the literal payload of the SSE data: line. The Anthropic provider populates it with the JSON it just synthesized in OpenAI shape (the Anthropic upstream emits a different wire format — see the caveat below).

The typed fields are also populated, so your gateway can inspect the chunk (read FinishReason, log Usage) without losing the original bytes you'll forward downstream.

When this matters

OpenAI-compatible proxies. Anything that fronts /v1/chat/completions and accepts the official OpenAI SDKs, LangChain, Cline, Continue, Cursor, etc. Those clients expect OpenAI's exact byte stream. Re-marshaling breaks them in subtle, hard-to-diagnose ways.
Audit / observability layers. If you're storing streams to S3 for replay later, you want the original bytes so the replay is byte-identical to what the client saw.
Cost-attribution gateways. Where you parse usage from the final chunk for billing, but the rest of the stream should flow through verbatim.
Schema-strict consumers. Any downstream that uses Pydantic, Zod, or another strict schema validator on OpenAI's response shape.

When it doesn't matter

CLI chatbots — you're concatenating Delta.Content into a string. Use the typed fields.
Multi-provider apps — you want the normalized shape, not the raw OpenAI/Anthropic bytes. Use the typed fields.
Server-side LLM workflows — you're parsing a single JSON object out of the response and using it. Use the typed fields.

The general rule: if you're displaying the response or using the response, use the typed fields. If you're forwarding the response, use Raw.

The request side: `ChatRequest.Raw`

The same idea on the way in:

type ChatRequest struct {
    Model    string    // ...
    Messages []Message // ...
    // ... other typed fields ...

    // Raw is the original request bytes for passthrough mode.
    Raw json.RawMessage `json:"-"`
}

When the OpenAI provider receives a ChatRequest with a non-empty Raw, it ignores the typed fields almost entirely. It does three minimal rewrites:

// buildRequestBody assembles the outgoing JSON. If req.Raw is supplied
// (passthrough mode), it is reused with the model field rewritten;
// otherwise the typed ChatRequest is marshaled. In both cases we force
// streaming on with include_usage so the upstream emits a final usage block.
func buildRequestBody(req llmrouter.ChatRequest) ([]byte, error) {
    var m map[string]json.RawMessage
    if len(req.Raw) > 0 {
        if err := json.Unmarshal(req.Raw, &m); err != nil {
            return nil, fmt.Errorf("openai: invalid raw request: %w", err)
        }
    } else {
        // ... marshal typed request ...
    }

    if req.Model != "" {
        mb, _ := json.Marshal(req.Model)
        m["model"] = mb
    }
    m["stream"] = json.RawMessage(`true`)
    if _, ok := m["stream_options"]; !ok {
        m["stream_options"] = json.RawMessage(`{"include_usage":true}`)
    }
    return json.Marshal(m)
}

Model is rewritten from req.Model, so your gateway can route gpt-5 requests to gpt-4o internally without the client knowing.
stream is forced true. Even if the client sent stream: false, the upstream call streams — and the proxy can decide to buffer-and-respond non-streaming if needed.
stream_options.include_usage is forced true if the client didn't set it, so the final chunk carries token counts.

Every other field — tools, tool_choice, response_format, logprobs, seed, parallel_tool_calls, anything OpenAI adds next year — flows through untouched.

The Anthropic caveat

Byte-passthrough cannot be byte-identical against the Anthropic upstream, because the wire format is different. Anthropic uses /v1/messages with named SSE event types (message_start, content_block_delta, message_delta, message_stop); OpenAI uses /v1/chat/completions with anonymous SSE data: frames containing OpenAI's chunk shape.

The Anthropic provider translates events into OpenAI-shape chunks and populates Chunk.Raw with that synthesized JSON. So downstream consumers of Raw still see consistent OpenAI-shape bytes — the abstraction holds — but those bytes are constructed by the library, not forwarded from Anthropic.

What this means in practice. A gateway built on llmrouter can transparently fail over from OpenAI to Anthropic and back; downstream OpenAI-SDK clients will see a consistent shape either way. They won't see bit-for-bit identical bytes per chunk across providers — there's no honest way to promise that — but they will see valid, parseable OpenAI chunks every time.

Full example: a 30-line OpenAI passthrough proxy

This is a complete http.Handler that accepts an OpenAI chat-completion request on any path, forwards it through llmrouter to OpenAI, and streams the raw response bytes back to the client as SSE. It's deliberately small — a real deployment would add auth, rate limiting, logging, and per-tenant routing.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "log"
    "net/http"
    "os"

    "github.com/elloloop/llmrouter"
    "github.com/elloloop/llmrouter/providers/openai"
)

func main() {
    p, err := openai.New(llmrouter.WithAPIKey(os.Getenv("OPENAI_API_KEY")))
    if err != nil {
        log.Fatal(err)
    }
    http.HandleFunc("/v1/chat/completions", proxyHandler(p))
    log.Fatal(http.ListenAndServe(":8080", nil))
}

// proxyHandler returns an SSE handler that forwards the original
// request body through llmrouter as a passthrough request and streams
// Chunk.Raw back to the client unmodified.
func proxyHandler(p llmrouter.Provider) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        body, err := io.ReadAll(r.Body)
        if err != nil {
            http.Error(w, err.Error(), http.StatusBadRequest)
            return
        }
        // Peek at the model so we can rewrite it if we wanted (we
        // don't, in this example — we just need it set on the
        // ChatRequest so the OpenAI provider knows what to forward).
        var head struct {
            Model string `json:"model"`
        }
        _ = json.Unmarshal(body, &head)

        stream, err := p.CompletionStream(r.Context(), llmrouter.ChatRequest{
            Model: head.Model,
            Raw:   body,
        })
        if err != nil {
            http.Error(w, err.Error(), http.StatusBadGateway)
            return
        }

        w.Header().Set("Content-Type", "text/event-stream")
        w.Header().Set("Cache-Control", "no-cache")
        flusher, _ := w.(http.Flusher)
        for chunk := range stream.Chunks() {
            fmt.Fprintf(w, "data: %s\n\n", chunk.Raw)
            if flusher != nil {
                flusher.Flush()
            }
        }
        // OpenAI clients expect a terminal [DONE] frame.
        fmt.Fprint(w, "data: [DONE]\n\n")
        if flusher != nil {
            flusher.Flush()
        }

        if err := stream.Err(); err != nil {
            log.Printf("stream ended with error: %v", err)
        }
    }
}

What the client sees: byte-identical OpenAI SSE chunks. What you get on the server: the ability to inspect every chunk (read the typed Choices[0].FinishReason, log the final Usage) without re-encoding any of the bytes you forward.

Tip. Wire r.Context() into CompletionStream as shown. When the client disconnects (e.g. closes the browser tab), the request context cancels, which cancels the upstream OpenAI HTTP call, which means you stop paying for tokens you're not going to deliver. See Context & Cancellation.

Next steps

The Streaming Model — full coverage of Stream, chunks, and cancellation.
OpenAI provider — WithBaseURL for OpenRouter, Groq, Together, vLLM, and other OpenAI-compatible endpoints.
Error Handling — how upstream 4xx/5xx errors surface and how to retry safely.

Byte Passthrough

The problem

The solution

When this matters

When it doesn't matter

The request side: ChatRequest.Raw

The Anthropic caveat

Full example: a 30-line OpenAI passthrough proxy

Next steps

The request side: `ChatRequest.Raw`