Last updated 2026-05-17

Structured outputs (JSON Schema)

Structured outputs let you constrain a model's response to a JSON schema you control. Instead of asking the model nicely in the prompt to “please respond with JSON matching this shape,” you attach a schema to the request and the provider enforces it on the wire. Your application receives JSON that always parses against your schema — no more {"answer": "Sure! Here's the JSON you asked for: ..."} wrappers, no more truncated braces.

llmrouter exposes this via a single ChatRequest.ResponseSchema field. Each provider that supports the feature translates it to whatever the underlying wire format calls it. Application code looks the same regardless of vendor.

The problem with prompt-engineered JSON

Before structured outputs, getting reliable JSON out of a model meant some combination of:

• Putting an example JSON object in the system prompt.
• Telling the model in CAPS that the response MUST be valid JSON and NOTHING ELSE.
• Wrapping the model call in a retry loop that re-prompts on parse failure.
• Hoping for the best.

This works often enough to be tempting and fails often enough to be expensive. Failure modes include the model wrapping the JSON in Markdown fences, prefacing it with an apology, hallucinating fields not in the example, omitting required fields, or truncating mid-object at the token limit.

Structured outputs eliminate all of these — the provider constrains the model's sampling at decode time so the only tokens that can be emitted are ones that keep the partial response on-schema.

The API

Two new types on the root package:

type ResponseSchema struct {
    Name        string          // a short identifier; passed through to OpenAI's json_schema.name
    Description string          // optional; helps the model understand intent
    Schema      json.RawMessage // the JSON Schema body
    Strict      bool            // request strict-mode enforcement (OpenAI; ignored by others)
}

type ChatRequest struct {
    // ... existing fields ...
    ResponseSchema *ResponseSchema // attach a schema to constrain the output
}

The field is a pointer so the zero value (nil) means “don't ask for structured output.” Setting it switches the request to constrained mode without changing anything else about how you consume the response.

Cross-vendor translation

Example: extract a structured product from text

Say you want to parse a free-form product description into a typed struct. Define the Go struct, derive the JSON Schema, set it on the request, parse the result.

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "log"
    "os"

    "github.com/elloloop/llmrouter"
    "github.com/elloloop/llmrouter/providers/openai"
)

// The shape we want the model to produce.
type Product struct {
    Name        string   `json:"name"`
    PriceCents  int      `json:"price_cents"`
    Currency    string   `json:"currency"`
    InStock     bool     `json:"in_stock"`
    Categories  []string `json:"categories"`
}

// The JSON Schema that describes it. Keep this hand-written — it's
// the source of truth the provider enforces.
const productSchema = `{
    "type": "object",
    "properties": {
        "name":        {"type": "string"},
        "price_cents": {"type": "integer", "minimum": 0},
        "currency":    {"type": "string", "enum": ["USD", "EUR", "GBP"]},
        "in_stock":    {"type": "boolean"},
        "categories":  {"type": "array", "items": {"type": "string"}}
    },
    "required": ["name", "price_cents", "currency", "in_stock", "categories"],
    "additionalProperties": false
}`

func main() {
    p, err := openai.New(llmrouter.WithAPIKey(os.Getenv("OPENAI_API_KEY")))
    if err != nil {
        log.Fatal(err)
    }

    stream, err := p.CompletionStream(context.Background(), llmrouter.ChatRequest{
        Model: "gpt-4o-2024-08-06",
        Messages: []llmrouter.Message{
            llmrouter.TextMessage("system", "Extract structured product data from the user's text."),
            llmrouter.TextMessage("user", "The new Aeropress XL is in stock at $79.99, in coffee gear and brewing categories."),
        },
        ResponseSchema: &llmrouter.ResponseSchema{
            Name:        "product",
            Description: "A single product extracted from free-form text.",
            Schema:      json.RawMessage(productSchema),
            Strict:      true,
        },
    })
    if err != nil {
        log.Fatal(err)
    }

    var raw []byte
    for chunk := range stream.Chunks() {
        for _, c := range chunk.Choices {
            raw = append(raw, []byte(c.Delta.Content)...)
        }
    }
    if err := stream.Err(); err != nil {
        log.Fatal(err)
    }

    var product Product
    if err := json.Unmarshal(raw, &product); err != nil {
        log.Fatalf("model returned invalid JSON despite schema: %v", err)
    }
    fmt.Printf("%+v
", product)
}

The same program runs against Anthropic by changing the import and the model — but you parse the result from Choice.Delta.ToolCalls instead of from the text deltas. See the next section.

Reading the response on Anthropic

Because Anthropic implements structured outputs via forced tool-use, the model's reply arrives as a tool call rather than as free text. Your consumer needs to look at Choice.Delta.ToolCalls:

stream, _ := p.CompletionStream(ctx, llmrouter.ChatRequest{
    Model:     "claude-3-5-sonnet-latest",
    MaxTokens: 1024,
    Messages:  []llmrouter.Message{ /* ... */ },
    ResponseSchema: &llmrouter.ResponseSchema{
        Name:   "product",
        Schema: json.RawMessage(productSchema),
    },
})

var argsBuf []byte
for chunk := range stream.Chunks() {
    for _, c := range chunk.Choices {
        for _, tc := range c.Delta.ToolCalls {
            // The synthetic tool's name matches ResponseSchema.Name.
            argsBuf = append(argsBuf, []byte(tc.Function.Arguments)...)
        }
    }
}

var product Product
_ = json.Unmarshal(argsBuf, &product)

Application code that wants vendor parity can wrap this in a helper that returns []byte regardless of whether the model spoke via text deltas (OpenAI / Vertex / Gemini) or tool-call argument deltas (Anthropic).

Caveats

• Not all models support strict mode. On OpenAI, Strict: true requires gpt-4o-2024-08-06 or newer; older models accept the schema as a hint but do not enforce it.
• Anthropic returns the result via tool-call args. Your consumer must read from Choice.Delta.ToolCalls, not from Choice.Delta.Content. Write a small helper to paper over the difference if you need true cross-vendor portability.
• Models can still fail. Rare but real: out-of-tokens truncation, content-filter interruptions, and genuine model bugs can produce invalid JSON. Wrap the parse in a retry once and surface a typed error on the second failure.
• Schema complexity has a cost. Deeply nested schemas with many oneOf / anyOf branches make constrained decoding slower and more token-hungry. Keep schemas flat and small where you can.
• Strict mode disallows additionalProperties: true on OpenAI. Set it to false on every nested object, or strict mode will reject the schema at request time.

When to use structured outputs vs prompting

Reach for ResponseSchema when:

• The downstream code parses the response and you need that parse to succeed every time.
• You're populating a database row, calling an API, or driving a typed pipeline.
• The schema is small enough to describe in JSON Schema cleanly.

Stick with prompt-engineered JSON when:

• The response is consumed by a human and parse failures are recoverable.
• You're targeting providers that don't support structured outputs and adding a schema would be a no-op.
• The output shape genuinely varies per call — a schema with too many alternatives is harder to write than a free-form prompt with examples.

See also

• Provider interface — the chat surface this field hangs off.
• OpenAI provider — native response_format support.
• Anthropic provider — forced tool-use translation.
• Vertex AI provider — Gemini schema support via the genai SDK.