Last updated 2026-05-17

The Router — Magic Model × Platform Resolution

The headline value proposition of llmrouter is to decouple the model vendor (OpenAI, Anthropic, Meta/Llama, Mistral, Cohere, Google, xAI, DeepSeek) from the hosting platform (the vendor's own API, AWS Bedrock, Google Vertex AI, Azure AI Foundry, OpenRouter, Together, Groq, Fireworks, Cerebras, DeepSeek, Perplexity, xAI). A user wants “Claude on Bedrock” or “Llama on Groq” — they should not need to know that the first answer lives in providers/bedrock and the second in providers/groq.

The router package is the resolver that turns a {Model, Platform, Credentials} tuple into a working llmrouter.Provider. It infers the model family from the id, validates that the platform can serve that family, gathers credentials (from Credentials or the environment), and constructs the underlying provider.

The minimal API

One function, one request type, one result.

import "github.com/elloloop/llmrouter/router"

p, err := router.Resolve(router.Request{
    Model:    "claude-3-5-sonnet-20241022",
    Platform: router.PlatformBedrock,
    Credentials: router.Credentials{
        AWSRegion: "us-east-1",
    },
})
if err != nil {
    log.Fatal(err)
}
// p is a working llmrouter.Provider — exactly the shape every other
// provider package returns.
stream, err := p.CompletionStream(ctx, llmrouter.ChatRequest{
    Model:     "claude-3-5-sonnet-20241022",
    MaxTokens: 1024,
    Messages:  []llmrouter.Message{ llmrouter.TextMessage("user", "hi") },
})

The 12 platforms

A Platform identifies where the model runs. Pass PlatformAuto (the zero value) to let the router pick the first platform whose credentials are present in the environment.

The 9 model families

A ModelFamily identifies which vendor built the model, independent of where it's hosted. The router infers the family from the model id via simple prefix matching after stripping any hosting-platform vendor prefix.

Hosting-platform vendor prefixes (anthropic., meta., mistral., cohere., amazon., ai21., stability.) are stripped before matching, so anthropic.claude-3-5-sonnet-20241022-v2:0 still resolves to FamilyAnthropic.

The full routing matrix

Rows are model families; columns are platforms. A cell shows the underlying provider package the router constructs.

Example: explicit — Claude on Bedrock

You know exactly what you want: Claude 3.5 Sonnet served from AWS Bedrock in us-east-1. Tell the router and skip the Bedrock model-id translation by hand.

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/elloloop/llmrouter"
    "github.com/elloloop/llmrouter/router"
)

func main() {
    p, err := router.Resolve(router.Request{
        Model:    "claude-3-5-sonnet-20241022",
        Platform: router.PlatformBedrock,
        Credentials: router.Credentials{
            AWSRegion: "us-east-1",
        },
    })
    if err != nil {
        log.Fatal(err)
    }

    // Optional: rewrite the model id to Bedrock's prefixed form
    // ("anthropic.claude-3-5-sonnet-20241022-v2:0").
    req := router.ApplyModelTranslation(llmrouter.ChatRequest{
        Model:     "claude-3-5-sonnet-20241022",
        MaxTokens: 1024,
        Messages: []llmrouter.Message{
            llmrouter.TextMessage("user", "What region am I served from?"),
        },
    }, router.PlatformBedrock)

    stream, err := p.CompletionStream(context.Background(), req)
    if err != nil {
        log.Fatal(err)
    }
    for chunk := range stream.Chunks() {
        for _, c := range chunk.Choices {
            fmt.Print(c.Delta.Content)
        }
    }
    if err := stream.Err(); err != nil {
        log.Fatal(err)
    }
}

Example: auto-resolve from env vars

For the common case of “use whatever credentials are set, pick the best available platform,” pass PlatformAuto (or use the ResolveFromEnv shortcut).

p, err := router.ResolveFromEnv("claude-3-5-sonnet-20241022")
if err != nil {
    log.Fatal(err)
}
// If ANTHROPIC_API_KEY is set, p talks to direct Anthropic.
// Else if AWS_REGION is set, p talks to Bedrock.
// Else if GOOGLE_CLOUD_PROJECT + Credentials.GCPRegion are set,
//   p talks to Vertex Model Garden.
// Else if AZURE_OPENAI_ENDPOINT + AZURE_OPENAI_API_KEY are set,
//   p talks to Azure AI Foundry — Anthropic.
// Else if OPENROUTER_API_KEY is set, p talks to OpenRouter.
// Else: ErrNoAutoPlatform.

PlatformAuto preference order

When PlatformAuto is in play the router scans this list in order and picks the first platform that (a) supports the inferred family and (b) has the required credentials:

1. PlatformDirect
2. PlatformBedrock
3. PlatformVertex
4. PlatformAzure
5. PlatformOpenRouter
6. PlatformTogether
7. PlatformGroq
8. PlatformFireworks
9. PlatformCerebras
10. PlatformDeepSeek
11. PlatformPerplexity
12. PlatformxAI

The order rewards specificity: direct first (you opted into a vendor by setting their key), then cloud accounts (Bedrock / Vertex / Azure, in alphabetical AWS-Google-Microsoft cycling order), then fan-out proxies (OpenRouter, Together, Groq, …). If you want a different priority, write your own preference loop around router.SupportedPlatforms.

Bedrock + Vertex model ID translation

Bedrock and Vertex address Claude / Llama / Mistral / Cohere with platform-specific prefixes and version suffixes. ApplyModelTranslation rewrites a vendor-neutral id to the right form so application code can keep using the canonical id and let the router translate at the boundary.

Pass a model id that already has the platform-specific prefix and the translator leaves it alone — so feeding anthropic.claude-3-5-sonnet-20241022-v2:0 to a Bedrock route is a no-op.

Env-var fallback registry

When Credentials fields are empty the router falls back to environment variables. The mapping lives in the global router.DefaultEnvVars — override it to swap in custom env var names (handy for multi-tenant CI runners).

var DefaultEnvVars = router.EnvVars{
    OpenAIAPIKey:    "OPENAI_API_KEY",
    AnthropicAPIKey: "ANTHROPIC_API_KEY",
    MistralAPIKey:   "MISTRAL_API_KEY",
    CohereAPIKey:    "COHERE_API_KEY",
    GoogleAPIKey:    "GOOGLE_API_KEY",
    GrokAPIKey:      "GROK_API_KEY",
    DeepSeekAPIKey:  "DEEPSEEK_API_KEY",

    OpenRouterAPIKey: "OPENROUTER_API_KEY",
    TogetherAPIKey:   "TOGETHER_API_KEY",
    GroqAPIKey:       "GROQ_API_KEY",
    FireworksAPIKey:  "FIREWORKS_API_KEY",
    CerebrasAPIKey:   "CEREBRAS_API_KEY",
    PerplexityAPIKey: "PERPLEXITY_API_KEY",

    AzureBaseURL: "AZURE_OPENAI_ENDPOINT",
    AzureAPIKey:  "AZURE_OPENAI_API_KEY",
    AWSRegion:    "AWS_REGION",
    AWSRegionAlt: "AWS_DEFAULT_REGION",
    GCPProject:   "GOOGLE_CLOUD_PROJECT",
}

// To use custom env var names — e.g. multi-tenant CI:
router.DefaultEnvVars.OpenAIAPIKey = "TENANT_A_OPENAI_KEY"

AWS credentials (access key, secret, session token) are not in this list — Bedrock uses the standard AWS credential chain (env vars, profile, IAM role) via the AWS SDK. The router only needs the AWS region from Credentials.AWSRegion or the AWS_REGION / AWS_DEFAULT_REGION env vars.

Errors

• router.ErrEmptyModel — Request.Model is empty. Always check this; an empty model bypasses the family inference entirely.
• router.ErrUnsupportedRoute — the inferred family cannot be served by the requested platform (e.g. Gemini on Bedrock).
• router.ErrMissingCredentials — required credentials are missing. The wrapped message names the field the caller forgot.
• router.ErrNoAutoPlatform — PlatformAuto could not find any supported platform with credentials.
• Wrapped llmrouter.ErrInvalidConfig — the underlying provider's New() rejected the assembled options (e.g. Azure missing api-version).

When to use the router vs constructing a provider directly

Use router.Resolve when:

• Your application is multi-vendor — you switch between Claude, GPT, Llama, Gemini based on cost, latency, or availability.
• You want a single config surface across environments (env vars in CI, Credentials struct in production).
• You're building a gateway that routes tenant-supplied model ids without knowing the family in advance.

Construct a provider directly (openai.New(...), anthropic.New(...), …) when:

• You only use one vendor and one platform. The direct constructor has a sharper API surface and no resolution overhead.
• You need provider-specific options not modelled in router.Credentials (custom HTTP middleware, vendor retry policies, specialised constructors like anthropic.NewRecommendedEmbedder).

See also

• API reference: router — every public symbol.
• Azure AI Foundry — Anthropic — one of the providers the router can resolve to.
• Vertex AI Model Garden — Anthropic — another.
• Azure AI Foundry — Serverless — the catch-all for Llama / Mistral / Cohere on Azure.
• Switching providers at runtime — pattern for hot-swapping providers behind a single config.