Last updated 2026-05-17

Azure AI Foundry — Serverless API (MaaS)

The azureserverless provider talks to non-OpenAI, non-Anthropic models hosted on Azure AI Foundry's Serverless API deployment mode (Models-as-a-Service, MaaS). This is the catch-all provider for the Foundry marketplace: Llama (Meta), Mistral, Cohere, Phi (Microsoft), Jais, Nemotron (NVIDIA), DeepSeek, and any other OpenAI-compatible chat-completions model that ships in Foundry's catalogue.

On the wire the request and response are OpenAI-shaped chat completions — Foundry's Serverless API exposes every model through the same /v1/chat/completions contract. The differences from direct OpenAI are the hostname pattern, the lack of an api-version query, and the Azure-style api-key (or AAD) auth.

Picking the right Foundry provider

  • providers/azureopenai — OpenAI's own GPT models on Foundry. Different URL shape (needs api-version query + /openai/deployments/<name> path).
  • providers/azureanthropic — Claude on Foundry. Different body shape (native Anthropic /messages).
  • providers/azureserverless (this page) — everything else on Foundry. OpenAI-shaped chat completions.

Import path

import (
    "github.com/elloloop/llmrouter"
    "github.com/elloloop/llmrouter/providers/azureserverless"
)

Two URL shapes

Foundry exposes Serverless API models under two endpoint patterns. Pick by whether you provisioned a single deployment per model or a shared hub endpoint that fans out to multiple deployments. Both are passed in via llmrouter.WithBaseURL — the provider appends /v1/chat/completions for you.

  1. Deployment-scoped (single model per deployment). The deployment name and region are baked into the hostname; the request body's model field is echoed back unchanged.
    https://<deployment>.<region>.models.ai.azure.com
    Example: https://my-llama.eastus.models.ai.azure.com.
  2. Hub-scoped (single hub, multiple models). The hostname is shared; the model field in the body picks which deployment.
    https://<hub-endpoint>

Authentication

Auth is XOR: provide exactly one of llmrouter.WithAPIKey(...) or azureserverless.WithAADToken(...). Providing both — or neither — fails at New() with ErrInvalidConfig.

  • API key. Flows as the Azure-specific api-key request header — not Authorization: Bearer. The deployment's primary or secondary key from the Foundry portal.
  • AAD bearer. Same shape as on azureanthropic: a per-request AADTokenSource callback returning a fresh bearer token. Use this when you want service-principal or managed-identity auth instead of a static key.

Construction

A single-model deployment with API-key auth: 

p, err := azureserverless.New(
    llmrouter.WithAPIKey(os.Getenv("AZURE_FOUNDRY_KEY")),
    llmrouter.WithBaseURL("https://my-llama.eastus.models.ai.azure.com"),
)
if err != nil {
    log.Fatal(err)
}

AAD-bearer auth instead:

p, err := azureserverless.New(
    llmrouter.WithBaseURL("https://my-llama.eastus.models.ai.azure.com"),
    azureserverless.WithAADToken(func(ctx context.Context) (string, error) {
        tok, err := myAzureCredential.GetToken(ctx, policy.TokenRequestOptions{
            Scopes: []string{"https://cognitiveservices.azure.com/.default"},
        })
        if err != nil {
            return "", err
        }
        return tok.Token, nil
    }),
)

Full example — Llama 3 70B Instruct on Foundry

package main


import (
    "context"
    "fmt"
    "log"
    "os"


    "github.com/elloloop/llmrouter"
    "github.com/elloloop/llmrouter/providers/azureserverless"
)


func main() {
    p, err := azureserverless.New(
        llmrouter.WithAPIKey(os.Getenv("AZURE_FOUNDRY_KEY")),
        llmrouter.WithBaseURL("https://my-llama3-70b.eastus.models.ai.azure.com"),
    )
    if err != nil {
        log.Fatal(err)
    }


    stream, err := p.CompletionStream(context.Background(), llmrouter.ChatRequest{
        // Deployment-scoped URL — the Model field is echoed back unchanged.
        Model: "Meta-Llama-3-70B-Instruct",
        Messages: []llmrouter.Message{
            llmrouter.TextMessage("system", "You are a concise assistant."),
            llmrouter.TextMessage("user", "What is Azure AI Foundry Serverless API?"),
        },
        MaxTokens: 256,
    })
    if err != nil {
        log.Fatal(err)
    }


    for chunk := range stream.Chunks() {
        for _, c := range chunk.Choices {
            fmt.Print(c.Delta.Content)
        }
    }
    if err := stream.Err(); err != nil {
        log.Fatal(err)
    }
    fmt.Println()
}

Model catalogue and deployment-name patterns

Foundry's Serverless API catalogue is large and changes often. The table below shows the families currently served via this provider and the typical deployment-name pattern Foundry uses when you provision them through the portal. The exact name varies by Foundry region and by what you typed in the deployment wizard — treat these as guidelines, not contracts.

Family Typical model id (ChatRequest.Model) Deployment-name pattern
Llama (Meta) Meta-Llama-3-70B-Instruct, Meta-Llama-3.1-405B-Instruct my-llama3-70b, llama-31-405b-instruct
Mistral Mistral-large-2407, Mistral-Nemo mistral-large, mistral-nemo-12b
Cohere Cohere-command-r-plus, Cohere-command-r cohere-command-r-plus
Phi (Microsoft) Phi-3.5-MoE-instruct, Phi-3-medium-128k-instruct phi-35-moe
Jais jais-30b-chat jais-30b
Nemotron (NVIDIA) Nemotron-3-8B-Chat nemotron-3-8b
DeepSeek DeepSeek-V3, DeepSeek-R1 deepseek-v3, deepseek-r1

For deployment-scoped URLs the deployment name lives in the hostname and the Model field is informational. For hub-scoped URLs the Model field is the deployment selector.

Capabilities

Whatever the underlying model supports on its OpenAI-compatible chat-completions endpoint. Foundry deployments expose the standard chat-completions surface plus byte passthrough — pass ChatRequest.Raw to send provider-specific fields the typed surface doesn't model, and read Chunk.Raw on the way back. Tool use, multimodal content, and JSON mode work when the underlying model supports them.

Error handling

Non-2xx responses surface as *llmrouter.ErrUpstream with Provider == "azureserverless". Common cases:

  • 401 — wrong api-key or expired AAD token.
  • 403 — the deployment's content-safety filter rejected the input or output.
  • 404 — wrong hostname (typo in the deployment / region segment) or the deployment was deleted.
  • 404 with ?api-version=... in the URL — the Serverless API does not accept that query parameter (see the gotcha above).
  • 429 — Foundry capacity throttle; honour Retry-After.
  • 400 — body shape mismatch; the most common cause is sending OpenAI tool-call shapes to a model that doesn't support tools.

Caveats

  • No default base URL. The caller must pass a hostname — there is no canonical Foundry-wide endpoint, because every deployment gets its own subdomain (or hub).
  • Capability varies per model. Llama, Mistral, and Cohere on Foundry have different feature sets — only some support tool use, only some support JSON mode. Check the model's Foundry catalogue page before relying on a feature.
  • Content safety is enforced server-side. Microsoft applies an Azure-managed content filter to every Foundry deployment by default. Filter rejections come back as 403 with a body describing which category triggered (hate, self-harm, sexual, violence). Inspect ErrUpstream.Body.
  • Auth is mutually exclusive. Providing both an API key and an AAD source — or neither — fails fast in New().

When to use this

  • You're on Azure and want a single billing / compliance surface across OpenAI's GPT models, Anthropic's Claude, and an open-weights model (Llama, Mistral, Cohere, Phi).
  • You need data residency in a Microsoft-managed region — Foundry runs the model inside Azure's own tenant, not as a passthrough to a third party.
  • You want to consolidate multiple open-weights models onto one billing surface instead of separate accounts at Together / Groq / Fireworks.

If you only need one or two models and aren't on Azure, the direct vendor providers (Mistral, Cohere) or the OpenAI-compatible specialists (Together / Groq / Fireworks / DeepSeek) are usually cheaper and have fresher model availability.

See also