Last updated 2026-05-17

Google Vertex AI Provider

The vertex provider talks to Google's Vertex AI platform — enterprise-grade hosting for Gemini, Anthropic Claude (via Vertex Marketplace), and Google's embedding models. Compared to the Gemini (AI Studio) provider, Vertex offers project-scoped quota, VPC service controls, and IAM-based access.

Capabilities: chat (Provider), embeddings (Embedder), and partial TTS via the Gemini audio path (Speaker). STT is not yet wrapped — use the standalone Cloud Speech-to-Text SDK or Deepgram.

Import path

import (
    "github.com/elloloop/llmrouter"
    "github.com/elloloop/llmrouter/providers/vertex"
)

Authentication

Vertex uses Application Default Credentials (ADC). No API key option; instead, the SDK resolves credentials via the standard Google chain (gcloud auth, service-account JSON, GCE/GKE/Cloud Run metadata server). 

p, err := vertex.New(
    vertex.WithProject(os.Getenv("GOOGLE_PROJECT_ID")),
    vertex.WithRegion("us-central1"),
)

Endpoint

The provider constructs the per-region endpoint: https://{region}-aiplatform.googleapis.com/v1/projects/{project}/locations/{region}/publishers/{publisher}/models/{model}:streamGenerateContent. Publisher is derived from the model id (gemini-* → google, claude-* → anthropic).

Chat example (Gemini)

stream, err := p.CompletionStream(ctx, llmrouter.ChatRequest{
    Model: "gemini-2.0-flash",
    Messages: []llmrouter.Message{
        llmrouter.TextMessage("system", "You are a helpful assistant."),
        llmrouter.TextMessage("user", "What's new in Vertex AI?"),
    },
})


for chunk := range stream.Chunks() {
    for _, c := range chunk.Choices {
        fmt.Print(c.Delta.Content)
    }
}

Claude on Vertex

Claude models on Vertex use the claude-*@* versioning convention: 

stream, err := p.CompletionStream(ctx, llmrouter.ChatRequest{
    Model:     "claude-3-5-sonnet-v2@20241022",
    MaxTokens: 1024,
    Messages: []llmrouter.Message{
        llmrouter.TextMessage("user", "Hello from Vertex."),
    },
})

Embeddings

Vertex's text embedding models live under the publishers/google/models/... path. The library routes via the same EmbedRequest.Model:

  • text-embedding-005 — current default, 768 dims.
  • text-embedding-004 — previous generation.
  • text-multilingual-embedding-002 — 250+ languages.
resp, err := p.Embed(ctx, llmrouter.EmbedRequest{
    Model:    "text-embedding-005",
    Inputs:   []string{"What is Vertex AI?"},
    TaskType: "RETRIEVAL_QUERY",
})

Audio (partial)

Gemini 2.0 Flash supports speech generation as part of its multimodal output. The library wraps this behind Speak for parity with other TTS providers, but support is partial: voice selection is limited compared to the dedicated TTS providers, and streaming chunks arrive at sentence granularity rather than word/phoneme. 

stream, err := p.Speak(ctx, llmrouter.SpeechRequest{
    Model:  "gemini-2.0-flash",
    Input:  "Hello from Vertex.",
    Voice:  "Aoede",
    Format: "wav",
})

For production TTS, prefer Cartesia or ElevenLabs.

Error handling

Non-2xx responses surface as *llmrouter.ErrUpstream with Provider == "vertex". Common cases:

  • 403 — service account missing roles/aiplatform.user.
  • 404 — model not available in the region; try us-central1.
  • 429 — per-project quota exhausted.
  • 400 — safety filter; inspect the response body.

Caveats

  • Pin the region. Model availability and pricing vary by region. Vertex does not auto-fall-back.
  • Safety filters. Vertex applies its own safety filters on top of model-level filters; tune via safetySettings on Raw.
  • VPC-SC compatible. Honours VPC Service Controls because all traffic uses Google's standard SDK transport.

See also