Last updated 2026-05-17

Rerank

Reranking is the “take a list of candidate documents and a query, and reorder them by how well each candidate actually answers the query” capability. It is the standard second-stage filter in a RAG pipeline: vector search is fast and recalls a wide set of plausibly-relevant chunks, and a rerank model then narrows that wide set down to the handful you actually feed into the LLM context window.

llmrouter exposes this via a single root interface, Reranker, implemented today by Cohere, Voyage, and Together. Application code never branches on which rerank vendor is upstream.

Why rerank is its own thing

Vector search is cheap and shallow. It compares embeddings — one fixed-dimension vector per document, one for the query — and returns whatever's nearest in the embedding space. That's a decent recall signal, but it does not capture term-level relevance the way a cross-encoder does. A cross-encoder reads the query and the candidate together and produces a single relevance score that's noticeably better than embedding similarity for the top-of-list ordering.

The standard pipeline is:

1. Vector search — recall the top ~100 candidates from your vector store. Cheap, fast, approximate.
2. Rerank — score those 100 with a cross-encoder and keep the top 5-10. One extra API call, much sharper ordering.
3. Generate — feed the top 5-10 into the LLM as context.

Skipping the rerank stage is the single most common reason RAG pipelines return technically-relevant-but-wrong answers.

The API

type Reranker interface {
    Name() string
    Rerank(ctx context.Context, req RerankRequest) (*RerankResponse, error)
}

type RerankRequest struct {
    Model     string   // vendor model id, e.g. "rerank-v3.5" (Cohere)
    Query     string   // the user's question
    Documents []string // candidates to rerank
    TopN      int      // cap on returned results; 0 means "all"
    Raw       json.RawMessage // byte passthrough for vendor-specific fields
}

type RerankResponse struct {
    Results []RerankResult
    Usage   *Usage          // tokens billed (when reported)
    Raw     json.RawMessage // original wire JSON
}

type RerankResult struct {
    Index    int     // position in the original Documents slice
    Score    float64 // 0.0-1.0; higher = more relevant
    Document string  // copy of the document text, for convenience
}

The shape is intentionally minimal — query, documents, optional cap, results sorted by descending score with the original index preserved so callers can join back to metadata they kept alongside the documents.

Provider models and endpoints

Example: rerank vector-search candidates

package main

import (
    "context"
    "fmt"
    "log"
    "os"

    "github.com/elloloop/llmrouter"
    "github.com/elloloop/llmrouter/providers/cohere"
)

// Pseudocode: the documents and their metadata came out of your
// vector store. Keep the metadata aligned with the documents slice
// by index so you can join back after rerank.
type Chunk struct {
    ID   string
    Text string
}

func vectorSearch(ctx context.Context, query string) ([]Chunk, error) {
    // ... whatever your store is (Qdrant, pgvector, ...) returns
    // its top 100 by embedding similarity ...
    return nil, nil
}

func main() {
    ctx := context.Background()

    query := "How do I cancel a streaming chat completion?"
    candidates, err := vectorSearch(ctx, query)
    if err != nil {
        log.Fatal(err)
    }

    // Cohere implements Reranker on its existing Provider type.
    p, err := cohere.New(llmrouter.WithAPIKey(os.Getenv("COHERE_API_KEY")))
    if err != nil {
        log.Fatal(err)
    }

    docs := make([]string, len(candidates))
    for i, c := range candidates {
        docs[i] = c.Text
    }

    resp, err := p.Rerank(ctx, llmrouter.RerankRequest{
        Model:     "rerank-v3.5",
        Query:     query,
        Documents: docs,
        TopN:      5,
    })
    if err != nil {
        log.Fatal(err)
    }

    // Results come back sorted by descending Score, with Index
    // pointing back into the original Documents slice.
    for _, r := range resp.Results {
        chunk := candidates[r.Index]
        fmt.Printf("score=%.3f id=%s text=%q
", r.Score, chunk.ID, chunk.Text)
    }
}

Swap the provider to Voyage or Together by changing the import, the constructor, and the model id. The Rerank call is identical:

// Voyage
import "github.com/elloloop/llmrouter/providers/voyage"
v, _ := voyage.New(llmrouter.WithAPIKey(os.Getenv("VOYAGE_API_KEY")))
resp, _ := v.Rerank(ctx, llmrouter.RerankRequest{
    Model: "rerank-2", Query: query, Documents: docs, TopN: 5,
})

// Together
import "github.com/elloloop/llmrouter/providers/openai"
t, _ := openai.New(
    llmrouter.WithAPIKey(os.Getenv("TOGETHER_API_KEY")),
    llmrouter.WithBaseURL("https://api.together.xyz/v1"),
)
resp, _ := t.Rerank(ctx, llmrouter.RerankRequest{
    Model: "Salesforce/Llama-Rank-V1", Query: query, Documents: docs, TopN: 5,
})

Picking a value for TopN

TopN is a soft cap. The vendor may return fewer results if its internal score floor isn't met by enough candidates — this is a feature, not a bug. It means the pipeline got a clean signal that the long tail isn't worth feeding to the LLM.

Rules of thumb:

• 5-10 for chat-style RAG where the LLM synthesises an answer from a few sources.
• 15-25 for “cite all relevant passages” use cases where recall matters more than precision.
• 1-3 when the downstream step is itself an LLM call with a tiny context budget (e.g. an on-device model).

Caveats

• One extra network round-trip. Rerank sits on the hot path between vector search and generation. Plan for 150-400 ms of added latency depending on candidate count and document length.
• Document length is billed. Cohere and Voyage both charge per “document token” pair. Sending the full document text when only the first 1-2 KB matters wastes money. Truncate aggressively before sending.
• Multilingual quality varies. Cohere's multilingual models cover more languages well than Voyage's; Together's open-weights model is mostly English-only. Test on your actual locale before committing.
• The score is a black box. A score of 0.8 from Cohere is not directly comparable to 0.8 from Voyage. Pick a provider and calibrate against your own evals; don't try to reuse score thresholds across vendors.
• No streaming. Rerank is a synchronous request-response call; there is no progressive result. Latency is end-to-end.

When to use rerank

Reach for rerank when:

• You have a RAG pipeline whose top-of-list ordering matters (i.e. always).
• Your vector store recall is set wide (top-50 to top-200) and you need to narrow before feeding the LLM.
• Eval shows that the right answer is in the candidate set but not always in the top-K by embedding similarity.

Skip it when:

• You're only retrieving 1-3 candidates total — there's nothing to reorder.
• Latency budget is critical and your evals show vector recall alone hits the precision floor you need.
• You're searching over very short candidates (tweets, log lines) where cross-encoder gains are smaller.

See also

• Embeddings concept — the first stage of the RAG pipeline.
• API reference: Rerank — full type definitions.
• Cohere provider — the canonical rerank vendor.
• Voyage AI provider — pairs naturally with Anthropic for embed + rerank.