Last updated 2026-05-17

Cancellation & Timeouts

Streaming LLM calls have an unusual property: the response can take anywhere from 200ms to several minutes, and the caller may want to abort it at any point — because the user navigated away, because the budget ran out, or because a higher-level deadline expired. There are three different knobs in llmrouter for controlling when a call ends, and which you use matters.

The three knobs

Stack them from the outside in: 

┌──────────────────────────────────────────────────────────┐
│ http.Client.Timeout — total wall-clock for the exchange  │
│  (configured via WithTimeout, or via your custom client) │
│                                                          │
│  ┌────────────────────────────────────────────────────┐  │
│  │ context.WithTimeout — deadline for THIS call       │  │
│  │  (passed as ctx to CompletionStream)               │  │
│  │                                                    │  │
│  │  ┌──────────────────────────────────────────────┐  │  │
│  │  │ stream.Cancel() — abort mid-stream by hand   │  │  │
│  │  │  (called from the consumer goroutine)        │  │  │
│  │  └──────────────────────────────────────────────┘  │  │
│  └────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────┘

1. http.Client.Timeout — defense in depth

The Go HTTP client has its own total-time-for-the-exchange timeout. When you provide an explicit *http.Client via WithHTTPClient, its Timeout field governs the entire request including the streaming body. If your stream takes 30 seconds and Timeout = 20 * time.Second, you'll see the client kill the response mid-stream.

The library default (WithTimeout(120 * time.Second)) is a sensible upper bound for most short chats, but it will cut off long-form completions. For streaming you typically want http.Client{Timeout: 0} and rely on the per-call context deadline instead.

2. context.WithTimeout — per-call deadline

This is the workhorse. Build a context.Context with a deadline and pass it as the first argument to CompletionStream. The provider attaches it to the underlying HTTP request, and the same context is used internally to govern the SSE pump goroutine — so when the deadline expires, everything tears down cleanly. 

ctx, cancel := context.WithTimeout(parent, 60*time.Second)
defer cancel()


stream, err := p.CompletionStream(ctx, req)
if err != nil {
    return err
}
for chunk := range stream.Chunks() {
    // ...
}
// stream.Err() will be context.DeadlineExceeded if we ran out of time.

3. stream.Cancel() — abort by hand

Sometimes you don't know in advance how long you want to wait. The most common case: a token budget. You count tokens as they come in, and when you cross a threshold you call stream.Cancel() to stop the upstream. The library propagates that to the underlying HTTP request and the connection closes. 

for chunk := range stream.Chunks() {
    if budgetExceeded(chunk) {
        stream.Cancel()
        break
    }
    forwardToClient(chunk)
}

Cancel() is safe to call multiple times. It's also safe to call concurrently from a different goroutine — for example, one goroutine drains chunks, another watches a "stop" channel.

Why the layering matters

Each knob protects against a different failure mode. You typically want all three:

  • HTTP client timeout catches the case where the upstream stops sending data without closing the connection. It's the ceiling you trust as a last resort.
  • Per-call deadline catches "the user only wants to wait a minute." Without it, a long upstream completion ties up the caller until the HTTP timeout kicks in — too long for an interactive UX.
  • stream.Cancel() catches dynamic conditions you can only detect mid-stream: budgets, user navigation, abuse signals.

Recipe: handler with a budget and a disconnect

The canonical chat handler combines all three timeouts: a 60-second deadline from r.Context(), a 4096-token budget enforced via stream.Cancel(), and a zero-timeout HTTP client because this request might be the long one. 

package main


import (
    "context"
    "encoding/json"
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
    "time"


    "github.com/elloloop/llmrouter"
    "github.com/elloloop/llmrouter/providers/openai"
)


const (
    perRequestDeadline = 60 * time.Second
    maxOutputTokens    = 4096
)


type onlyModel struct {
    Model string `json:"model"`
}


type handler struct {
    provider llmrouter.Provider
}


func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    // Layer 1: per-call deadline. We chain off r.Context() so client
    // disconnect ALSO aborts the stream — see the closer goroutine below.
    ctx, cancel := context.WithTimeout(r.Context(), perRequestDeadline)
    defer cancel()


    body, err := io.ReadAll(io.LimitReader(r.Body, 1<<20))
    if err != nil {
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }
    var peek onlyModel
    if err := json.Unmarshal(body, &peek); err != nil {
        http.Error(w, "invalid json", http.StatusBadRequest)
        return
    }


    stream, err := h.provider.CompletionStream(ctx, llmrouter.ChatRequest{
        Model: peek.Model,
        Raw:   body,
    })
    if err != nil {
        http.Error(w, err.Error(), http.StatusBadGateway)
        return
    }


    // Layer 2: client disconnect. r.Context() is the request context.
    // It cancels when the client TCP disconnects. We already wired it
    // into the call above, so this watcher is belt-and-suspenders.
    go func() {
        <-r.Context().Done()
        stream.Cancel()
    }()


    w.Header().Set("Content-Type", "text/event-stream; charset=utf-8")
    flusher, _ := w.(http.Flusher)


    // Layer 3: token budget. Count completion tokens as they arrive;
    // abort the stream once we cross the budget.
    var completionTokens int
    for chunk := range stream.Chunks() {
        for _, c := range chunk.Choices {
            // Roughly: every chunk's delta content adds one or more
            // tokens. For real budgeting use a proper tokenizer; this
            // string-length heuristic is just illustrative.
            completionTokens += approxTokens(c.Delta.Content)
        }
        if completionTokens > maxOutputTokens {
            log.Printf("budget exceeded after %d tokens — cancelling", completionTokens)
            stream.Cancel()
            break
        }
        _, _ = w.Write([]byte("data: "))
        _, _ = w.Write(chunk.Raw)
        _, _ = w.Write([]byte("\n\n"))
        if flusher != nil {
            flusher.Flush()
        }
    }


    if err := stream.Err(); err != nil {
        if ctx.Err() != nil {
            log.Printf("call ended via context: %v", ctx.Err())
        } else {
            log.Printf("stream error: %v", err)
        }
    }
    _, _ = w.Write([]byte("data: [DONE]\n\n"))
}


// approxTokens is a rough placeholder. Replace with a real tokenizer
// (tiktoken-go, etc.) in production.
func approxTokens(s string) int {
    // Roughly four characters per token in English text.
    if s == "" {
        return 0
    }
    n := len(s) / 4
    if n == 0 {
        return 1
    }
    return n
}


func main() {
    p, err := openai.New(
        llmrouter.WithAPIKey(os.Getenv("OPENAI_API_KEY")),
        // Disable the HTTP client timeout — we rely on the per-call
        // context deadline instead.
        llmrouter.WithHTTPClient(&http.Client{Timeout: 0}),
    )
    if err != nil {
        log.Fatal(err)
    }


    srv := &http.Server{
        Addr:              ":8080",
        Handler:           &handler{provider: p},
        ReadHeaderTimeout: 10 * time.Second,
        // Critically: no WriteTimeout — it would kill long streams.
    }
    fmt.Println("listening on :8080")
    log.Fatal(srv.ListenAndServe())
}

Recipe: client disconnect from a CLI

The same context plumbing works in CLI tools. Catch SIGINT, cancel the context, and the upstream tears down within a few milliseconds. 

package main


import (
    "context"
    "fmt"
    "log"
    "os"
    "os/signal"
    "syscall"


    "github.com/elloloop/llmrouter"
    "github.com/elloloop/llmrouter/providers/openai"
)


func main() {
    p, err := openai.New(llmrouter.WithAPIKey(os.Getenv("OPENAI_API_KEY")))
    if err != nil {
        log.Fatal(err)
    }


    // Ctrl-C cancels the context, which cancels the stream, which
    // closes the upstream HTTP connection. No goroutines left behind.
    ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
    defer stop()


    stream, err := p.CompletionStream(ctx, llmrouter.ChatRequest{
        Model: "gpt-4o-mini",
        Messages: []llmrouter.Message{
            llmrouter.TextMessage("user", "tell me a long story"),
        },
    })
    if err != nil {
        log.Fatal(err)
    }
    for chunk := range stream.Chunks() {
        for _, c := range chunk.Choices {
            fmt.Print(c.Delta.Content)
        }
    }
    fmt.Println()
    if err := stream.Err(); err != nil {
        log.Printf("stream ended: %v", err)
    }
}

When the consumer is slow

The Stream has a small buffered channel (size 16). If your consumer is slower than the upstream produces chunks, the producer backpressures naturally — it blocks on send until you drain.

If you can't keep up at all (e.g. you're writing to a TCP socket that's stuck), the producer eventually blocks indefinitely. That's why you always pair a streaming call with a context deadline — without one, a slow client can pin the upstream connection forever.

What happens on cancellation

When you cancel (any of the three ways):

  1. The Go HTTP layer closes the underlying TCP connection (or the HTTP/2 stream).
  2. The SSE pump goroutine inside the provider notices the context is done, stops reading, closes the response body.
  3. The chunks channel is closed; your for chunk := range stream.Chunks() loop exits.
  4. stream.Err() returns context.Canceled or context.DeadlineExceeded.

The upstream provider may have already counted the tokens it generated up to that point against your account — Anthropic and OpenAI both bill for the prefix they produced, even if you cancel. That's a billing fact about the providers, not something the library can shield you from.

Common mistakes

  • Setting WithTimeout(30*time.Second) for streaming. You'll get mysterious mid-stream cutoffs at the 30-second mark for every long completion. Use http.Client{Timeout: 0} plus a context deadline.
  • Forgetting to pass r.Context() from an HTTP handler. Without it, client disconnects don't reach the upstream, and the provider keeps generating tokens (and billing you) until the response completes.
  • Setting an aggressive http.Server.WriteTimeout. The standard library's http.Server kills the response after WriteTimeout elapses; for streaming endpoints this manifests as the SSE connection closing at exactly N seconds every time. Leave it zero for streaming routes.
  • Not draining stream.Chunks() after cancellation. The channel always closes when the producer finishes, but if you break out of the loop and then return, the producer goroutine may still be holding the response body. The cleanest pattern: call stream.Cancel() and let the loop drain to completion (which happens within milliseconds).

Related