Last updated 2026-06-29

Health & Metrics

This page covers everything you need to keep workspaces healthy in production: the liveness and readiness probes, the Prometheus metrics endpoint, the structured logs, and what to alert on.

Health probes

The service exposes two HTTP health endpoints. Both are unauthenticated — they bypass the service-auth check so an orchestrator can poll them without a token — and both return 200 OK when the process is serving.

Probe	Endpoint	Use
Liveness	`GET /healthz`	Is the process up? A failing probe means restart the container.
Readiness	`GET /readyz`	Is the process ready to take traffic? Gate routing on this.

curl -i http://localhost:8080/healthz   # liveness
curl -i http://localhost:8080/readyz    # readiness

The probes are served on both the Connect port (8080) and the metrics port (9090), so a Kubernetes livenessProbe and readinessProbe can target either.

Prometheus metrics

Metrics are exposed in Prometheus format at /metrics on the metrics port, 9090 by default (GATEWAY_METRICS_PORT). This port is separate from the RPC port so you can keep it on a private interface, away from application traffic.

curl http://localhost:9090/metrics

A scrape config that targets the metrics port:

scrape_configs:
  - job_name: workspaces
    metrics_path: /metrics
    static_configs:
      - targets: ["workspaces:9090"]

Exported metrics

The authorization decision metrics the service exports. Labels are kept to low-cardinality dimensions (namespace, relation, outcome, RPC, reason) — never object or subject identifiers. This list is generated from the service's metrics source, so it stays in lockstep with the code.

Metric	Meaning
`authz_batchcheck_items`	Items per BatchCheck request.
`authz_check_decisions_total`	Authorization Check/CheckSet/BatchCheck decisions by namespace, relation, and outcome.
`authz_check_duration_seconds`	Authorization decision RPC latency in seconds.
`authz_decision_errors_total`	Authorization decision RPC errors (validation + internal).
`authz_eval_backstop_total`	Engine per-request safety backstops that fired, by reason: depth/cycle (graceful fail-closed deny) or budget (read-budget exhausted, ResourceExhausted error). A rising rate signals an abusive tenant or a misconfigured deep/cyclic model.
`authz_region_refused_total`	Requests refused because the project's pinned data region differs from this instance's region (data-residency fail-closed).

Audit events

With GATEWAY_AUDIT_LOG=true, every relation-tuple change and admin mutation is emitted to the structured logger as an append-only audit record. The event-type vocabulary — the operation strings carried by those records — is generated from the service's audit source:

Event	Kind	Meaning
`create_project`	AdminAction	AdminActionCreateProject records a CreateProject mutation.
`delete`	TupleOpKind	TupleOpDelete is a revocation (a tuple was removed).
`insert`	TupleOpKind	TupleOpInsert is a grant (a tuple was written).
`update_project`	AdminAction	AdminActionUpdateProject records an UpdateProject mutation.

Structured logs

Workspaces logs with zap as structured JSON — each line is a discrete event with typed fields, ready to index in Loki, Elasticsearch, or any log pipeline. One startup line to watch for:

{"level":"warn","msg":"service_auth_disabled","reason":"no GATEWAY_SERVICE_AUTH_TOKENS configured","impact":"all callers trusted — deploy behind a private network/mesh or set the tokens"}

The service_auth_disabled warning is logged once at startup when GATEWAY_SERVICE_AUTH_TOKENS is empty: the bearer-token requirement is off and every caller is trusted. That is intentional behind a service mesh or mTLS, but in any other deployment it means the service is open — treat the warning as a misconfiguration signal and set the tokens (see Configuration).

What to alert on

Readiness failing — /readyz returning non-200, or the target down in Prometheus. Page on it; the service is not taking traffic.
Liveness flapping — repeated /healthz failures and restarts indicate a crash loop rather than a transient blip.
Scrape target down — Prometheus losing the workspaces target means you are flying blind; alert independently of the app's own metrics.
service_auth_disabled in production — alert on this log event anywhere that is not an mTLS/mesh deployment; it means the API is unauthenticated.
Sustained RPC errors — a rising rate of unauthenticated (bad or rotated service tokens) or permission_denied responses points at a broken caller or a misconfigured credential.

For the full list of error codes and their meaning, see Using the API.