Monitoring

Prometheus metrics, health checks, and what to alert on

The orchestrator exposes Prometheus metrics on /metrics (unauthenticated, the stock promhttp.Handler()) and liveness/readiness probes on /health and /ready. See Health, readiness, and metrics for the full endpoint reference; this page focuses on what to actually watch.

Scraping

scrape_configs:
  - job_name: fuse-orchestrator
    static_configs:
      - targets: ["orchestrator:8080"]

No authentication is needed for the scrape itself, /metrics is deliberately mounted outside the auth middleware chain so Prometheus doesn’t need a token. If the orchestrator is behind a reverse proxy that does require auth for everything else, carve out an exception for /metrics (and /health, /ready) at the proxy layer.

For a one-off look without a Prometheus in front of you, fuse metrics scrapes the same endpoint and prints the raw exposition. It sends the context’s token even though /metrics doesn’t require one.

What to watch

Reconcile health is the single most important signal, the reconcile loop is what keeps the state store and the real provider fleet in sync:

• orchestrator_reconcile_cycles_total should be increasing steadily (every 30 seconds by default). A flat rate with the scrape target still up means the loop is blocked inside a provider List or a state-store call: the loop has no panic recovery, so a panic would take the whole process down and the target with it rather than leaving the counter flat. A provider list that merely fails still completes the cycle and increments the counter, so flat means blocked, not erroring.
• orchestrator_reconcile_duration_seconds climbing over time suggests the fleet has grown past what a single reconcile pass handles comfortably, or a provider call is slow.
• orchestrator_reconcile_orphans_dead_lettered_total increasing means VMs are being given up on after repeated cleanup failures, worth a look at orchestrator logs for the specific VM IDs.
• orchestrator_reconcile_stuck_tasks_failed_total increasing means a running VM exceeded its runtime ceiling on two consecutive reconcile cycles and was forcibly torn down, its task failed and dead-lettered. The ceiling is the environment’s max_runtime_seconds, falling back to the fleet-wide TaskStuckTimeout (2 hours) when it isn’t set. orchestrator_reconcile_stuck_tasks_suspected_total is the first-strike leading indicator, it fires one cycle earlier on the same condition.
• orchestrator_reconcile_idle_vms_failed_total increasing means a running VM went longer than its idle timeout with no exec and no attach session on two consecutive cycles and was torn down. The window is the environment’s idle_timeout_seconds, falling back to the fleet-wide DefaultIdleTimeout, which is unset by default: environments that don’t ask for idle expiry never get it. orchestrator_reconcile_idle_vms_suspected_total is the first-strike leading indicator.

Fleet size drift: orchestrator_fleet_tracked_vms (what the state store thinks exists) versus orchestrator_fleet_provider_vms (what the provider actually reports) should track each other closely. A persistent gap means something is out of sync, check orchestrator_reconcile_vms_missing_provider_total for the specific failure mode.

HTTP health: the standard RED metrics apply directly.

• rate(orchestrator_http_requests_total{code=~"5.."}[5m]) for server error rate.
• histogram_quantile(0.99, rate(orchestrator_http_request_duration_seconds_bucket[5m])) for p99 latency, grouped by route.
• orchestrator_http_requests_in_flight as a saturation signal.

orchestrator_http_requests_total is labeled route/method/code and orchestrator_http_request_duration_seconds is labeled route/method, both using the chi route pattern (/v1/environments/{vmId}) rather than the raw path, so cardinality stays bounded even under a flood of requests to nonexistent IDs. orchestrator_http_requests_in_flight is a single unlabeled gauge, so saturation cannot be broken down by route.

Suggested alerts

What isn’t covered

There is no per-VM metrics API, orchestrator_fleet_tracked_vms is a fleet total, not broken out by task or host. For per-host capacity, poll GET /v1/hosts/{hostId} or use fuse host metrics instead, that data isn’t in the Prometheus registry.