Monitoring

Prometheus metrics, health checks, and what to alert on

The orchestrator exposes Prometheus metrics on /metrics (unauthenticated, the stock promhttp.Handler()) and liveness/readiness probes on /health and /ready. See Health, readiness, and metrics for the full endpoint reference; this page focuses on what to actually watch.

Scraping

scrape_configs:
  - job_name: fuse-orchestrator
    static_configs:
      - targets: ["orchestrator:8080"]

No authentication is needed for the scrape itself, /metrics is deliberately mounted outside the auth middleware chain so Prometheus doesn’t need a token. If the orchestrator is behind a reverse proxy that does require auth for everything else, carve out an exception for /metrics (and /health, /ready) at the proxy layer.

For a one-off look without a Prometheus in front of you, fuse metrics scrapes the same endpoint and prints the raw exposition. It sends the context’s token even though /metrics doesn’t require one.

What to watch

Reconcile health is the single most important signal, the reconcile loop is what keeps the state store and the real provider fleet in sync:

• orchestrator_reconcile_cycles_total should be increasing steadily (every 30 seconds by default). A flat rate with the scrape target still up means the loop is blocked inside a provider List or a state-store call: the loop has no panic recovery, so a panic would take the whole process down and the target with it rather than leaving the counter flat. A provider list that merely fails still completes the cycle and increments the counter, so flat means blocked, not erroring.
• orchestrator_reconcile_duration_seconds climbing over time suggests the fleet has grown past what a single reconcile pass handles comfortably, or a provider call is slow.
• orchestrator_reconcile_orphans_dead_lettered_total increasing means VMs are being given up on after repeated cleanup failures, worth a look at orchestrator logs for the specific VM IDs.
• orchestrator_reconcile_stuck_tasks_failed_total increasing means a running VM exceeded its runtime ceiling on two consecutive reconcile cycles and was forcibly torn down, its task failed and dead-lettered. The ceiling is the environment’s max_runtime_seconds, falling back to the fleet-wide TaskStuckTimeout (2 hours) when it isn’t set. orchestrator_reconcile_stuck_tasks_suspected_total is the first-strike leading indicator, it fires one cycle earlier on the same condition.
• orchestrator_reconcile_idle_vms_failed_total increasing means a running VM went longer than its idle timeout with no exec and no attach session on two consecutive cycles and was torn down. The window is the environment’s idle_timeout_seconds, falling back to the fleet-wide DefaultIdleTimeout, which is unset by default: environments that don’t ask for idle expiry never get it. orchestrator_reconcile_idle_vms_suspected_total is the first-strike leading indicator.
• orchestrator_fleet_egress_endpoints{provider,state} is the live count of proxy egress endpoints. A non-zero state="unhealthy" series means those environments currently have no outbound access at all: proxy mode is enforced on the host’s tap, so a dead backend never falls back to direct egress. orchestrator_egress_provision_failures_total{provider} increasing means proxy-mode creates are failing at boot because the backend would not come up; orchestrator_egress_teardown_failures_total{provider} means a backend is being leaked on destroy and the host needs a look.

Fleet size drift: orchestrator_fleet_tracked_vms (what the state store thinks exists) versus orchestrator_fleet_provider_vms (what the provider actually reports) should track each other closely. A persistent gap means something is out of sync, check orchestrator_reconcile_vms_missing_provider_total for the specific failure mode.

Fleet shape is read live on every scrape, not at the reconcile tick, so it is current to the second and a host or state that goes away stops being reported rather than freezing at its last value:

• orchestrator_fleet_vms{state,host} counts VMs by state and host. A growing state="provisioning" or state="draining" series is work that isn’t finishing. host is empty on a single-provider fleet.
• orchestrator_host_capacity{host,resource} and orchestrator_host_allocated{host,resource} give each host’s capacity and what’s allocated from it, for resource in cpus, ram_mb, storage_gb, vms and gpus. orchestrator_host_allocated / orchestrator_host_capacity is per-host utilization: alert on it approaching 1 before creates start failing placement.
• orchestrator_host_info{host,state,region,backend,arch} is always 1, one series per host, for joining and for spotting a cordoned or draining host. orchestrator_host_last_seen_timestamp_seconds{host} is when its agent last answered. time() - orchestrator_host_last_seen_timestamp_seconds climbing means the orchestrator has lost touch with that host.
• orchestrator_fleet_snapshots{state,mode} counts snapshots. A non-zero state="error" series is snapshots that failed and are still recorded. This is the one fleet series read from the state store, so a scrape while Postgres is unreachable leaves it out rather than failing.

Lifecycle operations: orchestrator_operations_total{op,result} and orchestrator_operations_duration_seconds{op} cover create, destroy, fork, migrate, migrate_live, snapshot and restore. destroy counts explicit destroys, from the API or a migrate removing its source. Teardowns the reconcile loop does on its own (stuck, idle, orphaned) and the teardown after a task completes aren’t included; the reconcile counters above cover the first group. snapshot includes the snapshots fork, migrate and checkpoint chains take internally, because each is a real checkpoint the host had to write. Watch rate(orchestrator_operations_total{result="error"}[5m]) by op, and the p99 of the duration histogram per op: a cold migrate taking minutes is normal, a fork taking minutes is not.

HTTP health: the standard RED metrics apply directly.

• rate(orchestrator_http_requests_total{code=~"5.."}[5m]) for server error rate.
• histogram_quantile(0.99, rate(orchestrator_http_request_duration_seconds_bucket[5m])) for p99 latency, grouped by route.
• orchestrator_http_requests_in_flight as a saturation signal.

orchestrator_http_requests_total is labeled route/method/code and orchestrator_http_request_duration_seconds is labeled route/method, both using the chi route pattern (/v1/environments/{vmId}) rather than the raw path, so cardinality stays bounded even under a flood of requests to nonexistent IDs. orchestrator_http_requests_in_flight is a single unlabeled gauge, so saturation cannot be broken down by route.

Suggested alerts

fuse-proxy

fuse-proxy serves its own Prometheus metrics on the admin listener at /metrics (127.0.0.1:7080 by default, set by FUSE_PROXY_ADMIN_LISTEN). Like /healthz it needs no admin token, so the scraper never holds the credential that moves routes. Every other path on that listener still does, so keep it off the public internet and scrape it from the same private network the orchestrator uses.

scrape_configs:
  - job_name: fuse-proxy
    static_configs:
      - targets: ["fuse-proxy:7080"]

Series labeled owner carry the environment’s vm id. They are deleted when the orchestrator unpublishes that vm, so the label set tracks the live fleet.

Did a live migrate keep its connections? A live migrate resumes the guest on the destination host with its memory intact, so the tunnel connection it already held, under the source vm’s id, starts sending from a new address. Look at the source vm’s id:

• fuse_proxy_tunnel_path_migrations_total{owner="<source vm>"} went up and fuse_proxy_tunnel_guest_connections_total{owner="<source vm>"} did not: the connection followed the guest, and client connections open across the pause kept running.
• guest_connections_total went up and path_migrations_total did not: the guest dialed in again, and every client connection open during the move was cut.

Separately, guest_connections_total{owner="<destination vm>"} goes to 1 once the destination’s sidecar connects under its own identity and adopts the url. That sidecar restart currently ends the source identity’s connection, along with any client connections still on it (#287).

Other things worth watching:

• fuse_proxy_port_exhaustion_total going up means creates are failing to get a url. Widen FUSE_PROXY_PORTS before it starts.
• fuse_proxy_tunnel_handshakes_total{result="duplicate"} going up steadily means a fork or copy is booting with its source’s tunnel identity and retrying against a live guest.
• rate(fuse_proxy_connections_total{result="failed"}[5m]) is clients reaching a url whose guest isn’t connected.

What isn’t covered

There is no per-VM metrics API: VMs are counted by state and host, never broken out one series per VM. For a browsable view of everything above, plus the hosts, environments and snapshots behind it, run fuse dashboard.