The reconcile loop

The background sweep that detects leaked VMs and runaway tasks

Every 30 seconds the orchestrator compares what it thinks is running against what the providers say is running, and acts on the difference. This is the reconcile loop. It is worth understanding early, because it is the only thing that destroys resources you did not explicitly ask it to.

The most important thing to know up front: it is a leak detector, not a supervisor. It never restarts, reschedules, or heals anything. Every action it takes is destructive, garbage collection, or read-only reporting. If you come from Kubernetes, this is the sharpest difference, there is no desired-state controller driving things back to health, and a failed task stays failed.

One cycle

A cycle runs on a fixed 30-second tick, with no jitter and no catch-up. One converge also runs immediately at startup, so the first sweep happens before the first tick, not 30 seconds in.

Each cycle, in order:

List every VM the providers actually have.
Mark missing VMs. A tracked VM the provider no longer reports is moved to destroying and its task is failed.
Destroy orphans, VMs the provider has that Fuse does not track.
Fail stuck tasks that have exceeded their runtime ceiling.
Read health verdicts back from the guests of environments that declared a healthcheck.
Garbage-collect snapshots past the retention window.
Reap destroyed VMs and return their capacity to the host.

Orphans

An orphan is a VM present in a provider but absent from Fuse’s own records. The definition is pure set difference, with no grace period, so anything the orchestrator has forgotten about is destroyed on the next tick.

Destruction is retried up to five times. After that the orphan is recorded in the dead-letter queue and skipped permanently, so a VM that cannot be destroyed stops consuming the loop rather than blocking it forever. Retry counters live in memory, so restarting the orchestrator gives every dead-lettered orphan a fresh budget.

Stuck tasks

A task is a candidate for failure when its VM is running, it still has a task attached, and it has exceeded its ceiling. The ceiling is the environment’s own max_runtime when set, and otherwise the fleet default of two hours.

Two properties matter more than the mechanism:

The ceiling measures total VM lifetime, from creation, not time since the last state change. A long-running job is not spared by making progress. Any workload legitimately outliving the default must set max_runtime explicitly, or it will be destroyed mid-flight at the two-hour mark.

Detection is not a health check. Nothing pings the guest and nothing reads a heartbeat. The loop knows only two things: how old a VM is, and whether the provider still lists it. The consequences run both ways, a wedged task that is young is completely invisible to this loop, and a healthy task that is old is a false positive unless it raised its own ceiling.

A VM must exceed its ceiling on two consecutive cycles before it is failed, which guards against a single cycle skewed by a clock jump or a scheduling pause. The practical effect is that the real time-to-failure is the ceiling plus one interval, and a task.stuck_suspected event fires on the first strike, giving you roughly 30 seconds of warning in the event stream.

Idle environments

Idle detection is the liveness check stuck-task detection is not. An environment is a candidate when its VM is running, no attach session is open, and it has gone longer than its idle timeout with no exec and no attach. The timeout is the environment’s own idle_timeout when set, and otherwise the fleet default, which is unset: an environment that does not ask for idle expiry never gets it.

The clock measures time since the last activity, not total lifetime, and activity means exactly two things: an exec call, and an open attach session. An attach session counts for as long as it stays open, however quiet, and the window restarts when it closes. In-guest CPU use and traffic on exposed ports are not observed, because the loop has no channel for either. A VM busy computing with nobody attached is idle as far as this predicate is concerned.

Detection is two-strike like stuck-task detection, with a vm.idle_suspected event on the first strike, so resolution is bounded by roughly two intervals. There is no pause or suspend in the stack, so the second strike destroys the environment down the same path a stuck task takes: the VM goes to destroying, its task (if any) is failed, and an idle_timeout dead letter is recorded.

The activity clock lives in memory only. Restarting the orchestrator restarts every environment’s idle window rather than inheriting a stale one.

Health verdicts

This is the one pass that is not a detector. It reads and reports; it never destroys anything.

An environment created with a healthcheck runs that probe inside its guest, and the guest agent leaves the current verdict in a file. Each cycle the loop reads that file back over the same exec channel the host agents already expose, and records the verdict on the VM, where it surfaces as the health field on the environment. A change of verdict emits a vm.health_changed event.

Only environments that declared a probe are read, so a fleet that declares none pays nothing for this pass. The reads are issued concurrently and each is bounded at 5 seconds, so one sick guest does not hold up the tick.

A read that fails is not a failing probe. An unreachable guest could equally be a host agent hiccup, and this pass has none of the two-strike structure the destructive detectors have, so the last known verdict is left in place and the next cycle tries again. The same is true of a guest that has not written a verdict yet.

Nothing acts on failing. The environment stays running, its task stays assigned, and no dead letter is recorded. The only destructive predicates in this loop remain the age ceiling and the idle window. Verdicts live in memory only, so restarting the orchestrator drops them until each guest is read again; a VM recovered across a restart is not polled at all, because the probe it was created with is not persisted.

Dead letters

Orphans that exhaust their retries, tasks failed for being stuck, and environments torn down for being idle are all recorded in a dead-letter queue, keyed by kind and entity so repeated failures update one row rather than piling up duplicates.

The queue is an inspection surface only. Nothing drains it, nothing retries from it, and writes to it are best-effort, a failure to record a dead letter is logged and swallowed rather than propagated. Treat it as a place to find out what went wrong, not as a work queue.

What the loop does not do

It does not detect a dead host agent. last_seen is never refreshed and nothing sweeps for unreachable hosts. A dead agent surfaces only as the aborted cycle described above.
It does not drain hosts. The draining host state exists but no eviction logic backs it.
It does not re-probe capacity. Host capacity is fixed at registration.
It does not rebalance. Once placed, an environment stays on its host for life.

Metrics

Every cycle emits its results, including on the aborted path, so a stuck loop is visible rather than silent:

Metric	Measures
`orchestrator_reconcile_cycles_total`	Cycles completed.
`orchestrator_reconcile_duration_seconds`	Cycle duration.
`orchestrator_fleet_tracked_vms`	VMs Fuse believes exist.
`orchestrator_fleet_provider_vms`	VMs the providers report.
`orchestrator_reconcile_orphans_destroyed_total`	Orphans destroyed.
`orchestrator_reconcile_orphans_failed_total`	Orphan destroy attempts that failed.
`orchestrator_reconcile_orphans_dead_lettered_total`	Orphans that exhausted retries.
`orchestrator_reconcile_stuck_tasks_suspected_total`	First-strike suspicions.
`orchestrator_reconcile_stuck_tasks_failed_total`	Tasks actually failed.
`orchestrator_reconcile_vms_missing_provider_total`	Tracked VMs the provider lost.
`orchestrator_reconcile_health_checked_vms`	Environments that answered their healthcheck this cycle.
`orchestrator_reconcile_health_failing_vms`	Environments whose healthcheck reported `failing` this cycle.

A persistent gap between fleet_tracked_vms and fleet_provider_vms is the signal worth alerting on, it means the two views of the world are diverging. Every counter is fleet-wide, there is no per-host breakdown.