Skip to content
Fuse
Esc
↑↓navigate↵open⌘Jpreview
On this page

Firecracker host setup

Bringing up a bare-metal or nested-virt host to run Firecracker microVMs

Requirements

Firecracker needs hardware virtualization (KVM). The host must expose /dev/kvm:

  • Bare-metal Linux, or a cloud instance type with nested virtualization enabled (e.g. GCP *-metal/nested-virt images, AWS *.metal, bare-metal providers like Equinix or Hetzner dedicated). It will not run inside an ordinary container or a VM without nested virt.
  • Confirm before you start: 
    ls -l /dev/kvm
[ -r /dev/kvm ] && [ -w /dev/kvm ] && echo ok
    If /dev/kvm is missing, the host can’t run Firecracker.
  • Linux only, x86_64 today (the baked rootfs pulls amd64 podman/iptables). The agent shells out to ip, iptables, firecracker, and SSH.
  • Needs sudo (TAP devices, iptables, mounting the rootfs to bake), plus python3 (the agent itself), podman (the bake extracts the iptables bundle with it), curl, tar, ssh, iptables, and e2fsck/resize2fs/truncate from e2fsprogs and util-linux.
  • Rather than installing that list by hand, run ./fc-deps.sh first, it installs the whole set on apt or dnf and then verifies every tool the scripts shell out to.

One-command bootstrap

On a host that meets the requirements above, bootstrap does everything: host dependencies, Firecracker, the agent service, a local Postgres, the orchestrator, the weekly auto-update timer, the guest agent and rootfs bake, and it self-registers the host. It then prints the token and the exact fuse connect line to run from your laptop.

git clone <this repo> ~/fc && cd ~/fc/host-agent/firecracker
sudo ./fc-agent.sh bootstrap

It’s idempotent, safe to re-run. Flags: --no-updater (skip the auto-update timer), --no-register (don’t self-register). Everything below is the manual, step-by-step equivalent, use it when you want to run or skip individual stages.

Manual, step-by-step setup

git clone <this repo> ~/fc && cd ~/fc/host-agent/firecracker

# 0. Install host dependencies (podman, python3, e2fsprogs, ...) and verify them.
./fc-deps.sh

# 1. Fetch firecracker binary + CI kernel + base rootfs + SSH key.
./fc-install.sh

# 2. Build the reference in-guest agent (produces host-agent/firecracker/fused). Needs Go.
#    To run your own agent instead, drop your binary here as `fused` and skip this.
../shared/fc-build-agent.sh

# 3. Bake the guest rootfs (rootfs-fused.ext4). Bakes in `fused` + fused.service.
#    Re-run whenever the agent binary changes -- the agent is baked into the image.
./fc-bake-rootfs.sh

# 4. Start the agent. Prints FIRECRACKER_BASE_URL + FIRECRACKER_TOKEN.
./fc-agent.sh start

# 5. Smoke-test the contract end to end.
./fc-agent-test.sh

Point Fuse at the printed values:

FIRECRACKER_BASE_URL=http://<host>:8090
FIRECRACKER_TOKEN=<generated>

See Deploying the orchestrator to wire these into an orchestrator instance.

What gets baked into the rootfs

See Artifacts for the conceptual explanation. Concretely, fc-bake-rootfs.sh builds rootfs-fused.ext4 on top of the Firecracker CI Ubuntu 22.04 rootfs, injecting /usr/local/bin/fused, podman-static and its runtime dependencies, iptables (symlinked to xtables-legacy-multi, the CI kernel has no nftables), the host’s CA bundle, and a fused.service systemd unit. Re-run it whenever the agent binary changes.

Known limitations of this rootfs: apt is unusable (the CI rootfs ships an empty /var/lib/dpkg/status, customize via the mounted ext4 from the host instead). podman run without --network=host falls back to host netns anyway (bridged per-container networking isn’t supported). The kernel lacks fuse and nftables support.

The host agent’s HTTP contract

VM routes are under /v1/vm, bearer auth (Authorization: Bearer $TOKEN), JSON in/out. Auth applies to every route including the health and capacity probes, there is no unauthenticated path on this agent:

Method Path Purpose
POST /v1/vm Create a microVM. Body: {name,cpus,memory_mb,storage_gb,region,image}. Returns {vm_id,url}.
GET /v1/vm/{id} {vm_id,url}
GET /v1/vm?prefix= {vms:[{vm_id,url}]}, prefix match on name.
DELETE /v1/vm/{id} Tear down, free TAP + DNAT.
POST /v1/vm/{id}/upload {path, content_b64}, writes into the guest.
POST /v1/vm/{id}/exec {cmd:[...], timeout_ms}, returns {exit_code, stdout, stderr} (base64).
POST /v1/vm/{id}/start-agent Preferred. Optionally fetches the agent binary via download_url, writes a systemd drop-in, starts it.
POST /v1/vm/{id}/start-surfd Frozen legacy wire, same as start-agent with fused defaults. Fuse falls back to this on a 404 from start-agent.
POST /v1/vm/{id}/snapshot {comment, live}. live defaults to false (rootfs copy only); true pauses the VM and writes guest memory and vCPU state alongside it. Returns {snapshot_id, digest, kind, size_bytes}, where digest covers the rootfs only.
GET /v1/vm/{id}/snapshots {snapshots:[...]}
POST /v1/vm/{id}/restore {snapshot_id}. Stops fc and swaps the rootfs, then branches on the snapshot’s recorded kind: disk reboots the VM, live loads the memory image into a fresh fc process, resumes, and pushes the host clock into the guest. There is no flag, the recorded kind decides.
POST /v1/vm/{id}/fork Seeds a brand-new VM from this VM’s snapshot. Returns the new VM’s {vm_id,url}.
GET /v1/vm/{id}/attach Hijacks the connection for an interactive pty session, fuse-attach/1 framing.
GET /v1/capacity Host CPU count, total RAM, and free disk. Probed once at host registration, not polled.
GET /healthz {ok:true, app_name:"fc-agent"}. GET / is the same handler.

image on create is optional: a base-image name resolved to $IMAGES_DIR/<image>.ext4, defaulting to the baked rootfs-fused.ext4 when unset. A name that resolves outside IMAGES_DIR and a name with no matching file both return 400, so bake and place the rootfs before referencing it.

timeout_ms on exec is clamped to a 600-second ceiling, and the ceiling applies when the field is omitted or zero.

A bring-your-own host agent has to serve fork and attach too. fuse environment fork is implemented on top of the former, and the API’s attach endpoint on top of the latter, rather than on any orchestrator-side fallback. Attach has no CLI command, it is reachable only over the HTTP API.

url is <public_host>:<host_port>, DNAT’d to the guest’s 9550. Host port is 19550 + vm_index. Public host is auto-detected; override with PUBLIC_HOST=... in the agent’s environment.

Networking model

  • One TAP per VM (fcv<N>), a /30 subnet 10.200.<N>.0/30, host .1, guest .2.
  • Host iptables: MASQUERADE for egress, FORWARD accept for the TAP, and a per-VM PREROUTING DNAT rule <public_host>:<port> -> <guest>:9550.
  • Open at your cloud / external firewall: 8090/tcp (the agent’s HTTP API) and 19551–19799/tcp (the per-VM guest-agent DNAT range).

Operating

./fc-agent.sh start               # launch agent on :8090, print env
./fc-agent.sh stop                 # stop
./fc-agent.sh restart              # stop+start; re-attaches to running VMs
./fc-agent.sh log                  # tail agent log
./fc-agent.sh env                  # print env keys for an already-running agent
./fc-agent-test.sh                 # contract smoke test

# systemd integration (optional, for long-lived hosts)
./fc-agent.sh install-service      # enable fc-agent.service (survives reboot)
./fc-agent.sh uninstall-service

To pick up a new agent binary: re-run ./fc-bake-rootfs.sh then ./fc-agent.sh restart.

On startup, the agent walks its state directory and for each VM either reuses it (if the process and socket are alive) or transparently recreates the TAP/DNAT and relaunches Firecracker with the same config. That means systemctl restart fc-agent and host reboots don’t lose VMs, as long as the systemd unit is installed.

Auto-update

fc-update.sh keeps a self-hosted box on the latest GitHub release. It pulls the checkout first, resolves the latest release tag, then compares that tag against the .fc-version marker file next to the script. If they differ it downloads the new fused, re-bakes the rootfs, and restarts the agent. The comparison is string equality, not a version ordering, so re-tagging or rolling back to an older release also triggers an update, and FC_FORCE=1 re-applies the current one. It never shells out to fused --version to make that decision:

./fc-update.sh                 # one-shot: update if the marker != the latest tag
./fc-agent.sh install-updater  # weekly systemd timer (Mon 04:00 UTC +/- 30m)
./fc-agent.sh uninstall-updater

No token needed (public repo). Optional host-agent/firecracker/.fc-updater.env can set GH_TOKEN=... to dodge API rate limits, or FUSE_ORCH_SERVICE/FUSE_ORCH_BIN to also update a co-located orchestrator. Override the source repo with FUSE_REPO=owner/name.

State

State lives under agent-state/vms/<vm_id>/, safe to rm -rf if the agent is stopped and you want a clean slate.

Hit an issue baking or booting? See Troubleshooting.

Was this page helpful?