Skip to content
Fuse
Esc
↑↓navigate↵open⌘Jpreview
On this page

Troubleshooting and known issues

Common install/setup problems and known limitations, with workarounds

KVM not available

Firecracker and QEMU both need hardware virtualization. If a host can’t run microVMs at all, this is almost always why.

ls -l /dev/kvm
[ -r /dev/kvm ] && [ -w /dev/kvm ] && echo ok

If /dev/kvm is missing: the host is either an ordinary container or a VM without nested virtualization enabled. You need bare-metal Linux, or a cloud instance type with nested virt explicitly enabled (e.g. GCP *-metal/nested-virt images, AWS *.metal, or a bare-metal provider). There is no software workaround: this is a hardware/hypervisor requirement.

Orchestrator silently runs in stub mode

Symptom: fuse up or fuse environment create succeeds, but nothing real ever boots, or environments behave oddly.

Cause: the orchestrator falls back to an in-memory stub provider whenever FIRECRACKER_BASE_URL is unset. This is intentional (it’s what the Quickstart relies on), but it’s easy to leave unset by accident when moving from local dev to a real host. Check the orchestrator’s startup logs; it logs which mode it’s running in, and confirm FIRECRACKER_BASE_URL and FIRECRACKER_TOKEN are set to the values fc-agent.sh start printed. See Deploying the orchestrator.

firecracker create vm: http 401

Symptom: fuse up or fuse environment create fails with firecracker create vm: http 401. The orchestrator is healthy and the host agent is running, only environment creation fails.

Cause: the per-host agent token the orchestrator has stored no longer matches the agent’s FC_AGENT_TOKEN. The agent authenticates every route, so the very first call the provider makes is rejected. This usually follows the agent’s .env being regenerated, or TOKEN_ENCRYPTION_KEY changing (which leaves the stored token undecryptable). The two are stored separately, so re-running the agent never updates the orchestrator’s copy on its own.

Fix: print the current value with ./fc-agent.sh env on the host, then re-register the host with it, fuse host register <id> --token <FIRECRACKER_TOKEN> .... Registration overwrites the stored token.

Rootfs bake fails or produces an incomplete guest

Symptom: environments get stuck in provisioning, or the guest boots but fused (or podman) isn’t present.

The most common cause is a bake that didn’t fully complete: fc-bake-rootfs.sh mounts the rootfs, injects several components (the fused binary, podman-static, iptables, the CA bundle, the systemd unit), and unmounts. If any step in that sequence is interrupted, the resulting rootfs-fused.ext4 can be missing pieces silently; new VMs will boot but the agent won’t be reachable.

Fix: re-run ./fc-bake-rootfs.sh from a clean rootfs.ext4 (re-run ./fc-install.sh first if you’re unsure the base rootfs itself is intact), and check its output for every step completing rather than assuming success. See Firecracker host setup for exactly what should be present afterward.

netavark / nftables errors during the bake

Symptom: the bake or an in-guest podman run fails with a netavark error referencing nftables or -m comment.

Cause: the Firecracker CI kernel (vmlinux-5.10.223) ships without CONFIG_NETFILTER_XT_MATCH_COMMENT, CONFIG_FUSE_FS, or CONFIG_NF_TABLES. Netavark unconditionally emits -m comment rules and fails without them.

Fix: this is why the baked rootfs’s containers.conf forces new containers into the host’s network namespace (netns = "host", firewall_driver = "none"); it sidesteps netavark’s netfilter dependency entirely. Isolation is still provided by the microVM boundary itself. If you need per-container network isolation inside one VM, you’d need a custom kernel enabling the missing netfilter modules; this is not supported by the reference bake.

apt doesn’t work inside the guest

The Firecracker CI base rootfs ships with an empty /var/lib/dpkg/status, so apt is unusable from inside a running guest. This is expected, not a bug; customize the rootfs by mounting the ext4 file from the host and modifying it there (the same way fc-bake-rootfs.sh itself does), not by running apt inside a live guest.

Snapshot restore can corrupt unsynced writes

Symptom (on an un-updated host): a clean snapshot-then-restore cycle (no writes in between) round-trips correctly. But if the guest had written data that hadn’t been flushed to disk when you called restore, the affected file read back as NUL bytes afterwards instead of the snapshot’s content.

Root cause: restore stopped the Firecracker process and copied the snapshot file over the live rootfs.ext4 in place. The just-killed guest could still have dirty host page-cache pages tied to that inode, and the in-place copy left those stale pages readable at offsets it didn’t overwrite.

The fix: restore now copies the snapshot to a temporary file and atomically renames it over the live rootfs, so the restored image always lands on a fresh inode with no page-cache history from the dead VM. No guest-side sync is required, and the old workaround (syncing inside the guest before every snapshot and restore) is no longer needed.

If you still see stale data after a restore, you are almost certainly running a host agent that predates the fix. Update it and restart fc-agent.

Where to look next

Was this page helpful?