GPU (QEMU) host setup

Whole-GPU passthrough via QEMU/VFIO

GPU environments use QEMU/KVM with whole IOMMU groups passed through via VFIO. This requires bare-metal Linux, IOMMU enabled, /dev/kvm, and a GPU that can be detached from its host driver.

Setup

cd host-agent

# install qemu, ovmf, a base cloud image, and the ssh keypair
./qemu-install.sh

# build the in-guest agent and bake a cuda image with an explicit driver branch
./fc-build-agent.sh
./qemu-bake-cuda-rootfs.sh 550

# inspect IOMMU groups and bind every member of each gpu group to vfio-pci.
# bind also snapshots nvidia-smi into vfio-inventory.txt -- see "IOMMU groups"
# below for why the snapshot must happen before the bind.
./qemu-vfio-bind.sh --list
sudo ./qemu-vfio-bind.sh

# install and start the host agent
sudo cp qemu-agent.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now qemu-agent

The reference bake produces rootfs-cuda.qcow2: a CUDA-capable qcow2 image with the generated SSH public key installed as root’s authorized_keys, plus vmlinuz.bin extracted for QEMU.

Ordering constraint: probe before bind

nvidia-smi is blind to a GPU once it is bound to vfio-pci, because the NVIDIA driver no longer owns the device. So the probe must run before the bind:

./qemu-vfio-bind.sh --list (or any nvidia-smi invocation) while devices are still on nvidia.
sudo ./qemu-vfio-bind.sh to move them to vfio-pci.

qemu-vfio-bind.sh enforces this internally: do_bind calls snapshot_smi before it touches any /sys/bus/pci/.../driver_override, and do_list reads that snapshot (VFIO_SMI_SNAPSHOT) to enrich each inventory line. If you bind first and then try to recover the metadata, the live probe returns nothing and you lose per-device detail (uuid, model, memory_mb, mig_mode) for the orchestrator’s gpu_devices list. The summary count survives (it comes from lspci, not nvidia-smi), but the rich per-device data does not.

IOMMU groups

The bind step writes vfio-inventory.txt (default $QEMU_DIR/vfio-inventory.txt, override with VFIO_INVENTORY), which qemu-agent.py consumes. Each line represents one indivisible IOMMU group, with an optional metadata suffix after a |:

<count> <kind> <pci_slot> [<pci_slot> ...] [| key=val;key=val;...]
1 a100 0000:17:00.0 0000:17:00.1
1 a100 0000:17:00.0 0000:17:00.1 | bus_id=0000:17:00.0;uuid=GPU-abc123;model=NVIDIA A100-SXM4-40GB;memory_mb=40960;mig_mode=Enabled

The count includes GPU display functions only. The PCI list includes every function in the group, such as a GPU’s companion audio function, because QEMU must attach the complete group. kind is lowercased when parsed, and blank lines and # comments are skipped.

The recognized metadata keys are bus_id, uuid, model, memory_mb, driver_version, compute_cap, and mig_mode. Unknown keys are ignored, so the format can grow without breaking older agents. qemu-vfio-bind.sh writes only five of them (bus_id, uuid, model, memory_mb, mig_mode), since that is what its pre-bind nvidia-smi query captures; driver_version and compute_cap are parsed if present but come from the live probe rather than the inventory file.

The metadata exists because of an ordering constraint: qemu-vfio-bind.sh takes an nvidia-smi snapshot before binding devices to vfio-pci, since once a card is bound to vfio-pci the NVIDIA driver can no longer report on it. Capturing the details up front is the only way to keep them. If no line in the inventory carries metadata, the agent falls back to probing nvidia-smi live. The fallback is all-or-nothing across the whole file, a single enriched line suppresses it for every other line, rather than being decided per line.

The agent serves this captured inventory from host_capacity() / GET /v1/capacity as a summary (gpus, gpu_kind) plus a per-device gpu_devices list, which is what the orchestrator probes at registration time. See the next section.

Registering the host

Set QEMU_AGENT_TOKEN in the service environment, then register. GPU capacity (count and kind) is probed from the host agent the same way cpus, ram-mb, and storage-gb are: omit --gpus / --gpu-kind (or leave them at 0 / empty) and the orchestrator pulls them from the inventory the bind step produced. No manual line-summing, no copy-pasted count:

fuse host register gpu-1 \
  --url http://gpu-host:8091 \
  --token "$QEMU_AGENT_TOKEN" \
  --backend qemu \
  --cpus 16 \
  --ram-mb 65536 \
  --storage-gb 500 \
  --max-vms 4

--max-vms is always required: it is a scheduling policy, not a hardware fact, so it is never probed. A host with zero GPUs is valid — register it the same way and the probe reports an empty GPU pool.

Overriding the probe

Pass --gpus and/or --gpu-kind to override the probed value. This is the escape hatch for a deliberate carve-out (e.g. reserving one card for the host) or a known overcommit. The orchestrator reconciles declared vs. probed:

Declared below or equal to probe: accepted silently. Useful for reserving a GPU you don’t want scheduled.
Declared above probe: still registers, with a warning surfaced in fuse host get (and the register response). Scheduling will then fight for capacity that isn’t really there — only do this intentionally.
Declared --gpu-kind that differs from the probe: kept as-declared, with a mismatch warning.

# reserve one GPU for the host: declare one fewer than probed
fuse host register gpu-1 \
  --url http://gpu-host:8091 \
  --token "$QEMU_AGENT_TOKEN" \
  --backend qemu \
  --gpus 3 \
  --gpu-kind a100 \
  --cpus 16 \
  --ram-mb 65536 \
  --storage-gb 500 \
  --max-vms 4

When the probe fails

A probe failure on a QEMU host with no declared --gpus becomes a warning, not a hard failure, because zero GPUs is a legitimate configuration. If the host agent is unreachable or its /v1/capacity errors, the host still registers with gpus=0 and the warning is attached to the host record. A QEMU host that should have GPUs but probes zero usually means the inventory file is empty or stale — re-run sudo ./qemu-vfio-bind.sh and check vfio-inventory.txt before re-registering.

Fractional GPUs with MIG

A host can also serve MIG (Multi-Instance GPU) slices, fractions of a physical card exposed as separate mdev devices. Creating those instances is a prerequisite you handle on the host, outside Fuse. Fuse consumes MIG instances, it never creates them, but the qemu-mig-setup.sh operator script automates the create/list/destroy lifecycle and writes the inventory file the agent probes.

Carving and listing MIG instances

qemu-mig-setup.sh is the MIG analogue of qemu-vfio-bind.sh. It enables MIG mode on each card, carves the requested profiles into GPU instances, and emits the inventory the agent reports to the orchestrator. Apply a layout with one --profile profile=count flag per profile:

sudo ./qemu-mig-setup.sh --profile 1g.10gb=4 --profile 2g.20gb=2

The layout is persisted to $QEMU_DIR/mig-layout.conf so a reboot re-applies it (the agent also re-applies it after a VM holding mdevs is destroyed, when MIG_LIFECYCLE_MANAGED is set). List the live instances with --list:

./qemu-mig-setup.sh --list

The inventory format is four fields, the last one optional:

<profile> <kind> <mdev_uuid> [<parent_gpu_uuid>]
1g.10gb a100 3f8e1a2b-0000-4000-8000-000000000001 GPU-abc123

Both profile and kind are lowercased when parsed. parent_gpu_uuid ties an instance back to the card it was carved from, which lets the scheduler match a MIG request’s gpu_kind against the parent device’s model when the instance itself reports no kind.

How the orchestrator allocates MIG

When the agent’s capacity probe finds a non-empty mig-inventory.txt, it reports mig_instances (one entry per carved instance) alongside the mig_profiles count summary. The orchestrator then switches from count-based to per-instance allocation: it binds specific instance UUIDs to VMs and records them on the VM, so it knows exactly which instance went to which VM. Releasing a VM frees its specific UUIDs, not a counter.

This is strictly additive. A host that reports no per-instance inventory (a hand-managed mig-inventory.txt without parent detail, or an operator who wants to override the probe) falls back to the count-map path, where mig_profiles is the scheduling unit and the qemu agent picks the UUID locally.

Registering MIG capacity

Register the host as a qemu backend. When the agent probes per-instance inventory, you do not need --mig-profile: the orchestrator derives the count map from the probed instances:

fuse host register gpu-1 \
  --url http://gpu-host:8091 \
  --token "$QEMU_AGENT_TOKEN" \
  --backend qemu \
  --gpu-kind a100 \
  --cpus 16 \
  --ram-mb 65536 \
  --storage-gb 500 \
  --max-vms 4

--mig-profile profile=count remains supported as an override: it sets the declared count map directly and is what a host with no probeable inventory falls back to. Use it only when you want to pin the counts yourself rather than let the probe report them.

The MIG pool and the whole-device pool are separate inventories. A MIG placement never consumes a whole device, so a host registered with both --gpus and MIG capacity serves the two request shapes independently.

Requesting a GPU

A Fusefile requests the device without naming a virtualization backend directly: the non-zero gpu count is what routes it to a matching QEMU host:

resources:
  gpu: 1
  gpu_kind: a100

Adding gpu_profile switches the request to a MIG instance. The gpu field then counts MIG instances rather than whole cards:

resources:
  gpu: 1
  gpu_kind: a100
  gpu_profile: 1g.10gb

See Fusefile for the full field reference.

Validating the setup

./qemu-agent-test.sh
FUSE_GPU_E2E=1 FUSE_GPU_KIND=a100 ./qemu-e2e.sh

The end-to-end test registers a QEMU host, creates a GPU environment, runs nvidia-smi in the guest, verifies snapshots are refused, and confirms destroy removes the QEMU VM. Without FUSE_GPU_E2E=1 and a reachable GPU agent, it reports SKIP rather than failing.

Firewall

Open at your cloud / external firewall:

8091/tcp: the qemu-agent HTTP API. GPU hosts listen on 8091, not the 8090 that fc-agent uses on Firecracker hosts.
19651-19899/tcp: the per-VM guest-agent DNAT range. This is not the 19551-19799 that Firecracker hosts use: qemu-agent defaults FUSE_HOST_PORT_BASE to 19650 where fc-agent defaults it to 19550, and each allocates one port per VM index from 1 to 249. Opening the Firecracker range on a GPU host blocks every GPU VM’s guest agent.