GPU batch and training jobs

Whole-device and fractional MIG GPU passthrough for workloads that need real hardware

Some workloads (model training, batch inference, anything CUDA-heavy) need real GPU hardware, not a virtualized fraction behind someone else’s driver stack. Fuse’s QEMU/VFIO backend passes GPUs into the guest directly, either as an entire device with its full IOMMU group, or as a MIG instance carved out of one.

The pattern

Register a GPU host with its device inventory, then request a GPU by kind in your Fusefile, the environment schedules onto a matching host automatically:

fuse host register gpu-1 \
  --url http://gpu-host:8091 \
  --token "$QEMU_AGENT_TOKEN" \
  --backend qemu \
  --gpus 1 \
  --gpu-kind a100 \
  --max-vms 4

version: 1
image: cuda-12 # a rootfs baked on the host, not an OCI ref

resources:
  cpus: 16
  memory: 64GB
  gpu: 1
  gpu_kind: a100

run: python train.py --epochs 50

fuse up

• You never name a backend directly in the Fusefile, requesting gpu: 1 is what routes the environment onto a QEMU-backed host with matching free GPU inventory. CPU-only Fusefiles keep scheduling onto the default Firecracker backend, untouched.
• gpu_kind is an optional model match (a100, or whatever label the host registered with). Omit it to accept any available GPU kind on a host that has one free. On a host reporting per-device inventory the match is a case-insensitive substring test against each device’s model, so a100 matches NVIDIA A100-SXM4-40GB.
• The whole GPU (and every PCI function in its IOMMU group, including a companion audio device if it has one) passes through to the guest. The workload sees real hardware, not an emulated or shared device.

Fractional GPUs with MIG

A whole A100 or H100 is more card than many jobs need. MIG (Multi-Instance GPU) is the NVIDIA feature that splits one such card into up to seven hardware-isolated instances, each with its own compute slices, memory, and cache paths. A MIG instance is real partitioned hardware, not a time-sliced share, so one tenant cannot starve or observe another. Ask for one by naming a profile:

resources:
  cpus: 4
  memory: 16GB
  gpu: 1
  gpu_kind: a100
  gpu_profile: 1g.10gb

run: python infer.py

Reading a profile name

A profile name is two numbers: compute slices, then memory.

Which profiles exist is a property of the card, not of Fuse: an 80GB A100 offers 1g.10gb through 7g.80gb, a 40GB A100 offers 1g.5gb through 7g.40gb. Use the names your hardware documents.

When to use MIG instead of a whole GPU

Use MIG when the job is smaller than a card and you want more of them running concurrently: single-stream inference, notebooks, CI jobs that just need CUDA to be present, per-tenant isolation on shared hardware.

Ask for a whole device when the job needs the full memory or the full SM count, when it uses NVLink or peer-to-peer transfers between GPUs (MIG instances are isolated from each other and cannot do that), when it wants more than 7 units of anything on one card, or when you do not yet know its ceiling. Most training runs want whole devices.

gpu counts instances, not cards

gpu is always a device count. Setting gpu_profile does not turn it into a fraction, it changes what is being counted:

gpu: 2 # two entire physical GPUs

gpu: 2
gpu_profile: 1g.10gb # two 1g.10gb instances, together about 2/7 of one A100

That is decision D5. The scheduler only ever allocates discrete units it can bind to a VM, so the count means the same thing whether the unit is a PCI device or a carved MIG instance. There is no way to write “half a GPU” as a fractional gpu: value, and none is planned.

What fails at compile time

Four GPU rules run client-side, in fuse up, fuse validate, and fuse compile, so a malformed GPU request fails on your machine rather than being scheduled and failing on a host. They are accumulated rather than short-circuited, so a Fusefile with several GPU mistakes reports all of them at once.

The parse rule is the nvidia mig-parted grammar: a slice count of 1 through 7, then g., then a memory size in GB, then an optional +me. 1g.10gb, 3g.20gb, and 1g.10gb+me all parse. Matching is case-insensitive and the compiled value is lowercased, so 1G.10GB is accepted and sent as 1g.10gb. Note that the grammar only checks shape: it cannot know which profiles your cards actually offer, so a well-formed name for a profile nobody carved is a scheduling failure later, not a compile error here.

The MIG-capability rule is a denylist, not an allowlist. Kinds known not to support MIG (v100, t4, p100, p40, k80, rtx, gtx, titan, l4, l40, l40s) are rejected outright, and an empty or unrecognized gpu_kind passes, on the assumption that the host’s reported per-device inventory is the better authority and that a future MIG-capable card should not be blocked by a stale list. The test is a case-insensitive substring match, so gpu_kind: rtx4090 is rejected by the rtx entry.

The same four rules are re-applied by the orchestrator against the compiled request, so a raw SDK or HTTP caller that skips the Fusefile gets the same errors as a 400.

Carving and scheduling instances

The operator carves the MIG instances on the host ahead of time with qemu-mig-setup.sh, they are not carved on demand. The agent probes the resulting inventory and reports one mig_instances entry per carved instance, so the orchestrator binds a specific instance UUID to each VM and knows exactly which slice went to which VM. Register the host as a qemu backend; when the probe finds instances you do not need --mig-profile:

fuse host register gpu-1 \
  --url http://gpu-host:8091 \
  --token "$QEMU_AGENT_TOKEN" \
  --backend qemu \
  --gpu-kind a100 \
  --max-vms 8

--mig-profile profile=count remains as an override for hosts that report no probeable per-instance inventory (a hand-managed mig-inventory.txt without parent detail). There the count map is the scheduling unit and the qemu agent picks the UUID locally.

MIG capacity and whole-device capacity are tracked as separate pools. A host can register both, and a 1g.10gb request never consumes a whole device from the passthrough pool, so slice workloads and whole-card workloads don’t contend for the same inventory.

MIG placement honors gpu_kind. A host is only a candidate for a gpu_profile request when it has free instances of that profile and the instance is on a MIG-capable card matching the requested kind, so a request pinned to a100 is not routed to a host that carries no MIG-capable a100 at all. Omit gpu_kind and any host offering the profile is eligible.

The tradeoff: no snapshots

GPU environments cannot be snapshotted or forked at all, a VFIO-passed-through device can’t be checkpointed the way a Firecracker microVM’s disk can. This applies to MIG environments too, a slice is still passthrough hardware. Snapshot and fork calls against a GPU environment fail with an explicit “not supported for gpu environments” message rather than silently doing nothing. Design long training runs to checkpoint their own progress to durable storage (object storage, a mounted volume) from inside the guest, the way you normally would for a long job that might restart, rather than relying on Fuse’s snapshot mechanism.

Bringing up a GPU host

Registering the host is the easy part, getting a bare-metal box IOMMU-ready and VFIO-bound is the real setup work. See GPU host setup for the full bring-up: driver branch selection, qemu-vfio-bind.sh, and the hardware validation e2e test.

Read next

• Providers for the Firecracker-vs-QEMU backend model.
• GPU host setup for the operational bring-up.