Skip to content

qemu.md

Latest commit

 

History

History
299 lines (239 loc) · 17.7 KB

qemu.md

File metadata and controls

299 lines (239 loc) · 17.7 KB
Raw

qemu-coreboot-(fb)whiptail-tpmX(-hotp) boards

The qemu-coreboot-fbwhiptail-tpm1-hotp configuration (and their variants) permits testing of most features of Heads.
It requires a supported USB token (which will be reset for use with the VM, do not use a token needed for a real machine). With KVM acceleration, speed is comparable to a real machine. If KVM is unavailable, lightweight desktops are still usable.

Heads is currently unable to reflash firmware within qemu, which means that OEM reset and re-ownership cannot be fully performed within the VM. Instead, a GPG key can be injected in the Heads image from the host during the build.

The TPM and disks for this configuration are persisted in the build/qemu-coreboot-fbwhiptail-tpm1-hotp/ directory by default.

Bootstrapping a working system

Important: The supported and tested workflow uses the provided Docker wrappers (./docker_repro.sh, ./docker_local_dev.sh, or ./docker_latest.sh). Host-side installation of QEMU, swtpm, or other QEMU-related tooling is unnecessary and is not part of the standard, supported workflow; only advanced or edge-case scenarios should install those tools on the host (see 'Troubleshooting' below for guidance).

  1. Install Docker

Note: the Nix-built Docker images used by ./docker_repro.sh include QEMU (qemu-system-x86_64), swtpm / libtpms, canokey-qemu (a virtual OpenPGP smartcard), and other userspace tooling required to build and test QEMU targets. These images are intended to be self-contained for QEMU testing; host-focused build instructions (e.g., building swtpm on the host) were removed to avoid divergence—use the Docker wrappers for the tested workflow.

If you do not specify USB_TOKEN when running QEMU targets, the container will use the included canokey-qemu virtual token by default. To forward a hardware token from the host, set USB_TOKEN or pass hostbus/hostport/vendorid,productid to the make invocation.

If you plan to manage disk images on the host (outside the container), install qemu-utils for qemu-img. Mounting uses losetup (from util-linux, present on all Linux distributions):

sudo losetup --find --show --partscan \
  ./build/x86/<board>/usb_fd.raw
# → /dev/loopN; use sudo fdisk -l /dev/loopN to check partitions
sudo mount /dev/loopNp1 /mnt   # partitioned, or /dev/loopN if flat

The Makefile creates usb_fd.raw (sparse — ~200K for a 64G virtual disk, grows only as ISOs are copied in) with an MBR partition table and ext4 filesystem. Older images (from before qemu-img create) may be flat — check with sudo fdisk -l first.

Note: the Docker container bind-mounts only the cloned Heads directory ($(pwd)), so images must reside within the clone — use the qemu_img/ directory inside the repo as a backing store (see hardlink workflow below).

  1. Build Heads
    • ./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp
  2. Install OS
    • ./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp INSTALL_IMG=<~/heads/path_to_iso.iso> run
    • Lightweight desktops (XFCE, LXDE, etc.) are recommended, especially if KVM acceleration is not available (such nested in Qubes OS)
    • When running nested in a qube, disable memory ballooning for the qube, or performance will be very poor.
    • Include QEMU_MEMORY_SIZE=6G to set the guest's memory (6G, 8G, etc.). The default is 4G to be conservative, but more may be needed depending on the guest OS.
    • Include QEMU_DISK_SIZE=30G to set the guest's disk size, the default is 20G.
  3. Shut down and boot Heads with the USB token attached, proceed with OEM reset
    • ./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp USB_TOKEN=<token> run
    • If you do not set USB_TOKEN, the included canokey-qemu virtual token will be used by default.
    • For <token>, use one of:
      • NitrokeyPro - a Nitrokey Pro by VID/PID
      • NitrokeyStorage - a Nitrokey Storage by VID/PID
      • Nitrokey3NFC - a Nitrokey 3 by VID:PID
      • LibremKey - a Librem Key by VID/PID
      • hostbus=#,hostport=# - indicate a host bus and port (see qemu usb-host)
      • vendorid=#,productid=# - indicate a device by VID/PID (decimal, see qemu usb-host)
    • You do need to export the GPG key to a USB disk, otherwise defaults are fine.
    • Head will show an error saying it can't flash the firmware, continue
    • Then Heads will indicate that there is no TOTP code yet, at this point shut down (Continue to main menu -> Power off)
  4. Get the public key that was saved to the virtual USB flash drive

    • sudo mkdir /media/fd_heads_gpg

    • Attach the raw image and mount the public partition:

      sudo losetup --find --show --partscan
      ./build/x86/qemu-coreboot-fbwhiptail-tpm1-hotp/usb_fd.raw

      → prints /dev/loopN with /dev/loopNp1, /dev/loopNp2, etc.

      sudo fdisk -l /dev/loopN # verify partition layout sudo mount /dev/loopNp2 /media/fd_heads_gpg # second/public partition

    • Look in /media/fd_heads_gpg and copy the most recent public key

    • sudo umount /media/fd_heads_gpg

    • sudo losetup -d /dev/loopN # detach (replace N with the actual number)

  5. Inject the GPG key into the Heads image and run again
    • ./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp PUBKEY_ASC=<path_to_key.asc> inject_gpg
    • ./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp USB_TOKEN=LibremKey PUBKEY_ASC=<path_to_key.asc> run
  6. Initialize the TPM - select "Reset the TPM" at the TOTP error prompt and follow prompts
  7. Select "Default boot" and follow prompts to sign /boot for the first time and set a default boot option

You can reuse an already created ROOT_DISK_IMG by passing its path at runtime. Ex: ./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm1 PUBKEY_ASC=~/pub_key_counterpart_of_usb_dongle.asc USB_TOKEN=NitrokeyStorage ROOT_DISK_IMG=~/heads/build/x86/qemu-coreboot-fbwhiptail-tpm1-hotp/root.qcow2 run

Saving Disk Images from Build-Dir Wipes

The Docker container can only see files inside the cloned Heads directory (docker/common.sh line 1446: -v "$(pwd):$(pwd)"). Any backup copy must live at a path inside the clone — ~/QemuImages/ and other user-home paths are invisible to Docker.

The build directory (build/x86/<board>/) is ephemeral. A make clean or fresh checkout deletes build/ entirely, including installed OS images and populated USB disks. Use hardlinks to keep safe copies inside the clone and share across board variants:

mkdir -p qemu_img                         # safe storage inside clone
cp build/x86/<board>/root.qcow2 qemu_img/ # copy OS install to safety
rm build/x86/<board>/root.qcow2           # remove build-tree copy
cp -alf qemu_img/root.qcow2 build/x86/<board>/  # hardlink back
# Now both paths point to the same data on disk.
# Wiping build/ won't touch qemu_img/.

# Restore after a wipe:
cp -alf qemu_img/root.qcow2 build/x86/<board>/root.qcow2
cp -alf qemu_img/usb_fd.img  build/x86/<board>/usb_fd.raw

cp -alf creates a hardlink — a second directory entry pointing to the same data blocks (zero additional space). Data is freed only when the last link is deleted. Caveat: writes to one link affect all links. Use qemu-img snapshot before modifying the root disk.

USB flash drive workflow

mkdir -p qemu_img                             # safe storage inside clone

# Step 1: Create the USB image via the Makefile.
./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 \
  QEMU_USB_SIZE=64G run
# → build/x86/.../usb_fd.raw now exists.

# Step 2: Save a master copy IMMEDIATELY (before population).
cp build/x86/qemu-coreboot-fbwhiptail-tpm2/usb_fd.raw qemu_img/usb_fd.img
rm build/x86/qemu-coreboot-fbwhiptail-tpm2/usb_fd.raw
cp -alf qemu_img/usb_fd.img build/x86/qemu-coreboot-fbwhiptail-tpm2/usb_fd.raw

# Step 3: Populate with ISOs.
sudo losetup --find --show --partscan build/x86/.../usb_fd.raw
sudo mount /dev/loop0p1 /mnt
cp ~/Downloads/ISOs/*.iso /mnt/
sudo umount /mnt && sudo losetup -d /dev/loop0

# Step 4: Hardlink into other board build directories.
cp -alf qemu_img/usb_fd.img build/x86/qemu-coreboot-fbwhiptail-tpm1-hotp/usb_fd.raw
cp -alf qemu_img/usb_fd.img build/x86/qemu-coreboot-fbwhiptail-tpm2-hotp/usb_fd.raw

# Next run uses the hardlink — Makefile skips creation since the file exists.

Daily development cycle

After OS install + USB provisioned, reference both from ./qemu_img/:

./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp \
  PUBKEY_ASC=pubkey.asc \
  USB_TOKEN=Nitrokey3NFC \
  ROOT_DISK_IMG=./qemu_img/root.qcow2 \
  inject_gpg run

Running via Docker wrappers

We provide convenient wrapper scripts at the repository root that encapsulate Docker invocation and automatically handle common host integrations needed for QEMU runs.

Wrapper comparison

Script Image Use
docker_latest.sh Defaults to pinned digest when available Convenience: run the latest published image
docker_local_dev.sh linuxboot/heads:dev-env Development: use local image built from the flake (rebuilds when flake files are dirty)
docker_repro.sh Image pinned from .circleci/config.yml Reproducible builds that match CircleCI

What the wrappers handle

Wrapper options: some runtime behavior is controlled via environment variables documented in the repository README (see 'Wrapper options & environment variables'). Wrapper scripts now have focused --help output for their own variables, and ./docker/common.sh prints the full environment reference. Important ones are HEADS_DISABLE_USB (set to 1 to disable automatic USB passthrough and cleanup) and HEADS_X11_XAUTH (force mounting your $HOME/.Xauthority).

Make variables such as USB_TOKEN, PUBKEY_ASC, INSTALL_IMG, QEMU_MEMORY_SIZE, QEMU_DISK_SIZE, QEMU_USB_SIZE, ROOT_DISK_IMG, CPUS and V are forwarded to the make invocation and affect how targets/qemu.mk runs QEMU. See targets/qemu.mk for token formats and examples.

The virtual USB flash drive (usb_fd.qcow2 by default, or usb_fd.raw if an existing raw image from a previous build is found) is created at build time under build/x86/<BOARD>/. Default virtual size is 64 GB — overridable via QEMU_USB_SIZE. QCOW2 is sparse: only written blocks consume host disk space (initial size ~200K). Raw files are also sparse when created via qemu-img create -f raw.

The Makefile auto-detects: if usb_fd.raw exists in the build directory, it's used directly; otherwise it creates a new usb_fd.qcow2 from a raw temp (partitioned + formatted via losetup, then converted to qcow2). The conversion is sparse — only written blocks are preserved.

See Hardlinks for Reusable Images above for the recommended workflow: save a master copy immediately, hardlink it back into the build directory, then populate it (which modifies the shared blocks).

Quick mount reference (after following the hardlink workflow):

sudo losetup --find --show --partscan \
  build/x86/qemu-coreboot-fbwhiptail-tpm2/usb_fd.raw
# → /dev/loopN; check partitions with sudo fdisk -l /dev/loopN
sudo mount /dev/loopNp1 /mnt   # or /dev/loopN if flat
cp ~/Downloads/ISOs/*.iso /mnt/
sudo umount /mnt && sudo losetup -d /dev/loopN

Note: when USB passthrough is active the wrapper will warn and, on
interactive shells, give a 3s abort window before attempting to kill
processes that hold the token (e.g., `scdaemon`/`pcscd`) to free the
device; set `HEADS_DISABLE_USB=1` to opt out.

- **KVM passthrough**: when `/dev/kvm` exists on the host the container is run with `/dev/kvm` mounted into the container, enabling KVM-accelerated QEMU.
- **X11 GUI support**: the wrappers mount the X11 socket and programmatically create a temporary Xauthority file (via `mktemp -t heads-docker-xauth-XXXXXX`, or `/tmp/.docker.xauth-<uid>` as fallback when mktemp is unavailable) when `xauth` is available; they fall back to mounting `${HOME}/.Xauthority` when needed and set `XAUTHORITY` inside the container so GTK/SDL QEMU windows work. The temp file is cleaned up automatically after `docker run` completes.
  - To force mounting your `${HOME}/.Xauthority` regardless of socket detection, set `HEADS_X11_XAUTH=1`.
- **USB passthrough**: when host USB buses exist `/dev/bus/usb` is mounted into the container so VMs can access hardware tokens. To explicitly disable automatic USB passthrough set `HEADS_DISABLE_USB=1`.
- **USB token cleanup**: the wrappers attempt to detect and stop local GPG/toolstack processes (e.g., `scdaemon`, `pcscd`) which might hold USB tokens. Behavior notes:
  - If `sudo` can be run without a password the cleanup runs silently.
  - The cleanup avoids prompting for a password in non-interactive shells; it will prompt only when running interactively (attached to a TTY). To skip the cleanup entirely set `HEADS_DISABLE_USB=1`.
- **Convenience variables accepted by the wrappers**: `V=1` for verbose make output, `CPUS=N` to set parallelism for builds, and any `make` variables may be passed through to the container command.
- **Argument forwarding**: arguments given to the wrapper are forwarded directly to the container command (no special separator needed). For example: `./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 run`.

Environment variables reference
---

| Variable | Default | Effect |
|---|---:|---|
| `HEADS_DISABLE_USB` | `0` | When `1`, disable automatic USB passthrough and USB cleanup |
| `HEADS_X11_XAUTH` | `0` | When `1`, mount `${HOME}/.Xauthority` into the container (force usage even when a programmatic Xauthority would otherwise be created) |
| `HEADS_SKIP_DOCKER_REBUILD` | `0` | When `1`, skip rebuilding the local Docker image when `flake.nix`/`flake.lock` are dirty |
| `HEADS_AUTO_INSTALL_NIX` | `0` | When `1`, automatically attempt single-user Nix install if `nix` is missing (suppresses prompt) |
| `HEADS_AUTO_ENABLE_FLAKES` | `0` | When `1`, automatically enable flakes by writing to `$HOME/.config/nix/nix.conf` (suppresses prompt) |
| `HEADS_MIN_DISK_GB` | `50` | Minimum free disk in GB required on `/nix` or `/` before attempting rebuild |
| `HEADS_SKIP_DISK_CHECK` | `0` | When `1`, skip the disk-space preflight check |

Examples
---

- Reproducible (uses image version from CircleCI config):
  - `./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 run`
  - `./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 PUBKEY_ASC=pubkey.asc USB_TOKEN=Nitrokey3NFC inject_gpg`
  - `HEADS_DISABLE_USB=1 ./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 PUBKEY_ASC=pubkey.asc run`
  - `HEADS_X11_XAUTH=1 ./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 run`

- Local development image (uses locally built `linuxboot/heads:dev-env`):
  - `./docker_local_dev.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2`

- Published latest image (convenience):
  - `./docker_latest.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 run`

How I tested these wrappers (smoke checks)
---

- Minimal: `source docker/common.sh && build_docker_opts` — should print a short description and show flags such as `--device=/dev/kvm` when KVM is available and `-v /tmp/heads-docker-xauth-XXXXXX:...` (or `-v /tmp/.docker.xauth-<uid>:...` as fallback) when Xauthority was created.
- Functional (examples tested by PR author): see the tests in the PR body (Ubuntu, Debian, Fedora installer flows). Consider testing `./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 run` locally to verify KVM+GTK behavior.

Troubleshooting
---

- Reuse provisioned canokey state across QEMU board build dirs:
  - QEMU boards that use the default virtual token store canokey state at `build/x86/<BOARD>/.canokey-file` (from `targets/qemu.mk`: `-device canokey,file=$(build)/$(BOARD)/.canokey-file`).
  - After provisioning via Heads OEM reset/re-ownership in one QEMU board, you can copy that file into another QEMU board build directory to reuse the same virtual smartcard identity/public key material.
  - Example:
    - `cp build/x86/qemu-coreboot-fbwhiptail-tpm2/.canokey-file build/x86/qemu-coreboot-fbwhiptail-tpm2-prod_quiet/.canokey-file`
  - This is useful when troubleshooting TPM workflows while keeping the same token identity across variants.

- TPM2 interaction capture (pcap) for debugging, similar to a bus sniffer:
  - On TPM2 boards, set `export CONFIG_TPM2_CAPTURE_PCAP=y` in the board config.
  - Heads `tpmr` then uses the pcap TCTI and writes captures to `/tmp/tpm0.pcap` inside the booted Heads environment.
  - Save/copy that file from the guest (mount-usb --mode rw) and inspect it with Wireshark to analyze TPM command/response traffic.
  - This is intended for TPM2 boards (for example the `qemu-coreboot-fbwhiptail-tpm2*` targets).

- Quick checks:
  - `echo $DISPLAY` — ensure `DISPLAY` is set on the host.
  - `command -v xauth` — preferred for programmatic Xauthority cookies.
  - `ls -l /dev/kvm` — verify `/dev/kvm` exists and is accessible.
  - `groups | grep -q kvm` — confirm your user is in a group with access to KVM (or run with appropriate privileges).
  - `source docker/common.sh && build_docker_opts` — inspect the options the wrapper will use without launching Docker.
- GUI issues: prefer installing `xauth` on the host so the wrappers can create a safe programmatic Xauthority file. As a last resort you can run `xhost +SI:localuser:root` (less secure).
- USB/GPG cleanup: if the cleanup is refusing to run due to non-interactive sudo, run the kill steps manually or set `HEADS_DISABLE_USB=1` to skip automatic cleanup.

Notes
---
- Ensure you have an X server available on the host; the wrappers forward `DISPLAY` automatically.
- If KVM is available but `/dev/kvm` is missing, load kernel modules (e.g., `kvm`, `kvm_intel`, `kvm_amd`) so `/dev/kvm` appears.