Skip to content

make-disk-image: Fix overriding kernel modules - #1277

Open
AlexLov wants to merge 2 commits into
nix-community:masterfrom
AlexLov:fix-vmTools-overried
Open

make-disk-image: Fix overriding kernel modules#1277
AlexLov wants to merge 2 commits into
nix-community:masterfrom
AlexLov:fix-vmTools-overried

Conversation

@AlexLov

@AlexLov AlexLov commented Jul 6, 2026

Copy link
Copy Markdown

After #530764 PR kernel modules should be passed as kernelModules instead of kernel.

Fixes error:

… while evaluating definitions from `/nix/store/1c1jbpjisa2249vrffd97vmjdflbcprx-source/lib/make-disk-image.nix':

(stack trace truncated; use '--show-trace' to show the full, detailed trace)

error: vmTools: the `kernel` argument (kernel-modules) has no
`target` attribute, so the kernel image filename cannot be determined.

If you are passing a module tree (e.g. from `pkgs.aggregateModules`) to
make extra modules available, pass it via `kernelModules` instead and
keep `kernel` pointing at a real kernel derivation. Alternatively, pass
`kernelImage` explicitly with the path of the bootable image relative
to the `kernel` derivation output (e.g. "bzImage" or "Image").

This should also fixes tests in #1273 PR

After [#530764](NixOS/nixpkgs#530764) PR
kernel modules should be passed as `kernelModules` instead of `kernel`.

Fixes error:
```
… while evaluating definitions from `/nix/store/1c1jbpjisa2249vrffd97vmjdflbcprx-source/lib/make-disk-image.nix':

(stack trace truncated; use '--show-trace' to show the full, detailed trace)

error: vmTools: the `kernel` argument (kernel-modules) has no
`target` attribute, so the kernel image filename cannot be determined.

If you are passing a module tree (e.g. from `pkgs.aggregateModules`) to
make extra modules available, pass it via `kernelModules` instead and
keep `kernel` pointing at a real kernel derivation. Alternatively, pass
`kernelImage` explicitly with the path of the bootable image relative
to the `kernel` derivation output (e.g. "bzImage" or "Image").
```

This should also fixes tests in [nix-community#1273](nix-community#1273) PR
Comment thread lib/make-disk-image.nix
Co-authored-by: Diffumist <git@diffumist.me>
JuneStepp added a commit to JuneStepp/disko that referenced this pull request Jul 12, 2026
@Bert-Proesmans

Copy link
Copy Markdown
Contributor

After applying this patch the virtual machine hangs in stage-1 script (from initrd) at kernel module loading, on my machine.
I encounter the same symptoms executing the relevant test case so that information is attached below.

In my own scripts I see "loading kernel modules..." printed on the console. That text is not visible in the test output.

I'm stuck on this, can't seem to figure out what is going wrong.


Reproduction

nix build .#checks.x86_64-linux.make-disk-image -L --show-trace --override-input nixpkgs "github:/NixOS/nixpkgs/241313f4e8e508cb9b13278c2b0fa25b9ca27163"

The commit hash is from nixos-unstable (2026-07-19)

Logs from test make-disk-image
<build lines omitted..>
initrd-linux> /etc/modules-load.d/nixos.conf -> /nix/store/imsf7wan48ilm50zbz8i00qmwgqqskc3-initrd-nixos.conf
initrd-linux> /etc/os-release -> /nix/store/qjnjisz6dj8q11h33xi4nmrba5bmcilc-initrd-release
initrd-linux> /etc/passwd -> /nix/store/k2ljzx427ibg5d2kskgndl8mzmwyy6n0-initrd-passwd
initrd-linux> /etc/shadow -> /nix/store/5pfl7dxlf4qiib8shlwypk0kay32qzkk-initrd-shadow
initrd-linux> /etc/shells -> /nix/store/i1vcz3gnrdna4bhw2zvq9ifsg2sg8qam-initrd-shells
initrd-linux> /etc/sysctl.d/nixos.conf -> /nix/store/3wwdhh96pn16fpy1j67h8kkprlz0c457-initrd-nixos.conf
initrd-linux> /etc/systemd/system -> /nix/store/a6a68rvdzizf5fa7w0g3fjn3qk1minan-initrd-units
initrd-linux> /etc/systemd/system.conf -> /nix/store/55kf633fbwd06jk1s6gn1qwpdylhsgkc-initrd-system.conf
initrd-linux> /etc/terminfo/l/linux -> /nix/store/62qhvy2m2lpqj7ca39j9wvp2kkcxwdcg-ncurses-6.6/share/terminfo/l/linux
initrd-linux> /etc/terminfo/v/vt100 -> /nix/store/62qhvy2m2lpqj7ca39j9wvp2kkcxwdcg-ncurses-6.6/share/terminfo/v/vt100
initrd-linux> /etc/terminfo/v/vt102 -> /nix/store/62qhvy2m2lpqj7ca39j9wvp2kkcxwdcg-ncurses-6.6/share/terminfo/v/vt102
initrd-linux> /etc/terminfo/v/vt220 -> /nix/store/62qhvy2m2lpqj7ca39j9wvp2kkcxwdcg-ncurses-6.6/share/terminfo/v/vt220
initrd-linux> /etc/udev/rules.d -> /nix/store/y0aqay023lr7h9zsjlzh3rnqapx26y6a-initrd-udev-rules
initrd-linux> /etc/vconsole.conf -> /nix/store/3kws1ydibxaig0manm8b71v5vr7paby4-vconsole.conf
initrd-linux> /init -> /nix/store/3v7kqg55hx4awhy5hdckniypbgqbqfmx-systemd-261/lib/systemd/systemd
initrd-linux> /lib -> /nix/store/racj06lagjb29da5maak14s49x8schp0-linux-6.18.39-modules-shrunk/lib
initrd-linux> /sbin -> /nix/store/4asc2mb1y6yhzqwgwxk4wcj4d0wb5igb-initrd-bin-env/sbin
initrd-linux> /usr/bin -> /nix/store/4asc2mb1y6yhzqwgwxk4wcj4d0wb5igb-initrd-bin-env/bin
initrd-linux> /usr/sbin -> /nix/store/4asc2mb1y6yhzqwgwxk4wcj4d0wb5igb-initrd-bin-env/sbin
initrd-linux> Warning: Couldn't satisfy dependency libcrypto.so.4 for "/nix/store/3v7kqg55hx4awhy5hdckniypbgqbqfmx-systemd-261/lib/systemd/systemd-tpm2-setup"
initrd-linux> Warning: Couldn't satisfy dependency libcrypto.so.4 for "/nix/store/3v7kqg55hx4awhy5hdckniypbgqbqfmx-systemd-261/lib/systemd/systemd-pcrextend"
initrd-linux> Warning: Couldn't satisfy dependency libbpf.so.0 for "/nix/store/3v7kqg55hx4awhy5hdckniypbgqbqfmx-systemd-261/lib/systemd/systemd-executor"
initrd-linux> Warning: Couldn't satisfy dependency libcrypt.so.1 for "/nix/store/3v7kqg55hx4awhy5hdckniypbgqbqfmx-systemd-261/lib/systemd/libsystemd-shared-261.so"
initrd-linux> Warning: Couldn't satisfy dependency libcrypt.so.1.1 for "/nix/store/3v7kqg55hx4awhy5hdckniypbgqbqfmx-systemd-261/lib/systemd/libsystemd-shared-261.so"
initrd-linux> Warning: Couldn't satisfy dependency libbpf.so.0 for "/nix/store/3v7kqg55hx4awhy5hdckniypbgqbqfmx-systemd-261/lib/systemd/libsystemd-core-261.so"
initrd-linux> Warning: Couldn't satisfy dependency libcrypto.so.4 for "/nix/store/3v7kqg55hx4awhy5hdckniypbgqbqfmx-systemd-261/bin/resolvectl"
initrd-linux> Warning: Couldn't satisfy dependency libcrypto.so.4 for "/nix/store/3v7kqg55hx4awhy5hdckniypbgqbqfmx-systemd-261/bin/.systemd-cryptsetup-wrapped"
initrd-linux> Warning: Couldn't satisfy dependency libcrypto.so.4 for "/nix/store/3v7kqg55hx4awhy5hdckniypbgqbqfmx-systemd-261/bin/bootctl"
initrd-linux> Warning: Couldn't satisfy dependency libcrypto.so.4 for "/nix/store/3v7kqg55hx4awhy5hdckniypbgqbqfmx-systemd-261/bin/systemd-repart"
initrd-linux> Warning: Couldn't satisfy dependency libcrypto.so.4 for "/nix/store/3v7kqg55hx4awhy5hdckniypbgqbqfmx-systemd-261/bin/homectl"
nixos-system-nixos> structuredAttrs is enabled
closure-info> structuredAttrs is enabled
nixos-disko-images> structuredAttrs is enabled
nixos-disko-images> Formatting '/nix/store/c2s01f0rxnczp7vs072r1jfp127wdj78-nixos-disko-images/main.raw', fmt=raw size=2147483648
nixos-disko-images> [2026-07-20T19:09:33Z WARN  virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)
nixos-disko-images> [2026-07-20T19:09:33Z WARN  virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether
nixos-disko-images> [2026-07-20T19:09:33Z WARN  virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)
nixos-disko-images> [2026-07-20T19:09:33Z WARN  virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether
nixos-disko-images> [2026-07-20T19:09:33Z INFO  virtiofsd] Waiting for vhost-user socket connection...
nixos-disko-images> [2026-07-20T19:09:33Z INFO  virtiofsd] Waiting for vhost-user socket connection...
nixos-disko-images> [2026-07-20T19:09:34Z INFO  virtiofsd] Client connected, servicing requests
nixos-disko-images> [2026-07-20T19:09:34Z INFO  virtiofsd] Client connected, servicing requests
nixos-disko-images> 
error: interrupted by the user
Logs from my own (similar) host configuration
Formatting '/home/bert-proesmans/nix-encore/main.raw', fmt=raw size=2147483648
[2026-07-20T17:47:13Z WARN  virtiofsd::limits] Failure when trying to set the limit to 1000000, the hard limit (524288) of open file descriptors is used instead.
[2026-07-20T17:47:13Z WARN  virtiofsd::limits] Failure when trying to set the limit to 1000000, the hard limit (524288) of open file descriptors is used instead.
[2026-07-20T17:47:13Z WARN  virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)
[2026-07-20T17:47:13Z WARN  virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether
[2026-07-20T17:47:13Z WARN  virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)
[2026-07-20T17:47:13Z WARN  virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether
[2026-07-20T17:47:13Z INFO  virtiofsd] Waiting for vhost-user socket connection...
[2026-07-20T17:47:13Z INFO  virtiofsd] Waiting for vhost-user socket connection...
[2026-07-20T17:47:13Z INFO  virtiofsd] Client connected, servicing requests
[2026-07-20T17:47:13Z INFO  virtiofsd] Client connected, servicing requests
loading kernel modules...
<Manual qemu termination here>
QEMU 11.0.2 monitor - type 'help' for more information
(qemu) q
[2026-07-20T17:48:04Z INFO  virtiofsd] Client disconnected, shutting down
[2026-07-20T17:48:04Z INFO  virtiofsd] Client disconnected, shutting down
Virtual machine didn't produce an exit code.

@Diffumist

Copy link
Copy Markdown

In my own scripts I see "loading kernel modules..." printed on the console. That text is not visible in the test output.

Could you provide the configuration or script that reproduces it?

I can't reproduce this here.

@Bert-Proesmans

Copy link
Copy Markdown
Contributor

My dev-stack is windows + hyper-v + nested virtualisation.
I did some more testing and disabling KVM kernel modules on the host-VM makes the command work for me. That narrows my root cause down to KVM.
This PR is unlikely linked to my issue with KVM. Not sure yet how I should proceed but I'll take this elsewhere.

nix build --extra-experimental-features "flakes nix-command" github:AlexLov/disko/fix-vmTools-overried#checks.x86_64-linux.make-disk-image -L --show-trace --override-input nixpkgs "github:/NixOS/nixpkgs/241313f4e8e508cb9b13278c2b0fa25b9ca27163" 2>&1 | tee out.log

Below the relevant log output, it explicitly mentions falling back to TCG backend instead of KVM. The command finished and produced a main.raw file successfully.

nixos-disko-images> Formatting '/nix/store/c2s01f0rxnczp7vs072r1jfp127wdj78-nixos-disko-images/main.raw', fmt=raw size=2147483648
nixos-disko-images> [2026-07-21T08:09:05Z WARN  virtiofsd::passthrough] Filesystem mounted on "/nix/store" (mount ID: 1295): Filesystem does not support file handles, falling back to O_PATH FDs
nixos-disko-images> [2026-07-21T08:09:05Z WARN  virtiofsd::passthrough] Failed to generate a file handle for the root node, disabling file handles altogether
nixos-disko-images> [2026-07-21T08:09:05Z WARN  virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)
nixos-disko-images> [2026-07-21T08:09:05Z WARN  virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether
nixos-disko-images> [2026-07-21T08:09:05Z INFO  virtiofsd] Waiting for vhost-user socket connection...
nixos-disko-images> [2026-07-21T08:09:05Z INFO  virtiofsd] Waiting for vhost-user socket connection...
nixos-disko-images> [2026-07-21T08:09:05Z INFO  virtiofsd] Client connected, servicing requests
nixos-disko-images> [2026-07-21T08:09:05Z INFO  virtiofsd] Client connected, servicing requests
<..HERE..>
nixos-disko-images> qemu-system-x86_64: Could not access KVM kernel module: No such file or directory
nixos-disko-images> qemu-system-x86_64: failed to initialize kvm: No such file or directory
nixos-disko-images> qemu-system-x86_64: falling back to tcg
<..HERE..>
nixos-disko-images>
nixos-disko-images>
nixos-disko-images>

<snip>
$ ls -la result/
total 1205820
dr-xr-xr-x 2 root root           60 Jan  1  1970 .
drwxrwxr-t 1 root nixbld     128560 Jul 21 08:11 ..
-r--r--r-- 1 root root   2147483648 Jan  1  1970 main.raw

kid added a commit to kid/home-ops that referenced this pull request Aug 8, 2026
Adds modules/hosts/test-vm.nix, a throwaway host for smoke-testing the
k3s/system-containerd/zfs-snapshotter wiring via
`nix build .#nixosConfigurations.test-vm.config.system.build.vmWithDisko`
without needing real hardware.

Booting it surfaced two real bugs:

- disko's lib/make-disk-image.nix passes an aggregated kernel-modules tree
  as vmTools' `kernel` arg, which broke against a nixpkgs vmTools API
  change (kernel/kernelModules split). Pin disko to nix-community/disko#1277
  (unmerged fix) until it lands upstream.
- k3s-containerd.nix's cni.bin_dir override was right, but dropping the
  transfer plugin's unpack_config as "redundant" wasn't: `ctr images pull
  --snapshotter zfs` failed with "no unpack platforms defined" without an
  explicit entry — the CRI snapshotter default doesn't extend to the
  transfer plugin's unpacker. Restored it.

Also makes zfs-disk-single.nix's boot.zfs.* settings mkDefault (disko's
vmVariantWithDisko needs to override devNodes/forceImportAll for qemu,
where /dev/disk/by-id is empty) and re-states persist/cache neededForBoot
under virtualisation.fileSystems, since vmWithDisko rebuilds fileSystems
from scratch and doesn't inherit the host-level overrides.

Verified live: booted test-vm, confirmed containerd.service active on the
right socket, both local/containerd(/snapshotter) zfs datasets mount
correctly, a zfs-snapshotter image pull succeeds, and k3s registers a
Node object reporting containerd://2.3.1 as its runtime (NotReady is
expected — no CNI in this isolated VM, Cilium ships via GitOps).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants