Basic NVME support built-in

[vazub]

NVME devices are more and more prevalent, especially in the desktop segment. It would make sense to consider them first-class citizen, without depending on initramfs. This would allow to drop initramfs altogether for even more cases, without growing the kernel size too much.

[ptr1337]

Related discussion:

https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/issues/126

[vazub]

Related discussion:

https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/issues/126

I would argue, that the discussion is not specifically applicable to CachyOS case, due to a different target audience defined for the project. For a desktop-first OS, the reported original reasons for building NVME support as a module, are not actually relevant. One is for VM-specific usage, another is for a 3rd-party manufacturer that can't seem to be bothered by using the default Linux driver.

The otherwise pros of providing a generic built-in support for the prevalent type of drives in modern desktop systems would outweigh the mentioned cons, in my opinion.

[ptr1337]

I just added it as context in general for the discussion to consider also their views :)

This was not directly a "no, we do not do it"

[damachine]

Improvements are always a good thing. We all love it and want it!
But :P
Excuse me if I raise a few points of criticism as a long-time Arch user.

• The NVMe controllers that have problems with the standard driver (FS#68021, FS#74271) don't suddenly work better just because the operating system has a different name.
• Possible anyone with a problematic controller is affected by CachyOS just the same.
• An integrated module cannot be disabled.
• Gamers also use VMs (for testing, for older games, etc.).
• Desktop users also have hardware problems.

How significant(~x) would the performance increase be during boot? Is it worth it?

The I/O runtime performance makes no difference.

• The NVMe driver is the same code.
• Whether integrated or as a module – the data transfer is identical.
• Latency, throughput, IOPS: all the same.

I think Arch took the right approach by rejecting this. as long as the points are not solved.
Of course, this is just my opinion.
Peace

[vazub]

@damachine

* An integrated module cannot be disabled.

This is simply a false and misleading statement. Easily done at kernel boot with an added parameter:

initcall_blacklist=nvme_init

Problem solved.

With this, your other argumentation points are simply irrelevant.

[damachine]

@vazub

Thanks for the response! However, I have to respectfully disagree.

initcall_blacklist=nvme_init is not equivalent to proper module blacklisting, as was already discussed in the original Arch issue #126:

The core issue:

With built-in drivers (=y):

• modprobe.blacklist doesn't work at all
• initcall_blacklist=nvme_init is the only option, but:
  • Users reported it doesn't work reliably
  • Requires knowing exact function names
  • And initcall_blacklist is Designed as a debugging tool, not a permanent solution

  https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
  initcall_blacklist=  [KNL] Do not execute a comma-separated list of
                        initcall functions.  Useful for debugging built-in
                        modules and initcalls.
  

I appreciate CachyOS pushing boundaries for performance, but forcing a built-in driver that breaks standard troubleshooting methods seems like a regression in system flexibility for a very minor gain.

[aidanharris]

Why were these issues not considered a problem for SATA drives which are built-in? Even if you build it in I think you should still be able to unbind the device like this:
https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Binding_vfio-pci_via_device_ID

For example:

# cd /sys/module/nvme/drivers/pci:nvme
# ls -al
total 0
drwxr-xr-x  2 root root    0 Mar  2 04:38 .
drwxr-xr-x 28 root root    0 Mar  2 04:38 ..
lrwxrwxrwx  1 root root    0 Mar  2 04:38 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:01:00.0
lrwxrwxrwx  1 root root    0 Mar  2 04:38 0000:04:00.0 -> ../../../../devices/pci0000:00/0000:00:01.2/0000:02:00.0/0000:03:01.0/0000:04:00.0
lrwxrwxrwx  1 root root    0 Mar  2 04:38 0000:0f:00.0 -> ../../../../devices/pci0000:00/0000:00:03.2/0000:0f:00.0
--w-------  1 root root 4096 Mar  2 04:38 bind
lrwxrwxrwx  1 root root    0 Mar  2 04:38 module -> ../../../../module/nvme
--w-------  1 root root 4096 Mar  2 04:38 new_id
--w-------  1 root root 4096 Mar  2 04:42 remove_id
--w-------  1 root root 4096 Mar  2 04:38 uevent
--w-------  1 root root 4096 Mar  2 04:38 unbind
# nvme="0000:01:00.0"
# /bin/echo 0000:01:00.0 > unbind
# modprobe vfio-pci
# nvme_vd="$(cat /sys/bus/pci/devices/$nvme/vendor) $(cat /sys/bus/pci/devices/$nvme/device)"
# echo $nvme_vd
0x144d 0xa80a
# /bin/echo 0000:01:00.0 > unbind
# At this point the device won't show in lsblk anymore
# echo "$nvme_vd" > /sys/bus/pci/drivers/vfio-pci/new_id
# ls -al /sys/bus/pci/drivers/vfio-pci/
total 0
drwxr-xr-x  2 root root    0 Mar  2 05:01 .
drwxr-xr-x 29 root root    0 Mar  2 04:53 ..
lrwxrwxrwx  1 root root    0 Mar  2 05:01 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:01:00.0
--w-------  1 root root 4096 Mar  2 04:56 bind
lrwxrwxrwx  1 root root    0 Mar  2 04:54 module -> ../../../../module/vfio_pci
--w-------  1 root root 4096 Mar  2 05:01 new_id
--w-------  1 root root 4096 Mar  2 04:54 remove_id
--w-------  1 root root 4096 Mar  2 04:54 uevent
--w-------  1 root root 4096 Mar  2 04:54 unbind

This can be done as a hook in the initramfs or with a Systemd unit.

[daedaevibin]

One possible approach would be to build in the core components required
to boot on the vast majority of modern desktop systems. This would allow
CachyOS to natively boot without relying on an initramfs in many cases,
while keeping the kernel size impact relatively small.

For example, building in components such as:

NVME
EXT4
BTRFS
XFS
USB HID
EFI
PCI

These represent the most common storage, filesystem, and input paths
required for a typical desktop boot process.

A compromise already exists for NVMe specifically, since it is effectively
the modern equivalent of SATA storage. Enabling the core NVMe support
directly in the kernel would cover most systems without significantly
reducing flexibility:

CONFIG_NVME_CORE=y
CONFIG_BLK_DEV_NVME=y
CONFIG_NVME_PCI=y