WIP: nvme-discoverd: initial daemon implementation by martin-belanger · Pull Request #3634 · linux-nvme/nvme-cli

martin-belanger · 2026-07-24T15:04:23Z

Summary

This is the initial implementation of nvme-discoverd, a small daemon that automatically connects NVMe-oF controllers discovered via NBFT, static config, or Discovery Log Pages, using systemd transient units (one per controller, no unit files written to disk). It's meant to eventually replace the 70-nvmf-autoconnect.rules/nvmf-connect@.service mechanism.

This is a work-in-progress submission, not ready for merge. I'm opening it now so you can start looking at the overall shape and design while I finish the remaining items below.

What's included

discoverd: add the daemon skeleton — CLI options, config knobs, meson wiring.
discoverd: track controllers and manage connections as systemd units — TID handling, in-memory cache, transient-unit management via D-Bus.
discoverd: react to events, fetch DLPs, and read the fabrics config (nvme-fabrics.conf) — udev monitoring, Discovery Log Page fetch, FC kickstart, the shared nvme-fabrics.conf as the connection source, and the exclusion-list/registry-ownership checks before connecting.
meson: split discoverd and legacy autoconnect into independent options — -Dnvme-discoverd/-Dnvmf-autoconnect, both enabled by default during this stabilization period.
discoverd: add the systemd service, config, and docs — unit file, discoverd.conf, README, man page.
discoverd: use ccan for linked lists and string helpers — shares the existing ccan static lib instead of hand-rolling lists/streq/ARRAY_SIZE.

Known gaps before this is review-ready

No automated tests yet.
nvme.spec.in doesn't reference the new binary/service/config yet.
Registry-ownership check is a courtesy check for intentional handoffs between orchestrators, not a race guard between two daemons managing the same controller.

Test plan

meson compile clean: default, -Dwerror=true, -Dfabrics=disabled
meson test: 81/83 (2 expected-fail unchanged)
make checkpatch-diff: clean
Live-tested against real nvmet dummy targets: discovery, connect, exclusion, and registry-ownership all behave as expected

igaw · 2026-07-27T12:42:49Z

All looks reasonable. Though, I am really struggling with the documentation again. It's really hard to read and understand. What about adding something like this here to the rules: linux-nvme/nvme-cli-ai#4 ?

The only thing which stands a bit out to me is the cache.c code. This doesn't seem to be a cache from the traditional point of view. As far I understand it, it reads from disk configuration and adds an API to query the loaded data, e.g. cache_is_desired. I think cache is just the wrong name for this API. Or I misunderstand it but in this case I really blame the documentation. I stopped reading it because it's just too complex to read.

igaw · 2026-07-27T12:43:43Z

And as discussed on matrix, I'd say we keep the defaults on 'off' and treat it as technology preview for the 3.0 release.

igaw · 2026-07-27T12:44:21Z

and without any configuration around:

# ./discoverd/nvme-discoverd
Failed to open (null): Bad address

martin-belanger · 2026-07-27T13:44:18Z

@igaw - Quick ask: discoverd is going through some refactoring right now since it uses a lot of the new "shared" code. It would help to conclude #3636 first, then I can properly address the #3634 issues.

igaw · 2026-07-27T14:20:04Z

@igaw - Quick ask: discoverd is going through some refactoring right now since it uses a lot of the new "shared" code. It would help to conclude #3636 first, then I can properly address the #3634 issues.

Sure thing. Let's get #3636 sorted out first. I am waiting for an update there.

dwsuse · 2026-07-27T14:28:26Z

After starting nvme-discoverd I see

$ systemctl -a | grep nvme-
  nvme-discoverd-14206fe13b52.service                                                                             loaded    active   exited    NVMe Discovery Controller nqn.2014-08.org.nvmexpress.discovery @ nn-0x201700a09890f5bf:pn-0x201800a09890f5bf:none
  nvme-discoverd-14eeec929051.service                                                                             loaded    active   exited    NVMe Discovery Controller nqn.2014-08.org.nvmexpress.discovery @ nn-0x200200a09890f5bf:pn-0x203700a09890f5bf:none
  nvme-discoverd-152e73862ab0.service                                                                             loaded    active   exited    NVMe Discovery Controller nqn.2014-08.org.nvmexpress.discovery @ nn-0x58ccf090c92006da:pn-0x58ccf099492906da:none
  nvme-discoverd-18ae8bfd5497.service                                                                             loaded    active   exited    NVMe Discovery Controller nqn.2014-08.org.nvmexpress.discovery @ nn-0x201700a09890f5bf:pn-0x201a00a09890f5bf:none
  nvme-discoverd-1f3a053dba1b.service                                                                             loaded    active   exited    NVMe Discovery Controller nqn.2014-08.org.nvmexpress.discovery @ nn-0x200200a09890f5bf:pn-0x203900a09890f5bf:none
  nvme-discoverd-2b737813a4e8.service                                                                             loaded    active   exited    NVMe Discovery Controller nqn.2014-08.org.nvmexpress.discovery @ nn-0x200200a09890f5bf:pn-0x203600a09890f5bf:none
  nvme-discoverd-2ca145afae18.service                                                                             loaded    active   exited    NVMe Discovery Controller nqn.2014-08.org.nvmexpress.discovery @ nn-0x201700a09890f5bf:pn-0x201b00a09890f5bf:none
[...]
 
$ systemctl status nvme-discoverd-14eeec929051.service
● nvme-discoverd-14eeec929051.service - NVMe Discovery Controller nqn.2014-08.org.nvmexpress.discovery @ nn-0x200200a09890f5bf:pn-0x203700a09890f5bf:none
     Loaded: loaded (/run/systemd/transient/nvme-discoverd-14eeec929051.service; transient)
  Transient: yes
     Active: active (exited) since Mon 2026-07-27 16:06:12 CEST; 16min ago
    Process: 80838 ExecStart=/usr/sbin/nvme connect --transport=fc --traddr=nn-0x200200a09890f5bf:pn-0x203700a09890f5bf --nqn=nqn.2014-08.org.nvmexpress.discovery --host-traddr=nn-0x20000024ff7fa449:pn-0x21000024ff7fa449 --idempotent --owner=discoverd --devid-file=/run/nvme/discoverd/units/nvme-discoverd-14eeec929051.devid (code=exited, status=0/>
    Process: 81070 ExecStartPost=/bin/sh -c DEV=$(cat /run/nvme/discoverd/units/nvme-discoverd-14eeec929051.devid 2>/dev/null) && mkdir -p /run/nvme/discoverd/controllers/$DEV && echo nvme-discoverd-14eeec929051.service > /run/nvme/discoverd/controllers/$DEV/unit (code=exited, status=0/SUCCESS)
   Main PID: 80838 (code=exited, status=0/SUCCESS)
        CPU: 37ms

Jul 27 16:05:45 dolin systemd[1]: Starting NVMe Discovery Controller nqn.2014-08.org.nvmexpress.discovery @ nn-0x200200a09890f5bf:pn-0x203700a09890f5bf:none...
Jul 27 16:06:12 dolin systemd[1]: Finished NVMe Discovery Controller nqn.2014-08.org.nvmexpress.discovery @ nn-0x200200a09890f5bf:pn-0x203700a09890f5bf:none.


$ nvme lists-subsys
nvme-subsys0 - NQN=nqn.2014-08.org.nvmexpress.discovery
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:62a4ab74-1e18-11f1-8bb7-6c1ff71ba506
\
 +- nvme0 fc nn-0x58ccf090c92006da:pn-0x58ccf099492806da live

nvme-subsys10 - NQN=nqn.2014-08.org.nvmexpress.discovery
                hostnqn=nqn.2014-08.org.nvmexpress:uuid:62a4ab74-1e18-11f1-8bb7-6c1ff71ba506
\
 +- nvme10 fc nn-0x58ccf090c92006da:pn-0x58ccf099492906da live

nvme-subsys7 - NQN=nqn.1992-08.com.netapp:sn.d646dc63336511e995cb00a0988fb732:discovery
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:62a4ab74-1e18-11f1-8bb7-6c1ff71ba506
\
 +- nvme11 fc nn-0x201700a09890f5bf:pn-0x201900a09890f5bf live
 +- nvme12 fc nn-0x201700a09890f5bf:pn-0x201800a09890f5bf live
 +- nvme13 fc nn-0x201700a09890f5bf:pn-0x201a00a09890f5bf live
 +- nvme14 fc nn-0x201700a09890f5bf:pn-0x201a00a09890f5bf live
 +- nvme16 fc nn-0x201700a09890f5bf:pn-0x201a00a09890f5bf live
 +- nvme17 fc nn-0x201700a09890f5bf:pn-0x201900a09890f5bf live
 +- nvme23 fc nn-0x201700a09890f5bf:pn-0x201a00a09890f5bf live
 +- nvme25 fc nn-0x201700a09890f5bf:pn-0x201b00a09890f5bf live
 +- nvme26 fc nn-0x201700a09890f5bf:pn-0x201b00a09890f5bf live
 +- nvme28 fc nn-0x201700a09890f5bf:pn-0x201800a09890f5bf live
 +- nvme29 fc nn-0x201700a09890f5bf:pn-0x201900a09890f5bf live
 +- nvme31 fc nn-0x201700a09890f5bf:pn-0x201800a09890f5bf live
 +- nvme32 fc nn-0x201700a09890f5bf:pn-0x201b00a09890f5bf live
 +- nvme7 fc nn-0x201700a09890f5bf:pn-0x201900a09890f5bf live
 +- nvme8 fc nn-0x201700a09890f5bf:pn-0x201800a09890f5bf live
 +- nvme9 fc nn-0x201700a09890f5bf:pn-0x201b00a09890f5bf live
[...]

$ ./nvme discover --transport=fc --traddr=nn-0x200200a09890f5bf:pn-0x203700a09890f5bf --nqn=nqn.2014-08.org.nvmexpress.discovery --host-traddr=nn-0x20000024ff7fa449:pn-0x21000024ff7fa449 --force

Discovery Log Number of Records 4, Generation counter 16
=====Discovery Log Entry 0======
trtype:  fc
adrfam:  fibre-channel
subtype: current discovery subsystem
treq:    not specified
portid:  5
trsvcid: none
subnqn:  nqn.1992-08.com.netapp:sn.44ede5aaa52511e8abb200a09890f5bf:discovery
traddr:  nn-0x200200a09890f5bf:pn-0x203800a09890f5bf
eflags:  explicit discovery connections, duplicate discovery information
=====Discovery Log Entry 1======
trtype:  fc
adrfam:  fibre-channel
subtype: current discovery subsystem
treq:    not specified
portid:  3
trsvcid: none
subnqn:  nqn.1992-08.com.netapp:sn.44ede5aaa52511e8abb200a09890f5bf:discovery
traddr:  nn-0x200200a09890f5bf:pn-0x203600a09890f5bf
eflags:  explicit discovery connections, duplicate discovery information
=====Discovery Log Entry 2======
trtype:  fc
adrfam:  fibre-channel
subtype: current discovery subsystem
treq:    not specified
portid:  4
trsvcid: none
subnqn:  nqn.1992-08.com.netapp:sn.44ede5aaa52511e8abb200a09890f5bf:discovery
traddr:  nn-0x200200a09890f5bf:pn-0x203700a09890f5bf
eflags:  explicit discovery connections, duplicate discovery information
=====Discovery Log Entry 3======
trtype:  fc
adrfam:  fibre-channel
subtype: current discovery subsystem
treq:    not specified
portid:  6
trsvcid: none
subnqn:  nqn.1992-08.com.netapp:sn.44ede5aaa52511e8abb200a09890f5bf:discovery
traddr:  nn-0x200200a09890f5bf:pn-0x203900a09890f5bf
eflags:  explicit discovery connections, duplicate discovery information

It seems many of those discovery controllers should not stay around because they don't set the NVMF_DISC_EFLAGS_EPCSD eflag. It seems we need to deal with PDC soon than later :)

MichaelRabek

I tested the daemon in the (NVMe/TCP boot POC)[https://github.com/timberland-sig/rh-linux-poc] QEMU environment and found the below described behaviour. I am detailing the steps to reproduce to make sure I didn't make a mistake in the build/installation process.

Steps to reproduce

Clone and set up (rh-linux-poc)[https://github.com/timberland-sig/rh-linux-poc] according to the README.md. Use make auto-start for the target-vm and select 2 boot attempts (dual path) to be put into the host-vm NBFT. Select Fedora 44 when asked to pick an OS.
Use make -C host-vm enter to connect to the VM that should be rebooted and running Fedora 44 over NVMe/TCP.
Clone this branch.
Configure with -Dnvme-discoverd=enabled, build and install
Use patchelf to patch in the correct libnvme3.so.1 path.
systemctl enable --now nvme-discoverd.service

Actual behaviour

# journalctl -eu nvme-discoverd
Jul 28 11:09:09 host-vm systemd[1]: Starting nvme-discoverd.service - NVMe-oF Discovery Daemon...
Jul 28 11:09:09 host-vm nvme-discoverd[4709]: Failed to open (null): Bad address
Jul 28 11:09:09 host-vm nvme-discoverd[4709]: cache_load_config() - loaded 0 DC(s), 0 IOC(s) from the fabrics config
Jul 28 11:09:09 host-vm nvme-discoverd[4709]: (tcp, 192.168.110.20, 4420, nqn.2014-08.org.nvmexpress:uuid:0c468c4d-a385-47e0-8299-6e95051277db, 192.168.110.30) | nvme1 - not desired, skipping
Jul 28 11:09:09 host-vm nvme-discoverd[4709]: (tcp, 192.168.101.20, 4420, nqn.2014-08.org.nvmexpress:uuid:0c468c4d-a385-47e0-8299-6e95051277db, 192.168.101.30) | nvme0 - not desired, skipping
Jul 28 11:09:09 host-vm systemd[1]: Started nvme-discoverd.service - NVMe-oF Discovery Daemon.

# nvme registry list
# nvme nbft show
/sys/firmware/acpi/tables/NBFT:

NBFT Subsystems:

Idx|NQN                                                                 |Trsp|Address       |SvcId|HFIs
---+--------------------------------------------------------------------+----+--------------+-----+----
1  |nqn.2014-08.org.nvmexpress:uuid:0c468c4d-a385-47e0-8299-6e95051277db|tcp |192.168.101.20|4420 |1   
2  |nqn.2014-08.org.nvmexpress:uuid:0c468c4d-a385-47e0-8299-6e95051277db|tcp |192.168.110.20|4420 |2   

NBFT HFIs:

Idx|Trsp|PCI Addr  |MAC Addr         |DHCP|IP Addr       |Mask|Gateway |DNS     
---+----+----------+-----------------+----+--------------+----+--------+--------
1  |tcp |0:0:5.0   |ea:eb:d3:58:89:58|no  |192.168.101.30|24  |0.0.0.0 |0.0.0.0 
2  |tcp |0:0:6.0   |ea:eb:d3:59:89:59|no  |192.168.110.30|24  |0.0.0.0 |0.0.0.0

# nvme list-subsys
nvme-subsys0 - NQN=nqn.2014-08.org.nvmexpress:uuid:0c468c4d-a385-47e0-8299-6e95051277db
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:f8131bac-cdef-4165-866b-5998c1e67890
\
 +- nvme0 tcp 192.168.101.20 live
 +- nvme1 tcp 192.168.110.20 live

Expected behaviour

I would expect the connections to be listed in the nvme registry with owner=nbft.
The NBFT connections should be in the daemon's desired set.

MichaelRabek · 2026-07-28T09:33:14Z

+ * Callbacks from the events layer to the main orchestrator.
+ * All callbacks are called from the event loop thread.
+ */
+struct events_cbacks {


nitpick: Could we rename this to struct events_callbacks? The full name with 3 extra letters added is not that long. I personally don't find the shortened version readable.

martin-belanger · 2026-07-28T11:05:20Z

@MichaelRabek - The issue you reported is the same that Daniel already reported (Failed to open (null): Bad address). There was a bug that prevented discoverd from reading the NBFT config. It's already fixed and I will push shortly.

martin-belanger · 2026-07-28T11:16:49Z

@MichaelRabek - Thanks for testing this against real NBFT boot

As mentioned earlier there was a bug reading the NBFT config. That also explains the second half of what you saw: since the NBFT table was never actually parsed, nbft_dcs/nbft_iocs stayed empty, so startup_audit()'s sysfs walk correctly-but-uselessly found nvme0/nvme1 already connected and logged them "not desired, skipping" -- they never matched anything in an empty desired set. With the fix, they'll be recognized as desired and NBFT-sourced, and discoverd will issue its normal --idempotent --owner=nbft connect against them -- a no-op on the actual connection, but it writes the registry entry, so they'll pick up owner=nbft even though discoverd didn't establish them itself.

On the empty nvme registry list at boot: I don't think that's a discoverd or connect-all bug. connect-all --nbft already defaults the owner to "nbft" before connecting anything, so if that's what ran at boot, ownership should have been recorded. Your steps don't show a reboot (or initramfs regen) after building/installing this branch, so the NBFT boot connections were most likely established by whatever nvme your image's initramfs had baked in from before nvme-cli 3.0. dracut's 95nvmf module inst_multiples the nvme/jq binaries into the image at generation time, so a live install doesn't reach early boot until the initramfs is rebuilt. Either way, it doesn't matter for the fix above -- discoverd takes ownership itself on startup regardless of what (if anything) boot recorded.

martin-belanger · 2026-07-28T11:21:11Z

Pushed a rebase onto current master plus a few fixes from this round of review:

Fixed cache_load_nbft() calling libnvmf_read_nbft() with a NULL filename -- it always failed and was silently swallowed as "no NBFT," which is what produced both the Failed to open (null): Bad address line and the "not desired, skipping" behavior Michael saw testing against real NBFT boot. Now uses libnvmf_nbft_read_files() to enumerate /sys/firmware/acpi/tables/NBFT*, same as fabrics.c and the nbft plugin.
Renamed cache.c/struct cache to inventory.c/struct inventory -- what it tracks isn't a cache in the traditional sense; the one part that is (the per-DC DLP data) is now its own dlp_cache field.
Renamed cback/cbacks to callback/callbacks throughout, per Michael's nitpick.
nvme-discoverd now defaults to disabled (technology preview for 3.0), matching what we discussed on matrix.

PDC (the NVMF_DISC_EFLAGS_EPCSD handling Daniel flagged) is a real gap but a bigger one -- coming in a follow-up commit.

A persistent daemon that connects the host's NVMe-oF controllers and keeps them connected: NBFT boot controllers, everything listed in the shared fabrics configuration, and whatever those Discovery Controllers' Discovery Log Pages (and FC kickstart) turn up along the way. Single-threaded (one sd_event loop), connect-only (never disconnects a live controller on its own initiative), and cooperative with other orchestrators via the ownership registry and exclusion list. Every connect goes through a systemd transient unit running nvme connect, never a direct /dev/nvme-fabrics write, since that ioctl blocks in D state until the kernel completes or times out the connection. Gated on a new -Dnvme-discoverd meson option, disabled by default (technology preview for 3.0). Signed-off-by: Martin Belanger <martin.belanger@dell.com>

Give the legacy udev-rule/systemd-unit NVMe-oF autoconnect mechanism its own meson option, enabled by default alongside nvme-discoverd. Neither excludes the other at install time -- switching a host over to discoverd is a deliberate systemctl operation for the administrator, not a build-time decision, until it's proven out in the field. Key provisioning and vendor-tuning udev rules stay unconditional; only the autoconnect-specific units, rules, dracut snippet, and NM dispatcher move under the new option. Signed-off-by: Martin Belanger <martin.belanger@dell.com>

MichaelRabek · 2026-07-28T13:46:01Z

@martin-belanger

On the empty nvme registry list at boot: I don't think that's a discoverd or connect-all bug. connect-all --nbft already defaults the owner to "nbft" before connecting anything, so if that's what ran at boot, ownership should have been recorded. Your steps don't show a reboot (or initramfs regen) after building/installing this branch, so the NBFT boot connections were most likely established by whatever nvme your image's initramfs had baked in from before nvme-cli 3.0. dracut's 95nvmf module inst_multiples the nvme/jq binaries into the image at generation time, so a live install doesn't reach early boot until the initramfs is rebuilt. Either way, it doesn't matter for the fix above -- discoverd takes ownership itself on startup regardless of what (if anything) boot recorded.

To clarify, I did not expect to see something in the nvme registry list right after boot. I merely used that as a proof that discoverd did not touch the registry. No, I did not re-build initramfs, but on purpose. I focused solely on discoverd.

martin-belanger · 2026-07-28T13:48:29Z

@MichaelRabek - BTW, you can use nvme list -v to see who owns each controller. There is a new "orchestrator" column, but only with the -v (verbose) option

Introduces man section 8 to the docs build -- the first daemon this project documents. Signed-off-by: Martin Belanger <martin.belanger@dell.com>

igaw · 2026-07-28T14:26:40Z

Thanks a lot!

MichaelRabek reviewed Jul 28, 2026

View reviewed changes

martin-belanger force-pushed the discoverd-v2 branch from 1f8a364 to 516e35d Compare July 28, 2026 11:17

Martin Belanger added 2 commits July 28, 2026 09:26

doc: add nvme-discoverd(8) and the design README

2e3f39d

Introduces man section 8 to the docs build -- the first daemon this project documents. Signed-off-by: Martin Belanger <martin.belanger@dell.com>

martin-belanger force-pushed the discoverd-v2 branch from 516e35d to 2e3f39d Compare July 28, 2026 14:03

igaw merged commit 73da15f into linux-nvme:master Jul 28, 2026
32 of 33 checks passed

martin-belanger deleted the discoverd-v2 branch July 28, 2026 14:53

Uh oh!

Conversation

martin-belanger commented Jul 24, 2026

Summary

What's included

Known gaps before this is review-ready

Test plan

Uh oh!

igaw commented Jul 27, 2026

Uh oh!

igaw commented Jul 27, 2026

Uh oh!

igaw commented Jul 27, 2026

Uh oh!

martin-belanger commented Jul 27, 2026

Uh oh!

igaw commented Jul 27, 2026

Uh oh!

dwsuse commented Jul 27, 2026

Uh oh!

MichaelRabek left a comment

Choose a reason for hiding this comment

Steps to reproduce

Actual behaviour

Expected behaviour

Uh oh!

MichaelRabek Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

martin-belanger Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

martin-belanger commented Jul 28, 2026

Uh oh!

martin-belanger commented Jul 28, 2026

Uh oh!

martin-belanger commented Jul 28, 2026

Uh oh!

MichaelRabek commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

martin-belanger commented Jul 28, 2026

Uh oh!

Uh oh!

igaw commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

MichaelRabek commented Jul 28, 2026 •

edited

Loading