Prevent NVMe-oF boot devices from being removed by disconnect-all

[johnmeneghini]

On systems that boot from SAN over NVMe-TCP or NVMe-FC, it is quite easy to make the system inoperable by issuing a 'nvme disconnect-all". For instance, after issuing the disconnect on a system which boots over NVMe-FC, the console reports the following:

[  103.827323] block nvme1n1: no available path - failing I/O 
[  103.827337] nvme1n1p4: writeback error on inode 67250623, offset 90112, sector 59489176 
[  103.827342] block nvme1n1: no available path - failing I/O 
[  103.827346] nvme1n1p4: writeback error on inode 67250623, offset 94208, sector 59489184 
[  103.832144] XFS (nvme1n1p2): Block device removal (0x20) detected at fs_bdev_mark_dead+0x43/0x60 (fs/xfs/xfs_super.c:1179).  Shutting down filesystem. 
[  103.832152] XFS (nvme1n1p2): Please unmount the filesystem and rectify the problem(s) 
[  103.832200] block nvme1n1: no available path - failing I/O 
[  103.832204] XFS (nvme1n1p4): log I/O error -5 
[  103.832206] XFS (nvme1n1p4): Filesystem has been shut down due to log error (0x2). 
[  103.832207] XFS (nvme1n1p4): Please unmount the filesystem and rectify the problem(s). 

[johnmeneghini]

Adding @martin-gpy @tbzatek for more information. This issue was reported to Red Hat and I think this is an upstream feature request.

[igaw]

The same problem exists with any FS mounted from an 'attached' storage. As far I am aware the general consensus here is, just don't do it. ip down, ifconfig etc. don't prevent you to shutdown an networking interface which is serving a mounted FS.

I am not convinced that adding even more logic to the ill defined 'connect-all' and 'disconnect-all' command is not going to make things better.

Anyway, I assume you are also attending the ALPSS conference; we can discuss this it there.

[tbzatek]

There are numerous occassions when you can shoot yourselves in the foot. Preventing that is often a battle with a wind. This bugreport has been sitting downstream for a while yet it was treated as low priority and seemingly not really viable for spending much time on it.

One idea was to somehow mark the nbftX network interfaces as private in NetworkManager, but after some other related discussions and the general upstream tendency to avoid network interface renaming this idea didn't make much sense after all.

Another idea was to use nvme-cli application strings to tag nbft-initiated connections, so they're excluded from 'disconnect-all' and such. No attempt to test this in practice was ever made.

Cc: @mwilck

[hreinecke]

I thought that's why we did introduced the '--context' options, precisely to identify which application/context created a particular connection.
What we can / might do here is to treat 'nvme disconnect-all' to mean 'disconnect all devices with no context set' (which would cover all manually created devices). And then to have a --force option to disconnect all device irrespective of the context.

[martin-belanger]

I thought that's why we did introduced the '--context' options, precisely to identify which application/context created a particular connection.

Unfortunately, the context is an attribute of the root object (i.e. struct nvme_root::application), which does not persist between invocations of nvme-cli. NBFT boot devices are created by one invocation of nvme-cli, while disconnet-all is performed by another invocation. And the context gets lost between them.

For this whole thing to work, the context would need to be saved somewhere (sysfs maybe, but how exactly?) and be an attribute of the controller so that we can tell which application created each controller.

[johnmeneghini]

I really need martin-gpy to reply to this. This request comes from NetApp.

See: https://issues.redhat.com/browse/RHEL-41068

[igaw]

From the RH bug report:

Consider "iscsiadm --mode node --logoutall=all", does that do the same thing on iSCSI?

And this is exactly my point. We have added way too much policy magic (local configuration) into connect-all and now we start to get the same discussions with disconnect-all. I am not convinced this is the right way to go any further. We need to figure out a better way to deal with this.

[johnmeneghini]

On 9/25/25 12:16 PM, Daniel Wagner wrote: *igaw* left a comment (linux-nvme/nvme-cli#2913) <#2913 (comment)> From the RH bug report: Consider "iscsiadm --mode node --logoutall=all", does that do the same thing on iSCSI?

And the answer to this question, from the iscsi maintainer, was: No, if the record is marked startup==onboot then logoutall never touches it

And this is exactly my point. We have added way too much policy magic (local configuration) into |connect-all| and now we start to get the same discussions with |disconnect-all|. I am not convinced this is the right way to go any further. We need to figure out a better way to deal with this.

I'm not disagreeing with you... disconnect-all is a mess. I'm just reporting what many customers and partners (not just NetApp) are saying. As NVMe-oF gets adopted, and more and more people use it, invariably what happens is people compare it to iSCSI and FCP. This is another one of those areas where NVMe/TCP is different from iSCSI... and people don't like it! So if you can't shoot yourself in the foot by mistakenly disconnecting your boot device with iSCSI or FCP, then people are going to expect the same from NVMe/TCP+FCP. We will need to address these expections, somehow. /John

[igaw]

Understood, though it like keep the discussion on technical ground. My point we have added horrible code in the connect logic and there seems no end in adding even more. It reached the point where it can't be maintained properly. Testing is almost impossible. This is the reasons why I push back.

Yes, this needs to be addressed and it might be that connect-all and disconnect-all have to go.

I need to familiarize myself with the iscsi setup. Anyway, is my assumption correct, that there is a daemon which holds state?

[mwilck]

Consider "iscsiadm --mode node --logoutall=all", does that do the same thing on iSCSI?
And the answer to this question, from the iscsi maintainer, was:

No, if the record is marked startup==onboot then logoutall never touches it

Well, "marked startup==onboot" requires that a node record exists in the iSCSI database, and that it has been assigned this property. The equivalent of this would be a matching entry in config.json with a respective property (not sure if there's a "startup mode" property in config.json, but could surely be added if not).

But neither for iSCSI nor NVMe do such entries necessarily exist ¹. iSCSI boot systems will run iscsiadm -m fw -l to set up sessions, which does not create node records. The user (or system software) needs to run iscsiadm -m discovery -t fw manually to creatte those. Only then can the user modify parameters like startup. Same for NVMe, connections initiated from the NBFT don't necessarily have entries in config.json.

However, for iSCSI I can confirm that iscsiadm -m node --logoutall=all does not disconnect sessions for which no node records exist (aka iBFT sessions). This is a distinctive difference wrt nvme disconnect-all, which disconnects everything.

We could make the semantics of disconnect-all match those of iscsiadm by disconnecting only those connections that have entries in config.json (and startup mode other than onboot). In practice, that would turn disconnect-all into a no-op on many systems, because more often than not users don't bother to have config.json entries for their connections.

Footnotes

1. And it's debatable wheter such entries should exist, because of the dynamic / variable nature of the iBFT and NBFT entries. ↩

[martin-belanger]

Following the discussion in this issue, I spent some time thinking more broadly about the underlying problem.

The issue appears to be less about disconnect-all specifically and more about the lack of administrative ownership and coordination semantics between NVMe-oF orchestration domains (manual nvme-cli usage, NBFT, nvme-stas, static configuration, future orchestrators, etc.).

Rather than adding additional special-case policy logic to connect-all or disconnect-all, this proposal explores a possible userspace-based coordination model built around:

• stable controller identities (TIDs - Transport ID)
• a shared libnvme-managed ownership registry
• ownership-aware controller operations
• explicit orchestration-domain separation

The proposal intentionally avoids kernel changes and instead focuses on shared userspace coordination infrastructure.

Several aspects of this model already exist today inside nvme-stas, which helped validate that the problem is real and not purely theoretical.

I am posting the document mainly as a discussion starter to gather feedback from the nvme-cli/libnvme maintainers and the broader Linux NVMe community.

Proposal: nvme_controller_ownership_registry_proposal.md

Thanks.

[igaw]

Thanks for the input. I’ve had the chance to discuss this topic with a few folks at LSFMM, including Lennart. Based on all the feedback, I think the overall solution should look something like this:

• Introduce /var/nvme with some sort of state files (the exact format is up for discussion), along with a file-locking protocol. This would allow nvme-cli to run completely without any daemons (nvme-discoverd, see the next point). Furthermore, if nvme-discoverd crashes or starts later in the boot process, it can read the current state.
• Link ownership will be covered by a protocol based on @martin-belanger’s proposal. I don’t have a strong opinion here, though I think we should introduce versioning for the files from the beginning so that we have the flexibility to extend or change them later.
• Introduce nvme-discoverd, which takes over discovery duties for the different transport protocols. This replaces the udev rule for FC. The daemon reads uevents directly. nvme-discoverd will create unit files that essentially run nvme connect $ARGS.
• For systems without systemd, nvme-discoverd will create shell scripts that run nvme connect $ARGS.
• nvme-discoverd will also learn how to handle TCP/RDMA discovery (see the various protocols to find a CDC).
• If we need some sort of IPC to communicate with nvme-discoverd, it will use the varlink protocol from the systemd project. This allows OS images to be built without D-Bus. If there is a real need for a D-Bus interface, that could also be added. However, I’d really like to avoid adding D-Bus as a default dependency. There is also ongoing work to add BUS1 to the kernel, which would make varlink even more useful.
• The root filesystem is handled outside of systemd to avoid circular dependencies. These connections are therefore added as /var/nvme state, which avoids all sorts of difficult heuristics to figure out which connection is owned by whom. For the shutdown sequence, it is the responsibility of the boot environment to handle the correct ordering.
• For block devices added while systemd is running, we will use the special blockdev@.target target. This will make the shutdown sequence work correctly.

With this in place, nvme disconnect-all is able to disconnect connections which have been added by the user with nvme connect-all or nvme connect. Obviously, we are also adding --force to overwrite this behavior.

WDYT?

[martin-belanger]

Thanks, this is really interesting and I think the overall direction makes a lot of sense.

A few questions/comments came to mind while reading through this.

1. Regarding /var/nvme vs /run/nvme

In the proposal I initially converged toward /run/nvme because I was mostly thinking about runtime orchestration ownership/state and there didn’t seem to be anything that fundamentally needed to survive a reboot.

From your description it sounds like /var/nvme would also serve as persistent orchestration state for nvme-discoverd (restart recovery, late startup, boot continuity, etc.). Is that the intended distinction?

2. BUS1

The BUS1 mention was a bit of a surprise to me. I thought that project had effectively died years ago, although I just noticed there seems to be renewed activity around it recently:
https://www.phoronix.com/news/BUS1-Linux-2026

I always thought BUS1 was a very interesting idea and wished I could have used it in several projects. That said, given its history and previous pushback, how likely do you think it is to actually materialize? Are you envisioning this mostly as a future optimization/opportunity, or as something the design may eventually depend on?

3. TCP/RDMA discovery

Regarding:

“nvme-discoverd will also learn how to handle TCP/RDMA discovery (see the various protocols to find a CDC)”

Are you referring to mDNS/DNS-SD discovery here?

That’s one of the core functions currently implemented in nvme-stas. Today stas leverages Avahi for this, using its D-Bus interface to discover CDC advertisements in a zeroconf fashion.

If the intention is to support mDNS discovery, are you thinking of leveraging Avahi, or implementing the discovery stack directly inside nvme-discoverd?

4. Varlink vs D-Bus

Varlink sounds like a very reasonable direction, especially since it can work during early boot before the D-Bus daemon is available.

That said, if nvme-discoverd ends up integrating with Avahi for mDNS/DNS-SD discovery, then some amount of D-Bus interaction may still be unavoidable anyway.

Overall though, I really like the direction this is heading.