Skip to content

e2e: add volume expansion tests for NVMe-oF#6058

Draft
nixpanic wants to merge 3 commits intoceph:develfrom
nixpanic:nvmeof/e2e/volume-expand
Draft

e2e: add volume expansion tests for NVMe-oF#6058
nixpanic wants to merge 3 commits intoceph:develfrom
nixpanic:nvmeof/e2e/volume-expand

Conversation

@nixpanic
Copy link
Member

@nixpanic nixpanic commented Feb 13, 2026 •
edited
Loading

NVMe-oF supports Volume Expansion, but there was no e2e test for it yet.

In order to test this, the following steps are done:

  • PR includes update to Ceph Tentacle (will cause failure for CephFS)
  • NVMe-F e2e tests are run (job)
  • Ceph Tentacle update is removed from this PR
  • Ready for review. No e2e needed, as NVMe-oF only run on Ceph Tentacle or newer

Depends-on: #6085 #6124 #6163

Show available bot commands

These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

  • /retest ci/centos/<job-name>: retest the <job-name> after unrelated
    failure (please report the failure too!)

@nixpanic nixpanic added ci/skip/multi-arch-build skip building on multiple architectures component/nvme-of Issues and PRs related to NVMe-oF. labels Feb 13, 2026
@nixpanic
Copy link
Member Author

nixpanic commented Feb 13, 2026

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

@mergify mergify bot added the component/testing Additional test cases or CI work label Feb 13, 2026
@nixpanic nixpanic force-pushed the nvmeof/e2e/volume-expand branch from 71b83d3 to 3f651a0 Compare February 13, 2026 12:17
@nixpanic
Copy link
Member Author

nixpanic commented Feb 13, 2026

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

@nixpanic nixpanic force-pushed the nvmeof/e2e/volume-expand branch from 3f651a0 to f3faa3c Compare February 13, 2026 13:16
@nixpanic
Copy link
Member Author

nixpanic commented Feb 13, 2026

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

@nixpanic
Copy link
Member Author

nixpanic commented Feb 13, 2026

/test ci/centos/mini-e2e/k8s-1.35/rbd

@nixpanic nixpanic force-pushed the nvmeof/e2e/volume-expand branch from f3faa3c to 070ac74 Compare February 13, 2026 16:14
@nixpanic
Copy link
Member Author

nixpanic commented Feb 13, 2026

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

@nixpanic
Copy link
Member Author

nixpanic commented Feb 13, 2026

/test ci/centos/mini-e2e/k8s-1.35/rbd

@nixpanic nixpanic force-pushed the nvmeof/e2e/volume-expand branch from 070ac74 to 77b03fb Compare February 13, 2026 16:45
@nixpanic nixpanic added the ci/skip/e2e skip running e2e CI jobs label Feb 13, 2026
@nixpanic nixpanic requested a review from a team February 13, 2026 16:45
@nixpanic nixpanic marked this pull request as ready for review February 13, 2026 16:45
@nixpanic nixpanic requested a review from gadididi February 13, 2026 16:46
gadididi
gadididi previously approved these changes Feb 16, 2026
View reviewed changes
Copy link
Contributor

@gadididi gadididi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Madhu-1
Madhu-1 reviewed Feb 16, 2026
View reviewed changes
@nixpanic nixpanic force-pushed the nvmeof/e2e/volume-expand branch from 77b03fb to 1fa5ae4 Compare February 16, 2026 10:48
@nixpanic nixpanic removed the ci/skip/e2e skip running e2e CI jobs label Feb 16, 2026
@nixpanic
Copy link
Member Author

nixpanic commented Feb 16, 2026

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

@mergify mergify bot dismissed gadididi’s stale review February 16, 2026 10:48

Pull request has been modified.

@nixpanic
Copy link
Member Author

nixpanic commented Feb 16, 2026

Deleting the NVMe-oF volume failed. A small bug has been introduced with the DH-CHAP feature:

  I0216 11:04:08.599664       1 utils.go:350] ID: 16 Req-ID: 0001-0024-9011eb60-3b65-4891-8d1f-846868ee065d-0000000000000007-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93 GRPC call: /csi.v1.Controller/DeleteVolume
  I0216 11:04:08.599742       1 utils.go:351] ID: 16 Req-ID: 0001-0024-9011eb60-3b65-4891-8d1f-846868ee065d-0000000000000007-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93 GRPC request: {"secrets":"***stripped***","volume_id":"0001-0024-9011eb60-3b65-4891-8d1f-846868ee065d-0000000000000007-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93"}
  I0216 11:04:08.601453       1 omap.go:89] ID: 16 Req-ID: 0001-0024-9011eb60-3b65-4891-8d1f-846868ee065d-0000000000000007-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93 got omap values: (pool="nvmeofpool", namespace="", name="csi.volume.4ad3ec9f-c916-4b67-ade3-d8d5e09caa93"): map[csi.imageid:124e7c5e59c0 csi.imagename:csi-vol-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93 csi.volname:pvc-e73a51b1-fbc8-4191-bd57-e774ae1a7000 csi.volume.owner:nvmeof-4145] 
  I0216 11:04:08.691558       1 controllerserver.go:211] ID: 16 Req-ID: 0001-0024-9011eb60-3b65-4891-8d1f-846868ee065d-0000000000000007-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93 No NVMe-oF metadata found, skipping NVMe-oF cleanup: metadata not found: failed to get .rbd.nvmeof.dhchapMode: RBD image not found: rbd: ret=-2, No such file or directory
  I0216 11:04:08.692639       1 omap.go:89] ID: 16 Req-ID: 0001-0024-9011eb60-3b65-4891-8d1f-846868ee065d-0000000000000007-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93 got omap values: (pool="nvmeofpool", namespace="", name="csi.volume.4ad3ec9f-c916-4b67-ade3-d8d5e09caa93"): map[csi.imageid:124e7c5e59c0 csi.imagename:csi-vol-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93 csi.volname:pvc-e73a51b1-fbc8-4191-bd57-e774ae1a7000 csi.volume.owner:nvmeof-4145] 
  E0216 11:04:08.731204       1 controllerserver.go:1091] ID: 16 Req-ID: 0001-0024-9011eb60-3b65-4891-8d1f-846868ee065d-0000000000000007-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93 rbd nvmeofpool/csi-getNVMeoFMetadatavol-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93 is still being used
  E0216 11:04:08.731268       1 utils.go:355] ID: 16 Req-ID: 0001-0024-9011eb60-3b65-4891-8d1f-846868ee065d-0000000000000007-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93 GRPC error: rpc error: code = Aborted desc = rbd csi-vol-4ad3ec9f-c916-4b67-ade3-d8d5e09caa93 is still being used

Because getNVMeoFMetadata returns an error, the RBD-image is not removed from the gateway, so it is really still in use.

@gadididi has reviewed the issue and will provide a fix for it.

@gadididi gadididi force-pushed the nvmeof/e2e/volume-expand branch from 7e238fb to a48627a Compare February 16, 2026 13:08
@gadididi
Copy link
Contributor

gadididi commented Feb 16, 2026

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

@nixpanic nixpanic marked this pull request as draft February 16, 2026 16:29
@gadididi
Copy link
Contributor

gadididi commented Feb 17, 2026

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

1 similar comment
@gadididi
Copy link
Contributor

gadididi commented Feb 17, 2026

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

@gadididi
Copy link
Contributor

gadididi commented Feb 17, 2026

I opened fix PR for the dh-chap (#6085) because that bug in not belong only to this PR
and I think it is better this fix will be merge ASAP because it critical issue (on deleting the volume)

@mergify
Copy link
Contributor

mergify bot commented Feb 19, 2026

This pull request now has conflicts with the target branch. Could you please resolve conflicts and force push the corrected changes? 🙏

@gadididi
Copy link
Contributor

gadididi commented Feb 19, 2026

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

@nixpanic nixpanic force-pushed the nvmeof/e2e/volume-expand branch from 13b5ce6 to 2d3fe45 Compare February 20, 2026 16:49
@nixpanic
Copy link
Member Author

nixpanic commented Feb 20, 2026

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

1 similar comment
@gadididi
Copy link
Contributor

gadididi commented Feb 22, 2026

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

@gadididi gadididi force-pushed the nvmeof/e2e/volume-expand branch from 2d3fe45 to 57867c7 Compare February 22, 2026 08:22
@gadididi
Copy link
Contributor

gadididi commented Feb 22, 2026

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

@gadididi gadididi force-pushed the nvmeof/e2e/volume-expand branch from 57867c7 to 76cf4f6 Compare February 22, 2026 08:34
@gadididi
Copy link
Contributor

gadididi commented Feb 22, 2026

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

This was referenced Mar 8, 2026
Merged
Closed
@gadididi gadididi force-pushed the nvmeof/e2e/volume-expand branch from 76cf4f6 to e479e02 Compare March 9, 2026 14:25
@gadididi
Copy link
Contributor

gadididi commented Mar 9, 2026

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

@gadididi
Copy link
Contributor

gadididi commented Mar 9, 2026

I removed the rescanning and the nvme downgrade.
not this PR has only the origin commits (beside the disconnection commits)
I rebased the disconnection commits, for testing.

@gadididi
Copy link
Contributor

gadididi commented Mar 9, 2026

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

@nixpanic
Copy link
Member Author

nixpanic commented Mar 10, 2026

@Mergifyio rebase

Cleanup things a little more. ci/centos/mini-e2e/k8s-1.35/nvmeof passed earlier.

@mergify
Copy link
Contributor

mergify bot commented Mar 10, 2026

rebase

✅ Branch has been successfully rebased

@ceph-csi-bot ceph-csi-bot force-pushed the nvmeof/e2e/volume-expand branch from e479e02 to cc957a2 Compare March 10, 2026 12:13
@nixpanic
Copy link
Member Author

nixpanic commented Mar 10, 2026

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

@mergify
Copy link
Contributor

mergify bot commented Mar 16, 2026

This pull request now has conflicts with the target branch. Could you please resolve conflicts and force push the corrected changes? 🙏

@gadididi gadididi force-pushed the nvmeof/e2e/volume-expand branch from cc957a2 to 8c8d840 Compare March 18, 2026 13:38
@gadididi
Copy link
Contributor

gadididi commented Mar 18, 2026

https://github.com/Mergifyio rebase

@mergify
Copy link
Contributor

mergify bot commented Mar 18, 2026

rebase

☑️ Nothing to do, the required conditions are not met

Details
  • -conflict [📌 rebase requirement]
  • -closed [📌 rebase requirement]
  • queue-position = -1 [📌 rebase requirement]
  • any of:
    • #commits-behind > 0 [📌 rebase requirement]
    • -linear-history [📌 rebase requirement]

nixpanic and others added 3 commits March 18, 2026 16:05
@nixpanic @gadididi
e2e: add volume expansion tests for NVMe-oF
cbfc1dd 
Tests for filesystem expansion and block-mode expansion are added. To
make the `resizePVCAndValidateSize()` helper a little more flexible, it
has been adapted to get a filename or PVC-object as 1st parameter.

Signed-off-by: Niels de Vos <ndevos@ibm.com>
@nixpanic @gadididi
ci: let Rook deploy a Ceph Tentacle cluster
b84b47e 
Signed-off-by: Niels de Vos <ndevos@ibm.com>
@gadididi
e2e: include the NVMe-oF gateway logs after a failure
f48d018 
add gw logs for e2e test log file.

Signed-off-by: gadi-didi <gadi.didi@ibm.com>
@gadididi gadididi force-pushed the nvmeof/e2e/volume-expand branch from 8c8d840 to f48d018 Compare March 18, 2026 14:06
@gadididi
Copy link
Contributor

gadididi commented Mar 18, 2026

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

@gadididi
Copy link
Contributor

gadididi commented Mar 18, 2026

rebased to devel.
@nixpanic , maybe we can open this PR for reviewing?
Or do you want to add (I can too) more e2e tests here?

@nixpanic
Copy link
Member Author

nixpanic commented Mar 19, 2026

rebased to devel. @nixpanic , maybe we can open this PR for reviewing? Or do you want to add (I can too) more e2e tests here?

We can, but the update to a Tentacle cluster would need to be dropped, as that currently has issues with CephFS. Tentacle v20.2.1 should fix that, but it isn't released yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Madhu-1 Madhu-1 Madhu-1 left review comments

@gadididi gadididi gadididi left review comments

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

ci/skip/multi-arch-build skip building on multiple architectures component/nvme-of Issues and PRs related to NVMe-oF. component/testing Additional test cases or CI work

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@nixpanic @gadididi @Rakshith-R @Madhu-1