Skip to content

ci: use Rook 1.19.2 and Ceph Tentacle#5672

Draft
nixpanic wants to merge 7 commits intoceph:develfrom
nixpanic:testing/ceph-tentacle
Draft

ci: use Rook 1.19.2 and Ceph Tentacle#5672
nixpanic wants to merge 7 commits intoceph:develfrom
nixpanic:testing/ceph-tentacle

Conversation

@nixpanic
Copy link
Member

@nixpanic nixpanic commented Oct 15, 2025

Recent versions of Rook use Ceph-CSI Operator by default. There is no need to have Ceph-CSI deployed by Rook, as our CI deploys it from the PR that is being tested.

Rook v1.18.4 works well with Ceph Squid (v19). But for NVMe-oF testing we need a Ceph Tentacle (v20) cluster.


Show available bot commands

These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

  • /retest ci/centos/<job-name>: retest the <job-name> after unrelated
    failure (please report the failure too!)

@nixpanic
Copy link
Member Author

/test ci/centos/mini-e2e/k8s-1.34

@mergify mergify bot added the component/testing Additional test cases or CI work label Oct 15, 2025
nixpanic added a commit to nixpanic/ceph-csi that referenced this pull request Oct 15, 2025
The devel branch will use a newer version of Rook and Ceph in the near
future. That requires the images to be mirrored in the CI registry.

See-also: ceph#5672
Signed-off-by: Niels de Vos <ndevos@ibm.com>
@nixpanic nixpanic added the ci/skip/multi-arch-build skip building on multiple architectures label Oct 15, 2025
@nixpanic
Copy link
Member Author

nixpanic commented Oct 15, 2025

@nixpanic
Copy link
Member Author

/test ci/centos/mini-e2e/k8s-1.34

@nixpanic
Copy link
Member Author

/test ci/centos/mini-e2e/k8s-1.34

1 similar comment
@nixpanic
Copy link
Member Author

/test ci/centos/mini-e2e/k8s-1.34

@nixpanic nixpanic changed the title ci: use Rook 1.18.4 ci: use Rook 1.18.4 and Ceph Tentacle Oct 15, 2025
mergify bot pushed a commit that referenced this pull request Oct 16, 2025
The devel branch will use a newer version of Rook and Ceph in the near
future. That requires the images to be mirrored in the CI registry.

See-also: #5672
Signed-off-by: Niels de Vos <ndevos@ibm.com>
@nixpanic nixpanic force-pushed the testing/ceph-tentacle branch from c2dabc9 to 03040f1 Compare October 16, 2025 09:07
@nixpanic
Copy link
Member Author

/test ci/centos/mini-e2e/k8s-1.34

@nixpanic
Copy link
Member Author

/test ci/centos/mini-e2e/k8s-1.33

@nixpanic nixpanic force-pushed the testing/ceph-tentacle branch from 03040f1 to 870215d Compare October 16, 2025 09:54
@nixpanic
Copy link
Member Author

/test ci/centos/mini-e2e/k8s-1.34

@nixpanic nixpanic force-pushed the testing/ceph-tentacle branch from 870215d to 025e47a Compare October 16, 2025 10:13
@nixpanic
Copy link
Member Author

/test ci/centos/mini-e2e/k8s-1.34

@nixpanic nixpanic force-pushed the testing/ceph-tentacle branch from 025e47a to a40f06c Compare October 16, 2025 11:19
@nixpanic
Copy link
Member Author

/test ci/centos/mini-e2e/k8s-1.34

@nixpanic nixpanic force-pushed the testing/ceph-tentacle branch from a40f06c to 9b9119c Compare October 16, 2025 11:59
@nixpanic
Copy link
Member Author

/test ci/centos/mini-e2e/k8s-1.34

1 similar comment
@nixpanic
Copy link
Member Author

/test ci/centos/mini-e2e/k8s-1.34

@nixpanic nixpanic force-pushed the testing/ceph-tentacle branch from 960e97e to 4b7516f Compare October 16, 2025 14:44
@nixpanic
Copy link
Member Author

/test ci/centos/mini-e2e/k8s-1.34

@nixpanic nixpanic force-pushed the testing/ceph-tentacle branch from 4b7516f to 48a4306 Compare October 16, 2025 14:59
@nixpanic
Copy link
Member Author

/test ci/centos/mini-e2e/k8s-1.34

1 similar comment
@nixpanic
Copy link
Member Author

/test ci/centos/mini-e2e/k8s-1.34

@nixpanic nixpanic force-pushed the testing/ceph-tentacle branch from b919ccd to 96abede Compare October 16, 2025 15:49
@nixpanic nixpanic force-pushed the testing/ceph-tentacle branch from 703779e to 1ad3fec Compare January 8, 2026 11:03
@nixpanic
Copy link
Member Author

nixpanic commented Jan 8, 2026

/test ci/centos/mini-e2e-operator/k8s-1.34/cephfs

@nixpanic nixpanic force-pushed the testing/ceph-tentacle branch from 1ad3fec to 249c286 Compare January 8, 2026 12:12
@nixpanic
Copy link
Member Author

nixpanic commented Jan 8, 2026

/test ci/centos/mini-e2e-operator/k8s-1.34/cephfs

@nixpanic nixpanic force-pushed the testing/ceph-tentacle branch from 249c286 to 87165cb Compare January 8, 2026 12:41
@nixpanic
Copy link
Member Author

nixpanic commented Jan 8, 2026

/test ci/centos/mini-e2e-operator/k8s-1.34/cephfs

@bumarcell
Copy link

I believe support for Tentacle was added in Rook v1.18.8

@nixpanic
Copy link
Member Author

I believe support for Tentacle was added in Rook v1.18.8

It is not so much an issue with Rook, more with a regression in Tentacle with CephFS. We're waiting for a Tentacle release that includes ceph/ceph#66611.

@bumarcell
Copy link

Got it 👍
That said, is there a reason to prefer upgrading to v1.18.4 rather than 1.18.8 or 1.18.9?

@nixpanic
Copy link
Member Author

This PR has been open for a while already. I guess at the time it was opened, Rook 1.18.4 was the most recent release. When we update this PR, we definitely should update the Rook version to whatever is current. Thanks for the reminder!

@nixpanic nixpanic force-pushed the testing/ceph-tentacle branch from 87165cb to 18859aa Compare February 6, 2026 16:30
@nixpanic nixpanic changed the title ci: use Rook 1.18.4 and Ceph Tentacle ci: use Rook 1.19.1 and Ceph Tentacle Feb 6, 2026
@github-actions
Copy link

github-actions bot commented Mar 8, 2026

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale label Mar 8, 2026
@mergify
Copy link
Contributor

mergify bot commented Mar 8, 2026

This pull request now has conflicts with the target branch. Could you please resolve conflicts and force push the corrected changes? 🙏

@github-actions github-actions bot removed the stale label Mar 9, 2026
Recent versions of Rook use Ceph-CSI Operator by default. There is no
need to have Ceph-CSI deployed by Rook, as our CI deploys it from the PR
that is being tested.

Signed-off-by: Niels de Vos <ndevos@ibm.com>
Signed-off-by: Niels de Vos <ndevos@ibm.com>
Signed-off-by: Niels de Vos <ndevos@ibm.com>
With Ceph Tentacle creating a (the first?) NFS-export fails with:

```
Error EINVAL: Traceback (most recent call last):
    File "/usr/share/ceph/mgr/mgr_module.py", line 2010, in _handle_command
      return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
    File "/usr/share/ceph/mgr/mgr_module.py", line 534, in call
      return self.func(mgr, **kwargs)
    File "/usr/share/ceph/mgr/object_format.py", line 592, in _format_response
      robj = f(*args, **kwargs)
    File "/usr/share/ceph/mgr/nfs/module.py", line 45, in
_cmd_nfs_export_create_cephfs
      earmark_resolver = CephFSEarmarkResolver(self)
    File "/usr/share/ceph/mgr/mgr_util.py", line 356, in __init__
      self._cephfs_client = client or CephfsClient(mgr)
    File "/usr/share/ceph/mgr/mgr_util.py", line 318, in __init__
      self.connection_pool = CephfsConnectionPool(self.mgr)
    File "/usr/share/ceph/mgr/mgr_util.py", line 223, in __init__
      self.timer_task.start()
    File "/usr/lib64/python3.9/threading.py", line 899, in start
      _start_new_thread(self._bootstrap, ())
  RuntimeError: can't start new thread
```

It seems this can only happen when the CLI is used, that means go-ceph
failed to create the export in the first place. It would be useful to
log the go-ceph failure, so that improving is possible.

Signed-off-by: Niels de Vos <ndevos@ibm.com>
Signed-off-by: Niels de Vos <ndevos@ibm.com>
Signed-off-by: Niels de Vos <ndevos@ibm.com>
@nixpanic nixpanic force-pushed the testing/ceph-tentacle branch from 18859aa to ff1d259 Compare March 20, 2026 10:31
@nixpanic nixpanic changed the title ci: use Rook 1.19.1 and Ceph Tentacle ci: use Rook 1.19.2 and Ceph Tentacle Mar 20, 2026
nixpanic added a commit to nixpanic/ceph-csi that referenced this pull request Mar 20, 2026
Rook 1.19.2 has been releases a few weeks ago. Include it in the CI
registry to that we can run e2e with it.

See-also: ceph#5672
Signed-off-by: Niels de Vos <ndevos@ibm.com>
@nixpanic
Copy link
Member Author

/test ci/centos/mini-e2e-operator/k8s-1.35/rbd

Use `dd` to sync the data, do not wait for unmounting to flush
outstanding I/O (and ignore failed writes).

See-also: https://tracker.ceph.com/issues/73997?#note-27
Co-authored-by: Patrick Donnelly <pdonnell@ibm.com>
Signed-off-by: Niels de Vos <ndevos@ibm.com>
@nixpanic nixpanic force-pushed the testing/ceph-tentacle branch from ff1d259 to 96626bb Compare March 20, 2026 16:33
@nixpanic
Copy link
Member Author

/test ci/centos/mini-e2e-operator/k8s-1.35/rbd

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/skip/multi-arch-build skip building on multiple architectures component/testing Additional test cases or CI work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants