cephfs: support MDS pinning via ControllerModifyVolume by ismaelpuerto · Pull Request #6390 · ceph/ceph-csi

ismaelpuerto · 2026-07-08T09:50:49Z

Expose the RPC_MODIFY_VOLUME capability on the CephFS controller and map the mutable VolumeAttributesClass parameters mds-pin-export, mds-pin-distributed and mds-pin-random to "ceph fs subvolume pin". This lets administrators change the MDS distribution policy of an existing subvolume at runtime without recreating the volume.

The three pin parameters are mutually exclusive and validated before being applied. A new PinVolume method on the SubVolumeClient wraps go-ceph's PinSubVolume; since go-ceph does not expose a subvolume-group aware variant, the pin is applied against Ceph's default subvolume group, which is documented as a known limitation.

Adds unit tests for the parameter validation, an example VolumeAttributesClass and documentation in the CephFS deployment guide.

Describe what this PR does

Provide some context for the reviewer

Is there anything that requires special attention

Do you have any questions?

Is the change backward compatible?

Are there concerns around backward compatibility?

Provide any external context for the change, if any.

For example:

Kubernetes links that explain why the change is required
CSI spec related changes/catch-up that necessitates this patch
golang related practices that necessitates this change

Related issues

Fixes: #6378
Depends-on: #6458

Future concerns

List items that are not part of the PR and do not impact it's
functionality, but are work items that can be taken up subsequently.

Checklist:

Commit Message Formatting: Commit titles and messages follow
guidelines in the developer
guide.
Reviewed the developer guide on Submitting a Pull
Request
Pending release
notes
updated with breaking and/or notable changes for the next major release.
Documentation has been updated, if necessary.
Unit tests have been added, if necessary.
Integration tests have been added, if necessary.

Show available bot commands

These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

/retest ci/centos/<job-name>: retest the <job-name> after unrelated
failure (please report the failure too!)

nixpanic

Thanks for the PR!

I had a first view over this, and it looks reasonable to me.

The only thing that is missing, is a test case under e2e/cephfs.go. See how e2e/nvmeof.go uses a VolumeAttributesClass to apply a change

Address review feedback on PR ceph#6390: the mutable VolumeAttributesClass parameters must also be applied during CreateVolume, not only through ControllerModifyVolume. Following the pattern in the NFS driver, the pin logic is moved into an applyMutableParameters helper so that locking and volume resolution happen only once, and both CreateVolume and ControllerModifyVolume reuse it. Also add an e2e test under e2e/cephfs.go that creates a PVC referencing a VolumeAttributesClass with an mds-pin-export rank, verifies the class is applied at creation time, and then updates the PVC to a second class to exercise the ControllerModifyVolume path. Assisted-by: goose <noreply@block.xyz> Signed-off-by: Ismael Puerto Freire <ismaelpf@inditex.com>

nixpanic · 2026-07-10T07:29:27Z

/test ci/centos/mini-e2e/k8s-1.37/cephfs

nixpanic · 2026-07-10T09:40:06Z

The new e2e test "create a PVC with a VolumeAttributesClass for MDS pinning and modify it" failed. It looks like an issue related to a missing RBAC:

  I0710 08:36:29.354794   80190 pvc.go:86] PVC csi-cephfs-pvc Event: ProvisioningFailed - volumeattributesclasses.storage.k8s.io "e2e-cephfs-3728-cephfs-vac1" is forbidden: User "system:serviceaccount:cephcsi-e2e-c2e72d32:cephfs-csi-provisioner" cannot get resource "volumeattributesclasses" in API group "storage.k8s.io" at the cluster scope

logs

In deploy/cephfs/kubernetes/csi-provisioner-rbac.yaml the permissions for VolumeAttributesClasses needs to be added. This is how RBD has them:

ceph-csi/deploy/rbd/kubernetes/csi-provisioner-rbac.yaml

Lines 84 to 86 in 7e9580b

    
           - apiGroups: ["storage.k8s.io"] 
        
             resources: ["volumeattributesclasses"] 
        
             verbs: ["get", "list", "watch"]

Commit ef8b07a added the permissions to the Helm Chart for RBD. This needs to be done for the CephFS Helm Chart as well.

ismaelpuerto · 2026-07-15T05:42:03Z

/test ci/centos/mini-e2e/k8s-1.37/cephfs

ismaelpuerto · 2026-07-15T12:58:20Z

/test ci/centos/mini-e2e/k8s-1.37/cephfs

nixpanic

Thanks, this looks good to me.

The CI job failed on an unrelated NFS test, but the new CephFS test passed. (logs)

nixpanic · 2026-07-24T08:53:16Z

@Mergifyio rebase

mergify · 2026-07-24T08:53:27Z

rebase

🛑 The pull request rule doesn't match anymore

Details

This action has been cancelled.

Address review feedback on PR ceph#6390: the mutable VolumeAttributesClass parameters must also be applied during CreateVolume, not only through ControllerModifyVolume. Following the pattern in the NFS driver, the pin logic is moved into an applyMutableParameters helper so that locking and volume resolution happen only once, and both CreateVolume and ControllerModifyVolume reuse it. Also add an e2e test under e2e/cephfs.go that creates a PVC referencing a VolumeAttributesClass with an mds-pin-export rank, verifies the class is applied at creation time, and then updates the PVC to a second class to exercise the ControllerModifyVolume path. Assisted-by: goose <noreply@block.xyz> Signed-off-by: Ismael Puerto Freire <ismaelpf@inditex.com>

nixpanic · 2026-07-24T08:53:58Z

/test ci/centos/mini-e2e/k8s-1.37/cephfs

Madhu-1 · 2026-07-28T04:37:15Z

+// go-ceph's typed PinSubVolume helper does not accept a subvolume group, so it
+// would always target Ceph's default group and fail for subvolumes created in
+// a non-default group (as ceph-csi does). To support any group, the "fs


@ismaelpuerto can we please fix this in go-ceph and open an issue to adopt to go-ceph API once its implemented?

Thanks for the review, I just create the issue : ceph/go-ceph#1293

iPraveenParihar · 2026-07-28T05:07:37Z

+	if len(req.GetMutableParameters()) != 0 {
+		if err = applyMutableParameters(ctx, volClient, vID.FsSubvolName,
+			req.GetMutableParameters()); err != nil {
+			return nil, err


if applyMutableParameters fail then we might need to clean the Subvolume orelse it will be orphaned.
You need to call PurgeVolume here.

iPraveenParihar · 2026-07-28T05:14:20Z

+	for _, key := range mdsPinParams {
+		value, ok := params[key]
+		if !ok {
+			continue


This will silently ignore unknown mutable parameters. CSI spec recommends InvalidArgument for unsupported parameters.

iPraveenParihar · 2026-07-28T05:23:19Z

+			if err != nil {
+				logAndFail("failed to get bound PV: %v", err)
+			}
+			if pv.Spec.VolumeAttributesClassName == nil ||


This only verifies the frontend part (validating the VAC reference in PV object), but doesn't validate on the backend if the pinning is applied on subvolume. Do you think that is possible to add coverage for this?

Thanks for the review. I looked into validating the pin on the backend and wanted to share what I found, as it affects how we can assert it in e2e.

1. ceph fs subvolume info does not expose the pin. Tested on Ceph 19 (Squid)

2. The pin is not visible from the application pod. ceph fs subvolume pin applies ceph.dir.pin on the subvolume directory (/volumes/csi/csi-vol-<id>/), but CSI mounts the data subdirectory (.../<uuid>/) as the volume root. So getfattr -n ceph.dir.pin from the pod always reports -1 even after a successful pin.

3. The only way to assert the pin is to mount the whole filesystem and read ceph.dir.pin on /volumes/csi/csi-vol-<id>/. Doable in the toolbox but adds mount setup/teardown and is more fragile than the rest of the CephFS e2e suite.

Would you prefer (a) a toolbox-side full-fs mount + getfattr verification on the subvolume directory, or keeping the current front-end assertion and documenting the backend-verification limitation?

@iPraveenParihar , do you have an opinion on this?

I guess it would be nice to check the xattr, but mounting the subvolume directory (parent directory of the user visible volume) is a little tricky...

Yea, lets keep it the current front-end assertion for now to not complex this PR. Let create a tracker to address backend assertion?

Sounds good, thanks! I've opened #6460 to track the backend assertion

Pull request has been modified.

nixpanic · 2026-08-03T08:19:20Z

Thanks again @ismaelpuerto! The code looks good to me. Can you please squash the last commit into the one that adds the new functionality (1st commit)? That makes the history of the commits cleaner and easier to follow.

Address review feedback on PR ceph#6390: the mutable VolumeAttributesClass parameters must also be applied during CreateVolume, not only through ControllerModifyVolume. Following the pattern in the NFS driver, the pin logic is moved into an applyMutableParameters helper so that locking and volume resolution happen only once, and both CreateVolume and ControllerModifyVolume reuse it. Also add an e2e test under e2e/cephfs.go that creates a PVC referencing a VolumeAttributesClass with an mds-pin-export rank, verifies the class is applied at creation time, and then updates the PVC to a second class to exercise the ControllerModifyVolume path. Assisted-by: goose <noreply@block.xyz> Signed-off-by: Ismael Puerto Freire <ismaelpf@inditex.com>

iPraveenParihar

LGTM
Thanks!

Rakshith-R · 2026-08-05T11:33:29Z

/test ci/centos/mini-e2e/k8s-1.36/cephfs

mergify · 2026-08-11T06:20:39Z

Merge Queue Status

✅ Entered queue — 2026-08-11 10:25 UTC · Rule: default · triggered by @Rakshith-R with the @mergifyio queue command
✅ Checks passed · in-place
✅ Merged — 2026-08-11 14:28 UTC · at 0df28bcab2412cb6150045a5fcf3a513d4355b29 · rebase

This pull request spent 4 hours 2 minutes 37 seconds in the queue, including 4 hours 1 minute 24 seconds running CI.

Required conditions to merge

depends-on = ceph/ceph-csi#6458 [⛓️ cleanup: validate unknown parameters in ControllerModifyVolume #6458]
- cephfs: support MDS pinning via ControllerModifyVolume #6390
github-review-approved [🛡 GitHub branch protection]
- cephfs: support MDS pinning via ControllerModifyVolume #6390

iPraveenParihar · 2026-08-11T06:54:58Z

/retest ci/centos/mini-e2e/k8s-1.36

iPraveenParihar · 2026-08-11T06:55:05Z

/retest ci/centos/mini-e2e/k8s-1.35

iPraveenParihar · 2026-08-11T06:55:09Z

/retest ci/centos/mini-e2e/k8s-1.34

mergify · 2026-08-11T10:25:42Z

Deprecation notice: This pull request comes from a fork and was queued with update_method=rebase and update_bot_account impersonation. This capability will be removed on July 1, 2026. After this date, the merge queue will no longer be able to rebase fork pull requests with this configuration. To avoid disruption, switch to update_method=merge in your queue rule.

Expose the RPC_MODIFY_VOLUME capability on the CephFS controller and map the mutable VolumeAttributesClass parameters mds-pin-export, mds-pin-distributed and mds-pin-random to "ceph fs subvolume pin". This lets administrators change the MDS distribution policy of an existing subvolume at runtime without recreating the volume. The three pin parameters are mutually exclusive and validated before being applied. A new PinVolume method on the SubVolumeClient wraps go-ceph's PinSubVolume; since go-ceph does not expose a subvolume-group aware variant, the pin is applied against Ceph's default subvolume group, which is documented as a known limitation. Adds unit tests for the parameter validation, an example VolumeAttributesClass and documentation in the CephFS deployment guide. Fixes: ceph#6378 Assisted-by: goose <noreply@block.xyz> Signed-off-by: Ismael Puerto Freire <ismaelpf@inditex.com>

Address review feedback on PR ceph#6390: the mutable VolumeAttributesClass parameters must also be applied during CreateVolume, not only through ControllerModifyVolume. Following the pattern in the NFS driver, the pin logic is moved into an applyMutableParameters helper so that locking and volume resolution happen only once, and both CreateVolume and ControllerModifyVolume reuse it. Also add an e2e test under e2e/cephfs.go that creates a PVC referencing a VolumeAttributesClass with an mds-pin-export rank, verifies the class is applied at creation time, and then updates the PVC to a second class to exercise the ControllerModifyVolume path. Assisted-by: goose <noreply@block.xyz> Signed-off-by: Ismael Puerto Freire <ismaelpf@inditex.com>

The CephFS provisioner needs to read VolumeAttributesClass objects to support MDS pinning via ControllerModifyVolume. Without this the e2e test fails with "cannot get resource volumeattributesclasses in API group storage.k8s.io at the cluster scope". Add the required RBAC rule to both the static deployment manifest and the CephFS Helm chart, matching what RBD already has. Assisted-by: goose <noreply@block.xyz> Signed-off-by: Ismael Puerto Freire <ismaelpf@inditex.com>

The "fs subvolume pin" manager command was issued through go-ceph's PinSubVolume helper, which does not accept a subvolume group. Ceph then looked the subvolume up in the default group and returned ENOENT for subvolumes created in a non-default group, which is the case for ceph-csi (and the e2e test uses the "e2e" group): failed to pin subvolume csi-vol-... in fs myfs: rados: ret=-2, No such file or directory: "subvolume 'csi-vol-...' does not exist" Issue the "fs subvolume pin" command directly with group_name set, via a new MgrCommand passthrough on ClusterConnection, so pinning works for any subvolume group. Assisted-by: goose <noreply@block.xyz> Signed-off-by: Ismael Puerto Freire <ismaelpf@inditex.com>

ceph-csi-bot · 2026-08-11T10:27:50Z

/test ci/centos/upgrade-tests-cephfs

ceph-csi-bot · 2026-08-11T10:27:50Z

/test ci/centos/upgrade-tests-rbd

ceph-csi-bot · 2026-08-11T10:27:51Z

/test ci/centos/k8s-e2e-external-storage/1.34

ceph-csi-bot · 2026-08-11T10:27:52Z

/test ci/centos/mini-e2e-helm/k8s-1.34

ceph-csi-bot · 2026-08-11T10:27:52Z

/test ci/centos/k8s-e2e-external-storage/1.35

ceph-csi-bot · 2026-08-11T10:27:52Z

/test ci/centos/mini-e2e/k8s-1.34

ceph-csi-bot · 2026-08-11T10:27:53Z

/test ci/centos/mini-e2e-helm/k8s-1.35

ceph-csi-bot · 2026-08-11T10:27:54Z

/test ci/centos/mini-e2e/k8s-1.35

ceph-csi-bot · 2026-08-11T10:27:57Z

/test ci/centos/k8s-e2e-external-storage/1.36

ceph-csi-bot · 2026-08-11T10:27:58Z

/test ci/centos/mini-e2e-helm/k8s-1.36

ceph-csi-bot · 2026-08-11T10:27:59Z

/test ci/centos/mini-e2e/k8s-1.36

Address review feedback on PR #6390: the mutable VolumeAttributesClass parameters must also be applied during CreateVolume, not only through ControllerModifyVolume. Following the pattern in the NFS driver, the pin logic is moved into an applyMutableParameters helper so that locking and volume resolution happen only once, and both CreateVolume and ControllerModifyVolume reuse it. Also add an e2e test under e2e/cephfs.go that creates a PVC referencing a VolumeAttributesClass with an mds-pin-export rank, verifies the class is applied at creation time, and then updates the PVC to a second class to exercise the ControllerModifyVolume path. Assisted-by: goose <noreply@block.xyz> Signed-off-by: Ismael Puerto Freire <ismaelpf@inditex.com>

mergify Bot added the component/cephfs Issues related to CephFS label Jul 8, 2026

nixpanic reviewed Jul 9, 2026

View reviewed changes

Comment thread internal/cephfs/controllerserver.go

ismaelpuerto force-pushed the pinning-subv branch from 4334ae7 to 3639d32 Compare July 10, 2026 06:28

nixpanic previously approved these changes Jul 15, 2026

View reviewed changes

nixpanic requested a review from a team July 15, 2026 14:59

ceph-csi-bot force-pushed the pinning-subv branch from 0063e9e to b8b5c07 Compare July 24, 2026 08:53

Madhu-1 previously approved these changes Jul 28, 2026

View reviewed changes

iPraveenParihar requested changes Jul 28, 2026

View reviewed changes

nixpanic requested a review from iPraveenParihar August 3, 2026 08:18

ismaelpuerto force-pushed the pinning-subv branch from 903daef to 47aa45f Compare August 5, 2026 09:32

iPraveenParihar approved these changes Aug 5, 2026

View reviewed changes

Rakshith-R requested review from Madhu-1 and nixpanic August 5, 2026 11:32

ismaelpuerto mentioned this pull request Aug 6, 2026

cephfs: e2e should assert MDS pinning is applied on the backend #6460

Open

nixpanic approved these changes Aug 6, 2026

View reviewed changes

mergify Bot added the queued label Aug 11, 2026

ismaelpuerto added 4 commits August 11, 2026 10:26

ceph-csi-bot force-pushed the pinning-subv branch from 2c0aa6a to 0df28bc Compare August 11, 2026 10:26

mergify Bot added the ok-to-test Label to trigger E2E tests label Aug 11, 2026

ceph-csi-bot added ci/in-progress/e2e This label acts like a guard and prevents Mergify from adding the `ok-to-test` label again. and removed ok-to-test Label to trigger E2E tests labels Aug 11, 2026

mergify Bot removed the ci/in-progress/e2e This label acts like a guard and prevents Mergify from adding the `ok-to-test` label again. label Aug 11, 2026

iPraveenParihar mentioned this pull request Aug 11, 2026

e2e: add acceptance minikube quick e2e smoke gate #6437

Merged

5 tasks

mergify Bot merged commit db4dcdc into ceph:devel Aug 11, 2026
43 checks passed

mergify Bot removed the queued label Aug 11, 2026

Uh oh!

Conversation

ismaelpuerto commented Jul 8, 2026 • edited by nixpanic Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe what this PR does

Is there anything that requires special attention

Related issues

Future concerns

Uh oh!

nixpanic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nixpanic commented Jul 10, 2026

Uh oh!

nixpanic commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ismaelpuerto commented Jul 15, 2026

Uh oh!

ismaelpuerto commented Jul 15, 2026

Uh oh!

nixpanic left a comment

Choose a reason for hiding this comment

Uh oh!

nixpanic commented Jul 24, 2026

Uh oh!

mergify Bot commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🛑 The pull request rule doesn't match anymore

Uh oh!

nixpanic commented Jul 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iPraveenParihar Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nixpanic commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iPraveenParihar left a comment

Choose a reason for hiding this comment

Uh oh!

Rakshith-R commented Aug 5, 2026

Uh oh!

mergify Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Uh oh!

iPraveenParihar commented Aug 11, 2026

Uh oh!

iPraveenParihar commented Aug 11, 2026

Uh oh!

iPraveenParihar commented Aug 11, 2026

Uh oh!

mergify Bot commented Aug 11, 2026

Uh oh!

ceph-csi-bot commented Aug 11, 2026

Uh oh!

ceph-csi-bot commented Aug 11, 2026

Uh oh!

ismaelpuerto commented Jul 8, 2026 •

edited by nixpanic

Loading

nixpanic commented Jul 10, 2026 •

edited

Loading

mergify Bot commented Jul 24, 2026 •

edited

Loading

iPraveenParihar Jul 28, 2026 •

edited

Loading

nixpanic commented Aug 3, 2026 •

edited

Loading

mergify Bot commented Aug 11, 2026 •

edited

Loading