rbd: use volumeattributesclass feature implement rbd volume qos by YiteGu · Pull Request #6160 · ceph/ceph-csi

YiteGu · 2026-03-06T09:34:06Z

Describe what this PR does

The solution using storageclass to implement RBD volume QoS has been cancelled because the storageclass parameters cannot be modified. The volumeattributesclass solution is more flexible because it supports mutable parameters.
Add the new parameter: volumeAttributesClassName when defining the PVC. For example:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vac-pvc-1
  namespace: rook-ceph
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi
  storageClassName: rook-ceph-block
  volumeMode: Block
  volumeAttributesClassName: silver

Implemented the ControllerModifyVolume interface. Users can upgrade their QoS by modifying the volumeAttributesClassName parameter of an existing PVC.

ref: https://kubernetes.io/docs/concepts/storage/volume-attributes-classes/

Is there anything that requires special attention

Do you have any questions?

Is the change backward compatible?

Are there concerns around backward compatibility?

Provide any external context for the change, if any.

For example:

Kubernetes links that explain why the change is required
CSI spec related changes/catch-up that necessitates this patch
golang related practices that necessitates this change

Related issues

Mention any github issues relevant to this PR. Adding below line
will help to auto close the issue once the PR is merged.

Fixes: #issue_number

Future concerns

List items that are not part of the PR and do not impact it's
functionality, but are work items that can be taken up subsequently.

Checklist:

Commit Message Formatting: Commit titles and messages follow
guidelines in the developer
guide.
Reviewed the developer guide on Submitting a Pull
Request
Pending release
notes
updated with breaking and/or notable changes for the next major release.
Documentation has been updated, if necessary.
Unit tests have been added, if necessary.
Integration tests have been added, if necessary.

Show available bot commands

These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

/retest ci/centos/<job-name>: retest the <job-name> after unrelated
failure (please report the failure too!)

nixpanic

This is a user visible change. Put a note in PendingReleaseNotes.md for this too. Maybe add a note in the main README.md for support of this as well, update other documentation?

internal/rbd/controllerserver.go

e2e/rbd.go

internal/util/idlocker.go

mergify · 2026-03-10T16:43:22Z

This pull request now has conflicts with the target branch. Could you please resolve conflicts and force push the corrected changes? 🙏

mergify · 2026-03-11T05:14:21Z

This pull request now has conflicts with the target branch. Could you please resolve conflicts and force push the corrected changes? 🙏

nixpanic

Two minor things that caught the eye of Claude Code, and look like good recommendations to me.

internal/rbd/controllerserver.go

examples/rbd/storageclass.yaml

YiteGu · 2026-03-11T11:21:33Z

/test ci/centos/mini-e2e/k8s-1.34

nixpanic

Another round of review (now completed by me, not Claude).

These are the only changes I would like to see, everything else looks good to me. Thanks!

PendingReleaseNotes.md

README.md

docs/volumeattributesclass.md

nixpanic

Thanks, looks good to me.

internal/rbd/controllerserver.go

YiteGu · 2026-03-19T12:24:37Z

In the logs of the csi-resizer sidecar of the provisioner:

  I0319 08:16:18.213061       1 event.go:389] "Event occurred" object="rbd-930/rbd-pvc" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="VolumeModify" message="external resizer is modifying volume rbd-pvc with vac gold"
  E0319 08:16:18.214624       1 controller.go:254] "Error syncing PVC" err="cannot get error status from modify volume err: error getting secret csi-rbd-secret in namespace default: secrets \"csi-rbd-secret\" not found"

The StorageClass probably points to the wrong secrets?

I checked the StorageClass's secret and found no problems.

   # controller-modify-secret
   csi.storage.k8s.io/controller-modify-secret-name: csi-rbd-secret
   csi.storage.k8s.io/controller-modify-secret-namespace: default

YiteGu · 2026-03-19T12:45:41Z

In the logs of the csi-resizer sidecar of the provisioner:

  I0319 08:16:18.213061       1 event.go:389] "Event occurred" object="rbd-930/rbd-pvc" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="VolumeModify" message="external resizer is modifying volume rbd-pvc with vac gold"
  E0319 08:16:18.214624       1 controller.go:254] "Error syncing PVC" err="cannot get error status from modify volume err: error getting secret csi-rbd-secret in namespace default: secrets \"csi-rbd-secret\" not found"

The StorageClass probably points to the wrong secrets?

I checked the StorageClass's secret and found no problems.

   # controller-modify-secret
   csi.storage.k8s.io/controller-modify-secret-name: csi-rbd-secret
   csi.storage.k8s.io/controller-modify-secret-namespace: default

Are there any other investigation approaches?

nixpanic · 2026-03-19T12:58:21Z

In createRBDStorageClass there is this, and it needs extending for the new StoragClass parameters:

ceph-csi/e2e/rbd_helper.go

Lines 136 to 146 in 9af58c4

    
           sc.Parameters["csi.storage.k8s.io/provisioner-secret-namespace"] = cephCSINamespace 
        
           sc.Parameters["csi.storage.k8s.io/provisioner-secret-name"] = rbdProvisionerSecretName 
        
           sc.Parameters["csi.storage.k8s.io/controller-expand-secret-namespace"] = cephCSINamespace 
        
           sc.Parameters["csi.storage.k8s.io/controller-expand-secret-name"] = rbdProvisionerSecretName 
        
           sc.Parameters["csi.storage.k8s.io/controller-publish-secret-namespace"] = cephCSINamespace 
        
           sc.Parameters["csi.storage.k8s.io/controller-publish-secret-name"] = rbdProvisionerSecretName 
        
           sc.Parameters["csi.storage.k8s.io/node-stage-secret-namespace"] = cephCSINamespace 
        
           sc.Parameters["csi.storage.k8s.io/node-stage-secret-name"] = rbdNodePluginSecretName

nixpanic · 2026-03-19T15:28:33Z

/test ci/centos/mini-e2e/k8s-1.34/rbd

nixpanic · 2026-03-19T15:42:44Z

@YiteGu , I've updated the commit e2e: add e2e test for rbd volume volumeattributesclass to include the new parameters. With these changes it should all be good 🤞

nixpanic · 2026-03-19T15:42:48Z

/test ci/centos/mini-e2e/k8s-1.34/rbd

nixpanic · 2026-03-19T15:42:55Z

/test ci/centos/mini-e2e/k8s-1.33/rbd

YiteGu · 2026-03-19T15:53:16Z

@YiteGu , I've updated the commit e2e: add e2e test for rbd volume volumeattributesclass to include the new parameters. With these changes it should all be good 🤞

Nice, I've already returned home from the company, and I'll continue working tomorrow.

YiteGu · 2026-03-20T02:43:26Z

@nixpanic @iPraveenParihar e2e test successful, I will execu a final push, remove the diagnostic log I had added.

Pull request has been modified.

1. Create a VolumeAttributesClass with a QoS parameter. Create a PVC with a VolumeAttributesClassName parameter. Verify the QoS results of the RBD image. 2. Modify the VolumeAttributesClassName of the PVC and verify the change in the QoS of the RBD image. 3. Create a VolumeAttributesClass that supports capacity-based QoS and verify the QoS results of the RBD image. Create a cloned volume of the PVC and verify the QoS results of the cloned volume. Signed-off-by: Yite Gu <guyite@bytedance.com> Co-authored-by: Niels de Vos <ndevos@ibm.com>

The addition of a new op caused the tryAcquire function to become too complex and fail CI checks. The tryAcquire function implementation was refactored, and the conflictMatrix design was introduced, which greatly reduced the complexity. Signed-off-by: Yite Gu <guyite@bytedance.com>

storageclass is no longer used as the qos parameters carrier for RBD volumes; these parameters are now defined in volumeattribetesclass. Signed-off-by: Yite Gu <guyite@bytedance.com>

Signed-off-by: Yite Gu <guyite@bytedance.com>

YiteGu · 2026-03-20T03:49:56Z

/test ci/centos/mini-e2e/k8s-1.34/rbd

YiteGu · 2026-03-20T03:50:07Z

/test ci/centos/mini-e2e/k8s-1.33/rbd

mergify bot added the component/rbd Issues related to RBD label Mar 6, 2026

YiteGu force-pushed the support-vac-feature-for-rbd branch 3 times, most recently from b78efe5 to 36c1c2f Compare March 6, 2026 11:43

YiteGu mentioned this pull request Mar 6, 2026

Use MutableParameters for NBD QOS in CreateVolume #5750

Closed

YiteGu force-pushed the support-vac-feature-for-rbd branch 4 times, most recently from c1e4f0a to dffa21e Compare March 9, 2026 13:41

nixpanic reviewed Mar 10, 2026

View reviewed changes

internal/rbd/controllerserver.go Outdated Show resolved Hide resolved

internal/rbd/controllerserver.go Show resolved Hide resolved

e2e/rbd.go Outdated Show resolved Hide resolved

internal/util/idlocker.go Show resolved Hide resolved

YiteGu force-pushed the support-vac-feature-for-rbd branch 3 times, most recently from cb885ba to a8df95a Compare March 11, 2026 05:13

YiteGu force-pushed the support-vac-feature-for-rbd branch 2 times, most recently from 43fa44e to 1a4ade8 Compare March 11, 2026 06:25

YiteGu requested a review from nixpanic March 11, 2026 07:36

YiteGu force-pushed the support-vac-feature-for-rbd branch from f7b6685 to e786b28 Compare March 11, 2026 07:44

nixpanic reviewed Mar 11, 2026

View reviewed changes

internal/rbd/controllerserver.go Outdated Show resolved Hide resolved

examples/rbd/storageclass.yaml Show resolved Hide resolved

YiteGu force-pushed the support-vac-feature-for-rbd branch from e786b28 to 650a107 Compare March 11, 2026 11:46

nixpanic reviewed Mar 11, 2026

View reviewed changes

PendingReleaseNotes.md Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

docs/volumeattributesclass.md Outdated Show resolved Hide resolved

YiteGu force-pushed the support-vac-feature-for-rbd branch 2 times, most recently from b2d68cb to a79ce62 Compare March 16, 2026 02:52

YiteGu requested a review from nixpanic March 16, 2026 03:34

nixpanic approved these changes Mar 16, 2026

View reviewed changes

nixpanic requested a review from a team March 16, 2026 09:05

nixpanic added the enhancement New feature or request label Mar 16, 2026

nixpanic requested changes Mar 16, 2026

View reviewed changes

internal/rbd/controllerserver.go Show resolved Hide resolved

YiteGu force-pushed the support-vac-feature-for-rbd branch from 19d57ab to b6400da Compare March 19, 2026 12:13

nixpanic force-pushed the support-vac-feature-for-rbd branch from b6400da to f838c0b Compare March 19, 2026 15:28

nixpanic force-pushed the support-vac-feature-for-rbd branch from f838c0b to c059c35 Compare March 19, 2026 15:42

nixpanic previously approved these changes Mar 19, 2026

View reviewed changes

nixpanic requested a review from a team March 19, 2026 15:44

YiteGu force-pushed the support-vac-feature-for-rbd branch from c059c35 to 137bf95 Compare March 20, 2026 02:58

YiteGu and others added 9 commits March 20, 2026 11:03

rbd: move qos parameters from sc to vac

d185863

storageclass is no longer used as the qos parameters carrier for RBD volumes; these parameters are now defined in volumeattribetesclass. Signed-off-by: Yite Gu <guyite@bytedance.com>

doc: update pendingreleasenotes for rbd qos vac feature

ca2b034

Signed-off-by: Yite Gu <guyite@bytedance.com>

doc: update README.md for rbd volumeattributesclass feature

1b56644

Signed-off-by: Yite Gu <guyite@bytedance.com>

doc: update how use volumeattributesclass for rbd volume qos

3a05746

Signed-off-by: Yite Gu <guyite@bytedance.com>

rbd: set controller-modify-secret in SC parameters

22d8834

Signed-off-by: Yite Gu <guyite@bytedance.com>

deploy: update csi-provisioner image to v6.2.0

c7a3504

Signed-off-by: Yite Gu <guyite@bytedance.com>

deploy: add volumeattributesclasses permissions for csi-provisioner

ff95af7

Signed-off-by: Yite Gu <guyite@bytedance.com>

YiteGu force-pushed the support-vac-feature-for-rbd branch from 137bf95 to ff95af7 Compare March 20, 2026 03:04

nixpanic approved these changes Mar 20, 2026

View reviewed changes

nixpanic requested a review from a team March 20, 2026 08:04

Conversation

YiteGu commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe what this PR does

Is there anything that requires special attention

Related issues

Future concerns

Uh oh!

nixpanic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mergify bot commented Mar 10, 2026

Uh oh!

mergify bot commented Mar 11, 2026

Uh oh!

nixpanic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

YiteGu commented Mar 11, 2026

Uh oh!

nixpanic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nixpanic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

YiteGu commented Mar 19, 2026

Uh oh!

YiteGu commented Mar 19, 2026

Uh oh!

nixpanic commented Mar 19, 2026

Uh oh!

nixpanic commented Mar 19, 2026

Uh oh!

nixpanic commented Mar 19, 2026

Uh oh!

nixpanic commented Mar 19, 2026

Uh oh!

nixpanic commented Mar 19, 2026

Uh oh!

YiteGu commented Mar 19, 2026

Uh oh!

YiteGu commented Mar 20, 2026

Uh oh!

YiteGu commented Mar 20, 2026

Uh oh!

YiteGu commented Mar 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

YiteGu commented Mar 6, 2026 •

edited

Loading