Best Practice for Multiple Namespaces and Sub -volume Groups

[woodz1024]

Good afternoon all, I am a new to using ceph-csi and am gaining experience with K8s. I am trying to deploy in such a way that I can access multiple namespaces and sub-volume groups for RBD and cephFS respectively. As far as I can tell that cant be done without deploying multiple ceph-csi instances since this values are set in the configmap that is used before everything is deployed. Based on this I have two questions:

Is there a way to specify a namespace/sub-volume group in a storage class or set it after the deployed is done that I am missing?

Would be it be a bad practice to have multiple deployments of ceph-csi for RBD and cephFs? My concern with this is a creating too much overhead on my cluster.

Any guidance is greatly appreciated, I want to be sure that I am doing everything correctly.

Thank you for your time!

[iPraveenParihar]

Hey @woodz1024, you could provide your such configurations in the ceph-csi-config ConfigMap and use the desired subvolume groupby specifying the clusterID in the StorageClass.

config.json: |-
    [
      {
        "clusterID": "<cluster-id-1>",
        "rbd": {
           "radosNamespace": "rbd-ns-1",
        },
        "monitors": [
          "<MONValue1>",
        ],
        "cephFS": {
          "subvolumeGroup": "svg-1"
          "radosNamespace": "cephfs-ns-1"
          ....
        }
      },
      {
        "clusterID": "<cluster-id-2>",
        "rbd": {
           "radosNamespace": "rbd-ns-2",
        },
        "monitors": [
          "<MONValue1>",
        ],
        "cephFS": {
          "subvolumeGroup": "svg-2"
          "radosNamespace": "cephfs-ns-2"
          ....
        }
      }
    ]

Use the desired clusterID in the StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
   # (required) String representing a Ceph cluster to provision storage from.
   # Should be unique across all Ceph clusters in use for provisioning,
   # cannot be greater than 36 bytes in length, and should remain immutable for
   # the lifetime of the StorageClass in use.
   # Ensure to create an entry in the configmap named ceph-csi-config, based on
   # csi-config-map-sample.yaml, to accompany the string chosen to
   # represent the Ceph cluster in clusterID below
   clusterID: <cluster-id-2>
   ...

Does this help you?

[woodz1024]

Hey @iPraveenParihar, thank you so much for answering. This worked and resolved the problems I was encountering. Have a great day!

[iPraveenParihar]

Great! 🎉