"reclaimPolicy = Delete", doesn't actually delete the remote content

[ryan-kraay-sigstore]

What happened:
This might not be a bug, rather intended (but misleading) behavior.

Given the following StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: drime-insecure
provisioner: rclone.csi.veloxpack.io
parameters:
  remote: "drime-insecure"
  remotePath: "rclone-20260206/insecure/${pvc.metadata.namespace}/${pvc.metadata.name}"
  csi.storage.k8s.io/node-publish-secret-name: "drime-insecure"
  csi.storage.k8s.io/node-publish-secret-namespace: "example"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true

I created a PVC and added some remote content. Upon deletion of the PVC the remote content remains, even though my reclaimPolicy is set to Delete.

What you expected to happen:

With having a reclaimPolicy: Delete - I would expect that at least the last directory in my remotePath would be deleted. (ie: rm -rf rclone-20260206/insecure/${pvc.metadata.namespace}/${pvc.metadata.name}).

note: I am aware that, in this example, this would leave some dangling empty namespace folders. However, perhaps this could be mitigated by changing the remotePath to rclone-20260206/insecure/${pvc.metadata.namespace}_${pvc.metadata.name}

How to reproduce it:

I don't have immediate access to s3, but my example was closely modeled after this example.

Deploy it, add some content to the PVC (via the pod), delete the pod/pvc or namespace, and you should find that the content still exists in the remote storage.

Anything else we need to know?:

Looking at the code this seems intentional.

However, it's unclear if there's a technical reason or simply not enough time/interest.

Environment:

• CSI Driver version: self-compiled based on your latest commit (I needed rclone v1.73.0)
• Kubernetes version (use kubectl version): v1.34.0
• OS (e.g. from /etc/os-release): talos v1.12.2
• Kernel (e.g. uname -a):
• Rclone version (use rclone version): 1.73.0
• Storage backend (S3, GCS, Azure, etc.): drime
• Install tools:
• Others:

[paulgrammer]

@ryan-kraay-sigstore Thank you for reporting this issue. I’ll look into it and keep you posted on any updates.

[paulgrammer]

@ryan-kraay-sigstore Thanks once again for raising this issue! You're right, the current behavior doesn't match the standard Kubernetes reclaimPolicy: Delete semantics, and I appreciate you taking the time to look at the code and provide such a detailed report.

Why the current implementation works this way:

The deletion behavior is intentional, but I recognize it's not well documented. The rclone CSI driver is different from traditional CSI drivers because it mounts paths within existing remote storage systems (S3 buckets, cloud storage, etc.) rather than creating isolated volumes.

I was concerned about scenarios like:

• Someone mounting the root of their S3 bucket as remote: "s3-bucket" with remotePath: "/"
• Multiple PVCs sharing parent directories
• Users storing non-Kubernetes-managed data in the same remote storage

In these cases, automatically deleting on PVC removal could cause catastrophic data loss - imagine losing your entire S3 bucket because a Pod was deleted!

However, you're absolutely right:

We should honor the standard Kubernetes reclaimPolicy. When reclaimPolicy: Delete is set, the driver should delete the resolved remotePath for that PVC. For your example with remotePath: "rclone-20260206/insecure/${pvc.metadata.namespace}/${pvc.metadata.name}", it would delete that entire resolved path.

For safety:

1. Add strong documentation warnings about using reclaimPolicy: Delete with static or root paths like "/"
2. Consider validation to warn or prevent dangerous configurations like remotePath: "/" with reclaimPolicy: Delete

I'm looking forward to finding a solution for this soon and will update the documentation accordingly!

[ryan-kraay-sigstore]

I greatly appreciate you wanting to turn this topic into a discussion. I completely understand where you're coming from and agree with the goal of "not breaking existing behavior".

The rclone CSI driver is different from traditional CSI drivers because it mounts paths within existing remote storage systems (S3 buckets, cloud storage, etc.) rather than creating isolated volumes.

tl;dr: Rather than introduce additional feature flags for deleting files, I'd suggest trying to create a mode where this CSI driver behaves like a "traditional CSI driver". A unified behavior for all CSI drivers is what makes the CSI framework so powerful and flexible. What this mode should be called is something to discuss. Maybe enforceReclaimPolicy: true|false (default false), legacyMode: true|false (default true)...

That said, as I've looked over quickly skimmed the code, there are two behaviors that this driver deviates a bit from:
1. The "default remotePath" of /
2. (as discussed) the reclaimPolicy: Delete/Retain

afaik, the good news is that your code already has all the logic. The bad news is, I can't think of a solution that would provide the expected CSI functionality, but doesn't cripple the current remotePath behavior.

What I mean is that the "default remotePath" is entirely dependent on the characteristics of the static PV or the dynamic PVC. In this respect "pinning" the remotePath to single templated value - will not work (at least, I cannot figure out - how to make it work).

For example:

If I have a dynamic PVC and a reclaimPolicy: Delete then my default remotePath would be "${pvc.spec.volumeName}" (?) (ie: remotePath: pvc-aabbccddd).

If I have a dynamic PVC and a reclaimPolicy: Retain then my default remotePath would be "${pvc.metadata.namespace}_${pvc.metadata.name}" (ie: remotePath: "default_pvc-s3-example")

If I have a static PV, regardless of my reclaimPolicy, my default remotePath would be what path is passed into the PV. As a sidenote, I've seen that the s3 csi driver (and others) use spec.mountOptions to provide a fixed path.

putting it all together

I'd suggest keeping the StorageClass remotePath, but remove the templating (from the user-facing interface) and use it as a "prefix path".

Let me see if an example helps put this all together (modeled after this example:

---
# Amazon S3 Storage Class
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rclone-s3
provisioner: rclone.csi.veloxpack.io
parameters:
  remote: "s3"
  remotePath: "/path/to/new/root"  # This can now _safely_ default to "/"
  csi.storage.k8s.io/node-publish-secret-name: "rclone-s3-secret"
  csi.storage.k8s.io/node-publish-secret-namespace: "default"
  legacyMode: false  # enforce our new behavior
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
---
# Example static PV:  The final path would be "/path/to/new/root/foo/bar"
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-rclone-example
  namespace: default
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany  # Rclone supports ReadWriteMany for cloud storage
  persistentVolumeReclaimPolicy: Retain
  storageClassName: rclone-s3
  mountOptions:
    - remotePath "/foo/bar"  # if spec.mountOptions.remotePath is not defined _and_ persistenceVolumeReclaimPolicy (or StorageClass.spec.reclaimPolicy) is Delete, we should error and force the user to explicitly add `spec.mountOptions.remotePath: "/"`.
---
# Example PVC using S3 storage
#  If StorageClass.spec.reclaimPolicy == Delete then the path will be "/path/to/new/root/pvc-aabbcc"
#  If StorageClass.spec.reclaimPolicy == Retain then the path will be "/path/to/new/root/default_pvc-s3-example"
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-s3-example
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: rclone-s3

Anyway, this is a (lengthy) but rough idea. I think it can certainly be refined and improved. What are you thoughts?

[paulgrammer]

Thank you so much for taking the time to share your thoughts and suggestions. I really appreciate it. Everything you’ve proposed makes sense, and I’m glad we’re aligned on the direction.
I’d like to spend a bit more time looking into how these changes would play out in practice and what side effects we might need to consider. Once I’ve done that, I’ll come back with an update. Thanks again for your patience and for the detailed input.

[paulgrammer]

@ryan-kraay-sigstore I went through all the sources you shared, their approach looks solid. They run a separate DaemonSet per storage mount, which means even if the rclone mount pod crashes, the FUSE process itself remains intact.

What do you think this implies for our driver? Right now, we’re using rclone as a Go package rather than as a CLI tool, so achieving similar isolation might be tricky. Maybe it’s worth considering running rclone as a CLI instead? Would love your thoughts.

[ryan-kraay-sigstore]

disclaimer: I've got no hands-on experience with this, so take it worth a grain of salt. However, I do have opinions ;)

tl;dr: I think switching to DaemonSets + rclone-cli would improve the robustness of this csi driver. However, given that it would effectively be a rewrite -- I'd avoid it, especially if no-one has complained about the current level of robustness.

Using the current code-base, I think most of the proposed changes (ie: legacyMode + the "proper" remotePath generation based on PV vs PVC) can consolidated as a web of branch logic applied here.

The only unknown for me is where/how would the RetainPolicy == Delete be implemented? I'd assume it should happen after the fuse filesystem is unmounted, probably in the DeleteVolume() call. In the case of having a PVC w/ a ReclaimPolicy == Delete: Would rclone be able to delete that /path/to/new/root/pvc-aabbcc? Would it be possible to pass that path in as a string? (It would be a heck of a lot easier, if before the umount, we'd simply run rm -rf * -- but it won't remove the pvc directory).

What do you think this implies for our driver? Right now, we’re using rclone as a Go package rather than as a CLI tool, so achieving similar isolation might be tricky. Maybe it’s worth considering running rclone as a CLI instead?

Actually, when I was looking for an rclone csi driver (there are a few options available): I chose this one because it used the Go package. My expectation was that by using the Go package, this csi driver would have better error handling and logging than if it just ran the cli. But, if you tell me that you can deal with the error handling as well via the cli -- I'll believe you.

Regarding isolation: tbh, I'm surprised the Go package wouldn't facilitate using goroutines to allow a single daemon to manage multiple rclone endpoints/filesystems.

They run a separate DaemonSet per storage mount, which means even if the rclone mount pod crashes, the FUSE process itself remains intact.

Many of the other CSI drivers, I've seen/used, use these DaemonSets are a way to isolate certain elevated resource/service-account privileges. For example, the S3 CSI driver needs elevated IAM Role permissions that you don't want associated with your workload. However, in the case of this rclone driver - there is no need for additional privileges.

That said, the DaemonSet would offer an extra degree of robustness. For example, if/when a PV/PVC w/ RetainPolicy == Delete get's hung or broken (due to networking disruptions, lack of rclone permissions, etc), or just takes a really long time (due to the amount of data to remove); the DaemonSet could in the background try, retry, and wait -- meanwhile, the pod that was using it can still release its mount and "move on".

However, as I mentioned, it would be a complete rewrite. If you want to do a rewrite -- great! Go for it! However, for the scope of this issue/feature request, I think it's overkill.

[paulgrammer]

@ryan-kraay-sigstore I went through all the sources you shared, their approach looks solid. They run a separate DaemonSet per storage mount, which means even if the rclone mount pod crashes, the FUSE process itself remains intact.

What do you think this implies for our driver? Right now, we’re using rclone as a Go package rather than as a CLI tool, so achieving similar isolation might be tricky. Maybe it’s worth considering running rclone as a CLI instead? Would love your thoughts.

This comment was not intended to address this issue, my apologies, it was to address #58