What steps did you take and what happened:
I installed Velero in an air-gapped Kubernetes cluster, configured to use an in-cluster MinIO instance as the backup storage location. The velero and velero-plugin-for-aws images were pre-loaded into a private registry and specified during installation.
The installation was run with the --use-node-agent flag to enable filesystem-level backups of persistent volumes.
When I create a backup, the main backup phase completes successfully. The command velero backup describe my-backup shows Phase: Completed for the top-level backup status.
However, the Volume Backups section at the bottom of the describe output consistently fails. The describe command hangs for about 30 seconds, then shows the standard describe output except for the error:
`<error getting (warnings|errors|backup volume info): Get http://minio-svc.gitlab.svc.cluster.local:9000/velero/backups/backupname/backupname-volumeinfo.json/<and a bunch of credentials>: context deadline exceeded`
The most critical finding is this: I used kubectl exec to get a shell inside a running node-agent pod. From that shell, I ran curl http://gitlab-minio-svc.gitlab.svc.cluster.local:9000. This command succeeded and returned an "Access Denied" XML dump from MinIO. This proves that the node-agent pod has a valid and working network path to the MinIO service endpoint.
The issue is that the node-agent process itself is timing out, even though the pod it's running in can successfully connect to the target S3 endpoint.
What did you expect to happen:
I expected the volume backup (performed by the node-agent) to complete successfully, just like the main resource backup. The Volume Backups section in velero backup describe should show Phase: Completed.
The following information will help us better understand what's going on:
I am running this on an airgapped system and cannot copy these files or outputs. I can add the most relevant information. Please let me know if you need more.
-
kubectl logs deployment/velero -n velero does not show any warnings or errors.
-
velero backup describe <backupname> shows an error getting warnings, errors, and Backup Volumes. Everything else appears correct. The error is <error getting (warnings|errors|backup volume info): Get http://minio-svc.gitlab.svc.cluster.local:9000/velero/backups/backupname/backupname-volumeinfo.json/<and a bunch of credentials>: context deadline exceeded
-
velero backup logs <backupname> gives the same error as above: context deadline exceeded
Anything else you would like to add:
This is not a simple configuration issue. I have exhaustively tested the following, and none of them are the cause:
- Image Pulls: The environment is air-gapped, and all images are pre-loaded. The node-agent pods are in a Running state, not ImagePullBackOff.
-
Network Policies: There are no NetworkPolicy resources in any namespace.
-
Firewalls: curl from within the node-agent pod works, proving the network path is open.
- SELinux: SELinux is running in Permissive mode and is not blocking any operations.
- Velero Configuration: I have tried multiple installation methods, including generating a raw manifest with velero install --dry-run and manually verifying its contents. I have also tried manually editing the node-agent DaemonSet to inject env: variables like VELERO_S3_URL, but the error persists.
The fact that curl works from inside the pod, but the Velero process itself fails, seems to point to a bug in how the node-agent's uploader (Kopia/Restic) is initializing or interacting with the S3 client library. It is failing to connect despite having a valid network path available to it.
Environment:
- Velero version (use
velero version): v1.17.2, git commit 7013a40
- Velero features (use
velero client config get features): NOT SET
- Kubernetes version (use
kubectl version): v1.34.3
- Kubernetes installer & version: On-prem RKE2 deployment
- OS (e.g. from
/etc/os-release): RHEL 9.7
Vote on this issue!
This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.
- 👍 for "I would like to see this bug fixed as soon as possible"
- 👎 for "There are more important bugs to focus on right now"
What steps did you take and what happened:
I installed Velero in an air-gapped Kubernetes cluster, configured to use an in-cluster MinIO instance as the backup storage location. The velero and velero-plugin-for-aws images were pre-loaded into a private registry and specified during installation.
The installation was run with the
--use-node-agentflag to enable filesystem-level backups of persistent volumes.When I create a backup, the main backup phase completes successfully. The command
velero backup describe my-backupshowsPhase: Completedfor the top-level backup status.However, the Volume Backups section at the bottom of the describe output consistently fails. The describe command hangs for about 30 seconds, then shows the standard describe output except for the error:
The most critical finding is this: I used kubectl exec to get a shell inside a running node-agent pod. From that shell, I ran
curl http://gitlab-minio-svc.gitlab.svc.cluster.local:9000. This command succeeded and returned an "Access Denied" XML dump from MinIO. This proves that the node-agent pod has a valid and working network path to the MinIO service endpoint.The issue is that the node-agent process itself is timing out, even though the pod it's running in can successfully connect to the target S3 endpoint.
What did you expect to happen:
I expected the volume backup (performed by the node-agent) to complete successfully, just like the main resource backup. The Volume Backups section in velero backup describe should show Phase: Completed.
The following information will help us better understand what's going on:
I am running this on an airgapped system and cannot copy these files or outputs. I can add the most relevant information. Please let me know if you need more.
kubectl logs deployment/velero -n velerodoes not show any warnings or errors.velero backup describe <backupname>shows an error getting warnings, errors, and Backup Volumes. Everything else appears correct. The error is<error getting (warnings|errors|backup volume info): Get http://minio-svc.gitlab.svc.cluster.local:9000/velero/backups/backupname/backupname-volumeinfo.json/<and a bunch of credentials>: context deadline exceededvelero backup logs <backupname>gives the same error as above: context deadline exceededAnything else you would like to add:
This is not a simple configuration issue. I have exhaustively tested the following, and none of them are the cause:
The fact that curl works from inside the pod, but the Velero process itself fails, seems to point to a bug in how the node-agent's uploader (Kopia/Restic) is initializing or interacting with the S3 client library. It is failing to connect despite having a valid network path available to it.
Environment:
velero version): v1.17.2, git commit 7013a40velero client config get features): NOT SETkubectl version): v1.34.3/etc/os-release): RHEL 9.7Vote on this issue!
This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.