B2 Unsupported header 'x-amz-tagging' received for this API call.

[runningman84]

I tried to setup this proxy to encrypt stuff which is being sent from longhorn to b2. Unfortunately this fails so far:

time="2025-12-07T21:03:18Z" level=info msg=listening ip=0.0.0.0 port=4433 region=eu-west-1
time="2025-12-07T21:03:18Z" level=warning msg="TLS is disabled"
time="2025-12-07T21:18:33Z" level=debug msg=forwarding host="s3proxy-b2.s3proxy.svc.cluster.local:4433" method=GET path=/example-longhorn-backup
time="2025-12-07T21:18:33Z" level=debug msg=forwarding host="s3proxy-b2.s3proxy.svc.cluster.local:4433" method=GET path=/example-longhorn-backup
time="2025-12-07T21:18:33Z" level=debug msg=forwarding host="s3proxy-b2.s3proxy.svc.cluster.local:4433" method=GET path=/example-longhorn-backup
time="2025-12-07T21:18:33Z" level=debug msg=forwarding host="s3proxy-b2.s3proxy.svc.cluster.local:4433" method=GET path=/example-longhorn-backup
time="2025-12-07T21:18:33Z" level=debug msg=forwarding host="s3proxy-b2.s3proxy.svc.cluster.local:4433" method=GET path=/example-longhorn-backup
time="2025-12-07T21:18:33Z" level=debug msg=forwarding host="s3proxy-b2.s3proxy.svc.cluster.local:4433" method=GET path=/example-longhorn-backup
time="2025-12-07T21:18:33Z" level=debug msg=forwarding host="s3proxy-b2.s3proxy.svc.cluster.local:4433" method=GET path=/example-longhorn-backup
time="2025-12-07T21:18:33Z" level=debug msg=forwarding host="s3proxy-b2.s3proxy.svc.cluster.local:4433" method=GET path=/example-longhorn-backup
time="2025-12-07T21:20:05Z" level=debug msg=forwarding host="s3proxy-b2.s3proxy.svc.cluster.local:4433" method=GET path=/example-longhorn-backup
time="2025-12-07T21:20:05Z" level=debug msg=intercepting host="s3proxy-b2.s3proxy.svc.cluster.local:4433" method=PUT path=/example-longhorn-backup/prod/backupstore/volumes/b7/f5/example-volume/locks/lock-480c10b5f7424f16.lck
time="2025-12-07T21:20:05Z" level=debug msg=putObject bucket=example-longhorn-backup key=prod/backupstore/volumes/b7/f5/example-volume/locks/lock-480c10b5f7424f16.lck requestID=38ca845c-b2d6-4ff2-8d12-e69fa92d6a84
time="2025-12-07T21:20:05Z" level=error msg="PutObject sending request to S3" error="operation error S3: PutObject, <?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\n<Error>\n    <Code>InvalidArgument</Code>\n    <Message>Unsupported header 'x-amz-tagging' received for this API call.</Message>\n</Error>\n" requestID=38ca845c-b2d6-4ff2-8d12-e69fa92d6a84
time="2025-12-07T21:20:05Z" level=debug msg=intercepting host="s3proxy-b2.s3proxy.svc.cluster.local:4433" method=PUT path=/example-longhorn-backup/prod/backupstore/volumes/b7/f5/example-volume/locks/lock-480c10b5f7424f16.lck
time="2025-12-07T21:20:05Z" level=debug msg=putObject bucket=example-longhorn-backup key=prod/backupstore/volumes/b7/f5/example-volume/locks/lock-480c10b5f7424f16.lck requestID=fd67d81b-bb5b-45fe-806e-e88485cb1162
time="2025-12-07T21:20:06Z" level=error msg="PutObject sending request to S3" error="operation error S3: PutObject, <?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\n<Error>\n    <Code>InvalidArgument</Code>\n    <Message>Unsupported header 'x-amz-tagging' received for this API call.</Message>\n</Error>\n" requestID=fd67d81b-bb5b-45fe-806e-e88485cb1162
time="2025-12-07T21:20:06Z" level=debug msg=intercepting host="s3proxy-b2.s3proxy.svc.cluster.local:4433" method=PUT path=/example-longhorn-backup/prod/backupstore/volumes/b7/f5/example-volume/locks/lock-480c10b5f7424f16.lck
time="2025-12-07T21:20:06Z" level=debug msg=putObject bucket=example-longhorn-backup key=prod/backupstore/volumes/b7/f5/example-volume/locks/lock-480c10b5f7424f16.lck requestID=4bdc0c0d-4a22-46ea-8ca0-847a7657b3ca
time="2025-12-07T21:20:07Z" level=error msg="PutObject sending request to S3" error="operation error S3: PutObject, <?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\n<Error>\n    <Code>InvalidArgument</Code>\n    <Message>Unsupported header 'x-amz-tagging' received for this API call.</Message>\n</Error>\n" requestID=4bdc0c0d-4a22-46ea-8ca0-847a7657b3ca
time="2025-12-07T21:20:07Z" level=debug msg=intercepting host="s3proxy-b2.s3proxy.svc.cluster.local:4433" method=PUT path=/example-longhorn-backup/prod/backupstore/volumes/b7/f5/example-volume/locks/lock-480c10b5f7424f16.lck
time="2025-12-07T21:20:07Z" level=debug msg=putObject bucket=example-longhorn-backup key=prod/backupstore/volumes/b7/f5/example-volume/locks/lock-480c10b5f7424f16.lck requestID=a562be07-424a-40d3-bc83-e7a27fd5f5bb
time="2025-12-07T21:20:07Z" level=error msg="PutObject sending request to S3" error="operation error S3: PutObject, <?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\n<Error>\n    <Code>InvalidArgument</Code>\n    <Message>Unsupported header 'x-amz-tagging' received for this API call.</Message>\n</Error>\n" requestID=a562be07-424a-40d3-bc83-e7a27fd5f5bb
time="2025-12-07T21:20:07Z" level=debug msg=forwarding host="s3proxy-b2.s3proxy.svc.cluster.local:4433" method=GET path=/example-longhorn-backup

This is my longhorn config:

---
apiVersion: v1
kind: Secret
metadata:
  name: s3proxy-b2-credentials
  namespace: longhorn-system
type: Opaque
stringData:
  # Backblaze B2 account ID
  AWS_ACCESS_KEY_ID: "xxx"
  # Backblaze B2 application key
  AWS_SECRET_ACCESS_KEY: "yyy"
  # Optional helper values that s3proxy or your deployment might expect
  AWS_ENDPOINTS: "http://s3proxy-b2.s3proxy.svc.cluster.local:4433"
---
apiVersion: longhorn.io/v1beta2
kind: BackupTarget
metadata:
  name: s3proxy-b2
  namespace: longhorn-system
spec:
  backupTargetURL: "s3://example-longhorn-backup@dummyregion/prod"
  credentialSecret: "s3proxy-b2-credentials"
  pollInterval: "0"

[slig2008]

See longhorn/longhorn#5220 (comment)

[ynsta]

Fixed in #44 (merged).

Root cause: PutObject unconditionally set Tagging: &tags on the SDK input, even when the incoming request carried no x-amz-tagging header. The AWS SDK then emitted x-amz-tagging: on the wire, which B2 rejects with InvalidArgument: Unsupported header 'x-amz-tagging' received for this API call.

Fix: only forward Tagging when the client supplied a non-empty value, mirroring the pattern already used for the SSE-C fields.

Related context: there is a public fork at https://github.com/k3s-argocd-cluster/s3proxy that takes a different approach — an explicit --no-tagging flag that strips the header even when the client sent a non-empty value. The minimal fix in #44 already resolves the longhorn use case (longhorn does not set x-amz-tagging). If a future client genuinely sends non-empty tags toward B2, that PUT will still fail — which is correct, since B2 does not support object tagging at all. If you need a silent-strip mode for mixed environments, please open a separate issue.

Will land in the next release. Please test against main (or a build from it) and reopen if anything regresses.