Dockerhub OCI repository returns 403 for http/1.1 but not for HTTP/2

[maxverbeek]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

Arguably, this issue resides on the side of Docker Hub.

OCI Helm chart tag listing fails with 403 Forbidden when the registry is Docker Hub (registry-1.docker.io). The token endpoint at auth.docker.io rejects the authenticated request, making it impossible to use targetRevision constraints (e.g. 1.*) that require tag enumeration.

The root cause is that ArgoCD's OCI HTTP transports set a custom TLSClientConfig on http.Transport, which silently disables HTTP/2 in Go. Docker Hub's token endpoint (auth.docker.io, fronted by Cloudflare) rejects HTTP/1.1 Basic Auth requests with 403, while identical HTTP/2 requests succeed.

This affects three http.Transport instantiations:

• util/helm/client.go — getIndex() (line 372)
• util/helm/client.go — GetTags() (line 492)
• util/oci/client.go — NewClientWithLock() (line 142)

To Reproduce

The following standalone Go program reproduces the issue using the same ORAS library that ArgoCD uses internally. It requires a Docker Hub account with at least one private OCI Helm chart:

package main

import (
	"context"
	"crypto/tls"
	"fmt"
	"net/http"
	"os"

	"oras.land/oras-go/v2/registry/remote"
	"oras.land/oras-go/v2/registry/remote/auth"
)

func main() {
	username := os.Getenv("DOCKERHUB_USERNAME")
	password := os.Getenv("DOCKERHUB_TOKEN")
	repo := os.Getenv("DOCKERHUB_REPO") // e.g. "myorg/myapp"

	if username == "" || password == "" || repo == "" {
		fmt.Fprintln(os.Stderr, "Usage: DOCKERHUB_USERNAME=... DOCKERHUB_TOKEN=... DOCKERHUB_REPO=myorg/myapp go run main.go")
		os.Exit(1)
	}

	ref := "registry-1.docker.io/" + repo
	credential := auth.StaticCredential("registry-1.docker.io", auth.Credential{
		Username: username,
		Password: password,
	})

	runTest := func(name string, transport http.RoundTripper) {
		fmt.Printf("=== %s ===\n", name)
		repository, _ := remote.NewRepository(ref)
		repository.Client = &auth.Client{
			Client:     &http.Client{Transport: transport},
			Cache:      nil,
			Credential: credential,
		}
		var tags []string
		err := repository.Tags(context.Background(), "", func(t []string) error {
			tags = append(tags, t...)
			return nil
		})
		if err != nil {
			fmt.Printf("FAILED: %v\n\n", err)
		} else {
			fmt.Printf("SUCCESS: Found %d tags\n\n", len(tags))
		}
	}

	// What ArgoCD does: custom TLS config disables HTTP/2
	runTest("Custom TLS (HTTP/1.1 only)", &http.Transport{
		TLSClientConfig:   &tls.Config{},
		DisableKeepAlives: true,
	})

	// With HTTP/2
	runTest("Custom TLS + ForceAttemptHTTP2", &http.Transport{
		TLSClientConfig:   &tls.Config{},
		DisableKeepAlives: true,
		ForceAttemptHTTP2:  true,
	})

	// DefaultTransport for comparison
	runTest("DefaultTransport (HTTP/2 via ALPN)", http.DefaultTransport)
}

DOCKERHUB_USERNAME=youruser DOCKERHUB_TOKEN=dckr_pat_... DOCKERHUB_REPO=yourorg/yourrepo go run main.go

Output:

=== Custom TLS (HTTP/1.1 only) ===
FAILED: GET "https://registry-1.docker.io/v2/yourorg/yourrepo/tags/list": GET "https://auth.docker.io/token?scope=repository%3Ayourorg%2Fyourrepo%3Apull&service=registry.docker.io": response status code 403: Forbidden

=== Custom TLS + ForceAttemptHTTP2 ===
SUCCESS: Found 55 tags

=== DefaultTransport (HTTP/2 via ALPN) ===
SUCCESS: Found 55 tags

The only difference between the failing and passing cases is ForceAttemptHTTP2: true on the transport. The credentials, headers, and request are otherwise identical.

Expected behavior

OCI Helm chart tag listing should succeed against Docker Hub, allowing targetRevision constraints like * or semver ranges to resolve correctly.

Version

argocd: v3.1.9
  BuildDate: unknown
  GitCommit: v3.1.9
  GitTreeState: clean
  GitTag: v3.1.9
  GoVersion: go1.25.7
  Compiler: gc
  Platform: linux/amd64
argocd-server: v3.3.2
  BuildDate: 2026-02-22T12:33:55Z
  GitCommit: 8a3940d8db27928931f0a85ba7c636e54786bddc
  GitTreeState: clean
  GitTag: v3.3.2
  GoVersion: go1.25.5
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.8.1 2026-02-09T16:15:27Z
  Helm Version: v3.19.4+g7cfb6e4
  Kubectl Version: v0.34.0
  Jsonnet Version: v0.21.0

Logs

time="2026-03-23T15:06:10Z" level=error msg="finished unary call with code Unknown"
  error="rpc error: code = Unknown desc = failed to get index:
  failed to get tags: GET \"https://registry-1.docker.io/v2/[redacted]/[redacted]/tags/list\":
  GET \"https://auth.docker.io/token?scope=repository%3A[redacted]%3Apull&service=registry.docker.io\":
  response status code 403: Forbidden"
  grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.time_ms=2906.028

Fix

Add ForceAttemptHTTP2: true to all three http.Transport instantiations. This is a one-line addition per transport that restores HTTP/2 ALPN negotiation while preserving custom TLS config support.

• util/helm/client.go:372 — getIndex()
• util/helm/client.go:492 — GetTags()
• util/oci/client.go:142 — NewClientWithLock()

From the Go net/http docs:

Programs that must disable HTTP/2 can do so by setting Transport.TLSNextProto (to a non-nil, empty map) or by using non-h2 TLS configuration. Alternatively, to configure transport-level HTTP/2 settings, use golang.org/x/net/http2 when setting TLSClientConfig, or set ForceAttemptHTTP2.