thar-be-registries: credentials entry creates a directory that shadows the `_default` wildcard mirror

[tim-mcdonald]

Image I'm using:

Bottlerocket 1.64.0, aws-k8s-1.33 (containerd 2.2). Also present on develop at v16.3.0.

What I expected to happen:

With a wildcard mirror plus credentials for a private registry, pulls from that registry go to the mirror.

[[settings.container-registry.mirrors]]
registry = "*"
endpoint = ["http://127.0.0.1:30020"]

[[settings.container-registry.credentials]]
registry = "registry.example.com"
username = "user"
password = "pass"

What actually happened:

Pulls from registry.example.com bypass the mirror entirely and go upstream. Registries with no credentials entry use the mirror correctly. There is no warning, no log line, and the settings apply successfully, so this fails silently.

How to reproduce the problem:

Apply the settings above, then on the node:

ls /etc/containerd/certs.d/_default/            # hosts.toml
ls /etc/containerd/certs.d/registry.example.com/ # credentials.toml only, no hosts.toml

Pull an image from registry.example.com and observe no request reaching the mirror endpoint.

Cause

write_hosts_toml maps registry = "*" to certs.d/_default/hosts.toml. write_credentials_toml writes to certs.d/<registry>/credentials.toml and calls fs::create_dir_all to do so, creating a directory that contains no hosts.toml.

containerd selects the host config directory by stat'ing the directory rather than by checking for hosts.toml (core/remotes/docker/config/config_unix.go, HostDirFromRoot in hosts.go):

hosts = append(hosts,
    filepath.Join(root, host),
    filepath.Join(root, "_default"),
)
...
for _, p := range hostPaths(root, host) {
    if _, err := os.Stat(p); err == nil {
        return p, nil
    }
}

The credentials-only directory matches first, so _default is never reached. loadHostDir then finds no hosts.toml there and returns no mirror hosts.

The effect is that a wildcard mirror silently stops applying to precisely the registries the user authenticates to. For anyone using a wildcard mirror for pull-through caching or P2P distribution, the private registries that benefit most are the ones excluded.

Docker Hub has an extra wrinkle: write_credentials_toml special-cases docker.io to registry-1.docker.io, which is also the host containerd resolves. A user-supplied registry = "docker.io" mirror writes to certs.d/docker.io/, which containerd never consults, so Docker Hub cannot be mirrored at all unless the user knows to write registry-1.docker.io in the mirror entry.

Workaround

Add an explicit mirror entry for every credentialed registry, duplicating the wildcard endpoints, and spell Docker Hub as registry-1.docker.io:

[[settings.container-registry.mirrors]]
registry = "registry.example.com"
endpoint = ["http://127.0.0.1:30020"]

thar-be-registries already supports both files in one directory, per test_workflow_mirror_and_credential_same_registry.

Suggested fix

Either of these would work, and I am happy to open a PR for whichever you prefer:

1. When a * mirror is configured, also write its hosts.toml into each credentialed registry's directory, with the appropriate server value for that registry rather than the _default form.
2. Have write_credentials_toml avoid creating a bare directory, for example by writing credentials into the _default directory keyed by host, if containerd's credential lookup allows it.

Option 1 is the smaller change and keeps the existing file layout.

At minimum, a warning at render time when a credentials entry has no matching mirror entry and a * mirror exists would turn a silent failure into a visible one.