calico/istio-ztunnel arm64 image does not exec

[nothingherenow]

calico/istio-ztunnel:v3.32.1 does not seem to have a valid arm64 image. It fails immediately to start up.

❯ uname -m
aarch64
❯ podman run --rm calico/istio-ztunnel:v3.32.1
{"msg":"exec container process `/usr/bin/ztunnel`: Exec format error","level":"error","time":"2026-07-09T18:41:06.368280Z"}

The index manifest seems to advertise an arm64 build, so this should work:

❯ crane manifest calico/istio-ztunnel:v3.32.1 | yq -P
schemaVersion: 2
mediaType: application/vnd.docker.distribution.manifest.list.v2+json
manifests:
  - mediaType: application/vnd.docker.distribution.manifest.v2+json
    size: 739
    digest: sha256:2044603ad673f511f229b8f11e74016413015b17e669deb08351943227c40773
    platform:
      architecture: amd64
      os: linux
  - mediaType: application/vnd.docker.distribution.manifest.v2+json
    size: 739
    digest: sha256:d8616d4c675d71bd6c2635c9386ea9a38ebb519209ebcb253193c4c676dfb3e9
    platform:
      architecture: arm64
      os: linux

Expected Behavior

The ztunnel container should start up properly.

Current Behavior

❯ uname -m
aarch64
❯ podman run --rm calico/istio-ztunnel:v3.32.1
{"msg":"exec container process `/usr/bin/ztunnel`: Exec format error","level":"error","time":"2026-07-09T18:41:06.368280Z"}

Possible Solution

Check the multi-arch build for istio-ztunnel. It seems that istio-install-cni and istio-pilot are built correctly for arm64.

❯ podman run --rm calico/istio-pilot:v3.32.1
Resolving "calico/istio-pilot" using unqualified-search registries (/home/user/.config/containers/registries.conf)
Trying to pull docker.io/calico/istio-pilot:v3.32.1...
Getting image source signatures
Copying blob b1ddc14fa8d9 skipped: already exists  
Copying blob acc5cbea5105 done   | 
Copying config fae7957156 done   | 
Writing manifest to image destination
2026/07/09 18:57:34 INFO memory is not limited, skipping package=github.com/KimMachineGun/automemlimit/memlimit
Istio Pilot provides mesh-wide traffic management, security and policy capabilities in the Istio Service Mesh.

Usage:
  pilot-discovery [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  discovery   Start Istio proxy discovery service.
  help        Help about any command
  request     Makes an HTTP request to Pilot metrics/debug endpoint
  version     Prints out build version information

Flags:
  -h, --help          help for pilot-discovery
      --vklog Level   number for the log level verbosity. Like -v flag. ex: --vklog=9

Use "pilot-discovery [command] --help" for more information about a command.
❯ podman run --rm calico/istio-install-cni:v3.32.1
Resolving "calico/istio-install-cni" using unqualified-search registries (/home/user/.config/containers/registries.conf)
Trying to pull docker.io/calico/istio-install-cni:v3.32.1...
Getting image source signatures
Copying blob 07372c2031f8 done   | 
Copying blob b1ddc14fa8d9 skipped: already exists  
Copying config c789ecdf2d done   | 
Writing manifest to image destination
2026-07-09T18:57:49.091535Z     info    cni-agent       CNI version: unknown-unknown-unknown
2026-07-09T18:57:49.091566Z     info    cni-agent       CNI logging level: info
2026-07-09T18:57:49.091601Z     info    cni-agent       CNI install configuration: 
MountedCNINetDir: /host/etc/cni/net.d
CNIConfName: 
ChainedCNIPlugin: true
CNIAgentRunDir: /var/run/istio-cni
IstioOwnedCNIConfigFilename: 
IstioOwnedCNIConfig: false
PluginLogLevel: info
KubeconfigMode: 0600
KubeCAFile: 
SkipTLSVerify: false
ExcludeNamespaces: 
PodNamespace: 
K8sServiceProtocol: 
K8sServiceHost: 
K8sServicePort: 
K8sNodeName: 65a33a9b054b
CNIBinSourceDir: /opt/cni/bin
CNIBinTargetDirs: /host/opt/cni/bin
MonitoringPort: 15014
ZtunnelUDSAddress: /var/run/ztunnel/ztunnel.sock
AmbientEnabled: false
AmbientEnablementSelector: 
AmbientDNSCapture: false
AmbientIPv6: false
AmbientDisableSafeUpgrade: false
AmbientReconcilePodRulesOnStartup: false
EnableAmbientDetectionRetry: false
NativeNftables: false
ForceIptablesBinary: 
MagicDSCPMark: 0

2026-07-09T18:57:49.091613Z     info    cni-agent       CNI race repair configuration: 
Enabled: true
NodeName: 
LabelKey: cni.istio.io/uninitialized
LabelValue: true
DeletePods: false
LabelPods: false
SidecarAnnotation: sidecar.istio.io/status
InitContainerName: istio-validation
InitTerminationMsg: 
InitExitCode: 126
LabelSelectors: 
FieldSelectors: 
NativeNftables: false
ForceIptablesBinary: 

2026-07-09T18:57:49.095548Z     info    ControlZ available at 127.0.0.1:9876
2026-07-09T18:57:49.097391Z     warn    Failed to create directory for /var/run/istio-cni/log.sock: mkdir /var: permission denied
2026-07-09T18:57:49.097448Z     error   cni-agent       Failed to start up UDS Log Server: failed to create UDS listener: failed to listen on unix socket "/var/run/istio-cni/log.sock": listen unix /var/run/istio-cni/log.sock: bind: no such file or directory
Error: failed to create UDS listener: failed to listen on unix socket "/var/run/istio-cni/log.sock": listen unix /var/run/istio-cni/log.sock: bind: no such file or directory

Steps to Reproduce (for bugs)

1. podman run --rm calico/istio-ztunnel:v3.32.1 # fails
2. podman run --rm calico/istio-ztunnel:v3.32.1-arm64 # also fails
3. Start an istio-ztunnel pod on an arm64 k8s worker

Context

It is blocking testing of the istio ambient mesh across my clusters, which include arm64 nodes.

Your Environment

• Calico version: v3.32.1
• Calico dataplane (bpf, nftables, iptables, windows etc.): bpf
• Orchestrator version (e.g. kubernetes, openshift, etc.): k8s
• Operating System and version: Talos Linux v1.12.4
• Link to your project (optional): N/A

[hjiawei]

@radixo Could this be caused by our ztunnel cross-compilation changes?

[caseydavenport]

@radixo @electricjesus boop - is this one in progress?