routing table is empty after bootstrapping on Talos + Cilium (v0.6.0)

[dbonne]

Spegel version

v0.6.0

Kubernetes distribution

Talos 1.12.4

Kubernetes version

v1.33.0

CNI

Cilium

Describe the bug

On a fresh 3-node Talos cluster with Cilium, Spegel v0.6.0 fails to bootstrap P2P routing. All 3 DaemonSet pods start simultaneously but the routing table never populates, causing the startup probe to kill them after 180s (60 × 3s). This results in an infinite CrashLoopBackOff cycle.

This is similar to #709 and #686, but those fixes are already included in v0.6.0. The error message is different — we see routing table is empty after bootstrapping rather than could not get peer id.

Key observations:

• DNS resolution works: spegel-bootstrap.kube-system.svc.cluster.local resolves to all 3 pod IPs
• TCP connectivity works: all 3 pods are reachable on port 5001 from other pods
• The spegel-bootstrap headless service has all 3 endpoints registered
• hostNetwork is not enabled (pods use pod IPs)
• The cluster is not dual-stack (IPv4 only)
• The init container (configuration) runs successfully and writes mirror config to /etc/cri/conf.d/hosts/_default/hosts.toml

Helm values:

spegel:
  containerdSock: /run/containerd/containerd.sock
  containerdRegistryConfigPath: /etc/cri/conf.d/hosts
service:
  registry:
    hostPort: 29999
serviceMonitor:
  enabled: true

Talos machine config (containerd customization):

machine:
  files:
    - op: create
      path: /etc/cri/conf.d/20-customization.part
      content: |
        [plugins."io.containerd.cri.v1.images"]
          discard_unpacked_layers = false

Container args (from DaemonSet):

registry
--log-level=INFO
--mirror-resolve-retries=3
--mirror-resolve-timeout=20ms
--registry-addr=:5000
--router-addr=:5001
--metrics-addr=:9090
--containerd-sock=/run/containerd/containerd.sock
--containerd-namespace=k8s.io
--containerd-registry-config-path=/etc/cri/conf.d/hosts
--bootstrap-kind=dns
--dns-bootstrap-domain=spegel-bootstrap.kube-system.svc.cluster.local.
--containerd-content-path=/var/lib/containerd/io.containerd.content.v1.content
--debug-web-enabled=true

Logs (from one pod, all 3 show the same pattern):

{"time":"2026-03-28T03:19:40.595013234Z","level":"INFO","msg":"skipping verification of containerd configuration"}
{"time":"2026-03-28T03:19:40.605749884Z","level":"INFO","msg":"starting p2p router","logger":"p2p","id":"12D3KooWN51k1SDTGTu3FtSvXopapPc3k9t2GDkrVTHDURsPgeaQ"}
{"time":"2026-03-28T03:19:40.605927582Z","level":"INFO","msg":"running Spegel","registry":":5000","router":":5001"}
{"time":"2026-03-28T03:19:40.634665457Z","level":"INFO","msg":"waiting for store events"}
{"time":"2026-03-28T03:19:50.609661248Z","level":"ERROR","msg":"failed to run bootstrap","logger":"p2p","err":"routing table is empty after bootstrapping","attempts":1}
{"time":"2026-03-28T03:20:00.674368879Z","level":"ERROR","msg":"failed to run bootstrap","logger":"p2p","err":"routing table is empty after bootstrapping","attempts":2}
{"time":"2026-03-28T03:20:10.723436888Z","level":"ERROR","msg":"failed to run bootstrap","logger":"p2p","err":"routing table is empty after bootstrapping","attempts":3}
{"time":"2026-03-28T03:20:21.016833924Z","level":"ERROR","msg":"failed to run bootstrap","logger":"p2p","err":"routing table is empty after bootstrapping","attempts":4}
{"time":"2026-03-28T03:20:31.035458863Z","level":"ERROR","msg":"failed to run bootstrap","logger":"p2p","err":"routing table is empty after bootstrapping","attempts":5}
{"time":"2026-03-28T03:20:41.259104375Z","level":"ERROR","msg":"failed to run bootstrap","logger":"p2p","err":"routing table is empty after bootstrapping","attempts":6}

The bootstrap retries every ~10s with exponential backoff, but never succeeds. After 60 failed startup probes (180s), the pod is killed and the cycle repeats.

Events:

Startup probe failed: HTTP probe failed with statuscode: 500
Container registry failed startup probe, will be restarted

Connectivity verification (from a test pod):

Testing 10.42.0.114:5001 → OK
Testing 10.42.1.242:5001 → OK
Testing 10.42.2.132:5001 → OK

DNS verification:

Name: spegel-bootstrap.kube-system.svc.cluster.local
Address: 10.42.0.114
Address: 10.42.2.132
Address: 10.42.1.242

DaemonSet status:

currentNumberScheduled: 3
desiredNumberScheduled: 3
numberReady: 0
numberUnavailable: 3

Startup probe config:

startupProbe:
  failureThreshold: 60
  httpGet:
    path: /readyz
    port: registry
    scheme: HTTP
  periodSeconds: 3

Expected behavior

The P2P routing table should eventually populate as pods discover each other via the DNS bootstrap service, and /readyz should return 200.

[phillebaba]

This is really weird. I can really tell with certainty what is going on. Could you try with v0.7.0 and see if this issue is still there?