k3s crash-loops with 100% CPU on boot when no default network route exists (no backoff, no restart limit)

[OscarAlmgren]

Summary

After a routine sudo apt upgrade on Ubuntu 25.10 (which updated netplan from 1.1.2-8ubuntu1~25.10.1 to ~25.10.2 and systemd from 257.9-0ubuntu2.1 to 257.9-0ubuntu2.3), rebooting the server caused k3s to enter an infinite crash loop at 100% CPU.

What happened

The netplan point-release changed the timing of network bring-up. k3s started before a default route existed in /proc/net/route, immediately fatally crashed, and systemd restarted it every 5 seconds — indefinitely. The service reached 100+ restart cycles, consuming all CPU, making the server completely unresponsive (SSH unreachable, all other services starved).

Fatal error (repeated every 5s)

level=fatal msg="Error: no default routes found in \"/proc/net/route\" or \"/proc/net/ipv6_route\""

Root issues

1. k3s ships with Wants=network-online.target — this is a soft dependency. If systemd-networkd-wait-online completes in a degraded state or the boot sequence races, k3s starts anyway and crashes.
2. No restart backoff or burst limit — Restart=always + RestartSec=5s with no StartLimitBurst means systemd will loop forever, pegging CPU.
3. A fatal "no route" error should not cause a tight crash loop — k3s could wait/retry internally instead of exiting immediately.

Expected behavior

• k3s should use Requires=network-online.target (hard dependency), OR
• Apply exponential backoff on restart, OR
• Internally retry route detection before fatally exiting

Workaround applied

Created a systemd drop-in override at /etc/systemd/system/k3s.service.d/10-network-wait.conf:

[Unit]
Requires=network-online.target

[Service]
RestartSec=30s
StartLimitBurst=3
StartLimitIntervalSec=300

Environment

• Ubuntu 25.10 (oracular)
• k3s (installed via official script)
• Kernel: 6.17.0-20-generic
• netplan: 1.1.2-8ubuntu1~25.10.2
• systemd: 257.9-0ubuntu2.3

---

Reported by a frustrated human user who lost several hours to this after a routine point-release upgrade. Discovered and diagnosed using Claude Code.

[brandond]

PR welcome

[sRiSh-JaNa-tech]

Hi team, I've been looking into #13895 and reviewing the approach in PR #14420.

I agree with the concerns raised on the PR—altering the systemd RestartSec slows down normal crash recovery, and relying on StartLimitBurst risks leaving K3s in a permanently failed state on Edge/IoT devices where the network might legitimately take longer than 15 seconds to initialize.

Instead of modifying the systemd unit, a safer fix would be to handle this gracefully inside K3s. I propose updating pkg/util/net.go (GetHostnameAndIPs) to wrap the apinet.ChooseHostInterface() call in a short retry loop with a timeout (e.g., polling for up to 60 seconds). This would eliminate the 100% CPU crash-loop, preserve the 5-second recovery for normal crashes, and gracefully accommodate slow-booting networks.

Would you be open to this approach? If so, I'd be happy to put together a PR for it!

[brandond]

Show me what you got.