Headplane Agent does requires a pre-auth key at boot due to non-persisted state

[RJDavison]

Description

headplane-agent fails after each reboot requiring new pre-auth keys

headplane | 2025-10-30T20:35:44.578Z [agent] INFO: Headplane agent started
headplane | 2025-10-30T20:35:44.632Z [agent] INFO: tsnet running state path /var/lib/headplane/agent/tailscaled.state
headplane | 2025-10-30T20:35:44.633Z [agent] INFO: tsnet starting with hostname "headplane-agent", varRoot "/var/lib/headplane/agent"
headplane | 2025-10-30T20:35:44.634Z [agent] INFO: Authkey is set; but state is NoState. Ignoring authkey. Re-run with TSNET_FORCE_LOGIN=1 to force use of authkey.
headplane | 2025-10-30T20:35:44.786Z [agent] WARN: Headplane agent exited with code 1 and signal undefined
headplane | 2025-10-30T20:35:44.787Z [agent] WARN: Headplane agent will restart in 296.514 seconds (attempt 40)

Headplane Version

0.6.1

Headscale Version

0.27.0

[tale]

This doesn't sound like a bug, it sounds like the pre-auth key you supplied is not reusable.

[LorenDB]

Why does the agent need a pre-auth key at every start? I'd think the agent could join the tailnet and then persist like a normal Tailscale connection.

[tale]

Is it not a reusable pre-auth-key? That should prevent a new one from being required constantly, along with making sure the folder that the Headplane agent uses to store its state is persisted if in Docker.

[LorenDB]

I don't think the reusable key is a good idea, because that technically opens a security hole to anyone who brute forces auth keys (plus I don't love having a persistent key hanging around in the UI).

[rhoriguchi]

I'm seeing the same issue but with ephemeral=0 and reusable=1 key.

Here is how create the key, so I'm sure that the key has the mentioned settings and is also always the same.
https://github.com/rhoriguchi/nixos-setup/blob/50e380f979912e8bd1a713c7626b766eced40e6d/configuration/devices/headless/nelliel/headscale/default.nix#L70-L88

Here is my headplane config: https://github.com/rhoriguchi/nixos-setup/blob/50e380f979912e8bd1a713c7626b766eced40e6d/configuration/devices/headless/nelliel/headscale/headplane.nix#L13-L49

> headscale version
headscale version 0.28.0
commit: v0.28.0
build time: unknown
built with: go1.25.5 linux/amd64

Headplane version: 931a7f8

Log output

Feb 09 16:09:19 XXLPitu-Nelliel systemd[1]: Started Headscale Web UI.
Feb 09 16:09:21 XXLPitu-Nelliel headplane[527405]: 2026-02-09T15:09:21.131Z [server] INFO: Running Node.js 22.22.0
Feb 09 16:09:21 XXLPitu-Nelliel headplane[527405]: (node:527405) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
Feb 09 16:09:21 XXLPitu-Nelliel headplane[527405]: (Use `node --trace-deprecation ...` to show where the warning was created)
Feb 09 16:09:21 XXLPitu-Nelliel headplane[527405]: 2026-02-09T15:09:21.279Z [agent] INFO: Using agent cache file at /var/lib/headplane/agent_cache.json
Feb 09 16:09:21 XXLPitu-Nelliel headplane[527405]: 2026-02-09T15:09:21.288Z [agent] INFO: Headplane agent started
Feb 09 16:09:21 XXLPitu-Nelliel headplane[527405]: 2026-02-09T15:09:21.315Z [agent] INFO: tsnet running state path /var/lib/headplane/agent/tailscaled.state
Feb 09 16:09:21 XXLPitu-Nelliel headplane[527405]: 2026-02-09T15:09:21.316Z [agent] INFO: tsnet starting with hostname "headplane-agent", varRoot "/var/lib/headplane/agent"
Feb 09 16:09:21 XXLPitu-Nelliel headplane[527405]: 2026-02-09T15:09:21.372Z [agent] INFO: Authkey is set; but state is NoState. Ignoring authkey. Re-run with TSNET_FORCE_LOGIN=1 to force use of authkey.
Feb 09 16:09:21 XXLPitu-Nelliel headplane[527405]: 2026-02-09T15:09:21.374Z [config] INFO: Found a valid Headscale configuration file at /nix/store/5v6xz6dw0j01j7bc7zhxdj1ggm0s3md6-headscale.yml
Feb 09 16:09:21 XXLPitu-Nelliel headplane[527405]: 2026-02-09T15:09:21.376Z [config] WARN: Headscale configuration file at /nix/store/5v6xz6dw0j01j7bc7zhxdj1ggm0s3md6-headscale.yml is not writable
Feb 09 16:09:21 XXLPitu-Nelliel headplane[527405]: 2026-02-09T15:09:21.479Z [config] ERROR: Error validating Headscale configuration:
Feb 09 16:09:21 XXLPitu-Nelliel headplane[527405]: 2026-02-09T15:09:21.479Z [config] ERROR:  - (0): dns.extra_records must be an array (was null)
Feb 09 16:09:21 XXLPitu-Nelliel headplane[527405]: 2026-02-09T15:09:21.480Z [config] INFO: Using Proc integration
Feb 09 16:09:21 XXLPitu-Nelliel headplane[527405]: 2026-02-09T15:09:21.515Z [config] INFO: Found headscale serve (PID 526936)
Feb 09 16:09:21 XXLPitu-Nelliel headplane[527405]: 2026-02-09T15:09:21.689Z [server] INFO: Running on 127.0.0.1:3001
Feb 09 16:09:22 XXLPitu-Nelliel headplane[527405]: 2026-02-09T15:09:22.973Z [agent] INFO: Connected to Tailnet (PublicKey: nodekey:f7bc17d48da95981caea9bced427e6073a1421b951cabf240e9d6cd8173e8b47)
Feb 09 16:09:22 XXLPitu-Nelliel headplane[527405]: 2026-02-09T15:09:22.974Z [agent] INFO: Headplane agent is ready and serving queries
Feb 09 16:09:26 XXLPitu-Nelliel headplane[527405]: 2026-02-09T15:09:26.347Z [agent] INFO: AuthLoop: state is Running; done
Feb 09 16:17:43 XXLPitu-Nelliel systemd[1]: Stopping Headscale Web UI...
Feb 09 16:17:43 XXLPitu-Nelliel headplane[527405]: 2026-02-09T15:17:43.439Z [server] INFO: Received SIGTERM, shutting down...
Feb 09 16:17:43 XXLPitu-Nelliel systemd[1]: headplane.service: Deactivated successfully.
Feb 09 16:17:43 XXLPitu-Nelliel systemd[1]: Stopped Headscale Web UI.
Feb 09 16:17:43 XXLPitu-Nelliel systemd[1]: headplane.service: Consumed 5.939s CPU time, 147.9M memory peak, 68K read from disk, 124K written to disk, 861.9K incoming IP traffic, 886.5K outgoing IP traffic.
Feb 09 16:17:43 XXLPitu-Nelliel systemd[1]: Started Headscale Web UI.
Feb 09 16:17:44 XXLPitu-Nelliel headplane[527884]: 2026-02-09T15:17:44.841Z [server] INFO: Running Node.js 22.22.0
Feb 09 16:17:44 XXLPitu-Nelliel headplane[527884]: (node:527884) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
Feb 09 16:17:44 XXLPitu-Nelliel headplane[527884]: (Use `node --trace-deprecation ...` to show where the warning was created)
Feb 09 16:17:44 XXLPitu-Nelliel headplane[527884]: 2026-02-09T15:17:44.972Z [agent] INFO: Using agent cache file at /var/lib/headplane/agent_cache.json
Feb 09 16:17:44 XXLPitu-Nelliel headplane[527884]: 2026-02-09T15:17:44.983Z [agent] INFO: Headplane agent started
Feb 09 16:17:45 XXLPitu-Nelliel headplane[527884]: 2026-02-09T15:17:45.069Z [agent] INFO: tsnet running state path /var/lib/headplane/agent/tailscaled.state
Feb 09 16:17:45 XXLPitu-Nelliel headplane[527884]: 2026-02-09T15:17:45.069Z [agent] INFO: tsnet starting with hostname "headplane-agent", varRoot "/var/lib/headplane/agent"
Feb 09 16:17:45 XXLPitu-Nelliel headplane[527884]: 2026-02-09T15:17:45.069Z [agent] INFO: Authkey is set; but state is NoState. Ignoring authkey. Re-run with TSNET_FORCE_LOGIN=1 to force use of authkey.
Feb 09 16:17:45 XXLPitu-Nelliel headplane[527884]: 2026-02-09T15:17:45.071Z [config] INFO: Found a valid Headscale configuration file at /nix/store/5v6xz6dw0j01j7bc7zhxdj1ggm0s3md6-headscale.yml
Feb 09 16:17:45 XXLPitu-Nelliel headplane[527884]: 2026-02-09T15:17:45.072Z [config] WARN: Headscale configuration file at /nix/store/5v6xz6dw0j01j7bc7zhxdj1ggm0s3md6-headscale.yml is not writable
Feb 09 16:17:45 XXLPitu-Nelliel headplane[527884]: 2026-02-09T15:17:45.120Z [config] ERROR: Error validating Headscale configuration:
Feb 09 16:17:45 XXLPitu-Nelliel headplane[527884]: 2026-02-09T15:17:45.121Z [config] ERROR:  - (0): dns.extra_records must be an array (was null)
Feb 09 16:17:45 XXLPitu-Nelliel headplane[527884]: 2026-02-09T15:17:45.121Z [config] INFO: Using Proc integration
Feb 09 16:17:45 XXLPitu-Nelliel headplane[527884]: 2026-02-09T15:17:45.159Z [config] INFO: Found headscale serve (PID 526936)
Feb 09 16:17:45 XXLPitu-Nelliel headplane[527884]: 2026-02-09T15:17:45.330Z [server] INFO: Running on 127.0.0.1:3001
Feb 09 16:17:47 XXLPitu-Nelliel headplane[527884]: 2026-02-09T15:17:47.253Z [agent] INFO: Connected to Tailnet (PublicKey: nodekey:f7bc17d48da95981caea9bced427e6073a1421b951cabf240e9d6cd8173e8b47)
Feb 09 16:17:47 XXLPitu-Nelliel headplane[527884]: 2026-02-09T15:17:47.256Z [agent] INFO: Headplane agent is ready and serving queries
Feb 09 16:17:50 XXLPitu-Nelliel headplane[527884]: 2026-02-09T15:17:50.047Z [agent] INFO: AuthLoop: state is Running; done

[WolfspiritM]

This problem only appears when the preauth key is tagged and the agent then belongs to the new "tagged-devices" service account. If I switch over to creating a personal preauth token for my account it works fine. It seems like the agent can't reconnect if it is tagged.

[rhoriguchi]

Hi @tale could you please take a look at this issue. It's kinda annoying to have a new agent entry after each service restart.

[tale]

Hey everybody, I'm well aware of this issue and I've been looking into it extensively. I think there's a few different problems here to deal with and I'm making progress on it for 0.7.0:

1. Auth keys should only be needed once. Tailscale's own docs say that tsnet services only need the auth key on first startup to join the tailnet and once they have persistent state the key is ignored entirely. My solution in 0.7.0 is to have Headplane automatically generate a non-ephemeral, non-reusable, tag-only key with a 5-minute expiry so the agent can complete the initial handshake and move on. No more manual key management required.

2. State persistence is properly handled. The tailscaled.state file in the agent work directory is preserved across restarts. On reboot, the agent reconnects using its existing identity without needing a new auth key. A fresh key is only generated if there's no state file at all (or if the agent cannot get data after more than 5 attempts). Importantly, it is up to the user to ensure this directory is persisted, because without it, Headscale will pick up the agent as a new node and Headplane will also generate a new pre-auth key because the state file was not persisted.

This changes will make the agent require Headscale 0.28+ since the agent uses tag-only pre-auth keys. I'm going to mark this as the answer for now and will keep everyone in the loop for 0.7.0 progress 🙂