Skip to content

Commit d07245b

Browse files
authored
feat(autotls): skip issuance when broker is down (#11397)
* feat(autotls): skip issuance when broker is down Bump github.com/ipshipyard/p2p-forge to the head of ipshipyard/p2p-forge#91: before first-time ACME issuance the client now confirms the broker responds with HTTP 204 on /v1/health, after the registration delay and once the node is publicly reachable. While the broker keeps failing the check, certificate setup is postponed with one ERROR and an hourly re-check (respecting Retry-After, capped at 24h) instead of doomed ACME retries for weeks. Ephemeral nodes (CI runners) still produce no broker traffic at all, and nodes with a certificate in storage are unaffected. * chore: update p2p-forge to v0.10.0
1 parent 4d21088 commit d07245b

9 files changed

Lines changed: 156 additions & 10 deletions

File tree

docs/changelogs/v0.43.md

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ This release was brought to you by the [Shipyard](https://ipshipyard.com/) team.
1212
- [🔦 Highlights](#-highlights)
1313
- [🔗 Native `ipfs://` and `ipns://` URIs work as input](#-native-ipfs-and-ipns-uris-work-as-input)
1414
- [🛜 One-time notice when behind CGNAT](#-one-time-notice-when-behind-cgnat)
15+
- [🩺 AutoTLS checks broker health before registration](#-autotls-checks-broker-health-before-registration)
1516
- [🐛 IPNS publishing validates lifetime and TTL](#-ipns-publishing-validates-lifetime-and-ttl)
1617
- [🛑 Clearer errors for invalid config at startup](#-clearer-errors-for-invalid-config-at-startup)
1718
- [🗂️ `ipfs files` no longer hangs when garbage collection runs](#-ipfs-files-no-longer-hangs-when-garbage-collection-runs)
@@ -49,6 +50,12 @@ Detection is best-effort and conservative: it fires only when a private or share
4950

5051
Silence the notice with [`Internal.CGNATCheck`](https://github.com/ipfs/kubo/blob/master/docs/config.md#internalcgnatcheck)`=false`. The dead-listener diagnostic added in v0.42 can now be toggled too, with [`Internal.DeadListenerCheck`](https://github.com/ipfs/kubo/blob/master/docs/config.md#internaldeadlistenercheck).
5152

53+
#### 🩺 AutoTLS checks broker health before registration
54+
55+
AutoTLS certificate issuance depends on the ACME DNS-01 broker at [`AutoTLS.RegistrationEndpoint`](https://github.com/ipfs/kubo/blob/master/docs/config.md#autotlsregistrationendpoint) (`registration.libp2p.direct` by default). Before, a publicly reachable node without a certificate would attempt ACME issuance even when that broker was unreachable (offline network, firewall, service outage) and keep retrying in the background for days, filling logs with errors that could not resolve themselves.
56+
57+
Now the broker's health endpoint is checked right before the first registration attempt, after the registration delay (1h by default, none when `AutoTLS.Enabled=true` is set explicitly) and once the node is publicly reachable. While the broker keeps failing the check, certificate setup is postponed with a single ERROR in the log and one cheap re-check per hour, and issuance starts automatically once the broker recovers. Nodes that already have a certificate are unaffected, and short-lived nodes (such as CI runners) produce no broker traffic at all.
58+
5259
#### 🐛 IPNS publishing validates lifetime and TTL
5360

5461
`ipfs name publish` now sanitizes its duration flags before creating an [IPNS record](https://specs.ipfs.tech/ipns/ipns-record/), instead of emitting one that fails verification later:
@@ -155,7 +162,7 @@ Removing files through the selection toolbar left their pins behind. It now offe
155162
- update `go-libp2p` to a pre-release pinned at [95be6665](https://github.com/libp2p/go-libp2p/commit/95be6665b014e3e29bdb639ec10f8944d141969d), ahead of v0.48.0 (no tagged release yet)
156163
- update `go-libp2p-pubsub` to [v0.17.0](https://github.com/libp2p/go-libp2p-pubsub/releases/tag/v0.17.0)
157164
- update `go-libp2p-kad-dht` to [v0.41.0](https://github.com/libp2p/go-libp2p-kad-dht/releases/tag/v0.41.0)
158-
- update `p2p-forge/client` to [v0.9.1](https://github.com/ipshipyard/p2p-forge/releases/tag/v0.9.1) (incl. [v0.9.0](https://github.com/ipshipyard/p2p-forge/releases/tag/v0.9.0), [v0.8.1](https://github.com/ipshipyard/p2p-forge/releases/tag/v0.8.1))
165+
- update `p2p-forge/client` to [v0.10.0](https://github.com/ipshipyard/p2p-forge/releases/tag/v0.10.0) (incl. [v0.9.1](https://github.com/ipshipyard/p2p-forge/releases/tag/v0.9.1), [v0.9.0](https://github.com/ipshipyard/p2p-forge/releases/tag/v0.9.0), [v0.8.1](https://github.com/ipshipyard/p2p-forge/releases/tag/v0.8.1))
159166
- update `go-ds-pebble` to [v0.5.12](https://github.com/ipfs/go-ds-pebble/releases/tag/v0.5.12)
160167
- updates `github.com/cockroachdb/pebble` to [v2.1.6](https://github.com/cockroachdb/pebble/releases/tag/v2.1.6)
161168

docs/config.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -774,6 +774,7 @@ When activated, together with [`AutoTLS.AutoWSS`](#autotlsautowss) (default) or
774774
**Note:**
775775

776776
- This feature requires a publicly reachable node. If behind NAT, manual port forwarding or UPnP (`Swarm.DisableNatPortMap=false`) is required.
777+
- On a node without a certificate yet, broker health is confirmed before the first registration attempt (HTTP GET of `/v1/health` at [`AutoTLS.RegistrationEndpoint`](#autotlsregistrationendpoint) must return HTTP 204). The check runs after [`AutoTLS.RegistrationDelay`](#autotlsregistrationdelay), once the node is publicly reachable. While the broker keeps failing the check, certificate setup is postponed and health is re-checked hourly; issuance starts automatically once the broker recovers.
777778
- The first time AutoTLS is used, it may take 5-15 minutes + [`AutoTLS.RegistrationDelay`](#autotlsregistrationdelay) before `/ws` listener is added. Be patient.
778779
- Avoid manual configuration. [`AutoTLS.AutoWSS=true`](#autotlsautowss) should automatically add `/ws` listener to existing, firewall-forwarded `/tcp` ports.
779780
- To troubleshoot, use `GOLOG_LOG_LEVEL="error,autotls=debug` for detailed logs, or `GOLOG_LOG_LEVEL="error,autotls=info` for quieter output.

docs/examples/kubo-as-a-library/go.mod

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,7 @@ require (
104104
github.com/ipld/go-car/v2 v2.17.0 // indirect
105105
github.com/ipld/go-codec-dagpb v1.7.0 // indirect
106106
github.com/ipld/go-ipld-prime v0.24.0 // indirect
107-
github.com/ipshipyard/p2p-forge v0.9.1 // indirect
107+
github.com/ipshipyard/p2p-forge v0.10.0 // indirect
108108
github.com/jackpal/go-nat-pmp v1.0.2 // indirect
109109
github.com/jbenet/go-temp-err-catcher v0.1.0 // indirect
110110
github.com/klauspost/compress v1.18.4 // indirect

docs/examples/kubo-as-a-library/go.sum

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -352,8 +352,8 @@ github.com/ipld/go-ipld-prime v0.24.0 h1:6th8Z6Peh5bCWuRAVZcDO1sHzZdVF6F2cCCDG36
352352
github.com/ipld/go-ipld-prime v0.24.0/go.mod h1:DYZxr/5caLNFbcuU6zLOgwSW7CgUEoC4wJiZMEU8Zhs=
353353
github.com/ipld/go-ipld-prime/storage/bsadapter v0.0.0-20250821084354-a425e60cd714 h1:cqNk8PEwHnK0vqWln+U/YZhQc9h2NB3KjUjDPZo5Q2s=
354354
github.com/ipld/go-ipld-prime/storage/bsadapter v0.0.0-20250821084354-a425e60cd714/go.mod h1:ZEUdra3CoqRVRYgAX/jAJO9aZGz6SKtKEG628fHHktY=
355-
github.com/ipshipyard/p2p-forge v0.9.1 h1:xHiauwYsHv8R6jIaXDV6e89xAeSS6tN++uMQuAC41Jw=
356-
github.com/ipshipyard/p2p-forge v0.9.1/go.mod h1:1keK1MRRCu5oNe9uFKfNIIZXOFEF9hgD1iK1DUsjsXQ=
355+
github.com/ipshipyard/p2p-forge v0.10.0 h1:OcfcyjoHkeiIskhT4fHN/N82HfpgAteb1Eg3ZNAGE/U=
356+
github.com/ipshipyard/p2p-forge v0.10.0/go.mod h1:1keK1MRRCu5oNe9uFKfNIIZXOFEF9hgD1iK1DUsjsXQ=
357357
github.com/jackpal/go-nat-pmp v1.0.2 h1:KzKSgb7qkJvOUTqYl9/Hg/me3pWgBmERKrTGD7BdWus=
358358
github.com/jackpal/go-nat-pmp v1.0.2/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc=
359359
github.com/jbenet/go-cienv v0.1.0/go.mod h1:TqNnHUmJgXau0nCzC7kXWeotg3J9W34CUv5Djy1+FlA=

go.mod

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ require (
4747
github.com/ipld/go-car/v2 v2.17.0
4848
github.com/ipld/go-codec-dagpb v1.7.0
4949
github.com/ipld/go-ipld-prime v0.24.0
50-
github.com/ipshipyard/p2p-forge v0.9.1
50+
github.com/ipshipyard/p2p-forge v0.10.0
5151
github.com/jbenet/go-temp-err-catcher v0.1.0
5252
github.com/julienschmidt/httprouter v1.3.0
5353
github.com/libp2p/go-doh-resolver v0.5.0

go.sum

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -425,8 +425,8 @@ github.com/ipld/go-ipld-prime v0.24.0 h1:6th8Z6Peh5bCWuRAVZcDO1sHzZdVF6F2cCCDG36
425425
github.com/ipld/go-ipld-prime v0.24.0/go.mod h1:DYZxr/5caLNFbcuU6zLOgwSW7CgUEoC4wJiZMEU8Zhs=
426426
github.com/ipld/go-ipld-prime/storage/bsadapter v0.0.0-20250821084354-a425e60cd714 h1:cqNk8PEwHnK0vqWln+U/YZhQc9h2NB3KjUjDPZo5Q2s=
427427
github.com/ipld/go-ipld-prime/storage/bsadapter v0.0.0-20250821084354-a425e60cd714/go.mod h1:ZEUdra3CoqRVRYgAX/jAJO9aZGz6SKtKEG628fHHktY=
428-
github.com/ipshipyard/p2p-forge v0.9.1 h1:xHiauwYsHv8R6jIaXDV6e89xAeSS6tN++uMQuAC41Jw=
429-
github.com/ipshipyard/p2p-forge v0.9.1/go.mod h1:1keK1MRRCu5oNe9uFKfNIIZXOFEF9hgD1iK1DUsjsXQ=
428+
github.com/ipshipyard/p2p-forge v0.10.0 h1:OcfcyjoHkeiIskhT4fHN/N82HfpgAteb1Eg3ZNAGE/U=
429+
github.com/ipshipyard/p2p-forge v0.10.0/go.mod h1:1keK1MRRCu5oNe9uFKfNIIZXOFEF9hgD1iK1DUsjsXQ=
430430
github.com/jackpal/go-nat-pmp v1.0.2 h1:KzKSgb7qkJvOUTqYl9/Hg/me3pWgBmERKrTGD7BdWus=
431431
github.com/jackpal/go-nat-pmp v1.0.2/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc=
432432
github.com/jbenet/go-cienv v0.1.0/go.mod h1:TqNnHUmJgXau0nCzC7kXWeotg3J9W34CUv5Djy1+FlA=

test/cli/autotls_test.go

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
package cli
2+
3+
import (
4+
"net/http"
5+
"net/http/httptest"
6+
"sync/atomic"
7+
"testing"
8+
"time"
9+
10+
"github.com/ipfs/kubo/config"
11+
"github.com/ipfs/kubo/test/cli/harness"
12+
"github.com/stretchr/testify/require"
13+
)
14+
15+
// autotlsPostponedMsg is the marker of the ERROR the p2p-forge client logs
16+
// when its pre-issuance broker health check fails and certificate setup is
17+
// postponed until the broker recovers.
18+
const autotlsPostponedMsg = "certificate setup postponed"
19+
20+
// wssWildcardFragment appears in swarm listen addrs only when the AutoTLS
21+
// machinery was wired up (AutoWSS appended the wildcard WSS listener).
22+
const wssWildcardFragment = "/tls/sni/"
23+
24+
// unroutableURL is guaranteed to refuse connections without depending on the
25+
// state of any real port (port 0 is never connectable).
26+
const unroutableURL = "http://127.0.0.1:0"
27+
28+
// countingBroker is a fake p2p-forge broker that records /v1/health probes.
29+
func countingBroker(t *testing.T, status int) (*httptest.Server, *atomic.Int32) {
30+
t.Helper()
31+
var probes atomic.Int32
32+
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
33+
if r.URL.Path == "/v1/health" {
34+
probes.Add(1)
35+
w.WriteHeader(status)
36+
return
37+
}
38+
w.WriteHeader(http.StatusNotFound)
39+
}))
40+
t.Cleanup(srv.Close)
41+
return srv, &probes
42+
}
43+
44+
// autotlsNode inits a node with AutoTLS explicitly enabled against the given
45+
// broker URL. Forced public reachability plus an announced public address
46+
// satisfy the conditions the p2p-forge client waits for before it contacts
47+
// the broker, so the pre-issuance health check can be observed in an
48+
// isolated test. CAEndpoint points at an unroutable address so no test can
49+
// ever reach a real ACME CA, even when issuance starts. GOLOG env is pinned
50+
// so stderr assertions do not depend on ambient GOLOG_* variables.
51+
func autotlsNode(t *testing.T, brokerURL string) *harness.Node {
52+
node := harness.NewT(t).NewNode().Init("--profile=test")
53+
node.Runner.Env["GOLOG_LOG_LEVEL"] = "error"
54+
node.Runner.Env["GOLOG_OUTPUT"] = "stderr"
55+
node.UpdateConfig(func(cfg *config.Config) {
56+
cfg.AutoTLS.Enabled = config.True
57+
cfg.AutoTLS.RegistrationEndpoint = config.NewOptionalString(brokerURL)
58+
cfg.AutoTLS.CAEndpoint = config.NewOptionalString(unroutableURL)
59+
cfg.Internal.Libp2pForceReachability = config.NewOptionalString("public")
60+
cfg.Addresses.Announce = []string{"/ip4/1.2.3.4/tcp/4001"}
61+
})
62+
return node
63+
}
64+
65+
func TestAutoTLSBrokerHealthCheck(t *testing.T) {
66+
t.Parallel()
67+
68+
t.Run("registration delay defers any broker traffic", func(t *testing.T) {
69+
t.Parallel()
70+
broker, probes := countingBroker(t, http.StatusNoContent)
71+
72+
node := autotlsNode(t, broker.URL)
73+
node.UpdateConfig(func(cfg *config.Config) {
74+
// implicit enable: the default 1h registration delay applies, so
75+
// an ephemeral node like this one must produce no broker traffic
76+
// at all, even while it looks publicly reachable
77+
cfg.AutoTLS.Enabled = config.Default
78+
})
79+
node.StartDaemon()
80+
defer node.StopDaemon()
81+
82+
require.Never(t, func() bool { return probes.Load() > 0 }, 2*time.Second, 100*time.Millisecond,
83+
"daemon must not contact broker before the registration delay")
84+
require.NotContains(t, node.Daemon.Stderr.String(), autotlsPostponedMsg)
85+
})
86+
87+
t.Run("postpones issuance while broker is unhealthy", func(t *testing.T) {
88+
t.Parallel()
89+
broker, probes := countingBroker(t, http.StatusServiceUnavailable)
90+
91+
node := autotlsNode(t, broker.URL)
92+
node.UpdateConfig(func(cfg *config.Config) {
93+
cfg.AutoTLS.RegistrationDelay = config.NewOptionalDuration(1 * time.Second)
94+
})
95+
node.StartDaemon()
96+
defer node.StopDaemon()
97+
98+
require.True(t, waitForLogMessage(node.Daemon.Stderr, autotlsPostponedMsg, 15*time.Second),
99+
"p2p-forge client should postpone certificate setup when broker is unhealthy")
100+
require.GreaterOrEqual(t, probes.Load(), int32(1))
101+
})
102+
103+
t.Run("proceeds with issuance when broker is healthy", func(t *testing.T) {
104+
t.Parallel()
105+
broker, probes := countingBroker(t, http.StatusNoContent)
106+
107+
node := autotlsNode(t, broker.URL)
108+
node.UpdateConfig(func(cfg *config.Config) {
109+
cfg.AutoTLS.RegistrationDelay = config.NewOptionalDuration(1 * time.Second)
110+
})
111+
node.StartDaemon()
112+
defer node.StopDaemon()
113+
114+
require.Eventually(t, func() bool { return probes.Load() >= 1 }, 15*time.Second, 100*time.Millisecond,
115+
"p2p-forge client should probe broker health before issuance")
116+
require.NotContains(t, node.Daemon.Stderr.String(), autotlsPostponedMsg)
117+
118+
// AutoTLS machinery is wired up: AutoWSS added the wildcard listener
119+
listenAddrs := node.IPFS("swarm", "addrs", "listen").Stdout.String()
120+
require.Contains(t, listenAddrs, wssWildcardFragment)
121+
})
122+
123+
t.Run("explicit enable goes through the same health check", func(t *testing.T) {
124+
t.Parallel()
125+
broker, probes := countingBroker(t, http.StatusServiceUnavailable)
126+
127+
// AutoTLS.Enabled=true with no custom delay means a zero registration
128+
// delay: the health check runs as soon as the node looks publicly
129+
// reachable, through the same code path as the delayed flow
130+
node := autotlsNode(t, broker.URL)
131+
node.StartDaemon()
132+
defer node.StopDaemon()
133+
134+
require.True(t, waitForLogMessage(node.Daemon.Stderr, autotlsPostponedMsg, 15*time.Second),
135+
"explicitly enabled AutoTLS should postpone certificate setup when broker is unhealthy")
136+
require.GreaterOrEqual(t, probes.Load(), int32(1))
137+
})
138+
}

test/dependencies/go.mod

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -154,7 +154,7 @@ require (
154154
github.com/ipld/go-car/v2 v2.17.0 // indirect
155155
github.com/ipld/go-codec-dagpb v1.7.0 // indirect
156156
github.com/ipld/go-ipld-prime v0.24.0 // indirect
157-
github.com/ipshipyard/p2p-forge v0.9.1 // indirect
157+
github.com/ipshipyard/p2p-forge v0.10.0 // indirect
158158
github.com/jackpal/go-nat-pmp v1.0.2 // indirect
159159
github.com/jbenet/go-temp-err-catcher v0.1.0 // indirect
160160
github.com/jgautheron/goconst v1.7.1 // indirect

test/dependencies/go.sum

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -357,8 +357,8 @@ github.com/ipld/go-ipld-prime v0.24.0 h1:6th8Z6Peh5bCWuRAVZcDO1sHzZdVF6F2cCCDG36
357357
github.com/ipld/go-ipld-prime v0.24.0/go.mod h1:DYZxr/5caLNFbcuU6zLOgwSW7CgUEoC4wJiZMEU8Zhs=
358358
github.com/ipld/go-ipld-prime/storage/bsadapter v0.0.0-20250821084354-a425e60cd714 h1:cqNk8PEwHnK0vqWln+U/YZhQc9h2NB3KjUjDPZo5Q2s=
359359
github.com/ipld/go-ipld-prime/storage/bsadapter v0.0.0-20250821084354-a425e60cd714/go.mod h1:ZEUdra3CoqRVRYgAX/jAJO9aZGz6SKtKEG628fHHktY=
360-
github.com/ipshipyard/p2p-forge v0.9.1 h1:xHiauwYsHv8R6jIaXDV6e89xAeSS6tN++uMQuAC41Jw=
361-
github.com/ipshipyard/p2p-forge v0.9.1/go.mod h1:1keK1MRRCu5oNe9uFKfNIIZXOFEF9hgD1iK1DUsjsXQ=
360+
github.com/ipshipyard/p2p-forge v0.10.0 h1:OcfcyjoHkeiIskhT4fHN/N82HfpgAteb1Eg3ZNAGE/U=
361+
github.com/ipshipyard/p2p-forge v0.10.0/go.mod h1:1keK1MRRCu5oNe9uFKfNIIZXOFEF9hgD1iK1DUsjsXQ=
362362
github.com/jackpal/go-nat-pmp v1.0.2 h1:KzKSgb7qkJvOUTqYl9/Hg/me3pWgBmERKrTGD7BdWus=
363363
github.com/jackpal/go-nat-pmp v1.0.2/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc=
364364
github.com/jbenet/go-temp-err-catcher v0.1.0 h1:zpb3ZH6wIE8Shj2sKS+khgRvf7T7RABoLk/+KKHggpk=

0 commit comments

Comments
 (0)