Failing to invalidate sessions again

[xamindar]

What happened?

I have re-organized my piholes and added some more. It worked for a little bit but then started failing again. I have tried recreating nebula-sync but it is just unable to invalidate sessions so they stack up until full.

2025-12-24T11:17:25-06:00 INF Invalidating sessions... 2025-12-24T11:17:27-06:00 WRN Failed to invalidate session for target: http://172.16.8.250 2025-12-24T11:17:29-06:00 WRN Failed to invalidate session for target: http://pihole1-1.pihole1-headless.pihole.svc.cluster.local 2025-12-24T11:17:31-06:00 WRN Failed to invalidate session for target: http://pihole2-0.pihole2-headless.pihole.svc.cluster.local 2025-12-24T11:17:33-06:00 WRN Failed to invalidate session for target: http://pihole2-1.pihole2-headless.pihole.svc.cluster.local 2025-12-24T11:17:33-06:00 FTL Sync failed error="authenticate: http://172.16.8.250/api/auth: unexpected status code: 429"

Expected Behaviour

It should successfully invalidate or somehow wipe out existing sessions so they do not fill up.

Nebula-sync Version

v0.11.1

Runtime

docker

Operating System

pi os bullseye

CPU Architecture

arm64

Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: nebula-sync-config
  namespace: pihole
data:
  PRIMARY: 'http://pihole1-0.pihole1-headless.pihole.svc.cluster.local|[REDACTED]'
  REPLICAS: 'http://172.16.8.250|[REDACTED],http://pihole1-1.pihole1-headless.pihole.svc.cluster.local|[REDACTED],http://pihole2-0.pihole2-headless.pihole.svc.cluster.local|[REDACTED],http://pihole2-1.pihole2-headless.pihole.svc.cluster.local|[REDACTED]'
  FULL_SYNC: 'true'
  RUN_GRAVITY: 'false'
  CRON: '*/10 * * * *'
  CLIENT_SKIP_TLS_VERIFICATION: 'true'
  TZ: 'America/Chicago'

Relevant log output

2025-12-24T11:24:36-06:00 INF Starting nebula-sync v0.11.1
2025-12-24T11:24:36-06:00 INF Running sync mode=full replicas=4
2025-12-24T11:24:36-06:00 INF Authenticating clients...
2025-12-24T11:24:45-06:00 INF Invalidating sessions...
2025-12-24T11:24:47-06:00 WRN Failed to invalidate session for target: http://172.16.8.250
2025-12-24T11:24:49-06:00 WRN Failed to invalidate session for target: http://pihole1-1.pihole1-headless.pihole.svc.cluster.local
2025-12-24T11:24:51-06:00 WRN Failed to invalidate session for target: http://pihole2-0.pihole2-headless.pihole.svc.cluster.local
2025-12-24T11:24:53-06:00 WRN Failed to invalidate session for target: http://pihole2-1.pihole2-headless.pihole.svc.cluster.local
2025-12-24T11:24:53-06:00 FTL Sync failed error="authenticate: http://172.16.8.250/api/auth: unexpected status code: 429"

Pi-hole Version

Docker Tag 2025.11.1Core v6.3FTL v6.4.1Web interface v6.4

[xamindar]

I guess this can be closed. I reset all my sessions on the piholes and adjusted the session lengths to be shorter.

However, nebula-sync seems to completely fail if only one pihole rejects it. It would be really helpful if it still continued to sync to the piholes that do still work, instead of just failing.

[xamindar]

I just want to add some more info in case it can help improve nebula-sync. It does fail to invalidate sessions sometimes. It is not always the same ones that fail so possibly it doesn't wait long enough before declaring a fail?

2025-12-24T22:50:04-06:00 INF Invalidating sessions...
2025-12-24T22:50:06-06:00 WRN Failed to invalidate session for target: http://pihole1-1.pihole1-headless.pihole.svc.cluster.local
2025-12-24T22:50:11-06:00 WRN Failed to invalidate session for target: http://172.16.8.250
2025-12-24T22:50:11-06:00 INF Sync completed

[Torstein-Eide]

I think it something how the ´Pihole web sever work´ I get the following:

nebula-sync, sucess to invalidate session for target

#./nebula-sync run --env-file .env
2026-01-03T14:26:22+01:00 INF Starting nebula-sync v0.11.1
2026-01-03T14:26:22+01:00 INF Running sync mode=selective replicas=1
2026-01-03T14:26:22+01:00 INF Authenticating clients...
14:26:22.249270 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [S], seq 3936918811, win 64240, options [mss 1460,sackOK,TS val 1147335126 ecr 0,nop,wscale 7], length 0
14:26:22.249843 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [S.], seq 1404521086, ack 3936918812, win 65160, options [mss 1460,sackOK,TS val 4276236223 ecr 1147335126,nop,wscale 7], length 0
14:26:22.249869 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [.], ack 1, win 502, options [nop,nop,TS val 1147335127 ecr 4276236223], length 0
14:26:22.250322 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [P.], seq 1:1454, ack 1, win 502, options [nop,nop,TS val 1147335127 ecr 4276236223], length 1453
14:26:22.250804 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [.], ack 1449, win 498, options [nop,nop,TS val 4276236224 ecr 1147335127], length 0
14:26:22.250838 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [.], ack 1454, win 498, options [nop,nop,TS val 4276236224 ecr 1147335127], length 0
14:26:22.293397 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [P.], seq 1:128, ack 1454, win 498, options [nop,nop,TS val 4276236267 ecr 1147335127], length 127
14:26:22.293415 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [.], ack 128, win 502, options [nop,nop,TS val 1147335170 ecr 4276236267], length 0
14:26:22.293886 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [P.], seq 128:134, ack 1454, win 498, options [nop,nop,TS val 4276236267 ecr 1147335170], length 6
14:26:22.293893 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [.], ack 134, win 502, options [nop,nop,TS val 1147335171 ecr 4276236267], length 0
14:26:22.294262 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [P.], seq 134:171, ack 1454, win 498, options [nop,nop,TS val 4276236268 ecr 1147335171], length 37
14:26:22.294267 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [.], ack 171, win 502, options [nop,nop,TS val 1147335171 ecr 4276236268], length 0
14:26:22.297236 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [P.], seq 171:704, ack 1454, win 498, options [nop,nop,TS val 4276236271 ecr 1147335171], length 533
14:26:22.297244 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [.], ack 704, win 501, options [nop,nop,TS val 1147335174 ecr 4276236271], length 0
14:26:22.309753 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [P.], seq 704:837, ack 1454, win 498, options [nop,nop,TS val 4276236283 ecr 1147335174], length 133
14:26:22.309766 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [.], ack 837, win 501, options [nop,nop,TS val 1147335186 ecr 4276236283], length 0
14:26:22.310597 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [P.], seq 837:906, ack 1454, win 498, options [nop,nop,TS val 4276236284 ecr 1147335186], length 69
14:26:22.310605 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [.], ack 906, win 501, options [nop,nop,TS val 1147335187 ecr 4276236284], length 0
14:26:22.311506 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [P.], seq 1454:1518, ack 906, win 501, options [nop,nop,TS val 1147335188 ecr 4276236284], length 64
14:26:22.311653 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [P.], seq 1518:1733, ack 906, win 501, options [nop,nop,TS val 1147335188 ecr 4276236284], length 215
14:26:22.311911 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [.], ack 1518, win 498, options [nop,nop,TS val 4276236285 ecr 1147335188], length 0
14:26:22.311965 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [.], ack 1733, win 497, options [nop,nop,TS val 4276236285 ecr 1147335188], length 0
2026-01-03T14:26:24+01:00 INF Syncing teleporters...
14:26:24.411584 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [P.], seq 906:959, ack 1733, win 497, options [nop,nop,TS val 4276238385 ecr 1147335188], length 53
14:26:24.451868 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [.], ack 959, win 501, options [nop,nop,TS val 1147337329 ecr 4276238385], length 0
14:26:24.452503 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [P.], seq 959:2281, ack 1733, win 497, options [nop,nop,TS val 4276238426 ecr 1147337329], length 1322
14:26:24.452524 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [.], ack 2281, win 524, options [nop,nop,TS val 1147337329 ecr 4276238426], length 0
2026-01-03T14:26:24+01:00 INF Syncing configs...
14:26:24.467275 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [P.], seq 1733:4127, ack 2281, win 524, options [nop,nop,TS val 1147337344 ecr 4276238426], length 2394
14:26:24.467305 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [P.], seq 4127:5873, ack 2281, win 524, options [nop,nop,TS val 1147337344 ecr 4276238426], length 1746
14:26:24.467356 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [P.], seq 5873:10639, ack 2281, win 524, options [nop,nop,TS val 1147337344 ecr 4276238426], length 4766
14:26:24.467383 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [P.], seq 10639:13535, ack 2281, win 524, options [nop,nop,TS val 1147337344 ecr 4276238426], length 2896
14:26:24.467746 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [.], ack 3181, win 486, options [nop,nop,TS val 4276238441 ecr 1147337344], length 0
14:26:24.467781 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [.], ack 4127, win 479, options [nop,nop,TS val 4276238441 ecr 1147337344], length 0
14:26:24.467789 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [P.], seq 13535:19327, ack 2281, win 524, options [nop,nop,TS val 1147337344 ecr 4276238441], length 5792
14:26:24.467938 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [.], ack 5873, win 469, options [nop,nop,TS val 4276238441 ecr 1147337344], length 0
14:26:24.468088 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [.], ack 7321, win 459, options [nop,nop,TS val 4276238442 ecr 1147337344], length 0
14:26:24.468093 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [P.], seq 19327:28015, ack 2281, win 524, options [nop,nop,TS val 1147337345 ecr 4276238442], length 8688
14:26:24.468224 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [.], ack 8769, win 448, options [nop,nop,TS val 4276238442 ecr 1147337344], length 0
14:26:24.468344 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [.], ack 10217, win 438, options [nop,nop,TS val 4276238442 ecr 1147337344], length 0
14:26:24.468382 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [.], ack 10639, win 435, options [nop,nop,TS val 4276238442 ecr 1147337344], length 0
14:26:24.468387 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [P.], seq 28015:29736, ack 2281, win 524, options [nop,nop,TS val 1147337345 ecr 4276238442], length 1721
14:26:24.468536 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [.], ack 12087, win 424, options [nop,nop,TS val 4276238442 ecr 1147337344], length 0
14:26:24.468642 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [.], ack 13535, win 413, options [nop,nop,TS val 4276238442 ecr 1147337344], length 0
14:26:24.468764 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [.], ack 14983, win 402, options [nop,nop,TS val 4276238442 ecr 1147337344], length 0
14:26:24.468881 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [.], ack 16431, win 391, options [nop,nop,TS val 4276238442 ecr 1147337344], length 0
14:26:24.468996 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [.], ack 17879, win 380, options [nop,nop,TS val 4276238442 ecr 1147337344], length 0
14:26:24.469135 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [.], ack 19327, win 369, options [nop,nop,TS val 4276238443 ecr 1147337344], length 0
14:26:24.469234 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [.], ack 20775, win 358, options [nop,nop,TS val 4276238443 ecr 1147337345], length 0
14:26:24.469403 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [.], ack 22223, win 347, options [nop,nop,TS val 4276238443 ecr 1147337345], length 0
14:26:24.469605 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [.], ack 23671, win 336, options [nop,nop,TS val 4276238443 ecr 1147337345], length 0
14:26:24.470106 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [.], ack 29736, win 289, options [nop,nop,TS val 4276238443 ecr 1147337345], length 0
14:26:24.529567 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [P.], seq 2281:2334, ack 29736, win 480, options [nop,nop,TS val 4276238503 ecr 1147337345], length 53
14:26:24.532287 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [.], seq 2334:3782, ack 29736, win 480, options [nop,nop,TS val 4276238506 ecr 1147337345], length 1448
14:26:24.532304 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [.], ack 3782, win 547, options [nop,nop,TS val 1147337409 ecr 4276238503], length 0
14:26:24.532408 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [P.], seq 29736:29760, ack 3782, win 547, options [nop,nop,TS val 1147337409 ecr 4276238503], length 24
14:26:24.532443 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [R.], seq 29760, ack 3782, win 547, options [nop,nop,TS val 1147337409 ecr 4276238503], length 0
14:26:24.533242 IP 192.168.2.10.443 > 192.168.3.2.51374: Flags [P.], seq 3782:3816, ack 29736, win 480, options [nop,nop,TS val 4276238506 ecr 1147337409], length 34
14:26:24.533262 IP 192.168.3.2.51374 > 192.168.2.10.443: Flags [R], seq 3936948547, win 0, length 0
14:26:24.533973 IP 192.168.3.2.51380 > 192.168.2.10.443: Flags [S], seq 4186229646, win 64240, options [mss 1460,sackOK,TS val 1147337411 ecr 0,nop,wscale 7], length 0
14:26:24.534506 IP 192.168.2.10.443 > 192.168.3.2.51380: Flags [S.], seq 4194995391, ack 4186229647, win 65160, options [mss 1460,sackOK,TS val 4276238508 ecr 1147337411,nop,wscale 7], length 0
14:26:24.534522 IP 192.168.3.2.51380 > 192.168.2.10.443: Flags [.], ack 1, win 502, options [nop,nop,TS val 1147337411 ecr 4276238508], length 0
14:26:24.535013 IP 192.168.3.2.51380 > 192.168.2.10.443: Flags [P.], seq 1:1454, ack 1, win 502, options [nop,nop,TS val 1147337412 ecr 4276238508], length 1453
14:26:24.535458 IP 192.168.2.10.443 > 192.168.3.2.51380: Flags [.], ack 1449, win 498, options [nop,nop,TS val 4276238509 ecr 1147337412], length 0
14:26:24.535491 IP 192.168.2.10.443 > 192.168.3.2.51380: Flags [.], ack 1454, win 498, options [nop,nop,TS val 4276238509 ecr 1147337412], length 0
2026-01-03T14:26:24+01:00 INF Running gravity...
14:26:24.579066 IP 192.168.2.10.443 > 192.168.3.2.51380: Flags [P.], seq 1:128, ack 1454, win 498, options [nop,nop,TS val 4276238552 ecr 1147337412], length 127
14:26:24.579080 IP 192.168.3.2.51380 > 192.168.2.10.443: Flags [.], ack 128, win 502, options [nop,nop,TS val 1147337456 ecr 4276238552], length 0
14:26:24.579557 IP 192.168.2.10.443 > 192.168.3.2.51380: Flags [P.], seq 128:134, ack 1454, win 498, options [nop,nop,TS val 4276238553 ecr 1147337456], length 6
14:26:24.579569 IP 192.168.3.2.51380 > 192.168.2.10.443: Flags [.], ack 134, win 502, options [nop,nop,TS val 1147337456 ecr 4276238553], length 0
14:26:24.579962 IP 192.168.2.10.443 > 192.168.3.2.51380: Flags [P.], seq 134:171, ack 1454, win 498, options [nop,nop,TS val 4276238553 ecr 1147337456], length 37
14:26:24.579968 IP 192.168.3.2.51380 > 192.168.2.10.443: Flags [.], ack 171, win 502, options [nop,nop,TS val 1147337457 ecr 4276238553], length 0
14:26:24.582921 IP 192.168.2.10.443 > 192.168.3.2.51380: Flags [P.], seq 171:704, ack 1454, win 498, options [nop,nop,TS val 4276238556 ecr 1147337457], length 533
14:26:24.582933 IP 192.168.3.2.51380 > 192.168.2.10.443: Flags [.], ack 704, win 501, options [nop,nop,TS val 1147337460 ecr 4276238556], length 0
14:26:24.595860 IP 192.168.2.10.443 > 192.168.3.2.51380: Flags [P.], seq 704:853, ack 1454, win 498, options [nop,nop,TS val 4276238569 ecr 1147337460], length 149
14:26:24.595875 IP 192.168.3.2.51380 > 192.168.2.10.443: Flags [.], ack 853, win 501, options [nop,nop,TS val 1147337473 ecr 4276238569], length 0
14:26:24.596734 IP 192.168.2.10.443 > 192.168.3.2.51380: Flags [P.], seq 853:922, ack 1454, win 498, options [nop,nop,TS val 4276238570 ecr 1147337473], length 69
14:26:24.596747 IP 192.168.3.2.51380 > 192.168.2.10.443: Flags [.], ack 922, win 501, options [nop,nop,TS val 1147337473 ecr 4276238570], length 0
14:26:24.597620 IP 192.168.3.2.51380 > 192.168.2.10.443: Flags [P.], seq 1454:1518, ack 922, win 501, options [nop,nop,TS val 1147337474 ecr 4276238570], length 64
14:26:24.597782 IP 192.168.3.2.51380 > 192.168.2.10.443: Flags [P.], seq 1518:2726, ack 922, win 501, options [nop,nop,TS val 1147337474 ecr 4276238570], length 1208
14:26:24.597798 IP 192.168.3.2.51380 > 192.168.2.10.443: Flags [P.], seq 2726:3711, ack 922, win 501, options [nop,nop,TS val 1147337474 ecr 4276238570], length 985
14:26:24.598038 IP 192.168.2.10.443 > 192.168.3.2.51380: Flags [.], ack 1518, win 498, options [nop,nop,TS val 4276238571 ecr 1147337474], length 0
14:26:24.598261 IP 192.168.2.10.443 > 192.168.3.2.51380: Flags [.], ack 2726, win 489, options [nop,nop,TS val 4276238572 ecr 1147337474], length 0
14:26:24.598458 IP 192.168.2.10.443 > 192.168.3.2.51380: Flags [.], ack 3711, win 482, options [nop,nop,TS val 4276238572 ecr 1147337474], length 0
14:26:24.603954 IP 192.168.2.10.443 > 192.168.3.2.51380: Flags [P.], seq 922:975, ack 3711, win 482, options [nop,nop,TS val 4276238577 ecr 1147337474], length 53
14:26:24.609374 IP 192.168.2.10.443 > 192.168.3.2.51380: Flags [.], seq 975:2423, ack 3711, win 482, options [nop,nop,TS val 4276238583 ecr 1147337474], length 1448
14:26:24.609387 IP 192.168.3.2.51380 > 192.168.2.10.443: Flags [.], ack 2423, win 524, options [nop,nop,TS val 1147337486 ecr 4276238577], length 0
14:26:24.609506 IP 192.168.2.10.443 > 192.168.3.2.51380: Flags [.], seq 2423:3871, ack 3711, win 482, options [nop,nop,TS val 4276238583 ecr 1147337474], length 1448
14:26:24.609554 IP 192.168.3.2.51380 > 192.168.2.10.443: Flags [P.], seq 3711:3735, ack 3871, win 547, options [nop,nop,TS val 1147337486 ecr 4276238583], length 24
14:26:24.609585 IP 192.168.3.2.51380 > 192.168.2.10.443: Flags [R.], seq 3735, ack 3871, win 547, options [nop,nop,TS val 1147337486 ecr 4276238583], length 0
14:26:24.609666 IP 192.168.2.10.443 > 192.168.3.2.51380: Flags [P.], seq 3871:5319, ack 3711, win 482, options [nop,nop,TS val 4276238583 ecr 1147337474], length 1448
14:26:24.609684 IP 192.168.3.2.51380 > 192.168.2.10.443: Flags [R], seq 4186233357, win 0, length 0
14:26:24.610000 IP 192.168.2.10.443 > 192.168.3.2.51380: Flags [P.], seq 5319:6441, ack 3711, win 482, options [nop,nop,TS val 4276238583 ecr 1147337486], length 1122
14:26:24.610013 IP 192.168.3.2.51380 > 192.168.2.10.443: Flags [R], seq 4186233357, win 0, length 0
14:26:24.614867 IP 192.168.3.2.51382 > 192.168.2.10.443: Flags [S], seq 441617630, win 64240, options [mss 1460,sackOK,TS val 1147337491 ecr 0,nop,wscale 7], length 0
14:26:24.615498 IP 192.168.2.10.443 > 192.168.3.2.51382: Flags [S.], seq 2936628407, ack 441617631, win 65160, options [mss 1460,sackOK,TS val 4276238589 ecr 1147337491,nop,wscale 7], length 0
14:26:24.615537 IP 192.168.3.2.51382 > 192.168.2.10.443: Flags [.], ack 1, win 502, options [nop,nop,TS val 1147337492 ecr 4276238589], length 0
14:26:24.616056 IP 192.168.3.2.51382 > 192.168.2.10.443: Flags [P.], seq 1:1454, ack 1, win 502, options [nop,nop,TS val 1147337493 ecr 4276238589], length 1453
14:26:24.616658 IP 192.168.2.10.443 > 192.168.3.2.51382: Flags [.], ack 1449, win 498, options [nop,nop,TS val 4276238590 ecr 1147337493], length 0
14:26:24.616721 IP 192.168.2.10.443 > 192.168.3.2.51382: Flags [.], ack 1454, win 498, options [nop,nop,TS val 4276238590 ecr 1147337493], length 0
14:26:24.658834 IP 192.168.2.10.443 > 192.168.3.2.51382: Flags [P.], seq 1:128, ack 1454, win 498, options [nop,nop,TS val 4276238632 ecr 1147337493], length 127
14:26:24.658869 IP 192.168.3.2.51382 > 192.168.2.10.443: Flags [.], ack 128, win 502, options [nop,nop,TS val 1147337535 ecr 4276238632], length 0
14:26:24.659321 IP 192.168.2.10.443 > 192.168.3.2.51382: Flags [P.], seq 128:171, ack 1454, win 498, options [nop,nop,TS val 4276238633 ecr 1147337535], length 43
14:26:24.659338 IP 192.168.3.2.51382 > 192.168.2.10.443: Flags [.], ack 171, win 502, options [nop,nop,TS val 1147337536 ecr 4276238633], length 0
14:26:24.662545 IP 192.168.2.10.443 > 192.168.3.2.51382: Flags [P.], seq 171:704, ack 1454, win 498, options [nop,nop,TS val 4276238636 ecr 1147337536], length 533
14:26:24.662563 IP 192.168.3.2.51382 > 192.168.2.10.443: Flags [.], ack 704, win 501, options [nop,nop,TS val 1147337539 ecr 4276238636], length 0
2026-01-03T14:26:24+01:00 INF Invalidating sessions...
14:26:24.675337 IP 192.168.2.10.443 > 192.168.3.2.51382: Flags [P.], seq 704:837, ack 1454, win 498, options [nop,nop,TS val 4276238649 ecr 1147337539], length 133
14:26:24.675363 IP 192.168.3.2.51382 > 192.168.2.10.443: Flags [.], ack 837, win 501, options [nop,nop,TS val 1147337552 ecr 4276238649], length 0
14:26:24.676087 IP 192.168.2.10.443 > 192.168.3.2.51382: Flags [P.], seq 837:906, ack 1454, win 498, options [nop,nop,TS val 4276238649 ecr 1147337552], length 69
14:26:24.676107 IP 192.168.3.2.51382 > 192.168.2.10.443: Flags [.], ack 906, win 501, options [nop,nop,TS val 1147337553 ecr 4276238649], length 0
14:26:24.677182 IP 192.168.3.2.51382 > 192.168.2.10.443: Flags [P.], seq 1454:1518, ack 906, win 501, options [nop,nop,TS val 1147337554 ecr 4276238649], length 64
14:26:24.677346 IP 192.168.3.2.51382 > 192.168.2.10.443: Flags [P.], seq 1518:1702, ack 906, win 501, options [nop,nop,TS val 1147337554 ecr 4276238649], length 184
14:26:24.677575 IP 192.168.2.10.443 > 192.168.3.2.51382: Flags [.], ack 1518, win 498, options [nop,nop,TS val 4276238651 ecr 1147337554], length 0
14:26:24.677749 IP 192.168.2.10.443 > 192.168.3.2.51382: Flags [.], ack 1702, win 497, options [nop,nop,TS val 4276238651 ecr 1147337554], length 0
14:26:24.705107 IP 192.168.2.10.443 > 192.168.3.2.51382: Flags [P.], seq 906:959, ack 1702, win 497, options [nop,nop,TS val 4276238678 ecr 1147337554], length 53
14:26:24.745873 IP 192.168.3.2.51382 > 192.168.2.10.443: Flags [.], ack 959, win 501, options [nop,nop,TS val 1147337623 ecr 4276238678], length 0
14:26:24.746643 IP 192.168.2.10.443 > 192.168.3.2.51382: Flags [P.], seq 959:2068, ack 1702, win 497, options [nop,nop,TS val 4276238720 ecr 1147337623], length 1109
14:26:24.746672 IP 192.168.3.2.51382 > 192.168.2.10.443: Flags [.], ack 2068, win 524, options [nop,nop,TS val 1147337623 ecr 4276238720], length 0
14:26:24.746883 IP 192.168.3.2.51382 > 192.168.2.10.443: Flags [P.], seq 1702:1726, ack 2068, win 524, options [nop,nop,TS val 1147337624 ecr 4276238720], length 24
14:26:24.746916 IP 192.168.3.2.51382 > 192.168.2.10.443: Flags [F.], seq 1726, ack 2068, win 524, options [nop,nop,TS val 1147337624 ecr 4276238720], length 0
14:26:24.751865 IP 192.168.3.2.51382 > 192.168.2.10.443: Flags [F.], seq 1726, ack 2068, win 524, options [nop,nop,TS val 1147337629 ecr 4276238720], length 0
14:26:24.752338 IP 192.168.2.10.443 > 192.168.3.2.51382: Flags [.], ack 1727, win 497, options [nop,nop,TS val 4276238726 ecr 1147337624,nop,nop,sack 1 {1726:1727}], length 0
14:26:24.797202 IP 192.168.3.2.51392 > 192.168.2.10.443: Flags [S], seq 3991377707, win 64240, options [mss 1460,sackOK,TS val 1147337674 ecr 0,nop,wscale 7], length 0
14:26:24.797896 IP 192.168.2.10.443 > 192.168.3.2.51392: Flags [S.], seq 216049776, ack 3991377708, win 65160, options [mss 1460,sackOK,TS val 4276238771 ecr 1147337674,nop,wscale 7], length 0
14:26:24.797921 IP 192.168.3.2.51392 > 192.168.2.10.443: Flags [.], ack 1, win 502, options [nop,nop,TS val 1147337675 ecr 4276238771], length 0
14:26:24.798379 IP 192.168.3.2.51392 > 192.168.2.10.443: Flags [P.], seq 1:1454, ack 1, win 502, options [nop,nop,TS val 1147337675 ecr 4276238771], length 1453
14:26:24.798976 IP 192.168.2.10.443 > 192.168.3.2.51392: Flags [.], ack 1449, win 498, options [nop,nop,TS val 4276238772 ecr 1147337675], length 0
14:26:24.798991 IP 192.168.2.10.443 > 192.168.3.2.51392: Flags [.], ack 1454, win 498, options [nop,nop,TS val 4276238772 ecr 1147337675], length 0
14:26:24.845637 IP 192.168.2.10.443 > 192.168.3.2.51392: Flags [P.], seq 1:128, ack 1454, win 498, options [nop,nop,TS val 4276238819 ecr 1147337675], length 127
14:26:24.845668 IP 192.168.3.2.51392 > 192.168.2.10.443: Flags [.], ack 128, win 502, options [nop,nop,TS val 1147337722 ecr 4276238819], length 0
14:26:24.846158 IP 192.168.2.10.443 > 192.168.3.2.51392: Flags [P.], seq 128:134, ack 1454, win 498, options [nop,nop,TS val 4276238820 ecr 1147337722], length 6
14:26:24.846170 IP 192.168.3.2.51392 > 192.168.2.10.443: Flags [.], ack 134, win 502, options [nop,nop,TS val 1147337723 ecr 4276238820], length 0
14:26:24.846569 IP 192.168.2.10.443 > 192.168.3.2.51392: Flags [P.], seq 134:171, ack 1454, win 498, options [nop,nop,TS val 4276238820 ecr 1147337723], length 37
14:26:24.846579 IP 192.168.3.2.51392 > 192.168.2.10.443: Flags [.], ack 171, win 502, options [nop,nop,TS val 1147337723 ecr 4276238820], length 0
14:26:24.850156 IP 192.168.2.10.443 > 192.168.3.2.51392: Flags [P.], seq 171:704, ack 1454, win 498, options [nop,nop,TS val 4276238823 ecr 1147337723], length 533
14:26:24.850173 IP 192.168.3.2.51392 > 192.168.2.10.443: Flags [.], ack 704, win 501, options [nop,nop,TS val 1147337727 ecr 4276238823], length 0
14:26:24.864029 IP 192.168.2.10.443 > 192.168.3.2.51392: Flags [P.], seq 704:837, ack 1454, win 498, options [nop,nop,TS val 4276238837 ecr 1147337727], length 133
14:26:24.864059 IP 192.168.3.2.51392 > 192.168.2.10.443: Flags [.], ack 837, win 501, options [nop,nop,TS val 1147337741 ecr 4276238837], length 0
14:26:24.864925 IP 192.168.2.10.443 > 192.168.3.2.51392: Flags [P.], seq 837:906, ack 1454, win 498, options [nop,nop,TS val 4276238838 ecr 1147337741], length 69
14:26:24.864938 IP 192.168.3.2.51392 > 192.168.2.10.443: Flags [.], ack 906, win 501, options [nop,nop,TS val 1147337742 ecr 4276238838], length 0
14:26:24.865892 IP 192.168.3.2.51392 > 192.168.2.10.443: Flags [P.], seq 1454:1518, ack 906, win 501, options [nop,nop,TS val 1147337743 ecr 4276238838], length 64
14:26:24.866007 IP 192.168.3.2.51392 > 192.168.2.10.443: Flags [P.], seq 1518:1675, ack 906, win 501, options [nop,nop,TS val 1147337743 ecr 4276238838], length 157
14:26:24.866317 IP 192.168.2.10.443 > 192.168.3.2.51392: Flags [.], ack 1518, win 498, options [nop,nop,TS val 4276238840 ecr 1147337743], length 0
14:26:24.866353 IP 192.168.2.10.443 > 192.168.3.2.51392: Flags [.], ack 1675, win 497, options [nop,nop,TS val 4276238840 ecr 1147337743], length 0
14:26:24.868960 IP 192.168.2.10.443 > 192.168.3.2.51392: Flags [P.], seq 906:959, ack 1675, win 497, options [nop,nop,TS val 4276238842 ecr 1147337743], length 53
2026-01-03T14:26:24+01:00 INF Sync completed 
14:26:24.909867 IP 192.168.3.2.51392 > 192.168.2.10.443: Flags [.], ack 959, win 501, options [nop,nop,TS val 1147337787 ecr 4276238842], length 0
14:26:24.910656 IP 192.168.2.10.443 > 192.168.3.2.51392: Flags [P.], seq 959:1946, ack 1675, win 497, options [nop,nop,TS val 4276238884 ecr 1147337787], length 987
14:26:24.910678 IP 192.168.3.2.51392 > 192.168.2.10.443: Flags [.], ack 1946, win 524, options [nop,nop,TS val 1147337787 ecr 4276238884], length 0
14:26:24.911877 IP 192.168.3.2.51392 > 192.168.2.10.443: Flags [F.], seq 1675, ack 1946, win 524, options [nop,nop,TS val 1147337789 ecr 4276238884], length 0
14:26:24.927601 IP 192.168.2.10.443 > 192.168.3.2.51392: Flags [P.], seq 1946:1983, ack 1676, win 497, options [nop,nop,TS val 4276238901 ecr 1147337789], length 37
14:26:24.927637 IP 192.168.3.2.51392 > 192.168.2.10.443: Flags [R], seq 3991379383, win 0, length 0
14:26:24.928047 IP 192.168.2.10.443 > 192.168.3.2.51392: Flags [F.], seq 1983, ack 1676, win 497, options [nop,nop,TS val 4276238901 ecr 1147337789], length 0
14:26:24.928062 IP 192.168.3.2.51392 > 192.168.2.10.443: Flags [R], seq 3991379383, win 0, length 0

nebula-sync, Failed to invalidate session for target

2026-01-03T14:16:15+01:00 INF Invalidating sessions...
14:16:15.425520 IP 192.168.3.2.45876 > 192.168.2.6:443: Flags [S], seq 5025158, win 64240, options [mss 1460,sackOK,TS val 2913369222 ecr 0,nop,wscale 7], length 0
14:16:15.425800 IP 192.168.2.6:443 > 192.168.3.2.45876: Flags [R.], seq 0, ack 5025159, win 0, length 0
2026-01-03T14:16:17+01:00 WRN Failed to invalidate session for target: https://192.168.2.6
2026-01-03T14:16:17+01:00 FTL Sync failed error="authenticate: https://192.168.2.6/api/auth: Post \"https://192.168.2.6/api/auth\": dial tcp 192.168.2.6:443: connect: connection refused"

nebula-sync, server2, Failed to invalidate session for target

#./nebula-sync run --env-file .env
2026-01-03T14:26:28+01:00 INF Starting nebula-sync v0.11.1
2026-01-03T14:26:28+01:00 INF Running sync mode=selective replicas=1
2026-01-03T14:26:28+01:00 INF Authenticating clients...
14:26:28.682393 IP 192.168.3.2.51404 > 192.168.2.10.443: Flags [S], seq 13580812, win 64240, options [mss 1460,sackOK,TS val 1147341559 ecr 0,nop,wscale 7], length 0
14:26:28.682896 IP 192.168.2.10.443 > 192.168.3.2.51404: Flags [R.], seq 0, ack 13580813, win 0, length 0
14:26:29.684165 IP 192.168.3.2.57620 > 192.168.2.10.443: Flags [S], seq 2474373643, win 64240, options [mss 1460,sackOK,TS val 1147342561 ecr 0,nop,wscale 7], length 0
14:26:29.684655 IP 192.168.2.10.443 > 192.168.3.2.57620: Flags [R.], seq 0, ack 2474373644, win 0, length 0
2026-01-03T14:26:30+01:00 INF Invalidating sessions...
14:26:30.685145 IP 192.168.3.2.57626 > 192.168.2.10.443: Flags [S], seq 120021144, win 64240, options [mss 1460,sackOK,TS val 1147343562 ecr 0,nop,wscale 7], length 0
14:26:30.685680 IP 192.168.2.10.443 > 192.168.3.2.57626: Flags [R.], seq 0, ack 120021145, win 0, length 0
2026-01-03T14:26:32+01:00 WRN Failed to invalidate session for target: https://192.168.2.10
2026-01-03T14:26:32+01:00 FTL Sync failed error="authenticate: https://192.168.2.10/api/auth: Post \"https://192.168.2.10/api/auth\": dial tcp 192.168.2.10:443: connect: connection refused"

[catanbri]

For me this seems to only be an issue with multiple replicas. In trying to understand what is breaking I took my replicas down to 1 from 2. I can have the service run fine with either of the replicas. but as soon as I add more than 1 I start seeing the break again.

I dont understand go or the HTTP stack, I am just throwing stuff at the wall here.

[bkwar201]

Thanks for some of the info int this, Just setting up syncing myself for a HA Ceph cluster of 2 piholes.. instructions that i found have it with /admin as part of the address...
Once i saw your example without that i was able to get it to start syncing!