Gravity database on primary Pi-hole (seemingly) getting munged after a sync

[PromoFaux]

What happened?

First of all - great project... it's exactly the kind of thing we include an API for!

---

It seems that shortly after nebula-sync does it's thing, Pi-hole starts logging errors about the state of the gravity database on the primary - the replica is fine!

I've tested this several times by compltely destroying all volumes on my running containers and starting with fresh databases. Each time I set up the primary with the relevant domains on my list.

Doing a manual teleport from the primary (and indeed reimporting that) does not cause this same issue. I have done the manual teleport both via the web UI and the API.

As soon as I turn the nebula-sync container on - I start seeing these issues.

Pretty stumped, because as far as I can tell nebula is only using the API to:

• grab teleport backup from primary
• push teleport backup to replica
• grab config from primary
• patch config on replica

Any ideas?

I have also tried both with and without RUN_GRAVITY enabled - after having read a thread suggesting it could be that.

Expected Behaviour

Gravity database to remain in tact :)

Nebula-sync Version

0.110

Runtime

docker

Operating System

synology DSM

CPU Architecture

x86_64

Configuration

services:
  nebula-sync:
    image: ghcr.io/lovelaze/nebula-sync:latest
    container_name: nebula-sync
    environment:
     PRIMARY: http://192.168.1.254|[Redacted]
     REPLICAS: http://[Redacted]:[Redacted]@192.168.0.15|
     FULL_SYNC: true
     NS_DEBUG: true    
     RUN_GRAVITY: true
     CRON: 0 * * * *

The replica is running as a home assistant addon which is exposed via nginx, hence the password up-front. It otherwise has no password

Relevant log output

2025-08-14T17:24:08Z INF github.com/lovelaze/nebula-sync/internal/service/service.go:66 > Starting nebula-sync v0.11.0
2025-08-14T17:24:08Z DBG github.com/lovelaze/nebula-sync/internal/api/server.go:38 > Starting http server
2025-08-14T17:24:08Z DBG github.com/lovelaze/nebula-sync/internal/service/service.go:67 > Settings config="{Primary:{URL:http://192.168.1.254 Password:[REDACTED]} Replicas:[{URL:http://[REDACTED]:[REDACTED]@192.168.0.15 Password:}] Client:{SkipTLSVerification:false RetryDelay:1 Timeout:20} Sync:{FullSync:true Cron:0xc00002c590 RunGravity:true GravitySettings:{DHCPLeases:false Group:false Adlist:false AdlistByGroup:false Domainlist:false DomainlistByGroup:false Client:false ClientByGroup:false} ConfigSettings:{DNS:{Enabled:false Filter:<nil>} DHCP:{Enabled:false Filter:<nil>} NTP:{Enabled:false Filter:<nil>} Resolver:{Enabled:false Filter:<nil>} Database:{Enabled:false Filter:<nil>} Webserver:{Enabled:false Filter:<nil>} Files:{Enabled:false Filter:<nil>} Misc:{Enabled:false Filter:<nil>} Debug:{Enabled:false Filter:<nil>}} WebhookSettings:{Failure:{Body: Headers:map[] Method:POST URL:} Success:{Body: Headers:map[] Method:POST URL:} Client:{SkipTLSVerification:false}}} API:{Enabled:true}}"
2025-08-14T17:24:08Z INF github.com/lovelaze/nebula-sync/internal/sync/sync.go:35 > Running sync mode=full replicas=1
2025-08-14T17:24:08Z INF github.com/lovelaze/nebula-sync/internal/sync/sync.go:52 > Authenticating clients...
2025-08-14T17:24:08Z DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:66 > PostAuth client=http://192.168.1.254
2025-08-14T17:24:08Z DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:66 > PostAuth client=http://[REDACTED]:[REDACTED]@192.168.0.15
2025-08-14T17:24:08Z INF github.com/lovelaze/nebula-sync/internal/sync/sync.go:84 > Syncing teleporters...
2025-08-14T17:24:08Z DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:149 > Get teleporter client=http://192.168.1.254
2025-08-14T17:24:08Z DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:175 > Post teleporter client=http://[REDACTED]:[REDACTED]@192.168.0.15 payload={"config":false,"dhcp_leases":true,"gravity":{"adlist":true,"adlist_by_group":true,"client":true,"client_by_group":true,"domainlist":true,"domainlist_by_group":true,"group":true}}
2025-08-14T17:24:09Z INF github.com/lovelaze/nebula-sync/internal/sync/sync.go:107 > Syncing configs...
2025-08-14T17:24:09Z DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:226 > Get config client=http://192.168.1.254
2025-08-14T17:24:09Z DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:261 > Patch config client=http://[REDACTED]:[REDACTED]@192.168.0.15 payload={"config":{"database":{"DBimport":true,"DBinterval":60,"maxDBdays":91,"network":{"expire":91,"parseARPcache":true},"useWAL":true},"debug":{"aliasclients":false,"all":false,"api":false,"arp":false,"caps":false,"clients":false,"config":false,"database":false,"dnssec":false,"edns0":false,"events":false,"extra":false,"flags":false,"gc":false,"helper":false,"inotify":false,"locks":false,"netlink":false,"networking":false,"ntp":false,"overtime":false,"queries":false,"regex":false,"reserved":false,"resolver":false,"shmem":false,"status":false,"tls":false,"vectors":false,"webserver":false},"dhcp":{"active":false,"end":"","hosts":[],"ignoreUnknownClients":false,"ipv6":false,"leaseTime":"","logging":false,"multiDNS":false,"netmask":"","rapidCommit":false,"router":"","start":""},"dns":{"CNAMEdeepInspect":true,"EDNS0ECS":true,"analyzeOnlyAandAAAA":false,"blockESNI":true,"blockTTL":2,"blocking":{"active":true,"edns":"TEXT","mode":"NULL"},"bogusPriv":true,"cache":{"optimizer":3600,"size":10000,"upstreamBlockedTTL":86400},"cnameRecords":[],"dnssec":false,"domain":"lan","domainNeeded":false,"expandHosts":false,"hostRecord":"","hosts":["192.168.1.254 ph1.lan","192.168.0.15 ph2.lan"],"ignoreLocalhost":false,"interface":"eth0","listeningMode":"ALL","piholePTR":"PI.HOLE","port":53,"queryLogging":true,"rateLimit":{"count":1000,"interval":60},"reply":{"blocking":{"IPv4":"","IPv6":"","force4":false,"force6":false},"host":{"IPv4":"","IPv6":"","force4":false,"force6":false}},"replyWhenBusy":"ALLOW","revServers":["true,192.168.0.0/16,192.168.0.1,lan"],"showDNSSEC":true,"specialDomains":{"designatedResolver":true,"iCloudPrivateRelay":true,"mozillaCanary":true},"upstreams":["8.8.8.8","8.8.4.4"]},"misc":{"addr2line":true,"check":{"disk":90,"load":true,"shmem":90},"delay_startup":0,"dnsmasq_lines":[],"etc_dnsmasq_d":false,"extraLogging":false,"nice":-10,"privacylevel":0,"readOnly":false},"ntp":{"ipv4":{"active":true,"address":""},"ipv6":{"active":true,"address":""},"sync":{"active":true,"count":8,"interval":3600,"rtc":{"device":"","set":false,"utc":true},"server":"pool.ntp.org"}},"resolver":{"networkNames":true,"refreshNames":"IPV4_ONLY","resolveIPv4":true,"resolveIPv6":true}}}
2025-08-14T17:24:09Z INF github.com/lovelaze/nebula-sync/internal/sync/sync.go:127 > Running gravity...
2025-08-14T17:24:09Z DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:297 > Post run gravity client=http://192.168.1.254
2025-08-14T17:24:09Z DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:297 > Post run gravity client=http://[REDACTED]:[REDACTED]@192.168.0.15
2025-08-14T17:24:09Z INF github.com/lovelaze/nebula-sync/internal/sync/sync.go:69 > Invalidating sessions...
2025-08-14T17:24:09Z DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:117 > Delete session client=http://192.168.1.254
2025-08-14T17:24:09Z DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:117 > Delete session client=http://[REDACTED]:[REDACTED]@192.168.0.15
2025-08-14T17:24:09Z DBG github.com/lovelaze/nebula-sync/internal/pihole/client.go:123 > Trying to delete empty session
2025-08-14T17:24:09Z INF github.com/lovelaze/nebula-sync/internal/service/service.go:95 > Sync completed

Shortly afterwards, Pi-hole logs spit out:

2025-08-14 17:24:11.013 UTC [167M] ERROR: SQLite3: no such table: main.gravity in "SELECT adlist_id from vw_gravity WHERE domain = ? AND group_id IN (0);" (1)
2025-08-14 17:24:11.013 UTC [167M] ERROR: gravityDB_open("SELECT(... vw_gravity ...)") - SQL error prepare: SQL logic error
2025-08-14 17:24:11.069 UTC [167/T623] INFO: Gravity database has been updated, reloading now
2025-08-14 17:24:11.113 UTC [167/T623] INFO: Compiled 0 allow and 1 deny regex for 18 clients in 41.0 msec

Or one from last night (I don't have the nebula log from that point)

2025-08-13 22:00:02.533 UTC [54M] ERROR: SQLite3: no such table: main.antigravity in "SELECT adlist_id from vw_antigravity WHERE domain = ? AND group_id IN (0);" (1)
2025-08-13 22:00:02.533 UTC [54M] ERROR: domain_in_list("scontent-lhr8-2.xx.fbcdn.net", 0x7f0b8ac7cd18, antigravity): Failed to perform step: SQL logic error
2025-08-13 22:00:02.533 UTC [54M] ERROR: SQLite3: no such table: main.gravity in "SELECT adlist_id from vw_gravity WHERE domain = ? AND group_id IN (0);" (1)
2025-08-13 22:00:02.533 UTC [54M] ERROR: domain_in_list("scontent-lhr8-2.xx.fbcdn.net", 0x7f0b8ac7ca98, gravity): Failed to perform step: SQL logic error
2025-08-13 22:00:02.942 UTC [54M] ERROR: SQLite3: no such table: main.antigravity in "SELECT adlist_id from vw_antigravity WHERE domain = ? AND group_id IN (0);" (1)
2025-08-13 22:00:02.942 UTC [54M] ERROR: domain_in_list("scontent-lhr8-1.xx.fbcdn.net", 0x7f0b8ac7cd18, antigravity): Failed to perform step: SQL logic error
2025-08-13 22:00:02.942 UTC [54M] ERROR: SQLite3: no such table: main.gravity in "SELECT adlist_id from vw_gravity WHERE domain = ? AND group_id IN (0);" (1)
2025-08-13 22:00:02.942 UTC [54M] ERROR: domain_in_list("scontent-lhr8-1.xx.fbcdn.net", 0x7f0b8ac7ca98, gravity): Failed to perform step: SQL logic error

Pi-hole Version

:latest

[mkrumbholz]

Exact same problem for me.
Had to recreate the gravity database and import a backup via the web teleport function.
Used this command for the recration of the database: pihole -g -r recreate

[Phill23]

same here

[lovelaze]

Hi @PromoFaux, thanks for reporting this!

First of all, yes, nebula-sync itself is pretty simple - it is basically just doing API calls to the piholes. It will only proceed and consider the sync successful if all pihole responses are http 200s.

It seems that shortly after nebula-sync does it's thing, Pi-hole starts logging errors about the state of the gravity database on the primary - the replica is fine!

This confuses me, because the only non read-only operation on the primary should be the run gravity operation with RUN_GRAVITY=true.

I have also tried both with and without RUN_GRAVITY enabled - after having read a thread suggesting it could be that.

It is contradicted by this, I don't see why the teleporter config GET requests should affect the primary database on the primary.

Pretty stumped, because as far as I can tell nebula is only using the API to:

* grab teleport backup from primary
* push teleport backup to replica
* grab config from primary
* patch config on replica 

Correct. As well as authenticating, deleting sessions and optionally running gravity.

Perhaps if you can start out with a clean state of the piholes and post the nebula-sync debug logs of the first sync, then we can rule out if there's something off with the requests to the piholes. But as they respond with 200 I'm not so sure that would be the case...

@mkrumbholz @mkrumbholz
How do your pihole setups diff from @PromoFaux?

[mkrumbholz]

@lovelaze
The only difference in the setup is: No Home Assitant or Synology disk station is involved and the primary and replica are running in a docker enviroment on Ubuntu 24.04 LTS servers VM x86_64. Everything is up to date.

RUN_GRAVITY was enabled.

Update:
Have RUN_GRAVITY disabled for another test, with that setting it seems to work now.
Maybe somehow multiple gravity runs were triggered?

My docker-compose.yml file:

services:
  nebula-sync:
    image: ghcr.io/lovelaze/nebula-sync:latest
    container_name: nebula-sync
    env_file: .env
    restart: unless-stopped

My .env file:

PRIMARY=https://address|pw
REPLICAS=https://address|pw
FULL_SYNC=true
RUN_GRAVITY=false
CRON=0 * * * *
CLIENT_TIMEOUT_SECONDS=60
CLIENT_RETRY_DELAY_SECONDS=60
CLIENT_SKIP_TLS_VERIFICATION=true

[michaelblight]

Same here - if I have RUN_GRAVITY=true then the first sync works and subsequent ones fail. It also breaks doing an Update Gravity from the pihole UI. Deleting the db and setting RUN_GRAVITY=false stops the errors. This is my first use of nebula-sync, and it along with 2 pihole instances are all running under docker on separate proxmox VMs/machines.

[ssavkar]

I was having similar issues which were solved as others note here by turning to false the run gravity flag. Now no issues on the primary.

[gregpakes]

I have the same issue - I only noticed because my keepalived would failover to another pihole daily for a short period.

It doesn't happen every update for me - as you can see by the logs.

2025-12-05 08:00:11.040 INFO Gravity database has been updated, reloading now
2025-12-05 08:00:11.041 INFO Compiled 0 allow and 0 deny regex for 45 clients in 0.0 msec
2025-12-05 08:27:41.903 INFO Received 8/8 valid NTP replies from pool.ntp.org
2025-12-05 08:27:41.903 INFO Time offset: -4.601005e+01 ms (excluded 0 outliers)
2025-12-05 08:27:41.903 INFO Round-trip delay: 1.385282e+02 ms (excluded 0 outliers)
2025-12-05 09:00:14.050 INFO Gravity database has been updated, reloading now
2025-12-05 09:00:14.052 INFO Compiled 0 allow and 0 deny regex for 46 clients in 0.0 msec
2025-12-05 09:27:42.215 INFO Received 8/8 valid NTP replies from pool.ntp.org
2025-12-05 09:27:42.216 INFO Time offset: -4.326071e+01 ms (excluded 1 outliers)
2025-12-05 09:27:42.216 INFO Round-trip delay: 1.386880e+02 ms (excluded 1 outliers)
2025-12-05 10:00:11.056 INFO Gravity database has been updated, reloading now
2025-12-05 10:00:11.058 INFO Compiled 0 allow and 0 deny regex for 46 clients in 0.0 msec
2025-12-05 10:27:42.715 INFO Received 8/8 valid NTP replies from pool.ntp.org
2025-12-05 10:27:42.715 INFO Time offset: -2.002913e+01 ms (excluded 0 outliers)
2025-12-05 10:27:42.715 INFO Round-trip delay: 1.566681e+02 ms (excluded 0 outliers)
2025-12-05 11:00:13.093 ERROR SQLite3: no such table: main.antigravity in "SELECT adlist_id from vw_antigravity WHERE domain = ? AND group_id IN (0);" (1)
2025-12-05 11:00:13.093 ERROR domain_in_list("onedscolprdweu17.westeurope.cloudapp.azure.com", 0x7fff42876b90, antigravity): Failed to perform step: SQL logic error
2025-12-05 11:00:13.093 ERROR SQLite3: no such table: main.gravity in "SELECT adlist_id from vw_gravity WHERE domain = ? AND group_id IN (0);" (1)
2025-12-05 11:00:13.093 ERROR domain_in_list("onedscolprdweu17.westeurope.cloudapp.azure.com", 0x7fff42876a50, gravity): Failed to perform step: SQL logic error
2025-12-05 11:00:13.172 ERROR SQLite3: no such table: main.antigravity in "SELECT adlist_id from vw_antigravity WHERE domain = ? AND group_id IN (0);" (1)
2025-12-05 11:00:13.172 ERROR domain_in_list("eu.events.data.trafficmanager.net", 0x7fff42876b90, antigravity): Failed to perform step: SQL logic error
2025-12-05 11:00:13.172 ERROR SQLite3: no such table: main.gravity in "SELECT adlist_id from vw_gravity WHERE domain = ? AND group_id IN (0);" (1)
2025-12-05 11:00:13.172 ERROR domain_in_list("eu.events.data.trafficmanager.net", 0x7fff42876a50, gravity): Failed to perform step: SQL logic error
2025-12-05 11:00:13.172 ERROR SQLite3: no such table: main.antigravity in "SELECT adlist_id from vw_antigravity WHERE domain = ? AND group_id IN (0);" (1)
2025-12-05 11:00:13.172 ERROR domain_in_list("onedscolprdweu07.westeurope.cloudapp.azure.com", 0x7fff42876b90, antigravity): Failed to perform step: SQL logic error
2025-12-05 11:00:13.172 ERROR SQLite3: no such table: main.gravity in "SELECT adlist_id from vw_gravity WHERE domain = ? AND group_id IN (0);" (1)
2025-12-05 11:00:13.172 ERROR domain_in_list("onedscolprdweu07.westeurope.cloudapp.azure.com", 0x7fff42876a50, gravity): Failed to perform step: SQL logic error
2025-12-05 11:00:13.172 ERROR SQLite3: no such table: main.antigravity in "SELECT adlist_id from vw_antigravity WHERE domain = ? AND group_id IN (0);" (1)
2025-12-05 11:00:13.172 ERROR domain_in_list("eu.events.data.trafficmanager.net", 0x7fff42876b90, antigravity): Failed to perform step: SQL logic error
2025-12-05 11:00:13.173 ERROR SQLite3: no such table: main.gravity in "SELECT adlist_id from vw_gravity WHERE domain = ? AND group_id IN (0);" (1)
2025-12-05 11:00:13.173 ERROR domain_in_list("eu.events.data.trafficmanager.net", 0x7fff42876a50, gravity): Failed to perform step: SQL logic error
2025-12-05 11:00:13.173 ERROR SQLite3: no such table: main.antigravity in "SELECT adlist_id from vw_antigravity WHERE domain = ? AND group_id IN (0);" (1)
2025-12-05 11:00:13.173 ERROR domain_in_list("onedscolprdweu17.westeurope.cloudapp.azure.com", 0x7fff42876b90, antigravity): Failed to perform step: SQL logic error
2025-12-05 11:00:13.173 ERROR SQLite3: no such table: main.gravity in "SELECT adlist_id from vw_gravity WHERE domain = ? AND group_id IN (0);" (1)
2025-12-05 11:00:13.173 ERROR domain_in_list("onedscolprdweu17.westeurope.cloudapp.azure.com", 0x7fff42876a50, gravity): Failed to perform step: SQL logic error
2025-12-05 11:00:14.080 INFO Gravity database has been updated, reloading now

Here it updates successfully with no errors at 08:00, 09:00, 10:00, but errors at 11:00