feat: WAL-based RocksDB replication with HTTP streaming and failover

[JackGuslerGit]

This relates to #35.

Summary:

• Adds a primary/secondary replication system using RocksDB's WAL (Write-Ahead Log) streamed over HTTP
• Secondary bootstraps from a full checkpoint on startup, then streams incremental WAL frames
• Failover is triggered via POST /_tuwunel/replication/promote — no process restart needed
• All replication endpoints are protected by a shared secret token

Test plan:

• Ran two Docker containers (primary on :8008, secondary on :8009)
• Secondary bootstrapped from primary checkpoint at seq 281 and began streaming
• Stopped primary with docker stop (graceful SIGTERM)
• Promoted secondary via curl — responded {"status":"promoted"}
• All messages from before the failover were present on the promoted instance
• Measured RPO ~0 on planned failover, RTO = seconds

Relevant config options added:

• rocksdb_primary_url — URL of primary for WAL streaming
• rocksdb_replication_token — shared secret for endpoint auth
• rocksdb_replication_interval_ms — heartbeat interval (default 250ms)

[pschichtel]

this implements async replication, so some dataloss is to be expected after an expected failover (node failure, disk failure process crash, ...), right?

[JackGuslerGit]

@pschichtel yes, that is correct. Under normal write load, RPO is determined just by network RTT.

[JackGuslerGit]

Hey @x86pup. It seems the CI failures on this PR are all runner-side cache issues. I am seeing two issues:

1: failed to create locked file '/opt/rust/cargo/debian/x86_64-linux-gnu/git/db/rust-rocksdb-eed8465c83fc7d81/config.lock': File exists; class=Os (2); code=Locked (-14)

2: could not open '/opt/rust/cargo/debian/x86_64-linux-gnu/git/checkouts/ruma-8006605ea0e2ea25/30d063c/crates/ruma-events/src/poll/unstable_start/unstable_poll_answers_serde.rs' for writing: No such file or directory; class=Os (2)

Can you clean these up and re-run?

[jevolk]

Hey @x86pup. It seems the CI failures on this PR are all runner-side cache issues. I am seeing two issues:

1: failed to create locked file '/opt/rust/cargo/debian/x86_64-linux-gnu/git/db/rust-rocksdb-eed8465c83fc7d81/config.lock': File exists; class=Os (2); code=Locked (-14)

2: could not open '/opt/rust/cargo/debian/x86_64-linux-gnu/git/checkouts/ruma-8006605ea0e2ea25/30d063c/crates/ruma-events/src/poll/unstable_start/unstable_poll_answers_serde.rs' for writing: No such file or directory; class=Os (2)

Can you clean these up and re-run?

These docker flakes sometimes occur when CI is really busy, apologies! We'll be happy to rerun as necessary.

[jevolk]

I haven't had a chance to thoroughly review this yet since I'm currently away, but a few things stand out as suspicious.

Foremost it's not clear why WAL streaming is necessary. RocksDB already has internal mechanisms to synchronize primary and secondary; all that's missing is the promotion signalling. What is the basis for concerning ourselves with binary framing of rocksdb inner-workings at the user level? Is the rocksdb synchronization API being invoked here? Perhaps I missed it...

[JackGuslerGit]

@jevolk Yes, you are correct, TryCatchUpWithPrimary() handles sync internally. Correct me if I'm wrong, but it requires the secondary instance to open the primary's database directory directly. This works when both instances share the same filesystem (same machine or NFS mount).

In our case, we have a cluster of physical servers where each server has its own local disk. We can't use NFS/shared storage in our infrastructure. So core2 has no direct filesystem access to core1's RocksDB directory.

That's the gap we're trying to fill by replicating the WAL and SST files over the network, so core2 can stay in sync with core1 without shared storage. Once core2 has a local copy of the data, TryCatchUpWithPrimary() could potentially still be used if we mirror the primary's files locally, or we apply the WAL batches ourselves.

Is there a mechanism in RocksDB you'd recommend for this case, or is shared storage assumed in your deployment model?

[jevolk]

@jevolk Yes, you are correct, TryCatchUpWithPrimary() handles sync internally. Correct me if I'm wrong, but it requires the secondary instance to open the primary's database directory directly. This works when both instances share the same filesystem (same machine or NFS mount).

In our case, we have a cluster of physical servers where each server has its own local disk. We can't use NFS/shared storage in our infrastructure. So core2 has no direct filesystem access to core1's RocksDB directory.

That's the gap we're trying to fill by replicating the WAL and SST files over the network, so core2 can stay in sync with core1 without shared storage. Once core2 has a local copy of the data, TryCatchUpWithPrimary() could potentially still be used if we mirror the primary's files locally, or we apply the WAL batches ourselves.

Is there a mechanism in RocksDB you'd recommend for this case, or is shared storage assumed in your deployment model?

Alright so this is not limited to shared filesystem mounts, that's rather exciting actually. Keep up the good work 👍

[JackGuslerGit]

@jevolk Thanks! I see it has passed all checks, what's the next step here?

[x86pup]

It needs to be thoroughly reviewed here especially since the usage of AI is apparent, Jason is on vacation and will get to it soon. Thank you for ensuring CI passes to help this along.

[JackGuslerGit]

Okay sounds good, thanks for letting me know!

[jevolk]

Thank you for your patience 🙏 I'm right around the corner now...

[JackGuslerGit]

Hey @jevolk. I ran into an issue with the checkpoint logic while testing. The original code swapped the RocksDB database directory while RocksDB was already running, causing file corruption because open file descriptors still pointed to the old files while new writes went to the checkpoint copy. The fix moves the checkpoint download and filesystem swap to before RocksDB opens, so the database always starts fresh from a clean checkpoint with no live files being touched.

[jevolk]

Hey @jevolk. I ran into an issue with the checkpoint logic while testing. The original code swapped the RocksDB database directory while RocksDB was already running, causing file corruption because open file descriptors still pointed to the old files while new writes went to the checkpoint copy. The fix moves the checkpoint download and filesystem swap to before RocksDB opens, so the database always starts fresh from a clean checkpoint with no live files being touched.

Thanks for finding this. I made an attempt at merging this but ran out of time before the 1.6 release with a few loose ends still. The main issue primarily dealt with switching to CBOR for the wire format which makes more sense for several reasons.

[jevolk]

I'll be revisiting this again at the top of the 1.6.1 dev cycle (start of next week). I only have a small number of re-organizations and applying CBOR (which is hugely simplifying) so this should go in pretty early on. Thank you again for your patience 🙏🏻

[jevolk]

Hi Jack, I took a second swipe at this and unfortunately I just haven't gotten it to where it needs to be. I should have taken notes to provide some details since there's way too much nuance to summarize here.

The tldr is that I have to revisit this after some higher priority tasks- either next week for 1.6.1 or on the backside of that release. Overall I think this feature has promise and we're very close now.

[JackGuslerGit]

Hey, no worries! Take your time. Let me know if there's anything I can do to help.

[JackGuslerGit]

Hey @jevolk, do you think this should also handle replicating media files?

[jevolk]

Hey @jevolk, do you think this should also handle replicating media files?

If we were to put media in RocksDB (and I have before) we would very likely be disabling the WAL for those columns and write-ops. We would need a different mode of transport. Now that we have S3 storage provider support, we have more possibilities for media backup. In fact I'm considering the ability to backup the database itself over an S3 connection- though that would be for "colder" storage and wouldn't replace this feature for "hot" failover of course 😅