feat: Raft-based High Availability using Apache Ratis#3731

Open

robfrank wants to merge 164 commits intomainfrom

ha-redesign

Commits on Apr 11, 2026

feat(ha-raft): add Maven module skeleton with Ratis dependencies

robfrank
and
claude
committed
feat(ha-raft): add HA_IMPLEMENTATION, HA_RAFT_PORT, HA_REPLICATION_LAG_WARNING config properties

robfrank
and
claude
committed
feat(ha-raft): add RaftLogEntryCodec for TX and SCHEMA entry serialization

robfrank
and
claude
committed
feat(ha-raft): add ArcadeStateMachine with WAL and schema entry application

robfrank
and
claude
committed
feat(ha-raft): add ClusterMonitor for replication lag tracking

robfrank
and
claude
committed
feat(ha-raft): add RaftHAServer wrapping Ratis RaftServer with peer list parsing

robfrank
and
claude
committed
feat(ha-raft): add SnapshotManager with checksum-based incremental sync

robfrank
and
claude
committed
feat(ha-raft): add RaftReplicatedDatabase wrapping LocalDatabase with Raft commit

robfrank
and
claude
committed
feat(ha-raft): add RaftHAPlugin with ServiceLoader registration and config validation

robfrank
and
claude
committed
docs: add HA Raft redesign design doc and implementation plan

robfrank
and
claude
committed
feat(ha-raft): wire Raft HA plugin into server startup with database wrapper hook

robfrank
and
claude
committed
feat(ha-raft): wire end-to-end Raft pipeline, 2-node integration test passing

robfrank
and
claude
committed
feat(ha-raft): add 3-node majority quorum integration test

robfrank
and
claude
committed
test(ha-raft): add failure scenario, schema replication, and quorum loss tests

robfrank
and
claude
committed
add notex to ignore
robfrank
committed
wip
robfrank
committed
feat(ha-raft): make Raft gRPC port configurable via HA_RAFT_PORT

robfrank
and
claude
committed
fixed pom
robfrank
committed
wip
robfrank
committed
feat(ha-raft): replica-to-leader auth forwarding with cluster token

robfrank
and
claude
committed
test(ha-raft): add ratis-test dependency for MiniRaftCluster support

robfrank
and
claude
committed
feat(ha-raft): add HA_RAFT_PERSIST_STORAGE and HA_RAFT_SNAPSHOT_THRESHOLD config; wire into RaftHAServer

robfrank
and
claude
committed
test(ha-raft): add persistentRaftStorage() + restartServer() to BaseRaftHATest

robfrank
and
claude
committed
feat(ha-raft): replica crash-and-recover test with Raft log replay

robfrank
and
claude
committed
test(ha-raft): add RaftLeaderCrashAndRecoverIT with leader rejoin and DatabaseComparator

robfrank
and
claude
committed
feat(ha-raft): add takeSnapshot() for log compaction + RaftFullSnapshotResyncIT stub

robfrank
and
claude
committed
fix(ha-raft): code quality fixes from review

robfrank
and
claude
committed
test(ha-raft): add BaseMiniRaftTest using MiniRaftClusterWithGrpc

robfrank
and
claude
committed
test(ha-raft): implement split-brain tests via MiniRaftClusterWithGrpc

robfrank
and
claude
committed
test params increased
robfrank
committed
wip
robfrank
committed
version fix
robfrank
committed
feat(ha-raft): implement leader command forwarding via HTTP and fix plugin discovery

robfrank
and
claude
committed
feat(ha-raft): improve Raft log readability with clean cluster event messages

robfrank
and
claude
committed
test parameters
robfrank
committed
feat(e2e-ha): add TestContainers + Toxiproxy e2e tests for Raft HA

robfrank
and
claude
committed
fix(e2e-ha): use Docker network disconnect for partition tests

robfrank
and
claude
committed
fix(e2e-ha): use DNS-valid container hostnames and improve test resilience

robfrank
and
claude
committed
add workflow for e2e-ha tests
robfrank
committed
fix sha
robfrank
committed
remove duplicated server dependency
robfrank
committed
try to have the wf visible on pr
robfrank
committed
add logback conf to ha-tests
robfrank
committed
fix(ha-raft): suppress Ratis DEBUG noise and fix missing root password in tests

robfrank
and
claude
committed
fix npe
robfrank
committed
fix npe
robfrank
committed
fix tentative
robfrank
committed
fix tentative
robfrank
committed
fix(ha-raft): refresh RaftClient gRPC channels on leader change to recover from partition

robfrank
and
claude
committed
fix(e2e-ha): wait for Raft leader election before writing during rolling restart

robfrank
and
claude
committed
fix(e2e-ha): retry assertThatUserCountIs to tolerate Raft replication lag

robfrank
and
claude
committed
rebasewd
robfrank
committed
build the image in workflow
robfrank
committed
fix(e2e-ha): fix flaky ConditionTimeoutException in NetworkPartitionIT and RollingRestartIT

robfrank
and
claude
committed
docs: add HA-raft test porting design spec

robfrank
and
claude
committed
docs: add ha-raft test porting implementation plan

robfrank
and
claude
committed
test(ha-raft): port HTTP layer IT tests from server/ha

robfrank
and
claude
committed
fix(ha-raft-tests): fix AssertJ withFailMessage ordering and add cross-node replication check

robfrank
and
claude
committed
fix(ha-raft-tests): fix thread exit on null, port literal, replication wait, and assertThat pattern

robfrank
and
claude
committed
test(ha-raft): port write forwarding and schema/view IT tests from server/ha

robfrank
and
claude
committed
fix(ha-raft): throw ServerIsNotTheLeaderException for schema changes on followers, fix schema version double-increment

robfrank
and
claude
committed
fix(ha-raft-tests): replace ArrayIndexOutOfBoundsException assertion with TODO, document HTTP forwarding

robfrank
and
claude
committed
test(ha-raft): port index compaction and operations IT tests from server/ha

robfrank
and
claude
committed
test(ha-raft): re-enable index tests that don't require compaction replication

robfrank
and
claude
committed
refactor(ha-raft-tests): promote findLeaderIndex to BaseRaftHATest, improve lsmVectorReplication Disabled reason

robfrank
and
claude
committed
test(ha-raft): port database utilities, config validation, and chaos IT tests from server/ha

robfrank
and
claude
committed
fix(ha-raft-tests): address code quality issues in Task 4 tests and production code

robfrank
and
claude
committed
fix(ha-raft): restore arcadedb-integration test dependency for backup SQL command

robfrank
and
claude
committed
style(ha-raft): replace em dashes with regular dashes in Task 4 files

robfrank
and
claude
committed
test(ha-raft): port HAInsertBenchmark from apache-ratis branch as RaftHAInsertBenchmark

robfrank
and
claude
committed
fix(benchmark): add 1s port-release delay between benchmark scenarios

robfrank
and
claude
committed
fix(benchmark): move to port range 3480/3434 to avoid Oracle port conflict

robfrank
and
claude
committed
fix(test): write only via leader in RaftServerDatabaseSqlScriptIT

robfrank
and
claude
committed
docs: add design spec for porting apache-ratis improvements to ha-redesign

robfrank
and
claude
committed
docs: add implementation plan for porting apache-ratis improvements

robfrank
and
claude
committed
feat(ha-raft): add HALog verbose logging utility with configurable levels

robfrank
and
claude
committed
feat(ha-raft): add LZ4 compression for WAL data in Raft log entries

robfrank
and
claude
committed
feat(ha-raft): add Quorum enum replacing string-based HA_QUORUM config

robfrank
and
claude
committed
perf(ha-raft): capture group-commit baseline benchmark numbers

robfrank
and
claude
committed
feat(ha-raft): upgrade ArcadeStateMachine to SimpleStateMachineStorage with election metrics

robfrank
and
claude
committed
feat(ha-raft): add RaftGroupCommitter for batched Raft submissions

robfrank
and
claude
committed
fix(ha-raft): update ArcadeStateMachineTest for BaseStateMachine INITIAL_VALUE behavior

robfrank
and
claude
committed
feat(studio): port enhanced cluster monitor UI from apache-ratis branch

robfrank
and
claude
committed
feat(ha-raft): wire snapshot install to Ratis + migrate debug logging to HALog

robfrank
and
claude
committed
test(ha-raft): add RaftReadConsistencyIT for linearizable read verification
robfrank
committed
docs: add ha-redesign vs apache-ratis branch comparison report

robfrank
and
claude
committed
docs: update ha-redesign vs apache-ratis comparison with latest changes

robfrank
and
claude
committed
fix(ha-raft): fix ALL quorum race in group committer, add HALog level caching

robfrank
and
claude
committed
fix(ha-raft): add cluster token auth for snapshots, NIO zip-slip, WAL deletion logging

robfrank
and
claude
committed
fix(ha-raft): PBKDF2 cluster token, configurable election timeouts and buffer sizes

robfrank
and
claude
committed
docs: update branch comparison after porting security and config fixes

robfrank
and
claude
committed
fix(ha-raft): guard against null raftHAServer during server restart

robfrank
and
claude
committed
fix(ha-raft): propagate ArcadeDB exceptions from Raft commit instead of wrapping in TransactionException

robfrank
and
claude
committed
- docs: Ratis version bump in comparison doc U
robfrank
committed
fix(ha-raft): fix stale raftHAServer reference after server restart

robfrank
and
claude
committed
fix(ha-raft): wait for Raft replication before database comparison in endTest

robfrank
and
claude
committed
fix(ha-raft): use 3 nodes in snapshot resync test so writes succeed with one down

robfrank
and
claude
committed
docs(ha-raft): update @Disabled message for lsmVectorReplication test

robfrank
and
claude
committed
fix(ha-raft): prevent Raft log purging in crash test that causes infinite hang

robfrank
and
claude
committed
fix(ha-raft): force leadership transfer after crash restart to fix stale gRPC channels

robfrank
and
claude
committed
fix(ha-raft): disable persistent Raft storage in crash test to avoid corrupted log recovery

robfrank
and
claude
committed
test diaabled
robfrank
committed
refactor(server): relocate ServerSocketFactory to com.arcadedb.server.network

robfrank
and
claude
committed
feat(ha): remove legacy HA implementation

robfrank
and
claude
committed
chore(config): remove legacy-only HA configuration entries

robfrank
and
claude
committed
feat(ha-raft): add dynamic membership, K8s auto-join, and read consistency

robfrank
and
claude
committed
fix(ha-raft): bounded Raft client retry and robust membership/leadership ops

robfrank
and
claude
committed
fix(ha-raft): fix verify database checksum comparison for cluster consistency

robfrank
and
claude
committed
fix(ha): forward server write commands to leader and make client cluster config resilient

robfrank
and
claude
committed
fix(e2e-ha): force leadership transfer after partition heals to refresh gRPC channels

robfrank
and
claude
committed
fix(e2e-ha): restore Docker network aliases on reconnect after partition

robfrank
and
claude
committed
fix(e2e-ha): replace unsupported 'none' quorum with 'majority' in Raft tests

robfrank
and
claude
committed
fix(network): remove redundant Authenticator from HttpClient to fix 401 handling

robfrank
and
claude
committed
fix(e2e-ha): increase PacketLossIT convergence timeouts to 180s

robfrank
and
claude
committed
fix(server): fix RaftHAPlugin auto-discovery when not in server.plugins list

robfrank
and
claude
committed
fix(server): include X-ArcadeDB-Forwarded-User header when forwarding server commands

robfrank
and
claude
committed
fix(server): forward Basic auth as-is instead of using cluster token for server commands

robfrank
and
claude
committed
docs: add 3-phase commit port design spec

robfrank
and
claude
committed
docs: add 3-phase commit port implementation plan

robfrank
and
claude
committed
refactor(ha-raft): add ReplicationPayload record for 3-phase commit

robfrank
and
claude
committed
docs: update branch comparison after 3-phase commit port

robfrank
and
claude
committed
fix(ha-raft): persist deferred schema save for read-only leader transactions

robfrank
and
claude
committed
fix(ha-raft): fix schema save under lock and exception propagation in 3-phase commit

robfrank
and
claude
committed
add benchmark results
robfrank
committed
feat(ha-raft): port logging improvements from apache-ratis branch

robfrank
and
claude
committed
fix(e2e-ha): fix schema comparison and test timeouts in HA e2e tests

robfrank
and
claude
committed
fix(e2e-ha): restart isolated nodes after partition to fix gRPC channel recovery

robfrank
and
claude
committed
feat(ha-raft): save benchmark results to target/reports/RaftHAInsertBenchmark.txt

robfrank
and
claude
committed
fix(e2e-ha): remove compareAllDatabases from HA test tearDown

robfrank
and
claude
committed
feat(ha-raft): add DROP_DATABASE_ENTRY and SECURITY_USERS_ENTRY log entry types
robfrank
committed
refactor(ha-raft): extend DecodedEntry with usersJson and forceSnapshot slots
robfrank
committed
feat(ha-raft): encode/decode DROP_DATABASE_ENTRY
robfrank
committed
feat(ha-raft): add forceSnapshot flag to INSTALL_DATABASE_ENTRY codec
robfrank
committed
feat(ha-raft): encode/decode SECURITY_USERS_ENTRY
robfrank
committed
feat(server): add HAReplicatedDatabase.dropInReplicas() interface method
robfrank
committed
feat(ha-raft): implement dropInReplicas and createInReplicas(forceSnapshot)
robfrank
committed
feat(ha-raft): apply DROP_DATABASE_ENTRY in state machine
robfrank
committed
feat(server): route drop database through Raft when HA is enabled
robfrank
committed
test(ha-raft): integration test for drop database propagation across 3 nodes
robfrank
committed
feat(ha-raft): honour forceSnapshot flag in applyInstallDatabaseEntry
robfrank
committed
feat(server): replicate restored database to replicas via forceSnapshot install
robfrank
committed
fix(ha-raft): fix snapshot install for drop+restore and concurrent requests
robfrank
committed
test(ha-raft): integration test for restore database propagation
robfrank
committed
feat(server): create imported database via Raft before running importer
robfrank
committed
fix(engine): unwrap database in ImportDatabaseStatement for HA replication
robfrank
committed
test(ha-raft): integration test for import database propagation
robfrank
committed
feat(security): expose JSON payload helpers for HA user replication
robfrank
committed
feat(server): add HAServerPlugin.replicateSecurityUsers default method
robfrank
committed
feat(ha-raft): implement RaftHAPlugin.replicateSecurityUsers
robfrank
committed
feat(ha-raft): apply SECURITY_USERS_ENTRY in state machine
robfrank
committed
feat(server): route create user through Raft when HA is enabled
robfrank
committed
feat(server): route drop user through Raft when HA is enabled
robfrank
committed
test(ha-raft): integration test for user replication across 3 nodes
robfrank
committed
fix(security): drop synchronized on ServerSecurity users hooks to unblock Raft apply
robfrank
committed
feat(ha-raft): seed new peer with current users after peer-add
robfrank
committed
test(ha-raft): integration test for peer-add user seed
robfrank
committed
test(e2e-ha): container scenario for drop database propagation
robfrank
committed
test(e2e-ha): container scenario for restore database propagation
robfrank
committed
test(e2e-ha): container scenario for import database propagation
robfrank
committed
test(e2e-ha): container scenario for user management propagation
robfrank
committed
test(e2e-ha): container smoke test for peer-add seed endpoint wiring
robfrank
committed

Commits on Apr 12, 2026

feat(ha): port production-resilience features from apache-ratis branch

robfrank
and
claude
committed
fix(server): share single LeaderProxy per HttpServer instead of per handler

robfrank
and
claude
committed
fix(ha-raft): disable HealthMonitor in tests to prevent thread exhaustion

robfrank
and
claude
committed

Uh oh!

Commits

Commits on Apr 11, 2026