Optimize mongodb_v2 and migrate to mongo-driver v2 - #442
Conversation
Performance optimizations for mongodb_v2 eventstore: - Move global position update outside transaction to reduce lock contention on the $all stream document - Extract updateStreamForEntity and updateGlobalPosition helpers - Update stream before inserting events for fail-fast optimistic locking with duplicate-key normalization - Add WithSortEventsOnDB option for version-based event ordering - Clone metadata map in newEvt to prevent caller mutation on transaction failure - Fix CountDocuments error check order in Replace - Handle both int32 and int64 in position ID verification with default case for unexpected types Migrate all MongoDB packages from mongo-driver v1 to v2: - Consolidate bsoncodec, bsonrw, bsontype imports into bson package - Export custom Registry for UUID codec with Marshal/Unmarshal helpers that ensure backward-compatible string encoding - UUID decoder handles both string and binary formats - Pass Registry via database options SetRegistry for all packages - Use bsoncodec.Marshal/Unmarshal for event data serialization - Replace mongo.Connect(ctx, opts) with mongo.Connect(opts) - Replace writeconcern.New(writeconcern.WMajority()) with writeconcern.Majority() - Replace mongo.SessionContext with context.Context - Replace options.Update() with options.UpdateOne() - Update FindOptions to FindOptionsBuilder for v2 Lister interface - Use errors.Is for ErrNoDocuments comparison - Fix import grouping to follow stdlib / 3rd party / internal Testing: - Add testcontainers for all MongoDB integration tests - Refactor TestMain to use runWithMongo helper to prevent container leaks on os.Exit - Add backward compatibility test for v1 encoded data Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
b85ab87 to
1c59b3e
Compare
|
@klowdo What do you think? |
Global position ordering trade-offMoving This is fine for many use cases, but not all. Some consumers (projectors, catch-up subscriptions) may rely on position reflecting commit order. Suggestion: Make this behavior configurable with three modes:
Something like: WithGlobalPositionStrategy(GlobalPositionInTX) // default, backward compat
WithGlobalPositionStrategy(GlobalPositionOutsideTX) // this PR's behavior
WithGlobalPositionStrategy(GlobalPositionDisabled) // skip $all entirelyThis way existing users get no behavior change, and users who understand the trade-off can opt into the faster path. The outbox is unaffected regardless — it uses |
|
Correction on my previous comment — disabling global position entirely is not viable since sagas depend on it. Updated suggestion: Two modes should be enough: WithGlobalPositionStrategy(GlobalPositionInTX) // default, backward compat
WithGlobalPositionStrategy(GlobalPositionOutsideTX) // this PR's behavior
|
|
@klowdo can you a quick review this, we are using this in production from our fork and everything is working fine |
|
only because you've used this in production doesn't mean that it work for everyone. in a heavy concurrency load the events can be stored in different order than creation. this gap could send out events in wrong order to saga/projection with can cause a ripple effect on the event stream. Why do you need to make this outside of the transaction? what kind of load do you have on your system? or this just a pre-optimization ? |
Co-authored-by: Felix Svensson <klowdo.fs@gmail.com>
…formance # Conflicts: # codec/bson/command.go # codec/bson/event.go # codec/bson/uuid.go # eventstore/mongodb/eventstore.go # eventstore/mongodb_v2/eventmaintenance.go # eventstore/mongodb_v2/eventstore.go # outbox/mongodb/outbox_test.go
Expose the global-position update strategy as a configurable option with two modes: - GlobalPositionInTX (default): increments the global position inside the save transaction. Backward-compatible: positions strictly reflect commit order and there are no gaps, at the cost of write-lock contention on the $all document under concurrent saves across aggregates. - GlobalPositionOutsideTX: increments the global position with an atomic FindOneAndUpdate before opening the save transaction. Removes $all as a serialization point and substantially improves throughput under high-concurrency / many-aggregate workloads. Trade-off: positions may have gaps and may not reflect commit order across aggregates. Per-aggregate version ordering is still guaranteed by the stream-document optimistic lock. The Save function dispatches to saveInTX or saveOutsideTX based on the strategy. saveInTX preserves the original upstream flow byte-for-byte for backward compatibility; saveOutsideTX implements the optimized path. Tests: - TestGlobalPositionOutsideTXIntegration runs the eventstore acceptance suite against the OutsideTX strategy. - TestGlobalPositionStrategyGapBehavior empirically demonstrates the trade-off by racing two saves on the same aggregate and asserting that $all advances by 1 in InTX (rolled back on abort) vs 2 in OutsideTX (gap on abort). - TestPerAggregateOrderingBothStrategies verifies per-aggregate version monotonicity holds under both modes. - BenchmarkSaveStrategiesConcurrent measures throughput and error rates under a disbursement-shaped workload (concurrent saves on different aggregates with a constrained connection pool).
…atrix
Replace BenchmarkSaveStrategiesConcurrent (a single concurrency/pool point)
with BenchmarkSaveStrategiesMatrix that sweeps:
- strategy: {InTX, OutsideTX}
- load profile: {burst, append}
- burst: a fresh aggregate per save (disbursement webhook pattern)
- append: sequential saves on pre-populated aggregates via an atomic
counter, so concurrent saves usually target distinct aggregates
(steady-state state-machine pattern)
- concurrency: {16, 64, 200}
- pool size: {20, 50, 200}
Add a warmupPool step so first-save TCP handshakes don't pollute the timed
window. Each cell uses one client, one DB, one EventStore shared by all
goroutines — matching how a real service is wired.
The matrix surfaces the contention knee: when concurrency exceeds pool size,
InTX latency grows ~5x to ~95x while OutsideTX stays roughly flat. This is
the operational pattern that motivated the optional OutsideTX strategy in
the first place.
Document the two global-position strategies (InTX default, OutsideTX opt-in), the trade-off they represent, which consumers are affected by gaps vs ordering, and the empirical benchmark matrix showing the contention knee at concurrency > pool size. Also documents how to reproduce the benchmark and how the trade-off behavior is asserted as a test.
|
Hey @klowdo — fair points, thanks for the pushback. You were right that "we use it in prod" alone isn't really an argument. Updated the PR so To answer your "why outside the transaction / what load / pre-optimization?" questions: Not pre-optimization. At AltScore we run a disbursements service that gets state-change webhooks from STP (Mexican payment network). Each webhook = Why it's safe for our specific workload:
On the gap behavior — you were also right. Added Benchmark:
Small heads up: looks like CI hasn't actually run since April — the runs sit on Let me know what you think — and if you want me to tweak the docs or add a godoc example for the option 🙏 |
|
coverage: 53.057% (+0.09%) from 52.965% — AltScore:feat/mongodb-v2-performance into looplab:main |
3 participants
Summary
updateStreamForEntityhelper for fail-fast optimistic locking. AddWithSortEventsOnDBoption for version-based event ordering on queries.Changes
mongodb_v2 eventstore optimizations
$allstream documentWithSortEventsOnDB()option for explicit version-based ordering on Load/LoadFromupdateStreamForEntity(two-value form)mongo-driver v2 migration (all packages)
bsoncodec,bsonrw,bsontypeconsolidated intobsonpackageRegistryexported fromcodec/bsonfor UUID string encoding (no global registry in v2)Marshal/Unmarshalhelpers ensure backward-compatible UUID encodingmongo.Connect(ctx, opts)replaced withmongo.Connect(opts)writeconcern.New(writeconcern.WMajority())replaced withwriteconcern.Majority()mongo.SessionContextreplaced withcontext.Contextin transaction callbacksoptions.Update()replaced withoptions.UpdateOne()FindOptionsreplaced withFindOptionsBuilderfor v2 Lister interfacebson.Dfor sort documents (ordered)Testing
Test plan
go test -short ./...— 41 packages passgo test ./eventstore/mongodb_v2/...— 6 integration tests passgo test ./eventstore/mongodb/...— 4 integration tests passgo test ./outbox/mongodb/...— 4 integration tests passgo test ./repo/mongodb/...— 2 integration tests pass🤖 Generated with Claude Code