node-event-storage

An optimized embedded event store for modern node.js, written in ES6.

📖 Full documentation on readthedocs.io

---

Why?

There is currently only a single other embedded event store for node/javascript: node-eventstore. It has a few drawbacks:

• Its API requires loading a full Event Stream before committing, making it unfit for frequently-restarting client applications.
• Its embeddable backends (TingoDB, NeDB) do not persist indexes and are slow on initial load.
• Events are fixed to one stream — creating overlapping projection streams is not possible.

node-event-storage is built from first principles for append-only workloads, giving you near-optimal write speed with no unnecessary overhead.

---

Installation

npm install event-storage

CommonJS / require() users: version 1.0 is ESM-only. If your project uses require() and migrating to ESM is not an option, install the 0.x series (npm install event-storage@0) which is functionally equivalent and retains full CJS support.

Quick Start

import { EventStore } from 'event-storage';

const eventstore = new EventStore('my-event-store', { storageDirectory: './data' });

eventstore.on('ready', () => {
    // Write events
    eventstore.commit('my-stream', [{ type: 'SomethingHappened', value: 42 }], 0, () => {
        console.log('Written!');
    });

    // Read events
    const stream = eventstore.getEventStream('my-stream');
    for (const event of stream) {
        console.log(event);
    }
});

Common Usage at a Glance

• Append events with optimistic concurrency via commit(stream, events, expectedVersion, cb).
• Read streams with getEventStream() and fluent options for revision/range/direction.
• Run cross-stream queries with query(types, matcher) and commit with its returned condition for DCB.
• Build projections with fromStreams() (joined streams) and createStream() for persistent filtered streams.
• Consume reliably with createConsumer() for durable at-least-once (or exactly-once with state) processing.

Key Features

Feature	Summary
Optimistic concurrency	Pass expectedVersion to commit() to guarantee conflict-free writes.
Flexible stream reading	Range queries, reverse iteration, and a fluent builder API.
Joined and derived streams	Combine any number of streams with a nested OR/AND selector tree into one globally-ordered virtual stream; filter events into reusable derived projection streams.
Object matchers	Support nested equality, array values (OR semantics), scalar operators ($gt / $gte / $lt / $lte / $eq / $ne), plus array-containment operators ($has / $hasAny), while still benefiting from O(1) discriminant routing on writes.
DCB	Configure typeAccessor to have per-type stream indexes maintained automatically, and use query() / Condition for fine-grained, query-scoped optimistic concurrency (Dynamic Consistency Boundaries).
Stream categories	Name streams <category>-<id> and query the whole category at once.
Durable consumers	At-least-once (and exactly-once with setState) event delivery with automatic position tracking.
Consistency guards	Build aggregates that enforce business invariants with built-in snapshotting.
Read-only mode	Open the store from a second process to build projections without touching the writer.
Crash safety	Torn writes detected and truncated on startup; automatic index repair via LOCK_RECLAIM; bounded, predictable data loss validated by a dedicated stress test.
Custom serialization	Plug in msgpack, protobuf, or any other codec.
Compression	Apply LZ4, zstd, or any other compression via the serializer option.
Access control hooks	preCommit / preRead hooks with per-stream metadata for authorization.

---

Joining Streams

Merge events from multiple streams in global insertion order with fromStreams. The selector tree alternates: OR at even depths, AND at odd depths:

// OR: all events from either stream, in insertion order
const joined = eventstore.fromStreams('view', ['orders', 'payments']);

// AND: only events indexed in both streams (intersection)
const matched = eventstore.fromStreams('view', [['customer-42', 'OrderPlaced']]);

// Nested — (stream-a AND stream-b) OR (stream-c AND stream-d)
const complex = eventstore.fromStreams('view', [
    ['stream-a', 'stream-b'],
    ['stream-c', 'stream-d']
]);

See Event Streams for the full selector algebra, categories, and derived projection streams.

---

DCB Example

const { stream, condition } = store.query(['OrderPlaced'], { payload: { customerId: 'cust-1' } });
const hasOpenOrder = stream.some((event) => event.status === 'open');

if (!hasOpenOrder) {
    store.commit('orders-cust-1', [{ type: 'OrderPlaced', customerId: 'cust-1' }], condition);
}

---

Object Matcher Syntax

Object matchers support nested equality, array values, scalar operators ($gt, $gte, $lt, $lte, $eq, $ne), and array-containment operators ($has, $hasAny).

• Scalar operators can be combined on one field (AND semantics), e.g. { $gte: 100, $lt: 1000 }.
• $has / $hasAny target array-valued fields and must be used on their own for that field.

For hot paths, reuse the same matcher object reference across calls when possible (instead of recreating equivalent objects each time) so matcher compilation and cache-based optimization paths can be reused.

{ payload: { type: ['OrderPlaced', 'OrderCancelled'] } }
{ payload: { amount: { $gte: 100, $lt: 1000 } } }
{ payload: { tags: { $has: 'featured' } } }
{ payload: { tags: { $hasAny: ['featured', 'priority'] } } }

---

HTTP API

To expose an event store over HTTP, see the companion package event-storage-http:

npm install event-storage-http

import EventStore from 'event-storage';
import { createEventStoreHttpServer } from 'event-storage-http';

const eventStore = new EventStore('my-store', { storageDirectory: './data' });
const server = createEventStoreHttpServer(eventStore);
server.listen(3000);

The package exposes NDJSON stream endpoints, durable consumer management, and an HttpEventStream client helper for consuming event streams over fetch.

---

Documentation

The full documentation is hosted at https://node-event-storage.readthedocs.io/en/latest/ and covers:

• Getting Started — installation, constructor options, basic usage.
• Event Streams — writing, reading, optimistic concurrency, fluent API, joining streams, categories, and event metadata.
• Dynamic Consistency Boundaries (DCB) — typeAccessor, query matchers, consistency tokens, and the full DCB workflow.
• Consumers — at-least-once and exactly-once delivery, consumer state, consistency guards, and read-only mode.
• Advanced Topics — ACID properties, reliability and crash-safety guarantees, storage configuration, partitioning, custom serialization, compression, security, and access control hooks.

---

Run Tests

npm test