craft: in-memory reference model + client-facing API surface

docs/craft/README.md

@@ -40,14 +40,18 @@ synchronization, and recovery bookkeeping. Write data never flows through the RA
 | **InternalLogin** | A RAFT log entry type. On apply, stores the new `client_token` and enforces single-writer exclusivity. |
 | **Missing** | A dLSN slot that a replica knows about (from a peer or from the RAFT log) but has not yet received data for. |
 | **Empty** | A dLSN proven never quorum-durable, declared by the leader; a permanent no-op the commit skips. |
-| **all_zeros (zero write)** | A payload-free write naming just an LBA range (the WRITE_ZEROES / discard-to-zero op). Takes a dLSN and merges like any write; allocates nothing and unmaps its range on apply. |
+| **zero write (all_zeros)** | A payload-free write naming just a byte range (WRITE_ZEROES / discard-to-zero), signaled by an **empty `sisl::sg_list`** (no `all_zeros` flag). Takes a dLSN and merges like any write; allocates nothing and unmaps its range on apply, reading back as a hole. |
 | **hole** | A read sub-range with no data (never written, zero-written, or an all-zero region collapsed at read time). Returned as a marker, read as zeros; **not** the same as `Missing`. |
+| **client_hdr** | The session + watermark fields stamped on every client IO: `{term, commit_lsn, all_committed_lsn}` (see the commit note below). |
+| **lba_size** | The volume block size in bytes, returned by `login`; the client aligns every byte `addr`/`len` to it and presents the geometry to the filesystem. |
+| **io_extent** | One sub-range of a read's sparse layout, in **bytes**: `{addr, len, hole}` — carries no bytes (they are in the caller's buffer). |
 
 ## Key design properties
 
 - **Single writer**: only one client at a time owns a partition (enforced by `InternalLogin` RAFT entry).
 - **Leaderless data path**: after login, the RAFT leader has no special role for writes or reads.
-- **Client drives commit**: replicas apply to the index only when told (via `commit` / `keep_alive`), strictly in dLSN order at the contiguous frontier. Readability is per-write and does not wait: appended entries are served from the journal-tail overlay (no index write on the read path).
+- **Client drives commit (piggybacked, no standalone verb)**: the client stamps its `commit_lsn` on every write / read / keep_alive (`client_hdr`); replicas apply to the index strictly in dLSN order at the contiguous frontier. `keep_alive` is the dedicated carrier (commit + watchdog reset, term-fenced). Readability is per-write and does not wait: appended entries are served from the journal-tail overlay (no index write on the read path).
+- **Byte-based, one buffer type**: `addr`/`len` are byte offsets/lengths (block-aligned to `lba_size`); `sisl::sg_list` is the single caller-owned buffer both ways (an **empty** write buffer is a zero write). A read fills the caller's buffer in place and returns a sparse `io_extent` layout (data vs holes).
 - **Client-routed reads**: reads are unicast, chosen by LBA-overlap against the client's per-replica Missing map (plus `Synced ≥ L`, the login dLSN). The read path never fetches from a peer; fetch is resync-only.
 - **Merge key, not serialization**: overlapping writes need no ordering; highest dLSN per LBA wins on every replica.
 - **Thin from the start**: a write may be `all_zeros` (WRITE_ZEROES) with no payload; reads return sparse results (data extents + holes), and the server collapses all-zero reads to holes, so reads and resync stay thin. A hole is not `Missing`.

docs/craft/rpcs.md

@@ -1,12 +1,35 @@
 # CRAFT RPCs
 
-8 RPCs total. 4 client↔server, 4 server↔server (2 via RAFT, 2 non-RAFT).
+8 RPCs total. 4 client↔server (Login, Write, Read, KeepAlive), 4 server↔server (2 via RAFT, 2 non-RAFT).
 RAFT internal RPCs (heartbeat, vote, membership) are not listed here.
 
+> **Commit is not a separate RPC.** The commit watermark rides on `client_hdr.commit_lsn`, piggybacked
+> on Write / Read / KeepAlive; KeepAlive is its dedicated carrier (commit + watchdog reset). Readability
+> is per-write and does not wait for it (journal-tail overlay reads).
+
 > **Canonical design:** the [**CRAFT Design**](https://github.com/eBay/HomeBlocks/wiki/CRAFT-Design)
 > wiki page is the source of truth for the protocol, and
 > [**CRAFT on HomeBlocks**](https://github.com/eBay/HomeBlocks/wiki/CRAFT-on-HomeBlocks) for the
-> implementation binding; this file is the RPC wire-format reference.
+> implementation binding; this file is the RPC wire-format reference. The C++ API is in [api.md](api.md).
+
+---
+
+## Common header
+
+Every client IO (Write, Read, KeepAlive) carries a `client_hdr`:
+
+```
+client_hdr: { term: uint64, commit_lsn: int64, all_committed_lsn: int64 }
+```
+
+- `term` — fences a stale writer (rejected `ETERM`); checked on **every** IO including KeepAlive, so a
+  deposed client cannot even reset the liveness watchdog.
+- `commit_lsn` — advance the contiguous commit frontier toward this, best-effort, in dLSN order
+  (`-1` = don't advance). This is CRAFT's commit, piggybacked.
+- `all_committed_lsn` — set-wide min commit_lsn; floors journal reclaim (`-1` = unknown).
+
+**Addressing is byte-based:** `addr` / `len` are byte offset/length, block-aligned to the volume's
+`lba_size` (from Login), else `EINVAL`.
 
 ---
 
@@ -16,15 +39,14 @@ RAFT internal RPCs (heartbeat, vote, membership) are not listed here.
 
 ```
 Request:  { client_token: uint64 }
-Response: { members: [endpoint], dLSN: int64, term: uint64 }
+Response: { members: [endpoint], dLSN: int64, term: uint64, lba_size: uint32 }
 ```
 
-Client sends to the RAFT leader. Leader runs the full login orchestration sequence
-(GetRSCommitLSN → optional FetchData → SyncRSCommitLSN RAFT → InternalLogin RAFT)
-and responds once both RAFT entries commit.
-
-`client_token` is an opaque 64-bit identity token. `dLSN` is the per-partition LSN, the only
-LSN CRAFT carries.
+Client sends to the RAFT leader (a follower replies `NOT_LEADER` + leader endpoint). Leader runs the
+full login orchestration (GetRSCommitLSN → optional FetchData → SyncRSCommitLSN RAFT → InternalLogin
+RAFT) and responds once both RAFT entries commit. `dLSN` is the starting per-partition LSN. `lba_size`
+is the volume block size in bytes — the client aligns every addr/len to it and presents the geometry to
+the filesystem.
 
 HomeBlocks handler: `CraftReplDev::login()`
 
@@ -33,20 +55,18 @@ HomeBlocks handler: `CraftReplDev::login()`
 ### 2. Write (Broadcast to all replicas)
 
 ```
-Request:  { term: uint64, lsn: int64, lba: uint64, len: uint32, all_zeros: bool, data: bytes }
+Request:  { hdr: client_hdr, dlsn: int64, addr: uint64, len: uint64, data: bytes }
 Response: { status: Status }
 ```
 
-Client sends to every replica in the set in parallel. Each replica appends the entry to its
-data journal at slot `lsn` and ACKs immediately. Write is durable once quorum ACKs, and
+Client sends to every replica in the set in parallel at the client-assigned `dlsn`. Each replica
+appends the entry to its data journal at slot `dlsn` and ACKs immediately. Durable once quorum ACKs,
 readable once Appended (served from the journal-tail overlay until applied).
 
-Two forms. A **data write** (`all_zeros=false`) carries `data`; zero-copy is required (`data`
-must not be copied during journal append). A **zero write** (`all_zeros=true`, the WRITE_ZEROES
-/ discard-to-zero op) omits `data`: the slot is metadata-only, allocates no blocks, and on apply
-unmaps its range. Both take a `dLSN` and merge by highest-`dLSN`-per-LBA. The client sets
-`all_zeros` when a client-side scan finds an all-zero buffer; the server does **not** re-scan a
-data write.
+**Empty `data` (length 0) is a zero write** (WRITE_ZEROES / discard-to-zero): the slot is metadata-only,
+allocates no blocks, and on apply unmaps its range (`len` is the range). Non-empty `data` is a data
+write of exactly `len` bytes. There is **no `all_zeros` flag** — the empty buffer is the signal. Both
+take a `dLSN` and merge by highest-`dLSN`-per-LBA. `hdr.commit_lsn` piggybacks a frontier advance.
 
 HomeBlocks handler: `CraftReplDev::write()`
 
@@ -55,102 +75,77 @@ HomeBlocks handler: `CraftReplDev::write()`
 ### 3. Read (Unicast to chosen replica)
 
 ```
-Request:  { term: uint64, readLSN: int64, lba: uint64, len: uint32 }
-Response: { status: Status, extents: [{ lba: uint64, len: uint32, hole: bool, data: bytes }] }
+Request:  { hdr: client_hdr, read_lsn: int64, addr: uint64, len: uint64 }
+Response: { status: Status, layout: [{ addr: uint64, len: uint64, hole: bool }], data: bytes }
 ```
 
-`readLSN` is the read horizon `H` (the client's contiguous quorum-acked prefix); the read
-reflects writes ≤ `H`. The client picks an *eligible* replica: one filled to the login dLSN
-`L` (`Synced ≥ L`) whose Missing set has no slot ≤ `H` overlapping `[lba, lba+len)`. The
-replica returns the latest version ≤ `H` for the range, from the LBA index if applied or
-straight from the journal-tail overlay if only appended (an **overlay read**; no index write
-on the read path). A replica **ignores any write above `H` even if it holds it** (the
-sub-quorum tail), which is the server-side half of read safety. Because the client only routes
-to a servable
-replica, **the read path never fetches from a peer**; large multi-LBA reads may be split
-across replicas.
+`read_lsn` is the read horizon `H`. The client picks an *eligible* replica (one filled to the login
+dLSN `L`, with no Missing slot ≤ `H` overlapping the range). The replica returns the latest version ≤
+`H` for `[addr, addr+len)`, from the LBA index if applied or straight from the journal-tail overlay if
+only Appended (an **overlay read**; no index write). It **ignores any write above `H` even if it holds
+it** (the sub-quorum tail). The read path **never fetches from a peer**; large reads may be split across
+replicas.
 
-The response is **sparse**: data extents interleaved with **holes** (`hole=true`, no `data`). A
-hole is an LBA sub-range that is never-written, zero-written (`all_zeros`), or an all-zero region
-the replica collapsed by a read-time scan; the client reads it as zeros. A hole is **not**
-`Missing` (the client routes around Missing slots, never around holes).
+The response is **sparse**: `layout` marks which byte sub-ranges are **data** vs **holes**
+(`hole=true`: never-written, zero-written, or an all-zero region collapsed by a read-time scan — read
+as zeros, **not** `Missing`); `data` carries the bytes for the data extents. The client places them
+into its caller-owned (iomgr) destination buffer per the layout (holes → zeros). `hdr.commit_lsn`
+piggybacks a frontier advance.
 
 HomeBlocks handler: `CraftReplDev::read()`
 
 ---
 
-### 4. Commit (Broadcast to all replicas)
-
-```
-Request:  { term: uint64, lsn: int64 }
-Response: { status: Status, commit_lsn: int64, last_append_lsn: int64 }
-```
-
-Tells replicas to advance `commit_lsn` to `lsn`. May be piggybacked on the next
-Write or KeepAlive instead of sent as a standalone RPC. The response carries the **achieved**
-`{commit_lsn, last_append_lsn}` (best-effort: `commit_lsn` stalls below the first Missing hole).
-
-HomeBlocks handler: `CraftReplDev::commit()`
-
----
-
-### 5. KeepAlive (Broadcast to all replicas)
+### 4. KeepAlive (Broadcast to all replicas)
 
 ```
-Request:  { commit_lsn: int64, all_committed_lsn: int64 }
+Request:  { hdr: client_hdr }
 Response: { status: Status, commit_lsn: int64, last_append_lsn: int64 }
 ```
 
-Advances `commit_lsn` (in-order apply) and resets the per-replica client-timeout watchdog.
-The response carries `{commit_lsn, last_append_lsn}` (feeds the client's Missing map and the
-reconfig promotion gate); the request carries `all_committed_lsn`, the set-wide min the
-client computed from those responses, letting each replica reclaim journal opportunistically
-below it. Sent periodically during idle periods and after every quorum-acknowledged write.
+The dedicated commit carrier: advances `commit_lsn` (in-order apply, skipping Empty, reclaiming
+superseded blocks) toward `hdr.commit_lsn` **and resets the per-replica client-timeout watchdog** (which
+is why it is term-fenced — a stale client must not keep the session alive). The response carries the
+**achieved** `{commit_lsn, last_append_lsn}` (best-effort: stalls below the first Missing hole), which
+feeds the client's Missing map and the reconfig promotion gate. `hdr.all_committed_lsn` lets the replica
+reclaim journal below it. Sent periodically during idle periods and after quorum-acknowledged writes.
 
 HomeBlocks handler: `CraftReplDev::keep_alive()`
 
 ---
 
 ## Server → Server (non-RAFT)
 
-### 6. GetRSCommitLSN (Broadcast, initiated by leader)
+### 5. GetRSCommitLSN (Broadcast, initiated by leader)
 
 ```
 Request:  { term: uint64, my_commit_lsn: int64, my_last_append_lsn: int64, is_login: bool }
 Response: { term: uint64, commit_lsn: int64, last_append_lsn: int64 }
 ```
 
-Leader sends to all peers to collect their current LSN state before a `SyncRSCommitLSN`
-RAFT proposal. Used during login and on timeout. `is_login=true` (the login poll) makes the
-peer **quiesce prior-session writes** before reporting `last_append` (the fencing barrier);
-watchdog / periodic polls carry `is_login=false` and never quiesce (they ride the live tail).
-Only `last_append_lsn` feeds the recovery watermark; `commit_lsn` is a contiguity
-certificate that bounds the leader's hole-resolution window, seeds `all_committed_lsn`
-(journal reclaim) when no client is attached, and carries the reconfig promotion gate.
+Leader polls peers for their LSN state before a `SyncRSCommitLSN` proposal (login and on timeout).
+`is_login=true` makes the peer **quiesce prior-session writes** before reporting `last_append` (the
+fencing barrier); watchdog/periodic polls carry `is_login=false` and never quiesce. Only `last_append`
+feeds the recovery watermark; `commit_lsn` is a contiguity certificate that bounds the leader's
+hole-resolution window, seeds `all_committed_lsn`, and carries the reconfig promotion gate.
 
 HomeBlocks handler: `CraftReplDev::get_rs_commit_lsn()` / `get_lsns()`
 Dispatched by: `CraftConnector` (inter-node channel, non-RAFT)
 
 ---
 
-### 7. FetchData (Unicast, from behind replica to an ahead peer)
+### 6. FetchData (Unicast, from behind replica to an ahead peer)
 
 ```
 Request:  { lsns: [int64] }
 Response: { slots: [{ lsn: int64, is_empty: bool, all_zeros: bool, lba: uint64, len: uint32, data: bytes }] }
 ```
 
-Called when a replica discovers it is missing data for certain LSNs after receiving a
-`SyncRSCommitLSN` RAFT entry. **Four-way per slot:** an entry with `is_empty=false,
-all_zeros=false` carries `data`; `all_zeros=true` is a **zero write** (no `data`, applied as a
-range unmap); `is_empty=true` means the peer has **positively** marked that slot `Empty` in a
-prior resync; a requested `lsn` **omitted** from `slots` means *not-present-here*. A peer never
-returns `is_empty=true` for a slot it merely lacks. The `Empty` **verdict itself is
-leader-only**: the leader runs the broadcast-and-accumulate quorum procedure (itself
-included; non-responders never count) before proposing `SyncRSCommitLSN`, and distributes
-verdicts in that entry (`empty_slots[]`). Lagging replicas fetch present slots and obey the
-verdict list; they never declare `Empty` unilaterally (see the wiki's Resync section for the
-intersection argument and the Empty-beats-held-data reconciliation rule).
+Called when a replica discovers it is missing data for certain LSNs after a `SyncRSCommitLSN` entry.
+**Four-way per slot:** `is_empty=false, all_zeros=false` carries `data`; `all_zeros=true` is a zero write
+(no data, applied as a range unmap); `is_empty=true` means the peer **positively** marked that slot
+`Empty` (leader-only verdict); a requested `lsn` **omitted** means *not-present-here*. A peer never
+returns `is_empty=true` for a slot it merely lacks.
 
 HomeBlocks handler: `CraftReplDev::fetch_data()`
 Dispatched by: `CraftConnector` (inter-node channel, non-RAFT)
@@ -159,42 +154,40 @@ Dispatched by: `CraftConnector` (inter-node channel, non-RAFT)
 
 ## Server → Server (RAFT)
 
-### 8. SyncRSCommitLSN (RAFT proposal, from leader)
+### 7. SyncRSCommitLSN (RAFT proposal, from leader)
 
 ```
 RAFT entry payload: { rs_commit_lsn: int64, client_token: uint64, empty_slots: [int64] }
 ```
 
-Proposed by the leader via `CraftReplDev::append()`. **Before proposing**, the leader
-resolves every unresolved slot ≤ `rs_commit_lsn`: fetch it from any holder, or record an
-`Empty` verdict on quorum-lacks evidence; it never proposes past an unresolved slot. On
-RAFT commit each replica applies the entry: verify the token against the current session,
-mark `empty_slots` as permanent no-op holes (discarding any local data in them), fetch the
-remaining missing slots from peers (the leader is guaranteed to hold every non-Empty slot ≤
-the watermark), then advance `commit_lsn`. Replicas never declare `Empty` unilaterally.
-This is the primary recovery mechanism — it carries no write data, only the watermark and
-verdicts.
+Proposed by the leader via `CraftReplDev::append()`. **Before proposing**, the leader resolves every
+unresolved slot ≤ `rs_commit_lsn`: fetch from any holder, or record an `Empty` verdict on
+quorum-lacks evidence; it never proposes past an unresolved slot. On RAFT commit each replica: verify
+the token, mark `empty_slots` as permanent no-op holes (discarding any local data there), fetch the
+remaining missing slots from peers, then advance `commit_lsn`. Replicas never declare `Empty`
+unilaterally. This is the primary recovery mechanism — it carries no write data, only the watermark
+and verdicts.
 
 ---
 
-### 9. InternalLogin (RAFT proposal, from leader during login)
+### 8. InternalLogin (RAFT proposal, from leader during login)
 
 ```
 RAFT entry payload: { client_token: uint64, term: uint64 }
 ```
 
-Proposed by the leader after `SyncRSCommitLSN` commits. On apply each replica stores
-`client_token` and `term`, rejecting any subsequent IO from a different token. Proposed
-immediately after the `SyncRSCommitLSN` entry during the login sequence.
+Proposed after `SyncRSCommitLSN` commits. On apply each replica stores `client_token` and `term`,
+rejecting any subsequent IO from a different term. Proposed immediately after the `SyncRSCommitLSN`
+entry during the login sequence.
 
 ---
 
 ## RPC Transport
 
 The transport layer for NubloxProto RPCs is decided by the **CRAFT-1 spike** (SDSTOR-22297
-dependency). `CraftConnector` is transport-agnostic: it will dispatch via whatever channel
-CRAFT-1 selects (likely gRPC or a custom framing over TCP). Server-to-server RPCs (6 and 7)
-use the same transport.
+dependency). `CraftConnector` is transport-agnostic: it will dispatch via whatever channel CRAFT-1
+selects (likely gRPC or a custom framing over TCP). Server-to-server RPCs use the same transport.
 
-During development, before CRAFT-1 lands, `CraftConnector` can use direct C++ function
-calls or a stub transport for unit/integration testing.
+During development, before CRAFT-1 lands, the calls are exercised in-process via direct function calls:
+the [in-memory reference model](../../src/lib/craft/memory/) implements the same per-replica surface
+(`craft_replica`) behind a `volume_handle`, with `MemTransport` standing in for the network.