diff --git a/docs/craft/README.md b/docs/craft/README.md index b957129..cd9ba6c 100644 --- a/docs/craft/README.md +++ b/docs/craft/README.md @@ -40,14 +40,18 @@ synchronization, and recovery bookkeeping. Write data never flows through the RA | **InternalLogin** | A RAFT log entry type. On apply, stores the new `client_token` and enforces single-writer exclusivity. | | **Missing** | A dLSN slot that a replica knows about (from a peer or from the RAFT log) but has not yet received data for. | | **Empty** | A dLSN proven never quorum-durable, declared by the leader; a permanent no-op the commit skips. | -| **all_zeros (zero write)** | A payload-free write naming just an LBA range (the WRITE_ZEROES / discard-to-zero op). Takes a dLSN and merges like any write; allocates nothing and unmaps its range on apply. | +| **zero write (all_zeros)** | A payload-free write naming just a byte range (WRITE_ZEROES / discard-to-zero), signaled by an **empty `sisl::sg_list`** (no `all_zeros` flag). Takes a dLSN and merges like any write; allocates nothing and unmaps its range on apply, reading back as a hole. | | **hole** | A read sub-range with no data (never written, zero-written, or an all-zero region collapsed at read time). Returned as a marker, read as zeros; **not** the same as `Missing`. | +| **client_hdr** | The session + watermark fields stamped on every client IO: `{term, commit_lsn, all_committed_lsn}` (see the commit note below). | +| **lba_size** | The volume block size in bytes, returned by `login`; the client aligns every byte `addr`/`len` to it and presents the geometry to the filesystem. | +| **io_extent** | One sub-range of a read's sparse layout, in **bytes**: `{addr, len, hole}` — carries no bytes (they are in the caller's buffer). | ## Key design properties - **Single writer**: only one client at a time owns a partition (enforced by `InternalLogin` RAFT entry). - **Leaderless data path**: after login, the RAFT leader has no special role for writes or reads. -- **Client drives commit**: replicas apply to the index only when told (via `commit` / `keep_alive`), strictly in dLSN order at the contiguous frontier. Readability is per-write and does not wait: appended entries are served from the journal-tail overlay (no index write on the read path). +- **Client drives commit (piggybacked, no standalone verb)**: the client stamps its `commit_lsn` on every write / read / keep_alive (`client_hdr`); replicas apply to the index strictly in dLSN order at the contiguous frontier. `keep_alive` is the dedicated carrier (commit + watchdog reset, term-fenced). Readability is per-write and does not wait: appended entries are served from the journal-tail overlay (no index write on the read path). +- **Byte-based, one buffer type**: `addr`/`len` are byte offsets/lengths (block-aligned to `lba_size`); `sisl::sg_list` is the single caller-owned buffer both ways (an **empty** write buffer is a zero write). A read fills the caller's buffer in place and returns a sparse `io_extent` layout (data vs holes). - **Client-routed reads**: reads are unicast, chosen by LBA-overlap against the client's per-replica Missing map (plus `Synced ≥ L`, the login dLSN). The read path never fetches from a peer; fetch is resync-only. - **Merge key, not serialization**: overlapping writes need no ordering; highest dLSN per LBA wins on every replica. - **Thin from the start**: a write may be `all_zeros` (WRITE_ZEROES) with no payload; reads return sparse results (data extents + holes), and the server collapses all-zero reads to holes, so reads and resync stay thin. A hole is not `Missing`. diff --git a/docs/craft/api.md b/docs/craft/api.md index 3625c4f..4585a9b 100644 --- a/docs/craft/api.md +++ b/docs/craft/api.md @@ -1,16 +1,50 @@ # HomeBlocks CRAFT C++ API -`CraftReplDev` is a new class (parallel to HomeStore's `ReplDisk`) that each CRAFT-mode -volume owns instead of a solo `repl_dev`. It exposes the following methods, which -`CraftConnector` calls 1-to-1 when translating incoming NubloxProto RPCs. +The CRAFT client-facing surface is a set of **free functions over a `volume_handle`** in the public +header `home_blocks.hpp` (parallel to the existing `async_read`/`async_write` block ops, which are now +`[[deprecated]]`). **One `volume_handle` == one replica device**: in production each is a different +server; the client holds one handle per member and does the CRAFT work itself (assign `dLSN`, +broadcast the write to every member, tally quorum, route the read to one member). The calls dispatch +through an internal per-replica interface, `craft_replica`, which both the production `CraftReplDev` +and the in-memory reference model (`src/lib/craft/memory/`) implement independently. -All methods are async/coroutine-style (`async_result` or `async_status`) matching the -existing HomeBlocks convention. +All methods are async/coroutine-style (`async_result` / `async_status`) matching the existing +HomeBlocks convention. > **Canonical design:** the [**CRAFT Design**](https://github.com/eBay/HomeBlocks/wiki/CRAFT-Design) > wiki page is the source of truth for the protocol, and > [**CRAFT on HomeBlocks**](https://github.com/eBay/HomeBlocks/wiki/CRAFT-on-HomeBlocks) for the -> implementation binding; this file is the C++ API reference. +> implementation binding; this file is the C++ API reference. Wire formats are in [rpcs.md](rpcs.md). + +--- + +## Addressing & buffer model + +- **Byte-based.** `addr` and `len` are **raw byte offsets/lengths** (not block indices), matching the + rest of the HomeBlocks public API. They must be aligned to the volume's `lba_size` (returned by + `login`), else the op fails with `std::errc::invalid_argument`. The byte↔block conversion is confined + to the server. +- **One buffer type both ways: `sisl::sg_list`** (caller-owned, e.g. iomgr-allocated and long-lived). + A **write** takes a source buffer; an **empty** `sg_list` (`size == 0`) *is* a zero write + (WRITE_ZEROES / discard-to-zero) — there is **no `all_zeros` flag**. A **read** fills a caller-provided + `dest` buffer in place (data → bytes, holes → zeros) and returns a sparse **layout**. +- **Holes are first-class.** A read returns `std::vector` marking which byte sub-ranges were + data vs holes; the same `hole` concept is what an empty-buffer write produces. A hole is **not** + `Missing`. + +```cpp +// The session + watermark fields the client stamps on EVERY CRAFT IO (write / read / keep_alive). +struct client_hdr { + uint64_t term{0}; // fences a stale writer (ETERM); every IO incl. keep_alive is checked + int64_t commit_lsn{-1}; // advance the frontier toward this, best-effort (-1 = don't). CRAFT's + // commit path: it piggybacks on the IO the client already sends -- + // there is NO standalone commit verb. + int64_t all_committed_lsn{-1}; // set-wide min commit_lsn; floors journal reclaim (-1 = unknown). +}; + +// One sub-range of a read's sparse layout, in BYTES. Carries no bytes (they live in the dest buffer). +struct io_extent { uint64_t addr; uint64_t len; bool hole; }; // hole => reads as zeros (unmapped) +``` --- @@ -21,11 +55,12 @@ struct CraftPartitionState { int64_t commit_lsn {-1}; // contiguous committed prefix (== Synced) int64_t last_append_lsn {-1}; // highest appended dLSN (may be uncommitted) uint64_t client_token {0}; // token from last successful InternalLogin - uint64_t term {0}; // current RAFT term + uint64_t term {0}; // current session term }; ``` -This state is authoritative in memory and recovered from the journal + superblock on restart. +Authoritative in memory; recovered from the journal + superblock on restart (the reference model does +not persist it). --- @@ -36,109 +71,77 @@ This state is authoritative in memory and recovered from the journal + superbloc ```cpp struct LoginResult { std::vector members; - int64_t dLSN; // starting (per-partition) LSN for new IO + int64_t dLSN; // starting (per-partition) LSN for new IO uint64_t term; + uint32_t lba_size; // block size in bytes: the client aligns addr/len to it and presents geometry to the FS }; -async_result -login(uint64_t client_token, volume_id_t vol_id); -``` - -Leader-only. Orchestrates the full login sequence: -1. `GetRSCommitLSN` broadcast to all peers (non-RAFT) -2. `FetchData` from an ahead peer if the leader is behind (non-RAFT) -3. Propose `SyncRSCommitLSN(rs_commit_lsn)` via RAFT -4. Propose `InternalLogin(client_token, new_term)` via RAFT -5. Return `LoginResult` after both RAFT entries commit - -**Preconditions:** caller is the RAFT leader. -**Postconditions:** all quorum members have `commit_lsn == rs_commit_lsn`; all reject IOs -from any token other than `client_token`. - ---- - -### `write` - -```cpp -async_status -write(uint64_t term, int64_t lsn, - lba_t lba, lba_count_t len, bool all_zeros, sisl::sg_list data); +async_result login(volume_handle const& vol, uint64_t client_token); ``` -Two forms, both journaled at slot `lsn` and both taking a `dLSN` that merges by -highest-per-LBA. A **data write** (`all_zeros=false`) lands `data` once and journals a -reference; zero-copy required on the hot path. A **zero write** (`all_zeros=true`, the -WRITE_ZEROES / discard-to-zero op) carries no `data`: it allocates nothing and journals a -metadata-only slot `{term, lsn, lba, len, all_zeros}`. - -Steps: -1. Reject if `term != state.term` → `ETERM`. -2. Data write: allocate blks, write `data` **once** via the data service (zero-copy), then - append the reference record `{term, lsn, lba, len, blkid}` to the journal at slot `lsn` (may - be out of order; homestore's `HS_DATA_LINKED` pattern). Zero write: no allocation and no - data-service write; append the metadata-only record. The client sets `all_zeros` after a - client-side zero scan; the server does **not** re-scan a data write. -3. `state.last_append_lsn = max(state.last_append_lsn, lsn)`. -4. ACK (on journal-append completion, which starts only after the data write completes). - -Does **not** apply data to the LBA index; that happens on `commit` (a zero write applies as an -index delete / range unmap). +Leader-only. Orchestrates the full login sequence (GetRSCommitLSN broadcast → optional FetchData → +`SyncRSCommitLSN` RAFT → `InternalLogin` RAFT) and returns once both RAFT entries commit. A follower +fails with `craft_error::NOT_LEADER`. Postconditions: all quorum members have +`commit_lsn == rs_commit_lsn`; all reject IO whose `term` != the new session term. --- -### `read` +### `async_write` ```cpp -async_result // sparse: data extents + holes -read(uint64_t term, int64_t readLSN, lba_t lba, lba_count_t len); +async_status async_write(volume_handle const& vol, client_hdr hdr, int64_t dlsn, + uint64_t addr, uint64_t len, sisl::sg_list data); ``` -`readLSN` is the read horizon `H`. Returns the latest version ≤ `H` for the range as a -**sparse** result: data extents interleaved with **holes** for sub-ranges with no data (never -written, zero-written, or an all-zero region collapsed by a read-time scan; the caller reads a -hole as zeros, and a hole is not `Missing`). Data extents come from the LBA index if applied, or -straight from the **journal-tail overlay** if only Appended (an **overlay read**; no index write -happens on the read path). **Entries above `H` are never served even if the replica holds them** -(the sub-quorum tail), the server-side half of read safety. The read never fetches from a peer; -the client routes only to a replica that holds every write ≤ `H` overlapping the range -(LBA-overlap eligibility, plus `Synced ≥ L`, the login dLSN). - -Rejects if `term != state.term`. +Append the client-assigned write at slot `dlsn`. `addr`/`len` are byte offset/length, block-aligned. +**Empty `data`** (`size == 0`) is a **zero write** (WRITE_ZEROES; `len` is the range to unmap; reads +back as a hole); non-empty `data` is a **data write** of exactly `len` bytes (single contiguous +buffer). Both take a `dLSN` and merge by highest-`dLSN`-per-LBA. Does **not** apply to the LBA index; +`hdr.commit_lsn` rides along and advances the contiguous commit frontier best-effort, in `dLSN` order +(this is CRAFT's commit — piggybacked on the writes the client is already broadcasting). Rejected with +`craft_error::STALE_TERM` if `hdr.term` != the session term, or `std::errc::invalid_argument` on +misalignment. ACK fires on journal-append completion. --- -### `commit` +### `async_read` ```cpp -async_result -commit(uint64_t term, int64_t lsn); +async_result> +async_read(volume_handle const& vol, client_hdr hdr, int64_t read_lsn, + uint64_t addr, uint64_t len, sisl::sg_list dest); ``` -Advance the contiguous commit watermark `commit_lsn` (≡ Synced) toward `lsn`, applying -present journal entries **strictly in dLSN order** (skipping Empty slots) and reclaiming -superseded blocks (a data entry applies as an index insert; a zero `all_zeros` entry as an -index delete / range unmap). Returns the **achieved** `{commit_lsn, last_append_lsn}`: -**best-effort**, -`commit_lsn` stalls just below the first Missing hole (resync fills it), which is not an -error and pauses only apply + reclaim, never reads. Readability is per-write and does not -wait for this watermark: appended entries above it are served from the journal-tail overlay, -so a higher LSN can be readable while a lower one is still a hole. +`read_lsn` is the read horizon `H`. Returns the latest version ≤ `H` for `[addr, addr+len)` by **filling +the caller-owned `dest` buffer in place** — data sub-ranges get their bytes, holes get zeros — and +returns the sparse **layout** (`io_extent[]`, ascending by `addr`) marking which byte sub-ranges were +data vs holes. Data comes from the LBA index if applied, or straight from the **journal-tail overlay** +if only Appended (an overlay read; no index write on the read path). **Entries above `H` are never +served even if the replica holds them** (the sub-quorum tail). The read never fetches from a peer; the +client routes only to a replica that holds every write ≤ `H` overlapping the range. `hdr.commit_lsn` +piggybacks a frontier advance. Rejects on term mismatch / misalignment. --- ### `keep_alive` ```cpp -async_result -keep_alive(int64_t commit_lsn, int64_t all_committed_lsn); +async_result keep_alive(volume_handle const& vol, client_hdr hdr); ``` -Same as `commit` plus resets the client-timeout watchdog. Returns the replica's -`{commit_lsn, last_append_lsn}` so the client can track the replica-set watermark (this feeds -the reconfig promotion gate, `commit_lsn ≥ startLSN`). `all_committed_lsn` is the set-wide -min the client computed from those responses; the replica may reclaim journal below -`min(all_committed_lsn, its checkpointed apply frontier)`. Sent periodically by the client -even during idle periods to prevent the server from triggering `SyncRSCommitLSN`. +The dedicated carrier for CRAFT's commit: advance the contiguous commit watermark toward +`hdr.commit_lsn` (applying present entries strictly in `dLSN` order, skipping Empty, reclaiming +superseded blocks) **and reset the client-timeout watchdog**. Returns the **achieved** +`{commit_lsn, last_append_lsn}` — best-effort: `commit_lsn` stalls just below the first Missing hole +(resync fills it; not an error; pauses only apply + reclaim, never reads). **Term-fenced** (`hdr.term`): +a stale client must not be able to reset the watchdog and keep the session alive. +`hdr.all_committed_lsn` lets the replica reclaim journal below +`min(all_committed_lsn, checkpointed apply frontier)`. Sent periodically (even when idle) and after +quorum-acked writes. + +> **There is no standalone `commit` verb.** The commit watermark advances via `hdr.commit_lsn` +> piggybacked on `async_write` / `async_read` / `keep_alive`; `keep_alive` is its dedicated carrier and +> also resets the watchdog. Readability is per-write and does not wait for the watermark (overlay reads). --- @@ -147,113 +150,73 @@ even during idle periods to prevent the server from triggering `SyncRSCommitLSN` ```cpp struct LSNPair { int64_t commit_lsn; int64_t last_append_lsn; }; -async_result -get_lsns(volume_id_t vol_id); +async_result get_lsns(volume_handle const& vol); ``` -Returns `{commit_lsn, last_append_lsn}` for the local partition. Used by peers via -`GetRSCommitLSN` during login and by the leader during `SyncRSCommitLSN`. +Returns `{commit_lsn, last_append_lsn}` for the local partition. Used by peers via `GetRSCommitLSN` +during login and by the leader during `SyncRSCommitLSN`. --- -### `truncate` +## Internal / peer & RAFT-entry API -```cpp -async_status -truncate(int64_t lsn); -``` +These are on the `craft_replica` backend, not the public client surface. -Drop all journal entries with dLSN > `lsn`. Called when a replica discovers it has -entries from a previous term that did not reach quorum (new `InternalLogin` forces -a truncate of stale appended entries). Also called during login to clean up followers -whose `last_append > agreed_dLSN`. - ---- - -## Internal / RAFT-entry API - -### `append` (propose SyncRSCommitLSN) +### `truncate` ```cpp -async_status -append(int64_t sync_to, uint64_t client_token); +async_status truncate(int64_t lsn); ``` -Proposes a `SyncRSCommitLSN` RAFT entry with value `sync_to`. Callable by the leader's -watchdog, the periodic checkpoint, or a client request (e.g. after a failed sub-quorum -write). The leader first resolves every unresolved slot ≤ `sync_to` (fetch from a holder, -or an `Empty` verdict on quorum-lacks evidence) and attaches the `empty_slots[]` verdict -list; it never proposes past an unresolved slot. `client_token` is embedded so followers -can verify the entry belongs to the current session. +Drop all journal entries with `dLSN > lsn` (stale prior-term tail; login truncation after +`InternalLogin` commits). ---- - -### `fetch_data` (for peer resync) +### `append` (propose SyncRSCommitLSN) ```cpp -async_result> -fetch_data(std::vector lsns); +async_status append(int64_t sync_to, uint64_t client_token); ``` -Returns raw journal data for the requested LSNs. Called server-to-server (not from the -client) during `SyncRSCommitLSN` apply when a replica discovers it is behind. Each `JournalSlot` -is one of: a **data** slot (`is_empty=false, all_zeros=false`), a **zero write** -(`all_zeros=true`, no data, applied as a range unmap), or **`Empty`** (`is_empty=true`); a -requested LSN absent from the result means not-present-here. - ---- +Propose a `SyncRSCommitLSN` RAFT entry. The leader first resolves every unresolved slot ≤ `sync_to` +(fetch from a holder, or an `Empty` verdict on quorum-lacks evidence) and attaches `empty_slots[]`; it +never proposes past an unresolved slot. -### `get_rs_commit_lsn` (for peer query) +### `fetch_data` (peer resync) ```cpp -async_result -get_rs_commit_lsn(); +async_result> fetch_data(std::vector lsns); ``` -Alias of `get_lsns` exposed to peer servers during the `GetRSCommitLSN` broadcast. +Server-to-server raw journal read during `SyncRSCommitLSN` apply. Each `JournalSlot` is one of: a data +slot, a zero write (`all_zeros=true`, no data), or `Empty` (`is_empty=true`); a requested LSN absent +from the result means not-present-here. ---- - -## RAFT state machine entries - -These are internal RAFT log entry types, not part of the public API. - -### `SyncRSCommitLSN` - -``` -payload: { rs_commit_lsn: int64, client_token: uint64, empty_slots: [int64] } -``` - -On RAFT apply (each replica): -1. Verify `client_token` against the current session (stale-session entries are no-ops). -2. Mark every slot in `empty_slots` as `Empty`, discarding any local data held there - (reconciliation). -3. If `last_append_lsn < rs_commit_lsn`: `fetch_data(missing non-Empty slots)` from peers - (the leader holds every non-Empty slot ≤ the watermark). -4. `commit_lsn = rs_commit_lsn`. +### `get_rs_commit_lsn` -### `InternalLogin` +Alias of `get_lsns` exposed to peers during the `GetRSCommitLSN` broadcast. -``` -payload: { client_token: uint64, term: uint64 } -``` +### RAFT state machine entries -On RAFT apply (each replica): -1. `state.client_token = client_token` -2. `state.term = term` -3. From this point, reject writes/reads whose `term` field != `state.term`. +`SyncRSCommitLSN { rs_commit_lsn, client_token, empty_slots[] }` and +`InternalLogin { client_token, term }` — see [rpcs.md](rpcs.md) and the wiki for apply semantics +(unchanged by the client-surface reshaping above). --- ## Replacing the existing API -`CraftReplDev` replaces the existing solo `ReplDev` for all volumes. The old -`async_read` / `async_write` surface in `home_blocks.hpp` (consumed by `ScstConnector`) -is superseded. `CraftConnector` is the new frontend; `ScstConnector` is removed. +The CRAFT free functions supersede the legacy byte block ops (now `[[deprecated]]`, kept for reference): -| Old API (removed) | CRAFT replacement | +| Old API (deprecated) | CRAFT replacement | |---|---| -| `async_write(vol, addr, sgs)` | `write(term, lsn, lba, len, data)` | -| `async_read(vol, addr, sgs)` | `read(term, readLSN, lba, len)` | -| `async_unmap` (stub) | `write(..., all_zeros=true)` (WRITE_ZEROES / discard-to-zero) | -| — | `login`, `commit`, `keep_alive`, `truncate`, `fetch_data`, `get_lsns`, `append` | +| `async_write(vol, addr, sgs)` | `async_write(vol, hdr, dlsn, addr, len, data)` | +| `async_read(vol, addr, sgs)` | `async_read(vol, hdr, read_lsn, addr, len, dest)` → `io_extent[]` layout | +| `async_unmap(vol, addr, len)` | `async_write(vol, hdr, dlsn, addr, len, /*empty*/ {})` (empty buffer = zero write) | +| — | `login`, `keep_alive` (commit carrier), `get_lsns` | + +The commit watermark rides on these calls (`client_hdr::commit_lsn`); there is no separate `commit` +call. Each handle is **one replica device**, never an aggregate / "whole volume" handle: the client +holds one handle per member and advances their commits independently (just like the protocol). A handle +comes from `create_volume` per server in production, or per-replica `create_memory_volume` for the +in-memory reference model (`make_memory_replica_set` is a test harness that loops it and wires the +in-process fabric). diff --git a/docs/craft/rpcs.md b/docs/craft/rpcs.md index 330d97f..1c3b6fb 100644 --- a/docs/craft/rpcs.md +++ b/docs/craft/rpcs.md @@ -1,12 +1,35 @@ # CRAFT RPCs -8 RPCs total. 4 client↔server, 4 server↔server (2 via RAFT, 2 non-RAFT). +8 RPCs total. 4 client↔server (Login, Write, Read, KeepAlive), 4 server↔server (2 via RAFT, 2 non-RAFT). RAFT internal RPCs (heartbeat, vote, membership) are not listed here. +> **Commit is not a separate RPC.** The commit watermark rides on `client_hdr.commit_lsn`, piggybacked +> on Write / Read / KeepAlive; KeepAlive is its dedicated carrier (commit + watchdog reset). Readability +> is per-write and does not wait for it (journal-tail overlay reads). + > **Canonical design:** the [**CRAFT Design**](https://github.com/eBay/HomeBlocks/wiki/CRAFT-Design) > wiki page is the source of truth for the protocol, and > [**CRAFT on HomeBlocks**](https://github.com/eBay/HomeBlocks/wiki/CRAFT-on-HomeBlocks) for the -> implementation binding; this file is the RPC wire-format reference. +> implementation binding; this file is the RPC wire-format reference. The C++ API is in [api.md](api.md). + +--- + +## Common header + +Every client IO (Write, Read, KeepAlive) carries a `client_hdr`: + +``` +client_hdr: { term: uint64, commit_lsn: int64, all_committed_lsn: int64 } +``` + +- `term` — fences a stale writer (rejected `ETERM`); checked on **every** IO including KeepAlive, so a + deposed client cannot even reset the liveness watchdog. +- `commit_lsn` — advance the contiguous commit frontier toward this, best-effort, in dLSN order + (`-1` = don't advance). This is CRAFT's commit, piggybacked. +- `all_committed_lsn` — set-wide min commit_lsn; floors journal reclaim (`-1` = unknown). + +**Addressing is byte-based:** `addr` / `len` are byte offset/length, block-aligned to the volume's +`lba_size` (from Login), else `EINVAL`. --- @@ -16,15 +39,14 @@ RAFT internal RPCs (heartbeat, vote, membership) are not listed here. ``` Request: { client_token: uint64 } -Response: { members: [endpoint], dLSN: int64, term: uint64 } +Response: { members: [endpoint], dLSN: int64, term: uint64, lba_size: uint32 } ``` -Client sends to the RAFT leader. Leader runs the full login orchestration sequence -(GetRSCommitLSN → optional FetchData → SyncRSCommitLSN RAFT → InternalLogin RAFT) -and responds once both RAFT entries commit. - -`client_token` is an opaque 64-bit identity token. `dLSN` is the per-partition LSN, the only -LSN CRAFT carries. +Client sends to the RAFT leader (a follower replies `NOT_LEADER` + leader endpoint). Leader runs the +full login orchestration (GetRSCommitLSN → optional FetchData → SyncRSCommitLSN RAFT → InternalLogin +RAFT) and responds once both RAFT entries commit. `dLSN` is the starting per-partition LSN. `lba_size` +is the volume block size in bytes — the client aligns every addr/len to it and presents the geometry to +the filesystem. HomeBlocks handler: `CraftReplDev::login()` @@ -33,20 +55,18 @@ HomeBlocks handler: `CraftReplDev::login()` ### 2. Write (Broadcast to all replicas) ``` -Request: { term: uint64, lsn: int64, lba: uint64, len: uint32, all_zeros: bool, data: bytes } +Request: { hdr: client_hdr, dlsn: int64, addr: uint64, len: uint64, data: bytes } Response: { status: Status } ``` -Client sends to every replica in the set in parallel. Each replica appends the entry to its -data journal at slot `lsn` and ACKs immediately. Write is durable once quorum ACKs, and +Client sends to every replica in the set in parallel at the client-assigned `dlsn`. Each replica +appends the entry to its data journal at slot `dlsn` and ACKs immediately. Durable once quorum ACKs, readable once Appended (served from the journal-tail overlay until applied). -Two forms. A **data write** (`all_zeros=false`) carries `data`; zero-copy is required (`data` -must not be copied during journal append). A **zero write** (`all_zeros=true`, the WRITE_ZEROES -/ discard-to-zero op) omits `data`: the slot is metadata-only, allocates no blocks, and on apply -unmaps its range. Both take a `dLSN` and merge by highest-`dLSN`-per-LBA. The client sets -`all_zeros` when a client-side scan finds an all-zero buffer; the server does **not** re-scan a -data write. +**Empty `data` (length 0) is a zero write** (WRITE_ZEROES / discard-to-zero): the slot is metadata-only, +allocates no blocks, and on apply unmaps its range (`len` is the range). Non-empty `data` is a data +write of exactly `len` bytes. There is **no `all_zeros` flag** — the empty buffer is the signal. Both +take a `dLSN` and merge by highest-`dLSN`-per-LBA. `hdr.commit_lsn` piggybacks a frontier advance. HomeBlocks handler: `CraftReplDev::write()` @@ -55,57 +75,40 @@ HomeBlocks handler: `CraftReplDev::write()` ### 3. Read (Unicast to chosen replica) ``` -Request: { term: uint64, readLSN: int64, lba: uint64, len: uint32 } -Response: { status: Status, extents: [{ lba: uint64, len: uint32, hole: bool, data: bytes }] } +Request: { hdr: client_hdr, read_lsn: int64, addr: uint64, len: uint64 } +Response: { status: Status, layout: [{ addr: uint64, len: uint64, hole: bool }], data: bytes } ``` -`readLSN` is the read horizon `H` (the client's contiguous quorum-acked prefix); the read -reflects writes ≤ `H`. The client picks an *eligible* replica: one filled to the login dLSN -`L` (`Synced ≥ L`) whose Missing set has no slot ≤ `H` overlapping `[lba, lba+len)`. The -replica returns the latest version ≤ `H` for the range, from the LBA index if applied or -straight from the journal-tail overlay if only appended (an **overlay read**; no index write -on the read path). A replica **ignores any write above `H` even if it holds it** (the -sub-quorum tail), which is the server-side half of read safety. Because the client only routes -to a servable -replica, **the read path never fetches from a peer**; large multi-LBA reads may be split -across replicas. +`read_lsn` is the read horizon `H`. The client picks an *eligible* replica (one filled to the login +dLSN `L`, with no Missing slot ≤ `H` overlapping the range). The replica returns the latest version ≤ +`H` for `[addr, addr+len)`, from the LBA index if applied or straight from the journal-tail overlay if +only Appended (an **overlay read**; no index write). It **ignores any write above `H` even if it holds +it** (the sub-quorum tail). The read path **never fetches from a peer**; large reads may be split across +replicas. -The response is **sparse**: data extents interleaved with **holes** (`hole=true`, no `data`). A -hole is an LBA sub-range that is never-written, zero-written (`all_zeros`), or an all-zero region -the replica collapsed by a read-time scan; the client reads it as zeros. A hole is **not** -`Missing` (the client routes around Missing slots, never around holes). +The response is **sparse**: `layout` marks which byte sub-ranges are **data** vs **holes** +(`hole=true`: never-written, zero-written, or an all-zero region collapsed by a read-time scan — read +as zeros, **not** `Missing`); `data` carries the bytes for the data extents. The client places them +into its caller-owned (iomgr) destination buffer per the layout (holes → zeros). `hdr.commit_lsn` +piggybacks a frontier advance. HomeBlocks handler: `CraftReplDev::read()` --- -### 4. Commit (Broadcast to all replicas) - -``` -Request: { term: uint64, lsn: int64 } -Response: { status: Status, commit_lsn: int64, last_append_lsn: int64 } -``` - -Tells replicas to advance `commit_lsn` to `lsn`. May be piggybacked on the next -Write or KeepAlive instead of sent as a standalone RPC. The response carries the **achieved** -`{commit_lsn, last_append_lsn}` (best-effort: `commit_lsn` stalls below the first Missing hole). - -HomeBlocks handler: `CraftReplDev::commit()` - ---- - -### 5. KeepAlive (Broadcast to all replicas) +### 4. KeepAlive (Broadcast to all replicas) ``` -Request: { commit_lsn: int64, all_committed_lsn: int64 } +Request: { hdr: client_hdr } Response: { status: Status, commit_lsn: int64, last_append_lsn: int64 } ``` -Advances `commit_lsn` (in-order apply) and resets the per-replica client-timeout watchdog. -The response carries `{commit_lsn, last_append_lsn}` (feeds the client's Missing map and the -reconfig promotion gate); the request carries `all_committed_lsn`, the set-wide min the -client computed from those responses, letting each replica reclaim journal opportunistically -below it. Sent periodically during idle periods and after every quorum-acknowledged write. +The dedicated commit carrier: advances `commit_lsn` (in-order apply, skipping Empty, reclaiming +superseded blocks) toward `hdr.commit_lsn` **and resets the per-replica client-timeout watchdog** (which +is why it is term-fenced — a stale client must not keep the session alive). The response carries the +**achieved** `{commit_lsn, last_append_lsn}` (best-effort: stalls below the first Missing hole), which +feeds the client's Missing map and the reconfig promotion gate. `hdr.all_committed_lsn` lets the replica +reclaim journal below it. Sent periodically during idle periods and after quorum-acknowledged writes. HomeBlocks handler: `CraftReplDev::keep_alive()` @@ -113,44 +116,36 @@ HomeBlocks handler: `CraftReplDev::keep_alive()` ## Server → Server (non-RAFT) -### 6. GetRSCommitLSN (Broadcast, initiated by leader) +### 5. GetRSCommitLSN (Broadcast, initiated by leader) ``` Request: { term: uint64, my_commit_lsn: int64, my_last_append_lsn: int64, is_login: bool } Response: { term: uint64, commit_lsn: int64, last_append_lsn: int64 } ``` -Leader sends to all peers to collect their current LSN state before a `SyncRSCommitLSN` -RAFT proposal. Used during login and on timeout. `is_login=true` (the login poll) makes the -peer **quiesce prior-session writes** before reporting `last_append` (the fencing barrier); -watchdog / periodic polls carry `is_login=false` and never quiesce (they ride the live tail). -Only `last_append_lsn` feeds the recovery watermark; `commit_lsn` is a contiguity -certificate that bounds the leader's hole-resolution window, seeds `all_committed_lsn` -(journal reclaim) when no client is attached, and carries the reconfig promotion gate. +Leader polls peers for their LSN state before a `SyncRSCommitLSN` proposal (login and on timeout). +`is_login=true` makes the peer **quiesce prior-session writes** before reporting `last_append` (the +fencing barrier); watchdog/periodic polls carry `is_login=false` and never quiesce. Only `last_append` +feeds the recovery watermark; `commit_lsn` is a contiguity certificate that bounds the leader's +hole-resolution window, seeds `all_committed_lsn`, and carries the reconfig promotion gate. HomeBlocks handler: `CraftReplDev::get_rs_commit_lsn()` / `get_lsns()` Dispatched by: `CraftConnector` (inter-node channel, non-RAFT) --- -### 7. FetchData (Unicast, from behind replica to an ahead peer) +### 6. FetchData (Unicast, from behind replica to an ahead peer) ``` Request: { lsns: [int64] } Response: { slots: [{ lsn: int64, is_empty: bool, all_zeros: bool, lba: uint64, len: uint32, data: bytes }] } ``` -Called when a replica discovers it is missing data for certain LSNs after receiving a -`SyncRSCommitLSN` RAFT entry. **Four-way per slot:** an entry with `is_empty=false, -all_zeros=false` carries `data`; `all_zeros=true` is a **zero write** (no `data`, applied as a -range unmap); `is_empty=true` means the peer has **positively** marked that slot `Empty` in a -prior resync; a requested `lsn` **omitted** from `slots` means *not-present-here*. A peer never -returns `is_empty=true` for a slot it merely lacks. The `Empty` **verdict itself is -leader-only**: the leader runs the broadcast-and-accumulate quorum procedure (itself -included; non-responders never count) before proposing `SyncRSCommitLSN`, and distributes -verdicts in that entry (`empty_slots[]`). Lagging replicas fetch present slots and obey the -verdict list; they never declare `Empty` unilaterally (see the wiki's Resync section for the -intersection argument and the Empty-beats-held-data reconciliation rule). +Called when a replica discovers it is missing data for certain LSNs after a `SyncRSCommitLSN` entry. +**Four-way per slot:** `is_empty=false, all_zeros=false` carries `data`; `all_zeros=true` is a zero write +(no data, applied as a range unmap); `is_empty=true` means the peer **positively** marked that slot +`Empty` (leader-only verdict); a requested `lsn` **omitted** means *not-present-here*. A peer never +returns `is_empty=true` for a slot it merely lacks. HomeBlocks handler: `CraftReplDev::fetch_data()` Dispatched by: `CraftConnector` (inter-node channel, non-RAFT) @@ -159,42 +154,40 @@ Dispatched by: `CraftConnector` (inter-node channel, non-RAFT) ## Server → Server (RAFT) -### 8. SyncRSCommitLSN (RAFT proposal, from leader) +### 7. SyncRSCommitLSN (RAFT proposal, from leader) ``` RAFT entry payload: { rs_commit_lsn: int64, client_token: uint64, empty_slots: [int64] } ``` -Proposed by the leader via `CraftReplDev::append()`. **Before proposing**, the leader -resolves every unresolved slot ≤ `rs_commit_lsn`: fetch it from any holder, or record an -`Empty` verdict on quorum-lacks evidence; it never proposes past an unresolved slot. On -RAFT commit each replica applies the entry: verify the token against the current session, -mark `empty_slots` as permanent no-op holes (discarding any local data in them), fetch the -remaining missing slots from peers (the leader is guaranteed to hold every non-Empty slot ≤ -the watermark), then advance `commit_lsn`. Replicas never declare `Empty` unilaterally. -This is the primary recovery mechanism — it carries no write data, only the watermark and -verdicts. +Proposed by the leader via `CraftReplDev::append()`. **Before proposing**, the leader resolves every +unresolved slot ≤ `rs_commit_lsn`: fetch from any holder, or record an `Empty` verdict on +quorum-lacks evidence; it never proposes past an unresolved slot. On RAFT commit each replica: verify +the token, mark `empty_slots` as permanent no-op holes (discarding any local data there), fetch the +remaining missing slots from peers, then advance `commit_lsn`. Replicas never declare `Empty` +unilaterally. This is the primary recovery mechanism — it carries no write data, only the watermark +and verdicts. --- -### 9. InternalLogin (RAFT proposal, from leader during login) +### 8. InternalLogin (RAFT proposal, from leader during login) ``` RAFT entry payload: { client_token: uint64, term: uint64 } ``` -Proposed by the leader after `SyncRSCommitLSN` commits. On apply each replica stores -`client_token` and `term`, rejecting any subsequent IO from a different token. Proposed -immediately after the `SyncRSCommitLSN` entry during the login sequence. +Proposed after `SyncRSCommitLSN` commits. On apply each replica stores `client_token` and `term`, +rejecting any subsequent IO from a different term. Proposed immediately after the `SyncRSCommitLSN` +entry during the login sequence. --- ## RPC Transport The transport layer for NubloxProto RPCs is decided by the **CRAFT-1 spike** (SDSTOR-22297 -dependency). `CraftConnector` is transport-agnostic: it will dispatch via whatever channel -CRAFT-1 selects (likely gRPC or a custom framing over TCP). Server-to-server RPCs (6 and 7) -use the same transport. +dependency). `CraftConnector` is transport-agnostic: it will dispatch via whatever channel CRAFT-1 +selects (likely gRPC or a custom framing over TCP). Server-to-server RPCs use the same transport. -During development, before CRAFT-1 lands, `CraftConnector` can use direct C++ function -calls or a stub transport for unit/integration testing. +During development, before CRAFT-1 lands, the calls are exercised in-process via direct function calls: +the [in-memory reference model](../../src/lib/craft/memory/) implements the same per-replica surface +(`craft_replica`) behind a `volume_handle`, with `MemTransport` standing in for the network. diff --git a/src/include/homeblks/craft_types.hpp b/src/include/homeblks/craft_types.hpp new file mode 100644 index 0000000..b8ddb53 --- /dev/null +++ b/src/include/homeblks/craft_types.hpp @@ -0,0 +1,117 @@ +/********************************************************************************* + * Modifications Copyright 2026 eBay Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed + * under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR + * CONDITIONS OF ANY KIND, either express or implied. See the License for the + * specific language governing permissions and limitations under the License. + * + *********************************************************************************/ +#pragma once + +// CRAFT client-facing data types -- the pure-data structs the public CRAFT API (home_blocks.hpp) takes +// and returns. Deliberately HomeStore-free and self-contained (only boost + std), so the public header +// and the in-memory reference model share one definition without pulling in the storage engine. +// Internal / peer-only types (CraftPartitionState, JournalSlot) live in lib/craft/craft_replica.hpp. +// The base types below are exactly what home_blocks.hpp's aliases resolve to (boost::uuids::uuid == +// peer_id_t, uint64_t == lba_t, uint32_t == lba_count_t). + +#include +#include +#include +#include + +#include // boost::uuids::uuid (== peer_id_t) +#include // ENUM + +namespace homeblocks { + +// Fundamental block-addressing aliases. Defined here (HomeStore-free) so both the public header and the +// CRAFT model share one definition; the byte-based CRAFT API uses raw uint64_t, but the rest of the impl +// (volume, io_req, the in-memory model's per-block index) speaks lba_t / lba_count_t. +using lba_t = uint64_t; +using lba_count_t = uint32_t; + +// Network address of a replica, as returned in login()'s member list. +struct replica_endpoint { + boost::uuids::uuid id; // == peer_id_t + std::string addr; // "host:port" +}; + +// {commit_lsn, last_append_lsn} snapshot -- returned by get_lsns() / keep_alive(). +struct LSNPair { + int64_t commit_lsn{-1}; + int64_t last_append_lsn{-1}; +}; + +// The session + watermark fields the client stamps on EVERY CRAFT IO (write / read / keep_alive). +// Addressing (dLSN / read horizon / LBA / len) stays op-specific; keep_alive carries only this header. +// - term: fences a stale writer (the protocol's ETERM). Every IO, incl. keep_alive, is +// rejected with craft_error::STALE_TERM if it != the replica's session term -- +// so a deposed client cannot even reset the liveness watchdog. +// - commit_lsn: advance the contiguous frontier toward this, best-effort, in dLSN order. +// -1 = do not advance. This is CRAFT's commit: it piggybacks on the IO the +// client is already sending; there is no standalone commit verb. +// - all_committed_lsn: the client-computed set-wide min commit_lsn; floors journal reclaim. -1 = unknown. +struct client_hdr { + uint64_t term{0}; + int64_t commit_lsn{-1}; + int64_t all_committed_lsn{-1}; +}; + +// Returned by login(): the replica set, the starting dLSN for new I/O, the session term, and the block +// size. `lba_size` is the volume's block size in bytes -- the client aligns every addr/len to it and +// uses it to present the volume geometry to the filesystem. +struct LoginResult { + std::vector< replica_endpoint > members; + int64_t dLSN{-1}; // starting (per-partition) LSN for new I/O + uint64_t term{0}; + uint32_t lba_size{0}; // block size in bytes (alignment unit for addr/len) +}; + +// One sub-range of a contiguous IO's sparse layout, in BYTES: a data extent (hole=false) or a hole +// (hole=true, reads as zeros -- unmapped / WRITE_ZEROES). Carries NO bytes -- the bytes live in the +// caller-owned (iomgr) sg_list buffer that write reads from and read fills. A read returns +// std::vector (ascending by addr) describing which byte sub-ranges of its dest buffer are +// data vs holes. A hole is NOT the same as Missing (a hole is a resolved "reads as zero"). +struct io_extent { + uint64_t addr{0}; // byte offset into the volume + uint64_t len{0}; // byte length + bool hole{false}; +}; + +// CRAFT-specific failures, registered as a std::error_condition enum (mirrors volume_error) so they +// ride result while staying branchable: if (r.error() == craft_error::STALE_TERM) { ... }. +// Anything with a standard equivalent is still returned via std::make_error_condition(std::errc::*). +ENUM(craft_error, uint16_t, + STALE_TERM = 1, // IO term != session term (the protocol's ETERM) + NOT_LEADER, // login / leader-only op sent to a follower + NO_QUORUM, // could not reach a quorum of live replicas + WRONG_TOKEN, // client_token is not the current owner + NOT_ELIGIBLE, // replica cannot serve this read (Missing overlap / below login-dLSN L) + REPLICA_DOWN); // addressed replica is down (fault injection / unreachable) + +class craft_error_category : public std::error_category { +public: + const char* name() const noexcept override { return "homeblocks.craft"; } + std::string message(int ev) const override { + return std::string{enum_name(static_cast< craft_error >(ev))}; + } +}; +inline std::error_category const& craft_error_category_inst() noexcept { + static craft_error_category inst; + return inst; +} +inline std::error_condition make_error_condition(craft_error e) noexcept { + return std::error_condition{static_cast< int >(e), craft_error_category_inst()}; +} + +} // namespace homeblocks + +template <> +struct std::is_error_condition_enum< homeblocks::craft_error > : std::true_type {}; diff --git a/src/include/homeblks/home_blocks.hpp b/src/include/homeblks/home_blocks.hpp index b972f82..3087f57 100644 --- a/src/include/homeblks/home_blocks.hpp +++ b/src/include/homeblks/home_blocks.hpp @@ -36,6 +36,8 @@ #include #include // result / async_result / status / ok +#include // client-facing CRAFT types (LoginResult, LSNPair, io_extent, client_hdr, craft_error); HomeStore-free + // Declares the homeblocks logging module so consumers can wire its log level into their own logging init. The // per-call LOG* shorthand macros are internal (lib/hb_internal.hpp) and intentionally not exposed here. SISL_LOGGING_DECL(homeblocks) @@ -143,9 +145,50 @@ inline std::error_condition make_error_condition(volume_error e) noexcept { // result is [[nodiscard]] -- dropping the task means the I/O never issues. read/write resolve to the byte count // transferred or a volume_error; the sg_list's iovecs point at caller-owned buffers that must outlive the await // (the descriptor itself is copied into the coroutine frame), and its total length is the transfer size. -[[nodiscard]] async_result< size_t > async_read(volume_handle const& vol, uint64_t addr, sisl::sg_list sgs); -[[nodiscard]] async_result< size_t > async_write(volume_handle const& vol, uint64_t addr, sisl::sg_list sgs); -[[nodiscard]] async_status async_unmap(volume_handle const& vol, uint64_t addr, uint64_t len); +[[nodiscard]] [[deprecated("legacy block op; use the CRAFT async_read/async_write overloads below (see docs/craft)")]] +async_result< size_t > async_read(volume_handle const& vol, uint64_t addr, sisl::sg_list sgs); +[[nodiscard]] [[deprecated("legacy block op; use the CRAFT async_read/async_write overloads below (see docs/craft)")]] +async_result< size_t > async_write(volume_handle const& vol, uint64_t addr, sisl::sg_list sgs); +[[nodiscard]] [[deprecated("legacy block op; use CRAFT async_write(..., all_zeros=true) (see docs/craft)")]] +async_status async_unmap(volume_handle const& vol, uint64_t addr, uint64_t len); + +// ---- CRAFT data plane: free functions over a volume_handle (one handle == one replica device) ---- +// +// The calls a CRAFT client issues against ONE replica. The client owns the CRAFT work: it assigns a +// per-partition dLSN to each write, broadcasts async_write to every member itself, tallies quorum, +// and routes async_read (carrying the horizon `read_lsn` = H) to one eligible member. The handle +// comes from create_volume (production, later) or create_memory_volume (the in-memory reference +// model in src/lib/craft/memory). Unlike the byte-based legacy API above, these are LBA-based and +// carry the session `term` and client-assigned `dLSN` / `read_lsn` that CRAFT defines. Errors ride +// craft_error (e.g. STALE_TERM, NOT_LEADER). See docs/craft/api.md and the CRAFT-Design wiki. + +// Leader-only. Establishes the session; returns the members, the starting dLSN for new I/O, and the +// term. Sent to the replica believed to be leader; a follower fails with craft_error::NOT_LEADER. +[[nodiscard]] async_result< LoginResult > login(volume_handle const& vol, uint64_t client_token); + +// Append one client-assigned write at slot `dlsn`. `addr`/`len` are BYTE offset/length and must be +// aligned to the volume's lba_size (from LoginResult), else std::errc::invalid_argument. `data` is a +// caller-owned (iomgr) buffer: EMPTY data (size 0) is a zero write (WRITE_ZEROES / unmap; reads back as +// a hole); non-empty data is a data write of exactly `len` bytes -- so the empty buffer, not a flag, +// signals a zero write. Not applied to the index directly; `hdr.commit_lsn` rides along and advances the +// frontier best-effort in dLSN order (CRAFT's piggybacked commit). STALE_TERM if hdr.term != session term. +[[nodiscard]] async_status async_write(volume_handle const& vol, client_hdr hdr, int64_t dlsn, uint64_t addr, + uint64_t len, sisl::sg_list data); + +// Read the latest version <= `read_lsn` (horizon H) for [addr, addr+len) (BYTE offset/length, aligned to +// lba_size). Fills the caller-owned `dest` buffer in place -- data sub-ranges get their bytes, holes get +// zeros -- and returns the sparse layout (which byte sub-ranges were data vs holes; the thin/hole info). +// Served from the index or the journal-tail overlay; never fetches from a peer. Advances the frontier to +// hdr.commit_lsn (piggybacked commit). std::errc::invalid_argument if addr/len are misaligned. +[[nodiscard]] async_result< std::vector< io_extent > > async_read(volume_handle const& vol, client_hdr hdr, + int64_t read_lsn, uint64_t addr, uint64_t len, + sisl::sg_list dest); + +// Advance the frontier toward hdr.commit_lsn (best-effort; stalls below the first hole) and reset the +// client-liveness watchdog (hence it is term-fenced). hdr.all_committed_lsn floors journal reclaim. +// Returns the achieved {commit_lsn, last_append_lsn}. The dedicated commit carrier; write/read piggyback +// it too, so there is no standalone commit() verb. +[[nodiscard]] async_result< LSNPair > keep_alive(volume_handle const& vol, client_hdr hdr); // ============================================= home_blocks ============================================= diff --git a/src/lib/CMakeLists.txt b/src/lib/CMakeLists.txt index fad3060..33b06fb 100644 --- a/src/lib/CMakeLists.txt +++ b/src/lib/CMakeLists.txt @@ -27,8 +27,6 @@ settings_gen_cpp( home_blks_config.fbs ) -#add_subdirectory(homestore_backend) -#add_subdirectory(memory_backend) add_subdirectory(craft) add_subdirectory(volume) add_subdirectory(tests) diff --git a/src/lib/craft/CMakeLists.txt b/src/lib/craft/CMakeLists.txt index 45ec4f5..68e7a86 100644 --- a/src/lib/craft/CMakeLists.txt +++ b/src/lib/craft/CMakeLists.txt @@ -3,7 +3,15 @@ cmake_minimum_required (VERSION 3.11) add_library(${PROJECT_NAME}_craft OBJECT) target_sources(${PROJECT_NAME}_craft PRIVATE craft_repl_dev.cpp + craft_api.cpp ) target_link_libraries(${PROJECT_NAME}_craft ${COMMON_DEPS} -) \ No newline at end of file +) + +# In-memory CRAFT reference model (HomeStore-free). Built by default; toggle off to exclude it and +# its tests from the build. +option(HB_BUILD_CRAFT_MEMORY_MODEL "Build the in-memory CRAFT reference model + tests" ON) +if (HB_BUILD_CRAFT_MEMORY_MODEL) + add_subdirectory(memory) +endif() diff --git a/src/lib/craft/craft_api.cpp b/src/lib/craft/craft_api.cpp new file mode 100644 index 0000000..7af4d21 --- /dev/null +++ b/src/lib/craft/craft_api.cpp @@ -0,0 +1,60 @@ +/********************************************************************************* + * Modifications Copyright 2026 eBay Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed + * under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR + * CONDITIONS OF ANY KIND, either express or implied. See the License for the + * specific language governing permissions and limitations under the License. + * + *********************************************************************************/ + +// Definitions of the public CRAFT data-plane free functions (declared in home_blocks.hpp). Each is a +// thin adapter: resolve the volume_handle to its per-replica craft_replica backend and forward 1:1. +// Dispatch is through the craft_replica INTERFACE, so this TU (part of libhomeblocks) does not depend +// on the in-memory model -- a production CraftReplDev-backed volume would work identically. + +#include + +#include "craft/craft_replica.hpp" // craft_replica interface +#include "volume/volume.hpp" // volume::craft_backend() + +namespace homeblocks { + +namespace { +craft_replica* craft_backend_of(volume_handle const& v) { return v ? v->craft_backend() : nullptr; } +// A volume with no CRAFT backend (e.g. a HomeStore-backed volume) cannot serve the CRAFT data plane. +auto no_craft_backend() { return std::unexpected(std::make_error_condition(std::errc::not_supported)); } +} // namespace + +async_result< LoginResult > login(volume_handle const& vol, uint64_t client_token) { + auto* b = craft_backend_of(vol); + if (!b) co_return no_craft_backend(); + co_return co_await b->login(client_token); +} + +async_status async_write(volume_handle const& vol, client_hdr hdr, int64_t dlsn, uint64_t addr, uint64_t len, + sisl::sg_list data) { + auto* b = craft_backend_of(vol); + if (!b) co_return no_craft_backend(); + co_return co_await b->write(hdr, dlsn, addr, len, std::move(data)); +} + +async_result< std::vector< io_extent > > async_read(volume_handle const& vol, client_hdr hdr, int64_t read_lsn, + uint64_t addr, uint64_t len, sisl::sg_list dest) { + auto* b = craft_backend_of(vol); + if (!b) co_return no_craft_backend(); + co_return co_await b->read(hdr, read_lsn, addr, len, std::move(dest)); +} + +async_result< LSNPair > keep_alive(volume_handle const& vol, client_hdr hdr) { + auto* b = craft_backend_of(vol); + if (!b) co_return no_craft_backend(); + co_return co_await b->keep_alive(hdr); +} + +} // namespace homeblocks diff --git a/src/lib/craft/craft_repl_dev.hpp b/src/lib/craft/craft_repl_dev.hpp index f90c9a5..0c5c462 100644 --- a/src/lib/craft/craft_repl_dev.hpp +++ b/src/lib/craft/craft_repl_dev.hpp @@ -15,6 +15,7 @@ #pragma once #include "../hb_internal.hpp" +#include "craft_replica.hpp" // internal CRAFT types (CraftPartitionState, JournalSlot) + the craft_replica interface #include #include @@ -25,44 +26,10 @@ namespace homeblocks { // ─── wire-protocol types ────────────────────────────────────────────────────── - -// Network address of a replica returned by login(). -struct replica_endpoint { - peer_id_t id; - std::string addr; // "host:port" -}; - -// Per-partition in-memory state. Authoritative in memory; recovered from -// journal + superblock on restart. -struct CraftPartitionState { - int64_t commit_lsn {-1}; // highest committed dLSN (applied to LBA index) - int64_t last_append_lsn {-1}; // highest appended dLSN (may be uncommitted) - uint64_t client_token {0}; // token from the last successful InternalLogin - uint64_t term {0}; // current RAFT term -}; - -// Returned by get_lsns() / get_rs_commit_lsn(). -struct LSNPair { - int64_t commit_lsn; - int64_t last_append_lsn; -}; - -// Returned by login(). -struct LoginResult { - std::vector< replica_endpoint > members; - int64_t dLSN; // starting LSN for new I/O - uint64_t term; -}; - -// One journal slot, returned by fetch_data(). is_empty == true means the LSN -// never reached any replica and the data fields are invalid. -struct JournalSlot { - int64_t lsn; - bool is_empty{false}; - lba_t lba{0}; - lba_count_t len{0}; - sisl::sg_list data; -}; +// +// replica_endpoint, CraftPartitionState, LSNPair, LoginResult, and JournalSlot are defined once in +// (HomeStore-free), pulled in via hb_internal.hpp, and shared by both +// this production class and the in-memory reference model. // ─── journal backend abstraction ───────────────────────────────────────────── // diff --git a/src/lib/craft/craft_replica.hpp b/src/lib/craft/craft_replica.hpp new file mode 100644 index 0000000..c74f1fd --- /dev/null +++ b/src/lib/craft/craft_replica.hpp @@ -0,0 +1,97 @@ +/********************************************************************************* + * Modifications Copyright 2026 eBay Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed + * under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR + * CONDITIONS OF ANY KIND, either express or implied. See the License for the + * specific language governing permissions and limitations under the License. + * + *********************************************************************************/ +#pragma once + +// The per-replica CRAFT server surface -- one replica device of a partition. This is the internal +// contract behind a CRAFT-mode volume_handle: `volume` delegates its CRAFT data-plane calls to a +// `craft_replica`, and the public CRAFT free functions in home_blocks.hpp resolve a volume_handle to +// its backend and call these methods 1:1. The in-memory reference model (MemCraftReplica) implements +// this now; a production HomeStore-backed CraftReplDev can implement/adapt it later, independently. +// +// HomeStore-free: only home_blocks.hpp (which pulls the header-only homestore/error.hpp aliases) and +// std/sisl. Do NOT include the storage engine here. + +#include +#include + +#include // async_result/async_status + client-facing CRAFT types (LoginResult, io_extent, ...) + +namespace homeblocks { + +// ── internal / peer-only data types (deliberately NOT on the public client surface) ── + +// Per-partition CRAFT state (internal to a replica implementation). Authoritative in memory; the +// production impl recovers it from the journal + superblock on restart (the reference model does not). +struct CraftPartitionState { + int64_t commit_lsn {-1}; // contiguous committed prefix (== Synced) + int64_t last_append_lsn {-1}; // highest appended dLSN (may be uncommitted) + uint64_t client_token {0}; // token from the last successful InternalLogin + uint64_t term {0}; // current session term +}; + +// One journal slot returned by fetch_data() (server-to-server resync; not client-facing). Four-way: +// data (is_empty=false, all_zeros=false), zero write (all_zeros=true, no data), Empty (is_empty=true), +// or omitted from the response (not-present-here). +struct JournalSlot { + int64_t lsn{-1}; + bool is_empty{false}; + bool all_zeros{false}; + lba_t lba{0}; + lba_count_t len{0}; + sisl::sg_list data{}; +}; + +class craft_replica { +public: + virtual ~craft_replica() = default; + + // ── client-facing (what a CRAFT client issues against this one replica) ── + + // Leader-only: run the login orchestration and return the members, starting dLSN, and term. + // A non-leader replica returns craft_error::NOT_LEADER. + virtual async_result< LoginResult > login(uint64_t client_token) = 0; + + // Append one client-assigned write at slot `dlsn`. `addr`/`len` are BYTE offset/length, aligned to + // the volume's lba_size (else std::errc::invalid_argument). `data` is a caller-owned (iomgr) buffer: + // empty (size 0) => a zero write (WRITE_ZEROES / unmap; `len` is the range); non-empty => a data + // write of exactly `len` bytes. Does NOT apply to the index; the frontier is advanced by + // hdr.commit_lsn (piggybacked commit). craft_error::STALE_TERM if hdr.term != the session term. + virtual async_status write(client_hdr hdr, int64_t dlsn, uint64_t addr, uint64_t len, sisl::sg_list data) = 0; + + // Latest version <= read_lsn (horizon H) for [addr, addr+len) (BYTE offset/length, aligned). Fills the + // caller-owned `dest` buffer in place (data sub-ranges get bytes, holes get zeros) and returns the + // sparse layout (data vs holes). Advances the frontier to hdr.commit_lsn. STALE_TERM on term mismatch. + virtual async_result< std::vector< io_extent > > read(client_hdr hdr, int64_t read_lsn, uint64_t addr, + uint64_t len, sisl::sg_list dest) = 0; + + // Advance the frontier toward hdr.commit_lsn + reset the client-liveness watchdog -- which is WHY + // it is term-fenced: a stale client must not be able to keep the session alive. Returns the + // achieved {commit_lsn, last_append_lsn}. No standalone commit verb; keep_alive is its carrier. + virtual async_result< LSNPair > keep_alive(client_hdr hdr) = 0; + + // Snapshot {commit_lsn, last_append_lsn}. + virtual async_result< LSNPair > get_lsns() = 0; + + // ── peer-facing (server-to-server; driven by the cold path / resync, never by a client) ── + + virtual async_result< LSNPair > get_rs_commit_lsn() = 0; + virtual async_result< std::vector< JournalSlot > > fetch_data(std::vector< int64_t > lsns) = 0; + virtual async_status truncate(int64_t lsn) = 0; + + // This replica's endpoint id (for routing / membership). + virtual peer_id_t id() const = 0; +}; + +} // namespace homeblocks diff --git a/src/lib/craft/memory/CMakeLists.txt b/src/lib/craft/memory/CMakeLists.txt new file mode 100644 index 0000000..1b9aac4 --- /dev/null +++ b/src/lib/craft/memory/CMakeLists.txt @@ -0,0 +1,15 @@ +cmake_minimum_required (VERSION 3.11) + +# The HomeStore-FREE CRAFT reference model (MemCraftReplica + MemTransport). It is compiled with +# COMMON_DEPS' include paths (the header-only homestore/error.hpp + iomgr_types.hpp, plus sisl / fmt / +# boost) but references NO homestore symbol -- COMMON_DEPS is PRIVATE so nothing is propagated, and the +# model unit test links WITHOUT homestore to enforce that. This OBJECT lib is intentionally NOT +# aggregated into libhomeblocks; only the tests (and any reference driver) pull its objects. +add_library(${PROJECT_NAME}_craft_mem OBJECT) +target_sources(${PROJECT_NAME}_craft_mem PRIVATE + mem_craft_replica.cpp + mem_craft_cluster.cpp +) +target_link_libraries(${PROJECT_NAME}_craft_mem PRIVATE ${COMMON_DEPS}) + +add_subdirectory(tests) diff --git a/src/lib/craft/memory/mem_craft_cluster.cpp b/src/lib/craft/memory/mem_craft_cluster.cpp new file mode 100644 index 0000000..1171e63 --- /dev/null +++ b/src/lib/craft/memory/mem_craft_cluster.cpp @@ -0,0 +1,142 @@ +/********************************************************************************* + * Modifications Copyright 2026 eBay Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed + * under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR + * CONDITIONS OF ANY KIND, either express or implied. See the License for the + * specific language governing permissions and limitations under the License. + * + *********************************************************************************/ + +#include "mem_craft_cluster.hpp" + +#include +#include + +namespace homeblocks::craft { + +namespace { +auto fail(craft_error e) { return std::unexpected(make_error_condition(e)); } +} // namespace + +MemTransport::MemTransport(std::vector< replica_endpoint > members, uint32_t lba_size) : + members_{std::move(members)}, lba_size_{lba_size} { + if (!members_.empty()) leader_ = members_.front().id; +} + +void MemTransport::register_replica(MemCraftReplica* r) { + std::lock_guard< std::mutex > g{mu_}; + by_id_[r->id()] = r; +} + +peer_id_t MemTransport::leader() const { + std::lock_guard< std::mutex > g{mu_}; + return leader_; +} +void MemTransport::set_leader(peer_id_t id) { + std::lock_guard< std::mutex > g{mu_}; + leader_ = id; +} + +bool MemTransport::is_up(peer_id_t id) const { + std::lock_guard< std::mutex > g{mu_}; + return !down_.contains(id); +} +bool MemTransport::write_allowed(peer_id_t id) const { + std::lock_guard< std::mutex > g{mu_}; + if (down_.contains(id)) return false; + if (write_keep_ && !write_keep_->contains(id)) return false; + return true; +} + +void MemTransport::set_up(peer_id_t id, bool up) { + std::lock_guard< std::mutex > g{mu_}; + if (up) { + down_.erase(id); + } else { + down_.insert(id); + } +} +void MemTransport::force_subquorum(std::vector< peer_id_t > keep) { + std::lock_guard< std::mutex > g{mu_}; + write_keep_ = std::unordered_set< peer_id_t, uuid_hash >(keep.begin(), keep.end()); +} +void MemTransport::clear_faults() { + std::lock_guard< std::mutex > g{mu_}; + down_.clear(); + write_keep_.reset(); +} + +std::vector< MemCraftReplica* > MemTransport::live_replicas_locked() const { + std::vector< MemCraftReplica* > live; + for (auto const& m : members_) { + if (down_.contains(m.id)) continue; + if (auto it = by_id_.find(m.id); it != by_id_.end()) live.push_back(it->second); + } + return live; +} + +result< LoginResult > MemTransport::run_login(MemCraftReplica* caller, uint64_t client_token) { + std::vector< MemCraftReplica* > live; + std::vector< replica_endpoint > members_copy; + uint64_t nt{0}; + { + std::lock_guard< std::mutex > g{mu_}; + if (down_.contains(caller->id())) return fail(craft_error::REPLICA_DOWN); + if (caller->id() != leader_) return fail(craft_error::NOT_LEADER); + live = live_replicas_locked(); + if (live.size() < quorum(members_.size())) return fail(craft_error::NO_QUORUM); + nt = ++term_; + members_copy = members_; + } + // Drive the cold path WITHOUT holding mu_ (each replica call takes only its own lock). + // Phase 1: GetRSCommitLSN -> rs_commit_lsn = max(quorum.last_append_lsn). + int64_t rs = -1; + for (auto* r : live) rs = std::max(rs, r->peek_lsns().last_append_lsn); + // Phase 2: SyncRSCommitLSN(rs) applied to all live members. + for (auto* r : live) r->cold_apply_sync(rs, client_token); + // Phase 3: InternalLogin(token, term+1) applied to all live members. + for (auto* r : live) r->cold_apply_login(client_token, nt); + // Phase 4: truncate any stale tail above rs. + for (auto* r : live) { + if (r->peek_lsns().last_append_lsn > rs) r->cold_truncate_above(rs); + } + return LoginResult{std::move(members_copy), rs, nt, lba_size_}; +} + +// ── group factory (model level; no volumes attached) ── + +peer_id_t mem_replica_id(volume_id_t vol_id, uint32_t index) { + peer_id_t id = vol_id; // copy the 16 bytes, then perturb the tail deterministically with `index` + auto* p = reinterpret_cast< uint8_t* >(&id); + p[12] ^= static_cast< uint8_t >(index & 0xff); + p[13] ^= static_cast< uint8_t >((index >> 8) & 0xff); + p[14] ^= static_cast< uint8_t >((index >> 16) & 0xff); + p[15] ^= static_cast< uint8_t >((index >> 24) & 0xff); + return id; +} + +MemReplicaGroup make_mem_replica_group(volume_id_t vol_id, uint32_t n, uint32_t page_size) { + std::vector< replica_endpoint > members; + members.reserve(n); + for (uint32_t i = 0; i < n; ++i) { + members.push_back(replica_endpoint{mem_replica_id(vol_id, i), "mem://replica-" + std::to_string(i)}); + } + auto net = std::make_shared< MemTransport >(members, page_size); + MemReplicaGroup group; + group.net = net; + group.replicas.reserve(n); + for (uint32_t i = 0; i < n; ++i) { + auto r = std::make_shared< MemCraftReplica >(members[i], page_size, net); + net->register_replica(r.get()); + group.replicas.push_back(std::move(r)); + } + return group; +} + +} // namespace homeblocks::craft diff --git a/src/lib/craft/memory/mem_craft_cluster.hpp b/src/lib/craft/memory/mem_craft_cluster.hpp new file mode 100644 index 0000000..8e642ef --- /dev/null +++ b/src/lib/craft/memory/mem_craft_cluster.hpp @@ -0,0 +1,92 @@ +/********************************************************************************* + * Modifications Copyright 2026 eBay Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed + * under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR + * CONDITIONS OF ANY KIND, either express or implied. See the License for the + * specific language governing permissions and limitations under the License. + * + *********************************************************************************/ +#pragma once + +// MemTransport -- the in-process stand-in for the CRAFT network AND the (RAFT) cold path, for the +// in-memory reference model. It routes replica<->replica cold-path calls, holds the membership and a +// designated leader, and is the fault-injection surface for tests (set_up / force_subquorum). It does +// NOT own the replicas' lifetime: they are co-owned by their volume_handles (production shape) or by +// the MemReplicaGroup below (model unit tests); MemTransport keeps raw pointers registered at creation. +// +// Membership, replica count, leader, and fault injection are control-plane / orchestration concerns +// that sit OUTSIDE the CRAFT protocol -- here they are just wired up directly (a CLI flag / harness). + +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "mem_craft_replica.hpp" + +namespace homeblocks::craft { + +using uuid_hash = boost::hash< peer_id_t >; + +class MemTransport { +public: + MemTransport(std::vector< replica_endpoint > members, uint32_t lba_size); + + // Called by create_memory_volume / make_mem_replica_group as each replica is built. + void register_replica(MemCraftReplica* r); + + // ── cold path: leader-only login orchestration (GetRSCommitLSN -> SyncRSCommitLSN -> InternalLogin) ── + // `caller` is the replica the client invoked login() on. Returns craft_error::NOT_LEADER if it is + // not the leader, or NO_QUORUM if a live majority is not reachable. + result< LoginResult > run_login(MemCraftReplica* caller, uint64_t client_token); + + // ── membership / routing (consulted by the replicas' client-facing ops) ── + peer_id_t leader() const; + bool is_up(peer_id_t id) const; + bool write_allowed(peer_id_t id) const; // false => this write is dropped (sub-quorum) + std::size_t member_count() const { return members_.size(); } + + // ── fault injection (tests / CLI) ── + void set_up(peer_id_t id, bool up); // bring a replica down / back up + void force_subquorum(std::vector< peer_id_t > keep); // subsequent writes land ONLY on `keep` + void clear_faults(); + void set_leader(peer_id_t id); + +private: + std::vector< MemCraftReplica* > live_replicas_locked() const; // requires mu_ held + static std::size_t quorum(std::size_t n) { return n / 2 + 1; } + + std::vector< replica_endpoint > members_; + std::unordered_map< peer_id_t, MemCraftReplica*, uuid_hash > by_id_; + peer_id_t leader_{}; + uint64_t term_{0}; + uint32_t lba_size_{0}; // volume block size (bytes) + std::unordered_set< peer_id_t, uuid_hash > down_; + std::optional< std::unordered_set< peer_id_t, uuid_hash > > write_keep_; + mutable std::mutex mu_; +}; + +// An N-replica in-process group with NO volumes attached -- the model layer used directly by unit +// tests (and the core of make_memory_replica_set, which additionally wraps each replica in a +// volume_handle). HomeStore-free. +struct MemReplicaGroup { + std::shared_ptr< MemTransport > net; + std::vector< std::shared_ptr< MemCraftReplica > > replicas; // index 0 is the default leader +}; +MemReplicaGroup make_mem_replica_group(volume_id_t vol_id, uint32_t n = 3, uint32_t page_size = 4096); + +// Deterministic per-replica id derived from the volume id + index (so tests are reproducible). +peer_id_t mem_replica_id(volume_id_t vol_id, uint32_t index); + +} // namespace homeblocks::craft diff --git a/src/lib/craft/memory/mem_craft_replica.cpp b/src/lib/craft/memory/mem_craft_replica.cpp new file mode 100644 index 0000000..d6d5f69 --- /dev/null +++ b/src/lib/craft/memory/mem_craft_replica.cpp @@ -0,0 +1,266 @@ +/********************************************************************************* + * Modifications Copyright 2026 eBay Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed + * under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR + * CONDITIONS OF ANY KIND, either express or implied. See the License for the + * specific language governing permissions and limitations under the License. + * + *********************************************************************************/ + +#include "mem_craft_replica.hpp" +#include "mem_craft_cluster.hpp" // the full MemTransport type + +#include +#include + +namespace homeblocks::craft { + +namespace { +// Copy a scatter/gather list into an owned, contiguous byte buffer (deliberately not zero-copy). +std::shared_ptr< std::vector< uint8_t > > own_bytes(sisl::sg_list const& s) { + auto b = std::make_shared< std::vector< uint8_t > >(); + b->reserve(s.size); + for (auto const& io : s.iovs) { + auto const* p = static_cast< uint8_t const* >(io.iov_base); + b->insert(b->end(), p, p + io.iov_len); + } + return b; +} +bool all_zero(uint8_t const* p, std::size_t n) { + for (std::size_t i = 0; i < n; ++i) { + if (p[i] != 0) return false; + } + return true; +} +auto fail(craft_error e) { return std::unexpected(make_error_condition(e)); } +} // namespace + +MemCraftReplica::MemCraftReplica(replica_endpoint ep, uint32_t page_size, std::shared_ptr< MemTransport > net) : + ep_{std::move(ep)}, page_size_{page_size}, net_{std::move(net)} {} + +// ── async wrappers: all real work is synchronous, so these complete inline (no reactor) ── + +async_result< LoginResult > MemCraftReplica::login(uint64_t client_token) { + if (!net_) co_return fail(craft_error::NO_QUORUM); + if (!net_->is_up(ep_.id)) co_return fail(craft_error::REPLICA_DOWN); + co_return net_->run_login(this, client_token); +} +async_status MemCraftReplica::write(client_hdr hdr, int64_t dlsn, uint64_t addr, uint64_t len, + sisl::sg_list data) { + co_return do_write(hdr, dlsn, addr, len, std::move(data)); +} +async_result< std::vector< io_extent > > MemCraftReplica::read(client_hdr hdr, int64_t read_lsn, uint64_t addr, + uint64_t len, sisl::sg_list dest) { + co_return do_read(hdr, read_lsn, addr, len, std::move(dest)); +} +async_result< LSNPair > MemCraftReplica::keep_alive(client_hdr hdr) { co_return do_keep_alive(hdr); } +async_result< LSNPair > MemCraftReplica::get_lsns() { co_return do_lsns(); } +async_result< LSNPair > MemCraftReplica::get_rs_commit_lsn() { co_return do_lsns(); } +async_result< std::vector< JournalSlot > > MemCraftReplica::fetch_data(std::vector< int64_t > lsns) { + co_return do_fetch(lsns); +} +async_status MemCraftReplica::truncate(int64_t lsn) { co_return do_truncate(lsn); } + +// ── synchronous cores ── + +status MemCraftReplica::do_write(client_hdr hdr, int64_t dlsn, uint64_t addr, uint64_t len, + sisl::sg_list data) { + if (net_ && !net_->is_up(ep_.id)) return fail(craft_error::REPLICA_DOWN); + if (net_ && !net_->write_allowed(ep_.id)) return fail(craft_error::REPLICA_DOWN); // sub-quorum drop + // byte-based API: addr/len must be block-aligned (the model works in page_size blocks internally). + if (addr % page_size_ != 0 || len % page_size_ != 0 || len == 0) { + return std::unexpected(std::make_error_condition(std::errc::invalid_argument)); + } + std::lock_guard< std::mutex > g{mu_}; + if (hdr.term != state_.term) return fail(craft_error::STALE_TERM); + + MemJournalSlot slot; + slot.term = hdr.term; + slot.lba = addr / page_size_; // byte offset -> block index + slot.len = static_cast< lba_count_t >(len / page_size_); // byte length -> block count + slot.all_zeros = (data.size == 0); // empty sg_list => zero write (WRITE_ZEROES); no all_zeros flag + if (!slot.all_zeros) { + // a data write is a single contiguous buffer of exactly `len` bytes (redundant with `len`, which + // is the authoritative range for a zero write, so it stays on the signature). + if (data.iovs.size() != 1 || data.size != len) { + return std::unexpected(std::make_error_condition(std::errc::invalid_argument)); + } + slot.bytes = own_bytes(data); + } + journal_[dlsn] = std::move(slot); + state_.last_append_lsn = std::max(state_.last_append_lsn, dlsn); + apply_up_to(hdr.commit_lsn); // piggybacked commit: advance the frontier best-effort, in dLSN order + return ok(); +} + +result< std::vector< io_extent > > MemCraftReplica::do_read(client_hdr hdr, int64_t read_lsn, uint64_t addr, + uint64_t len, sisl::sg_list dest) { + if (net_ && !net_->is_up(ep_.id)) return fail(craft_error::REPLICA_DOWN); + // byte-based API: addr/len block-aligned; dest is a single contiguous buffer covering [addr,addr+len) + if (addr % page_size_ != 0 || len % page_size_ != 0 || len == 0) { + return std::unexpected(std::make_error_condition(std::errc::invalid_argument)); + } + if (dest.iovs.size() != 1 || dest.size < len) { + return std::unexpected(std::make_error_condition(std::errc::invalid_argument)); + } + std::lock_guard< std::mutex > g{mu_}; + if (hdr.term != state_.term) return fail(craft_error::STALE_TERM); + apply_up_to(hdr.commit_lsn); // piggybacked commit: advance the frontier opportunistically + return read_range(read_lsn, addr, len, dest); +} + +result< LSNPair > MemCraftReplica::do_keep_alive(client_hdr hdr) { + if (net_ && !net_->is_up(ep_.id)) return fail(craft_error::REPLICA_DOWN); + std::lock_guard< std::mutex > g{mu_}; + // Term-fenced: a stale client must NOT reset the liveness watchdog (that would block failover). + if (hdr.term != state_.term) return fail(craft_error::STALE_TERM); + apply_up_to(hdr.commit_lsn); + // watchdog reset + journal reclaim below min(hdr.all_committed_lsn, commit_lsn) are deferred seams. + return LSNPair{state_.commit_lsn, state_.last_append_lsn}; +} + +result< LSNPair > MemCraftReplica::do_lsns() { + if (net_ && !net_->is_up(ep_.id)) return fail(craft_error::REPLICA_DOWN); + std::lock_guard< std::mutex > g{mu_}; + return LSNPair{state_.commit_lsn, state_.last_append_lsn}; +} + +status MemCraftReplica::do_truncate(int64_t lsn) { + std::lock_guard< std::mutex > g{mu_}; + journal_.erase(journal_.upper_bound(lsn), journal_.end()); + state_.last_append_lsn = std::min(state_.last_append_lsn, lsn); + return ok(); +} + +result< std::vector< JournalSlot > > MemCraftReplica::do_fetch(std::vector< int64_t > const& lsns) { + std::lock_guard< std::mutex > g{mu_}; + std::vector< JournalSlot > out; + for (auto lsn : lsns) { + auto it = journal_.find(lsn); + if (it == journal_.end()) continue; // not-present-here => omit + auto const& s = it->second; + JournalSlot js; + js.lsn = lsn; + js.is_empty = s.is_empty; + js.all_zeros = s.all_zeros; + js.lba = s.lba; + js.len = s.len; + if (!s.all_zeros && !s.is_empty && s.bytes) { + // seam: the sg_list points into the slot's owned buffer (valid while the slot lives). + js.data.size = s.bytes->size(); + js.data.iovs.push_back(iovec{s.bytes->data(), s.bytes->size()}); + } + out.push_back(std::move(js)); + } + return out; +} + +// ── apply / read helpers (mu_ held) ── + +void MemCraftReplica::apply_slot(int64_t dlsn, MemJournalSlot const& s) { + if (s.all_zeros) { + for (lba_count_t i = 0; i < s.len; ++i) { + index_.erase(s.lba + i); // unmap => hole + } + } else { + for (lba_count_t i = 0; i < s.len; ++i) { + index_[s.lba + i] = IndexCell{dlsn, s.bytes, static_cast< std::size_t >(i) * page_size_}; + } + } +} + +void MemCraftReplica::apply_up_to(int64_t target) { + int64_t next = state_.commit_lsn + 1; + while (next <= target) { + auto it = journal_.find(next); + if (it == journal_.end()) break; // Missing hole -> stall (best-effort) + if (!it->second.is_empty) apply_slot(next, it->second); + state_.commit_lsn = next; // Empty slots are skipped on apply but still advance the frontier + ++next; + } +} + +// Highest-dLSN journal-tail slot with commit_lsn < dLSN <= H that covers `x` (the journal-tail overlay, +// materialized on demand). Slots above H are never examined -- that is the horizon clamp. +MemCraftReplica::MemJournalSlot const* MemCraftReplica::highest_slot_le(lba_t x, int64_t H) const { + for (auto it = journal_.upper_bound(H); it != journal_.begin();) { + --it; + if (it->first <= state_.commit_lsn) break; // reached the applied prefix (served from index_) + auto const& s = it->second; + if (s.is_empty) continue; + if (s.lba <= x && x < s.lba + s.len) return &s; + } + return nullptr; +} + +std::vector< io_extent > MemCraftReplica::read_range(int64_t H, uint64_t addr, uint64_t len, + sisl::sg_list const& dest) { + auto* out = static_cast< uint8_t* >(dest.iovs[0].iov_base); // caller buffer, >= len bytes + lba_t const lba0 = addr / page_size_; // byte offset -> first block + lba_count_t const nblk = static_cast< lba_count_t >(len / page_size_); + std::vector< io_extent > layout; + for (lba_count_t i = 0; i < nblk; ++i) { + lba_t const x = lba0 + i; + // Winner = highest-dLSN version <= H covering block x, between the applied index cell and the + // journal tail. A tail slot's dLSN is always > commit_lsn >= any applied cell's, so the tail wins. + uint8_t const* page = nullptr; + bool hole = true; + if (auto* s = highest_slot_le(x, H)) { + if (!s->all_zeros) { + page = s->bytes->data() + static_cast< std::size_t >(x - s->lba) * page_size_; + hole = false; + } // else: zero write => hole + } else if (auto it = index_.find(x); it != index_.end()) { + page = it->second.buf->data() + it->second.off; + hole = false; + } + // read-time scan: an all-zero data page reads back thin (as a hole). + if (!hole && all_zero(page, page_size_)) hole = true; + + // fill the caller's buffer in place: a data block gets its bytes, a hole gets zeros. + uint8_t* dst = out + static_cast< std::size_t >(i) * page_size_; + if (hole) { + std::memset(dst, 0, page_size_); + } else { + std::memcpy(dst, page, page_size_); + } + // coalesce the returned layout, in BYTES, with the previous extent if contiguous. + uint64_t const x_addr = static_cast< uint64_t >(x) * page_size_; + if (!layout.empty() && layout.back().hole == hole && layout.back().addr + layout.back().len == x_addr) { + layout.back().len += page_size_; + } else { + layout.push_back(io_extent{x_addr, page_size_, hole}); + } + } + return layout; +} + +// ── cold-path hooks (driven by MemTransport, which does NOT hold its own lock while calling these) ── + +LSNPair MemCraftReplica::peek_lsns() { + std::lock_guard< std::mutex > g{mu_}; + return LSNPair{state_.commit_lsn, state_.last_append_lsn}; +} +void MemCraftReplica::cold_apply_sync(int64_t rs_commit_lsn, uint64_t /*client_token*/) { + std::lock_guard< std::mutex > g{mu_}; + // SyncRSCommitLSN: (seam) fetch any missing non-Empty slots <= rs from peers, then advance commit. + apply_up_to(rs_commit_lsn); +} +void MemCraftReplica::cold_apply_login(uint64_t client_token, uint64_t term) { + std::lock_guard< std::mutex > g{mu_}; + state_.client_token = client_token; + state_.term = term; +} +void MemCraftReplica::cold_truncate_above(int64_t rs_commit_lsn) { + std::lock_guard< std::mutex > g{mu_}; + journal_.erase(journal_.upper_bound(rs_commit_lsn), journal_.end()); + state_.last_append_lsn = std::min(state_.last_append_lsn, rs_commit_lsn); +} + +} // namespace homeblocks::craft diff --git a/src/lib/craft/memory/mem_craft_replica.hpp b/src/lib/craft/memory/mem_craft_replica.hpp new file mode 100644 index 0000000..51adad6 --- /dev/null +++ b/src/lib/craft/memory/mem_craft_replica.hpp @@ -0,0 +1,105 @@ +/********************************************************************************* + * Modifications Copyright 2026 eBay Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed + * under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR + * CONDITIONS OF ANY KIND, either express or implied. See the License for the + * specific language governing permissions and limitations under the License. + * + *********************************************************************************/ +#pragma once + +// In-memory, HomeStore-free implementation of ONE CRAFT replica device -- the reference model a +// client is built against. It owns an in-memory data journal (dLSN -> slot), an applied LBA index, +// and derives the journal-tail overlay on demand from the unapplied journal tail. All real work is +// synchronous under a per-replica mutex; the async_result/async_status methods are 1-line coroutine +// wrappers that complete inline (no iomgr reactor). NOT crash-resilient and NOT performant by design. +// +// Peer-to-peer concerns (leader election, login orchestration, resync, fault injection) live in +// MemTransport (mem_craft_cluster.hpp), the in-process stand-in for the network. + +#include +#include +#include +#include +#include + +#include "craft/craft_replica.hpp" // craft_replica interface + CRAFT data types + +namespace homeblocks::craft { + +class MemTransport; // in-process network + cold path + +class MemCraftReplica final : public craft_replica { +public: + MemCraftReplica(replica_endpoint ep, uint32_t page_size, std::shared_ptr< MemTransport > net); + + // ── craft_replica: client-facing ── + async_result< LoginResult > login(uint64_t client_token) override; + async_status write(client_hdr hdr, int64_t dlsn, uint64_t addr, uint64_t len, + sisl::sg_list data) override; + async_result< std::vector< io_extent > > read(client_hdr hdr, int64_t read_lsn, uint64_t addr, uint64_t len, + sisl::sg_list dest) override; + async_result< LSNPair > keep_alive(client_hdr hdr) override; + async_result< LSNPair > get_lsns() override; + + // ── craft_replica: peer-facing (driven by MemTransport) ── + async_result< LSNPair > get_rs_commit_lsn() override; + async_result< std::vector< JournalSlot > > fetch_data(std::vector< int64_t > lsns) override; + async_status truncate(int64_t lsn) override; + peer_id_t id() const override { return ep_.id; } + +private: + friend class MemTransport; // the cold path drives the cold_* / peek helpers below directly + + struct MemJournalSlot { + uint64_t term{0}; + lba_t lba{0}; + lba_count_t len{0}; + bool all_zeros{false}; + bool is_empty{false}; // Empty verdict (resync seam) + std::shared_ptr< std::vector< uint8_t > > bytes; // len*page_size bytes; null iff all_zeros/empty + }; + struct IndexCell { + int64_t dlsn{-1}; + std::shared_ptr< std::vector< uint8_t > > buf; // one page at buf->data()+off + std::size_t off{0}; + }; + + // synchronous cores (each takes mu_); the public methods co_return these inline + status do_write(client_hdr hdr, int64_t dlsn, uint64_t addr, uint64_t len, sisl::sg_list); + result< std::vector< io_extent > > do_read(client_hdr hdr, int64_t read_lsn, uint64_t addr, uint64_t len, + sisl::sg_list dest); + result< LSNPair > do_keep_alive(client_hdr hdr); + result< LSNPair > do_lsns(); + status do_truncate(int64_t lsn); + result< std::vector< JournalSlot > > do_fetch(std::vector< int64_t > const& lsns); + + // helpers (mu_ held by caller) + void apply_up_to(int64_t target); + void apply_slot(int64_t dlsn, MemJournalSlot const& s); + // Fill `dest` for byte range [addr,addr+len) (data -> bytes, holes -> zeros) and return the layout. + std::vector< io_extent > read_range(int64_t H, uint64_t addr, uint64_t len, sisl::sg_list const& dest); + MemJournalSlot const* highest_slot_le(lba_t x, int64_t H) const; // journal-tail slot, honoring the horizon clamp + + // cold-path hooks used by MemTransport (each takes mu_) + LSNPair peek_lsns(); + void cold_apply_sync(int64_t rs_commit_lsn, uint64_t client_token); + void cold_apply_login(uint64_t client_token, uint64_t term); + void cold_truncate_above(int64_t rs_commit_lsn); + + replica_endpoint ep_; + uint32_t page_size_; + std::shared_ptr< MemTransport > net_; + CraftPartitionState state_; + std::map< int64_t, MemJournalSlot > journal_; // dLSN -> slot (out-of-order arrival tolerated) + std::map< lba_t, IndexCell > index_; // applied prefix (<= commit_lsn); an absent LBA is a hole + mutable std::mutex mu_; +}; + +} // namespace homeblocks::craft diff --git a/src/lib/craft/memory/mem_craft_volume.cpp b/src/lib/craft/memory/mem_craft_volume.cpp new file mode 100644 index 0000000..e51bdaf --- /dev/null +++ b/src/lib/craft/memory/mem_craft_volume.cpp @@ -0,0 +1,41 @@ +/********************************************************************************* + * Modifications Copyright 2026 eBay Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed + * under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR + * CONDITIONS OF ANY KIND, either express or implied. See the License for the + * specific language governing permissions and limitations under the License. + * + *********************************************************************************/ + +#include "mem_craft_volume.hpp" + +#include "volume/volume.hpp" // full volume type + mem_craft_tag ctor + +namespace homeblocks::craft { + +volume_handle create_memory_volume(volume_info info, std::shared_ptr< MemCraftReplica > replica) { + // volume's memory-mode ctor: HomeStore-free, stores `replica` as the CRAFT backend. + return std::make_shared< volume >(std::move(info), std::move(replica), volume::mem_craft_tag{}); +} + +MemReplicaHandles make_memory_replica_set(volume_info info, uint32_t n) { + auto group = make_mem_replica_group(info.id, n, static_cast< uint32_t >(info.page_size)); + + MemReplicaHandles out; + out.net = group.net; + out.handles.reserve(n); + for (uint32_t i = 0; i < n; ++i) { + // One volume_handle per replica device -- the same shape create_volume yields per server. + volume_info vi{info.id, info.size_bytes, info.page_size, info.name}; + out.handles.push_back(create_memory_volume(std::move(vi), group.replicas[i])); + } + return out; +} + +} // namespace homeblocks::craft diff --git a/src/lib/craft/memory/mem_craft_volume.hpp b/src/lib/craft/memory/mem_craft_volume.hpp new file mode 100644 index 0000000..d4b94a6 --- /dev/null +++ b/src/lib/craft/memory/mem_craft_volume.hpp @@ -0,0 +1,51 @@ +/********************************************************************************* + * Modifications Copyright 2026 eBay Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed + * under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR + * CONDITIONS OF ANY KIND, either express or implied. See the License for the + * specific language governing permissions and limitations under the License. + * + *********************************************************************************/ +#pragma once + +// Bridge between the HomeStore-free CRAFT reference model (MemCraftReplica) and the public +// volume_handle surface. `create_memory_volume` wraps ONE in-process replica behind a volume_handle +// so it is drivable via the public CRAFT free functions (login/async_write/async_read/...), exactly +// like a production CRAFT volume_handle -- the API works whether the replicas are three servers or +// three objects in one process. `make_memory_replica_set` stands up the whole in-process set. +// +// This is a NON-public header (src/lib): it pulls in volume.hpp, so a TU using it links HomeStore. +// The model itself (mem_craft_replica/cluster) stays HomeStore-free and is testable on its own. + +#include +#include +#include + +#include // volume_handle, volume_info + +#include "mem_craft_cluster.hpp" // MemTransport, MemCraftReplica, make_mem_replica_group + +namespace homeblocks::craft { + +// Wrap one MemCraftReplica behind an opaque volume_handle (via volume's memory-mode ctor). +volume_handle create_memory_volume(volume_info info, std::shared_ptr< MemCraftReplica > replica); + +// Stand up an N-replica in-process set and return the N INDEPENDENT per-replica handles a client +// drives (index 0 is the default leader), plus the shared transport (fault injection only). There is +// deliberately NO aggregate / "whole volume" handle: each element is one replica device -- exactly what +// create_volume yields on each server in production -- and the client advances commits across them +// independently. This helper is just a harness loop over create_memory_volume (orchestration, outside +// the CRAFT protocol); `info` supplies the volume id / page_size / size. +struct MemReplicaHandles { + std::vector< volume_handle > handles; // one per replica device; the client drives each separately + std::shared_ptr< MemTransport > net; // in-process network + cold path + faults (no production analog) +}; +MemReplicaHandles make_memory_replica_set(volume_info info, uint32_t n = 3); + +} // namespace homeblocks::craft diff --git a/src/lib/craft/memory/tests/CMakeLists.txt b/src/lib/craft/memory/tests/CMakeLists.txt new file mode 100644 index 0000000..950fb13 --- /dev/null +++ b/src/lib/craft/memory/tests/CMakeLists.txt @@ -0,0 +1,39 @@ +cmake_minimum_required (VERSION 3.11) + +# Model unit test. It links the HomeStore-free model objects + sisl + gtest, deliberately WITHOUT +# homestore::homestore: a successful link is the proof that the model references no homestore symbol +# (an undefined-symbol error here would mean the model touched the storage engine). +add_executable(test_craft_memory) +target_sources(test_craft_memory PRIVATE + test_craft_memory.cpp + craft_test_main.cpp + $ +) +target_link_libraries(test_craft_memory + sisl::sisl + GTest::gmock + -rdynamic +) +# home_blocks.hpp (pulled in transitively) needs homestore/iomgr HEADERS only (the header-only +# error.hpp / iomgr_types.hpp). Supply their include dirs WITHOUT linking the engine. +target_include_directories(test_craft_memory SYSTEM PRIVATE + $ + $ +) +add_test(NAME CraftMemoryModelTest COMMAND test_craft_memory) + +# End-to-end test: drives the model through the PUBLIC volume_handle API (login/async_write/ +# async_read/commit in libhomeblocks) via create_memory_volume. This one DOES link HomeStore (through +# libhomeblocks + volume), as expected for the integration surface. +add_executable(test_craft_e2e) +target_sources(test_craft_e2e PRIVATE + test_craft_e2e.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/../mem_craft_volume.cpp # volume<->model bridge (needs volume.hpp) + $ +) +target_link_libraries(test_craft_e2e + ${PROJECT_NAME} + ${COMMON_TEST_DEPS} + -rdynamic +) +add_test(NAME CraftMemoryE2ETest COMMAND test_craft_e2e) diff --git a/src/lib/craft/memory/tests/craft_test_main.cpp b/src/lib/craft/memory/tests/craft_test_main.cpp new file mode 100644 index 0000000..bfca7a0 --- /dev/null +++ b/src/lib/craft/memory/tests/craft_test_main.cpp @@ -0,0 +1,38 @@ +/********************************************************************************* + * Modifications Copyright 2026 eBay Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed + * under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR + * CONDITIONS OF ANY KIND, either express or implied. See the License for the + * specific language governing permissions and limitations under the License. + * + *********************************************************************************/ + +// Minimal, HomeStore-free gtest main for the CRAFT memory-model tests (sisl logging/options only -- +// deliberately does NOT include hb_internal.hpp / homestore, so the binary links no storage engine). + +#include + +#include +#include +#include + +// home_blocks.hpp declares the `homeblocks` log module (SISL_LOGGING_DECL); since this HomeStore-free +// test does not link the homeblocks lib that normally defines it, define it here. +SISL_LOGGING_DEF(homeblocks) +SISL_LOGGING_INIT(homeblocks) +SISL_OPTIONS_ENABLE(logging) + +int main(int argc, char* argv[]) { + int parsed_argc = argc; + ::testing::InitGoogleTest(&parsed_argc, argv); + SISL_OPTIONS_LOAD(parsed_argc, argv, logging); + sisl::logging::SetLogger(std::string(argv[0])); + sisl::logging::SetLogPattern("[%D %T%z] [%^%L%$] [%t] %v"); + return RUN_ALL_TESTS(); +} diff --git a/src/lib/craft/memory/tests/test_craft_e2e.cpp b/src/lib/craft/memory/tests/test_craft_e2e.cpp new file mode 100644 index 0000000..03243ae --- /dev/null +++ b/src/lib/craft/memory/tests/test_craft_e2e.cpp @@ -0,0 +1,143 @@ +/********************************************************************************* + * Modifications Copyright 2026 eBay Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed + * under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR + * CONDITIONS OF ANY KIND, either express or implied. See the License for the + * specific language governing permissions and limitations under the License. + * + *********************************************************************************/ + +// End-to-end test of the in-memory CRAFT reference model through the PUBLIC volume_handle API +// (home_blocks.hpp: login / async_write / async_read / keep_alive) -- exactly the surface a real CRAFT +// client is built against. The handles come from make_memory_replica_set (create_memory_volume), +// the in-process stand-in for "create the volume on each of three servers". + +#include +#include +#include + +#include +#include +#include +#include + +#include + +#include "coro_helpers.hpp" +#include "craft/memory/mem_craft_volume.hpp" + +// libhomeblocks defines the `homeblocks` log module in homeblks_impl.o, but this test never odr-uses +// that object (it drives create_memory_volume, not init_homeblocks), so the static archive does not +// pull it in. Define the module here so logging init finds it. +SISL_LOGGING_DEF(homeblocks) +SISL_LOGGING_INIT(homeblocks) +SISL_OPTIONS_ENABLE(logging) + +using namespace homeblocks; + +namespace { +constexpr uint32_t PAGE = 512; +constexpr uint64_t TOKEN = 0xABCDEFULL; + +constexpr uint64_t blk(uint64_t n) { return n * uint64_t{PAGE}; } // block index/count -> byte offset/length + +// The per-IO client header: session term + commit watermark to piggyback (-1 = don't advance). +client_hdr chdr(uint64_t term, int64_t commit = -1) { return client_hdr{term, commit, -1}; } + +template < class T > +auto rg(T&& t) { + return detail::sync_get(std::forward< T >(t)); +} +sisl::sg_list one_iov(std::vector< uint8_t >& b) { + sisl::sg_list s; + s.size = b.size(); + s.iovs.push_back(iovec{b.data(), b.size()}); + return s; +} +std::vector< uint8_t > page_of(uint8_t f) { return std::vector< uint8_t >(PAGE, f); } + +volume_info mk_info() { + volume_info vi; + vi.id = boost::uuids::random_generator()(); + vi.size_bytes = 1ULL << 20; + vi.page_size = PAGE; + vi.name = "craft-e2e"; + vi.repl_mode = replication_mode::CRAFT; + return vi; +} +} // namespace + +// Full path: stand up the in-process replica set, then drive login -> write (broadcast) -> keep_alive +// (commit) -> read entirely through the public free functions over volume_handle. +TEST(CraftMemE2E, LoginWriteCommitReadViaPublicApi) { + auto set = craft::make_memory_replica_set(mk_info(), 3); + ASSERT_EQ(set.handles.size(), 3u); + + auto lr = rg(login(set.handles[0], TOKEN)); // leader is replica 0 + ASSERT_TRUE(lr.has_value()); + EXPECT_EQ(lr->members.size(), 3u); + EXPECT_EQ(lr->term, 1u); + EXPECT_EQ(lr->dLSN, -1); + EXPECT_EQ(lr->lba_size, PAGE); + uint64_t const term = lr->term; + + // The client assigns dLSN 0, broadcasts async_write to every replica handle, and counts acks. + auto buf = page_of(0x5A); + int acks = 0; + for (auto& h : set.handles) { + if (rg(async_write(h, chdr(term), /*dlsn*/ 0, /*addr*/ blk(9), /*len*/ blk(1), one_iov(buf))).has_value()) + ++acks; + } + EXPECT_EQ(acks, 3); // quorum (all three) acked + + for (auto& h : set.handles) { // keep_alive carries the commit watermark (no standalone commit verb) + ASSERT_TRUE(rg(keep_alive(h, chdr(term, /*commit_lsn*/ 0))).has_value()); + } + std::vector< uint8_t > dest(PAGE, 0xEE); + auto rr = rg(async_read(set.handles[2], chdr(term), /*H*/ 0, blk(9), blk(1), one_iov(dest))); + ASSERT_TRUE(rr.has_value()); + ASSERT_EQ(rr->size(), 1u); + EXPECT_FALSE((*rr)[0].hole); + EXPECT_EQ(dest, buf); +} + +// login to a follower handle is rejected NOT_LEADER through the public API. +TEST(CraftMemE2E, LoginOnFollowerRejected) { + auto set = craft::make_memory_replica_set(mk_info(), 3); + auto lr = rg(login(set.handles[1], TOKEN)); + ASSERT_FALSE(lr.has_value()); + EXPECT_EQ(lr.error(), make_error_condition(craft_error::NOT_LEADER)); +} + +// a zero write (empty buffer) reads back as a hole through the public API. +TEST(CraftMemE2E, ZeroWriteHoleViaPublicApi) { + auto set = craft::make_memory_replica_set(mk_info(), 3); + auto lr = rg(login(set.handles[0], TOKEN)); + ASSERT_TRUE(lr.has_value()); + uint64_t const term = lr->term; + auto& h = set.handles[0]; + + ASSERT_TRUE(rg(async_write(h, chdr(term), 0, blk(3), blk(1), /*empty => zero write*/ sisl::sg_list{})).has_value()); + ASSERT_TRUE(rg(keep_alive(h, chdr(term, 0))).has_value()); + std::vector< uint8_t > dest(PAGE, 0xEE); + auto rr = rg(async_read(h, chdr(term), 0, blk(3), blk(1), one_iov(dest))); + ASSERT_TRUE(rr.has_value()); + ASSERT_EQ(rr->size(), 1u); + EXPECT_TRUE((*rr)[0].hole); + EXPECT_EQ(dest, page_of(0)); +} + +int main(int argc, char* argv[]) { + int parsed_argc = argc; + ::testing::InitGoogleTest(&parsed_argc, argv); + SISL_OPTIONS_LOAD(parsed_argc, argv, logging); + sisl::logging::SetLogger(std::string(argv[0])); + sisl::logging::SetLogPattern("[%D %T%z] [%^%L%$] [%t] %v"); + return RUN_ALL_TESTS(); +} diff --git a/src/lib/craft/memory/tests/test_craft_memory.cpp b/src/lib/craft/memory/tests/test_craft_memory.cpp new file mode 100644 index 0000000..e496870 --- /dev/null +++ b/src/lib/craft/memory/tests/test_craft_memory.cpp @@ -0,0 +1,347 @@ +/********************************************************************************* + * Modifications Copyright 2026 eBay Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software distributed + * under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR + * CONDITIONS OF ANY KIND, either express or implied. See the License for the + * specific language governing permissions and limitations under the License. + * + *********************************************************************************/ + +// Unit tests for the in-memory CRAFT reference model, driving MemCraftReplica / MemTransport +// directly (no volume, no HomeStore, no iomgr reactor -- ops complete inline via detail::sync_get). +// +// The IO surface is BYTE-based: addr/len are byte offsets/lengths, block-aligned to the volume's +// lba_size (returned by login). sisl::sg_list is the one buffer type both ways (caller-owned): a WRITE +// takes a source buffer -- EMPTY (size 0) means a zero write (WRITE_ZEROES), non-empty is a data write. +// A READ fills a caller `dest` buffer (data -> bytes, holes -> zeros) and returns a std::vector +// byte layout marking which sub-ranges were data vs holes. Every IO carries a client_hdr {term, +// commit_lsn, all_committed_lsn}; commit_lsn (-1 = don't advance) piggybacks the frontier -- there is no +// standalone commit verb. keep_alive is the dedicated commit carrier and is term-fenced. + +#include +#include +#include +#include + +#include +#include + +#include "coro_helpers.hpp" +#include "craft/memory/mem_craft_cluster.hpp" + +using namespace homeblocks; +using namespace homeblocks::craft; + +namespace { +constexpr uint32_t PAGE = 512; +constexpr uint64_t TOKEN = 0xC0FFEEULL; +constexpr int64_t NO_COMMIT = -1; // commit_lsn sentinel: don't advance the frontier + +constexpr uint64_t blk(uint64_t n) { return n * uint64_t{PAGE}; } // block index/count -> byte offset/length + +volume_id_t new_vol() { return boost::uuids::random_generator()(); } + +// The per-IO client header: session term + the commit watermark to piggyback (-1 = don't advance). +client_hdr chdr(uint64_t term, int64_t commit = NO_COMMIT) { return client_hdr{term, commit, /*all_committed*/ -1}; } + +// A one-iovec sg_list over `buf` (which must outlive the call). +sisl::sg_list one_iov(std::vector< uint8_t >& buf) { + sisl::sg_list s; + s.size = buf.size(); + s.iovs.push_back(iovec{buf.data(), buf.size()}); + return s; +} +std::vector< uint8_t > page_of(uint8_t fill, uint32_t n = PAGE) { return std::vector< uint8_t >(n, fill); } + +template < class Task > +auto rg(Task&& t) { + return detail::sync_get(std::forward< Task >(t)); +} + +// Read `nblk` blocks starting at block `lba`, at horizon H (optionally piggybacking commit), into a +// fresh dest buffer pre-filled with 0xEE (so a hole read proves the model actually zeroed it). +struct read_out { + bool ok{false}; + std::error_condition err{}; + std::vector< io_extent > layout; // which byte sub-ranges were data vs holes + std::vector< uint8_t > data; // the filled dest buffer +}; +read_out rd(MemCraftReplica& r, uint64_t term, int64_t H, uint64_t lba, uint64_t nblk, int64_t commit = NO_COMMIT) { + std::vector< uint8_t > dest(nblk * PAGE, 0xEE); + auto rr = rg(r.read(chdr(term, commit), H, blk(lba), blk(nblk), one_iov(dest))); + if (!rr.has_value()) return {false, rr.error(), {}, {}}; + return {true, {}, std::move(*rr), std::move(dest)}; +} + +// Log the leader in; assert success and return {term, dLSN}. +std::pair< uint64_t, int64_t > login_ok(MemReplicaGroup& g) { + auto lr = rg(g.replicas[0]->login(TOKEN)); + EXPECT_TRUE(lr.has_value()); + return {lr->term, lr->dLSN}; +} +} // namespace + +// 1. login establishes a session across the set and reports the block size. +TEST(CraftMemModel, LoginEstablishesSession) { + auto g = make_mem_replica_group(new_vol(), 3, PAGE); + auto lr = rg(g.replicas[0]->login(TOKEN)); + ASSERT_TRUE(lr.has_value()); + EXPECT_EQ(lr->members.size(), 3u); + EXPECT_EQ(lr->term, 1u); + EXPECT_EQ(lr->dLSN, -1); + EXPECT_EQ(lr->lba_size, PAGE); // the client aligns addr/len to this +} + +// login sent to a follower is rejected with the leader hint. +TEST(CraftMemModel, LoginOnFollowerIsNotLeader) { + auto g = make_mem_replica_group(new_vol(), 3, PAGE); + auto lr = rg(g.replicas[1]->login(TOKEN)); + ASSERT_FALSE(lr.has_value()); + EXPECT_EQ(lr.error(), make_error_condition(craft_error::NOT_LEADER)); +} + +// 2. broadcast a write, advance the frontier via keep_alive, read it back. +TEST(CraftMemModel, WriteKeepAliveRead) { + auto g = make_mem_replica_group(new_vol(), 3, PAGE); + auto [term, dl] = login_ok(g); + ASSERT_EQ(dl, -1); + auto buf = page_of(0xAB); + for (auto& r : g.replicas) { + ASSERT_TRUE(rg(r->write(chdr(term), 0, blk(5), blk(1), one_iov(buf))).has_value()); + } + for (auto& r : g.replicas) { // keep_alive is the dedicated commit carrier; returns the achieved pair + auto c = rg(r->keep_alive(chdr(term, /*commit_lsn*/ 0))); + ASSERT_TRUE(c.has_value()); + EXPECT_EQ(c->commit_lsn, 0); + EXPECT_EQ(c->last_append_lsn, 0); + } + auto out = rd(*g.replicas[1], term, /*H*/ 0, 5, 1); + ASSERT_TRUE(out.ok); + ASSERT_EQ(out.layout.size(), 1u); + EXPECT_FALSE(out.layout[0].hole); + EXPECT_EQ(out.layout[0].addr, blk(5)); + EXPECT_EQ(out.layout[0].len, blk(1)); + EXPECT_EQ(out.data, buf); +} + +// 3. an appended-but-uncommitted entry is served from the journal-tail overlay, above a committed hole. +TEST(CraftMemModel, OverlayReadAboveCommittedHole) { + auto g = make_mem_replica_group(new_vol(), 3, PAGE); + auto [term, dl] = login_ok(g); + auto& r = *g.replicas[0]; + auto b0 = page_of(1), b1 = page_of(2); + ASSERT_TRUE(rg(r.write(chdr(term), 0, blk(5), blk(1), one_iov(b0))).has_value()); + ASSERT_TRUE(rg(r.write(chdr(term), 1, blk(6), blk(1), one_iov(b1))).has_value()); + ASSERT_TRUE(rg(r.keep_alive(chdr(term, 0))).has_value()); // apply dLSN 0 only; dLSN 1 stays in overlay + + auto out = rd(r, term, /*H*/ 1, 5, 2); // block 5 applied, block 6 from overlay + ASSERT_TRUE(out.ok); + std::vector< uint8_t > want; + want.insert(want.end(), b0.begin(), b0.end()); + want.insert(want.end(), b1.begin(), b1.end()); + ASSERT_EQ(out.layout.size(), 1u); // both data + contiguous -> one coalesced extent + EXPECT_FALSE(out.layout[0].hole); + EXPECT_EQ(out.layout[0].len, blk(2)); + EXPECT_EQ(out.data, want); + + auto below = rd(r, term, /*H*/ 0, 6, 1); // dLSN1 > H -> hole + ASSERT_TRUE(below.ok); + ASSERT_EQ(below.layout.size(), 1u); + EXPECT_TRUE(below.layout[0].hole); + EXPECT_EQ(below.data, page_of(0)); // dest was zeroed +} + +// 4. the horizon clamp: a locally-held write above H is never returned. +TEST(CraftMemModel, HorizonClampHidesHeldWrite) { + auto g = make_mem_replica_group(new_vol(), 3, PAGE); + auto [term, dl] = login_ok(g); + auto& r = *g.replicas[0]; + auto b5 = page_of(9); + ASSERT_TRUE(rg(r.write(chdr(term), 5, blk(10), blk(1), one_iov(b5))).has_value()); // sub-quorum tail + + auto clamped = rd(r, term, /*H*/ 4, 10, 1); + ASSERT_TRUE(clamped.ok); + ASSERT_EQ(clamped.layout.size(), 1u); + EXPECT_TRUE(clamped.layout[0].hole); // dLSN 5 > H=4 + EXPECT_EQ(clamped.data, page_of(0)); + + auto seen = rd(r, term, /*H*/ 5, 10, 1); + ASSERT_TRUE(seen.ok); + ASSERT_EQ(seen.layout.size(), 1u); + EXPECT_FALSE(seen.layout[0].hole); + EXPECT_EQ(seen.data, b5); +} + +// 5. a zero (empty-buffer) write reads back as a hole after its frontier passes (range unmap). +TEST(CraftMemModel, ZeroWriteReadsAsHole) { + auto g = make_mem_replica_group(new_vol(), 3, PAGE); + auto [term, dl] = login_ok(g); + auto& r = *g.replicas[0]; + auto data = page_of(7); + ASSERT_TRUE(rg(r.write(chdr(term), 0, blk(7), blk(1), one_iov(data))).has_value()); + ASSERT_TRUE(rg(r.keep_alive(chdr(term, 0))).has_value()); + EXPECT_FALSE(rd(r, term, 0, 7, 1).layout[0].hole); // data present + + // empty sg_list => zero write. `len` (blk(1)) is the authoritative range to unmap. + ASSERT_TRUE(rg(r.write(chdr(term), 1, blk(7), blk(1), sisl::sg_list{})).has_value()); + ASSERT_TRUE(rg(r.keep_alive(chdr(term, 1))).has_value()); + auto out = rd(r, term, 1, 7, 1); + ASSERT_TRUE(out.ok); + ASSERT_EQ(out.layout.size(), 1u); + EXPECT_TRUE(out.layout[0].hole); // unmapped -> hole + EXPECT_EQ(out.data, page_of(0)); +} + +// 6. a DATA write of all-zero bytes reads back as a hole via the read-time scan. +TEST(CraftMemModel, AllZeroDataCollapsesToHole) { + auto g = make_mem_replica_group(new_vol(), 3, PAGE); + auto [term, dl] = login_ok(g); + auto& r = *g.replicas[0]; + auto zeros = page_of(0); // all-zero bytes, written as a normal (non-empty) data write + ASSERT_TRUE(rg(r.write(chdr(term), 0, blk(8), blk(1), one_iov(zeros))).has_value()); + ASSERT_TRUE(rg(r.keep_alive(chdr(term, 0))).has_value()); + auto out = rd(r, term, 0, 8, 1); + ASSERT_TRUE(out.ok); + ASSERT_EQ(out.layout.size(), 1u); + EXPECT_TRUE(out.layout[0].hole); +} + +// 7. commit is best-effort: keep_alive stalls just below the first missing slot, while the overlay +// still serves the higher write. +TEST(CraftMemModel, CommitStallsAtGap) { + auto g = make_mem_replica_group(new_vol(), 3, PAGE); + auto [term, dl] = login_ok(g); + auto& r = *g.replicas[0]; + auto buf = page_of(3); + ASSERT_TRUE(rg(r.write(chdr(term), 0, blk(5), blk(1), one_iov(buf))).has_value()); + ASSERT_TRUE(rg(r.write(chdr(term), 2, blk(6), blk(1), one_iov(buf))).has_value()); // gap at dLSN 1 + + auto c = rg(r.keep_alive(chdr(term, 2))); + ASSERT_TRUE(c.has_value()); + EXPECT_EQ(c->commit_lsn, 0); // stalled below the missing dLSN 1 + EXPECT_EQ(c->last_append_lsn, 2); + + auto out = rd(r, term, /*H*/ 2, 6, 1); // dLSN 2 still readable via the overlay + ASSERT_TRUE(out.ok); + EXPECT_FALSE(out.layout[0].hole); + EXPECT_EQ(out.data, buf); +} + +// 8. commit piggybacks on a WRITE: a later write carrying commit_lsn applies the earlier entry. +TEST(CraftMemModel, CommitPiggybacksOnWrite) { + auto g = make_mem_replica_group(new_vol(), 3, PAGE); + auto [term, dl] = login_ok(g); + auto& r = *g.replicas[0]; + auto b0 = page_of(0x11), b1 = page_of(0x22); + ASSERT_TRUE(rg(r.write(chdr(term), 0, blk(5), blk(1), one_iov(b0))).has_value()); // no commit yet + ASSERT_TRUE(rg(r.write(chdr(term, /*commit_lsn*/ 0), 1, blk(6), blk(1), one_iov(b1))).has_value()); // rides commit 0 + + auto ls = rg(r.get_lsns()); + ASSERT_TRUE(ls.has_value()); + EXPECT_EQ(ls->commit_lsn, 0); // dLSN 0 applied via the piggyback on the dLSN-1 write + EXPECT_EQ(ls->last_append_lsn, 1); + auto out = rd(r, term, 0, 5, 1); // served from the index (applied), not the overlay + ASSERT_TRUE(out.ok); + EXPECT_FALSE(out.layout[0].hole); + EXPECT_EQ(out.data, b0); +} + +// 9. commit piggybacks on a READ: reading with commit_lsn advances the frontier as a side-effect. +TEST(CraftMemModel, CommitPiggybacksOnRead) { + auto g = make_mem_replica_group(new_vol(), 3, PAGE); + auto [term, dl] = login_ok(g); + auto& r = *g.replicas[0]; + auto buf = page_of(0x33); + ASSERT_TRUE(rg(r.write(chdr(term), 0, blk(5), blk(1), one_iov(buf))).has_value()); + ASSERT_EQ(rg(r.get_lsns())->commit_lsn, -1); // not committed yet + + auto out = rd(r, term, /*H*/ 0, 5, 1, /*commit_lsn*/ 0); // read advances the frontier to 0 + ASSERT_TRUE(out.ok); + EXPECT_FALSE(out.layout[0].hole); + EXPECT_EQ(out.data, buf); + EXPECT_EQ(rg(r.get_lsns())->commit_lsn, 0); +} + +// 10. term fencing on write: an IO whose term != the session term is rejected (the protocol's ETERM). +TEST(CraftMemModel, WriteTermFencing) { + auto g = make_mem_replica_group(new_vol(), 3, PAGE); + auto [term, dl] = login_ok(g); + auto& r = *g.replicas[0]; + auto buf = page_of(1); + auto bad = rg(r.write(chdr(term + 1), 0, blk(5), blk(1), one_iov(buf))); + ASSERT_FALSE(bad.has_value()); + EXPECT_EQ(bad.error(), make_error_condition(craft_error::STALE_TERM)); + EXPECT_TRUE(rg(r.write(chdr(term), 0, blk(5), blk(1), one_iov(buf))).has_value()); +} + +// 11. keep_alive is ALSO term-fenced -- a stale client must not reset the liveness watchdog. +TEST(CraftMemModel, KeepAliveIsTermFenced) { + auto g = make_mem_replica_group(new_vol(), 3, PAGE); + auto [term, dl] = login_ok(g); + auto& r = *g.replicas[0]; + auto stale = rg(r.keep_alive(chdr(term + 1, 0))); + ASSERT_FALSE(stale.has_value()); + EXPECT_EQ(stale.error(), make_error_condition(craft_error::STALE_TERM)); + ASSERT_TRUE(rg(r.keep_alive(chdr(term, -1))).has_value()); // current term is accepted +} + +// 12. misaligned addr/len are rejected server-side with invalid_argument. +TEST(CraftMemModel, MisalignedIoRejected) { + auto g = make_mem_replica_group(new_vol(), 3, PAGE); + auto [term, dl] = login_ok(g); + auto& r = *g.replicas[0]; + auto buf = page_of(1); + auto const einval = std::make_error_condition(std::errc::invalid_argument); + + auto bad_addr = rg(r.write(chdr(term), 0, /*addr*/ 1, blk(1), one_iov(buf))); + ASSERT_FALSE(bad_addr.has_value()); + EXPECT_EQ(bad_addr.error(), einval); + + auto bad_len = rg(r.write(chdr(term), 0, blk(5), /*len*/ 100, one_iov(buf))); + ASSERT_FALSE(bad_len.has_value()); + EXPECT_EQ(bad_len.error(), einval); + + std::vector< uint8_t > dest(PAGE); + auto bad_read = rg(r.read(chdr(term), 0, /*addr*/ 3, blk(1), one_iov(dest))); + ASSERT_FALSE(bad_read.has_value()); + EXPECT_EQ(bad_read.error(), einval); +} + +// 13a. fault hook: a downed replica fails addressed I/O; the rest keep a quorum. +TEST(CraftMemModel, ReplicaDownFault) { + auto g = make_mem_replica_group(new_vol(), 3, PAGE); + auto [term, dl] = login_ok(g); + auto buf = page_of(1); + g.net->set_up(g.replicas[1]->id(), false); + auto down = rg(g.replicas[1]->write(chdr(term), 0, blk(5), blk(1), one_iov(buf))); + ASSERT_FALSE(down.has_value()); + EXPECT_EQ(down.error(), make_error_condition(craft_error::REPLICA_DOWN)); + EXPECT_TRUE(rg(g.replicas[0]->write(chdr(term), 0, blk(5), blk(1), one_iov(buf))).has_value()); + EXPECT_TRUE(rg(g.replicas[2]->write(chdr(term), 0, blk(5), blk(1), one_iov(buf))).has_value()); + g.net->set_up(g.replicas[1]->id(), true); +} + +// 13b. fault hook: force_subquorum makes a broadcast land on only a minority (< quorum acks). +TEST(CraftMemModel, SubQuorumFault) { + auto g = make_mem_replica_group(new_vol(), 3, PAGE); + auto [term, dl] = login_ok(g); + auto buf = page_of(1); + g.net->force_subquorum({g.replicas[0]->id()}); // only replica 0 accepts the next writes + int acks = 0; + for (auto& r : g.replicas) { + if (rg(r->write(chdr(term), 0, blk(5), blk(1), one_iov(buf))).has_value()) ++acks; + } + EXPECT_EQ(acks, 1); // minority: sub-quorum, not durable + g.net->clear_faults(); + acks = 0; + for (auto& r : g.replicas) { + if (rg(r->write(chdr(term), 1, blk(5), blk(1), one_iov(buf))).has_value()) ++acks; + } + EXPECT_EQ(acks, 3); // all members accept again +} diff --git a/src/lib/hb_internal.hpp b/src/lib/hb_internal.hpp index d4f4361..12b37f1 100644 --- a/src/lib/hb_internal.hpp +++ b/src/lib/hb_internal.hpp @@ -46,9 +46,6 @@ constexpr uint64_t Gi = Ki * Mi; namespace homeblocks { -using lba_t = uint64_t; -using lba_count_t = uint32_t; - template < class T > using shared = std::shared_ptr< T >; template < class T > diff --git a/src/lib/memory_backend/CMakeLists.txt b/src/lib/memory_backend/CMakeLists.txt deleted file mode 100644 index ea22f0f..0000000 --- a/src/lib/memory_backend/CMakeLists.txt +++ /dev/null @@ -1,22 +0,0 @@ -cmake_minimum_required (VERSION 3.11) - -add_library ("${PROJECT_NAME}_memory") -target_sources("${PROJECT_NAME}_memory" PRIVATE - mem_homeblks.cpp - $ -) -target_link_libraries("${PROJECT_NAME}_memory" - ${COMMON_DEPS} -) - -add_executable (memory_test) -target_sources(memory_test PRIVATE - $ -) - -target_link_libraries(memory_test - ${PROJECT_NAME}_memory - ${COMMON_TEST_DEPS} - -rdynamic -) -add_test(NAME MemoryTestCPU COMMAND memory_test -csv error) diff --git a/src/lib/memory_backend/mem_homeblks.cpp b/src/lib/memory_backend/mem_homeblks.cpp deleted file mode 100644 index ea4f088..0000000 --- a/src/lib/memory_backend/mem_homeblks.cpp +++ /dev/null @@ -1,12 +0,0 @@ -#include "mem_homeblks.hpp" - -namespace homeblocks { - -/// NOTE: We give ourselves the option to provide a different HR instance here than libhomeblocks.a -result< shared< home_blocks > > init_homeblocks(home_blocks_config cfg) { - return shared< home_blocks >(std::make_shared< MemoryHomeBlocks >(std::move(cfg))); -} - -MemoryHomeBlocks::MemoryHomeBlocks(home_blocks_config&& cfg) : HomeBlocksImpl::HomeBlocksImpl(std::move(cfg)) {} - -} // namespace homeblocks diff --git a/src/lib/memory_backend/mem_homeblks.hpp b/src/lib/memory_backend/mem_homeblks.hpp deleted file mode 100644 index 503c304..0000000 --- a/src/lib/memory_backend/mem_homeblks.hpp +++ /dev/null @@ -1,16 +0,0 @@ -#pragma once - -#include -#include - -#include "lib/homeblks_impl.hpp" - -namespace homeblocks { - -class MemoryHomeBlocks : public HomeBlocksImpl { -public: - MemoryHomeBlocks(home_blocks_config&& cfg); - ~MemoryHomeBlocks() override = default; -}; - -} // namespace homeblocks diff --git a/src/lib/volume/tests/CMakeLists.txt b/src/lib/volume/tests/CMakeLists.txt index 5163b8a..ed0dfa9 100644 --- a/src/lib/volume/tests/CMakeLists.txt +++ b/src/lib/volume/tests/CMakeLists.txt @@ -1,5 +1,9 @@ cmake_minimum_required (VERSION 3.11) +# These tests exercise the legacy byte-based async_read/async_write, now [[deprecated]] in favor of +# the CRAFT overloads. Demote (not silence) the warning so it stays visible without failing -Werror. +add_flags("-Wno-error=deprecated-declarations") + add_executable(test_volume) target_sources(test_volume PRIVATE test_volume.cpp diff --git a/src/lib/volume/volume.hpp b/src/lib/volume/volume.hpp index 206e908..39bb926 100644 --- a/src/lib/volume/volume.hpp +++ b/src/lib/volume/volume.hpp @@ -112,6 +112,11 @@ class VolumeMetrics : public sisl::MetricsGroup { void on_gather(); }; +// The CRAFT per-replica backend a memory-mode volume delegates its data plane to (defined in +// craft/craft_replica.hpp; only src/lib/craft needs the full type, so a forward decl here keeps +// volume.hpp free of that include). HomeStore-free. +class craft_replica; + class volume : public std::enable_shared_from_this< volume > { public: inline static auto const VOL_META_NAME = std::string("Volume2"); // different from old releae; @@ -181,6 +186,19 @@ class volume : public std::enable_shared_from_this< volume > { volume& operator=(volume const& volume) = delete; volume& operator=(volume&& volume) = default; + // Tag + constructor for the in-memory CRAFT reference model (src/lib/craft/memory). Builds a + // HomeStore-free volume whose CRAFT data plane is served by `backend`; skips all HomeStore setup + // (no repl_dev / index table / superblk write / chunk selectors / metrics). rd() and the index + // paths must not be used on such a volume -- only the CRAFT free functions, which route to + // craft_backend(). + struct mem_craft_tag {}; + explicit volume(volume_info&& info, shared< craft_replica > backend, mem_craft_tag) : + sb_{VOL_META_NAME}, craft_backend_{std::move(backend)} { + vol_info_ = + std::make_shared< volume_info >(info.id, info.size_bytes, info.page_size, info.name, info.ordinal); + m_state_ = volume_state::ONLINE; + } + virtual ~volume() = default; // static APIs exposed to HomeBlks Implementation Layer; @@ -210,6 +228,10 @@ class volume : public std::enable_shared_from_this< volume > { std::string id_str() const { return boost::uuids::to_string(vol_info_->id); }; const ReplDevPtr& rd() const { return rd_; } + // The CRAFT per-replica backend for a memory-mode volume (null for HomeStore-backed volumes). + // The CRAFT free functions in home_blocks.hpp route here. + craft_replica* craft_backend() const { return craft_backend_.get(); } + volume_info_ptr info() const { return vol_info_; } std::string to_string() { return vol_info_->to_string(); } @@ -284,6 +306,7 @@ class volume : public std::enable_shared_from_this< volume > { false}; // indicates if volume destroy has started, avoid destroy to be executed more than once. std::atomic< volume_state > m_state_; // in-memory sb state, avoid taking lock in IO path; std::unique_ptr< VolumeMetrics > metrics_; + shared< craft_replica > craft_backend_{nullptr}; // CRAFT per-replica backend (memory-mode volumes only) }; // RAII: marks an IO in-flight on its volume for the op's duration -- so destroy()/shutdown() wait for it (and diff --git a/src/test/CMakeLists.txt b/src/test/CMakeLists.txt index 15def27..792f36c 100644 --- a/src/test/CMakeLists.txt +++ b/src/test/CMakeLists.txt @@ -1,8 +1,9 @@ cmake_minimum_required (VERSION 3.11) # Test-only ublk adapter. Relax pedantic/unused-parameter for the kernel UAPI headers (ublk_cmd.h, -# ublksrv.h) pulled in transitively via ublkpp; keep -Werror/-Wall/-Wextra. -add_flags("-Wno-pedantic -Wno-unused-parameter") +# ublksrv.h) pulled in transitively via ublkpp; keep -Werror/-Wall/-Wextra. Demote (not silence) the +# deprecation warning: homeblk_disk.cpp still calls the legacy async_read/async_write on purpose. +add_flags("-Wno-pedantic -Wno-unused-parameter -Wno-error=deprecated-declarations") # ublkpp is a test_requires; if it isn't available (e.g. a tests-disabled build) skip the adapter+CLI. find_package(UblkPP QUIET)