Skip to content

feat(eql_v3): SteVec value-inclusive selectors + envelope wire format#412

Open
coderdan wants to merge 1 commit into
eql-v3-json-entry-cross-type-operatorsfrom
dan/cip-3551-eql-value-selectors
Open

feat(eql_v3): SteVec value-inclusive selectors + envelope wire format#412
coderdan wants to merge 1 commit into
eql-v3-json-entry-cross-type-operatorsfrom
dan/cip-3551-eql-value-selectors

Conversation

@coderdan

Copy link
Copy Markdown
Contributor

Reworks the encrypted-JSON (SteVec) surface so exact field equality is value-selector presence for every scalar type, and restructures the SteVec document wire format around a per-document key header with selector-derived entry nonces.

Stacks on #410 (json_entry ↔ query_ cross-type operators) — GitHub will retarget this to main when #410 merges. This is the EQL half of a cross-repo change; the encryption-client and protect-ffi halves ship in their own branches and merge bottom-up (client release first).

What changed

Value-inclusive selectors. Each JSON node emits a path entry {s, c, op?} plus a value entry whose selector SEL(type-tag ‖ path ‖ canonical(value)) bakes in the value, so the selector's presence in a stored document is an exact, injective match — correct for text / bigint / numeric, where the old op-based comparison collided ("café" == "cafe", 9007199254740993 == …992). Field equality is now document containment: WHERE col @> $1::eql_v3.query_json. The per-value hm term is retired; extract-surface = / <> is blocked for every family (a path entry carries no value selector), while ranges stay on the extract surface. The single-entry containment operators are dropped.

Envelope wire format. A document is now {v, k:"sv", i, h, sv:[…]}. The key-retrieval material (IV, tag, descriptor, keyset) is hoisted once to a document-level h field instead of being repeated inside every entry; there is no root c (the root ciphertext is the root sv entry). Each entry's c is the raw AEAD output only, and its nonce derives from the entry's own selector (hex_decode(s)[..12]) rather than a shared per-document IV. This shrinks a document ~30% (measured on the fixture: 2238 → 1505 chars) and closes an intra-document leak — equal values at different paths, and the constant value-entry plaintext, no longer produce byte-identical ciphertexts, and each ciphertext is cryptographically bound to its selector (a within-document graft fails to decrypt). Value entries encrypt a fixed versioned sentinel (not the empty string), so a value entry stays distinguishable from a genuine "" leaf. eql_v3.jsonb_array_elements_text is removed (a bare-ciphertext stream is no longer independently decryptable).

Full design rationale (including the options considered and the sentinel / descriptor / array-marker decisions) is in docs/plans/stevec-envelope-key-header-rfc.md.

Breaking

Stored SteVec documents must be re-encrypted — there is no mechanical v2→v3 conversion (from_v2 fails closed on a SteVec document and on hm-bearing query entries). The changeset is a major bump and upgrade note U-010 documents the migration.

Verification

  • Bindings gain the KeyHeader / EntryCiphertext newtypes and a term-less-optional SteVecTerm; TS/JSON-Schema regenerated; types:check / codegen:parity are a deterministic no-op after regeneration.
  • Fixtures regenerated through the new client; the SteVec document confirms the shape (h present, no root c, no hm, all term-less ciphertexts distinct).
  • Green: encrypted_domain 2403, v3_jsonb_tests 68, v3_json_entry_cross_type 7, v3_json_entry_query_operand_e2e 9, v3_json_storage 8, v3_jsonb_bindings 2, v3_jsonb_operator_surface 7, payload_schema 32, v3_public_surface 3 (snapshot regenerated), v3_privilege 5, v3_operator_equivalents 7, v3_uninstall 3; eql-bindings all suites.

Docs updated: json-support.md, sql-support.md, docs/upgrading/v3.0.md (U-010).

https://claude.ai/code/session_01FnhDhqdknNLWrHWPZhQtWq

Exact equality on an encrypted-JSON field now works for every scalar type
(text, bigint, numeric included) via document containment on a value-inclusive
selector: each JSON node emits a value entry whose selector bakes in the value
(SEL(type-tag ‖ path ‖ canonical(value))), so the selector's presence is an
exact, injective match. Field equality is `col @> $1::eql_v3.query_json`; the
per-value `hm` term is retired, and extract-surface `=`/`<>` is blocked for
every family (a path entry carries no value selector). Ranges stay on the
extract surface. The single-entry containment operators are dropped.

The SteVec document wire format is restructured to `{v, k:"sv", i, h, sv}`:
the key-retrieval material is hoisted once to a document-level key header `h`
(no root `c`), and each entry's `c` is raw AEAD output whose nonce derives
from the entry's own selector rather than a shared per-document IV. This
shrinks a document ~30% and closes an intra-document leak — equal values at
different paths, and the constant value-entry plaintext, no longer produce
byte-identical ciphertexts, and each ciphertext is bound to its selector.
Value entries encrypt a fixed versioned sentinel (not the empty string) so a
value entry stays distinguishable from a genuine "" leaf.
`eql_v3.jsonb_array_elements_text` is removed (a bare-ciphertext stream is no
longer independently decryptable).

Breaking wire-format change: stored SteVec documents must be re-encrypted;
there is no mechanical v2->v3 conversion (`from_v2` fails closed on a SteVec
document and on `hm`-bearing query entries). Bindings gain the KeyHeader and
EntryCiphertext newtypes; the SteVec term becomes an optional `op` (term-less
entries take an explicit empty arm). Docs, changeset, and upgrade note U-010
updated; fixtures and the public-surface snapshot regenerated. Full spec in
docs/plans/stevec-envelope-key-header-rfc.md.

Claude-Session: https://claude.ai/code/session_01FnhDhqdknNLWrHWPZhQtWq
@coderabbitai

coderabbitai Bot commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: e2bf9fe2-8a48-4d05-84f4-4b0d3cdddf75

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch dan/cip-3551-eql-value-selectors

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant