feat(eql_v3): SteVec value-inclusive selectors + envelope wire format#412
Open
coderdan wants to merge 1 commit into
Open
feat(eql_v3): SteVec value-inclusive selectors + envelope wire format#412coderdan wants to merge 1 commit into
coderdan wants to merge 1 commit into
Conversation
Exact equality on an encrypted-JSON field now works for every scalar type
(text, bigint, numeric included) via document containment on a value-inclusive
selector: each JSON node emits a value entry whose selector bakes in the value
(SEL(type-tag ‖ path ‖ canonical(value))), so the selector's presence is an
exact, injective match. Field equality is `col @> $1::eql_v3.query_json`; the
per-value `hm` term is retired, and extract-surface `=`/`<>` is blocked for
every family (a path entry carries no value selector). Ranges stay on the
extract surface. The single-entry containment operators are dropped.
The SteVec document wire format is restructured to `{v, k:"sv", i, h, sv}`:
the key-retrieval material is hoisted once to a document-level key header `h`
(no root `c`), and each entry's `c` is raw AEAD output whose nonce derives
from the entry's own selector rather than a shared per-document IV. This
shrinks a document ~30% and closes an intra-document leak — equal values at
different paths, and the constant value-entry plaintext, no longer produce
byte-identical ciphertexts, and each ciphertext is bound to its selector.
Value entries encrypt a fixed versioned sentinel (not the empty string) so a
value entry stays distinguishable from a genuine "" leaf.
`eql_v3.jsonb_array_elements_text` is removed (a bare-ciphertext stream is no
longer independently decryptable).
Breaking wire-format change: stored SteVec documents must be re-encrypted;
there is no mechanical v2->v3 conversion (`from_v2` fails closed on a SteVec
document and on `hm`-bearing query entries). Bindings gain the KeyHeader and
EntryCiphertext newtypes; the SteVec term becomes an optional `op` (term-less
entries take an explicit empty arm). Docs, changeset, and upgrade note U-010
updated; fixtures and the public-surface snapshot regenerated. Full spec in
docs/plans/stevec-envelope-key-header-rfc.md.
Claude-Session: https://claude.ai/code/session_01FnhDhqdknNLWrHWPZhQtWq
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reworks the encrypted-JSON (SteVec) surface so exact field equality is value-selector presence for every scalar type, and restructures the SteVec document wire format around a per-document key header with selector-derived entry nonces.
Stacks on #410 (json_entry ↔ query_ cross-type operators) — GitHub will retarget this to
mainwhen #410 merges. This is the EQL half of a cross-repo change; the encryption-client and protect-ffi halves ship in their own branches and merge bottom-up (client release first).What changed
Value-inclusive selectors. Each JSON node emits a path entry
{s, c, op?}plus a value entry whose selectorSEL(type-tag ‖ path ‖ canonical(value))bakes in the value, so the selector's presence in a stored document is an exact, injective match — correct fortext/bigint/numeric, where the oldop-based comparison collided ("café"=="cafe",9007199254740993==…992). Field equality is now document containment:WHERE col @> $1::eql_v3.query_json. The per-valuehmterm is retired; extract-surface=/<>is blocked for every family (a path entry carries no value selector), while ranges stay on the extract surface. The single-entry containment operators are dropped.Envelope wire format. A document is now
{v, k:"sv", i, h, sv:[…]}. The key-retrieval material (IV, tag, descriptor, keyset) is hoisted once to a document-levelhfield instead of being repeated inside every entry; there is no rootc(the root ciphertext is the rootsventry). Each entry'scis the raw AEAD output only, and its nonce derives from the entry's own selector (hex_decode(s)[..12]) rather than a shared per-document IV. This shrinks a document ~30% (measured on the fixture: 2238 → 1505 chars) and closes an intra-document leak — equal values at different paths, and the constant value-entry plaintext, no longer produce byte-identical ciphertexts, and each ciphertext is cryptographically bound to its selector (a within-document graft fails to decrypt). Value entries encrypt a fixed versioned sentinel (not the empty string), so a value entry stays distinguishable from a genuine""leaf.eql_v3.jsonb_array_elements_textis removed (a bare-ciphertext stream is no longer independently decryptable).Full design rationale (including the options considered and the sentinel / descriptor / array-marker decisions) is in
docs/plans/stevec-envelope-key-header-rfc.md.Breaking
Stored SteVec documents must be re-encrypted — there is no mechanical v2→v3 conversion (
from_v2fails closed on a SteVec document and onhm-bearing query entries). The changeset is amajorbump and upgrade note U-010 documents the migration.Verification
KeyHeader/EntryCiphertextnewtypes and a term-less-optionalSteVecTerm; TS/JSON-Schema regenerated;types:check/codegen:parityare a deterministic no-op after regeneration.hpresent, no rootc, nohm, all term-less ciphertexts distinct).encrypted_domain2403,v3_jsonb_tests68,v3_json_entry_cross_type7,v3_json_entry_query_operand_e2e9,v3_json_storage8,v3_jsonb_bindings2,v3_jsonb_operator_surface7,payload_schema32,v3_public_surface3 (snapshot regenerated),v3_privilege5,v3_operator_equivalents7,v3_uninstall3; eql-bindings all suites.Docs updated:
json-support.md,sql-support.md,docs/upgrading/v3.0.md(U-010).https://claude.ai/code/session_01FnhDhqdknNLWrHWPZhQtWq