Web platform support: Store, Box and queries on IndexedDB#822
Open
mechaadi wants to merge 6 commits into
Open
Conversation
The objectbox package, code generated by objectbox_generator, and objectbox_flutter_libs now compile for the web platform with both dart2js and dart2wasm. There is no database implementation on web yet: this fills the empty lib/src/web/ stubs (added with the original conditional-import scaffolding in objectbox#189) with API-compatible implementations that throw UnsupportedError. Details: - Enable the store conditional export (was commented out) and switch all platform conditionals from dart.library.html to dart.library.js_interop so they also work under dart2wasm. - Keep shared code free of dart:ffi: copy OBXVectorDistanceType and OBXHnswFlags into modelinfo/enums.dart (same approach the file already used for other OBX enums), move EntityDefinition.objectFromData into a native-only extension in bindings/helpers.dart, re-point to_one.dart and model_hnsw_params.dart at platform-neutral imports, and export InternalStoreAccess through the conditional store facade. - Move StoreConfiguration and the vector search result types to shared files used by both platform variants. - Mirror the full public API in lib/src/web/ (store, box, query incl. builder/params/property, transaction, model, sync, admin). Query property constructors are functional so generated static finals can initialize; Admin.isAvailable(), Sync.isAvailable() and Store.isOpen() return false on web so guard code keeps working; everything else throws UnsupportedError pointing at issue objectbox#185. - Generator: emit IdUid.fromString('id:uid') instead of integer literals; UIDs are random 64-bit values that dart2js rejects as literals above 2^53. Relax the IdUid parse range check, whose (1 << 63) - 1 upper bound wrapped negative under JS bit semantics (the bound is enforced by int.parse on the VM anyway). - flutter_libs: split into native/web variants behind a conditional export; on web defaultStoreDirectory() returns a logical name and loadObjectBoxLibraryAndroidCompat() is a no-op. Verified: all 213 objectbox_test tests pass on the VM; a smoke package with entities, relations and an HNSW index generates, compiles with dart2js and dart2wasm, and passes behavior tests in Chrome under both compilers (model loads, entities construct, openStore throws UnsupportedError, availability checks return false).
…#185) Store and Box are now fully functional in the browser: an in-memory engine (keeping the synchronous ObjectBox API) persisted to IndexedDB with a write-behind queue. Engine (lib/src/web/engine.dart): - In-memory records per entity (id-sorted, so getAll() returns objects in id order like native), loaded once asynchronously on open. - Write-behind persistence: mutations mark records dirty; a microtask flushes the batch in a single IndexedDB transaction. close() flushes; a re-open of the same path waits for the previous connection to finish closing. Requests navigator.storage.persist() best-effort. - Monotonic id sequences persisted in a meta store (ids of removed objects are not reused, also across sessions). - @unique enforcement via in-memory value indexes rebuilt on load, including UNIQUE_ON_CONFLICT_REPLACE; throws UniqueViolationException. - Standalone ToMany relations in a relation store; removing an object cleans up relation rows on both sides. - Write transactions use an undo log: on error, records, id sequences, unique indexes and relation rows are restored and the restored state re-queued for persistence. Mirrors the native Transaction protocol so shared ToOne/ToMany code works unchanged. - Change events power store.watch<T>() and store.entityChanges. Box: put/putMany with the native relation protocol (reserve id first for ToOne cycles, apply ToOne targets, serialize, apply ToMany), PutMode semantics, get/getAll/getMany, count/contains/isEmpty, remove/removeMany/removeAll, async variants (same-thread), Store.attach and attachByConfiguration with engine ref-counting (used by lazy ToOne loading). Queries remain unsupported until phase 3. FlatBuffers on web: dart2js does not support ByteData.get/setInt64, so serialization crashed on every put. Added a vendored copy of package:flat_buffers (Apache-2.0) with JavaScript-safe 64-bit accessors built from two 32-bit halves, exposed through the conditional export package:objectbox/flatbuffers.dart. Native platforms keep using the real package via the same export, so types are identical; generated code now imports the shim. Also new: Store.ready (completes when persisted data is loaded; immediate on native) awaited by generated Flutter openStore(), and PutMode moved into the shared facade (same pattern as TxMode). Also: minimum Dart SDK is now 3.4 (dart:js_interop / package:web); the ffigen bindings are pinned to language 2.19 until regenerated (Dart 3 requires Struct/Opaque subtypes to be final); fixed newly-flagged lints from the language bump (unreachable switch defaults, redundant non-null assertions). Verified: 17 browser tests pass under both dart2js and dart2wasm in Chrome - CRUD roundtrips of all property types, id assignment and PutMode rules, unique enforcement, ToOne auto-put + lazy load, ToOne backlinks, ToMany add/remove and target-removal cleanup, transaction rollback (including a failed put inside a relations transaction), watch/entityChanges, store registry/attach, and IndexedDB persistence across close/reopen (data, relations, unique index, id sequence). All 213 native tests pass (an occasional single-test failure in the suite reproduces on unmodified upstream main and is a pre-existing test race, not a regression).
box.query() now works on web: a pure-Dart evaluator over the Condition
tree running against the web engine's in-memory records, reading
property values generically from the stored FlatBuffers (fb_reader).
- Conditions: string (equals/notEquals/contains/startsWith/endsWith/
greaterThan/lessThan/oneOf with per-condition case sensitivity
falling back to queriesCaseSensitiveDefault), integer/date/dateNano
(equals/notEquals/compare/between/oneOf/notOneOf), double
(compare/between), bool, byte vector (lexicographic compare), string
vector containsElement, isNull/notNull, and/or groups (andAll/orAny,
& and | operators). Null property values only match isNull, like the
native core.
- order(): descending, caseSensitive (string ordering is
case-insensitive by default), nullsLast, nullsAsZero, unsigned;
default order is ascending by id. offset/limit setters.
- Results: find/findFirst/findUnique (NonUniqueResultException)/
findIds/count/remove/stream + async variants, describe/
describeParameters.
- Query parameters: query.param(property, alias:) with value/values/
twoValues setters, matching conditions by property identity or alias
(also inside link conditions), and nearestNeighborsF32 re-targeting.
- Property queries: min/max/sum/average/count/find with distinct and
string case sensitivity; nulls skipped or replaced via
replaceNullWith; offset/limit are not applied (like native).
- Links: link/backlink (ToOne, via the relation property), linkMany/
backlinkMany (standalone ToMany via the engine's relation store),
nested links on the returned sub-builder, and
QueryBacklinkToMany.relationCount. Sub-builders only support link
calls; build()/watch()/order() throw StateError like the native
private _QueryBuilder split.
- Vector search: nearestNeighborsF32 evaluated as an exact brute-force
scan ordered by score, capped at maxResultCount, combinable with
filter conditions; findWithScores/findIdsWithScores. Distances:
Euclidean (squared, default), cosine, dot product (normalized and
non-normalized); Geo throws UnsupportedError.
- QueryBuilder.watch({triggerImmediately}) emits the query on entity
changes.
Verified: 13 new browser query tests plus the existing 17 engine/stub
tests (30 total) pass under both dart2js and dart2wasm in Chrome; all
213 native tests pass (no native code touched this phase).
…x#185) Same dart2js constraint as the IdUid change: retired entity/index/ property/relation UIDs are 64-bit values that cannot be written as integer literals in code compiled to JavaScript. Went unnoticed until generating for a model with schema-evolution history (retired UIDs); verified against the fitloop-business model.
Ports the browser test suite for the web implementation into the repo as an internal (unpublished) package, per the contribution guidelines: 30 tests covering Store/Box CRUD of all property types, PutMode rules, @unique enforcement, ToOne/ToMany/backlinks, transaction rollback, watch/entityChanges, the store registry/attach, IndexedDB persistence across close/reopen, and the full query engine (conditions, ordering, offset/limit, parameters, property queries, links, relationCount and nearestNeighborsF32 with scores). The new web-test workflow runs the suite in Chrome with both dart2js and dart2wasm. Unlike objectbox_test, no native library is required. Also mention the retired-UID generator fix in the CHANGELOG.
The name described the phase-1 stub implementation; the file now tests the deliberately unsupported web APIs (graceful degradation: Sync/Admin availability checks return false, unsupported APIs throw UnsupportedError) and that generated code loads without a native library.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #185 (phases 1–3 of the approach discussed there).
This PR makes objectbox, objectbox_generator-generated code and objectbox_flutter_libs compile and run on the web platform, under both dart2js and dart2wasm. It builds on the conditional-import scaffolding that was added for this purpose back in #189 (the empty lib/src/web/ placeholders and per-file facades), and follows the layering sketched by @greenrobot in this comment, with pure Dart taking the place of the WASM core: since the native core can't currently target WASM (mmap), the web implementation is a self-contained Dart engine behind the existing public API.
What works on web
Store/Box: all CRUD operations with PutMode semantics, monotonic ID assignment (ids of removed objects are not reused, also across sessions), @unique enforcement incl. ConflictStrategy.replace (UniqueViolationException), ToOne/ToMany/backlinks via the unchanged shared relation code, runInTransaction with rollback on error (undo log), watch()/entityChanges streams, Store.attach, in-memory databases via memory: paths.
Queries: the full condition set (string ops with case-sensitivity flags, integer/date/double/bool/byte-vector/string-vector conditions, isNull/oneOf/between, and/or groups), order() flags, offset/limit, find*/count/remove/stream, query parameters with alias support, property queries (min/max/sum/average/distinct/find), relation links (link/backlink/linkMany/backlinkMany, nested, relationCount), and nearestNeighborsF32 evaluated as an exact brute-force scan with findWithScores (Euclidean/cosine/dot-product; Geo throws).
Persistence: an in-memory working set (preserving the synchronous API) flushed to IndexedDB in batched transactions via a write-behind queue; data, unique indexes, relations and ID sequences survive reload. navigator.storage.persist() is requested best-effort.
Design notes
New public API: Store.ready, a future that completes when persisted data is loaded (completes immediately on native). Generated Flutter openStore() awaits it.
FlatBuffers on web: dart2js has no ByteData.get/setInt64, so serialization cannot use package:flat_buffers as-is. The web build uses a vendored copy (Apache-2.0 headers retained) with JS-safe 64-bit accessors, exposed through a conditional export package:objectbox/flatbuffers.dart; native platforms re-export the real package through the same entrypoint, so types are identical and native behavior is unchanged. Generated code imports the shim.
Generator: model IDs/UIDs and retired-UID lists are emitted as runtime-parsed strings (IdUid.fromString('1:…')) — UIDs are 64-bit values that dart2js rejects as literals above 2^53. IdUid's former (1 << 63) - 1 range check wrapped negative under JS 32-bit shift semantics and is now a lower-bound check (the upper bound is enforced by int.parse on the VM).
Shared code hygiene: modelinfo/relations no longer transitively import dart:ffi (EntityDefinition.objectFromData moved to a native-only extension; the two HNSW enums are mirrored in enums.dart like the other OBX enums already were; PutMode moved into the facade alongside TxMode).
Minimum SDK is now 3.4 (dart:js_interop / package:web). The ffigen bindings are pinned // @dart=2.19 until regenerated (Dart 3 class-modifier rules).Not supported on web
Sync and Admin (isAvailable() returns false so guard code keeps working), Store.fromReference, isolate APIs (runAsync/runInTransactionAsync run on the same thread), Geo vector distance. Queries scan the in-memory records (no index acceleration). Durability is write-behind: writes flush within a microtask, so a hard browser crash can lose the final moments; close() flushes. Multi-tab access is not coordinated.
Testing
Validated end-to-end in a production Flutter app (entities with unique indexes, relations and schema-evolution history):
flutter build web --releaseand Android builds green, app verified working in the browser on IndexedDB.I'm happy to split this into smaller PRs (the commits are already staged that way: compile-support → engine → queries → generator fixes), adjust naming/placement, or rework the FlatBuffers approach if you'd prefer a different mechanism.