VSAG new-format index snapshot fails to deserialize after restart

[footka]

Problem

An HNSW_SQ vector index snapshot written with VSAG's new serialization format (use_old_serial_format=false) cannot be loaded after seekdb restarts. The first ANN query triggers lazy loading and fails with errors similar to:

alloc failed because a single allocation larger than 4 GiB is not allowed
alloc_size: 4294967312
Vsag failed to Deserialize: std::bad_alloc
Using the vsag interface returns an error

This prevents ANN queries from running after restart.

Root cause

The new VSAG format seeks to footer metadata during deserialization. seekdb provides VSAG with an ObIStreamBuf callback stream whose seek support is scoped to the current callback buffer rather than the complete serialized stream.

The footer seek is therefore interpreted with the wrong offset. A length derived from that position underflows/wraps to approximately UINT32_MAX, which leads to an attempted allocation close to 4 GiB and causes deserialization to fail.

Reproduction

1. Create HNSW_SQ vector indexes.
2. Insert 100,000 rows and wait for the snapshot to be persisted.
3. Insert another 30,000 rows and wait for the incremental data to be persisted.
4. Restart seekdb.
5. Execute the first ANN query so the indexes are lazily loaded.

Expected behavior

The persisted snapshot and incremental data load successfully after restart, and the ANN query runs normally. In the reproduction above, the loaded index statistics should report snap_cnt=100000 and incr_cnt=30000.

Proposed short-term fix

Serialize VSAG indexes with use_old_serial_format=true until the callback stream implements global seek semantics required by the new format.

This setting affects newly written snapshots. Existing snapshots written in the incompatible new format may need to be rebuilt or migrated.

Validation

The short-term fix was validated on macOS arm64 with the complete vector_reboot_lite HNSW_SQ restart case:

• process restart completed successfully;
• pre-query index statistics confirmed that indexes were not loaded;
• the first ANN query lazily loaded all indexes;
• all indexes reported incr_cnt=30000 and snap_cnt=100000;
• the test passed in 833.48 seconds;
• no single-allocation or VSAG deserialization error was present in the server log.