feat : recover map and arrays resizing (#37)

Author: virtexalejandro

CHANGELOG.md

@@ -37,6 +37,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **common/schema**: the schema's top-level **`"buckets"` array is now parsed and is the source of truth for the frame bucket layout** -- `SchemaRegistry::GetBucketNames()` returns the declared names in order (`"buckets"[i]` names Frame bucket index `i`), and `PropertyAddressCache` carries them (`bucket_names`) so the reader can use them too.  A new `Buckets` validation rule rejects a malformed key when present (non-array, non-string entries, empty or duplicate names); schemas without the key stay valid (legacy)
 - **writer/api**: **schema-driven bucket normalization** -- before finalization the writer rearranges every frame's buckets into the schema-declared layout: buckets are reordered to schema order, declared buckets missing from the frame are created empty, and a bucket the schema does not declare rejects the frame with the new **`VtxErrorCode::BucketUnresolved`** (observable via `TryRecordFrame`).  Frames built positionally (no `bucket_map`) adopt the schema layout as long as they do not exceed the declared bucket count.  Schemas without a `"buckets"` array skip the normalization entirely
 - **reader/api**: **bucket names restored on read** -- bucket names never hit the wire (both formats serialize buckets positionally), so deserialized frames used to come back with an empty `bucket_map`.  The reader now stamps `bucket_map` from the embedded schema's `"buckets"` array when chunks are deserialized, so by-name lookups (`Frame::GetBucket(name)` const) and `bucket_map` iteration work on read frames.  Replays written without a `"buckets"` array keep the old positional-only behavior
+- **common/schema**: **array & map pre-sizing** -- extends the existing scalar pre-sizing (`type_max_indices`) so declared *array* and *map* fields are also materialized on load.  `SchemaStruct` / `StructSchemaCache` gain `array_max_indices` (max Array-container index per `FieldType`) and `map_max_index` (count of Struct-valued map fields); `Helpers::ResizeContainerToMaxIndices` now pre-creates one empty subarray per declared array field in the matching `FlatArray` (via new `FlatArray::EnsureSubArrayCount`) and pre-sizes `map_properties`.  A declared-but-unpopulated array/map field is now present-and-empty instead of absent, symmetric with scalars.  Applies wherever scalar pre-sizing already ran (the four frame loaders + `schema_dynamic_loader`); the loader CRTP hook `GetTypeMaxIndices` became `GetStructSizing` (returns the `StructSchemaCache`)
+- **reader/api**: **declared-empty arrays restored on read** -- an array with no data is not serialized (nothing to store), so deserialized entities used to come back missing those subarrays even for schema-declared array fields (maps already round-trip their slot count). The reader now re-creates the declared-but-empty subarrays from the embedded schema (`Helpers::EnsureDeclaredArrays`, grow-only, recursing into nested structs / struct-array elements / map values) so a read frame mirrors the array layout of an ingest-loaded frame. Arrays that carry data round-trip unchanged (their `offsets` already encode empty subarrays); scalars and maps are untouched
 
 ### Changed
 
@@ -60,7 +62,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **writer/reader (Protobuff -> Protobuf rename)** (#28): the misspelled "Protobuff" was removed from the public API -- **breaking, no aliases kept**.  `SerializationFormat::Protobuffs` -> **`SerializationFormat::Protobuf`**; `CreateProtobuffWriterFacade` -> **`CreateProtobufWriterFacade`**; `CreateProtobuffNetworkWriterFacade` -> **`CreateProtobufNetworkWriterFacade`**; reader `CreateProtobuffFacade` -> **`CreateProtobufFacade`** (internal `ProtobuffFacadeImpl` -> `ProtobufFacadeImpl`).  Migrate call sites with a literal `Protobuff` -> `Protobuf` rename
 - **common/types**: `FlatArray::GetSubArray` & `FlatArray::GetMutableSubArray` did not have a check for StartIndex being greater or equal to EndIndex. In exceptional cases, like an empty array this can cause a crash or undefined behaviour in the best case scenario.
 - **writer/flatbuffers**: `FlatBuffersVtxPolicy::FromNative` hardcoded exactly two bucket slots -- `buckets[0]` was renamed to `"data"`, `buckets[1]` to `"bone_data"`, and **any bucket at index >= 2 was silently dropped** from the file.  It now iterates all buckets generically (same shape as the Protobuf policy, shared `Serialization::SortBucketByTypeId` helper in `bucket_type_sort.h`), preserving every bucket and the frame's `bucket_map`.  Bucket naming is schema-driven (see the `"buckets"` entries above), not serializer-driven
-- **reader/schema**: the reader-side `PropertyAddressCache` built from the embedded schema (`PopulateCacheFromJsonString`) hand-copied the registry cache and **omitted `type_max_indices`**, so `ValidateEntity` on read frames treated every per-type max as 0 (any populated property would flag `FieldIndexOutOfRange` once frame validation ran).  It now reuses `SchemaRegistry::GetPropertyCache()` verbatim.  Also makes `ValidateReplay`'s per-frame pass effective: it iterates `frame.bucket_map`, which was always empty on read frames before bucket-name restoration
+- **reader/schema**: the reader-side `PropertyAddressCache` built from the embedded schema (`PopulateCacheFromJsonString`) hand-copied the registry cache and **omitted `type_max_indices`**, so `ValidateEntity` on read frames treated every per-type max as 0 (any populated property would flag `FieldIndexOutOfRange` once frame validation ran).  It now reuses `SchemaRegistry::GetPropertyCache()` verbatim.  This also makes `ValidateReplay`'s per-frame pass effective **for replays whose schema declares a `"buckets"` array** -- it iterates `frame.bucket_map`, which was always empty on read frames before bucket-name restoration.  Replays whose schema has no `"buckets"` array still read back with an empty `bucket_map`, so their per-frame pass remains a no-op (unchanged from before)
 - **samples/benchmarks**: `basic_write.cpp` and `bench_writer.cpp` created a `"Players"` bucket while writing against the arena schema, which declares `"buckets": ["entity"]` -- under schema-driven normalization those frames would now be rejected.  Both use `"entity"`
 
 

sdk/include/vtx/reader/core/vtx_reader.h

@@ -667,6 +667,7 @@ namespace VTX {
                 SerializerPolicy::ProcessChunkData(idx, compressed_blob, stop_token, cc.native_frames,
                                                    cc.decompressed_blob, cc.raw_frames_spans);
                 RestoreBucketNames(cc.native_frames);
+                RestoreDeclaredArrays(cc.native_frames);
                 return cc;
             } catch (const std::exception& e) {
                 VTX_ERROR("[READER] Chunk {} deserialization failed: {}", idx, e.what());
@@ -696,6 +697,43 @@ namespace VTX {
             }
         }
 
+        // A declared array field with no data is not serialized (an empty array has
+        // nothing to store), so deserialized entities come back missing those
+        // subarrays. Re-create the declared-but-empty subarrays from the schema so a
+        // read frame mirrors the same array layout an ingest-loaded frame has (maps
+        // already round-trip their slot count, so only arrays need this). Recurses
+        // into nested structs / struct-array elements / map values, matching the
+        // loader's recursive PrepareContainer. Grow-only: never touches populated
+        // arrays, scalars, or maps.
+        void RestoreDeclaredArrays(std::vector<VTX::Frame>& frames) const {
+            if (property_address_cache_.structs.empty())
+                return;
+            for (auto& frame : frames) {
+                for (auto& bucket : frame.GetMutableBuckets()) {
+                    for (auto& entity : bucket.entities) {
+                        PreSizeDeclaredArraysRecursive(entity);
+                    }
+                }
+            }
+        }
+
+        void PreSizeDeclaredArraysRecursive(VTX::PropertyContainer& container) const {
+            auto it = property_address_cache_.structs.find(container.entity_type_id);
+            if (it != property_address_cache_.structs.end()) {
+                Helpers::EnsureDeclaredArrays(container, it->second.array_max_indices);
+            }
+            for (auto& nested : container.any_struct_properties)
+                PreSizeDeclaredArraysRecursive(nested);
+            for (auto& nested : container.any_struct_arrays.data)
+                PreSizeDeclaredArraysRecursive(nested);
+            for (auto& map_item : container.map_properties)
+                for (auto& value : map_item.values)
+                    PreSizeDeclaredArraysRecursive(value);
+            for (auto& map_item : container.map_arrays.data)
+                for (auto& value : map_item.values)
+                    PreSizeDeclaredArraysRecursive(value);
+        }
+
     private:
         std::string filepath_;
 

sdk/src/vtx_writer/src/vtx/writer/formatters/flatbuffers_vtx_policy.cpp

@@ -21,14 +21,7 @@ std::unique_ptr<VTX::FlatBuffersVtxPolicy::FrameType> VTX::FlatBuffersVtxPolicy:
     sorted_frame->GetMutableBuckets().resize(native_buckets.size());
 
     for (size_t b_idx = 0; b_idx < native_buckets.size(); ++b_idx) {
-        const auto& src_bucket = native_buckets[b_idx];
-        auto& dst_bucket = sorted_frame->GetBucket(static_cast<int32_t>(b_idx));
-
-        if (b_idx == 0) {
-            Serialization::SortBucketByTypeId(src_bucket, dst_bucket);
-        } else {
-            dst_bucket = src_bucket;
-        }
+        Serialization::SortBucketByTypeId(native_buckets[b_idx], sorted_frame->GetBucket(static_cast<int32_t>(b_idx)));
     }
 
     return sorted_frame;

sdk/src/vtx_writer/src/vtx/writer/formatters/protobuff_vtx_policy.cpp

@@ -16,14 +16,7 @@ std::unique_ptr<VTX::ProtobufVtxPolicy::FrameType> VTX::ProtobufVtxPolicy::FromN
     sorted_native.GetMutableBuckets().resize(native_buckets.size());
 
     for (size_t b_idx = 0; b_idx < native_buckets.size(); ++b_idx) {
-        const auto& src_bucket = native_buckets[b_idx];
-        auto& dst_bucket = sorted_native.GetBucket(static_cast<int>(b_idx));
-
-        if (b_idx == 0) {
-            Serialization::SortBucketByTypeId(src_bucket, dst_bucket);
-        } else {
-            dst_bucket = src_bucket;
-        }
+        Serialization::SortBucketByTypeId(native_buckets[b_idx], sorted_native.GetBucket(static_cast<int>(b_idx)));
     }
 
     auto proto = std::make_unique<cppvtx::Frame>();

tests/common/test_schema_registry.cpp

@@ -184,3 +184,92 @@ TEST(SchemaRegistry, ReloadReplacesBucketNames) {
     ASSERT_TRUE(schema.LoadFromRawString(raw));
     EXPECT_TRUE(schema.GetBucketNames().empty());
 }
+
+// ---------------------------------------------------------------------------
+// Array / Map pre-sizing (symmetric with scalar pre-sizing)
+// ---------------------------------------------------------------------------
+
+namespace {
+    // Player declares: 2 int32 scalars, 1 float array, 2 string arrays,
+    // 1 struct array (Inventory of Item) and 1 struct map (AmmoByWeapon of Ammo).
+    constexpr const char* kArrayMapSchema = R"({
+        "version": "1.0.0",
+        "buckets": ["entity"],
+        "property_mapping": [
+            { "struct": "Item", "values": [
+                {"name":"Id","typeId":"Int32","containerType":"None","keyId":"None","structType":"","meta":{"defaultValue":"0","fixedArrayDim":1}}
+            ]},
+            { "struct": "Ammo", "values": [
+                {"name":"Count","typeId":"Int32","containerType":"None","keyId":"None","structType":"","meta":{"defaultValue":"0","fixedArrayDim":1}}
+            ]},
+            { "struct": "Player", "values": [
+                {"name":"Score","typeId":"Int32","containerType":"None","keyId":"None","structType":"","meta":{"defaultValue":"0","fixedArrayDim":1}},
+                {"name":"Level","typeId":"Int32","containerType":"None","keyId":"None","structType":"","meta":{"defaultValue":"0","fixedArrayDim":1}},
+                {"name":"Cooldowns","typeId":"Float","containerType":"Array","keyId":"None","structType":"","meta":{"defaultValue":"","fixedArrayDim":0}},
+                {"name":"Tags","typeId":"String","containerType":"Array","keyId":"None","structType":"","meta":{"defaultValue":"","fixedArrayDim":0}},
+                {"name":"Names","typeId":"String","containerType":"Array","keyId":"None","structType":"","meta":{"defaultValue":"","fixedArrayDim":0}},
+                {"name":"Inventory","typeId":"Struct","containerType":"Array","keyId":"None","structType":"Item","meta":{"defaultValue":"","fixedArrayDim":0}},
+                {"name":"AmmoByWeapon","typeId":"Struct","containerType":"Map","keyId":"String","structType":"Ammo","meta":{"defaultValue":"","fixedArrayDim":1}}
+            ]}
+        ]
+    })";
+} // namespace
+
+TEST(SchemaRegistry, PreSizesArraysAndMapsFromSchema) {
+    VTX::SchemaRegistry schema;
+    ASSERT_TRUE(schema.LoadFromRawString(kArrayMapSchema));
+
+    const VTX::SchemaStruct* player = schema.GetStruct("Player");
+    ASSERT_NE(player, nullptr);
+
+    VTX::PropertyContainer c;
+    VTX::Helpers::PreparePropertyContainer(c, *player);
+
+    // Scalars: unchanged (two int32 fields).
+    EXPECT_EQ(c.int32_properties.size(), 2u);
+
+    // Arrays: one empty subarray per declared array field, grouped per type.
+    EXPECT_EQ(c.float_arrays.SubArrayCount(), 1u);
+    EXPECT_EQ(c.string_arrays.SubArrayCount(), 2u);
+    EXPECT_EQ(c.any_struct_arrays.SubArrayCount(), 1u);
+    EXPECT_TRUE(c.float_arrays.GetSubArray(0).empty());
+    EXPECT_TRUE(c.string_arrays.GetSubArray(1).empty());
+
+    // Array types with no declared fields stay untouched.
+    EXPECT_EQ(c.int32_arrays.SubArrayCount(), 0u);
+    EXPECT_EQ(c.vector_arrays.SubArrayCount(), 0u);
+
+    // Map: one empty, Struct-valued map slot.
+    ASSERT_EQ(c.map_properties.size(), 1u);
+    EXPECT_TRUE(c.map_properties[0].keys.empty());
+    EXPECT_TRUE(c.map_properties[0].values.empty());
+}
+
+TEST(SchemaRegistry, ArrayAndMapSizingLandInPropertyCache) {
+    VTX::SchemaRegistry schema;
+    ASSERT_TRUE(schema.LoadFromRawString(kArrayMapSchema));
+
+    const int32_t player_id = schema.GetStructTypeId("Player");
+    ASSERT_GE(player_id, 0);
+
+    const auto& sc = schema.GetPropertyCache().structs.at(player_id);
+    const auto string_idx = static_cast<size_t>(VTX::FieldType::String);
+    const auto float_idx = static_cast<size_t>(VTX::FieldType::Float);
+    ASSERT_GT(sc.array_max_indices.size(), string_idx);
+    EXPECT_EQ(sc.array_max_indices[string_idx], 2);
+    EXPECT_EQ(sc.array_max_indices[float_idx], 1);
+    EXPECT_EQ(sc.map_max_index, 1);
+}
+
+// A schema without array/map fields leaves those sizes empty/zero.
+TEST(SchemaRegistry, ScalarOnlySchemaHasNoArrayOrMapSizing) {
+    const std::string raw = std::string(R"({ "version": "1.0.0", )") + kTinyMapping + "}";
+    VTX::SchemaRegistry schema;
+    ASSERT_TRUE(schema.LoadFromRawString(raw));
+
+    const int32_t id = schema.GetStructTypeId("Tiny");
+    ASSERT_GE(id, 0);
+    const auto& sc = schema.GetPropertyCache().structs.at(id);
+    EXPECT_TRUE(sc.array_max_indices.empty());
+    EXPECT_EQ(sc.map_max_index, 0);
+}