File Format

File Format

The VTX file format is a self-describing binary format built for speed, streaming, and long-term data durability. This page is the wiki-level overview. For the bit-exact layout see docs/FILE_FORMAT.md in the repo.

File structure

A VTX file has four sections.

Header

Format versioning, schema versioning, and source metadata. A reader can identify what it is looking at and whether it has a compatible schema immediately, before parsing any frames.

The magic bytes identify the serialisation backend:

• VTXP for Protocol Buffers.
• VTXF for FlatBuffers.

VTX::OpenReplayFile auto-detects the backend from these magic bytes. Integrators do not need to know in advance which one was used.

Embedded schema

The complete property-to-name-and-type mapping for this file. The schema acts as a self-contained decoder. Property ID 1 maps to Health, property ID 2 maps to Position, and so on.

This decouples data durability from the software development lifecycle. A file recorded today can be read years from now without needing the original application code or external documentation. Every VTX file carries its own decoder ring.

Chunks

Actual frame data is segmented into compressed chunks. Each chunk contains multiple frames, serialised with the chosen binary format (Protobuf or FlatBuffers) and compressed with Zstd. Chunks are the unit of streaming and on-demand download.

Footer / time index

The timeline metadata is separated from the frame data. The footer maps game timestamps to file byte offsets.

When you scrub a timeline, the system binary-searches this footer to find the exact byte offset needed. That gives O(1) seeking: instant jumps to any point in time without sequential parsing.

Serialisation backends

The VTX SDK is not locked to a single binary serialisation format. The entire SDK (writer, reader, differ) supports both Protocol Buffers and FlatBuffers out of the box. You pick per file; the file announces the choice in its magic bytes.

The architecture is extensible. Any other binary format can be plugged in by implementing the serialisation interface (plus a matching diff implementation).

Protobuf vs. FlatBuffers

	Protocol Buffers	FlatBuffers
File size	Smaller (varint encoding, efficient sparse data)	Larger (vtable overhead for nested structures)
Diff performance	Standard	Faster diffs and data access (zero-copy reads)
Best for	Storage, network transmission, archival	Real-time playback where diff speed is critical
Language support	Broad (C++, Python, Go, Rust, Java, JS, ...)	Broad (C++, Python, Go, Rust, Java, JS, ...)
Schema evolution	Excellent (unknown fields safely ignored)	Good (vtable-based field access)

In practice: Protobuf produces smaller files, FlatBuffers is faster at diffing and data access. Storage-constrained or network-heavy applications benefit from Protobuf. Latency-sensitive playback applications may prefer FlatBuffers.

Both satisfy the core requirement of schema-defined, evolvable formats. Schemaless options (Cereal, Bits) were evaluated and rejected: the diffing system requires files to be self-describing.

Compression

Chunks are compressed with Zstd (Zstandard). Strong compression ratio with very fast decompression. This matters for:

• Reducing file sizes for storage and network transfer.
• Fast seeking. Only the target chunk needs decompressing, not the whole file.
• Operating on memory-constrained platforms (mobile, web).

Why not Unreal's FArchive?

Legacy replay tooling at Zenos used Unreal's FArchive serialisation, which tied replay files to a specific Unreal Engine version. Moving to standard binary formats (Protobuf / FlatBuffers):

• Removes the UE version dependency.
• Makes replay files readable outside the engine (external tools, web viewers, Python notebooks).
• Ensures a stable long-term format not subject to engine changes.
• Makes the data effectively engine-agnostic.
• Lets SDK users pick the serialisation format that best fits their use case.

Chunk size

Chunk sizes are tuned for the balance between:

• Compression efficiency. Larger chunks compress better.
• Seek responsiveness. Smaller chunks mean less data to decompress per seek.
• Memory budget, especially on constrained platforms.

See Performance, Cache-window sizing for guidance on the reader-side cache that pairs with chunk layout.

Streaming vs. storage

The same format works in two modes:

• Streaming: chunks are written and flushed live; downstream consumers (broadcast overlays, analytics pipelines) read them as they land. Low latency.
• Storage / replay: full matches persisted for later scrubbing, analysis, or training. Random access by frame or timestamp is O(1) via the footer.

A tool written once reads either.