[Data] Dictionary-encode schema_data in MCAP blocks, and document the format

[marwan116]

Description

A schema is a per-channel attribute in MCAP: the file stores each definition once in the data section, repeats it in the summary for lookup, and every channel that uses it refers to it by id. MCAPDatasource._message_to_dict copies the whole definition onto every row, so a block holds one value as many times as it has messages.

Real ROS 2 definitions are not small. In the recording used below, sensor_msgs/msg/Imu is 2,581 bytes and sensor_msgs/msg/CompressedImage is 1,505. On a high-rate topic with a small payload — IMU, odometry, transforms — that column dominates the block.

Fix. Dictionary-encode schema_data once per block, on the way out of _read_stream. Arrow keeps one copy of each distinct definition and gives every row an int32 index into it. The value a row reads back is byte-identical.

Measurements

A FAST-LIVO ROS 2 recording — 105 s, 24,456 messages: Livox IMU at ~200 Hz, Livox lidar, and two compressed camera streams. Before and after go through the same code path (read_mcap(...).materialize().size_bytes()), differing only in this change:

read	rows	before	after	smaller
one shard, all topics	286	11.95 MB	11.30 MB	5.5%
one shard, IMU only	247	741.0 KB	124.2 KB	83.2%
one shard, cameras only	26	4.95 MB	4.92 MB	0.5%
one shard, include_metadata=False	286	11.24 MB	11.24 MB	0.0%
full recording, IMU only	21,300	63.90 MB	9.00 MB	85.9%

The saving tracks payload size against schema size. It is large exactly where robotics pipelines read a high-rate, small-payload topic, and negligible where camera or lidar payloads outweigh the schema. Reading the recording whole gains 5.5%; reading only its IMU gains 85.9%, because 54.9 MB of that 63.9 MB was one 2,581-byte definition repeated 21,300 times.

include_metadata=False is byte-identical before and after, which is the column being absent rather than the encoding being skipped.

Scope and costs

Only schema_data is encoded. topic and schema_name repeat as well, but they are short enough that the saving is marginal, and callers sort and group by them — which Arrow cannot do on a dictionary-encoded column. Encoding them would trade a real capability for a rounding error.

Two costs, verified rather than assumed:

• sort_by("schema_data") now raises ArrowNotImplementedError. Narrow, but real.
• pa.concat_tables refuses an encoded block beside an unencoded one, so unioning a new read with a dataset cached before this change fails. This is the argument for doing it in one step rather than behind a flag — a mixed corpus of blocks is the state worth avoiding.

The helper leaves the column untouched when it is absent (include_metadata=False), already encoded, all-null (no schema on any message), or of a type Arrow declines to encode.

Also in this PR

The module docstring becomes a brief on the format: the record layout, what a message, channel and schema each are, and the three properties of MCAP that this module's behaviour follows from — the summary section sitting at the end of the file behind a footer offset, schemas being stored once per file, and chunks being the unit of compression. Each is load-bearing for code in the file rather than general background.

Related issues

Related to #65640 (file-level packed output), #65641 and #65787 (batch_size) — none of which touches the row schema.

Overlaps #65912 (in-memory size estimation) in two places, both small:

• Both edit the module docstring. This PR's version is a superset, so the resolution is to take it.
• [Data] Estimate MCAP in-memory size from sampled messages, not file size #65912's sampling path builds a block directly rather than through _read_stream, so once both are in it should also call _dictionary_encode_schema_data to keep its estimate faithful. Without that line the estimate overstates by roughly the saving above. I will add it in whichever of the two rebases second.

Tests

PYTHONPATH=python/ray/data/tests python -u -m pytest \
    python/ray/data/tests/datasource/test_mcap.py -v

20 passed, 1 failed. The failure is test_read_mcap_invalid_time_range, which fails identically on unmodified master — its pytest.raises pattern omits the values the error message interpolates, so it can never match. It is fixed in #65912; I have left it alone here to avoid a second conflict between the two branches. Baseline on this branch point is 17 passed, 1 failed.

Three new tests: the column is dictionary-encoded and every row reads back the original definition; the encoded block is smaller than the unencoded schema column alone would be; and include_metadata=False omits the column with the encoding a no-op.

Lint: black, ruff clean.

---

AI assistance (Claude) was used for this change. I reviewed every changed line and ran the tests locally.

[gemini-code-assist]

Code Review

This pull request introduces dictionary encoding for the schema_data column in the MCAP datasource to optimize memory usage when reading MCAP files with large schema definitions. It also adds comprehensive unit tests to verify the behavior. The reviewer suggested a performance optimization to reverse the order of combine_chunks() and dictionary_encode() on PyArrow columns to avoid the overhead of unifying dictionaries across multiple chunks.