Skip to content

[Data] Install mcap in CI so the MCAP datasource tests actually run - #65914

Open
marwan116 wants to merge 6 commits into
masterfrom
marwan/mcap-ci-deps
Open

[Data] Install mcap in CI so the MCAP datasource tests actually run#65914
marwan116 wants to merge 6 commits into
masterfrom
marwan/mcap-ci-deps

Conversation

@marwan116

@marwan116 marwan116 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Description

python/ray/data/tests/datasource/test_mcap.py guards its whole module with
pytestmark = skipif(not MCAP_AVAILABLE), and mcap appears in no requirements file
anywhere in the repo. //python/ray/data:test_mcap has therefore reported success
since the datasource landed in #55716 without executing a single one of its 18 tests.

This PR adds mcap to python/requirements/ml/data-test-requirements.txt — the file
consumed by the Data CI image (ci/env/install-dependencies.sh:303, ci/ci.sh:60) and
by ci/raydepsets/configs/ci_data.depsets.yaml:19 — and regenerates the locks that
depend on it.

Related issues

Related to #65912, which fixes estimate_inmemory_data_size for this datasource. The
two PRs are independent: both carry the identical one-line fix to
test_read_mcap_invalid_time_range, deliberately, so neither blocks the other. Being
byte-identical, whichever lands second should rebase without a conflict.

Related to #61222 (audit and refactor file-based datasource tests), which covers
adjacent ground but not this.

Additional information

What turning the tests on exposed

Three defects were sitting behind that skip. Each is fixed in its own commit here,
because without them the target goes red on its first real run:

  1. test_read_mcap_invalid_time_range can never pass. It searches for
    "start_time must be less than end_time"; the code raises
    "start_time (2000) must be less than end_time (1000)". Fails on unmodified master.

  2. _should_include_message and _message_to_dict annotate schema as a bare
    Schema,
    but McapReader.iter_messages() yields
    Tuple[Optional[Schema], Channel, Message] — a channel may declare no schema. Both
    bodies already guard with if schema else None, so this is an annotation fix only.
    It was invisible while mcap was absent: pyrefly could not resolve mcap.reader,
    treated the return as Any, and reported nothing. With the package installed its
    py.typed annotations apply and ci/lint/pyrefly-check.sh fails on the two
    pre-existing call sites in _read_stream.

  3. python/requirements_compiled.txt was missing mcap. That file is a separate
    lock from python/deplocks/, compiled by compile_pip_dependencies in ci/ci.sh
    from an input list that includes data-test-requirements.txt.

The dependency itself costs the image nothing

The regenerated locks introduce exactly one package:

mcap==1.4.0 \
    --hash=sha256:0528e2f86a61bfec73779e0628e6cf27af83d01d89e20b27d5ec9f0b556a63ac \
    --hash=sha256:0b48b1cc951b8d5aabd2599e60d410bae4f1be1819094f54117b7cbf6b3ee2e9
    # via -r python/requirements/ml/data-test-requirements.txt

mcap's own dependencies, lz4 and zstandard, are already present via
clickhouse-connect, dask, ray and pyiceberg. Every other changed line across the
ten python/deplocks/ci/*.lock files and requirements_compiled.txt is a # via
comment recording mcap as a new consumer.

How the locks were produced

python/deplocks/*.lock with the repo's own tooling, using the same command the
raydepsets: compile all dependencies job runs:

bazelisk run //ci/raydepsets:raydepsets -- build --all-configs
bazelisk run //ci/raydepsets:raydepsets -- build --all-configs --check
# Lock files are up to date.

python/requirements_compiled.txt could not be regenerated locally:
compile_pip_dependencies returns early on arm64 (ci/ci.sh:20-27). Its four added
lines were reconstructed from the dependencies job's own diff and land at the exact
line numbers it reported — mcap at 1093–1094, zstandard's # via entry at 2712.

Worth knowing for anyone adding a dependency here: the two lock systems are coupled
through a symlink. python/requirements_compiled_py3.10py3.13 are all symlinks to
requirements_compiled.txt (only py3.14 is a real file), and
ci/raydepsets/pre_hooks/remove-compiled-headers.sh copies
python/requirements_compiled_py3.13.txt into /tmp/ray-deps/, where it becomes the
constraint the data depsets compile against. So adding a package to
requirements_compiled.txt changes the # via provenance recorded in
python/deplocks/ci/*.lock — uv then lists the package as coming from both the
constraint and data-test-requirements.txt. Regenerating one lock system without the
other leaves the tree inconsistent, which is why this PR has a separate commit for it.

Tests

With mcap installed on unmodified master, the previously-skipped module runs and one
test fails — what the target would have been reporting all along:

1 failed, 17 passed in 15.50s
FAILED test_read_mcap_invalid_time_range
E   AssertionError: Regex pattern did not match.
E    Regex: 'start_time must be less than end_time'
E    Input: 'start_time (2000) must be less than end_time (1000)'

With the fixes in this PR:

PYTHONPATH=python/ray/data/tests python -u -m pytest \
    python/ray/data/tests/datasource/test_mcap.py -v
18 passed in 10.95s

pyrefly, run exactly as ci/lint/pyrefly-check.sh does, reports no errors in
mcap_datasource.py. Lint hooks black, ruff, pydoclint, docstyle, check-ast,
check-import-order, trailing-whitespace and end-of-file-fixer all pass on the
changed files.

size = "small" is left as-is: the 18 tests take ~11s, and test_json is small with
58 tests, so the 60-second budget has ample headroom.


AI assistance (Claude) was used for this change. I reviewed every changed line and ran
the commands above locally.

`python/ray/data/tests/datasource/test_mcap.py` guards its whole module
with `pytestmark = skipif(not MCAP_AVAILABLE)`, and mcap appears in no
requirements file in the repo. The //python/ray/data:test_mcap target has
therefore reported success since the datasource landed in #55716 without
executing a single one of its 18 tests.

Add mcap to python/requirements/ml/data-test-requirements.txt, which feeds
the Data CI image (ci/env/install-dependencies.sh, ci/ci.sh) and the
dependency sets in ci/raydepsets/configs/ci_data.depsets.yaml, and
regenerate the affected locks with

    bazelisk run //ci/raydepsets:raydepsets -- \
        build ci/raydepsets/configs/ci_data.depsets.yaml

The regenerated locks add exactly one package. mcap's own dependencies,
lz4 and zstandard, are already present via clickhouse-connect, dask, ray
and pyiceberg, so every other changed line is a `# via` comment.

Enabling the tests also exposes test_read_mcap_invalid_time_range, whose
assertion can never match: it searches for
"start_time must be less than end_time" while the code raises
"start_time (2000) must be less than end_time (1000)". Relax the pattern
to what the code actually raises. Without it the target would go red on
its first real run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
@marwan116
marwan116 requested a review from a team as a code owner September 4, 2026 03:37

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds the mcap dependency to the ML data test requirements and updates the corresponding CI lockfiles. This ensures that the MCAP datasource tests in python/ray/data/tests/datasource/test_mcap.py are actually executed rather than being silently skipped. Additionally, a minor assertion adjustment was made in test_mcap.py to match a more general error message. There are no review comments, and I have no feedback to provide.

@marwan116 marwan116 added the go add ONLY when ready to merge, run all tests label Sep 4, 2026
@marwan116 marwan116 self-assigned this Sep 4, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 2f6912f. Configure here.

Comment thread python/deplocks/ci/relaxed_data-ci_depset_py3.12.lock
`data-pyarrow-latest-ci_depset_py3.12.lock` and
`relaxed_data-ci_depset_py3.12.lock` pinned mcap==1.3.0 while the other
eight locks pinned 1.4.0, and the relaxed lock paired that 1.3.0 pin with
1.4.0 hashes, so a --require-hashes install from it would fail.

Both derive from data-base-ci_depset_py3.12.lock. They were produced from
a momentarily incorrect copy of that base: `raydepsets build --check`
snapshots the existing outputs and then runs a full build that overwrites
the real lock files before comparing (ci/raydepsets/cli.py:116-119), so
running it against a bad input writes that input's results to disk. The
relax step rewrites the version constraint without re-resolving hashes,
which is why one file ended up internally inconsistent.

Regenerated with a clean base. All ten locks now pin mcap==1.4.0 and
carry a single, identical hash pair.

Reported by Cursor Bugbot on #65914.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
@ray-gardener ray-gardener Bot added the data Ray Data-related issues label Sep 4, 2026
marwan116 and others added 2 commits September 4, 2026 07:33
`McapReader.iter_messages()` is typed
`Iterator[Tuple[Optional[Schema], Channel, Message]]` -- a channel may
declare no schema -- but `_should_include_message` and `_message_to_dict`
annotated that parameter as a bare `Schema`. Both bodies already guard
with `if schema else None`, so this is an annotation fix, not a behaviour
change.

The mismatch was invisible until this PR. mcap was in no requirements
file, so pyrefly could not resolve `mcap.reader`, treated its return as
`Any`, and reported nothing. Installing mcap gives pyrefly the package's
`py.typed` annotations, and `ci/lint/pyrefly-check.sh` then fails on the
two pre-existing call sites in `_read_stream`.

Reproduced and verified locally with the same command CI runs:

    pip install pyrefly==0.51.0
    find python/ray/data -name '*.py' \
      | grep -vFf ci/lint/pyrefly-excluded-files.txt | xargs pyrefly check

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
`python/requirements/ml/data-test-requirements.txt` is one of the inputs
to `compile_pip_dependencies` in ci/ci.sh, so adding mcap there also
changes python/requirements_compiled.txt. That file is a separate lock
from python/deplocks/, which the earlier commits regenerated, and the
`dependencies` CI job regenerates it and diffs it against the committed
copy.

mcap's dependencies lz4 and zstandard are already pinned here, so the
only changes are mcap's own entry and two `# via` lines recording it as a
new consumer.

`compile_pip_dependencies` returns early on arm64 (ci/ci.sh:20-27), so
this cannot be regenerated on an Apple Silicon checkout. The insertions
were reconstructed from the job's own diff and land at the exact line
numbers it reported -- mcap at 1093-1094 and zstandard's entry at 2712 --
so the result is byte-identical to what the job produced.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
@marwan116
marwan116 requested a review from a team as a code owner September 4, 2026 15:52
marwan116 added a commit that referenced this pull request Sep 4, 2026
…ly see

pyrefly.toml lists "mcap.*" under ignore-missing-imports, and the comment at
the foot of that file explains the consequence: ignore-missing-imports covers
modules that cannot be found, not installed ones. mcap is in no requirements
file, so the ray/data pyrefly job resolves every mcap type as Any and reports
nothing. Installing mcap and running the same check surfaces 31 errors across
these two files -- 13 in the datasource, 18 in its tests.

Four classes, all real:

- iter_messages yields Optional[Schema]: a channel need not declare one.
  _should_include_message and _message_to_dict were typed Schema, at four call
  sites now that sampling calls them too.
- summary.statistics is Optional. _estimate_file_inmemory_size narrows it, but
  the helpers it calls take the Summary and reach through to .statistics
  themselves, which pyrefly cannot follow across a call. Narrow in each.
- fill_column is defined on ArrowBlockAccessor and PandasBlockAccessor, not on
  the BlockAccessor base. Carry the include_paths column in the row dict
  instead, which needs no downstream fill and is what the read path's own
  conversion would do.
- _file_sizes() returns List[float], so _sample_files receives
  list[tuple[str, float]] against a list[tuple[str, int]] annotation.

The tests compared Optional[int] estimates directly; they now go through one
narrowing helper rather than eighteen scattered assertions.

pyrefly reports 0 errors with mcap installed. Estimates are unchanged:
no filter, topic-filtered, time-windowed and include_paths=True all still land
at 1.00x of measured on a 21.8 MB episode, and the two real ROS 2 corpora are
byte-identical to before.

The Optional[Schema] class overlaps #65914, which changes the same two
signatures. Fixing it here lets that PR stay confined to CI plumbing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
marwan116 and others added 2 commits September 4, 2026 09:19
`python/requirements_compiled_py3.{10,11,12,13}.txt` are symlinks to
`python/requirements_compiled.txt`, and the `remove-compiled-headers`
pre-hook copies that file to `/tmp/ray-deps/`, where the data depsets
consume it as a constraint. Adding mcap to requirements_compiled.txt
therefore made mcap a constraint as well as a requirement, and uv records
both sources in the package's `# via` provenance.

The earlier regeneration ran before requirements_compiled.txt was
edited, so it recorded only the requirement. `raydepsets build
--all-configs --check`, which is what CI runs, caught the difference.

Provenance comments only -- the pin and hashes are unchanged in all five
files.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
The file comments an entry only when a version pin or Python marker needs
justifying -- zarr, numcodecs, lerobot, torchcodec. The other 38 entries,
including comparable optional test deps like fastavro and datasketches,
are bare. mcap is bare and unpinned, so it takes no comment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data Ray Data-related issues go add ONLY when ready to merge, run all tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant