[Data] Install mcap in CI so the MCAP datasource tests actually run - #65914
[Data] Install mcap in CI so the MCAP datasource tests actually run#65914marwan116 wants to merge 6 commits into
Conversation
`python/ray/data/tests/datasource/test_mcap.py` guards its whole module with `pytestmark = skipif(not MCAP_AVAILABLE)`, and mcap appears in no requirements file in the repo. The //python/ray/data:test_mcap target has therefore reported success since the datasource landed in #55716 without executing a single one of its 18 tests. Add mcap to python/requirements/ml/data-test-requirements.txt, which feeds the Data CI image (ci/env/install-dependencies.sh, ci/ci.sh) and the dependency sets in ci/raydepsets/configs/ci_data.depsets.yaml, and regenerate the affected locks with bazelisk run //ci/raydepsets:raydepsets -- \ build ci/raydepsets/configs/ci_data.depsets.yaml The regenerated locks add exactly one package. mcap's own dependencies, lz4 and zstandard, are already present via clickhouse-connect, dask, ray and pyiceberg, so every other changed line is a `# via` comment. Enabling the tests also exposes test_read_mcap_invalid_time_range, whose assertion can never match: it searches for "start_time must be less than end_time" while the code raises "start_time (2000) must be less than end_time (1000)". Relax the pattern to what the code actually raises. Without it the target would go red on its first real run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
There was a problem hiding this comment.
Code Review
This pull request adds the mcap dependency to the ML data test requirements and updates the corresponding CI lockfiles. This ensures that the MCAP datasource tests in python/ray/data/tests/datasource/test_mcap.py are actually executed rather than being silently skipped. Additionally, a minor assertion adjustment was made in test_mcap.py to match a more general error message. There are no review comments, and I have no feedback to provide.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 2f6912f. Configure here.
`data-pyarrow-latest-ci_depset_py3.12.lock` and `relaxed_data-ci_depset_py3.12.lock` pinned mcap==1.3.0 while the other eight locks pinned 1.4.0, and the relaxed lock paired that 1.3.0 pin with 1.4.0 hashes, so a --require-hashes install from it would fail. Both derive from data-base-ci_depset_py3.12.lock. They were produced from a momentarily incorrect copy of that base: `raydepsets build --check` snapshots the existing outputs and then runs a full build that overwrites the real lock files before comparing (ci/raydepsets/cli.py:116-119), so running it against a bad input writes that input's results to disk. The relax step rewrites the version constraint without re-resolving hashes, which is why one file ended up internally inconsistent. Regenerated with a clean base. All ten locks now pin mcap==1.4.0 and carry a single, identical hash pair. Reported by Cursor Bugbot on #65914. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
`McapReader.iter_messages()` is typed
`Iterator[Tuple[Optional[Schema], Channel, Message]]` -- a channel may
declare no schema -- but `_should_include_message` and `_message_to_dict`
annotated that parameter as a bare `Schema`. Both bodies already guard
with `if schema else None`, so this is an annotation fix, not a behaviour
change.
The mismatch was invisible until this PR. mcap was in no requirements
file, so pyrefly could not resolve `mcap.reader`, treated its return as
`Any`, and reported nothing. Installing mcap gives pyrefly the package's
`py.typed` annotations, and `ci/lint/pyrefly-check.sh` then fails on the
two pre-existing call sites in `_read_stream`.
Reproduced and verified locally with the same command CI runs:
pip install pyrefly==0.51.0
find python/ray/data -name '*.py' \
| grep -vFf ci/lint/pyrefly-excluded-files.txt | xargs pyrefly check
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
`python/requirements/ml/data-test-requirements.txt` is one of the inputs to `compile_pip_dependencies` in ci/ci.sh, so adding mcap there also changes python/requirements_compiled.txt. That file is a separate lock from python/deplocks/, which the earlier commits regenerated, and the `dependencies` CI job regenerates it and diffs it against the committed copy. mcap's dependencies lz4 and zstandard are already pinned here, so the only changes are mcap's own entry and two `# via` lines recording it as a new consumer. `compile_pip_dependencies` returns early on arm64 (ci/ci.sh:20-27), so this cannot be regenerated on an Apple Silicon checkout. The insertions were reconstructed from the job's own diff and land at the exact line numbers it reported -- mcap at 1093-1094 and zstandard's entry at 2712 -- so the result is byte-identical to what the job produced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
…ly see pyrefly.toml lists "mcap.*" under ignore-missing-imports, and the comment at the foot of that file explains the consequence: ignore-missing-imports covers modules that cannot be found, not installed ones. mcap is in no requirements file, so the ray/data pyrefly job resolves every mcap type as Any and reports nothing. Installing mcap and running the same check surfaces 31 errors across these two files -- 13 in the datasource, 18 in its tests. Four classes, all real: - iter_messages yields Optional[Schema]: a channel need not declare one. _should_include_message and _message_to_dict were typed Schema, at four call sites now that sampling calls them too. - summary.statistics is Optional. _estimate_file_inmemory_size narrows it, but the helpers it calls take the Summary and reach through to .statistics themselves, which pyrefly cannot follow across a call. Narrow in each. - fill_column is defined on ArrowBlockAccessor and PandasBlockAccessor, not on the BlockAccessor base. Carry the include_paths column in the row dict instead, which needs no downstream fill and is what the read path's own conversion would do. - _file_sizes() returns List[float], so _sample_files receives list[tuple[str, float]] against a list[tuple[str, int]] annotation. The tests compared Optional[int] estimates directly; they now go through one narrowing helper rather than eighteen scattered assertions. pyrefly reports 0 errors with mcap installed. Estimates are unchanged: no filter, topic-filtered, time-windowed and include_paths=True all still land at 1.00x of measured on a 21.8 MB episode, and the two real ROS 2 corpora are byte-identical to before. The Optional[Schema] class overlaps #65914, which changes the same two signatures. Fixing it here lets that PR stay confined to CI plumbing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
`python/requirements_compiled_py3.{10,11,12,13}.txt` are symlinks to
`python/requirements_compiled.txt`, and the `remove-compiled-headers`
pre-hook copies that file to `/tmp/ray-deps/`, where the data depsets
consume it as a constraint. Adding mcap to requirements_compiled.txt
therefore made mcap a constraint as well as a requirement, and uv records
both sources in the package's `# via` provenance.
The earlier regeneration ran before requirements_compiled.txt was
edited, so it recorded only the requirement. `raydepsets build
--all-configs --check`, which is what CI runs, caught the difference.
Provenance comments only -- the pin and hashes are unchanged in all five
files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
The file comments an entry only when a version pin or Python marker needs justifying -- zarr, numcodecs, lerobot, torchcodec. The other 38 entries, including comparable optional test deps like fastavro and datasketches, are bare. mcap is bare and unpinned, so it takes no comment. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>

Description
python/ray/data/tests/datasource/test_mcap.pyguards its whole module withpytestmark = skipif(not MCAP_AVAILABLE), andmcapappears in no requirements fileanywhere in the repo.
//python/ray/data:test_mcaphas therefore reported successsince the datasource landed in #55716 without executing a single one of its 18 tests.
This PR adds
mcaptopython/requirements/ml/data-test-requirements.txt— the fileconsumed by the Data CI image (
ci/env/install-dependencies.sh:303,ci/ci.sh:60) andby
ci/raydepsets/configs/ci_data.depsets.yaml:19— and regenerates the locks thatdepend on it.
Related issues
Related to #65912, which fixes
estimate_inmemory_data_sizefor this datasource. Thetwo PRs are independent: both carry the identical one-line fix to
test_read_mcap_invalid_time_range, deliberately, so neither blocks the other. Beingbyte-identical, whichever lands second should rebase without a conflict.
Related to #61222 (audit and refactor file-based datasource tests), which covers
adjacent ground but not this.
Additional information
What turning the tests on exposed
Three defects were sitting behind that skip. Each is fixed in its own commit here,
because without them the target goes red on its first real run:
test_read_mcap_invalid_time_rangecan never pass. It searches for"start_time must be less than end_time"; the code raises"start_time (2000) must be less than end_time (1000)". Fails on unmodified master._should_include_messageand_message_to_dictannotateschemaas a bareSchema, butMcapReader.iter_messages()yieldsTuple[Optional[Schema], Channel, Message]— a channel may declare no schema. Bothbodies already guard with
if schema else None, so this is an annotation fix only.It was invisible while
mcapwas absent: pyrefly could not resolvemcap.reader,treated the return as
Any, and reported nothing. With the package installed itspy.typedannotations apply andci/lint/pyrefly-check.shfails on the twopre-existing call sites in
_read_stream.python/requirements_compiled.txtwas missingmcap. That file is a separatelock from
python/deplocks/, compiled bycompile_pip_dependenciesinci/ci.shfrom an input list that includes
data-test-requirements.txt.The dependency itself costs the image nothing
The regenerated locks introduce exactly one package:
mcap's own dependencies,
lz4andzstandard, are already present viaclickhouse-connect,dask,rayandpyiceberg. Every other changed line across theten
python/deplocks/ci/*.lockfiles andrequirements_compiled.txtis a# viacomment recording mcap as a new consumer.
How the locks were produced
python/deplocks/*.lockwith the repo's own tooling, using the same command theraydepsets: compile all dependenciesjob runs:python/requirements_compiled.txtcould not be regenerated locally:compile_pip_dependenciesreturns early on arm64 (ci/ci.sh:20-27). Its four addedlines were reconstructed from the
dependenciesjob's own diff and land at the exactline numbers it reported — mcap at 1093–1094, zstandard's
# viaentry at 2712.Worth knowing for anyone adding a dependency here: the two lock systems are coupled
through a symlink.
python/requirements_compiled_py3.10…py3.13are all symlinks torequirements_compiled.txt(onlypy3.14is a real file), andci/raydepsets/pre_hooks/remove-compiled-headers.shcopiespython/requirements_compiled_py3.13.txtinto/tmp/ray-deps/, where it becomes theconstraint the data depsets compile against. So adding a package to
requirements_compiled.txtchanges the# viaprovenance recorded inpython/deplocks/ci/*.lock— uv then lists the package as coming from both theconstraint and
data-test-requirements.txt. Regenerating one lock system without theother leaves the tree inconsistent, which is why this PR has a separate commit for it.
Tests
With
mcapinstalled on unmodified master, the previously-skipped module runs and onetest fails — what the target would have been reporting all along:
With the fixes in this PR:
pyrefly, run exactly as
ci/lint/pyrefly-check.shdoes, reports no errors inmcap_datasource.py. Lint hooksblack,ruff,pydoclint,docstyle,check-ast,check-import-order,trailing-whitespaceandend-of-file-fixerall pass on thechanged files.
size = "small"is left as-is: the 18 tests take ~11s, andtest_jsonissmallwith58 tests, so the 60-second budget has ample headroom.
AI assistance (Claude) was used for this change. I reviewed every changed line and ran
the commands above locally.