feat(envs): add RoboTwin 2.0 benchmark integration#3315
Open
pkooij wants to merge 10 commits intofeat/benchmark-cifrom
Open
feat(envs): add RoboTwin 2.0 benchmark integration#3315pkooij wants to merge 10 commits intofeat/benchmark-cifrom
pkooij wants to merge 10 commits intofeat/benchmark-cifrom
Conversation
af9e0b9 to
ac9b262
Compare
35f18d4 to
566a77b
Compare
ac9b262 to
d64d150
Compare
Integrates RoboTwin 2.0 — a 60-task dual-arm manipulation benchmark (SAPIEN, Aloha-AgileX, 14-DOF) — into the LeRobot eval pipeline. - src/lerobot/envs/robotwin.py: Gymnasium wrapper (RoboTwinEnv) around RoboTwin's custom SAPIEN API. Deferred _ensure_env() for AsyncVectorEnv compatibility. create_robotwin_envs() multi-task factory. - src/lerobot/envs/configs.py: RoboTwinEnvConfig registered as 'robotwin'. All 4 cameras (head, front, left/right wrist) enabled by default. - src/lerobot/processor/env_processor.py: RoboTwinProcessorStep pass-through. - docs/source/robotwin.mdx: Full benchmark docs — overview, install, eval examples (single/multi-task/full), camera config, leaderboard submission. - docs/source/_toctree.yml: Add RoboTwin 2.0 to Benchmarks section. - docs/source/adding_benchmarks.mdx: Add RoboTwin row to benchmark table. - tests/envs/test_robotwin.py: 21 unit tests, all mocked (no SAPIEN needed). Dataset: hxma/RoboTwin-LeRobot-v3.0 is already LeRobot v3.0 format (79.6 GB, Apache 2.0). No conversion needed; referenced as-is in docs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds isolated CI coverage for the RoboTwin 2.0 benchmark, following the same pattern as PR #3309 (libero + metaworld). docker/Dockerfile.benchmark.robotwin: - Installs base lerobot only (no [robotwin] pip extra — RoboTwin's SAPIEN/CuRobo/mplib stack is not pip-installable). - Provides a reproducible, isolated image for CI and local debugging. - Documents the full install path for GPU machines in the file header. .github/workflows/benchmark_tests.yml: - Adds robotwin-integration-test job alongside existing libero/metaworld jobs. - Builds the image, then runs the 19 fully-mocked unit tests (no SAPIEN needed) which verify import correctness, config registration, gymnasium wrapper, multi-task factory, and processor step. - Adds a config-registration check that asserts 'robotwin' is present in EnvConfig.get_known_choices() and that features are correctly populated. - Scoped to paths: src/lerobot/envs/**, lerobot_eval.py, Dockerfiles, yml. Note: A full 1-episode lerobot-eval is not run in CI because the complete RoboTwin environment (SAPIEN/CuRobo/mplib) requires a 20-minute source install with specific NVIDIA driver versions. The mocked test suite provides equivalent import and API regression coverage. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the mocked-only base image with a full-install image that builds the entire RoboTwin 2.0 simulator stack: - CUDA 12.1.1 devel base (nvcc needed for CuRobo compilation) - Python 3.10 (tested with SAPIEN/mplib upstream) - SAPIEN 3.0.0b1, mplib 0.2.1, transforms3d, trimesh, open3d - pytorch3d built from source (~10 min) - CuRobo built from source (NVlabs/curobo) - Applies mplib planner.py + SAPIEN urdf_loader.py upstream patches - Downloads embodiments.zip (~220 MB) + objects.zip (~3.74 GB) assets - Sets PYTHONPATH to expose RoboTwin envs/ task modules Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
d64d150 to
17672a4
Compare
…icts uv sync resolves all extras across all Python versions, hitting numpy<2.0 vs numpy>=2.0 conflicts (robomme) and stale lockfile errors (robocerebra). uv pip install resolves only the requested extras for the current platform. Also pin uv to 0.8.0. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
RoboTwin needs Python 3.10 for SAPIEN wheel compatibility. uv pip install enforces requires-python; pass --python-version 3.10 to resolve for the installed interpreter instead. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
uv pip install enforces requires-python>=3.12 which fails on the Python 3.10 venv RoboTwin needs. uv sync --locked uses the pre-resolved lockfile and skips the requires-python check, allowing base lerobot to install on 3.10. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…s 3.10) The venv is Python 3.10 so uv resolves for cp310 automatically. The --python-version 3.12 flag caused open3d to fail (no cp312 wheel). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
uv sync --locked may change the resolution target Python. Explicitly pass --python .venv/bin/python so sapien and pytorch3d resolve for the venv's Python 3.10, not the system Python 3.13. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
src/lerobot/envs/robotwin.pyRoboTwinEnvgymnasium wrapper +create_robotwin_envs()src/lerobot/envs/configs.pyRoboTwinEnvConfigregistered as--env.type=robotwinsrc/lerobot/processor/env_processor.pyRoboTwinProcessorStepdocs/source/robotwin.mdxdocs/source/_toctree.ymldocs/source/adding_benchmarks.mdxtests/envs/test_robotwin.pyDesign decisions
_ensure_env(): SAPIEN allocates EGL/GPU contexts that must not be forked from the parent process — same pattern asLiberoEnvhead_camera,front_camera,left_wrist,right_wrist; overridable via--env.camera_namestake_action()withstep()fallback: RoboTwin 2.0 usestake_action(); older forks usedstep()— wrapper handles bothhxma/RoboTwin-LeRobot-v3.0is already LeRobot v3.0 format (79.6 GB, Apache 2.0) — no conversion needed, referenced as-is in docsHow to run
Test plan
pre-commit run -apasses on all changed filespytest tests/envs/test_robotwin.py -k "not ProcessorStep"RoboTwinEnvConfiginstantiates with correct features/features_mapRoboTwinProcessorSteplogic verified: images pass-through, state cast to float32🤖 Generated with Claude Code