Open
Conversation
pkooij
force-pushed
the
feat/robomme-benchmark
branch
from
aaf286c to
208234d
Compare
pkooij
force-pushed
the
feat/benchmark-ci
branch
from
e89e6d9 to
927118e
Compare
pkooij
force-pushed
the
feat/robomme-benchmark
branch
from
208234d to
f654d7e
Compare
Adds RoboMME — a 16-task memory-augmented manipulation benchmark (ManiSkill/SAPIEN) — to lerobot's eval system, rebased on the feat/async-vector-env branch and using the create_envs() dispatch pattern. - src/lerobot/envs/robomme.py: RoboMMEGymEnv wrapper + create_robomme_envs() - src/lerobot/envs/lazy_vec_env.py: LazyVectorEnv for deferred env init - src/lerobot/envs/configs.py: RoboMMEEnv config with create_envs() - pyproject.toml: robomme extras (Linux only, git dep) - docker/Dockerfile.eval-robomme: gymnasium/numpy pin overrides for ManiSkill - tests/test_robomme_env.py: 12 unit tests (all passing, mocked sim) - docs/source/robomme.mdx: install instructions, task table, dataset info - docs/source/_toctree.yml: add robomme to Benchmarks section Dataset pepijn223/robomme_data_lerobot is already in LeRobot v3.0 format (1,600 episodes, 768K frames). No conversion needed. Initial integration prototyped in feat/robomme-integration. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…val) Follows the pattern from feat/async-vector-env (PR #3309) — each benchmark gets an isolated Docker image so its dependency tree cannot conflict with other benchmarks. - docker/Dockerfile.benchmark.robomme: CUDA image installing lerobot[robomme] only; adds Vulkan ICD loader for ManiSkill/SAPIEN rendering; applies gymnasium==0.29.1 + numpy==1.26.4 overrides post-install (mani-skill pins). Uses `uv sync` without --locked because robomme is a git dep not in uv.lock. - .github/workflows/benchmark_tests.yml: full workflow with libero + metaworld jobs (from PR #3309) plus new robomme-integration-test job; triggers on changes to src/lerobot/envs/**, lerobot_eval.py, Dockerfiles. TODO: update --policy.path in the robomme eval step once a robomme-trained policy is published on the Hub. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pkooij
force-pushed
the
feat/robomme-benchmark
branch
from
f654d7e to
489f681
Compare
…icts uv sync resolves all extras across all Python versions, hitting numpy<2.0 vs numpy>=2.0 conflicts (robomme) and stale lockfile errors (robocerebra). uv pip install resolves only the requested extras for the current platform. Also pin uv to 0.8.0. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…, not cross-extra) mani-skill hard-pins gymnasium==0.29.1 which directly conflicts with lerobot's gymnasium>=1.1.1. This isn't a cross-extra resolution issue — it's a genuine incompatibility. Fix: install lerobot base first, then force-install robomme with --override to downgrade gymnasium and numpy. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Title
Short, imperative summary (e.g., "fix(robots): handle None in sensor parser"). See CONTRIBUTING.md for PR conventions.
Type / Scope
Summary / Motivation
Related issues
What changed
How was this tested (or how to run locally)
Example:
Ran the relevant tests:
Reproduce with a quick example or CLI (if applicable):
Checklist (required before merge)
pre-commit run -a)pytest)Reviewer notes