ci(robotwin): run full 1-episode smoke eval instead of mocked tests

The robotwin-integration-test job now builds the full-install image
(SAPIEN, mplib, CuRobo, pytorch3d + assets) and runs a real 1-episode
lerobot-eval against beat_block_hammer, matching the libero/metaworld
CI pattern.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: pkooij

.github/workflows/benchmark_tests.yml

@@ -170,18 +170,12 @@ jobs:
             "
 
   # ── ROBOTWIN 2.0 ──────────────────────────────────────────────────────────
-  # RoboTwin 2.0 is NOT pip-installable (SAPIEN / CuRobo / mplib require a
-  # 20-minute source installation with specific NVIDIA driver versions).
-  # The image therefore installs only the base lerobot package and runs the
-  # fully-mocked unit-test suite, which catches:
-  #   - import-time regressions in lerobot.envs.robotwin
-  #   - RoboTwinEnvConfig registration under --env.type=robotwin
-  #   - Gymnasium wrapper, multi-task factory, and processor-step correctness
-  #
-  # For a full end-to-end eval on a NVIDIA GPU machine, follow the install guide
-  # at https://robotwin-platform.github.io/doc/usage/robotwin-install.html.
+  # Isolated image: full RoboTwin 2.0 stack — SAPIEN, mplib, CuRobo,
+  # pytorch3d, + simulation assets (~4 GB).
+  # Build takes ~20 min on first run; subsequent runs hit the layer cache.
+  # Requires an NVIDIA GPU runner with CUDA 12.1 drivers.
   robotwin-integration-test:
-    name: RoboTwin 2.0 — build image + mocked unit tests
+    name: RoboTwin 2.0 — build image + 1-episode eval
     runs-on:
       group: aws-g6-4xlarge-plus
     env:
@@ -198,6 +192,9 @@ jobs:
         with:
           cache-binary: false
 
+      # Build the full-install image: SAPIEN, mplib, CuRobo, pytorch3d +
+      # simulation assets (~4 GB). Layer cache lives in the runner's local
+      # Docker daemon — reused across re-runs on the same machine.
       - name: Build RoboTwin 2.0 benchmark image
         uses: docker/build-push-action@v6 # zizmor: ignore[unpinned-uses]
         with:
@@ -209,22 +206,21 @@ jobs:
           cache-from: type=local,src=/tmp/.buildx-cache-robotwin
           cache-to: type=local,dest=/tmp/.buildx-cache-robotwin,mode=max
 
-      # Run the fully-mocked test suite — no SAPIEN installation required.
-      # 19 tests verify the gymnasium wrapper, config registration, multi-task
-      # factory, error handling, and task list integrity.
-      - name: Run RoboTwin 2.0 unit tests (mocked, no SAPIEN)
+      - name: Run RoboTwin 2.0 smoke eval (1 episode)
         run: |
-          docker run --rm \
-            --shm-size=2g \
+          docker run --rm --gpus all \
+            --shm-size=4g \
+            -e HF_HOME=/tmp/hf \
+            -e HF_USER_TOKEN="${HF_USER_TOKEN}" \
             lerobot-benchmark-robotwin:ci \
             bash -c "
-              pytest tests/envs/test_robotwin.py -k 'not ProcessorStep' \
-                --tb=short -q
+              hf auth login --token \"\$HF_USER_TOKEN\" --add-to-git-credential 2>/dev/null || true
+              lerobot-eval \
+                --policy.path=pepijn223/smolvla_robotwin \
+                --env.type=robotwin \
+                --env.task=beat_block_hammer \
+                --eval.batch_size=1 \
+                --eval.n_episodes=1 \
+                --eval.use_async_envs=false \
+                --policy.device=cuda
             "
-
-      # Verify that RoboTwinEnvConfig is correctly registered and importable
-      # as --env.type=robotwin, independent of the SAPIEN installation.
-      - name: Verify RoboTwin config registration
-        run: |
-          docker run --rm lerobot-benchmark-robotwin:ci \
-            python -c "from lerobot.envs.configs import EnvConfig; choices = EnvConfig.get_known_choices(); assert 'robotwin' in choices, f'robotwin not in registered envs: {list(choices)}'; cfg = choices['robotwin'](); assert cfg.task == 'beat_block_hammer'; assert len(cfg.features) >= 5; print('RoboTwinEnvConfig registered OK:', cfg.task, '| features:', sorted(cfg.features))"