Skip to content

feat(benchmarks): add LIBERO training benchmark pipeline#3337

Open
pkooij wants to merge 1 commit intomainfrom
feat/libero-benchmark
Open

feat(benchmarks): add LIBERO training benchmark pipeline#3337
pkooij wants to merge 1 commit intomainfrom
feat/libero-benchmark

Conversation

@pkooij
Copy link
Copy Markdown
Member

@pkooij pkooij commented Apr 9, 2026

Summary

  • Add a complete benchmark pipeline to train and evaluate all 9 LeRobot policies (pi0, pi0_fast, pi05, groot, act, diffusion, smolvla, xvla, multi_task_dit) on LIBERO
  • Generates SLURM sbatch scripts for cluster submission (train + eval + GPU monitoring per policy)
  • Publishes results as a HuggingFace leaderboard dataset with full training metadata (steps, BS, LR, GPU memory, training time, eval success rates, etc.)
  • Handles pi0_fast tokenizer pre-training as a dependency job

Test plan

  • Dry-run generation of all 9 policy scripts verified locally
  • Run single-policy (ACT) on cluster to validate end-to-end flow
  • Run collect_results.py on completed outputs
  • Verify HuggingFace dataset push with correct schema

🤖 Generated with Claude Code

@github-actions github-actions bot added documentation Improvements or fixes to the project’s docs CI Issues related to the continuous integration pipeline github_actions labels Apr 9, 2026
@pkooij @claude
feat(benchmarks): add LIBERO training benchmark pipeline
fd00e38 
Single-script benchmark that trains and evaluates all 9 LeRobot policies
on LIBERO. Each SLURM job self-publishes its result row to a HuggingFace
leaderboard dataset — no separate collection step needed.

Policies: pi0, pi0_fast, pi05, groot, act, diffusion, smolvla, xvla,
multi_task_dit. 5000 steps, BS 256, with per-policy GPU allocation and
default LR/scheduler presets.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@pkooij pkooij force-pushed the feat/libero-benchmark branch from 3c74bf6 to fd00e38 Compare April 9, 2026 15:01
@github-actions github-actions bot removed CI Issues related to the continuous integration pipeline github_actions labels Apr 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

documentation Improvements or fixes to the project’s docs

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@pkooij