feat: CI benchmark regression detection on PRs (#2013)

* feat: CI benchmark regression detection

Add a consolidated benchmark suite (`ci_benchmark`) and a GitHub Actions
workflow that automatically compares performance against the main branch
baseline on every PR.

Benchmark coverage (13 measurements, ~4 min on CI):
  - BPE GPT-2: encode, batch, no-cache, batch-no-cache
  - Llama-3: encode, batch, encode-fast, char-offsets, concurrent-4t
  - Serialization: GPT-2 load, Llama-3 load, Llama-3 save
  - Training: BPE small corpus

CI workflow:
  - On push to main: run benchmarks, store baseline in gh-pages branch
  - On PR: run benchmarks, compare vs baseline, post/update a single
    PR comment with the delta table
  - Alert threshold: 15% regression (warn, don't fail)

Uses benchmark-action/github-action-benchmark with criterion's bencher
output format for machine-readable results.

* fix: rustfmt + add github-token for PR comments

* ci: require approval for benchmark runs on PRs

* bench: add from_file + deserialize benchmarks to ci_benchmark

* fix: split workflow into two jobs to avoid empty environment value

* fix: download benchmark data in CI + skip if output empty

* fix: use huggingface-cli for data download (handles gated models like Llama-3)

* fix: use uvx for huggingface-cli, add setup-uv step

* fix: hf not huggingface-cli

* fix: single download from hf-internal-testing/tokenizers-bench-data

* fix: use hf-internal-testing dataset for BOTH jobs, remove all curl/gated refs

* Initialize gh-pages for benchmark data

* upupdate

* Initialize gh-pages for benchmark data

* ci: sccache, workflow_dispatch with PR comment, push-to-main only (no PR trigger)

* ci: temporarily trigger on ci-benchmarks branch for testing

* ci: touch tokenizers/ to trigger path filter

* ci: store baselines on HF Hub, drop github-action-benchmark + gh-pages

* ci: also trigger on workflow file changes + doc touch

* ci: skip upload if HF_TOKEN missing, remove ci-benchmarks branch trigger

* feat: add Python benchmark suite + CI matrix for Python bindings perf

- pytest-benchmark based test suite covering:
  - BPE GPT-2: encode, encode_batch, multithreaded (4 workers)
  - Llama-3: encode, encode_batch, encode_fast, multithreaded, decode_batch
  - Async: async_encode_batch, async_encode_batch_fast
  - Serialization: from_file, to_str, from_str (roberta, llama3, albert)
  - Training: BPE small corpus
- CI workflow: separate benchmark-python job with sccache + maturin
- Re-enabled ci-benchmarks branch trigger for testing

* fix: skip benchmark tests when pytest-benchmark is not installed

* ci: Python bench comparison against saved baseline + PR comment

* ci: compare against baseline BEFORE uploading new one (both Rust and Python)

* ci: /benchmark comment trigger + only upload baseline on push to main

- New benchmark-trigger.yml: maintainer comments '/benchmark' on a PR
  to dispatch the benchmark workflow on the PR's ref
- Upload steps gated on github.event_name == 'push' so workflow_dispatch
  (PR runs) never overwrite the baseline
- Trigger requires MEMBER/OWNER/COLLABORATOR association

* ci: remove ci-benchmarks branch trigger

* ci: /benchmark creates a check run on the PR, updated with results when done

- benchmark-trigger.yml creates a 'Benchmark Results' check on the PR head SHA
- benchmarks.yml marks it in_progress at start, completed (success/failure) at end
- The check body contains the comparison markdown table
- Can be made a required check in branch protection rules

* ci: use criterion --save-baseline/--baseline + export artifacts

Rust:
- Push to main: runs with --save-baseline main, uploads criterion
  data (tar.gz) + bencher output to HF Hub
- workflow_dispatch: downloads criterion baseline, runs with --baseline main
  for automatic criterion comparison
- criterion HTML report uploaded as GitHub Actions artifact (30 day retention)

Python:
- bench_output.json uploaded as GitHub Actions artifact
- Baseline stored/compared via HF Hub as before

Both:
- Artifacts downloadable from the workflow run page for manual inspection
- Comparison tables posted to PR comments

* update

* ci: pin macOS Python to 3.13 (3.14 breaks abi3 linking)

* ci: add -undefined dynamic_lookup for macOS abi3 cross-compilation linking

* ?

Author: ArthurZucker

.github/workflows/CI.yml

@@ -146,6 +146,11 @@ jobs:
           target: ${{ matrix.platform.target }}
           args: --release --out dist --manifest-path bindings/python/Cargo.toml
           sccache: 'true'
+        env:
+          # abi3 extension modules resolve Python symbols at runtime;
+          # explicit --target triggers cross-compilation mode which
+          # doesn't pass this flag automatically on macOS.
+          RUSTFLAGS: "-C link-arg=-undefined -C link-arg=dynamic_lookup"
       - name: Upload wheels
         uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02  # v4
         with:

.github/workflows/benchmark-trigger.yml

@@ -0,0 +1,63 @@
+name: Benchmark Trigger
+
+# A maintainer comments "/benchmark" on a PR to trigger benchmark comparison.
+# Creates a check run on the PR's head SHA so it appears in the PR's checks
+# list and can be made a required check in branch protection.
+
+on:
+  issue_comment:
+    types: [created]
+
+permissions:
+  actions: write
+  checks: write
+  pull-requests: read
+
+jobs:
+  trigger:
+    name: Trigger benchmarks
+    if: >-
+      github.event.issue.pull_request &&
+      contains(github.event.comment.body, '/benchmark') &&
+      (github.event.comment.author_association == 'MEMBER' ||
+       github.event.comment.author_association == 'OWNER' ||
+       github.event.comment.author_association == 'COLLABORATOR')
+    runs-on: ubuntu-latest
+    steps:
+      - name: Get PR info
+        id: pr
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          pr_number=${{ github.event.issue.number }}
+          head_sha=$(gh api "repos/${{ github.repository }}/pulls/$pr_number" --jq '.head.sha')
+          head_ref=$(gh api "repos/${{ github.repository }}/pulls/$pr_number" --jq '.head.ref')
+          echo "pr_number=$pr_number" >> "$GITHUB_OUTPUT"
+          echo "head_sha=$head_sha" >> "$GITHUB_OUTPUT"
+          echo "head_ref=$head_ref" >> "$GITHUB_OUTPUT"
+
+      - name: Create check run on PR
+        id: check
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          check_id=$(gh api "repos/${{ github.repository }}/check-runs" \
+            -X POST \
+            -f "name=Benchmark Results" \
+            -f "head_sha=${{ steps.pr.outputs.head_sha }}" \
+            -f "status=queued" \
+            -f "output[title]=Benchmarks queued" \
+            -f "output[summary]=Benchmark workflow has been dispatched. Results will appear here when complete." \
+            --jq '.id')
+          echo "check_id=$check_id" >> "$GITHUB_OUTPUT"
+
+      - name: Dispatch benchmark workflow
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          gh workflow run benchmarks.yml \
+            --repo "${{ github.repository }}" \
+            --ref "${{ steps.pr.outputs.head_ref }}" \
+            -f pr_number="${{ steps.pr.outputs.pr_number }}" \
+            -f check_run_id="${{ steps.check.outputs.check_id }}" \
+            -f head_sha="${{ steps.pr.outputs.head_sha }}"