Skip to content

PERFORMANCE.md

Latest commit

 

History

History
166 lines (124 loc) · 7.05 KB

PERFORMANCE.md

File metadata and controls

166 lines (124 loc) · 7.05 KB
Raw

Performance & Benchmarking

Scope

This document tracks runtime and SDK benchmark coverage for Sovereign-Mohawk and explains how to reproduce and compare benchmark outputs.

Benchmark Surfaces

  1. Python SDK regression gate (pytest-benchmark) in .github/workflows/performance-gate.yml
  2. Go runtime FedAvg aggregation matrix (BenchmarkAggregateParallel) in test/accelerator_test.go
  3. Base-vs-current FedAvg benchmark comparison report generated by scripts/benchmark_fedavg_compare.sh
  4. Bridge JSON-vs-zero-copy compression benchmark comparison generated by scripts/benchmark_bridge_compression_compare.sh

Benchmark Contract (Canonical Defaults)

Surface Default Parameters Notes
FedAvg base-vs-current compare BASE_REF=origin/main, BENCH_TIME=300ms, BENCH_COUNT=10, BENCH_CPU=2, USE_BENCHSTAT=always, BENCHSTAT_ALPHA=0.01 Fails if expected benchmark symbol BenchmarkAggregateParallel/ is missing in base or current output.
Bridge format compare BENCH_TIME=200ms, BENCH_COUNT=5, BENCH_CPU=2, BENCHSTAT_ALPHA=0.01 JSON vs zero-copy on same commit; PR regression compares against cached main baseline.
Python SDK gate pytest-benchmark JSON artifact plus absolute and optional trend checks Trend checks compare against cached main baseline when available.

Regression Policy (Fail-Closed)

Gate Threshold Fail Behavior
FedAvg PR regression +5.0% geomean time regression Fails on threshold breach, missing baseline cache, or unparsable geomean delta.
Bridge PR regression +5.0% geomean time regression Fails on threshold breach, missing baseline cache, or unparsable geomean delta.
Python SDK absolute Per-benchmark mean/p99 limits in .github/workflows/performance-gate.yml Fails on any threshold breach or missing expected benchmarks.
Python SDK trend PY_PERF_MEAN_DELTA_MAX_PCT=10.0, PY_PERF_P99_DELTA_MAX_PCT=15.0 Fails on regression breach when baseline exists; logs skip when baseline is not available.

Current Benchmark Matrix (FedAvg Runtime)

The FedAvg microbenchmark matrix currently includes:

  • Workloads:
    • clients32_dim2048
    • clients128_dim4096
    • clients256_dim8192
    • clients512_dim8192
  • Worker configurations:
    • workers1
    • workers2
    • workers4
    • workers8
    • workersAuto

workersAuto resolves through ResolveAggregateWorkers(...) in internal/accelerator/aggregate.go, which uses single-worker fallback on small reductions and parallel worker selection for larger workloads.

Latest Runtime Findings (March 26, 2026)

Representative result trends from the current benchmark run:

  • Small workload (32x2048): single worker or workersAuto is typically fastest due to parallel overhead.
  • Medium/large workloads (128x4096 and above): multi-worker and workersAuto materially improve throughput.
  • High workload (512x8192): parallel paths substantially outperform workers1 in typical runs.

This is expected behavior for reduction-heavy workloads where synchronization overhead dominates at small tensor sizes but amortizes at higher element counts.

Live Stress Capture Evidence (March 26, 2026)

A live stress capture was executed on a running full stack with temporary stress node agents (node-agent-stress-4 through node-agent-stress-10) over a 10-minute scope using 3 windows of 200 seconds each.

Artifacts:

  • results/metrics/stress_metrics_capture_10m.json
  • results/metrics/stress_metrics_capture_10m.md

Highlights from the 10-minute capture:

  • bridge_total delta: +60 (mean rate 0.100/s)
  • proof_total delta: +120 (mean rate 0.200/s)
  • hybrid_proof_total delta: +60 (mean rate 0.100/s)
  • accel_ops_total delta: +240 (mean rate 0.400/s)
  • accel_gradient_submit_success_total delta: +60
  • accel_gradient_submit_failure_total delta: 0

Resource snapshot during stress run showed low utilization headroom remained available (for example, orchestrator ~1.45% CPU and ~51.46 MiB memory in the sampled interval).

How To Run

1. Python SDK benchmark gate (local equivalent)

cd sdk/python
python -m pytest tests/test_benchmarks.py --benchmark-only -q

2. Go FedAvg benchmark matrix

TOOLROOT=/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.9.linux-amd64 \
GOROOT=$TOOLROOT PATH=$TOOLROOT/bin:$PATH GOTOOLCHAIN=local \
go test ./test -run '^$' -bench BenchmarkAggregateParallel -benchmem -benchtime=300ms -count=10 -cpu=2

3. Generate base-vs-current comparison report

TOOLROOT=/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.9.linux-amd64 \
BASE_REF=origin/main BENCH_TIME=300ms BENCH_COUNT=10 \
BENCH_CPU=2 \
USE_BENCHSTAT=always BENCHSTAT_ALPHA=0.01 \
REPORT_PATH=results/metrics/fedavg_benchmark_compare.md \
./scripts/benchmark_fedavg_compare.sh

This enforces benchstat output and uses a stricter significance threshold (alpha=0.01).

4. Run extended live stress capture (10-minute scope)

# Full stack should already be running.
for i in 4 5 6 7 8 9 10; do
  docker rm -f node-agent-stress-$i >/dev/null 2>&1 || true
  docker compose run -d --no-deps --name node-agent-stress-$i \
    -e NODE_ID=node-stress-$i \
    -e MOHAWK_LIBP2P_PORT=$((4100+i)) \
    node-agent-1 >/dev/null
done

# Capture script path used in this run:
python3 - <<'PY'
print('See results/metrics/stress_metrics_capture_10m.json and .md artifacts from captured run')
PY

# Cleanup temporary stress agents.
for i in 4 5 6 7 8 9 10; do
  docker rm -f node-agent-stress-$i >/dev/null 2>&1 || true
done

5. Compare bridge compression paths (JSON vs zero-copy)

BENCH_TIME=200ms REPORT_PATH=results/metrics/bridge_compression_benchmark_compare.md \
BENCH_COUNT=5 BENCH_CPU=2 BENCHSTAT_ALPHA=0.01 \
./scripts/benchmark_bridge_compression_compare.sh

This benchmark report compares JSON vs zero-copy serialization formats on the same commit. It is not a base-ref regression report.

CI Artifacts

  • Python performance artifact:
    • benchmark-results.json (uploaded by .github/workflows/performance-gate.yml)
  • FedAvg comparison artifact:
    • results/metrics/fedavg_benchmark_compare.md (uploaded by .github/workflows/fedavg-benchmark-compare.yml)
    • Baseline source in CI: cached main benchmark output (fedavg-main-* cache key prefix) and artifact fedavg-baseline-main
  • Bridge compression comparison artifacts:
    • results/metrics/bridge_compression_benchmark_compare.md
    • results/metrics/bridge_compression_benchmark_raw.txt
    • PR regression artifact: results/metrics/bridge_compression_regression_compare.md
    • Baseline source in CI: cached main raw benchmark output (bridge-main-* cache key prefix) and artifact bridge-baseline-main
  • Extended live stress capture artifacts:
    • results/metrics/stress_metrics_capture_10m.json
    • results/metrics/stress_metrics_capture_10m.md

Notes

  • If benchmark symbols are missing from either base or current FedAvg runs, scripts/benchmark_fedavg_compare.sh fails fast with an explicit error.
  • For stable trend analysis, run with BENCH_COUNT>=3 and compare medians across runs.