Skip to content

Examples validation harness: scripts/validate_examples.py + manifest + CI tiers #323

Description

@Pendu

Part of the 0.3.7 testing program (#322). Prerequisite: #324
(step knobs). Absorbs the CI-execution side of #319.

Problem

The 17 examples are the de-facto integration suite: they are the only runnable
artifacts that exercise the full launcher -> aggregator -> training pipeline
the way users do. No CI job runs any of them. During the 0.3.6 pre-merge
validation, a scripted sweep of the CPU-runnable examples caught more product
surface per minute than any test directory: verdict flips, per-rank rows,
null-vs-zero semantics, manifest lifecycle, stderr cleanliness.

Proposal

  1. scripts/validate_examples.py: runs manifest-listed examples sequentially
    under the real launcher, one TRACEML_LOGS_DIR per example, and asserts per
    run: exit 0; final_summary.json present with the current schema version;
    run manifest completed; h2d_ms null (never 0.0) on CPU; the expected
    diagnosis assertion (see tiers below); no [TraceML]-prefixed
    error/traceback lines in stderr.
  2. A manifest (checked in, single source of truth) where each entry declares:
    script, args, hardware (cpu/gpu/multi-gpu), timeout (~300s cap), and an
    assertion tier:
    • structural: rc/schema/manifest/nulls only (default for anything
      timing-sensitive on shared runners),
    • family: verdict must be in an allowlist (only where an injected knob
      gives a large signal margin, e.g. the bottleneck demo's slow scenario),
    • exact: reserved for structurally-forced outcomes (e.g. a
      no-instrumentation example must produce INCOMPLETE_DATA).
  3. CI wiring per the umbrella tier model: CPU subset on PRs targeting
    version_* branches (paths-filtered) + nightly full sweep; NOT per-PR
    (except the Add a 2-process gloo DDP end-to-end test to CI #319 DDP entry, which is this harness invoked on one manifest
    entry). Job hardening: OMP_NUM_THREADS=1, job timeout-minutes, file-based
    or port-0 rendezvous for the DDP entry, artifacts (logs dir + summary +
    stdout/stderr) uploaded on failure, one-line contributor-legible failure
    messages (example, assertion class, expected vs actual), and a post-run
    no-orphan-process check.
  4. The examples/README expected-verdict table is GENERATED from the manifest
    (or CI asserts they agree); two hand-maintained copies of the same truth
    is how fixtures drifted before.

Notes

  • All current CPU-runnable examples are synthetic and download-free; the
    manifest rule "no network in CI entries" keeps it that way.
  • The maintainers' release-validation checklist invokes this same script;
    no forked sweep logic (engine invariant in the umbrella).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions