You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Part of the 0.3.7 testing program (#322). Prerequisite: #324
(step knobs). Absorbs the CI-execution side of #319.
Problem
The 17 examples are the de-facto integration suite: they are the only runnable
artifacts that exercise the full launcher -> aggregator -> training pipeline
the way users do. No CI job runs any of them. During the 0.3.6 pre-merge
validation, a scripted sweep of the CPU-runnable examples caught more product
surface per minute than any test directory: verdict flips, per-rank rows,
null-vs-zero semantics, manifest lifecycle, stderr cleanliness.
Proposal
scripts/validate_examples.py: runs manifest-listed examples sequentially
under the real launcher, one TRACEML_LOGS_DIR per example, and asserts per
run: exit 0; final_summary.json present with the current schema version;
run manifest completed; h2d_ms null (never 0.0) on CPU; the expected
diagnosis assertion (see tiers below); no [TraceML]-prefixed
error/traceback lines in stderr.
A manifest (checked in, single source of truth) where each entry declares:
script, args, hardware (cpu/gpu/multi-gpu), timeout (~300s cap), and an
assertion tier:
structural: rc/schema/manifest/nulls only (default for anything
timing-sensitive on shared runners),
family: verdict must be in an allowlist (only where an injected knob
gives a large signal margin, e.g. the bottleneck demo's slow scenario),
exact: reserved for structurally-forced outcomes (e.g. a
no-instrumentation example must produce INCOMPLETE_DATA).
CI wiring per the umbrella tier model: CPU subset on PRs targeting
version_* branches (paths-filtered) + nightly full sweep; NOT per-PR
(except the Add a 2-process gloo DDP end-to-end test to CI #319 DDP entry, which is this harness invoked on one manifest
entry). Job hardening: OMP_NUM_THREADS=1, job timeout-minutes, file-based
or port-0 rendezvous for the DDP entry, artifacts (logs dir + summary +
stdout/stderr) uploaded on failure, one-line contributor-legible failure
messages (example, assertion class, expected vs actual), and a post-run
no-orphan-process check.
The examples/README expected-verdict table is GENERATED from the manifest
(or CI asserts they agree); two hand-maintained copies of the same truth
is how fixtures drifted before.
Notes
All current CPU-runnable examples are synthetic and download-free; the
manifest rule "no network in CI entries" keeps it that way.
The maintainers' release-validation checklist invokes this same script;
no forked sweep logic (engine invariant in the umbrella).
Part of the 0.3.7 testing program (#322). Prerequisite: #324
(step knobs). Absorbs the CI-execution side of #319.
Problem
The 17 examples are the de-facto integration suite: they are the only runnable
artifacts that exercise the full launcher -> aggregator -> training pipeline
the way users do. No CI job runs any of them. During the 0.3.6 pre-merge
validation, a scripted sweep of the CPU-runnable examples caught more product
surface per minute than any test directory: verdict flips, per-rank rows,
null-vs-zero semantics, manifest lifecycle, stderr cleanliness.
Proposal
scripts/validate_examples.py: runs manifest-listed examples sequentiallyunder the real launcher, one TRACEML_LOGS_DIR per example, and asserts per
run: exit 0; final_summary.json present with the current schema version;
run manifest
completed;h2d_msnull (never 0.0) on CPU; the expecteddiagnosis assertion (see tiers below); no
[TraceML]-prefixederror/traceback lines in stderr.
script, args, hardware (cpu/gpu/multi-gpu), timeout (~300s cap), and an
assertion tier:
structural: rc/schema/manifest/nulls only (default for anythingtiming-sensitive on shared runners),
family: verdict must be in an allowlist (only where an injected knobgives a large signal margin, e.g. the bottleneck demo's slow scenario),
exact: reserved for structurally-forced outcomes (e.g. ano-instrumentation example must produce INCOMPLETE_DATA).
version_* branches (paths-filtered) + nightly full sweep; NOT per-PR
(except the Add a 2-process gloo DDP end-to-end test to CI #319 DDP entry, which is this harness invoked on one manifest
entry). Job hardening: OMP_NUM_THREADS=1, job timeout-minutes, file-based
or port-0 rendezvous for the DDP entry, artifacts (logs dir + summary +
stdout/stderr) uploaded on failure, one-line contributor-legible failure
messages (example, assertion class, expected vs actual), and a post-run
no-orphan-process check.
(or CI asserts they agree); two hand-maintained copies of the same truth
is how fixtures drifted before.
Notes
manifest rule "no network in CI entries" keeps it that way.
no forked sweep logic (engine invariant in the umbrella).