Skip to content

CI-CONTRACT.md

Latest commit

 

History

History
384 lines (305 loc) · 14.8 KB

CI-CONTRACT.md

File metadata and controls

384 lines (305 loc) · 14.8 KB
Raw

CI Contract: Rul1an/Assay-Harness

Draft status: review contract before workflow implementation.

Rul1an/Assay-Harness is the public recipe, gate, and report layer for canonical Assay artifacts. It is not a runtime, capture engine, or second source of artifact semantics. The CI contract must therefore preserve the existing fixture/golden-output coverage, prove compatibility with released Assay shapes, and keep public docs/examples honest about what the harness consumes and reports.

This contract is a diff from today's repository state. The first rule is no required-coverage regressions: existing useful gates should stay required unless this file is updated with a clear replacement.

0. As-Is Inventory

Repository state observed on 2026-06-11:

  • Visibility: public.
  • Default branch: main.
  • Languages: TypeScript/JavaScript, Python.
  • Workflows:
    • .github/workflows/harness-ci.yml
    • .github/workflows/release.yml
    • .github/workflows/sbom.yml
    • .github/workflows/zizmor.yml
  • Reusable local action: .github/actions/setup-node-harness/action.yml.
  • Workflow posture already present:
    • workflow-level permissions: {} in the main workflows;
    • per-job permissions;
    • persist-credentials: false;
    • PR concurrency in harness-ci.yml;
    • zizmor required-check no-op behavior for unrelated PRs;
    • SBOM generation on main;
    • release artifact generation and build provenance attestation on tags.
  • Current action pinning: third-party actions use version tags such as actions/checkout@v6.0.2, actions/setup-python@v6, actions/setup-node@v6, actions/upload-artifact@v7, github/codeql-action/upload-sarif@v4.36.0, and actions/attest-build-provenance@v4, not commit SHAs.
  • Main CI required-shape jobs in harness-ci.yml:
    • Node Tests + Type Check
    • Contract Validation
    • Golden Contract Tests
    • Hardening Tests
    • Policy Validation
    • Verify Evidence
    • Regression Gate
    • Evidence Export
    • manual Assay Release Compatibility Recipes
  • TypeScript package: harness/package.json, package assay-harness, version 0.8.0, scripts for build, typecheck, tests, verify, and harness run.
  • Python utilities and tests:
    • mapper/map_to_assay.py
    • ci/emit_*.py
    • adapter/probe code under adapters/, patterns/, and probes/
    • Python unittest suites under tests/
  • Public artifacts and fixtures:
    • fixtures/*.harness.json
    • fixtures/*.assay.ndjson
    • adapter fixtures under fixtures/claude_agent_sdk/ and fixtures/deepagents/
    • examples under examples/
    • emitted JUnit/SARIF paths in CI
  • Dependency automation: .github/dependabot.yml for npm and GitHub Actions.
  • CI notes already document why job consolidation, broad path filtering, and release npm caching were not chosen.
  • Required branch-protection contexts: to be confirmed from a live PR through GitHub's checks API before settings are changed.

No target workflow should downgrade this inventory unless the contract is updated with an explicit rationale.

1. Required PR Checks

Required checks must be cheap, stable, and relevant to public contract drift. They should preserve cross-language fixture coverage rather than path-filtering away the exact regressions this repo exists to catch.

Existing Harness Contract Coverage

Keep these behaviors required:

  • Node build/typecheck and Node tests.
  • Mapping valid and failure fixtures to golden NDJSON output.
  • Malformed fixture rejection.
  • Python golden contract tests.
  • Python hardening tests.
  • Policy decisions for allowed, denied, and approval-required tools.
  • Evidence verification over valid and failure fixtures.
  • Regression compare output and exit behavior.
  • JUnit/SARIF evidence export shape.
  • SARIF upload tolerance where code-scanning upload is unavailable.

Do not split or path-filter these checks in a way that lets TypeScript changes skip Python contract tests, or Python mapper changes skip TypeScript consumer tests. This repository's value is the language boundary.

Assay Compatibility

Keep the release-binary compatibility recipe workflow available through workflow_dispatch, and treat it as the proving lane for a specific Assay release tag.

Follow-up target:

  • Add a scheduled compatibility canary against the latest supported Assay release line after the release-binary download path has a stable cost and failure profile.
  • Keep Assay main compatibility non-blocking and scheduled/manual only. It is useful early warning, not a PR gate.

Workflow And Action Hygiene

Keep zizmor required for workflow-relevant changes and weekly drift. Add or confirm:

  • actionlint for all workflow files.
  • shellcheck for shell blocks in workflows and local action shell blocks, either through actionlint integration or an explicit extractor.
  • timeout-minutes on every job.
  • Per-job permissions only; workflow-level default remains permissions: {}.
  • persist-credentials: false on all checkouts unless a job explicitly writes.
  • Ordinary setup/check actions may keep tag pins as a recorded tradeoff while this repo is still tightening.
  • High-trust actions that can affect release artifacts, provenance, SBOM, or security-tab output must move to commit-SHA pins in an implementation PR. This includes release, SBOM, attestation, and SARIF-upload actions.

Public Artifact Sanitization Guard

Add a required fail-closed public-artifact sanitization check for the whole repository, excluding only generated, vendored, build, dependency, and cache output explicitly allowlisted by the implementation. In a public repository, any source file, fixture, doc, test, code comment, sample, or workflow can leak public text; do not rely on path enumeration.

Hard rule:

  • The sensitive vocabulary list must not be present in this public repository and must not be printed in CI logs.

Acceptable implementation patterns:

  • Compare normalized tokens or n-grams against HMAC-SHA256 entries supplied from a private source plus a separate private HMAC key.
  • Run plaintext sensitive-list checks only in trusted private contexts where logs are not public and untrusted PR code cannot read the list.
  • On fork PRs, run only the public-safe structural portion.

Required-gate split:

  • The public-safe structural portion is required on every PR, including forks.
  • The full HMAC-denylist comparison runs only for trusted same-repository PRs, pushes, and scheduled checks that can access the private source and HMAC key safely.
  • A degraded fork run must say that private-list comparison was skipped without exposing the list, while still enforcing structural public-artifact rules.
  • The trusted HMAC-list layer is part of the sanitizer workflow, not a separate required context, until a future context-capture/import review says otherwise.
  • When the trusted HMAC-list layer runs, the list must include the digest for the committed public canary fixture. The scanner fails closed on a canary miss so key encoding, normalization, or generator drift cannot silently turn the trusted layer into a no-op.
  • The trusted list must enumerate every spelling, casing, and spacing variant of a term. Normalization lowercases, splits on non-alphanumerics, and HMACs one-to-five-token windows per line, so a compound spelling and a spaced or hyphenated spelling of the same term produce different digests. Variant completeness is a property of the trusted list, not the scanner.

Logging contract:

  • Report only counts and locations, for example 3 matches in README.md:42.
  • Never print matched text.
  • Never print the sensitive term, phrase, unhashed denylist entry, digest, or HMAC key.
  • Treat printing the matched term as a CI bug and a sanitization failure.

The guard is a backstop, not a guarantee. Human sanitization review remains primary because fixed matchers miss variants and context.

Claims And Boundary Guard

Add a lightweight required check for public wording in changed docs/examples:

  • Claims should name the consumed artifact or fixture shape.
  • Claims should name whether the check is informational or gating.
  • Claims should state degradation/non-claim when evidence cannot support a stronger statement.
  • Docs must not imply that Harness captures runtime behavior, performs live provider verification, owns Assay artifact semantics, or upgrades evidence beyond the observed source.

This guard can start as a checklist-backed script plus tests over known public docs. It should not block legitimate examples that explicitly state their boundary.

2. Scheduled Checks

Scheduled checks are for ecosystem drift and compatibility warning, not for ordinary PR cost.

  • Existing weekly zizmor drift canary.
  • OpenSSF Scorecard for public supply-chain posture. The first implementation uses the default GITHUB_TOKEN, which can read repository rulesets but may not fully measure classic branch-protection or webhook settings unless a future read/admin token is intentionally added.
  • OSV-Scanner for npm dependency surfaces with resolved lockfiles. Today this is harness/package-lock.json.
  • CodeQL or equivalent code scanning for TypeScript/JavaScript, Python, and workflow glue if GitHub default setup is not enabled.
  • Scheduled SBOM remains useful on main.
  • Optional scheduled Assay release compatibility against the latest supported release.
  • Optional non-blocking Assay main compatibility canary.

Scheduled supply-chain posture workflows are advisory only. They run on a weekly cadence plus manual dispatch, do not run on ordinary pull requests, and must not be promoted to required contexts without a separate context-capture review.

Do not schedule:

  • Live model/provider runs.
  • Self-hosted runner requirements.
  • Broad matrix expansion without a measured failure class.

3. Release-Only Checks

Release-only checks should preserve the existing artifact and provenance posture.

Keep:

  • Release tests before artifact generation.
  • Evidence artifact generation for JUnit, SARIF, and golden fixtures.
  • GitHub release creation/update through the GitHub CLI.
  • Build provenance attestation for release artifacts.
  • Release workflow's explicit no-cache choice unless a future contract update changes the supply-chain tradeoff.

Add or confirm:

  • Release notes/examples validate against current CLI inputs and outputs.
  • SBOM attached to releases if the release starts distributing package artifacts beyond generated evidence fixtures.
  • Floating tag policy if this repo later adopts floating major/minor tags.

Current release boundary:

  • Attestations apply to the generated release artifacts in this repository.
  • They do not prove Assay runtime truth, live provider behavior, or correctness beyond the emitted artifact subject and workflow boundary.

4. Manual Checks

Manual workflows are acceptable for checks that are useful but not routinely needed:

  • Assay release compatibility for a specific input tag.
  • Compatibility against Assay main.
  • Expanded adapter/probe suites that require optional SDKs.
  • Regeneration of large example artifacts.

Manual workflow requirements:

  • Inputs must be read through environment variables or otherwise avoid template injection.
  • Downloaded release assets must be checksum-verified.
  • Any generated public artifacts must pass sanitization before publication or release attachment.

5. Non-Goals And Non-Claims

Non-goals:

  • No fuzzing by default.
  • No live model/provider dependency in required PR checks.
  • No self-hosted runner dependency.
  • No broad OS matrix by default.
  • No path-filtered skip that hides cross-language fixture regressions.
  • No second artifact semantics layer beside Assay.

Allowed language:

  • "Recipe, gate, and report layer."
  • "Composes released Assay artifact shapes."
  • "Emits JUnit/SARIF projections."
  • "Gates a fixture-backed regression."
  • "Reports observed support and degradation."

Disallowed without explicit boundary:

  • Claims that Harness captures runtime effects.
  • Claims that Harness verifies live provider behavior.
  • Claims that Harness proves compliance.
  • Claims that Harness replaces Assay verification.
  • Claims that attestation changes observed support.

The sanitization guard is separate from these claim-boundary rules. It protects private strategy vocabulary from appearing in public artifacts and must do so without reprinting protected vocabulary.

6. Required Context Names

Branch protection is enforced by exact check context names, not by this file. Before making any branch-protection changes:

  1. Open a draft PR that implements the workflows.
  2. Query the live check runs for that PR.
  3. Copy the exact check names into this section.
  4. Treat future job renames as breaking changes because they can silently un-gate protected branches.

Proposed required context groups:

  • Existing Harness CI contract jobs.
  • Existing Audit workflow security check.
  • Public artifact sanitization.
  • Action/workflow lint.
  • Claims and boundary guard.

Observed from the CI baseline implementation PR #106:

  • Node Tests + Type Check
  • Contract Validation
  • Golden Contract Tests
  • Hardening Tests
  • Policy Validation
  • Verify Evidence
  • Regression Gate
  • Evidence Export
  • Audit workflow security
  • Public Artifact Sanitization
  • Assay Release Compatibility Recipes

Proposed required context names for the next branch-protection review:

  • Node Tests + Type Check
  • Contract Validation
  • Golden Contract Tests
  • Hardening Tests
  • Policy Validation
  • Verify Evidence
  • Regression Gate
  • Evidence Export
  • Audit workflow security
  • Public Artifact Sanitization

Checked-in ruleset activation lives at .github/rulesets/main-required-ci-contexts.json.

Import note: the checked-in ruleset is config-as-code only until imported in GitHub settings. Add bypass_actors only if the repository owner intentionally wants to preserve an admin bypass path; otherwise strict_required_status_checks_policy: true means merges must be rebased-current and green.

Do not require Assay Release Compatibility Recipes on ordinary pull requests. It is a manual or compatibility recipe lane, not a stable required merge gate.

External advisory checks should remain non-required unless the repository owner explicitly accepts their availability as a merge dependency.

7. Target Workflow Files

Expected target workflow set:

  • .github/workflows/harness-ci.yml kept and tightened, not removed.
  • .github/workflows/zizmor.yml kept.
  • .github/workflows/sbom.yml kept.
  • .github/workflows/release.yml kept.
  • .github/workflows/action-lint.yml if actionlint/shellcheck are not folded into the existing zizmor workflow.
  • .github/workflows/sanitize.yml for public-artifact sanitization.
  • .github/workflows/claims-boundary.yml for public wording and non-claim checks, unless folded into sanitization.
  • .github/workflows/compatibility-canary.yml only if scheduled Assay release compatibility is promoted after review.

Implementation should happen in small follow-up PRs after this contract is reviewed.