@chio/conformance

Third-party conformance test suite for @chio/bridge. A fresh plugin developer points this at their own bond/check flow (or at a running chio-test-harness) and gets a yes/no answer plus actionable diagnostics for every part of the bridge surface.

This is plugin-author infrastructure, not an end-user tool.

Install

From a local checkout (this package is not yet on the registry):

cd chio-conformance
bun install
bun run build

You now have chio-conformance on your local node_modules/.bin/ path, or you can invoke node dist/index.js directly.

Usage

Against a running chio-test-harness

# In one shell: start the harness.
cd chio-test-harness
bash bin/start.sh
source bin/env.sh

# In the same shell: run the conformance suite.
cd ../chio-conformance
node dist/index.js run

The runner reads $CHIO_TRUST_URL, $CHIO_MCP_URL, and $CHIO_TOKEN from the sourced env.

Against an arbitrary daemon

chio-conformance run \
  --daemon http://my-trust-plane.example:8940 \
  --mcp    http://my-mcp-edge.example:8931 \
  --token  $MY_SERVICE_TOKEN \
  --policy ./policies/my-canonical.yaml

When --daemon and --token are both passed, the runner will NOT spawn a harness; it uses the supplied endpoints verbatim.

Against your own plugin's bond flow

Your plugin already imports @chio/bridge. Stand up a harness (or your own trust plane + MCP edge), run your plugin's initialization/bond code once so the trust plane has receipts to return, and then run the conformance suite against the same daemon URLs your plugin used. This proves the bridge behavior your plugin relies on is reproducible against a vanilla set of fixtures.

CI integration

- name: Conformance
  run: |
    # 1. Spin up chio-test-harness (or your equivalent).
    bash chio-test-harness/bin/start.sh
    source chio-test-harness/bin/env.sh
    # 2. Run the suite with JSON output for machine-readable gating.
    node chio-conformance/dist/index.js run --json > conformance.json
    # 3. Gate on non-zero exit; the runner exits 1 if any test failed.

Flags

Flag	Description
--daemon <url>	Trust plane base URL. If unset, uses $CHIO_TRUST_URL or spawns a local chio-test-harness.
--mcp <url>	MCP edge URL. Defaults to $CHIO_MCP_URL or http://127.0.0.1:8931.
--token <t>	Service token used for both trust plane and MCP edge.
--policy <yaml>	Override the canonical policy fixture shipped with this package.
--suite <name>	Run one named test (see chio-conformance list) or all (default).
--json	Emit the full RunnerReport as JSON (CI-friendly). Exits 1 on any failure.

Output shape

Human-readable table (default):

chio-conformance 0.1.0
daemon:  http://127.0.0.1:8940
mcp:     http://127.0.0.1:8931
policy:  ./src/util/fixtures/canonical.yaml
harness: ../chio-test-harness (pre-existing)

  ✓ policy-parse           (0.12s)
  ✓ bond-cycle             (0.43s)
  ✓ check-allow            (0.18s)
  ✓ check-deny-path        (0.15s)
  …
  ✗ extensions-chio        (0.10s)  — policy parse error: unknown field "chio"

12 tests · 11 passed · 1 failed · 0 skipped
score: 11/12 (92%)  [total 3.14s]

JSON (--json):

{"suite":"all","harness":{"trustUrl":"http://127.0.0.1:8940","mcpEdgeUrl":"http://127.0.0.1:8931","harnessDir":"…","owned":false},"policy":"…/canonical.yaml","passed":11,"failed":1,"skipped":0,"score":0.92,"totalMs":3140,"tests":[{"name":"policy-parse","status":"pass","detail":"…","durationMs":120}, …]}

What each test exercises

Test	Bridge surface	Assertion
policy-parse	loadPolicy, lintPolicy	Canonical policy lints clean; a policy with unknown rules.* key surfaces a specific per-path error.
bond-cycle	bond, status, revoke, verifyPassport({passportId})	Lifecycle round-trip: bond → bonded=true → revoke → verifyPassport false.
check-allow	check, receipts, verifyReceipt	Permitted tool yields allow; recent receipt for that tool passes ed25519 verification.
check-deny-path	check	delete_file against /etc/hosts denies (forbidden_paths / tool_access).
check-deny-egress	check (CLI mode)	fetch targeting a blocked host denies under egress: default=block.
budget-cancel	check (CLI mode)	tiny-budget.yaml velocity guard cancels before 6 paid_action calls complete.
human-in-loop	check (CLI mode)	require_confirmation: ['refund'] gates a refund call (not allow).
receipt-verify	receipts, verifyReceipt, parseReceipt	100% of live receipts verify; bit-flipping a signature breaks verification.
attenuate	issueCapability, attenuate	Narrower capability is a fresh id with strictly-narrower grants.
evidence-export	exportEvidence	Bundle JSON loads; every nested receipt verifies; tamper breaks verification.
revoke-all-subject	bond, revokeAllForSubject, verifyPassport	Two bonds share a DID; revokeAllForSubject(did) kills both.
extensions-chio	loadPolicy, lintPolicy	extensions.chio.market_hours + human_in_loop.approve_when are accepted.

Tests that can't exercise their surface against the resolved runtime (missing CLI binary, /v1/evidence/export unimplemented, kernel-split subject DIDs) report status: "skip" with a reason rather than hard-failing.

Exit codes

Code	Meaning
0	All tests passed (or only passed/skipped).
1	One or more tests failed.
2	Bad CLI flag.
3	Runner bootstrap failed (couldn't resolve a harness or construct the bridge).
4	Uncaught fatal in main.

Determinism notes

• bond-cycle, revoke-all-subject, and attenuate mutate real trust-plane state. They don't clean up after themselves — the harness retains history across runs. That's intentional for repeatable audit, but re-running the suite against a pre-existing harness will accumulate records.
• check-deny-egress, budget-cancel, and human-in-loop drive the chio CLI directly (not the MCP edge) because those guards require a policy the MCP edge's synthetic grant shape doesn't satisfy. They skip when CHIO_BIN is unset.
• evidence-export skips if the trust plane on the resolved build doesn't expose /v1/evidence/export.

See VERIFY.md for a reference run against the current arc binary.

License

Apache-2.0.