test(project-engine-client): reusable live-vs-mock parity probe

[rainer-friederich]

1. Abstract

Adds a reusable, token-gated live-vs-mock parity probe (scripts/parity-probe.mjs) for the Semrush Project Engine and User Manager gateways — a drift detector that replays one request catalog against any origin and diffs the captures. It is a dev tool under scripts/, outside the published surface.

2. Reasoning

The mock-fidelity sweep behind adobe/spacecat-shared issue 1745 was a manual, one-off comparison of the Counterfact mocks against the live Semrush gateway. The findings need to stay true as the mocks change (the fixes land in PR 1746), so the comparison should be repeatable rather than re-done by hand each time.

3. High-level overview of the changes

Adds a single Node ESM CLI (no dependencies, no runtime/app behaviour change — it does not ship with the package; it lives in scripts/, outside the src/ + mock/ coverage globs). Two modes:

• capture — replays a canonical PE + User Manager request catalog against any origin (the live Semrush gateway OR a running Counterfact mock), normalising volatile uuids/timestamps into a JSON capture. --mutate adds a self-cleaning create → publish → duplicate → delete lifecycle (it deletes exactly what it creates); --only pe|um restricts to one gateway for a single-container mock; --insecure accepts the mock's self-signed TLS.
• diff — compares two captures and splits KNOWN / documented divergences (a curated catalog, each tagged fixed-in-1746 or intentional with a reason) from UNEXPECTED drift. Comparison is shape + behavioural-sentinel based (field presence/type plus a small set of value-bearing fields such as language.name, publish_status, existing_count, initialized), so a mock seeded with different data than the live workspace does not flood the diff. It exits non-zero on unexpected drift, so it can gate a manual parity check.

Going-forward use: after the 1746 mock bump, capture live + the new mock and diff — the UNEXPECTED list is new drift to investigate; the KNOWN list is the documented-gap ledger.

4. Required information

• Jira / issue: PE mock: created+published project read-view drops the market (empty settings.ai.language.name; publish_status not flipped) — blocks serenity market round-trip #1745
• Other: complements PR fix(project-engine-client, user-manager-client): round-trip created+published market + live-fidelity sweep (#1745) #1746 (the actual mock fixes); findings posted as comments on issue 1745.

6. Additional information outside the code

Validated this session against the live Semrush prod gateway (https://adobe-hackathon.semrush.com, IMS bearer) and the published ghcr.io/adobe/spacecat-shared-project-engine-client-mock:1.3.1 image (pulled and booted locally on :8443). The diff correctly auto-classified every documented divergence from issue 1745 into the KNOWN section (the language.name / country.name / publish_status / published_at / existing_count / family-403 ones as fixed-in-1746; the 500 / 403 / init_status / publish-content-type ones as intentional), and surfaced the broader read-view drift that 1746 closes in the UNEXPECTED section. The --mutate lifecycle left zero leaked projects on the live workspace (verified by listing the workspace afterward).

7. Test plan

(a) Done this session: ran capture against live + the published mock image and diff-ed them; confirmed the KNOWN/UNEXPECTED split is correct and the --mutate run self-cleaned.

(b) To re-verify (manual, token-gated — not a CI job):

• node scripts/parity-probe.mjs capture --only pe --base https://adobe-hackathon.semrush.com --token "$(mysticat auth token --ims)" --pe-ws <child-ws> --pe-project <published-project> --pe-unowned-ws <ws-you-do-not-own> --mutate --out live.json
• boot the mock (npm run mock or the GHCR image), then ... capture --only pe --base <mock-origin> --token dummy --insecure --mutate --out mock.json
• node scripts/parity-probe.mjs diff live.json mock.json — expect the documented gaps under KNOWN and nothing genuinely new under UNEXPECTED.

8. Deployment & merge order

Independent — no merge-order dependency. Related to PR #1746 (the mock fixes this tool measures); either can merge first.

🤖 Generated with Claude Code