carbon a11y tests fetch the ACE engine and rule archive at eval time (~4.5% flake)

[john-b-yang]

accessibility-checker reaches the network during evaluation, which makes carbon's
a11y-bearing tests non-deterministic. Measured rate on a full 133-instance run: 6 flaked
(127/133 first pass), all recovered by one automatic retry.

Failure signature:

TypeError: ace.Checker is not a constructor      (ACEngineManager.ts)
TypeError: ace_ibma.Checker is not a constructor

What is fetched

Two separate things, both at eval time:

1. the engine — lib/engine/ace-node.js
2. the rule archive itself

.achecker.yml (ruleArchive: 07Oct2020, set in constants.py) pins which archive is
used, so results are reproducible, but it does not make either fetch local.

Investigated: vendoring the engine

Works, but is only half the fix. Verified in a container:

• copy ace-node.js into node_modules/accessibility-checker/lib/engine/
• patch aChecker.loadLocalEngine to use it when present — necessary, because the function
  otherwise calls request.get(rulePack + "/ace-node.js") unconditionally and overwrites
  whatever is there. Its only short-circuit is an in-memory ace variable.

With able.ibm.com blackholed (v4 and v6) the engine loaded fine — 0 ACE errors — but
the a11y suites then emitted 419 ECONNREFUSED fetching the rule archive, failing 10
component suites (Button, Link, Accordion, Checkbox, Tag, Loading, ...).

What a complete fix needs

• mirror /archives/2020.10.07/ into the image
• point rulePack at a local server (ACConfigLoader supports a custom rule server)

That means rebuilding + re-pushing 133 images and running a static server inside every
carbon evaluation. Judged not worth it for a 4.5% transient that one retry clears.

Gotcha for anyone who picks this up

The archive id does not map to a URL of the same name. 07Oct2020 resolves via
https://able.ibm.com/rules/archives.json to path /archives/2020.10.07.
https://able.ibm.com/rules/archives/07Oct2020/js/ace-node.js returns 404;
https://able.ibm.com/rules/archives/2020.10.07/js/ace-node.js returns 200 (404 KB).
Building an image from the guessed URL would silently bake a 404 page into 133 images.

Current mitigation

Retry once on failure (carbon rebuild pipeline does this: 127/133 → 133/133), plus
--maxWorkers=4 for carbon, which appears to reduce the rate further — two instances that
failed uncapped passed with the cap.

[john-b-yang]

Not planned. Documenting the investigation so nobody repeats it.

A build-time fix was prototyped and does work — vendor ace-node.js into the image and
short-circuit the loader so nothing is fetched during evaluation. Verified twice with
able.ibm.com blackholed over IPv4 and IPv6, once by hand and once on a generator-built
image: 163/163 suites, 1347 tests, 0 ACE errors, fully offline.

It was not adopted because carbon spans two accessibility-checker generations:

generation	loader	patchable
older	ACHelper.js / loadLocalEngine	yes — plain JS
3.1.32	ACEngineManager.js / loadEngineLocal	no — transpiled __awaiter/__generator state machine

8 of 9 randomly sampled instances hit the newer generation, so the patch covers only part
of the repo. A version-agnostic fix is possible, since both generations do
GET rulePack + "/ace-node.js", but the newer one resolves rulePack by fetching
archives.json from ruleServer. That means running a local HTTP server inside every
carbon evaluation serving two files, plus .achecker.yml keys for both generations
(customRuleServer/rulePack for the old, ruleServer for the new). A background
process per evaluation is a worse trade than the flake it removes.

Empirical mitigation, which is what we use: re-run the affected instance. Measured over
a full 133-instance carbon rebuild, 127 passed first time and all 6 stragglers passed on a
single automatic retry (133/133). Capping jest at --maxWorkers=4 also appears to reduce
the rate — two instances that failed uncapped passed with the cap.

Gotcha if anyone revisits: the archive id is not its URL. 07Oct2020 resolves via
https://able.ibm.com/rules/archives.json to /archives/2020.10.07;
/archives/07Oct2020/js/ace-node.js returns 404, so building from the guessed URL would
silently bake an error page into the images.