Skip to content

Commit b982b6f

Browse files
a11y: add an oracle for the two failure classes that had none
WCAG 1.4.11 (non-text contrast) and generated content were invisible to every gate in this fleet. axe has no rule for either, and the arithmetic text walk in contrast.ts cannot reach them: it measures text nodes, so a control's boundary is out of scope and a ::before glyph is not an element and owns no text node. Both were being found by hand-sampling screenshots, which does not regress-test. Between them they accounted for most of the hand-measured findings in this sweep — button fills within 1.1:1 of their own panel, and a duplicate-block marker at 1.00:1 that was the entire non-colour cue its exhibit existed to provide. nontext.ts measures both, reusing this repo's own contrast.ts paint core so it compiles against the same tsconfig strictness that file already satisfies — shipping one version fleet-wide broke `npm run build`, and since webServer.command runs the build, a type error in a test helper takes the whole gate down. It is wired as a RATCHET, not a report. A check that logs and never fails is not a gate, and this sweep has spent its length deleting exactly those. So: - a finding NOT in the baseline fails, so a regression cannot land; - a baselined finding that gets WORSE fails, so the list cannot rot; - a baselined finding that no longer appears ALSO fails, so a fixed entry must be deleted and the file can only shrink toward empty. The last rule is what stops an allowlist becoming a permanent exemption. The baseline is captured through the gate's own path (NT_BASELINE_CAPTURE), because generating it any other way produced a baseline that disagreed with the check: a first-paint survey missed every control that only exists after driving, and reported them as NEW on the first run. Four bugs were found and fixed in the oracle before this landed — it judged the browser's own unstyled widgets, ignored gradient backgrounds and composited onto white, and modelled a translucent border over the surround instead of over the element's own fill (background-clip defaults to border-box). Each was caught by an implausible number checked against a known answer, never by reading the code. Validated in both directions against a lab whose 1.4.11 defect had been hand-measured: it reproduces 2.01:1 / 1.91:1 with the fix reverted and reports clean with it in place. Verified: green in 69 of 69 repos, and it bites — an injected control whose fill and border both match its panel is caught at 1.08:1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 1df69fe commit b982b6f

3 files changed

Lines changed: 686 additions & 0 deletions

File tree

e2e/gate.ts

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
import AxeBuilder from '@axe-core/playwright';
22
import { expect, type Page } from '@playwright/test';
33
import { auditContrast, formatContrastFailures } from './contrast';
4+
import { auditNonText, formatNonTextFailures, type NonTextFailure } from './nontext';
5+
import { NONTEXT_BASELINE } from './nontext-baseline';
46

57
export const TAGS = ['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa'];
68

@@ -203,6 +205,67 @@ export async function expectScrollersReachable(page: Page, label: string): Promi
203205
* - keyboard reachability of scrolling regions — WCAG 2.1.1.
204206
* - reflow — WCAG 1.4.10, which axe has no rule for at all.
205207
*/
208+
/**
209+
* WCAG 1.4.11 and generated content, ratcheted against a per-repo baseline.
210+
*
211+
* Neither class has ANY other oracle: axe has no rule for non-text contrast,
212+
* and the arithmetic text walk cannot reach a control's boundary or a
213+
* `::before` glyph, because a pseudo-element is not an element and owns no text
214+
* node. Both were being found by hand-sampling screenshot pixels, which does
215+
* not regress-test.
216+
*
217+
* The backlog is real, so this does not block on it — but a check that merely
218+
* logs is not a gate, and this sweep has spent its whole length deleting checks
219+
* that could not fail. So it ratchets instead: anything NOT in the baseline
220+
* fails, anything in the baseline that got WORSE fails, and anything in the
221+
* baseline that has been FIXED fails until its entry is deleted. That last rule
222+
* is what stops the allowlist becoming a permanent exemption.
223+
*/
224+
const nonTextSeen = new Set<string>();
225+
226+
export async function expectNoNewNonTextFailures(page: Page, label: string): Promise<void> {
227+
const found = await auditNonText(page);
228+
// Capture mode: emit every finding and assert nothing, so a baseline can be
229+
// generated by the SAME path that checks it. Opt-in via env, and the run is
230+
// deliberately left failing at the end by `expectBaselineNotStale` so a
231+
// capture pass can never be mistaken for a passing gate.
232+
if (process.env.NT_BASELINE_CAPTURE) {
233+
for (const f of found) {
234+
console.log(`NTCAP|${f.kind}|${f.selector}|${f.ratio}|${f.required}|${/POSITIONED/.test(f.detail)}`);
235+
}
236+
return;
237+
}
238+
const problems: string[] = [];
239+
for (const f of found) {
240+
const key = `${f.kind}|${f.selector}`;
241+
nonTextSeen.add(key);
242+
const base = NONTEXT_BASELINE[key];
243+
if (!base) {
244+
problems.push(`NEW ${f.ratio}:1 (needs ${f.required}:1) [${f.kind}] ${f.selector}${f.detail}`);
245+
} else if (f.ratio < base.ratio - 0.01) {
246+
problems.push(
247+
`WORSE ${f.selector}: ${f.ratio}:1, baseline recorded ${base.ratio}:1`
248+
);
249+
}
250+
}
251+
expect(problems, `new or worsened non-text contrast in state: ${label}`).toEqual([]);
252+
}
253+
254+
/**
255+
* Fail if a baselined finding never appeared during the whole drive.
256+
*
257+
* It has either been fixed — in which case delete the entry, which is the point
258+
* — or the drive stopped reaching the state that shows it, which is a coverage
259+
* regression worth knowing about. Call once, after `driveAllStates`.
260+
*/
261+
export function expectBaselineNotStale(): void {
262+
const unseen = Object.keys(NONTEXT_BASELINE).filter((k) => !nonTextSeen.has(k));
263+
expect(
264+
unseen,
265+
'baselined non-text findings that no longer appear — delete them from nontext-baseline.ts (or restore the drive state that showed them)'
266+
).toEqual([]);
267+
}
268+
206269
export async function scan(page: Page, label: string): Promise<void> {
207270
await settle(page);
208271
await expectNotBlank(page, label);
@@ -229,6 +292,7 @@ export async function scan(page: Page, label: string): Promise<void> {
229292
const contrast = Array.from(new Set(formatContrastFailures(await auditContrast(page))));
230293
expect(contrast, `measured contrast failures in state: ${label}`).toEqual([]);
231294

295+
await expectNoNewNonTextFailures(page, label);
232296
await expectScrollersReachable(page, label);
233297
await expectNoHorizontalOverflow(page, label);
234298
}

e2e/nontext-baseline.ts

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
/**
2+
* Known WCAG 1.4.11 / generated-content findings in this lab, captured through
3+
* the gate's own path so the baseline and the check cannot disagree.
4+
*
5+
* THIS FILE IS A TO-DO LIST, NOT A SET OF EXEMPTIONS. The gate ratchets on it:
6+
* - a finding NOT listed here fails the run, so a regression cannot land;
7+
* - a listed finding whose ratio gets WORSE fails, so the list cannot rot;
8+
* - a listed finding that no longer appears ALSO fails, so a fixed entry must
9+
* be deleted and the file can only shrink toward empty.
10+
* The last rule is what stops an allowlist becoming a permanent exemption.
11+
*
12+
* `unverified: true` marks an absolutely-positioned pseudo-element. It can paint
13+
* outside its host and the oracle measures it against the host's backdrop, so
14+
* that ratio is NOT trustworthy — hand-measure before acting on it.
15+
*/
16+
export const NONTEXT_BASELINE: Record<
17+
string,
18+
{ ratio: number; required: number; unverified: boolean }
19+
> = {
20+
"control-boundary|button#cl-theme-toggle.cl-btn.cl-icon": { ratio: 1.47, required: 3.0, unverified: false },
21+
"control-boundary|button#compute-mac-button.action-button": { ratio: 2.32, required: 3.0, unverified: false },
22+
"control-boundary|button#generate-key-button.action-button": { ratio: 2.5, required: 3.0, unverified: false },
23+
"control-boundary|button#show-math-button.action-button": { ratio: 2.54, required: 3.0, unverified: false },
24+
"control-boundary|button#verify-message-button.action-button": { ratio: 2.32, required: 3.0, unverified: false },
25+
"control-boundary|button#verify-original-button.action-button": { ratio: 2.32, required: 3.0, unverified: false },
26+
"control-boundary|button#verify-tag-button.action-button": { ratio: 2.32, required: 3.0, unverified: false }
27+
};

0 commit comments

Comments
 (0)