Skip to content

Commit d1bf5c0

Browse files
committed
CI: retry testem ci once on failure
The Browserstack job occasionally fails on transient browser-side network errors (e.g. Safari `TypeError: Load failed` seen in PR #32 run 24655116088). Those failures come from the BrowserStack tunnel, not from code under test. On first failure, re-run the whole suite once; if that also fails the original exit code propagates and the job fails normally. Tradeoff: masks flaky tests with <=50% failure rate. Acceptable here because the observed browserstack-only flakes are infra, not product. For product flakiness we'd want per-test retry, which testem does not provide natively. Rejected alternatives: - `retry_count: 1` in testem config — not a real testem option (verified against testem 3.19.1 source and docs); silently ignored. - `.catch()` on `ember browserstack:results` / `:disconnect` — `process.exit(0)` on the success path skips the finally entirely, so the cleanup `.catch()` only fires when testem already failed. The linked PR #32 failure was a real test error, not a cleanup failure; the `.catch()` would not have changed the outcome.
1 parent 2297ebc commit d1bf5c0

1 file changed

Lines changed: 13 additions & 2 deletions

File tree

bin/run-browserstack-tests.js

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,15 +17,26 @@ async function run(command, args = []) {
1717
// Calling testem directly here instead of `ember test` so that
1818
// we do not have to do a double build (by the time this is run
1919
// we have already ran `ember build`).
20-
await run('testem', [
20+
const testemArgs = [
2121
'ci',
2222
'-f',
2323
'testem.browserstack.js',
2424
'--host',
2525
'127.0.0.1',
2626
'--port',
2727
'7774',
28-
]);
28+
];
29+
30+
// One whole-suite retry to absorb transient BrowserStack network
31+
// failures (e.g. Safari `TypeError: Load failed`). Masks flaky
32+
// tests with <=50% failure rate; acceptable for this browserstack
33+
// job because observed flakes are infra, not product.
34+
try {
35+
await run('testem', testemArgs);
36+
} catch (e) {
37+
console.log(chalk.yellow('testem ci failed; retrying once.'));
38+
await run('testem', testemArgs);
39+
}
2940

3041
console.log('success');
3142
process.exit(0); // eslint-disable-line n/no-process-exit

0 commit comments

Comments
 (0)