feat: add max-auto-reruns parameter to run-tests command and run job#577
Open
esetnik wants to merge 4 commits intocypress-io:masterfrom
Open
feat: add max-auto-reruns parameter to run-tests command and run job#577esetnik wants to merge 4 commits intocypress-io:masterfrom
esetnik wants to merge 4 commits intocypress-io:masterfrom
Conversation
Threads CircleCI's step-level `max_auto_reruns` and `auto_rerun_delay` through to the internal "Run Cypress" run step so consumers can scope auto-retry to the Cypress invocation itself, instead of applying workflow-level `max_auto_reruns` (which retries every failed job in the workflow, including non-flaky ones like lint/typecheck). Both parameters default to their disabled values (0 reruns, 0s delay), so behavior is unchanged for callers who don't set them. Reference: https://circleci.com/docs/guides/orchestrate/automatic-reruns/ Note for reviewers: the local CircleCI CLI's orb validator (0.1.34950) does not yet recognize step-level `max_auto_reruns` and `auto_rerun_delay` keys on `run` steps and reports them as "extraneous". The runtime accepts them — the docs page above shows the exact syntax used here. The orb-tools/validate job in CI may report the same false positive until the CLI schema is updated.
|
|
Collaborator
|
The CircleCI orb validator pre-checks the value of `auto_rerun_delay` against `^((10|[1-9])m|([1-9][0-9]*)s)$` *before* substituting the parameter expression, so any `<< parameters.x >>` placeholder fails validation. Reverting to the runtime default (0s, no delay) keeps the integer max_auto_reruns parameter working without breaking orb-tools/pack. If users need a custom delay later, the orb can hardcode a default or wait for the CLI validator to defer regex checks until after parameter substitution.
The orb validator's value-schema for max_auto_reruns requires the field to be >= 1. Threading `max_auto_reruns: << parameters.max-auto-reruns >>` unconditionally meant test-deploy.yml's `cypress/run` instantiations substituted in the parameter default of 0, which then failed the "0 is not greater or equal to 1" check on the integration tests. Fork the Run Cypress step into two variants: one without the field (when max-auto-reruns is 0/unset) and one with it (when 1+). Only one runs per job. Behavior is unchanged for callers who don't set the parameter.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
max-auto-rerunsparameter to both thecypress/run-testscommand andcypress/runjob. The parameter is threaded through to CircleCI's step-levelmax_auto_rerunson the internal "Run Cypress" run step.Motivation
Today the only way for an orb consumer to get auto-retry on flaky Cypress runs is to apply
max_auto_rerunsat the workflow level, which retries every failed job in the workflow — including non-flaky ones like lint/typecheck/spellcheck — and burns CI minutes on real failures that would never recover.Step-level
max_auto_rerunslets you scope the retry budget to just the Cypress invocation, where it's actually needed (Vite startup races, browser crashes, transient network blips, etc.). Combined with Cypress's per-testretries: { runMode: N }, you get layered flake protection without affecting other jobs.Behavior
max-auto-rerunsdefaults to0(disabled), so callers who don't set it see no behavior change.What was dropped from the original draft
The first commit also added an
auto-rerun-delayparameter, butorb-tools/packrejected it: the CircleCI orb validator pre-checks the time-format regex (^((10|[1-9])m|([1-9][0-9]*)s)$) against the literal<< parameters.auto-rerun-delay >>string before substituting the parameter, which always fails. Integer parameters likemax_auto_rerunsaren't affected because they're typedintegerand skip the pattern step.I dropped
auto-rerun-delayso this PR doesn't depend on a CLI fix. Reruns happen with the runtime's default0sdelay; if a delay becomes important later, the orb can hardcode one or wait for the upstream CLI fix tracked at CircleCI-Public/circleci-cli#1202.Test plan
circleci orb pack ./src && circleci orb validate orb.ymlpasses locallyorb-tools/packandorb-tools/lintpass in CIdev:build, validate end-to-end against a real Cypress job by triggering a flaky run and confirming the step retries (will be done from a downstream consumer pipeline)