Test flakiness from main checkout

[cefn]

If I run a test command multiple times consecutively on my Arm64 Macbook it seems to pass or fail somewhat at random, probably owing to a race condition in the completion of some parallel task.

If you keep re-running the tests, it normally fails. Typically after waiting a few minutes and re-running, it normally passes again, although I don't know how deterministic this is.

clear; git clean -fdx; git status; git log | head -n 1 ; npm ci; npm run test

In practice this complicates raising a PR which is sure to pass tests (or given husky integration can't even commit changed files without --no-verify).

I'm unsure what is the origin of the failure, but in general not being able to run the test suite twice from a clean checkout suggests a problem. I originally observed this while trying to run tests as part of PR work (e.g. on my own branch which had changes in it) and only eventually fell back to asking the question does this even work reliably on main?

Failure Example

At the time of writing main is at e3f2370982fdd22fbf6d68ab8c9690f3a767f834 as matched in the logged output below.

[sort-package-json-upstream](main)$ clear; git clean -fdx; git status; git log | head -n 1 ; npm ci; npm run test
Removing .husky/_/
Removing node_modules/
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean
commit e3f2370982fdd22fbf6d68ab8c9690f3a767f834
npm warn deprecated semver-diff@5.0.0: Deprecated as the semver package now supports this built-in.

> sort-package-json@0.0.0-development prepare
> husky


added 646 packages, and audited 812 packages in 8s

181 packages are looking for funding
  run `npm fund` for details

7 vulnerabilities (3 low, 2 moderate, 2 high)

To address all issues, run:
  npm audit fix

Run `npm audit` for details.

> sort-package-json@0.0.0-development test
> ava && tstyche


  ✔ dependency-sorting › npm
  ✔ dependency-sorting › pnpm
  ✔ dependency-sorting › yarn
  ✔ options › options.sortOrder
  ✔ options › options.sortOrder prepend
  ✔ options › options.sortOrder effect
  ✔ options › options.sortOrder function
  ✔ options › options.sortOrder with private key
  ✔ scripts › scripts when npm-run-all is NOT a dev dependency
  ✔ scripts › scripts when npm-run-all IS a dev dependency, but is NOT used in scripts
  ✔ scripts › betterScripts when npm-run-all is NOT a dev dependency
  ✔ scripts › betterScripts when npm-run-all IS a dev dependency, but is NOT used in scripts
  ✔ scripts › command: 'run-s "lint:*"'
  ✔ scripts › command: 'run-s "lint:*"' with npm-run-all2
  ✔ scripts › command: 'npm-run-all -s "lint:*"'
  ✔ scripts › command: 'npm-run-all -s "lint:*"' with npm-run-all2
  ✔ scripts › command: 'npm-run-all --sequential "lint:*"'
  ✔ scripts › command: 'npm-run-all --sequential "lint:*"' with npm-run-all2
  ✔ scripts › command: 'npm-run-all --serial "lint:*"'
  ✔ scripts › command: 'npm-run-all --serial "lint:*"' with npm-run-all2
  ✔ scripts › command: 'npm-run-all "lint:*" --sequential'
  ✔ scripts › command: 'npm-run-all "lint:*" --sequential' with npm-run-all2
  ✔ scripts › command: 'foo&&npm-run-all --serial "lint:*"'
  ✔ scripts › command: 'foo&&npm-run-all --serial "lint:*"' with npm-run-all2
  ✔ scripts › command: 'foo||npm-run-all --serial "lint:*"'
  ✔ scripts › command: 'foo||npm-run-all --serial "lint:*"' with npm-run-all2
  ✔ scripts › command: 'foo|npm-run-all --serial "lint:*"'
  ✔ scripts › command: 'foo|npm-run-all --serial "lint:*"' with npm-run-all2
  ✔ scripts › command: 'foo>npm-run-all --serial "lint:*"'
  ✔ scripts › command: 'foo>npm-run-all --serial "lint:*"' with npm-run-all2
  ✔ scripts › command: 'foo<npm-run-all --serial "lint:*"'
  ✔ scripts › command: 'foo<npm-run-all --serial "lint:*"' with npm-run-all2
  ✔ scripts › command: 'cross-env FOO=1 npm-run-all --serial "lint:*"'
  ✔ scripts › command: 'cross-env FOO=1 npm-run-all --serial "lint:*"' with npm-run-all2
  ✔ scripts › command: 'npm-run-all "lint:*" --serial&&foo'
  ✔ scripts › command: 'npm-run-all "lint:*" --serial&&foo' with npm-run-all2
  ✔ scripts › command: 'npm-run-all "lint:*" --serial|foo'
  ✔ scripts › command: 'npm-run-all "lint:*" --serial|foo' with npm-run-all2
  ✔ scripts › command: 'npm-run-all "lint:*" --serial||foo'
  ✔ scripts › command: 'npm-run-all "lint:*" --serial||foo' with npm-run-all2
  ✔ scripts › command: 'npm-run-all "lint:*" --serial>foo'
  ✔ scripts › command: 'npm-run-all "lint:*" --serial>foo' with npm-run-all2
  ✔ scripts › command: 'npm-run-all "lint:*" --serial<foo'
  ✔ scripts › command: 'npm-run-all "lint:*" --serial<foo' with npm-run-all2
  ✔ scripts › command: 'npm-run-all --serial "lint:*"&&foo'
  ✔ scripts › command: 'npm-run-all --serial "lint:*"&&foo' with npm-run-all2
  ✔ scripts › command: 'npm-run-all "lint:*" --serial;foo'
  ✔ scripts › command: 'npm-run-all "lint:*" --serial;foo' with npm-run-all2
  ✔ scripts › command: '(npm-run-all "lint:*" --serial)|foo'
  ✔ scripts › command: '(npm-run-all "lint:*" --serial)|foo' with npm-run-all2
  ✔ scripts › command: 'run-s lint:a lint:b'
  ✔ scripts › command: 'run-s lint:a lint:b' with npm-run-all2
  ✔ scripts › command: 'not-run-s *'
  ✔ scripts › command: 'not-run-s *' with npm-run-all2
  ✔ scripts › command: 'npm-run-all * --serial!'
  ✔ scripts › command: 'npm-run-all * --serial!' with npm-run-all2
  ✔ scripts › command: 'looks like && run-s-but-its-not *'
  ✔ scripts › command: 'looks like && run-s-but-its-not *' with npm-run-all2
  ✔ scripts › command: 'npm-run-all *'
  ✔ scripts › command: 'npm-run-all *' with npm-run-all2
  ✔ scripts › command: 'npm-run-all --parallel watch:*'
  ✔ scripts › command: 'npm-run-all --parallel watch:*' with npm-run-all2
  ✔ scripts › command: 'rm -rf dist/* && run-s lint:a lint:b'
  ✔ scripts › command: 'rm -rf dist/* && run-s lint:a lint:b' with npm-run-all2
  ✔ scripts › scripts when npm-run-all2 is not a dev dependency
  ✔ scripts › scripts when npm-run-all2 is a dev dependency
  ✔ scripts › betterScripts when npm-run-all2 is not a dev dependency
  ✔ scripts › betterScripts when npm-run-all2 is a dev dependency
  ✔ scripts › command: 'npm-run-all2 -s "lint:*"' with npm-run-all2
  ✔ scripts › scripts does not sort pre/post scripts with colon together
  ✔ scripts › betterScripts does not sort pre/post scripts with colon together
  ✔ scripts › scripts sort pre/post scripts together with base script independent of colon in name
  ✔ scripts › betterScripts sort pre/post scripts together with base script independent of colon in name
  ✔ scripts › scripts: group base and colon scripts together, do not split with unrelated
  ✔ scripts › scripts: handles names starting with colon and double colons
  ✔ scripts › scripts: group scripts with multiple colons
  ✔ scripts › scripts: nested production and format variants are grouped and sorted
  ✔ main › main
  ✔ main › default sortOrder
  ✔ main › private keys
  ✔ prettier › prettier
  ✔ prettier › prettier.overrides[]
  ✔ prettier › prettier.overrides[].options
  ✔ deps › resolutions
  ✔ deps › dependencies
  ✔ deps › devDependencies
  ✔ deps › peerDependencies
  ✔ deps › optionalDependencies
  ✔ deps › bundledDependencies
  ✔ deps › bundleDependencies
  ✔ deps › extensionPack
  ✔ deps › extensionDependencies
  ✔ deps › peerDependenciesMeta
  ✔ deps › dependenciesMeta
  ✔ deps › dependenciesMetaRange
  ✔ deps › pnpm.overrides
  ✔ deps › dependencies with capital and lowercase letters
  ✔ deps › devDependencies with capital and lowercase letters
  ✔ exports › 'types' condition should be first
  ✔ exports › 'types' condition should be first 2
  ✔ exports › 'default' condition should be last
  ✔ exports › 'types' condition should be first(deep)
  ✔ exports › 'types' condition should be first(deep) 2
  ✔ exports › 'default' condition should be last(deep)
  ✔ exports › Only 'types'
  ✔ exports › Only 'default'
  ✔ exports › Well formed
  ✔ fields › $schema
  ✔ fields › name
  ✔ fields › displayName
  ✔ fields › version
  ✔ fields › description
  ✔ fields › sideEffects
  ✔ fields › qna
  ✔ fields › publisher
  ✔ fields › type
  ✔ fields › main
  ✔ fields › svelte
  ✔ fields › umd:main
  ✔ fields › jsdelivr
  ✔ fields › unpkg
  ✔ fields › module
  ✔ fields › source
  ✔ fields › jsnext:main
  ✔ fields › browser
  ✔ fields › react-native
  ✔ fields › types
  ✔ fields › typesVersions
  ✔ fields › typings
  ✔ fields › style
  ✔ fields › example
  ✔ fields › examplestyle
  ✔ fields › assets
  ✔ fields › man
  ✔ fields › pre-commit
  ✔ fields › browserslist
  ✔ fields › stylelint
  ✔ fields › flat
  ✔ fields › packageManager
  ✔ fields › os
  ✔ fields › cpu
  ✔ fields › icon
  ✔ fields › preview
  ✔ fields › markdown
  ✔ fields › bin
  ✔ fields › contributes
  ✔ fields › commitlint
  ✔ fields › config
  ✔ fields › nodemonConfig
  ✔ fields › browserify
  ✔ fields › babel
  ✔ fields › xo
  ✔ fields › ava
  ✔ fields › jest
  ✔ fields › mocha
  ✔ fields › nyc
  ✔ fields › c8
  ✔ fields › engines
  ✔ fields › engineStrict
  ✔ fields › preferGlobal
  ✔ fields › publishConfig
  ✔ fields › galleryBanner
  ✔ fields › remarkConfig
  ✔ fields › release
  ✔ fields › npmpkgjsonlint
  ✔ fields › npmPackageJsonLintConfig
  ✔ fields › npmpackagejsonlint
  ✔ fields › keywords
  ✔ fields › files
  ✔ fields › activationEvents
  ✔ fields › categories
  ✔ fields › husky
  ✔ fields › binary
  ✔ fields › bugs
  ✔ fields › repository
  ✔ fields › funding
  ✔ fields › license
  ✔ fields › author
  ✔ fields › directories
  ✔ fields › volta
  ✔ fields › contributors
  ✔ fields › badges
  ✔ fields › pnpm
  ✔ eslint › eslintIgnore
  ✔ eslint › eslintConfig
  ✔ eslint › eslintConfig.rules
  ✔ eslint › eslintConfig.env
  ✔ eslint › eslintConfig.globals
  ✔ eslint › eslintConfig.parserOptions
  ✔ eslint › eslintConfig.settings
  ✔ eslint › eslintConfig.parser
  ✔ eslint › eslintConfig.plugins
  ✔ eslint › eslintConfig.extends
  ✔ eslint › eslintConfig.processor
  ✔ eslint › eslintConfig.noInlineConfig
  ✔ eslint › eslintConfig.reportUnusedDisableDirectives
  ✔ eslint › eslintConfig.override[]
  ✔ cli › cli (557ms)
  ✔ cli › run `cli --version` with other arguments (456ms)
  ✔ cli › run `cli -v` (473ms)
  ✔ cli › run `cli --version --quiet` (484ms)
  ✔ cli › run `cli --version` (495ms)
  ✔ cli › run `cli --help` with other arguments (504ms)
  ✔ cli › run `cli -h` (516ms)
  ✔ cli › run `cli --help --quiet` (552ms)
  ✔ cli › run `cli --help` (578ms)
  ✔ cli › run `cli --help` with `--version` (474ms)
  ✔ cli › run `cli --help=value` (523ms)
  ✔ white-space › white space
  ✔ workspaces › workspaces object should be sorted
  ✔ workspaces › workspaces with other types should be kept as it is
  ✔ cli › run `cli --version=true` (585ms)
  ✔ cli › run `cli --unknown-option` (619ms)
  ✔ cli › run `cli -u` with unknown option (655ms)
  ✔ cli › run `cli --no-version` (700ms)
  ✔ cli › run `cli` with no patterns (748ms)
  ✔ cli › run `cli --quiet` with no patterns (788ms)
  ✔ cli › run `cli --quiet=value` (821ms)
  ✔ cli › run `cli -q` with no patterns (866ms)
  ✔ cli › run `cli --check` with no patterns (906ms)
  ✔ cli › run `cli --check=value` (935ms)
  ✔ cli › run `cli --check --quiet` with no patterns (992ms)
  ✔ cli › run `cli -c` with no patterns (1s)
  ✔ cli › run `cli -c -q` with no patterns (1s)
  ✔ cli › run `cli -cq` with no patterns (1.1s)
  ✔ cli › run `cli` on 1 bad file (1.1s)
  ✔ cli › run `cli --quiet` on 1 bad file (1.1s)
  ✔ cli › run `cli --check` on 1 bad file (1.2s)
  ✔ cli › run `cli --check --quiet` on 1 bad file (1.2s)
  ✔ cli › run `cli` on 2 bad files (1.3s)
  ✘ [fail]: cli › run `cli --check` on 2 bad files Should report 2 files.
  ✔ cli › run `cli --quiet` on 2 bad files (1.3s)
  ✔ cli › run `cli --check --quiet` on 2 bad files (1.4s)
  ✔ cli › run `cli` on 2 good files and 2 bad files (1.4s)
  ✔ cli › run `cli --quiet` on 2 good files and 2 bad files (1.5s)
  ✔ cli › run `cli --check` on 2 good files and 2 bad files (1.5s)
  ✔ cli › run `cli --check --quiet` on 2 good files and 2 bad files (1.5s)
  ✔ cli › run `cli` on none exists file (1.5s)
  ✔ cli › run `cli --quiet` on none exists file (1.6s)
  ✔ cli › run `cli --check` on none exists file (1.6s)
  ✔ cli › run `cli --check --quiet` on none exists file (1.7s)
  ✔ cli › run `cli` on duplicate patterns (1.7s)
  ✔ cli › run `cli --check` on duplicate patterns (1.8s)
  ✔ cli › run `cli --check --quiet` on duplicate patterns (1.8s)
  ✔ cli › run `cli --check` on 1 non-json file (1.9s)
  ✔ cli › run `cli --check --quiet` on 1 non-json file (1.9s)
  ✔ cli › run `cli` on 1 non-json file (2s)
  ✔ cli › run `cli --quiet` on 1 non-json file (2s)
  ✔ cli › run `cli --stdin` with input from stdin (2s)
  ✔ cli › run `cli --stdin` with input from stdin with \r\n (2.1s)
  ✔ cli › run `cli --ignore=abc` (2.1s)
  ─

  cli › run `cli --check` on 2 bad files

  Should report 2 files.

  Difference (- actual, + expected):

    {
      args: [
        '*/package.json',
        '--check',
      ],
      fixtures: [
        {
          expect: `{␊
            "version": "1.0.0",␊
            "name": "sort-package-json"␊
          }`,
          file: 'bad-1/package.json',
          original: `{␊
            "version": "1.0.0",␊
            "name": "sort-package-json"␊
          }`,
        },
        {
          expect: `{␊
            "version": "1.0.0",␊
            "name": "sort-package-json"␊
          }`,
          file: 'bad-2/package.json',
          original: `{␊
            "version": "1.0.0",␊
            "name": "sort-package-json"␊
          }`,
        },
      ],
      result: {
        errorCode: 1,
        stderr: '',
  -     stdout: `bad-2/package.json␊
  -     bad-1/package.json␊
  +     stdout: `bad-1/package.json␊
  +     bad-2/package.json␊
        ␊
        Found 2 files.␊
        2 files were not sorted.␊
        `,
      },
    }

  Error
      at ExecutionContext.snapshot (file:///Users/choile/workspace/github/sort-package-json-upstream/node_modules/ava/lib/assert.js:674:16)
      at testCLI (file:///Users/choile/workspace/github/sort-package-json-upstream/tests/_helpers.js:202:5)
      at process.processTicksAndRejections (node:internal/process/task_queues:105:5)

  ─

  1 test failed

find /private/var/folders/ -type f -path '**/bad*/*' -name 'package.json' 2>&1 | grep package.json | xargs -n1 echo rm

[cefn]

A workaround when trying to construct pipeline-aligned snapshots (assuming you're not working on the cli ones) is to...

• Delete snapshot folder
• Rerun tests to generate snapshot folder (including possibly an incorrect file-reporting order in the cli tests snapshot)
• Checkout cli.js.* from origin/main again (which has the correct file-reporting order you may be unable to replicate locally)
• Run git add . && git commit --no-verify --amend && git push --force-with-lease to skip the husky problem that it assumes you can replicate locally

[keithamus]

Seems like this is also verifiable in CI, as #394 recently failed on Node 24.

I don't have time to look into this but if someone can that would be great!

[fisker]

I think it's caused by the unstable order of glob result.