feat: add automated dependency bump checker and changelog validator#267
feat: add automated dependency bump checker and changelog validator#267cryptodev-2s wants to merge 24 commits intomainfrom
Conversation
a6b2b18 to
cd10640
Compare
16089f8 to
3d0c830
Compare
- Rename defaultBranch to baseBranch (more accurate terminology)
- Add --remote option (default: 'origin') for customizing remote name
- Default base branch is now ${remote}/main instead of local 'main'
- Remove misleading fallback logic in getMergeBase that could give wrong results
- Rename getGitDiff to getManifestGitDiff for clarity
- Remove unnecessary try/catch in getManifestGitDiff (git diff returns 0 for no changes)
63df851 to
5804a00
Compare
|
@metamaskbot publish-preview |
|
A preview build for this branch has been published. You can configure your project to use the preview build with this identifier: See these instructions for more information about preview builds. |
…angelog validation - Integrate dependency checking into validate command with --checkDeps flag - Add new CLI options: --fromRef, --toRef, --remote, --baseBranch, --currentPr - Auto-detect git reference from merge-base when --fromRef not provided - Validate dependency/peerDependency bumps with exact version matching - Auto-fix missing entries with --fix flag (requires --currentPr) - Support renamed packages via existing --tagPrefixBeforePackageRename option - Add comprehensive unit tests for parseDependencyDiff and getDependencyChangesForPackage - Add unit tests for updateChangelogWithDependencies - Add dependency validation tests for validateChangelog - Refactor parseDependencyDiff to be internal (not exported) - Update CHANGELOG.md with new feature documentation
2e6df05 to
57a884e
Compare
- Rename DiffParseResult to DependencyCheckResult - Pass dependencyResult object directly to validateChangelog - Fix validation to check correct changelog section (Unreleased vs version)
|
@metamaskbot publish-preview |
|
A preview build for this branch has been published. You can configure your project to use the preview build with this identifier: See these instructions for more information about preview builds. |
|
@cursor review |
| ); | ||
| } catch { | ||
| return null; | ||
| } |
There was a problem hiding this comment.
git show errors treated as missing file
Medium Severity
getFileAtRef catches all failures and returns null, and the caller treats oldContent === null as “new package”. This can silently skip dependency bump validation when git show fails for unrelated reasons (bad ref, permissions, repo state), producing false passes and missing required changelog entries.
Additional Locations (1)
|
|
||
| // Any-version match (same dep + type, different versions) | ||
| return { hasExactMatch: false, existingEntry: entry, entryIndex: i }; | ||
| } |
There was a problem hiding this comment.
Dependency entry matcher stops at first match
Medium Severity
hasChangelogEntry returns the first same-dependency/same-type entry as an “any-version match”, without scanning the rest of the list for an exact match. If a changelog contains multiple entries for the same dependency/type (e.g., stale then corrected), validation and auto-fix can target the wrong entry or report a missing entry incorrectly.
| if (prNumbers !== undefined) { | ||
| change.prNumbers = prNumbers; | ||
| } | ||
| } |
There was a problem hiding this comment.
updateChange can’t remove dependencyBump
Low Severity
updateChange only updates dependencyBump when dependencyBump !== undefined, so callers can’t clear an existing change.dependencyBump. This can leave stale structured metadata attached to an entry even after switching it back to a manual description, causing stringification/validation to keep treating it as a dependency bump.
src/validate-cli.test.ts
Outdated
|
|
||
| describe('validate', () => { | ||
| beforeEach(() => { | ||
| jest.resetAllMocks(); |
There was a problem hiding this comment.
Is this needed? I believe we've already configured Jest to do this by default:
| jest.resetAllMocks(); |
There was a problem hiding this comment.
Done, removed (already configured globally).
src/validate-changelog.test.ts
Outdated
| const repoUrl = | ||
| 'https://github.com/ExampleUsernameOrOrganization/ExampleRepository'; | ||
|
|
||
| it('passes when dependency changes are found in changelog', async () => { |
There was a problem hiding this comment.
We seem to be using should to name tests in this file. Should we match the same style?
Also, just a nit:
| it('passes when dependency changes are found in changelog', async () => { | |
| it('should pass when expected dependency changes are found in changelog', async () => { |
There was a problem hiding this comment.
Done, all dependency test names now use should prefix.
src/validate-cli.test.ts
Outdated
| const actual = jest.requireActual('./validate-changelog'); | ||
| return { | ||
| ...actual, | ||
| validateChangelog: jest.fn(), |
There was a problem hiding this comment.
Should we really be mocking this function or any of these other functions? This is basically the highest file that we have in the test hierarchy (as there are no tests for cli.ts directly). So if we mock out the underlying functions then we have no confirmation that the validate command calls validateChangelog correctly.
I would think that we would mock things like commands (because those are difficult to simulate otherwise) but for file access or the other logic we are testing, is there a way that we can not mock that?
There was a problem hiding this comment.
Done, rewrote to only mock I/O (fs) and git (get-dependency-changes), letting validateChangelog, generateDiff, and updateChangelogWithDependencies run for real.
src/validate-changelog.test.ts
Outdated
| ).rejects.toThrow('Missing changelog entries for dependency bumps'); | ||
| }); | ||
|
|
||
| it('validates peerDependency entry with BREAKING prefix', async () => { |
There was a problem hiding this comment.
The next test also validates the peerDependency, should we be more clear about what this test does?
| it('validates peerDependency entry with BREAKING prefix', async () => { | |
| it('passes when peer dependency changes are found in changelog', async () => { |
src/validate-changelog.ts
Outdated
| // Check in the appropriate section: use versionBump if provided, otherwise Unreleased | ||
| const changesSection = dependencyResult.versionBump | ||
| ? changelog.getReleaseChanges(dependencyResult.versionBump) | ||
| : changelog.getUnreleasedChanges(); | ||
|
|
There was a problem hiding this comment.
validateChangelog already takes isReleaseCandidate and currentVersion option which indicates whether auto-changelog is being run on a release branch and what the version-to-release is. Additionally, validateChangelog sets releaseChanges at the top of the function. Currently this variable is undefined if currentVersion is not provided, however, it only seems to be used in the case where currentVersion is provided.
So, could we change releaseChanges to be what you've set changesSection to here, and then remove changesSection?
| // Check in the appropriate section: use versionBump if provided, otherwise Unreleased | |
| const changesSection = dependencyResult.versionBump | |
| ? changelog.getReleaseChanges(dependencyResult.versionBump) | |
| : changelog.getUnreleasedChanges(); |
There was a problem hiding this comment.
Refactored to use dependencyCheckResult.versionChanged instead of isReleaseCandidate to determine which section to check. This correctly handles packages on a release branch that are not themselves being released.
src/check-dependency-bumps.ts
Outdated
| * Minimal type for the parts of package.json we care about. | ||
| */ | ||
| type PackageJson = { | ||
| version?: string; |
There was a problem hiding this comment.
Nit: I think we can assume this is always set:
| version?: string; | |
| version: string; |
There was a problem hiding this comment.
Done, changed to version: string.
src/check-dependency-bumps.ts
Outdated
| } | ||
| } | ||
|
|
||
| const versionBump = |
There was a problem hiding this comment.
I've made a comment about whether this property is necessary, but if we do need this, it's a bit confusing at the moment what it does just based on the name. Could you explain how this is used or why it's needed?
There was a problem hiding this comment.
Replaced with versionChanged: boolean on DependencyCheckResult. It indicates whether the package's own version changed between the two refs, used to determine whether dependency entries go in the release section or Unreleased.
src/check-dependency-bumps.ts
Outdated
| absolutePath: string, | ||
| projectRoot: string, | ||
| ): Promise<string> { | ||
| const topLevel = await getStdoutFromCommand( |
There was a problem hiding this comment.
If we know the project root can we use path.relative to accomplish this instead of running git again?
There was a problem hiding this comment.
Done, replaced with synchronous path.relative.
mcmire
left a comment
There was a problem hiding this comment.
I looked through this PR and made some comments as you can see. I did another round but considering there are a lot I will stop for now and let you process these.
src/changelog.ts
Outdated
| } | ||
|
|
||
| // Auto-generate description from dependencyBump when not provided | ||
| let effectiveDescription = description ?? ''; |
There was a problem hiding this comment.
Nit: This line seems to allow description to be an empty string. That is corrected in the next line, but it could be misleading.
What are your thoughts on changing AddChangeOptions to the following?
type AddChangeOptions = {
addToStart?: boolean;
category: ChangeCategory;
version?: Version;
prNumbers?: string[];
} & (
| {
description: string;
}
| {
description?: string;
dependencyBump: DependencyBump;
}
);Then you could change the beginning of this function to:
const {
addToStart = true,
category,
description,
version,
prNumbers = [],
} = options;
if (!category) {
throw new Error('Category required');
} else if (!orderedChangeCategories.includes(category)) {
throw new Error(`Unrecognized category: '${category}'`);
} else if (!description && !('dependencyBump' in options)) {
throw new Error('Description required');
} else if (version !== undefined && !this.#changes[version]) {
throw new Error(`Specified release version does not exist: '${version}'`);
}
const effectiveDescription =
'dependencyBump' in options
? (options.description ??
formatDependencyBumpDescription(options.dependencyBump))
: options.description;
const release = version
? this.#changes[version]
: this.#changes[unreleased];
const releaseCategory = release[category] ?? [];
const change: Change = {
description: effectiveDescription,
prNumbers,
};
if ('dependencyBump' in options) {
change.dependencyBump = options.dependencyBump;
}There was a problem hiding this comment.
Done per your suggestion: two-variant union with 'dependencyBump' in options narrowing. Avoided as string by using an if/else that lets TypeScript narrow naturally.
src/changelog.ts
Outdated
| description, | ||
| prNumbers, | ||
| dependencyBump, | ||
| }: { |
There was a problem hiding this comment.
Nit: Since we have types to represent the options for other methods, should we extract these options to a type too?
There was a problem hiding this comment.
Done, extracted as a flat type with all optional fields since update operations allow partial updates (e.g. only prNumbers).
src/changelog.ts
Outdated
|
|
||
| if (!release) { | ||
| throw new Error( | ||
| `Specified release version does not exist: '${String(version)}'`, |
There was a problem hiding this comment.
Do we want to account for when version is not provided? Otherwise this would say Specified release version does not exist: ''
| `Specified release version does not exist: '${String(version)}'`, | |
| `Could not find ${version ? `release: '${version}'` : `'${unreleased}' section`}`, |
There was a problem hiding this comment.
Done, shows Could not find 'Unreleased' section when version is undefined.
| const releaseCategory = release[category]; | ||
| if ( | ||
| !releaseCategory || | ||
| entryIndex < 0 || | ||
| entryIndex >= releaseCategory.length | ||
| ) { | ||
| throw new Error( | ||
| `No change at index ${entryIndex} in category ${category}`, | ||
| ); | ||
| } | ||
|
|
||
| const change = releaseCategory[entryIndex]; |
There was a problem hiding this comment.
Thoughts on splitting this into two?
| const releaseCategory = release[category]; | |
| if ( | |
| !releaseCategory || | |
| entryIndex < 0 || | |
| entryIndex >= releaseCategory.length | |
| ) { | |
| throw new Error( | |
| `No change at index ${entryIndex} in category ${category}`, | |
| ); | |
| } | |
| const change = releaseCategory[entryIndex]; | |
| const releaseCategory = release[category]; | |
| if (!releaseCategory) { | |
| throw new Error( | |
| `Could not find category '${category}' in release section '${version ?? unreleased}'`, | |
| ); | |
| } | |
| const change = releaseCategory[entryIndex]; | |
| if (!change) { | |
| throw new Error( | |
| `Could not find change at index ${entryIndex} in category '${category}' in release section '${version ?? unreleased}'`, | |
| ); | |
| } | |
There was a problem hiding this comment.
Done, separate errors for missing category and out-of-bounds index.
| if (dependencyBump !== undefined) { | ||
| change.dependencyBump = dependencyBump; | ||
| // Auto-update the description to match the new dependencyBump data. | ||
| // Note: if both description and dependencyBump are provided, | ||
| // dependencyBump takes precedence for the description. | ||
| change.description = formatDependencyBumpDescription(dependencyBump); | ||
| } else if (description !== undefined) { | ||
| change.description = description; | ||
| } |
There was a problem hiding this comment.
Since addChange will prioritize description if it's given, maybe it makes sense for updateChange to do a similar thing?
| if (dependencyBump !== undefined) { | |
| change.dependencyBump = dependencyBump; | |
| // Auto-update the description to match the new dependencyBump data. | |
| // Note: if both description and dependencyBump are provided, | |
| // dependencyBump takes precedence for the description. | |
| change.description = formatDependencyBumpDescription(dependencyBump); | |
| } else if (description !== undefined) { | |
| change.description = description; | |
| } | |
| if (description !== undefined) { | |
| change.description = description; | |
| } else if (dependencyBump !== undefined) { | |
| change.dependencyBump = dependencyBump; | |
| change.description = formatDependencyBumpDescription(dependencyBump); | |
| } |
There was a problem hiding this comment.
Done, description now takes precedence over dependencyBump, matching addChange.
src/changelog.ts
Outdated
| version?: Version; | ||
| category: ChangeCategory; | ||
| entryIndex: number; | ||
| description?: string; |
There was a problem hiding this comment.
Similar to the suggestion I made for addChange, should the options for this method be a union? And should we have a similar check to confirm that if dependencyBump is not given then description is required?
There was a problem hiding this comment.
Kept as a flat type since updates are partial (you may update only prNumbers without touching description or dependencyBump).
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
| change.description = formatDependencyBumpDescription(dependencyBump); | ||
| } else if (description !== undefined) { | ||
| change.description = description; | ||
| } |
There was a problem hiding this comment.
Inconsistent description/dependencyBump priority between addChange and updateChange
Medium Severity
addChange and updateChange handle the description/dependencyBump priority differently. In addChange, when both are provided, the explicit description is preserved and dependencyBump only auto-generates when description is absent. In updateChange, dependencyBump always overrides any provided description, silently discarding it. This asymmetric behavior means callers passing both parameters will get different results depending on which method they use, which is a confusing API contract.
Additional Locations (1)
There was a problem hiding this comment.
Already consistent, both addChange and updateChange prioritize description over dependencyBump. No change needed.
- Rename types/files per review (DependencyBump.isBreaking, findChangelogEntry, validate-command.ts, get-dependency-changes.ts, etc.) - Make AddChangeOptions a discriminated union requiring description or dependencyBump - Extract UpdateChangeOptions as a flat type for partial updates - Replace private getStdoutFromCommand with shared runCommand (add cwd support) - Use path.relative instead of git rev-parse for repo-relative paths - Rewrite validate-command tests as integration tests (only mock I/O and git) - Add versionChanged to DependencyCheckResult to fix entries going to wrong changelog section when package is not being released - Document --baseBranch usage for stacked PRs - Improve error messages, test names, and test coverage
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
| } else if (dependencyBump !== undefined) { | ||
| change.dependencyBump = dependencyBump; | ||
| change.description = formatDependencyBumpDescription(dependencyBump); | ||
| } |
There was a problem hiding this comment.
updateChange silently drops dependencyBump metadata update
Low Severity
In updateChange, when both description and dependencyBump are provided, only description is applied — the dependencyBump property on the change object retains its old value. This creates an inconsistency where change.dependencyBump has stale metadata that no longer matches change.description. Subsequent calls to findChangelogEntry (which matches on dependencyBump fields) could produce incorrect results if this code path is exercised.
There was a problem hiding this comment.
Fixed in 52755f4. updateChange now always updates change.dependencyBump when provided, regardless of whether description is also set.
| const changesSection = | ||
| dependencyCheckResult.versionChanged && currentVersion | ||
| ? releaseChanges | ||
| : changelog.getUnreleasedChanges(); |
There was a problem hiding this comment.
Validation checks wrong section when version bumped without release header
Medium Severity
When dependencyCheckResult.versionChanged is true and currentVersion is set but the changelog has no release section for that version, releaseChanges is undefined. The ternary falls to releaseChanges (which is undefined), coalesced to {}, causing ALL dependency bumps to be reported as missing — even if they already exist in the Unreleased section. The fix path in validate-command.ts would also fail because addChange throws when the target version section doesn't exist.
There was a problem hiding this comment.
Fixed in 52755f4. Added && releaseChanges guard so validation falls back to Unreleased when the release header does not exist yet.
…Unreleased when release header missing
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
| currentVersion: dependencyCheckResult?.versionChanged | ||
| ? currentVersion | ||
| : undefined, | ||
| prNumbers, |
There was a problem hiding this comment.
Fix handler crashes when release section doesn't exist
Medium Severity
The validation logic in validateChangelog correctly falls back to the Unreleased section when versionChanged is true but the release header for currentVersion doesn't exist in the changelog (because releaseChanges is undefined). However, the fix handler in validate-command.ts unconditionally passes currentVersion to updateChangelogWithDependencies when versionChanged is true, without checking whether the release section exists. This causes addChange inside updateChangelogWithDependencies to throw "Specified release version does not exist" — making --fix fail in a scenario where validation itself succeeds with the fallback.
Additional Locations (1)
There was a problem hiding this comment.
Fixed in cf07c99. The fix handler now checks whether the release header actually exists in the changelog before targeting that section, mirroring the fallback logic in validateChangelog.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
src/parse-changelog.ts
Outdated
| description, | ||
| version: mostRecentRelease, | ||
| prNumbers, | ||
| dependencyBump, |
There was a problem hiding this comment.
Passing dependencyBump: undefined defeats union type guard
Low Severity
parsePossibleDependencyBumpDescription can return undefined, and this undefined is passed as an explicit dependencyBump: undefined property to addChange. Inside addChange, the 'dependencyBump' in options check at line 583 returns true for every parsed entry (because JavaScript's in operator checks key existence, not value), making this type-narrowing check meaningless. The code works only because of the subsequent !== undefined guard. Using a conditional spread like ...(dependencyBump != null && { dependencyBump }) would make the in check behave as intended and keep the discriminated union meaningful.
Additional Locations (1)
There was a problem hiding this comment.
Fixed in 0de6b70. Changed to conditional spread ...(dependencyBump \!= null && { dependencyBump }) so the key is omitted when undefined, making the 'dependencyBump' in options union guard meaningful.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
| return entry.prNumbers; | ||
| } | ||
| // Otherwise extract from the description text | ||
| const matches = entry.description.matchAll(/\[#(\d+)\]|(?<!\[)#(\d+)\)/gu); |
There was a problem hiding this comment.
Regex misses PR numbers in comma-separated short links
Medium Severity
The fallback regex in extractPrNumbersFromEntry — \[#(\d+)\]|(?<!\[)#(\d+)\) — only captures PR numbers immediately followed by ). For comma-separated short PR links like (#123, #456), the #123 is followed by , not ), so only the last number (#456) is captured. This causes earlier PR numbers to be silently dropped when merging PR numbers during entry updates.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
| description, | ||
| version: mostRecentRelease, | ||
| prNumbers, | ||
| ...(dependencyBump !== null && { dependencyBump }), |
There was a problem hiding this comment.
Wrong nullish check spreads undefined dependencyBump on all entries
Medium Severity
parsePossibleDependencyBumpDescription returns DependencyBump | undefined, never null. The guard dependencyBump !== null is always true (since undefined !== null is true in JS), so { dependencyBump: undefined } is spread into every parsed change entry's options — not just dependency bump entries. This causes 'dependencyBump' in options inside addChange to evaluate to true for all entries, incorrectly entering the dependency-bump code path. The check needs to be dependencyBump !== undefined or dependencyBump != null.


Summary
Adds
--checkDepsflag tovalidatecommand to detect missing dependency bump entries in changelogs.dependenciesandpeerDependenciesinpackage.jsonbetween git refs usinggit showJSON comparison (ignores devDependencies)--fix --currentPr <number>, automatically adds or updates missing entriesgit log**BREAKING:**prefixAPI changes
Changelog.addChange()accepts optionaldependencyBumpfor structured bump metadataChangelog.updateChange()method for modifying existing entriesparseDependencyBumpDescription()enables round-trip fidelity (parse -> stringify)getDependencyChangesForPackage,DependencyCheckResult,DependencyChange,MissingDependencyEntriesErrorCLI options
--checkDeps— enable dependency bump validation--fromRef <ref>— starting git ref (auto-detects from merge-base if omitted)--toRef <ref>— ending git ref (defaults toHEAD)--remote <name>— remote for base branch auto-detection (defaults toorigin)--baseBranch <ref>— base branch ref (defaults to<remote>/main)--currentPr <number>— PR number for changelog entries (required with--fix)Usage
Testing in MetaMask/core
Note
Medium Risk
Adds new git-driven dependency diffing and changelog auto-modification paths to
validate, which could cause false positives/negatives or unexpected changelog rewrites if ref detection or entry matching is off. Changes are mostly additive but touch core validation/CLI behavior.Overview
Adds an opt-in dependency-bump validator to the
validateCLI via--checkDeps, diffingdependencies/peerDependenciesbetween git refs (auto-detected via merge-base or provided via--fromRef/--toRef/--remote/--baseBranch) and extracting PR numbers fromgit log.Extends the changelog model with structured
dependencyBumpentries (auto-generatingBump ...text and**BREAKING:**for peer deps) plus a newChangelog.updateChange()API, and updates parsing/validation to round-trip these entries and throwMissingDependencyEntriesError/breaking-prefix mismatches.When
--fix --currentPris used,validatenow auto-adds/updates dependency bump entries in the appropriate section (Unreleased vs release) using a newupdateChangelogWithDependencieshelper; also introduces small CLI/file I/O helpers, exports the new APIs fromindex.ts, and adds extensive tests (with raised coverage thresholds).Written by Cursor Bugbot for commit ccefe50. This will update automatically on new commits. Configure here.