Skip to content

chore(ci): add PR quality gate and improve PR template#4052

Open
bibhuti230185 wants to merge 1 commit intoeclipse-sw360:mainfrom
siemens:chore/improve-pr-template-and-quality-gate
Open

chore(ci): add PR quality gate and improve PR template#4052
bibhuti230185 wants to merge 1 commit intoeclipse-sw360:mainfrom
siemens:chore/improve-pr-template-and-quality-gate

Conversation

@bibhuti230185
Copy link
Copy Markdown
Contributor

Summary

Add an automated PR Quality Gate workflow and rewrite the PR template to enforce type-specific quality evidence, AI disclosure accountability, and a 60% completeness scoring threshold. Also update the Copilot instructions to reflect the new PR standards.

Type of Change

  • feat — New feature
  • fix — Bug fix
  • refactor — Code restructuring (no behavior change)
  • test — Adding or updating tests
  • docs — Documentation only
  • chore — Build, CI, dependencies, cleanup
  • build — Build system or Docker changes

Related Issue

Issue: N/A — This addresses a systemic quality gap observed across recent PRs: missing root cause analysis in bug fixes, missing architecture context in features, undisclosed AI usage, and no manual testing evidence. No single issue tracks this.


Suggest Reviewer

@GMishx

How To Test

  1. PR template — Open a new draft PR against main and verify the template auto-populates with all type-specific sections (Bug Fix Details, Feature Details, Refactor Details), expanded checklist, and AI Disclosure fields.
  2. Quality gate — empty description — Edit the PR body to be nearly empty (remove summary, uncheck all boxes). Verify the PR Quality Gate workflow fails with hard errors listing missing sections.
  3. Quality gate — partial description — Fill in Summary + Type + a few checklist items but leave How To Test empty. Verify the workflow reports a score below 60% and fails.
  4. Quality gate — good description — Fill in all sections properly for a chore type (Summary, Type, Issue, How To Test, 4+ checklist items). Verify the workflow passes with ≥60% score.
  5. AI disclosure enforcement — Check the "AI-assisted code" checkbox but leave "Your understanding" blank. Verify the workflow produces a hard error.
  6. AI hint detection — Mention "Copilot" in the body without checking the AI disclosure box. Verify a soft warning appears.
  7. Trivial type exemption — Select docs as type and leave type-specific sections empty. Verify no hard error for missing type-specific content (auto-pass).
  8. Concurrency — Push two quick commits to the same PR branch. Verify the older workflow run is cancelled.

Checklist

Must:

  • Code is formatted (mvn spotless:apply)
  • No new compiler warnings introduced
  • I have manually tested these changes against a running SW360 instance (or explained why not)

Note on manual testing: This PR contains only Markdown and YAML (GitHub Actions workflow). It cannot be tested against a running SW360 instance. Testing is done by opening PRs with various description completeness levels and observing the workflow output (see How To Test above).

AI Disclosure

  • This PR includes AI-assisted code

Tool & model: GitHub Copilot (Claude 3.5 Sonnet) in VS Code agent mode

What AI generated vs. what you wrote/changed:
AI generated the initial drafts of all three files based on detailed requirements and iterative feedback. I designed the overall approach (type-specific sections, 60% scoring threshold, hard errors vs. soft warnings split, AI accountability fields), reviewed every line, adjusted the scoring weights, verified all GitHub Action SHAs against the repository's existing workflows, and manually cross-referenced the Harden Runner SHA and actions/github-script SHA via GitHub Tags API.

Your understanding of the changes (in your own words):
The PR template now forces contributors to provide evidence appropriate to their change type — bug fixes need root cause analysis, features need architecture explanation, refactors need no-behavior-change confirmation. The Quality Gate workflow parses the PR body with regex helpers, validates structural requirements (summary, type, issue, how-to-test), runs type-specific checks as soft warnings, enforces AI disclosure accountability as a hard error when the checkbox is checked but understanding is missing, and computes a weighted completeness score across 7 dimensions. PRs below 60% or with any hard errors are blocked via core.setFailed(). Trivial types (docs/chore/build) get automatic credit for type-specific sections. The workflow uses the same hardening conventions as all existing SW360 workflows: SHA-pinned actions, step-security/harden-runner, least-privilege permissions, and PR concurrency groups.

Edge cases you verified:

  1. Empty PR body — workflow fails gracefully with clear error messages instead of crashing on null matches
  2. AI tools mentioned in body text (e.g., "tested with Copilot suggestions") without the checkbox checked — produces a soft warning, not a hard error
  3. docs/chore/build type PRs without type-specific sections — auto-pass (15 pts) instead of penalizing trivial changes
  4. test type PRs without "How To Test" — exempt since the tests themselves are the verification

Breaking Changes

None.

Additional Notes

File Lines Purpose
.github/pull_request_template.md 145 Complete rewrite — type-specific sections, AI accountability, expanded checklist
.github/workflows/pr_description_check.yml 261 New workflow — PR Quality Gate with 60% scoring threshold
.github/instructions/git-commit.instructions.md 240 Updated — replaced inline template copy with pointer to actual template, documented Quality Gate

After merge: A repository admin should add validate-pr as a required status check in Settings → Branches → main branch protection rules to make the gate mandatory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant