Add structured telemetry fields to ARM API reviewer agent comment marker#42824
Open
Add structured telemetry fields to ARM API reviewer agent comment marker#42824
Conversation
Next Steps to MergeNext steps that must be taken to merge this PR:
Comment generated by summarize-checks workflow run. |
- Fix #1: Label management conditional on blocking findings (agent Step 9) - Fix #2: Add CI cross-reference step (agent Step 6a) - Fix #3: Add comment volume control caps (agent Step 8) - Fix #4: Add severity calibration section (agent) - Fix #6: Add design-decisions.md reference in severity calibration - Fix #7: Renumber steps (4a->5, 5->6, 6->7, 7->8, 8->9) - Fix #8: Expand Scenario E with agent-vs-human distinction (copilot-review) - Fix #10: Add source field to telemetry marker (agent/code-review)
…ions" This reverts commit e333875.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updates the hidden HTML marker appended to ARM API reviewer agent comments from
a simple identifier to a structured format with rule ID, severity, and
new/existing classification fields. This enables querying agent-posted comments
by rule, severity, and classification for telemetry purposes.
Updated files:
.github/agents/arm-api-reviewer.agent.md.github/copilot-review-instructions.mdDetection logic in reconciliation scenarios (B and C) updated to use substring
matching for backward compatibility with existing comments.
Marker Format
Each agent-posted comment ends with a hidden HTML marker:
<!-- posted-by: arm-api-reviewer-agent | rule: RPC-Put-V1-11 | severity: blocking | classification: new -->Telemetry Example
Extract structured telemetry from agent comments on a PR using the
ghCLI:Sample output:
[ { "id": 123456, "rule": "RPC-Put-V1-11", "severity": "blocking", "classification": "new" }, { "id": 123457, "rule": "OAPI027", "severity": "warning", "classification": "existing" } ]This enables aggregate analysis such as top violated rules, new-vs-existing
issue ratios, and comments-per-PR trends.