Skip to content

Add structured telemetry fields to ARM API reviewer agent comment marker#42824

Open
ravimeda wants to merge 5 commits intomainfrom
raeda/extend-agent-telemetry-marker
Open

Add structured telemetry fields to ARM API reviewer agent comment marker#42824
ravimeda wants to merge 5 commits intomainfrom
raeda/extend-agent-telemetry-marker

Conversation

@ravimeda
Copy link
Copy Markdown
Member

@ravimeda ravimeda commented May 1, 2026

Updates the hidden HTML marker appended to ARM API reviewer agent comments from
a simple identifier to a structured format with rule ID, severity, and
new/existing classification fields. This enables querying agent-posted comments
by rule, severity, and classification for telemetry purposes.

Updated files:

  • .github/agents/arm-api-reviewer.agent.md
  • .github/copilot-review-instructions.md

Detection logic in reconciliation scenarios (B and C) updated to use substring
matching for backward compatibility with existing comments.

Marker Format

Each agent-posted comment ends with a hidden HTML marker:

<!-- posted-by: arm-api-reviewer-agent | rule: RPC-Put-V1-11 | severity: blocking | classification: new -->

Telemetry Example

Extract structured telemetry from agent comments on a PR using the gh CLI:

gh api "/repos/Azure/azure-rest-api-specs/pulls/{number}/comments" \
  --jq '[.[] | select(.body | test("posted-by: arm-api-reviewer-agent"))
         | {id, rule: (.body | capture("rule: (?<r>[^ |]+)")).r,
            severity: (.body | capture("severity: (?<s>[^ |]+)")).s,
            classification: (.body | capture("classification: (?<c>[^ ]+)")).c}]'

Sample output:

[
  { "id": 123456, "rule": "RPC-Put-V1-11", "severity": "blocking", "classification": "new" },
  { "id": 123457, "rule": "OAPI027", "severity": "warning", "classification": "existing" }
]

This enables aggregate analysis such as top violated rules, new-vs-existing
issue ratios, and comments-per-PR trends.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 1, 2026

Next Steps to Merge

Next steps that must be taken to merge this PR:
  • ❌ The required check named Protected Files has failed. Refer to the check in the PR's 'Checks' tab for details on how to fix it and consult the aka.ms/ci-fix guide


Comment generated by summarize-checks workflow run.

ravimeda added 4 commits May 1, 2026 07:55
- Fix #1: Label management conditional on blocking findings (agent Step 9)
- Fix #2: Add CI cross-reference step (agent Step 6a)
- Fix #3: Add comment volume control caps (agent Step 8)
- Fix #4: Add severity calibration section (agent)
- Fix #6: Add design-decisions.md reference in severity calibration
- Fix #7: Renumber steps (4a->5, 5->6, 6->7, 7->8, 8->9)
- Fix #8: Expand Scenario E with agent-vs-human distinction (copilot-review)
- Fix #10: Add source field to telemetry marker (agent/code-review)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant