Summary
After a systematic review of the ARM API Reviewer agent, its instruction files, shared skill references, and eval suite, I have identified several improvements that could strengthen reliability, accuracy, and maintainability as adoption scales.
Suggestions
High Impact
1. Chunked rule loading strategy
The agent must load approximately 2,500+ lines of instruction content simultaneously (ARM ~1,200 + OpenAPI ~600 + TypeSpec ~400 + reference files). This creates risk of context truncation in LLM inference. Consider a phased approach (e.g., Phase 1: security + breaking changes; Phase 2: resource model; Phase 3: property design) where each phase loads only relevant sections, or a condensed "critical rules" summary for fast-path reviews.
2. Explicit error recovery guidance
The agent lacks guidance for partial failures, such as when get_file_contents fails for a previous version, GitHub API rate limits are hit, or authentication lapses mid-review. A "Failure Modes" section in the agent file would improve robustness (e.g., file not found: skip comparison, note it, classify all issues as [NEW]).
3. Consistent rule IDs across all rules
Some rules have formal IDs (RPC-Put-V1-11, OAPI027, SEC-SECRET-DETECT) while others are cited by section name ("Section 6.1 - Naming"). Assigning IDs to every rule would improve cross-referencing, traceability, and the linter-rule-coverage map.
4. Quick review mode for small PRs
For small PRs (1-3 files, fewer than 200 lines changed, example-only or description-only changes), the full 8-step workflow with version comparison and suppression continuity analysis adds overhead. A lightweight fast path could skip Steps 3-5 for qualifying PRs.
Medium Impact
5. Version discovery optimization
Step 3 requires directory enumeration on the base branch to find the previous API version. Checking the readme.md tag configuration first (which encodes version history) would reduce API calls for services with many versions.
6. Suppression continuity parity for suppressions.yaml
The readme.md suppression continuity analysis (Step 4) is thorough, but the equivalent guidance for suppressions.yaml is much thinner ("apply the same approval criteria" without the same inventory-compare-classify workflow).
7. TypeSpec conversion operation count verification
Section 8 of typespec-review.instructions.md covers conversions but does not mention verifying that all operations from the original OpenAPI are present in the TypeSpec output. This is a common conversion bug (especially list-by-subscription or operations API).
8. Concrete reconciliation examples
The comment reconciliation logic (Scenarios A-E) is well-designed but dense. Adding 2-3 concrete examples, especially for Scenario C (different reviewer's comment with line shift) and Scenario E (violation fixed), would improve consistency.
Lower Impact
9. Linter rule coverage summary
Add a coverage count at the top of linter-rule-coverage.md (e.g., "Coverage: 125/130 rules, 5 gaps") so maintainers can assess status at a glance.
10. Top 10 common mistakes quick-start guide
A short reference showing the most frequently flagged violations with before/after examples would help PR authors self-review before submission and reduce review round-trips.
Related Files
.github/agents/arm-api-reviewer.agent.md.github/instructions/armapi-review.instructions.md.github/instructions/openapi-review.instructions.md.github/instructions/typespec-review.instructions.md.github/skills/azure-api-review/SKILL.md.github/skills/azure-api-review/references/
Summary
After a systematic review of the ARM API Reviewer agent, its instruction files, shared skill references, and eval suite, I have identified several improvements that could strengthen reliability, accuracy, and maintainability as adoption scales.
Suggestions
High Impact
1. Chunked rule loading strategy
The agent must load approximately 2,500+ lines of instruction content simultaneously (ARM ~1,200 + OpenAPI ~600 + TypeSpec ~400 + reference files). This creates risk of context truncation in LLM inference. Consider a phased approach (e.g., Phase 1: security + breaking changes; Phase 2: resource model; Phase 3: property design) where each phase loads only relevant sections, or a condensed "critical rules" summary for fast-path reviews.
2. Explicit error recovery guidance
The agent lacks guidance for partial failures, such as when
get_file_contentsfails for a previous version, GitHub API rate limits are hit, or authentication lapses mid-review. A "Failure Modes" section in the agent file would improve robustness (e.g., file not found: skip comparison, note it, classify all issues as[NEW]).3. Consistent rule IDs across all rules
Some rules have formal IDs (
RPC-Put-V1-11,OAPI027,SEC-SECRET-DETECT) while others are cited by section name ("Section 6.1 - Naming"). Assigning IDs to every rule would improve cross-referencing, traceability, and the linter-rule-coverage map.4. Quick review mode for small PRs
For small PRs (1-3 files, fewer than 200 lines changed, example-only or description-only changes), the full 8-step workflow with version comparison and suppression continuity analysis adds overhead. A lightweight fast path could skip Steps 3-5 for qualifying PRs.
Medium Impact
5. Version discovery optimization
Step 3 requires directory enumeration on the base branch to find the previous API version. Checking the
readme.mdtag configuration first (which encodes version history) would reduce API calls for services with many versions.6. Suppression continuity parity for
suppressions.yamlThe
readme.mdsuppression continuity analysis (Step 4) is thorough, but the equivalent guidance forsuppressions.yamlis much thinner ("apply the same approval criteria" without the same inventory-compare-classify workflow).7. TypeSpec conversion operation count verification
Section 8 of
typespec-review.instructions.mdcovers conversions but does not mention verifying that all operations from the original OpenAPI are present in the TypeSpec output. This is a common conversion bug (especially list-by-subscription or operations API).8. Concrete reconciliation examples
The comment reconciliation logic (Scenarios A-E) is well-designed but dense. Adding 2-3 concrete examples, especially for Scenario C (different reviewer's comment with line shift) and Scenario E (violation fixed), would improve consistency.
Lower Impact
9. Linter rule coverage summary
Add a coverage count at the top of
linter-rule-coverage.md(e.g., "Coverage: 125/130 rules, 5 gaps") so maintainers can assess status at a glance.10. Top 10 common mistakes quick-start guide
A short reference showing the most frequently flagged violations with before/after examples would help PR authors self-review before submission and reduce review round-trips.
Related Files
.github/agents/arm-api-reviewer.agent.md.github/instructions/armapi-review.instructions.md.github/instructions/openapi-review.instructions.md.github/instructions/typespec-review.instructions.md.github/skills/azure-api-review/SKILL.md.github/skills/azure-api-review/references/