OCPEDGE-2727: Add agent-eval-harness CI strategy and setup script

[dhensel-rh]

Summary

• Add docs/agent-eval-harness/ci-strategy.md documenting the eval CI
  integration strategy for Claude Code skills
• Add scripts/eval-setup-ci.sh for dynamic skill discovery and
  judge/case generation in CI

Strategy doc covers

• Three approaches to running evals (local, container, step registry)
  with flow diagrams for each
• Dynamic skill discovery from PR diffs — one CI job scales to any
  number of skills
• Judge design: skill-specific judges (completeness, accuracy) and
  behavioral judges (tool-usage, data-grounding, efficiency)
• Two maturity levels: fresh judges (smoke test) vs committed judges
  with baselines (regression detection)
• What CI reporting actually tells you today and what it could tell
  you with committed judges

Setup script

eval-setup-ci.sh is called by the openshift-claude-agent-eval
shared step registry ref via EVAL_SETUP_SCRIPT. It:

1. Diffs PULL_BASE_SHA...HEAD to find the changed SKILL.md
2. Runs /eval-analyze to generate scoring judges
3. Runs /eval-dataset to generate fresh scenarios
4. Outputs the eval config path for the ref to use

Dependencies

• Step registry ref changes in openshift/release (PR #80177)
• claude-payload-agent-github-token secret needs to be created in
  test-credentials

Test plan

• Verify strategy doc renders correctly on GitHub (Mermaid diagrams)
• Review setup script for correctness
• End-to-end validation after release repo PR merges

🤖 Generated with Claude Code

Summary by CodeRabbit

• Documentation

  • Added guidance describing the evaluation CI strategy, including recommended presubmit flow, setup options, and what evaluation reporting currently includes.
  • Documented how criteria and scenarios are generated, along with prerequisites.

• New Features

  • Added CI setup support that detects which skills changed in a pull request, generates an evaluation configuration, and produces fresh evaluation scenarios for the run.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dhensel-rh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [dhensel-rh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: fdea045b-2f05-4811-9c5b-1c825b15ed60

📥 Commits

Reviewing files that changed from the base of the PR and between c89a060 and 4ce9854.

📒 Files selected for processing (1)

• scripts/eval-setup-ci.sh

🚧 Files skipped from review as they are similar to previous changes (1)

• scripts/eval-setup-ci.sh

---

Walkthrough

Adds CI documentation and a Bash setup script for discovering changed skills, generating eval config and datasets, and handing the config to the next CI step.

Changes

Agent eval CI

Layer / File(s)	Summary
CI strategy overview docs/agent-eval-harness/ci-strategy.md	Introduces the eval execution options and selects the shared step-registry path for CI.
Presubmit skill discovery docs/agent-eval-harness/ci-strategy.md	Describes the presubmit trigger, changed-skill detection, path extraction, and eval pipeline handoff.
Judges, scenarios, and reporting docs/agent-eval-harness/ci-strategy.md	Defines judge and scenario generation, current CI reporting, the fresh and committed judge levels, related components, and prerequisites.
Setup script flow scripts/eval-setup-ci.sh	Validates CI inputs, discovers the changed skill, derives plugin and skill names, generates the eval config and dataset, and prints the config path.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Suggested reviewers

• pablofontanilla
• fonta-rh

🚥 Pre-merge checks | ✅ 9 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Ai-Attribution	⚠️ Warning	PR text says it was generated with Claude Code, but the commit only has Co-Authored-By: Claude Opus 4.6 and no Red Hat Assisted-by/Generated-by trailer.	Replace AI Co-Authored-By attribution with the required Red Hat Assisted-by or Generated-by trailer in the commit/PR metadata.

✅ Passed checks (9 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: adding the CI strategy doc and eval setup script for agent-eval-harness.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
No-Weak-Crypto	✅ Passed	Touched files are docs and a bash CI script; no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret/token comparisons were found.
Container-Privileges	✅ Passed	No privileged, host namespace, SYS_ADMIN, or allowPrivilegeEscalation settings appear in the PR diff; the new files are docs and a shell script.
No-Sensitive-Data-In-Logs	✅ Passed	The new script only logs PULL_BASE_SHA, skill paths, and eval config paths; it never prints tokens, passwords, PII, or customer data.
No-Hardcoded-Secrets	✅ Passed	No hardcoded secret values, credential URLs, or private-key blocks found; only secret names are mentioned in docs.
No-Injection-Vectors	✅ Passed	No SQL/exec/pickle/yaml-load/os.system/dangerous HTML patterns appear in the new doc or setup script; shell variables are quoted.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 5

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/agent-eval-harness/ci-strategy.md`:
- Around line 143-164: The discovery behavior in the CI strategy doc does not
match `scripts/eval-setup-ci.sh`, which currently selects only the first changed
`SKILL.md` via its discovery logic. Either update the script to iterate over
every changed skill and run the full `/eval-analyze` and `/eval-dataset`
pipeline for each, or narrow the documented contract to clearly state that one
discovered skill is processed per run; keep the wording aligned with the actual
behavior of `run_if_changed`, `PULL_BASE_SHA...HEAD` discovery, and the setup
handoff.
- Around line 365-373: The Components table in the strategy doc has an outdated
reference for the Strategy doc entry. Update the location text in that row to
point to docs/agent-eval-harness/ci-strategy.md instead of
docs/eval-ci-strategy.md, keeping the rest of the table unchanged.

In `@scripts/eval-setup-ci.sh`:
- Around line 23-33: The SKILL_FILE detection in eval-setup-ci.sh is swallowing
git diff failures, so an invalid PULL_BASE_SHA or missing ancestor can
incorrectly look like “no skill files changed.” Update the git diff handling so
the script checks the command’s exit status before relying on empty output, and
only logs “No skill files changed, nothing to evaluate.” when the diff succeeds
and truly returns no matches. Keep the logic around SKILL_FILE, git diff, and
the no-change exit path aligned so CI misconfigurations fail loudly instead of
exiting 0.
- Line 1: Update the shell script header in eval-setup-ci.sh to use the required
Bash shebang instead of the env-based one. Replace the existing interpreter line
at the top of the script with the project-standard `#!/usr/bin/bash`, keeping
the rest of the script unchanged.
- Around line 50-53: The Claude invocation is using the wrong redirection order,
so stdout is leaking into the script’s stdout instead of staying on stderr and
can corrupt the config-path handoff. Update the claude command in both places in
the eval-setup flow so its output is explicitly redirected to stderr with the
correct ordering, keeping stdout reserved for the final echoed config path and
preserving the EVAL_CONFIG contract.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 9896a84c-95ac-4532-aa15-ca3e98c0ee62

📥 Commits

Reviewing files that changed from the base of the PR and between d06c004 and 2986f89.

📒 Files selected for processing (2)

• docs/agent-eval-harness/ci-strategy.md
• scripts/eval-setup-ci.sh

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/eval-setup-ci.sh`:
- Around line 27-55: Add tests for the new validation flow in eval-setup-ci.sh,
since the script now includes diff parsing and SKILL.md path validation logic.
Cover the main branches in the script’s discovery and validation steps: valid
plugin skill detection, empty git diff/no matching SKILL.md files, deleted
SKILL.md handling, invalid plugin/skills path shapes, and the empty generated
config case. Use the script’s key symbols and behavior around DIFF_OUTPUT,
SKILL_FILE, PLUGIN, and SKILL_NAME to ensure both positive and negative cases
are exercised.
- Around line 33-49: The skill discovery in eval-setup-ci.sh is too broad and
can pick up non-plugin SKILL.md files before the path validation runs. Tighten
the discovery grep in the script’s skill-file selection logic to only match
plugins/<plugin>/skills/<skill>/SKILL.md, and keep the existing path-shape
validation in the same eval setup flow as a safety check. Add tests covering
both the discovery filter and the invalid-path rejection cases so the new
validation logic is exercised with positive and negative examples.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 7f7fee93-85b1-4867-b786-1e13d82b9582

📥 Commits

Reviewing files that changed from the base of the PR and between 2986f89 and c89a060.

📒 Files selected for processing (2)

• docs/agent-eval-harness/ci-strategy.md
• scripts/eval-setup-ci.sh

✅ Files skipped from review due to trivial changes (1)

• docs/agent-eval-harness/ci-strategy.md

[kasturinarra]

After comparing all four eval harness PRs against ai-helpers' production setup, recommending to close this PR.

Why

• On-the-fly judge generation prevents regression tracking — eval-setup-ci.sh calls /eval-analyze and /eval-dataset every run, producing different judges and test cases each
  time. You can't compare scores across runs.
• Level 1 (generated everything) provides only a smoke test — it answers "did the skill crash?" not "did the skill get worse?" As the strategy doc itself acknowledges, Level 2
  (committed judges) is needed for real regression detection.

What's valuable from this PR

The strategy doc (docs/agent-eval-harness/ci-strategy.md) is excellent documentation — the Level 1 → Level 2 progression concept, the comparison of local vs container vs step registry approaches. Let us get this into the evals/README.md in PR #178 ?

cc: @dhensel-rh @ggiguash