From c15b2884cf03565844165b525eff19f872bce35e Mon Sep 17 00:00:00 2001 From: Brendan McMullen Date: Thu, 25 Jun 2026 11:30:46 -0700 Subject: [PATCH 1/3] add report generation skills --- .../claude/hypaware.plugin.json | 6 +- .../hypaware-ai-adoption-report/SKILL.md | 57 ++++++++++++++++ .../hypaware-ai-improvement-report/SKILL.md | 60 ++++++++++++++++ .../hypaware-ai-security-report/SKILL.md | 57 ++++++++++++++++ .../skills/hypaware-ai-spend-report/SKILL.md | 68 +++++++++++++++++++ .../plugins-workspace/claude/src/index.js | 10 ++- 6 files changed, 256 insertions(+), 2 deletions(-) create mode 100644 hypaware-core/plugins-workspace/claude/skills/hypaware-ai-adoption-report/SKILL.md create mode 100644 hypaware-core/plugins-workspace/claude/skills/hypaware-ai-improvement-report/SKILL.md create mode 100644 hypaware-core/plugins-workspace/claude/skills/hypaware-ai-security-report/SKILL.md create mode 100644 hypaware-core/plugins-workspace/claude/skills/hypaware-ai-spend-report/SKILL.md diff --git a/hypaware-core/plugins-workspace/claude/hypaware.plugin.json b/hypaware-core/plugins-workspace/claude/hypaware.plugin.json index 6f107141..02d3954f 100644 --- a/hypaware-core/plugins-workspace/claude/hypaware.plugin.json +++ b/hypaware-core/plugins-workspace/claude/hypaware.plugin.json @@ -34,7 +34,11 @@ "skills": [ { "name": "hypaware-query", "clients": ["claude"] }, { "name": "hypaware-ignore", "clients": ["claude"] }, - { "name": "hypaware-unignore", "clients": ["claude"] } + { "name": "hypaware-unignore", "clients": ["claude"] }, + { "name": "hypaware-ai-adoption-report", "clients": ["claude"] }, + { "name": "hypaware-ai-improvement-report", "clients": ["claude"] }, + { "name": "hypaware-ai-security-report", "clients": ["claude"] }, + { "name": "hypaware-ai-spend-report", "clients": ["claude"] } ], "agents": [ { "name": "hypaware-analyst", "clients": ["claude"] } diff --git a/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-adoption-report/SKILL.md b/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-adoption-report/SKILL.md new file mode 100644 index 00000000..e573a36b --- /dev/null +++ b/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-adoption-report/SKILL.md @@ -0,0 +1,57 @@ +--- +name: hypaware-ai-adoption-report +description: AI Adoption Profile for a HypAware server — descriptive "who's using the fleet and how": per-gateway utilization (volume + focus — top models, tools, repos, themes) and parallelism/fan-out (multi-agent adoption, concurrency, main-vs-subagent split, payoff). +--- + +# AI Adoption Profile + +The one **descriptive** report on *who's using the fleet and how* — no actions, just a clear +picture. Two lenses over one window: + +- **Utilization** — per `gateway_id` (≈ one machine/user): *how much* (messages, sessions, + active days, tokens, cache-read ratio) and *what it's focused on* (top models, tools, repos, + work themes). +- **Parallelism / fan-out** — *how sophisticated*: multi-agent adoption, fan-out breadth/depth, + true concurrency vs serial, the main-loop-vs-subagent token split, and whether fan-out appears + to earn its cost. + +IMPORTANT: Query the server, not local logs. **Descriptive only** — route every *action* out: +"fan out more/less" and tooling go to **hypaware-ai-improvement-report**, token waste to **hypaware-ai-spend-report**. +Query mechanics live in the **hypaware-query** skill; reuse hypaware-ai-spend-report's deduped token spine +for any token figure. + +## Procedure +1. **Scope + coverage.** Distinct `gateway_id` (the unit — `user_id` is ~always null, so don't + measure reach by it); window; coverage of `gateway_id`, token usage, and subagent provenance + (`agent_id` / `is_sidechain` / `parent_thread_id` — transcript-enriched, may not survive + ingest). Decide cost-capable vs volume-only, and which parallelism dimensions are real vs + proxied (the `Task`-call proxy). State N; if it's effectively one gateway / dogfood, say so. +2. **Per-gateway utilization.** One row per gateway: volume (messages, sessions, active days, + first/last seen), tokens + cache-read ratio `cache_read/(cache_read+input)`, then its focus — + top models (+ `(unknown)`), tools (Bash dominance + top commands), repos, client + (claude/codex), and 2–4 recurring work themes (sampled, redacted). Distill each into a + one-line **focus label**. Use the activity graph for structural focus (Session→Repo/PR via + GitHub enrichment) where projected. +3. **Parallelism / fan-out.** Adoption (% conversations with ≥1 subagent, incl. the zero + bucket); breadth (subagents per parent) and depth (`parent_thread_id` chains); concurrency + (do subagent time-spans actually *overlap*, or is it serial?); cost split (token share main + loop vs subagent). Read the payoff *descriptively* — fan-out vs tokens-to-resolution — and + route the "fan out more/less" recommendation to hypaware-ai-improvement-report. + +## Output — SAVE A MARKDOWN FILE +- **Path:** `docs/adoption-profiles/-adoption-profile.md` (create the dir if needed). + Dated so reports accumulate. +- **Bottom line (1 sentence):** how many gateways are active, the busiest one and its focus, and + the headline fan-out adoption %. +- **Per-gateway profile (sortable table):** one row per gateway — **gateway** · **volume** + (messages / sessions) · **tokens** · **cache-read %** · **fan-out** (adoption + main-vs-subagent + split) · **focus label**. Busiest first — this table is the spine. +- **Then the detail**, in these sections: **Scope & coverage · Per-gateway utilization · + Per-gateway focus · Parallelism & fan-out · Payoff (descriptive) · Fleet view · Caveats** — + plus every query you ran. +- **Formatting (human-readable):** open each section with its takeaway; **bold** headline + numbers; one sortable footprint table as the centerpiece; keep the bottom line + table a + ~1-minute read. +- **Capture-health note:** if subagent provenance doesn't reach the server, the standing #1 + caveat is "subagent identity must survive ingest" — run adoption off the `Task`-call proxy and + flag it. diff --git a/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-improvement-report/SKILL.md b/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-improvement-report/SKILL.md new file mode 100644 index 00000000..8308db7f --- /dev/null +++ b/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-improvement-report/SKILL.md @@ -0,0 +1,60 @@ +--- +name: hypaware-ai-improvement-report +description: Analyze AI agent logs and generate a markdown report identifying additions or modifications to skills, subagents, and AGENTS.md/CLAUDE.md, to improve quality and efficiency of the AI agent performance. +--- + +# AI Improvement Review + +Read how the team's agents actually behave in the logs, then propose the **additions and +edits to skills, subagents, and AGENTS.md/CLAUDE.md** that make them do **better work +(quality)** with **less wasted effort and spend**. + +IMPORTANT: Query server logs not local logs + +Focus on these signals: + +- **Repeated work** — work the team repeats successfully → package it once (skill / subagent). +- **Sticking points** — where agents get stuck or redo work (errors, loops, refusals, + abandonment) → the missing or too-weak instruction that would prevent it (an AGENTS.md + rule, or a skill). +- **Inefficiency** — expensive patterns (model over-spec, context bloat, low cache-read, + retry loops) → the setup change that costs less (right-size the model in AGENTS.md / a + subagent, a context-hygiene rule, a skill that avoids the redo). + +## Procedure +1. **Footprint + basis.** Window; distinct contributors (`gateway_id`), repos, sessions; + claude/codex mix. State N and breadth; flag if single-contributor. +2. **Scan the signals for possible improvements.** Work the three signals above; each turns + up candidates. Note frequency (sessions, distinct gateways) and redact examples. + - **Repeated work** — cluster sessions by shared-file overlap and tool-set signature into + recurring kinds of work (reuse a recent `docs/spend-reviews/` *Leverage candidates* + handoff if present). Flag parallelizable work done serially (low `is_sidechain`, few + `agent_id`), and recurring asks / multi-step workflows / re-sent instructions in + sampled prompts + `system_text`. + - **Sticking points** — where agents got stuck or redid work, ranked by impact: failing + tools (`is_error` by `tool_name`), retry loops (same tool + first `tool_args` token + ≥3×/session), refusals/truncations (stop-reason), abandoned costly sessions, + repeatedly-violated conventions. + - **Inefficiency** (reuse the spend spine) — model over-spec, context bloat (no + `is_compact_summary`), low cache-read (`cache_read/(cache_read+input)`), redo loops. +3. **Collect, dedup, prioritize.** Gather the candidates into a list of possible + improvements; drop anything the inventory already covers. For each, suggest a likely form + where one obviously fits — a **skill**, a **subagent**, or an **AGENTS.md/CLAUDE.md + edit** (propose the concrete line(s) as a diff, not "consider documenting") — but don't + force a mapping. Attach evidence (frequency/impact + distinct gateways + token prize) and + rank by it. + +## Output — SAVE A MARKDOWN FILE +- **Path:** `docs/improvement-reviews/-improvement-review.md` (create the dir + if needed). Dated so reviews accumulate. +- **Bottom line (1 sentence):** "Create/edit these N things and what it will improve.". +- **Improvements (ranked table):** one row per improvement — **what** · **suggested form** + (skill / subagent / AGENTS.md edit, where one fits) · **evidence** (recurs N× across G + gateways, sticking-point impact, or token prize) · **where it lands**. Highest-leverage first. +- **Then the detail**, in these sections: **Inventory (what exists) · Basis · Skill + candidates · Subagent candidates (fan-out + roles) · AGENTS.md/CLAUDE.md edits · + Efficiency → setup changes · Caveats** — plus every query you ran and the proposed + AGENTS.md lines verbatim. +- **Formatting (human-readable):** every candidate states its evidence inline; show + AGENTS.md edits as real diffs/code blocks, not prose; **bold** the artifact type; keep + it a ~1-minute read. diff --git a/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-security-report/SKILL.md b/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-security-report/SKILL.md new file mode 100644 index 00000000..bafe0576 --- /dev/null +++ b/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-security-report/SKILL.md @@ -0,0 +1,57 @@ +--- +name: hypaware-ai-security-report +description: Security & Risk Review for a HypAware server — audits gateway logs for risky autonomous activity (destructive commands, remote-exec, privilege escalation, secret reads, network egress, package installs), severity-ranks findings, and recommends guardrails. Saves a dated report under docs/security-reviews/; queries the server via the hypaware-query skill. Audits recorded logs, NOT pending code — for a code diff use the built-in /security-review. +--- + +# Security & Risk Review + +Turn a window of central AI-gateway logs into a severity-ranked picture of **what agents did +unsupervised that carries risk**, and the guardrails to contain it. This audits *recorded +activity*, not pending code (for a diff, use the built-in `/security-review`). Be +proportionate: separate "ran a risky command" from "caused harm". + +IMPORTANT: Query the server, not local logs. **REDACT everything** — never echo a secret, +token, key, or credential value in the report, even one you found in `tool_args`; a raw secret +appearing in args is itself a capture finding (flag it, don't reproduce it). Query mechanics +live in the **hypaware-query** skill; read it first. + +## Risk classes (match over the FULL command text, not just the first token) +| Class | Match signals | Why it matters | +| --- | --- | --- | +| Destructive | `rm -rf`, `git reset --hard`, `git push --force`, `drop`/`truncate`, `dd`, `mkfs` | irreversible data/history loss | +| Remote-exec | `curl`/`wget … \| sh`/`bash`, piped installers | runs unreviewed remote code | +| Privilege | `sudo`, `su`, `chmod 777`, writing system paths | escalation beyond the workspace | +| Secret read | `.env`, `id_rsa`, `.ssh`, `.aws`, `.netrc`, `AWS_*`, `*token*`, `credentials` | credential exposure | +| Network egress | outbound `curl`/`scp`/`nc` POSTs to external hosts | potential exfiltration | +| Package install | `npm i`, `pip install`, `brew install`, `apt` | supply-chain surface | + +Risk is **amplified by autonomy** — weight anything that ran under `bypassPermissions` higher +than the same command in a gated session. + +## Procedure +1. **Coverage + autonomy baseline.** Window; coverage of `tool_args` on Bash calls and of + `permission_mode` (bound the read if sparse); distinct `gateway_id` (the unit — `user_id` is + ~always null). Total tool calls, % Bash, % conversations in `bypassPermissions`. State N; if + it's a dogfood window, say so. +2. **Scan + rank.** Histogram the first token of each Bash command (top 30 = the normal), then + match the full text against each risk class: count, ONE redacted example, and the + `gateway_id`/`git_branch` where it ran. Rank by frequency × blast radius × autonomy; keep + "ran" vs "caused harm" distinct, and note anything that ran in a *gated* (non-bypass) session. +3. **Guardrails.** For each top risk, a concrete control + estimated coverage: a PreToolUse + **hook** (deny/allow), a **permission policy / allowlist**, a **CLAUDE.md** rule, or a + **gateway redaction** fix for any secret seen in args. + +## Output — SAVE A MARKDOWN FILE +- **Path:** `docs/security-reviews/-security-review.md` (create the dir if needed). + Dated so reports accumulate. +- **Bottom line (1 sentence):** the posture — how autonomous the fleet is (% `bypassPermissions`) + and whether anything needs urgent attention. +- **Findings (severity-ranked table):** one row per finding — **risk class** · **severity** + (high / med / low) · **count** · **example (redacted)** · **guardrail** · **where it ran**. + Highest severity first. +- **Then the detail**, in these sections: **Autonomy baseline · Command profile · Risk findings + · Guardrails · Trends · Capture caveats** — plus every query you ran. **All examples redacted.** +- **Formatting (human-readable):** open each section with its takeaway; **bold** severities; one + severity-ranked findings table as the centerpiece; keep the bottom line + table a ~1-minute read. +- **Capture-health note:** if raw credentials appear in `tool_args`, the standing #1 finding is + "gateway must redact secrets in tool_args" — put it in Capture caveats without reproducing it. diff --git a/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-spend-report/SKILL.md b/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-spend-report/SKILL.md new file mode 100644 index 00000000..163da783 --- /dev/null +++ b/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-spend-report/SKILL.md @@ -0,0 +1,68 @@ +--- +name: hypaware-ai-spend-report +description: AI Spend Review for a HypAware server — where tokens go (by user/repo/model/gateway), what work they go on (graph-clustered), and how to spend less (waste scorecard + cost levers). Deduped token volume, never dollars. Saves a dated report under docs/spend-reviews/; queries the server via the hypaware-query skill. +--- + +# AI Spend Review + +Where the team's tokens go, on what work, and how to spend less — read off one deduped token +spine. One story in three moves: **where it goes** (by user / repo / model / gateway, over +time), **on what work** (sessions clustered into recurring work-types, sized by tokens), and +**how to spend less** (a waste scorecard + ranked cost levers). + +IMPORTANT: Query the server, not local logs. **Tokens, never dollars** — capture is partial, +so stop at token volume and behavioral proxies. Query mechanics (`--server`, date-pruning the +30s timeout, the SQL dialect, sampling) live in the **hypaware-query** skill; read it first. + +## Token math (get this right — every breakdown reconciles to it) +Usage is in `attributes.usage` (NOT `raw_frame`): `input_tokens`, `output_tokens`, +`cache_read_tokens`, `cache_write_tokens` (+ `reasoning_tokens` for Codex). Dedup by message +and `max()` — a plain `SUM` overcounts ~3×. Report the four types separately (cache-read is +usually the bulk; output the scarce slice). + +```sql +WITH msg AS ( + SELECT COALESCE(CAST(JSON_EXTRACT(raw_frame,'$.message_id') AS VARCHAR), message_id) mid, + max(CAST(JSON_EXTRACT(attributes,'$.usage.input_tokens') AS BIGINT)) inp, + max(CAST(JSON_EXTRACT(attributes,'$.usage.output_tokens') AS BIGINT)) outp, + max(CAST(JSON_EXTRACT(attributes,'$.usage.cache_write_tokens') AS BIGINT)) cwrite, + max(CAST(JSON_EXTRACT(attributes,'$.usage.cache_read_tokens') AS BIGINT)) cread + FROM ai_gateway_messages + WHERE date BETWEEN '' AND '' + AND role='assistant' AND JSON_EXTRACT(attributes,'$.usage') IS NOT NULL + GROUP BY mid) +SELECT sum(inp) t_in, sum(outp) t_out, sum(cwrite) t_cw, sum(cread) t_cr FROM msg; +-- slice by adding max(gateway_id)/max(model)/max(repo_root)/max(date) inside, GROUP BY outside. +``` + +## Procedure +1. **Coverage + footprint.** Window; usage coverage, `model`-column coverage (token-weighted), + user/repo coverage; distinct gateways, sessions, claude/codex mix. `user_id` is ~always null + → attribute by `gateway_id` (≈ one machine/user) and always show an explicit **unattributed** + row. State N; if it's a dogfood window, say so. If usage is thin, fall back to behavioral + proxies (turns, tool calls, length), labeled as estimates. +2. **Where it goes + on what work.** Slice the spine by user / repo / model (→ `(unknown)` + bucket) / gateway with shares, plus trends (by week, WoW deltas vs the last report) and + top-spend outliers. Then cluster sessions into recurring **work-types** — shared-file overlap + for code work, tool-set signature for no-file work (context graph if projected, else SQL) — + and size each by deduped tokens. +3. **How to spend less.** Score the waste — cache-read ratio `cache_read/(cache_read+input)` + (the biggest lever — feature it), retry loops, abandoned costly sessions, model over-spec, + context bloat — and turn the top few into ranked cost levers, each with an estimated weekly + token saving. Hand the expensive recurring work-types to **hypaware-ai-improvement-report** as packaging + candidates (surface them; don't build them here). + +## Output — SAVE A MARKDOWN FILE +- **Path:** `docs/spend-reviews/-spend-review.md` (create the dir if needed). Dated + so reviews accumulate. +- **Bottom line (1 sentence):** total tokens this window, the trend vs the last report (▲/▼ %), + and the single biggest driver. +- **Cost actions (ranked table):** one row per lever — **action** · **evidence** (tokens/share + + cause) · **est. weekly saving** · **where it lands**. Highest-saving first. +- **Then the detail**, in these sections: **Coverage & footprint · Total spend (4 token types) + · Allocation (user / repo / model / gateway) · Trends · Outliers · Spend by work-type · Waste + scorecard · Leverage candidates (handoff to hypaware-ai-improvement-report) · Caveats** — plus every query + you ran. +- **Formatting (human-readable):** open each section with its one-sentence takeaway, then the + table; **bold** headline numbers; split the four token types (never collapse them); keep the + bottom line + ranked table a ~1-minute read. diff --git a/hypaware-core/plugins-workspace/claude/src/index.js b/hypaware-core/plugins-workspace/claude/src/index.js index 9f547615..cee9af5b 100644 --- a/hypaware-core/plugins-workspace/claude/src/index.js +++ b/hypaware-core/plugins-workspace/claude/src/index.js @@ -254,7 +254,15 @@ export async function activate(ctx) { const skillsRoot = path.resolve(skillsRootDir(), 'skills') // @ref LLP 0011#interactive-walkthrough [implements] — contributes client skills the first-run walkthrough installs - for (const skillName of ['hypaware-query', 'hypaware-ignore', 'hypaware-unignore']) { + for (const skillName of [ + 'hypaware-query', + 'hypaware-ignore', + 'hypaware-unignore', + 'hypaware-ai-adoption-report', + 'hypaware-ai-improvement-report', + 'hypaware-ai-security-report', + 'hypaware-ai-spend-report', + ]) { ctx.skills.register({ name: skillName, plugin: PLUGIN_NAME, From 281a8dabb5b5ab42fb62e7896bf5ba2201129c86 Mon Sep 17 00:00:00 2001 From: Brendan McMullen Date: Thu, 25 Jun 2026 16:00:06 -0700 Subject: [PATCH 2/3] add codex skills --- .../hypaware-ai-adoption-report/SKILL.md | 25 ++++--- .../hypaware-ai-improvement-report/SKILL.md | 16 ++-- .../hypaware-ai-security-report/SKILL.md | 30 +++++--- .../skills/hypaware-ai-spend-report/SKILL.md | 21 ++++-- .../claude/skills/hypaware-query/SKILL.md | 31 +++++++- .../codex/hypaware.plugin.json | 8 +- .../hypaware-ai-adoption-report/SKILL.md | 64 ++++++++++++++++ .../hypaware-ai-improvement-report/SKILL.md | 66 ++++++++++++++++ .../hypaware-ai-security-report/SKILL.md | 65 ++++++++++++++++ .../skills/hypaware-ai-spend-report/SKILL.md | 75 +++++++++++++++++++ .../codex/skills/hypaware-query/SKILL.md | 31 +++++++- .../plugins-workspace/codex/src/index.js | 23 ++++-- 12 files changed, 412 insertions(+), 43 deletions(-) create mode 100644 hypaware-core/plugins-workspace/codex/skills/hypaware-ai-adoption-report/SKILL.md create mode 100644 hypaware-core/plugins-workspace/codex/skills/hypaware-ai-improvement-report/SKILL.md create mode 100644 hypaware-core/plugins-workspace/codex/skills/hypaware-ai-security-report/SKILL.md create mode 100644 hypaware-core/plugins-workspace/codex/skills/hypaware-ai-spend-report/SKILL.md diff --git a/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-adoption-report/SKILL.md b/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-adoption-report/SKILL.md index e573a36b..5c6ab15b 100644 --- a/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-adoption-report/SKILL.md +++ b/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-adoption-report/SKILL.md @@ -15,10 +15,18 @@ picture. Two lenses over one window: true concurrency vs serial, the main-loop-vs-subagent token split, and whether fan-out appears to earn its cost. -IMPORTANT: Query the server, not local logs. **Descriptive only** — route every *action* out: -"fan out more/less" and tooling go to **hypaware-ai-improvement-report**, token waste to **hypaware-ai-spend-report**. -Query mechanics live in the **hypaware-query** skill; reuse hypaware-ai-spend-report's deduped token spine -for any token figure. +IMPORTANT: Don't assume which logs to read — **ask first.** Start by listing the data sources +and let the user choose which to query: **local logs** (this machine's own recordings — +`hyp query sql …`, no `--remote`) and **each remote HypAware server** (every target from +`hyp remote list`, plus any hypaware MCP server already available to you as MCP tools (a +`query_sql` / `graph_neighbors` tool in your toolset); the same server can appear both ways — +list it once). Present the options, +ask which one (or more) to profile, then proceed against the chosen source. + +**Descriptive only** — route every *action* out: "fan out more/less" and tooling go to +**hypaware-ai-improvement-report**, token waste to **hypaware-ai-spend-report**. Query mechanics +live in the **hypaware-query** skill; reuse hypaware-ai-spend-report's deduped token spine for +any token figure. ## Procedure 1. **Scope + coverage.** Distinct `gateway_id` (the unit — `user_id` is ~always null, so don't @@ -39,7 +47,7 @@ for any token figure. route the "fan out more/less" recommendation to hypaware-ai-improvement-report. ## Output — SAVE A MARKDOWN FILE -- **Path:** `docs/adoption-profiles/-adoption-profile.md` (create the dir if needed). +- **Path:** `hypaware-reports/-adoption-profile.md` (create the dir if needed). Dated so reports accumulate. - **Bottom line (1 sentence):** how many gateways are active, the busiest one and its focus, and the headline fan-out adoption %. @@ -47,11 +55,10 @@ for any token figure. (messages / sessions) · **tokens** · **cache-read %** · **fan-out** (adoption + main-vs-subagent split) · **focus label**. Busiest first — this table is the spine. - **Then the detail**, in these sections: **Scope & coverage · Per-gateway utilization · - Per-gateway focus · Parallelism & fan-out · Payoff (descriptive) · Fleet view · Caveats** — - plus every query you ran. + Per-gateway focus · Parallelism & fan-out · Payoff (descriptive) · Fleet view · Caveats**. - **Formatting (human-readable):** open each section with its takeaway; **bold** headline numbers; one sortable footprint table as the centerpiece; keep the bottom line + table a ~1-minute read. - **Capture-health note:** if subagent provenance doesn't reach the server, the standing #1 - caveat is "subagent identity must survive ingest" — run adoption off the `Task`-call proxy and - flag it. + caveat is "subagent identity must survive ingest" — run adoption off the sub-agent-invocation + proxy (the tool calls that spawn sub-agents) and flag it. diff --git a/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-improvement-report/SKILL.md b/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-improvement-report/SKILL.md index 8308db7f..50a17b99 100644 --- a/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-improvement-report/SKILL.md +++ b/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-improvement-report/SKILL.md @@ -9,7 +9,14 @@ Read how the team's agents actually behave in the logs, then propose the **addit edits to skills, subagents, and AGENTS.md/CLAUDE.md** that make them do **better work (quality)** with **less wasted effort and spend**. -IMPORTANT: Query server logs not local logs +IMPORTANT: Don't assume which logs to read — **ask first.** Start by listing the data sources +and let the user choose which to query: **local logs** (this machine's own recordings — +`hyp query sql …`, no `--remote`) and **each remote HypAware server** (every target from +`hyp remote list`, plus any hypaware MCP server already available to you as MCP tools (a +`query_sql` / `graph_neighbors` tool in your toolset); the same server can appear both ways — +list it once). Present the options, +ask which one (or more) to run the review against, then proceed against the chosen source. +Query mechanics live in the **hypaware-query** skill; read it first. Focus on these signals: @@ -27,7 +34,7 @@ Focus on these signals: 2. **Scan the signals for possible improvements.** Work the three signals above; each turns up candidates. Note frequency (sessions, distinct gateways) and redact examples. - **Repeated work** — cluster sessions by shared-file overlap and tool-set signature into - recurring kinds of work (reuse a recent `docs/spend-reviews/` *Leverage candidates* + recurring kinds of work (reuse a recent `hypaware-reports/` *Leverage candidates* handoff if present). Flag parallelizable work done serially (low `is_sidechain`, few `agent_id`), and recurring asks / multi-step workflows / re-sent instructions in sampled prompts + `system_text`. @@ -45,7 +52,7 @@ Focus on these signals: rank by it. ## Output — SAVE A MARKDOWN FILE -- **Path:** `docs/improvement-reviews/-improvement-review.md` (create the dir +- **Path:** `hypaware-reports/-improvement-review.md` (create the dir if needed). Dated so reviews accumulate. - **Bottom line (1 sentence):** "Create/edit these N things and what it will improve.". - **Improvements (ranked table):** one row per improvement — **what** · **suggested form** @@ -53,8 +60,7 @@ Focus on these signals: gateways, sticking-point impact, or token prize) · **where it lands**. Highest-leverage first. - **Then the detail**, in these sections: **Inventory (what exists) · Basis · Skill candidates · Subagent candidates (fan-out + roles) · AGENTS.md/CLAUDE.md edits · - Efficiency → setup changes · Caveats** — plus every query you ran and the proposed - AGENTS.md lines verbatim. + Efficiency → setup changes · Caveats** — plus the proposed AGENTS.md lines verbatim. - **Formatting (human-readable):** every candidate states its evidence inline; show AGENTS.md edits as real diffs/code blocks, not prose; **bold** the artifact type; keep it a ~1-minute read. diff --git a/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-security-report/SKILL.md b/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-security-report/SKILL.md index bafe0576..4eafe2fb 100644 --- a/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-security-report/SKILL.md +++ b/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-security-report/SKILL.md @@ -1,19 +1,26 @@ --- name: hypaware-ai-security-report -description: Security & Risk Review for a HypAware server — audits gateway logs for risky autonomous activity (destructive commands, remote-exec, privilege escalation, secret reads, network egress, package installs), severity-ranks findings, and recommends guardrails. Saves a dated report under docs/security-reviews/; queries the server via the hypaware-query skill. Audits recorded logs, NOT pending code — for a code diff use the built-in /security-review. +description: Security & Risk Review for a HypAware server — audits gateway logs for risky autonomous activity (destructive commands, remote-exec, privilege escalation, secret reads, network egress, package installs), severity-ranks findings, and recommends guardrails. Saves a dated report under hypaware-reports/; first asks which HypAware source to query (local logs or a remote server) via the hypaware-query skill. Audits recorded logs, NOT pending code — for a code diff use a dedicated code-review tool. --- # Security & Risk Review Turn a window of central AI-gateway logs into a severity-ranked picture of **what agents did unsupervised that carries risk**, and the guardrails to contain it. This audits *recorded -activity*, not pending code (for a diff, use the built-in `/security-review`). Be +activity*, not pending code (for a diff, use a dedicated code-review tool). Be proportionate: separate "ran a risky command" from "caused harm". -IMPORTANT: Query the server, not local logs. **REDACT everything** — never echo a secret, -token, key, or credential value in the report, even one you found in `tool_args`; a raw secret -appearing in args is itself a capture finding (flag it, don't reproduce it). Query mechanics -live in the **hypaware-query** skill; read it first. +IMPORTANT: Don't assume which logs to read — **ask first.** Start by listing the data sources +and let the user choose which to query: **local logs** (this machine's own recordings — +`hyp query sql …`, no `--remote`) and **each remote HypAware server** (every target from +`hyp remote list`, plus any hypaware MCP server already available to you as MCP tools (a +`query_sql` / `graph_neighbors` tool in your toolset); the same server can appear both ways — +list it once). Present the options, +ask which one (or more) to audit, then proceed against the chosen source. + +**REDACT everything** — never echo a secret, token, key, or credential value in the report, +even one you found in `tool_args`; a raw secret appearing in args is itself a capture finding +(flag it, don't reproduce it). Query mechanics live in the **hypaware-query** skill; read it first. ## Risk classes (match over the FULL command text, not just the first token) | Class | Match signals | Why it matters | @@ -37,12 +44,13 @@ than the same command in a gated session. match the full text against each risk class: count, ONE redacted example, and the `gateway_id`/`git_branch` where it ran. Rank by frequency × blast radius × autonomy; keep "ran" vs "caused harm" distinct, and note anything that ran in a *gated* (non-bypass) session. -3. **Guardrails.** For each top risk, a concrete control + estimated coverage: a PreToolUse - **hook** (deny/allow), a **permission policy / allowlist**, a **CLAUDE.md** rule, or a - **gateway redaction** fix for any secret seen in args. +3. **Guardrails.** For each top risk, a concrete control + estimated coverage: a tool-exec + **guardrail** (a pre-exec hook or approval policy, deny/allow), a **permission / allowlist + policy**, an **agent-instructions rule** (AGENTS.md / CLAUDE.md), or a **gateway redaction** + fix for any secret seen in args. ## Output — SAVE A MARKDOWN FILE -- **Path:** `docs/security-reviews/-security-review.md` (create the dir if needed). +- **Path:** `hypaware-reports/-security-review.md` (create the dir if needed). Dated so reports accumulate. - **Bottom line (1 sentence):** the posture — how autonomous the fleet is (% `bypassPermissions`) and whether anything needs urgent attention. @@ -50,7 +58,7 @@ than the same command in a gated session. (high / med / low) · **count** · **example (redacted)** · **guardrail** · **where it ran**. Highest severity first. - **Then the detail**, in these sections: **Autonomy baseline · Command profile · Risk findings - · Guardrails · Trends · Capture caveats** — plus every query you ran. **All examples redacted.** + · Guardrails · Trends · Capture caveats**. **All examples redacted.** - **Formatting (human-readable):** open each section with its takeaway; **bold** severities; one severity-ranked findings table as the centerpiece; keep the bottom line + table a ~1-minute read. - **Capture-health note:** if raw credentials appear in `tool_args`, the standing #1 finding is diff --git a/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-spend-report/SKILL.md b/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-spend-report/SKILL.md index 163da783..b04ffa93 100644 --- a/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-spend-report/SKILL.md +++ b/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-spend-report/SKILL.md @@ -1,6 +1,6 @@ --- name: hypaware-ai-spend-report -description: AI Spend Review for a HypAware server — where tokens go (by user/repo/model/gateway), what work they go on (graph-clustered), and how to spend less (waste scorecard + cost levers). Deduped token volume, never dollars. Saves a dated report under docs/spend-reviews/; queries the server via the hypaware-query skill. +description: AI Spend Review for a HypAware server — where tokens go (by user/repo/model/gateway), what work they go on (graph-clustered), and how to spend less (waste scorecard + cost levers). Deduped token volume, never dollars. Saves a dated report under hypaware-reports/; first asks which HypAware source to query (local logs or a remote server) via the hypaware-query skill. --- # AI Spend Review @@ -10,9 +10,17 @@ spine. One story in three moves: **where it goes** (by user / repo / model / gat time), **on what work** (sessions clustered into recurring work-types, sized by tokens), and **how to spend less** (a waste scorecard + ranked cost levers). -IMPORTANT: Query the server, not local logs. **Tokens, never dollars** — capture is partial, -so stop at token volume and behavioral proxies. Query mechanics (`--server`, date-pruning the -30s timeout, the SQL dialect, sampling) live in the **hypaware-query** skill; read it first. +IMPORTANT: Don't assume which logs to read — **ask first.** Start by listing the data sources +and let the user choose which to query: **local logs** (this machine's own recordings — +`hyp query sql …`, no `--remote`) and **each remote HypAware server** (every target from +`hyp remote list`, plus any hypaware MCP server already available to you as MCP tools (a +`query_sql` / `graph_neighbors` tool in your toolset); the same server can appear both ways — +list it once). Present the options, +ask which one (or more) to run the review against, then proceed against the chosen source. + +**Tokens, never dollars** — capture is partial, so stop at token volume and behavioral proxies. +Query mechanics (`--remote`, date-pruning the 30s timeout, the SQL dialect, sampling) live in +the **hypaware-query** skill; read it first. ## Token math (get this right — every breakdown reconciles to it) Usage is in `attributes.usage` (NOT `raw_frame`): `input_tokens`, `output_tokens`, @@ -53,7 +61,7 @@ SELECT sum(inp) t_in, sum(outp) t_out, sum(cwrite) t_cw, sum(cread) t_cr FROM ms candidates (surface them; don't build them here). ## Output — SAVE A MARKDOWN FILE -- **Path:** `docs/spend-reviews/-spend-review.md` (create the dir if needed). Dated +- **Path:** `hypaware-reports/-spend-review.md` (create the dir if needed). Dated so reviews accumulate. - **Bottom line (1 sentence):** total tokens this window, the trend vs the last report (▲/▼ %), and the single biggest driver. @@ -61,8 +69,7 @@ SELECT sum(inp) t_in, sum(outp) t_out, sum(cwrite) t_cw, sum(cread) t_cr FROM ms + cause) · **est. weekly saving** · **where it lands**. Highest-saving first. - **Then the detail**, in these sections: **Coverage & footprint · Total spend (4 token types) · Allocation (user / repo / model / gateway) · Trends · Outliers · Spend by work-type · Waste - scorecard · Leverage candidates (handoff to hypaware-ai-improvement-report) · Caveats** — plus every query - you ran. + scorecard · Leverage candidates (handoff to hypaware-ai-improvement-report) · Caveats**. - **Formatting (human-readable):** open each section with its one-sentence takeaway, then the table; **bold** headline numbers; split the four token types (never collapse them); keep the bottom line + ranked table a ~1-minute read. diff --git a/hypaware-core/plugins-workspace/claude/skills/hypaware-query/SKILL.md b/hypaware-core/plugins-workspace/claude/skills/hypaware-query/SKILL.md index 748a99ef..f53b9624 100644 --- a/hypaware-core/plugins-workspace/claude/skills/hypaware-query/SKILL.md +++ b/hypaware-core/plugins-workspace/claude/skills/hypaware-query/SKILL.md @@ -5,7 +5,7 @@ description: Inspect local HypAware recordings with the hyp query CLI. Use when # HypAware Query -Use `hyp query` to inspect local HypAware recordings. It reads local JSONL recordings and an explicit local query cache; it does not query the central server. +Use `hyp query` to inspect local HypAware recordings. By default it reads local JSONL recordings and an explicit local query cache, not the central server. To run the same query against a remote HypAware host (a fleet server) over its MCP endpoint, add `--remote ` — see [Remote queries](#remote-queries-other-hypaware-hosts). ## Workflow @@ -33,6 +33,33 @@ hyp collect remove These are the only subcommands in the installed CLI (`hyp query`: schema, status, sql, refresh, maintain; `hyp collect`: list, remove). There are no high-level `catalog`/`logs`/`traces`/`metrics` query commands — answer questions with `hyp query sql`, and discover datasets from the `hyp query status` output. +## Remote queries (other HypAware hosts) + +By default `hyp query` is local-only. To run a verb against a remote HypAware host (a fleet server) over its MCP endpoint (`/v1/mcp`), add `--remote `: `hyp` acts as an MCP client, runs the same SQL against the remote `query_sql` tool, and renders the result with the same formatter. Only read-class tools are reachable remotely (`query_sql`, `graph_neighbors`); the credential is **query-scoped** (read/compute only — it cannot author configs or mint tokens), distinct from the server's operator/admin token, which never leaves the server. + +- **Discover configured targets:** `hyp remote list` (`--json` for machine output). Each row shows the target URL and a `token:` status — `env` (a `HYP_REMOTE_TOKEN_` var is set), `stored` (saved by `hyp remote login`), or `missing`. This reflects local config + credentials only; it is **not** a liveness check. The real connectivity/auth test is running a `--remote` query: rows back means reachable + authorized; a 401/timeout tells you which half failed. +- **Set up a target (two steps):** `hyp remote add ` registers the URL — pass the full endpoint including the path (e.g. `https://host:8740/v1/mcp`); it is used verbatim, never auto-suffixed. Then supply the query-scoped token one of two ways: `hyp remote login ` (token via `--token-file ` or piped stdin — never a CLI argument, never an interactive prompt), or a per-target env var `HYP_REMOTE_TOKEN_` (name uppercased, non-alphanumeric runs → `_`; e.g. `prod` → `HYP_REMOTE_TOKEN_PROD`). The env var is checked first and wins. +- **Query it:** `hyp query sql "" --remote --format json`. +- **Truncation is doubled on remote — read both stderr lines.** A server-side data cap (`remote: showing first N rows (server cap …)`) clips before rows leave the server and you **cannot** lift it; the usual local display budget (`notice:` / `--output`) clips again on your side. Never `2>/dev/null` a remote query. +- **`--remote` together with `--refresh` is a hard error** — refresh is a local-cache operation, meaningless against a server that owns its own freshness. +- A remote target may be reachable only over a private network (e.g. a tailnet / `100.x` address); a timeout often means you are off that network, not that the server is down. + +### Two ways a HypAware host's MCP may be attached + +A HypAware host exposes its read-class verbs (`query_sql`, `graph_neighbors`) as an **MCP tool**, and that MCP can be attached by **two independent routes** — be aware of both: + +- **Via `hyp --remote`** — the CLI path above: `hyp` acts as the MCP client (`hyp query sql … --remote `) and renders locally. Discover these by running `hyp remote list`. +- **Via a direct client connection** — the host's `/v1/mcp` endpoint is registered in this client's MCP config (out of band), surfacing the `query_sql` / `graph_neighbors` **tools** directly as the `mcp__hypaware__*` tools already in your toolset — no `hyp` in the data path. + +The routes are independent, so the **same server may be attached both ways at once** — an `mcp__hypaware__*` tool and a `hyp remote list` target can point at the identical `/v1/mcp` URL. Expect that overlap; don't treat them as two different servers. + +Both routes run the identical `query_sql` operation, so **the data is the same** — but the surfaces are **not byte-identical**: + +- **MCP tool:** returns the **full structured result** (every matching row, as JSON) with **no ~32 KB display budget**; a large result can overflow the AI client's own output limit and spill to a file. +- **`hyp --remote` CLI:** applies the ~32 KB display budget and prints `notice: showing N of M rows …` on stderr; lift it with `--max-bytes 0` or `--output ` to recover the tool's full set. + +Never read a smaller CLI row count as "fewer rows matched" — it is the display budget, not the result set. + ## SQL dialect notes - `json_extract_scalar()` does not exist. `JSON_EXTRACT` does, but it errors on rows where a JSON-typed column (notably `tool_args`) holds a plain string instead of a JSON object ("first argument must be JSON string or object, got string"). @@ -49,6 +76,8 @@ Key columns: - `tool_name`, `tool_call_id`, `tool_args`, `status` — tool-call/result joins and sparse status such as `finish_reason`. - `attributes` (JSON) — request settings, usage, propagated `dev_run_id`, and gateway diagnostics under `attributes.gateway`. +**Token counts** live under `attributes.usage` on `role='assistant'` rows (NOT in `raw_frame`): `input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_write_tokens`. Codex (`provider='openai'`) omits `cache_write_tokens` and adds `reasoning_tokens` + `total_tokens`. Extract with `CAST(JSON_EXTRACT(attributes,'$.usage.input_tokens') AS BIGINT)`; usage repeats across a message's content parts, so dedup per `message_id` (e.g. `max(...)`) before summing. + Claude transcript enrichment adds `provider_uuid`, `parent_uuid`, `request_id`, `entrypoint`, `client_version`, `user_type`, `permission_mode`, and `hook_event` when the local Claude Code JSONL transcript can be matched. Run `hyp query schema ai_gateway_messages --format markdown` for the authoritative column reference. diff --git a/hypaware-core/plugins-workspace/codex/hypaware.plugin.json b/hypaware-core/plugins-workspace/codex/hypaware.plugin.json index 389e522c..2244ed87 100644 --- a/hypaware-core/plugins-workspace/codex/hypaware.plugin.json +++ b/hypaware-core/plugins-workspace/codex/hypaware.plugin.json @@ -2,7 +2,7 @@ "schema_version": 1, "name": "@hypaware/codex", "version": "2.0.0", - "description": "OpenAI Codex client adapter for HypAware. Registers the OpenAI-compatible upstream preset on the local AI gateway, configures Codex's config.toml to route through the gateway, and ships the hypaware-query helper skill.", + "description": "OpenAI Codex client adapter for HypAware. Registers the OpenAI-compatible upstream preset on the local AI gateway, configures Codex's config.toml to route through the gateway, and ships the hypaware-query helper skill plus the AI adoption/improvement/security/spend report skills.", "hypaware_api": "^1.0.0", "runtime": "node", "node_engine": ">=20", @@ -30,7 +30,11 @@ "required_upstreams": ["openai", "chatgpt"] }, "skills": [ - { "name": "hypaware-query", "clients": ["codex"] } + { "name": "hypaware-query", "clients": ["codex"] }, + { "name": "hypaware-ai-adoption-report", "clients": ["codex"] }, + { "name": "hypaware-ai-improvement-report", "clients": ["codex"] }, + { "name": "hypaware-ai-security-report", "clients": ["codex"] }, + { "name": "hypaware-ai-spend-report", "clients": ["codex"] } ] } } diff --git a/hypaware-core/plugins-workspace/codex/skills/hypaware-ai-adoption-report/SKILL.md b/hypaware-core/plugins-workspace/codex/skills/hypaware-ai-adoption-report/SKILL.md new file mode 100644 index 00000000..5c6ab15b --- /dev/null +++ b/hypaware-core/plugins-workspace/codex/skills/hypaware-ai-adoption-report/SKILL.md @@ -0,0 +1,64 @@ +--- +name: hypaware-ai-adoption-report +description: AI Adoption Profile for a HypAware server — descriptive "who's using the fleet and how": per-gateway utilization (volume + focus — top models, tools, repos, themes) and parallelism/fan-out (multi-agent adoption, concurrency, main-vs-subagent split, payoff). +--- + +# AI Adoption Profile + +The one **descriptive** report on *who's using the fleet and how* — no actions, just a clear +picture. Two lenses over one window: + +- **Utilization** — per `gateway_id` (≈ one machine/user): *how much* (messages, sessions, + active days, tokens, cache-read ratio) and *what it's focused on* (top models, tools, repos, + work themes). +- **Parallelism / fan-out** — *how sophisticated*: multi-agent adoption, fan-out breadth/depth, + true concurrency vs serial, the main-loop-vs-subagent token split, and whether fan-out appears + to earn its cost. + +IMPORTANT: Don't assume which logs to read — **ask first.** Start by listing the data sources +and let the user choose which to query: **local logs** (this machine's own recordings — +`hyp query sql …`, no `--remote`) and **each remote HypAware server** (every target from +`hyp remote list`, plus any hypaware MCP server already available to you as MCP tools (a +`query_sql` / `graph_neighbors` tool in your toolset); the same server can appear both ways — +list it once). Present the options, +ask which one (or more) to profile, then proceed against the chosen source. + +**Descriptive only** — route every *action* out: "fan out more/less" and tooling go to +**hypaware-ai-improvement-report**, token waste to **hypaware-ai-spend-report**. Query mechanics +live in the **hypaware-query** skill; reuse hypaware-ai-spend-report's deduped token spine for +any token figure. + +## Procedure +1. **Scope + coverage.** Distinct `gateway_id` (the unit — `user_id` is ~always null, so don't + measure reach by it); window; coverage of `gateway_id`, token usage, and subagent provenance + (`agent_id` / `is_sidechain` / `parent_thread_id` — transcript-enriched, may not survive + ingest). Decide cost-capable vs volume-only, and which parallelism dimensions are real vs + proxied (the `Task`-call proxy). State N; if it's effectively one gateway / dogfood, say so. +2. **Per-gateway utilization.** One row per gateway: volume (messages, sessions, active days, + first/last seen), tokens + cache-read ratio `cache_read/(cache_read+input)`, then its focus — + top models (+ `(unknown)`), tools (Bash dominance + top commands), repos, client + (claude/codex), and 2–4 recurring work themes (sampled, redacted). Distill each into a + one-line **focus label**. Use the activity graph for structural focus (Session→Repo/PR via + GitHub enrichment) where projected. +3. **Parallelism / fan-out.** Adoption (% conversations with ≥1 subagent, incl. the zero + bucket); breadth (subagents per parent) and depth (`parent_thread_id` chains); concurrency + (do subagent time-spans actually *overlap*, or is it serial?); cost split (token share main + loop vs subagent). Read the payoff *descriptively* — fan-out vs tokens-to-resolution — and + route the "fan out more/less" recommendation to hypaware-ai-improvement-report. + +## Output — SAVE A MARKDOWN FILE +- **Path:** `hypaware-reports/-adoption-profile.md` (create the dir if needed). + Dated so reports accumulate. +- **Bottom line (1 sentence):** how many gateways are active, the busiest one and its focus, and + the headline fan-out adoption %. +- **Per-gateway profile (sortable table):** one row per gateway — **gateway** · **volume** + (messages / sessions) · **tokens** · **cache-read %** · **fan-out** (adoption + main-vs-subagent + split) · **focus label**. Busiest first — this table is the spine. +- **Then the detail**, in these sections: **Scope & coverage · Per-gateway utilization · + Per-gateway focus · Parallelism & fan-out · Payoff (descriptive) · Fleet view · Caveats**. +- **Formatting (human-readable):** open each section with its takeaway; **bold** headline + numbers; one sortable footprint table as the centerpiece; keep the bottom line + table a + ~1-minute read. +- **Capture-health note:** if subagent provenance doesn't reach the server, the standing #1 + caveat is "subagent identity must survive ingest" — run adoption off the sub-agent-invocation + proxy (the tool calls that spawn sub-agents) and flag it. diff --git a/hypaware-core/plugins-workspace/codex/skills/hypaware-ai-improvement-report/SKILL.md b/hypaware-core/plugins-workspace/codex/skills/hypaware-ai-improvement-report/SKILL.md new file mode 100644 index 00000000..50a17b99 --- /dev/null +++ b/hypaware-core/plugins-workspace/codex/skills/hypaware-ai-improvement-report/SKILL.md @@ -0,0 +1,66 @@ +--- +name: hypaware-ai-improvement-report +description: Analyze AI agent logs and generate a markdown report identifying additions or modifications to skills, subagents, and AGENTS.md/CLAUDE.md, to improve quality and efficiency of the AI agent performance. +--- + +# AI Improvement Review + +Read how the team's agents actually behave in the logs, then propose the **additions and +edits to skills, subagents, and AGENTS.md/CLAUDE.md** that make them do **better work +(quality)** with **less wasted effort and spend**. + +IMPORTANT: Don't assume which logs to read — **ask first.** Start by listing the data sources +and let the user choose which to query: **local logs** (this machine's own recordings — +`hyp query sql …`, no `--remote`) and **each remote HypAware server** (every target from +`hyp remote list`, plus any hypaware MCP server already available to you as MCP tools (a +`query_sql` / `graph_neighbors` tool in your toolset); the same server can appear both ways — +list it once). Present the options, +ask which one (or more) to run the review against, then proceed against the chosen source. +Query mechanics live in the **hypaware-query** skill; read it first. + +Focus on these signals: + +- **Repeated work** — work the team repeats successfully → package it once (skill / subagent). +- **Sticking points** — where agents get stuck or redo work (errors, loops, refusals, + abandonment) → the missing or too-weak instruction that would prevent it (an AGENTS.md + rule, or a skill). +- **Inefficiency** — expensive patterns (model over-spec, context bloat, low cache-read, + retry loops) → the setup change that costs less (right-size the model in AGENTS.md / a + subagent, a context-hygiene rule, a skill that avoids the redo). + +## Procedure +1. **Footprint + basis.** Window; distinct contributors (`gateway_id`), repos, sessions; + claude/codex mix. State N and breadth; flag if single-contributor. +2. **Scan the signals for possible improvements.** Work the three signals above; each turns + up candidates. Note frequency (sessions, distinct gateways) and redact examples. + - **Repeated work** — cluster sessions by shared-file overlap and tool-set signature into + recurring kinds of work (reuse a recent `hypaware-reports/` *Leverage candidates* + handoff if present). Flag parallelizable work done serially (low `is_sidechain`, few + `agent_id`), and recurring asks / multi-step workflows / re-sent instructions in + sampled prompts + `system_text`. + - **Sticking points** — where agents got stuck or redid work, ranked by impact: failing + tools (`is_error` by `tool_name`), retry loops (same tool + first `tool_args` token + ≥3×/session), refusals/truncations (stop-reason), abandoned costly sessions, + repeatedly-violated conventions. + - **Inefficiency** (reuse the spend spine) — model over-spec, context bloat (no + `is_compact_summary`), low cache-read (`cache_read/(cache_read+input)`), redo loops. +3. **Collect, dedup, prioritize.** Gather the candidates into a list of possible + improvements; drop anything the inventory already covers. For each, suggest a likely form + where one obviously fits — a **skill**, a **subagent**, or an **AGENTS.md/CLAUDE.md + edit** (propose the concrete line(s) as a diff, not "consider documenting") — but don't + force a mapping. Attach evidence (frequency/impact + distinct gateways + token prize) and + rank by it. + +## Output — SAVE A MARKDOWN FILE +- **Path:** `hypaware-reports/-improvement-review.md` (create the dir + if needed). Dated so reviews accumulate. +- **Bottom line (1 sentence):** "Create/edit these N things and what it will improve.". +- **Improvements (ranked table):** one row per improvement — **what** · **suggested form** + (skill / subagent / AGENTS.md edit, where one fits) · **evidence** (recurs N× across G + gateways, sticking-point impact, or token prize) · **where it lands**. Highest-leverage first. +- **Then the detail**, in these sections: **Inventory (what exists) · Basis · Skill + candidates · Subagent candidates (fan-out + roles) · AGENTS.md/CLAUDE.md edits · + Efficiency → setup changes · Caveats** — plus the proposed AGENTS.md lines verbatim. +- **Formatting (human-readable):** every candidate states its evidence inline; show + AGENTS.md edits as real diffs/code blocks, not prose; **bold** the artifact type; keep + it a ~1-minute read. diff --git a/hypaware-core/plugins-workspace/codex/skills/hypaware-ai-security-report/SKILL.md b/hypaware-core/plugins-workspace/codex/skills/hypaware-ai-security-report/SKILL.md new file mode 100644 index 00000000..4eafe2fb --- /dev/null +++ b/hypaware-core/plugins-workspace/codex/skills/hypaware-ai-security-report/SKILL.md @@ -0,0 +1,65 @@ +--- +name: hypaware-ai-security-report +description: Security & Risk Review for a HypAware server — audits gateway logs for risky autonomous activity (destructive commands, remote-exec, privilege escalation, secret reads, network egress, package installs), severity-ranks findings, and recommends guardrails. Saves a dated report under hypaware-reports/; first asks which HypAware source to query (local logs or a remote server) via the hypaware-query skill. Audits recorded logs, NOT pending code — for a code diff use a dedicated code-review tool. +--- + +# Security & Risk Review + +Turn a window of central AI-gateway logs into a severity-ranked picture of **what agents did +unsupervised that carries risk**, and the guardrails to contain it. This audits *recorded +activity*, not pending code (for a diff, use a dedicated code-review tool). Be +proportionate: separate "ran a risky command" from "caused harm". + +IMPORTANT: Don't assume which logs to read — **ask first.** Start by listing the data sources +and let the user choose which to query: **local logs** (this machine's own recordings — +`hyp query sql …`, no `--remote`) and **each remote HypAware server** (every target from +`hyp remote list`, plus any hypaware MCP server already available to you as MCP tools (a +`query_sql` / `graph_neighbors` tool in your toolset); the same server can appear both ways — +list it once). Present the options, +ask which one (or more) to audit, then proceed against the chosen source. + +**REDACT everything** — never echo a secret, token, key, or credential value in the report, +even one you found in `tool_args`; a raw secret appearing in args is itself a capture finding +(flag it, don't reproduce it). Query mechanics live in the **hypaware-query** skill; read it first. + +## Risk classes (match over the FULL command text, not just the first token) +| Class | Match signals | Why it matters | +| --- | --- | --- | +| Destructive | `rm -rf`, `git reset --hard`, `git push --force`, `drop`/`truncate`, `dd`, `mkfs` | irreversible data/history loss | +| Remote-exec | `curl`/`wget … \| sh`/`bash`, piped installers | runs unreviewed remote code | +| Privilege | `sudo`, `su`, `chmod 777`, writing system paths | escalation beyond the workspace | +| Secret read | `.env`, `id_rsa`, `.ssh`, `.aws`, `.netrc`, `AWS_*`, `*token*`, `credentials` | credential exposure | +| Network egress | outbound `curl`/`scp`/`nc` POSTs to external hosts | potential exfiltration | +| Package install | `npm i`, `pip install`, `brew install`, `apt` | supply-chain surface | + +Risk is **amplified by autonomy** — weight anything that ran under `bypassPermissions` higher +than the same command in a gated session. + +## Procedure +1. **Coverage + autonomy baseline.** Window; coverage of `tool_args` on Bash calls and of + `permission_mode` (bound the read if sparse); distinct `gateway_id` (the unit — `user_id` is + ~always null). Total tool calls, % Bash, % conversations in `bypassPermissions`. State N; if + it's a dogfood window, say so. +2. **Scan + rank.** Histogram the first token of each Bash command (top 30 = the normal), then + match the full text against each risk class: count, ONE redacted example, and the + `gateway_id`/`git_branch` where it ran. Rank by frequency × blast radius × autonomy; keep + "ran" vs "caused harm" distinct, and note anything that ran in a *gated* (non-bypass) session. +3. **Guardrails.** For each top risk, a concrete control + estimated coverage: a tool-exec + **guardrail** (a pre-exec hook or approval policy, deny/allow), a **permission / allowlist + policy**, an **agent-instructions rule** (AGENTS.md / CLAUDE.md), or a **gateway redaction** + fix for any secret seen in args. + +## Output — SAVE A MARKDOWN FILE +- **Path:** `hypaware-reports/-security-review.md` (create the dir if needed). + Dated so reports accumulate. +- **Bottom line (1 sentence):** the posture — how autonomous the fleet is (% `bypassPermissions`) + and whether anything needs urgent attention. +- **Findings (severity-ranked table):** one row per finding — **risk class** · **severity** + (high / med / low) · **count** · **example (redacted)** · **guardrail** · **where it ran**. + Highest severity first. +- **Then the detail**, in these sections: **Autonomy baseline · Command profile · Risk findings + · Guardrails · Trends · Capture caveats**. **All examples redacted.** +- **Formatting (human-readable):** open each section with its takeaway; **bold** severities; one + severity-ranked findings table as the centerpiece; keep the bottom line + table a ~1-minute read. +- **Capture-health note:** if raw credentials appear in `tool_args`, the standing #1 finding is + "gateway must redact secrets in tool_args" — put it in Capture caveats without reproducing it. diff --git a/hypaware-core/plugins-workspace/codex/skills/hypaware-ai-spend-report/SKILL.md b/hypaware-core/plugins-workspace/codex/skills/hypaware-ai-spend-report/SKILL.md new file mode 100644 index 00000000..b04ffa93 --- /dev/null +++ b/hypaware-core/plugins-workspace/codex/skills/hypaware-ai-spend-report/SKILL.md @@ -0,0 +1,75 @@ +--- +name: hypaware-ai-spend-report +description: AI Spend Review for a HypAware server — where tokens go (by user/repo/model/gateway), what work they go on (graph-clustered), and how to spend less (waste scorecard + cost levers). Deduped token volume, never dollars. Saves a dated report under hypaware-reports/; first asks which HypAware source to query (local logs or a remote server) via the hypaware-query skill. +--- + +# AI Spend Review + +Where the team's tokens go, on what work, and how to spend less — read off one deduped token +spine. One story in three moves: **where it goes** (by user / repo / model / gateway, over +time), **on what work** (sessions clustered into recurring work-types, sized by tokens), and +**how to spend less** (a waste scorecard + ranked cost levers). + +IMPORTANT: Don't assume which logs to read — **ask first.** Start by listing the data sources +and let the user choose which to query: **local logs** (this machine's own recordings — +`hyp query sql …`, no `--remote`) and **each remote HypAware server** (every target from +`hyp remote list`, plus any hypaware MCP server already available to you as MCP tools (a +`query_sql` / `graph_neighbors` tool in your toolset); the same server can appear both ways — +list it once). Present the options, +ask which one (or more) to run the review against, then proceed against the chosen source. + +**Tokens, never dollars** — capture is partial, so stop at token volume and behavioral proxies. +Query mechanics (`--remote`, date-pruning the 30s timeout, the SQL dialect, sampling) live in +the **hypaware-query** skill; read it first. + +## Token math (get this right — every breakdown reconciles to it) +Usage is in `attributes.usage` (NOT `raw_frame`): `input_tokens`, `output_tokens`, +`cache_read_tokens`, `cache_write_tokens` (+ `reasoning_tokens` for Codex). Dedup by message +and `max()` — a plain `SUM` overcounts ~3×. Report the four types separately (cache-read is +usually the bulk; output the scarce slice). + +```sql +WITH msg AS ( + SELECT COALESCE(CAST(JSON_EXTRACT(raw_frame,'$.message_id') AS VARCHAR), message_id) mid, + max(CAST(JSON_EXTRACT(attributes,'$.usage.input_tokens') AS BIGINT)) inp, + max(CAST(JSON_EXTRACT(attributes,'$.usage.output_tokens') AS BIGINT)) outp, + max(CAST(JSON_EXTRACT(attributes,'$.usage.cache_write_tokens') AS BIGINT)) cwrite, + max(CAST(JSON_EXTRACT(attributes,'$.usage.cache_read_tokens') AS BIGINT)) cread + FROM ai_gateway_messages + WHERE date BETWEEN '' AND '' + AND role='assistant' AND JSON_EXTRACT(attributes,'$.usage') IS NOT NULL + GROUP BY mid) +SELECT sum(inp) t_in, sum(outp) t_out, sum(cwrite) t_cw, sum(cread) t_cr FROM msg; +-- slice by adding max(gateway_id)/max(model)/max(repo_root)/max(date) inside, GROUP BY outside. +``` + +## Procedure +1. **Coverage + footprint.** Window; usage coverage, `model`-column coverage (token-weighted), + user/repo coverage; distinct gateways, sessions, claude/codex mix. `user_id` is ~always null + → attribute by `gateway_id` (≈ one machine/user) and always show an explicit **unattributed** + row. State N; if it's a dogfood window, say so. If usage is thin, fall back to behavioral + proxies (turns, tool calls, length), labeled as estimates. +2. **Where it goes + on what work.** Slice the spine by user / repo / model (→ `(unknown)` + bucket) / gateway with shares, plus trends (by week, WoW deltas vs the last report) and + top-spend outliers. Then cluster sessions into recurring **work-types** — shared-file overlap + for code work, tool-set signature for no-file work (context graph if projected, else SQL) — + and size each by deduped tokens. +3. **How to spend less.** Score the waste — cache-read ratio `cache_read/(cache_read+input)` + (the biggest lever — feature it), retry loops, abandoned costly sessions, model over-spec, + context bloat — and turn the top few into ranked cost levers, each with an estimated weekly + token saving. Hand the expensive recurring work-types to **hypaware-ai-improvement-report** as packaging + candidates (surface them; don't build them here). + +## Output — SAVE A MARKDOWN FILE +- **Path:** `hypaware-reports/-spend-review.md` (create the dir if needed). Dated + so reviews accumulate. +- **Bottom line (1 sentence):** total tokens this window, the trend vs the last report (▲/▼ %), + and the single biggest driver. +- **Cost actions (ranked table):** one row per lever — **action** · **evidence** (tokens/share + + cause) · **est. weekly saving** · **where it lands**. Highest-saving first. +- **Then the detail**, in these sections: **Coverage & footprint · Total spend (4 token types) + · Allocation (user / repo / model / gateway) · Trends · Outliers · Spend by work-type · Waste + scorecard · Leverage candidates (handoff to hypaware-ai-improvement-report) · Caveats**. +- **Formatting (human-readable):** open each section with its one-sentence takeaway, then the + table; **bold** headline numbers; split the four token types (never collapse them); keep the + bottom line + ranked table a ~1-minute read. diff --git a/hypaware-core/plugins-workspace/codex/skills/hypaware-query/SKILL.md b/hypaware-core/plugins-workspace/codex/skills/hypaware-query/SKILL.md index 748a99ef..7d4abc2e 100644 --- a/hypaware-core/plugins-workspace/codex/skills/hypaware-query/SKILL.md +++ b/hypaware-core/plugins-workspace/codex/skills/hypaware-query/SKILL.md @@ -5,7 +5,7 @@ description: Inspect local HypAware recordings with the hyp query CLI. Use when # HypAware Query -Use `hyp query` to inspect local HypAware recordings. It reads local JSONL recordings and an explicit local query cache; it does not query the central server. +Use `hyp query` to inspect local HypAware recordings. By default it reads local JSONL recordings and an explicit local query cache, not the central server. To run the same query against a remote HypAware host (a fleet server) over its MCP endpoint, add `--remote ` — see [Remote queries](#remote-queries-other-hypaware-hosts). ## Workflow @@ -33,6 +33,33 @@ hyp collect remove These are the only subcommands in the installed CLI (`hyp query`: schema, status, sql, refresh, maintain; `hyp collect`: list, remove). There are no high-level `catalog`/`logs`/`traces`/`metrics` query commands — answer questions with `hyp query sql`, and discover datasets from the `hyp query status` output. +## Remote queries (other HypAware hosts) + +By default `hyp query` is local-only. To run a verb against a remote HypAware host (a fleet server) over its MCP endpoint (`/v1/mcp`), add `--remote `: `hyp` acts as an MCP client, runs the same SQL against the remote `query_sql` tool, and renders the result with the same formatter. Only read-class tools are reachable remotely (`query_sql`, `graph_neighbors`); the credential is **query-scoped** (read/compute only — it cannot author configs or mint tokens), distinct from the server's operator/admin token, which never leaves the server. + +- **Discover configured targets:** `hyp remote list` (`--json` for machine output). Each row shows the target URL and a `token:` status — `env` (a `HYP_REMOTE_TOKEN_` var is set), `stored` (saved by `hyp remote login`), or `missing`. This reflects local config + credentials only; it is **not** a liveness check. The real connectivity/auth test is running a `--remote` query: rows back means reachable + authorized; a 401/timeout tells you which half failed. +- **Set up a target (two steps):** `hyp remote add ` registers the URL — pass the full endpoint including the path (e.g. `https://host:8740/v1/mcp`); it is used verbatim, never auto-suffixed. Then supply the query-scoped token one of two ways: `hyp remote login ` (token via `--token-file ` or piped stdin — never a CLI argument, never an interactive prompt), or a per-target env var `HYP_REMOTE_TOKEN_` (name uppercased, non-alphanumeric runs → `_`; e.g. `prod` → `HYP_REMOTE_TOKEN_PROD`). The env var is checked first and wins. +- **Query it:** `hyp query sql "" --remote --format json`. +- **Truncation is doubled on remote — read both stderr lines.** A server-side data cap (`remote: showing first N rows (server cap …)`) clips before rows leave the server and you **cannot** lift it; the usual local display budget (`notice:` / `--output`) clips again on your side. Never `2>/dev/null` a remote query. +- **`--remote` together with `--refresh` is a hard error** — refresh is a local-cache operation, meaningless against a server that owns its own freshness. +- A remote target may be reachable only over a private network (e.g. a tailnet / `100.x` address); a timeout often means you are off that network, not that the server is down. + +### Two ways a HypAware host's MCP may be attached + +A HypAware host exposes its read-class verbs (`query_sql`, `graph_neighbors`) as an **MCP tool**, and that MCP can be attached by **two independent routes** — be aware of both: + +- **Via `hyp --remote`** — the CLI path above: `hyp` acts as the MCP client (`hyp query sql … --remote `) and renders locally. Discover these by running `hyp remote list`. +- **Via a direct client connection** — the host's `/v1/mcp` endpoint is registered in this client's MCP config (e.g. an `[mcp_servers]` entry in `~/.codex/config.toml`, set up out of band), surfacing the `query_sql` / `graph_neighbors` **tools** directly as hypaware MCP tools already available to you — no `hyp` in the data path. + +The routes are independent, so the **same server may be attached both ways at once** — an attached hypaware MCP tool and a `hyp remote list` target can point at the identical `/v1/mcp` URL. Expect that overlap; don't treat them as two different servers. + +Both routes run the identical `query_sql` operation, so **the data is the same** — but the surfaces are **not byte-identical**: + +- **MCP tool:** returns the **full structured result** (every matching row, as JSON) with **no ~32 KB display budget**; a large result can overflow the client's own output limit and spill to a file. +- **`hyp --remote` CLI:** applies the ~32 KB display budget and prints `notice: showing N of M rows …` on stderr; lift it with `--max-bytes 0` or `--output ` to recover the tool's full set. + +Never read a smaller CLI row count as "fewer rows matched" — it is the display budget, not the result set. + ## SQL dialect notes - `json_extract_scalar()` does not exist. `JSON_EXTRACT` does, but it errors on rows where a JSON-typed column (notably `tool_args`) holds a plain string instead of a JSON object ("first argument must be JSON string or object, got string"). @@ -49,6 +76,8 @@ Key columns: - `tool_name`, `tool_call_id`, `tool_args`, `status` — tool-call/result joins and sparse status such as `finish_reason`. - `attributes` (JSON) — request settings, usage, propagated `dev_run_id`, and gateway diagnostics under `attributes.gateway`. +**Token counts** live under `attributes.usage` on `role='assistant'` rows (NOT in `raw_frame`): `input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_write_tokens`. Codex (`provider='openai'`) omits `cache_write_tokens` and adds `reasoning_tokens` + `total_tokens`. Extract with `CAST(JSON_EXTRACT(attributes,'$.usage.input_tokens') AS BIGINT)`; usage repeats across a message's content parts, so dedup per `message_id` (e.g. `max(...)`) before summing. + Claude transcript enrichment adds `provider_uuid`, `parent_uuid`, `request_id`, `entrypoint`, `client_version`, `user_type`, `permission_mode`, and `hook_event` when the local Claude Code JSONL transcript can be matched. Run `hyp query schema ai_gateway_messages --format markdown` for the authoritative column reference. diff --git a/hypaware-core/plugins-workspace/codex/src/index.js b/hypaware-core/plugins-workspace/codex/src/index.js index 626be84e..6b669c60 100644 --- a/hypaware-core/plugins-workspace/codex/src/index.js +++ b/hypaware-core/plugins-workspace/codex/src/index.js @@ -24,7 +24,8 @@ const CHATGPT_UPSTREAM_NAME = 'chatgpt' * * Resolves the `hypaware.ai-gateway` capability, registers the * OpenAI-compatible upstream preset, wires Codex's config.toml - * attach/detach, and contributes the `hypaware-query` skill for + * attach/detach, and contributes the `hypaware-query` skill plus + * the AI report skills (adoption/improvement/security/spend) for * Codex installs. * * Each attach/detach emits a `client.attach`/`client.detach` span @@ -211,12 +212,20 @@ export async function activate(ctx) { }) const skillsRoot = path.resolve(skillsRootDir(), 'skills') - ctx.skills.register({ - name: 'hypaware-query', - plugin: PLUGIN_NAME, - clients: ['codex'], - sourceDir: path.join(skillsRoot, 'hypaware-query'), - }) + for (const skillName of [ + 'hypaware-query', + 'hypaware-ai-adoption-report', + 'hypaware-ai-improvement-report', + 'hypaware-ai-security-report', + 'hypaware-ai-spend-report', + ]) { + ctx.skills.register({ + name: skillName, + plugin: PLUGIN_NAME, + clients: ['codex'], + sourceDir: path.join(skillsRoot, skillName), + }) + } } /** From 63683d833401adbee8e3e41bfafe718640a5541c Mon Sep 17 00:00:00 2001 From: Brendan McMullen Date: Thu, 25 Jun 2026 16:43:33 -0700 Subject: [PATCH 3/3] don't report inventory and estimate token waste in improvments --- .../hypaware-ai-improvement-report/SKILL.md | 21 ++++++++++++++----- .../hypaware-ai-improvement-report/SKILL.md | 21 ++++++++++++++----- 2 files changed, 32 insertions(+), 10 deletions(-) diff --git a/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-improvement-report/SKILL.md b/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-improvement-report/SKILL.md index 50a17b99..d45a4a4a 100644 --- a/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-improvement-report/SKILL.md +++ b/hypaware-core/plugins-workspace/claude/skills/hypaware-ai-improvement-report/SKILL.md @@ -45,11 +45,20 @@ Focus on these signals: - **Inefficiency** (reuse the spend spine) — model over-spec, context bloat (no `is_compact_summary`), low cache-read (`cache_read/(cache_read+input)`), redo loops. 3. **Collect, dedup, prioritize.** Gather the candidates into a list of possible - improvements; drop anything the inventory already covers. For each, suggest a likely form + improvements; drop anything an existing artifact already covers — a quick scan of the + repo's `.claude/skills/`, subagents, and AGENTS.md/CLAUDE.md (the only repo read — every + other signal is the logs), used to dedup and to mark each survivor **new** vs **edit to + an existing artifact**. For each, suggest a likely form where one obviously fits — a **skill**, a **subagent**, or an **AGENTS.md/CLAUDE.md edit** (propose the concrete line(s) as a diff, not "consider documenting") — but don't force a mapping. Attach evidence (frequency/impact + distinct gateways + token prize) and rank by it. + - **Size the token prize** off the spend spine (deduped `max()` per `message_id`; a plain + `SUM` overcounts ~3×). Give two numbers, kept distinct: **exposure (measured)** — tokens + currently flowing through the issue (sum across a repeated cluster, or tokens in + error/retry/abandoned turns) — and **est. saving (assumption)** only where the + counterfactual is clean (cache-read ratio, model right-size). Both are floors — capture + is partial; never present a saving as if it were measured. ## Output — SAVE A MARKDOWN FILE - **Path:** `hypaware-reports/-improvement-review.md` (create the dir @@ -57,10 +66,12 @@ Focus on these signals: - **Bottom line (1 sentence):** "Create/edit these N things and what it will improve.". - **Improvements (ranked table):** one row per improvement — **what** · **suggested form** (skill / subagent / AGENTS.md edit, where one fits) · **evidence** (recurs N× across G - gateways, sticking-point impact, or token prize) · **where it lands**. Highest-leverage first. -- **Then the detail**, in these sections: **Inventory (what exists) · Basis · Skill - candidates · Subagent candidates (fan-out + roles) · AGENTS.md/CLAUDE.md edits · - Efficiency → setup changes · Caveats** — plus the proposed AGENTS.md lines verbatim. + gateways, sticking-point impact) · **token prize** (exposure measured / est. saving, + labeled) · **where it lands**. Highest-leverage first. +- **Then the detail**, in these sections: **Basis · Skill candidates · Subagent candidates + (fan-out + roles) · AGENTS.md/CLAUDE.md edits · Efficiency → setup changes · Caveats** — + plus the proposed AGENTS.md lines verbatim. Each candidate is marked **new** or **edit to + **; don't list artifacts that aren't being changed. - **Formatting (human-readable):** every candidate states its evidence inline; show AGENTS.md edits as real diffs/code blocks, not prose; **bold** the artifact type; keep it a ~1-minute read. diff --git a/hypaware-core/plugins-workspace/codex/skills/hypaware-ai-improvement-report/SKILL.md b/hypaware-core/plugins-workspace/codex/skills/hypaware-ai-improvement-report/SKILL.md index 50a17b99..d45a4a4a 100644 --- a/hypaware-core/plugins-workspace/codex/skills/hypaware-ai-improvement-report/SKILL.md +++ b/hypaware-core/plugins-workspace/codex/skills/hypaware-ai-improvement-report/SKILL.md @@ -45,11 +45,20 @@ Focus on these signals: - **Inefficiency** (reuse the spend spine) — model over-spec, context bloat (no `is_compact_summary`), low cache-read (`cache_read/(cache_read+input)`), redo loops. 3. **Collect, dedup, prioritize.** Gather the candidates into a list of possible - improvements; drop anything the inventory already covers. For each, suggest a likely form + improvements; drop anything an existing artifact already covers — a quick scan of the + repo's `.claude/skills/`, subagents, and AGENTS.md/CLAUDE.md (the only repo read — every + other signal is the logs), used to dedup and to mark each survivor **new** vs **edit to + an existing artifact**. For each, suggest a likely form where one obviously fits — a **skill**, a **subagent**, or an **AGENTS.md/CLAUDE.md edit** (propose the concrete line(s) as a diff, not "consider documenting") — but don't force a mapping. Attach evidence (frequency/impact + distinct gateways + token prize) and rank by it. + - **Size the token prize** off the spend spine (deduped `max()` per `message_id`; a plain + `SUM` overcounts ~3×). Give two numbers, kept distinct: **exposure (measured)** — tokens + currently flowing through the issue (sum across a repeated cluster, or tokens in + error/retry/abandoned turns) — and **est. saving (assumption)** only where the + counterfactual is clean (cache-read ratio, model right-size). Both are floors — capture + is partial; never present a saving as if it were measured. ## Output — SAVE A MARKDOWN FILE - **Path:** `hypaware-reports/-improvement-review.md` (create the dir @@ -57,10 +66,12 @@ Focus on these signals: - **Bottom line (1 sentence):** "Create/edit these N things and what it will improve.". - **Improvements (ranked table):** one row per improvement — **what** · **suggested form** (skill / subagent / AGENTS.md edit, where one fits) · **evidence** (recurs N× across G - gateways, sticking-point impact, or token prize) · **where it lands**. Highest-leverage first. -- **Then the detail**, in these sections: **Inventory (what exists) · Basis · Skill - candidates · Subagent candidates (fan-out + roles) · AGENTS.md/CLAUDE.md edits · - Efficiency → setup changes · Caveats** — plus the proposed AGENTS.md lines verbatim. + gateways, sticking-point impact) · **token prize** (exposure measured / est. saving, + labeled) · **where it lands**. Highest-leverage first. +- **Then the detail**, in these sections: **Basis · Skill candidates · Subagent candidates + (fan-out + roles) · AGENTS.md/CLAUDE.md edits · Efficiency → setup changes · Caveats** — + plus the proposed AGENTS.md lines verbatim. Each candidate is marked **new** or **edit to + **; don't list artifacts that aren't being changed. - **Formatting (human-readable):** every candidate states its evidence inline; show AGENTS.md edits as real diffs/code blocks, not prose; **bold** the artifact type; keep it a ~1-minute read.