Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions BUGS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@

These are bugs (or missing features) I've observed while working with `multi checks`.

- [ ] Remove the `Claude -p` executor.

- [ ] No use of Cersei workflows to chain multiple prompts together.

- [ ] No GitHub Action available.

- [ ] Logs no longer report the id of the check that failed (or the number of attempted retries)

- [ ] Assemble_instructions is hard-coded: src/checks/executor/mod.rs:98 (definition), called from src/checks/executor/cersei.rs:110
Expand All @@ -18,10 +18,6 @@ These are bugs (or missing features) I've observed while working with `multi che

- [ ] No limit on max turns.

- [ ] No support for Fireworks AI.

- [ ] No GitHub Action available.

- [ ] No loading of skills.

- [ ] No loading of RULES.md files from the .claude directory.
Expand All @@ -43,6 +39,10 @@ guaranteeing the terminal is restored on the way out.

## Fixes

- [x] No support for Fireworks AI.

- [x] Remove the `Claude -p` executor.

- [x] No trace capture. We need a way to record all session traces so that we can analyze why they failed.

- [x] Output is now hanging. I suspect this is recent (within the last few commits) and it started
Expand Down
14 changes: 7 additions & 7 deletions CHECKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,19 @@

# Requirement Provider-Agnostic Execution

The agent executor is swappable behind a trait so execution can run over different engines (the in-process `cersei-agent` executor, the legacy `claude -p` fallback, or a test fake) without rewriting the execution phase. This extensibility is the whole point of the configuration/executor seam.
The agent executor is defined behind a trait so execution can run over the in-process `cersei-agent` executor or a test fake without rewriting the execution phase. This extensibility is the whole point of the configuration/executor seam.

## Check Boxed Executor Trait

Inspect the Rust sources for the `multi check` feature. Confirm that agent execution is defined behind a trait (for example `CheckExecutor`) that is consumed as a boxed or `Arc`-wrapped, dynamically-dispatched trait object — for example a type alias of the form `Box<dyn CheckExecutor + Send + Sync>`, mirroring the existing `BoxedIngress`, `BoxedMonitor`, and `BoxedPlatform` aliases. The check passes only if the execution phase depends on this trait rather than naming a concrete executor type. Report a failure if the execution path references a concrete executor struct (such as the cersei or claude executor) directly instead of the trait object.
Inspect the Rust sources for the `multi check` feature. Confirm that agent execution is defined behind a trait (for example `CheckExecutor`) that is consumed as a boxed or `Arc`-wrapped, dynamically-dispatched trait object — for example a type alias of the form `Box<dyn CheckExecutor + Send + Sync>`, mirroring the existing `BoxedIngress`, `BoxedMonitor`, and `BoxedPlatform` aliases. The check passes only if the execution phase depends on this trait rather than naming a concrete executor type. Report a failure if the execution path references a concrete executor struct (such as the cersei executor) directly instead of the trait object.

# Requirement In-Process Agent Execution

The default executor runs each check's agent **inside the CLI process** via the `cersei-agent` library — there is no longer any requirement to shell out to the `claude` CLI for a check to run. A `claude -p` executor is retained only as an optional, explicitly-selected migration fallback.
The executor runs each check's agent **inside the CLI process** via the `cersei-agent` library — there is no requirement to shell out to any external CLI for a check to run.

## Check Default Executor Runs In-Process

Inspect the executor implementations and how the default executor is selected from configuration. Confirm that the default execution engine builds and runs a `cersei_agent::Agent` in-process (calling its `run`/`run_stream` method) rather than spawning the `claude` CLI. Confirm that any use of `std::process::Command` / `tokio::process` to launch `claude -p` lives only in the non-default fallback executor, gated behind an explicit `executor` selection. The check fails if running a check with the default configuration requires spawning an external `claude` process.
Inspect the executor implementation and how it is constructed from configuration. Confirm that the execution engine builds and runs a `cersei_agent::Agent` in-process (calling its `run`/`run_stream` method) rather than spawning an external process. The check fails if running a check requires spawning an external `claude` process.

# Requirement Trustworthy In-Process Reporting

Expand Down Expand Up @@ -66,7 +66,7 @@ Inspect the execution phase. Confirm that checks are dispatched concurrently —

# Requirement Concurrency Defaults To The Host's Core Count

The concurrency cap is user-configurable (a `--concurrency` flag, layered the same way as `--provider`/`--model`/`--effort`/`--executor`), but absent any override its default must equal the number of CPU cores available on the machine running `multi check` — not a hardcoded constant. A fixed default either strands cores on big machines or overcommits small ones.
The concurrency cap is user-configurable (a `--concurrency` flag, layered the same way as `--provider`/`--model`/`--effort`), but absent any override its default must equal the number of CPU cores available on the machine running `multi check` — not a hardcoded constant. A fixed default either strands cores on big machines or overcommits small ones.

## Check Default Concurrency Equals Available Parallelism

Expand All @@ -90,11 +90,11 @@ Confirm that the command exits with status code 0 when every requirement is sati

# Requirement Layered Configuration

The model, provider, effort, and executor are resolved from three sources with standard CLI precedence — flag, then environment, then config file (flag wins) — while credentials are read only from each provider's native environment variable, never from the config file.
The model, provider, and effort are resolved from three sources with standard CLI precedence — flag, then environment, then config file (flag wins) — while credentials are read only from each provider's native environment variable, never from the config file.

## Check Precedence And Validation

Inspect the configuration phase of `multi check`. Confirm that provider/model/effort/executor are merged from a config file, `MULTI_`-prefixed environment variables, and CLI flags, with flags overriding environment overriding file. Confirm the selected model is validated against a hardcoded allowlist of known IDs for the provider (an unknown ID is a clear error). The check fails if any of these values cannot be set from configuration, or if the merge precedence is not flag > env > file.
Inspect the configuration phase of `multi check`. Confirm that provider/model/effort are merged from a config file, `MULTI_`-prefixed environment variables, and CLI flags, with flags overriding environment overriding file. Confirm the selected model is validated against a hardcoded allowlist of known IDs for the provider (an unknown ID is a clear error). The check fails if any of these values cannot be set from configuration, or if the merge precedence is not flag > env > file.

## Check Credentials Are Environment-Only

Expand Down
20 changes: 10 additions & 10 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

25 changes: 9 additions & 16 deletions guides/checks.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,9 @@ requirement is satisfied only if **all** of its checks pass (logical AND).
## ✅ Prerequisites

- [ ] An API key for your chosen provider in the environment (e.g.
`ANTHROPIC_API_KEY`) — see [Configuration](#️-configuration). The default
`ANTHROPIC_API_KEY`) — see [Configuration](#️-configuration). The
in-process executor talks to the provider directly; **no `claude` CLI is
required**. (The optional `claude -p` fallback — `--executor claude` — does
need the [`claude`](https://docs.claude.com/en/docs/claude-code) CLI on your
`PATH`.)
required**.
- [ ] **macOS** — the MVP sandboxes each check with an APFS copy-on-write clone.
Other operating systems are not yet supported.

Expand Down Expand Up @@ -159,15 +157,14 @@ Glob, plus the judge tool) — a verification agent observes, it does not mutate

## ⚙️ Configuration

The default **provider**, **model**, **effort**, **executor**, and
**concurrency** are resolved from three sources, in order of precedence
(highest wins):
The default **provider**, **model**, **effort**, and **concurrency** are
resolved from three sources, in order of precedence (highest wins):

1. **Flags** — `--provider`, `--model`, `--effort`, `--executor`,
`--concurrency` on `multi check`.
1. **Flags** — `--provider`, `--model`, `--effort`, `--concurrency` on
`multi check`.
2. **Environment** — `MULTI_`-prefixed vars mapped into the `checks` namespace,
e.g. `MULTI_CHECKS_MODEL`, `MULTI_CHECKS_PROVIDER`, `MULTI_CHECKS_EFFORT`,
`MULTI_CHECKS_EXECUTOR`, `MULTI_CHECKS_CONCURRENCY`.
`MULTI_CHECKS_CONCURRENCY`.
3. **Config file** — the `[checks]` table of `MultiTool.toml` (or `.json` /
`.jsonc`), discovered up the directory tree like any MultiTool manifest.

Expand All @@ -176,7 +173,6 @@ The default **provider**, **model**, **effort**, **executor**, and
provider = "anthropic" # anthropic | openai | gemini
model = "claude-sonnet-4-6" # must be a known model ID for the provider
effort = "low" # low | medium | high → thinking-token budget
executor = "cersei" # cersei (in-process, default) | claude (fallback)
concurrency = 8 # checks run at once; must be > 0 (default: CPU core count)

# optional, non-secret base-URL overrides per provider
Expand All @@ -192,11 +188,8 @@ enable extended thinking (4096- and 8192-token budgets respectively), while
`low` — the default — keeps thinking off for speed and cost, running the agent
deterministically instead.

The **`executor`** selects the execution engine. The default `cersei` runs each
check as an in-process agent (native multi-provider model swapping, no external
CLI). `claude` is the legacy `claude -p` shell-out fallback, kept selectable for
migration while the in-process path is validated; it requires the `claude` CLI on
your `PATH` and will be removed once cersei is proven out.
Each check runs as an in-process agent (native multi-provider model swapping,
no external CLI or subprocess).

The **`concurrency`** flag caps how many checks run at once; it must be a
positive integer (`0` is rejected with a clear error). Its default matches the
Expand Down
Loading