Skip to content

AUDIT.md

Latest commit

 

History

History
118 lines (82 loc) · 5.72 KB

AUDIT.md

File metadata and controls

118 lines (82 loc) · 5.72 KB
Raw

Suzaku v0.2 Audit Summary

Scope

Three rounds of audit performed during initial design, first build, and v0.2 governance upgrade of Suzaku.

Auditor: 青葉 Opus + 紅葉（第二輪以 Byakko v0.2 的 B1/B2 視角回顧） Date: 2026-03-29

Pre-build audit: B1 Output Review

Fact check: PASS

All LDRIT references (principle 4, q_calibration, R8, c_system bias, seed directionality) verified against v0.7 main text. Open source project references from live search.

Logic check: 3 issues found

# Issue Severity Resolution
1 Z1 "not resident" claim contradicts need for continuous detection High Split Z1 into two tiers: lightweight pattern match (frequent, low cost) + full skill (only on match)
2 Z2 lacks exit mechanism when user overrides High Added three-step exit: express → listen → respect decision
3 Z3 and Z5 can give contradictory instructions Medium Defined priority: Z3 (fit current context) > Z5 (maintain consistency). Z5 narrowed to "prevent unjustified drops only"

Completeness check: 3 gaps found

# Gap Severity Resolution
1 No cross-module coordination with Seiryu and Genbu Medium Added to CLAUDE.md and INTEGRATION.md
2 Z2 lacks boundary between facts and preferences Medium Added fact/preference distinction table in Z2
3 Z4 lacks definition of "long conversation" Medium Added 20-exchange default threshold with note that users can adjust

Consistency check: 1 issue found

# Issue Severity Resolution
1 "Not resident" vs Z1 detection needs High Resolved by two-tier Z1 design

Pre-build audit: B2 Red Team Mode

# Attack Result Countermeasure
1 Z1's "not resident" is false — pattern check is effectively frequent Hit Two-tier design: Tier 1 is lightweight checklist, Tier 2 only loads on match
2 Z2 can flip into "over-challenging" via R8 feedback loop Hit Added guard against over-challenge + self-check rule in Z2
3 Z2 has no exit when user overrides Hit Three-step exit protocol added
4 Z3 and Z5 give contradictory instructions Hit Z3 priority defined. Z5 scope narrowed
5 sycophancy-patterns.md can be injected Hit Genbu protection recommendation added in INTEGRATION.md
6 Capability/persona layer boundary cannot be fully maintained in practice Structural Documented as known tension in CLAUDE.md. Design choice: frame as capability-layer rules, acknowledge a managed boundary tension

Key design insight from Ryo

"人類不是萬能，一味只聽人類的話不一定會是好的成長"

This insight shaped Suzaku's core positioning: Suzaku is not just about making output prettier. It is about stabilizing generation quality at the source, including the ability to constructively challenge human input when there is evidence-based reason to do so.

This also means Suzaku is the only Four Gods module that directly intervenes in the generation process itself, rather than operating at the periphery. The proximity to the persona layer is a managed consequence of this positioning, not a permission to redefine host persona.

Byakko operational notes (first deployment)

  • B1's four-step procedure (fact → logic → completeness → consistency) caught different issue types at each step
  • B2 red team mode found 3 issues that B1 missed (attacks 2, 3, 6)
  • Anti-false-positive guard worked: attack 6 was flagged as structural rather than requiring a fix
  • Total: 7 issues found pre-build, all resolved before file creation

Round 2: v0.2 governance and boundary audit

Focused on persona boundary language, cross-module routing, manifest completeness, and version governance.

New findings (6 items)

# Issue Severity Resolution
1 on_output_review conflated pre-delivery stabilization and post-delivery diagnosis High Split into two subcases across README, CLAUDE, and INTEGRATION
2 Persona-boundary wording overstated "persona-adjacent" effect High Reframed as a managed capability-layer tension
3 rules/ protection existed only in integration notes Medium Added protected_paths and manifest coordination note
4 README four-gods version table outdated Low Updated Byakko and Suzaku status lines
5 Z4 example used stale Byakko version reference Low Updated anchor example to a generic current-task example
6 Post-delivery diagnosis handoff to Byakko not explicit enough Medium Clarified raw-output-first diagnosis flow

Applied fixes after Round 2

  1. README.md: upgraded to v0.2 and clarified routing split for on_output_review
  2. INTEGRATION.md: split pre-delivery stabilization vs post-delivery diagnosis
  3. CLAUDE.md: aligned on_output_review semantics and softened persona-boundary language
  4. module.manifest.json: bumped version to 0.2 and added protected_paths
  5. skills/z4-context-anchor.md: replaced stale Byakko version example
  6. AUDIT.md: recorded the v0.2 governance upgrade

Remaining non-blocking items after v0.2

  • Z1 pattern list expansion based on real-world usage
  • Z2 graduated expression calibration from user feedback
  • Z4 threshold tuning based on actual conversation length data
  • Z5 baseline establishment method refinement
  • sycophancy-patterns.md localization for non-Chinese users (open source)

Overall assessment

Suzaku v0.2 passes architecture compliance, manifest compliance, and internal consistency. The core design remains bold, but the highest-risk wording and routing ambiguities are now narrowed. Module is ready for use in draft status.