|
| 1 | +# Compliance Audit Copilot |
| 2 | + |
| 3 | +A four-stage, **token-budgeted** audit workflow: upload policy documents, |
| 4 | +get an AI-generated checklist of audit questions, write raw observations |
| 5 | +and get them rewritten into professional phrasing, get an AI severity |
| 6 | +score per finding, and generate an executive summary across the whole |
| 7 | +audit. Every LLM call — in every feature — is routed through a real, |
| 8 | +tested 8,000-token budget with automatic map-reduce fallback for oversized |
| 9 | +input, not a "keep it short" comment. |
| 10 | + |
| 11 | + |
| 12 | + |
| 13 | + |
| 14 | + |
| 15 | +## The four features |
| 16 | + |
| 17 | +1. **Checklist generation** — upload a PDF (extracted via `pymupdf4llm`), |
| 18 | + get back a structured list of specific, answerable audit questions. |
| 19 | +2. **Observation optimizer** — write a raw, informal observation; get three |
| 20 | + professionally-phrased alternatives to choose from. |
| 21 | +3. **AI severity scoring** — a continuous 0.0–1.0 severity rating per |
| 22 | + observation, with a short rationale, relative to its question. |
| 23 | +4. **Executive summary** — synthesizes every question, observation, and |
| 24 | + severity score in the audit into one coherent summary for leadership. |
| 25 | + |
| 26 | +## Token budgeting — the actual point of this project |
| 27 | + |
| 28 | +`token_budget.py` enforces a hard 8,000-token context budget |
| 29 | +(`MAX_CONTEXT_TOKENS`), with ~7,000 tokens reserved for input after |
| 30 | +accounting for the expected completion. Every feature routes its prompt |
| 31 | +construction through it: |
| 32 | + |
| 33 | +- **Observation optimizer & severity scoring** operate on one question + |
| 34 | + one observation — comfortably within budget by construction, but still |
| 35 | + routed through the same `fit_to_budget` guard defensively (a |
| 36 | + pathologically long raw observation shouldn't be able to blow the budget). |
| 37 | +- **Checklist generation & executive summary** are the two places a real |
| 38 | + audit can genuinely exceed the budget (a long policy document; a large |
| 39 | + number of findings). Both fall back to **map-reduce**: split into |
| 40 | + token-sized chunks, process each chunk independently, then merge/combine |
| 41 | + the partial results into one final output — all individually |
| 42 | + budget-checked, including the merge/combine step itself. |
| 43 | + |
| 44 | +This isn't asserted, it's tested: `tests/test_checklist.py` and |
| 45 | +`tests/test_summary.py` each feed in a deliberately oversized input and |
| 46 | +assert that **every single prompt actually sent to the LLM** — across every |
| 47 | +map call and the final reduce call — measured with `tiktoken`, fits the |
| 48 | +budget. That's the proof the constraint is real, not just a docstring. |
| 49 | + |
| 50 | +### A real off-by-one bug this caught |
| 51 | + |
| 52 | +Early on, a test failed with the prompt at *6,977 tokens against a 6,976 |
| 53 | +budget* — off by exactly one token. The cause: BPE tokens can merge |
| 54 | +differently at the boundary where a prompt template and inserted content |
| 55 | +join, so `count(template) + count(content)` isn't always exactly |
| 56 | +`count(template.format(content))`. The fix wasn't tighter arithmetic, it |
| 57 | +was accepting that boundary effects exist: `content_budget()` reserves a |
| 58 | +small safety margin (`BOUNDARY_SAFETY_MARGIN`) specifically for this, |
| 59 | +found by a test catching a real violation, not by inspection. |
| 60 | + |
| 61 | +## Architecture |
| 62 | + |
| 63 | +- **Direct LLM calls, not an agent framework.** `llm.py` / `llm_json.py` |
| 64 | + wrap a plain OpenAI-compatible client against the NVIDIA NIM endpoint. |
| 65 | + This system is four independent, linear, structured-output calls behind |
| 66 | + a CRUD workflow — not a multi-step agentic loop — so pulling in |
| 67 | + LangGraph/LangChain (used in the other two portfolio projects) would be |
| 68 | + unjustified complexity for this task shape. Right tool for the job, not |
| 69 | + a hammer looking for a nail. |
| 70 | +- **Structured output, verified not trusted.** `llm_json.py` prompts for |
| 71 | + strict JSON, extracts it with a balanced-brace parser that tolerates |
| 72 | + markdown fences and surrounding prose, validates against a Pydantic |
| 73 | + schema, and retries once with corrective feedback on failure — the same |
| 74 | + "don't just trust the model's JSON" lesson learned (and re-learned) |
| 75 | + across all three portfolio projects. |
| 76 | +- **Real persistence.** SQLAlchemy + SQLite — audits, documents, checklist |
| 77 | + items, and observations survive a restart; this is a system, not a demo |
| 78 | + script that forgets everything on refresh. |
| 79 | + |
| 80 | +```mermaid |
| 81 | +flowchart LR |
| 82 | + UI[React UI] -->|REST| API[FastAPI] |
| 83 | + API --> DB[(SQLite)] |
| 84 | + API --> PDF[pymupdf4llm] |
| 85 | + API --> Budget[token_budget.py] |
| 86 | + Budget --> LLM[NVIDIA Nemotron] |
| 87 | + subgraph Features |
| 88 | + C[checklist.py] |
| 89 | + O[observation.py] |
| 90 | + S[severity.py] |
| 91 | + Sum[summary.py] |
| 92 | + end |
| 93 | + API --> Features |
| 94 | + Features --> Budget |
| 95 | +``` |
| 96 | + |
| 97 | +## A note on originality |
| 98 | + |
| 99 | +This project deliberately mirrors a pattern (checklist → observation → |
| 100 | +severity → summary) similar to production audit-automation work. To avoid |
| 101 | +any overlap with employer IP, this is an independent implementation built |
| 102 | +from scratch, applied to a different domain (generic IT security / |
| 103 | +operational compliance, not banking/regulatory), using entirely synthetic |
| 104 | +sample documents — no real organization's data or proprietary logic. |
| 105 | + |
| 106 | +## Getting started |
| 107 | + |
| 108 | +```bash |
| 109 | +git clone <your-fork-url> |
| 110 | +cd compliance-audit-copilot |
| 111 | +cp .env.example .env |
| 112 | +# edit .env and add your NVIDIA_API_KEY (free key: https://build.nvidia.com) |
| 113 | + |
| 114 | +python3.12 -m venv .venv |
| 115 | +source .venv/bin/activate |
| 116 | +pip install -r requirements.txt |
| 117 | + |
| 118 | +# terminal 1 — backend |
| 119 | +uvicorn backend.main:app --host 127.0.0.1 --port 8001 |
| 120 | + |
| 121 | +# terminal 2 — frontend |
| 122 | +cd frontend |
| 123 | +npm install |
| 124 | +npm run dev |
| 125 | +``` |
| 126 | + |
| 127 | +Open http://localhost:5173 (proxies `/api` to the backend on `:8001`). |
| 128 | + |
| 129 | +## Testing |
| 130 | + |
| 131 | +```bash |
| 132 | +pytest -q # 38 tests |
| 133 | +ruff check . |
| 134 | + |
| 135 | +cd frontend && npm run build && npm run lint |
| 136 | +``` |
| 137 | + |
| 138 | +- `tests/test_token_budget.py` — counting, truncation, chunking, and the |
| 139 | + boundary safety margin. |
| 140 | +- `tests/test_llm_json.py` — JSON extraction (fences, embedded prose) and |
| 141 | + the validate-then-retry-once structured output flow. |
| 142 | +- `tests/test_pdf_extraction.py` — real extraction from a generated PDF. |
| 143 | +- One test file per feature, each with an oversized-input case proving the |
| 144 | + map-reduce path actually engages and stays under budget. |
| 145 | +- `tests/test_api.py` — the full workflow (create → upload → generate |
| 146 | + checklist → observe → optimize → select → score → summarize) through the |
| 147 | + real FastAPI app and a real (temporary) SQLite database, LLM mocked. |
| 148 | + |
| 149 | +## Project structure |
| 150 | + |
| 151 | +``` |
| 152 | +compliance-audit-copilot/ |
| 153 | +├── config.py # settings, including token budget constants |
| 154 | +├── token_budget.py # the core differentiator — see above |
| 155 | +├── llm.py, llm_json.py # NVIDIA client + verified structured output |
| 156 | +├── pdf_extraction.py # pymupdf4llm wrapper |
| 157 | +├── models.py, db.py # SQLAlchemy models + session management |
| 158 | +├── features/ |
| 159 | +│ ├── checklist.py # + map-reduce for oversized documents |
| 160 | +│ ├── observation.py |
| 161 | +│ ├── severity.py |
| 162 | +│ └── summary.py # + map-reduce for oversized findings |
| 163 | +├── backend/main.py # FastAPI REST endpoints |
| 164 | +├── frontend/src/ |
| 165 | +│ ├── pages/{AuditList,AuditDetail}.tsx |
| 166 | +│ └── components/{ChecklistItemCard,DocumentUpload,ExecutiveSummary,...}.tsx |
| 167 | +├── tests/ |
| 168 | +└── .github/workflows/ci.yml |
| 169 | +``` |
| 170 | + |
| 171 | +## Known limitations |
| 172 | + |
| 173 | +- Token counting uses `tiktoken`'s `cl100k_base` encoding as a standard |
| 174 | + proxy — Nemotron doesn't publish a public tiktoken-compatible encoding, |
| 175 | + but this is close enough in practice for budgeting purposes. |
| 176 | +- SQLite is fine for a portfolio demo; a real multi-user deployment would |
| 177 | + need a proper server-based database and auth. |
| 178 | +- Regenerating a checklist replaces the previous one (and, via cascade, its |
| 179 | + observations) — simplest consistent behavior for a demo, not full |
| 180 | + version history. |
| 181 | + |
| 182 | +## License |
| 183 | + |
| 184 | +MIT — see [LICENSE](LICENSE). |
0 commit comments