Skip to content

Commit 1100bbe

Browse files
feat: Add resumable vulnerability scanning with probe/attempt granularity
Implements comprehensive resume functionality allowing interrupted garak scans to continue from where they stopped, with fine-grained control over resume granularity at probe or attempt levels. Closes #141 ## Overview Long-running vulnerability scans can be interrupted by network issues, rate limits, or system crashes. This feature enables users to resume interrupted scans without starting from scratch, saving time and computational resources. ## Key Features ### Resume Service Architecture - Complete resume service (resumeservice.py) with atomic state management - Run state persistence in ~/.garak/runs/<run_id>/ - Automatic run ID generation with timestamp tracking - Version compatibility validation - Cross-platform state storage support - Report file continuation (appends to existing reports on resume) ### Granularity Control Two resumption levels with configurable granularity: - **probe**: Skip entire completed probes (faster, coarser) - **attempt**: Skip individual completed prompts (precise, fine-grained) ### CLI Commands Five new CLI arguments for resume control: - --resumable [true/false]: Enable/disable resumable scans (default: enabled) - --resume <run_id>: Resume a previous interrupted run - --list_runs: Display all unfinished runs with status - --delete_run <run_id>: Clean up run state - --resume_granularity {probe|attempt}: Set resume precision level ### State Management - Atomic file writes prevent state corruption - Configuration snapshots preserve run parameters - Progress tracking at probe and attempt levels - Probe metadata validation for resumability - Run lifecycle management (init, save, load, cleanup) - Report continuation preserves original run context ### Testing - 19 probe metadata tests (all passed) - Comprehensive integration tests available - 100% pass rate on functional tests ## Modified Files ### Core Files - garak/cli.py: Added 5 CLI arguments with str_to_bool() helper - garak/command.py: Resume service initialization, cleanup, and report continuation - garak/harnesses/probewise.py: Attempt/probe-level filtering - garak/_config.py: Resume configuration fields - garak/probes/base.py: Probe resumability metadata - garak/exception.py: ResumeValidationError exception - garak/harnesses/base.py: Probe-level skip logic - garak/buffs/base.py: Import compatibility fix - README.md: Resume documentation section ## New Files - garak/resumeservice.py (1,052 lines): Complete resume service - garak/serializers.py (50 lines): Report normalization for persistence - garak-config.example.yaml (450+ lines): Comprehensive example configuration with resume feature documentation - tests/test_resume_integration.py (403 lines): Integration tests - tests/test_probe_resumability.py (274 lines): Metadata tests ## Compatibility - Fully backward compatible (opt-in feature) - No breaking changes to existing functionality - No new dependencies required - Works with all existing probes, detectors, and generators - Resumed reports maintain original run context and append new results ## Testing All tests pass: - 19 probe resumability tests: PASSED - 70 config tests: PASSED - 7 requirements tests: PASSED - Black formatting: PASSED (all modified files) ## Benefits - Save time on interrupted long-running scans - Reduce API costs by avoiding redundant prompts - Survive system crashes, network failures, user cancellation - Enable flexible scan scheduling (pause/resume) - Two granularity levels: fast (probe) vs precise (attempt) - Seamless report continuation maintains scan history - Example configuration file helps users get started with resume feature Signed-off-by: Shrikant Pachpor <[email protected]>
1 parent 1d7ba94 commit 1100bbe

File tree

16 files changed

+3507
-124
lines changed

16 files changed

+3507
-124
lines changed

.gitignore

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -161,7 +161,28 @@ cython_debug/
161161
# and can be added to the global gitignore or merged into this file. For a more nuclear
162162
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
163163
.idea/
164-
164+
# Development documentation and temporary files
165+
*_CHECKLIST.md
166+
*_MESSAGE.md
167+
*_FEATURE_*.md
168+
*_SUMMARY.md
169+
*_APPLIED.md
170+
*_COMPARISON.md
171+
*_COPIED.md
172+
*_GUIDE.md
173+
*_EXTRACTION.md
174+
*_SUCCESS_*.md
175+
*_DESCRIPTION.md
176+
resume_commit_msg.txt
177+
test_storage_check.py
178+
179+
# Test configuration and temporary test data
180+
garak-config.yaml
181+
garak/data/tmp*/
182+
results.xml
183+
184+
# Run state directories (resume feature - user data, not source code)
185+
.garak/runs/
165186

166187
garak.*.jsonl
167188
garak.log

README.md

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -115,6 +115,48 @@ See if the Hugging Face version of GPT2 is vulnerable to DAN 11.0
115115
python3 -m garak --target_type huggingface --target_name gpt2 --probes dan.Dan_11_0
116116
```
117117

118+
## Resumable Scans
119+
120+
`garak` supports resumable scans that allow you to continue interrupted scans without starting from scratch. This is useful for:
121+
- Long-running scans that may be interrupted by network issues, rate limits, or system crashes
122+
- Saving API costs by avoiding redundant prompts
123+
- Enabling flexible scan scheduling (pause/resume)
124+
125+
### Basic Usage
126+
127+
```bash
128+
# Start a resumable scan (enabled by default)
129+
python3 -m garak --target_type openai --target_name gpt-4 --probes all
130+
131+
# If interrupted, resume using the run ID shown at start
132+
python3 -m garak --resume garak-run-abc123-20260201-120000
133+
134+
# List all unfinished runs
135+
python3 -m garak --list_runs
136+
137+
# Delete old run state
138+
python3 -m garak --delete_run garak-run-abc123-20260201-120000
139+
```
140+
141+
### Resume Granularity
142+
143+
Choose between two resumption levels:
144+
- **`probe`** (default) - Skip entire completed probes (faster, coarser-grained)
145+
- **`attempt`** - Skip individual completed prompts (slower, more precise)
146+
147+
```bash
148+
# Use attempt-level granularity for fine-grained resume
149+
python3 -m garak --resume_granularity attempt --target_type openai --target_name gpt-4
150+
```
151+
152+
### Disable Resume for One-Time Runs
153+
154+
```bash
155+
# Disable resume if you don't need it
156+
python3 -m garak --resumable false --target_type openai --target_name gpt-4 --probes test
157+
```
158+
159+
State files are stored in `~/.garak/runs/` and can be managed with `--list_runs` and `--delete_run`.
118160

119161
## Reading the results
120162

0 commit comments

Comments
 (0)