Commit 648826a
fix(callgraph,dsl): Add thread-safety, improve logging, and fix progress messages (#451)
* fix(dsl): Add early filtering for empty container rule directories
Prevent system hang when loading container rules from directories without
@dockerfile_rule or @compose_rule decorators. Exit early with clear error
message instead of attempting Python execution.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
* fix(dsl,callgraph): Add early filtering for rules and progress logging
1. Add early filtering for code analysis rules in LoadRules()
- Skip executing Python files without @rule decorator
- Prevents hanging when normal app.py files are used as rules
- Add hasCodeAnalysisRuleDecorators() to check for @rule or codepathfinder imports
2. Add progress logging to BuildCallGraph()
- Add progress messages for 4 passes through all Python files
- Extracting return types, variable assignments, class attributes, call sites
- Prevents appearance of hanging with large repos (20k+ files)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
* feat(dsl): Add verbose logging for loaded rules with ID and pathname
- Add Logger interface to dsl package to avoid import cycle with output package
- Update LoadRules() and LoadContainerRules() to accept logger parameter
- Log each loaded rule with ID and file path in debug mode
- Log container rules (Dockerfile and docker-compose) separately
- Update all callers in cmd/scan.go and cmd/ci.go to pass logger
- Update all tests to pass nil logger
Fixes: Shows loaded rules when --debug flag is used
Example output:
[00:00.030] Loaded docker-compose rule: COMPOSE-SEC-008 from rules/docker-compose/dangerous_capabilities.py
[00:00.362] Loaded code analysis rule: CUSTOM-001 from /tmp/test-rules/test_eval.py
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
* fix(callgraph): Make taint summary progress message consistent
- Remove 'Pass 5:' prefix from taint summary generation message
- Align with other progress messages style (present continuous without pass numbers)
Before: 'Pass 5: Generating taint summaries...'
After: 'Generating taint summaries...'
Matches the style of:
- 'Extracting return types from X modules (parallel)...'
- 'Extracting variable assignments (parallel)...'
- 'Extracting class attributes (parallel)...'
- 'Resolving call sites (parallel)...'
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
* fix(callgraph): Add thread-safety to TypeInferenceEngine for parallel processing
Fixes race condition: 'fatal error: concurrent map read and map write' during
variable assignment extraction phase.
Root cause: Multiple goroutines accessing TypeInferenceEngine.Scopes and
TypeInferenceEngine.ReturnTypes maps concurrently without synchronization.
Changes:
- Add sync.RWMutex fields (scopeMutex, typeMutex) to TypeInferenceEngine struct
- Protect GetScope() and AddScope() with scopeMutex
- Add GetReturnType() method with typeMutex protection
- Protect AddReturnTypesToEngine() with typeMutex
- Update all direct map accesses to use thread-safe methods:
- chaining.go: 6 occurrences
- attribute.go: 1 occurrence
Thread-safety pattern:
- Use RLock/RUnlock for read operations (GetScope, GetReturnType)
- Use Lock/Unlock for write operations (AddScope, AddReturnTypesToEngine)
This allows parallel extraction passes to safely share the TypeInferenceEngine.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
* fix(dsl): Show loaded rules list with --verbose flag instead of --debug
Changes:
- Update Logger interface to include IsVerbose() and Statistic() methods
- Change rule loading logging from IsDebug() + Debug() to IsVerbose() + Statistic()
- This makes rule list visible with --verbose flag (more user-friendly)
- Remove duplicate logging in LoadContainerRules (already logged in loadContainerRulesFromFile)
Output example with --verbose:
Loading container rules...
- Loaded docker-compose rule COMPOSE-SEC-008 from rules/docker-compose/dangerous_capabilities.py
- Loaded docker-compose rule COMPOSE-SEC-001 from rules/docker-compose/privileged_service.py
Loading rules from /tmp/test-rules...
- Loaded rule CUSTOM-001 from /tmp/test-rules/test_eval.py
Loaded 1 rules
Before: Only visible with --debug flag
After: Visible with --verbose flag
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
* fix(dsl): Add nolint comment for intentional error ignore in hasAnyContainerRulesInPath
- Suppress nilerr linter warning for filepath.Walk error handling
- We intentionally ignore errors during walk and return false
- This is safe because we're just checking if any container rules exist
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
* chore: Bump version to 1.1.5
* fix(tests): Update test filenames to avoid module registry filtering
The module registry now filters out test files (test_*.py) which is correct
for production use. Updated test files to use non-test filenames:
- test_frameworks.py → frameworks.py
- test_os.py → file_ops.py
- test_pathlib.py → dir_utils.py
- test_json.py → json_handler.py
- test_sys.py → cli_args.py
- test_comprehensive.py → comprehensive_stdlib.py
- test_alias.py → alias_module.py
- test_from.py → from_imports.py
- test_multi.py → multi_module.py
- test_baseline.py → baseline.py
- test.py → app.py, process.py, handler.py
This fixes TestFrameworkResolution, TestStdlibRegressionSuite,
TestStdlibEdgeCases, and TestStdlibNoRegression tests.
* feat(scan,ci): Add --skip-tests flag to control test file filtering
Add --skip-tests flag (default: true) to scan and ci commands to give users
control over whether test files should be excluded from analysis.
Changes:
- Add skipTests parameter to BuildModuleRegistry(rootPath, skipTests)
- Update shouldSkipFile() to respect skipTests parameter
- Add --skip-tests flag to scan command (default true)
- Add --skip-tests flag to ci command (default true)
- Update all test/integration callers to pass skipTests parameter
- Add debug logging when test files are being skipped
Usage:
# Default behavior: skip test files
pathfinder scan --rules rules/ --project .
# Include test files in analysis
pathfinder scan --rules rules/ --project . --skip-tests=false
# CI mode with test file analysis
pathfinder ci --rules rules/ --project . --output sarif --skip-tests=false
Rationale:
- Production scanning should skip tests by default (cleaner results)
- Development/testing may need to analyze test files
- User control via flag provides flexibility
- Tests use --skip-tests=false to validate registry behavior
* test(python): Improve test coverage to 98.25% and fix formatting
Add comprehensive tests for cli and decorators modules to exceed 95% coverage requirement.
**Test Coverage:**
- cli/__init__.py: 35% → 99% (added 24 tests)
- get_binary_path() - all 3 priority levels (bundled, PATH, download)
- _is_musl() - musl libc detection for Alpine Linux
- _download_binary() - tar.gz and zip archive handling
- main() - entry point execution with argument passing
- decorators.py: 75% → 90% (added 6 tests)
- _enable_auto_execute() - atexit registration
- _register_rule() - rule registry management
- Auto-execution in __main__ context
- JSON output format validation
**Overall Results:**
- 300 total tests passing (was 280)
- 98.25% total coverage (exceeds 95% requirement)
- All black formatting checks passing
**Files Changed:**
- python-dsl/tests/test_cli.py: Added TestIsMusl, TestGetBinaryPath, TestDownloadBinary, TestMain
- python-dsl/tests/test_decorators.py: Added TestAutoExecution class
- python-dsl/codepathfinder/cli/__init__.py: Black formatting
- python-dsl/compile_container_rules.py: Black formatting
- python-dsl/rules/container_*.py: Black formatting (6 files)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
* fix(python): Remove unused imports from test files
Fix ruff linting errors by removing unused imports:
- test_cli.py: Removed subprocess, os, tempfile, Path, call
- test_decorators.py: Removed sys, atexit, MagicMock, _auto_execute_enabled
All 35 tests still pass. Linting now passes:
- lintPython: ✓ All checks passed
- lintGo: ✓ 0 issues
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
* chore: Add test coverage files to .gitignore
Add coverage artifacts to gitignore:
- .coverage (Python coverage)
- coverage.out (Go coverage)
- htmlcov/ (HTML coverage reports)
- *.coverage (any coverage files)
These are temporary test artifacts and should not be tracked in version control.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
* test(registry): Add comprehensive tests for shouldSkipFile and skipTests parameter
Add tests to improve module registry coverage from 23.52% to 92%+ for new code:
**New Tests:**
- TestShouldSkipFile: Tests file filtering logic with skipTests parameter
- Validates test_ prefix filtering
- Validates _test suffix filtering
- Validates conftest.py, setup.py, __main__.py filtering
- Tests skipTests=false includes all files
- TestBuildModuleRegistry_SkipTestsParameter: Integration test for skipTests
- Creates 2 regular files + 4 test files
- Verifies skipTests=true excludes test files (2 modules)
- Verifies skipTests=false includes all files (6 modules)
- TestBuildModuleRegistry_SkipTestDirectories: Tests directory filtering
- Verifies tests/, test/, fixtures/, mocks/ directories are skipped
- Files within these directories are not indexed
**Coverage Impact:**
- shouldSkipFile(): 0% → 100%
- BuildModuleRegistry(): 85% → 92%
- module.go overall: 23.52% patch → 90.6% package
Addresses Codecov report: 87 lines missing → ~70 lines remaining
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
* chore: Remove result.json and update .gitignore for output files
Remove tracked result.json file (37KB test output) and update .gitignore to prevent output files from being tracked:
**Changes:**
- Removed sast-engine/result.json (test output artifact)
- Updated .gitignore to ignore result.json and scan.json under "output files" section
- Consolidated scan.json entry (was duplicated)
These files are generated during testing/scanning and should not be version controlled.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
* chore: Update .gitignore to exclude output files
Add result.json and reorganize .gitignore output files section:
**Changes:**
- Add result.json to gitignore (test output artifact)
- Consolidate output files under dedicated section
- Remove duplicate scan.json entry
Output files like result.json and scan.json are generated during testing/scanning and should not be version controlled.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
---------
Co-authored-by: Claude Sonnet 4.5 <[email protected]>1 parent e98e5db commit 648826a
File tree
35 files changed
+1255
-273
lines changed- python-dsl
- codepathfinder
- cli
- rules
- tests
- sast-engine
- cmd
- dsl
- graph/callgraph
- builder
- extraction
- registry
- resolution
35 files changed
+1255
-273
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
37 | 46 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
25 | | - | |
| 25 | + | |
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
| |||
49 | 50 | | |
50 | 51 | | |
51 | 52 | | |
| 53 | + | |
52 | 54 | | |
53 | 55 | | |
54 | 56 | | |
| |||
60 | 62 | | |
61 | 63 | | |
62 | 64 | | |
63 | | - | |
64 | | - | |
| 65 | + | |
| 66 | + | |
65 | 67 | | |
66 | 68 | | |
67 | | - | |
| 69 | + | |
68 | 70 | | |
69 | | - | |
| 71 | + | |
70 | 72 | | |
71 | 73 | | |
72 | 74 | | |
| |||
155 | 157 | | |
156 | 158 | | |
157 | 159 | | |
158 | | - | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
159 | 163 | | |
160 | 164 | | |
161 | 165 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
58 | 58 | | |
59 | 59 | | |
60 | 60 | | |
| 61 | + | |
61 | 62 | | |
62 | 63 | | |
63 | 64 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
| 13 | + | |
13 | 14 | | |
14 | | - | |
| 15 | + | |
15 | 16 | | |
16 | 17 | | |
17 | 18 | | |
18 | 19 | | |
19 | 20 | | |
20 | | - | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
24 | 25 | | |
25 | | - | |
26 | | - | |
27 | | - | |
28 | | - | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
| 32 | + | |
36 | 33 | | |
37 | 34 | | |
38 | 35 | | |
| |||
47 | 44 | | |
48 | 45 | | |
49 | 46 | | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
| 47 | + | |
54 | 48 | | |
55 | 49 | | |
56 | 50 | | |
| |||
65 | 59 | | |
66 | 60 | | |
67 | 61 | | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | | - | |
| 62 | + | |
72 | 63 | | |
73 | 64 | | |
74 | 65 | | |
| |||
83 | 74 | | |
84 | 75 | | |
85 | 76 | | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
| 77 | + | |
90 | 78 | | |
91 | 79 | | |
92 | 80 | | |
93 | 81 | | |
94 | 82 | | |
| 83 | + | |
95 | 84 | | |
96 | 85 | | |
97 | 86 | | |
98 | 87 | | |
99 | 88 | | |
100 | 89 | | |
101 | 90 | | |
| 91 | + | |
102 | 92 | | |
103 | 93 | | |
104 | 94 | | |
105 | | - | |
| 95 | + | |
106 | 96 | | |
107 | 97 | | |
108 | 98 | | |
| |||
112 | 102 | | |
113 | 103 | | |
114 | 104 | | |
115 | | - | |
| 105 | + | |
116 | 106 | | |
117 | 107 | | |
118 | 108 | | |
119 | 109 | | |
120 | 110 | | |
121 | 111 | | |
122 | | - | |
| 112 | + | |
123 | 113 | | |
124 | 114 | | |
125 | 115 | | |
| |||
138 | 128 | | |
139 | 129 | | |
140 | 130 | | |
141 | | - | |
| 131 | + | |
142 | 132 | | |
143 | 133 | | |
144 | 134 | | |
145 | 135 | | |
146 | 136 | | |
147 | 137 | | |
148 | | - | |
| 138 | + | |
149 | 139 | | |
150 | 140 | | |
151 | 141 | | |
| |||
157 | 147 | | |
158 | 148 | | |
159 | 149 | | |
160 | | - | |
| 150 | + | |
161 | 151 | | |
162 | 152 | | |
163 | 153 | | |
164 | 154 | | |
165 | 155 | | |
166 | 156 | | |
| 157 | + | |
167 | 158 | | |
168 | 159 | | |
169 | 160 | | |
170 | 161 | | |
171 | | - | |
172 | | - | |
173 | | - | |
174 | | - | |
| 162 | + | |
175 | 163 | | |
176 | 164 | | |
177 | 165 | | |
| |||
213 | 201 | | |
214 | 202 | | |
215 | 203 | | |
216 | | - | |
| 204 | + | |
217 | 205 | | |
218 | 206 | | |
219 | 207 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| 15 | + | |
15 | 16 | | |
16 | 17 | | |
17 | 18 | | |
| |||
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
| 28 | + | |
27 | 29 | | |
28 | 30 | | |
29 | 31 | | |
| |||
32 | 34 | | |
33 | 35 | | |
34 | 36 | | |
| 37 | + | |
35 | 38 | | |
36 | 39 | | |
37 | 40 | | |
| |||
63 | 66 | | |
64 | 67 | | |
65 | 68 | | |
| 69 | + | |
66 | 70 | | |
67 | 71 | | |
68 | 72 | | |
| |||
100 | 104 | | |
101 | 105 | | |
102 | 106 | | |
| 107 | + | |
103 | 108 | | |
104 | 109 | | |
105 | 110 | | |
106 | 111 | | |
107 | 112 | | |
108 | | - | |
| 113 | + | |
109 | 114 | | |
110 | 115 | | |
111 | 116 | | |
| |||
115 | 120 | | |
116 | 121 | | |
117 | 122 | | |
118 | | - | |
| 123 | + | |
119 | 124 | | |
120 | 125 | | |
121 | 126 | | |
| |||
154 | 159 | | |
155 | 160 | | |
156 | 161 | | |
| 162 | + | |
157 | 163 | | |
158 | 164 | | |
159 | 165 | | |
160 | | - | |
| 166 | + | |
161 | 167 | | |
162 | 168 | | |
163 | 169 | | |
| |||
166 | 172 | | |
167 | 173 | | |
168 | 174 | | |
169 | | - | |
| 175 | + | |
170 | 176 | | |
171 | 177 | | |
172 | 178 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
100 | 100 | | |
101 | 101 | | |
102 | 102 | | |
103 | | - | |
| 103 | + | |
104 | 104 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
| 12 | + | |
12 | 13 | | |
13 | 14 | | |
14 | 15 | | |
15 | 16 | | |
16 | 17 | | |
17 | | - | |
18 | | - | |
19 | | - | |
20 | | - | |
| 18 | + | |
21 | 19 | | |
22 | 20 | | |
23 | 21 | | |
24 | 22 | | |
| 23 | + | |
25 | 24 | | |
26 | 25 | | |
27 | 26 | | |
| |||
158 | 157 | | |
159 | 158 | | |
160 | 159 | | |
| 160 | + | |
161 | 161 | | |
162 | 162 | | |
163 | 163 | | |
| |||
0 commit comments