feat(palace): add .obsidian, .terraform, vendor to SKIP_DIRS - #1348
feat(palace): add .obsidian, .terraform, vendor to SKIP_DIRS#1348Phizix wants to merge 1 commit into
Conversation
Closes #1329. Resolves the palace bloat reproducer where `.obsidian/plugins/` JavaScript bundles (and similar tooling directories) get ingested by the convo_miner / sweeper passes. These directories never contain user content worth indexing — they're vendored framework code, terraform provider binaries, or editor plugin bundles. Tested against an Obsidian vault: prior to this change, 26K+ contamination drawers from `.obsidian/plugins/` were ingested in a single pass — a single plugin (excalidraw) contributed ~11.9K bundled JS lines. The vault contained ~9 real markdown files. With SKIP_DIRS expanded, the contamination class is eliminated at the directory walk level. Updates four SKIP_DIRS sets for consistency across the file-walk surface: - mempalace/palace.py — canonical SKIP_DIRS imported by miner.py + convo_miner.py (the file-walk path that runs during `mempalace mine`) - mempalace/entity_detector.py — auto-detection walk - mempalace/project_scanner.py — project-init scan - mempalace/room_detector_local.py — both detect_rooms_from_folders and detect_rooms_from_files paths Adds tests/test_skip_dirs.py with 6 tests: - 3 membership tests (palace.SKIP_DIRS contains .obsidian, .terraform, vendor) - 1 each for project_scanner.SKIP_DIRS, entity_detector.SKIP_DIRS - 1 end-to-end against miner.scan_project: builds a tiny Obsidian-shaped fixture (Notes/index.md + .obsidian/plugins/.../main.js + .obsidian/themes/.../theme.css), verifies real markdown is found and no .obsidian/* paths leak into the corpus. Full upstream pytest: 1495 passed + 1 skipped (was 1489 + 1; +6 = the 6 new SKIP_DIRS tests). No regressions. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
Hi, tests/test_skip_dirs.py builds rel_paths using str(Path(...)) which uses backslashes on Windows, but asserts against a POSIX-style path ("Notes/index.md"). This makes the new test fail on Windows runners. Severity: action required | Category: reliability How to fix: Normalize paths with as_posix() Agent prompt to fix - you can give this to your LLM of choice:
Qodo code review - free for open-source. |
|
Withdrawing this PR. Submitter is not contributing to open source projects at this time. The SKIP_DIRS additions (.obsidian, .terraform, vendor) work locally and the underlying issue #1329 reproducer remains valid — feel free to lift the diff if useful. |
Summary
palace.SKIP_DIRS(the canonical set imported byminer.pyandconvo_miner.py) to include.obsidian,.terraform, andvendor.entity_detector.py,project_scanner.py,room_detector_local.py) where any of the three were missing, for symmetry.Why
.obsidian/plugins/ships large vendored JS bundles per Obsidian plugin. When a user runs mempalace's miner against an Obsidian vault, those bundles get ingested as drawers despite carrying no semantic value — purely framework / plugin code. Same pattern for.terraform(provider binaries) andvendor/(Go / PHP vendored dependencies).Real-world reproducer (Obsidian vault, ~9 real markdown files): 26K+ contamination drawers from
.obsidian/plugins/were ingested in a single mine pass before this fix; one plugin (excalidraw) alone contributed ~11.9K bundled JS lines.Tests
tests/test_skip_dirs.py(new): 6 testspalace.SKIP_DIRS(the canonical set imported by miner / convo_miner)project_scanner.SKIP_DIRSandentity_detector.SKIP_DIRSNotes/index.md+.obsidian/plugins/.../main.js+.obsidian/themes/.../theme.css) and verifiesminer.scan_projectyields the markdown but no.obsidian/*pathsOut of scope (Codex preflight finding, deferred)
Codex preflight flagged that
sweeper.py:311usesdir_p.rglob("*.jsonl")and does NOT route through any SKIP_DIRS set, somempalace sweep <directory>would still sweep.jsonlfiles under.obsidian/. This is a pre-existing coverage hole ondevelop, not introduced by this PR. It's intentionally deferred to a separate fix because (a) it requires its own behavior choice (filterrglobresults vs. switch toos.walkwithdirs[:]pruning) plus separate test coverage, and (b) bundling would expand the PR scope beyond the targeted #1329 fix. Happy to follow up with a separate PR if maintainers want it tackled here.Test plan
test_skip_dirs.pyexercises end-to-end miner walkmempalace mine <obsidian-vault>against a vault with.obsidian/plugins/; confirm zero drawers ingested from skip-dir pathsCloses #1329