Skip to content

feat(palace): add .obsidian, .terraform, vendor to SKIP_DIRS - #1348

Closed
Phizix wants to merge 1 commit into
MemPalace:developfrom
Phizix:feat/skip-obsidian-vendor-terraform
Closed

feat(palace): add .obsidian, .terraform, vendor to SKIP_DIRS#1348
Phizix wants to merge 1 commit into
MemPalace:developfrom
Phizix:feat/skip-obsidian-vendor-terraform

Conversation

@Phizix

@Phizix Phizix commented May 4, 2026

Copy link
Copy Markdown

Summary

  • Expands palace.SKIP_DIRS (the canonical set imported by miner.py and convo_miner.py) to include .obsidian, .terraform, and vendor.
  • Mirrors the addition into the three other tooling-walk SKIP_DIRS sets (entity_detector.py, project_scanner.py, room_detector_local.py) where any of the three were missing, for symmetry.
  • Eliminates the contamination class behind issue Stop hook: 1.9 TB palace bloat + ChromaDB Rust bindings segfault despite #1231 fix #1329.

Why

.obsidian/plugins/ ships large vendored JS bundles per Obsidian plugin. When a user runs mempalace's miner against an Obsidian vault, those bundles get ingested as drawers despite carrying no semantic value — purely framework / plugin code. Same pattern for .terraform (provider binaries) and vendor/ (Go / PHP vendored dependencies).

Real-world reproducer (Obsidian vault, ~9 real markdown files): 26K+ contamination drawers from .obsidian/plugins/ were ingested in a single mine pass before this fix; one plugin (excalidraw) alone contributed ~11.9K bundled JS lines.

Tests

  • tests/test_skip_dirs.py (new): 6 tests
    • 3 membership tests on palace.SKIP_DIRS (the canonical set imported by miner / convo_miner)
    • 1 each on project_scanner.SKIP_DIRS and entity_detector.SKIP_DIRS
    • 1 end-to-end test that builds a tiny Obsidian-shaped fixture (Notes/index.md + .obsidian/plugins/.../main.js + .obsidian/themes/.../theme.css) and verifies miner.scan_project yields the markdown but no .obsidian/* paths
  • Full suite: 1495 pass + 1 skip (was 1489 + 1; +6 new). No regressions.

Out of scope (Codex preflight finding, deferred)

Codex preflight flagged that sweeper.py:311 uses dir_p.rglob("*.jsonl") and does NOT route through any SKIP_DIRS set, so mempalace sweep <directory> would still sweep .jsonl files under .obsidian/. This is a pre-existing coverage hole on develop, not introduced by this PR. It's intentionally deferred to a separate fix because (a) it requires its own behavior choice (filter rglob results vs. switch to os.walk with dirs[:] pruning) plus separate test coverage, and (b) bundling would expand the PR scope beyond the targeted #1329 fix. Happy to follow up with a separate PR if maintainers want it tackled here.

Test plan

  • Existing test suite passes (1495 + 1 skipped)
  • New test_skip_dirs.py exercises end-to-end miner walk
  • Manual verification: run mempalace mine <obsidian-vault> against a vault with .obsidian/plugins/; confirm zero drawers ingested from skip-dir paths

Closes #1329

Closes #1329.

Resolves the palace bloat reproducer where `.obsidian/plugins/`
JavaScript bundles (and similar tooling directories) get ingested by
the convo_miner / sweeper passes. These directories never contain user
content worth indexing — they're vendored framework code, terraform
provider binaries, or editor plugin bundles.

Tested against an Obsidian vault: prior to this change, 26K+
contamination drawers from `.obsidian/plugins/` were ingested in a
single pass — a single plugin (excalidraw) contributed ~11.9K bundled
JS lines. The vault contained ~9 real markdown files. With SKIP_DIRS
expanded, the contamination class is eliminated at the directory walk
level.

Updates four SKIP_DIRS sets for consistency across the file-walk
surface:
  - mempalace/palace.py — canonical SKIP_DIRS imported by miner.py +
    convo_miner.py (the file-walk path that runs during `mempalace mine`)
  - mempalace/entity_detector.py — auto-detection walk
  - mempalace/project_scanner.py — project-init scan
  - mempalace/room_detector_local.py — both detect_rooms_from_folders
    and detect_rooms_from_files paths

Adds tests/test_skip_dirs.py with 6 tests:
  - 3 membership tests (palace.SKIP_DIRS contains .obsidian, .terraform, vendor)
  - 1 each for project_scanner.SKIP_DIRS, entity_detector.SKIP_DIRS
  - 1 end-to-end against miner.scan_project: builds a tiny
    Obsidian-shaped fixture (Notes/index.md + .obsidian/plugins/.../main.js
    + .obsidian/themes/.../theme.css), verifies real markdown is found and
    no .obsidian/* paths leak into the corpus.

Full upstream pytest: 1495 passed + 1 skipped (was 1489 + 1; +6 = the
6 new SKIP_DIRS tests). No regressions.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@Qodo-Free-For-OSS

Copy link
Copy Markdown

Hi, tests/test_skip_dirs.py builds rel_paths using str(Path(...)) which uses backslashes on Windows, but asserts against a POSIX-style path ("Notes/index.md"). This makes the new test fail on Windows runners.

Severity: action required | Category: reliability

How to fix: Normalize paths with as_posix()

Agent prompt to fix - you can give this to your LLM of choice:

Issue description

tests/test_skip_dirs.py is not portable to Windows because it converts Path objects to str, producing backslash-separated paths, while assertions expect forward slashes.

Issue Context

Other tests in this repo already normalize relative paths with .as_posix() for OS-independent comparisons.

Fix Focus Areas

  • tests/test_skip_dirs.py[71-76]

Suggested change

Change:

rel_paths = {str(Path(f).relative_to(tmp_path)) for f in files}

To:

rel_paths = {Path(f).relative_to(tmp_path).as_posix() for f in files}

(Or equivalent normalization before assertions.)


Qodo code review - free for open-source.

@Phizix

Phizix commented May 6, 2026

Copy link
Copy Markdown
Author

Withdrawing this PR. Submitter is not contributing to open source projects at this time. The SKIP_DIRS additions (.obsidian, .terraform, vendor) work locally and the underlying issue #1329 reproducer remains valid — feel free to lift the diff if useful.

@Phizix Phizix closed this May 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stop hook: 1.9 TB palace bloat + ChromaDB Rust bindings segfault despite #1231 fix

2 participants