feat: incremental graph rebuild (3-PR sequence: G1+G2+G3)

[HumanBean17]

Scope

Implements the full incremental graph rebuild plan from plans/active/PLAN-INCREMENTAL-GRAPH.md as a single PR containing three logically distinct layers:

PR-G1: Hash tracker + source_file edge schema

• Adds source_file STRING column to all 12 edge table DDLs for file-scoped deletion
• Implements FileHashTracker class (SHA-256, atomic save, change detection)
• Bumps ONTOLOGY_VERSION from 16 → 17 (re-index required)

PR-G2: Incremental rebuild orchestrator

• Adds incremental_rebuild() function with scoped pass 1–4 and global pass 5–6
• Implements _load_existing_types(), _load_existing_members() for cross-file resolution
• Implements _find_dependents() for single-hop dependent expansion (cap: 50 files)
• Implements phase-based _delete_file_scope() (all edges before any nodes)
• Implements _scoped_write() for writing into existing DB without schema drop
• Crash safety via .graph_increment_in_progress marker file with fallback to full rebuild
• Adds --incremental CLI flag to build_ast_graph.py
• Adds run_incremental_graph() wrapper to pipeline.py

PR-G3: CLI integration

• Updates _cmd_increment() to run incremental graph update after CocoIndex
• Adds --vectors-only flag to preserve old Lance-only behavior
• Removes stale _INCREMENT_WARNING_LINES / _emit_increment_kuzu_warning()
• Updates README CLI cheat sheet and roadmap
• Updates docs/JAVA-CODEBASE-RAG-CLI.md

Files changed

File	PR	Changes
ast_java.py	G1	ONTOLOGY_VERSION 16 → 17
build_ast_graph.py	G1	source_file STRING on all 12 edge DDLs + edge-write queries + FileHashTracker
build_ast_graph.py	G2	incremental_rebuild(), _load_existing_*(), _find_dependents(), _delete_file_scope(), _scoped_write(), --incremental flag
java_codebase_rag/pipeline.py	G2/G3	run_incremental_graph() wrapper
java_codebase_rag/cli.py	G3	increment calls graph update, --vectors-only, removes stale warning
tests/test_incremental_graph.py	G1+G2	22 tests (9 G1 + 13 G2)
tests/test_java_codebase_rag_cli.py	G3	Updated stale warning test + 5 new CLI tests
README.md	G3	CLI cheat sheet + roadmap update
docs/JAVA-CODEBASE-RAG-CLI.md	G3	increment command docs

Manual Evidence

# All 22 incremental tests pass
.venv/bin/python -m pytest tests/test_incremental_graph.py -v
============================== 22 passed ==============================

# CLI tests pass
.venv/bin/python -m pytest tests/test_java_codebase_rag_cli.py -v
============================== 19 passed ==============================

# Lint checks pass
.venv/bin/ruff check build_ast_graph.py ast_java.py tests/test_incremental_graph.py java_codebase_rag/cli.py java_codebase_rag/pipeline.py README.md docs/JAVA-CODEBASE-RAG-CLI.md tests/test_java_codebase_rag_cli.py
All checks passed!

# Full test suite passes
.venv/bin/python -m pytest tests -v
============ 670 passed, 9 skipped, 0 failed ============

Design Notes

• Phase-based deletion: _delete_file_scope deletes ALL edges across all scope files first, then deletes nodes. This prevents Kuzu errors when file A's nodes have incoming edges from file B that haven't been cleaned up yet.
• Pass 5–6 always global: Client/producer extraction and cross-service matching iterate all members/routes — cheap in-memory operations that ensure consistency.
• source_file semantics: Origin-side file only (e.g., for CALLS edges, it's the caller's filename). Dependent expansion covers target-side changes.

Reindex Required

Existing installations must run java-codebase-rag reprocess once after upgrading to add the source_file column to edge tables. This is a one-time migration triggered by the ONTOLOGY_VERSION bump from 16 to 17.

🤖 Generated with Claude Code

[HumanBean17]

PR #284 Review: Incremental Graph Rebuild (G1+G2+G3)

+2003 / -67 lines across 8 files — architecture is sound, but there's a critical runtime bug that must be fixed before merge.

---

Critical Bug: _write_clients_producers_and_calls will crash at runtime

This function has multiple issues that would cause failures on any real codebase with routes/clients/producers:

1. Wrong parameter names — The Cypher templates use $sid/$cid/$pid/$rid for MATCH node lookups, but the function passes $src/$dst:

# _CREATE_DECLARES_CLIENT expects: MATCH (s:Symbol {id: $sid}), (c:Client {id: $cid})
conn.execute(_CREATE_DECLARES_CLIENT, {
    "src": row.symbol_id,   # BUG: should be "sid"
    "dst": row.client_id,   # BUG: should be "cid"
    ...
})

Same mismatch for DECLARES_PRODUCER ($sid/$pid → "src"/"dst"), HTTP_CALLS ($cid/$rid → "src"/"dst"), and ASYNC_CALLS ($pid/$rid → "src"/"dst").

2. Missing parameters — All four edge-writing loops omit required template parameters:

Edge type	Missing params
DECLARES_CLIENT	strategy
DECLARES_PRODUCER	strategy
HTTP_CALLS	strategy, method_call, raw_uri, match
ASYNC_CALLS	strategy, direction, raw_topic, match

3. O(n²) lookup — Source file resolution for HTTP_CALLS/ASYNC_CALLS builds a new list and does .index() for every row:

tables.client_rows[[c.id for c in tables.client_rows].index(row.client_id)].filename

Should use a dict lookup like the existing _write_routes_and_exposes does.

Why tests don't catch this: The minimal test fixtures (class A {}, class B extends A {}) have no routes/clients/producers, so these loops are never entered. A test with Spring-annotated classes would surface the crash.

---

Significant Issues (should fix)

4. Duplicated _load_existing_types / _load_existing_types_filtered — These two functions (and their _members counterparts) are ~50 lines each and differ only by the AND NOT (s.filename IN $exclude_files) clause. Extract common logic into a single function with an optional exclude_files param.

5. Repetitive _find_dependents — Six nearly identical if/elif branches that only differ by the edge label name. Loop over a list of edge type strings instead.

6. _write_nodes_merge duplicates _write_nodes — These ~70-line functions differ only by using _MERGE_SYMBOL vs _CREATE_SYMBOL. Factor to a shared helper that accepts the query template as a parameter.

7. _file_by_node_id built twice independently — Once in _write_edges and once in _write_routes_and_exposes. Compute once and share.

---

Minor Issues (nice to fix)

8. Duplicate comment in pass1_parse — "Skip files not in scope" appears twice.

9. import json inside main() and _cmd_increment — should be top-level imports.

10. FileHashTracker.save() silently swallows OSError — consider logging a warning so it's discoverable when writes fail repeatedly.

11. _fallback_to_full duplicates hash-init logic from the no-DB branch of incremental_rebuild — extract to a shared helper.

12. AGENTS.md says ontology_version is 15 but is now 17 — worth updating.

---

Test Coverage Gap

No test with Spring annotations (routes, clients, producers) that exercises the pass 5-6 global path with actual Client/Producer/HTTP_CALLS/ASYNC_CALLS edges. This is why the critical bug above slipped through. Adding one test against the bank-chat-system fixture (which has Spring controllers) would catch it.

---

Summary

The design — phase-based deletion, single-hop dependent expansion with cap, crash marker, automatic fallback — is well-considered. The critical bug in _write_clients_producers_and_calls (wrong param names + missing params) must be fixed before merge. The duplication issues are worth addressing but not blocking.