feat: binary hook engine with lexer, streaming, and rewrite_compound integration

[ahundt]

Summary

This PR #536 integrates the hooks-v2 engine (from PR #156) with the develop branch (note that #537 is the same code as 536 and 156 but it is stripped of tests for easier reviewing), and #361 describes the bug fixes in this pr, this pr #536 demonstrates that:

1. The lexer is a drop-in replacement for rewrite_compound()'s inline byte scanner — all 155 existing rewrite tests pass unchanged
2. The binary hook path (rtk hook claude) works alongside the shell hook path (rtk rewrite)
3. The streaming infrastructure (stream.rs) enables bidirectional process shims with exit code transparency

Key components

• src/cmd/lexer.rs — Shell-aware tokenizer (56 tests): handles quoting, escapes, $(), backticks, globs, redirects, 2>&1
• src/cmd/analysis.rs — Command chain parser (61+ tests): needs_shell() detection, parse_chain() for &&/||/; chains
• src/cmd/hook/mod.rs — Binary hook engine (98 tests): conservative hook_lookup() whitelist, Claude Code JSON protocol, safe suffix handling
• src/cmd/exec.rs — Native command executor: runs simple chains natively, delegates complex shell to /bin/sh
• src/stream.rs — Streaming process execution (39 tests): StreamFilter trait, StdinMode::Inherit for pipe support, SIGPIPE handling, 1 MiB raw cap
• src/pipe_cmd.rs — rtk pipe --filter dispatch for grep/rg/find/fd output filtering
• src/discover/registry.rs — rewrite_compound() replaced: lexer-based splitting instead of 130-line inline byte scanner

Lexer integration in rewrite_compound()

The byte-level quote tracker in rewrite_compound() (~130 lines) is replaced with lexer::tokenize() calls (~80 lines). The lexer adds:

• Escape sequence handling (\\)
• $() and backtick detection (Shellism tokens)
• Glob detection (*, ? → Shellism)
• Redirect parsing — not confused with && operator (2>&1, &>/dev/null)

Token byte offsets (ParsedToken.offset) enable extracting original substrings from the command string, so rewrite_segment() receives exact original text — no reconstruction fidelity issues.

Binary hook architecture

Two complementary routing paths:

• Shell hook: rtk rewrite → registry::rewrite_command() → rewrite_compound() → rewrite_segment() (TOML filters + RULES table)
• Binary hook: rtk hook claude → check_for_hook() → lexer::tokenize() → analysis::parse_chain() → hook_lookup() → route_native_command() (conservative whitelist)

The binary hook uses a strict whitelist (hook_lookup()) that only routes subcommands RTK optimizes well (e.g., git status but not git rebase, cargo test but not cargo publish).

Init system (src/init.rs)

• --hook-type binary|script flag to select hook mode
• Plugin cache patching: removes Bash from plugin matchers, writes manifest
• Manifest-based handler dispatch for both NoOpinion and Allow paths
• backup_file_once() + atomic_write() for safe file operations
• OpenCode plugin support (develop's --install-opencode flag preserved)

Test plan

• cargo test — 1309 passed, 0 failed, 6 ignored
• cargo fmt --all — clean
• cargo clippy --all-targets — clean
• All 155 registry rewrite tests pass (including develop's TOML rule tests)
• All 56 lexer tests pass
• All 98 hook tests pass
• All 39 streaming tests pass
• Manual: rtk rewrite "cargo test && git status" produces rtk cargo test && rtk git status
• Manual: rtk rewrite "cargo test | grep FAIL" produces rtk cargo test | grep FAIL
• Manual: rtk rewrite "cargo test 2>&1" produces rtk cargo test 2>&1

Stats

46 files changed, ~9500 insertions, ~450 deletions

Fixes #222

[ahundt]

@KuSh @pszymkowiak everything that is outstanding should be addressed including the comments above! There are also no conflicts with the base branch and it is ready to be merged w.r.t. git conflicts at the time of writing.

[ahundt]

I updated again to match upstream, it is ready to merge to the best of my knowledge! Additionally this pr fixes the multi-hook conflict bug that is now independently confirmed by #361 (comment)