fix: guard stale heartbeat rescue and restore discovery

[elliot245]

Summary

Closes #167.

• Add a serialized final stale-rescue revalidation immediately before any stop.
• Carry the exact originating HB_ID and pane-tail fingerprint through rescue timers.
• Skip acknowledged, superseded, busy, pane-progress, inbound-progress, missing-session, and newer-heartbeat cases with an auditable HB_RESCUE_STALE_SKIP disposition and no stop/start/prime.
• Rediscover Codex sessions by exact owner marker and normalized cwd when the provider-session mapping is stale.
• Update the rescue runbook and add regression coverage.

Verification

• python3 -m unittest discover -s agent-manager/scripts/tests -p 'test_*.py'
• Result: Ran 326 tests in 81.312s — OK
• Head: e6261af
• Existing dirty checkout and live Agent sessions were not touched.

[elliot245]

QA Verdict: FAIL

Exact head: e6261affc18079102996862136f2825189f5ffc9
Exact tree: c9c7cf9a4a66b1bb68feba997c317e9a1b88de86

P0/P1/P2: 0/2/0

P1 — auto rescue still has fail-open pane-evidence paths

The in-process pending-heartbeat rescue calls _restart_heartbeat_session_restore(...) with the heartbeat ID but without baseline_pane_hash (main.py:3115-3122). The final guard only reads the pane when that optional hash is truthy (main.py:1305-1308), so pane progress between preflight and stop is never observed on this real caller path.

The delayed path also turns a failed capture_output() into SHA-1(empty) (main.py:2301), and revalidation turns another capture failure into the same value (main.py:1306). Two missing observations therefore compare equal and authorize stop/start rather than failing closed.

Exact-head adversarial reproduction:

REPRO immediate_no_baseline: stop=1 start=1 pane_capture=0
REPRO capture_failure_collision: pane_hash=da39a3ee5e6b4b0d3255bfef95601890afd80709 stop=1 start=1

Please make pane evidence mandatory for every automatic rescue caller and treat either scheduling-time or final capture failure as stale-skip/no mutation. Add regressions through the immediate caller and failed-capture path, not only the helper with an explicitly supplied hash.

P1 — the “final” inbound guard is not concurrency-safe with enqueue

has_pending_inbound_messages() is checked at main.py:1309, then the guard returns. The caller next performs another session_exists() and stops at main.py:2356-2357. The rescue flock only covers the rescue/audit lock file; inbound append/enqueue (services/inbound_queue.py:90-96,152-190) does not participate in it. A new inbound message can therefore become pending after the check and before stop_session().

Exact-head deterministic interleaving:

REPRO inbound_after_guard: stop=1 inbound_pending_at_stop=1

Please bind inbound enqueue and rescue finalization through a shared agent-scoped barrier/lock (or an equivalent atomic protocol) so no inbound can appear between the authoritative check and stop, and add a concurrent regression.

Passing evidence

• All six exact-head GitHub checks: SUCCESS.
• python3 -m unittest discover -s agent-manager/scripts -p 'test_*.py': 326 tests, OK.
• python3 -m compileall -q agent-manager/scripts: PASS.
• git diff --check base..head: PASS.
• Provider stale-mapping rediscovery does resume the discovered exact owner/cwd session in the covered path.

Review evidence is bound to the exact head above. No source changes, merge, or release action were performed.

[elliot245]

QA Verdict: PASS

Exact head: 4fef7125fea443cbbd9853bd3711470328c419cc
Exact tree: 0e1529827654b7fafd0778b3dce7d0d5685e4733

P0/P1/P2: 0/0/0

The two findings from the previous exact-head FAIL are resolved:

1. Every automatic rescue now carries mandatory pane evidence. The immediate caller captures and passes a baseline; scheduling-time capture failure prevents timer creation; missing baseline, failed final capture, and changed pane hash all stale-skip with zero stop/start.
2. New inbound enqueue and rescue final revalidation/stop share the same flock. received -> queued cannot become visible between the authoritative no-inbound check and stop_session(); it is linearized after the stop boundary and remains queued for restore/drain.

Independent adversarial evidence:

• immediate baseline capture failure: stop=0 start=0 reason=pane_baseline_missing;
• final pane capture failure: stop=0 start=0;
• cross-process enqueue interleaving: child remained blocked at stop, then completed after lock release with pending_after_stop=1.

Validation:

• Exact-head GitHub checks: 6/6 SUCCESS (Quality, Integration Matrix, Python 3.9–3.12).
• Full exact-head CI test/coverage gate: PASS; local full suite observed 330 tests, OK.
• Final focused heartbeat-recovery suite: 62 tests, OK.
• Integration suite, compileall, and git diff --check: PASS.
• Provider stale-mapping exact owner/cwd rediscovery remains PASS.

Residual: the lock is repo-scoped rather than agent-scoped, which is conservative and may briefly serialize unrelated agent enqueue, but does not affect correctness or the issue #167 acceptance criteria.

No P0/P1/P2 findings remain. Ready to merge.

[elliot245]

@sulabs2022 Could you please review/approve this exact head so the protected-branch merge gate can complete?

• Head: 4fef7125fea443cbbd9853bd3711470328c419cc
• CI: 6/6 SUCCESS
• Independent QA: PASS, P0/P1/P2=0/0/0 (evidence)
• The two prior P1 findings are closed; no commits have landed after the QA-bound head.

The author/admin account cannot self-approve and branch protection enforces one write-level approval for admins. Please leave an APPROVED review if the exact head is acceptable.