fix: deadlocks between kernel close, the event loop and reactive updates

[maartenbreddels]

Summary

Audit of releases 1.58-1.61 for deadlocks and leaks. Three lock cycles, all reproduced live before the fix:

• context.close() ran on the uvicorn event loop (evict route, lifespan shutdown) while the kernel's message thread holds context.lock for a whole handler and needs the loop for every widget update: evict hung the server permanently, shutdown with --timeout-graceful-shutdown left the loop dead for the handler's duration. Both now run off the loop; shutdown closes all kernels in parallel under a deadline (SOLARA_SERVER_SHUTDOWN_CLOSE_TIMEOUT, default 10s, daemon threads because the interpreter joins executor workers at exit).
• The cull ran close() inline on the shared keep-alive loop: one kernel wedged in a handler stalled the cull of every other kernel (a process-wide leak). It now closes on its own thread.
• update() and field sets fired listeners under the store lock; listeners render, reacton holds the render lock for the whole pass, and the render thread can need the store lock (with mutation detection even on a read). Stores now separate storing (_set_deferred, under the lock) from firing (after).

Found in review and also fixed: Thread.start() hang when the kernel closes between Thread() and start(); close() aborting on a KeyError before closed_event; send_bytes from the loop thread sending twice; in non-threaded mode close() from a plain thread raising on the unset context-id ContextVar; a reconnect marking a page connected on a kernel the cull had just decided to close (CLOSING is now set under the decision's lock hold).

Known trade-off, documented: two threads writing the same reactive at once may notify out of store order (plain .value = x never ordered either; every solara listener re-reads the store).

Docs and tests

• docs/deadlock-rules.md: the lock inventory and six rules.
• tests/unit/deadlock_test.py: deterministic two-thread cycles for each rule, all red before the fix.
• tests/integration/deadlock_test.py: evict while a handler is busy (red with the inline close, green after).
• docs/tla/: PlusCal models; the lock model re-finds the three cycles under the old rules, the lifecycle model found the cull/reconnect race and validated its fix.

Verification

Full unit suite green. Leak integration tests green; 10x10 memory harness on the kitchen-sink app shows no leak and a flat thread count. Live: evict returns while the loop keeps answering /readyz; SIGTERM with a 60s handler exits in 13.6s. Remaining integration failures in server_test.py and test_reconnect_fail are pre-existing on master (verified on a clean checkout).

🤖 Generated with Claude Code