Add web-pod observability: memory/OOM watchdog, in-flight-request logging, concurrency counters

[grossir]

Problem

PR #114 (part of #113) captures the stack of a timed-out worker and logs per-request durations. That closes the hang attribution gap, but three diagnostic blind spots remain — all observability concerns that were deliberately scoped out of #114 to keep it low-risk and reviewable.

Gaps to close

1. OOM / SIGKILL is a black box (highest value)

scanning-web workers are SIGKILL'd (SCANNING-1W, "perhaps out of memory"), and SIGKILL is uncatchable — the only way to get evidence is to log before the kill. There is currently no in-process memory signal at all. Proposed:

• A lightweight background thread logging worker RSS every N seconds (a memory-over-time trace in pod logs, line up against the kill timestamp).
• A pre-OOM watchdog: when RSS crosses a soft threshold (e.g. 85% of the pod limit), dump all-thread stacks + RSS to stderr/Sentry. The only in-process route to "what was the worker doing when it ran out of memory."

2. No "which request + params" attribution

faulthandler names the view function, but not the request context (which scan.pk, user, URL params). And gunicorn's access log writes a line on response completion, so a request that hangs until the worker is killed never logs — the access log shows everything except the culprit. Proposed:

• An in-flight-request logger: middleware records (thread, url, scan_pk, user, start_time) on entry into a shared dict; a background thread logs any entry older than N seconds (with elapsed + context) while it's still hung, and clears it on completion. Names the frozen endpoint with its params, before the kill.

3. Threadpool / concurrency saturation is invisible

A leading hypothesis is asgiref-threadpool exhaustion / GIL starvation, but nothing logs it. Proposed:

• Periodically log in-flight request count and threading.active_count() (pairs naturally with #2's shared state).

Optional

• send_default_pii=True in Sentry init to attach user/request details to error events (privacy tradeoff).
• Structured/JSON logs for searchability in the EKS log aggregator.

Definition of done

• Worker RSS is visible over time in logs; a pre-OOM threshold dump exists, letting SCANNING-1W be classified (pod OOM vs node pressure).
• A hung request is logged with its endpoint + params (scan_pk, user) while still in flight.
• In-flight/active-thread counts are logged, enough to confirm or rule out threadpool saturation.

Notes

• Higher-risk than feat(scanning): report worker-timeout stack traces to Sentry #114: touches the hot request path and adds background threads + shared state — hence a separate PR.
• Relates to Investigate scanning-web WORKER TIMEOUT freezes — add instrumentation to identify the blocking code path #113 (investigation) and feat(scanning): report worker-timeout stack traces to Sentry #114 (stack + duration capture).

[grossir]

Scope update: dropping the memory/OOM watchdog

While investigating the web-pod churn this issue was opened to instrument, the evidence pointed away from memory as the cause, so the shipped work (PR #121) is narrowed to in-flight-request + concurrency logging and leaves out the RSS trace and pre-OOM watchdog originally scoped here.

Why

• The pod deaths aren't OOM. An hour of kubectl get events -n scanning showed zero OOMKilled. Pods are cycled by infrastructure: HPA flapping (4↔8 on CPU), spot-node reclamation → NodeNotReady/TaintManagerEviction, scheduler Preempted, and slow starts behind a 1.5 GB image pull. A specific pod we traced (jtr2d) died via a graceful Killing/Stopping container web — a node drain, not a kill.
• Container memory is healthy. container_memory_working_set_bytes (cAdvisor) plateaus flat around ~400 MB — no leak, no runaway growth toward the pod's request.
• RSS is already observable. The memory-over-time trace this issue proposed is redundant with Prometheus/Grafana (container_memory_working_set_bytes), so logging it from the app added noise without new signal.
• The pod has no resources.limits.memory, so the cgroup reports memory.max=max; a cgroup-driven pre-OOM watchdog is inert until a limit exists, and even the worker-level SCANNING-1W ("Worker sent SIGKILL, perhaps out of memory") looks like gunicorn escalating a timeout (SIGABRT→SIGKILL on a wedged worker), not an actual OOM.

What we kept, and why it's the right lever

The real app-level problem is worker saturation: all 4 sync workers wedged (GIL-bound heavy requests) so nothing answers /monitoring/heartbeat/ → WORKER TIMEOUT (SCANNING-1V) and kubelet liveness-probe kills (#120). The gunicorn access log never logs a hung request and faulthandler (#114) names only the view function — so PR #121 adds the piece that names which request/scan_pk/user saturated the workers while it's still hung, plus in-flight/active-thread counts and per-step breadcrumbs.

Follow-ups spun out of the investigation

• Slim the container image to cut pull time and reduce scale-up/reschedule churn #119 — slim the 1.5 GB image (aggravates churn via slow pulls)
• Web liveness/readiness probes time out under worker saturation → kubelet kills healthy pods (restart storm) #120 — probe timeouts under saturation kill healthy pods
• (proposed) reduce churn from spot reclamation / HPA flapping / preemption

If a real resources.limits.memory is ever set and pod-level OOMs appear, a memory watchdog can be revisited as a focused follow-up — but there's no evidence for it today.