Bound celery pool-child RSS so one runaway task can't take the container

[blarghmatey]

What are the relevant tickets?

https://github.com/mitodl/hq/issues/12673 — the application-side fix is mitodl/mit-learn#3780. This is the infrastructure-side guard; the two are independent.

Description (What does it do?)

mitlearn-default-celery-worker has been OOMKilling on applications-production continuously since 2026-08-13, at 15–24 container restarts an hour.

The celery worker command sets --max-tasks-per-child 100 with no memory-based recycle trigger, so a child that balloons on task 1 stays resident for the next 99 and the kernel kills the whole container — taking unrelated in-flight tasks with it.

• Adds max_memory_per_child_kib to OLApplicationK8sCeleryWorkerConfig, emitting --max-memory-per-child only when set. Defaults to None, so every other OLApplicationK8s consumer is unchanged — this flag decides when celery retires a pool child, and shouldn't arrive as a silent default across every application.
• Sets it to 655360 (640Mi) on mit_learn's default worker. A healthy child on that queue sits at ~165Mi — the whole container runs ~480Mi for master + 2 children — so it only fires on a child that has genuinely ballooned.
• Sized against the floor-derived limit (1Gi request × the declared 2:1 ratio = 2Gi), not the 2560Mi declared or the 6144Mi the VPA currently enforces, because the floor is the smallest limit a pod can run under. Coupled to --concurrency=2 and _worker_vpa_bounds["min_allowed"] — revisit the three together.
• Deliberately not applied to the embeddings or edx_content workers. The embeddings worker holds an ML model, so recycling on memory would mean reloading it.

How can this be tested?

pytest tests/ol_infrastructure/components/services/ — 114 passed, including three new cases: flag absent by default, flag and value emitted as separate argv entries, and non-positive values rejected by the validator.

pre-commit run on the changed files — all hooks pass, including mypy.

pulumi preview --stack Production shows the celery Deployment as an update, not a replacement, touching only the command array:

~ kubernetes:apps/v1:Deployment: (update)
    [id=mitlearn/mitlearn-default-celery-worker]
  ~ command: [
      ~ [11]: "--concurrency=2"        => "--max-memory-per-child"
      ~ [12]: "--prefetch-multiplier=1" => "655360"
      + [13]: "--concurrency=2"
      + [14]: "--prefetch-multiplier=1"
    ]

Only the default worker's Deployment appears — edx-content and embeddings are untouched, which confirms the opt-in default behaves as intended.

Additional Context

This is the guard, not the cure. Celery checks child RSS after a task returns, so it bounds what a child carries into the next task. A single task that blows the cgroup limit within one execution still OOM-kills the container. That is what mitodl/mit-learn#3780 fixes — the PostHog view-event ETL was retaining the entire S3 backlog in memory, on both the extract and load sides. This PR does not clear the alert on its own.

Resizing is not an option here. The VPA on this deployment is already pegged at its ceiling: requests read exactly 3Gi (= _worker_vpa_bounds["max_allowed"]["memory"]) against a 6144MiB enforced limit vs 2560MiB declared. The 2:1 ratio and 1Gi floor added after earlier incidents guard the floor; here the ceiling binds, so raising max_allowed only moves the wall.

Follow-up worth doing once memory is bounded: re-tighten max_allowed back down from 3Gi. Every replica currently reserves 3Gi against up to 20 replicas while fresh pods sit at 480Mi.

Unrelated pending drift, visible in the same pulumi preview and not from this branch — worth knowing before applying, since it would ride along: the fastly-provider version, the fastly-mit_learn-production gzip content-type/extension list ordering, and the mitlearn-webapp-pod-monitor selector labels.

[Copilot]

Pull request overview

Adds an opt-in Celery child-memory recycle guard, applied only to MIT Learn’s default worker.

Changes:

• Adds validated max_memory_per_child_kib configuration.
• Emits --max-memory-per-child when configured.
• Adds coverage for defaults, argument rendering, and validation.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
src/ol_infrastructure/components/services/k8s.py	Adds and renders the optional Celery memory threshold.
src/ol_infrastructure/applications/mit_learn/__main__.py	Configures a 640 MiB threshold for the default worker.
tests/ol_infrastructure/components/services/test_k8s_extra_containers.py	Tests threshold behavior and validation.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.