Batch catalog reindex into sequential chords to bound Redis broker pressure

[feoh]

What are the relevant tickets?

Closes #3623

Description (What does it do?)

start_recreate_index and start_update_index previously materialized a subtask for every ~100-item chunk of the entire catalog and dispatched all of them as one giant celery chord. With the Redis result backend, every subtask RPUSHes its result into a single per-group list, so a full reindex produced a massive burst of Redis list/set commands and result keys on the shared mitlearn-redis broker/cache instance — the failure mode behind the 2026-07-15 memory saturation (#3624) and the ongoing paging.

This change bounds the in-flight footprint:

• replace_with_batched_reindex splits the index subtask list into batches of at most OPENSEARCH_REINDEX_BATCH_SIZE (new setting, default 100) and dispatches only the first batch as a chord.
• continue_reindex_batches (each batch chord's callback) accumulates that batch's error strings, then dispatches the next batch — so only one batch's chord bookkeeping lives on Redis at a time — and finally invokes the original finish signature with all accumulated results.
• Batch state (remaining subtask signatures, interim errors, the finish signature) is stored in the durable database cache keyed by the originating task's run id, keeping the multi-MB batch snapshot off Redis entirely. Keys are deleted when the run completes, with a 7-day TTL as a leak backstop.
• Existing semantics are preserved: finish_recreate_index still receives the merged error strings and performs the alias swap exactly once after all batches complete (merge_strings aggregation unchanged), and finish_update_index still clears the views cache. Callers waiting on the original task still wait for the full chain via Task.replace.

If there are zero index subtasks, the task skips straight to the finish signature (previously an empty chord was dispatched).

How can this be tested?

• pytest learning_resources_search/tasks_test.py — includes new tests covering batch splitting/storage (test_replace_with_batched_reindex), the no-op path, sequential dispatch + error accumulation (test_continue_reindex_batches_*), cache cleanup on completion, and loud failure when batch state is missing/expired.
• Full affected suite run locally: learning_resources_search/tasks_test.py, learning_resources/tasks_test.py, vector_search/tasks_test.py, learning_resources_search/management — 163 passed.
• Manual validation: run ./manage.py recreate_index --all (or update_index) against a populated environment and observe on the Redis side that chord result lists stay bounded (≤ OPENSEARCH_REINDEX_BATCH_SIZE entries) instead of growing to catalog size, while the final alias swap still occurs once at the end.

Additional Context

• Companion mitigations already merged: Bound Celery result retention to reduce mitlearn Redis memory pressure #3625 (CELERY_RESULT_EXPIRES bound). Move Celery result backend off the shared mitlearn broker/cache Redis #3622 (separate result backend) remains available as defense in depth.
• OPENSEARCH_REINDEX_BATCH_SIZE is env-configurable; 100 subtasks/batch ≈ 10k documents in flight per batch at the default chunk sizes.
• The mocked_celery fixture now also patches celery.chord since the reindex canvas uses chords directly instead of chain(group, ...).

[github-actions]

OpenAPI Changes

No changes detected

View full changelog

Unexpected changes? Ensure your branch is up-to-date with main (consider rebasing).

[Copilot]

Pull request overview

This PR updates the OpenSearch reindex Celery canvas to avoid dispatching one catalog-wide chord (which can overwhelm the shared Redis broker/result backend) by splitting the work into sequential chord batches and persisting batch state in the durable (DB) cache.

Changes:

• Introduces replace_with_batched_reindex + continue_reindex_batches to run reindex subtasks as sequential chords of bounded size, accumulating errors across batches.
• Adds OPENSEARCH_REINDEX_BATCH_SIZE (default 100) to control max subtasks per chord batch.
• Updates test utilities and task tests to account for direct celery.chord usage and to validate batching behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
main/settings.py	Adds OPENSEARCH_REINDEX_BATCH_SIZE setting to bound chord batch size.
learning_resources_search/tasks.py	Implements sequential chord batching with durable-cache persisted state and updates reindex entrypoints to use it.
learning_resources_search/tasks_test.py	Adds/updates tests to validate batching, sequential dispatch, error accumulation, and no-op behavior.
fixtures/common.py	Extends mocked_celery fixture to patch celery.chord for the new canvas pattern.

[feoh]

@copilot Please fix pre-commit errors.

[Copilot]

@copilot Please fix pre-commit errors.

Fixed in commit b1a8deb. The pre-commit.ci bot had added blank lines to separate the if return block in continue_reindex_batches, but the code block that follows (reading accumulated errors, validating the finish signature, cleaning up cache keys, and doing the final replace) was accidentally at module level instead of indented inside the function body. The fix properly indents those lines as part of the function. Ruff lint and format checks now pass cleanly.

[feoh]

@rhysyngsun points out that this is too risky to be desirable before the weekend. I'll leave it for now.