HNT-1890: shared curated recommendation cache

[mmiermans]

References

JIRA: HNT-1890

Description

Adds a shared Redis cache between the per-pod in-memory cache and the Corpus GraphQL API. ~300 pods currently hit the API independently every ~60s. With this, one pod fetches and the rest read from Redis.

Disabled by default. Enable with corpus_cache.cache = "redis".

How it works

Requests are always served from the in-memory cache (L1) — either fresh or stale. Stale data is served whenever possible so that the Redis and API calls never block the request. Revalidation happens in a background task: it checks Redis (L2) first, and only the pod that acquires the distributed lock fetches from the API.

flowchart TB
    req["Firefox NewTab Request"]

    subgraph L1 ["L1 — Per-Pod In-Memory SWR"]
        check_l1{{"Check in-memory cache"}}
    end

    respond_fresh["Respond with fresh data"]
    respond_stale["Respond with stale data"]

    subgraph bg ["Background Revalidation Task"]
        direction TB

        subgraph L2 ["L2 — Shared Redis"]
            check_l2{{"Check Redis cache"}}
            acquire_lock{{"Try distributed lock"}}
        end

        api["Fetch from Corpus GraphQL API"]
        write["Write to Redis + release lock + update L1"]
    end

    req --> check_l1

    check_l1 -- "FRESH HIT" --> respond_fresh
    check_l1 -- "STALE" --> respond_stale
    check_l1 -. "MISS (cold start, blocks)" .-> check_l2

    respond_stale -. "spawns task" .-> check_l2

    check_l2 -- "FRESH HIT" --> done_l2["Update L1 cache"]
    check_l2 -. "STALE" .-> acquire_lock
    check_l2 -. "MISS" .-> acquire_lock

    acquire_lock -- "LOCK ACQUIRED" --> api
    acquire_lock -. "LOCK HELD + stale exists" .-> serve_stale["Return stale data"]
    acquire_lock -. "LOCK HELD + no data" .-> retry["Wait → retry Redis → fetch from API"]

    api --> write --> done_api["Update L1 cache"]

    style req fill:#2c3e50,stroke:#1a252f,color:#ecf0f1,stroke-width:2px
    style check_l1 fill:#2980b9,stroke:#1f6da0,color:#fff,stroke-width:2px
    style check_l2 fill:#d35400,stroke:#a04000,color:#fff,stroke-width:2px
    style acquire_lock fill:#e67e22,stroke:#bf6516,color:#fff,stroke-width:2px
    style api fill:#1e8449,stroke:#145a32,color:#fff,stroke-width:2px
    style write fill:#1e8449,stroke:#145a32,color:#fff,stroke-width:2px
    style respond_fresh fill:#27ae60,stroke:#1e8449,color:#fff,stroke-width:2px
    style respond_stale fill:#27ae60,stroke:#1e8449,color:#fff,stroke-width:2px
    style serve_stale fill:#f4d03f,stroke:#d4ac0f,color:#333
    style retry fill:#f4d03f,stroke:#d4ac0f,color:#333
    style done_l2 fill:#27ae60,stroke:#1e8449,color:#fff
    style done_api fill:#27ae60,stroke:#1e8449,color:#fff
    style L1 fill:#eaf2f8,stroke:#2980b9,stroke-width:2px,color:#2c3e50
    style L2 fill:#fef5e7,stroke:#d35400,stroke-width:2px,color:#2c3e50
    style bg fill:#f4f6f7,stroke:#95a5a6,stroke-width:2px,stroke-dasharray: 8 4,color:#2c3e50

Loading

Design decisions

Decision	Choice	Why
Cache layer	Redis L2 behind existing in-memory L1	Keeps per-pod latency low, Redis only consulted on L1 miss
Write pattern	Distributed stale-while-revalidate	One pod revalidates, others serve stale. Avoids thundering herd
Lock mechanism	SET NX EX 30	Simple, self-expiring. Unconditional DELETE on release — worst case is one extra API call
Soft/hard TTL	2 min soft, 10 min hard	Soft triggers revalidation. Hard is a Redis eviction safety net
Cache format	Pydantic model dicts via orjson	Processed models cached, not raw GraphQL. Saves CPU across pods
Redis cluster	Same shared cluster as other providers	~36 keys, ~4MB total. Not worth a separate instance
Failure mode	All Redis errors fall through to API	Redis is an optimization, never a requirement. No new failure modes
Rollout	Global config flag, off by default	Enable in staging first, then production

Follow-up

• Add StatsD metrics (hit/miss/stale/error) before enabling in production
• Enable in staging, validate API call reduction
• Enable in production

PR Review Checklist

Put an x in the boxes that apply

• This PR conforms to the Contribution Guidelines
• The PR title starts with the JIRA issue reference, format example [DISCO-####], and has the same title (if applicable)
• [load test: (abort|skip|warn)] keywords are applied to the last commit message (if applicable)
• Documentation has been updated (if applicable)
• Functional and performance test coverage has been expanded and maintained (if applicable)

┆Issue is synchronized with this Jira Task

[mmiermans]

Returning a 201 code in the scenario where we do not have any (stale) data would improve reliability, by avoiding connections piling up.

TODO: Check how Firefox Desktop & Mobile would handle 201 status code? Does it retry?

[mmiermans]

Redis L2 behind existing in-memory L1

Q: Are we consistent, or does weather and other providers hit Redis on every request? Are the use-cases different?

Q: What's the existing pattern for testing with Redis?

Q: What are the existing Redis patterns?

[mmiermans]

We probably want this to be high: more like 1-24 hours? Days? To match our response time.

[mmiermans]

Again, check if this is a new pattern, and whether it should be one.

[mmiermans]

Suggested change

	hard_ttl_sec = 600
	hard_ttl_sec = 3600

Long hard ttl?

[mmiermans]

TODO: Check if there's a client network timeout of 30s?

[mmiermans]

Is there a good reason to do this differently from the usual config validation?

[mmiermans]

Should backend_type and surface_id be an enum? I believe we already have an enum for the latter.

[mmiermans]

Use named arguments.

[mmiermans]

Let's verify whether we still need this? Do we need to fetch yesterday's data for the schedule?

[mmiermans]

Ideally extra would be structured data.

[mmiermans]

Let's check if there's a pattern to create an envelope or have a separate expiration key?

[mmiermans]

Return 201 if this happens?

[mmiermans]

Why is this the first time we need to do a shutdown in the curated-recommendations provider?