Skip to content

Health reports critical from heapUsed/heapTotal, so /agentmemory/health returns 503 on a healthy process #1223

Description

@inix-x

evaluateHealth measures memory severity against heapTotal, which is the heap V8 has already committed, not the ceiling it is allowed to grow to. V8 sizes heapTotal to demand, so a busy healthy process sits near 100% of it more or less permanently.

src/health/thresholds.ts:62-65

const memPercent =
  snapshot.memory.heapTotal > 0
    ? (snapshot.memory.heapUsed / snapshot.memory.heapTotal) * 100
    : 0;

Since heapTotal <= heap_size_limit always holds, this can only ever over-report severity, never under-report it.

What I saw

On a live deployment:

bytes
heapUsed 390,771,624
heapTotal 404,930,560
rss 584,327,168
v8.getHeapStatistics().heap_size_limit 6,492,782,592 (6192 MB)

That is 6.0% of the heap the process may actually use. The alert read memory_critical_97%_rss557mb and status was critical.

GET /agentmemory/health then returned HTTP 503, because src/triggers/api.ts:274 maps critical to 503:

const statusCode = status === "critical" ? 503 : 200;

/agentmemory/livez returned 200 at the same moment. So an uptime monitor pointed at /health reads the service as down while it is serving normally. At that point the container was using 3.0 GB of an 8 GB limit, the circuit breaker was closed, CPU was 3.4%, and event loop lag was 0.1 ms.

Worth noting for anyone deploying from this repo: deploy/railway/railway.json sets healthcheckPath to /agentmemory/livez, so Railway's own check keeps passing and the service never restart-loops. The shipped template dodges this by accident. It is the richer /agentmemory/health endpoint, the one you would point an external monitor at, that reports the false 503.

What I expected

healthy, and HTTP 200. Six percent of the heap limit is not memory pressure.

How this relates to #158

#158 reported the same formula misfiring at heapUsed ~45 MB, heapTotal ~46 MB, and rss ~120 MB. The fix that landed added memoryRssFloorBytes (512 MB, src/health/thresholds.ts:20) so the alert stays quiet on small processes. That covers the case as reported, but the ratio itself was never corrected, so the bug comes back for any process whose RSS clears the floor.

There is also no way to tune around it from the outside. src/health/monitor.ts:87 calls evaluateHealth(snapshot) with no config argument, so DEFAULTS always wins and neither the percentages nor the RSS floor are reachable from the environment.

Repro

  1. Run agentmemory under a workload that keeps RSS above 512 MB.
  2. curl -si "$AGENTMEMORY_URL/agentmemory/health" | head -1
  3. You get 503, with memory_critical_NN% in health.alerts, while the process is using a small fraction of v8.getHeapStatistics().heap_size_limit.

The viewer shows the same thing. src/viewer/index.html:1582 recomputes the identical heapUsed / heapTotal ratio client-side, so the HEAP gauge renders red on a healthy process.

Environment

  • agentmemory 0.9.28 (latest published: 0.9.29)
  • Node v22.23.2
  • Platform: Railway, self-hosted from this repo's own deploy/railway template
  • Linux container, 8 GB memory limit, single replica
  • Reproduced against the deployed service, not a local dev server

I have a fix with tests and can open a PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions