[NemoClaw][Sandbox] HEALTHCHECK stays unhealthy after the in-container gateway is restarted outside the supervisor — #4952 pidfile fallback trusts a stale /tmp/nemoclaw-gateway.pid #5568

TL;DR: On OpenShell docker-driver sandboxes the #4952 HEALTHCHECK fallback reports a fully-working sandbox as (unhealthy) until rebuild, because it trusts a /tmp/nemoclaw-gateway.pid that nothing refreshes once the supervisor exits — and the supervisor can exit, since nemoclaw-start is not PID 1 here. Fix: clear the /tmp/nemoclaw-gateway-local marker on supervisor exit so the healthcheck takes the existing marker-absent → healthy path (#4503) instead of a stale PID. Details below; the table is the quick read.

Root cause

The #4952 HEALTHCHECK fallback (Dockerfile, the HEALTHCHECK CMD) only runs when the in-container curl …/health gets connection-refused (rc=7) and pgrep -f 'openclaw[ -]gateway' misses (the gateway re-execs to a plain openclaw argv). In that case it trusts the recorded PID:

[ -f /tmp/nemoclaw-gateway-local ] || exit 0          # not our gateway → healthy (#4503)
if ! pgrep --ignore-ancestors -f 'openclaw[ -]gateway' >/dev/null 2>&1; then
    gwpid="$(cat /tmp/nemoclaw-gateway.pid 2>/dev/null)"
    case "$(ps -p "$gwpid" -o comm= 2>/dev/null)" in openclaw*) ;; *) exit 1 ;; esac
fi
[ -s /tmp/gateway.log ]

/tmp/nemoclaw-gateway.pid is written only by record_gateway_pid (scripts/nemoclaw-start.sh), at gateway launch and inside the respawn loop. The supervise loop itself is correct — every respawn it controls refreshes the pidfile.

The gap is the loop's own exit assumption. The loop is guarded by a comment stating "This script is PID 1; if it exits, Docker kills all children." On OpenShell docker-driver sandboxes that is not true — PID 1 is /opt/openshell/bin/openshell-sandbox and nemoclaw-start runs as its child:

PID  PPID  COMMAND
  1     0  /opt/openshell/bin/openshell-sandbox    ← real PID 1
 35     1  sleep infinity                          ← OPENSHELL_SANDBOX_COMMAND, keeps the container alive
 36     1  bash /usr/local/bin/nemoclaw-start      ← supervisor (NOT PID 1)
467    36  openclaw                                ← gateway (re-execed argv)

Two things follow from this layout, both observed on a live container (not inferred):

1. Container liveness is decoupled from the supervisor. OpenShell launches the container with OPENSHELL_SANDBOX_COMMAND=sleep infinity (PID 35), a sibling of nemoclaw-start directly under PID 1. The container stays up as long as sleep infinity + openshell-sandbox live — independent of nemoclaw-start. So nemoclaw-start (PID 36) can exit and the container keeps running, gateway-less, indefinitely.
2. The loop's exit comment is wrong for this deployment. It assumes "if it exits, Docker kills all children" — true only at PID 1; at PID 36 its exit reaps nothing.

So when the supervise loop exits — a clean gateway exit (wait returns 0 → exit 0) or a forwarded signal — nemoclaw-start terminates but the container keeps running (the sleep infinity sibling holds it open). The /tmp/nemoclaw-gateway-local marker stays in place, and any gateway process started afterward (by recovery, by an external restart, by anything other than this loop) is never recorded. The pidfile is now stale, and because the fallback can't observe the gateway any other way in this namespace (see below), it returns exit 1 on every probe — permanent (unhealthy). This is exactly the state my unhealthy container was found in: only sleep infinity running, no openclaw, the supervisor already gone.

nemoclaw doctor does not help: recoverDockerDriverSandbox no-ops on an already-running container, and recoverNamedGatewayRuntime only touches the host-side OpenShell gateway — neither refreshes the in-container pidfile.

Why the fallback can't just re-discover the gateway

From inside the sandbox netns there is no other signal to fall back to. The gateway binds off-namespace (the #3975/#4503 shape), so the dashboard port isn't visible here:

$ ss -tlnp        # inside the sandbox container
LISTEN  127.0.0.11:36183   ← docker DNS
LISTEN  10.200.0.1:3128    ← openshell-sandbox proxy (pid 1)
# no 18789 / 18790 — the gateway's listeners live in another namespace

curl can't reach it, pgrep can't match it (re-execed name), and the listener isn't in this netns. The pidfile is the only usable signal — which is exactly why it must stay fresh.

Reproduction

Deterministic and non-destructive — replays the exact fallback logic on a live, healthy container, varying only the pidfile, without touching the real one:

C=<sandbox-container>   # state: gateway alive, curl 127.0.0.1:18789 → rc=7, pgrep 'openclaw[ -]gateway' misses
docker exec "$C" sh -c '
  fresh=$(cat /tmp/nemoclaw-gateway.pid)
  for pid in "$fresh" 999999; do
    if case "$(ps -p "$pid" -o comm= 2>/dev/null)" in openclaw*) true;; *) false;; esac && [ -s /tmp/gateway.log ]
    then echo "pidfile=$pid -> HEALTHY"; else echo "pidfile=$pid -> UNHEALTHY"; fi
  done'
# pidfile=467     -> HEALTHY      (fresh: points at the live gateway)
# pidfile=999999  -> UNHEALTHY    (stale: any out-of-supervisor restart reproduces this)

The natural trigger is: the gateway process is replaced while the supervise loop is not the one doing it — possible because the loop has already exited (nemoclaw-start isn't PID 1, so its exit doesn't tear down the container). The found state of my unhealthy container — only sleep infinity, no openclaw, no supervisor — is reachable only after that exit.

Proposed fix

Tie the /tmp/nemoclaw-gateway-local marker to "a supervisor is actively managing the gateway and keeping the pidfile fresh". Clear it on every supervisor exit with a single trap clear_in_container_gateway_marker EXIT, armed right after the existing cleanup_on_signal trap in both the root and non-root launch blocks. One EXIT trap covers every way the supervisor leaves — a clean gateway exit (wait returns 0 → exit 0), a forwarded signal (cleanup_on_signal ends in exit), and errexit — without having to thread cleanup through each individual exit site or touch the shared cleanup_on_signal in sandbox-init.sh. Once the supervisor stops, the marker is gone and the HEALTHCHECK takes the existing marker-absent branch: report healthy and defer to host-side delivery monitoring (#4503), rather than trust a pidfile nobody refreshes.

This completes the #4710 marker semantics (marker = "this container runs a supervised in-container gateway") rather than adding a new mechanism, and it keeps the restart-detection the pidfile fallback was added for while the supervisor is running — the marker is re-dropped at each (re)launch by mark_in_container_gateway, so the respawn loop, which never exits the script, keeps it in place. Coverable with tests alongside the existing marker cases in test/nemoclaw-start-gateway-marker.test.ts.

Why fix the marker rather than the exit? The respawn loop's exit 0 on wait-returns-0 is intentional (#2757): it exists so a graceful gateway shutdown lets the supervisor exit cleanly instead of relaunching the gateway during docker stop / nemoclaw destroy. It must stay. The deeper issue is that, on docker-driver deployments, the supervisor has no local signal to tell a shutdown rc=0 apart from a self-restart/handoff rc=0 — both arrive as "wait returned 0, no signal seen". I confirmed this: signalling the supervisor takes the cleanup_on_signal trap path, but a SIGTERM delivered only to the gateway (or a gateway that exits 0 on its own) both fall through to the respawn loop's exit 0 with no shutdown flag set, indistinguishable from each other. Truly disambiguating them needs the orchestrator to forward the stop signal to the supervisor (an OpenShell-side concern, out of scope here). Clearing the marker on exit is the in-repo fix that holds for every exit cause: it converts a permanent false-unhealthy into, at worst, a brief honest gap until a new supervisor takes over — never a stale-PID false reading.

Open question (does not affect the fix)

Everything above was observed on a live container, not inferred. The one thing I couldn't capture before the container was rebuilt is which supervisor-exit path fired (clean exit 0 vs. signal). Per the harness in the previous section, that doesn't matter: the two rc=0 causes are indistinguishable to the script, so the fix clears the marker on every exit and holds regardless.

Not a duplicate of

Three nearby issues share a fact or a line with this one but describe different defects:

• #4951 (open) — nemoclaw tunnel stop mis-detects the gateway because the process rewrites its argv to a bare openclaw. Same underlying fact (argv rewrite defeats ps/pgrep matching), but #4951 is a stop-command false-negative in src/lib/tunnel/services.ts (reports "not running" while the gateway runs). This issue is a HEALTHCHECK false-negative in nemoclaw-start.sh (reports unhealthy while the gateway runs). Different surface, different file, different fix; they happen to be downstream of the same argv behavior.
• #2042 (closed) — K8s pod reconcile leaves the gateway and port-forward genuinely dead; recovery is a side-effect of connect. It corroborates the structural fact I rely on — "the new pod has sleep infinity as its command and does not re-run nemoclaw-start" — but there the service is actually down (connection refused is correct). Here the gateway is fully alive and only the healthcheck is wrong.
• #1015 (closed) — added the trap/signal handling to nemoclaw-start.sh, and questioned the very same "This script is PID 1; if it exits, Docker kills all children" comment. It fixed signal forwarding / graceful teardown; it did not touch pidfile freshness or the healthcheck. This issue is the freshness consequence of that same wrong PID-1 assumption.

Environment

• NemoClaw v0.0.65-58-gb025230fb (tracking main @ b025230fb)
• OpenShell docker-driver sandbox · OpenClaw gateway re-execs to plain openclaw argv
• Docker 29.5.3 · Node.js v22.22.3
• macOS 26.5.1 (arm64)

Checklist

• Reproducible (deterministic fallback replay above)
• Searched existing issues — not a duplicate. Closest prior art is the #3975/#4503/#4710/#4952 HEALTHCHECK chain (this is the remaining pidfile-freshness gap in #4952); nearest open/closed neighbors #4951/#2042/#1015 are distinguished above.