Commit 77db6fb
fix(sandbox): clear gateway marker when the start supervisor exits (#5569)
## Summary
Fixes #5568.
On OpenShell docker-driver sandboxes, `scripts/nemoclaw-start.sh` is not
PID 1 — OpenShell's `sleep infinity` keeps the container alive as a
sibling. The Docker HEALTHCHECK's pidfile fallback (#4952) trusts
`/tmp/nemoclaw-gateway.pid`, which is refreshed only by
`record_gateway_pid` inside this script's launch and respawn paths. When
the supervise loop exits, the container lives on but nothing refreshes
the pidfile. The in-container gateway marker stays in place, so the
healthcheck keeps trusting a now-stale PID forever and reports a working
sandbox as permanently `unhealthy`.
## Change
- Add `clear_in_container_gateway_marker`, symmetric to
`mark_in_container_gateway`.
- Arm it via `trap ... EXIT` right after the existing `SIGTERM`/`SIGINT`
trap, in both the non-root and root supervise paths.
The EXIT trap covers every way the supervisor leaves: a clean gateway
exit, a forwarded signal (`cleanup_on_signal` ends in `exit`), and
errexit. This keeps the fix inside `nemoclaw-start.sh` and out of the
shared `cleanup_on_signal` in `sandbox-init.sh`, which
`agents/hermes/start.sh` also sources. The marker is re-dropped at each
(re)launch by `mark_in_container_gateway`, so the respawn loop — which
never exits the script — keeps it in place; only a supervisor *exit*
clears it.
The marker semantics become: "a supervisor is actively managing the
gateway and keeping the pidfile fresh." Once the marker is gone, the
healthcheck takes the marker-absent → healthy branch (#4503) instead of
trusting a stale PID.
## Tests
Extends `test/nemoclaw-start-gateway-marker.test.ts`, reusing its
existing `extractShellFunctionFromSource` / `safeTmpHelpers` harness so
each case runs the real shell functions in isolation against a temp
marker path:
- `clear_in_container_gateway_marker` removes the marker and is a no-op
when it is absent.
- The marker is dropped when the supervisor reaches a clean exit (real
`trap ... EXIT` wiring, not a direct helper call).
- The marker is dropped when the supervisor is terminated by a signal,
exercising the `cleanup_on_signal` → `exit` → EXIT-trap path.
- The marker survives respawns while the supervisor keeps running, and
is cleared only once the script finally exits.
`shfmt`, `shellcheck`, Biome, and `typecheck:cli` pass; the new and
existing marker tests pass (14/14). Happy to revisit the choice of an
EXIT trap over explicit cleanup at each exit site if maintainers prefer
a different shape.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved gateway supervisor shutdown to reliably remove the
in-container gateway marker on normal exit, SIGTERM/SIGINT termination,
early launch failures, and `errexit`/`EXIT` flows (including non-PID 1
scenarios).
* Reduces risk of stale marker files misleading container health checks.
* **Tests**
* Added shell-based behavioral coverage for marker cleanup on both
launch paths, SIGTERM teardown, and marker persistence across respawn
iterations until final exit.
* **Documentation**
* Clarified the marker lifecycle in the deployment/architecture
reference.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
---------
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
Co-authored-by: Charan Jagwani <cjagwani@nvidia.com>1 parent 89e19f0 commit 77db6fb
4 files changed
Lines changed: 446 additions & 67 deletions
File tree
- docs/reference
- scripts
- test
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
87 | 87 | | |
88 | 88 | | |
89 | 89 | | |
90 | | - | |
| 90 | + | |
| 91 | + | |
91 | 92 | | |
92 | 93 | | |
93 | 94 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
350 | 350 | | |
351 | 351 | | |
352 | 352 | | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
353 | 369 | | |
354 | 370 | | |
355 | 371 | | |
| |||
4049 | 4065 | | |
4050 | 4066 | | |
4051 | 4067 | | |
4052 | | - | |
| 4068 | + | |
| 4069 | + | |
| 4070 | + | |
4053 | 4071 | | |
4054 | 4072 | | |
4055 | 4073 | | |
| |||
4872 | 4890 | | |
4873 | 4891 | | |
4874 | 4892 | | |
| 4893 | + | |
| 4894 | + | |
| 4895 | + | |
| 4896 | + | |
| 4897 | + | |
| 4898 | + | |
| 4899 | + | |
4875 | 4900 | | |
| 4901 | + | |
| 4902 | + | |
| 4903 | + | |
| 4904 | + | |
| 4905 | + | |
| 4906 | + | |
| 4907 | + | |
| 4908 | + | |
| 4909 | + | |
| 4910 | + | |
4876 | 4911 | | |
4877 | 4912 | | |
4878 | 4913 | | |
| |||
4894 | 4929 | | |
4895 | 4930 | | |
4896 | 4931 | | |
| 4932 | + | |
| 4933 | + | |
| 4934 | + | |
| 4935 | + | |
| 4936 | + | |
| 4937 | + | |
| 4938 | + | |
| 4939 | + | |
| 4940 | + | |
| 4941 | + | |
4897 | 4942 | | |
4898 | 4943 | | |
4899 | 4944 | | |
| |||
5492 | 5537 | | |
5493 | 5538 | | |
5494 | 5539 | | |
5495 | | - | |
5496 | | - | |
5497 | | - | |
5498 | | - | |
5499 | | - | |
5500 | | - | |
| 5540 | + | |
5501 | 5541 | | |
5502 | 5542 | | |
5503 | 5543 | | |
| |||
5515 | 5555 | | |
5516 | 5556 | | |
5517 | 5557 | | |
5518 | | - | |
5519 | 5558 | | |
5520 | 5559 | | |
5521 | 5560 | | |
| |||
5720 | 5759 | | |
5721 | 5760 | | |
5722 | 5761 | | |
| 5762 | + | |
5723 | 5763 | | |
5724 | 5764 | | |
5725 | 5765 | | |
| |||
5769 | 5809 | | |
5770 | 5810 | | |
5771 | 5811 | | |
5772 | | - | |
5773 | 5812 | | |
5774 | 5813 | | |
5775 | 5814 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
50 | 54 | | |
51 | 55 | | |
52 | 56 | | |
| |||
101 | 105 | | |
102 | 106 | | |
103 | 107 | | |
| 108 | + | |
104 | 109 | | |
105 | 110 | | |
106 | 111 | | |
| |||
111 | 116 | | |
112 | 117 | | |
113 | 118 | | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
114 | 131 | | |
115 | 132 | | |
116 | 133 | | |
| |||
994 | 1011 | | |
995 | 1012 | | |
996 | 1013 | | |
997 | | - | |
998 | | - | |
999 | | - | |
1000 | | - | |
1001 | | - | |
1002 | | - | |
| 1014 | + | |
| 1015 | + | |
1003 | 1016 | | |
1004 | 1017 | | |
1005 | 1018 | | |
| |||
1021 | 1034 | | |
1022 | 1035 | | |
1023 | 1036 | | |
| 1037 | + | |
1024 | 1038 | | |
1025 | 1039 | | |
1026 | 1040 | | |
1027 | 1041 | | |
| 1042 | + | |
1028 | 1043 | | |
1029 | 1044 | | |
1030 | 1045 | | |
| |||
1038 | 1053 | | |
1039 | 1054 | | |
1040 | 1055 | | |
1041 | | - | |
1042 | | - | |
1043 | | - | |
1044 | | - | |
1045 | | - | |
1046 | | - | |
1047 | | - | |
1048 | | - | |
1049 | | - | |
1050 | | - | |
1051 | | - | |
1052 | | - | |
1053 | | - | |
1054 | | - | |
1055 | 1056 | | |
1056 | 1057 | | |
1057 | 1058 | | |
| |||
1074 | 1075 | | |
1075 | 1076 | | |
1076 | 1077 | | |
1077 | | - | |
1078 | | - | |
1079 | | - | |
| 1078 | + | |
| 1079 | + | |
| 1080 | + | |
| 1081 | + | |
| 1082 | + | |
1080 | 1083 | | |
1081 | 1084 | | |
1082 | 1085 | | |
| |||
1106 | 1109 | | |
1107 | 1110 | | |
1108 | 1111 | | |
1109 | | - | |
| 1112 | + | |
| 1113 | + | |
1110 | 1114 | | |
1111 | 1115 | | |
1112 | 1116 | | |
| |||
1124 | 1128 | | |
1125 | 1129 | | |
1126 | 1130 | | |
| 1131 | + | |
1127 | 1132 | | |
1128 | 1133 | | |
1129 | | - | |
| 1134 | + | |
| 1135 | + | |
| 1136 | + | |
| 1137 | + | |
| 1138 | + | |
| 1139 | + | |
| 1140 | + | |
| 1141 | + | |
| 1142 | + | |
| 1143 | + | |
1130 | 1144 | | |
1131 | 1145 | | |
1132 | 1146 | | |
1133 | 1147 | | |
1134 | 1148 | | |
1135 | | - | |
| 1149 | + | |
1136 | 1150 | | |
1137 | 1151 | | |
1138 | | - | |
1139 | | - | |
1140 | | - | |
| 1152 | + | |
| 1153 | + | |
| 1154 | + | |
| 1155 | + | |
1141 | 1156 | | |
1142 | 1157 | | |
1143 | 1158 | | |
| |||
1356 | 1371 | | |
1357 | 1372 | | |
1358 | 1373 | | |
1359 | | - | |
1360 | | - | |
1361 | | - | |
| 1374 | + | |
| 1375 | + | |
1362 | 1376 | | |
1363 | 1377 | | |
1364 | 1378 | | |
1365 | | - | |
1366 | | - | |
1367 | | - | |
1368 | | - | |
1369 | | - | |
1370 | | - | |
1371 | | - | |
1372 | | - | |
1373 | | - | |
1374 | | - | |
1375 | | - | |
1376 | | - | |
1377 | | - | |
1378 | | - | |
1379 | | - | |
1380 | 1379 | | |
1381 | 1380 | | |
1382 | 1381 | | |
| |||
1417 | 1416 | | |
1418 | 1417 | | |
1419 | 1418 | | |
1420 | | - | |
1421 | | - | |
1422 | | - | |
1423 | | - | |
1424 | | - | |
1425 | | - | |
1426 | | - | |
1427 | | - | |
| 1419 | + | |
| 1420 | + | |
| 1421 | + | |
| 1422 | + | |
1428 | 1423 | | |
1429 | 1424 | | |
| 1425 | + | |
| 1426 | + | |
| 1427 | + | |
| 1428 | + | |
1430 | 1429 | | |
1431 | 1430 | | |
1432 | 1431 | | |
1433 | | - | |
| 1432 | + | |
1434 | 1433 | | |
1435 | 1434 | | |
1436 | 1435 | | |
| |||
1457 | 1456 | | |
1458 | 1457 | | |
1459 | 1458 | | |
1460 | | - | |
| 1459 | + | |
1461 | 1460 | | |
1462 | 1461 | | |
1463 | 1462 | | |
| |||
0 commit comments