feat(sentinel): report the critical level whether or not the restart is enabled

[thiagoesteves]

What changed and why

Follows #292, now merged, and targets main.

A resource that crossed the restart threshold with enable_restart: false said nothing at all. The restart clause did not match, the warning clause had already fired on the way past the warning threshold, and the check fell through to its catch-all. The only signal an operator ever got was the warning at 75%, and that is exactly the shape the installer templates hand out for the DeployEx atom, process and port limits, where enable_restart is off by default. enable_restart decides what DeployEx may do about a resource, not whether the resource is worth reporting.

Sentinel.Watchdog now tracks each resource through the level its usage falls into, :ok, :warning or :critical, which is the model Components.Monitoring already uses to colour the card. That replaces the warning_log_flag boolean and collapses three near-identical clause families into one, a net 105 lines lighter.

• Reaching the restart threshold is logged and notified whether or not a restart follows.
• Reporting follows the changes of level, so a resource parked above a threshold no longer repeats its log line and its notification on every check, once per second.
• A usage that jumps past both thresholds between two checks is reported as critical instead of being announced as a warning.
• Restarting still follows the level itself: while the host memory stays critical each check restarts the current heaviest application, the cascade the existing test pins. Terminating DeployEx is the exception, the node is already on its way down and a repeat would only re-announce the same shutdown.

Behaviour changes to be aware of

watchdog_threshold_exceeded carries an action of :restart or :no_restart. Anything treating the event as proof that something was restarted needs to read it. Slack and PagerDuty say which one happened, and the PagerDuty severity stays critical either way, the resource is at its limit regardless of what DeployEx is allowed to do about it.

The memory event now carries the DeployEx node, the node the measurement is about, instead of the application that happened to be restarted. Previously the payload read type: :memory next to a monitored application's node, which looks like that application's memory rather than the host's. The restarted application is named in the log line.

The comparisons are now inclusive, >= for both thresholds. The UI card already read Critical at exactly the restart threshold while the watchdog stayed quiet. The log lines print >= and < accordingly, so log greps built on the old > and <= wording need updating.

The DeployEx host memory is reported at critical even when there is no monitored application to restart for it. The nil clause used to come first and suppressed the whole check, warnings included.

Two unrelated fixes, each in its own commit

Both came out of chasing intermittent suite failures and are kept separate so they can be dropped on their own.

Foundation.CatalogTest ran async: true while its setup wipes var_path, the tree Foundation.CatalogExDocTest walks in the setup_all_apps/0 doctest. When the two overlapped, mkdir_p failed with :enotdir, about one full-suite run in seven. A module that destroys state the whole suite shares cannot be async.

Host.Terminal.Server opens its connection in a continue, after Terminal.new/1 has returned, so the terminal page renders while the LiveView still has no process to write to. A key pressed inside that window reached Commander.send/2 with a nil process and took the terminal down. Those keys are dropped now, and the test waits for the connection through a fixture helper instead of racing the server. That one failed about one run in twelve.

Risk assessment

Impact: operators get a log line and a notification when a resource reaches its restart threshold with the restart disabled, which is the default the templates ship for the three DeployEx Beam VM limits. Repeated notifications for a resource that stays above a threshold stop. The payload of watchdog_threshold_exceeded gains action, and its node changes for the memory resource. A keystroke typed before the terminal connection is open no longer crashes the page.

Blast radius: the threshold evaluation in Sentinel.Watchdog, the formatting of watchdog_threshold_exceeded in the Slack and PagerDuty adapters, and the key handler of the terminal LiveView. Metric collection, the ETS layout, the restart mechanics, the YAML parsing and every other event are untouched.

Regression risk: low for the actions, which keep their existing behaviour including the memory cascade, and moderate for anything parsing the log lines, since the comparisons they show changed. The tests cover every level transition, the critical level with the restart disabled, and the absence of a repeat while a level lasts.

Rollback: plain commit revert, per commit. No data or config migration, the level lives only in the runtime ETS table.

Verification

Compile --warnings-as-errors, mix format --check-formatted, mix credo --strict and mix dialyzer pass. Around 90 suite runs went into the two flake fixes: foundation and web went from one failure in twelve to zero in fifteen, the terminal tests zero in twenty-five, sentinel zero in twenty, deployer zero in ten, and the CI invocation mix test --cover --warnings-as-errors zero in eight.

Follow-up

The CHANGELOG is not touched here. The action field, the node change on the memory event and the new log wording belong under Backwards incompatible changes for 0.9.13.

ObserverWeb.Telemetry.Storage was seen crashing once with System.os_time/1 is undefined, meck reloading System for a with_mocks block while the observer_web poller calls into it. It crashes a GenServer without failing a test and is left alone here, the fix is to give Foundation.Yaml an injectable config path so the tests stop mocking System globally.

🤖 Generated with Claude Code