Pangolin compose doesn't start in working condition on weak machine - fix with delayed start

[KusAdama]

I have unattended udates, which frequently require system restart. Over several months, I observed that all my Pangolin instances (running on 1c/1G RAM VPS) became after each reboot non-functional. No resource nor the dashboard working.

Can't connect at all:

Secure Connection Failed
An error occurred during a connection to my.domain.net. PR_CONNECT_RESET_ERROR
Error code: PR_CONNECT_RESET_ERROR

The stack remaining running, can SSH in, restarting the Pagolin containers one by one doesn't help.
Only entire stack down & up again restores functionality.

Figured out it is possibly because the stack is heavy for weak machine and somehow it mangled itself if systemd initializes compose immediately after machine boot.

(Have it with CrowdSec and there is the ufw-docker - but it is of no difference if ufw-docker not used.)

Didn't add this as Issue as it's solved by config for OS rather to changing anything with Pangolin.

The root cause is unknown to me.

Solution for me is to delay start of the compose by some time and let the OS handle what is necessary; start the compose later.

How:

1. Change in docker compose the unless-stopped to > no

2. create file: sudo vim /etc/systemd/system/docker-pangolin-delay.service

add in:

[Unit]
Description=Pangolin Docker Compose Stack Delay
# Ensure Docker and Network are ready first
Requires=docker.service
After=docker.service network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/home/_change_here_/pangolin
# Wait 60 seconds before starting
ExecStartPre=/bin/sleep 60
# Start the stack
ExecStart=/usr/bin/docker compose up -d
# Stop the stack on shutdown
ExecStop=/usr/bin/docker compose down

[Install]
WantedBy=multi-user.target

3. reload the systemctl so the file is recognized and enable the new service

sudo systemctl daemon-reload

sudo systemctl enable my-stack.service

4. reboot the VPS and verify it does work as intended

reboot

docker ps - only non-delayed will be started already (if anything is beside the Pangolin), Pangolin containers show itself after the delay time

Can watch the spike in CPU/RAM usage with htop when the start hit.

Hope it helps.

[KusAdama]

This log appears relevant to the problem:

gerbil | 03:00:02 Stopping SNI Proxy...
gerbil | 03:00:02 All connections closed gracefully
gerbil | 03:00:02 SNI Proxy stopped
gerbil | panic: close of closed channel
gerbil |
gerbil | goroutine 1 [running]:
gerbil | github.com/fosrl/gerbil/relay.(*UDPProxyServer).Stop(0xc0000f52c0)
gerbil | /app/relay/relay.go:237 +0x99
gerbil | main.main()
gerbil | /app/main.go:420 +0x217a
gerbil | INFO: 03:00:02 Context cancelled, shutting down
gerbil | INFO: 03:00:33 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
gerbil | ERROR: 03:00:33 Error fetching remote config http://pangolin:3001/api/v1/gerbil/get-config: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp 172.18.0.4:3001: connect: connection refused
gerbil | ERROR: 03:00:33 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp 172.18.0.4:3001: connect: connection refused
gerbil | INFO: 03:00:38 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config

6x the error above, deleted to shorter log

gerbil | ERROR: 03:00:58 Failed to load configuration: Post "http://pangolin:3001/api/v1/gerbil/get-config": dial tcp 172.18.0.4:3001: connect: connection refused
gerbil | INFO: 03:01:03 Fetching remote config from http://pangolin:3001/api/v1/gerbil/get-config
gerbil | INFO: 03:01:06 Created WireGuard interface wg0
gerbil | INFO: 03:01:06 Assigned IP address 100.89.128.1/24 to interface wg0
gerbil | INFO: 03:01:06 Attempting to delete existing MSS clamping rule for chain INPUT
gerbil | INFO: 03:01:06 Attempting to delete existing MSS clamping rule for chain OUTPUT
gerbil | INFO: 03:01:06 Attempting to delete existing MSS clamping rule for chain FORWARD
gerbil | INFO: 03:01:06 Adding MSS clamping rule for chain INPUT
gerbil | INFO: 03:01:06 Successfully added and verified MSS clamping rule for chain INPUT
gerbil | INFO: 03:01:06 Adding MSS clamping rule for chain OUTPUT
gerbil | INFO: 03:01:06 Successfully added and verified MSS clamping rule for chain OUTPUT
gerbil | INFO: 03:01:06 Adding MSS clamping rule for chain FORWARD
gerbil | INFO: 03:01:06 Successfully added and verified MSS clamping rule for chain FORWARD
gerbil | INFO: 03:01:06 WireGuard interface wg0 created and configured
gerbil | INFO: 03:01:06 Peer yJxm5sGtP5IDeNZN/HhsaFCZPSCHvFis9j7MJutwmhM= added successfully
gerbil | INFO: 03:01:06 Received initial mappings: {"mappings":{}}
gerbil | INFO: 03:01:06 Loaded 0 initial proxy mappings
gerbil | INFO: 03:01:06 UDP server listening on :21820
gerbil | INFO: 03:01:06 Starting HTTP server on :3004

==

pangolin | [Error: Failed to find Server Action "95a72bb7". This request might be from an older or newer deployment.
pangolin | Read more: https://nextjs.org/docs/messages/failed-to-find-server-action]
pangolin | Making login request to: http://localhost:3000/api/v1/auth/login
pangolin | Making login request to: http://localhost:3000/api/v1/auth/login
pangolin | npm error path /app
pangolin | npm error command failed
pangolin | npm error signal SIGTERM
pangolin | npm error command sh -c ENVIRONMENT=prod node dist/migrations.mjs && ENVIRONMENT=prod NODE_ENV=development node --enable-source-maps dist/server.mjs
pangolin | npm notice
pangolin | npm notice New minor version of npm available! 11.9.0 -> 11.12.1
pangolin | npm notice Changelog: https://github.com/npm/cli/releases/tag/v11.12.1
pangolin | npm notice To update run: npm install -g npm@11.12.1
pangolin | npm notice
pangolin | npm error A complete log of this run can be found in: /root/.npm/_logs/_00_29_902Z-debug-0.log
pangolin |
pangolin | > @fosrl/pangolin@0.0.0 start
pangolin | > ENVIRONMENT=prod node dist/migrations.mjs && ENVIRONMENT=prod NODE_ENV=development node --enable-source-maps dist/server.mjs
pangolin |
pangolin | Starting migrations from version 1.16.0
pangolin | Migrations to run:
pangolin | All migrations completed successfully
pangolin | T03:00:56+00:00 [warn]: Email SMTP configuration is missing. Emails will not be sent.
pangolin | T03:00:59+00:00 [info]: Analytics usage statistics collection is disabled. If you enable this, you can help us make Pangolin better for everyone. Learn more at: https://docs.pangolin.net/telemetry
pangolin | T03:01:01+00:00 [info]: API server is running on http://localhost:3000
pangolin | T03:01:01+00:00 [info]: Internal server is running on http://localhost:3001
pangolin | T03:01:05+00:00 [info]: Next.js server is running on http://localhost:3002
pangolin | T03:01:06+00:00 [info]: Updated exit node with reachableAt to http://gerbil:3004
pangolin | T08:34:10+00:00 [info]: Updated exit node with reachableAt to http://gerbil:3004