feature request: keepalived

[toxic0berliner]

What problem does this solve?

As you have the agent version I suspect I am probably not the only one to run traefik in failover on several instances and having looked long and hard I believe there's nothing foss better than keepalived that I went with.
For me it could make sense to have some stats and maybe actions to manage keepalived.

Proposed solution

I would like to know if you'r interested before going too much in the details but in short:

• get the current status of keepalived
• allow for entering or leaving maintenance mode
• possibly 'ssh' into the other identified nodes or fetch their status somehow
• track it over time?

Currently my getstate.sh prints the keepalived status as follows

cat /var/run/keepalive.*.*.state

kill -USR2 $(cat /var/run/keepalived.pid) && sleep 0.35 && cat /tmp/keepalived.stats && rm /tmp/keepalived.stats

And I found online a guide so I'm probably not the only one to use an empty file maintenance on the traefik folder, when it's present on a node it's keepalived consoders itself as faulty and leaves the other nodes as master. Removing the maintenance file and keepalived comes back alive if traefik is up.

Alternatives you've considered

Keepalived being very minimal the maintenance mode might be implemented differently by other users (i considered curl into traefik on a custom maintenance page that one might also be easily doable here as you already manipulate the API, it was harder for me at the time)

Also tracking it's status over time might be a full endeavor...

And no idea how to get the status of the "cluster" cleanly, I suspect it can only be done cleanly with one agent per node and a page that shows data of all agents.

I considered vibe coding this as a one-time use app for my own need but finding noone has a gui for keepalived as it's still widely used lets me think there's a slot for a formal solution, your intégration of a mobile app too is amazing so I wanted to ask at least and see how you feel.

Which area does this relate to?

Other

Additional context

Testing it might be a chore...

[chr0nzz]

Interested - I'll put it on the list. TM already has an agent on every node, which is the natural shape for this: each agent reads its own keepalived state, the Host shows the cluster.

Before I commit to a design, please share more details so I can see what you really want day to day:

• Your keepalived.conf (redacted is fine) - mainly whether you use notify scripts, and what writes your /var/run/keepalive.*.state files
• How many nodes, and whether every node could run a TM agent
• Your maintenance flow today: who touches the maintenance file, from where, and what you check before doing it
• What "status" means to you at a glance - just MASTER/BACKUP/FAULT per node, or also priority, VIPs, last transition?
• Would read-only state plus a maintenance toggle cover it, or is there an action you reach for that I'm not seeing?

No promises on timing yet, but it fits the agent model well and I'd take your setup as the reference fixture when it lands.

[toxic0berliner]

• My keepalived.conf:

`keepalived.conf`

global_defs {
    router_id proxy
    enable_script_security
    script_user nobody
    max_auto_priority 99
}

vrrp_script chk_https {
    script "/usr/bin/nc -zv localhost 443"
    interval 1
}

vrrp_script chk_maintenance {
    script "/mnt/local-persist/proxy/check_maintenance.sh"
    interval 1
}

vrrp_instance VRRP1 {
    state MASTER
    interface eth0
    virtual_router_id 70
    priority 99
    advert_int 1
    virtual_ipaddress {
        X.X.X.X/24
    }
    unicast_peer {
        X.X.X.X # MASTER_IP_VALUE
        X.X.X.Y   # BACKUP_IP_VALUE
        X.X.X.Z   # BACKUP_IP_VALUE
        X.X.X.W   # BACKUP_IP_VALUE
    }
    notify /mnt/local-persist/proxy/notify.sh root root
    track_script {
        chk_https
        chk_maintenance
    }
}

Here is chech_maintenance.sh :

#!/bin/bash

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )

echo "Checking for file $SCRIPT_DIR/maintenance"

if [ -f "$SCRIPT_DIR/maintenance" ]; then
    echo "MAINTENANCE IS ENABLED: ON !"
    exit 1
else
    echo "Maintenance is NOT enabled: off !"
    exit 0
fi

And notify.sh

#!/bin/bash
# notify.sh

echo $1 $2 is in $3 state > /var/run/keepalive.$1.$2.state
chmod go+r /var/run/keepalive.$1.$2.state

• I think this answers (and reminds me) what writes the /var/run/keepalive.x.y.state (I totaly forgot I did it this way)
• all 4 nodes are deployed by ansible using the same script, they get "templated" based on their hostname to get different priorities for example.
• I use keepalived for my 4 traefik reverse proxies that are my single entrypoint and forward trafic to other LXCs also running traefik plus a service, every service on my lan has valid https using ACME from my step-ca CA.
• I also use keepalived for my 4 DNS servers (technitium), the same way templated based on hostname, same keepalived but 2 VRRPs one ipv4 one ipv6
• I would have 2 clusters of 4 , and yes all already run traefik and will be running traefik-manager-agent (in fact all my 4 reverse proxyes already run both traefik-manager and traefik-manager-agent, I saw no drawback on having 4 instances of the manager so far)
• currently, I just ssh into one proxy and touch maintenance and rm maintenance whenevery I need to try out things, it's my 'home lab' so nothing else to it. My plan is to have watchtower do this before &after upgrading but I didn't get around getting this to work yet, so the maintenance is not really useful, the VRRP is useful when I reboot one of the host though ;-)
• status to me is as simple as you proposed first : MASTER/BACKUP/FAULT per VRRP
• here is my keepalived.conf for the 2 VRRP with ipv6:

Details

global_defs {
    router_id dns
    enable_script_security
    script_user nobody
    max_auto_priority 99
}

vrrp_sync_groupVI_1_100 {
  group {
    VRRP2
    VRRP2_6
  }
}

vrrp_script chk_https {
    script "/usr/bin/nc -zv localhost 443"
    interval 1
}

vrrp_script chk_dns {
    script "/usr/bin/nc -zv localhost 53"
    interval 1
}

vrrp_script chk_maintenance {
    script "/mnt/local-persist/dns/check_maintenance.sh"
    interval 1
}

vrrp_instance VRRP2 {
    state MASTER
    interface eth0
    virtual_router_id 8
    priority 99
    advert_int 1
    virtual_ipaddress {
        10.X.X.8/24
    }
    unicast_peer {
        10.X.X.8   # VIP_IP_VALUE, this is probably a mistake to have this here
        10.X.X.7   # MASTER_IP_VALUE
        10.X.X.6   # BACKUP_IP_VALUE
        10.X.X.5   # BACKUP_IP_VALUE
        10.X.X.9   # BACKUP_IP_VALUE
    }
    notify /mnt/local-persist/dns/notify.sh root root
    notify_master /mnt/local-persist/dns/notify_master.sh root root
    track_script {
        chk_https
        chk_maintenance
    }
}

vrrp_instance VRRP2_6 {
    state MASTER
    interface eth0
    virtual_router_id 8
    priority 99
    advert_int 1
    virtual_ipaddress {
        2a01:x:x:x::8/64
    }
    unicast_peer {
        2a01:X::8   # MASTER_IP_VALUE
        2a01:X::7   # BACKUP_IP_VALUE
        2a01:X::6   # BACKUP_IP_VALUE
        2a01:X::5   # BACKUP_IP_VALUE
        2a01:X::9   # BACKUP_IP_VALUE
    }
    unicast_src_ip 2a01:X::7
    notify /mnt/local-persist/dns/notify.sh root root
    track_script {
        chk_https
        chk_dns
        chk_maintenance
    }
}

seems I have some more check on notify_master.sh

#set -x

# Configuration
INITIAL_DELAY=3          # Time to wait for Keepalived to assign the VIP
RETRY_TIMEOUT=60         # Maximum time to wait for the container to become healthy
POST_HEALTH_DELAY=5      # Time to wait after container is healthy before re-querying DNS
VIP="10.X.X.8"
DNS_NAME="dns.lan"
DNS_CONTAINER_NAME="dns"

# Wait for initial delay
sleep "$INITIAL_DELAY"

# Function to check if DNS resolves to an IPv4 address using dig
check_dns_ipv4() {
  dig +short @"$VIP" "$DNS_NAME" A | grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}' | grep -q .
}

# First attempt: try DNS query
if check_dns_ipv4; then
  exit 0
fi

# Restart DNS container
docker restart "$DNS_CONTAINER_NAME" >/dev/null

# Wait for it to become healthy (up to RETRY_TIMEOUT)
END=$((SECONDS + RETRY_TIMEOUT))
while [ $SECONDS -lt $END ]; do
  health_status=$(docker inspect --format='{{.State.Health.Status}}' "$DNS_CONTAINER_NAME" 2>/dev/null)
  if [ "$health_status" == "healthy" ]; then
    break
  fi
  sleep 2
done

# If still not healthy, exit with failure
if [ "$health_status" != "healthy" ]; then
  exit 1
fi

# Wait additional delay to let services settle
sleep "$POST_HEALTH_DELAY"

# Try DNS query again
if check_dns_ipv4; then
  exit 0
else
  exit 1
fi

• currently I have no visibility at all on which is the master (except using the VIP and looking at the hostname that is in all http pages title for example) and maintenance is only manual ssh into the LXC, so yeah, viewing who is master, who is backup and probably who is offline would be pretty much all I need with the maintenance toggle the cherry on top.
  Hope this clarifies

[chr0nzz]

Thanks for this, and for the detail in your follow-up. The configs, the notify and maintenance scripts, and the answers about your two clusters gave me enough to research this properly rather than guess, which I did.

I'm going to hold off on it for now, and I'd rather give you the real reasons.

There's no zero-configuration way to read keepalived state. No control socket, no status command, no metrics endpoint. Every mechanism (notify scripts, notify_fifo, SIGUSR1/SIGJSON dumps, DBus, SNMP) requires the user to add something to keepalived.conf. Signalling is out entirely for TM's agent, which has no host PID namespace and so can't see or signal the host's keepalived process at all. I'm not granting it one just for this. That leaves TM defining and owning a file contract, then supporting it across keepalived versions and every deployment shape, generalising from a single reference setup. That's the part I'm not comfortable with.

Stale state is actively dangerous here. A notify-written state file says MASTER forever after keepalived dies or is stopped. An HA dashboard showing a confident green MASTER on a dead node is worse than no dashboard at all, and it's a documented failure in the Prometheus exporter world. Fixing it properly needs a heartbeat, which is more setup on your side again.

Maintenance and FAULT are the same signal. With weight 0, your chk_maintenance script drives the instance to FAULT, which is exactly what you want operationally, but it means a node in maintenance is indistinguishable from a genuinely broken one. Any honest UI has to say "FAULT, possibly intentional", which is weak.

The maintenance toggle is the part that worries me most. It moves production traffic. Traefik Manager has no roles or permissions today, so every logged-in user has full access. Putting a failover control in the UI without an authorization model behind it is a bigger change to TM's security posture than the feature is worth, and doing it properly means building RBAC first.

None of this is a criticism of the request. It's a good idea and your setup is a clean fit for the agent model, which is why I looked seriously at it. The problem is that the honest version costs an owned config contract, a liveness protocol, and an authorization model, and the first time something goes wrong in someone's failover they'll be looking at TM.

One thing that might help you directly: if you're on keepalived 2.0.0 or newer, track_file is the native mechanism for what your chk_maintenance script is doing. It's inotify-driven, so it reacts immediately instead of on the script interval, and it drops the shell out of the path. The vrrp_script sentinel approach you found in that guide predates it.

On prior art: mehdy/keepalived-exporter with Grafana covers the status half today. Roxy-WI is the one established web UI that touches keepalived, though it delegates the actual state reading to an exporter rather than doing it itself. So the gap you're describing is real, it's just narrower than "nothing exists".

I'll leave this open rather than closing it. If keepalived grows a proper read-only status interface, the calculus changes. In the meantime, if you want to link this around and it picks up interest from other people running Traefik behind a VIP, that's a genuine signal and I'll revisit.

[toxic0berliner]

Thanks a lot for the time invested already!
I trust your research more than mine and agree with all your points. Well except one regarding the RBAC it's up to me to put TM behind the proper auth and that should stay true on all nodes part of a VIP, so unless I misunderstood you this seems moot to me.
But yes, for sure all other points are crystal clear and understandable.
I do believe VIP with keepalived is a niche within a niche so I doubt I'll be able to gather much support here so no problem I fully understand this ain't going to be done, for now.

I will indeed look into the enhancements you proposed I believe keepalived was a mystery to myself for a long time and this old config merits a dust-off

Also, thanks a lot for the long and clear explanation, you didn't owe me anything but it's really appreciated 👍

Keep up the good work ❤️