Can't see real client IPs passed with PROXY protocol v2

[bam80]

mtg/contrib/sni-router/README.md

Lines 49 to 50 in 8d143d7

	v2 header so both backends see the real client IP instead of HAProxy's
	container address. The three pieces must stay in sync:

170346b doesn't work for me - still see internal container addresses everywhere in the logs:

haproxy-1  | 172.28.0.1:51416 [06/May/2026:20:45:14.098] tls mtg/mtg 2/0/61067 4887 cD 5/5/4/4/0 0/0
mtg-1      | {"level":"info","dc":203,"client-ip":"172.28.0.1" ...

Note I have #478 applied.

mtg/contrib/sni-router/Caddyfile

Line 20 in 170346b

allow 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16

I also tried to add 172.28.0.0/24 here, didn't help.

@dolonet

[dolonet]

@bam80 strong candidate: Docker's userland-proxy.

With it enabled (older Docker defaults, some distro packages), every connection arriving on a published port (ports: "443:443") is accepted by the docker-proxy userspace daemon and re-opened from the bridge gateway into the container. HAProxy then sees 172.28.0.1 as the source, stamps it into PROXY v2, and mtg/Caddy log it faithfully.

Quick check on the host:

docker info 2>&1 | grep -i "userland proxy"

If Userland Proxy: true, two ways forward:

1. Disable it — add {"userland-proxy": false} to /etc/docker/daemon.json, then sudo systemctl restart docker. Docker switches to iptables DNAT for the port forward and the original source IP is preserved end-to-end. No real downside on a public-facing server.
2. network_mode: host on the haproxy service in docker-compose.yml. HAProxy binds in the host netns directly — no NAT involved. One-line change, costs bridge isolation and conflicts with anything else on the host's :443/:80.

If option 1 doesn't fix it, share docker --version + docker info and I'll keep digging.

Side note on the cD 5/5/4/4/0 flags in the haproxy line — that's a normal client-initiated close mid-stream, not a misroute. The connection itself worked, the source IP is just attributed to the gateway instead of the client.

[dolonet]

Self-correction on the diagnosis sequence above — jumped to a single hypothesis without asking for the diagnostic first. Reordering:

Step 1: confirm what Docker is doing on your host

docker --version
docker info 2>&1 | grep -iE 'userland proxy|server version|cgroup'
sudo iptables -t nat -L DOCKER -n -v 2>&1 | head -40

Step 2: leading causes, in priority order

1. Userland Proxy: true — most common, fix as in the comment above ({"userland-proxy": false} in /etc/docker/daemon.json + systemctl restart docker).
2. MASQUERADE on the compose bridge — some hosts apply SNAT on bridge egress that survives even with userland-proxy: false. Symptom is identical: source IP becomes the bridge gateway. Check the iptables output above for a MASQUERADE line on the 172.28.0.0/24 source. If present, the fix is to remove that rule (or switch haproxy to network_mode: host).
3. Reverse proxy / tunnel in front of the Docker host — Cloudflare tunnel, an Nginx terminating :443 on the host before forwarding into Docker, etc. If anything sits between the public internet and the haproxy container, it'll strip or rewrite source IP unless it sends PROXY protocol itself.

Side note correction: cD in the haproxy log line is "client closed during data transfer" — a normal client-initiated session end, not an abnormal abort. The earlier "mid-stream close" wording was sloppy of me.

Once you share the Step 1 output we can pinpoint which of the three it is.

[bam80]

Thanks for following up,
Usually I use podman, but on docker I see similar behavior with the IP addresses.
I run locally to test, so no Cloudflare here.

$ docker --version
Docker version 29.4.0, build 1.fc43

$ docker info 2>&1 | grep -iE 'userland proxy|server version|cgroup'
 Server Version: 29.4.0
 Cgroup Driver: systemd
 Cgroup Version: 2
  cgroupns

$ sudo iptables -t nat -L DOCKER -n -v 2>&1 | head -40
Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination

Please let me know if I can provide more info, thanks.

PS:
I also don't have /etc/docker/ here on Fedora.

[dolonet]

Empty nat/DOCKER is expected on Docker 29 + Fedora — v29 added a native nftables firewall backend, and the DNAT rules live in nft table docker-bridges instead of the legacy nat/DOCKER chain. Not your problem.

The 172.28.0.1 source on inbound to a published port is the docker-proxy fingerprint: the userspace daemon accepts on the host, opens a fresh connection from the bridge gateway into the container, and the container sees the gateway as source. Standard fix:

sudo mkdir -p /etc/docker
echo '{"userland-proxy": false}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker
docker compose down && docker compose up -d

That forces kernel DNAT and preserves the source IP end-to-end.

If the haproxy log line still shows 172.28.0.1 after that, paste:

docker info
docker compose ps
sudo nft list ruleset | head -200

and we'll go straight into the rules.

[bam80]

Hmm, for some reason I can't connect to the docker's frontend at all if I set '{"userland-proxy": false}' - nothing in the logs.
And apparently podman doesn't have such thing as "userland-proxy", and still - it doesn't show IPs either.

[dolonet]

Quick follow-up with a self-correction.

docker info output. The missing "Userland Proxy" line isn't a signal — Docker 29 replaced that field with "Firewall Backend" (release notes). Default is still userland-proxy: true, so your starting state was the standard one.

The actual ingress story. Your podman observation is the key clue — same wrong source IP without userland-proxy even existing as a concept. The pattern is bridge networking + published port = ingress rewrites source IP, with different mechanisms per runtime:

• Docker default — docker-proxy userspace forwarder; source becomes bridge gateway.
• Docker userland-proxy: false — kernel DNAT on PREROUTING; source preserved if br_netfilter / nf_conntrack / sysctls are healthy. Docker 29 also changed MASQUERADE behavior (moby/moby#48854), so corner cases can drop traffic silently — that's likely your "frontend goes silent" symptom.
• Podman rootless — slirp4netns / pasta does userspace forwarding from a netns gateway; source rewrite is a property of the rootless network stack, no daemon flag fixes it.
• Podman rootful — kernel DNAT, source preserved.

PROXY v2 from HAProxy to the backends faithfully carries whatever source IP HAProxy itself observed, and in bridge mode that's the gateway. The fix can't live on the haproxy→backend leg — it has to lift HAProxy out of the rewrite path.

Universal fix: network_mode: host for HAProxy. It binds in the host netns directly, sees real source IPs, and stamps them into the PROXY v2 header. mtg and Caddy stay on the compose bridge; HAProxy reaches them via the host loopback.

Patch (untested in your specific environment, but should work for both Podman rootless and Docker on Fedora):

# docker-compose.yml
   haproxy:
     image: haproxy:lts-alpine
-    ports:
-      - "443:443"
-      - "80:80"
+    network_mode: host                 # binds host:443 / host:80, sees real IPs
     volumes:
       - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
   mtg:
     image: nineseconds/mtg:2
     volumes:
       - ./mtg-config.toml:/config/config.toml:ro
-    expose:
-      - "3128"
+    ports:
+      - "127.0.0.1:3128:3128"          # host-mode haproxy reaches via loopback
   web:
     image: caddy:alpine
     volumes:
       ...
-    expose:
-      - "80"
-      - "8443"
+    ports:
+      - "127.0.0.1:8080:80"
+      - "127.0.0.1:8443:8443"

# haproxy.cfg
 backend mtg
-    server mtg mtg:3128 send-proxy-v2
+    server mtg 127.0.0.1:3128 send-proxy-v2

 backend web
-    server web web:8443 send-proxy-v2
+    server web 127.0.0.1:8443 send-proxy-v2

 backend web_acme
     mode http
-    server web web:80
+    server web 127.0.0.1:8080

# Caddyfile
                 proxy_protocol {
                         timeout 5s
-                        allow 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16
+                        allow 127.0.0.0/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16
                 }

Notes:

• 127.0.0.0/8 in Caddy's allow covers the case where Caddy sees the connection arriving from 127.0.0.1 (Docker userland-proxy: false does kernel DNAT and preserves the loopback source). With the default Docker setup or Podman rootless, the source is the bridge gateway and the existing 172.16.0.0/12 already covers it — but adding 127/8 makes the example work in all three modes.
• mtg's PROXY listener (pires/go-proxyproto) accepts headers from any source, so no allow-list change needed there.
• Trade-off of network_mode: host: HAProxy occupies the host's :443 and :80. For a dedicated mtg-router host that's expected; if you run anything else on those ports, this layout isn't for you.

If this works on your Fedora + Docker setup (and on Podman rootless), I'll send it as a follow-up PR to the contrib example, with the README updated to explain the host-mode choice.

[bam80]

Progress:
To make IPv6 work, needed to swap 127... and * wherever feasible to [::1] and [::], respectively.
That's all for tonight.

PS:
IP logging works for sure, thanks!

[bam80]

@dolonet would you make draft PR? I'll make my comments.

[bam80]

If this works on your Fedora + Docker setup (and on Podman rootless), I'll send it as a follow-up PR to the contrib example, with the README updated to explain the host-mode choice.

Friendly reminder - as it works fore me now (with some IPv6-related changes above), the PR is welcome.