Incomplete uWSGI header with nginx 1.26 and gunicorn 25.2+

[lonvia]

I have a setup with nginx(uwsgi_pass) + gunicorn + falcon(async) running on Debian Trixie:

gunicorn -b unix:/tmp/gunicorn.sock -w 4 --worker-class asgi --protocol uwsgi "nominatim_api.server.falcon.server:run_wsgi()"

with nginx set up as explained in Nominatim's deploy docs.

Starting with gunicorn 25.2.0, I get a 502 back, with gunicorn reporting:

[2026-04-20 09:34:52 +0200] [8420] [DEBUG] uWSGI parse error: Invalid uWSGI header: incomplete header

After installing gunicorn_h1c the report is a bit more verbose:

Fatal error on transport UnixTransport
protocol: <gunicorn.asgi.protocol.ASGIProtocol object at 0x7f9787531fd0>
transport: <UnixTransport closed=False reading=False 0x7f9786f386c0>
Traceback (most recent call last):
  File "uvloop/handles/stream.pyx", line 841, in uvloop.loop.__uv_stream_on_read_impl
  File "uvloop/loop.pyx", line 109, in uvloop.loop.run_in_context1
  File "/tmp/gc/lib/python3.13/site-packages/gunicorn/asgi/protocol.py", line 543, in data_received
    if not self._feed_callback_parser(data):
           ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/tmp/gc/lib/python3.13/site-packages/gunicorn/asgi/protocol.py", line 556, in _feed_callback_parser
    self._callback_parser.feed(data)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
gunicorn_h1c._parser.InvalidRequestMethod: Invalid character '\x00' in HTTP method
[2026-04-20 09:28:05 +0200] [7936] [DEBUG] uWSGI parse error: Invalid uWSGI header: incomplete header

The problem persists on current master, so likely unrelated to #3539, which has been fixed lately afaiu. It works with gunicorn 25.1.0, which indicates nginx' configuration isn't the problem either. Switching to HTTP protocol works as well.

Furthermore I do have the same setup (with gunicorn 25.3.0) running on a Debian Bookworm without issues. The two differences that might be relevant: nginx updated from 1.22.1 to 1.26.3 and Python updated from 3.11 to 3.13.

[benoitc]

hrm it may be also the introduction of the rules enforcing checks :| checking

[thestinger]

This is still an issue with gunicorn 36.0.0, nginx 1.30.2 and python 3.14.5. We're also using Nominatim and wanted to migrate from using uvicorn.workers.UvicornWorker to the gunicorn asgi worker with uWSGI to follow the current Nominatim documentation. Using HTTP instead of uWSGI avoids the issue.

It looks similar to what was fixed by 385a921 but it isn't working for ASGI.

[lonvia]

You can pin gunicorn to 25.1.0 as a workaround. I'll add a warning in the Nominatim docs..

[thestinger]

@lonvia By the way, we noticed the standard epoll load balancing using the workers in the order they added the socket to epoll works very poorly with Nominatim and figured out a good way to work around it. The standard behavior gives events to the workers in a fixed order as long as they're listening for events so it only starts distributing the load once they're getting overloaded. It mixes very poorly with having a maximum PostgreSQL connection pool for each worker since too many requests are being assigned. The workers have enough to do that they do end up being busy and forcing load to be spread out but it's not efficient and is creating poor latency at lower loads and unnecessarily bottlenecking throughput. It makes it harder for it to use all the available resources.

In nginx itself, they work around it by readding the socket to epoll every 16 requests to spread out the load in batches, which isn't ideal but does spread it out a lot more:

nginx/nginx@96c342e

The same thing could be done in gunicorn to spread out the load without needing SO_REUSEPORT which isn't usable with Unix sockets and has major drawbacks itself. Using a TCP socket with SO_REUSEPORT spreads out load quite evenly via hashing but tends to hurt latency because it distributes the load regardless of which workers are ready to accept a connection.

We solved this by moving to a separate Unix socket for each worker via a systemd service template. The load is very evenly distributed across those via nginx least_conn with keepalive connections so they get a relatively even number of keepalive connections and nginx cycles through workers and keepalive connections spreading out the load across those quite evenly. The services are grouped under nominatim.target so it can still be restarted as a whole with systemctl restart nominatim.target.

https://github.com/GrapheneOS/nominatim.grapheneos.org/blob/main/nginx/nginx.conf#L88-L108
https://github.com/GrapheneOS/nominatim.grapheneos.org/blob/main/systemd/system/nominatim%40.service#L13

gunicorn saves memory by using fork including even more when passing --preload so it would be both simpler and more memory efficient if it had a way to make a separate socket for each worker itself.

Our systemd unit for Nominatim also has standard systemd service hardening and stepped auto-restart. It would be nice to have those in the standard one in the docs which is being copied by Linux distributions.

[thestinger]

@lonvia The Nominatim documentation should also probably be updated to pass --no-control-socket since it's currently trying to create a control socket in /srv/nominatim-project which it won't be able to do as www-data in a normal setup so it logs an error. It could also be changed to use a RuntimeDirectory for the socket(s) and put the control socket there which is what we're doing since we have many sockets for per-worker sockets.