Heads up - the local, pbs and slurm jobs are failing on main right now, and on pretty much any PR that pulls fresh deps from PyPI. I ran into it on #941 and dug in a bit; it doesn't look like anything in gateway itself broke, just two dependencies that moved under us. The k8s and kerberos jobs are fine. (@consideRatio, this is the "other CI failure" you mentioned over there.)
The first one hits local. distributed 2026.6.0 removed the old dask-scheduler / dask-worker console scripts – they live under dask scheduler / dask worker now (dask/distributed#9240). Gateway still calls the old names, both as the scheduler_cmd/worker_cmd defaults in backends/base.py and hardcoded as /opt/python/bin/dask-scheduler in the pbs/slurm tests, so cluster startup just dies with:
FileNotFoundError: [Errno 2] No such file or directory: 'dask-scheduler'
The second one hits pbs and slurm, and it's actually why they don't even get to pytest. Pillow 12.3.0 stopped publishing manylinux2014 wheels, and our CI image is still CentOS 7 (glibc 2.17), so pip grabs sdist and tries to build it – but there's no gcc in the container:
error: command 'gcc' failed: No such file or directory
Pillow only shows up here because bokeh drags it in.
The k8s and kerberos jobs dodge all of this: k8s builds from frozen images still pinned to distributed==2026.3.0, and the kerberos test never spins up a real cluster.
Fixing it isn't much: point the scheduler/worker commands at dask scheduler / dask worker, and add pillow to the mamba install -c conda-forge ... line the pbs/slurm _install.sh already uses for contourpy and pandas.
Full logs if useful: https://github.com/dask/dask-gateway/actions/runs/29776604255
I'm happy to send a PR for both – just say the word.
Heads up - the
local,pbsandslurmjobs are failing onmainright now, and on pretty much any PR that pulls fresh deps from PyPI. I ran into it on #941 and dug in a bit; it doesn't look like anything in gateway itself broke, just two dependencies that moved under us. Thek8sandkerberosjobs are fine. (@consideRatio, this is the "other CI failure" you mentioned over there.)The first one hits
local.distributed2026.6.0 removed the olddask-scheduler/dask-workerconsole scripts – they live underdask scheduler/dask workernow (dask/distributed#9240). Gateway still calls the old names, both as thescheduler_cmd/worker_cmddefaults inbackends/base.pyand hardcoded as/opt/python/bin/dask-schedulerin the pbs/slurm tests, so cluster startup just dies with:The second one hits
pbsandslurm, and it's actually why they don't even get to pytest. Pillow 12.3.0 stopped publishingmanylinux2014wheels, and our CI image is still CentOS 7 (glibc 2.17), so pip grabs sdist and tries to build it – but there's no gcc in the container:Pillow only shows up here because
bokehdrags it in.The k8s and kerberos jobs dodge all of this: k8s builds from frozen images still pinned to
distributed==2026.3.0, and the kerberos test never spins up a real cluster.Fixing it isn't much: point the scheduler/worker commands at
dask scheduler/dask worker, and addpillowto themamba install -c conda-forge ...line the pbs/slurm_install.shalready uses for contourpy and pandas.Full logs if useful: https://github.com/dask/dask-gateway/actions/runs/29776604255
I'm happy to send a PR for both – just say the word.