Skip to content

ci job failes on main due to upstream dependency changes #944

Description

@GorML

Heads up - the local, pbs and slurm jobs are failing on main right now, and on pretty much any PR that pulls fresh deps from PyPI. I ran into it on #941 and dug in a bit; it doesn't look like anything in gateway itself broke, just two dependencies that moved under us. The k8s and kerberos jobs are fine. (@consideRatio, this is the "other CI failure" you mentioned over there.)

The first one hits local. distributed 2026.6.0 removed the old dask-scheduler / dask-worker console scripts – they live under dask scheduler / dask worker now (dask/distributed#9240). Gateway still calls the old names, both as the scheduler_cmd/worker_cmd defaults in backends/base.py and hardcoded as /opt/python/bin/dask-scheduler in the pbs/slurm tests, so cluster startup just dies with:

FileNotFoundError: [Errno 2] No such file or directory: 'dask-scheduler'

The second one hits pbs and slurm, and it's actually why they don't even get to pytest. Pillow 12.3.0 stopped publishing manylinux2014 wheels, and our CI image is still CentOS 7 (glibc 2.17), so pip grabs sdist and tries to build it – but there's no gcc in the container:

error: command 'gcc' failed: No such file or directory

Pillow only shows up here because bokeh drags it in.

The k8s and kerberos jobs dodge all of this: k8s builds from frozen images still pinned to distributed==2026.3.0, and the kerberos test never spins up a real cluster.

Fixing it isn't much: point the scheduler/worker commands at dask scheduler / dask worker, and add pillow to the mamba install -c conda-forge ... line the pbs/slurm _install.sh already uses for contourpy and pandas.

Full logs if useful: https://github.com/dask/dask-gateway/actions/runs/29776604255

I'm happy to send a PR for both – just say the word.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions