What happened + What you expected to happen
Problem
PR #62485 introduced ci_docgpu.depsets.yaml with a single depset combining both dl-cpu-requirements.txt and dl-gpu-requirements.txt. This creates an unresolvable PyTorch/PyG dependency conflict:
Because you require torch-spline-conv==1.2.2+pt27cu128 and
torch-spline-conv==1.2.2+pt27cpu, your requirements are unsatisfiable.
The docgpu Docker image is GPU-only (oss-ci-base_gpu), so it should never need CPU wheels. The conflict should not exist.
Bare Minimum Solution
Split into two separate depsets (following the ci_ml pattern):
-
Update ci/raydepsets/configs/ci_docgpu.depsets.yaml:
- Split
ci_docgpu_depset into two:
ci_docgpu_cpu_depset: Uses --index https://download.pytorch.org/whl/cpu
ci_docgpu_gpu_depset: Uses --index https://download.pytorch.org/whl/cu128
-
Update Docker files to reference GPU lock only:
ci/docker/docgpu.build.wanda.yaml (line 5): Change to docgpu_gpu_depset_py$PYTHON.lock
ci/docker/docgpu.build.Dockerfile (line 7): Change to docgpu_gpu_depset_py$PYTHON.lock
This resolves the torch-spline-conv conflict immediately.
Extensive Solution (Including Lock Regeneration)
After the config changes above, regenerate lock files:
-
Run: bazel run //ci/raydepsets:raydepsets -- build ci/raydepsets/configs/ci_docgpu.depsets.yaml
-
This generates four new lock files:
python/deplocks/ci/docgpu_cpu_depset_py3.10.lock
python/deplocks/ci/docgpu_cpu_depset_py3.12.lock
python/deplocks/ci/docgpu_gpu_depset_py3.10.lock
python/deplocks/ci/docgpu_gpu_depset_py3.12.lock
-
Remove old lock files:
python/deplocks/ci/docgpu_depset_py3.10.lock
python/deplocks/ci/docgpu_depset_py3.12.lock
Known Blocker: etils Version Conflict
Lock regeneration is currently blocked by a pre-existing etils version mismatch:
python/requirements/ml/py313/dl-cpu-requirements.txt: etils==1.5.2; python_version < '3.12'
/tmp/ray-deps/requirements_compiled_py3.13.txt: etils==1.14.0
This conflict exists independently of this fix. It must be resolved separately (see related issue for etils resolution).
Once etils is fixed, lock regeneration will succeed and lock files can be committed.
Related information that helped me
Versions / Dependencies
Versions:
Python 3.10, 3.12
Ray master branch
Dependencies :
PyTorch (CPU vs GPU indices)
PyG packages (torch-spline-conv, torch-scatter, torch-sparse, torch-cluster)
Reproduction script
Run the bash script in below in the project root. Shows the problem explicitly. The expected output
Step 1: Showing the conflicting wheels in docgpu requirements
==========================================
CPU variant (dl-cpu-requirements.txt):
--find-links https://data.pyg.org/whl/torch-2.7.0+cpu.html # for CPU versions of torch-scatter, torch-sparse, torch-cluster, torch-spline-conv
torch-spline-conv==1.2.2
GPU variant (dl-gpu-requirements.txt):
--find-links https://data.pyg.org/whl/torch-2.7.0+cu128.html # for GPU versions of torch-scatter, torch-sparse, torch-cluster, torch-spline-conv
torch-spline-conv==1.2.2+pt27cu128
Step 2: Attempting to compile docgpu depset
==========================================
Running: bazel run //ci/raydepsets:raydepsets -- build ci/raydepsets/configs/ci_docgpu.depsets.yaml
This should FAIL with conflict like:
'Because you require torch-spline-conv==1.2.2+pt27cu128 and'
'torch-spline-conv==1.2.2+pt27cpu, your requirements are unsatisfiable.'
............
Attempting compile...
No solution found when resolving dependencies:
╰─▶ Because you require torch-spline-conv==1.2.2+pt27cu128 and
torch-spline-conv==1.2.2+pt27cpu, we can conclude that your requirements
are unsatisfiable.
Below is the script to reproduce the problem in the project root
#!/bin/bash
# Reproduce the torch-spline-conv conflict error in docgpu depset
# This script demonstrates the problem before the fix
set -e
echo "========================================"
echo "Reproducing torch-spline-conv conflict"
echo "========================================"
echo ""
echo "The issue: docgpu tries to merge CPU and GPU PyTorch wheels"
echo "in a SINGLE depset, which creates an unsolvable conflict."
echo ""
if [ ! -f "ci/raydepsets/configs/ci_docgpu.depsets.yaml" ]; then
echo "ERROR: Must run from ray repository root"
exit 1
fi
if [ ! -d "/tmp/ray-deps" ]; then
echo "ERROR: /tmp/ray-deps not found (needed for constraint file)"
echo "This script requires the ray-deps directory with requirements_compiled_py3.13.txt"
exit 1
fi
echo "Step 1: Showing the conflicting wheels in docgpu requirements"
echo "=========================================="
echo ""
echo "CPU variant (dl-cpu-requirements.txt):"
grep "torch-spline-conv" python/requirements/ml/py313/dl-cpu-requirements.txt || echo " (not found)"
echo ""
echo "GPU variant (dl-gpu-requirements.txt):"
grep "torch-spline-conv" python/requirements/ml/py313/dl-gpu-requirements.txt || echo " (not found)"
echo ""
echo "Step 2: Attempting to compile docgpu depset"
echo "=========================================="
echo "Running: bazel run //ci/raydepsets:raydepsets -- build ci/raydepsets/configs/ci_docgpu.depsets.yaml"
echo ""
echo "This should FAIL with a conflict error like:"
echo " 'Because you require torch-spline-conv==1.2.2+pt27cu128 and'"
echo " 'torch-spline-conv==1.2.2+pt27cpu, your requirements are unsatisfiable.'"
echo ""
echo "Attempting compile (first 100 lines of output)..."
echo "=========================================="
echo ""
if ! command -v bazel &> /dev/null; then
echo "bazel not found - skipping compilation test"
echo "Install bazel to see the full error"
else
bazel run //ci/raydepsets:raydepsets -- build ci/raydepsets/configs/ci_docgpu.depsets.yaml 2>&1 | head -100
fi
echo ""
echo "=========================================="
echo "✓ Error reproduced!"
echo "=========================================="
echo ""
echo "The conflict occurs because:"
echo " - dl-cpu-requirements.txt requires: torch-spline-conv==1.2.2 (plain)"
echo " - dl-gpu-requirements.txt requires: torch-spline-conv==1.2.2+pt27cu128 (CUDA)"
echo " - Single depset tries to resolve BOTH in one index → impossible"
echo ""
echo "Solution: Split into two depsets with separate indices:"
echo " - ci_docgpu_cpu_depset: --index https://download.pytorch.org/whl/cpu"
echo " - ci_docgpu_gpu_depset: --index https://download.pytorch.org/whl/cu128"
echo ""
Issue Severity
High: It blocks me from completing my task.
What happened + What you expected to happen
Problem
PR #62485 introduced
ci_docgpu.depsets.yamlwith a single depset combining bothdl-cpu-requirements.txtanddl-gpu-requirements.txt. This creates an unresolvable PyTorch/PyG dependency conflict:The docgpu Docker image is GPU-only (
oss-ci-base_gpu), so it should never need CPU wheels. The conflict should not exist.Bare Minimum Solution
Split into two separate depsets (following the
ci_mlpattern):Update
ci/raydepsets/configs/ci_docgpu.depsets.yaml:ci_docgpu_depsetinto two:ci_docgpu_cpu_depset: Uses--index https://download.pytorch.org/whl/cpuci_docgpu_gpu_depset: Uses--index https://download.pytorch.org/whl/cu128Update Docker files to reference GPU lock only:
ci/docker/docgpu.build.wanda.yaml(line 5): Change todocgpu_gpu_depset_py$PYTHON.lockci/docker/docgpu.build.Dockerfile(line 7): Change todocgpu_gpu_depset_py$PYTHON.lockThis resolves the torch-spline-conv conflict immediately.
Extensive Solution (Including Lock Regeneration)
After the config changes above, regenerate lock files:
Run:
bazel run //ci/raydepsets:raydepsets -- build ci/raydepsets/configs/ci_docgpu.depsets.yamlThis generates four new lock files:
python/deplocks/ci/docgpu_cpu_depset_py3.10.lockpython/deplocks/ci/docgpu_cpu_depset_py3.12.lockpython/deplocks/ci/docgpu_gpu_depset_py3.10.lockpython/deplocks/ci/docgpu_gpu_depset_py3.12.lockRemove old lock files:
python/deplocks/ci/docgpu_depset_py3.10.lockpython/deplocks/ci/docgpu_depset_py3.12.lockKnown Blocker: etils Version Conflict
Lock regeneration is currently blocked by a pre-existing etils version mismatch:
python/requirements/ml/py313/dl-cpu-requirements.txt:etils==1.5.2; python_version < '3.12'/tmp/ray-deps/requirements_compiled_py3.13.txt:etils==1.14.0This conflict exists independently of this fix. It must be resolved separately (see related issue for etils resolution).
Once etils is fixed, lock regeneration will succeed and lock files can be committed.
Related information that helped me
ci_ml.depsets.yamlsplits CPU (ci_ml_build_depset) and GPU (ci_ml_gpubuild_depset) variantsVersions / Dependencies
Versions:
Python 3.10, 3.12
Ray master branch
Dependencies :
PyTorch (CPU vs GPU indices)
PyG packages (torch-spline-conv, torch-scatter, torch-sparse, torch-cluster)
Reproduction script
Run the bash script in below in the project root. Shows the problem explicitly. The expected output
Below is the script to reproduce the problem in the project root
Issue Severity
High: It blocks me from completing my task.