Skip to content

deps: split ci_docgpu CPU/GPU depsets and regenerate locks #62595

Description

@ans9868

What happened + What you expected to happen

Problem

PR #62485 introduced ci_docgpu.depsets.yaml with a single depset combining both dl-cpu-requirements.txt and dl-gpu-requirements.txt. This creates an unresolvable PyTorch/PyG dependency conflict:

Because you require torch-spline-conv==1.2.2+pt27cu128 and
torch-spline-conv==1.2.2+pt27cpu, your requirements are unsatisfiable.

The docgpu Docker image is GPU-only (oss-ci-base_gpu), so it should never need CPU wheels. The conflict should not exist.

Bare Minimum Solution

Split into two separate depsets (following the ci_ml pattern):

  1. Update ci/raydepsets/configs/ci_docgpu.depsets.yaml:

    • Split ci_docgpu_depset into two:
      • ci_docgpu_cpu_depset: Uses --index https://download.pytorch.org/whl/cpu
      • ci_docgpu_gpu_depset: Uses --index https://download.pytorch.org/whl/cu128
  2. Update Docker files to reference GPU lock only:

    • ci/docker/docgpu.build.wanda.yaml (line 5): Change to docgpu_gpu_depset_py$PYTHON.lock
    • ci/docker/docgpu.build.Dockerfile (line 7): Change to docgpu_gpu_depset_py$PYTHON.lock

This resolves the torch-spline-conv conflict immediately.

Extensive Solution (Including Lock Regeneration)

After the config changes above, regenerate lock files:

  1. Run: bazel run //ci/raydepsets:raydepsets -- build ci/raydepsets/configs/ci_docgpu.depsets.yaml

  2. This generates four new lock files:

    • python/deplocks/ci/docgpu_cpu_depset_py3.10.lock
    • python/deplocks/ci/docgpu_cpu_depset_py3.12.lock
    • python/deplocks/ci/docgpu_gpu_depset_py3.10.lock
    • python/deplocks/ci/docgpu_gpu_depset_py3.12.lock
  3. Remove old lock files:

    • python/deplocks/ci/docgpu_depset_py3.10.lock
    • python/deplocks/ci/docgpu_depset_py3.12.lock

Known Blocker: etils Version Conflict

Lock regeneration is currently blocked by a pre-existing etils version mismatch:

  • python/requirements/ml/py313/dl-cpu-requirements.txt: etils==1.5.2; python_version < '3.12'
  • /tmp/ray-deps/requirements_compiled_py3.13.txt: etils==1.14.0

This conflict exists independently of this fix. It must be resolved separately (see related issue for etils resolution).

Once etils is fixed, lock regeneration will succeed and lock files can be committed.

Related information that helped me

Versions / Dependencies

Versions:
Python 3.10, 3.12

Ray master branch
Dependencies :
PyTorch (CPU vs GPU indices)
PyG packages (torch-spline-conv, torch-scatter, torch-sparse, torch-cluster)

Reproduction script

Run the bash script in below in the project root. Shows the problem explicitly. The expected output

 Step 1: Showing the conflicting wheels in docgpu requirements
==========================================

CPU variant (dl-cpu-requirements.txt):
--find-links https://data.pyg.org/whl/torch-2.7.0+cpu.html  # for CPU versions of torch-scatter, torch-sparse, torch-cluster, torch-spline-conv
torch-spline-conv==1.2.2

GPU variant (dl-gpu-requirements.txt):
--find-links https://data.pyg.org/whl/torch-2.7.0+cu128.html  # for GPU versions of torch-scatter, torch-sparse, torch-cluster, torch-spline-conv
torch-spline-conv==1.2.2+pt27cu128

Step 2: Attempting to compile docgpu depset
==========================================
Running: bazel run //ci/raydepsets:raydepsets -- build ci/raydepsets/configs/ci_docgpu.depsets.yaml

This should FAIL with conflict like:
  'Because you require torch-spline-conv==1.2.2+pt27cu128 and'
  'torch-spline-conv==1.2.2+pt27cpu, your requirements are unsatisfiable.'

............

Attempting compile...



No solution found when resolving dependencies:
  ╰─▶ Because you require torch-spline-conv==1.2.2+pt27cu128 and
      torch-spline-conv==1.2.2+pt27cpu, we can conclude that your requirements
      are unsatisfiable.

Below is the script to reproduce the problem in the project root

#!/bin/bash
# Reproduce the torch-spline-conv conflict error in docgpu depset
# This script demonstrates the problem before the fix

set -e

echo "========================================"
echo "Reproducing torch-spline-conv conflict"
echo "========================================"
echo ""
echo "The issue: docgpu tries to merge CPU and GPU PyTorch wheels"
echo "in a SINGLE depset, which creates an unsolvable conflict."
echo ""

if [ ! -f "ci/raydepsets/configs/ci_docgpu.depsets.yaml" ]; then
  echo "ERROR: Must run from ray repository root"
  exit 1
fi

if [ ! -d "/tmp/ray-deps" ]; then
  echo "ERROR: /tmp/ray-deps not found (needed for constraint file)"
  echo "This script requires the ray-deps directory with requirements_compiled_py3.13.txt"
  exit 1
fi

echo "Step 1: Showing the conflicting wheels in docgpu requirements"
echo "=========================================="
echo ""
echo "CPU variant (dl-cpu-requirements.txt):"
grep "torch-spline-conv" python/requirements/ml/py313/dl-cpu-requirements.txt || echo "  (not found)"
echo ""
echo "GPU variant (dl-gpu-requirements.txt):"
grep "torch-spline-conv" python/requirements/ml/py313/dl-gpu-requirements.txt || echo "  (not found)"
echo ""

echo "Step 2: Attempting to compile docgpu depset"
echo "=========================================="
echo "Running: bazel run //ci/raydepsets:raydepsets -- build ci/raydepsets/configs/ci_docgpu.depsets.yaml"
echo ""
echo "This should FAIL with a conflict error like:"
echo "  'Because you require torch-spline-conv==1.2.2+pt27cu128 and'"
echo "  'torch-spline-conv==1.2.2+pt27cpu, your requirements are unsatisfiable.'"
echo ""
echo "Attempting compile (first 100 lines of output)..."
echo "=========================================="
echo ""

if ! command -v bazel &> /dev/null; then
  echo "bazel not found - skipping compilation test"
  echo "Install bazel to see the full error"
else
  bazel run //ci/raydepsets:raydepsets -- build ci/raydepsets/configs/ci_docgpu.depsets.yaml 2>&1 | head -100
fi

echo ""
echo "=========================================="
echo "✓ Error reproduced!"
echo "=========================================="
echo ""
echo "The conflict occurs because:"
echo "  - dl-cpu-requirements.txt requires: torch-spline-conv==1.2.2 (plain)"
echo "  - dl-gpu-requirements.txt requires: torch-spline-conv==1.2.2+pt27cu128 (CUDA)"
echo "  - Single depset tries to resolve BOTH in one index → impossible"
echo ""
echo "Solution: Split into two depsets with separate indices:"
echo "  - ci_docgpu_cpu_depset: --index https://download.pytorch.org/whl/cpu"
echo "  - ci_docgpu_gpu_depset: --index https://download.pytorch.org/whl/cu128"
echo ""

Issue Severity

High: It blocks me from completing my task.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething that is supposed to be working; but isn'tcommunity-backlogreefstabilitytech-debtThe issue that's due to tech debt

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions