[QST] rapids-singlecell-cu12 container ships CUDA 13 binaries, fails on driver 570 (CUDA 12.8)

[chris-hegde]

Environment

• Container: ghcr.io/scverse/rapids-singlecell-cu12:latest (pulled May 2026)
• HPC GPU nodes: NVIDIA A100-PCIE-40GB and RTX 3090
• Driver version: 570.148.08
• CUDA version reported by nvidia-smi: 12.8
• Apptainer version: 1.3.6
• OS: RHEL 9.6

Problem

Importing rapids_singlecell fails with:

ImportError: cuda.bindings.cydriver does not export expected C function cuDeviceGetUuid_v2

CuPy itself works correctly — cp.cuda.runtime.getDeviceCount() returns 1 and the GPU is visible. The failure is specific to cuml/cudf imports.

Evidence of CUDA 13 internals in cu12 container

During container build, pip reports the following dependency conflicts inside the cu12 container:

cuml 26.4.0 requires cupy-cuda13x>=13.6.0
cudf 26.4.0 requires cupy-cuda13x!=14.0.0,>=13.6.0
cuda-python 12.9.6 requires cuda-bindings~=12.9.6, but cuda-bindings 13.2.0 is present

This indicates the RAPIDS stack inside the cu12 container was built against CUDA 13,
which requires driver 580+ — but the cu12 container should support driver 535+
per the RAPIDS platform support matrix.

Expected behaviour

The cu12 container should work with driver 570 (CUDA 12.8), which satisfies the
stated CUDA 12 requirement of driver 535+.

Is the only workaround to pull the older version ? can someone help me figure this out, thank you!!