Incompatibility break at mamba-ssm==2.2.5

[leeminju531]

Hi Mamba maintainers, Thanks for sharing this awesome repository 🙏

I'd like to report an incompatibility issue I encountered when installing mamba-ssm==2.2.5.
The same setup works fine with mamba-ssm==2.2.4, so it looks like a regression introduced in 2.2.5.

Environment

Host OS: Ubuntu (see attached screenshots)
GPU + Driver: (see nvidia-smi screenshot)
CUDA Toolkit (Host): 12.6
Docker Base Image: pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel
Python: 3.11 (comes with the base image)

Ubuntu :

Output of nvidia-smi :

Cuda Toolkit : 12.6

Steps to reproduce :

Step 1 - Docker File :

FROM pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel
ARG DEBIAN_FRONTEND=noninteractive

RUN apt-get update && apt-get install -y --no-install-recommends \
    git build-essential cmake ninja-build pkg-config

RUN apt-get install -y libgl1-mesa-glx libglib2.0-0

RUN python -m pip install --upgrade pip

# RUN apt-get update && apt-get install -y libgl1-mesa-glx libglib2.0-0 qt5-default

# RUN printf "numpy==1.26.4\n" > /etc/pip-constraints.txt
# ENV PIP_CONSTRAINT=/etc/pip-constraints.txt
# ENV PIP_UPGRADE_STRATEGY=only-if-needed

RUN pip install open3d==0.18.0 scipy pyquaternion tqdm opencv-python configobj einops numba plyfile transformers==4.37.2
# # RUN pip install -U setuptools wheel ninja
RUN pip install spconv-cu124
# RUN pip install torch-scatter -f https://data.pyg.org/whl/torch-2.1.0+${cu124}.html
# # RUN pip install nuscenes-devkit
# # RUN pip install mamba-ssm
# # RUN pip install git+https://github.com/openai/CLIP.git

# RUN apt-get install -y python3-tk ffmpeg

RUN pip install ftfy regex
RUN pip install git+https://github.com/openai/CLIP.git
RUN pip install --no-cache-dir --no-deps --no-build-isolation --no-binary :all: mamba-ssm==2.2.5

Step 2 : Build + run

docker build -t ilikemamba .
docker run -it --rm -v /tmp/.X11-unix:/tmp/.X11-unix --gpus all ilikemamba /bin/bash

Step 3 : Import Test

export LD_LIBRARY_PATH=/opt/conda/lib/python3.11/site-packages/torch/lib:$LD_LIBRARY_PATH

python -c "import selective_scan_cuda; print('selective_scan_cuda OK')"

Output :

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: /opt/conda/lib/python3.11/site-packages/selective_scan_cuda.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN3c107WarningC1ESt7variantIJNS0_11UserWarningENS0_18DeprecationWarningEEERKNS_14SourceLocationENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb

Expected Result :

The import should succeed and print:
selective_scan_cuda OK

This works correctly if I change the Dockerfile version to mamba-ssm==2.2.4:

Thanks again for maintaining this project 🙏
Please let me know if I can provide any additional logs or debugging output to help investigate this issue.

I’m still new to this ecosystem, so if my setup approach is incorrect (or there’s a better way to install/build), I’d really appreciate your guidance! 🙂

[leeminju531]

Additionally, I would like to use version 2.2.5. I would really appreciate it if you could guide me on how to set it up and use it properly.

[ibitec7]

Hi,

The mistake you are making is using pip to install from PyPI. The error you are getting is because the wheel pip installed from PyPI doesn't match how your environment was compiled. Since mamba uses custom triton kernels etc. that are compiled on for a specific CUDA version, you need to choose the right wheel version for your machine from here: releases

pip uses compatibility tags that get information like python version, operating system versions and hardware architecture. What it does not see is stuff like the CUDA version, whether Pytorch uses ABI or not. Here is a docker base image I wrote to download mamba v2.2.5 that works:

`# For build
FROM pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel

WORKDIR /app
ENV CUDA_HOME=/usr/local/cuda
ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH}

RUN apt-get update && apt-get install -y build-essential python3-dev git ninja-build cmake && \
    rm -rf /var/lib/apt/lists/*

# Install mamba, ensure correct versions

# Pytorch: 2.6.0
# Cuda: 12.0+
# Python/CPython: 3.11

RUN pip install --no-cache-dir \
    https://github.com/state-spaces/mamba/releases/download/v2.2.5/mamba_ssm-2.2.5+cu12torch2.6cxx11abiTRUE-cp311-cp311-linux_x86_64.whl

# COPY requirements.txt requirements.txt
# RUN pip install --no-cache-dir -r requirements.txt --prefer-binary

# COPY . .`

You can copy this Dockerfile and modify it to how you want. Here is how you can break down the wheel to tell if its the right version:

mamba_ssm-2.2.5+cu12torch2.6cxx11abiTRUE-cp311-cp311-linux_x86_64.whl

• "..cu12torch2.6..": means you need to have Pytorch with CUDA 12.0+ installed, since you download the base image 2.6 with CUDA 12.4 this is ok.
• "..cxx11abiTRUE..": whether your Pytorch was installed with C++11 ABI, Pytorch images always have this TRUE, but you can always check by running this in your base image in an interactive terminal to see if the output is true or false: python -c "import torch; print(torch._C._GLIBCXX_USE_CXX11_ABI)"
• "..cp311..": the CPython/Python version for your environment, use python --version in the interactive terminal of your base image to verify.

Hope this helps.