[bug] EFA/OFI plugin fails to load — Nemo 26.02.00 container

[sudostock]

Problem

During a DGXC benchmark run on AWS H100, NCCL fails to load the OFI network plugin and falls back to Socket transport (with terrible performance as a result):

  NCCL INFO NET/Plugin: libnccl-net-ofi.so: /opt/rdma-core/build/lib/libibverbs.so.1: version `IBVERBS_PRIVATE_34' not found (required by
  /usr/lib/x86_64-linux-gnu/libefa.so.1)
  NCCL INFO NET/Plugin: Could not find: ofi

Appears to be a version conflict with the rdma-core libs installed:

• System (/usr/lib/x86_64-linux-gnu/): libibverbs 1.14.56.0 exporting IBVERBS_PRIVATE_34, installed via .deb from the networking image. ibverbs-providers (libefa, libmlx5, etc.) are built against this version.
• HPC-X (/opt/rdma-core/build/lib/): libibverbs 1.15.60.0 exporting IBVERBS_PRIVATE_59, from HPC-X's bundled rdma-core source build.

The version with IBVERBS_PRIVATE_59 gets loaded during job init.

Minimal repro

1. Run any DGXC benchmark recipe on an AWS system with NCCL_DEBUG=INFO.
2. Alternatively try loading the library directly in the container:

import ctypes, os
ctypes.CDLL('/opt/rdma-core/build/lib/libibverbs.so.1', mode=os.RTLD_NOW | os.RTLD_GLOBAL)
ctypes.CDLL('libnccl-net-ofi.so', mode=os.RTLD_NOW)

Expected behavior

I expect it to correctly load the EFA libraries on an EFA system.

Affected area

area:build

Regression?

Yes

Environment

• NeMo Container: 26.02.00
• DGXC Benchmarks 26.02 (pre-release)
• AWS H100 cluster

Logs

pool0-1521:2460198:2460198 [3] NCCL INFO NCCL_NET_PLUGIN set by environment to ofi
pool0-1521:2460198:2460198 [3] NCCL INFO NET/Plugin: libnccl-net-ofi.so: /opt/rdma-core/build/lib/libibverbs.so.1: version `IBVERBS_PRIVATE_34' not found (required by /usr/lib/x86_64-linux-gnu/libefa.so.1)
pool0-1521:2460198:2460198 [3] NCCL INFO NET/Plugin: Could not find: ofi
pool0-1521:2460189:2460189 [0] NCCL INFO NET/IB : No device found.
pool0-1521:2460189:2460189 [0] NCCL INFO NET/IB : Using [RO]; OOB eth0:10.12.183.174<0>
pool0-1521:2460189:2460189 [0] NCCL INFO Failed to initialize NET plugin IB
pool0-1521:2460189:2460189 [0] NCCL INFO NET/Socket : Using [0]eth0:10.12.183.174<0>
pool0-1521:2460189:2460189 [0] NCCL INFO Initialized NET plugin Socket
pool0-1521:2460189:2460189 [0] NCCL INFO Could not get speed from /sys/class/net/eth0/speed. Defaulting to 10 Gbps.
pool0-1521:2460189:2460189 [0] NCCL INFO Assigned NET plugin Socket to comm
pool0-1521:2460189:2460189 [0] NCCL INFO Using network Socket

[ko3n1g]

hey, could you try if LD_LIBRARY_PATH=/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/opt/amazon/aws-ofi-nccl/lib:/usr/local/tensorrt/lib:/usr/local/cuda/compat/lib.real:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/opt/amazon/aws-ofi-nccl/lib:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/usr/local/lib/python3.12/dist-packages/torch/lib:/usr/local/lib/python3.12/dist-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64,PATH=/opt/amazon/efa/bin:/opt/amazon/openmpi/bin:/opt/amazon/aws-ofi-nccl/bin:/opt/venv/bin:/opt/venv/bin:/root/.local/bin:/opt/amazon/aws-ofi-nccl/bin:/opt/amazon/efa/bin:/opt/amazon/openmpi/bin:/opt/venv/bin:/usr/local/lib/python3.12/dist-packages/torch_tensorrt/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/mpi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/ucx/bin:/opt/amazon/efa/bin:/opt/tensorrt/bin,OPAL_PREFIX=/opt/amazon/openmpi,FI_PROVIDER=efa fixes the issue for you?

Note that the 26.02.00 container comes with aws-ofi-nccl 1.17.0 which has a known memory leak issue. We're publishing a patch to 26.02 soon that'll upgrade it to 1.17.3

[sudostock]

I tried and I get the same error. I went down a similar rabbit hole with my debug symlinking, ldconfig, etc inside the container to force it to pickup the correct library, with no luck.

The workaround that does work is nuking the newer (/opt/rdma-core/build/lib/) rdma-core libraries from the container.

Good to know on the memory leak, don't think I've encountered that yet.