Install

Prerequisites

UCX depends on the following system libraries being present:

Please install the packages above with your Linux system’s package manager. When building from source you will also need the *-dev (*-devel on Enterprise Linux) packages as well.

Optional Packages

Enabling InfiniBand requires that host is running a build of Linux kernel 5.6 or higher with InfiniBand active or NVIDIA MLNX_OFED Drivers 5.0 or higher.

Once the existence of either Linux kernel 5.6 or higher or MOFED 5.0 or higher is confirmed, verify that InfiniBand support is active by checking for the presence of /dev/infiniband/rdma_cm and /dev/infiniband/uverbs*:

$ ls -l /dev/infiniband/{rdma_cm,uverbs*}
crw-rw-rw- 1 root root  10,  58 May 18 20:43 /dev/infiniband/rdma_cm
crw-rw-rw- 1 root root 231, 192 May 18 20:43 /dev/infiniband/uverbs0
crw-rw-rw- 1 root root 231, 193 May 18 20:43 /dev/infiniband/uverbs1
crw-rw-rw- 1 root root 231, 194 May 18 20:43 /dev/infiniband/uverbs2
crw-rw-rw- 1 root root 231, 195 May 18 20:43 /dev/infiniband/uverbs3

Conda

Use one of the commands below to install conda packages. Replace -c rapidsai with -c rapidsai-nightly to pull in the newer but less stable nightly packages. Change cuda-version to pin to a different CUDA minor version if you’d like.

# CUDA 12
conda create -n ucx -c conda-forge -c rapidsai \
  cuda-version=12.9 ucx-py

Starting with the UCX 1.14.1 conda-forge package, InfiniBand support is available again via rdma-core, thus building UCX from source is not required solely for that purpose anymore but may still be done if desired (e.g., to test for new capabilities or bug fixes).

PyPI

PyPI installation is possible and currently supports CUDA version 12. Packages are compatible with CPU-only workloads and any one can be chosen if the application doesn’t use CUDA, but currently there are no pre-built CPU-only packages available, so the CUDA package must be installed instead.

# CUDA 12
pip install ucx-py-cu12

UCX-Py has no direct dependency on CUDA, but the package specifies the -cu12 prefix so that the correct libucx-cu12 package is selected. This is also the reason why there are no CPU-only UCX-Py packages available at the moment, CPU-only builds of the UCX library are not currently available in PyPI.

Source

Conda

The following instructions assume you’ll be using UCX-Py on a CUDA-enabled system and using a Conda environment.

Build Dependencies

conda create -n ucx -c conda-forge \
    automake make libtool pkg-config \
    "python=3.13" "setuptools>=64.0" "cython>=3.0.0"

Note

The Python version must be explicitly specified here, UCX-Py currently supports Python versions 3.10, 3.11, 3.12, and 3.13.

Test Dependencies

conda install -n ucx -c rapidsai -c nvidia -c conda-forge \
    pytest pytest-asyncio \
    cupy "numba>=0.57" cudf \
    dask distributed cloudpickle

UCX >= 1.15.0

Instructions for building UCX >= 1.15.0 (minimum version supported by UCX-Py), make sure to change git checkout v1.15.0 to a newer version if desired:

conda activate ucx
git clone https://github.com/openucx/ucx
cd ucx
git checkout v1.15.0
./autogen.sh
mkdir build
cd build
# Performance build
../contrib/configure-release --prefix=$CONDA_PREFIX --with-cuda=$CUDA_HOME --enable-mt
# Debug build
../contrib/configure-devel --prefix=$CONDA_PREFIX --with-cuda=$CUDA_HOME --enable-mt
make -j install

UCX + rdma-core

It is possible to enable InfiniBand support via the conda-forge rdma-core package. To do so, first activate the environment created previously and install conda-forge compilers and rdma-core:

conda activate ucx
conda install -c conda-forge c-compiler cxx-compiler gcc_linux-64=11.* rdma-core=28.*

After installing the necessary dependencies, it’s now time to build UCX from source, make sure to change git checkout v1.15.0 to a newer version if desired:

git clone https://github.com/openucx/ucx
cd ucx
git checkout v1.15.0
./autogen.sh
mkdir build
cd build
# Performance build
../contrib/configure-release --prefix=$CONDA_PREFIX --with-cuda=$CUDA_HOME --enable-mt --with-verbs --with-rdmacm
# Debug build
../contrib/configure-devel --prefix=$CONDA_PREFIX --with-cuda=$CUDA_HOME --enable-mt --with-verbs --with-rdmacm
make -j install

UCX + MOFED

It is still possible to build UCX and use the MOFED system install. Unlike the case above, we must not install conda-forge compilers, this is because conda-forge compilers can’t look for libraries in the system directories (e.g., /usr). Additionally, the rdma-core conda-forge package should not be installed either, because compiling with a newer MOFED version will cause ABI incompatibilities.

Before continuing, first ensure MOFED 5.0 or higher is installed, for example in the example below we have MOFED 5.4-3.5.8.0:

(ucx) user@dgx:~$ ofed_info -s
MLNX_OFED_LINUX-5.4-3.5.8.0:

If MOFED drivers are not installed on the machine, you can download drivers directly from NVIDIA.

Building from source as shown previously should automatically include InfiniBand support if available in the system. It is possible to explicitly activate those, ensuring the system satisfies all dependencies or fail otherwise, by including the --with-rdmacm and --with-verbs build flags. Additionally, we want to make sure UCX uses compilers from the system, we do so by specifying CC=/usr/bin/gcc and CXX=/usr/bin/g++, be sure to adjust that for the path to your system compilers. For example:

CC=/usr/bin/gcc CXX=/usr/bin/g++ \
../contrib/configure-release \
--enable-mt \
--prefix="$CONDA_PREFIX" \
--with-cuda="$CUDA_HOME" \
--enable-mt \
--with-rdmacm \
--with-verbs

UCX-Py

Building and installing UCX-Py can be done via pip install. For example:

conda activate ucx
git clone https://github.com/rapidsai/ucx-py.git
cd ucx-py
pip install -v .
# or for develop build
pip install -v -e .

PyPI

The following instructions assume you’ll be installing UCX-Py on a CUDA-enabled system, in a pip-only environment.

Installing UCX-Py from source in a pip-only environment has additional limitations when compared to conda environments.

UCX-Py with UCX from PyPI

CUDA-enabled builds of the UCX libraries are available from PyPI, under the name libucx-cu12 (CUDA 12). Notice that those builds do not currently include InfiniBand support, if InfiniBand is required you will need to provide a custom UCX install as described in the “UCX-Py with custom UCX install” section.

To build UCX-Py using those UCX packages (to avoid needing to build UCX from source), run the following.

conda activate ucx
git clone https://github.com/rapidsai/ucx-py.git
cd ucx-py
pip install -C 'rapidsai.disable-cuda=false' .
# or for develop build
pip install -v -e .

This will automatically handle installing appropriate, compatible libucx-cu12 packages for build-time and runtime use. When you run UCX-Py code installed this way, it will load UCX libraries from the installed libucx-cu12 package.

UCX-Py packages are built against the oldest version of UCX that UCX-Py supports, and can run against a range of ABI-compatible UCX versions.

You can use packages from PyPI to customize the UCX version used at runtime. For example, to switch to using UCX 1.18 at runtime, run the following.

# CUDA 12
pip install 'libucx-cu12>=1.18.0,<1.19'

UCX-Py with UCX system install

If a UCX system install is available, building and installing UCX-Py can be done via pip install with no additional requirements. For example:

conda activate ucx
git clone https://github.com/rapidsai/ucx-py.git
cd ucx-py
pip install -v .
# or for develop build
pip install -v -e .

To ensure that system install of UCX is always used at runtime (and not the libucx-cu12 wheels), set the following environment variable in the runtime environment.

export RAPIDS_LIBUCX_PREFER_SYSTEM_LIBRARY=true

UCX-Py with custom UCX install

If UCX is installed in a non-default path (as it might be if you built it from source), some additional configuration is required to build and run UCX-Py against it. To check if the loader can find your custom UCX installation, run the following.

ldconfig -p | grep libucs

If that returns that filepath you expect, then you can just use the “UCX-Py with UCX system install” instructions above. If that doesn’t show anything, then you need to help the loader find the UCX libraries. At build time, add your install of UCX to LD_LIBRARY_PATH.

conda activate ucx
git clone https://github.com/rapidsai/ucx-py.git
cd ucx-py
CUSTOM_UCX_INSTALL="wherever-you-put-your-ucx-install"
LD_LIBRARY_PATH="${CUSTOM_UCX_INSTALL}:${LD_LIBRARY_PATH}" \
    pip install -v .
# or for develop build
LD_LIBRARY_PATH="${CUSTOM_UCX_INSTALL}:${LD_LIBRARY_PATH}" \
    pip install -v -e .

Set the following in the environment to ensure that those libraries are preferred at run time as well.

RAPIDS_LIBUCX_PREFER_SYSTEM_LIBRARY=true
LD_LIBRARY_PATH="${CUSTOM_UCX_INSTALL}:${LD_LIBRARY_PATH}" \
  python -c "import ucp; print(ucp.get_ucx_version())"