UCX depends on the following system libraries being present:
For MOFED 4.x support:
librdmacm. Ideally installed from Mellanox OFED Drivers
For MOFED 5.x support: Mellanox OFED Drivers
For system topology identification:
numactlon Enterprise Linux)
Please install the packages above with your Linux system’s package manager.
When building from source you will also need the
Enterprise Linux) packages as well.
Some preliminary Conda packages can be installed as so. Replace
<CUDA version> with either
11.2. These are
available both on
With GPU support:
conda create -n ucx -c conda-forge -c rapidsai \ cudatoolkit=<CUDA version> ucx-proc=*=gpu ucx ucx-py python=3.7
Without GPU support:
conda create -n ucx -c conda-forge -c rapidsai \ ucx-proc=*=cpu ucx ucx-py python=3.7
The following instructions assume you’ll be using ucx-py on a CUDA enabled system and is in a Conda environment.
As of version 0.15, the UCX conda package build will no longer include IB/RDMA support. This is largely due to compatibility issues between OFED versions. We do however provide instructions below for how to build UCX with IB/RDMA support in the UCX + OFED section.
conda create -n ucx -c conda-forge \ automake make libtool pkg-config \ psutil \ "python=3.7" setuptools "cython>=0.29.14,<3.0.0a0"
conda install -n ucx -c rapidsai -c nvidia -c conda-forge \ pytest pytest-asyncio \ cupy "numba>=0.46" rmm \ distributed
UCX >= 1.11.1
Instructions for building UCX >= 1.11.1 (minimum version supported by UCX-Py), make sure to change git checkout v1.11.1 to a newer version if desired:
conda activate ucx git clone https://github.com/openucx/ucx cd ucx git checkout v1.11.1 ./autogen.sh mkdir build cd build # Performance build ../contrib/configure-release --prefix=$CONDA_PREFIX --with-cuda=$CUDA_HOME --enable-mt # Debug build ../contrib/configure-devel --prefix=$CONDA_PREFIX --with-cuda=$CUDA_HOME --enable-mt make -j install
UCX + OFED
As noted above, the UCX conda package no longer builds support for IB/RDMA. To build UCX with IB/RDMA support first confirm OFED is installed properly:
(ucx) user@dgx:~$ ofed_info -s OFED-internal-4.7-3.2.9
If OFED drivers are not installed on the machine, you can download drivers at directly from Mellanox. For versions older than 5.1 click on, archive versions.
Building UCX >= 1.11.1 as shown previously should automatically include IB/RDMA support if available in the system. It is possible to explicitly activate those, ensuring the system satisfies all dependencies or fail otherwise, by including the
--with-verbs build flags. For example:
../contrib/configure-release \ --enable-mt \ --prefix="$CONDA_PREFIX" \ --with-cuda="$CUDA_HOME" \ --enable-mt \ --with-rdmacm \ --with-verbs
conda activate ucx git clone https://github.com/rapidsai/ucx-py.git cd ucx-py pip install -v . # or for develop build pip install -v -e .