These are my notes for setting up RoCE (RDMA over Converged Ethernet) on aarch64 running Ubuntu Server. The instructions below are based off the guide found here http://www.rdmamojo.com/2014/11/08/working-rdma-ubuntu/. However, some package dependencies are not available on arm64. I've updated the list to work with the Ubuntu packages available on arm64.
I am going to be using the Utah CloudLab cluster, which contains a bunch of HP Moonshot nodes with the following hardware:
The nodes are running the following software:
Install the following packages:
root@node:~$ apt-get install libibverbs1 ibverbs-utils librdmacm1 rdmacm-utils libcxgb3-1 libmlx4-1 libmthca1 libnes1 libibcommon1
Next we need to install some kernel modules, as Ubuntu currently doesn't autoload these.
Now we can test it using the guide http://community.mellanox.com/docs/DOC-1465. Start server:
end 0 means success.