These are my notes for setting up RoCE (RDMA over Converged Ethernet) on aarch64 running Ubuntu Server. The instructions below are based off the guide found here http://www.rdmamojo.com/2014/11/08/working-rdma-ubuntu/. However, some package dependencies are not available on arm64. I've updated the list to work with the Ubuntu packages available on arm64.

I am going to be using the Utah CloudLab cluster, which contains a bunch of HP Moonshot nodes with the following hardware:

HP Moonshot m400
Eight 64-bit ARMv8 (Atlas/A57) cores at 2.4 GHz (APM X-GENE)
64GB ECC Memory (8x 8 GB DDR3-1600 SO-DIMMs)
120 GB of flash (SATA3 / M.2, Micron M500)
Dual-port Mellanox ConnectX-3 10 GB NIC (PCIe v3.0, 8 lanes)

The nodes are running the following software:

root@node:~$ uname -a
Linux node.nwatkins-qv1183.emulab-net.utah.cloudlab.us 3.13.0-40-generic #69-Ubuntu SMP Thu Nov 13 19:05:44 UTC 2014 aarch64 aarch64 aarch64 GNU/Linux
root@node:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.1 LTS
Release:    14.04
Codename:   trusty

Install the following packages:

root@node:~$ apt-get install libibverbs1 ibverbs-utils librdmacm1 rdmacm-utils libcxgb3-1 libmlx4-1 libmthca1 libnes1 libibcommon1

Next we need to install some kernel modules, as Ubuntu currently doesn't autoload these.

modprobe rdma_cm
modprobe ib_uverbs
modprobe rdma_ucm
modprobe ib_ucm
modprobe ib_umad
modprobe ib_ipoib
modprobe mlx4_ib
modprobe mlx4_en
modprobe iw_cxgb3
modprobe iw_cxgb4
modprobe iw_nes
modprobe iw_c2

Now we can test it using the guide http://community.mellanox.com/docs/DOC-1465. Start server:

nwatkins@node-1:~$ rdma_server
rdma_server: start
rdma_server: end 0

Run client:

nwatkins@node-1:~$ rdma_client -s 10.10.1.2
rdma_client: start
rdma_client: end 0
nwatkins@node-1:~$ 

Apparently end 0 means success.