In this post I'll show you how to use iSER, iSCSI, and LIO to setup a remote
RAM disk. This is useful if you need high IOPS but don't have access to a
bunch of SSDs or NVRAM. Note that the performance achieved in this post is
quite low compared to what you should be able to achieve with different
hardware. Currently the arm64 machines we are using aren't getting the
performance expected, and tuning is on going. However, the description of the
steps here are relevant for other installations. Once you create several
remote RAM disks, tie them together with RAID 0 or dm-linear.
We'll use the following hardware provided by CloudLab.
Next I'll show you the basic server and client setup, and then demonstrate
usage with some basic benchmarks.
Target (Server) Setup
Make the RAMDisk backing store:
Make the iSCSI target:
Create a LUN backed by the RAMDisk:
Create a portal for the iSCSI target:
Enable iSER on the portal:
Here is the final configuration
Finally, disable all security
Initiator (Client) Setup
From the client, also called the initiator, we can use the iscsiadm tool to
look for the targets we have created. In this case we've setup one iSCSI
target on the node with address 10.10.1.3:
To access the iSCSI targets as local devices we need to login to the
targets. We can login to all of the targets that have been discovered with the
Now that we have logged into the target, we should be able to access the LUN
we setup as a local device. We can see the device has been attached by
The iscsiadm tool also lets us look at a lot of information about the
targets. Using the following command we can see some of the networking
configuration for the targets:
Notice that the Iface Transport option is set to tcp. In order to get
maximum performance using RDMA, we want to use the iSER transport instead. To
set this we first need to logout of the targets:
Next, set the iface.transport_name option to iser for our target:
Now we can log back in to the target and check to be sure that the transport
has been set to iSER. Login:
We are going to use the fio tool to do 512 byte direct I/O random reads and
writes to the remote RAM disk devices. With a single device we are able to get
around 70,000 random read and write IOPS. Here is the write workload:
And the read workload:
I would expect the performance to be better.
One iSCSI Target with 2 LUNs
Next we try again with one iSCSI target hosting two LUNs, and instruct fio
to send IOs to both devices. For writes we get slightly less than 2x speed-up
at 113,000 IOPS:
And right about 2x speed-up for reads:
This still seems really slow.
Two iSCSI portals With 2 LUNs
Next we try to create separate portals. We associate one LUN with each portal.
Writes are now performing a bit better than 2x at 131K IOPS, but this is
probably a peak we caught. For the most part its about 2x:
And reads are a bit better too at 126K IOPS:
Still not that great.
Four Targets With Four LUNs
The next experiment is four targets each with a separate LUN. Now we get
roughly 200K IOPS for read and write workloads. Write workload:
And the read workload:
I really expect much higher IOPS.
According to this page
there are a lot of different optimizations that can be applied to help squeeze
out more performance. However, after applying most of the optimizations the
performance doesn't really improve for me. While the page above isn't using
RoCE and they are using x86 rather than arm64, they are able to get almost 2
million IOPS. I'm hoping I can figure out how to get more IOPS out of our