This is Chapter One of VPP on Arm (VPP is one of the projects from fd.io - Fast Data Input/Output collection of projects).
VPP stands for Vector Packet Processing, it is the open source version of Cisco's Vector Packet Processing Software. VPP is a framework/toolkit which provides the functionality of Router/Switch in userspace. You can read more on What is VPP?
Even before we get started with VPP and running external traffic we need a NIC. The first ones I encountered were Mellanox NICs (CX-4 and CX-5).
In order to access the NIC in userspace, VPP can sometimes use DPDK (Data Plane Development Toolkit).
Anyone familiar with DPDK would know that DPDK consists of different Userspace Poll Mode Drivers (PMD). The manner in which they operate is: The control of the Interface is taken away from Kernel using Kernel Modules like igb_uio, vfio_pci (although you can still do Flow Bifurcation).
Mellanox Cards, however, operate in what is popularly known as the Bifurcated Driver Model, i.e. control of the NIC is still with the Kernel but Userspace PMD can directly access data plane.
In order to achieve this, you need a bunch of Kernel Modules and Verb libraries on top of which the PMD is built (libibverbs, libmlnx4/5). All of these are provided by Mellanox MOFED on their website; Mellanox OpenFabrics Enterprise Distribution.
This is usually the price you have to pay when you are the new kid in town, although you can find binary versions for older Distributions along with the older versions of MOFED.
In order to see all the possible versions for download, scroll down and see the downloads tab on the Mellanox product website.
For the purpose of this blog post, I have chosen MOFED version 4.4-2.0.7.0 with OS Distribution (Ubuntu 18.04 - bionic).
At the time of writing the post, I see two architectures (x86_64, ppc64). It doesn't matter which architecture we choose, as we plan to download the source, compile and install it on Arm (aarch64).
$ wget http://www.mellanox.com/downloads/ofed/MLNX_OFED-4.4-2.0.7.0/MLNX_OFED_SRC-debian-4.4-2.0.7.0.tgz
$ tar xzvf MLNX_OFED_SRC-debian-4.4-2.0.7.0.tgz $ cd MLNX_OFED_SRC-4.4-2.0.7.0/ $ sudo ./install.pl --dpdk --upstream-libs
The installation is likely going to fail with errors suggesting missing dependencies. Please resolve it accordingly.
Once the installation is done please restart the drivers:
$ sudo service openibd restart
You should see the openibd service running along with the Kernel Modules loaded:
$ sudo service openibd status ● openibd.service - openibd - configure Mellanox devices Loaded: loaded (/lib/systemd/system/openibd.service; enabled; vendor preset: enabled) Active: active (exited) since Mon 2018-09-24 14:39:34 CDT; 22h ago Docs: file:/etc/infiniband/openib.conf Process: 956 ExecStart=/etc/init.d/openibd start bootid=1cf2538f8bae4917a21a7cbf7fcf8a8a (code=exited, status=0/SUCCESS) Main PID: 956 (code=exited, status=0/SUCCESS) Tasks: 0 (limit: 7065) CGroup: /system.slice/openibd.service
$ lsmod | grep mlx mlx5_fpga_tools 24576 0 mlx4_en 180224 0 mlx4_ib 258048 0 mlx4_core 430080 2 mlx4_en,mlx4_ib mlx5_ib 335872 0 ib_core 364544 10 ib_cm,rdma_cm,ib_umad,ib_uverbs,ib_ipoib,iw_cm,mlx5_ib,ib_ucm,rdma_ucm,mlx4_ib mlx5_core 1064960 2 mlx5_fpga_tools,mlx5_ib mlxfw 24576 1 mlx5_core devlink 53248 4 mlx4_en,mlx5_core,mlx4_core,mlx4_ib mlx_compat 16384 15 ib_cm,rdma_cm,ib_umad,ib_core,mlx5_fpga_tools,ib_uverbs,mlx4_en,ib_ipoib,mlx5_core,iw_cm,mlx5_ib,mlx4_core,ib_ucm,rdma_ucm,mlx4_ib ptp 28672 3 i40e,mlx4_en,mlx5_core
This post described source compilation and installation. Ideally it's a architecture independent method. You go with this method only if you don't find the MOFED libraries in binary form, or you want to compile MOFED with a specific compiler (gcc-8 in my case).
Watch out for my next chapter, which I will link to below once live.
Chapter 2: VPP on ARM: Simple IPv4 forwarding Test case using Mellanox NICs(PFs and VFs)