Chapter 1: VPP on Arm: Installing MLNX_OFED, for using Mellanox Cards

This is Chapter One of VPP on Arm (VPP is one of the projects from - Fast Data Input/Output collection of projects).

What is VPP?

VPP stands for Vector Packet Processing, it is the open source version of Cisco's Vector Packet Processing Software. VPP is a framework/toolkit which provides the functionality of Router/Switch in userspace. You can read more on What is VPP?

Even before we get started with VPP and running external traffic we need a NIC. The first ones I encountered were Mellanox NICs (CX-4 and CX-5).

In order to access the NIC in userspace, VPP can sometimes use DPDK (Data Plane Development Toolkit).

The Curious Case of Mellanox Cards

Anyone familiar with DPDK would know that DPDK consists of different Userspace Poll Mode Drivers (PMD). The manner in which they operate is: The control of the Interface is taken away from Kernel using Kernel Modules like igb_uio, vfio_pci (although you can still do Flow Bifurcation).

Mellanox Cards, however, operate in what is popularly known as the Bifurcated Driver Model, i.e. control of the NIC is still with the Kernel but Userspace PMD can directly access data plane.

In order to achieve this, you need a bunch of Kernel Modules and Verb libraries on top of which the PMD is built (libibverbs, libmlnx4/5). All of these are provided by Mellanox MOFED on their website; Mellanox OpenFabrics Enterprise Distribution.

Installing from Source

This is usually the price you have to pay when you are the new kid in town, although you can find binary versions for older Distributions along with the older versions of MOFED.

In order to see all the possible versions for download, scroll down and see the downloads tab on the Mellanox product website.

For the purpose of this blog post, I have chosen MOFED version 4.4- with OS Distribution (Ubuntu 18.04 - bionic).

At the time of writing the post, I see two architectures (x86_64, ppc64). It doesn't matter which architecture we choose, as we plan to download the source, compile and install it on Arm (aarch64).

Download Source

$ wget

Untar and Install

$ tar xzvf MLNX_OFED_SRC-debian-4.4-
$ cd MLNX_OFED_SRC-4.4-
$ sudo ./ --dpdk --upstream-libs

The installation is likely going to fail with errors suggesting missing dependencies. Please resolve it accordingly.  

Restart Drivers

Once the installation is done please restart the drivers:

$ sudo service openibd restart


You should see the openibd service running along with the Kernel Modules loaded:

$ sudo service openibd status
● openibd.service - openibd - configure Mellanox devices
   Loaded: loaded (/lib/systemd/system/openibd.service; enabled; vendor preset: enabled)
   Active: active (exited) since Mon 2018-09-24 14:39:34 CDT; 22h ago
     Docs: file:/etc/infiniband/openib.conf
  Process: 956 ExecStart=/etc/init.d/openibd start bootid=1cf2538f8bae4917a21a7cbf7fcf8a8a (code=exited, status=0/SUCCESS)
 Main PID: 956 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 7065)
   CGroup: /system.slice/openibd.service

$ lsmod | grep mlx
mlx5_fpga_tools        24576  0
mlx4_en               180224  0
mlx4_ib               258048  0
mlx4_core             430080  2 mlx4_en,mlx4_ib
mlx5_ib               335872  0
ib_core               364544  10 ib_cm,rdma_cm,ib_umad,ib_uverbs,ib_ipoib,iw_cm,mlx5_ib,ib_ucm,rdma_ucm,mlx4_ib
mlx5_core            1064960  2 mlx5_fpga_tools,mlx5_ib
mlxfw                  24576  1 mlx5_core
devlink                53248  4 mlx4_en,mlx5_core,mlx4_core,mlx4_ib
mlx_compat             16384  15 ib_cm,rdma_cm,ib_umad,ib_core,mlx5_fpga_tools,ib_uverbs,mlx4_en,ib_ipoib,mlx5_core,iw_cm,mlx5_ib,mlx4_core,ib_ucm,rdma_ucm,mlx4_ib
ptp                    28672  3 i40e,mlx4_en,mlx5_core

Summary and peek into the future

This post described source compilation and installation. Ideally it's a architecture independent method. You go with this method only if you don't find the MOFED libraries in binary form, or you want to compile MOFED with a specific compiler (gcc-8 in my case).

Watch out for my next chapter, which I will link to below once live.

Chapter 2: VPP on ARM: Simple IPv4 forwarding Test case using Mellanox NICs(PFs and VFs)