Switchdev on Fedora

Switchdev on the Mellanox platform

Switchdev is a linux project to directly support networking ASICs in a standard linux environment.

It provides in kernel support for the ASIC, so existing tools like ifconfig, ethtool, ip link just work, and are used to configure the forwarding hardware.

Mellanox provides a switchdev driver that they include into a fedora remix iso that can easily be installed.

We will use this platform to take a look what can be done in this new model.

As for several other examples we will une ONIE to install the switch OS.

It is important to mention here that besides the OS there also exists a switch firmware in the forwarding ASIC that must be the correct / current version.

If this is not the case Mellanox provides images and tools to update the firmware. As this is not a naive Linux functionality the tools ofr this must be provided by the hardware vendor.

The ONIE install is a bit more complex for this system, some additional steps are necessary:

1. We need to extract the ISO image onto the webroot of our installation server

2. The kickstart file needs to be in a directory named ks and point to the installation server

After preparing this, and starting the installation it ran through without any problem, and we were able to log into the device.

Running ifconfig only shows us the management ports at the moment. We need to update the kernel to have the correct version to work with our firmware. Firmware in this case refers to the microcode on the linecard / ASIC itself. The management ports are "standard" ethernet ports running with the standard linux drivers from the OS perspective. The dataplane ports are special ethernet interfaces that need the "advanced" switchdev driver to work. The switchdev driver needs to talk directly to the hardware accelerated ports using an API exposed by the linecard firmware code. This API is called by the kernel, which means kernel version and firmware need to be in sync for the dataplane ports to work. Here is the state after installing, we see only the management interfaces:

To get the updates, we need to configure the networking accordingly to reach the fedora rpm servers.

If not done automatically through dhcp, you need to set ip, gateway and DNS servers manually as you do on the Linux distribution used. The distribution provided for the Mellanox is based on Fedora.

To update the kernel (remember, switchdev is a kernel driver), we use dnf. Dnf is a newer replacement for yum which works much the same way:

after a reboot the new driver is active, and we can see all the interfaces now. If this would not be the case we would need to update the linecard firmware too. As this is not a standard Linux functionallity, we would need to use a switch verndor specific tool, and download the most current firmware from the vendor site. The whole process is described at the Mellanox switchdev WIKI in detail.

Here we see the output of ifconfig after things have been updated, and the dataplane interfaces are active:

We can now use standard Linux commands / tools to configure the 100G dataplane ports just as the standard management interfaces. First, let's look at an example how to get hardware related information like temperature or fan speeds:

Hardware monitoring:

Find the RPM of a fan:

Find the current ASIC temperature:

Another way is using the lm_sensors package. As often in Linux, there are several different ways to access the same information:

We can use ethtool to get port state information, again ethtool works the same way as we are used to:

We can use ip link to change port states and get link information:

Now we look at configuring some basic networking functionallity on the dataplane ports. In this example we use ip link to configure a L2 bridge:

These are just some examples to demonstrate the way we can now configure standard networking through linux tools. This as an approach that has a certain learning curve in how to utilize linux commands to configure networking behavior, but this is the same as learning a new CLI when switching to a new networking vendor. The nice thing is that the linux commands will work the same with different vendor devices supporting switchdev, so there is no other CLI to learn.

In addition to this, we are also able to install standard Linux software this way.

As an example, we will use this platform to install perfsonar tools directly on the device. You will find the information regarding the perfsonar installation in a later blog post.