PerfSonar

Perfsonar is a toolset for active performance measurements on high speed networks. You will find more detailed information at https://www.perfsonar.net/

From their webpage: "PerfSONAR is a network measurement toolkit designed to provide federated coverage of paths, and help to establish end-to-end usage expectations. There are 1000s of perfSONAR instances deployed world wide, many of which are available for open testing of key measures of network performance. This global infrastructure helps to identify and isolate problems as they happen, making the role of supporting network users easier for engineering teams, and increasing productivity when utilizing network resources. perfSONAR has been developed through an international collaboration led by Internet2, ESnet, Indiana University, and GEANT."

Current implementations use a server directly connected to the device that represents the measurement point. Using the switchdev environment it should be possible to install perfsonar directly onto the device and by this avoid having to use external hardware. This might also allow to install perfsonar in an ad hoc mod when needing additional measurement points for better debugging of an issue.

We will focus our experiments on the perfsonar tools, not the backend / frontend of the full suite, as it does not make sense to have this functionality on the switch hardware.

Our proof of concept install will look in particular at three question:

1. Does the install run into any problems not seen on other platforms? Additionally, we will provide the information needed to install it on the switchdev environment.

2. What is the performance we can achieve, taking into account that the connection between CPU and dataplane might be a bottleneck.

3. Do some test measurements against public test points to verify correct functionality

In this example we will use a switchdev environment to install perfsonar directly on a 100G switch. As a switchdev environment is a standard Linux environment with a driver that exposes the 100G network ports / the high speed dataplane ASIC as a standard linux ethernet interface, there should be no special treatment (like having to compile source code on the platform) needed. We will use the standard dfn/yum install process using the OS rpm repositories. We are following the install procedure as published at the perfsonar website:

https://www.perfsonar.net/deploy/installation-and-configuration/

First, we need to install some prerequisites, like epel and the Internet2 repository that holds the perfsonar rpms:

We additionally need to edit the Internet2.repo file. The mirrorlist entry does not work, so we comment it out, and instead we add a direct entry for the baseurl. You can see the working configuration below:

Now we can install and use the perfsonar tools:

As we see this looks exactly as installs on a standard Linux machine looks, there is nothing special happening in regards to us using the hardware accelerated ASIC.

Now we are able to run perfsonar tests against other machines in the net as needed, for example we run bwctl against albq-pt1.es, a publicily accessible test endpoint provided by esnet.

bwctl uses iperf(3) for testing by default. When using iperf3 on one of the dataplane ports there is an additional bottleneck that influences the performance we can achieve. The interface between the CPU where the iperf packets are generated and the dataplane ASIC is a PCI3 bus not providing the full dataplane linerate. Also the CPU is not fast enough to 100G data streams. We did some example testing of the maximum transfer speeds that we can expect in this environment. Without any tuning, iperf3 was able to transfer about 5G/s to a directly connected iperf3 process on a server. When setting the MTU to 9000, we were able to get around 8G/s bandwidth in our setup. So testing on 10G links seems to be reasonably achieveable on this platform, but higher speeds will be challenging. On this platform the vendor is currently working to enable dataplane side generation of test and debugging packets, which might provide a way to do higher speed tests in the future avoiding the CPU and CPU to dataplane connection bottlenecks. As soon as this functionallity is available we will do some more in depth testing of what performance can be achieved.