The Soft RoCE distribution is available now as a specially patched OFED-1.5.2 distribution. Users familiar with the installation and configuration of OFED software will find this easy to use.
If you are not already experienced with installing and configuring OFED distributions, you should probably visit the Open Fabrics Alliance and read their installation documentation. The Soft RoCE distribution contains the entire OFED-1.5.2 distribution, with the addition of the Soft RoCE code.
Installation of the OFED-1.5.2-rxe distribution works exactly the same as a “standard” OFED distribution. Installation can be accomplished interactively, via the “install.pl” program, or automatically via “install.pl –c ofed.conf”. The required new components are “librxe” and “ofa-kernel.” The latter is not new, but in our "rxe" version of the OFED distribution it includes the rxe/Soft RoCE kernel module.
Once you have installed the OFED-1.5.2-rxe distribution, Soft RoCE (aka rxe) configuration tasks are handled via the “rxe_cfg” program. Following is a summary of basic functions.
Load kernel module and configure previously added persistent instances
Add an instance on Ethernet interface “eth4”
Display the status of all rxe instances:
Remove a rxe instance:
Set the rxe MTU
To set the rxe MTU for all rxe instances to 4K, type the following:
To set the mtu for a single rxe instance:
Notes about MTU settings
The Soft RoCE specification allows power-of-two MTUs from 512 to 4096, although the currently available Hard RoCE implementation only supports up to 2K. Setting the MTU as described above will only take effect if the underlying Ethernet interface has a large enough MTU. For example, to support an MTU of 4096, the Ethernet MTU must be at least 4176 (80 bytes larger). If the Soft RoCE MTU is too large to fit within the Ethernet MTU, the Soft RoCE software will use the largest MTU that does fit. For example, if the Soft RoCE MTU is 4096, and the Ethernet MTU is 4096, Soft RoCE will use a 2048 byte MTU. If you subsequently increase the Ethernet MTU to 9000 (or any value greater than or equal to 4176), the Soft RoCE MTU will automatically switch to 4096. If you use the "-f" option, as in "rxe_cfg -f mtu rxe0 4096", the relevant Ethernet MTU will be changed, if necessary, to 4176 (but will not be altered if it is already larger than 4176).
A note about persistent instances
The rxe driver, as presently shipped, is not automatically loaded after boot. However, when you start up the subsystem (rxe_cfg start), the driver will be loaded and any previously configured instances will be recreated.
By default, “rxe_cfg add” is persistent. The file /var/rxe/rxe is created, and contains the list of Ethernet interfaces for which rxe instances should be created. You can remove this file if you want to delete existing records of persistent rxe instances.
You can also add the “-n” option to the rxe_cfg add and remove subcommands. “rxe_cfg –n add eth3” will add a rxe instance on eth3 without adding a persistence entry in /var/rxe/rxe. “rxe_cfg –n remove rxe0” will remove the rxe0 instance without removing it from the /var/rxe/rxe persistence file.
The Soft RoCE / RXE source code can be browsed and searched on the OFED Cross Reference Server, hosted by System Fabric Works:
Source code is contained in the OFED-1.5.2-rxe tarball, in source rpm format.
If you need to download another copy of the previous release, it is still available on this site. Please upgrade to the new release as soon as possible.
For more information email firstname.lastname@example.org