Downloads‎ > ‎

RoCE

Download

The Soft RoCE distribution is available now as a specially patched OFED-1.5.2 distribution.  Users familiar with the installation and configuration of OFED software will find this easy to use.

Download Soft RoCE OFED Distribution

Installation

If you are not already experienced with installing and configuring OFED distributions, you should probably visit the Open Fabrics Alliance and read their installation documentation.  The Soft RoCE distribution contains the entire OFED-1.5.2 distribution, with the addition of the Soft RoCE code.

Installation of the OFED-1.5.2-rxe distribution works exactly the same as a “standard” OFED distribution.  Installation can be accomplished interactively, via the “install.pl” program, or automatically via “install.pl –c ofed.conf”.  The required new components are “librxe” and “ofa-kernel.” The latter is not new, but in our "rxe" version of the OFED distribution it includes the rxe/Soft RoCE kernel module.

Configuration Instructions

Once you have installed the OFED-1.5.2-rxe distribution, Soft RoCE (aka rxe) configuration tasks are handled via the “rxe_cfg” program.  Following is a summary of basic functions.

Load kernel module and configure previously added persistent instances

rxe_cfg start

Add an instance on Ethernet interface “eth4”

rxe_cfg add eth4

Display the status of all rxe instances:

rxe_cfg status (or just rxe_cfg – status is the default)

Remove a rxe instance:

rxe_cfg remove rxe0

Set the rxe MTU

To set the rxe MTU for all rxe instances to 4K, type the following:

rxe_cfg mtu 4096

To set the mtu for a single rxe instance:

rxe_cfg mtu rxe0 2048

Notes about MTU settings

The Soft RoCE specification allows power-of-two MTUs from 512 to 4096, although the currently available Hard RoCE implementation only supports up to 2K. Setting the MTU as described above will only take effect if the underlying Ethernet interface has a large enough MTU. For example, to support an MTU of 4096, the Ethernet MTU must be at least 4176 (80 bytes larger). If the Soft RoCE MTU is too large to fit within the Ethernet MTU, the Soft RoCE software will use the largest MTU that does fit. For example, if the Soft RoCE MTU is 4096, and the Ethernet MTU is 4096, Soft RoCE will use a 2048 byte MTU. If you subsequently increase the Ethernet MTU to 9000 (or any value greater than or equal to 4176), the Soft RoCE MTU will automatically switch to 4096. If you use the "-f" option, as in "rxe_cfg -f mtu rxe0 4096", the relevant Ethernet MTU will be changed, if necessary, to 4176 (but will not be altered if it is already larger than 4176).

A note about persistent instances

The rxe driver, as presently shipped, is not automatically loaded after boot.  However, when you start up the subsystem (rxe_cfg start), the driver will be loaded and any previously configured instances will be recreated.

By default, “rxe_cfg add” is persistent.  The file /var/rxe/rxe is created, and contains the list of Ethernet interfaces for which rxe instances should be created.  You can remove this file if you want to delete existing records of persistent rxe instances.

You can also add the “-n” option to the rxe_cfg add and remove subcommands.  “rxe_cfg –n add eth3” will add a rxe instance on eth3 without adding a persistence entry in /var/rxe/rxe.  “rxe_cfg –n remove rxe0” will remove the rxe0 instance without removing it from the /var/rxe/rxe persistence file.

Source Code

The Soft RoCE / RXE source code can be browsed and searched on the OFED Cross Reference Server, hosted by System Fabric Works:

OFED-1.5.1-rxe Cross Reference

Source code is contained in the OFED-1.5.2-rxe tarball, in source rpm format.

Previous Release

If you need to download another copy of the previous release, it is still available on this site. Please upgrade to the new release as soon as possible.

Download Soft RoCE OFED Distribution 1.5.1 (OBSOLETE VERSION)

Further Information

For more information email info@systemfabricworks.com