Winners:

We are pleased to announce the winners of the 2024 Radar in Robotics competition:
CFEAR++ (S. Li, Y. Chen, Z. Hong and L. Hu)

The top three submissions were:
CFEAR++ (S. Li, Y. Chen, Z. Hong and L. Hu) with 0.51% translation error

CFEAR (D. Adolfsson, Max. Hilger) with 0.61% translation error

STEAM-RIO++ (K. Burnett, A. P. Schoellig, T. D. Barfoot) with 0.62% translation error

Our detailed leaderboard is available at: https://www.boreas.utias.utoronto.ca/#/leaderboard
(see the SE(2) odometry leaderboard)

Competition:

The competition involves developing and benchmarking radar odometry using the Navtech radar data collected in the Boreas dataset. The Boreas dataset has an IJRR paper associated with it which can be found here. Submissions and leaderboard will be located at boreas.utias.utoronto.ca

Restrictions:

• Submissions must run “online”, which means that the odometry estimates cannot use future information in an acausal manner, batch optimization over the entire trajectory is not permitted.

• Submissions must be real-time, capable of estimating odometry for each new radar frame in under 250ms (the framerate of the Navtech radar). Participants are free to use whatever compute hardware they think is appropriate to achieve this.

• Participants may NOT use lidar or camera data; however, participants are free to use IMU and/or wheel encoder data.

The top 3 submissions will be invited to deliver a 5-minute talk about their submission at the workshop. This will likely be conducted over zoom. However, if any of these top three participants is already planning to attend ICRA 2024 in person, we may be able to make some talks in-person as part of the workshop. We encourage participants to open-source their implementations after the competition for the benefit of the community. 

Downloading the dataset:

First, note that there are training sequences that include groundtruth poses and test sequences where we have withheld the groundtruth poses for the sake of the competition. The split can be found here.

The dataset is stored on Amazon Web Services, S3. You will need to install the AWS command line interface, instructions here. You can download our dataset without an AWS account by appending --no-sign-request after your CLI commands. Here’s an example command for downloading a sequence from the training split:

aws s3 sync s3://boreas/boreas-2020-11-26-13-58 </path/to/boreas/>boreas-2020-11-26-13-58 --exclude "*" --include "radar/*" --include "applanix/*" --include "calib/*"

• The extrinsic calibration between the sensors may change between sequences, so please use the extrinsics contained in each sequence for that corresponding sequence.

• Radar data collected before UNIX epoch time 1632182400 has a resolution of 0.0596m, afterwards it has a resolution of 0.04381.

Submitting to the benchmark:

We will benchmark submissions using the KITTI odometry metrics which estimate the drift as a % for translation and deg/m for rotation.

General submission instructions can be found here. Note that we provide a submission_checker.py in our git repo, please use this to check for any obvious submission format problems before uploading to the website. In your yaml file, please use the same email that you used to sign in to our submission form on the website. You should receive an email with your submission results. You will also have the option to keep your submission private or make it public for others to view on the website.

More detailed submission instructions for the odometry benchmark can be found here. Please focus on the “2D” instructions part, and ignore instructions for the 3D benchmark, you can also ignore the section about “interpolation”.

You can only submit to the benchmark once for the competition. Submitting multiple times to fine-tune your results to the test set is not permitted. The ground truth poses for the test sequences have been held out. However the remaining training sequences can be used to develop and validate your approach before running on the test sequences and submitting to the benchmark.

Please include “ICRA2024” at the beginning of your method name, as in the yaml file which is a part of the zip file to be submitted to the benchmark. We will use this tag to separate regular submissions from competition submissions and to create a competition-specific leaderboard.

Additional submission requirements:

Participants are also required to submit a 2-page+references write-up on their approach. Format is dual-column single-space, IEEE conference format.

Submission rankings:

Submissions will be ranked based on their translational drift [%] on the benchmark. The submission with the lowest translational drift will win.

If you encounter any problems:

If you have any problems, please open a github issue at https://github.com/utiasASRL/pyboreas/issues

We can also be reached at boreas@robotics.utias.utoronto.ca

Disclaimer:

We reserve the right to change the rules of the competition at any point and to disqualify individuals who break any rules mentioned above or cheat in any other manner or who violate the IEEE code of ethics.