Benchmark Leaderboard

Benchmark Overview

TEMPO-VINE supports SLAM, sensor fusion, and place recognition evaluation on real vineyard trajectories all over the year. Benchmarking state-of-the-art solutions is one of the final research goals of the dataset.

An extended benchmark study is currently under development as future work of the dataset, to five insights about what type of localization, mapping and representation learning techniques better perform on the real field. The project aims to build a SLAM leaderboard and a Place Recognition leaderboard, instruction to test your own method on the dataset and access the ranking will be released soon.

SLAM Leaderboard

Currently, SLAM algorithms are evaluated using the following metrics:

Absolute Trajectory Error (ATE) [m]
Relative Pose Error - translation (RPE-t) [m]
Relative Pose Error - rotation (RPE-r) (unitless)

The following algorithms have included in the evaluation so far:

RTAB-Map (RGB-D Camera) [1]
Fast-LIO (VelodyneVLP-16 and Livox MID360) [2]
LIO-SAM (Livox MID360) [3]

SLAM Benchmark table

Results of SLAM in Winter (March) and Summer (July)in both Trellis and Pergola Vineyard

We evaluated RGB-D and LiDAR SLAM methods using RTAB-Map, Fast-LIO and LIO-SAM across winter and summer conditions in both trellis and pergola vineyards.

Results show that seasonal vegetation and row geometry can cause significant trajectory drift and occasional divergence.
LIO-SAM with VLP-16 is the most robust keeping ATE below 4m.

Vineyard (Trellis)

Trajectories of SLAM algorithms on the vineyard (Trellis) in Winter (March - Run 02).

Pergola

Trajectories of SLAM algorithms on the vineyard (pergola) in Winter (March - Run 02).

Place Recognition Leaderboard

Scan Context [4] (LPR)

Place Recognition Quantitative Analysis with SCAN CONTEXT [4]

Metrics: RECALL@5 AND PRECISION@5

Settings:Different LiDARS, Seasons and Runs.

Learning Methods Baselines [5,6] (LPR)

Place Recognition Analysis of learned method of LiDAR Place Recognition with both LiDARS.

Metrics: RECALL@1% AND RECALL@1

Baselines: PointNetVLAD [5] and MinkLoc3Dv2 [6].

Instructions

To access the TEMPO-VINE benchmark leaderboard, instructions will be given soon.

References

M. Labb´e and F. Michaud, “Rtab-map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation,” Journal of Field Robotics, vol. 36, no. 2, pp. 416–446, 2019. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/rob.21831
Xu, W., Cai, Y., He, D., Lin, J., & Zhang, F. (2022). Fast-lio2: Fast direct lidar-inertial odometry. IEEE Transactions on Robotics, 38(4), 2053-2073.
T. Shan, B. Englot, D. Meyers, W. Wang, C. Ratti, and R. Daniela, “Lio-sam: Tightly-coupled lidar inertial odometry via smoothing and mapping,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020, pp. 5135–5142.
G. Kim and A. Kim, “Scan context: Egocentric spatial descriptor for place recognition within 3d point cloud map,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2018, pp. 4802–4809.
M. A. Uy and G. H. Lee, “PointNetVLAD: Deep point cloud based retrieval for large-scale place recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4470–4479.
J. Komorowski, “Improving point cloud based place recognition with ranking-based loss and large batch training,” in 2022 26th international conference on pattern recognition (ICPR). IEEE, 2022, pp. 3699–3705

Page updated

Google Sites

Report abuse