Challenge
Challenge
The workshop consists of a map-based visual re-localization challenge for autonomous driving. The goal of the challenge is to estimate the 6DOF relative pose between individual images from a reference sequence to a target sequence. The winner of the challenge will receive 1000 USD, and the second place will receive 500 USD.
Evaluation
For the challenge, we will evaluate the translational accuracy of a method. For each pair of the re-localization file, we evaluate the translational error between the estimated and the ground truth pose, respectively.
Dataset
The sequences for the challenge are based on the 4Seasons dataset.
The following sequences will be used for the challenge:
Reference: reference_sequence
Training: training_sequence
Validation: validation_sequence
Test: test_sequence0, test_sequence1
See this GitHub repository for the README.
OPTIONAL: MLAD-pcviewer for visualizing the data.
Submission
We strongly encourage all participants to use only the reference and training sequences for finding parameters and report results on the provided test re-localization files to enable a meaningful comparison.
Please submit your results as a single .zip file. The results for each re-localization file must be stored in a separate .txt file in the archive's root folder. The file name must be exactly like the provided ones (case sensitive).
The file format should be the same as the ground truth file, which is a text-file containing one re-localization pair per line. Each line must contain 9 values:
source_kf target_kf t_x t_y t_z q_x q_y q_z q_w
In order to submit your results, you will have to send an email with the single .zip file which contains the re-localization results for the 6 provided test files. (3x test0, 3x test1)
Send to: mlad-eccv2020@googlegroups.com.
You have to decide whether the results will be publicly visible on the leaderboard.
Rules
If you want to participate in the challenge, you need to describe your approach in a publication. This can be in form of a published conference or journal paper (incl. ECCV2020), a pre-print paper on arXiv or a regular submission to our workshop.
You are not allowed to hand in results obtained with open-source projects without any added novelty.
Both, the winner, and the second place of the challenge must attend the workshop and present their approach (one member is sufficient)
The use of additional training data or the use of pre-trained models is allowed. However, one needs to specify which data was used to train the models. Using the test sequence for training is not allowed.
Deadlines
Challenge deadline: August 16, 2020
For questions, please send an email to: mlad-eccv2020@googlegroups.com.
Baselines
We provide the following baseline results for the re-localization challenge. The re-localization results are reported as the percentage of re-localization candidates (between reference and target sequence) which where localized within the given translational thresholds.
Thresholds: 0.1m, 0.2m, 0.5m
test_sequence0 (recording_2020-03-24_17-45-31)
SuperPoint/SuperGlue + PnP + RANSAC: 21.2 / 33.9 / 60.0
R2D2 + PnP + RANSAC: 21.5 / 33.1 / 53.0
D2-Net + PnP + RANSAC: 12.5 / 29.3 / 56.7
SuperPoint + PnP + RANSAC: 15.5 / 27.5 / 47.5
test_sequence1 (recording_2020-04-23_19-37-00)
SuperPoint/SuperGlue + PnP + RANSAC: 12.4 / 26.5 / 54.4
R2D2 + PnP + RANSAC: 12.3 / 23.7 / 42.0
D2-Net + PnP + RANSAC: 7.5 / 21.4 / 47.7
SuperPoint + PnP + RANSAC: 9.0 / 19.4 / 36.4
Leaderboard
Localization thresholds: 0.1m, 0.2m, 0.5m