Track 1: Generic Event Boundary Detection Challenge

This track aims at encouraging our participants to advance SOTA GEBD system.
The competition is based on our Kinetics-GEBD test set only.
For training data & supervision, there are two different scenarios (i.e. sub-track 1.1 and 1.2); participants can attend only 1 sub-track or both:
- Sub-track 1.1: no constraint of additional supervision for training upstream models and additional training video data (our previous track 1 setting)
- Sub-track 1.2: cannot use additional supervision for training upstream models or additional training video data (so that is friendly to the team of limited compute resource)
Participants will be ranked within each sub-track and the best performer of each sub-track will be awarded 1K USD.
Top 3 winners in each sub-track will be mentioned at the workshop and formally recognized.
Note that if you participate in both Track 1 and Track 2, duplicate submissions or submissions with significant content overlap are NOT allowed.

For more details, please refer to our Challenge White Paper. For any questions about codalab, please post in its forum.

Evaluation Protocol

We use Relative Distance (Rel.Dis) to determine the correctness of each prediction. Rel.Dis is the error between detected and ground-truth timestamps, divided by the length of the whole video. Given a fixed threshold for Rel.Dis, we can determine whether a detection is correct (i.e. <=threshold) or incorrect (i.e. >threshold), then compute precision, recall and F1 score on the whole dataset.
Note that for each video, we multiple raters who make annotations independently. We (1) compare our detection result with each rater’s annotation and (2) select GT for this video as the rater’s annotation which leads to the best F1 score among all raters.
The official metric used in this task is F1@5%, which is defined as the F1 score computed with threshold 5%.
We evaluate performance on Kinetics-GEBD Test Set in this competition, video list can be found here.
We have provided starter baseline code for Track 1 at Github

Baselines

Table: Performance on Kinetics-GEBD for various supervised and unsuperivsed GEBD methods.

Submission Format

To submit your results to the leaderboard, you must construct a submission zip file containing two files: submit_val.pkl, submit_test.pkl for validation data and test data, respectively. Use the following command to generate the submission file.

zip -r test_submit.zip submit_val.pkl, submit_test.pkl

The pickle format is composed of a dictionary containing keys with video names and values with boundary lists. For example,

{'6Tz5xfnFl4c': [5.9, 9.4], 'zJki61RMxcg': [0.1, 0.4, 0.6, 1.5, 2.7]}.

If you have a question about the submission format or if you are still having problems with your submission, please create a topic in the competition forum (rather than contact the organizers directly by e-mail) and we will answer it as soon as possible.

Submission Policy

There are two phases for the challenge:

Development phase (corresponding to the validation split) will be launched on March 25. It is used to validate your method and tune the hyper-parameters. Submission to this phase will not be taken into account for the challenge. Submission limits: 100 submissions in total, no more than 2 per day.
Final phase (corresponding to the test split) will be launched on May 1. The scores of the development phase will not be automatically copied over. Therefore, you must re-submit your solution to the final phase if you want to be considered in the final leaderboard. Submission limits: 10 submissions in total, no more than 1 per day.

Report Format

Use CVPR style (double column) in the form of 3-6 pages or NeurIPS style (single column) in the form of 6-10 pages inclusive of any references. Please explain clearly what data, supervision, pre-trained models you have used so that we can make sure your results are comparable to others.
Please include your github link in the report. The top 3 winners in each track are required to release their codebases and final models so that other people can reproduce in the future. Please contact us if you have any questions.

Report Submission Portal

For report submission, please send an email to loveu.cvpr21@gmail.com.

Format of email subject: “YourName-Submission-LOVEU21-Track1.X”;
Attach your technical report and other relevant materials in the email;
Include your Codalab account (registered email) and username for our challenge in the email. Include meta info like team members, institution, etc.

For more details, please refer to our Challenge White Paper.

Timeline

March 25, 2021 (11:59PM Pacific Time): evaluation server open for the val set
May 01, 2021 (11:59PM Pacific Time): evaluation server open for the test set, with leaderboard available
~~May 23, 2021~~ ~~(11:59PM Pacific Time): hide leaderboard~~
Jun 01, 2021 (11:59PM Pacific Time): evaluation server close
Jun 08, 2021 (11:59PM Pacific Time): report submission due

Communication & QA

- For Challenge Policies: CodaLab Challenge Forum
- For starter baseline code: Github