DataCV Challenge @ CVPR 2023
Overview
The 1st DataCV Challenge is held in conjunction with the CVPR 2023 Visual Dataset Understanding workshop. It is the first of its kind in the community, where we focus on evaluating model performance under various test sets without ground truths. The competition is held via CodaLab and includes two phases. Go to this website to participate in the challenge.
Important Dates and News
Monday, January 30th, 2023: Training and validation data release.
Thursday, March 4th , 2023: Test data release. online evaluation available.
Monday, March 7th , 2023: Result submission closed.
Competition
Task. Label-free model evaluation is the competition task. It is different from standard evaluation that calculates model accuracy based on model outputs and corresponding test labels. Label-free model evaluation (AutoEval), on the other hand, has no access to test labels. In this competition, the participant needs to design a method that can estimate the model accuracies on test sets without ground truths.
Evaluation website. We use CodaLab as the competition platform. The URL is https://codalab.lisn.upsaclay.fr/competitions/10221
Datasets. We use classification datasets CIFAR-10 label space. We use existing CIFAR-10 variants for training and validation and provide new CIFAR-10 variants and newly collected sets for testing. In training, teams will be required to learn to predict model accuracy on train/validation sets, and in testing, need to predict classification accuracy numbers on the test sets.
Ethical considerations. Because we focus on classification, and there is no “human” class, the ethics is mostly fine. We will double check the images to remove those containing humans and follow the practice in (Asano et al., NeurIPS 2021) to validate the copyright before distributing the test sets.
Organizers
Australian National University
Australian National University
Australian National University
Xingjian Leng
Australian National University
Yasong Dai
Australian National University