Crowd Bias Challenge

Register now to take part in the Crowd Bias Challenge and contest through Kaggle. Amazon will provide $250 in AWS credits to the first 15 teams who commit to participating in the challenge and to using AWS Ground Truth as part of their participation.

We invite you to participate in the CrowdBias challenge organised with CSCW 2021 Workshop - Investigating and Mitigating Biases in Crowdsourced Data. Register here to take part in teams of up to 4 members.

The broad task goal is to collect correct, unbiased labels for an annotation task in which each item to annotate has a true answer, known to the organizers. Teams are free to pursue whatever research designs they see fit, such as interface design, annotation workflows, assignment methods, aggregation approaches, etc. Teams will be evaluated based on a combination of accuracy and fairness metrics. Successful teams will also be invited to present their approach during the workshop.

Participants may use any crowdsourcing platform of their choice to collect human annotations.

For any teams interested in using Amazon’s Sagemaker Ground Truth (GT) or Augmented Artificial Intelligence (A2I) service to collect human annotations, Amazon will provide $250 in AWS credits to the first 15 teams who commit to participating in the challenge and to using GT as part of their participation. AWS credits cannot be used with Amazon Mechanical Turk.

Challenge participants are also invited to attend our workshop at CSCW. Please note that you will have to register for both the CSCW conference and the workshop in order to attend (per the registration link above, one can see that prices are fairly inexpensive as a virtual conference, and further discounted with early registration through October 1, 2021).

Step 1 - Follow this link to register for the CrowdBias challenge and receive AWS credits

https://forms.gle/m2esEZegkDYyQ8jH7

Step 2 - Optional - Register for CSCW Conference and Workshop (use the code “MIBI09”)

https://cscw.acm.org/2021/registration/

Challenge Details

In this challenge, you need to collect apparent age labels for a given set of face images. The challenge is available in Kaggle.

Challenge Input

324 face images with the file name <image_id.jpg>. You can download a zip file that contains the dataset from the following link or through Kaggle competition page. Please note that the image size can vary.

final-images.zip

Expected Output

You need to submit an abstract (maximum length of 200 words) describing your method and a plain text file (in CSV format) that includes the final consolidated age labels. The CSV file should contain two columns (‘image_id’ and ‘age’) and only a single entry for each image. You are also encouraged to document the process. We will invite successful teams to present during the workshop.

You can make multiple submissions in Kaggle. However, the final challenge output should be submitted through the workshop submission website by 15th October 2021 20th October 2021.

Evaluation

The final ranking will be determined based on 1) MAE calculated using the real age and the submitted apparent age and 2) MAE difference between groups (determined based on the demographics and other attributes of people appearing in the images).

Please note that the Kaggle leaderboard is only based on metric 1 and the final ranking will be different to the Kaggle rankings.

How to use AWS Sagemaker GroundTruth (GT)

Participants may use any crowdsourcing platform of their choice to collect human annotations.

For any teams interested in using free AWS credits with Amazon’s Sagemaker Ground Truth (GT), we provide some helpful information below for getting started.

You can create a task in GT using the AWS management console or the AWS SDK (Create a Labeling Job (API) - Amazon SageMaker). Check the following document for a step by step guide on setting up the challenge task in AWS Sagemaker Ground Truth.

AWS Sagemaker Ground Truth - How to Guide

Additional Resources

• A Complete Tutorial on Annotating Datasets with Amazon SageMaker Ground Truth

• Creating Custom Labeling Workflows - Amazon SageMaker

• Github - Example task UIs for Amazon SageMaker Ground Truth

Acknowledgement

Original data from ChaLearn LAP datasets licensed under CC BY-NC 4.0.