Responsibly Working with Crowdsourced Data

November 9th @ HCOMP 2023  @  3pm CET / 9am ET / 6am PT

Please indicate your interest in joining us here! 

Crowdsourced datasets are commonly used to develop a range of machine learning tools, such as hate speech classifiers and object detection systems. At the same time, scholars have pointed out a range of related ethical concerns, ranging from the conditions in which raters work and data is collected to the role of social experiences in shaping raters’ data judgments. One result of this work is a set of recent frameworks designed to guide data collection and bring increased transparency to crowdsourced dataset development. This half-day workshop will discuss data annotation and crowdsourcing for a variety of goals– including training dataset development, benchmark development, and model finetuning. A panel will kick off a discussion of ethical issues underlying the development and use of crowdsourced datasets and a breakout activity will guide participants in 1) unpacking crowdsourcing tensions in their own domain, 2) identifying the strengths of different dataset development frameworks in addressing these tensions, and 3) identifying tensions that remain to be addressed.


Key Information


Conference Site:

Interest Form (for meeting link):

Location:

Date:

Time:

Panelists

Milagros Miceli

Milagros Miceli is a sociologist and computer scientist with the Weizenbaum Institute. Her research is centered on exploring the production of ground-truth data for machine learning, with a specific focus on labor conditions and power dynamics involved in data generation and labeling.

Christopher Homan

Christopher Homan is an Associate Professor at Rochester Institute of Technology. His research centers on the gap between human-computer interaction (HCI) and machine learning (ML), specifically on mechanisms for human- and machine-based intelligence to learn from each other in computer-mediated settings.

Krystal Kauffman

Krystal Kauffman is a gig worker and lead organizer of Turkopticon, a nonprofit organization dedicated to fighting for the rights of gig workers. She is also a research fellow at the DAIR Institute.

Alex Chávez

Alex Chávez is based in South America and has been a data worker on Amazon Mechanical Turk for over 6 years.

Organizing Team

Mark Díaz (Google)

Andrew Zaldivar (Google)

Remi Denton (Google)

Vinodkumar Prabhakaran (Google)

Rachel Rosen (Google)