Workshop Schedule

Date and Time - Saturday, 23rd October 2021 from 3.00 PM - 7.45 PM (EDT)

The workshop will include the following activities. All times at given in EDT (Eastern Daylight Time).

Instructions for attendees: Workshop attendees will be able to join the workshop Zoom room via the CSCW program/schedule. When you go into Midspace, you can click on the CSCW Program or Full Schedule and select your workshop in the popup window. Sessions in Midspace will be available 15 minutes prior to the scheduled start time. Only those registered for the workshop will be able to attend.


3.00 PM - 3.45 PM - Introduction

We will provide a brief introduction to the workshop outlining planned activities, goals and themes of the workshop. We also plan to run a quick ice-breaking round to get to know the participants.

3.45 PM - 5.30 PM - Position Paper Discussion

Participants will share the accepted position papers with the audience.

  • 3.45 PM - Senjuti Dutta, Rhema Linder, Doug Lowe, Matthew Rosenbalm, Anastasia Kuzminykh and Alex Williams. The Productivity Paradox: Understanding Tooling Biases in Crowdwork. [PDF]

  • 4.00 PM - Hossein A. Rahmani and Jie Yang. Demographic Biases of Crowd Workers in Key Opinion Leaders Finding. [PDF]

  • 4.15 PM - Tim Draws, Alisa Rieger, Oana Inel, Ujwal Gadiraju and Nava Tintarev. Introducing the Cognitive-Biases-in-Crowdsourcing Checklist. [PDF]

  • 4.30 PM - Rahul Pandey and Hemant Purohit. CrowdRL: A Reinforcement Learning Framework for Human Error Mitigation in Crowdsourcing-based Stream Processing. [PDF]

  • 4.45 PM - Gianluca Demartini, Kevin Roitero and Stefano Mizzaro. Bias Management in Crowdsourced Data: Moving Beyond Bias Removal. [PDF]

  • 5.00 PM - Chien-Ju Ho and Ming Yin. Designing and Optimizing Cognitive Debiasing Strategies for Crowdsourcing Annotation. [PDF]

  • 5.15 PM - Caifan Du. Bias Mitigation through ML pipeline in Crowd-based Knowledge Creation. [PDF]

5.30 PM - 5.45 PM - Break


5.45 PM - 6.15 PM - Crowd Bias Challenge Recap

The challenge outcomes will be presented and discussed. We will invite three leading teams to discuss their solutions with the audience.

6.15 PM - 7.15 PM - Blue Sky session - Beyond the Crowd

Facilitated by Prof. Julian Thomas. He is the Director of the ARC Centre of Excellence for Automated Decision-Making and Society (ADM+S Centre), and a Distinguished Professor in the School of Media and Communication at RMIT University.

This ideation session will be facilitated by an invited expert outside the crowdsourcing research domain to explore challenging future research directions. We will probe participants to discuss more broad research questions around biases and crowd data. For example, how can we scale crowdsourcing platforms to wider population groups while preserving bias considerations? We plan to conduct small group discussions under four themes.

7.15 PM - 7.45 PM - Closing

This session will summarise the workshop output and obtain feedback from participants regarding possible future activities.

Ujwal Gadiraju will provide a brief overview on Third symposium on Biases in Human Computation and Crowdsourcing (BHCC)