Abstracts

List of accepted 18 abstracts:

Title: Phrase Detectives

Authors: Jon Chamberlain

Abstract: The online game-with-a-purpose Phrase Detectives has been collecting decisions about anaphoric coreference in human language for over 10 years (4 million judgements from 40,000 players). The game was originally designed to collect multiple valid solutions for a single task, which complicated aggregation but created a very rich (and noisy) dataset. Analysis of the ambiguous player decisions highlights the need for understanding and resolving disagreement that is inherent in language interpretation. This talk will present some of the interesting cases of ambiguity found by the players of Phrase Detectives (a dataset that will be made available to the research community later this year) and discuss the statistical methods we have been working on to harness crowds that disagree with each other.

Slides: https://drive.google.com/open?id=1xV2xCpwGp_0N6tF8vPNZF1eLqDWJUOSm

Title: Exploring Bias in Crowd-Powered Social Robotics for Stress Mitigation Tasks

Authors: Tahir Abbas, Vassilis-Javed Khan, Ujwal Gadiraju and Panos Markopoulos

Abstract: Real-time crowdsourcing (RTC) is an area in Human Computation research in which online workers carry out tasks under real time constraints. RTC has been used in a wide variety of domains including tele-operating a robot’s locomotion and supporting or facilitating information retrieval tasks. When it comes to handling complex social conversational tasks with RTC, the crowds’ collective input is vulnerable to bias due to a variety of factors ranging from the cultural background of workers to their personal interests or viewpoints. In a recent study, we investigated how effectively a crowd of workers, recruited in real-time, can act as a coach for assisting stressed students via a robot. The crowd’s input includes interesting data on how the workers’ cultural background biases their recommended strategies in the context of mitigating the stress of students. More specifically, we developed a system where Softbank’s Pepper robot broadcasts a live audio-video (AV) feed of a student who is stressed to crowd workers. Workers are then asked to converse with the students by typing messages which are in turn spoken out by Pepper, in an attempt to alleviate the stress of students and propose coping strategies. Our results highlight the cultural bias among workers. We found that Indian workers asked more personal questions (e.g. about the age or partner of the student), suggested solutions influenced by their spiritual beliefs (e.g. to pray, or to believe in God) and proposed unique solutions (e.g. practicing yoga). Conversely, US workers let the students express themselves by asking open questions, explored students’ strengths and coping skills, and suggested practical exercises (e.g. walking) as a main strategy to alleviate stress. Based on this study, we reflect on the potential challenges when using real-time crowdsourcing to support conversational tasks, and envision opportunities to reduce bias in such complex social conversational tasks. Our proposed solutions include training workers, building intelligent workflows and UIs.

Slides: https://drive.google.com/open?id=1cr3Sixn6Fghvlks-QTZfe_6vHVkKoLKx

Title: Biases in Human Computation and Crowdsourcing - Current Legal Perspectives

Authors: Stefanie Pletz

Abstract: “Justice outweighs all other values.”

C. Perelman [1]

Human-computer interaction inevitably consists of value transfer processes[2].

Specifically, the transfer of these values is subject to human biases and discrimination in algorithms, machine learning and computation at large. This particular discriminatory bias appears topically manifold and wide-ranging due to the complexities in data processing. Demographic information such as attributes [3] referring to age, national origin and race, have been identified as negative social discriminatory bias in computation which opposes legal analytical logic and methodology that statutorily [4] enshrines fundamental human values and standards such as fairness, justice and equality. The paper seeks to identify and discuss human bias and human bias research methodologies to further explore the ‘efficiency vs. equality trade-off’ caused by discriminatory biases in human computation including institutional and legal perspectives. It contends that existing legislative frameworks are insufficient to support the detection and identification of human biases in computation. The paper then examines the rationale for legal intervention and illustrates how recent legislative developments and proposals such as the use of super-soft law, or the Algorithmic Accountability Act could mitigate discriminatory human biases. The paper closes by discussing and contextualising specific current practices such as testing and evaluating algorithms and proposes to follow the approach of dynamic legislation to reduce human bias in computation.

[1] Chaim Perelman, Justice, Law and Argument – Essays on Moral and Legal Reasoning (first published 1980) 1

[2] Value transfer processes Ger. for Wertübertragunsprozesse in Thilo Hagendorff, ‘Maschinelles Lernen und Diskriminierung: Probleme und Lösungsansätze’ [2019] ÖZS 53

[3] Yanai Elazar and Yoav Goldberg, ‘ Adversarial Removal of Demographic Attributes from Text Data’ [2018] Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing 11-21

[4] Protocol No. 12 to the Convention for the Protection of Human Rights and Fundamental Freedoms, an additional protocol to the European Convention on Human Rights (ECHR) provides for a general prohibition of discrimination states in Art. 1 that: "The enjoyment of the rights and freedoms set forth in this Convention shall be secured without discrimination on any ground such as sex, race, colour, language, religion, political or other opinion, national or social origin, association with a national minority, property, birth or other status." And in Art. 2 respectively: “No one shall be discriminated against by any public authority on any grounds such as those mentioned above.”

Slides: Author/s do not authorize to publish

Title: A human in the loop approach to detect episodic and thematic framing in news videos

Authors: Panagiotis Mavridis, Markus De Jong, Alessandro Bozzon, Tobias Kuhn and Lora Aroyo

Abstract: A vast majority of people prefer the use of news videos to get informed. However, news videos, like any kind of news information, are prone to bias that leads to miscommunication. This bias comes from differences between groups of news consumers. Citizens that interpret the news have different political orientations and thus understand news differently. Also, media scholars interpret and study news differently compared to the broader public. While automated methods exist for different types of bias detection, their capabilities and accuracy are limited or there is a need to fine-tune specific parameters to make them work out of the box. For instance, episodic and thematic framing is very difficult to be captured by existing automatic techniques when applied to news videos. Instead, we propose the use of humans in the loop in order to detect framing. We propose the design of crowdsourcing tasks that will help people identify through a set of particular questions the existence of thematic vs. episodic framing. Then, we evaluate the result of the crowdsourcing experiment and compare the results with our expert ground truth. Conversely, when applied for a particular news event the occurring word patterns and annotations of crowdsourcing can help determine framing in an automatic manner. After having evaluated the skillset of the crowd to determine framing, we finally evaluate our semi-automatic method with the use of humans in the loop.

Slides: Author/s do not authorize to publish

Title: Comparing measurement of perceptions of energy density and carbon footprint using Citizen Science and a UK representative sample

Authors: Christian Reynolds

Abstract: There is a food knowledge disconnect between the food researcher and the general population. Indeed researchers cannot measure easily what citizens understand or perceive to know about food. Standard surveys take time and are costly.

One digital engagement and data collection methodology that would allow the measurement of perception around food, as well as educate participants and measure changes in knowledge is citizen science. The citizen science method invites members of the public to participate in scientific thinking and data collection around a set theme or experiment and to collaboratively become researchers.

This paper provides comparison of two methods 1) the Zooniverse citizen science platform (n=~516 participants); and 2) a UK representative sample (n=400) deployed over the Qualtrics Panel, to measure perceptions foods of energy density (kcal) and carbon footprint (g of CO2). Ten foods were selected with a range of energy and carbon densities.

Slides: https://drive.google.com/open?id=11qlS6CdW7twtv__aD1hQXFSAx1TQw3M1

Title: Understanding Bias and Subjectivity in Conversational Microtask Crowdsourcing Using Conversational Style Estimation

Authors: Sihang Qiu, Ujwal Gadiraju and Alessandro Bozzon

Abstract: Crowdsourcing platforms such as Amazon Mechanical Turk and Figure Eight have created marketplaces for millions of people to gain profit or make a living. As crowdsourcing has grown to be an important source of income of these people (i.e. online crowd workers), the study of human factor recently became a popular direction in the crowdsourcing field. With the booming of chatbots, previous works have shown that text-based conversational agents can effectively support crowdsourcing task execution with high user satisfaction. However, bias and subjectivity can still affect crowdsourcing result as long as human is involved.

During the last few decades, linguists and psychologists have found that the conversational style could reflect the personality, thinking style, and growth environment of a person, which inspires us to investigate whether we can better understand bias in crowdsourcing by estimating the conversational style of the worker. To this end, we design and implement a web-based conversational agent, which can be easily embedded in any web-based crowdsourcing platforms, to assist workers in microtask execution. Furthermore, we proposed a coding scheme to manually classified conversational styles into two categories (involvement vs. considerateness) according to relevant linguistic works. We use this coding scheme to produce ground-truth dataset, and then train a classification model to automatically measure conversational styles. We intend to conduct a crowdsourcing experiment, where workers need to collect and understand the information acquired from the Internet (about a controversial issue) and then give a brief summary. Finally, we plan to evaluate the quality of the answers given by workers, and to investigate whether bias significantly correlate with conversational style.

Slides: https://drive.google.com/open?id=1jWk4PHnwvPbHMI2Hjq674Pcl3KYvXtig

Title: Correcting dataset biases for machine learning applications with human-in-the-loop systems

Authors: Agathe Balayn

Abstract: Real-life applications of machine learning (ML) suffer from unfairness due to biases emerging along the ML pipeline. Research mostly focuses on solving algorithmic aspects of unfairness and biases within high-stake applications using structured data (e.g. characteristics of individuals to predict their probability to reimburse a loan). Conversely, here we investigate data biases in classification tasks of unstructured data, assuming that these applications have different important challenges to address.

Specifically we explore the creation of unbiased datasets for fairer ML applications, assuming that dataset representativeness is the primary source of unfairness. Through two use-cases (profession prediction from images, and toxicity prediction based on textual sentences), we identify data biases stemming from the content of the samples (latent features of images giving away protected attributes) and from the labels (aggregated crowd-provided labels representing majority opinions) that contribute to the unfairness of current systems. We then propose human-in-the-loop approaches in order to mitigate these biases and the consequent unfairness, relying on the analysis of the outputs of ML models. First, we show that training ML models on the dataset to correct and evaluating their fairness provides an indicative bias measure. Then, we propose a visualization tool to help ML experts understand the causes of label bias and identify the sentences in the dataset and the type of crowd workers that could provide labels correcting unfairness. We also propose a methodology to pinpoint the causes of sample biases in images, combining crowdsourced knowledge on stereotypes and information from methods to explain ML models. We present initial experiments that show the potential of these methodologies and discuss future work.

Slides: https://drive.google.com/open?id=1ftiYlSQV_H44FER0Hxf5eIVySHuczKmz

Title: Wisdom of the crowds vs expert knowledge: experiments in genre annotation

Authors: Serge Sharoff

Abstract: This talk will report differences in annotation experiments aimed at the same task, ie. genre annotation, while involving different annotation setups. More specifically, this study compares crowd-sourced annotation of commonly recognised genres, such as news or editorial cite:asheghi16, vs expert annotation of text functions, such as argumentative or reporting texts cite:sharoff18genres.

Most of the previous genre annotation studies have not been tested for inter-annotator reliability. When they have been tested cite:sharoff10lrec, they were found to exhibit low inter-annotator agreement. With precise and consistent annotation guidelines for well-defined and well-recognized categories, it is possible to use crowdsourcing (via Amazon Mechanical Turk) to obtain reasonably good agreement. However, this applies to a small portion of the training corpus. Many documents in the crowd-sourced annotation experiment received the non-informative `Other' label.

One of the reasons for this is genre hybridism. Even with strict editorial control, the authors may choose to combine distinct styles of writing in a single text, such as reportage and expressions of opinions in a newspaper article. This leads to possible proliferation of genre labels, e.g., /editorial, column, opinion, analytic, feature article/. On the Web there are far fewer explicit gate-keepers, and far more authors with varying levels of expertise or willingness to express themselves according to traditionally accepted ways which are recognised as genres. This leads to such phenomena as citizen journalism or research blogs. From the annotation perspective, different annotators can interpret a hybrid Web text in different ways, thus producing different annotations for stylisticallly similar texts.

Expert annotation can be based on recognition of communicative, in which the texts are described in terms of their similarity to prototype genres. The suggested set of 15 functions is designed to be applicable to any text on the Web and to be reliable in annotation practice. Inter-annotator agreement results show that the suggested categories produce Krippendorff's α at above 0.76.

Slides: https://drive.google.com/open?id=1uZ-ex6FGWm_E9kjZiuclPAYKnuNgRNtn

Title: Measuring social biases in human annotators using counterfactual queries in Crowdsourcing

Authors: Bhavya Ghai, Q. Vera Liao, Yunfeng Zhang and Klaus Mueller

Abstract: Algorithmic bias has been termed as the imminent AI danger faced by our society. Recent studies have shown that machine learning(ML) algorithms are capable of exhibiting social biases like gender, race, etc. A major source of bias in the ML pipeline arises from the training dataset. Crowdsourcing is a popular way to gather labeled data for different ML tasks. As crowdsourcing tasks might involve a subjective component, it’s important to gauge implicit social biases of human annotators and prevent them from spreading into the curated dataset.

We propose a novel way to measure social biases in crowdworkers using counterfactuals queries. Here, we are considering a supervised learning scenario with numeric or categorical input features. A counterfactual to a given query is the most similar query in an alternate world where its sensitive attributes like race, gender, etc. is flipped. Counterfactual queries can be generated using causal inference by measuring the impact of flipping the sensitive attribute on other input features. During the training phase of the user study, we can ask human annotators to label a set of counterfactual queries and measure the deviation in the responses. Zero deviation characterizes perfect unbiased behavior and higher values symbolize more biased behaviour. If the deviation is beyond a specific threshold, we can consider such annotators to be unfit for the study and terminate the study for those labelers. This methodology doesn’t need unbiased labels and biased labelers are disqualified in the training phase itself. Hence, this can serve as a cost-effective way to tackle social biases in crowdsourcing.

Slides: https://drive.google.com/open?id=17KQwUByRPBWzCCrLRf1BnovMxLo69avp