A Tutorial on the Normative Design and Evaluation of Information Access Systems
The 2024 ACM SIGIR Conference on Human Information Interaction and Retrieval in Sheffield, UK between the 10th and 14th of March, 2024
Embedded Files
One CHIIR for ... NORMalize!
One CHIIR for ... NORMalize!
The workshop on the Normative Design and Evaluation of Recommender Systems (NORMalize) has been accepted as a tutorial at ACM CHIIR 2024, March 10-14, in Sheffield. It is featured as a half-day interactive tutorial and aims to promote normative thinking in the design of recommender/information retrieval systems.
Objectives
Objectives
NORMalize sets out to provide a platform for researchers and practicioners from different domains to discuss challenges related to the normative design and evaluation of recommender systems.
Recommender systems are used in a many different domains, from recommending recipes to song playlists. They increasingly act as "gatekeepers" to what users are exposed to online. Therefore it is important that we, as designers of these systems, think about the norms and values appropriate to the domain when designing and evaluating these recommender systems, such that they have a positive impact on society.
Should recipe recommenders prioritize healthy choices over not so healthy choices? Or should they prioritize sustainability and recommend recipes that make use of local, fresh ingredients? Or maybe we need both?
Should news recommenders provide sufficiently diverse recommendations? Should we strive to expose users to content that contradicts their political beliefs?
How can we operationalize these norms and values in recommender systems?
How can we design normative recommender systems?
How can we evaluate recommender systems according to our norms and values?
At NORMalize, we aim to provide some preliminary answers to the above questions, and many more through guided discussions that bring together industry and academia and scholars from different research domains.
Interactive Tutorial
Interactive Tutorial
The on-site interactive tutorial is intended for the norm-curious and will allow for knowledge sharing and the creation of a shared vocabulary that can facilitate research collaborations. Participants are divided into breakout groups of four to five people each. In these groups, they discuss a specific use case of an information access system, for example, Google News or YouTube. First, they identify when, where and how the system is used and where ranking algorithms are used to decide what is shown to a user. Then, they identify the stakeholders of the system and the norms and values that matter to them. Next, they consider how values might be related to each other. For instance, are diversity and a user’s right to relevant content at odds with each other? Or, if we value freedom of speech, could that lead to hate speech and misinformation? Subsequently, each group is allocated a total of one hundred points, to be divided amongst various values. Each member within the group is given the responsibility to represent a stakeholder of the recommender system and to champion their respective values. The group work concludes with a discussion of what a system that prioritizes values and stakeholders in such a way would look like. Finally, each group presents the findings of their discussion to all participants and organizers.
Page updated
Google Sites
Report abuse