Reading

Some recent academic literature describes the importance of socially subjective annotations and the nuances in handling disagreements therein. We recommend these following papers as relevant reading for this tutorial. It is not required to read them before attending the tutorial, and we will abridge and cover some material from them.

1. Whose  Ground  Truth?   Accounting  for  Individual   and   Collective   Identities   Underlying  Dataset  Annotation;   Emily  Denton, Mark Díaz, Ian Kivlichan, Vinodkumar Prabhakaran,  Rachel  Rosen;   Data  Centric  AI Workshop at NeurIPS 2021 (Denton et al., 2021)

2. SeeGULL:  A  Stereotype  Benchmark  withBroad  Geo-Cultural  Coverage  Leveraging Generative   Models; Akshita   Jha,   Aida Mostafazadeh  Davani,  Chandan  K  Reddy, Shachi   Dave,    Vinodkumar   Prabhakaran, Sunipa Dev; ACL 2023 ( Jha et al., 2023)

3. Probing  pre-trained  language  models  for cross-cultural  differences  in  values;  Arnav Arora, Lucie-Aimée Kaffee, Isabelle Augenstein;   Proceedings  of  the  First  Workshop on  Cross-Cultural  Considerations  in  NLP (C3NLP) at EACL 2023 ((Arora et al., 2023))

4. Cultural Incongruencies in Artificial Intelligence; Vinodkumar Prabhakaran, Rida Qadri, Ben Hutchinson; Cultures and AI Workshop at NeurIPS 2022 (Prabhakaran et al., 2022)

5. Assessing cross-cultural alignment between ChatGPT  and  human  societies:   An  empirical  study;  Yong  Cao,  Li  Zhou,  SeolhwaLee, Laura Cabello, Min Chen,  Daniel Hershcovich;  Proceedings of the First Workshop on  Cross-Cultural  Considerations  in  NLP (C3NLP) at EACL 2023 (Cao et al., 2023)

6. Disentangling disagreements  on  offensiveness:   A  cross-cultural study; Aida Mostafazadeh Davani, Mark Diaz, Dylan Baker, Vinodkumar Prabhakaran; Proceedings of  Association for Computational Linguistics (ACL) 2023 (Davani et al., 2023)