1st workshop on critical evaluation of

generative models and their impact on society

at ECCV 2024, Milan, Italy

Visual generative models have revolutionized our ability to generate realistic images, videos, and other visual content. However, with great power comes great responsibility. While the computer vision community continues to innovate with models trained on vast datasets to improve visual quality, questions regarding the adequacy of evaluation protocols arise. Automatic measures such as CLIPScore and FID may not fully capture human perception, while human evaluation methods are costly and lack reproducibility. Alongside technical considerations, critical concerns have been raised by artists and social scientists regarding the ethical, legal, and social implications of visual generative technologies. The democratization and accessibility of these technologies exacerbate issues such as privacy, copyright violations, and the perpetuation of social biases, necessitating urgent attention from our community.

This interdisciplinary workshop aims to convene experts from computer vision, machine learning, social sciences, digital humanities, and other relevant fields. By fostering collaboration and dialogue, we seek to address the complex challenges associated with visual generative models and their evaluation, benchmarking, and auditing.

Invited speakers

John Collomosse is a principal scientist at Adobe Research, where he leads research for the Content Authenticity Initiative (CAI) and two cross-industry task forces within the C2PA open standards body for media authenticity. He is a professor at the University of Surrey, where he is the founder and director of DECaDE, the UKRI Research Centre for the Decentralized Creative Economy. His research focuses on media provenance to fight misinformation and online harms, and on improving data integrity and attribution for responsible AI.

Yukino Baba is an Associate Professor in the Graduate School of Arts and Sciences at University of Tokyo. Her research interests are in Human-AI Collaboration and Human Computation. Prior to joining the faculty at University of Tokyo, she was an Associate Professor at University of Tsukuba from 2018 to 2022, Assistant Professor at Kyoto University from 2015 to 2018, and a postdoctoral fellow at National Institute of Informatics (2014-2015) and University of Tokyo (2012-2014). She received her Ph.D. from University of Tokyo in 2012.

Eva Cetinic is a researcher at the University of Zurich (UZH), Switzerland, where she is currently leading the research project “The Canon of Latent Spaces: How Large AI Models Encode Art and Culture”, funded by the Swiss National Science Foundation. Prior to that, she was a postdoctoral fellow at the Digital Society Initiative at UZH and a postdoctoral fellow at the Center for Digital Visual Studies at UZH. Before joining the University of Zurich, she was a postdoctoral researcher in Digital Humanities and Machine Learning at the Department of Computer Science, Durham University, UK. She received her Ph.D. degree in Computer science from the Faculty of Electrical Engineering and Computing, University of Zagreb in 2019. Her research interests focus on exploring deep learning techniques for computational image understanding and multimodal learning in the context of visual art and culture.

Debora Nozza is an Assistant Professor in Computing Sciences at Bocconi University. She was awarded a €1.5m ERC Starting Grant project 2023 for research on personalized and subjective approaches to Natural Language Processing. Her research interests mainly focus on Natural Language Processing, specifically on the detection and counter-acting of hate speech and algorithmic bias on Social Media data in multilingual context.

Call for papers

Format. We call for novel work and work in progress that has not been published elsewhere. Submissions must follow the ECCV 2024 template and will be peer-reviewed in a single-blind fashion. We welcome the following formats:

Full papers: 14 pages (excluding references). By default, accepted papers will be included in the ECCV workshop proceedings, unless authors choose to opt out of inclusion in the proceedings, in which case the paper should be made available on arXiv.
Extended abstracts: 4 pages (excluding references). Accepted extended abstracts will not be published in the proceedings but should be made available on arXiv.

Additionally, we also accept recently published papers elsewhere for poster presentation; e.g., CVPR, NeurIPS, EMNLP, ECCV main conference. We do not review these papers again and it is not necessary to change the format to the ECCV 2024 template. A jury of organizers will select these papers.

Topics. The workshop revolves around two main topics, visual quality assessment and impact on society:

Visual generation quality assessment
- Image-quality evaluation metrics aligned with human perception.
- Language and vision alignment metrics in text-to-image generation.
- Protocols for reproducible human evaluation.
- New benchmarks for visual generative models.
Visual generation impact on society
- Data audition and analysis.
- Social bias evaluation.
- Privacy threads detection.
- Intellectual property violation detection.
- Impact on the informational environment.
- Impact on the cultural environment.
- Impact on the natural environment.