In recent years, we have witnessed a remarkable rise of deep learning (DL), whose impressive performance on benchmark tasks has led to increasing ambitions to deploy DL in real-world applications across all fields and disciplines [1, 2, 3, 4, 5]. However, despite its potential, DL still faces many challenges during deployment in dynamic, real-world conditions, exposing practical limitations that are often overlooked in controlled benchmarks.

Current publication mechanisms tend to prioritize solutions that work on standard bench, lacking a platform to systematically collect real-world failure cases. Moreover, discussions about these failures are usually confined within specific domains, with limited cross-domain interaction, even though these failures may have similar underlying causes. Establishing a platform for collecting and sharing real-world challenges and failures of DL can address fundamental issues to facilitate more successful deployment of DL across domains, and enhance understanding of theoretical and empirical weaknesses in machine learning (ML) research.

Building such a platform and fostering this community has been the continuous goal of our I Can’t Believe It’s Not Better (ICBINB) initiative. As DL systems have become increasingly present in everyday life also for non-scientific people, we want to put a special focus on real-world applications now. Therefore, in this proposed ICBINB workshop, we aim to explore the challenges, unexpected outcomes, and common principles underlying similar issues and failure modes encountered across various fields and disciplines when deploying DL models in real-world scenarios. We will focus the discussion on:

Challenges & failure modes: We will invite papers from diverse fields including but not limited to healthcare, scientific discovery, robotics, education, equality & fairness, and social sciences to discuss the challenges and failure modes when deploying DL models for domain-specific applications as well as the underlying reasons. The failure modes may include suboptimal performance, concerns with the safety and reliability of applying DL models in unpredictable real-world applications, as well as ethical and societal challenges.

Common challenges across domains & underlying reasons: We aim to discuss common reasons or patterns in challenges and failure modes across disciplines, which may include, but are not limited to, data-related issues (e.g., distribution shift, bias, label quality), model limitations (e.g., ethics, fairness, interpretability, scalability, domain alignment), and deployment challenges (e.g., computational demands, hardware constraints).

This workshop forms one workshop in a series as part of the larger I Can't Believe It's Not Better (ICBINB) activities. We are a diverse group of researchers promoting the idea that there is more to machine learning research than tables with bold numbers. We believe that understanding in machine learning can come through more routes than iteratively improving upon previous methods and as such this workshop aims to focus on understanding through negative results. Previous workshops have focused on ideas motivated by beauty and gaps between theory and practice in probabilistic ML, we also run a monthly seminar series aiming to crack open the research process and showcase what goes on behind the curtain. Read more about our activities and our members here.

ICBINB aims to foster an inclusive and welcoming community. If you have any questions, comments, or concerns, please contact us at: cant.believe.it.is.not.better@gmail.com

Whilst we will have a range of fantastic speakers appearing in person at the workshop we understand that many people are not able to travel to ICLR at this moment in time. It is our aim to make this workshop accessible to all, all talks will be viewable remotely.

Subscribe to the ICBINB mailing list here: The ICBINB Initiative - Google Groups

Follow us on X and Bluesky!