The 2022 ICBINBworkshop successfully took place at NeurIPS on December 3, 2022. The event was held in person with talks live streamed and recorded. These talks are now available to be viewed here. You can also view the accepted papers from the workshop here.

Deep learning has flourished in the last decade. Recent breakthroughs have shown stunning results [1,2,3], and yet, researchers still cannot fully explain why neural networks generalise so well [4,5,6] or why some architectures or optimizers work better than others [7,8,9]. There is a lack of understanding of existing deep learning systems, which led NeurIPS 2017 test of time award winners Rahimi & Recht [10] to compare machine learning with alchemy and to call for the return of the 'rigour police'.

Despite excellent theoretical work in the field [11], deep neural networks are so complex that they might not be able to be fully comprehended with theory alone. Unfortunately, the experimental alternative - rigorous work that neither proves a theorem nor proposes a new method - is currently under-valued in the machine learning community [12].

To change this, this workshop aims to promote the method of empirical falsification.

We solicit contributions which explicitly formulate a hypothesis related to deep learning or its applications (based on first principles or prior work), and then empirically falsify it through experiments. We further encourage submissions to go a layer deeper and investigate the causes of an initial idea not working as expected. This workshop will showcase how negative results offer important learning opportunities for deep learning researchers, possibly far greater than the incremental improvements found in conventional machine learning papers!

Why empirical falsification? In the words of Karl Popper [13], "It is easy to obtain confirmations, or verifications, for nearly every theory—if we look for confirmations. Confirmations should count only if they are the result of risky predictions."

We believe that similarly to physics, which seeks to understand nature, the complexity of deep neural networks makes any understanding about them built inductively likely to be brittle [14].

The most reliable method with which physicists can probe nature is by experimentally validating (or not) the falsifiable predictions made by their existing theories. We posit the same could be the case for deep learning and believe that the task of understanding deep neural networks would benefit from adopting the approach of empirical falsification.

This workshop forms one workshop in a series as part of the larger I Can't Believe It's Not Better (ICBINB) activities. We are a diverse group of researchers promoting the idea that there is more to machine learning research than tables with bold numbers. We believe that understanding in machine learning can come through more routes than iteratively improving upon previous methods and as such this workshop aims to focus on understanding through negative results. Previous workshops have focused on ideas motivated by beauty and gaps between theory and practice in probabilistic ML, we also run a monthly seminar series aiming to crack open the research process and showcase what goes on behind the curtain. Read more about our activities and our members here.

ICBINB aims to foster an inclusive and welcoming community. If you have any questions, comments, or concerns, please reach out to us.

Whilst we will have a range of fantastic speakers appearing in person at the workshop we understand that many people are not able to travel to NeurIPS at this moment in time. It is our aim to make this workshop accessible to all, so the workshop will include a virtual attendance option. This means,

• Authors may choose to attend remotely if they are unable to come in person

• All talks will be viewable remotely

• The workshop program will include specific activities for remote attendees