Viacheslav Komisarenko, Estonia, University of Tartu: On Calibration Improvement with Focal Activation

Markus Kängsepp, Estonia, University of Tartu: Calibrated Perception Uncertainty Across Objects and Regions in Bird’s-Eye-View

Novin Shahroudi, Estonia, University of Tartu: Evaluation of Trajectory Distribution Predictions with Energy Score
Predicting the future trajectory of surrounding objects is inherently uncertain and vital in the safe and reliable planning of autonomous systems such as self-driving cars. Although trajectory prediction models have become increasingly sophisticated in dealing with the complexities of spatiotemporal data, the evaluation methods used to assess these models have not kept pace. "Minimum of N" is a common family of metrics used to assess such rich outputs. We critically examine the Minimum of N within the proper scoring rules framework to show that it is not strictly proper and demonstrate how that could lead to an uninformative and even misleading assessment of multimodal trajectory predictions. As an alternative, we propose using Energy Score-based evaluation measures, leveraging their proven propriety for a more reliable evaluation of trajectory distribution predictions.

David Rügamer, Germany, MCML, LMU Munich: Towards Tractable Bayesian Deep Learning

Göran Kauermann, Germany, LMU Munich: Labeling Uncertainty

Thomas Augustin , Germany, LMU Munich: Machine learning under measurement error --- yesterday's concerns or renaissance of a classical sub-discipline?
The talk looks at machine learning under noisy data from a measurement error perspective, which has been developed in statistical modelling in biometrics, econometrics and psychometrics since the eighties of the last century. We discuss the folklore statement that measurement error is no longer an issue after the prediction-oriented turn in data science.  In doing so, we argue from an interpretable machine learning and a privacy preserving machine learning perspective. To make the argumentation a bit more concrete and technical, we then look at measurement error in recursive partitioning. We discuss structural effects of measurement error in regression trees and finally derive a corrected score function in the context of model-based recursive partitioning of survival data.

Slides

Christian Fröhlich, Germany, University of Tübingen: Data Models With Underlying Imprecision
Most work in the field of imprecise probabilities has been rooted in the subjectivist paradigm. We aim to revive two largely discontinued threads of research on imprecision from a frequentist perspective. We mainly focus on constructing data models which exhibit true underlying imprecision, that is, where the imprecision is not due to epistemic uncertainty, but inherent to the data generating process. Such data models then pave the way for deriving generalizations of concepts like calibration and scoring rules, notions which we claim are only meaningful relative to a data model. As to applications, it turns out that this view on imprecision can be meaningfully linked to the machine learning literature on learning from multiple distributions. 

Andrea  Campagner, Italy , IRCCS Ospedale Galeazzi Sant'Ambrogio: Credal Learning: Weakly Supervised Learning from Credal Sets

Michele Caprio and Kuk Jin Jang , USA, University of Pennsylvania: Imprecise Evidential Classification
Uncertainty quantification presents a pivotal challenge in the field of artificial intelligence, profoundly impacting decision-making, risk assessment, and model reliability. In this paper, we introduce Imprecise Evidential Neural Networks (IENN) as a novel approach to address this challenge in classification tasks. IENN leverages a credal set of evidential predictive distributions. They allow to avoid overfitting to the training data, and to systematically assess both epistemic and aleatoric uncertainties (EU and AU, respectively). When these uncertainties surpass acceptable thresholds, IENN has the capability to abstain from classification and indicate an excess of EU or AU. Conversely, within acceptable uncertainty bounds, IENN provides a label class with robust probabilistic guarantees. IENN is trained using standard backpropagation and a loss function that draws from the theory of evidence. Our approach overcomes the shortcomings of previous efforts, and extends the current evidential deep learning literature.  Through extensive experiments, we demonstrate the competitive performance of IENN in classification with abstention and out-of-distribution (OOD) evaluation settings, showcasing its effectiveness on benchmark datasets. 

Volker Tresp, Germany, LMU Munich: Where Heisenberg, and not Bayes, Rules

Tanmoy Mukherjee, Belgium, University of Antwerp/imec: OOD Detection and the Tale of Two Uncertainties 

David Strieder, Germany, MCML, TUM: Confidence in Causal Inference under Structure Uncertainty
Inferring the effect of interventions within complex systems is a fundamental problem of statistics. A widely studied approach employs structural causal models that postulate noisy functional relations among a set of interacting variables. The underlying causal structure is then naturally represented by a directed graph whose edges indicate direct causal dependencies.  In a recent line of work, additional assumptions on the causal models have been shown to render this causal graph identifiable from observational data alone.  One example is the assumption of linear causal relations with homoscedastic errors that we will take up in this work.  When the graph structure is known, classical methods may be used for calculating estimates and confidence intervals for causal effects.  However, in many applications, expert knowledge that provides an a priori valid causal structure is not available. Lacking alternatives, a commonly used two-step approach first learns a graph and then treats the graph as known in inference.  This, however, yields confidence intervals that are overly optimistic and fail to account for the data-driven model choice.  We argue that to draw reliable conclusions, it is necessary to incorporate the remaining uncertainty about the underlying causal structure in confidence statements about causal effects.  To address this issue, we present a framework based on test inversion that allows us to give confidence regions for total causal effects that capture both sources of uncertainty: causal structure and numerical size of nonzero effects. 

Nis Meinert, Germany, German Aerospace Center (DLR): Your Epistemic Uncertainty is Secretly a Density Estimation and You Should Treat it Like One

Mira Jürgens, Belgium, Ghent University: Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods?
In recent years, evidential deep learning methods have become a popular approach of quantifying epistemic (and aleatoric) uncertainty. The resulting second-order distributions are obtained by a single forward-pass of a second-order learner, which is trained in a direct way. In this talk, we will investigate whether the resulting distributions represent epistemic uncertainty in a faithful manner. To this end, we will introduce the notion of a reference distribution and argue that second-order methods should provide an estimate of this distribution if they intend to model epistemic uncertainty in a frequentist way. We both theoretically and empirically show that this is often not the case. 

Bálint Mucsanyi, Germany, University of Tübingen: Are Uncertainty Quantification Methods Doing What We Think They Are Doing?
In recent years, a multitude of uncertainty quantification methods have been developed with different ideas in mind. In this paper, we wonder if they are doing what they are intended to do. We benchmark widely used uncertainty quantification methods on various practical tasks ranging from out-of-distribution detection to abstained prediction. We also investigate the performance of uncertainty disentanglement methods on real-life tasks, the correlation of their components, and the alignment of popular uncertainty estimators with these components. We find that simple uncertainty often estimators dominate on the practical tasks and that the performance of the methods is saturated. We further observe that, although uncertainty disentanglement methods might be principled, their components are not disentangled in our experiments. 

Konstantin Hess, Germany, LMU Munich: Bayesian Neural Controlled Differential Equations for Treatment Effect Estimation
Treatment effect estimation in continuous time is crucial for personalized medicine. However, existing methods for this task are limited to point estimates of the potential outcomes, whereas uncertainty estimates have been ignored. Needless to say, uncertainty quantification is crucial for reliable decision-making in medical applications. To fill this gap, we propose a novel Bayesian Neural Controlled Differential Equations for treatment effect estimation in continuous time. In our \method, the time dimension is modeled through a coupled system of neural controlled differential equations and neural stochastic differential equations, where the neural stochastic differential equations allow for tractable variational Bayesian inference. Thereby, for an assigned sequence of treatments, our \method provides meaningful posterior predictive distributions of the potential outcomes. To the best of our knowledge, ours is the first tailored neural method to provide uncertainty estimates of treatment effects in continuous time. As such, our method is of direct practical value for promoting reliable decision-making in medicine. 

Nikos Nikolaou, United Kingdom, University College London: On the Interplay Between Uncertainty Quantification and Model Explainability

Nina Schmid, Germany, University of Bonn: Advancing Uncertainty Quantification in Universal Differential Equations: Formulation, Assessment, and Comparison
Universal Differential Equations for ScientificMachine Learning (UDEs) are a dynamic mod-elling approach to combine prior knowledge inform of mechanistic formulations with univer-sal function approximators, like neural networks.The neural network component is used to describeunknown components of a differential equation.The estimation of uncertainty, with respect to plau-sible values of mechanistic parameters, as wellas the prediction uncertainty of the whole modelor model components, is crucial for the interpre-tation of the modeling results. With this work,we attempt a formalisation of uncertainty quantifi-cation (UQ) for UDEs and investigate importantfrequentist and Bayesian methods. By analyzingthree synthetic examples of increasing complexityand estimating the underlying ground truth uncer-tainty, we evaluate the validity and efficiency ofdifferent UQ methods.

Cornelia  Gruber, Germany, LMU Munich: More Labels or Cases? Assessing Label Variation in Natural Language Inference
In this work, we analyze the uncertainty that is inherently present in the labels used for supervised machine learning in natural language inference (NLI). In cases where multiple annotations per instance are available, neither the majority vote nor the frequency of individual class votes is a trustworthy representation of the labeling uncertainty.
We propose modeling the votes via a Bayesian mixture model to recover the data-generating process, i.e., the posterior distribution of the "true" latent classes, and thus gain insight into the class variations. This will enable a better understanding of the confusion happening during the annotation process. We also assess the stability of the proposed estimation procedure by systematically varying the numbers of i) instances and ii) labels. Thereby, we observe that few instances with many labels can predict the latent class borders reasonably well, while the estimation fails for many instances with only a few labels. This leads us to conclude that multiple labels are a crucial building block for properly analyzing label uncertainty.

Alaa Othman, Center for Applied Data Science Gütersloh (CfADS) Hochschule Bielefeld-University of Applied Sciences

Andrea Belles Ferreres, German Aerospace Center (DLR)

Miro Miranda Lorenz, German Research Center for Artificial Intelligence, University of Kaiserslautern-Landau

Thomas Mortier, Ghent University

Paul Hofman, LMU Munich

Amir Hossein Kargaran, LMU Munich

Valentin Margraf, LMU Munich

Sepideh Saran, Max Delbrueck Center / Technical University of Berlin

Alessandro Scagliotti, MCML,  TUM

Frederik Hoppe, RWTH Aachen University

Dominik Fuchsgruber, TUM

Viacheslav Komisarenko, University of Tartu

Markus Kängsepp, University of Tartu

Novin Shahroudi, University of Tartu

Kornelius Raeth, University of Tübingen

Alireza Javanmardi, LMU Munich

Nina Maria Gottschling, German Aerospace Center (DLR)

Ghifari Adam Faza, KU Leuven

Kaizheng Wang, KU Leuven

Cornelia Gruber, LMU Munich

Raziyeh Hosseini, LMU Munich

Leonardo Galli, LMU Munich

Long XUE, The Hong Kong Polytechnic University

Carina Newen, TU Dortmund

Nikos Nikolaou, University College London

Tanmoy Mukherjee, University of Antwerp/imec

Corrado Mencar, University of Bari Aldo Moro

Cornelius Emde, University of Oxford

Mihkel Lepson, University of Tartu

Joonas Järve, University of Tartu

Michael Deistler, University of Tübingen

Matias Pizarro, Ruhr Universität & Dorothea Kolossa, TU Berlin