Without any doubt, the notion of uncertainty is of major importance in machine learning and constitutes a key element of modern machine learning methodology. In recent years, it has gained in importance due to the increasing relevance of machine learning for practical applications, many of which are coming with safety requirements. In this regard, new problems and challenges have been identified by machine learning scholars, many of which call for novel methodological developments. Indeed, while uncertainty has a long tradition in statistics, and a broad range of useful concepts for representing and quantifying uncertainty have been developed on the basis of probability theory, recent research has gone beyond traditional approaches and also leverages more general formalism and uncertainty calculi.

In the statistics literature, two inherently different sources of uncertainty are commonly distinguished, referred to as aleatoric and epistemic. While the former refers to variability due to inherently random effects, the latter is uncertainty caused by a lack of knowledge and hence relates to the epistemic state of an agent. Thus, epistemic uncertainty can in principle be reduced on the basis of additional information, while aleatoric uncertainty is non-reducible. The distinction between different types of uncertainty and their quantification has also been adopted in the recent ML literature, and various methods for quantifying aleatoric and epistemic uncertainty have been proposed. In the context of supervised learning, the focus is typically on predictive uncertainty, i.e., the learner’s uncertainty in the outcome given a query instance for which a prediction is sought. The aleatoric part of this uncertainty is due to the supposedly stochastic nature of the dependence between instances and outcomes, e.g. due to a lack of informative features or noisy class annotations. Therefore, the “ground-truth” is a conditional probability distribution, i.e., each outcome has a certain probability to occur. Even with full knowledge about the underlying data-generating process, the outcome cannot be predicted with certainty.

Obviously, the learner does not have full knowledge of this ground-truth. Instead, it produces a “guess” on the basis of the sample data provided for training. Broadly speaking, epistemic uncertainty is uncertainty about the true probability and hence the discrepancy between the ground-truth and the learner's guess. This (second-order) uncertainty can be captured and represented in different ways. One approach is to train a probabilistic predictor in a more or less standard way, and to quantify the uncertainty of that predictor in a kind of post hoc manner. An example is the estimation of epistemic uncertainty in terms of the mutual information between the target variable and the model parameters.

Another idea is to estimate uncertainty in a more direct way, and to let the learner itself predict, not only the target variable, but also its own uncertainty about the prediction. For example, instead of predicting a probability distribution, the learner may predict a second-order distribution (distribution of distributions) or a set of distributions. Its epistemic uncertainty is then represented by the “peakedness” of the former or the size of the latter.  

This tutorial aims to provide an overview of uncertainty quantification in machine learning, a topic that has received increasing attention in the recent past. Starting with a recapitulation of classical statistical concepts, we specifically focus on novel approaches for distinguishing and representing so-called aleatoric and epistemic uncertainty. By the end of the tutorial, attendees will have a comprehensive understanding of the fundamental concepts and recent advances in this field.

PRESENTERS

This tutorial will be held in-person spread over four days.

Day 1 – Introduction and First-Order Uncertainty Representation

Instructor: Willem Waegeman

The first day introduces methods for representing uncertainty in machine learning predictions. The focus will be on calibration techniques, which aim to produce probabilistic models whose confidence estimates accurately reflect reality. Topics include uncertainty representation and calibration for binary classification, multi-class classification, and regression problems.

Day 2 – Conformal Prediction

Instructor: Willem Waegeman

The second day is devoted to conformal prediction, a versatile framework for uncertainty representation that has gained significant attention in recent years. Conformal prediction represents uncertainty through prediction sets and provides strong statistical guarantees under mild assumptions. The course will cover both classification and regression settings, as well as recent extensions to structured prediction problems.

Day 3 – Second-Order Uncertainty

Instructor: Eyke Hüllermeier

On the third day, the focus shifts to methods that distinguish between aleatoric uncertainty (inherent randomness) and epistemic uncertainty (lack of knowledge). Topics include Bayesian inference, ensemble methods, evidential deep learning, and credal machine learning. The course will also discuss emerging challenges in uncertainty representation for modern AI systems, including large language models.

Day 4 – Uncertainty Quantification and Evaluation

Instructor: Eyke Hüllermeier

The final day focuses on measuring and evaluating uncertainty. Participants will learn about established uncertainty measures, such as entropy- and variance-based approaches, and how these can be decomposed into aleatoric and epistemic components. The course will also address the evaluation of uncertainty quantification methods, including their use in downstream applications such as out-of-distribution detection and active learning.

Slides are available.

Day 1: here

Day 2: here

Day 3 and 4: here

For registration, see the ESSAI 2026 website.