Publications

2019

A deterministic and computable Bernstein-von Mises theorem

Guillaume Dehaene

In order to make Bayesian inference possible on large datasets, approximations are required. For example, computing the Laplace approximation is straightforward since it only requires finding the maximum of the posterior. However, while the Bernstein-von Mises theorem guarantees that the error goes to in the limit of infinitely large datasets, it is hard to measure precisely the size of the error in a given example.

This article derives a tight and computable elegant approximation of the size of this error. I show that the Kullback-Leibler divergence between a given probability distribution and its Laplace approximation can be approximated using the “Kullback-Leibler variance”

2017

Computing the quality of the Laplace approximation. AABI NIPS 2017 Workshop.

Guillaume Dehaene

Bayesian inference requires approximations because the posterior distribution is generally uncomputable. The Laplace approximation is a fairly basic one which gives us a Gaussian approximation of the posterior distribution. This begs the question: how good is the approximation? The classical answer to this question is the Bernstein-von Mises theorem, which asserts that in the large-data limit, the Laplace approximation becomes exact. However, this theorem is mostly useless in practice, mostly because its assumptions are hard to check.

This article presents a computationally-relevant extension of the classical result: we give an explicit upper-bound for the distance between a given posterior and its Laplace approximation. The approach we follow can be extended to more advanced Gaussian approximation methods which we will do in further work.

Expectation Propagation in the large-data limit. Journal of the Royal Statistical Society, series B.

Guillaume Dehaene and Simon Barthelmé

Expectation Propagation is a popular method for variational inference which can be remarkably effective despite the fact that there's very little theory supporting it. Our two main contributions consist in showing that EP is closely related to Newton's method for finding the maximum of the posterior, and showing that EP is asymptotically exact, meaning that when the number of datapoints goes to infinity the method recovers the posterior exactly.

We also introduce some new theoretical tools that help analysing EP formally, including a simpler variant, called Average-EP (or Stochastic-EP), that is asymptotically equivalent to EP.

2016

Expectation Propagation performs a smoothed gradient descent. AABI NIPS 2016 Workshop.

Guillaume Dehaene

If one wants to compute a Gaussian approximation of a probability distribution, there are three popular alternatives: the Laplace approximation, the Gaussian Variational Approximation, and Expectation Propagation.

I show in this work that the approximations found by these three methods are actually very closely related, as they all correspond to variants from the same algorithm. This shines a bright light on the deep connections between these three algorithms.

2015

Bounding errors of expectation-propagation. NIPS 2015.

Guillaume Dehaene and Simon Barthelmé

Expectation Propagation is a popular method for variational inference which can be remarkably effective despite the fact that there's very little theory supporting it.

Our contribution in this work consists in showing that, in the large-data limit, EP is asymptotically exact and, furthermore, more precise than the alternative Laplace approximation. However, our results only hold for strongly log-concave distributions, which very rarely exist.