SIR Sequential Design. Left plots show benchmark time and utility evaluation between LFIRE and the Variational estimator for a fixed design (d = 1.0s) over 10 runs each at a range of sample sizes. The variational estimator is orders of magnitude more efficient (top left) and shows lower variance at each sample (top right). The first (bottom left) and fourth (bottom right) sequential BED iterations yield comparable designs between both methods (GP posterior MI shown).

Abstract: Computing the mutual information (MI) between sets of random variables lacks a closed-form solution in nontrivial models. Variational MI approximations are widely used as flexible estimators for this purpose, but computing these estimators typically requires solving a costly non-convex optimization. In this paper we demonstrate a class of variational estimators whose solution is not only convex, but efficiently solved via moment matching conditions. We show that the moment-matched variational estimator provides optimal upper and lower bounds on MI in models with explicit forward models. Furthermore, we show that the same moment matching solution yields accurate MI approximations in so-called "implicit likelihood models", where the observation likelihood lacks an analytic representation. In further analysis on implicit likelihood models we prove that our moment matching solution is equivalent to existing gradient based methods but at a significantly reduced computational cost. Our theoretical results are supported by numerical evaluation, showing that the proposed approach elegantly applies to fully parameterized (explicit likelihood) Gaussian mixture models, as well as implicit models arising from marginalization of nuisance variables. Finally, using the SIR model in epidemiology, we show that our approach easily applies to implicit simulation-based likelihood models, while avoiding costly Likelihood-Free Inference Ratio Estimation (LFIRE) common to such models.