Variational Inference

Exact Bayesian inference is not possible for nonlinear models. Instead, one must use approximate inference frameworks, and Variational Inference is one such approach. It factorises the posterior density and optimises the parameters of the factors so as to minimise the KL divergence between the true and approximate posterior. It also provides a lower bound on the the model evidence. This bound is the "negative variational free energy".

• W. Penny, S. Kiebel, and K. Friston. Variational Bayes. In K. Friston, J. Ashburner, S. Kiebel, T. Nichols, and W. Penny, editors, Statistical Parametric Mapping: The analysis of functional brain images. Elsevier, London, 2006. PDF

• K.Friston, J. Mattout, N. Trujillo-Barreto, J. Ashburner, and W. Penny. Variational free energy and the Laplace approximation. Neuroimage, 34(1):220-234, 2007. PDF

• W. D. Penny, J. Kilner, and F. Blankenburg. Robust Bayesian General Linear Models. Neuroimage, 36(3):661-671, 2007. PDF

• W.D. Penny, S.J. Kiebel, and K.J. Friston. Variational Bayesian Inference for fMRI time series. NeuroImage, 19(3):727-741, 2003. PDF

• M.J. Cassidy and W.Penny. Bayesian nonstationary autogregressive models for biomedical signal analysis. IEEE Transactions on Biomedical Engineering, 49(10):1142-1152, 2002. PDF

• W.D. Penny and S.J. Roberts. Bayesian Multivariate Autoregresive Models with structured priors. IEE Proceedings on Vision, Image and Signal Processing, 149(1):33-41, 2002. PDF

• S.J. Roberts and W.D. Penny. Variational Bayes for Generalised Autoregressive models. IEEE Transactions on Signal Processing, 50(9):2245-2257, 2002. PDF

The original idea behind variational inference was proposed in the context of neural networks, whereby the suggestion was to replace the standard cost function (e.g. mean square error or "energy") with the "variational free energy", F, so as to improve generalisation performance.