Embedded Files
Here we study Bayes free energy which has many other names such as minus log marginal likelihood, Bayesian stochastic complexity, and Bayesian code length. Sometimes the same concept is called as Bayes factor. The importance of this concept in Bayesian statistics was firstly proposed by statistician I. J. Good.
Here we study Bayes free energy which has many other names such as minus log marginal likelihood, Bayesian stochastic complexity, and Bayesian code length. Sometimes the same concept is called as Bayes factor. The importance of this concept in Bayesian statistics was firstly proposed by statistician I. J. Good.
(Remark) In this page, we do not explain the meaning of free energy from the viewpoint of statistical mechanics. If you are interested in the relation between statistical mechanics and machine learning, please see here. It is interesting that the mathematically same concept is called by many different names.
(Remark) In this page, we do not explain the meaning of free energy from the viewpoint of statistical mechanics. If you are interested in the relation between statistical mechanics and machine learning, please see here. It is interesting that the mathematically same concept is called by many different names.
This is a basic setting; a statistical model, a prior distribution, and a sample which is generated from unknown distribution q(x).
This is a basic setting; a statistical model, a prior distribution, and a sample which is generated from unknown distribution q(x).
Posterior and predictive distributions are defined by these equations, respectively. Here the normalizing constant Z is called the marginal likelihood.
Posterior and predictive distributions are defined by these equations, respectively. Here the normalizing constant Z is called the marginal likelihood.
Generalization loss and Free energy are defined. Free energy is equal to the minus log marginal likelihood.
Generalization loss and Free energy are defined. Free energy is equal to the minus log marginal likelihood.
Average and empirical loss functions, L(w) and Ln(w), are defined. The parameter that minimizes L(w) is denoted by w0. In practical applications, q(x) is unknown, hence w0 is also unknown.
Average and empirical loss functions, L(w) and Ln(w), are defined. The parameter that minimizes L(w) is denoted by w0. In practical applications, q(x) is unknown, hence w0 is also unknown.
Fisher information matrix is defined. If Fisher information matrix is positive definite, if q(x) is realizable by p(x|w), and if the sample size is sufficiently large, then the posterior distribution can be approximated by a normal distribution whose covariance matrix is (nI(w0))^{-1}. Such a case is called regular.
Fisher information matrix is defined. If Fisher information matrix is positive definite, if q(x) is realizable by p(x|w), and if the sample size is sufficiently large, then the posterior distribution can be approximated by a normal distribution whose covariance matrix is (nI(w0))^{-1}. Such a case is called regular.
It is easy to show that the increase of the free energy is equal to the generalization loss.
It is easy to show that the increase of the free energy is equal to the generalization loss.
The generalization loss is - log p(Xn|X1,X2,...,Xn-1), where as the free energy is -log p(X1,X2,...,Xn). Both are important in evaluation of a statistical model and a prior distribution, however, they are different. Therefore the model that minimizes the generalization loss is, in general, different from the model that minimizes the free energy. This is not a paradox.
The generalization loss is - log p(Xn|X1,X2,...,Xn-1), where as the free energy is -log p(X1,X2,...,Xn). Both are important in evaluation of a statistical model and a prior distribution, however, they are different. Therefore the model that minimizes the generalization loss is, in general, different from the model that minimizes the free energy. This is not a paradox.
In regular cases, Bayesian information criterion (BIC) was proposed by G. E. Schwarz (1). In singular cases, (d/2) is replaced by the estimated real log canonical threshold (RLCT) and sBIC was proposed by M. Drton and M. Plummer (2).
In regular cases, Bayesian information criterion (BIC) was proposed by G. E. Schwarz (1). In singular cases, (d/2) is replaced by the estimated real log canonical threshold (RLCT) and sBIC was proposed by M. Drton and M. Plummer (2).
(1) G. E. Schwarz, Estimating the dimension of a model, Annals of Statistics, Vol.6, pp.461-464, 1978.
(2) M. Drton, M. Plummer, A Bayesian information criterion for singular models. Journal of Royal Statistical Society Series B. Vol.79, pp.323-380, 2017.
Widely applicable Bayesian information criterion (WBIC) is defined by this equation (3).
Widely applicable Bayesian information criterion (WBIC) is defined by this equation (3).
(3) S. Watanabe, A widely applicable Bayesian information criterion, Journal of Machine Learning Research, vol.14, pp.867-897, 2013.
There are several relations between the free energy, BIC, and WBIC.
There are several relations between the free energy, BIC, and WBIC.
In general, it needs heavy computational cost to numerically calculate the free energy.
In general, it needs heavy computational cost to numerically calculate the free energy.
Experiments are conducted using this model and prior. In this special model, the free energy can be calculated analytically, hence it is easy to compare the free energy with BIC and WBIC.
Experiments are conducted using this model and prior. In this special model, the free energy can be calculated analytically, hence it is easy to compare the free energy with BIC and WBIC.
(Remark) Unfortunately, in almost all singular models, the free energies cannot be analytically calculated.
This is the data-generating distribution used in experiments. We study three different cases for q(x).
This is the data-generating distribution used in experiments. We study three different cases for q(x).
Note: This p(y|x,a,s) is a regular model, however, when the support of q(x) is narrow, some eigen values of Fisher information matrix is close to zero.
The empirical entropy term n Ln(w0) = n Sn is the common term in the free energy, BIC, and WBIC. Hence it is reduced from each value in comparison.
The empirical entropy term n Ln(w0) = n Sn is the common term in the free energy, BIC, and WBIC. Hence it is reduced from each value in comparison.
In the first case (1), q(x) was set as a uniform distribution on [-3,3]. Horizontal line shows the sample size. In this case, Fisher information matrix is positive definite, hence difference between the free energy and BIC is a constant order term and the difference between BIC and WBIC is a smaller order term than constant.
In the first case (1), q(x) was set as a uniform distribution on [-3,3]. Horizontal line shows the sample size. In this case, Fisher information matrix is positive definite, hence difference between the free energy and BIC is a constant order term and the difference between BIC and WBIC is a smaller order term than constant.
Eigenvalues of Fisher information matrix are shown. All of them are positive.
Eigenvalues of Fisher information matrix are shown. All of them are positive.
In the second case (2), q(x) is set as a uniform distribution on [-1,1]. In this case, some eigenvalues of Fisher information matrix are close to zero. Difference between free energy and BIC is larger than that of free energy and WBIC.
In the second case (2), q(x) is set as a uniform distribution on [-1,1]. In this case, some eigenvalues of Fisher information matrix are close to zero. Difference between free energy and BIC is larger than that of free energy and WBIC.
The eigenvalues of Fisher information matrix are shown. Remark that they are quite different.
The eigenvalues of Fisher information matrix are shown. Remark that they are quite different.
In the third case (3), q(x) is set as a uniform distribution on [-1/3,1/3]. Some eigenvalues of Fisher information matrix are almost zero. Difference between free energy and BIC is larger than that of free energy and WBIC.
In the third case (3), q(x) is set as a uniform distribution on [-1/3,1/3]. Some eigenvalues of Fisher information matrix are almost zero. Difference between free energy and BIC is larger than that of free energy and WBIC.
The eigenvalues of this case are shown. It is not appropriate that the posterior distribution is understood as regular.
The eigenvalues of this case are shown. It is not appropriate that the posterior distribution is understood as regular.
Free energy is an important concept in Bayesian statistics in general, however, it needs heavy computational cost to calculate it. BIC cannot be employed when some eigenvalues of Fisher information matrix are close to zero. This is because the posterior distribution cannot be approximated by any normal distribution. WBIC can be employed even in singular cases.
Free energy is an important concept in Bayesian statistics in general, however, it needs heavy computational cost to calculate it. BIC cannot be employed when some eigenvalues of Fisher information matrix are close to zero. This is because the posterior distribution cannot be approximated by any normal distribution. WBIC can be employed even in singular cases.
If you need more information about applications to statistics such as model selection, please see here.
Page updated
Google Sites
Report abuse