Embedded Files
Cross validation is useful and employed in many situations in statistics and machine learning. In this subpage, let us study a mathematical property of the Bayes leave-one-out cross validation.
Cross validation is useful and employed in many situations in statistics and machine learning. In this subpage, let us study a mathematical property of the Bayes leave-one-out cross validation.
Let p(x|w) be a statistical model, phi(w) be a prior, and X1,X2,...,Xn be independent random variables whose data-generating distribution q(x) is unknown.
Let p(x|w) be a statistical model, phi(w) be a prior, and X1,X2,...,Xn be independent random variables whose data-generating distribution q(x) is unknown.
Both the posterior and predictive distributions are defined by the usual way. The predictive distribution is an estimated probability density of unknown data-generating distribution. The estimated p(x|data) is different from unknown q(x).
Both the posterior and predictive distributions are defined by the usual way. The predictive distribution is an estimated probability density of unknown data-generating distribution. The estimated p(x|data) is different from unknown q(x).
Both the generalization loss and cross validation loss are defined. Here we study the Bayes leave-one-out cross validation loss.
Both the generalization loss and cross validation loss are defined. Here we study the Bayes leave-one-out cross validation loss.
Average and empirical loss functions are defined. Note that the average of Ln(w) is equal to L(w). If q(x) is realizable by p(x|w), then q(x)=p(x|w0), resulting that L(w0) and Ln(w0) are equal to the average and empirical entropies of q(x), respectively.
Average and empirical loss functions are defined. Note that the average of Ln(w) is equal to L(w). If q(x) is realizable by p(x|w), then q(x)=p(x|w0), resulting that L(w0) and Ln(w0) are equal to the average and empirical entropies of q(x), respectively.
These are two main theorems about Bayes cross validation.
These are two main theorems about Bayes cross validation.
Theorem 1: The averages of the cross validation loss and the generalization loss are the same.
Theorem 1: The averages of the cross validation loss and the generalization loss are the same.
Theorem 2: The cross validation error (Cn-Ln(w0)) and the generalization error (Gn-L(w0)) have the negative correlation. Here lambda is the real log canonical threshold (RLCT).
Theorem 2: The cross validation error (Cn-Ln(w0)) and the generalization error (Gn-L(w0)) have the negative correlation. Here lambda is the real log canonical threshold (RLCT).
Theorem 1 may be trivial but Theorem 2 is not trivial. Unfortunately, even expert statisticians may not know this fact.
Theorem 1 may be trivial but Theorem 2 is not trivial. Unfortunately, even expert statisticians may not know this fact.
Note: Gn and Cn do not have negative correlation in general. However, (Gn-L(w0)) and (Cn-Ln(w0)) have negative correlation. It should be emphasized that such a property cannot be derived from the definition of the cross validation. We need singular learning theory. As is shown in the following, it is easy to check this negative correlation by a computer simulation.
If you need the proof of these theorems and the definition of RLCT, please see,
(1) S. Watanabe, Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory. Journal of Machine Learning Research, vol11, pp.3571-3594, 2010.
These theorems show that minimization of the cross validation error may be different from minimization of the generalization error.
These theorems show that minimization of the cross validation error may be different from minimization of the generalization error.
When the cross validation error is understood as an estimator of the generalization error, it is unbiased but has a negative correlation. Note that information criteria have asymptotically the same property. When two models are compared, the model that has the smaller average cross validation loss has the average smaller generalization loss, however, the model that has the smaller cross validation loss as a random variable may not have the smaller generalization loss as a random variable.
When the cross validation error is understood as an estimator of the generalization error, it is unbiased but has a negative correlation. Note that information criteria have asymptotically the same property. When two models are compared, the model that has the smaller average cross validation loss has the average smaller generalization loss, however, the model that has the smaller cross validation loss as a random variable may not have the smaller generalization loss as a random variable.
It is conjectured that there is no estimator of the generalization error that is unbiased and has nonnegative correlation. If you solved this conjecture positively or negatively, it is worthwhile to publish it.
It is conjectured that there is no estimator of the generalization error that is unbiased and has nonnegative correlation. If you solved this conjecture positively or negatively, it is worthwhile to publish it.
Note : Hold-out cross validation, which is often referred to as out-of-sample test, has much larger variance than leave-one-out cross validation. Therefore hold-out cross-validation is generally not recommended, except in cases where there is no other viable option.
Let us illustrate the negative correlation between the generalization error and the cross-validation error using a polynomial model.
Let us illustrate the negative correlation between the generalization error and the cross-validation error using a polynomial model.
This is a data-generating distribution used in the experiment. You can use another a0 or another q(y|x) which is not realizable by p(y|x,a).
This is a data-generating distribution used in the experiment. You can use another a0 or another q(y|x) which is not realizable by p(y|x,a).
Note: We recommend trying cross-validation using real-world data. However, because the generalization error cannot be calculated for real-world data, it is difficult to observe the negative correlation between the cross-validation error and the generalization error. Only comparing a sample and predictive distribution, such negative correlation cannot be discovered.
Note: We recommend trying cross-validation using real-world data. However, because the generalization error cannot be calculated for real-world data, it is difficult to observe the negative correlation between the cross-validation error and the generalization error. Only comparing a sample and predictive distribution, such negative correlation cannot be discovered.
This is a result for the first case, d=5 and n=10. We can find the negative correlation between (Cn-Ln(w0)) and (Gn-L(w0)), even in a non-asymptotic case.
This is a result for the first case, d=5 and n=10. We can find the negative correlation between (Cn-Ln(w0)) and (Gn-L(w0)), even in a non-asymptotic case.
This is a result for the second case, d=5 and n=30. Also we find negative correlation.
This is a result for the second case, d=5 and n=30. Also we find negative correlation.
This is the third case, d=5, n=100. Also we find negative correlation. This case becomes closer to an asymptotic case.
This is the third case, d=5, n=100. Also we find negative correlation. This case becomes closer to an asymptotic case.
This is the fourth case, d=5, n=500. Also we find negative correlation. Note that the standard deviation of (Gn-L(w0)) is in proportion to 2 lambda/n. Thus the negative correlation does not disappear, even if n becomes large.
This is the fourth case, d=5, n=500. Also we find negative correlation. Note that the standard deviation of (Gn-L(w0)) is in proportion to 2 lambda/n. Thus the negative correlation does not disappear, even if n becomes large.
Even for the maximum likelihood method, the same negative correlation can be shown. The tendency of the negative correlation between cross validation error and generalization error can be found in many statistical estimation methods, because statistical estimation uses optimization for the training data. By the definition, the averages of cross validation error and generalization error are the same. However, many people even expert researchers may not know this negative correlation. We had better know this fact when we use cross validation in practical applications. For example, when you compare cross validation losses of two statistical models, if difference of them is very large, model choice by cross validation works well, however, if it is not so large, may be not. When many models have almost same cross validation losses, it is not appropriate to choose the model that has the minimum cross validation loss. In such cases, the free energy, which is the minus log marginal likelihood, may serve as a helpful aid for model comparison. However, of course, the free energy itself is not a criterion for predictive performance.
Even for the maximum likelihood method, the same negative correlation can be shown. The tendency of the negative correlation between cross validation error and generalization error can be found in many statistical estimation methods, because statistical estimation uses optimization for the training data. By the definition, the averages of cross validation error and generalization error are the same. However, many people even expert researchers may not know this negative correlation. We had better know this fact when we use cross validation in practical applications. For example, when you compare cross validation losses of two statistical models, if difference of them is very large, model choice by cross validation works well, however, if it is not so large, may be not. When many models have almost same cross validation losses, it is not appropriate to choose the model that has the minimum cross validation loss. In such cases, the free energy, which is the minus log marginal likelihood, may serve as a helpful aid for model comparison. However, of course, the free energy itself is not a criterion for predictive performance.
In conclusion, cross validation is useful and powerful in practical applications, however, it is important to be cautious about relying on it too heavily. The same caution applies to information criteria.
In conclusion, cross validation is useful and powerful in practical applications, however, it is important to be cautious about relying on it too heavily. The same caution applies to information criteria.
Note: When a leverage sample point is contained in a sample, then the cross validation error becomes large, whereas the generalization error small. In such cases, information criteria may be more useful than cross validations. Cross validation requires {(Xi,Yi)} are independent, however, information criteria does only {(Yi|xi)} are independent and allows that {xi} may be dependent or fixed. Cross validation is not necessary superior to information criteria in every situation.
Note: When a leverage sample point is contained in a sample, then the cross validation error becomes large, whereas the generalization error small. In such cases, information criteria may be more useful than cross validations. Cross validation requires {(Xi,Yi)} are independent, however, information criteria does only {(Yi|xi)} are independent and allows that {xi} may be dependent or fixed. Cross validation is not necessary superior to information criteria in every situation.
Sumio Watanabe, Information criteria and cross validation for Bayesian inference in regular and singular cases. Japanese Journal of Statistics and Data Science, vol.4, pp.1-19, 2021. link.springer.com/article/10.1007/s42081-021-00121-3
Sumio Watanabe, Information criteria and cross validation for Bayesian inference in regular and singular cases. Japanese Journal of Statistics and Data Science, vol.4, pp.1-19, 2021. link.springer.com/article/10.1007/s42081-021-00121-3
Page updated
Google Sites
Report abuse