Embedded Files
Cross validation is useful and employed in many situations in statistics and machine learning. In this subpage, let us study a mathematical property of the Bayes leave-one-out cross validation.
Cross validation is useful and employed in many situations in statistics and machine learning. In this subpage, let us study a mathematical property of the Bayes leave-one-out cross validation.
Let p(x|w) be a statistical model, phi(w) be a prior, and X1,X2,...,Xn be independent random variables whose data-generating distribution q(x) is unknown.
Let p(x|w) be a statistical model, phi(w) be a prior, and X1,X2,...,Xn be independent random variables whose data-generating distribution q(x) is unknown.
Both the posterior and predictive distributions are defined by the usual way. The predictive distribution is an estimated probability density of unknown data-generating distribution. The estimated p(x|data) is different from unknown q(x).
Both the posterior and predictive distributions are defined by the usual way. The predictive distribution is an estimated probability density of unknown data-generating distribution. The estimated p(x|data) is different from unknown q(x).
Both the generalization loss and cross validation loss are defined. Here we study the Bayes leave-one-out cross validation loss.
Both the generalization loss and cross validation loss are defined. Here we study the Bayes leave-one-out cross validation loss.
Average and empirical loss functions are defined. Note that the average of Ln(w) is equal to L(w). If q(x) is realizable by p(x|w), then q(x)=p(x|w0), resulting that L(w0) and Ln(w0) are equal to the average and empirical entropies of q(x), respectively.
Average and empirical loss functions are defined. Note that the average of Ln(w) is equal to L(w). If q(x) is realizable by p(x|w), then q(x)=p(x|w0), resulting that L(w0) and Ln(w0) are equal to the average and empirical entropies of q(x), respectively.
These are two main theorems about Bayes cross validation. Theorem 1: The averages of the cross validation loss and the generalization loss are the same. Theorem 2: The cross validation error (Cn-Ln(w0)) and the generalization error (Gn-L(w0)) have the negative correlation. Here lambda is the real log canonical threshold (RLCT). Theorem 1 may be trivial but Theorem 2 is not trivial.
These are two main theorems about Bayes cross validation. Theorem 1: The averages of the cross validation loss and the generalization loss are the same. Theorem 2: The cross validation error (Cn-Ln(w0)) and the generalization error (Gn-L(w0)) have the negative correlation. Here lambda is the real log canonical threshold (RLCT). Theorem 1 may be trivial but Theorem 2 is not trivial.
If you need the proof of these theorems and the definition of RLCT, please see,
(1) S. Watanabe, Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory. Journal of Machine Learning Research, vol11, pp.3571-3594, 2010.
These theorems show that minimization of the cross validation error may be different from minimization of the generalization error.
These theorems show that minimization of the cross validation error may be different from minimization of the generalization error.
When the cross validation error is understood as an estimator of the generalization error, it is unbiased but has a negative correlation. Note that information criteria have asymptotically the same property. When two models are compared, the model that has the smaller average cross validation loss has the average smaller generalization loss, however, the model that has the smaller cross validation loss as a random variable may not have the smaller generalization loss as a random variable.
When the cross validation error is understood as an estimator of the generalization error, it is unbiased but has a negative correlation. Note that information criteria have asymptotically the same property. When two models are compared, the model that has the smaller average cross validation loss has the average smaller generalization loss, however, the model that has the smaller cross validation loss as a random variable may not have the smaller generalization loss as a random variable.
It is conjectured that there is no estimator of the generalization error that is unbiased and has nonnegative correlation. If you solved this conjecture positively or negatively, it is worthwhile to publish it.
It is conjectured that there is no estimator of the generalization error that is unbiased and has nonnegative correlation. If you solved this conjecture positively or negatively, it is worthwhile to publish it.
Note : Hold-out cross validation, which is often referred to as out-of-sample test, has much larger variance than leave-one-out cross validation. Therefore hold-out cross-validation is generally not recommended, except in cases where there is no other viable option.
Let us illustrate the negative correlation of the generalization error and the cross validation error in a case of the polynomial regression.
Let us illustrate the negative correlation of the generalization error and the cross validation error in a case of the polynomial regression.
This is a data-generating distribution used in the experiment.
This is a data-generating distribution used in the experiment.
This is a result for the first case, d=5 and n=10. We can find the negative correlation between (Cn-Ln(w0)) and (Gn-L(w0)), even in a non-asymptotic case.
This is a result for the first case, d=5 and n=10. We can find the negative correlation between (Cn-Ln(w0)) and (Gn-L(w0)), even in a non-asymptotic case.
This is a result for the second case, d=5 and n=30. Also we find negative correlation.
This is a result for the second case, d=5 and n=30. Also we find negative correlation.
This is the third case, d=5, n=100. Also we find negative correlation. This case becomes closer to an asymptotic case.
This is the third case, d=5, n=100. Also we find negative correlation. This case becomes closer to an asymptotic case.
Even for the maximum likelihood method, the same negative correlation can be shown. The tendency of the negative correlation between training and generalization can be found in many statistical estimation methods, because statistical estimation uses optimization for the traning data. By the definition, the averages of cross validation error and generalization error are the same. However, many people may not know this negative correlation. We had better know this fact when we use cross validation in practical applications. For example, when you compare cross validation losses of two statistical models, if difference of them is very large, model choice by cross validation works well, however, if it is not so large, may be not. When many models have almost same cross validation losses, it is not appropriate to choose the model that has the minimum cross validation loss.
Even for the maximum likelihood method, the same negative correlation can be shown. The tendency of the negative correlation between training and generalization can be found in many statistical estimation methods, because statistical estimation uses optimization for the traning data. By the definition, the averages of cross validation error and generalization error are the same. However, many people may not know this negative correlation. We had better know this fact when we use cross validation in practical applications. For example, when you compare cross validation losses of two statistical models, if difference of them is very large, model choice by cross validation works well, however, if it is not so large, may be not. When many models have almost same cross validation losses, it is not appropriate to choose the model that has the minimum cross validation loss.
In conclusion, cross validation is useful and powerful in practical applications, however, it is important to be cautious about relying on it too heavily. The same remark is necessary for information criteria.
In conclusion, cross validation is useful and powerful in practical applications, however, it is important to be cautious about relying on it too heavily. The same remark is necessary for information criteria.
Page updated
Google Sites
Report abuse