Permutation tests for binary classification

Project "Permutation tests for binary classification" partially supported by the Mathematisches Forschungsinstitut Oberwolfach

Permutation tests are being widely used for various statistical hypotheses testing problems with small sample size. In these cases they can provide more accurate hypothesis testing than the parametric alternatives. Binary classification problems with small sample sizes, which are common in neuroimaging and genomic studies, can also benefit from permutation tests. However, full permutation tests for binary classification problems can be computationally costly since the retraining cost of a learning algorithm is often very high for problems with high feature/representation dimensions, so retraining even for a couple of thousand times is not a desirable option for many studies. Furthermore, their relationship to the parametric approaches, such as Bayesian intervals, are not completely understood so it is hard to understand the advantages of one test with respect to another. Therefore, a theoretical analysis of the permutation tests for binary classification problem is needed. Such an analysis would link the Bayesian intervals and the permutation tests suggesting which test to use in what circumstance. Furthermore, it would yield approximation techniques and bounds for the permutation tests to reduce the computational cost.

Publications:

• Permutation tests for classification:Revisited.