Classification using Versatile Voxels

We use the Haxby's data set and perform classification

We use only logistic regression as the classifier. In this table, we just need to decide if what choice of lanbda we should use. And the result shows that lambda = 0.1 is probably the best.

We then proceed to pick voxels with V>=V_thres to do classification.

The plot of V versus the accuracy (averaged, min, max) and the ratio of voxels used

This is the version 1, using only training set to train the model

Raw output

V: 0.00 min: 75.00% avg: 87.50% max: 100.00% std: 8.33% #voxel: 577

V: 0.04 min: 75.00% avg: 90.00% max: 100.00% std: 7.91% #voxel: 559

V: 0.07 min: 75.00% avg: 87.50% max: 100.00% std: 8.33% #voxel: 542

V: 0.11 min: 75.00% avg: 91.25% max: 100.00% std: 8.44% #voxel: 511

V: 0.14 min: 87.50% avg: 93.75% max: 100.00% std: 6.59% #voxel: 480

V: 0.18 min: 87.50% avg: 91.25% max: 100.00% std: 6.04% #voxel: 453

V: 0.21 min: 87.50% avg: 92.50% max: 100.00% std: 6.45% #voxel: 422

V: 0.25 min: 87.50% avg: 93.75% max: 100.00% std: 6.59% #voxel: 381

V: 0.29 min: 87.50% avg: 93.75% max: 100.00% std: 6.59% #voxel: 322

V: 0.32 min: 87.50% avg: 93.75% max: 100.00% std: 6.59% #voxel: 273

V: 0.36 min: 87.50% avg: 93.75% max: 100.00% std: 6.59% #voxel: 231

V: 0.39 min: 87.50% avg: 93.75% max: 100.00% std: 6.59% #voxel: 186

V: 0.43 min: 75.00% avg: 90.00% max: 100.00% std: 7.91% #voxel: 151

V: 0.46 min: 75.00% avg: 93.75% max: 100.00% std: 8.84% #voxel: 121

V: 0.50 min: 75.00% avg: 91.25% max: 100.00% std: 8.44% #voxel: 88

V: 0.54 min: 75.00% avg: 90.00% max: 100.00% std: 7.91% #voxel: 64

V: 0.57 min: 75.00% avg: 88.75% max: 100.00% std: 9.22% #voxel: 47

V: 0.61 min: 50.00% avg: 85.00% max: 100.00% std: 16.46% #voxel: 32

V: 0.64 min: 62.50% avg: 81.25% max: 100.00% std: 12.15% #voxel: 20

V: 0.68 min: 50.00% avg: 78.75% max: 100.00% std: 13.24% #voxel: 13

V: 0.71 min: 37.50% avg: 62.50% max: 75.00% std: 13.18% #voxel: 7

V: 0.79 min: 12.50% avg: 45.00% max: 75.00% std: 20.58% #voxel: 5

V: 0.82 min: 12.50% avg: 31.25% max: 50.00% std: 12.15% #voxel: 3

V: 0.86 min: 12.50% avg: 40.00% max: 62.50% std: 14.19% #voxel: 2

V: 0.89 min: 25.00% avg: 41.25% max: 62.50% std: 11.86% #voxel: 1

This is version 2, using both training and cv set to train the model

V: 0.00 min: 75.00% avg: 87.50% max: 100.00% std: 10.21% #voxel: 577

V: 0.04 min: 87.50% avg: 95.00% max: 100.00% std: 6.45% #voxel: 559

V: 0.07 min: 75.00% avg: 91.25% max: 100.00% std: 8.44% #voxel: 542

V: 0.11 min: 75.00% avg: 93.75% max: 100.00% std: 10.62% #voxel: 511

V: 0.14 min: 87.50% avg: 90.00% max: 100.00% std: 5.27% #voxel: 480

V: 0.18 min: 75.00% avg: 91.25% max: 100.00% std: 8.44% #voxel: 453

V: 0.21 min: 87.50% avg: 92.50% max: 100.00% std: 6.45% #voxel: 422

V: 0.25 min: 87.50% avg: 95.00% max: 100.00% std: 6.45% #voxel: 381

V: 0.29 min: 87.50% avg: 93.75% max: 100.00% std: 6.59% #voxel: 322

V: 0.32 min: 87.50% avg: 93.75% max: 100.00% std: 6.59% #voxel: 273

V: 0.36 min: 87.50% avg: 96.25% max: 100.00% std: 6.04% #voxel: 231

V: 0.39 min: 87.50% avg: 96.25% max: 100.00% std: 6.04% #voxel: 186

V: 0.43 min: 87.50% avg: 96.25% max: 100.00% std: 6.04% #voxel: 151

V: 0.46 min: 87.50% avg: 96.25% max: 100.00% std: 6.04% #voxel: 121

V: 0.50 min: 75.00% avg: 92.50% max: 100.00% std: 10.54% #voxel: 88

V: 0.54 min: 87.50% avg: 95.00% max: 100.00% std: 6.45% #voxel: 64

V: 0.57 min: 75.00% avg: 88.75% max: 100.00% std: 7.10% #voxel: 47

V: 0.61 min: 62.50% avg: 88.75% max: 100.00% std: 13.76% #voxel: 32

V: 0.64 min: 62.50% avg: 78.75% max: 87.50% std: 10.29% #voxel: 20

V: 0.68 min: 62.50% avg: 77.50% max: 100.00% std: 11.49% #voxel: 13

V: 0.71 min: 25.00% avg: 60.00% max: 87.50% std: 17.48% #voxel: 7

V: 0.79 min: 0.00% avg: 47.50% max: 87.50% std: 26.22% #voxel: 5

V: 0.82 min: 0.00% avg: 35.00% max: 50.00% std: 15.37% #voxel: 3

V: 0.86 min: 25.00% avg: 41.25% max: 50.00% std: 8.44% #voxel: 2

V: 0.89 min: 25.00% avg: 42.50% max: 62.50% std: 10.54% #voxel: 1

Again, this is a "peek-type" classification though--namely, I calculate the distinguishability for all the voxels in the brain using all the samples available regardless of they are in the train or test set. It makes sense for now because this experiment does not aim for the highest classification accuracy, but just to see how much the classification accuracy can be improved when the non-informative voxels are gradually removed step by step. So, from this curve we can say the following:

We can use distinguishability as a preprocessing part just to remove the non-informative voxels, say remove voxels with V<0.4. The number of the remaining voxels are less than 40%, but the averaged accuracy is as high as 90%.
Version 2 is better than version 1. Training set's size does matter!

Now, the whole brain version1

The raw output

V: 0.00 min: 12.50% avg: 37.50% max: 62.50% std: 17.68% #voxel: 43193

V: 0.04 min: 37.50% avg: 57.50% max: 87.50% std: 16.87% #voxel: 27899

V: 0.07 min: 50.00% avg: 63.75% max: 87.50% std: 12.43% #voxel: 19595

V: 0.11 min: 50.00% avg: 76.25% max: 87.50% std: 12.43% #voxel: 13718

V: 0.14 min: 62.50% avg: 78.75% max: 100.00% std: 11.86% #voxel: 9354

V: 0.18 min: 62.50% avg: 77.50% max: 100.00% std: 14.19% #voxel: 6280

V: 0.21 min: 62.50% avg: 82.50% max: 87.50% std: 8.74% #voxel: 4206

V: 0.25 min: 75.00% avg: 90.00% max: 100.00% std: 7.91% #voxel: 2787

V: 0.29 min: 75.00% avg: 88.75% max: 100.00% std: 7.10% #voxel: 1839

V: 0.32 min: 75.00% avg: 90.00% max: 100.00% std: 7.91% #voxel: 1275

V: 0.36 min: 87.50% avg: 93.75% max: 100.00% std: 6.59% #voxel: 886

V: 0.39 min: 75.00% avg: 92.50% max: 100.00% std: 8.74% #voxel: 653

V: 0.43 min: 75.00% avg: 93.75% max: 100.00% std: 8.84% #voxel: 461

V: 0.46 min: 75.00% avg: 96.25% max: 100.00% std: 8.44% #voxel: 317

V: 0.50 min: 75.00% avg: 96.25% max: 100.00% std: 8.44% #voxel: 221

V: 0.54 min: 75.00% avg: 93.75% max: 100.00% std: 8.84% #voxel: 160

V: 0.57 min: 75.00% avg: 93.75% max: 100.00% std: 8.84% #voxel: 107

V: 0.61 min: 75.00% avg: 93.75% max: 100.00% std: 10.62% #voxel: 69

V: 0.64 min: 75.00% avg: 92.50% max: 100.00% std: 8.74% #voxel: 40

V: 0.68 min: 75.00% avg: 88.75% max: 100.00% std: 10.94% #voxel: 22

V: 0.71 min: 50.00% avg: 71.25% max: 87.50% std: 13.24% #voxel: 10

V: 0.79 min: 12.50% avg: 45.00% max: 75.00% std: 20.58% #voxel: 5

V: 0.82 min: 12.50% avg: 31.25% max: 50.00% std: 12.15% #voxel: 3

V: 0.86 min: 12.50% avg: 40.00% max: 62.50% std: 14.19% #voxel: 2

V: 0.89 min: 25.00% avg: 41.25% max: 62.50% std: 11.86% #voxel: 1

Now, whole brain version 2

V: 0.00 min: 12.50% avg: 37.50% max: 75.00% std: 20.41% #voxel: 43193

V: 0.04 min: 37.50% avg: 55.00% max: 75.00% std: 14.67% #voxel: 27899

V: 0.07 min: 50.00% avg: 66.25% max: 87.50% std: 13.24% #voxel: 19595

V: 0.11 min: 75.00% avg: 80.00% max: 100.00% std: 8.74% #voxel: 13718

V: 0.14 min: 62.50% avg: 80.00% max: 100.00% std: 12.08% #voxel: 9354

V: 0.18 min: 75.00% avg: 86.25% max: 100.00% std: 7.10% #voxel: 6280

V: 0.21 min: 75.00% avg: 86.25% max: 100.00% std: 9.22% #voxel: 4206

V: 0.25 min: 62.50% avg: 83.75% max: 100.00% std: 14.49% #voxel: 2787

V: 0.29 min: 50.00% avg: 78.75% max: 87.50% std: 13.24% #voxel: 1839

V: 0.32 min: 62.50% avg: 82.50% max: 100.00% std: 10.54% #voxel: 1275

V: 0.36 min: 75.00% avg: 92.50% max: 100.00% std: 8.74% #voxel: 886

V: 0.39 min: 87.50% avg: 97.50% max: 100.00% std: 5.27% #voxel: 653

V: 0.43 min: 87.50% avg: 96.25% max: 100.00% std: 6.04% #voxel: 461

V: 0.46 min: 75.00% avg: 97.50% max: 100.00% std: 7.91% #voxel: 317

V: 0.50 min: 75.00% avg: 96.25% max: 100.00% std: 8.44% #voxel: 221

V: 0.54 min: 75.00% avg: 97.50% max: 100.00% std: 7.91% #voxel: 160

V: 0.57 min: 75.00% avg: 97.50% max: 100.00% std: 7.91% #voxel: 107

V: 0.61 min: 87.50% avg: 97.50% max: 100.00% std: 5.27% #voxel: 69

V: 0.64 min: 87.50% avg: 96.25% max: 100.00% std: 6.04% #voxel: 40

V: 0.68 min: 75.00% avg: 90.00% max: 100.00% std: 9.86% #voxel: 22

V: 0.71 min: 62.50% avg: 73.75% max: 87.50% std: 10.94% #voxel: 10

V: 0.79 min: 0.00% avg: 47.50% max: 87.50% std: 26.22% #voxel: 5

V: 0.82 min: 0.00% avg: 35.00% max: 50.00% std: 15.37% #voxel: 3

V: 0.86 min: 25.00% avg: 41.25% max: 50.00% std: 8.44% #voxel: 2

V: 0.89 min: 25.00% avg: 42.50% max: 62.50% std: 10.54% #voxel: 1

Comparison of the accuracy from all the methods

Things I should do:

In this peek type, use both train and cv to train the model, because we don't have to pick free parameters using cv set anyway.
1. Compare the performance curves from {WB, VTC} x {version1, 2}
It is good to test if the model is overfitting or underfitting the data, and wether or not the number of samples affect the result. (Following the steps in Andrew's sheet.)
Prove that the information in the brain is distributed by using the non-informative voxels. First, calculate the accuracy using the smallest-V voxel alone, then gradually add the better-V voxel one by one and see if the accuracy increases a lot and have "joint behavior" or not.
Plot the correlation between single-voxel classification accuracy vs its V. I expect to see high correlation between V and single-voxel accuracy.
Now let's find the synergic behavior on both versatile and non-informative voxels:
1. For every voxel in the brain, calculate its single-voxel accuracy, that is, using one voxel to train/test.
2. In versatile group (V>0.4), randomly pick 10 voxels at a time, calculate the average accuracy using all the voxels simultaneously.
3. Plot the average accuracy vs the average V from the group.
4. We expect to see high correlation in this versatile group.
5. Now, we perform 2-4 on the non-informative voxels, but I expect to see the correlation is low because of the synergy.