sparse regularization on lexical data

Here are some results from using L1- and L2-norm with logistic regression on lexical data

Data matrices produced

The datamatrix is stored in the dir:

/NAS_II/Projects/MVPA_Language/lexical/${subj}/stats_images_fmri

each with the name:

${data}_sexemplar_beta_tstat_matrix_${mask}.mat

where

${data} = 'animtool', 'mamnonmam', 'allmamnonmam'

${mask} = 'lh_mask_vtc2', 'mask_vtc2', 'gray_mask2'

The work flow and the data for each step

Rawdata --> whole scan datamatrix --> task-specific data matrix --> sweep-parameter classification results --> best-parameter classification result for each subject

Raw data --->

some code, I don't remember

---> whole scan datamatrix --->

/mnt/home/kittipat/Dropbox/random_matlab_code/fMRI/lexical_project/rdm/

make_datamatrix_allmamnonmam.m

make_datamatrix_animtool.m

make_datamatrix_mamnonmam.m

---> task-specific data matrix --->

/mnt/home/kittipat/Dropbox/random_matlab_code/fMRI/lexical_project/lr_regularization/

experiments_animtool_4fold.m

experiments_mamnonmam_4fold.m

experiments_animtool_lou.m

experiments_mamnonmam_lou.m

---> task-specific data matrix --->

/mnt/home/kittipat/Dropbox/random_matlab_code/fMRI/lexical_project/lr_regularization/

summarize_out_4fold.m

summarize_out_4fold.m (not completed yet)

---> the best results from a selected criterion

More details regarding the stored files.

raw data

whole scan data matrix

task-mask-specific data matrix

sweep-parameter classification results

best-parameter classification result for each subject

Experimental data produced

Each data can be treated by 2 ways:

• 4fold = 4-fold cross validation, sweep over alpha, lambda. Report only one best accuracy. Sweeping lambda = 0 --> 1, alpha = 0-->1. We evaluate train, validation and test.
  • file: ${subj}_${mask}_combined_${type}_4fold
• lou = leave-one-out, sweep over alpha, lambda
  • file: ${subj}_${mask}_combined_${type}_lou

Official results#1: 4-fold cross validation

(using the code summarize_out_4fold.m)

There are 2 experiments:

• animals vs tools
• mammals vs nonmammals

Each reported on 3 masks

• lh_mask_vtc2
• mask_vtc2
• gray_mask2

There are 3 types of classifiers used in this experiment:

• LR+lasso
• LR+ridge
• LR+elastic net
• LR alone without regularization

For optimal parameter selection, I use 2 criteria:

• criterion#1: pick the parameter (lambda, alpha) based on the maximum validation accuracy alone. However, the consistence of voxels across 4 folds using this criterion is not desirable, so I propose using another criterion.
• criterion#2: pick the parameter (lambda, alpha) based on 1) the train accuracy must greater than 0.85, 2) the number of nonzeros coefficients (selected voxels) must greater than 20 3) pick the parameters where the validation accuracy is greatest 4) Use alpha and lambda as the tie-breaker, the greater is preferable as it's more sparse--> greater interpretability.

The report results include

• The optimally selected parameters alpha* lambda* for 2 criteria.
• The accuracy for train, validation and test set: acc_train, acc_valid, acc_test
• The sparsity in terms of number of non-zero coefficients
• The consistency of selected voxels across 4 folds in the same subject. The consistency can be measured by how many times it is included in each fold. There are 4 levels:
  • #intersect>0: number of voxels that appears at least in one fold
  • #intersect>0.25: number of voxels that appears at least 1/4 of the folds
  • #intersect>0.5: number of voxels that appears at least 1/2 of the folds
  • #intersect>0.75: : number of voxels that appears at least 3/4 of the folds
  • #intersect=1: number of voxels that appears in every fold.
  • #intersect=0: number of voxels that never appears in any fold.

The data log for post-processing data for each subject (run using the code summarize_out_4fold.m).

Correlation of the posterior

The summary table for lexical data

The summary bar plot when averaging across subjects.

Official results#2: leave-one-observation-out cross validation

(using the code summarize_out_lou.m)

There are 2 experiments:

• animals vs tools
• mammals vs nonmammals

Each reported on 3 masks

• lh_mask_vtc2
• mask_vtc2
• gray_mask2

There are 3 types of classifiers used in this experiment:

• LR+lasso
• LR+ridge
• LR+elastic net
• LR alone without regularization

For optimal parameter selection, I use 2 criteria:

• criterion#1: pick the parameter (lambda, alpha) based on the maximum validation accuracy alone. However, the consistence of voxels across 4 folds using this criterion is not desirable, so I propose using another criterion.
• criterion#2: pick the parameter (lambda, alpha) based on 1) the train accuracy must greater than 0.85, 2) the number of nonzeros coefficients (selected voxels) must greater than 20 3) pick the parameters where the validation accuracy is greatest 4) Use alpha and lambda as the tie-breaker, the greater is preferable as it's more sparse--> greater interpretability.

The report results include

• The optimally selected parameters alpha* lambda* for 2 criteria.
• The accuracy for train, validation and test set: acc_train, acc_valid, acc_test
• The sparsity in terms of number of non-zero coefficients
• The consistency of selected voxels across 4 folds in the same subject. The consistency can be measured by how many times it is included in each fold. There are 4 levels:
  • #intersect>0: number of voxels that appears at least in one fold
  • #intersect>0.25: number of voxels that appears at least 1/4 of the folds
  • #intersect>0.5: number of voxels that appears at least 1/2 of the folds
  • #intersect>0.75: : number of voxels that appears at least 3/4 of the folds
  • #intersect=1: number of voxels that appears in every fold.
  • #intersect=0: number of voxels that never appears in any fold.