questions

If we show all the experiments, we would have 6*2*6*3 = 144 experiments, which is too many!

Now, though I have the results, I don't exactly know which measures are good because I don't have gold standard to compare with.

2) If possible, I plan to select only one biclustering algorithm in our work.

Here is the table showing pros and cons for each biclustering method.

I would go for non-negative matrix factorization (NNMF) because

1. NNMF outputs the numbers of biclusters according to what we want exactly.
2. The biclusters are usually (>80%) on the diagonal, so the row winner biclusters and the column winner biclusters are the same. So, I don't need to pick a row/column/robust winner biclusters. Unless we have some other criteria to pick subspaces to analyze.

3) How can we pick the number of biclusters k to run in the first place?

- If possible, I try to use only 1 value of k for the whole experiment; otherwise it's too much.

- From prior knowledge I know that the categories "face"and "house" are stand out, so the choice of k is 2 < k <=8. However, I also know that some tools might be grouped together, so it can be narrowed down to 2 < k < 8. In fact, k=4-6 also gives a nice result, but we might want to use only one for our convenience.

- Any suggestion on this?

4) Selecting biclusters to analyze

Once we obtain the biclustering results already, how people select the biclusters to analyze?

So far, I select the winner biclusters based on the maximum mean in each row, column and the robust winner.

I know that some people in gene expression community check the homogeneity of each bicluster.

If you can give me a pointer to some resources, that would save me lots of time.

5) How to evaluate our results without the gold standard?

We will obtain a few voxel-descriptor biclusters for each measure for each subject. We can plot the biclusters on the brain space, but how can we evaluate the results since we don't have a ground truth? Here is what I think:

1) consistency of the bicluster across subjects

2) compare the results with other methods. In fact, the multiple subject data from Haxby provide mask for "house" and "face", but not other categories.

3) Compare with the expected regions from medical/neuroscience knowledge

If possible, I want to show the results like the tables below, and report the results:

1) comparing the dissimilarity measures

2) comparing the matching modes (OAO or OAA).

3) comparing the "house" and "face" regions to what has been reported in Haxby's paper

4) new finding for the unknown categories

Evaluation criterion#1: Rand's index on "face" only

Evaluation criterion#2: Jaccard index on all categories