Post date: Nov 06, 2013 4:37:14 PM
I ran fit HMM's based solely on the data, and we do indeed see that the states with lower means tend to have higher variance. In fact many of the highest Fst SNPs end up in the lower mean higher variance state. I decided to see whether things would change if we used three states like Excoffier's group. It doesn't. You still get higher means with lower variance and vice versa (and again many of the highest Fst SNPs are in the low mean high variance state). So, we are not really capturing what we want. So, I started playing with runs where I constrained all of the variances to be the empirical variances and estimated the means and transition probabilities. I haven't yet finished this for all populations, but it looks much better. I started with the three state model (I just put an example in dropbox called threestateHMM.png, this is for LA x PRC with red = high Fst, black = normal, blue = low, you can barely see the blue but its at the very bottom). I need to play more with the two state model and with the other populations, but I think this will give us something meaningful (and it nicely shows the heterogeneity in fst).