Maximum Entropy

One of my believes 

During my second year in Hopkins, I met Dr. Sanjeev Khudanpur, who became my dissertation advisor later. A bright and thoughtful scientist, Sanjeev brought me to the wonderland of maximum entropy. There, you can felt the beauty of mathematics. We tried to use the method to build a better language model than n-gram models and used some tools. Unfortunately, there're no efficient training tools for maximum entropy models. So I proposed a serial of fast training algorithms for maximum entropy models. We got probably the best language models one could get in the world using maximum entropy methods and we were awarded the best paper award in Eurospeech99.

The principle of maximum entropy is very simple and nature. When we are going to estimate something with incomplete information, we want to choose a model which satisfies all the facts we have known and at the same time leave the uncertainty as much as possible. Why it is correct? It has been proved in many areas, from language processing to hedge funds management. It's more like a philosophical belief than a thereom.