My greatest concern was what to call it. I thought of calling it 'information,' but the word was overly used, so I decided to call it 'uncertainty.' When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, 'You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, no one really knows what entropy really is, so in a debate you will always have the advantage.'
-- Claude Elwood Shannon, (1971) Scientific American, Vol 225, p. 180.
In this course we consider important problems of representing sources of information efficiently (source coding) and recovering information reliably over a noisy medium (statistical inference).
This course is an introduction to how data processing systems are designed and analyzed. The aim of this course is to build a solid (theoretical as well as intuitive) foundation for topics in information theory and data processing.