Computer Vision: Algorithms and Applications, written by Richard Szeliski, The book is still in the Draft stage will be published later this year. While it is in this stage, it can be downloaded from his website here. I have read only a few chapters like image stitching, Stereo correspondence and 3D reconstruction. The book assumes that you are reasonably familiar with what is happening in this field and is recommended for some advanced and eclectic topics.
Computer Vision: A Modern Approach, written by David Forsyth. This is perhaps the best book in this field for beginners. The explanations are very simple and very well written. I have still not completed more that half of the book. But it is must read before starting your Ph.D.
Robot Vision, written by K.P.Horn. It is one of the oldest book and does not have a lot of the modern day algorithms. But If you want to know about how it all began, this is the way to go.( I have only glimpsed through the first few chapters)
Multiple View Geometry in Computer Vision, written by Richard Hartley and Andrew Zisserman. Undoubtedly, THE book as far as multiview geometry is concerned. Contains a range of algorithms but requires some knowledge on linear algebra to understand it. I suggest you go through the lectures given here. I have completed most of this book including the appendix. :)
Linux Device Drivers, 2nd Edition, written by Alessandro Rubini and Jonathan Corbet. This is important book and I found it extremely useful in some areas like writing Webcam drivers for both desktop and embedded processors.
Learning OpenCV, Gary Bradski and Adrian Kaehler. This is how it all began for me. I can only thank Gary and Adrian for the effort. Although the new version of OpenCV has a significant number of new functions which are well documented here.
Linear Algebra : A very integral part of computer Vision. (Gilbert Strang)
Machine Learning : A great set of lectures if you want to understand the principle of feature based object detection and other AI based methods that are catching on. Helped me a lot with the Viola and Jones algorithm. (Andrew Y. Ng)
A Statistical method of Machine Learning by Prof. Marina Meila at UW. The link to it is here.
I am looking for a set of lectures on Probability if you guys know any good lectures on it, please mail be about the same.
Besides these books, I enjoy reading P.G.Wodehouse, especially the Jeeves and Wooster series. Agatha Christie, Arthur Hailey, Alexander Dumas, ( not many people are aware of the books after three musketeers, there are really good, you should read it), Douglas Adams, Richard Feynman.
My favorite T.V shows includes House( almost anything that has Hugh Laurie is worth a watch), Jeeves and Wooster, Blackadder.