David MacKay: Information Theory, Inference, and Learning Algorithms, available free online at http://www.inference.org.uk/mackay/itila/book.html . An excellent read!
Andrew Gelman, John Carlin, Hal Stern, David Dunson, Aki Vehtari, and Donald Rubin: Bayesian Data Analysis, available free online at http://www.stat.columbia.edu/~gelman/book . Very useful and a standard in the field.
E. T. Jaynes (and L. Bretthorst): Probability Theory: The Logic of Science, available at least partially online at https://bayes.wustl.edu/etj/prob/book.pdf . A classic. Written in a very readable and somewhat provocative style it introduces and develops the concepts of Bayesian reasoning for science. There are couple of annotated editions availalbe for purchase at the usual websites.
Revised standards for statistical evidence — Recent PNAS paper comparing classical statistical hypothesis testing with Bayesian evidence-based model comparison.
Radford Neal's classic overview of Hamiltonian Monte Carlo, made available for your convenience as 1206.1901.pdf in Ben Wandelt's files
slice sampling paper — Radford Neal's paper and software on slice sampling
I used to cover this in depth in previous versions of the course. This is still very interesting, so I encourage you to read the references below, but I had to drop some of the material; instead I shifted focus to implicit likelihood inference techniques.