08: Hypothesis Testing
"Every experiment may be said to exist only in order to give the facts a chance of 'disproving' the null hypothesis." - Ronald Fisher."The great tragedy of science -- the slaying of a beautiful hypothesis by an ugly fact." - Thomas Huxley.
"Still, it is an error to argue in front of your data. You find yourself insensibly twisting them round to fit your theories." - Arthur Conan Doyle.
Lecture outline: how to test the validity of an hypothesis through statistics?
1. Intuition of hypothesis testing
Analogy with legal system
Types of errors in hypothesis testing: Type I and Type II errors
Impact of sample size
2. Types of tests and test statistics
Steps of hypothesis testing
Test statistics: the z-statistic; the t-statistic
One tailed and two tailed hypothesis tests
Significance testing; p-value
Chances of error: Type I and Type II errors
Power of a test
Primary reference for this lecture:
1. “The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling” by Raj Jain; Chapter 16: “Introduction to Experimental Design”.
2. "Probability and Statistics with Reliability, Queueing and Computer Science Applications", Kishor Trivedi, Chapter 10: Statistical Inference (Section 10.3: Hypothesis Testing).
Secondary references for this lecture:
1. “Cartoon Guide to Statistics” by Larry Gonick; Chapter 8: “Hypothesis Testing”
2. "Head First Statistics", Dawn Griffiths, Chapter 13: "Using Hypothesis Tests: Look at the Evidence"
3. "Probability, Statistics and Queueing Theory with Computer Science Applications" by Arnold Allen, Chapter 8, Hypothesis Testing.