09: Confidence Intervals

"Most quantities of interest, most projections, most numerical assessments are not point estimates. Rather they are rough distributions — not always normal, sometimes bi-modal, sometimes exponential, sometimes something else." - John Allen Paulos."Variation is the hard reality, not a set of imperfect measures for a central tendency. Means and medians are the abstractions." — Stephen Jay Gould.

"Although this may seem a paradox, all exact science is dominated by the idea of approximation." - Bertrand Russell.

Lecture outline: study the issues relating to the confidence in measurements of sampled data.

1. Intuition of confidence intervals

Population and Sampling distribution

Z-distribution vs. t-distribution

Point estimates and Interval estimates

Interval estimates: Margin of error

Critical values: t* and z*

Sample size for desired margin of error

90%, 95%, 99% confidence intervals

2. Types of confidence intervals

Confidence intervals of sample mean

Confidence intervals of proportion

Confidence intervals of difference of mean (2 samples)

Primary reference for this lecture:

1. “The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling” by Raj Jain; Chapter 12: “Summarizing Measured Data”

2. "Statistics for Experimenters", Box and Hunter.

3. "Probability and Statistics with Reliability, Queueing and Computer Science Applications", Kishor Trivedi, Chapter 10: Statistical Inference

Secondary references for this lecture:

1. “Measuring Computer Performance: A Practitioner's Guide” by David Lilja; Chapter 4: “Errors in Experimental Measurements”

2. “Performance Evaluation of Computer and Communication Systems” by Le Boudec, Chapter 2: “Summarizing Performance Data, Confidence Intervals”

3. “Cartoon Guide to Statistics” by Larry Gonick; Chapter 7: “Confidence Intervals”