Lecture 10
For today you should:
1) Read the rest of Chapter 9 of Think Stats 2e
2) Prepare for a quiz on Chapters 8 and 9, primarily
3) Start the Estimation section of your journal
Today:
1) Hypothesis testing part 2
2) Quiz
For next time you should:
1) Read Chapter 10 of Think Stats 2e
2) Also read The Jimmy Nut Company Problem
2) Start the hypothesis test section of your journal
Project reminders
0) Bring two laptops, one with the meeting notes, one with the journal. Also bring results you want to discuss on paper. We'll start with the TODO list from last time. After the meeting, review the new TODO list with your partner.
1) Many groups I have talked to are a little behind schedule in the journals. That's ok, as many of you have spent time refining the project description and getting data. But let's work on closing the gap.
2) Many of you are diving into work on the core project goal, and that's good. But don't skip the simple basic stuff.
Example: You have run log data that has minutes and miles for each workout. You want to work with pace information, so you compute pace in minutes per mile. Then you compute the mean, and report that the average pace for the people in your dataset is 7:15 minutes per mile.
What are the 17 things you just did wrong?
3) The required elements are required. Some of them might feel contrived, but I believe they are good for your learning and for the project.
The element descriptions are scattered in the lecture notes. You should paste them into your journal, at least temporarily.
We'll have a major journal checkpoint on March 13 (the Friday before break). Details to follow.
Hypothesis test
For the next section of your journal, you should:
1) Find an apparent effect: a difference between groups is an obvious choice. Ideally, find one that is not obviously significant.
2) Choose an appropriate test statistic and compute it.
3) Model the null hypothesis.
4) Use the HypothesisTest framework to compute the distribution of test statistics and a p-value.
For now this one is mostly an exercise and you can keep it simple. We will cycle back to this section later when you have a better idea what the important quantities are to estimate.
There is only one test
Power
Suppose you think there is a difference between groups, run a test, and get p=0.54. That means that the apparent effect could easily be due to chance.
Does that mean there is no difference between groups? Not necessarily. There are two explanations:
1) The difference between groups is small.
2) The test was "underpowered"
Power is the probability of a "true positive". It depends on the actual effect size, which is usually unknown.
So how do you make an argument for a negative result?
1) There is a difference between groups, but it is only 𝛿*, and it is not statistically significant.
2) We ran a power analysis (as in Section 9.10) and found that if the actual effect size is 𝛿*, our test has a 80% chance of producing a true positive.
3) We conclude that the test had sufficient power to yield a true positive, so the non-significance we found is evidence that the apparent effect is not real.
OR
2) We ran a power analysis (as in Section 9.10) and found that if the actual effect size is 𝛿*, our test has a 20% chance of producing a true positive.
3) We conclude that the test was underpowered to detect an effect as small as 𝛿*. We can't interpret this result as evidence that the apparent effect is not real.