Intro to Hypothesis Testing

The total length of the videos in this section is approximately 19 minutes, but you will also spend time answering short questions while completing this section.

You can also view all the videos in this section at the YouTube playlist linked here.

The Lady tasting tea

The lady was Dr. B. Muriel Bristol, a colleague of Fisher's. Here is a photo of her in an article from Significance Magazine, along with more information about the history of this problem:

There is a photo! She did a PhD on algae!

Imagine that your drink order at a party became the basis for an idea as foundational as the p-value!

Question 1: The lady might have correctly identified 0, 1, 2, 3, or 4 milk-first cups. Which of these numbers, if observed, would convince you that the lady is indeed able to distinguish between milk-first and tea-first cups?

Question 2: Suppose that you know the lady is not able to distinguish milk-first and tea-first cups. If you had to bet on how many of the four cups she labels milk-first will be correct, what number would you bet on?

The Lady can't taste tea

Question 3: Consider all the subsets consisting of 4 cups that could be chosen from the 8 cups. How many of these subsets contain 4 milk-first cups?

Counting

Question 4: What is the probability that the lady will correctly choose exactly half of the milk cups, assuming that she is just guessing randomly?

Optionally, you can refresh your memory of combinations/permutations here. If the videos and the problem above make sense to you, it's not important (for this course) for you to be able to calculate the number of ways to allocate 2 milk and 2 tea cups, etc.

Comparing truth to distribution

Question 5: If something happens that should only happen one out of every 70 times, are you surprised?

Hypothesis Tests

Question 6: Suppose a neuroscientist compares brain measurements from two types of mice. Based on five mice of each type, she calculates the difference in median measurements between the two groups. Can the difference she calculates be called the null hypothesis, the test statistic, or the p-value?

Example: Harvard Legal Aid Bureau Evaluation

Question 7: The difference in win rates between those who are offered help from HLAB and those who are not offered help from HLAB depends on which people are randomized to which group. Is this uncertainty due to sampling from the target population or assignment to treatment groups within a sample?

In case you're interested in law or this example, the (long) HLAB paper is here. 

Ethical issues surrounding R. A. Fisher and current discussions about p-values

R. A. Fisher was a eugenicist. In 2020, the Committee of the Presidents of Statistical Societies "retired" their most prominent award, which was named after Fisher, for this reason.  This article discusses Fisher's history in more detail. There are important ongoing conversations about how statistics and other fields should handle the remaining influences of scholars who were actively racist and whose scholarly contributions were sometimes separate from but often intertwined with their racism. Interestingly, the academic community is talking about de-emphasizing p-values at the same time that we are talking about de-emphasizing Fisher, who first proposed p-values.

You may know that the most common convention is to use 0.05 as a cutoff for the p-value of a hypothesis test, and it's helpful to have a convention, but there is no particular justification for any cutoff. In fact, there has been a lot of discussion in recent years about the drawbacks of hypothesis testing as the primary way to report study results. The main idea is that p-values do not include all of the information about study results: they produce exactly the number that is their definition, the probability that we'd see results at least as extreme when some null hypothesis is true. But, it's not very helpful to report a p-value without also visualizing the data, reporting the means/medians in each groups, perhaps calculating a confidence interval, etc. Over time, some fields have become over-reliant on comparing each p-value to 0.05 as the primary way to identify important results, but that is not what p-values are meant to do. One psychology journal has even banned p-values, which is a bit of an over-correction! The journal The American Statistician recently published an issue with guidelines for researchers who want to avoid over-relying on p-values. The opening article in that issue is an interesting read - I recommend it: "Moving to a world beyond p<0.05."

We will come back to the drawbacks of strict p-value cutoffs when we talk about practical v. statistical significance and when we talk about multiple comparisons.

In this second-level statistics course, I think it is important for you to have a solid understanding of hypothesis testing and p-values not only because they are still the most popular approach to data analysis, but also because you can't fully engage in conversations about the appropriate role of p-values and how to handle Fisher's legacy without understanding the underlying concepts.

Question 9: Very briefly, what do you think is the best way for society to handle influential contributions by people with unacceptable, offensive, and harmful views?

Now you are ready to move on to a discussion of non-parametric hypothesis tests. Please proceed!

During this tutorial you learned:

• About hypothesis tests, using the Lady Tasting Tea example, including reference distributions, p-values, null hypotheses, proof by contradiction, and test statistics

• About hypothesis testing using the Harvard Legal Bureau (HLAB) example

• About the ethical issues related to R .A. Fisher and the lineage of racism in the field of statistics

• About the arbitrariness of 0.05 as a conventional cutoff value for the p-value of a hypothesis test

Terms and concepts:

Hypothesis test, reference distribution, p-value, null hypothesis, proof by contradiction, test statistic