Exploratory Hypothesis Tests can be More Compelling THan confirmatory hypothesis tests

Researchers often distinguish between:

1⃣ Exploratory hypothesis tests - unplanned tests of post hoc hypotheses that may be based on the current results, and

2⃣ Confirmatory hypothesis tests - planned tests of a priori hypotheses that are independent from the current results

This distinction is supposed to be useful because exploratory results are assumed to be more “tentative” and “open to bias” than confirmatory results. In this paper, we challenge this assumption and argue that exploratory results can be more compelling than confirmatory results.

Our article has three parts. In the first part, we demonstrate that the same data can be used to generate and test a hypothesis in a transparently valid manner. We agree that circular reasoning can invalidate some exploratory hypothesis tests. However, circular reasoning can be identified by checking the contents of the reasoning without knowing the timing of that reasoning (i.e., a priori or post hoc).

In the second part of our article, we argue that exploratory hypothesis tests can have several evidential advantages over confirmatory tests and, consequently, they have the potential to deliver more compelling research conclusions. In particular, exploratory hypothesis tests:

✅ avoid researcher commitment and prophecy biases

✅ reduce the motive for data fraud

✅ are more appropriate following unplanned deviations

✅ facilitate inference to the best explanation

✅ allow peer reviewers to contribute to exploratory analyses

Finally, in the third part of our article, we consider several potential disadvantages of exploratory hypothesis tests and conclude that these potential disadvantages may not be problematic. In particular, exploratory hypotheses tests are not necessarily disadvantaged due to:

✅ overfitting

✅ bias

✅ HARKing

✅ unacceptable research practices

And they:

✅ are usually necessary

✅ can be falsified

✅ can predict anything but may suffer an evaluative cost in doing so

To be clear, our claim is not that exploratory hypothesis tests are always more compelling than confirmatory tests or even that they are typically more compelling. Our claim is only that exploratory tests can be more compelling in specific research situations. More generally, we encourage researchers to evaluate specific tests and results on a case-by-case basis rather than to follow simplistic heuristics such as “exploratory results are more tentative,” which represents a form of methodolatory.

Further Information

Article

Rubin, M., & Donkin, C. (2024). Exploratory hypothesis tests can be more compelling than confirmatory hypothesis tests. Philosophical Psychology, 37(8), 2019-2047. https://doi.org/10.1080/09515089.2022.2113771 

Publisher’s open access version.

Related Work

Our paper builds on some of my previous work on preregistration and HARKing. And please check out Szollosi and Donkin’s (2021) paper on “the misguided distinction between exploratory and confirmatory research.”