When Does HARKing Hurt?

Source: Dirk-Jan Hoek. https://www.flickr.com/photos/23868780@N00/7374874302

This paper (Rubin, 2017) considers the issue of hypothesising after the results are known or HARKing (Kerr, 1998). HARKing occurs when researchers check their research results and then add or remove hypotheses on the basis of those results without acknowledging this process in their research report.

In my paper, I consider three type of HARKing: (1) Constructing hypotheses after the results are known, or CHARKing, involves surreptitiously constructing hypotheses after observing the current results and then implying that the CHARKed hypotheses are a priori hypotheses rather than post hoc hypotheses. (2) Retrieving hypotheses after the results are known, or RHARKing, involves retrieving previously published hypotheses and then using them to explain the current research results without declaring the post hoc timing of this retrieval. Finally, (3) suppressing hypotheses after the results are known, or SHARKing, occurs when researchers fail to report a priori hypotheses that they had originally considered during their study.

Using arguments from the philosophy of science, I conclude that it is unacceptable to CHARK, but acceptable to RHARK. In support of this position, I note that RHARKed hypotheses may be confirmed or disconfirmed by the current results, and that they cannot overfit the current results because they are constructed (by other researchers) before the results are known. Effectively, RHARKed hypotheses have been "preregistered" in the existing literature. All that is left to do is to appraise the validity with which these hypotheses have been tested in the current study, and this appraisal can be made by readers without knowing that the hypotheses are RHARKed.

RHARKing: Sometimes, someone else has previously published a substantive hypothesis that can be tested in your study.

To provide another example of RHARKing, imagine that you observe an apple falling from a tree, but you don't know why. You then read up on Newton's theory of gravity and declare that you've found support for his theory without disclosing your prior ignorance of this theory. Although you are secretly RHARKing, your observation can still count as evidence in support of Newton's theory.

I also argue that it is acceptable to SHARK but only under certain conditions. Specifically, SHARKing is acceptable if the suppressed hypotheses are both (a) poorly tested and (b) unrelated to the reported research conclusions. In this case, the SHARKing helps rather than harms scientific progress by enabling a clearer and more efficient communication of the research results.

I should note that I have recently changed my view about the acceptability of CHARKing. In my 2017 paper, I argued that CHARKing was unethical because CHARKed hypotheses cannot predict or be falsified by the evidence and because overfitting is possible during the hypothesis construction process. I have revised this view in a more recent paper (Rubin, 2019). I now assume that CHARKed hypotheses can predict the current results if they are deduced on the basis of demonstrably a priori theory and evidence, although poor quality theorizing may sometimes detract from the informativeness of this prediction. I also assume that readers may increase or decrease their estimated relative verisimilitude (closeness to the truth) of hypotheses in a valid manner, even if those hypotheses are CHARKed (cf. falsification). Finally, I distinguish overfitting from flexible theorizing. I argue that overfitting is not possible when researchers deduce hypotheses from a priori theory and evidence. Flexible theorizing is possible. However, it is equally identifiable for CHARKed and non-CHARKed hypotheses, and it may be taken into account in estimates of relative verisimilitude. In summary, my view on CHARKing is now much closer to my view on RHARKing. I summarise my revised position here.

Further Information

Article

Rubin, M. (2017). When does HARKing hurt? Identifying when different types of undisclosed post hoc hypothesizing harm scientific progress. Review of General Psychology, 21(4), 308-320. https://doi.org/10.1037/gpr0000128 *Self-archived version*