Replication Crisis
I’ve published several articles on issues connected with the replication crisis in science. For example, I’ve argued that it's not always problematic to engage in questionable research practices such as hypothesising after the results are known (HARKing; Rubin, 2017c, 2022) and uncorrected multiple testing (Rubin, 2017b, 2021a, 2021c, 2022a, 2023, 2024). I've also criticised some science reforms, such as preregistration (e.g., Rubin, 2020a; Rubin & Donkin, 2022) and stricter adherence to Neyman-Pearson hypothesis testing (Rubin, 2020b, 2021b), and I've written about what I call questionable metascience practices (Rubin, 2023). Nonetheless, I am generally in favour of other open science reforms, such as open data and materials in the case of postpositivist science.
Summaries
Publications
Rubin, M. (2024). Inconsistent multiple testing corrections: The fallacy of using family-based error rates to make inferences about individual hypotheses. Methods in Psychology,10, Article 100140.
Publisher's open access version.
Rubin, M. (2024). Type I error rates are not usually inflated. MetaArXiv.
Rubin, M. (2023). The replication crisis is less of a “crisis” in Lakatos' philosophy of science. MetaArXiv.
Rubin, M. (2023). Questionable metascience practices. Journal of Trial and Error.
Publisher’s open access version.
Rubin, M., & Donkin, C. (2022). Exploratory hypothesis tests can be more compelling than confirmatory hypothesis tests. Philosophical Psychology.
Publisher’s open access version.
Rubin, M. (2022a). That’s not a two-sided test! It’s two one-sided tests! Significance, 19(2), 50-53.
Publisher’s version. Self-archived version.
Rubin, M. (2022b). The costs of HARKing. The British Journal for the Philosophy of Science, 73(2), 535-560.
Publisher’s version. Publisher’s free access. Self-archived version.
Rubin, M. (2021a). There’s no need to lower the significance threshold when conducting single tests of multiple individual hypotheses. Academia Letters, Article 610.
Publisher’s open access version. Direct open access version.
Rubin, M. (2021b). What type of Type I error? Contrasting the Neyman-Pearson and Fisherian approaches in the context of exact and direct replications. Synthese, 198, 5809–5834.
Publisher’s version. Self-archived version.
Rubin, M. (2021c). When to adjust alpha during multiple testing: A consideration of disjunction, conjunction, and individual testing. Synthese, 199, 10969–11000.
Publisher’s version. Self-archived version.
Rubin, M. (2020a). Does preregistration improve the credibility of research findings? The Quantitative Methods in Psychology, 16(4), 376–390.
Publisher’s version. Self-archived version.
Rubin, M. (2020b). “Repeated sampling from the same population?” A critique of Neyman and Pearson’s responses to Fisher. European Journal for Philosophy of Science, 10, Article 42, 1-15.
Publisher’s version. Publisher's open access version. Self-archived version.
Rubin, M. (2017a). An evaluation of four solutions to the forking paths problem: Adjusted alpha, preregistration, sensitivity analyses, and abandoning the Neyman-Pearson approach. Review of General Psychology, 21, 321-329.
Publisher’s version. Self-archived version.
Rubin, M. (2017b). Do p values lose their meaning in exploratory analyses? It depends how you define the familywise error rate. Review of General Psychology, 21, 269-275.
Publisher’s version. Self-archived version.
Rubin, M. (2017c). When does HARKing hurt? Identifying when different types of undisclosed post hoc hypothesizing harm scientific progress. Review of General Psychology, 21, 308-320.
Comments and Responses
Rubin, M. (2023, June 7). The preregistration prescriptiveness trade-off and unknown unknowns in science: Comments on Van Drimmelen (2023). Critical Metascience: MetaArXiv.
Open access self-published version.
Rubin, M. (2023, May 4). Opening up open science to epistemic pluralism: Comment on Bazzoli (2022) and some additional thoughts. Critical Metascience.
Open access self-published version.
Rubin, M. (2022). Green jelly beans and studywise error rates: A “theory first” response to Goeman (2022). PsyArXiv.
Open access self-published version.
Rubin, M. (2017). The implications of significance testing based on hypothesiswise and studywise error. PsyArXiv.
Presentations
Rubin, M. (2022). Does preregistration improve the interpretability and credibility of research findings? In Research transparency: From preregistration to open access. Erasmus Research Institute of Management Research Transparency Campaign, Erasmus University Rotterdam.