This is a non-exhaustive, chronological list of over 100 articles that take a step back and provide a critical perspective on some of the fundamental issues associated with open science, science reform, and/or the replication crisis. For example, these articles put forward critical views on preregistration, questionable research practices, and the importance of replication in science. For want of a better term, I've labelled this work "critical metascience." Some people have described it as meta-meta-science! 🙂

If I've left out a key article, or there's new work you'd like added, please send me the reference. For other lists of articles related to open science, please see the Framework for Open and Reproducible Research Training's collection at https://forrt.org/syllabus/ 

2024

Dames, H., Musfeld, P., Popov, V., Oberauer, K., & Frischkorn, G. T. (2024). Responsible research assessment should prioritize theory development and testing over ticking open science boxes. Meta-Psychology. https://doi.org/10.15626/MP.2023.3735 

Feest, U. (2024). What is the replication crisis a crisis of? Philosophy of Science. https://doi.org/10.1017/psa.2024.2 

Guest, O. (2024). What makes a good theory, and how do we make a theory good? Computational Brain & Behavior. https://doi.org/10.1007/s42113-023-00193-2 

Hostler, T. J. (2024). Research assessment using a narrow definition of “research quality” is an act of gatekeeping: A comment on Gärtner et al. (2022). Meta-Psychology, 8. https://doi.org/10.15626/MP.2023.3764 

Jost, J. T. (2024). Grand challenge: Social psychology without hubris. Frontiers in Social Psychology, 1, Article 1283272. https://doi.org/10.3389/frsps.2023.1283272 

Khan, S., Hirsch, J. S., & Zubida, O. Z. (2024). A dataset without a code book: Ethnography and open science. Frontiers in Sociology, 9, Article 1308029. https://doi.org/10.3389/fsoc.2024.1308029 

Klonsky, E. D. (2024, April 20). Campbell’s law explains the replication crisis: Pre-registration badges are history repeating. PsyArXiv. https://doi.org/10.31234/osf.io/7czpr 

Maziarz, M. (2024). Conflicting results and statistical malleability: Embracing pluralism of empirical results. Perspectives on Science. https://doi.org/10.1162/posc_a_00627 

Phaf, R. H. (2024). Positive deviance underlies successful science: Normative methodologies risk throwing out the baby with the bathwater. Review of General Psychology. https://doi.org/10.1177/10892680241235120 

Pownall, M. (2024). Is replication possible for qualitative research? A response to Makel et al. (2022). Educational Research and Evaluation. https://doi.org/10.1080/13803611.2024.2314526 

Prosser, A. M., Bagnall, R., Higson-Sweeney, N. (2024). Reflection over compliance: Critiquing mandatory data sharing policies for qualitative research. Journal of Health Psychology. https://doi.org/10.1177/13591053231225903 

Reyes Elizondo, A., & Kaltenbrunner, W. (2024). Navigating the science system: Research integrity and academic survival strategies. Science and Engineering Ethics, 30, Article 12. https://doi.org/10.1007/s11948-024-00467-3 

Rubin, M. (2024). Inconsistent multiple testing corrections: The fallacy of using family-based error rates to make inferences about individual hypotheses. Methods in Psychology, 10, Article 100140. https://doi.org/10.1016/j.metip.2024.100140 

Rubin, M. (2024). Type I error rates are not usually inflated. MetaArXiv. https://doi.org/10.31222/osf.io/3kv2b

Ting. C., & Greenland, S. (2024). Forcing a deterministic frame on probabilistic phenomena: a communication blind spot in media coverage of the “replication crisis.” Science Communication. https://doi.org/10.1177/10755470241239947 

Ulpts, S. (2024). Responsible assessment of what research? Beware of epistemic diversity! Meta-Psychology. https://doi.org/10.15626/MP.2023.3797 

2023

Bak-Coleman, J. B., & Devezer, B. (2023, November 20). Causal claims about scientific rigor require rigorous causal evidence. MetaArXiv. https://doi.org/10.31234/osf.io/5u3kj 

Baumeister, R., Bushman, B., & Tice, D. (2023). Multi-site replications in social psychology: Reflections, implications, and future directions. The Spanish Journal of Psychology, 26, E3. https://doi.org/10.1017/SJP.2023.6

Buzbas, E. O., & Devezer, B. (2023). Tension between theory and practice of replication. Journal of Trial & Error. https://doi.org/10.36850/mr9 

Buzbas, E. O., Devezer, B., & Baumgaertner, B. (2023). The logical structure of experiments lays the foundation for a theory of reproducibility. Royal Society Open Science, 10(3). https://doi.org/10.1098/rsos.221042 

Dal Santo, T., Rice, D. B., Amiri, L. S., Tasleem, A., Li, K., Boruff, J. T., .Geoffroy, M.-C., Benedetti, A., & Thombs, B. D. (2023). Methods and results of studies on reporting guideline adherence are poorly reported: A meta-research study. Journal of Clinical Epidemiology. https://doi.org/10.1016/j.jclinepi.2023.05.017

Darda, K. M., Conry-Murray, C., Schmidt, K., Elsherif, M. M., Peverill, M., Yoneda, T., … Gernsbacher, M. (2023, October 29). Promoting civility in formal and informal open science contexts. PsyArXiv. https://doi.org/10.31234/osf.io/rfkyu 

Devezer, B., & Buzbas, E. O. (2023). Rigorous exploration in a model-centric science via epistemic iteration. Journal of Applied Research in Memory and Cognition, 12(2), 189–194. https://doi.org/10.1037/mac0000121 

Devezer, B., & Penders, B. (2023). Scientific reform, citation politics and the bureaucracy of oblivion. Quantitative Science Studies. https://doi.org/10.1162/qss_c_00274 

Field, S. M., Kaznatcheev, A., van Dongen, N., Volz, L., & Wright, C. (2023, November 28). Good theory from bad ingredients? https://osf.io/preprints/osf/sz9y7 

Hicks, D. J. (2023). Open science, the replication crisis, and environmental public health. Accountability in Research, 30(1), 34-62. https://doi.org/10.1080/08989621.2021.1962713 

Hostler, T. J. (2023). The invisible workload of open research. Journal of Trial & Error. https://doi.org/10.36850/mr5 

Jacobucci, R. (2022). A critique of using the labels confirmatory and exploratory in modern psychological research. Frontiers in Psychology, 13, Article 1020770. https://doi.org/10.3389/fpsyg.2022.1020770 

Lavelle, J. S. (2023, October 2). Growth from uncertainty: Understanding the replication 'crisis' in infant psychology. PhilSci Archive. https://philsci-archive.pitt.edu/22679/ 

Leonelli, S. (2023). Philosophy of open science. Cambridge University Press. https://www.cambridge.org/core/elements/philosophy-of-open-science/0D049ECF635F3B676C03C6868873E406 

Liu, M. (2023). Whose open science are we talking about? From open science in psychology to open science in applied linguistics. Language Teaching, 1-8. https://doi.org/10.1017/S0261444823000307 

Peterson, D., & Panofsky, A. (2023). Metascience as a scientific social movement. Minerva. https://doi.org/10.1007/s11024-023-09490-3

Rubin, M. (2023). Questionable metascience practices. Journal of Trial & Error. https://doi.org/10.36850/mr4

Rubin, M. (2023). The replication crisis is less of a “crisis” in Lakatos' philosophy of science. MetaArXiv. https://doi.org/10.31222/osf.io/2dz9s

Schwartz, B. (2023, March 20). Psychology’s increased rigor is good news. But is it only good news? Behavioural Scientist. https://behavioralscientist.org/psychologys-increased-rigor-is-good-news-but-is-it-only-good-news/  

Schimmelpfennig, R., Spicer, R., White, C., Gervais, W. M., Norenzayan, A., Heine, S., … Muthukrishna, M. (2023, February 9). A problem in theory and more: Measuring the moderating role of culture in Many Labs 2. PsyArxiv. https://psyarxiv.com/hmnrx/

Steltenpohl, C. N., Lustick, H., Meyer, M. S., Lee, L. E., Stegenga, S. M., Reyes, L. S., & Renbarger, R. L. (2023). Rethinking transparency and rigor from a qualitative open science perspective. Journal of Trial & Error. https://doi.org/10.36850/mr7 

Syed, M. (2023, December 8). Some data indicating that editors and reviewers do not check preregistrations during the review process. PsyArXiv. https://doi.org/10.31234/osf.io/nh7qw 

Syrjänen, P. (2023). Novel prediction and the problem of low-quality accommodation. Synthese, 202, Article 182, 1-32. https://doi.org/10.1007/s11229-023-04400-2  

van Drimmelen, T., Slagboom, N., Reis, R., Bouter, L., & van der Steen, J. T. (2023, August 7). Decisions, decisions, decisions: An ethnographic study of researcher discretion in practice. PsyArXiv. https://osf.io/preprints/metaarxiv/7dh3t/ 

Wilson, B. M., & Wixted, J. T. (2023). On the importance of modeling the invisible world of underlying effect sizes. Social Psychological Bulletin, 18, 1-16. https://doi.org/10.32872/spb.9981 

2022

Bak-Coleman, J. B., Mann, R. P., West, J., & Bergstrom, C. T. (2022, April 28). Replication does not measure scientific productivity. SoArXiv. https://doi.org/10.31235/osf.io/rkyf7

Baumeister, R. F., Tice, D. M., & Bushman, B. J. (2022). A review of multisite replication projects in social psychology: Is it viable to sustain any confidence in social psychology’s knowledge base? Perspectives on Psychological Science, 0(0). https://doi.org/10.1177/17456916221121815 

Bazzoli, A. (2022). Open science and epistemic pluralism: A tale of many perils and some opportunities. Industrial and

Organizational Psychology, 15(4), 525-528. https://doi.org/10.1017/iop.2022.67 

Berberi, I., & Roche, D. G. (2022). No evidence that mandatory open data policies increase error correction. Nature Ecology & Evolution, 6(11), 1630-1633. https://doi.org/10.1038/s41559-022-01879-9

Derksen, M., & Field, S. (2022). The tone debate: Knowledge, self, and social order. Review of General Psychology, 26(2), 172-183. https://doi.org/10.1177/10892680211015636

Derksen, M., & Morawski, J. (2022). Kinds of replication: Examining the meanings of “conceptual replication” and “direct replication”. Perspectives on Psychological Science, 17(5), 1490-1505. https://doi.org/10.1177/17456916211041116

Devezer, B., & Buzbas, E. (2022, November 25). Minimum viable experiment to replicate. PhilSci Archive. http://philsci-archive.pitt.edu/id/eprint/21475 

Flis, I. (2022). The function of literature in psychological science. Review of General Psychology, 26(2), 146-156. https://doi.org/10.1177/10892680211066466

Fox Tree, J., Lleras, A., Thomas, A., & Watson, D. (2022, August 30). The inequitable burden of open science. Psychonomic Society Featured Content. https://featuredcontent.psychonomic.org/the-inequitable-burden-of-open-science/

Gollwitzer, M., & Schwabe, J. (2022). Context dependency as a predictor of replicability. Review of General Psychology, 26(2), 241-249. https://doi.org/10.1177/10892680211015635 

Guzzo, R. A., Schneider, B., & Nalbantian, H. R. (2022). Open science, closed doors: The perils and potential of open science for research in practice. Industrial and Organizational Psychology: Perspectives on Science and Practice, 15, 495–515. https://doi.org/10.1017/iop.2022.61 

Haig, B. D. (2022). Understanding replication in a way that is true to science. Review of General Psychology, 26(2), 224-240. https://doi.org/10.1177/10892680211046514

Khalil, A. T., Shinwari, Z. K., & Islam, A. (2022). Fostering openness in open science: An ethical discussion of risks and benefits. Frontiers in Political Science, 4, 930574. https://doi.org/10.3389/fpos.2022.930574 

Lash, T. L. (2022). Getting over TOP. Epidemiology, 33(1), 1-6. https://doi.org/10.1097/EDE.0000000000001424 

Leonelli, S. (2022). Open science and epistemic diversity: Friends or foes? Philosophy of Science, 89(5), 991-1001. https://doi.org/10.1017/psa.2022.45 

Liu, B., & Wei, L. (2022). Unintended effects of open data policy in online behavioral research: An experimental investigation of participants’ privacy concerns and research validity. Computers in Human Behavior, Article 107537. https://doi.org/10.1016/j.chb.2022.107537 

Lohmann, A., Astivia, O. L., Morris, T. P., & Groenwold, R. H. (2022). It's time! Ten reasons to start replicating simulation studies. Frontiers in Epidemiology, 2, 973470. https://doi.org/10.3389/fepid.2022.973470 

Malich, L., & Rehmann-Sutter, C. (2022). Metascience is not enough - A plea for psychological humanities in the wake of the replication crisis. Review of General Psychology, 26(2), 261-273. https://doi.org/10.1177/10892680221083876

McDermott, R. (2022). Breaking free: How preregistration hurts scholars and science. Politics and the Life Sciences, 41(1), 55-59.  https://doi.org/10.1017/pls.2022.4 

Morawski, J. (2022). How to true psychology’s objects. Review of General Psychology, 26(2), 157-171. https://doi.org/10.1177/10892680211046518

Penders, B. (2022). Process and bureaucracy: Scientific reform as civilisation. Bulletin of Science, Technology & Society, 42(4), 107-116. https://doi.org/10.1177/02704676221126388 

Pownall, M., & Hoerst, C. (2022). Slow science in scholarly critique. The Psychologist, 35, 2. https://thepsychologist.bps.org.uk/volume-35/february-2022/slow-science-scholarly-critique

Prosser, A. M. B., Hamshaw, R., Meyer, J., Bagnall, R., Blackwood, L., Huysamen, M., ... & Walter, Z. (2022). When open data closes the door: Problematising a one size fits all approach to open data in journal submission guidelines. British Journal of Social Psychology. https://doi.org/10.1111/bjso.12576 

Rubin, M. (2022). The costs of HARKing. British Journal for the Philosophy of Science, 73(2), 535-560. https://doi.org/10.1093/bjps/axz050

Rubin, M., & Donkin, C. (2022). Exploratory hypothesis tests can be more compelling than confirmatory hypothesis tests. Philosophical Psychology. https://doi.org/10.1080/09515089.2022.2113771 

Wegener, D. T., Fabrigar, L. R., Pek, J., & Hoisington-Shaw, K. (2022). Evaluating research in personality and social psychology: Considerations of statistical power and concerns about false findings. Personality and Social Psychology Bulletin, 48(7), 1105-1117. https://doi.org/10.1177/01461672211030811 

2021

Anonymous. (2021, November 25). It’s 2021… and we are still dealing with misogyny in the name of open science. University of Sussex School of Psychology Blog. https://blogs.sussex.ac.uk/psychology/2021/11/25/its-2021-and-we-are-still-dealing-with-misogyny-in-the-name-of-open-science/

Bastian, H. (2021, October 31). The metascience movement needs to be more self-critical. PLOS Blogs: Absolutely Maybe. https://absolutelymaybe.plos.org/2021/10/31/the-metascience-movement-needs-to-be-more-self-critical/

Bennett, E. A. (2021). Open science from a qualitative, feminist perspective: Epistemological dogmas and a call for critical examination. Psychology of Women Quarterly, 45(4), 448-456. https://doi.org/10.1177/03616843211036460

Bryan, C. J., Tipton, E., & Yeager, D. S. (2021). Behavioural science is unlikely to change the world without a heterogeneity revolution. Nature Human Behaviour, 5(8), 980-989. https://doi.org/10.1038/s41562-021-01143-3 

Devezer, B., Navarro, D. J., Vandekerckhove, J., & Ozge Buzbas, E. (2021). The case for formal methodology in scientific reform. Royal Society Open Science, 8(3), Article 200805. https://doi.org/10.1098/rsos.200805

Field, S. M., & Derksen, M. (2021). Experimenter as automaton; experimenter as human: Exploring the position of the researcher in scientific research. European Journal for Philosophy of Science, 11, Article 11. https://doi.org/10.1007/s13194-020-00324-7

Gervais, W. M. (2021). Practical methodological reform needs good theory. Perspectives on Psychological Science, 16(4), 827-843. https://doi.org/10.1177/1745691620977471

Guest, O., & Martin, A. E. (2021). How computational modeling can force theory building in psychological science. Perspectives on Psychological Science, 16(4), 789–802. https://doi.org/10.1177/1745691620970585 

Jacobs, A., Büthe, T., Arjona, A., Arriola, L., Bellin, E., Bennett, A.,...Yashar, D. (2021). The qualitative transparency deliberations: Insights and implications. Perspectives on Politics, 19(1), 171-208. https://doi.org/10.1017/S1537592720001164 

Kessler, A., Likely, R., & Rosenberg, J. M. (2021). Open for whom? The need to define open science for science education. Journal of Research in Science Teaching, 58(10), 1590-1595. https://doi.org/10.1002/tea.21730

Peterson, D., & Panofsky, A. (2021). Arguments against efficiency in science. Social Science Information, 60(3), 350-355. https://doi.org/10.1177/05390184211021383

Pham, M. T., & Oh, T. T. (2021). Preregistration is neither sufficient nor necessary for good science. Journal of Consumer Psychology, 31(1), 163-176. https://doi.org/10.1002/jcpy.1209

Pham, M. T., & Oh, T. T. (2021). On not confusing the tree of trustworthy statistics with the greater forest of good science: A comment on Simmons et al.’s perspective on pre‐registration. Journal of Consumer Psychology, 31(1), 181-185. https://doi.org/10.1002/jcpy.1213 

Proulx, T., & Morey, R. D. (2021). Beyond statistical ritual: Theory in psychological science. Perspectives on Psychological Science, 16(4), 671-681. https://doi.org/10.1177/17456916211017098

Rubin, M. (2021). What type of Type I error? Contrasting the Neyman-Pearson and Fisherian approaches in the context of exact and direct replications. Synthese, 198, 5809–5834. https://doi.org/10.1007/s11229-019-02433-0  

Szollosi, A., & Donkin, C. (2021). Arrested theory development: The misguided distinction between exploratory and confirmatory research. Perspectives on Psychological Science, 16(4), 717-724. https://doi.org/10.1177/1745691620966796

Wentzel, K. R. (2021). Open science reforms: Strengths, challenges, and future directions. Educational Psychologist, 56(2), 161-173. https://doi.org/10.1080/00461520.2021.1901709

2020

Andreoletti, M. (2020). Replicability crisis and scientific reforms: Overlooked issues and unmet challenges. International Studies in the Philosophy of Science, 33(3), 135-151. https://doi.org/10.1080/02698595.2021.1943292

Bird, A. (2020). Understanding the replication crisis as a base rate fallacy. The British Journal for the Philosophy of Science, 72(4), 965-993. https://doi.org/10.1093/bjps/axy051 

Guttinger, S. (2020). The limits of replicability. European Journal for Philosophy of Science, 10(2), 1-17. https://doi.org/10.1007/s13194-019-0269-1

Iso-Ahola, S. E. (2020). Replication and the establishment of scientific truth. Frontiers in Psychology, 11, Article 2183. https://doi.org/10.3389/fpsyg.2020.02183 

Lewandowsky, S., & Oberauer, K. (2020). Low replicability can support robust and efficient science. Nature Communications, 11, Article 358. https://doi.org/10.1038/s41467-019-14203-0

Navarro, D. (2020, September 23). Paths in strange spaces: A comment on preregistration. PsyArXiv. https://doi.org/10.31234/osf.io/wxn58

Rubin, M. (2020). Does preregistration improve the credibility of research findings? The Quantitative Methods for Psychology, 16(4), 376–390. https://doi.org/10.20982/tqmp.16.4.p376

Szollosi, A., Kellen, D., Navarro, D. J., Shiffrin, R., van Rooij, I., Van Zandt, T., & Donkin, C. (2020). Is preregistration worthwhile? Trends in Cognitive Science, 24(2), 94-95. https://doi.org/10.1016/j.tics.2019.11.009

Ulrich, R., & Miller, J. (2020). Meta-research: Questionable research practices may have little effect on replicability. ELife, 9, Article e58237. https://doi.org/10.7554/eLife.58237

Whitaker, K., & Guest, O. (2020). # bropenscience is broken science. The Psychologist, 33, 34-37. https://thepsychologist.bps.org.uk/volume-33/november-2020/bropenscience-broken-science

2019

Bryan, C. J., Yeager, D. S., & O’Brien, J. M. (2019). Replicator degrees of freedom allow publication of misleading failures to replicate. Proceedings of the National Academy of Sciences, 116(51), 25535-25545. https://doi.org/10.1073/pnas.1910951116 

Derksen, M. (2019). Putting Popper to work. Theory & Psychology, 29(4), 449-465. https://doi.org/10.1177/0959354319838343

Devezer, B., Nardin, L. G., Baumgaertner, B., & Buzbas, E. O. (2019). Scientific discovery in a model-centric framework: Reproducibility, innovation, and epistemic diversity. PloS one, 14(5), Article e0216125. https://doi.org/10.1371/journal.pone.0216125

Drummond, C. (2019). Is the drive for reproducible science having a detrimental effect on what is published? Learned Publishing, 32(1), 63-69. https://doi.org/10.1002/leap.1224

Feest, U. (2019). Why replication is overrated. Philosophy of Science, 86(5), 895-905. https://doi.org/10.1086/705451

Flis, I. (2019). Psychologists psychologizing scientific psychology: An epistemological reading of the replication crisis. Theory & Psychology, 29(2), 158-181. https://doi.org/10.1177/0959354319835322

Lewandowsky, S. (2019, January 22). Avoiding Nimitz Hill with more than a Little Red Book: Summing up #PSprereg. Psychonomic Society. https://featuredcontent.psychonomic.org/avoiding-nimitz-hill-with-more-than-a-little-red-book-summing-up-psprereg/

MacEachern, Steven N., & Van Zandt, T. (2019). Preregistration of modeling exercises may not be useful. Computational Brain & Behavior, 2, 179-182. https://doi.org/10.1007/s42113-019-00038-x 

Morawski, J. (2019). The replication crisis: How might philosophy and theory of psychology be of use? Journal of Theoretical and Philosophical Psychology, 39(4), 218–238. https://doi.org/10.1037/teo0000129

Morey, R. (2019). You must tug that thread: Why treating preregistration as a gold standard might incentivize poor behavior. Psychonomic Society. https://featuredcontent.psychonomic.org/you-must-tug-that-thread-why-treating-preregistration-as-a-gold-standard-might-incentivize-poor-behavior/

Oberauer, K. (2019, January 15). Preregistration of a forking path – What does it add to the garden of evidence? Psychonomic Society. https://featuredcontent.psychonomic.org/preregistration-of-a-forking-path-what-does-it-add-to-the-garden-of-evidence/

Oberauer, K., & Lewandowsky, S. (2019). Addressing the theory crisis in psychology. Psychonomic Bulletin & Review, 26(5), 1596-1618. https://doi.org/10.3758/s13423-019-01645-2

Penders, B., Holbrook, J. B., & de Rijcke, S. (2019). Rinse and repeat: Understanding the value of replication across different ways of knowing. Publications, 7(3), 52. https://doi.org/10.3390/publications7030052 

Shiffrin, R. (2019). Complexity of science v. #PSprereg? Psychonomic Society. https://featuredcontent.psychonomic.org/complexity-of-science-v-psprereg/

Sacco, D. F., Brown, M., & Bruton, S. V. (2019). Grounds for ambiguity: Justifiable bases for engaging in questionable research practices. Science and Engineering Ethics, 25(5), 1321-1337. https://doi.org/10.1007/s11948-018-0065-x

Stroebe, W. (2019). What can we learn from many labs replications? Basic and Applied Social Psychology, 41(2), 91-103. https://doi.org/10.1080/01973533.2019.1577736 

van Rooij, I. (2019). Psychological science needs theory development before preregistration. Psychonomic Society. https://featuredcontent.psychonomic.org/psychological-science-needs-theory-development-before-preregistration/  

Wiggins, B. J., & Christopherson, C. D. (2019). The replication crisis in psychology: An overview for theoretical and philosophical psychology. Journal of Theoretical and Philosophical Psychology, 39(4), 202–217. https://doi.org/10.1037/teo0000137

Wood, W., & Wilson, T. D. (2019, August 22). No crisis but no time for complacency. APS Observer, 32(7). https://www.psychologicalscience.org/observer/no-crisis-but-no-time-for-complacency

2018

Fanelli, D. (2018). Opinion: Is science really facing a reproducibility crisis, and do we need it to? Proceedings of the National Academy of Sciences, 115(11), 2628-2631. https://doi.org/10.1073/pnas.1708272114

Fiedler, K. (2018). The creative cycle and the growth of psychological science. Perspectives on Psychological Science, 13(4), 433-438. https://doi.org/10.1177/1745691617745651 

Gigerenzer, G. (2018). Statistical rituals: The replication delusion and how we got there. Advances in Methods and Practices in Psychological Science, 1(2), 198-218. https://doi.org/10.1177/2515245918771329 

Leonelli, S. (2018). Rethinking reproducibility as a criterion for research quality. Including a symposium on Mary Morgan: Curiosity, imagination, and surprise. Research in the History of Economic Thought and Methodology, 36B (pp. 129-146). Emerald Publishing. https://doi.org/10.1108/S0743-41542018000036B009

Mirowski, P. (2018). The future(s) of open science. Social Studies of Science, 48(2), 171-203. https://doi.org/10.1177/030631271877208 

Redish, D. A., Kummerfeld, E., Morris, R. L., & Love, A. C. (2018). Reproducibility failures are essential to scientific inquiry. Proceedings of the National Academy of Sciences, 115, 5042-5046. https://doi.org/10.1073/pnas.1806370115  

Vancouver, J. N. (2018). In defense of HARKing. Industrial and Organizational Psychology. 11(1), 73–80. https://doi.org/10.1017/iop.2017.89

Wilson, B. M., & Wixted, J. T. (2018). The prior odds of testing a true effect in cognitive and social psychology. Advances in Methods and Practices in Psychological Science, 1(2), 186–197. https://doi.org/10.1177/2515245918767122 

2017

Finkel, E. J., Eastwick, P. W., & Reis, H. T. (2017). Replicability and other features of a high-quality science: Toward a balanced and empirical approach. Journal of Personality and Social Psychology, 113(2), 244-253. http://dx.doi.org/10.1037/pspi0000075 

Greenfield, P. M. (2017). Cultural change over time: Why replicability should not be the gold standard in psychological science. Perspectives on Psychological Science, 12(5), 762-771. https://doi.org/10.1177/1745691617707314

Hamlin, J. K. (2017). Is psychology moving in the right direction? An analysis of the evidentiary value movement. Perspectives on Psychological Science, 12(4), 690-693. https://doi.org/10.1177/1745691616689062

Hollenbeck, J. R., & Wright, P. M. (2017). Harking, sharking, and tharking: Making the case for post hoc analysis of scientific data. Journal of Management, 43(1), 5-8. https://doi.org/10.1177/0149206316679487

Iso-Ahola, S. E. (2017). Reproducibility in psychological science: When do psychological phenomena exist? Frontiers in Psychology, 8, 879. https://doi.org/10.3389/fpsyg.2017.00879 

Levin, N., & Leonelli, S. (2017). How does one “open” science? Questions of value in biological research. Science, Technology, & Human Values, 42(2), 280-305. https://doi.org/10.1177/0162243916672071

Rubin, M. (2017). An evaluation of four solutions to the forking paths problem: Adjusted alpha, preregistration, sensitivity analyses, and abandoning the Neyman-Pearson approach. Review of General Psychology, 21(4), 321-329. https://doi.org/10.1037/gpr0000135  

Rubin, M. (2017). Do p values lose their meaning in exploratory analyses? It depends how you define the familywise error rate. Review of General Psychology, 21(3), 269-275. https://doi.org/10.1037/gpr0000123  

Rubin, M. (2017). When does HARKing hurt? Identifying when different types of undisclosed post hoc hypothesizing harm scientific progress. Review of General Psychology, 21(4), 308-320. https://doi.org/10.1037/gpr0000128

2016

Crandall, C. S., & Sherman, J. W. (2016). On the scientific superiority of conceptual replications for scientific progress. Journal of Experimental Social Psychology, 66, 93-99. https://doi.org/10.1016/j.jesp.2015.10.002 

Fiedler, K., & Schwarz, N. (2016). Questionable research practices revisited. Social Psychological and Personality Science, 7(1), 45-52. https://doi.org/10.1177/1948550615612150

Firestein, S. (2016, February 14). Why failure to replicate findings can actually be good for science. LA Times. https://www.latimes.com/opinion/op-ed/la-oe-0214-firestein-science-replication-failure-20160214-story.html

Fiske, S. T. (2016, October 31). A call to change science’s culture of shaming. APS Observer, 29. https://www.psychologicalscience.org/observer/a-call-to-change-sciences-culture-of-shaming

Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016). Comment on “Estimating the reproducibility of psychological science”. Science, 351(6277), 1037-1037. https://doi.org/10.1126/science.aad7243  

Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016). More on “Estimating the reproducibility of psychological science'”. https://gking.harvard.edu/files/gking/files/gkpw_post_publication_response.pdf 

Guest, O. (2016). Crisis in what exactly? The Winnower. https://doi.org/10.15200/winn.146590.01538 

Trafimow, D., & Earp, B. D. (2016). Badly specified theories are not responsible for the replication crisis in social psychology: Comment on Klein. Theory & Psychology, 26(4), 540–548. https://doi.org/10.1177/0959354316637136 

Tsai, A. C., Kohrt, B. A., Matthews, L. T., Betancourt, T. S., Lee, J. K., Papachristos, A. V., ... & Dworkin, S. L. (2016). Promises and pitfalls of data sharing in qualitative research. Social Science & Medicine, 169, 191-198. https://doi.org/10.1016/j.socscimed.2016.08.004 

2015

Barrett, L. F. (2015, September 1). Psychology is not in crisis. The New York Times. https://www3.nd.edu/~ghaeffel/ScienceWorks.pdf

Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist, 70(6), 487–498. https://doi.org/10.1037/a0039400

2014

Stanley, D. J., & Spence, J. R. (2014). Expectations for replications: Are yours realistic? Perspectives on Psychological Science, 9(3), 305-318. https://doi.org/10.1177/1745691614528518

Stroebe, W., & Strack, F. (2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9(1), 59-71. https://doi.org/10.1177/1745691613514450

2013

Scott, S. (2013, July 25). Pre-registration would put science in chains. Times Higher Education. https://www.timeshighereducation.com/comment/opinion/pre-registration-would-put-science-in-chains/2005954.article 

2012

Fiedler, K., Kutzner, F., & Krueger, J. I. (2012). The long way from α-error control to validity proper: Problems with a short-sighted false-positive debate. Perspectives on Psychological Science, 7(6), 661-669. https://doi.org/10.1177/1745691612462587 

Klein, O., Doyen, S., Leys, C., Magalhães de Saldanha da Gama, P. A., Miller, S., Questienne, L., & Cleeremans, A. (2012). Low hopes, high expectations: Expectancy effects and the replicability of behavioral experiments. Perspectives on Psychological Science, 7(6), 572-584. https://doi.org/10.1177/1745691612463704

Lash, T. L., & Vandenbroucke, J. P. (2012). Commentary: Should preregistration of epidemiologic study protocols become compulsory?: Reflections and a counterproposal. Epidemiology, 23(2), 184-188. https://doi.org/10.1097/EDE.0b013e318245c05b 

Further Information

Reference for this Resource

Rubin, M. (2024). Critical metascience articles. https://sites.google.com/site/markrubinsocialpsychresearch/replication-crisis/list-of-articles-critical-of-open-science 

Associated Research Rabbit Collection

https://www.researchrabbitapp.com/collection/public/V467G42MLX