Questionable Research Practices

In this new article, I consider questionable research practices in the field of metascience. A questionable metascience practice (QMP) is a research practice, assumption, or perspective that’s been questioned by several commentators as being potentially problematic for metascience and/or the science reform movement. I discuss 10 QMPs that relate to criticism, replication, bias, generalization, and the characterization of science. My aim is not to cast aspersions on the field of metascience but to encourage a deeper consideration of its more questionable research practices, assumptions, and perspectives.

10 QMPs

1. Rejecting or ignoring self-criticism: Rejecting or ignoring criticisms of metascience and/or science reform

2. Fast ‘n’ bropen scientific criticism: A quick, superficial, dismissive, and/or mocking style of scientific criticism

3. Overplaying the role of replication: Assuming that replication is essential to science, and that it indexes “the truth”

4. Unspecified replication rate targets: Assuming a replication rate is “too low” without specifying an “acceptable” rate

5. Metabias: An unacknowledged bias towards explaining the replication crisis in terms of researcher bias

6. A bias reduction assumption: Focusing on selective reporting as the primary form of researcher bias and assuming that it can be reduced without increasing other forms of bias

7. Devaluing exploratory results: Devaluing an exploratory result as being more “tentative” than a confirmatory result without considering other relevant issues (e.g., quality of associated theory, methods, analyses, transparency)

8. Presuming QRPs are problematic: Presuming that questionable research practices (e.g., HARKing) are always problematic research practices

9. A focus on knowledge accumulation: Conceiving knowledge accumulation as the primary objective of science without considering (a) the role of specified ignorance or (b) different objectives in other philosophies of science

10. Homogenizing science: Focusing on specific approaches (e.g., quantitative methods; replicable effects) as “the scientific method”

A Caveat

In my article, I stress that only some metascientists engage in some QMPs some of the time, and that these QMPs may not always be problematic. Research is required to estimate the prevalence and impact of QMPs. In the meantime, I think that QMPs should be viewed as invitations to ask questions about how we go about doing better metascience.

Further Information

The Article

Rubin, M. (2023). Questionable metascience practices. Journal of Trial and Error. https://doi.org/10.36850/mr4 

Journal of Trial and Error Special Issue

My article is part of a special issue in the Journal of Trial and Error: “Consequences of the scientific reform movement: Is the scientific reform movement headed in the right direction?” https://journal.trialanderror.org/pub/callscientificreform/release/4

List of Critical Metascience Articles

Many of the articles that I cite in my article are included in this list of critical metascience articles.

Everything Hertz Podcast

Dan Quintana and James Heathers review my paper on their podcast Everything Hertz here. They conclude "more papers like this please!"