Why is this topic so important?

Put it this way: When the most popular, most central, and most fundamental studies, concepts, and methods once widely celebrated by researchers start to fall and fail before our eyes, you've got to ask why the world (at least, the psychological universe) is falling apart.

The answer? Sometimes, it's because what we think is there is just there due to chance. It's not likely to happen again, or at least not all the time. Sometimes, with the pressure to publish or the greed to achieve fame, researchers plunge into the dark side and make up amazing conclusions from thin air.

That's the replication crisis part. So where's the revolution?

More and more, psychologists are realizing that the old ways of doing science isn't working. They say that we have to look at other statistics, other ways of publishing, and other ways of being scientific citizens.

With the field scarred and marred by doubts after the replication fiasco, psychologists now hold on to more rigorous considerations of participant welfare and scientific integrity - quite literally, a revolution into a more credible, transparent, inclusive, and human science.

This guide is a walkthrough for the uninitiated - the casual traveler who's curious about the revolution in psychology. While you're here, we're going to look at both academic papers and more approachable op-eds from scientific journalism. Welcome aboard, brave the storm, and join psychology in its journey.

• Repeat After Me (Naro 2016 / The Nib)

• The Replication Crisis in Psychology (Diener & Biswas-Diener 2019 / NOBA Project)

• The Replication Crisis [CrashCourse Statistics]

• I Fooled Millions Into Thinking Chocolate Helps Weight Loss. Here’s How. (Bohannon 2015 / Gizmodo)

• Psychology experiments in top journals - are they true? (80000Hours)

• The current crisis in psychology research in the Philippines: Reproducibility or producibility? (Doble, Sio, & Clemente 2018)

• Psychology is in crisis whether it's in crisis (Palmer 2016 / WIRED)

  • How reliable are psychology studies? (Yong 2015 / Atlantic)

  • (based on) Estimating the reproducibility of psychological science (Open Science Collaboration 2015 / Science)

• More social science studies just failed to replicate. Here’s why this is good. (Resnick 2018 / Vox)

  • (based on) Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015 (Camerer et al. 2018 / Nature Human Behavior)

• Psychology’s Replication Crisis Can’t Be Wished Away (Yong 2016 / Atlantic)

Many Ways that Findings Fail

• Scientists often fail when they try to replicate studies. This psychologist explains why. (Belluz 2015 / Vox)

• The replication crisis may also be a theory crisis (O'Grady 2019 / Ars Technica)

A short survey of failed findings

Chance, Confounds, and Chaotic Constructs

• Power Posing: When the Revolution Came for Amy Cuddy (Dominus 2017 / New York Times)

  • (on the ethics of calling out) The Trials of Amy Cuddy (Engber 2017 / Slate)

• Embodied Cognition: The replication crisis is killing psychologists’ theory of how the body influences the mind (Goldhill 2019 / Quartz)

• Ego Depletion: Everything Is Crumbling (Engber 2016 / Slate)

• ESP: Daryl Bem Proved ESP Is Real (Engber 2017 / Slate)

• Delay of Gratification: Why Rich Kids Are So Good at the Marshmallow Test (Calarco 2018 / Atlantic)

Intersecting Replication and Ethics

• 'Gaydar' AI: Why Stanford Researchers Tried to Create a ‘Gaydar’ Machine (Murphy 2017 / New York Times)

  • (what AI was really doing) Machine learning about sexual orientation? (Calling Bullshit)

  • (ethical considerations) AI Gaydar Study Gets Another Look (Flaherty 2017 / Inside Higher Ed)

• Criminal Machine Learning: China’s big brother: how artificial intelligence is catching criminals and advancing health care (Aldama 2017 / South China Morning Post)

  • (reconsidering) Criminal machine learning (Calling Bullshit)

Data, Deception, and Downright Fraud

• Meet the ‘data thugs’ out to expose shoddy and questionable research (Marcus & Oransky 2018 / ScienceMag)

• Why Most Published Research Findings Are False (Ioannidis 2005 / PLOS)

• p-Hacking

  • Hack Your Way to Scientific Glory (demo)

  • p-Hacker (demo)

  • CrashCourse Statistics Ep. 30

• Retraction Watch

• Brian Wansink: Here’s How Cornell Scientist Brian Wansink Turned Shoddy Data Into Viral Studies About How We Eat (Lee 2018 / BuzzFeed News)

• Diederik Stapel: The Mind of a Con Man (Bhattacharjee 2013 / New York Times)

• Philip Zimbardo: The Lifespan of a Lie (Blum 2018 / Medium)

Redefining Significance & The New Statistics

• The Value of p-Values: What a nerdy debate about p-values shows about science — and how to fix it (Resnick 2017 / Vox)

  • Statistical Errors (Nuzzo 2014 / Nature)

  • When the statistical tail wags the scientific dog (Morey 2017 / Medium)

  • Redefine statistical significance (Benjamin et al. 2018 / Nature Human Behavior)

  • Remove, rather than redefine, statistical significance (Amrhein & Greenland 2018 / Nature Human Behavior)

• The New Statistics - and the Bayesian-Frequentist Debates: Redefining statistical significance: the statistical arguments (Morey 2018 / Medium)

  • The Flawed Reasoning Behind the Replication Crisis (Clayton 2019)

  • The New Statistics: Why and How (Cumming 2013 / Psychological Science)

• More Statistics Resources

  • Specification Curve (Simonsohn, Simmons, & Nelson 2015 / SSRN)

  • Seeing Theory

  • Open Stats Lab

  • Calling Bullshit

Open Science

• Open science is now the only way forward for psychology (Chambers & Etchells 2018 / Guardian)

  • A manifesto for reproducible science (Munafo et al. 2017 / Nature Human Behavior)

  • Open Science: One Term, Five Schools of Thought (Fecher & Friesike 2013)

• Open Access: Open Science Isn't Always Open to All Scientists (Bahlai et al. 2019 / American Scientist)

  • The war to free science (Resnick & Belluz 2019 / Vox)

  • Paywall: The Business of Scholarship

• Intellectual humility: the importance of knowing you might be wrong (Resnick 2019)

  • The importance of stupidity in scientific research (Schwartz 2008)

• Failure Is Moving Science Forward (Aschwanden 2016 / FiveThirtyEight)

• What psychology’s crisis means for the future of science (Resnick 2016 / Vox)