The Problem in Behavioral Priming Research is Equal Parts of Theory and Data

Post date: Jan 16, 2014 10:17:40 AM

In this recent article (http://pps.sagepub.com/content/9/1/72.abstract) Ap Dijksterhuis welcomes back theory in research on behavioral priming. Some have argued that social priming research lacks a strong theoretical foundation (e.g., Cesario, 2014, in the same Perspectives on Psychological Science issue, which is regrettably not open access). I am actually pretty impressed with early theoretical work on social priming, and recently re-read a lot of papers by Wyer, Srull, Smith, Carlston, Hastie, and others who provided social priming research with a very solid foundation in the early eighties. It is a real shame that this work has been almost completely ignored in the last 20 years, or at least has seen practically no progress. So yes, welcome back, theory!

Here, I want to focus on a problem that is just as big, but much less appreciated: Lack of data. It seems some people do not really appreciate how very, very little data they have to build theories on. In the article by Dijksterhuis, it is clear that there are 2 things he fails to appreciate. The first is the file-drawer problem. The second is the reliability of the available empirical support for social priming. This becomes clear from the following quote (Dijksterhuis, 2014, p 73):

“It is interesting that the few published nonreplications have led some to suggest that behavioral priming may not exist. However, there are good reasons to believe that the fear that psychology is infested with false positives is largely unnecessary (Dalton, Aguinis, Dalton, Bosco, & Pierce, 2012; Murayama, Pekrun, & Fiedler, in press); in the case of behavioral priming, the hundreds of papers cannot be erased by the mere flick of a skeptic magic wand, no matter how hard you try.”

Let me explain why this statement is wrong, and in my view represents a fundamental misunderstanding of the current state of psychological science (or any other discipline that relies on statistical inferences and suffers from publication bias).

1 The file-drawer: No biggie?

First of all, I encountered the references to Dalton, Aguinis, Dalton, Bosco, & Pierce (2012), and Murayama, Pekrun, & Fiedler (in press) in a submission for a special issue of Social Cognition I reviewed (I was asked to submit a manuscript about embodiment research for the same special issue, which is currently under editorial consideration, perhaps more on that later). In this manuscript by John Bargh (cited by Dijksterhuis, 2014) the same reference was used to downplay the file-drawer problem. I tried to prevent these claims from getting into the published literature in my review of the manuscript (which I signed, as I always do), but since the argument is now part of the published record through the publication by Dijksterhuis, I will share my criticisms on this line of reasoning here.

The work by Dalton et al (2012) states that the file drawer problem does not pose a serious threat to the validity of meta-analytically derived conclusions (it appeared in a journal called 'Personnel Psychology' - I wonder how it would be treated in a methods journal). This must seem complete nonsense to anyone familiar with the file-drawer problem, and it obviously is. The conclusion by Dalton et al cannot be applied to social priming research, because Dalton et al (2012) focus on nonexperimental research (more specifically, correlation matrices), and there is no reason to assume their conclusions generalize to experimental research, where a single test of an hypothesis is significant or not, and is only published in the first, but not the latter instance (unlike correlation matrices). It might be interesting to read how the article is discussed by others. For example, Kepes & McDaniel (2013) write, in a footnote:

We acknowledge that Dalton, Aguinis, Dalton, Bosco, and Pierce (2012) concluded that publication bias is not worrisome in our literature. However, that paper stands alone in that conclusion. We note that the Dalton et al. (2012) effort differed from all other published publication bias analyses in that did not examine any speciﬁc research topic in the literature (Kepes, McDaniel, Brannick, & Banks, 2013). As such, we do not ﬁnd it an informative contribution to the publication bias literature (i.e., publication bias is concerned with the availability of effect sizes on a particular relation of interest).

Although I would have liked to read in the footnote that the conclusion does not generalize to experimental research (instead of the focus on a specific relation of interest), these authors understand that the article stands alone in its conclusion. It might hold for correlation matrices in some sub disciplines, but the conclusion does not hold for social priming research. In my view, the citation of Dalton et al. in the article by Dijksterhuis is bad scholarship.

Similarly, referring to Muryama et al as support for the fact that the number of Type 1 errors is not a big problem is not really fair. As Muryama et al state: “We do not by any means intend to argue that current research in social psychology is this healthy and has sufficient self-cleansing capabilities” p. 1). The research practices discussed by Muryama (replications, a-priori hypotheses, etc) WOULD reduce Type 1 error IF they are used – and we should interpret the likelihood that there are Type 1 errors in a research area based on the extent that these practices have been used. In social priming research (as in most other domains), there are no pre-registered hypotheses, very few close replications, and no way to estimate the number of failed experiments (but see the p-curve section below, for a key to the file-drawer). This citation seems to suggest the article by Muryama argues all is well in social psychology – but nothing could be further from the truth.

2 Erasing Social Priming by the Flick of a P-curve.

Dijksterhuis correctly notes that you cannot erase hundreds of papers by the flick of a wand. Indeed, magic is not real. Luckily for researchers, p-curve analyses (Simonsohn, Nelson, & Simmons, 2013) are real. Let’s imagine social priming studies have observed significant effects with the following p-values:

p = .032, p = .001, p = .021, p = .045, p = .012, p = .002, p = .028, p = .038, p = .016, p = .044, p = .015, p = .023, etc.

Anyone familiar with the distribution of p-values under the null-hypothesis will see what I was typing in: a completely uniform distribution of p-values. If the null-hypothesis is true, every p-value is equally likely. If all significant studies that reveal social priming effects have a distribution that is uniform, the studies would lack evidential value. We would have to treat social priming as a research area that consists primarily (there might be some exceptions) as research findings that represent selection bias (either of possible tests performed on the same set of data, or of the datasets themselves). It’s not the flick of a wand, but it’s much better: It’s a thorough understanding of statistics, applied at a meta-level, to draw inferences from published findings, without being limited by publication bias. If the p-values would consist of a lot of p-values between .00 and .01, and only very few p-values between .04 and .05, social priming would show evidential value.

What would a p-curve reveal? I started p-curving social embodiment findings (closely related to social priming, and many studies are done by authors who have also published social priming studies) and based on an analyses of 100 effects from over 40 published articles, I am not optimistic (but more on that later as well). I will share my analysis soon, so you can judge the results for yourself, but when it comes to studies that examine effects of concrete primes on behavior, decisions, evaluations, etc, I think there is reason to worry. If the 100 effects in my analyses can, taken together, lack evidential value, so can hundreds of social priming studies. Never underestimate the number of Type 1 errors hundreds of researchers can produce if they apply themselves to a specific research area for more than a decade. Do the math.

The empirical basis of social priming research is not as strong as authors such as Dijksterhuis and Bargh think. With an increase in pre-registered replications, combined with meta-analytic analyses that are not influenced by publication bias such as p-curve analyses, we will start to get a much more realistic view on the reliability of our research, in social priming, and beyond.