Assessing the Belief Bias Effect with ROCs: It’s a Response Bias Effect

A belief bias effect in syllogistic reasoning (Evans, Barston, & Pollard, 1983) is observed when subjects accept more valid than invalid arguments, more believable than unbelievable conclusions, and show greater overall accuracy in judging arguments with unbelievable conclusions. The effect is measured with a contrast-of-contrasts, comparing the acceptance rates for valid and invalid arguments with believable and unbelievable conclusions. We show that use of this measure entails the assumption of a threshold model, which predicts linear receiver operating characteristics (ROCs). In 3 experiments, subjects made valid/invalid responses to syllogisms, followed by confidence ratings that allowed the construction of empirical ROCs, the form of which was inconsistent with the predictions of Klauer, Musch, and Naumer’s (2000) multinomial model of belief bias. We propose a more appropriate, signal detection based, model of belief bias. We then use that model to develop theoretically sound and empirically justified measures of decision accuracy and response bias; those measures demonstrate that the belief bias effect is simply a response bias effect. Thus, our data and analyses challenge all existing theories of belief bias, because those theories depend entirely on an assessment of the effect, the contrast-of-contrasts, which is inappropriate for the structure of the data.