(to be published in Philosophical Psychology)
Maarten Boudry[*] and Johan Braeckman
This paper offers an epistemological discussion of self-validating belief systems and the recurrence of “epistemic defense mechanisms” and “immunizing strategies” across widely different domains of knowledge. We challenge the idea that typical “weird” belief systems are inherently fragile, and we argue that, instead, they exhibit a surprising degree of resilience in the face of adverse evidence and criticism. Borrowing from the psychological research on belief perseverance, rationalization and motivated reasoning, we argue that the human mind is particularly susceptible to belief systems that are structurally self-validating. On this cognitive-psychological basis, we construct an epidemiology of beliefs, arguing that the apparent convenience of escape clauses and other defensive ‘tactics’ used by believers may well derive not from conscious deliberation on their part, but from more subtle mechanisms of cultural selection.
Keywords: self-validating belief systems, epidemiology of beliefs, cultural selection, immunizing strategies, epistemic defense mechanisms, cognitive constraints
"When men wish to construct or support a theory, how they torture facts into their service!" Charles Mackay (Mackay, 1974 , p. 459)
The satirist H. L. Mencken once wrote that: “the most common of all follies is to believe passionately in the palpably not true. It is the chief occupation of mankind”. Mencken was clearly engaging in hyperbole, but he has some statistics on his side (Irwin, 2009; Hines, 2003): according to a 1999 Gallup poll, 45% of all Americans believe that the earth is less than 10,000 years old and that all living species were created by God in their present form. A vast number of people believe that the murder of John F. Kennedy was a conspiracy, that the moon landing in 1969 never happened, or that the Bush administration was involved in the 9/11 attacks (a conspiracy theory that is also popular in Europe). The number of people who put faith in bogus medical treatments, or even the number of different bogus therapies, is alarmingly high. A 2005 Gallup poll conducted in the United States, Canada, and Great Britain found that ca. 25% of all persons questioned believe in astrology and around 20% believe that extraterrestrials have visited the earth recently. Even higher numbers were found for the belief in haunted houses and communication with the dead. Similarly, all over the world, sects and religious cults continue to attract followers.
In this paper, we discuss belief systems across widely different domains and focus on their self-validating nature as part of an explanation of their wide appeal and enduring popularity. After an introductory section on the received view about weird beliefs and irrationality (section 2), we review the relevant literature on belief perseverance, cognitive dissonance and motivated reasoning (section 3). We proceed by analyzing the recurrence of certain epistemic defense mechanisms and immunizing strategies in many of the most widespread “irrational” belief systems. By demonstrating that these belief structures exhibit a remarkable degree of natural resistance or resilience in the face of disconfirmation and criticism, we challenge the received idea that they are intrinsically vulnerable or fragile. (sections 4 and 5). In line with the emerging literature on cognitive constraints on the formation and distribution of beliefs (Boyer, 1994; Atran, 2002; Barrett, 2007), we explain the success of these belief systems in terms of ordinary modes of human cognition, and within the framework of an epidemiology of beliefs (section 6) (Boyer, 1994; Sperber, 1996; Lienard & Boyer, 2006). Our epistemological approach is informed by, but not reducible to, the cognitive research on motivated reasoning and cognitive dissonance, and makes for a level of explanation in its own right. Finally, we show that the epistemic “engineering” of certain belief systems may well derive not from conscious deliberation on the part of believers, but from more subtle mechanisms of cultural selection (section 7 and 8).
The tenacity of belief systems that are highly implausible, or whose content contradicts well-established scientific knowledge, has often exasperated skeptical scientists and philosophers alike. Dyed-in-the-wool skeptics, however, have long come to realize that firm believers are very difficult to convince with evidence and rational arguments. To believe otherwise is to commit the “rationalistic fallacy” (Pigliucci, 2002, pp. 234-236), the idea that all one has to do to make people abandon their pseudoscientific beliefs is to explain things a little more clearly. Experience with debating “believers” has conclusively shattered this illusion, and the phenomenon of belief perseverance in the face of disconfirming evidence has been well researched in psychology (Anderson, Lepper, & Ross, 1980; Carretta & Moreland, 1982). But why is it that rational arguments are generally to little avail in the domain of pseudoscience and the paranormal?
Among skeptics engaged in the scientific evaluation of these claims, there is a widespread conviction that believers in the paranormal and pseudoscience are obviously irrational, if not plain stupid. Carl Sagan once noted that, to his regret, many of his skeptical colleagues find that “those [...] who believe in all these stupid doctrines are morons” (C. Sagan, 1995, pp. 29-30). For example, Richard Dawkins, annoyed by the continuing opposition to evolution by religious fundamentalists, once remarked that “[i]t is absolutely safe to say that if you meet somebody who claims not to believe in evolution, that person is ignorant, stupid or insane” (Dawkins, 1989). After all, so the argument goes, pseudoscientists are not capable of rational argumentation, they simply ignore or fail to understand evidence that does not fit their dogmatic ideas, and they keep committing fallacies of reasoning that have long been laid to rest (e.g. Godfrey, 1979; Estling, 2005). The sentiment that some belief systems are so obviously wrong or absurd that no sane person would ever come to accept them, has also pervaded academic research on for example superstition and religious cults. In an attempt to explain why seemingly normal people succumb to pseudoscience and cults, some researchers have argued that such belief commitments can only be sustained by elaborate organizational devices, psychological indoctrination and other techniques for distorting normal reasoning functions. Other researchers have speculated on the existence of a special mode of “magical thinking”, which is disconnected from normal reasoning faculties, or which constitutes a pre-rational stage in the development of the human mind (for an overview, see Atran, 2002, p. 141).
Modern research in cognitive psychology belies these speculations and suggests that many of our false beliefs stem from mundane reasoning errors and biases which are inherent in the way the human mind processes information (Nisbett & Ross, 1980; Gilovich, 1991). For example, instead of attributing superstition to a special mode of “magical thinking”, researchers have documented the role of our flawed understanding of randomness and coincidence in the formation and persistence of superstitious beliefs (Gilovich, 1991; Vyse, 1997). Thus, the persistence of superstition emerges as a side-effect of our natural ability for pattern recognition and causal inference making (e.g. Foster & Kokko, 2009). In evolutionary terms, the cost of overlooking causal relations is higher than that of occasional false positives, which explains the high sensitivity of our cognitive faculties to correlations in the environment. Nesse (2001) has termed this the “smoking detector principle”, because smoking alarms are designed to err on the side of caution for the same reason. In a similar vein, researchers are starting to approach the human penchant for magic and the paranormal as a natural by-product of the way our brain employs ontological categories (e.g. physical, biological, mental) to make sense of the world (Lindeman & Aarnio, 2007). Paranormal beliefs are then regarded as the result of a confusion between the core attributes of these ontological categories. In general, cognitive psychologists have argued that local irrationality emerges as the inevitable by-product of our brain’s ability to efficiently gather and process information (Talmont-Kaminski, 2008, 2009).
As we will see, evidence against the sheer “irrationality” or “stupidity” of believers has also come from sociological studies of cults and sects, which have shown that even the most outlandish belief systems have a form of internal logic and coherence, and that believers are often not the simple-minded fanatics they are taken to be (e.g. Dein, 2001; Tumminia, 1998; Lukes, 2007). Moreover, cults and sects who did rely on crude brainwashing and mind control devices to recruit new members have generally met with little success (Mercier, in press; Streatfeild, 2007). It seems that attempts to impair people’s normal reasoning abilities is an inefficient way of winning new converts for a “weird” belief system.
In this paper, we want to focus on one particular misconception underlying the sentiments of skeptical outsiders of weird belief systems. Echoing the remarks of H. L. Mencken, people often express puzzlement about what they perceive as the “palpable” falsity of these belief systems: “How could anyone in his right mind believe such nonsense?” In other words, people assume that pseudoscientific belief systems are “inherently fragile” (Snow & Machalek, 1982) – that believers are constantly faced with overwhelming adverse evidence, which would compel any reasonable person to immediately give up such beliefs. We argue that this assumption is largely misguided, and that a closer look at the structure of weird belief systems reveals that believers are in fact well-prepared to withstand such difficulties. More specifically, once believers accept the central premises of the belief system in question, they have ample explanatory resources at their disposal to reason their way through apparent disconfirmations, and to withstand criticism from outsiders. Before delving into this discussion, however, we need to cover some psychological ground about irrationality, motivated reasoning and belief perseverance.
Almost half of the population in the United States believe that all living species were created in their present form by God, and that the earth is less than 10,000 years old. Adhering to a literalist reading of Scripture, these people also believe in the Garden of Eden, Noah’s Ark and the Great Flood. It is difficult to find a belief that flies more in the face of modern science. Consequently, it is tempting to argue that, since the creationist belief system is so “palpably not true”, those who endorse it surely must be completely insensible to rational arguments and evidence.
On the other hand, it is surprising to note how often creationists go to considerable lengths to “massage” scientific evidence into their preconceived Biblical framework. Moreover, they have – often with amusing results – taken great pains to overcome exegetical inconsistencies and other difficulties in their belief systems to answer questions like: How did Noah manage to get all the animals into the Ark? If Adam and Eve were the only humans around, did Cain marry his own sister? What did T. Rex use his claws and teeth for in the Garden of Eden, before death and carnivore diets entered the world? Why does the book of Genesis contain two apparently inconsistent accounts of creation?
Of course, the arguments invented by creationists to address these problems are selective, distorted and heavily biased (Kitcher, 1982; Pennock, 1999; Pigliucci, 2002), but there is a more interesting point to be made: for people who are blindly and “irrationally” committed to religious faith, many creationists have a surprising concern with inconsistencies and adverse evidence. Why do they not just ignore anything that does not fit their ideas, instead of bothering themselves with elaborate rationalizations and ad hoc explanations? One might object that creationists merely need these elaborate arguments to attract new converts, but that explanation pushes the question ahead. If religious fundamentalism is simply about committing oneself blindly to the truth of a holy book, why do people need arguments to be convinced in the first place? In other words, why are creationists not more irrational?
3. The Psychology of Belief Perseverance
3.1 Motivated reasoning and confirmation bias
Psychologists have found some truth in the received psychological wisdom about irrationality. For example, research on “confirmation bias” suggests that people actively avoid being confronted with disconfirming evidence (Nickerson, 1998), and they sometimes simply fail to notice discrepancies between their beliefs and the available evidence (Benassi, Singer, & Reynolds, 1980). This literature on “confirmation bias” is sometimes taken to imply that people simply “forget” failures and ignore adverse evidence, but the actual psychological mechanism may be more interesting: for example, researchers found that people betting on sport games remember their losses better than their wins. Rather than just “forgetting” their apparent failures, people take the time to scrutinize them intensely in search of elements that allow them to rationalize these failures away. Typically, they accept their wins at face value but rely on ad hoc explanations to turn their losses into “near wins” (Gilovich, 1983). Researchers have shown that humans are remarkably creative in inventing such ad hoc explanations for events (Ross, Lepper, Strack, & Steinmetz, 1977) and in explaining away adverse evidence to rescue cherished beliefs from refutation. (Gilovich, 1991; Tumminia, 1998). In a classical experiment, Lord and his colleagues (Lord, Ross, & Lepper, 1979) asked defenders and opponents of capital punishment to read two studies, one of which suggested that the death penalty deterred people from committing crimes, whereas the other suggested that it was not an effective deterrent. Both groups detected more methodological problems in the study disfavoring their own beliefs, and hence rated this study lower, while they took the study in favor of their own beliefs at face value.
Kunda (1990) explains that, ironically, this pervasiveness of ad hoc reasoning and special pleading suggests that there are limits to the extent to which people are engaged in “motivated reasoning”: “[T]he biasing role of goals is [...] constrained by one’s ability to construct a justification for the desired conclusion: People will come to believe what they want to believe only to the extent that reason permits” (Kunda, 1990, p. 483). This is because people like to think of themselves as objective and unbiased reasoners. In psychological terms, they place a high premium on consistency and impartiality (Kunda, 1990; Tavris & Aronson, 2008; von Hippel & Trivers, in press). When people are motivated to cling to a belief, they do not feel comfortable with blithely ignoring adverse evidence or simply shutting their ears to anyone who opposes their views. Instead, they engage in more subtle forms of ad hoc reasoning, rationalization and special pleading to arrive at their desired conclusions and to justify their beliefs to others, e.g. reinterpreting the facts, weighing them against background knowledge, finding some reason to discredit the source, etc. (Gilovich, 1991, pp. 54-56). This practice allows them to uphold an “illusion of objectivity concerning the manner in which [...] inferences were derived” (Pyszczynski & Greenberg, 1987, p. 302).
The conventional wisdom on so-called “wishful thinking” is that, as Francis Bacon put it, man “prefers to believe what he prefers to be true”. The psychological evidence, however, suggests that there are constraints on the ways in which people let their desires and goals guide their beliefs (Ditto & Lopez, 1992). The cognitive premium on a flattering self-image of being unbiased and reasonable explains why many people, even defenders of weird belief systems, will typically scrutinize adverse evidence until they find some justification for rejecting or ignoring it (see the use of immunizing strategies in section 5).
Apart from that, it is clear that the power of “wishful thinking” is limited in yet another respect. Often enough, people are firmly committed to weird beliefs that show no signs of wishful thinking at all, but in fact confirm their worst fears. For example, many among the most widespread irrational beliefs – eternal damnation, witchcraft, evil conspiracies – are positively frightening and menacing (Guthrie, 1993; Atran, 2002, pp. 75-78). If it were true that people believe what they prefer to be true, why do they not just prefer not to believe in these phantoms, a fortiori when the evidence for them is so scant? Certainly, as psychological research has shown, if people have already committed themselves to belief in witchcraft or hell, and have acted accordingly over a period of time, they may be motivated to persevere in that belief. That brings us to the next section.
3.2 Cognitive dissonance
In many everyday situations, believers just accept adverse evidence and revise their beliefs accordingly (Sperber, et al., in press). If I think the capital of Ghana is Abidjan and I find in an atlas that is in fact Accra, I do not write a letter of complaint to the publisher of that atlas. If I thought I left my keys in the drawer and I cannot find them there, I usually revise my belief (still, if I am really confident or just stubborn, I can persist that someone else must have taken them away). Under what circumstances can we expect people to persevere in their beliefs and to explain away such evidence? Cognitive dissonance theory suggests that, from a psychological and motivational point of view, there has to be something at stake.
According to cognitive dissonance theory (Festinger, Schachter, & Riecken, 1964; Aronson, 1992; Tavris & Aronson, 2008), when people are presented with new evidence that conflicts with their previously held beliefs, this results in a form of cognitive tension called “dissonance”. Importantly, the strength of this uncomfortable tension depends on the degree to which people have invested in their beliefs, for example by way of public commitment, or by the time and effort spent acting in accordance with these beliefs (Batson, 1975). If the psychological investment in a belief is high, people are more motivated to reduce dissonance by rationalizing away disconfirming data. In the refined version of dissonance theory, dissonance arises not so much because of two conflicting cognitions, but because adverse evidence conflicts with one’s self-esteem as a competent and reasonable person. This accords with our earlier observation that, when people explain away unwelcome evidence, they do so in a way that allows them to uphold an illusion of objectivity. For example, if a psychic has publicly professed his powers and risks losing his credibility, he is unlikely to be put off his balance by blatant failure. Or if a believer has spent a substantial amount of time and money on astrology consults, typically no amount of rational argumentation and debunking efforts will make him renounce his beliefs. As Nicholas Humphrey noted: “psychic phenomena can, it seems, survive almost any amount of subsequent disgrace” (Humphrey, 1996, p. 150). By contrast, if the psychological stakes are low, as in the everyday situations we mentioned above, the motivation for belief perseverance will be greatly reduced. Consider another example related to paranormal beliefs: suppose that Anna and Paul both start to suspect that they have psychic powers, but their level of confidence is not very high. While Paul hastens to tell his friends that he may be psychic and even performs some psychic readings, Anna decides to conduct an experiment on herself at an early point, when her beliefs are still privately held. All other things being equal, it is much more likely that Anna will abandon her beliefs silently when she discovers that they do not pan out (Humphrey, 1996, p. 105), while Paul will rationalize his failures because he has already made a public commitment. Thus, we would predict that people with an inquisitive and cautious mindset are more likely to put their hunches to the test early on, and are less likely to be sucked into commitment to wrong beliefs like these. By contrast, people who rush to conclusions and start spreading the news right away will more often find themselves in a situation where they obstinately refuse to abandon a false belief.
A classic illustration of cognitive dissonance can be found in the landmark study by Leon Festinger and his colleagues, who infiltrated a doomsday cult and observed the behavior of the followers when the prophesized end of the world failed to come true (Festinger, et al., 1964). The followers who had resigned from their jobs, given away their material belongings and were present at the arranged place and time with full conviction in their imminent salvation, became even more ardent believers after the prophecy failed, and started to proselytize even more actively for the cult. However, those for whom the cognitive stakes were lower (e.g. those who kept their belongings and stayed home in fearful expectation of what was supposedly to come), were more likely to abandon their beliefs afterwards.
Early cognitive dissonance theory required that prophecies be sufficiently specific and unequivocal, and that believers themselves acknowledge them to be roundly refuted. This aspect of Festinger’s theory was belied by more recent studies of millennial cults (for an overview, see Dawson, 1999), which suggest that believers afterwards rarely if ever recognize that the prophecy which they issued has not been borne out. Indeed, the denial of failure is “the common mode of adaptation of millennial groups” (Melton, 1985, p. 21). Instead of recognizing failure and proceeding to ignore it completely, as early cognitive dissonance theory suggested, committed believers explain away apparent failure by means of semi-plausible post hoc rationalizations (Dein, 2001; Dawson, 1999), consistent with the psychological findings on the illusion of objectivity.
Cognitive dissonance theory is concerned with the conditions that give rise to dissonance and that motivate for dissonance reduction, but as such it does not describe the way in which this tension is resolved. In section 3.1, we have already shown that belief perseverance is typically achieved by ad hoc reasoning and rationalization rather than blunt denial, and in the next section, we consider what conditions are conducive to this practice.
Cervantes’ classic novel Don Quixote tells the tale of an elderly gentleman who is obsessed with books of chivalry, and has succumbed to the delusion that he is an errant-knight on an epic mission to restore the golden age of chivalry. Although he is confronted with a series of tragic defeats and humiliations, Don Quixote is able to persevere in his grand delusion by invoking invisible malicious wizards thwarting his every action. But Quixote is not stupid. When the canon, one of the characters in the novel, is confronted with the “extraordinary nature of Don Quixote’s madness”, he marvels that “in all his remarks and replies he should show such excellent sense, and only lose his stirrups […] when the subject of chivalry was broached” (Cervantes, 2008, p. 644).
If an ardent believer is confronted with what outsiders perceive as clearly disconfirming evidence of his belief system, he will tend to resist belief revision to the extent that he is able to come up with plausible rationalizations and excuses in the face of difficulties. As we saw, these rationalizations allow the believer to uphold an “illusion of objectivity”, thus reducing the level of cognitive dissonance. If we now raise the question as to what non-motivational factors facilitate this ability to rationalize away apparent failures, we can take up two different perspectives, one cognitive-psychological and one epistemological.
From a psychological point of view, one may ask in what way intelligence is related to belief perseverance. Although the received view holds that intelligent people are less likely to accept wrong beliefs, we submit that, once intelligent people become highly committed to a belief, it will prove more difficult to put them off their balance with adverse evidence and criticism. Just as Don Quixote had no difficulties in explaining away his failures and in brushing aside counterarguments, skilled reasoners are more proficient at inventing and constructing rationalizations in the face of difficulties, and they will be more prone to belief perseverance when they experience cognitive dissonance. Even worse, intelligent people may be more vulnerable to wrong beliefs in the first place. Mercier (in press) has argued that the more people rely on reasoning through communication to achieve epistemic improvement, the more they are likely to accept a number of wrong beliefs amidst the bulk of true beliefs. According to Mercier, this explains why highly intelligent people have the habit of endorsing some quite weird beliefs. For example, Michael Shermer has discussed the cases of such scientific luminaries as the renowned cosmologist Frank Tipler, who has tried to demonstrate the dogmas of Christianity with intricate mathematical and physical equations, and Harvard psychiatrist John Mack, who is a firm believer in alien abductions. Shermer argues that precisely the intellectual skills of these individuals make them highly proficient in defending implausible beliefs, and hence prone to perseverance in the face of overwhelming adverse evidence: “Smart people believe weird things because they are skilled at defending beliefs they arrived at for non-smart reasons” (Shermer, 2003, p. 64).
In addition to this psychological take on the issue, we argue that it is fruitful to approach the problem of irrationality and belief perseverance from an epistemological perspective as well. What if some systems of beliefs are more resilient in the face of adverse evidence and criticism? What if some of them provide more explanatory and conceptual resources for believers to draw upon in the face of difficulties? In that case, the belief system itself would be conducive to belief perseverance and rationalization on the part of individual believers. If we accept this possibility, we can develop a straightforward epidemiological model of beliefs (Sperber, 1990, 1996; Dawkins, 1993; Boyer, 1994, 1998): over the course of the history of human culture, millions upon millions of beliefs have been tried out and entertained by as many different persons, and only some of these are remembered, acquired and transmitted. If we accept that there is constant cultural and interpersonal variation in the generation and transmission of beliefs, it is inevitable that, for a variety of complex reasons, some ideas will be more successfully remembered, recalled and propagated. As Pascal Boyer noted: “certain features are recurrent because they are more likely to be entertained, acquired, and transmitted by human minds” (Boyer, 1994, p. ix; 1998). Importantly, as Scott Atran suggested, considerations of “cognitive optimality might be at work not only at the level of individual beliefs but at the level of belief structures as well” (Atran, 2002, p. 101). The claim we want to develop, is that one of the constraints that channel beliefs and belief structures is the degree of structural resilience they exhibit to adverse evidence and critical arguments. We argue that this epistemological consideration partly accounts for the puzzling popularity of certain “weird” beliefs systems. Note that our account must be distinguished from straightforward selectionist approaches of culture, typically in terms of memes or culturgens (Blackmore, 2000; Dawkins, 1976; Richerson & Boyd, 2005). Following the critiques of Boyer, Atran and Sperber, we think that memetics and similar approaches take a too simplistic view of the notion of replication, and they largely obscure the shaping role of our cognitive architecture (Sperber, 2000; Boyer, 1994). Nevertheless, the seminal work on memes and mind viruses by Dawkins (1976, 1993) and Dennett (1991, 1996) contains valuable insights into the ways self-validating belief structures coalesce and are rendered impervious to criticism and adverse evidence.
Elsewhere (Boudry & Braeckman, 2010) we have documented how many “weird” belief systems exhibit certain internal, structural features that render them invulnerable to adverse evidence and critical arguments. We have termed these “epistemic defense mechanisms” and distinguished them from “immunizing strategies”, which are defined as arguments brought forward in support of a belief system. In contrast to epistemic defense mechanisms, immunizing strategies are independent from the belief system at hand. The distinction between two types is not always clear-cut, as immunizing strategies may loosely draw on theory-internal resources, or develop into an integral part of the theory over time.
For the purposes of this paper, we provide a brief sample of epistemic defense mechanisms, which will suffice to demonstrate that the alleged fragility of typical weird belief systems rests on a superficial analysis.
5. Epistemic Defense Mechanisms
In astrology and in prophetical works such as those of Nostradamus, it is typical to be presented with a series of ambiguous statements having what psychologists call “multiple endpoints” (Gilovich, 1991, pp. 58-59; Hines, 2003), as in the parody prediction by Woody Allen: “Two nations will go to war, but only one will win”. In fact, typical astrological descriptions are amenable both to a specific interpretation and a range of broader and more metaphorical ones, e.g. “a father-figure stands behind you”. This creates an asymmetry between what will count as hits and misses of the predictions in question, allowing the astrologer – or gullible believer – to switch back and forth between specific and broad interpretations. In this way, practically any outcome will be perceived as a fulfillment of astrological predictions. In a variation on this theme, a belief system consists of statements that are specific and exciting on first inspection, but when running into trouble, they are belatedly modified so as to make them trivial or uninteresting. The deflationary re-interpretation of a failed doomsday prediction is a standard example of this move: a typical solution for apparent prophetic failure is to reinterpret the promised events on an invisible and spiritual level (Tumminia, 1998, p. 168; Melton, 1985; Zygmunt, 1970). For example, when their prediction of the Second Coming of Christ in 1873–74 failed to come true, Jehovah’s Witnesses argued that Christ had returned as predicted, but as an invisible spirit being (Zygmunt, 1970, p. 931). We also find the strategy in the use of conceptual “moving targets” in the writings of pseudoscientists (Cioffi, 1998; Boudry, Blancke, & Braeckman, in press). More generally still, the indeterminate and mysterious nature of many religious and pseudoscientific propositions ensures that they are closed to normal epistemic evaluation (Sperber, et al., in press), and that contradictions and adverse evidence will go largely unnoticed to the believers. (Sperber, 1996, pp. 91-92; see also the discussion of "quasi-propositions" in religion, Atran, 2002)
In certain belief systems, invisible causes are postulated to account for a range of phenomena, in such a way that their working can only be inferred ex post facto from the observed effects. If the causal relations and conditions in the belief system are not sufficiently specified, and allow for all sorts of secondary elaborations, believers can get entangled in subtle feedback loops between theory and observations, which keep the belief system forever outside the reach of empirical refutation (Boyer, 1994). Consider the belief in magical rituals, healing crystals, shamanic powers etc. In such cases, the effects are typically used to retrospectively determine the activity of whatever invisible cause is dictated by the belief system: such-and-such must have happened to account for the observed effects. Any apparent failure, then, can be explained away by arguing that, apparently, the intervention was not of the right type, or not performed properly, or interfered with another invisible cause, etc. This pattern of spurious postdiction is also apparent in the way parapsychologists explain away null results and cherry pick data to “determine” where and when psi forces were active (Wiseman, 2010). Likewise, cults groups often draw upon a range of unfalsifiable concepts and events to avert disconfirmation (e.g. Dein, 2001).
5.3 Conspiracy thinking
(a) Conspiracy theorists typically believe, against the received view of a historical event, that a group of interested agents have been secretly pulling the strings to bring about the event in question, all the while carefully covering up their actions. They argue for this view on the basis of “errant data” (Keeley, 1999, p. 118), i.e. anomalies, unexplained details and inconsistencies in the official story. On the other hand, when investigations fail to reveal the conspiracy or even flatly contradict it, believers typically turn the evidence on its head, arguing that this is exactly what would be predicted by their view. After all, conspirators, being who they are, can be reasonably expected to erase all traces of evidence leading to their plot, and to lead the rest of us astray with forged evidence (Clarke, 2002). From this perspective, conflicting testimonies and disconfirming material evidence merely bear witness to the power and cunning of the conspirators.
(b) Furthermore, the conspiratorial pattern of reasoning allows believers to explain away the motives for disbelief and criticism within their own belief system, for example by accusing the skeptics of being somehow implicated in the conspiracy themselves. In this way, criticism of any sort is immediately deflected and transformed into further confirmation of the belief system. Take, for example, the creationist idea that the theory of evolution was concocted by Satan himself to lure the faithful into disbelief (Morris, 1963, p. 93), the Freudian argument regarding unconscious “resistance” on the part of critics of psychoanalysis (Freud, 1957, p. 39), and the idea that extraterrestrials have lulled us into disbelief by manipulating our minds (Jacobs, 1998, p. 258).
5.4 Invisible escape clauses
In many pseudoscientific belief systems, we are confronted with an imponderable force or cause that, when push comes to shove, confounds the expectations initially engendered by the theory, and conveniently explains away apparent failure. A host of such escape clauses can be found in the field of parapsychology: e.g. the idea that the presence of inquisitive minds tends to disturb psychical phenomena, which is known as “negative psi vibration” or “catapsi” (for a sceptical discussion, see Humphrey, 1996; Wiseman, 2010); the argument that psi is “actively evasive” because its primary function is to “induce a sense of mystery and wonder” (Kennedy, 2003, p. 67); the “decline effect” (Beloff, 1994, p. 11), which refers to the puzzling tendency of psychics to lose their powers as they are tested more extensively; and so on. Other examples include the ad hoc creationist argument that God forged geological evidence for an ancient universe to test our faith in Scripture, or the astrologer’s belated invocation of the formation of stars at the moment of conception when his prediction on the basis of the birth date has failed. In some cases, we are dealing with an immunizing strategy that is independent from the belief system at hand. In other cases (e.g. parapsychology), these escape clauses may develop into fully-fledged epistemic defense mechanisms, forming an integral part of the theory.
The central claim developed in this paper is that beliefs that develop into systems which are more successful in withstanding empirical failures and in “surviving” the onslaught of critical arguments, be it from the inquisitive believers themselves or from skeptical outsiders, will be more readily acquired, remembered and selected among their competitors.
Of course, beliefs do not “develop” into self-validating structures all by themselves. Beliefs are entertained by individual agents, and they are modified and revised by individual agents. Over time, the problems these agents encounter within their system of beliefs will inspire solutions in the form of modifications, reinterpretations and elaborations. Not all of these changes will be equally successful from a psychological point of view, and hence not all of them will tend to survive. We claim that certain “successful” configurations of beliefs may be expected to become recurrent in widely different domains, despite huge cultural and interpersonal variation. Thus, rather than turning into full-blown self-validating structures all at once, we maintain that beliefs crystallize into such systems after a number of successive modifications and elaborations, which result from attempts to resolve inconsistencies and to rescue the belief system from apparent refutation.
Consider again the case of doomsday cults, which are literally confronted with the problem of surviving the day on which prophecy fails. If the day of truth arrives and the predicted events are not borne out, the belief system is faced with serious institutional crisis (Zygmunt, 1970). If, on the other hand, the system is flexible enough to cope with eventual failure, by allowing for some convenient escape clauses, excuses or reinterpretations, it may withstand the impact of reality (Balch, Domitrovich, Mahnke, & Morrison, 1997). For example, the cult of Jehovah’s Witnesses has a long history of what outsiders perceive as blatant prophetic failures, but the movement does not show any signs of disappearing. This is partly because, as Zygmunt’s study on Jehovah’s Witnesses makes clear, the prophecies of the cult were phrased “in a manner that made them only partially open to disconfirmation” (Zygmunt, 1970, p. 944). As they allow for enough “wiggling room”, the failed prophecies can always be retrospectively related to real historical events, and thus be “converted into partial successes” (Zygmunt, 1970, pp. 944-945), strengthening the conviction of the followers and renewing their proselytizing efforts. In other words, the belief system of Jehovah’s Witnesses has made use of the defense mechanism of multiple endpoints and deflationary revisions which we described above.
Of course, these reinterpretations do not present themselves spontaneously, but that does not mean that they are deliberately constructed by believers with strategic purposes in mind (see section 8). In the doomsday cult, a plausible post hoc rationalization of prophetic failure is typically suggested by the group leader and taken up by the other members (Dawson, 1999, p. 65). Alternatively, group members may entertain different rationalizations and reinterpretations, in the full conviction that the prophecy must be true in some sense, and the solutions that emerge as cognitively “optimal” are taken up by other believers. In this way the belief system may slowly develop an increasing resilience in the face of adverse data.
Thus, if the believers succeed in constructing elaborations on or carrying out reinterpretations of their belief system that make it impervious to empirical failure (to which they will often be strongly motivated), the belief system will survive the day on which the prophecy fails, and live on in this more resistant form. All other things being equal, the weird belief systems that reach a cultural level of dissemination tend to be the ones that have stabilized on a form that is immune to the empirical refutation and criticism from outsiders. Those that are too fragile wither away and are simply no longer there for us to observe.
In contrast to doomsday prophecies, most belief system do not hinge on a single moment of truth sometime in the (distant) future. Most weird belief systems – conspiracy theories, homeopathy, magic healing, parapsychology, ufology, etc. – involve claims that are supposed to derive support from currently available evidence. Essentially, however, the epistemological predicament of these belief systems is no different, as they too have to provide for the resources to cope with unwelcome facts and with disbelievers. Conspiracy theories, which are equipped with built-in protection against empirical failure and hostile criticism from outsiders, have “succeeded” in this regard. Their epistemic structure guarantees that believers will always have some way of explaining away difficulties. As we noticed above, Don Quixote’s delusional belief system was fundamentally conspiratorial in nature (Farrell, 1996). Anything that was in apparent conflict with Quixote’s fantasy world of knights, castles, maidens and dragons, was interpreted by him in terms of malicious sorcerers who make things appear different than they are. As with any conspiracy thinker, nothing and nobody could convince Quixote that the world of chivalrous knights existed only in his imagination.
In many conspiracy theories that are currently popular on the Internet, apparently disconfirming evidence is interpreted as forged evidence and false information spread by the conspirators, and detractors are suspected of being part of the conspiracy itself, having been bribed by the government, or having merely been misled by the cunning of the evil plotters. As in the case of doomsday prophecies, the social dynamic of a group of believers may further facilitate this process. If a new piece of evidence turns up that seems to be in conflict with the conspiracy hypothesis, or a new argument is voiced by critics, different ways of explaining away these difficulties may be tried out, and the ones that are most “successful” from a psychological perspective, in virtue of their allowing believers to preserve an illusion of objectivity, are taken up by other members to become part of the belief system.
As we saw, the conspiracy template turns up in a variety of different belief systems, as it is such a convenient way of dealing with problems. For example, creationists in the second half of the 20th century have cultivated the idea that evolutionary theory and all the evidence supporting it is nothing less than a satanic ploy to lure the faithful into disbelief (Morris, 1963). Similarly, UFO believers have proclaimed for over several decades that there is a vast government conspiracy to obscure the real evidence for extraterrestrial visits to the earth (Hines, 2003, pp. 257-298; Park, 2002, Ch. 9). In general, if a group of people is firmly committed to a system of beliefs, which is then increasingly threatened by mounting adverse evidence, the community of (remaining) believers will often settle on a form of conspiracy defense. The reader may object that this is not what usually happens in disputes between scientists, even though many of them are also highly committed to a cherished theory or hypothesis. Be that as it may, resistance to change and belief perseverance are certainly not alien to scientific disputes, even though science is valued as a self-corrective enterprise that depends on the relentless correction and overthrow of old theories. Scientists too can be unduly conservative in their beliefs, but typically they resort to more sophisticated ways for rescuing a theory from falsification (Boudry, in press). Nevertheless, in some heated scientific controversies, the losing party does resort to conspiracy theorizing: see for example the downfall of cold fusion, the Duesberg hypothesis about the non-infectious nature of AIDS, and more recently, the small pockets of continuing scientific resistance to theories of anthropogenic climate change (Pigliucci, 2010).
The epidemiological argument outlined in this paper emerges in light of persistent cognitive features that conspire to make us vulnerable to self-validating belief systems: (a) our proficiency at ad hoc reasoning and rationalization; (b) the motivation to reduce cognitive dissonance; (c) the persistence of the confirmation bias; and (d) the psychological premium placed on being rational and free from bias.
Note that an analysis pertaining to the epistemic structure of belief systems accounts for only one factor in a more general epidemiological model of culture. In Sperber’s epidemiological model of representations, a host of cognitive, psychological, ecological and cultural factors channel the formation and dissemination of beliefs. In particular, Sperber has focused on the “relevance” (Sperber, 1985; Sperber & Wilson, 1995) of representations to domain-specific cognitive modules. Briefly, according to Sperber, representations that are relevant independently of a local context will stand a greater chance of cultural success: “independence of an immediate context means that relevance will be maintained in spite of changes of local circumstances – that is, it will be maintained on a social scale” (Sperber, 1996, p. 140).
By way of illustrating how these relevance considerations apply likewise to the belief systems discussed here, consider again the case of conspiracy theories. Our susceptibility to evil conspiracy theories is not only a result of their self-validating epistemic structure, but is arguably also a function of at least two specific cognitive modules: a mechanism for agency detection (Guthrie, 1993; Barrett, 2000; Atran, 2002; Barrett, 2004) that is biased towards over-attribution of agency in our environment, and a “hazard-precaution system” (Lienard & Boyer, 2006) geared towards detecting danger and acting in dangerous situation. Seeing that conspiracies involve the secret and potentially threatening actions of hidden agents, we realize why they tend to activate the cognitive processes mentioned above, and hence why they never fail to command our attention. In the words of Liénard and Boyer, cognitive modules such as these are liable to “cognitive capture” (Lienard & Boyer, 2006, p. 821) by specific representations that meet their input conditions. Our epistemological analysis further contributes to an understanding of how evil conspiracies of all stripes – cover-ups by the government, secret plans of the Illuminati or the Elders of Zion, etc. – often reach a level of wide cultural dissemination and why they are so resistant to adverse evidence.
Although our epistemological argument is informed by research on motivated reasoning and cognitive dissonance, we view it as a level of explanation in its own right, which allows for the identification of patterns and trends that are not visible from the lower level of cognition. Our susceptibility to self-validating belief systems becomes only transparent when we connect different cognitive and psychological findings and pursue an epistemological approach to the problem.
Epistemic defense mechanisms and immunizing strategies may appear convenient for the believer who is motivated to cling to his beliefs, but in fact to think in strategic and intentional terms may be misleading here. If our analysis of the epistemic rationale of self-validating belief systems is accurate, this appearance of strategic “convenience” may well be the outcome of cultural-selection processes rather than straightforward and conscious deliberation. In this respect, it is interesting to have a brief look at the suspicions which skeptics of pseudoscience have often voiced regarding the sincerity of believers. We rehearse two often-heard arguments to that effect and proceed to show why they are largely misguided (or at least inconclusive).
7.1 Avoiding tricky situations
Many pseudoscientists seem to carefully avoid situations that would put them at a risk of empirical refutation. For example, mediums and clairvoyants have all sorts of excuses for refusing to participate in the type of controlled test that is bound to expose their lack of powers all too clearly (Hines, 2003). As we noted, believers consider proper scientific investigation of the paranormal inappropriate or impossible, arguing that the phenomena in question are, in the words of a skeptic, “unpredictable, unrepeatable, shy, highly context-dependent, droll, evanescent, dreamlike” (Humphrey, 1996, p. 73). Likewise, not only do successful astrologers and soothsayers avoid making claims that are too vulnerable to refutation, but they are particularly reluctant to do so when questioned by skeptics. Some pseudoscientists even expressly warn against giving demonstrations in the vicinity of inquisitive minds. As in the words of one of Franz Anton Mesmer’s followers: “Never magnetize before inquisitive persons!” (quoted in Mackay, 1974 , p. 290). For many skeptics, this suspiciously evasive behavior on the part of believers is a telltale sign of insincerity.
Many people proclaim to believe in supernatural or paranormal causation, all the while relying on more mundane courses of action when push comes to shove. For example, as Nicholas Humphrey noted, many people profess to believe in telepathy, but “when they themselves want to communicate to a distant friend, they play safe and write or call them” (Humphrey, 1996, p. 55). Interestingly, anthropologists have noted that, in many cultures where supernatural spells and magic are used for achieving a certain material goal (rainfall, a good harvest, victory in a battle), people always make sure to rely on more down-to-earth methods as well, which suggests that they are not all too confident in supernatural causation as they claim to be. In a paper discussing magic and religion, anthropologist Eli Sagan noted:
A people going to war may sing over their spears in order to make them more effective. If there ever have been people who felt they could defeat an enemy in war merely by singing and who therefore dispensed with spears we have not heard of them (E. Sagan, 1979, p. 93).
Similarly, many believers suddenly lose their professed faith in the paranormal and supernatural when their own lives are at stake. For example, when the chakra healer himself falls seriously ill, he will make sure to consult a regular doctor. In other cases, supernatural faith suddenly becomes somewhat half-hearted. For example, when Pope John Paul II was shot and critically wounded in an assassination attempt in 1981, he asked the surgeons not to remove his Brown Scapular during the operation, stating that Our Lady of Fátima would help him recover. But why did the Pope rely on scientific medicine and surgery in the first place, instead of putting faith in supernatural help? On the basis of similar examples, Humphrey and other skeptics have suggested “that most people know only too well how things stand” (Humphrey, 1996, p. 56).
Against the two arguments presented in section 7, we submit that believers’ suspicious behavior may well result from more subtle mechanisms of self-deception and rationalization (von Hippel & Trivers, in press). We briefly show how the epidemiological argument presented here supports that conclusion. As for the first argument, it is instructive to imagine the fate of a psychic who is not so careful to avoid tricky situations or who is not equipped with a bag of excuses for doing so. For example, an astrologer who is confident enough to make very risky predictions is bound to have a hard time explaining his failure after the fact. A self-proclaimed psychic who recklessly accepts the invitation for a scientific experiment, unprepared for refutation, will sorely disappoint himself, not to mention the followers who witness the failure. By contrast, psychic healers and mediums who happen to come across an argument that suggests to them that scientific investigation of psi is impossible or inappropriate, will be less likely to be confronted with cases of blatant failure. Those who have learnt and cultivated successful excuses for shying away from tricky tests are precisely those who are still in the game to defend their beliefs and to convince others.
Of course, this sketchy explanation leaves open many unanswered psychological questions. Further discussion may bear on how selectionist explanations translate on the psychological level, and how they relate to issues of intentionality. If an agent starts to rationalize away adverse evidence for her beliefs, either through arguments of her own or through ones that she has picked up elsewhere, does this not suggest a certain level of awareness on her part? It seems that, at least, the agent has come to realize that a certain situation may prove threatening to her belief. Moreover, in section 3.1 we suggested that agents consciously scrutinize conflicting evidence in search for reasons to dismiss it. Still, we think this is not incompatible with an agent sincerely holding a weird belief: precisely the agent’s solid conviction of being right arguably motivates her search for flaws in unwelcome evidence (“something must be wrong with this data”). In any case, a thorough discussion of intentionality, self-deception and motivated reasoning falls outside the scope of this paper (see for example Mele, 1994; von Hippel & Trivers, in press).
In relation to the second argument, we can apply a similar reasoning. People who expect the sort of supernatural or paranormal causation that would make more mundane courses of action to the same effect superfluous, are bound to be disappointed by the results. The person who asks a friend for dinner by telepathic means alone will surely be spending a lonely night. In general, those who expect tangible results from psi powers will be forced either to abandon their belief in psi, or to correct – by way of rationalization – their expectations on the causal power of psi in a way that does not make ordinary modes of action superfluous.
As for the cases of faltering faith when life is at stake, an even more obvious selection process is at work. For example, it is not difficult to imagine what will happen to a people who relies wholeheartedly on magical spellbinding and dispensed with weaponry. As Sagan (1979, p. 93) dryly noted, they will probably be “all dispatched in the midst of their spellbinding”. Indeed, there are many documented cases of people who go to battle virtually unarmed because they believe they enjoy supernatural protection. Similarly, up until this day many religious people who are inflicted by a lethal disease relinquish any form of medical treatment, convinced as they are that faith alone – or “alternative” therapy – can save them (e.g. Peters, 2008; Edgerton, 1992). If anything, the anthropological evidence suggests that people are perfectly capable of sincere conviction in highly dangerous beliefs. In any case, to return to our present argument, even when self-proclaimed supernaturalists pursue more mundane courses of action as soon as life is at stake, this does not necessarily mean that their beliefs are insincere. Again, the apparent design may be “authorless”, resulting from a process of cultural selection.
In summary, we cannot take the “convenience” of believers’ suspicious and evasive behavior at face value, i.e. as a token of strategic deliberation. Rather than being the result of conscious deliberation on the part of individual believers, epistemic defense mechanisms and evasive behavior in general may exhibit what Daniel Dennett has termed a “free-floating” rationale (Dennett, 1996, p. 78, see also pp. 164-165).
Note that our epidemiological argument does not deny the existence of conscious deliberation to the same effect. Even if the question of sincerity may often be difficult to resolve, it is obvious that there are a lot of conscious impostors among pseudoscientists and paranormal mediums (Wiseman, 1997, p. 12), and that religious leaders need not always be sincere in their beliefs (Dennett & LaScola, 2010). For example, mediums such as Uri Geller and faith healers like Peter Popoff have been caught cheating several times during performances. Thus, the decision to protect one’s self-proclaimed paranormal powers from exposure can be perfectly deliberate. However, as Dennett notes:
A tactic that works can be used deliberately and viciously, but it can also work – sometimes better – in the hands of an innocent enthusiast who would never dream of doing anything duplicitous (Dennett, 2006, p. 365).
Consider magician Robert-Houdin’s advice to performers “never [to] announce beforehand the nature of the effect which you intend to produce” (quoted in Wiseman, 1997, p. 42). In case something goes wrong, this golden rule allows the magician to finish the trick in another way, without having failed in the eyes of the public. But the same rationale can work without intentional action, for example in the hands of a psychic who has “learnt” not to announce the psi effects he intends to produce.
In this paper, we challenged the common assumption that pseudosciences and other forms of weird belief systems are inherently fragile. Instead, they exhibit a surprising degree of resilience in the face of apparently adverse evidence and criticism from outsiders. Based on a number of findings in cognitive psychology, we argued that this invulnerability of belief systems may in part explain their unabated popularity. All other things being equal, belief systems that allow the believer to remain outside the reach of refutations, or that provide some convenient ways of coping with difficulties, will be more likely to be selected among competing beliefs and belief systems, and more likely to be disseminated. In this way, our argument is intended as a contribution to the general question about human culture set forth by, among others, Dan Sperber: “[W]hy are some representations more successful in a human population, more “catching” than others?” (Sperber, 1996, p. 58). We also noted that the use of epistemic defense mechanisms and immunizing strategies, together with the generally evasive behavior of pseudoscientists, often strike the outsider as suspiciously convenient. However, rather than being the outcome of conscious deliberation on the part of believers, this strategic convenience may well be authorless – resulting from mechanisms of cultural selection.
Our susceptibility to self-validating belief systems is a function of several aspects of the way our human “belief engine” works: its inclination towards confirmation bias, its proficiency at rationalization and ad hoc reasoning, its valuation of an appearance of objectivity, and its motivation for cognitive dissonance reduction. If we view these insights from cognitive psychology in an epistemological light, and if we insert them in an epidemiological model of beliefs, then the enduring popularity of self-validating belief systems and the recurrence of defense mechanisms and immunizing strategies is hardly surprising.
The authors are grateful to Amber Griffioen, Helen De Cruz, Hugo Mercier, Nicholas Humphrey, Stefaan Blancke and the anonymous reviewers of Philosophical Psychology for valuable comments and suggestions. Parts of this paper were presented at the Fourth Conference of the Dutch-Flemish Association for Analytic Philosophy at the University of Leuven (2010).
This work was supported by the Fund for Scientific Research Flanders (FWO).
Anderson, C. A., Lepper, M. R., & Ross, L. (1980). Perseverance of social theories: The role of explanation in the persistence of discredited information. Journal of Personality and Social Psychology, 39(6), 1037-1049.
Aronson, E. (1992). The return of the repressed: Dissonance theory makes a comeback. Psychological Inquiry, 3(4), 303.
Atran, S. (2002). In gods we trust : The evolutionary landscape of religion. Oxford: Oxford University Press.
Balch, R. W., Domitrovich, J., Mahnke, B. L., & Morrison, V. (1997). Fifteen years of failed prophecy: Coping with cognitive dissonance in a Baha’i sect. In T. Robbins & S. J. Palmer (Eds.), Millennium, messiahs, and mayhem (pp. 73-90). New York: Routledge.
Barrett, J. L. (2000). Exploring the natural foundations of religion. Trends in Cognitive Sciences, 4(1), 29-34.
Barrett, J. L. (2004). Why would anyone believe in God? Walnut Creek (Calif.): AltaMira Press.
Barrett, J. L. (2007). Cognitive science of religion: What is it and why is it? Religious Compass, 1(6), 768-786.
Batson, C. D. (1975). Rational processing or rationalization?: The effect of discontinuing information on stated religious belief. Journal of Personality and Social Psychology, 32(1), 176-184.
Beloff, J. (1994). Lessons of history. Journal of the American Society for Psychical Research, 88(7), 7-22.
Benassi, V. A., Singer, B., & Reynolds, C. B. (1980). Occult belief: Seeing is believing. Journal for the Scientific Study of Religion, 19(4), 337-349.
Blackmore, S. J. (2000). The meme machine. Oxford: Oxford University Press.
Boudry, M. (in press). The hypothesis that saves the day. Ad hoc reasoning in pseudoscience. Logique et Analyse.
Boudry, M., Blancke, S., & Braeckman, J. (in press). Irreducible incoherence and intelligent design. A look into the conceptual toolbox of a pseudoscience. Quarterly Review of Biology, 85(4).
Boudry, M., & Braeckman, J. (2010). Immunizing strategies & epistemic defense mechanisms. Philosophia. doi:10.1007/s11406-010-9254-9
Boyer, P. (1994). The naturalness of religious ideas : A cognitive theory of religion. Berkeley (Calif.): University of California Press.
Boyer, P. (1998). Cognitive tracks of cultural inheritance: How evolved intuitive ontology governs cultural transmission. American Anthropologist, 100(4), 876-889.
Boyer, P. (2001). Religion explained: The evolutionary origins of religious thought. New York (N.Y.): Basic Books.
Carretta, T. R., & Moreland, R. L. (1982). Nixon and Watergate: A field demonstration of belief perseverance. Personality and Social Psychology Bulletin, 8(3), 446.
Cervantes, M. (2008). The history of Don Quixote, Volume 1. Charleston, SC: Bibliobazaar.
Cioffi, F. (1998). Freud and the Question of Pseudoscience. Chicago: Open Court.
Clarke, S. (2002). Conspiracy theories and conspiracy theorizing. Philosophy of the Social Sciences, 32(2), 131.
Dawkins, R. (1976). The selfish gene. Oxford: Oxford University Press.
Dawkins, R. (1989, April 9, 1989). Review of blueprints: Solving the mystery of evolution. By Maitland A. Edey and Donald C. Johanson. New York Times, p. 34, from http://www.philvaz.com/apologetics/p88.htm
Dawkins, R. (1993). Viruses of the mind. In B. Dahlbom (Ed.), Dennett and his critics: Demystifying mind (pp. 13-27). Oxford: Blackwell.
Dawson, L. L. (1999). When prophecy fails and faith persists: A theoretical overview. Nova Religio, 3(1), 60-82.
Dein, S. (2001). What really happens when prophecy fails: The case of Lubavitch. Sociology of Religion, 62(3), 383-401.
Dennett, D. C. (1991). Consciousness explained. Boston: Little, Brown and Company.
Dennett, D. C. (1996). Darwin's dangerous idea : Evolution and the meanings of life. New York: Simon and Schuster.
Dennett, D. C. (2006). Breaking the spell: Religion as a natural phenomenon. New York, N.Y.: Viking (Penguin).
Dennett, D. C., & LaScola, L. (2010). Preachers who are not believers. Evolutionary Psychology, 8(1), 122-150.
Ditto, P. H., & Lopez, D. F. (1992). Motivated skepticism: Use of differential decision criteria for preferred and nonpreferred conclusions. Journal of Personality and Social Psychology, 63, 568-568.
Edgerton, R. B. (1992). Sick societies: Challenging the myth of primitive harmony. New York & Toronto: Free Press.
Estling, R. (2005). Stupid things. Skeptical Inquirer, 29(1), 56-57.
Farrell, J. (1996). Freud's paranoid quest: Psychoanalysis and modern suspicion. London: New York University Press.
Festinger, L., Schachter, S., & Riecken, H. W. (1964). When prophecy fails : a social and psychological study of a modern group that predicted the destruction of the world. New York (N.Y.): Harper and Row.
Foster, K. R., & Kokko, H. (2009). The evolution of superstitious and superstition-like behaviour. Proceedings of the Royal Society of London, Series B, 276, 31-37.
Freud, S. (1957). The standard edition of the complete psychological works of Sigmund Freud. Vol. 11, (1910) : Five lectures on psycho-analysis, Leonardo da Vinci, and other works. London: The Hogarth Press and the Institute of Psycho-analysis.
Gilovich, T. (1983). Biased evaluation and persistence in gambling. Journal of Personality and Social Psychology, 44(6), 1110-1126.
Gilovich, T. (1991). How we know what isn't so : the fallibility of human reason in everyday life. New York (N.Y.): Free press.
Godfrey, L. (1979). Science and Evolution in the Public Eye. Skeptical Inquirer, 4(1), 21-32.
Guthrie, S. (1993). Faces in the clouds : a new theory of religion. New York (N.Y.): Oxford university press.
Hines, T. (2003). Pseudoscience and the paranormal (2nd ed.). Amherst, NY: Prometheus Books
Humphrey, N. (1996). Soul searching : Human nature and supernatural belief. London: Vintage.
Irwin, H. J. (2009). The psychology of paranormal belief. Hatfield, Hertfordshire: University of Hertfordshire Press.
Jacobs, D. M. (1998). The threat: The secret agenda: What the aliens really want … and how they plan to get it. New York: Simon & Schuster.
Keeley, B. L. (1999). Of conspiracy theories. Journal of Philosophy, 96(3), 109-126.
Kennedy, J. E. (2003). The Capricious, Actively Evasive, Unsustainable Nature of Psi: A Summary and Hypotheses. The Journal of Parapsychology, 67(1), 53-75.
Kitcher, P. (1982). Abusing science: The case against creationism. Cambridge (Mass.): MIT Press.
Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108(3), 480-480.
Lienard, P., & Boyer, P. (2006). Whence collective rituals? A cultural selection model of ritualized behavior. American Anthropologist, 108(4), 814.
Lindeman, M., & Aarnio, K. (2007). Superstitious, magical, and paranormal beliefs: An integrative model. Journal of Research in Personality, 41(4), 731-744.
Lord, C. G., Ross, L., & Lepper, M. R. (1979). Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence. Journal of Personality and Social Psychology, 37(11), 2098-2109.
Lukes, S. (2007). The problem of apparently irrational beliefs. In S. P. Turner & M. W. Risjord (Eds.), Philosophy of anthropology and sociology (pp. 591-606). Amsterdam: Elsevier/North-Holland.
Mackay, C. (1974 ). Extraordinary popular delusions and the madness of crowds. New York: Barnes & Noble Publishing.
Mele, A. R. (1994). Self-control and belief. Philosophical Psychology, 7(4), 419-435.
Melton, J. G. (1985). Spiritualization and reaffirmation: What really happens when prophecy fails. American Studies, 26(2), 17.
Mercier, H. (in press). Our pigheaded core: How we became smarter to be influenced by other people. In B. Calcott, R. Joyce & K. Sterelny (Eds.), Signaling, Commitment and Emotion. Cambridge (Mass.): MIT Press.
Morris, H. M. (1963). Twilight of evolution. Grand Rapids, Mich: Baker Pub Group.
Nesse, R. M. (2001). The smoke detector principle. Annals of the New York Academy of Sciences, 935, 75-85.
Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2, 175-220.
Nisbett, R., & Ross, L. (1980). Human inference: Strategies and shortcomings of social judgment. Englewood Cliffs (N.J.): Prentice-Hall.
Park, R. L. (2002). Voodoo science: the road from foolishness to fraud. New York: Oxford University Press.
Pennock, R. T. (1999). Tower of Babel : the evidence against the new creationism. Cambridge (Mass.): MIT press.
Peters, S. F. (2008). When prayer fails: Faith healing, children, and the law. Oxford: Oxford University Press.
Pigliucci, M. (2002). Denying evolution: Creationism, scientism, and the nature of science. Sunderland, MA: Sinauer Associates.
Pigliucci, M. (2010). Nonsense on stilts. How to tell science from bunk. Chicago and London: University of Chicago Press.
Pyszczynski, T., & Greenberg, J. (1987). Toward an integration of cognitive and motivational perspectives on social inference: A biased hypothesis-testing model. In L. Berkowitz (Ed.), Advances in experimental social psychology (Vol. 20) (pp. 297–340). New York: Academic Press.
Richerson, P. J., & Boyd, R. (2005). Not by genes alone: How culture transformed human evolution. Chicago (Ill.): University of Chicago press.
Ross, L., Lepper, M. R., Strack, F., & Steinmetz, J. (1977). Social explanation and social expectation: Effects of real and hypothetical explanations on subjective likelihood. Journal of Personality, 35(11), 817-829.
Sagan, C. (1995). Wonder and skepticism. Skeptical Enquirer, 19(1), 24–30.
Sagan, E. (1979). Religion and magic: A developmental view. Sociological Inquiry, 49(2-3), 87-116.
Shermer, M. (2003). Why smart people believe weird things. Skeptic, 10(2), 62-73.
Snow, D. A., & Machalek, R. (1982). On the presumed fragility of unconventional beliefs. Journal for the Scientific Study of Religion, 21(1), 15-26.
Sperber, D. (1985). Anthropology and psychology: Towards an epidemiology of representations. Man, 20(1), 73-89.
Sperber, D. (1990). The epidemiology of beliefs. In G. Gaskell & C. Fraser (Eds.), The social psychological study of widespread beliefs (pp. 25-44). Oxford: Clarendon Press.
Sperber, D. (1996). Explaining culture: A naturalistic approach. Cambridge, Mass.: Blackwell.
Sperber, D. (2000). An objection to the memetic approach to culture. In R. Aunger (Ed.), Darwinizing culture: The status of memetics as a science (pp. 163-173). Oxford: Oxford University Press.
Sperber, D., Clément, F., Heintz, C., Mascaro, O., Mercier, H., Origgi, G., et al. (in press). Epistemic Vigilance. Mind & Language.
Sperber, D., & Wilson, D. (1995). Relevance : communication and cognition (2nd ed.). Oxford: Blackwell.
Streatfeild, D. (2007). Brainwash: The secret history of mind control. New York: Thomas Dunne Books.
Talmont-Kaminski, K. (2008). In a mirror, darkly: Does superstition reflect rationality? Skeptical Enquirer, 32(4), 48-51.
Talmont-Kaminski, K. (2009). The fixation of superstitious beliefs. Teorema, 28(3), 81-95.
Tavris, C., & Aronson, E. (2008). Mistakes were made (but not by me) : why we justify foolish beliefs, bad decisions, and hurtful acts. London: Pinter and Martin.
Tumminia, D. (1998). How prophecy never fails: Interpretive reason in a flying-saucer group. Sociology of Religion, 59(2), 157-170.
von Hippel, & Trivers, R. (in press). The evolution and psychology of self-deception. Behavioral and Brain Sciences.
Vyse, S. A. (1997). Believing in magic : the psychology of superstition. New York (N.Y.): Oxford university press.
Wiseman, R. (1997). Deception & self-deception: Investigating psychics. Amherst, N.Y.: Prometheus Books.
Wiseman, R. (2010). 'Heads I win, tails you lose'. How parapsychologists nullify null results. Skeptical Inquirer, 34(1), 36-39.
Zygmunt, J. F. (1970). Prophetic failure and chiliastic identity: The case of Jehovah's Witnesses. The American Journal of Sociology, 75(6), 926-948.
[*] Correspondence to: Maarten Boudry, Dept. of Philosophy & Moral Sciences. Blandijnberg 2, Ghent University, 9000 Ghent, Belgium. Email: email@example.com
 Aronson distinguishes three components of a positive self-image that are shared by most people: a consistent and stable self-image; a sense of self as a competent person; and a sense of self as a morally good person. In this paper, we are mainly concerned with people’s self-image of being competent and reasonable.
 The James Randi Educational Foundation offers a $1,000,000 prize to anyone who is able to demonstrate, under controlled observing conditions, evidence of paranormal or supernatural powers. Claimants for the challenge are always asked to conduct a private experiment on themselves before coming to the official test. Interestingly, after being instructed as to how a proper self-test can be performed, many of them are never to be heard of again. By contrast, according to Randi’s experience, those who do turn up for the real test, because they failed to conduct such a self-test (or did not do it properly), always make recourse to rationalizations to explain away their failure.
 As we will see, the very notion of “clearly disconfirming evidence” will become problematic once we take into account the defense mechanisms and escape clauses inherent in the system.
 This effect of increased epistemic resilience can be relatively small, compared to other cultural and cognitive constraints. However, as Liénard and Boyer noted: “In cultural transmission […] very small effects aggregated over many cycles of transmission are sufficient to create massive trends” (Lienard & Boyer, 2006, p. 824).
 Of course, it is quite possible that the success of such a belief system is compromised by other factors offsetting the gain in epistemic resistance. For example, too many elaborations and defense mechanisms may render the belief system cumbersome and/or too complicated, hampering recall and transmission to other believers. In this way, the very features that account for its epistemic immunity may make the belief system less successful in other respects. As in many biological adaptations (e.g. the peacock’s tail), it is plausible that, in such cases, a trade-off will take place.
 And even in the case of Jehovah’s Witnesses, the prophecy is embedded in a complex network of beliefs and practices.
 In the newly emerging field of Cognitive Science of Religion (CSR) (Boyer, 1994; Barrett, 2000; Boyer, 2001; Atran, 2002; Barrett, 2004; for a recent overview, see Barrett, 2007), researchers have brought these insights from cognitive psychology to bear on the study of religion: “much of what is typically called religion may be understood as the natural product of aggregated ordinary cognitive processes” (Barrett, 2000, p. 29).
 According to cognitive and evolutionary psychologists, the hypersensitivity of these cognitive mechanisms makes good evolutionary sense: traces in the grass or a rustle of leaves may signal the presence of enemies, sickness may be the result of poisoning by a rival, etc. The cost incurred by a false negative (failing to detect agents) is significantly greater than the cost of a false positive (detecting agency where there is none).
 Another example, which we have already mentioned, is our inclination towards superstitious beliefs, which may be partly explained by our cognitive proficiency at pattern detection and our difficulties with evaluating random coincidences (Gilovich, 1991).
 Interestingly, the skeptical literature does contain a few cases of psychics who made a “reckless” claim, agreed to be put to the test, and afterwards did accept the negative verdict, or at least started to doubt their powers.