American paleontologist Stephen Jay Gould was one of the most popular and highly regarded science writers of the late 20th century. All 300 of the monthly columns that he wrote for Natural History magazine were collected in books, and his numerous awards include the National Book Award (1981), the Distinguished Scientist Award (1997), Humanist of the Year (2001), and the Darwin-Wallace Medal (2008).
Gould was also a devoted baseball fan. In his essay "The Streak of Streaks" (first published in The New York Review of Books in 1988), he explored the significance of Joe Dimaggio's 56-game hitting streak in 1941, generally considered to be "the greatest accomplishment in the history of baseball, if not all modern sport." In excerpts from this essay, Gould relies on examples and statistics to explain--and debunk--the popular belief known as "hot hands," a version of the gambler's fallacy.
Statistics and mythology may seem the most unlikely bedfellows. How can we quantify Caruso or measure Middlemarch? But if God could mete out heaven with the span (Isaiah 40:12), perhaps we can say something useful about hitting streaks. The statistics of “runs,” defined as continuous series of good or bad results (including baseball’s streaks and slumps), is a well-developed branch of the profession, and can yield clear—but wildly counterintuitive—results.
Start with a phenomenon that nearly everyone both accepts and considers well understood—”hot hands” in basketball. Now and then, someone just gets hot, and can’t be stopped. Basket after basket falls in—or out as with “cold hands,” when a man can’t buy a bucket for love or money (choose your cliché). The reason for this phenomenon is clear enough; it lies embodied in the maxim: “When you’re hot, you’re hot; and when you’re not, you’re not.” You get that touch, build confidence; all nervousness fades, you find your rhythm; swish, swish, swish. Or you miss a few, get rattled, endure the booing, experience despair; hands start shaking and you realize that you shoulda stood in bed.
Everybody knows about hot hands. The only problem is that no such phenomenon exists. The Stanford psychologist Amos Tversky studied every basket made by the Philadelphia 76ers for more than a season. He found, first of all, that probabilities of making a second basket did not rise following a successful shot. Moreover, the number of “runs,” or baskets in succession, was no greater than what a standard random, or coin-tossing, model would predict. (If the chance of making each basket is 0.5, for example, a reasonable value for good shooters, five hits in a row will occur, on average, once in thirty-two sequences—just as you can expect to toss five successive heads about once in thirty-two times, or 0.55.)
Of course Larry Bird, the great forward of the Boston Celtics, will have more sequences of five than Joe Airball—but not because he has greater will or gets in that magic rhythm more often. Larry has longer runs because his average success rate is so much higher, and random models predict more frequent and longer sequences. If Larry shoots field goals at 0.6 probability of success, he will get five in a row about once every thirteen sequences (0.65). If Joe, by contrast, shoots only 0.3, he will get his five straight only about once in 412 times. In other words, we need no special explanation for the apparent pattern of long runs. There is no ineffable “causality of circumstance” (if I may call it that), no definite reason born of the particulars that make for heroic myths—courage in the clinch, strength in adversity, etc. You only have to know a person’s ordinary play in order to predict his sequences. (I rather suspect that we are convinced of the contrary not only because we need myths so badly, but also because we remember the successes and simply allow the failures to fade from memory. More on this later.)
My colleague Ed Purcell, Nobel laureate in physics but, for purposes of this subject, just another baseball fan,2 has done a comprehensive study of all baseball streak and slump records. His firm conclusion is easily and swiftly summarized. Nothing ever happened in baseball above and beyond the frequency predicted by coin-tossing models. The longest runs of wins or losses are as long as they should be, and occur about as often as they ought to.
That old Persian tentmaker, Omar Khayyám, understood the quandary of our lives:
Into this Universe, and Why not knowing,
Nor Whence, like Water willy-nilly flowing; And out of it, as Wind along the Waste,
I know not Whither, willy-nilly blowing.
But we cannot bear it. We must have comforting answers. We see pattern, for pattern surely exists, even in a purely random world. (Only a highly nonrandom universe could possibly cancel out the clumping that we perceive as pattern. We think we see constellations because the stars are dispersed at random in the heavens, and therefore clump in our sight.) Our error lies not in the perception of pattern but in automatically imbuing pattern with meaning, especially with meaning that can bring us comfort, or dispel confusion.
Amos Tversky, who studied “hot hands,” has performed a series of elegant psychological experiments with Daniel Kahneman.5 These long-term studies have provided our finest insight into “natural reasoning” and its curious departure from logical truth. To cite an example, they construct a fictional description of a young woman: “Linda is thirty-one years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.” Subjects are then given a list of hypothetical statements about Linda: they must rank these in order of presumed likelihood, most to least probable. Tversky and Kahneman list eight statements, but five are a blind, and only three make up the true experiment:
Linda is active in the feminist movement;
Linda is a bank teller;
Linda is a bank teller and is active in the feminist movement.
Now it simply must be true that the third statement is least likely, since any conjunction has to be less probable than either of its parts considered separately. Everybody can understand this when the principle is explained explicitly and patiently. But all groups of subjects, sophisticated students who ought to understand logic and probability as well as folks off the street corner, rank the last statement as more probable than the second. (I am particularly fond of this example because I know that the third statement is least probable, yet a little homunculus in my head continues to jump up and down, shouting at me—”but she can’t just be a bank teller; read the description.”)
Why do we so consistently make this simple logical error? Tversky and Kahneman argue, correctly I think, that our minds are not built (for whatever reason) to work by the rules of probability, though these rules clearly govern our universe. We do something else that usually serves us well, but fails in crucial instances: we “match to type.” We abstract what we consider the “essence” of an entity, and then arrange our judgments by their degree of similarity to this assumed type. Since we are given a “type” for Linda that implies feminism, but definitely not a bank job, we rank any statement matching the type as more probable than another that only contains material contrary to the type. This propensity may help us to understand an entire range of human preferences, from Plato’s theory of form to modern stereotyping of race or gender.
Probability does pervade the universe—and in this sense, the old chestnut about baseball imitating life really has validity. The statistics of streaks and slumps, properly understood, do teach an important lesson about epistemology, and life in general. The history of a species, or any natural phenomenon that requires unbroken continuity in a world of trouble, works like a batting streak. All are games of a gambler playing with a limited stake against a house with infinite resources. The gambler must eventually go bust. His aim can only be to stick around as long as possible, to have some fun while he’s at it, and, if he happens to be a moral agent as well, to worry about staying the course with honor. The best of us will try to live by a few simple rules: do justly, love mercy, walk humbly with thy God, and never draw to an inside straight.