Risky Choices
How Unknowns undermine Individual Decisions
while Existential Risk challenges Social Policy
Overview:
I live in Southern California where it rarely rains. I don’t own a raincoat, let alone an umbrella. But if I did, it would be foolhardy to schlep the umbrella around when I go outside. The likelihood of rain is too low to make the inconvenience of bringing it along worthwhile. In rainier climes it might be a different story, and it is a story that is often used to illustrate the concept of expected utility as a way of analyzing choice under uncertainty where risk is in play. As long as we have complete knowledge of our choices, and our preferences about them, this standard model works beautifully at both the individual and social level, even when uncertainty comes into the picture about the likelihood of outcomes between which we are choosing.That's true about everyday choices with a limited time horizon. But the more the choices we face take us into the unfamiliar, about either our preferences or our knowledge, the harder it is to apply the standard model. Likewise, the further we go out in time, the harder it is to make rational decisions and the less well-equipped our standard model becomes. In the case of the individual, time opens up the chance for changes of preferences which are hard to predict. In the case of the social, the further we go out in time the larger the size of the class of people who have yet to be born whose interests are at stake and affected by the decisions we make. In both cases, time forces us to confront choices in which the facts are not only not known to us, but as yet not even set themselves. In this book, I aim to show how the inadequacy of the standard model in these settings plays out in different ways in the arena of the individual and the social, and does so for different reasons. And, as a result of these differences, I argue for different alternatives to the standard model for each of these arenas. When it comes to individuals, instead of thinking about risk as something that is about alternatives we have to choose between, we would do better to think of it as a matter of a psychological stance toward risk itself - what I call naked risk. When it comes to the social, I argue that when our choices implicate the well-being of future generations, instead of accepting risk in exchange for the chance of increased benefits, we would do better to be intolerant of risk altogether whenever possible – a position I call absolutism.
Table of contents:
Chapter One: Individuals
Chapter Two: The Social
Chapter Three: Absolutism
Chapter Four: Trinity
Chapter Five: Climate
Chapter Six: Genes
Chapter Seven: Artificial Intelligence
Chapter Eight: Complications
Chapter Nine: Individuals Redux
Appendix
The argument of the book in 25 steps:
1. Expected utility theory works well for short-term choices.
2. Yet the most significant choices, be they individual or social, implicate the long-term.
3. For the individual, long-term choices of risk are fraught with uncertainty of two kinds that are both dynamical - one is uncertainty about how feelings about a choice may change after the fact and the other is uncertainty about how facts underlying that choice may change after the fact.
4. For social choices, the hazard of the long-term creates a problem for a different reason. The further we go out in time, the greater the number of people whose lives are at stake.
5. At least they are assuming we take the well-being of future generations into account.
6. Why expected utility is the wrong approach when existential risk is at stake.
7. Avoiding existential risk is a moral imperative – we, in the here and now, have no right to jeopardize the lives of those who may follow us.
8. Absolutism when it comes to avoiding existential risk is defended.
9. This view is differentiated from maximin utility approaches and the precautionary principle as well as The Dismal Theorem.
10. Why it is insulated from Pascal’s mugging.
11. Teller’s Trinity Test account of existential risk is shown to be inconsistent on theoretical grounds before the test took place.
12. Absolutism is applied to some of the most prominent alleged cases of existential risk today to show that in each case the account of how extinction is possible, either does not hold or can be blocked for a small price.
13. In the case of Hansen’s Venus scenario, the account of existential risk from runaway global warming was consistent with our best theories when he proposed it but later shown to be inconsistent as our theories developed.
14. In the case of CRISPR germline research, risk can be controlled in the laboratory, and once released into the gene pool, the risk is no different than that associated with the natural rate of mutation.
15. In the case of Artificial Intelligence, assuming the lowest bar of the requirements a system would have to have to constitute a danger to us, such systems would have to be limited in their ability to autonomously form goals de novo.
16. What we would give up in the power of AI and its usefulness if we limited systems to prevent them from meeting those requirements.
17. Even if climate change, genetic engineering and artificial intelligence are the only prima facie candidates for existential risk today, there is nothing to say there won’t be others in the future.
18. Moreover, a world of rogue players, be they states or individuals, makes for a much darker picture than the argument so far.
19. How trade theory and partial compliance game theory show how management of rogue state actors is plausible.
20. On the other hand, individual rogue actors present much more of a challenge, particularly in the area of genetic engineering.
21. Obligations to future generations when existential risk is not in play.
22. Existential risk may be avoidable for us collectively in a way that it is not to us as individuals.
23. How then should we make rational choices involving risk that reach beyond the short-term that are subject to dynamical uncertainty?
24. Why this is the wrong question. Instead of thinking of risk as a feature of choice, we should think of it as a psychological feature of our lives.
25. How that account challenges us to examine our tolerance for uncertainty in the face of the unknown.
Chapter Outline:
Chapter One: Individuals
Faced with a choice, expected utility theory directs us to choose the alternative which maximizes the product of its likelihood and the utility we attach to it. But consider the risk-reward calculus a woman faces the choice of whether or not to have a prophylactic bi-lateral mastectomy. She has to place a valuation in the here and now on the gain in the chance of survival versus the loss of her breasts. She has to make a judgment about how much she might regret opting for surgery after the fact. But in doing so she has to cope with uncertainty about how her valuation of her choice may change if she opts for surgery (following Paul). But if that is not hard enough, she also has to make that choice with uncertainty about the likelihood of scientific research that might offer less drastic ways of lowering her risk in the near future, or new treatments that might increase the odds of achieving long term remission after cancer. As such, judgments of risk are fraught with uncertainty of two kinds that are both dynamical - one is uncertainty about how feelings about a choice may change after the fact and the other is uncertainty about how facts underlying that choice may change after the fact. And while the first of these may be few and far between, the second is pervasive once the horizon of rational choice extends beyond the short-term, like whether to take an umbrella on a hike. But, of course, it is in the domain of the long-term not the short-term that making the right choices usually matter the most.
Chapter Two: The Social
The element of uncertainty that plagues individual decisions does not infect the domain of social choice because, in making such social choices, we strip individuals of their individuating features – be they a matter of facts or preferences – and use “everyman” stand ins. Still, like the domain of the individual, in the case of the social, time is our enemy, all be it for different reasons. In the case of the individual, the further we go out in time, the less we know. In the case of the social, the further we go out in time, the larger the number of lives at stake. At least it is if we take into consideration the lives of those not yet born. Suppose we destroy the planet. Then there will be no lives of those not yet born. This gives rise to what I call the extinction paradox. We do wrong if we leave a planet largely destroyed for future generations but not if we destroy it so much that there are no future generations. I argue that to avoid this paradox we have to swallow hard (at least, if you are a philosopher) and allow that we have a duty to possible people. But once we allow that, I argue that we should never trade the risk of human extinction for a benefit, however small the risk. I call this view absolutism when it comes to avoiding existential risk.
Chapter Three: Absolutism
The defense of absolutism depends on a plausible account of the notion of possibility. I argue it can’t be the layman’s notion – “surely anything is possible”, rather I defend the idea that possibility is relative to our favored theory of the world as captured by the physical laws we take to govern it. The objection to absolutism is the idea that, even relative to our laws, there will always be some risk of extinction however small. Hence, we face an infinite array of risks which absolutism dictates we should avoid. In the face of this objection, I recast the argument in terms of choice. Faced with a choice between two courses of action, one of which has a higher risk of extinction than the other, always choose the second over the first. I go on to contrast my account to both the Precautionary Principle and Maximin utility accounts as well as Pascal’s Mugging (contra Bostrom). In doing so there are two central features of the account: probability is cashed in subjectively, and absolutism is based on moral grounds unrelated to utility. To make the case for this approach I defend a burden of proof argument: to assert a risk of extinction for a course of action that is higher in risk than another course of action you must provide an account consistent with our favored theory of the world. The four chapters that follow illustrate this strategy by examining some of the most prominent alleged cases of existential risk today. In each case I argue how the demand that there be an account of how extinction is possible, consistent with our favored theory of the world, either does not hold or can be avoided for a small price (contra Ord).
Chapter Four: Trinity
Existential risk has its origins in the Trinity test – the test of the first nuclear bomb. Prior to the test, Edward Teller, one of the team developing it, argued there was a possibility that the explosion might set off a fusion reaction burning up all of the atmosphere. On the standard account (which is the central claim of the movie, Oppenheimer) the test proceeds despite the risk, given the priority of defeating the Nazis. I show that, to the contrary, a committee set up by Oppenheimer before the test had shown that Teller’s scenario was theoretically impossible relative to the laws of physics – the rate of diffusion in the atmosphere was too great for an atomic explosion to be sustained in a fusion reaction.
Chapter Five: Climate
In 2005, James Hansen et al. used climate modeling to argue for the existence of amplifying feedbacks that could realize runaway scenarios in which the planet stabilizes in a (cold) state akin to Mars at -50oC or a (hot) state akin to Venus at +450oC. In the Mars syndrome, increasing surface albedo (reflective power) as the planet cools is the amplifying force. In the Venus syndrome, increasing water vapor as the planet warms drives things. But there is a big difference between these two scenarios. Runaway cooling has happened before (most recently 640 million years ago) but is subject to change as weathering of sedimentary rocks and rising atmospheric CO2 reverse the process. However, Hansen argued that there is no such reversing process in the case of runaway heating, hence the potential for mass permanent extinction. Hansen went on to argue that there was a non-zero chance that the Venus Scenario could be initiated on Earth once the parts per million of atmospheric CO2 reached about 950 ppm which was the level projected by the IPCC in its business-as-usual scenario for the end of this century. Today we know that this argument is in fact incorrect, as Hansen himself admitted in 2013. In fact, once surface temperatures reach 1127oC, radiation is emitted in the range of long wavelengths which pass through water vapor unhindered and into space. The climate case illustrates the idea that existential risk is relative to our laws, and as they change, so does what counts as an existential risk.
Chapter Six: Genes
Changes to somatic cells can affect the fitness of host organisms, but these changes themselves are not passed on to their offspring, unlike changes to the germ line. So, while making somatic changes carries risk, it is only a risk for that generation. That contrast is what led the Federal ban funding of germline editing in the United States. It is also what led to community outrage among researchers when He Jiankui announced that he had modified the CCR5 genes in the germline of twin babies in 2018. Prior to the He affair, some researchers had already called for a voluntary moratorium on germline interventions (except for research use). But, in light of his work, a new call was put out for government bans on such research. That said, these calls were couched in terms of the need for a “pause” followed by the promulgation of guidelines to ensure, among other things, that the benefits outweigh the risks. But here is the rub – even with a pause and the relative rarity of off-target effects, how can we be sure that such specific interventions might not “… inadvertently target other loci in the genome and that such unanticipated genetic manipulations could alter biological functions in problematic ways.” (Jeremy Sugarman, “Ethics and germline gene editing,” EMBO Reports 16, no. 8 (2015): 879–880.) The incidence of off-target mutations may be low, and may become vanishingly small over time, but it is unlikely to be totally eliminated. But if and when they do occur, the most problematic would be an unintended mutation in the germ line. However, protocols are available to isolate and test for the occurrence of such outcomes, thereby eliminating any risk of diffusion in the gene pool by conducting interventions ex-vivo (outside the body) and then resequencing the altered DNA. What if the moratorium on making changes in germ cells ends? How can we be sure that further mutations in germ cells won’t occur with unanticipated results even if the initial, deliberately induced changes are carefully studied in laboratory settings before being inserted into the germ line? We can’t. This is the stuff of popular nightmares about genetic engineering gone rogue in our gene pool. But the risk associated with those follow on mutations will be no different from the risk of naturally occurring mutations in general, primarily involving small changes in a few base pairs. As such, I argue, the existential risk posed by genetic engineering of the germline is no greater than the risk from natural mutation once it is introduced outside the laboratory.
Chapter Seven: Artificial Intelligence
The assessment of the existential risk posed by artificial intelligence is extremely challenging because of competing models of the requirements for a system to have the capacities to act on its own and act against our interests. I argue that the idea that such systems can be imbued with our values or ethics to ensure they our aligned with our interests is philosophically naïve. For purposes of argument, I assume the lowest bar of the requirements a system would have to have to constitute a danger to us. Such systems would have to be limited in their ability to autonomously form goals de novo. From there I outline what we would give up in the power of AI and its usefulness if we limited systems to prevent them meeting those requirements. I argue the price we pay is relatively low, we still have most of what we want. In a philosophical coda, I warn that limiting the capacity for autonomous goal formation might not limit the capacity for such systems to express emergent sentience, which raises the specter of a new class of slaves.
Chapter Eight: Complications
All of the foregoing has been in the realm of the ideal. It assumes we act collaboratively in advancing our collective well-being while avoiding existential risk. I make no claim that climate, genetic engineering and artificial intelligence are the only prima facie candidates for existential risk. Even if there are no others in play now, there is nothing to say there won’t be others in the future. But what happens when we introduce bad (human) actors into the picture? I address this in this chapter by distinguishing between bad state actors and bad individual actors. And I do so by contrasting the real-world management of existential risk for each of the areas discussed in the previous four chapters. Regulation of rogue states is obviously much easier than regulation of individual actors and I illustrate this with how states have successfully enforced compliance using trade sanctions. On the other hand, regulation of individual bad actors is much harder and dependent on the willingness of states to police their citizens. Using partial compliance game theory, I show how the problem of regulation cuts quite differently in the case of climate as compared to AI and genetic engineering. And while the cost of entry for individuals when it comes to AI is prohibitively high, that is not true of genetic engineering. However, the same cannot be said of corporations. That has led some to analogize AI defense to the need for nuclear mutually assured destruction. I explain why this is a bad idea – nuclear bombs cannot combine to target us, while AI systems of suitable sophistication might be able and motivated to do just that. In the final section of the chapter I extend the account of obligations to future generations when existential risk is not in play.
Chapter Nine: Individuals Redux
If we are lucky, as a species, we will survive until the sun burns out and maybe beyond, if we send explorers to other solar systems. But as individuals, extinction is our lot. In public policy, most risk-benefit analysis of the social is limited to 100 years. I argue this has all the virtues of theft over honest toil (see Bertrand Russell). I argue the same is true of individual choice when it is based only on short term considerations when what matters most is choice over the long term. What is the alternative? In the face of all of the unknowns when it comes to the long-term, I argue in favor of forsaking risk as an element of rational choice in favor of embracing it as a psychological construct. What I call “naked risk”. Naked risk is a matter of your stance toward the unknown. A choice you have about how you stand relative to the future. What if you want to embrace risk about the future but instead feel dread? (This is where the book panders to the self-improvement market in the footsteps of Agnes Callard on whose work I draw!) Faced with the unknown, three people are each filled with dread. One embraces the idea of the unknown except when actually faced with it. The second stays loyal to his enthusiasm for the unknown with equanimity but in the face of it, dread overwhelms him. The third disvalues the idea of the unknown and experiences dread in the face of it but wishes he did neither. Each wants to become a person they are not. But merely changing their preferences will not be enough. Each has to change their affect in the face of the unknown. But if they can’t do that by way of changing their preferences, how are they to do so? Many years ago, I taught an undergraduate seminar on the topic of disgust. An assignment for the semester was for each student to identify an object of disgust and see if they could extinguish that association during the semester. It turns out that whether disgust can be extinguished behaviorally or not varies from object to object. For some objects, like putting dead insects in your mouth, it can be. For others, like the smell of rotting meat, it can’t be. What about dread of the unknown then? Is the unknown more like insects or rotting meat? This is not a question that needs to be answered a priori. Experimental data will yield an answer as to how variable it is across different populations, how difficult it is to extinguish, and, if it can be, by what means.