Milkman, K.L., Ellis, S.F., Gromet, D.M., Luscher, A.S., Mobarak, R.S., Paxson, M.K., Silvera Zumaran, R.A., Kuan, R., Berman, R., Lewis, N.A., List, J.A., Patel, M.S., Van den Bulte, C., Volpp, K.G., Beauvais, M.V., Bellows, J.K., Marandola, C.A., and Duckworth, A.L. 2024. “Megastudy shows that reminders boost vaccination but adding free rides does not.” Nature. 631, 179-188.
Encouraging routine COVID-19 vaccinations is likely to be a crucial policy challenge for decades to come. To avert hundreds of thousands of unnecessary hospitalizations and deaths, adoption will need to be higher than it was in the autumn of 2022 or 2023, when less than one-fifth of Americans received booster vaccines. One approach to encouraging vaccination is to eliminate the friction of transportation hurdles. Previous research has shown that friction can hinder follow-through and that individuals who live farther from COVID-19 vaccination sites are less likely to get vaccinated. However, the value of providing free round-trip transportation to vaccination sites is unknown. Here we show that offering people free round-trip Lyft rides to pharmacies has no benefit over and above sending them behaviourally informed text messages reminding them to get vaccinated. We determined this by running a megastudy with millions of CVS Pharmacy patients in the United States testing the effects of (1) free round-trip Lyft rides to CVS Pharmacies for vaccination appointments and (2) seven different sets of behaviourally informed vaccine reminder messages. Our results suggest that offering previously vaccinated individuals free rides to vaccination sites is not a good investment in the United States, contrary to the high expectations of both expert and lay forecasters. Instead, people in the United States should be sent behaviourally informed COVID-19 vaccination reminders, which increased the 30-day COVID-19 booster uptake by 21% (1.05 percentage points) and spilled over to increase 30-day influenza vaccinations by 8% (0.34 percentage points) in our megastudy. More rigorous testing of interventions to promote vaccination is needed to ensure that evidence-based solutions are deployed widely and that ineffective but intuitively appealing tools are discontinued.
Ellis, S.F., D. Ganguly, M. Kecinski, and K.D. Messer. 2023. “Back to the Source: Consumers Response to Produce Irrigated with Different Sources of Recycled Water.” Water Resources Research. 59(7).
Using recycled water to irrigate agricultural products can be an effective solution to water scarcity and security. However, a better understanding of how society values different sources of recycled water provides insights into potential demand-side barriers to adoption of these solutions. This paper implements a framed field experiment conducted in the Southwest and Mid-Atlantic regions of the United States that evaluates consumers' willingness-to-pay (WTP) for three sources of recycled irrigation water: “gray,” “black,” and “produced.” Our analysis indicates that people consider certain sources of recycled water more acceptable for irrigating produce than others. Recycled gray water is preferred to recycled produced water, and both are preferred to recycled black water. We also explore how people respond to scientific information about the benefits and risks of using recycled irrigation water and found no evidence to support that this information changes people's behaviors.
Ellis, S.F., O.M. Savchenko, and K.D. Messer. 2023. “Is a non‑representative convenience sample of adults good enough? Insights from an economic experiment” Journal of the Economic Science Association.
Recruitment of representative and generalizable adult samples is a major challenge for researchers conducting economic field experiments. Limited access to representative samples or the high cost of obtaining them often leads to the recruitment of non-representative convenience samples. This research compares the findings from two field experiments involving 860 adults: one from a non-representative in-person convenience sample and one from a representative online counterpart. We find no meaningful differences in the key behaviors of interest between the two samples. These findings contribute to a growing body of literature demonstrating that non-representative convenience samples can be sufficient in certain contexts.
Ellis, S.F., O.M. Savchenko, and K.D. Messer. 2022 “Mitigating Stigma Associated with Recycled Water.” American Journal of Agricultural Economics. 104(3): 1077-1099.
Stigmatization of water and food products can constrain markets and prevent the implementation of scientifically safe solutions to environmental problems, such as water scarcity. Recycled water can be a cost-effective, dependable, and safe solution to water shortages. However, consumers generally either require a large reduction in price to purchase products made with recycled water or reject such products outright. If emerging sustainable agricultural technologies, such as recycled water, are to be used to address growing water shortages worldwide, policymakers, water managers, and industry stakeholders must identify effective strategies for mitigating the stigma associated with recycled water. Using field experiments involving 1,420 adult participants, we test the effectiveness of two stigma-mitigating techniques. We also demonstrate a novel twist to the collection of representative samples in non-hypothetical field experimental settings and then compare the results to a more traditional field experiment that recruited participants at large public gatherings. The analysis of these two different samples suggests a common finding: passing recycled water through a natural barrier, such as an aquifer, removes the stigma consumers would otherwise attach to it. We also find that the trophic level an organism occupies in the food chain influences stigmatizing behavior. The greater the steps in the food chain between an organism and the use of recycled water, the less it is stigmatized by consumers. These results have important implications for efforts to promote large-scale potable and non-potable water recycling projects and the use of recycled water in the agricultural industry.
Milkman, K.L, L. Ghandhi, S.F. Ellis, H.N. Graci, D.M Gromet, R.S. Mobarak, A.M. Buttenheim, A.L. Duckworth, D.G. Pope, A. Stanford, R.H. Thaler, and K.G. Volpp. 2022. "A City-Wide Experiment Testing the Impact of Geographically-Targeted, High-Payoff Vaccine Lotteries." Nature Human Behaviour. 6: 1515-1524.
Lotteries have been shown to motivate behavior change in many settings, but their value as a policy tool is relatively untested. We implemented a pre-registered, city-wide experiment to test the effects of three, high-payoff, geographically-targeted lotteries designed to motivate adult Philadelphians to get their COVID-19 vaccine. In each drawing, residents of a randomly selected “treatment” zip code received half the lottery prizes, boosting their chances of winning to 50-100x those of other Philadelphians. The first treated zip code, which drew considerable media attention, may have experienced a small bump in vaccinations compared to control zip codes: average weekly vaccinations rose by an estimated 61 per 100,000 people per week (+11%). Pooling results from all three zip codes treated during our six-week experiment, however, we do not detect any evidence of any overall benefits. Further, our 95% confidence interval provides a 9% upper bound on the net benefits of treatment in our study.
Ellis, S.F., M. Kecinski, K.D. Messer, and C. Lipchin. 2021. “Consumer Perceptions After Long Term Use of Alternative Irrigation Water: A Field Experiment in Israel.” Applied Economic Perspectives and Policy. 44(2): 1003-1020.
This study provides the first revealed preference estimates of Israeli’s willingness-to-pay for produce irrigated with alternative water. Results show that Israeli’s prefer produce irrigated with conventional water over any type of alternative water, and that preferences for alternative water varies by type. These results indicate there may be long-run limits to how high consumer demand for alternative water can rise. Policymakers should be cognizant of these findings and gradually expose the public to the unavoidable future of widespread alternative water use. Increased public awareness of water scarcity and alternative water technology will encourage adoption when it is possible and necessary.
Ellis, S.F., M. Masters, K.D. Messer, C. Weigel, and P.J. Ferraro. 2020. “The Problem of Feral Hogs and the Challenges of Providing a Weak-link Public Good.” Applied Economic Perspectives and Policy. 43(3): 985-1002.
Feral hogs are an invasive species in the United States annually inflicting $1.5 billion in damages. Although interest in mitigating this grows, control spans properties and jurisdictions, creating challenges. Because landowners and local governments cannot capture the benefits of control that accrue beyond their boundaries, they generally underinvest. Worse, the weakest contributors constrain efforts. To offer insights into this problem, we synthesized studies on best management practices, invasive species, and weak-link public goods. We also provide estimates of farmers’ willingness-to-pay for control efforts, finding they require financial assistance. Together, this suggests that more investment is necessary to manage feral hogs.
Ellis, S.F., O.M. Savchenko, and K.D. Messer. 2019. “What's in a Name? Branding Reclaimed Water.” Environmental Research. 172: 384-393.
Reclaimed water is often presented as a cost-effective, reliable, and safe solution to increasingly common water shortages in the United States and across the globe, but studies have shown that consumers tend to object to the use of this water. Broad adoption of this technology will require consumer acceptance or at least tolerance of it, and studies have suggested that better branding could minimize consumers' concerns. In this study, we first test twenty-one potential branding names for reclaimed water using survey responses to identify the top-six most favored names. Second, we test whether an opportunity for consumers to drink actual reclaimed water changes their preferences. The results suggest that the common names for this water, such as Recycled, Reclaimed, Nontraditional, Treated Wastewater, and Reused, are the least appealing, as they all scored at the bottom. In contrast, names that invoke desirable characteristics of the water—Pure, Eco-Friendly, and Advanced Purified, were viewed significantly more favorable than the others. Having an opportunity to taste this water seems to clarify consumers' preferences by increasing the differences in favorability between the names. Based on these results, it appears that while there are a couple of appealing names, the most consistently preferred is Pure Water.
Ellis, S.F., J.R. Fooks, K.D. Messer, and M.J. Miller. 2016. "The Effects of Climate Change Information on Charitable Giving for Water Quality Protection: A Field Experiment." Agricultural and Resource Economics Review. 45(2): 319-337.
This study uses a field experiment involving 251 adult participants to determine which messages related to climate change, extreme weather events, and decaying infrastructure are most effective in encouraging people to pay more for investments that could alleviate future water-quality risks. The experiment also assesses whether people prefer the investments to be directed toward gray or green infrastructure projects. Messages about global warming induced climate change and decaying infrastructure lead to larger contributions than messages about extreme weather events. The results suggest that people are likely to pay more for green infrastructure projects than for gray infrastructure projects.
Gromet, D.M., S.F. Ellis, J.S. Kay, and H.N. Graci. 2023. “The Megastudy Approach for Changing Behavior at Scale.” in Michael Sanders, Syon Bhanot, and Shibeal O’Flaherty (editors), Behavioral Public Policy in a Global Context: Practical Lessons from Outside the Nudge Unit. Palgrave Macmillan Cham.
The Behavior Change for Good (BCFG) Initiative was founded in 2016 to accelerate scientific discovery and identify the best behavioral science strategies to help people make meaningful and lasting changes to improve their lives. There have been many important insights from behavioral science, yet we still know very little about which insights have the best chance at cost-effectively changing behavior in any given situation. This chapter reviews BCFG’s novel, collaborative approach to combat these issues in behavioral science—the “megastudy.” A megastudy is a large field experiment in which many sub-studies are tested simultaneously on the same objective outcome, allowing for apples-to-apples (and dollars-to-dollars) comparisons across many different interventions developed from multiple disciplines. The authors review the benefits and drawback of the megastudy approach, explain how BCFG conducts megastudies, provide an in-depth look into BCFG’s 54-intervention megastudy on encouraging exercise with a large U.S. gym chain, and share the lessons learned for implementing this new approach to conduct behavioral science at scale.
Ellis, S.F., P. Ferraro, and K.D. Messer. 2020. “The Feral Hog Problem: Can a Cost-Share Program Help?” Center for Behavioral and Experimental Agri-Environmental Research. CBEAR Behavioral Insight Brief. No. 9.
Trapping systems can be an effective method for managing feral hogs, but they are expensive. Would financial assistance through a cost-share program increase their adoption?
Cochrane, S., S. Koropeckyj, A. Smith, and S. Ellis. 2014. “Central Cities and Metropolitan Areas: Manufacturing and Nonmanufacturing Employment as Drivers of Growth,” in Susan Wachter and Kimberly A. Zeuli (editors), Revitalizing America’s Cities. Philadelphia: University of Pennsylvania Press.
Using a broad approach, this chapter examines trends in employment and population growth in cities and suburbs of the nation’s metro areas, focusing on the historical deindustrialization and the loss of manufacturing jobs in cities. Throughout the twentieth century and into the first decade of the twenty-first, most U.S. economic growth has taken place in metropolitan areas; this has been the case since 1970, for both employment and population growth. The aggregate of the central cities has not enjoyed growth in manufacturing employment since the 1970's. The most recent data, however, show a break in these long-term historical trends, with central city population growth rate 1.0 percent greater than the 0.8 percent growth rate in the suburbs for 2010-2011. Nonmanufacturing employment growth in central cities in the aggregate has also, for the first time in the data, outpaced that of the suburbs in the same time period. Furthermore, more than half of the central cities expanded their manufacturing bases between the second half of 2009 through the fourth quarter of 2011. This may be an outcome of the recovery from the preceding bubble and bust. While it is too soon to call this a trend, cities throughout the country have recovered from decline and are growing; decline is not inevitable.
Milkman, K.L., S.F. Ellis, D.M. Gromet, I.M. DeMay, H.N. Graci, Y. Jung, R.S. Mobarak, R.A. Silvera Zumaran, M.N. Simmons, C. Van den Bulte, S. Benartzi, M. Hilchey, L. Goodyear, D. Karlan, N. Mazar, D. Mochon, A.M. Shah, D. Soman, J. Zinman, A.L. Duckworth. “Can Reminder Emails Compel Americans to Save? A Two-Million Person Megastudy.” Revise and Resubmit at PNAS Nexus.
In the United States, increasing savings is a common goal. We present a 1,925,785 person megastudy testing seven different email campaigns encouraging one-time and recurring savings deposits to increase savings balances. Our campaigns increased one-time savings deposits, on average, by 0.05 percentage points (or 0.51%). The best-performing campaign sent weekly savings reminders to people who hadn’t saved in the last week but congratulatory messages to those who had successfully saved. This increased the monthly likelihood a customer made a one-time savings deposit by 1.32%, and in one model, increased monthly savings by 47.07%. If such light-touch email nudges can generate benefits, it suggests behaviorally-informed interventions should be far more widely deployed by financial institutions.
Ellis, S.F., M. Kecinski, K.D. Messer, and J.L. Lusk. “Feelings Vs. Behavior – A Neuroeconomic Perspective on Food and Water Stigma.”
Dealing with large-scale societal problems, such as water scarcity, often requires changes in behavior that consumers resist. Some sustainable, cost-effective, and safe solutions are even rejected because of a psychological response of disgust, such as food produced with recycled water to supplement traditional water supplies and crickets as a replacement for water-intensive proteins like beef. Adding the “right” positive elements to a stigmatized item has been shown to mitigate disgust. However, this can be difficult and expensive to do as it requires the stigmatized object to go through a process that restores its “natural” and “pure” state. This study combines fMRI and a choice experiment to test if a behavioral intervention that emphasizes the existing, positive dimensions of a stigmatized object can lessen the weight of disgust in the decision process, mitigating its effects. Results suggest that it can. A video that promotes recycled water was shown to ameliorate consumers’ aversion to food produced with it. However, consumers’ greater acceptance of recycled irrigation water came not from overcoming whatever psychological reactions of disgust they experienced but from reweighting its importance in their decisions. These findings are relevant to many real-world scenarios where stigma-driven behavior produces inefficiencies, such as farm-level adoption of recycled irrigation – a safe, cost-effective and environmentally sound solution to water shortages.
Kuan, R.A., K.L. Milkman, S.F. Ellis, D.M. Gromet, Y. Jung, M.K. Paxson, R.A Silvera Zumaran, R. Berman, E. Dimant, C. Van den Bulte, M.V. Beauvais, J.K. Bellows, C.A. Marandola, S.K. Nelson Jr., A.L. Duckworth. “Do planning prompts suggesting a default plan increase follow-through? Two 1-million-person field experiments.”
Sean F. Ellis, Ph.D.
3720 Walnut St., Solomon Labs, Philadelphia, PA 19104
sfellis@wharton.upenn.edu | LinkedIn | GoogleScholar
Updated March 2025