Learning curve: Progress in the Replication Crisis
with Noam Angrist, Micheal Ainomugisha, Sai Pramod Bathena, Peter Bergman, Colin Crossley, Thato Letsomo, Moitshepi Matsheng, Rene Marlon Panti, Shwetlena Sabarwal & Tim Sullivan. AEA Papers and Proceedings, 2023., Vol.113, pp. 482-88.
Abstract: We present detailed monitoring data across a five-country randomized trial of phone-based targeted tutoring—one of the largest multicountry replication efforts in education to date. We study an approach shown to work in Botswana and replicated in India, Kenya, Nepal, the Philippines, and Uganda. While the existing literature often finds diminishing effects as proof-of-concept studies are replicated and scaled, we find the opposite: implementation fidelity (the degree of targeted educational instruction) improves across replications and over time. This demonstrates that replication is not intractable; rather, equipped with mechanisms to learn from experience, organizational "learning curves" can enable effective replication and scale-up.
Method Matters: Underreporting of intimate partner violence. World Bank Economic Review, 2023.
Abstract: This paper analyzes the magnitude and predictors of misreporting on intimate partner violence. Women in Nigeria were randomly assigned to answer questions using either an indirect method (list experiment) that gives respondents anonymity, or the standard, direct face-to-face method. Intimate partner violence rates were up to 35 percent greater when measured using the list method than the direct method. Misreporting was associated with indicators often targeted in empowerment and development programs, such as education and vulnerability. These results suggest that standard survey methods may generate significant underestimates of the prevalence of intimate partner violence, and biased correlations and treatment effect estimates.
World Bank Working Paper version: Method Matters: Underreporting of Intimate Partner Violence in Nigeria and Rwanda
cited in: Violence against women is a scourge on poor countries. (2021, March 13). The Economist.
Effectiveness of community mobilisation and group-based interventions for preventing intimate partner violence: A systematic review and meta-analysis. Journal of Global Health, 2024.
with Jessica Leight, Meghna Ranganathan, and Alexa Yakubovich.
This study systematically reviews and meta-analyzes randomized controlled trials (RCTs) of community-level or group-based interventions aimed at preventing IPV in lower- and middle-income countries. The research focuses on whether these gender-transformative interventions are more effective in reducing IPV compared to control groups with standard programming. The study included a systematic search up to July 20, 2021, and assessed outcomes related to IPV experiences and perpetration. Using multilevel random-effects meta-analysis, the study evaluates the pooled effect and explores how various study characteristics influence the results.
Building Resilient Education Systems: Evidence from Large-Scale Randomized Trials in Five Countries
with Noam Angrist, Micheal Ainomugisha, Sai Pramod Bathena, Peter Bergman, Colin Crossley, Thato Letsomo, Moitshepi Matsheng, Rene Marlon Panti, Shwetlena Sabarwal & Tim Sullivan.
NBER Working paper here. Revise and Resubmit: Nature.
Education systems need to withstand frequent shocks, including conflict, disease, natural disasters, and climate events, all of which routinely close schools. During these emergencies, alternative models are needed to deliver education. However, rigorous evaluation of effective educational approaches in these settings is challenging and rare, especially across multiple countries. We present results from large-scale randomized trials evaluating the provision of education in emergency settings across five countries: India, Kenya, Nepal, Philippines, and Uganda. We test multiple scalable models of remote instruction for primary school children during COVID-19, which disrupted education for over 1 billion schoolchildren worldwide. Despite heterogeneous contexts, results show that the effectiveness of phone call tutorials can scale across contexts. We find consistently large and robust effect sizes on learning, with average effects of 0.30-0.35 standard deviations. These effects are highly cost-effective, delivering up to four years of high-quality instruction per $100 spent, ranking in the top percentile of education programs and policies. In a subset of trials, we randomized whether the intervention was provided by NGO instructors or government teachers. Results show similar effects, indicating scalability within government systems. These results reveal it is possible to strengthen the resilience of education systems, enabling education provision amidst disruptions, and to deliver cost-effective learning gains across contexts and with governments.
Discussed in: CSAE Research Podcasts: Under the Hood: Randomised Controlled Trials on Distance Education During COVID 19. + Follow up here; ABC News & News Radio; El Pais, JPAL blog; Nature Human Behaviour - Behind the paper summary; VoxEU and VoxDev summaries; Ideas for India; Financial Times.
The Unintended Impacts of an Intimate Partner Violence Prevention Program: Experimental Evidence from Rwanda
with Arthur Alik-Lagrange, Muthoni Ngatia, and Julia Vaillant.
World Bank Policy Research Working Paper here.
Abstract: One in three women worldwide experiences intimate partner violence (IPV) at some point in her lifetime, but little is known about how to prevent it. We study the impact of an IPV prevention program for couples in rural Rwanda using a randomized control trial. The program involved 22 weekly facilitated group discussions designed to improve couples’ communication skills, change gender attitudes, and spread a message of gender equality in the community. Randomizing at two levels– villages and couples– we assess the program’s effect on trained couples and on ‘spillover couples’– control couples in treated villages. Unlike previous studies but consistent with a backlash theory of violence, we find that the program caused large, unintended increases in IPV. Our results indicate that this represents a true increase in violence rather than increased reporting. Results suggest that violence may have increased because men backlashed against the perceived threat to their identity posed by program messages about women’s empowerment, and against their wives’ more progressive gender attitudes and greater aspirations for agency.
Female empowerment and male backlash: Experimental evidence from India
with Sarthak Joshi, Joseph Vecci, and Julia Talbot-Jones.
Working paper here. Revise and Resubmit: JPE Micro.
Abstract: The unintended consequences of women’s empowerment are rarely measured and remain poorly understood. We study the impact of female empowerment on male backlash through a series of experiments involving 1,007 households in rural India. We find that men pay to punish empowered women at double the rate of women in an otherwise identical control group. We find that backlash occurs regardless of how women are empowered, with social image concerns being a key driver. Finally, we test several policies to reduce backlash and find that reframing empowerment programs to emphasize broader community benefits can help mitigate backlash.
Optimizing tutoring programs for scale: Evidence from a dozen randomized iterative A/B tests
with Noam Angrist, Colin Crossley, and Janica Magat.
Abstract: Over the course of 12 rapid randomized experiments, we optimize an educational tutoring program. Tutoring is one of the most effective educational approaches yet has remained difficult to scale due to high costs. We adaptively test and improve a technology-enabled tutoring program to enhance cost-effectiveness and scalability. Results show that seven of twelve tests led to efficiency improvements, ranging from 9 to 30 percent. Efficiency gains were driven by cost-reducing modifications that streamlined labor-intensive implementation processes and effectiveness-enhancing innovations that actively involved caregivers in their child's education. The largest efficiency gains came from caregiver engagement interventions, demonstrating that parental involvement can more than double program impact at minimal additional cost. We find that while practitioner priors often underestimate the potential of further program refinement, rigorous testing updates practitioner beliefs and facilitates more accurate predictions of `what works.' Our findings both reveal the returns to iterative testing in social programs and contribute new evidence on simple, cost-effective strategies to improve learning outcomes.
Conference presentation recording here at 28min.
Testing, one, two, three: A randomized trial of radio education relative to alternatives
with Noam Angrist, Peter Bergman, Konstantin Büchel, and Colin Crossley. Submitted.
Empowering Girls: A Randomized Trial on mHealth 'Safe Spaces' During COVID-19 - Drafting
Types of Tech: Experimental evidence on cheap, scalable education interventions - Drafting
Remote Education for Afghan Girls - Fieldwork. With Noam Angrist, Michael Callen, Oeindrila Dube, and Anaya Dam
Radhakrishnan, Karthika; Sabarwal, Shwetlena; Sharma, Uttam; Cullen, Claire; Crossley, Colin; Letsomo, Thato; Angrist, Noam. 2021. Remote Learning : Evidence from Nepal during COVID-19. World Bank, Washington, DC.
Radhakrishnan, Karthika; Angrist, Noam; Bergman, Peter; Cullen, Claire; Matsheng, Moitshepi; Ramakrishnan, Anusha; Sabarwal, Shwetlena; Sharma, Uttam. 2021. Learning in the Time of COVID-19 : Insights from Nepal. World Bank, Washington, DC.
Research and policy work underpinning the launch of Australia's Pacific Microstates Worker Pilot Program, summarised here: Center for Global Development. Northern Australia Worker Pilot Program (NAWPP). Migration Pathways. Washington, D.C.