Research

Publications

Learning curve: Progress in the Replication Crisis

with Noam Angrist, Micheal Ainomugisha, Sai Pramod Bathena, Peter Bergman, Colin Crossley, Thato Letsomo, Moitshepi Matsheng, Rene Marlon Panti, Shwetlena Sabarwal & Tim Sullivan. AEA Papers and Proceedings, 113 : 482-88.

Abstract: We present detailed monitoring data across a five-country randomized trial of phone-based targeted tutoring—one of the largest multicountry replication efforts in education to date. We study an approach shown to work in Botswana and replicated in India, Kenya, Nepal, the Philippines, and Uganda. While the existing literature often finds diminishing effects as proof-of-concept studies are replicated and scaled, we find the opposite: implementation fidelity (the degree of targeted educational instruction) improves across replications and over time. This demonstrates that replication is not intractable; rather, equipped with mechanisms to learn from experience, organizational "learning curves" can enable effective replication and scale-up.

Method Matters: Underreporting of intimate partner violence. World Bank Economic Review, 2023.  

Abstract: This paper analyzes the magnitude and predictors of misreporting on intimate partner violence. Women in Nigeria were randomly assigned to answer questions using either an indirect method (list experiment) that gives respondents anonymity, or the standard, direct face-to-face method. Intimate partner violence rates were up to 35 percent greater when measured using the list method than the direct method. Misreporting was associated with indicators often targeted in empowerment and development programs, such as education and vulnerability. These results suggest that standard survey methods may generate significant underestimates of the prevalence of intimate partner violence, and biased correlations and treatment effect estimates. 

World Bank Working Paper version: Method Matters: Underreporting of Intimate Partner Violence in Nigeria and Rwanda

cited in: Violence against women is a scourge on poor countries. (2021, March 13). The Economist.

Working papers

Building Resilient Education Systems: Evidence from Large-Scale Randomized Trials in Five Countries 

with Noam Angrist, Micheal Ainomugisha, Sai Pramod Bathena, Peter Bergman, Colin Crossley, Thato Letsomo, Moitshepi Matsheng, Rene Marlon Panti, Shwetlena Sabarwal & Tim Sullivan

NBER Working paper here.

Education systems need to withstand frequent shocks, including conflict, disease, natural disasters, and climate events, all of which routinely close schools. During these emergencies, alternative models are needed to deliver education. However, rigorous evaluation of effective educational approaches in these settings is challenging and rare, especially across multiple countries. We present results from large-scale randomized trials evaluating the provision of education in emergency settings across five countries: India, Kenya, Nepal, Philippines, and Uganda. We test multiple scalable models of remote instruction for primary school children during COVID-19, which disrupted education for over 1 billion schoolchildren worldwide. Despite heterogeneous contexts, results show that the effectiveness of phone call tutorials can scale across contexts. We find consistently large and robust effect sizes on learning, with average effects of 0.30-0.35 standard deviations. These effects are highly cost-effective, delivering up to four years of high-quality instruction per $100 spent, ranking in the top percentile of education programs and policies. In a subset of trials, we randomized whether the intervention was provided by NGO instructors or government teachers. Results show similar effects, indicating scalability within government systems. These results reveal it is possible to strengthen the resilience of education systems, enabling education provision amidst disruptions, and to deliver cost-effective learning gains across contexts and with governments.

Discussed in: CSAE Research Podcasts: Under the Hood: Randomised Controlled Trials on Distance Education During COVID 19. (2021, December 16) + Follow up here; ABC News & News Radio; El Pais, JPAL blog; Nature Human Behaviour - Behind the paper summary; VoxEU and VoxDev summaries; Ideas for India

The Impacts of an Intimate Partner Violence Prevention Program: Experimental Evidence from Rwanda

with Arthur Alik-Lagrange, Muthoni Ngatia, and Julia Vaillant.

Abstract: One in three women worldwide experiences intimate partner violence (IPV) at some point in her lifetime, but little is known about how to prevent it. We study the impact of an IPV prevention program for couples in rural Rwanda using a randomized control trial. The program involved 22 weekly facilitated group discussions designed to improve couples’ communication skills, change gender attitudes, and spread a message of gender equality in the community. Randomizing at two levels– villages and couples– we assess the program’s effect on trained couples and on ‘spillover couples’– control couples in treated villages. Unlike previous studies but consistent with a backlash theory of violence, we find that the program caused large, unintended increases in IPV. Our results indicate that this represents a true increase in violence rather than increased reporting. Results suggest that violence may have increased because men backlashed against the perceived threat to their identity posed by program messages about women’s empowerment, and against their wives’ more progressive gender attitudes and greater aspirations for agency.

AEA Trial Registry: https://www.socialscienceregistry.org/trials/2282. 

Work in progress

Empowering Girls: A Randomized Trial on mHealth 'Safe Spaces' During COVID-19 

Bargaining power and backlash: experimental evidence on the impacts of a cash transfer and livelihoods program on intimate partner violence in Nigeria (with Paula Gonzalez-Martinez and Sreelakshmi Papineni). 

AEA Trial Registry: https://www.socialscienceregistry.org/trials/3015. 

Can role models and changing norms change gender attitudes? Evidence from a survey experiment in Rwanda.  

AEA Trial Registry: https://www.socialscienceregistry.org/trials/5869 

Rapid optimization as you scale: AB testing in in the social sector. 

Policy

Radhakrishnan, Karthika; Sabarwal, Shwetlena; Sharma, Uttam; Cullen, Claire; Crossley, Colin; Letsomo, Thato; Angrist, Noam. 2021. Remote Learning : Evidence from Nepal during COVID-19. World Bank, Washington, DC. 

Radhakrishnan, Karthika; Angrist, Noam; Bergman, Peter; Cullen, Claire; Matsheng, Moitshepi; Ramakrishnan, Anusha; Sabarwal, Shwetlena; Sharma, Uttam. 2021. Learning in the Time of COVID-19 : Insights from Nepal. World Bank, Washington, DC.  

Research and policy work underpinning the launch of Australia's Pacific Microstates Worker Pilot Program, summarised here: Center for Global Development. Northern Australia Worker Pilot Program (NAWPP). Migration Pathways. Washington, D.C.