Publications

NaijaHate: Evaluating Hate Speech Detection on Nigerian Twitter Using Representative Data

Manuel Tonneau, Pedro Vitor Quinta de Castro, Karim Lasri, Ibrahim Farouq, Lakshminarayanan Subramanian, Victor Orozco-Olvera, Samuel Fraiberger

To address the global issue of hateful content proliferating in online platforms, hate speech detection (HSD) models are typically developed on datasets collected in the United States, thereby failing to generalize to English dialects from the Majority World. Furthermore, HSD models are often evaluated on curated samples, raising concerns about overestimating model performance in real-world settings. In this work, we introduce NaijaHate, the first dataset annotated for HSD which contains a representative sample of Nigerian tweets. We demonstrate that HSD evaluated on biased datasets traditionally used in the literature largely overestimates real-world performance on representative data. We also propose NaijaXLM-T, a pretrained model tailored to the Nigerian Twitter context, and establish the key role played by domain-adaptive pretraining and finetuning in maximizing HSD performance. Finally, we show that in this context, a human-in-the-loop approach to content moderation where humans review 1% of Nigerian tweets flagged as hateful would enable to moderate 60% of all hateful content. Taken together, these results pave the way towards robust HSD systems and a better protection of social media users from hateful content in low-resource settings.


Media and Health Outcomes 

Víctor Orozco-Olvera and Niyati Malhotra

This chapter takes stock of the research that studies the relationship between mass media campaigns and behavioral outcomes that have an impact on health. It first discusses the theoretical frameworks underpinning health-focused entertainment media and social and behavior change communications (SBCCs), followed by recent natural experiments that exploit new datasets and statistical approaches for rigorously evaluating the causal effect of “real-world” media on health outcomes. This discussion covers both the intentional use of media through entertainment education, or “edutainment,” and also the varied effects of commercial media consumption on health. The chapter then discusses the literature linking traditional and new media with health by considering reviews and high-quality evaluations of SBCC media campaigns targeting sexual and reproductive health, child survival, violence against women, non-communicable diseases – in particular those associated with tobacco consumption and other substances – and healthcare-seeking and health-preserving behaviors, particularly in the context of the COVID-19 pandemic. The chapter concludes with a discussion highlighting the main findings and existing knowledge gaps of the literature and a discussion of new media for improving health outcomes at scale.


Large-Scale Demographic Inference of Social Media Users in a Low-Resource Scenario

Karim Lasri, Manuel Tonneau, Haaya Naushan, Niyati Malhotra, Ibrahim Farouq, Víctor Orozco-Olvera, and Samuel Fraiberger

Characterizing the demographics of social media users enables a diversity of applications, from better targeting of policy interventions to the derivation of representative population estimates of social phenomena. Achieving high performance with supervised learning, however, can be challenging as labeled data is often scarce. Alternatively, rule-based matching strategies provide well-grounded information but only offer partial coverage over users. It is unclear, therefore, what features and models are best suited to maximize coverage over a large set of users while maintaining high performance. In this paper, we develop a cost-effective strategy for large-scale demographic inference by relying on minimal labeling efforts. We combine a name-matching strategy with graph-based methods to map the demographics of 1.8 million Nigerian Twitter users. Specifically, we compare a purely graph-based propagation model, namely Label Propagation (LP), with Graph Convolutional Networks (GCN), a graph model that also incorporates node features based on user content. We find that both models largely outperform supervised learning approaches based purely on user content that lack graph information. Notably, we find that LP achieves comparable performance to the state-of-the-art GCN while providing greater interpretability at a lower computing cost. Moreover, performance does not significantly improve with the addition of user-specific features, such as textual representations of user tweets and user geolocation. Leveraging our data collection effort, we describe the demographic composition of Nigerian Twitter finding that it is a highly non-uniform sample of the general Nigerian population.


Improving Enrollment and Learning Through Videos and Mobiles: Experimental Evidence from Northern Nigeria 

Victor Orozco-Olvera and Ericka Rascón-Ramirez 

In northern Nigeria, half of primary school-age children attend school, half of girls are married before turning 15, and one in five people can read a whole sentence. Conducted in rural, low literate communities governed by traditional norms, this paper presents the results of a cluster randomized controlled trial that tested community screenings to reshape parental aspirations and attitudes toward education, and as a reinforcing arm, the distribution of mobiles with engaging apps to teach 6-9-year-old children to read. Twelve months after the screenings, children were 42 percent less likely to be out of school, but as expected, their learning levels did not improve. In the communities that were provided the mobile reinforcer, literacy and numeracy skills increased by 0.46 and 0.63 standard deviation, respectively. The impacts of the combined intervention on school attendance and learning gains were similar for boys and girls. For non-targeted older siblings, the intervention increased learning by 0.34 and 0.47 standard deviation and reduced the likelihood of teenage pregnancy and early entrance into the labor market by 13 and 14 percent, respectively. The mechanisms behind these effects include improved parental aspirations and expectations, improved attitudes and social norms, higher self-efficacy beliefs of parents, and increased time for home learning activities. Relative to other educational investments that have been evaluated in developing countries, the combined intervention is highly effective and cost-effective.


Using Social Media to Change Gender Norms: An Experimental Evaluation Within Facebook Messenger in India

Dante Donati, Victor Orozco-Olvera, Nandan Rao

This paper experimentally tests the effectiveness of two short edutainment campaigns (under 25 minutes) delivered through Facebook Messenger at reshaping gender norms and reducing social acceptability of violence against women (VAW) in India. Participants were randomly assigned to watch video-clips with implicit or explicit messaging formats (respectively a humorous fake reality TV drama or a docu-series with clear calls to action). After one week, intent-to-treat effects of the implicit format on knowledge, gender norms, and acceptability of VAW oscillated between 0.16 and 0.21 standard deviations yet impacts diminished after four months. On other hand, the explicit format was more impactful in the short-term in increasing willingness to share video-clips with friends and promoting online information-seeking behaviors; and in the medium-term, individuals exposed to the docu-series were 91% (7.5 p.p.) more likely to add a frame against VAW in their Facebook profile picture, a public display of their disapproval of this harmful practice. The general lack of heterogeneous effects across social status indicators suggest social media as a potential medium for reaching different online populations, including vulnerable ones.


To pay or not to pay: Measuring risk preferences in lab and field 

Pablo Brañas, Lorenzo Estepa, Diego Jorrat, Victor Orozco-Olvera and Ericka Rascón-Ramirez 

Measuring risk preferences using monetary incentives is costly. In the field, it might be also unfair and unsafe. The commonly used measure of Holt and Laury (2002) relies on a dozen lottery choices and payments, which make it time consuming and expensive. It also raises moral concerns as a result of the unequal payments generated by good and bad luck. Paying some but not all subjects may also create tensions between the researcher and subjects. In a pre-registered study in Honduras, Nigeria and Spain, we use a short version of Holt and Laury where we address all three concerns. We find in the field that not paying at all or paying with and without probabilistic rules makes no difference. Our hypothetical and short version makes our measurement of risk cheaper, fairer and safer.

SSRN working paper 


Conducting Surveys and Interventions Entirely Online : A Virtual Lab Practitioner’s Manual

Dante Donati, Victor Orozco-Olvera, Nandan Rao

Online and social media campaigns reach billions of people every day. While commercial companies have built up extensive expertise in using these tools to recruit and build relationships with customers, researchers and policymakers have been slower to take full advantage of these new digital tools. Virtual Lab is an open-source set of tools developed by the World Bankʼs Development Impact Department (DIME) for online recruitment, intervention, and surveying via digital advertising and social-media platforms. It was built specifically for researchers and policymakers. An online study can only work if it performs inference on groups that can at least be found online, even if they are underrepresented, and only if such inference can be performed entirely from those individuals who are found online, even if they are hard to find. This manual provides an introduction to the survey theory underpinning Virtual Lab, lays out detailed guidelines for designing online studies within the platform, shares costs data and lessons learned from recent trials, and lays out promising areas for future research. Virtual Lab can be self-hosted on private or public cloud infrastructure. All code is open source.


The Entertaining Way to Behavioral Change: Fighting HIV with MTV 

Abhijit Banerjee, Eliana La Ferrara, and Victor Orozco-Olvera

This paper tests the effectiveness of an entertainment education television series, MTV Shuga, aimed at providing information and changing attitudes and behaviors related to HIV/AIDS. Using a simple model, the paper shows that “edutainment” can work through an individual or a social channel. This study is a randomized controlled trial conducted in urban Nigeria, where young viewers were exposed to MTV Shuga or a placebo television series. Among those exposed to MTV Shuga, the trial created additional variation in the social messages they received and the people with whom they watched the show. The study finds significant improvements in knowledge and attitudes toward HIV and risky sexual behavior. Treated subjects are twice as likely to get tested for HIV eight months after the intervention. The study also finds reductions in sexually transmitted diseases among women. These effects are stronger for viewers who reported being more involved with the narrative, consistent with the psychological underpinnings of edutainment. The trial’s experimental manipulations of the social norm component did not produce significantly different results from the main treatment. The individual effect of edutainment thus seems to have prevailed in the context of this study. 


Entertainment, Education, and Attitudes Toward Domestic Violence 

Abhijit Banerjee, Eliana La Ferrara, and Victor Orozco-Olvera

We experimentally measure the effectiveness of the edu-tainment TV series MTV Shuga on domestic violence outcomes, a secondary theme of the studied season. We find that MTV Shuga induced an improvement in men’s attitudes toward women eight months later. We also find that attitudes toward GBV significantly improve for men and women who report occasionally thinking about the characters and who remember specific facts about them. These findings call for a deeper analysis of the links between attention, empathy and the policy impact of edu-tainment programs.


Payments for environmental services supported social capital while increasing land management

Alix-Garcia, Jennifer M., Katharine R. E. Sims, Victor H. Orozco-Olvera, Laura E. Costica, Jorge David Fernández Medina, Sofía Romo Monroy

Payments for environmental services (PES) programs incentivize landowners to protect or improve natural resources. Many conservationists fear that introducing compensation for actions previously offered voluntarily will reduce social capital (the institutions, relationships, attitudes, and values that govern human interactions), yet little rigorous research has investigated this concern. We examined the land cover management and communal social capital impacts of Mexico’s federal conservation payments program, which is a key example for other countries committed to reducing deforestation, protecting watersheds, and conserving biodiversity. We used a regression discontinuity (RD) methodology to identify causal program effects, comparing outcomes for PES participants and similar rejected applicants close to scoring cutoffs. We found that payments increased land cover management activities, such as patrolling for illegal activity, building fire breaks, controlling pests, or promoting soil conservation, by ∼50%. Importantly, increases in paid activities as a result of PES did not crowd out unpaid contributions to land management or other prosocial work. Community social capital increased by ∼8–9%, and household-level measures of trust were not affected by the program. These findings demonstrate that major environmental conditional cash transfer programs can support both land management and the attitudes and institutions underpinning prosocial behavior. Rigorous empirical research on this question can proceed only country by country because of methodological limitations, but will be an important line of inquiry as PES continues to expand worldwide. 


Hard Skills or Soft Talk: Unintended consequences of a vocational training and an inspiration talk on childbearing and sexual behavior in vulnerable youth

Maria Jones, Victor Orozco-Olvera and Ericka Rascon-Ramirez

Hard and soft skills have been shown to be relevant for labour market and  educational outcomes, however, little is known about their influence on fertility  decision-making and sexual behaviour. This paper analyses to what extent a hard  skill (vocational training) and a soft skill (inspirational talk) intervention affected  childbearing decisions, HIV testing and transactional sex in young people. Using  baseline and follow-up data of a randomised control trial in Malawi, we find that  receiving an offer to attend a vocational training programme decreased the chances  of becoming a mother and increased the chances of being HIV-tested in both  females and males when comparing with the control group. The chances of being  demanded for transactional sex decreased for those women receiving the offer to  attend the vocational training. Comparing the effects of both interventions between  adolescents (under 20) and young adults (20-24), we observe that both ‘hard and  soft skills’ reduced the chances of becoming a parent and increased the chances of  being HIV-tested in both groups. For transactional sex, we observe opposite effects.  While vocational training reduced the chances of being demanded for transactional  services, inspirational talks increased the chances of demanding such services.  Adolescents who received ‘hard skills’ are less likely to offer transactional sex after  the intervention than those in the control group. These results shed light on gender and age differences in the impact of vocational training interventions on non-labour  outcomes and on how low-cost ‘soft skills’ interventions, such as inspirational talks,  may affect long term outcomes. 


Can Environmental Cash Transfers Reduce Deforestation and Improve Social Outcomes? A Regression Discontinuity Analysis of Mexico’s National Program (2011-2014) 

Alix-Garcia, Jennifer M., Katharine R. E. Sims, Victor H. Orozco-Olvera, Laura E. Costica, Jorge David Fernández Medina, Sofía Romo Monroy, and Stefano Pagiola

Environmental conditional cash transfers, or “payments for ecosystem services” are a centerpiece of global efforts to protect biodiversity, safeguard watersheds, and mitigate climate change by reducing forest loss. This paper evaluates the impacts of Mexico’s national payments for ecosystem services program, which provides five years of payments to landowners in exchange for maintaining and managing natural land cover. Using a regression discontinuity design, the paper studies impacts on environmental, socioeconomic, and social capital outcomes for the 2011–14 program cohorts. The analysis finds that treated communities increased management activities to protect land cover, such as patrolling for illegal conversion or combating soil erosion (by 48 percent compared to controls). The program reduced the loss of tree cover in areas at high risk of deforestation (by 29 percent compared to controls), with effects being larger for those that have been in the program the longest (38 percent compared to controls). 


Paying Attention to Technology Innovations - Experimental Evidence from Solar Lighting in Africa 

Aidan Coville, Victor Orozco-Olvera and Arndt Reichert 

This paper provides an explanation for why many information campaigns fail to affect decision-making. The authors experimentally show that a large information intervention about a profitable and climate-friendly household investment had limited effects if it only provided generic data. In contrast, it caused households to make new investments when it followed a campaign strategy designed to minimize information processing costs. This finding is consistent with a model of selective attention, where individuals prioritize information believed to be valuable after accounting for the costs of attending to the data that arise due to limited mental energy and time. The paper studies a range of possible mechanisms and finds corroborative evidence of selective attention as an inhibitor to learning. 


The effectiveness of using entertainment education narratives to promote safer sexual behaviors of youth: A meta-analysis, 1985-2017 

Victor Orozco-Olvera, Fuyuan Shen and Lucie Cluver

Risky sexual behaviors are associated with the transmission of sexually transmitted infections (STIs) and unwanted pregnancies, both major health concerns for youth worldwide. This review studies the effectiveness of narrated mass media programs in promoting safer sexual practices among youth in developed and developing countries. Electronic and manual searches were conducted to identify experimental and quasi-experimental studies with robust counterfactual designs published between 1985 and the first quarter of 2017. Effect sizes were meta-analyzed using mixed-effects models. Eight experimental and two quasi-experimental studies met our inclusion criteria. The aggregated sample size was 23,476 participants, with a median of 902 participants per study. Entertainment education narratives had small but significant effects for three sexual behaviors. It reduced the number of sexual partners [standardized mean difference, (SMD) = 0.17, 95% confidence interval (CI) = 0.02–0.33, three effect sizes], reduced unprotected sex (SMD = 0.08, 95% CI = 0.03–0.12, nine effect sizes), and increased testing and management for STIs (SMD = 0.29, 95% CI = 0.11–0.46, two effect sizes). The interventions were not effective in reducing inter-generational sex, measured through the age-gap with sexual partners (SMD = 0.06, 95% CI = -0.06–0.19, four effect sizes). Entertainment education had medium-size effects on knowledge outcomes (SMD = 0.67, 95% CI = 0.32–1.02, seven effect sizes), where a time-decay relationship is observed. No effects were found on attitudes. Although mass media entertainment had small effects in promoting safer sexual practices, its economies of scales over face-to-face interventions suggest its potential to be a cost-effective tool above an audience threshold. The use of study participants from the general youth population and the use of mostly effectiveness trials mitigate concerns regarding its scalability. The overall paucity of high-quality studies affirms the need for strengthening the evidence base of entertainment education. Future research should be undertaken to understand the moderator effects for different subgroups and intervention characteristics.


Gender Differences in the Effects of Vocational Training in Malawi: Constraints on Women and Drop-Out Behavior

Y. Cho, D. Kalomba, M. Mobarak, V. Orozco-Olvera and D. Wolfson 

We provide experimental evidence on the effects of vocational and entrepreneurial training for Malawian youth in an environment where access to schooling and formal sector employment is extremely low. The training results in skills development, continued investment in human capital and improved well-being, with more positive effects for men, but no improvements in labor market outcomes in the short-run. We find that women make decisions in a more constrained environment, and their participation is affected by family obligations. We also find that participation is more expensive for women compared to their male counterparts, resulting in a worse training experience. Additionally, we track a large fraction of program dropouts – a common phenomenon in the training evaluation literature – which allows us to examine the determinants and consequences of the dropout decision and how it mediates the effects of such programs. 


Moving from efficacy to effectiveness: Using behavioral economics to improve the impact of water, sanitation and hygiene interventions

Aidan Coville and Victor Orozco-Olvera

This paper explores cognitive and behavioural constraints identified in the behavioural economics literature and how interventions have successfully accounted for these constraints in their design to increase demand for services and positive habit formation.