Abstract: Sustained progress on child mortality requires better curative care. However, policy instruments intended to increase access to healthcare may only incompletely reduce underuse or create overuse. We conducted an RCT of 1,768 children in Mali that cross-randomized primary care subsidies and community health worker visits. We analyze how these interventions affect the targeting of acute care, which depends not only on overall demand, but on whether children receive care when actually sick. We collect nine weeks of daily symptom and health care data to measure demand conditional on need for care, as defined by WHO standards. Parents are over five times more likely to seek care when it is medically indicated, yet the probability of getting needed care remains below 5% in the control. Subsidies increase utilization by over 250%, significantly reducing underuse with moderate effects on overuse. Health worker visits have little aggregate effect on care usage. 

Earlier version with additional details on CHW interventions: World Bank Policy Research Working Paper WPS9486.

• • Online appendix of the working paper version here.

  • Innovations for Poverty Action policy brief / Write-up in the "Economiste du Faso".
    
    

Abstract: We explore prompt designs for a lightweight integration of a large language model (LLM) into clinical decision support in primary care in Nigeria. The LLM integration is designed to give immediate, actionable "second opinions" to frontline healthworkers on their patient interaction notes.  The assessment of a physician serves as a benchmark for the quality of the LLM feedback.  A particular challenge was to counter the LLM's tendency to over-recommend laboratory testing, which is more in line with medical practice in high-income countries.  We evaluate the ability of a range of prompt engineering approaches to better align the LLM's medical test recommendations with locally appropriate standards of clinical care. 

Abstract: Subsidized primary care and community health worker (CHW) visits are important demand-side policies in the effort to achieve universal health care for children aged under 5. Causal evidence on the interaction between these policies is still sparse. This paper reports the effects on diarrhea prevention, curative care, and incidence as well as anthropometrics for 1,649 children from a randomized controlled trial in Bamako that cross-randomized CHW visits and access to free health care. CHW visits improve prevention and subsidies increase the use of curative care for acute illness, with some indication of positive interaction effects. There is no evidence of moral hazard, such as reduced preventive care among families receiving the subsidy. Although there are no significant improvements in malnutrition, diarrhea incidence is reduced by over 70 percent in the group that receives both subsidies and CHWs. Positive effects are concentrated among children under age 2. 

• Earlier version: World Bank Policy Research Working Paper WPS9986.
  Funded by The Population Studies and Training Center and the Rhodes Center at Brown University.

Abstract: Multiple price lists are a convenient tool to elicit willingness to pay (WTP) in surveys and experiments, but choice patterns such as "multiple switching'' and "never switching'' indicate high error rates. Existing measurement approaches often do not provide accurate standard errors and cannot correct for bias due to framing and order effects. We propose to combine a randomization approach with a random-effects latent utility model to detect bias and account for error. Data from a choice experiment in South Africa shows that significant order effects exist which, if uncorrected, would lead to distorted conclusions about subjects' preferences. We provide templates to create an MPL survey instrument in SurveyCTO and analyze the resulting data using our proposed methods.

• • Earlier versions: World Bank Policy Research Working Paper WPS 10173, NBER Working Paper 30433.

  • Technical Appendix describing data collection and analysis tools contained in the implementation package.

  • GitHub repository with the implementation package (by Kelsey Jack, Kathryn McDermott, Anja Sautmann, Kenneth Chan, and Lei Yue).
    
    Field work supported by J-PAL Africa, J-PAL’s Urban Services Initiative and the International Growth Center.
    Software development funded by a World Bank RSB grant.

Abstract: We conducted an experiment in Mali to test whether patients pressure doctors to prescribe medical treatment they do not necessarily need. The experiment varied patients’ information about a discount for antimalarial tablets and measured demand for both tablets and costlier antimalarial injections. We find evidence of patient-driven demand: informing patients about the discount, instead of letting doctors decide to share this information, increased discount use by 35 percent and overall malaria treatment by 10 percent. These marginal patients rarely had malaria, worsening the illness-treatment match.

• • Video (GSE Barcelona, 2018) / Short summary (VoxDev, 2019).

  • A Policy Research Talk at the World Bank about this and other papers on quality of care, using the example of subsidized health care in Mali.

Abstract: Standard experimental designs are geared toward point estimation and hypothesis testing, while bandit algorithms are geared toward in‐sample outcomes. Here, we instead consider treatment assignment in an experiment with several waves for choosing the best among a set of possible policies (treatments) at the end of the experiment. We propose a computationally tractable assignment algorithm that we call “exploration sampling,” where assignment probabilities in each wave are an increasing concave function of the posterior probabilities that each treatment is optimal. We prove an asymptotic optimality result for this algorithm and demonstrate improvements in welfare in calibrated simulations over both non‐adaptive designs and bandit algorithms. An application to selecting between six different recruitment strategies for an agricultural extension service in India demonstrates practical feasibility.

• • NOTE: The proof to theorem 1 in this paper contains errors, as pointed out by Ariu et al (2021). Items 1 and 2 built on arguments in Russo (2016) that do not apply to Bernoulli distributions, but were proven for Bernoulli outcome distributions by Shang et al. (2020). Accordingly, equivalent statements to item 1 and 2 regarding the speed of convergence of the treatment arm assignment shares to a limit allocation continue to hold. Item 3 claiming constrained optimal speed of convergence of (frequentist) expected policy regret is incorrect.  A weaker statement on the constrained optimal convergence of the posterior probability optimal and therefore posterior expected regret continues to hold. Please see a more detailed correction note here (Kasy & Sautmann, Nov 2021).

  • A visual dashboard of the experiment with Precision Agriculture in Development in the paper. 

  • An interactive dashboard to determine treatment assignment using exploration sampling for your own experiments.

  • A blog post explaining exploration sampling and adaptive experimentation more generally on VoxDev.
    More blogs: GlobalDev / "Phenomenal world".

Abstract: Incentivized experiments are often used to identify the time preferences of households in developing countries. We argue theoretically and empirically that experimental measures may not identify preference parameters, but are a useful tool for understanding financial shocks and constraints. Using data from an experiment in Mali, we find that subject responses vary with savings and financial shocks, meaning they provide information about credit constraints and can be used to test models of risk sharing.

Abstract: This paper considers a matching market with two-sided search and transferable utility where match payoffs depend on age at marriage (time until match) and search is finite. We define and prove existence of equilibrium, and provide sufficient conditions for positive assortative matching that build on restricting the slope and curvature of the marriage payoff function to generate single-peaked preferences in age and therefore convex matching sets. Payoff functions that are incompatible with positive sorting by age include all strictly increasing functions and constant flow payoffs enjoyed for some finite period.

Abstract: This paper experimentally tests the predictions of a principal-agent model in which the agent has biased beliefs about his ability. Overconfident workers are found to earn lower wages than underconfident ones because they overestimate their expected payoff, and principals adjust their offers accordingly. Moreover, the profit-maximizing contract distorts effort by varying incentives according to self-confidence, although only the most successful principals use this strategy. These findings have implications for the labor market; in particular, self-confidence is often correlated with gender, implying that principals would prefer to hire men over women simply because they are more overconfident.

Funded by NSF grant SES-0849465.

Abstract: Can information about the value of diagnostic tests improve provider practice and help patients recognize higher quality of care? In a randomized experiment at public clinics in Mali,  health providers and patients received tailored information about the importance of rapid diagnostic tests (RDTs) for malaria. The provider training increased provider reliance on RDTs, which improved the match between a patient's malaria status and treatment with antimalarials by 15-30 percent. Nonetheless, patients were significantly less satisfied with the care they received, driven by those whose prior beliefs did not match their true malaria status. The patient information intervention did not affect treatment outcomes or patient satisfaction and reduced malaria testing. These findings are consistent with highly persistent patient beliefs that translate into low demand for diagnostic testing and limit patients' ability to recognize improved quality of care.

Abstract: The use of randomized controlled trials (RCTs) in the social sciences has greatly expanded, resulting in newly abundant, high-quality data that can be reused to perform methods research in program evaluation, to systematize evidence for policymakers, and for replication and training purposes. However, potential users of RCT data often face significant barriers to discovery and reuse. This paper proposes a metadata schema that standardizes RCT data documentation and can serve as the basis for one—or many, interoperable —data catalogs that make such data easily findable, searchable, and comparable, and thus more readily reusable for secondary research. The schema is designed to document the unique properties of RCT data. Its set of fields and associated encoding schemes (acceptable formats and values) can be used to describe any dataset associated with a social science RCT. The paper also makes recommendations for implementing a catalog or database based on this metadata schema. 

Abstract: Adaptive sampling in experiments with multiple waves can improve learning for "policy choice problems" where the goal is to select the optimal intervention or treatment among several options. This paper uses a real-world policy choice problem to demonstrate the advantages of adaptive sampling and propose solutions to common issues in applying the method. The application is a test of six formats for automated calls to parents in Kenya that encourage reading with children at home. The adaptive 'exploration sampling' algorithm is used to efficiently identify the call with the highest rate of engagement. Simulations show that adaptive sampling increased the posterior probability of the chosen arm being optimal from 86 to 93 percent and more than halved the posterior expected regret. The paper discusses a range of implementation aspects, including how to decide about research design parameters such as the number of experimental waves.

Abstract: Using in-home health records for 1,763 children in Mali, we examine gender differences in uptake and duration of treatment with antibiotics. The detailed data provide a window into parents’ day-to-day decisions while accounting for symptoms. We find no gender differences in starting treatment, but boys are over 10 percentage points more likely to complete a course of antibiotics than girls. This difference is driven by families with an educated household head. An explanation may be that (male) household heads are less involved in caring for girls, so that benefits from education that lead to better care accrue overproportionally to boys.