Research

How Much Should We Trust Observational Estimates? Accumulating Evidence Using Randomised Controlled Trials with Imperfect Compliance, with David Rhys Bernard, Gharad Bryan, Sylvain Chabé-Ferret, Jon de Quidt and Roland Rathelot.

Associated CEDIL Research Project Paper 9. Available at DOI: https://doi.org/10.51744/CRPP9.
Financial support by IPA and CEDIL.

The use of observational methods remains common in program evaluation. How much should we trust these studies, which lack clear identifying variation? We propose adjusting confidence intervals to incorporate the uncertainty due to observational bias. Using data from 44 development RCTs with imperfect compliance (ICRCTs), we estimate the parameters required to construct our confidence intervals. The results show that, after accounting for potential bias, observational studies have low effective power. Using our adjusted confidence intervals, a hypothetical infinite sample size observational study has a minimum detectable effect size of over 0.3 standard deviations. We conclude that – given current evidence – observational studies are uninformative about many programs that in truth have important effects. There is a silver lining: collecting data from more ICRCTs may help to reduce uncertainty about bias, and increase the effective power of observational program evaluation in the future.

A Metadata Schema for Data from Experiments in the Social Sciences, with Jack Cavanagh, Sarah Kopper and Anja Sautmann. 

World Bank Policy Research Working Paper WPS10296.
Supported by J-PAL Global.

The use of randomized controlled trials (RCTs) in the social sciences has greatly expanded, resulting in newly abundant, high-quality data that can be reused to perform methods research in program evaluation, to systematize evidence for policymakers, and for replication and training purposes. However, potential users of RCT data often face significant barriers to discovery and reuse. We propose a metadata schema that standardizes RCT data documentation and can serve as the basis for one  - or many, interoperable - data catalogs that make such data easily findable, searchable, and comparable, and thus more readily reusable for secondary research. The schema is designed to document the unique properties of RCT data. Its set of fields and associated encoding schemes (acceptable formats and values) can be used to describe any dataset associated with a social science RCT. We also make recommendations for implementing a catalog or database based on this metadata schema.

GitHub repository with latest version and related materials.

Large Sample Inference for a Class of Estimators Based on Unconfoundedness

Matching-type methods are widely used to estimate the causal effect of a treatment on a set of outcomes under unconfoundedness. This paper proposes an inference procedure for a broad class of estimators imputing the missing counterfactual outcomes as a weighted sum of outcomes from the opposite treatment group. This encompasses the vast majority of matching-type estimators. In this context, a marginal variance estimator is proposed for the population average treatment effect and the population average treatment effect on the treated. To establish the results, I generalize the methodology suggested by Abadie and Imbens (2006) for nearest neighbor matching estimators. A Monte Carlo study assesses the performance of the inference procedure for a local linear kernel matching estimator. I obtain precise standard errors and coverage rates that perform equally well, if not better, than the naive bootstrap. An empirical application illustrates the good performance of the inference procedure in practice.

Demand-Side Interventions for Economic Integration of Refugees, with David Rhys Bernard, Yvonne Giesing, Jakob Hennig and Sekou Keita.

Funded by J-PAL's European Social Inclusion Initiative.

J-PAL Research Resources: Data Analysis, Maya Duru and Sarah Kopper,
with contributors Jack Cavanagh, Jasmin Claire Fliegner and Anja Sautmann.