Empirical Evidence on the Bias of Observational Methods

The goal of this part of my research is to gather empirical evidence on the bias of observational methods. More precisely, we aim at generating a dataset containing the bias of observational methods for a wide range of applications. Such a dataset would provide researchers with access to information on the bias of observational methods in precise sets of conditions, e.g. for a given outcome, method and set of observed covariates. Equipped with such knowledge, researchers could decide which observational method to use, or alternatively decide to run an RCT, decide to collect critical covariates, correct the results of observational methods ex-post and have a more informed assessment of the results stemming from observational methods.

The empirical leg of my approach leverages on LaLonde tests. LaLonde tests compare observational and experimental estimates of the same program. Together with Jasmin Fliegner and Roland Rathelot, we propose a cheaper and more rigorous way to run LaLonde tests using "Intention to Treat" Randomized Controlled Trials. We believe that our approach has the potential to increase the number of LaLonde tests and thus to increase our knowledge on the bias of observational methods in a variety of different contexts. We apply our approach on data from 7 RCTs conducted in developing countrires and for which data is available online, thus adding almost as many new LaLonde tests as already exist in the literature. Our goal is also to enrich our dataset using experiments for which the data is not accessible online. To that aim, we are developing a statistical package that automatically analyzes any experimental dataset and sends an estimate to our database. The database will be eventually made freely accessible to all researchers and practitioners. With Jasmin Fliegner, we also conduct a meta-analysis of the results of previous LaLonde tests and we compare them to the ones we have obtained.

Our main results can be summarized as follows:

1. For social programs implemented in developing countries, the bias of observational methods is large and it remains large even after conditioning on the observed covariates present in the datasets.
2. For Job Training Programs in developed countries, the bias of observational methods is small and consistently negative.