Modeling the Bias of Observational Methods

In this part of my work, I develop realistic models in order to understand and gauge the bias of observational methods in the absence of experimental data. This theoretical leg of work on observational methods complements the empirical leg in several ways. First, LaLonde tests are not always feasible, whereas it is almost always feasible to set up a realistic model of selection and outcome dynamics. The theoretical leg leverages on the modeling of outcome dynamics and of selection into programs in order to predict the expected direction of the bias of observational methods. Second, the theoretical leg has the potential to help us make sense of the results of the empirical leg. It is one thing to spot an empirical regularity among all programs of the same type, it is something else to be able to understand why this regularity is there. Being able to explain a regularity increases our confidence in its existence and makes us more confident when we extend it to other contexts. Third, the results of the theoretical leg of our approach are predictions that can be tested against the results of the empirical leg. If our model predicts the sign and the scope of the bias of a given method well, it is strongly vindicated, especially if the model was not cooked to reproduce this result.

In the modeling leg of my work, I very often resort to the example of Job Training Programs (JTPs) and their impact on earnings, for several reasons. First, JTPs are crucial components of the modern welfare state, especially in a context in which innovations and trade disrupt entire sectors in developed countries and require the retooling of millions of workers. Second, earnings are the main outcome that a JTP seeks to influence, especially by increasing the human capital of workers. Third, earnings dynamics are described extensively by some well-known processes whose parameters have been estimated in labor economics. Fourth, both observational methods and Randomized Controlled Trials (RCTs) have been and still are extensively used to evaluate JTPs. Fifth, most previous LaLonde tests have been conducted for the effect of JTPs on earnings and their results can be contrasted with the predictions of the model.

As of writing, my main results are:

1. The observational method considered to reproduce the results of RCTs best (the combination of Difference In Differences (DID) with matching on observed covariates) does not succeed because it combines DID with conditioning on pre-treatment outcomes, as currently believed in the literature. This approach might even increase bias. DID matching performs best when applied symmetrically around the treatment date and not conditioning on pre-treatment outcomes.
2. The bias of matching estimators is expected to be negative, and should decrease as we adjust for more observations of pre-treatment earnings.
3. DID matching and matching should become very similar as we condition on additional observations of pre-treatment outcomes and should dominate DID applied symmetrically around the treatment date.