Econometrics of Program Evaluation

This course covers the basic theoretical knowledge and technical skills required for implementing Econometric Methods of Causal Inference. These methods are used to tests predictions of economic theories and to measure the impacts of programs. These tools have been developed by social scientists, natural scientists and statisticians over the course of the last century. Over the last thirty years, economists have regrouped most of these tools in a standard toolkit. In this class, we will study this basic set of tools. These tools have been developed and/or are heavily used in labor, education, development, health and environmental economics. They are also used by funding agencies and governments to conduct evaluations of public policies and start being used by firms to evaluate product design, auction design, advertisement, etc.

The aim of this class is threefold:

1. Provide the (minimal) mathematical underpinning required to apply Econometric Methods of Causal Inference.
2. Provide the R code in order to apply these methods. I use knitr in order to nicely intricate code and text so that every notion is immediately illustrated by the accompanying R code simulation. Each chapter or section is available as a .pdf file, but also as a .Rnw file where the latex code and the R code to generate the examples are intertwined. In the text (for which only the first section of the class is completed yet) the R code is shown explicitly, so that there is no need for looking inside the .Rnw file. For slides, integrating code is more difficult so that the .Rnw files bring valuable information on the R code.
3. Make extremely clear the statistical issues that these methods face and try to suggest solutions. I especially focus on the notions of counterfactual and of sampling noise. I use simulated data to make apparent these specific problems. Simulations enable to show the unobserved counterfactuals and how they behave and also to illustrate the size of sampling noise and how the methods we use to recover treatment effects and sampling noise perform. I devoid time to explain the perils of significance testing. I provide a description of the statistical tools required to detect and correct for publication bias: pre-registration, blind data analysis, meta-analaysis, p-curves, etc.

The last version of the course is available publicly on the following GitLab repo: git@gitlab.com:chabefer/PEM.git. Comments, branching and pull proposals are welcome.

The course is structured in three broad sequences:

1. The Two Fundamental Problems of Inference:
   • Rubin Causal Model: the basic language to encode causality.
   • Treatment Effects: our causal parameters of interest.
   • The Fundamental Problem of Causal Inference (FPCI): the Treatment Effects of interest can NEVER be observed, even with a sample of infinite size (a very acute problem indeed!). What we can do instead is to use transformations of the observed data that, under certain assumptions, are equal to the Treatment Effect of interest when the sample size is infinite.
   • The Biases of Intuitive Comparisons: the intuitive comparisons that we use for causal inference (the before/after and with/without comparisons) are generally biased because of factors that determine both the outcomes of the program and who receives it. These factors are called confounding factors.
   • The Fundamental Problem of Statistical Inference (FPSI): in practice, sample sizes are finite. As a consequence, in each sample, our estimator differs from the Treatment Effect of interest. This phenomenon is called sampling noise. We will cover two useful statistical tools to help with this problem: gauging the size of the sampling noise ex-post; choosing sample size ex-ante to decrease sampling noise.
   • The perils of significance testing: specification search and publication bias. I suggest to NEVER use statistical tests and I explain why. I suggest to gauge sampling noise instead.
2. Methods of Causal Inference: In this section, we learn the three sets of methods that are used by economists in order to suppress the influence of confounding factors and estimate Treatment Effects. For each estimator, we will cover identification (how it solves the fundamental problem of causal inference absent sampling noise), estimation (how to compute an estimator with a sample) and precision (how to gauge the sensitivity of our estimate to sampling noise with independently and identically distributed (i.i.d.) observations).
   • Randomized Controlled Trials (RCTs) solve for the problem of the confounding factors by allocating the treatment at random, i.e. independently of the confounders. We will cover the four most used RCT designs: randomization by brute force, after self-selection, after eligibility and encouragement designs.
   • Natural Experiments leverage on features of the implementation of the program that approximate the conditions of a RCT. We are going to cover the three most used natural experiment methods: Instrumental Variables (IV), Difference-In-Differences (DID) and Regression Discontinuity Designs (RDD).
   • Observational methods try to measure the confounders and to account separately for their effects on the outcomes. Standard observational methods that we are going to study are OLS and Matching. I am also going to dedicate some time to more recent Observational Methods based on Machine Learning (ML).
3. Additional important topics.
   • Power analysis: before implementing a given method, we want either to choose the sample size required to reach a pre-specified level of precision or to gauge the level of precision we might reach with a pre-specified sample size.
   • How to estimate precision when observations are not i.i.d.
   • Placebo tests: tests that we implement in order to check the validity of natural experiments and of observational methods.
   • LaLonde tests: check whether observational methods and natural experiments can reproduce the results of RCTs.
   • Analysis of diffusion effects.
   • Analysis of distributive effects.
   • Meta-analysis.