Counterfactual impact evaluation

Impact evaluation assesses the degree to which changes in a specific outcome or variable of interest as measured by a pre-specified set of indicators can be attributed to a program rather than to other factors. Such evaluation generally requires a counterfactual analysis to assess what the outcome would have looked like in the absence of the intervention.

Broadly speaking, one needs to compare a group that received the intervention, the “treatment group”, against a similar group, the “comparison group”, which did not receive the intervention. The observed difference in mean outcome between the treatment group and the comparison group can then be inferred to be caused by the intervention. What is observed in the comparison group serves as the counterfactual of what would have happened in the absence of the intervention.

Two types of methods can be used to generate the counterfactual: randomized controlled experiments and quasi-experiments. Both approaches rely on the estimation of the average causal effect in a population. In the first case, the treatment group and the comparison group (also termed “control group” in this case) are selected randomly from the same population. Similarly, quasi-experimental evaluation estimates the causal impact of an intervention, the difference being that it does not randomly assign the units between the treatment group and the comparison group. Hence, a key issue with quasi-experimental methods is to find a proper comparison group that resembles the treatment group in everything but the fact of receiving the intervention.

Examples are provided all along this lecture, with detailed R-CRAN programs, to provide the students with a complete description of these approaches.