Before-after comparison and the beauty of difference-in-differences
Consider that we want to understand whether the productivity of a business unit has changed since we have introduced a new technology (“AI” or “robot” for simplicity). First, we need to define what change means, relative to what. We need to clarify the level of comparison. A first natural comparison would be the productivity of the same unit before the implementation of AI. Say we compare the unit’s productivity in 16 weeks before the AI arrives, to its productivity in 16 weeks thereafter. When done with multiple units, we can quantify the average change. We call this research design an event study.
Event study designs are not optimal because something else might have happened at the same time as the introduction of the AI. For example, local management might have changed, workers in the region might have gone on strike, supply chain issues may have led to underutilization of capacity. Whether or not these things are observable in the data, they pose a problem because we cannot know whether changes in productivity happened because of robots, or because the other things that happened at the same time. The change that we quantify in an event study design would be incorrectly attributed to the introduction of robots, but in reality, is the effect of robots and other things.
A research design that can help us in such a situation is the so-called difference-in-differences approach. We add another comparison: in addition to the before/after of the event study, we also compare a group of units that eventually receive robots to a group of units that do not. We can call the former the treatment group and the latter the control group. Now if the things that happen at the same time as the introduction of the robots happen also in the control group, we can separate out the effect of robots from the effect of the other things. We essentially do two event studies: we compare productivity before/after in the treatment group, as well as in the control group. When we subtract the before/after difference in the control group from the before/after difference in the treatment group, we remove the changes that are due to the other things. Hence the name difference-in-differences.
The difference-in-differences approach is already much better but still has important problems. What if units that introduce robots differ in their productivity in the first place? For example, robots might be introduced to improve the productivity of less-than-average units. Or they might be introduced in units that are already highly productive, for whatever reason. When we take the difference-in-differences, we still do not know whether the difference is coming from robots per se. We could also just be measuring differences in productivity among different types of units where one type happens to also have robots, and the other type not. The tricky bit is that it is perfectly possible that there are no productivity effects of robots at all, although our difference-in-differences approach indicates so.