Natural Experiment

A natural experiment is an observational study in which the assignment of treatments to subjects has been haphazard, chosen by nature or policy rather than an experimenters. Typically the variation is "exogenous" to the model; it must also be orthogonal to other determinants of the outcome variable.

Natural experiments are most useful when there has been a clearly defined and large change in the treatment or exposure to a clearly defined subpopulation, so that changes in responses can be plausibly attributed to the change in treatments or exposure.

Heckman and Deaton criticize the experimentalist method here and here, arguing that parameters from natural experiments apply convincingly only to the studied subpopulation of induced compliers. Imbens responds here that recovered parameters are usually the best we can do.

The argument seems to decompose into those arguing structural methods offer "external validity" against those arguing so-called reduced-form methods strive for "internal validity." It's usually lost that internal validity is necessary for external validity.

Moreover, natural experiments are less costly...and natural experiments derived from uneven policy application are always policy relevant. Angrist argues that natural experiments have given economists a new wind of credibility.

There are a few papers from recent job candidates that I think show the value of natural experiments and how they can be done well.

Consider the increasingly sexy problem of network effects. How can the econometrician measure out the effect of a network when networks are inherently endogenous? One of my great colleagues at University of Pennsylvania, Anton Badev, approaches the problem by modeling error term, imposing some structure that networks are made in a particular temporal manner. This soundingly simple approach is technically excruciating. A single iteration of estimates took one month of Minnesota supercomputers working 24-7. And the results are sensitive to how you model the networking. And since there's a lot economists don't know about networks (because of our endogeneity problems) we just don't have a great idea for how that error should be modeled.

This type of vexing question is appropriate grounds for a natural experiment if you can find one. We look for a situation in which networks are somehow determined orthogonally to other determinants of the outcome of interest. Enter Kelly Shue who used the random assignment of sections among Harvard MBAs over the past 65 years. She finds some really marvelous results that are completely compelling because of the natural experiment setting. She placed at Booth with this paper.

The trouble is that the results don't necessarily generalize. Estimates derived from one environment transfer only with great faith or additional assumptions; it's difficult to say whether and how estimates apply to other settings because experimentalists often ignore the underlying mechanisms which inform how results will transfer to new settings. But the results are transparent and convincing.

Another nice use of natural experiments are those in which (1) the effects may be small (and thus hard to see in small samples feasible in randomized controlled trials (RCTs), (2) experimentation may be prohibited (exposing people to pollution), or (3) you want to test a new hypothesis where the academy has strong contrary priors making funding for a large experiment difficult.

A great example of all three is Doug Almond's paper on neonatal nutrition in which he uses the lunar calendar and Ramadan as exogenous variation in nutrition for expecting mothers. He finds that years where daylight is shorter during Ramadan induces healthier offspring. Very clever economist.

The effect was too small to see in a controlled experiment, you probably could not have done the experiment, and the academy had strong priors against the need to test the hypothesis. A great use of natural experiments.

More on natural experiments from Wikipedia.

Keywords: natural experiment, internal validity, external validity, Andrew Johnston, Andrew, Johnston, economics, applied economics, economist, microeconomics, empirics, empirical economics, Wharton Applied Economics, applied economics, Wharton, Andrew Johnston, Andrew, Johnston, economics, applied economics, economist, microeconomics, empirics, empirical economics, Wharton, Andrew Johnston, Economics, Andrew Johnston economics