Throw the Ball to Naturally
February 16, 2021 1:22 PM   Subscribe

A List of Natural Experiments "Suppose you want to know whether being exposed to pollution stunts a child’s cognitive development. A university ethics board is unlikely to allow you to put some children into a treatment group, give them a hearty dose of pollution and then see what happens. Similarly, if you want to know how strong institutions affect long-run economic growth, you cannot assign good institutions to one country, bad institutions to a neighbouring one and then wait 300 years to watch it all unfold. Natural experiments can provide answers to these kind of questions."

"For this reason, and hopefully to provoke some inspiration, I put together the below list of natural experiments. This list is mostly drawn from the economics literature and is in no way comprehensive, representative or a ranking. For all these examples, remember that the key assumption is that there are comparable groups, and one group is randomly affected by forces outside their control (“nature”). If this assumption holds, we can causally interpret the results. All these papers typically include many tests in order to convince the reader that their data meet these conditions, but there are always interesting possible objections to think of. "

Background reading:

1. The Oregon Health Insurance Experiment

2. Angrist(1990), Lifetime Earnings and the Vietnam Era Draft Lottery: Evidence from Social Security Administrative Records, AER.

3. Card & Krueger(1994), Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania, AER

4. Almond (2006), Is the 1918 Influenza Pandemic Over? Long Term Effects of In Utero Influenza Exposure in the Post-1940 U.S. Population, Journal of Political

5. Almond, Mazumder, & van Ewijk (2014), In Utero Ramadan Exposure and Children's Academic Performance, Economic Journal

6. The Dutch Famine Study

7. Card (1990), The Impact of the Mariel Boatlift on the Miami Labor Market, Industrial and Labor Relations Review.

8. Rich et al (2015), Differences in Birth Weight Associated with the 2008 Beijing Olympic Air Pollution Reduction: Results from a Natural Experiment, Environmental Health Perspectives

9. Angrist & Lavy (1999), Using Maimonides' Rule to Estimate the Effect of Class Size on Scholastic Achievement, QJE.

10. Kearney& Levine (2014), Media Influences on Social Outcomes: The Impact of MTV’s 16 and Pregnant on Teen Childbearing, NBER Working Paper.

11. Costello et al. (2003), Relationships Between Poverty and Psychopathology A Natural Experiment, JAMA.

12. Do the Effects of Early Severe Deprivation on Cognition Persist Into Early Adolescence? Findings From the English and Romanian Adoptees Study, Child Development.

13. Bouchard et al. (1990), Sources of Human Psychological Differences: The Minnesota Study of Twins Reared Apart, Science.

14. Acemoglu, Johnson, & Robinson (2001), The Colonial Origins of Comparative Development: An Empirical Investigation, AER.

15. Acconcia, Corsetti & Simonelli (2014), Mafia and Public Spending: Evidence on the Fiscal Multiplier from a Quasi-experiment, AER

16. Lacetera, Macis & Slonim (2012), Will There Be Blood? Incentives and Displacement Effects in Pro-Social Behaviour, American Economic Journal

17. Almond, Edlund & Palme (2009), Chernobyl's Subclinical Legacy: Prenatal Exposure to Radioactive Fallout and School Outcomes in Sweden.
posted by storybored (12 comments total) 49 users marked this as a favorite
I'm surprised the Corrupted Blood incident wasn't on the list, being one of the more well known natural experiments for epidemiology.
posted by NoxAeternum at 1:29 PM on February 16 [1 favorite]

A university ethics board is unlikely to allow you to put some children into a treatment group, give them a hearty dose of pollution and then see what happens

I reject this unscientific idle speculation.

We conducted a randomized controlled trial in which 100 university ethics boards were presented with either a 'placebo' research proposal in which the hearty dose of pollution is administered to mice, or the 'experiment' research proposal...
posted by allegedly at 1:35 PM on February 16 [25 favorites]

there are always interesting possible objections to think of

E.g., Did Reality TV Really Cause a Decline in Teenage Childbearing? A Cautionary Tale of Evaluating Identifying Assumptions

And the 1994 Card and Kreuger minimum wage research has spawned an entire subfield; a recent paper reviewed 66 similar studies.
posted by Mr.Know-it-some at 1:46 PM on February 16 [3 favorites]

Boy do these confirm my priors.

Cool post!
posted by latkes at 1:47 PM on February 16

The Corrupted Blood one seems even more relevant these days. At the time when it was first brought up as a pandemic analog, I assumed it wouldn't be that great because obviously in a real deadly pandemic people wouldn't deliberately spread it to others, right?


Stares dismally at dozens of videos of anti-maskers and people spitting on other people.
posted by tavella at 2:12 PM on February 16 [1 favorite]

Since this blog post was written, there's of course been more work -- although it's not my field, natural experiments are the basis for a lot of the angrist and pischke 'mostly harmless' approach to identification in microeconometrics.

Two more review articles for the interested:
Nicola Fuchs-Schuendeln & Tarek Alexander Hassan (2016), "Natural Experiments in Macroeconomics"
Davide Cantoni & Noam Yuchtman, (2020), "Historical Natural Experiments: Bridging Economics and Economic History"

The Journal of Economic Perspectives (ungated) had a 2010 symposium on the randomization/quasi-randomization approach to econometrics, which is available here. Chris Sims's contribution to that symposium has a characteristically blunt opening:
The fact is, economics is not an experimental science and cannot be. "Natural" experiments and "quasi" experiments are not in fact experiments. They are rhetorical devices that are often invoked to avoid having to confront real econometric difficulties. Natural, quasi-, and computational experiments, as well as regression discontinuity design, can all, when well applied, be useful, but none are panaceas. The essay by Angrist and Pischke, in its enthusiasm for some real accomplishments in certain subfields of economics, makes overbroad claims for its favored methodologies. What the essay says about macroeconomics is mainly nonsense.
(I think Sims is overstating here -- obviously, there are people who do experimental, lab-based work in economics, and RCTs are common in empirical development economics and other fields. But I also I share some of his concern about the 'natural experiment' rhetorical device and its limitations).
posted by dismas at 2:29 PM on February 16 [3 favorites]

What, no Kansas Brownback experiment? I hope that puts the stake in the coffin of the "cutting taxes always good" policies.
posted by Monochrome at 5:26 PM on February 16 [1 favorite]

Why is there no academic study on the mall with two minimum wages?
posted by GuyZero at 11:34 AM on February 17 [2 favorites]

17 hit close to home, I grew up in the one the worst effected areas. Both me and my brother are data points in that study... hmm reads paper...
But thankfully not in the cohort that was exposed in utero. So we don't have any significant effects and we both made it to uni so it can't be that bad ;)
posted by skaggig at 10:55 PM on February 17

I like Fisman and Miguel's natural experiment look at corruption; whether official corruption comes more from cultural norms or from the lack of legal enforcement is hard to evaluate because places with strong norms against corruption tend to also have strong enforcement cultures and vice versa.

The natural experiment took officials from across a wide range of cultures and presented them with the ability to gain a valuable benefit in a corrupt fashion without any enforcement consequences. The premise is that the effect cultural norms would be at least partially isolated.

Specifically, United Nations diplomats in New York have essentially no enforcement of parking violations (and easy parking in NYC is very valuable); the study found that diplomats from countries with higher corruption tended to have many more unpaid parking violations than those from less corrupt countries.
posted by Superilla at 11:44 PM on February 17 [4 favorites]

A lot of the work done in Astrophysics is technically natural experiments. Not for ethical reasons, obviously. Most people just don't have the patience to set up a few galaxies and see how they evolve over a few billion years - no discipline these days!
posted by vibratory manner of working at 8:35 PM on February 18 [2 favorites]

« Older What happened?   |   Il Maestro Newer »

You are not currently logged in. Log in or create a new account to post comments.