The pivot from the quantitative finding to the speculative explanation

July 13, 2011 12:27 AM

Everyone knows that correlation doesn't imply causation, but researchers invariably need to come up with plausible explanations (i.e., models) for the patterns found in their data. However, very different models can "explain" the same pattern. The books The Bottom Billion: Why the Poorest Countries are Failing and What Can Be Done About It and Wars, Guns and Votes: Democracy in Dangerous Places by Oxford economist Paul Collier try to explain why some countries have remained poor using data from econometric studies. In his very interesting review (PDF), Mike McGovern, a political anthropologist at Yale, critiques the types of explanations found in popular economics books. Statistician Andrew Gelman has further thoughts on descriptive statistics, causal inference, and story time.
posted by Jasper Friendly Bear (59 comments total)

Easterly Reviews Collier

The New York Review of Books: Foreign Aid Goes Military!

This is a book review of the Paul Collier's The Bottom Billion: Why the Poorest Countries Are Failing and What Can Be Done About It by William Easterly, the author of The White Man's Burden: Why the West's Efforts to Aid the Rest Have Done So Much Ill and So Little Good. I think Easterly's book is still the best of the books I've read on Africa. In this lengthy review, Easterly takes Collier to task for his openness to military intervention in failing nations. I don't think Easterly's criticism invalidates the basic themes coming from Collier's research but I think it does provide a helpful check on how we move from research to policy.
posted by infini at 1:54 AM on July 13, 2011

Economics are modern day alchemy.

Scientific sounding terminology, activities that look like research and analysis, but in the end it's just handwaving accompanied by technobabble, trying to convince people that they can turn goose shit into gold.
posted by unigolyn at 1:56 AM on July 13, 2011 [12 favorites]

Everyone knows that correlation doesn't imply causation

I'm not sure about that. I think there may be a strong link between a lack of correlation and a lack of causation, but I don't think it proves anything.
posted by twoleftfeet at 2:33 AM on July 13, 2011 [5 favorites]

The trap of identifiability (from the descriptive statistics link):

Rogeberg has a theoretical model explaining how economists can be so rigorous in parts of their analysis and so unrigorous in others. Rogeberg sounds very much like McGovern when he writes:

The puzzle that we try to explain is this frequent disconnect between high-quality, sophisticated work in some dimensions, and almost incompetently argued claims about the real world on the other.

This sort of Freakonomical analysis sells books, but doesn't do much else.
posted by three blind mice at 2:47 AM on July 13, 2011

An Economist is somebody who will be able to explain tomorrow why today didn't happen what he predicted yesterday.
posted by DreamerFi at 3:11 AM on July 13, 2011 [8 favorites]

Fabulous, fabulous post. There is so much here, and the links and blog comments keep spinning off into other posts, reviews, editorials, comments that are all of a uniformly high quality - agree with them or not.

I think McGovern is bang on the money when he talks about sleight of hand a lot of popular economists pull by bludgeoning readers with data and then sneakily using older - and flawed - rhetorical techniques to rustle up a point.

On the other hand, though, I think Chris Blattman is absolutely right to say:

Economists often take their models too seriously, and too far. Unfortunately, no one else takes them seriously enough. In social science, models are like maps; they are useful precisely because they don't explain the world exactly as it is, in all its gory detail. Economic theory and statistical evidence doesn't try to fit every case, but rather find systematic tendencies. We go wrong to ignore these regularities, but we also go wrong to ignore the other forces at work-especially the ones not so easily modeled with the mathematical tools at hand..

For me, personally, I feel like a lot of popular economists are more than happy to wander off the disciplinary trail and apply their economic discourse to any discipline, any problem, any setting with a kind of blithe assurance that because their research has some maths in it, it's a science, unlike those silly mystical sociologists, political scientists, anthropologists etc that have preceded them.

Unfortunately, I think a lot of *popular* (not condemning the whole field here, far from it) economists fail to consider the full context, and the years of experience, insight and knowledge that other disciplines working in the field - whatever the field is - might have. It's a shame, because it cheapens the reputation of the field as a whole, and often means that the valuable contributions a truly cross-disciplinary approach could yield never really eventuate.

This all said, I also think the problem is that regular, orthodox economists rarely get as much attention as the heterodox, out-there economists, making politicised judgments in zany fields. It's a damned shame; if more people had paid attention to the orthodox economists, the GFC would have been nowhere so bad, and the response to it and subsequent recovery much, much more effective.

Instead people equate economics with lunatic libertarians, Friendmanite fools, etc. The reality is nowhere near so incendiary or outrageous.
posted by smoke at 3:25 AM on July 13, 2011 [4 favorites]

Great post. I've just finished the McGovern review and am about to start on the Gelman one; I love a solidly argued academic slam-down.
posted by A Thousand Baited Hooks at 3:53 AM on July 13, 2011

It's wrong to say "correlation doesn't imply causation", it just doesn't tell you which way causation flows. So lets say you drive over a nail and your tire goes flat. That doesn't mean The flat tire caused the nail to appear, but the nail did cause the flat tire.

If there is a statistically valid correlation, then there is some causal link between the two things (up to the statistical certainty of the data)

Mostly "Correlation is not causation" is something people say whenever they hear a scientific result they don't like.
posted by delmoi at 3:57 AM on July 13, 2011 [4 favorites]

Never reading anyone on Metafilter hashing out facile causation/correlation arguments would correlate very strongly with intense jubilation in my soul.
posted by smoke at 4:01 AM on July 13, 2011 [1 favorite]

related (somewhat): Andrew Gelman on The Browser recommending five books for people interested in statistics.
posted by smoke at 4:03 AM on July 13, 2011

Everyone knows that correlation doesn't imply causation

I wonder how "everyone" learns not to put their hands on hot things, then. Do they develop a working model of physics and wave/particle duality before they stop burning themselves?
posted by rodgerd at 4:07 AM on July 13, 2011 [3 favorites]

If there is a statistically valid correlation, then there is some causal link between the two things (up to the statistical certainty of the data)

Not true.

Like for instance lets examine the correlation between two things don't have a causal link, that is one doesn't cause the other or vice versa. Like say the certainty with which you assert your opinions as fact and the frequency of such assertions. Neither causes the other yet there is a correlation that is derived from the fact that the two independent things have a shared cause - an engineering degree.
posted by srboisvert at 4:13 AM on July 13, 2011 [5 favorites]

Correlation is not proof of causation, but it is often strong evidence.
posted by rocket88 at 4:48 AM on July 13, 2011 [1 favorite]

Neither causes the other yet there is a correlation that is derived from the fact that the two independent things have a shared cause - an engineering degree.

If there is some cause that causes both of them, then they have a causal link, just an indirect one. There is a path from one to the other. Causal link isn't a technical term as far as I know. I just meant that you can get from one to the other. Either one causes the other or there is some third cause that causes both of them.

But, if you have a statistically valid sample, then you can say with statistical certainty that there is some causal relationship.
posted by delmoi at 4:57 AM on July 13, 2011 [1 favorite]

The issue with the statement is "imply". I can imply untrue things all day long. This is why, in the Citadel of Science, we run experiments where our goal is (or at least should be) to beat the crap out of this suggested link and see if we can break it. If we can, well, that isn't it, is it. If we can't - Fame! Fortune! Women throwing themselves at us!* A conclusions statement in a report that isn't absolutely painful to write!

When economists run experiments it tends to be more like a rain dance. No controls. No rigidly defined conditions. Just keep applying the treatment (or pay lip service to applying the treatment) until it rains.

*I'm not sure if women throw themselves at my female collages or if men do. Hell, I'm not sure how those women even get into the building!
posted by Kid Charlemagne at 4:57 AM on July 13, 2011

The issue with the statement is "imply". I can imply untrue things all day long.

Okay those two sentences use the word "imply" in completely different ways. The correlation/causation thing means logical implication while you're using the word "imply" to mean "hint at something" in the second sentence. Totally different meaning.

A Logical implication is something like if A then B When you say Correlation does not imply causation you're saying it is not true that If correlation then causation.

You can't make logically untrue implications and still be logically consistent. It's true that you can "imply untrue things all day long" but you can also simply state untrue things all day long if you want.
posted by delmoi at 5:08 AM on July 13, 2011 [2 favorites]

If there is a statistically valid correlation, then there is some causal link between the two things (up to the statistical certainty of the data)

You say this like there are not a dozen odd kinds of bias that a social scientist should know. Even if the correlation is not a technical / design artifact, most of the time that causal path is just an uninteresting one, like confounding or selection bias.

Mostly "Correlation is not causation" is something people say whenever they hear a scientific result they don't like.

People who take weak studies at face value get the Wishful Thinking Merit Badge issued by the Daily Mail.
posted by a robot made out of meat at 5:16 AM on July 13, 2011 [3 favorites]

When economists run experiments it tends to be more like a rain dance. No controls. No rigidly defined conditions. Just keep applying the treatment (or pay lip service to applying the treatment) until it rains.

Do economists run experiments at all? My recollection of economics from the year I spent studying it at uni was that is was 60% an inseparable mixture of abstract mathematics and amateur psychological speculation and 40% using post hoc reasoning to argue whatever ideological point you felt like running with at the time. Maybe closer to 50-50. I certainly don't remember any "experiments", unless you count the breakup of the USSR.

I love the bit of the McGovern piece that Chris Blattman quotes, especially the comparison of development economics to literary criticism.
posted by A Thousand Baited Hooks at 5:18 AM on July 13, 2011 [1 favorite]

I find it persuasive that Paul Collier doesn't understand the Ivory Coast, and his policy recommendations seem sort of dumb, but McGovern doesn't once mention instrumental variables, so the larger point about mistaking correlation for causation has not been proven.
posted by anotherpanacea at 5:30 AM on July 13, 2011

Not a logical implication, a statistical implication. And when you get right down to it, they only hint at something. It's just that they put numbers on those hints, but those numbers are just a very precise way of saying that whatever is being suggested may only be a bit of froth floating on a sea of noise. (Sorta like the crawl at the bottom of Fox News.)

And at the end of the day, just because you have proven that two phenomenon are linked, that does not mean that the link works the way you think it does.
posted by Kid Charlemagne at 5:32 AM on July 13, 2011

Oh, for pete's sake, correlation DOES imply causation. It just doesn't prove it.
posted by ocschwar at 5:41 AM on July 13, 2011 [1 favorite]

Oh, that's an XKCD link. Never mind...
posted by ocschwar at 5:43 AM on July 13, 2011

Oh, for pete's sake, correlation DOES imply causation. It just doesn't prove it.

That depends on what you mean by "implication"...
posted by Philosopher Dirtbike at 5:45 AM on July 13, 2011

On reading more carefully, what demoi said.
posted by Philosopher Dirtbike at 5:47 AM on July 13, 2011

On rereading more carefully, only partially what delmoi said. The ability to infer causation is not directly related to having a "statistically valid" sample; rather, it is related to the research design itself. Experiments, in which a factor is manipulated (often at random) and the outcomes are observed, give the ability to infer causation. Other designs, such as pure observation (which is typically used in economics, because of the difficulty of manipulating factors effectively in an economy) cannot be used to infer causation. There is middle ground; sometimes nature, or culture, sets up a pseudoexperiment for you.

But the main point is that statistical properties of a sample are not directly involved in being able to infer causation.
posted by Philosopher Dirtbike at 5:55 AM on July 13, 2011

The ability to infer causation is not directly related to having a "statistically valid" sample; rather, it is related to the research design itself. Experiments, in which a factor is manipulated (often at random) and the outcomes are observed, give the ability to infer causation.

Well, right because you're know you're controlling one of the variables yourself. I think when people say "correlation does not imply causation" they only mean when looking at data that's been gathered without doing experiments.

Not a logical implication, a statistical implication

What is the difference between a statistical implication and a logical implication, other then the addition of probability bounds rather then 1/0 truth or falsehood?
posted by delmoi at 6:23 AM on July 13, 2011

Sometimes the explanation is there is no explanation. But that's probably not what the folks funding your research are expecting.
posted by tommasz at 6:46 AM on July 13, 2011

The general issue that McGovern and Gelman are raising is how should researchers interpret (i.e., explain or model) their quantitative findings, regardless of the specific statistical technique used. The problem of explanation remains whether the pattern is found through descriptive statistics, correlation analysis, regression analysis (including instrumental variable techniques), or randomized controlled trials. Here’s an example (What Do We Learn from Narrow Randomized Studies?) of Gelman discussing an explanation given for the findings arising from an experiment. Gelman’s idea that there are always two models involved in quantitative research is very useful: “I would separate the conceptually simple statistical models that are crucial to understanding evidence in any complex-data setting, from the economics (or, more generally, social science) models that are needed to apply empirical correlations to real-world decisions.”
posted by Jasper Friendly Bear at 7:05 AM on July 13, 2011 [1 favorite]

I'm afraid I haven't had time to read any of the FPP links, yet, but related to the correlation/causation question people are discussing, "Tackling the widespread and critical impact of batch effects in high-throughput data" is worth a read. Abstract:

High-throughput technologies are widely used, for example to assay genetic variants, gene and protein expression, and epigenetic modifications. One often overlooked complication with such studies is batch effects, which occur because measurements are affected by laboratory conditions, reagent lots and personnel differences. This becomes a major problem when batch effects are correlated with an outcome of interest and lead to incorrect conclusions. Using both published studies and our own analyses, we argue that batch effects (as well as other technical and biological artefacts) are widespread and critical to address. We review experimental and computational approaches for doing so.

When you realize the extent of the biological research resources which have been committed to experimental designs which ignore this elementary source of spurious correlation, it's really kind of gobsmacking. (Here's a more specific example.)
posted by Coventry at 7:12 AM on July 13, 2011

Ladies and gentlemen, I have discovered that I can easily refute any statement if I interpret that statement in a certain way.

Example 1
Statement: Neither lender nor borrower be.
Me: This statement means that lenders and borrowers do not exist. This is clearly false.

Example 2
Statement: Correlation does not imply causation.
...
posted by -jf- at 7:16 AM on July 13, 2011

The point is, you need to know the intended interpretation. "Correlation does not imply causation" means something like this: if you find that variable A and variable B are correlated, that does not necessarily mean that A caused B nor that B caused A.
posted by -jf- at 7:25 AM on July 13, 2011 [1 favorite]

-jf-, I assume that you are talking about a first order, direct causation?
posted by kuatto at 7:27 AM on July 13, 2011

There is the old joke that economists are like blind javelin throwers. They seldom hit the mark but they hold your attention.
posted by Bitter soylent at 7:30 AM on July 13, 2011

unigolyn: "Economics are modern day alchemy.

Scientific sounding terminology, activities that look like research and analysis, but in the end it's just handwaving accompanied by technobabble, trying to convince people that they can turn goose shit into gold."

And that's the magic - get enough people to believe it and that shit (economics) turns into gold! But alchemy is a great fucking term. I always laugh when people say economics is a science.
posted by symbioid at 7:31 AM on July 13, 2011

Correlation does not imply causation, but correlation is correlated with causation.
posted by grouse at 7:34 AM on July 13, 2011

The silliness of an argument correlates strongly with the semantic uncertainty in the proposition being considered. (Nail down what you mean by causation, and the dispute goes away.)
posted by Coventry at 7:39 AM on July 13, 2011

Question for Delmoi, (viz Coventry above)

How do you arrive at your causal categories? For instance in the example of the nail and the tire, why are we not talking about the atoms in the rubber splitting apart with some certainty and the crystalline structure of the steel in the nail holding firm? It seems you are operating in a purely logical domain with a very restrictive ontology.

To put it another way, why is there not a logical proposition for each atom in the nail? Have you chosen to ignore that detail in your model?
posted by kuatto at 7:48 AM on July 13, 2011 [1 favorite]

Nail down what you mean by causation, and the dispute goes away.

Literally no one who has ever seriously studied this subject has been able to find a rigorous way to do that, though.

The idea of causation is largely axiomatic--there's no rigorous way to prove that anything causes anything else; any correlation of two events or phenomena could merely be coincidence no matter how many recurrences of the coincidence are observed. So, in practice, causation can always formally be analyzed as mere correlation. That's what David Hume pointed out long, long ago, and no one has ever satisfactorily answered him, as far as I know.

Even real causation (assuming such a thing exists) would under observation be essentially indistinguishable from mere correlation.

Reasoning can only take us so far; at a certain point, we have to accept uncertainty and use discriminating wisdom (or if you like, "common sense") to arrive at our conclusions. There's no getting around the fact that, at a certain point, you just have to use higher-level judgment, even though such higher-level mental functions aren't completely understood.
posted by saulgoodman at 7:52 AM on July 13, 2011 [4 favorites]

kuatto: that's a good point. that's also along the same lines as Hume's critique of the epistemology of causality.
posted by saulgoodman at 7:53 AM on July 13, 2011

kuatto: Have you chosen to ignore that detail in your model?

It is the nature of all models, by their very definition to "ignore details". The model that does not ignore any details is called The Universe.
posted by mondo dentro at 8:02 AM on July 13, 2011 [3 favorites]

saulgoodman: That's true, but when "correlation does not imply causation" is invoked, the meaning of causation is usually pretty clear in that specific context. For instance, in biology experiments the batch-effect explanation competes with the explanation that the observed correlations are caused by some relatively general biological mechanism. A batch correlation is a kind of causal link, but one which applies only to the samples in the experiment, so you can't generalize to a cause in the population of inference and it's not a very interesting result.
posted by Coventry at 8:03 AM on July 13, 2011

(Part of the problem is that this thread is looking at the "correlation is causation" proposition in very general terms, which means it's poorly defined.)
posted by Coventry at 8:04 AM on July 13, 2011

By the way: fabulous FPP, and great comments. The interplay of the various scientific, empirical, and philosophical aspects of this issue are being well represented by MeFites.
posted by mondo dentro at 8:05 AM on July 13, 2011

If there is some cause that causes both of them, then they have a causal link, just an indirect one. There is a path from one to the other.

This isn't right, I think. If A causes B and A also causes C, then there is no path from B to C; the paths go the wrong way. On the other hand, if your notion of "causally linked" is symmetric, so that two events are said to be "causally linked" whenever they have a common cause, then everything is causally linked to everything else, since all are caused by the creation of the universe, and the notion becomes vacuous.

But this gets away a bit from the FPP, so let me say also that Andrew Gelman's blog is a national treasure.
posted by escabeche at 8:15 AM on July 13, 2011 [1 favorite]

There's always Granger causality
posted by dismas at 8:21 AM on July 13, 2011 [1 favorite]

"Correlation does not imply causation" means something like this: if you find that variable A and variable B are correlated, that does not necessarily mean that A caused B nor that B caused A.

-jf-, I assume that you are talking about a first order, direct causation?

No, it's true even if you broaden "causation" to "second-order" or "indirect" causation, as escabeche explains.
posted by John Cohen at 8:46 AM on July 13, 2011

Friendmanite fools,

A telling friedian slip, my freud.

Dude makes up economic theories based on his misunderstandings of what his friends tell him.
posted by Herodios at 8:50 AM on July 13, 2011 [1 favorite]

For me, personally, I feel like a lot of popular economists are more than happy to wander off the disciplinary trail and apply their economic discourse to any discipline, any problem, any setting with a kind of blithe assurance that because their research has some maths in it, it's a science, unlike those silly mystical sociologists, political scientists, anthropologists etc that have preceded them.

If I were a professional sociologist, anthropologist, political scientist, etc. I'd be plotting with my colleagues to colonize economics the way the economists have colonized every other field.

It's the only way to get at some of those sweet Nobel Kronor.
 
posted by Herodios at 8:59 AM on July 13, 2011

Nail down what you mean by causation, and the dispute goes away.

Related: Feinmann on causation in physics, and how difficult questions of "why" are.
posted by Philosopher Dirtbike at 9:12 AM on July 13, 2011

Mike McGovern's response (linked to in the OP) to Collier's work is pretty compelling as a reality check. From the examples--assuming McGovern is not selectively quoting out of context--it seems that Collier often waves his hands around and makes up motivations for people that fit his theories, and he isn't rigorous about fact-checking when creating a narrative about historical events. The picture McGovern paints of Collier reminds me of evolutionary psychologists and their "just-so" stories.

McGovern on Collier's explanation of what rebel soldiers were thinking when they joined the cause:

How do we know them [the hypothesized motivations supplied by Collier] to be true? [The motivations] must either come from conversations with the fighters themselves, a type of source that is usually excluded from Collier’s account ... or from the author’s own imagination. There are many aspects left unexplored, and no justification given for privileging one explanation over another.

That's a strong criticism. But McGovern introduces his criticism by saying, At one point, while summarizing Jeremy Weinstein’s work on rebel group recruitment in Mozambique and Sierra Leone, [Collier] lapses into an imaginary account of would-be rebels’ states of mind... Is the problem Collier, imposing an imaginary viewpoint on people he knows nothing about, or is the problem that Collier trusted Weinstein's research and attempted to restate it in simpler terms? It's not clear from McGovern's response, and I've read nothing of Collier or Weinstein myself.

McGovern on the fuzziness of Collier's facts:

I will not belabor the point here, but there are basic factual errors about who did what to whom in almost every paragraph of the chapter from the point where he begins to analyze the cascade of events that began with the country’s 1999 coup d’état. Of these, two bear special mention....

I would hope that anyone arguing for more military intervention in impoverished, war-torn countries would, above all, seek to accurately understand the history of those countries and their worldview/culture/way of thinking. McGovern has me mostly convinced that those are two areas where Collier falls down.
posted by jsturgill at 9:20 AM on July 13, 2011

At the same time, I am struck by the extent to which economists, at least when they are writing about poor people in out of the way places, seem to rely on half-baked ethnographic insights, the kinds gleaned from corridor talk at meetings and by talking with taxi drivers on the way in from the airport and bartenders in business-class hotels.

-From McGovern's review essay

The level of indifference of presumed experts and policymakers toward the poor and developing countries continues to shock me. Their - and our - arrogance and disinterest in getting even the basic facts right on which to base their policies is the root of the problem. We see this in the microcosms of our own countries; the way we treat our own poor, our own homeless, our own children.
posted by Atrahasis at 9:36 AM on July 13, 2011 [1 favorite]

Scientific sounding terminology, activities that look like research and analysis, but in the end it's just handwaving accompanied by technobabble...

I'm not going to say that economics is a pure science, but what you described (except the part of turning shit into gold) is also the charge that's leveled against theoretical astrophysics or evolutionary biology. Of course, those two fields dont' have nearly the amount of money thrown at them as business/policy economics.
posted by FJT at 9:40 AM on July 13, 2011

Well, theoretical astrophysics at least produces testable results sometimes. When was the last time an article of faith in economic theory was successfully ruled out by contrary results? Economic ideas don't ever die, they just circulate forever, coming into and out of fashion.
posted by saulgoodman at 10:03 AM on July 13, 2011 [1 favorite]

Mercantilism, for one. That's probably the biggest one I can think of.
posted by FJT at 10:13 AM on July 13, 2011

And just what form of "proof" did the scientists of economics present in the lab to disprove mercantilism? The Wikipedia article on the subject notes that many Asian countries still adhere to and practice various aspects of mercantilism.

As far as I can tell, the practices that define what economists have dubbed "mercantilism" seem to have become unfashionable in the West, but they weren't theoretically disproven in the same way that, say, flat earth theory was. It's not like there's some specific, well-formulated system of economic thought with better, more easily testable assumptions to replace or augment mercantilism. It's not even clear to me that "theory" means the same thing when we talk economics as it does in the hard sciences, as mercantilism seems more like a set of political and social practices than anything rising to the level of a coherent "theory."
posted by saulgoodman at 11:12 AM on July 13, 2011 [1 favorite]

These passages from the "very interesting review" link are on point:

Looked at over the fifty-year span since the publication of W.W. Rostow’s The Stages of Economic Growth: A Non-Communist Manifesto, development economics as a field looks far more like literary criticism than like those natural sciences it emulates. As we contemplate the series of enthusiasms that have characterized different moments in development theory and policy, it looks an awful lot like movements from romantic to symbolist to modernist to beatnik poetry.

....

The difference between poets and economists, however, is that for poets, as for literary critics, there are rivalries and certainly individual claims to preeminence, but as a general rule, there is an acceptance that there are many ways to write a great poem, just as there are many enlightening ways to read any great poem. Bound as it is to the model of the natural sciences, economics cannot accept that there might be two incommensurable but equally valuable ways of explaining a given group of data points.

posted by saulgoodman at 12:16 PM on July 13, 2011

economics cannot accept that there might be two incommensurable but equally valuable ways of explaining a given group of data points.

I can't count the number of times I've had economists try to be funny with the following:

"Put two economists in a room, and you get three opinions."
"If economists had three hands they would say: On one hand, on the other hand, but on the third hand..."
posted by FJT at 5:07 PM on July 13, 2011

I'm not going to say that economics is a pure science, but what you described (except the part of turning shit into gold) is also the charge that's leveled against theoretical astrophysics or evolutionary biology. Of course, those two fields dont' have nearly the amount of money thrown at them as business/policy economics.

Well, it's hardly handwaving, at least in astrophysics. The difference here is that while people may propose ideas that are backed by somewhat flimsy evidence, their ideas will have to withstand further inquiry.

Now, I'd have to agree that string/M theorists are kind of like economists, with The Elegant Universe being the Freakanomics equivalent. The difference between the groups is that string theorists would give up their ideas if they ever became falsifiable (and were promptly falsified). Economists will stick to their guns as long as they can keep doing statistical hocus pocus to produce superficially plausible numbers, i.e. forever.
posted by unigolyn at 11:13 PM on July 13, 2011

It used to be that I thought the problem was that you couldn't do experiments in social science, or at least, not easily. Science without repeatable experiments is really at loose ends: economics, political science, sociology, evolutionary biology, even cosmology and particle physics -- they all have vastly more researchers than they have experiments to support it all. A healthy science works by proposing a theory, proposing a clever test, designing and executing the experiment, and then passing that on to everyone else to execute in their own lab. Without that infinity of data -- the replicable experiment -- you're stuck reworking the same datasets over and over, with all the problems of over-fitting and consistency-with-multiple-theories that that entails. When there are too many scientists working in these sorts of fields, everything goes a bit wonky, and you get a vastly inflated ratio of theories to empirical proofs. There are of course lots of ways of getting around that -- natural experiments, new datasets and surveys, field experiments, new econometric methods -- but those things are a lot harder than coming up with a new experiment, or even building a new device that allows a new kind of experiment.

But after working a bit longer in social science, I've realized that economics has self-inflicted problems that go way beyond these. For one, huge numbers of them would never grant that most of their theories are empirically unvalidated and probably unvalidatable -- unlike, say, string theorists. Second, as the articles linked here discuss, they are much more loyal to the procedure of rational choice plus differential equation manipulation than they are to what little empirical validation there does exist. But third and most importantly, many economic theories (like rational addiction) do make quite specific micro-predictions than can be very much tested in the lab with human subjects. As Rogeberg points out, rational addiction, to the extent it has been tested in the lab, has been thoroughly disproven. Since rational choice theories are so often utterly implausible, it's not surprising that economists have resisted testing their foundations systematically in the lab. But it does make economics uniquely anti-empirical. Thankfully though, as experiments become more popular and people realize they don't have to disprove the super-technical theories if they can disprove the micro-foundations, the hegemony may be on the wane. I'd like to think the utter embarrassment of fresh water macro since 2008 also helps -- but alas, there you have the single-data-set, multiple-interpretation problem, even though the Keynesian version seems so obviously superior.
posted by chortly at 12:08 AM on July 14, 2011 [2 favorites]