Tags:





equal economic ignorance
December 3, 2011 6:03 PM   Subscribe

I was Wrong and So are You. I needed to retract the conclusions I’d trumpeted in The Wall Street Journal. The new results invalidated our original result: under the right circumstances, conservatives and libertarians were as likely as anyone on the left to give wrong answers to economic questions. The proper inference from our work is not that one group is more enlightened, or less. It’s that “myside bias”—the tendency to judge a statement according to how conveniently it fits with one’s settled position—is pervasive among all of America’s political groups. The bias is seen in the data, and in my actions.

In 2010, Daniel Klein published a Wall Street Journal op-ed entitled: Are You Smarter Than a Fifth Grader? based on his research survey that seemed to show that progressives were more ignorant about basic economics than conservatives. After realizing the interesting cognitive flaw in the study, he and his co-researcher did another survey and found that when it comes to economics, ideology does not convey an advantage.
posted by storybored (41 comments total) 9 users marked this as a favorite

I Was Wrong, and So Are You

Well, you're half right.
posted by Horace Rumpole at 6:07 PM on December 3, 2011 [15 favorites]


Was that an attempt to illustrate the point, Horace Rumpole?
posted by The World Famous at 6:09 PM on December 3, 2011 [7 favorites]


Did he control for reality having a left wing bias?

More seriously, I'm amazed they didn't recognise the issue with the first survey sooner.
posted by knapah at 6:12 PM on December 3, 2011


What I find particularly hard to believe is that Klein still seems to think the various assertions in the survey are indisputably true facts.

Example

"Third World workers working for American companies overseas are being exploited (unenlightened answer: agree)."

"Free trade leads to unemployment (unenlightened answer: agree)."

It doesn't take a fifth grader to spot the problems with marking these answers "unenlightened".
posted by mikeand1 at 6:14 PM on December 3, 2011 [29 favorites]


Of course, I'm sure many people will happily trumpet the initial findings and ignore the follow-up.
posted by rmd1023 at 6:14 PM on December 3, 2011


You lost me at "Wall Street journal op-ed".
posted by facetious at 6:22 PM on December 3, 2011 [9 favorites]


Was that an attempt to illustrate the point, Horace Rumpole?

There's nothing more honorable than admitting a mistake, and describing what you learned from that mistake. But the problems with this ideological-purity-test-masquerading-as-survey were obvious from the moment it was first trumpeted, and the damage it did cannot and will not be undone, as rmd1023 points out. He'd like to use "see, we're all biased" as a get out of jail free card, but that doesn't change the fact that his survey was a tendentious piece of bullshit.
posted by Horace Rumpole at 6:24 PM on December 3, 2011 [36 favorites]


This is one of those things that lends itself to lazy snark, but is actually pretty neat to see happen.

I agree that it seems obvious that the "exploited third world workers" question should never have been used if you want to have any sort of hope of vague objectivity. It's not at all hard to imagine that a worker might both be exploited while at the same time gaining some benefit. There was a bit in the BBC's Human Planet special about workers somewhere that walked into an active volcanic area to harvest sulfur. The air was, as you'd expect, highly toxic and acidic, and their safety equipment amounted to "an old t-shirt wrapped around their mouth and nose".

They'd carry out a couple hundred pounds of sulfur in baskets suspended from a pole across their back, of course developing these huge callused knobs on their shoulders and they'd only be able to do the job for a few years before things like lungs, eyes, and backs began to fall apart.

Were they getting exploited? Obviously.
Were they making more money than their peers that didn't do the work? Definitely.

The original experimental design also suffers from employee performance survey weakness, where they're measuring yes/no/other, but instead write a survey that provides significantly more granular data, which they then throw out. It's shitty, bad analysis and design when companies do it ("please mark all fives otherwise I get yelled at") and it's shitty, bad analysis and design here. It's easy to imagine someone that responds to any of these questions with a "slightly agree" answer that would choose to mark "other" if she knew that her real choices were yes/no/other.

Ironically, this all goes to lend even more weight to his most recent results and makes the linked article even more persuasive and valuable.

It also makes me somewhat - I dunno - melancholy about the progressive condition. Or at least my progressive condition. I know about confirmation bias. I know about groupthink. I know about the Milgram experiment. And so I have to constantly second-guess myself and wonder if I'm simply masking my own wrongness with certainty. If I get into a verbal discussion about something, I have a hard time dismissing any arguments out of hand, while sparring partners can torpedo delicate arguments pretty easily.

I feel like this happens a lot in the sphere of actual policy, too. If you want to be exhaustive with your evaluation of a position, you have to admit that there are costs for the things you want. But if you're convinced that you're Totally Correct, you can just sort of charge ahead while people like me are dithering about and trying to formulate a sentence with enough clauses and caveats to actually address a thing fairly.
posted by kavasa at 6:37 PM on December 3, 2011 [21 favorites]


Was that an attempt to illustrate the point, Horace Rumpole?

There's nothing more honorable than admitting a mistake, and describing what you learned from that mistake. But the problems with this ideological-purity-test-masquerading-as-survey were obvious from the moment it was first trumpeted, and the damage it did cannot and will not be undone, as rmd1023 points out. He'd like to use "see, we're all biased" as a get out of jail free card, but that doesn't change the fact that his survey was a tendentious piece of bullshit.

So . . . an illustration, but not an attempt?
posted by The World Famous at 6:40 PM on December 3, 2011 [1 favorite]


The original questions are all designed to challenge progressive economic ideas. The new ones, however, are not all designed to challenge conservative/libertarian economic ideas. We have: Klein is attempting to obscure the war on the American middle class by bringing in noneconomic issues on which liberals and libertarians often side against social conservatives and hoping we won't notice.
posted by Ralston McTodd at 6:48 PM on December 3, 2011 [8 favorites]


"[I]mmigrants reduce the economic well-being of American citizens ([unenlightened answer:] agree)"

There's also a not-uncommon liberal critique of illegal immigration (a distinction which the survey question elides) that it provides an exploitable permanent underclass that subverts the protections available to documented workers and depresses wages. Just because immigration is a conservative bugaboo doesn't make this a question that divides neatly on ideological lines.
posted by Horace Rumpole at 6:59 PM on December 3, 2011 [7 favorites]


So when is he writing a new op-ed for the Wall Street Journal to trumpet his new results?

Should I hold my breath?
posted by never used baby shoes at 7:13 PM on December 3, 2011


He asked the WSJ to publish a follow up op-Ed. They declined. That's how the article ended up in The Atlantic.
posted by to sir with millipedes at 7:16 PM on December 3, 2011 [5 favorites]


@kavasa: Isn't that Yeats all over again?

"The best lack all conviction, while the worst
Are full of passionate intensity."

I would say "certainty" there instead of "intensity" but there you go.

http://www.secretdoors.com/weavermoon/secondcoming.html
posted by aleph at 7:39 PM on December 3, 2011


Nope. Conservatives are still retards.

I'd like my "favorites" in twenties and fifties, thanks.
posted by codswallop at 8:05 PM on December 3, 2011 [1 favorite]


codswallop, no no no, that's not true. The bizarre thing is that there are a lot of very smart, educated conservatives. It makes you wonder how they manage to reconcile their beliefs with reality. For example, do they really think that people in other developed countries envy the American health care system? (Yes, they really believe that!) This article goes a long way to explaining these beliefs.

We liberals should guard against confirmation bias too.
posted by Loudmax at 8:45 PM on December 3, 2011 [2 favorites]


@codswallop: I like your furious certainty.

@Loudmax: Knew an EMT who came down from Canada who had a fierce hatred of "managed care" Medicine. One big reason he moved down here. From talking to him, mostly on an email list, I got a sense of the different "world" he lived in. The horror stories that went around his circle of all the huge waiting periods for life-critical care there. The waste, the corruption. All were fuel for his World view. With no balance of anything the system was good for or successes. When I pointed out some AMA articles that were critical of the U.S. system he was astonished. All the criticism of the U.S. system was from bleeding heart liberals who had no concept of reality, etc.

As I like to say, "Everyone walks around in their own little made-up version of Reality." With the important caveat: "Some are closer than others."
posted by aleph at 9:00 PM on December 3, 2011 [2 favorites]


It’s that “myside bias”—the tendency to judge a statement according to how conveniently it fits with one’s settled position—is pervasive among all of America’s political groups.

myside bias = American Exceptionalism
posted by stinkycheese at 9:45 PM on December 3, 2011


codswallop, no no no, that's not true.

I know, I was just funnin'. One of the things I reliably get from Metafilter is a snicker at religious certainty of the "reality based community".

The bizarre thing is that there are a lot of very smart, educated conservatives. It makes you wonder how they manage to reconcile their beliefs with reality.

Ah, there it is :)
posted by codswallop at 9:45 PM on December 3, 2011 [1 favorite]


Dataset please.
posted by quadog at 9:56 PM on December 3, 2011


The true wtf is that this fella still thinks or pretends to think economics (worse, a specific brand of economics) is an ideologically neutral discipline, that there can be answers that are objectively true and that questioning the underlying assumptions is like questioning the fundamentals of physics.
posted by MartinWisse at 11:08 PM on December 3, 2011 [8 favorites]


[I]mmigrants reduce the economic well-being of American citizens ([unenlightened answer:] agree)

Either agreeing or disagreeing with that statement would be unenlightened, since the issue is not as straightforward as "immigration hurts/helps Americans." Setting aside questions about how to best model the economy (which makes a difference on the theoretical side at least), many recent studies indicate that both legal and illegal immigration depresses wages for poor, unskilled native workers (many of the links are to pdfs), while at the same time, immigration increases the wages of higher-earning natives. Probably, the survey authors are equating the well-being of American citizens with the aggregate well-being of American citizens, since the positive wage effect on higher-earning citizens is larger than the negative effect on lower-earning citizens. But that doesn't seem to me to be the best way to read the question as-stated. If it were qualified with an "on average" or some such, then maybe. As far as I'm concerned, the question as stated does not have a correct agree-or-disagree answer, because the effect of immigration on native wages is not uniform across the native population.

Anyway, many of the questions in these surveys have this kind of problem. Some of them are worse, in that they ask questions that use politically loaded terms, like "exploited," in such a way that two people with different ideologies could agree about all of the facts and still disagree about the truth of the survey sentence.

And aside from what amounts to journalistic misconduct on the part of the Wall Street Journal, even if the Journal had printed the follow-up piece, cognitive psychology tells us that beliefs are sticky. People will be more likely to continue to believe the first set of results, even if they become aware of the second set.
posted by Jonathan Livengood at 12:03 AM on December 4, 2011 [1 favorite]


The #1 thing that any leadership group with a decided agenda can want is that the people's aspirations can be made to pull in a hundred directions at once - keeping them divided. Actions cancel one another; efforts are dissipated.

To contrive such an energetic but powerless vacuum - schisming repeatedly into endless splinters - is to make it possible for any significant momentum in a constant direction to become a nearly unstoppable force.
posted by Twang at 12:41 AM on December 4, 2011 [1 favorite]


I agree that it seems obvious that the "exploited third world workers" question should never have been used if you want to have any sort of hope of vague objectivity.

None of the questions should ever have been used, in the first place. This shit is not even undergraduate, and he purports to be an economist. The follow-up is just as bad. The fact that people touting "research" like this have tenure and work at universities in the world makes a mockery of both tenure, and the "prestige" associated with certain universities.

I would note most of his so-called economic "Facts", are not even just simplistic to the point of illiteracy, but many turn their backs on decades of proven economic orthdoxy, in a frothy waterfall of neocon free-market jism that has about as much in common with real world economics as the film Gladiator did to ancient Rome.
posted by smoke at 2:19 AM on December 4, 2011 [3 favorites]


Fuck Klein. He's a bullshit artist and the WSJ is Murdoch propaganda. Any idiot could have seen the "bias" (er, intentional skewed design ain't bias, it's "fraud") in the first survey.

Wonder if his reputational cost was too high? But really, fuck him.
posted by spitbull at 3:14 AM on December 4, 2011 [1 favorite]


And what this really shows is the truly unscientific stupidity of survey-based social psychology research. What intellectually vacuous, bankrupt crap.
posted by spitbull at 3:44 AM on December 4, 2011 [1 favorite]


Wait a second, this guy is a Professor of economics? Really?

He might want to talk to someone who has a clue about how to write a research survey. Also, to someone else who can explain to him the responsibility and importance of reflexive self-critique in the practice of research.

I suppose it helps when you have a journal that you edit to publish your ditherings in without feedback or peer review. Given the survey questions I'd assumed that he was the type of 'economist' that works at well funded 'think-tanks' publicising 'research' through 'newspapers'. And there goes my quota of scare quotes for the month.

On preview: Not all survey-based research is worthless, of course those that use it properly don't represent their results as 'scientific true facts' look I had a few left! and are very careful to consider how the questions are written and how they might be responded to before drawing conclusions from their data. I.e. the conclusion to the original piece would, if done properly, likely would have concluded that progressives disagree with doctrinaire neo-liberal economics. Of course that result doesn't get you an op-ed.
posted by Hello, I'm David McGahan at 3:54 AM on December 4, 2011


"Third World workers working for American companies overseas are being exploited (unenlightened answer: agree)."

"Free trade leads to unemployment (unenlightened answer: agree)."

It doesn't take a fifth grader to spot the problems with marking these answers "unenlightened".

and

Wait a second, this guy is a Professor of economics? Really?

A small note: probably the experience that did most to drive me leftwards was this: I had been reading a few books about the impact of the IMF and so-caled "free trade" on other countries, particularly in Latin America and the former Soviet Bloc. They were interesting, but I wasn't sure I agreed with them.

What really stood out to me was that these books pointed to genuine problems. Whether you looked at the problems they described as a matter of too much government or too little, they were talking about something that you would think economists would want to talk about.

Then I read a book about economics by a Professor at Oxford. He was strongly in favour of globalization and free trade and I assumed that he would have something to say about why its critics were wrong. I was shocked and dismayed to find that he didn't say anything at all. He described the people who protested in events like Seattle as being hippie wierdos, deliberate ignoramuses.

"My God," I thought. "He doesn't even bother to engage with what they have to say. Maybe he doesn't know the first thing about these people; or maybe he doesn't even have an answer."

I have a similar feeling looking at these statements.

It simply isn't the case that the "enlightened answer" to the question of whether or not free trade leads to unemployment is "disagree". This is a complex issue. Among other things, I suspect it is entirely possible for someone to be more "enlightened" (however you define the term) than the author of this survey. Similarly, whether or not third world workers working for American companies overseas are exploited is also a complex question, whose answer depends not just on economic theories but on actual circumstances.

So, to me, it isn't just that these points of view are partisan. It's that they are extraordinarily parochial. It would be perfectly okay to hold a conservative point of view if you had considered the viewpoints that challenge it and rejected them. But if you can honestly say that these are statements that someone "enlightened" would automatically agree with, then you are either shockingly ignorant or dishonest or intellectually lazy - and someone who has those traits, conservative or liberal, has no business being a professor of anything.
posted by lucien_reeve at 4:22 AM on December 4, 2011 [9 favorites]


The problem with this test isn't that it didn't present equal challenges to libertarians and liberals. The problem is that he thinks he can determine the value of someone's thought with a test that boils down tremendously complex issues to simplistic generalisations with crude yes-or-no answers.

For instance:

"Third World workers working for American companies overseas are being exploited (unenlightened answer: agree)."

That is extremely sloppy phrasing.

Which Third World countries? How do you define a Third World country anyway?

Workers in what capacity? Factory line workers? Foremen? Regional managers? What?

How do you classify exploitation? Exploited relative to their American equivalents, or to their compatriots working for locally-based businesses, or to international law, or to some objective standard of exploitation, and in which case, who defines it?

This isn't a question designed to test your actual knowledge or understanding. It's a basic 'Whose side are you on?' If you'd had a question that read, say, 'Assembly-line workers in USA-owned factories based in South America generally work in conditions that are more uncomfortable and unsafe than USA health and safety laws would permit for assembly-line workers in the US', then there would be a correct answer. Not an 'enlightened' or an 'unenlightened' one, but one that was factually correct. (I don't know what the answer would be, because, not being a test of ideology, that question requires proper data. As a liberal, my answer would be 'It wouldn't surprise me, but I'd like to have the information before I said something definite.')

This isn't a study. It's the kind of personality test you'd find at the back of a high school magazine.
posted by Kit W at 5:00 AM on December 4, 2011 [13 favorites]


I suppose it helps when you have a journal that you edit to publish your ditherings in without feedback or peer review

Yes, I noticed that as well. The right wing bubble extends to science as well.
posted by spitbull at 5:35 AM on December 4, 2011 [1 favorite]


How do you classify exploitation?

Exploitation is being paid less than your marginal product.

It's still a stupid question. It's not hard to imagine that some people in such factories, especially the relatively powerless like children and young women, are paid less than their marginal product.
posted by ROU_Xenophobe at 8:25 AM on December 4, 2011


There's nothing more honorable than admitting a mistake, and describing what you learned from that mistake. But the problems with this ideological-purity-test-masquerading-as-survey were obvious from the moment it was first trumpeted, and the damage it did cannot and will not be undone, as rmd1023 points out. He'd like to use "see, we're all biased" as a get out of jail free card, but that doesn't change the fact that his survey was a tendentious piece of bullshit.

I'd just like to thank Horace for introducing me to the word "tendentious".
posted by heathkit at 9:34 AM on December 4, 2011


No word better describes conservative political discourse in the age of Fox News, which was founded on the principle of presenting the most extreme political opinions as objective facts.
posted by Horace Rumpole at 10:00 AM on December 4, 2011


Daniel Klein is an economist at George Mason University. GMU's economics department is funded almost entirely by the Koch brothers, and control over academic appointments is run by a Koch-funded "advisory council."

GMU economists definitely deserve scare quotes around the "economist" title. You'll notice they appear as news sources a lot, and they produce a ton of popular books and a major podcast, but they do very little peer-reviewed research. Many of these guys wouldn't be qualified to teach at any other large research university, even disregarding their conservative viewpoint. It's the only example in the US of a corporation completely corrupting an entire academic department (though there are other cases of corporate-controlled endowed chairs). Needless to say, the school only interested Koch because happens to be in DC, and presumably the other schools there were too ethical to take their money.

Please don't tar all economists with the same brush. There are a lot of economists in the world. Probably 95% of them don't do anything that has any national policy implications whatsoever. Of the ones who do, the vast majority would not support absolutely unrestricted free trade with zero government regulation. But publicizing that point of view happens to be beneficial for some very very rich people, so those economists are the ones who get the big grants and the Wall Street Journal op-eds. It's not bribery exactly -- they really believe what they're saying -- but they drown out the other voices.

Given that situation, it's a small miracle that Klein even published a retraction. I hope he already has tenure.

I'm very disappointed in the Atlantic for not mentioning the GMU-Koch connection; I consider that unethical in itself.
posted by miyabo at 10:03 AM on December 4, 2011 [10 favorites]


Exploitation is being paid less than your marginal product.

Well, that's the Marxist definition of the word. By that standard, professional athletes and engineers at Google are being exploited. I think a more commonly accepted definition would be being paid a *lot* less than your marginal product, due to lack of alternatives or understanding.

Which just furthers the original point, that these questions are not constructed such anyone should pretend they have absolute "correct" or "enlightened" answers. The reasonable response to almost all of them is "Wait, what?"
posted by Bokononist at 10:04 AM on December 4, 2011 [6 favorites]


In an academic career that now extends past 20 years, if you count from the date of my first peer-reviewed publication (mid-grad school years), as I like to do, I have seen so very little survey-based social psychology research (much of which is done by economists and the even stranger perversion of social science that is "marketing" as a field of research) that wasn't designed to reliably "prove" the assumptions it presumed to test. It is a field lost in the mud of methodological minutiae -- whether to use brightly colored stamps for the return envelope for higher rates of return! -- and impervious to the sustained criticism it routinely receives from those of us on the qualitative side of social research, on the one hand, or on the neurobiological/cognitive side of behavioral science, on the other, for being patently and obviously full of shit, distilling the worst flaws of both qualitative social science (subjectivism, observer bias, limits of sampling, language dependency -- the things that make us call it qualitative in the first place, meaning the researcher is presumed to have a bias and account for it explicitly) and the worst flaws of reductionist behavioral science (insufficiently attentive to context, problems of meaning/value, overconfidence in the ability of discrete data to describe gradient phenomena, etc.).

That's why social attitudes survey research is an instrument of political rhetoric and entertainment ("news") to the extent that it is. And why it is so subject to fraud. Far more science goes into tuning this stuff to yield desired results predictably than goes into making sure it describes something actually out there in the world.

It is simply and factually evident that there are many factors other than abstracted political ideology that would determine any given respondent's ability to reason their way to the (ideologically) "correct" answer in this particular study. Obviously class, education, age, field of employment, exposure to various media, etc. all enter into what one "knows" about concepts as complex as immigration or international trade. And obviously, these same things overdetermine one's particular ideological stance toward simplifying these complex subjects into policy-implicating, binary distinctions, View A vs. View B.

Retch. It's the same market-driven pseudoscientific crap-for-hire "research" as you get out of political think tanks, marketing firms, and economists on the payroll of the major investment banks, informed by the same blinkered model of human decision making, emotional experience, or social identity.
posted by spitbull at 11:14 AM on December 4, 2011 [2 favorites]


GMU economists definitely deserve scare quotes around the "economist" title.

That's interesting - I did not know that. Thank you.

This sort of thing is why I find "think tanks" quite disturbing and distasteful - particularly the ones designed to defend the interests of the rich and powerful. I'm sure their defenders would characterise them as as a useful source of alternative points of view or research directly relevant to current political issues.

The problem is, though, in practice they often debase the coin of academic knowledge by flooding the market with too much bad quality research. This particular study is obviously flawed. But that isn't always the case. Sometimes the mistakes are only visible to experts.

It can be really quite difficult and time consuming to find out what is true and what is false on quite a large number of issues, many of them of considerable importance. This can waste a lot of time for any ordinary person who is trying to make an informed decision - which, in a democracy, we are all sometimes called upon to do.

Now, it might sound like I'm just complaining about having to learn about the world and use my own judgment. But honestly, I'm not. I like learning things. It's sort of my "happy place". What I don't like is being deceived.

Robert Heinlein had a famous line about how "specialisation is for insects", which sounds wonderful in theory, but runs into all sorts of problems in practice. We all have limited time on this Earth and in that limited time we can only become expert in so many things. Having to waste some of that finite time and energy battling through ideologically motivated and misleading pretend academic work is a great shame.

Of course, it is very hard to know what can be done about this. Free speech, a natural feeling that most people have that there are two sides to every story, a suspicion of self-proclaimed expertise or academic cliques... These things are not always wrong. I don't think that I know the answer to this problem, alas.
posted by lucien_reeve at 12:50 PM on December 4, 2011


Previously. I am forwarding Klein's piece to my conservative friend to see what she thinks -- she was quite convinced the original survey was valid, of course... Hardly a sweeter moment than a mea culpa from an antagonist.
posted by rleamon at 1:29 PM on December 4, 2011


How do you classify exploitation?

Exploitation is being paid less than your marginal product.

So are all productive volunteers exploited?
posted by John Cohen at 3:15 PM on December 4, 2011


They are self-exploitative. That's the nature of the idea of "volunteering," is it not?
posted by spitbull at 3:47 PM on December 4, 2011


"... he and his co-researcher did another survey and found that when it comes to economics, ideology does not convey an advantage."

I am wearing my surprised face all day.
posted by clvrmnky at 8:40 AM on December 5, 2011


« Older Rule no. 1: Catch the first Pokemon you encounter ...  |  Kodak's long fade to black.... Newer »


This thread has been archived and is closed to new comments