The prophets prophesy falsely, and the priests bear rule by their means; and my people love to have it so: and what will ye do in the end thereof?

May 4, 2011 1:19 PM

In this paper, we report on the first-ever test of the accuracy of figures who made political predictions. We sampled the predictions of 26 individuals.... We discovered that a few factors impacted a prediction's accuracy. The first is whether or not the prediction is a conditional; conditional predictions were more likely to not come true. The second was partisanship; liberals were more likely than conservatives to predict correctly. The final significant factor in a prediction's outcome....

[PDF] Are Talking Heads Blowing Hot Air?

The top prognosticators – led by New York Times columnist Paul Krugman – scored above five points and were labeled “Good,” while those scoring between zero and five were “Bad.” Anyone scoring less than zero (which was possible because prognosticators lost points for inaccurate predictions) were put into “The Ugly” category. Syndicated columnist Cal Thomas came up short and scored the lowest of the 26.

posted by orthogonality (41 comments total)

They're talking alot, but they're not saying anything.
posted by Bromius at 1:24 PM on May 4, 2011 [17 favorites]

Buncha Nostradumbasses.
posted by Sys Rq at 1:26 PM on May 4, 2011 [11 favorites]

Finally, those prognosticators with a law degree were more likely to be wrong.

Oh, I know who you're talking about.
posted by fuq at 1:31 PM on May 4, 2011 [4 favorites]

And obviously a conditional prediction is going to be less accurate; it's really two predictions, with the accuracy of the second entirely dependent on the accuracy of the first -- and even then, the second might not be accurate. It's like aiming your shot before the target is visible.
posted by Sys Rq at 1:34 PM on May 4, 2011 [2 favorites]

Glad to see Tom Friedman way down there at the bottom.
posted by thsmchnekllsfascists at 1:40 PM on May 4, 2011 [2 favorites]

I like the fact that Paul Krugman came out on top. He seems like someone who both has a very good sense of how economics and politics relate, but also willing to tell the truth even when it's unpopular. Lots of people in politics just say whatever it is they need to say for their party to do better. And then you have another class of contrarians who just say the opposite of whatever everyone else thinks regardless of how wrong it is.
posted by delmoi at 1:41 PM on May 4, 2011 [9 favorites]

extrapolation using mathematical models typically does better than human predictors

And the followers of Hari Seldon nod approvingly.
posted by chambers at 1:42 PM on May 4, 2011 [15 favorites]

Glad to see Tom Friedman way down there at the bottom.

Give it a few Friedman Units and everything should be better for him.
posted by NoMich at 1:44 PM on May 4, 2011 [4 favorites]

The earth is flat, this I know, because my spirit level tells me so.
posted by Hoopo at 1:47 PM on May 4, 2011 [1 favorite]

The final significant factor in a prediction's outcome was having a law degree; lawyers predicted incorrectly more often.

Did you ever ask a lawyer the time of day? He told you how to make a watch, didn't he? Ever ask a lawyer how to get to Mr. Jones' house in the country? You got lost, didn't you?
posted by benzenedream at 1:50 PM on May 4, 2011 [2 favorites]

Sys Rq: "And obviously a conditional prediction is going to be less accurate; it's really two predictions, with the accuracy of the second entirely dependent on the accuracy of the first "

Yes, this is discussed briefly in the hamilton.edu link. I'm surprised that William Kristol isn't listed -- or is he retired and in the Hall of Shame?
posted by boo_radley at 1:52 PM on May 4, 2011

I'm surprised that William Kristol isn't listed

Christ that would have skewed the results.
posted by thsmchnekllsfascists at 1:59 PM on May 4, 2011 [3 favorites]

It seems like they have a pretty significant conflating factor here by doing this study on the 2008 election. Conservative pundits routinely predict conservative candidates will win, and liberal pundits routinely predict liberal candidates will win, regardless of what the actual facts on the ground are - saying a candidate will win is advocacy for them, and pundits advocate for their political viewpoint.

With the sweeping Democratic victories in 2008, all the liberal columnists got an automatic accuracy bonus by virtue of being on the winning side for that particular election. I don't doubt that Krugman's a genius and Will's an idiot, but the study needs to span more than a single one-sided election to make these numbers mean anything. You could probably re-run the study on the 2010 midterms and watch Gingrich flip from moron to maven as his continuous stream of "The GOP candidate will win" predictions came true.
posted by 0xFCAF at 1:59 PM on May 4, 2011 [12 favorites]

I came in just to say the same thing re: Bill Kristol.
posted by zoinks at 2:11 PM on May 4, 2011

Whoever decided to put those big color photos and, worse yet, the analog liberal-conservative "meters" in the pdf really trvialized their own work; all of that should have been a chart (and a really savvy author would have put it up as a web page, with some javascript to allow readers to sort by any column).
posted by orthogonality at 2:12 PM on May 4, 2011 [1 favorite]

Ah, never mind, they actually did account for this (missed this on first skim):

Our first instinct was that the election outcome (with Democrats and Barack Obama winning by large margins) could have had a confounding effect on the partisanship results. Democrats predicted optimistically that their candidates would win election, and the election swung in their favor, thus making their rhetorically-driven predictions correct. In order to decipher whether or not the election had skewed our results, we decided to run the regression again, this time excluding all the predictions having to do with the November election. Partisanship was once again statistically significant, showing that the liberal prognosticators had actually predicted more accurately than their Republican counterparts during our time period

Democrats really are smarter than Republicans. We now return to your regularly scheduled programming.
posted by 0xFCAF at 2:12 PM on May 4, 2011 [12 favorites]

It seems like they have a pretty significant conflating factor here by doing this study on the 2008 election. Conservative pundits routinely predict conservative candidates will win, a...

The study authors claim to have accounted for that, though they word it awkwardly:

According to our regression analysis, liberals are better predictors than conservatives—even when taking out the Presidential and Congressional election questions. Whether this holds true from election season to election season needs further evaluation; liberals may have implicit benefits from Obama winning the 2008 election.

I'm not sure why, if liberals do better even when "Presidential and Congressional election questions" are taken out, they then qualify that "liberals may have implicit benefits from Obama winning the 2008 election."

I think that means, "liberals do better when election predictions are not included in the mix, and they still did better in '08 when election predictions are included."
posted by orthogonality at 2:17 PM on May 4, 2011

The fact that they include politicians hurts the study, I think. Of course conservative politicians are going to be rosy going into the 2008 elections, and of course liberal ones will advocate that we need a change of direction. If you take out the politicians, you have 3 good liberals and 2 good conservatives. Hardly convincing.
posted by Vhanudux at 2:19 PM on May 4, 2011

Sure like to see Krugman in the "better then coin" list. Apparently he has been agreeing with me most of the times, even before the day I was born - good for him ;) ...but then again, what is the difference between a coin and an human being, when possible events are either A or not A ?

Maybe some expert in probability theory may enlighten me on this subject: if expert are asked to predict the outcome of a phenomenon which admits only two outcomes, A or !A, wouldn't that basically reduce their abilities to that of a coin? In other words, assuing a coin knows exactly nothing (being a piece of metal), shouldn't the coin be able to predict the outcomes exactly as well as an human being would, if the odds of the examined events are indeed by their own nature close to 50/50?
posted by elpapacito at 2:24 PM on May 4, 2011

I agree with 0xFCAF.

In addition, pundits and politicians often don't predict what they think is true. They say what they want to be true, or what they think will motivate their base. For example, if the right wing told the Tea Party "2010 is a lost cause," they'd probably discourage a lot of voters from going. Instead, they said it was going to be close, but they could make sweeping victories if they got out the vote, which is very motivating, and a self-fulfilling prophecy.

Add in the factor that saying what your audience wants gets you viewers and/or fundraising money, and you have a clear conflict of interest if you really expect pundits/politicians to be prognosticators. But we don't. We see them as sources of spin.

It'd be more interesting to see the case with pollsters, but there's already a ton of studies on that, and FiveThirtyEight did great reporting on the issue for laypeople and used statistics and meta-analysis to get fairly accurate predictions on recent elections.
posted by mccarty.tim at 2:32 PM on May 4, 2011 [1 favorite]

Two words: Sampling size.

It worries me (well, okay, not really) that five college seniors might believe that this silliness is at all meaningful.

Assuming, of course, that this is not largely a practical joke
posted by IndigoJones at 2:38 PM on May 4, 2011

Sys Rq: "And obviously a conditional prediction is going to be less accurate; it's really two predictions, with the accuracy of the second entirely dependent on the accuracy of the first -- and even then, the second might not be accurate. It's like aiming your shot before the target is visible."

They did account for it somewhat. They introduced a variable PredTrue that indicates the truth of the first prediction. So if it definitely didn't happen, you don't get penalized for predicting what comes next:
"we decided to omit predictions for which the PredTrue variable value was 0, or in other words, the conditional part of the prediction statement did not end up happening. When the conditional portion of the prediction (the “if” portion) is not accurate, then the rest of the statement (the “then” part) is by default inaccurate as well."

I would quibble with their interpretation of conditionals -- when the if clause is wrong, the statement as a whole remains true. But the math of their methodology seems correct to me.
posted by pwnguin at 2:39 PM on May 4, 2011 [1 favorite]

See also Why Foxes Are Better Forecasters Than Hedgehogs (video).
posted by Skorgu at 2:44 PM on May 4, 2011 [1 favorite]

Two words: Sampling size.

This is such a knee-jerk and meaningless objection; it's basically noise. What precisely is your objection?
posted by Justinian at 3:28 PM on May 4, 2011

lawyers predicted incorrectly more often

I've never thought of lawyers as expert predictors of the future, and would be inclined to avoid one that claimed to be such. The skill of a lawyer is in foreseeing a variety of possibilities, developing contingency plans for them, and being willing to accept a high degree of uncertainty. Lawyers' general reputation from risk aversity surely stems from a painful awareness of how unreliable forecasting can be.
posted by anigbrowl at 3:42 PM on May 4, 2011

Two words: Sampling size.

It is usually widely believed that the large the size of the sample, the more accurate the estimation or analysis. That's not always the case; self-selected/non random samples, poorly stratified samples may lead to wrong conclusion regardless of how large the sample size is.
posted by elpapacito at 3:46 PM on May 4, 2011

If I understood what that paper was, it was basically a student assignment to demonstrate that they had grasped some or other set of concepts and were able to do some analysis. i.e. Not meant as a serious proof of anything.

The main thing I discovered from it was learning about the work of Tetlock.

Some good things I turned up looking for more info:

- New Yorker
- A nice blog post
- A Long Now seminar with Tetlock
posted by philipy at 4:51 PM on May 4, 2011

Here's a video of the Long Now talk I mentioned in the last comment.

I haven't seen all of it yet, but it looks well worth a watch if you are interested in how accurate political judgments, including your own, are likely to prove.
posted by philipy at 5:08 PM on May 4, 2011

I took intro to polisci from a professor who was a bit of a showboat, and something of a public pundit in the state.

The first day of the course was a week before the 1996 presidential election. In a dramatic gesture he waved a white envelope in the air and said something like: "Political science is a real science. To demonstrate, this envelope contains the predicted results of the coiming election. Tonight I'm going to put it in a safety deposit box and I'll be opening it in front of you a week from now." At the end of the hour he proudly assigned us the task of watching his TV appearance later that evening.

He never brought up the supposed prediction again. At the end of the course I asked him about the envelope.

"WHAT envelope?"
"The one in the deposit box. With the election results."

In exactly the tone of voice you'd expect from a busted unrepentant con man he replied "Ohhhh.. THAT envelope!" Many laughs were had.


After the revolution they'll all be hung, of course.
posted by clarknova at 6:50 PM on May 4, 2011 [4 favorites]

This may be a flawed study, but they got the part about Joe Lieberman being full of shit right.
posted by jefbla at 9:14 PM on May 4, 2011 [3 favorites]

Pundits and politicians don't prosper when their predictions are correct; they prosper when their predictions are believed.
posted by BitterOldPunk at 9:20 PM on May 4, 2011 [2 favorites]

The first day of the course was a week before the 1996 presidential election.

A week before the election in '96, it was obvious Bob Dole was going to lose terribly, so obvious that I for various personal reasons gave him my vote out of sympathy.

And the polls were predicting an even bigger Clinton victory than actually occurred.

There's no way a polisci professor, much less one with a reputation as a pundit, gets that one wrong.
posted by orthogonality at 9:50 PM on May 4, 2011

elpapacito:

If you want to predict whether it will rain today, you may consult a coin, as you are certainly free to do so. It will come up either Rain or Not Rain, with each outcome having a 50% chance. This chance does not change whether you are in the mojave desert or the amazon during monsoon season, but a reasonable prognosticator would be able to account for such things.

The reasonable prognosticator would also look to the sky. Are there clouds? Do they seem threatening? S/he would check audible signals as well. Do I hear thunder?

Even more reasonably, they would check temperature and barometric pressure, and then check those results against past data. Essentially, the idea reasonable prognosticator would become a meteorologist, and while his predictions would not be 100% accurate, most likely, they would be better by a damn sight than flipping a coin.

But now, picture yourself in a location which gets roughly half of its economy from agriculture and the other half from tourism. Ideally, the meteorologist stays impartial, merely reporting the facts, but now the facts themselves have an effect on people's livelihoods and happiness to a greater, and more divided, amount.

What happens if the two competing publications in that town are The Daily Almanac and Surf's Up? What happens if the elected officials in the town have access to an even minimally effective weather machine? What if the majority of residents are subsistence farmers but a small minority are very rich rental agents? What if the town is in a drought phase or flood phase?

And what if the readers of the local publications start to care less about accuracy and more about feeling happy about their state every morning?

So yeah, expert prognosticators are obviously better than a coin flip, until human nature enters the equation.
posted by Navelgazer at 10:13 PM on May 4, 2011 [1 favorite]

There's no way a polisci professor, much less one with a reputation as a pundit, gets that one wrong.

No, I think the real lesson is that there was no prediction, it's not a science, and his life was a lie, but as long as everyone played along we could have a great time.

In the new Democratic Socialist Republic, shameless political & academic hypocrites will answer to the hangman. Unconscious hypocrisy will only earn you the final comfort of the the hypobaric chamber.
posted by clarknova at 10:14 PM on May 4, 2011

We now return to your regularly scheduled programming.

I'm looking forward to the day when that phrase has no meaning.
posted by telstar at 3:04 AM on May 5, 2011

Or at least only its ostensible meaning, eh.
posted by clarknova at 7:08 AM on May 5, 2011

In the new Democratic Socialist Republic, shameless political & academic hypocrites will answer to the hangman. Unconscious hypocrisy will only earn you the final comfort of the the hypobaric chamber.

What the fuck?
posted by grubi at 8:33 AM on May 5, 2011

Conditional predictions should be more accurate. After all, if I predict "all life on earth will die in 2012", that's unlikely to be true, but if I predict "if a huge meteor hits the earth in 2012, all life on earth will die in 2012", that's far more likely to be true.

I'm surprised that they found the reverse, even accounting for the fact that they just didn't count the prediction at all if the conditional were false.

This might tell us something about the psychology of making predictions, though I don't know what yet. It's something to investigate.

Perhaps what's going on is something like this: instead of generating a prediction, then adding conditional clauses to qualify that prediction, people start by assuming the conditional, then attempt to make a prediction based on that assumption. Naively, we might think that this would give accuracy equal to non-conditional predictions. But it may be that when people assume the conditional, they're also (consciously or not) assuming a bunch of other crap that they believe must be the case in order to fulfill the condition, and some of those assumptions may be incorrect. Those assumptions would screw up the subsequent prediction-making process.
posted by Jpfed at 8:51 AM on May 5, 2011

What the fuck?

Just some wishful ideation. Pay it no mind as you watch Helpful Political Reeducation Film #2373, comrade.
posted by clarknova at 9:21 AM on May 5, 2011

Jpfed: "
I'm surprised that they found the reverse, even accounting for the fact that they just didn't count the prediction at all if the conditional were false."

Well, they only discounted the strongly didn't happen, and left in the just plain didn't happen and neutral. That alone may well jigger the facts.

But if handling conditionals correctly doesn't account for all of the reverse, I expect it's because people use conditionals as a hedging strategy for events they believe are improbable. So you say things like, "Trump will be the 2012 Republican candidate... if he wins the Vermont primary."
posted by pwnguin at 9:32 AM on May 5, 2011

lawyers predicted incorrectly more often

I think there's a nuance here, and folks can correct me if I have this wrong. But the "worst predictors" aren't "Lawyers" per se, they are people who hold a law degree. I don't mean to quibble, but there are a lot of folks out there with law degrees who've never practiced law.

I think what the authors are trying to say is more along the lines of: "the method of thinking that they teach in law school leads to poor prognostication." And law school is supposed to be about teaching you how to think in a certain way, amongst other skills of course.
posted by indiebass at 9:42 AM on May 5, 2011