DEWEY DEFEATS TRUMAN
March 12, 2013 10:56 PM   Subscribe

Respected polling firm Gallup had a disastrous 2012, with a final presidential poll showing Romney 49-48 over Obama, some five points off of the final result. So what exactly went wrong?
posted by Chrysostom (85 comments total) 13 users marked this as a favorite
 
This is a great article -- especially for a Huffington Post link -- so I don't want to start the thread out too snarky, but I can't help myself seeing this:

And in January,Gallup and USA Today ended their 20-year polling partnership. While both parties described the breakup as amicable,the pollster's misfire on the 2012 election loomed large in the background.

I've often asked "how shitty would reporting have to be before USA Today thought 'um, this isn't a brand we want to associate with'"; a younger, forensics-winning, Gallup-sider would never thought this.
posted by MCMikeNamara at 11:05 PM on March 12, 2013 [4 favorites]


I think it's getting harder and harder to reach random people who don't necessarily want to be reached for polls. There is so much more competition from awful robocalls and "polls" that are thinly-veiled marketing come-ons that I can't believe anyone in their right mind answers their phone anymore if they don't recognize the number. I know I don't.
posted by overleaf at 11:13 PM on March 12, 2013 [26 favorites]


They weren't five points off, they were 3.1 points off on Obama and 1.8 points off on Romney. The points don't add together. The Romney figure is within the margin of error. The Obama figure... well if you do enough polls, sometimes you hit the other 5% of the confidence interval.
posted by Jahaza at 11:14 PM on March 12, 2013 [6 favorites]


I've read half the article but it seems to be taking a long time to say that Gallup relied too much on telephone surveys.

I also still think that the establishment still has not come to terms with Obama's highly focused, highly efficient and data-driven effort to get the vote out. The dumb fucks.

It remains to be seen if subsequent candidates will be able to implement a similar campaign.
posted by KokuRyu at 11:17 PM on March 12, 2013 [3 favorites]


@KokuRyu"[t]he establishment still has not come to terms with Obama's highly focused, highly efficient and data-driven effort to get the vote out." How is Obama not a part of "the establishment"?
posted by koavf at 11:22 PM on March 12, 2013 [13 favorites]


They consistently skewed GOP on every poll this cycle and have for years. It wasn't a fluke, Jahaza, there's a bias in their methodology.
posted by empath at 11:23 PM on March 12, 2013 [19 favorites]


I did think to myself as I pressed "post" that, yeah, Obama got a lot of money, so the elites must love him, but I guess I was thinking of whatever consensus Romney and Gallup represent, if that makes any sense.
posted by KokuRyu at 11:41 PM on March 12, 2013 [4 favorites]


The Romney figure is within the margin of error.

For one poll, sure, but they took many of them, and they were all wrong, by fairly similar amounts, as I recall. That's no fluke, that's a systematic problem.
posted by Malor at 12:20 AM on March 13, 2013 [5 favorites]


Respected*

* according to Gallup poll
posted by BrotherCaine at 12:26 AM on March 13, 2013 [7 favorites]


It wasn't a fluke, Jahaza, there's a bias in their methodology.

Well then it is an obvious liberal bias. Putting the thumb on the scale for the GOP is more likely to suppress Republican turnout and increase Democratic turnout. If you think you're candidate is going to win there is less reason to leave the country club to go and stand in line with the plebes at the dingy local school. On the other hand, if you think your candidate is behind, but close, then you're more likely to make the effort.
posted by three blind mice at 1:32 AM on March 13, 2013


seems to be taking a long time to say that Gallup relied too much on telephone surveys.

that's not even close to what it is saying. It is asking why this telephone survey was wrong when other telephone surveys were more accurate.
posted by jacalata at 1:53 AM on March 13, 2013 [1 favorite]


Well then it is an obvious liberal bias.

Conversely, putting the thumb on the scale for the Democratic party would also be obvious liberal bias, because that would be trying to create the impression of momentum and get the Democratic base excited about Obama. Everything is evidence of liberal bias. Everything.

That's because all of the media has a liberal bias. Fox News, who favored Romney in much the same way that Gallup did? The same explanation holds — liberal bias. You didn't know about Fox News' hidden liberal agenda? You must be a victim of liberal bias.

There are names for propositions that can never contradicted by any evidence, and they aren't "science" and they aren't "based in reality".

(Is this a derail? Probably. Someday better than me will have to end it.)
posted by benito.strauss at 1:58 AM on March 13, 2013 [74 favorites]


Well then it is an obvious liberal bias

Are you being serious here with that thought?
posted by flaterik at 2:09 AM on March 13, 2013


I'm glad when polls go wrong. An unpredictable electorate is a less manipulable electorate.
posted by Segundus at 2:09 AM on March 13, 2013 [8 favorites]


An unpredictable electorate is a less manipulable electorate

That's what they want us to think!
posted by aubilenon at 2:13 AM on March 13, 2013 [5 favorites]


On the other hand, if you think your candidate is behind, but close, then you're more likely to make the effort.

Once again, Gallup had Romney at 49 and Obama at 48. I think that counts as a sufficiently slim lead to motivate GOP voters to show up and keep that 1% advantage.
posted by maudlin at 2:19 AM on March 13, 2013 [2 favorites]


Obama's turnout operation had to be a huge factor in this as well. This was a turnout election - if you just get your voters to cast their ballots, you'll win. The 2008 campaign for Obama was amazing to see as far as identifying voters abd getting them to turn out, but the 2012 campaign was even better, and much more thorough. This had to wreak havoc on likely voter screens. Romney never knew what hit him, and Gallup looked hopelessly stuck in the past.
posted by azpenguin at 2:36 AM on March 13, 2013 [2 favorites]


because that would be trying to create the impression of momentum and get the Democratic base excited about Obama. Everything is evidence of liberal bias. Everything.

That's because you're thinking politics and not maths. In the primaries the impression of momentum would have helped Romney - GOP voters were lukewarm to him. But in the general election when voting is more partisan the only reason to get out to vote is to keep the other guy from winning.

I think that counts as a sufficiently slim lead to motivate GOP voters to show up and keep that 1% advantage.

And that is a fair point which would suggest a LACK of bias in the Gallup results and simply faulty methods.

And let's face it, there was never that much Democratic enthusiasm behind the Drone Ranger in 2012. There were many more reasons to vote against Romney than for Obama and that's what won the day for him.
posted by three blind mice at 2:41 AM on March 13, 2013 [1 favorite]


Nate Silver said many times going into the election Gallup was fubar. He was right.
posted by professor plum with a rope at 2:41 AM on March 13, 2013 [2 favorites]


In a thread about how inaccurate Gallup's poll results were, using a Gallup poll to make a point seems sort of self-defeating.
posted by Elementary Penguin at 2:46 AM on March 13, 2013 [12 favorites]


Hell, Gallup did worse than Rasmussen! I mean, that takes skill.
posted by professor plum with a rope at 2:47 AM on March 13, 2013 [6 favorites]


"Well then it is an obvious liberal bias."

Uh-huh. As mentioned, us dirty libs have used advanced mind-control techniques on traditionally GOP-leaning pollsters. Just ask Rasmussen! Crazy obsese "unskewed polls" guy! Fuck me, just take it up with Karl Rove! Right-wing stupidity/magical thinking/non-rationality can't possibly explain it, it must be Nobama and his drones again!

"I'm glad when polls go wrong. An unpredictable electorate is a less manipulable electorate."

Rational adults like Nate Silver (who happens to lean left, but is more interested in his reputation for accuracy above all else) actually have this down to a tee, for now. Part of their methodology is to rate polls according to their historical accuracy. Gee, guess what -- it turns out that traditionally "liberal leaning" polls are also the most objectively accurate, in no small part because they call people with cell phones and not just the olds on their landlines. Republican leaning polls rely on an outdated, if not downright false, notion that "firing up the base" with "momentum" is how you do it.

IMO USian elections are going to become more and more predictable and make "centrist," "serious," "moderate" pundits like Peggy Noonan (OH HAI she's actually a life-long Republican and former Reagan speech writer!) cry into her fifth glass of wine.

The fact is, American conservatives need to embrace a paradigm shift in the American electorate. The other fact is, half of them aren't capable and half of them are deeply in denial.

And it's beautiful to watch.
posted by bardic at 2:51 AM on March 13, 2013 [2 favorites]


Also, you know who was, arguably, more accurate than Silver? Commie pinko Markos Moulitsas!

And who was really, really wrong about 2012? Pretty much every Republican alive!

One party, despite it's many flaws, is still grounded in an objective reality. The other is not. It's really as simple as that.
posted by bardic at 2:54 AM on March 13, 2013 [4 favorites]


Well then it is an obvious liberal bias

I wasn't talking about political bias.
posted by empath at 2:55 AM on March 13, 2013 [10 favorites]


" there was never that much Democratic enthusiasm"

Bullshit. Overall turnout was down but Obama actually did better in some regions and with some important demographics.

And dude, did you just link to Gallup to defend your point in a thread about how Gallup is constantly wrong?

Well played, sir.
posted by bardic at 2:58 AM on March 13, 2013 [12 favorites]


"In a thread about how inaccurate Gallup's poll results were, using a Gallup poll to make a point seems sort of self-defeating."

It says a hell of a lot, don't it?
posted by bardic at 3:00 AM on March 13, 2013


What really breaks my heart about Gallup's performance in the last election is that Gallup is the hero in the canonical explanation of why proper sampling is so important that is used in basically every Intro to Stats class ever. You just wonder what went wrong.
posted by Elementary Penguin at 3:11 AM on March 13, 2013 [3 favorites]


You just wonder what went wrong.

Over-reliance on an outdated model of the electorate, I'd wager. All things considered, this is kind of one of those "everything is amazing, but everyone complains" things that Louis CK talks about. It's kind of amazing that we live in a world where failing to predict the collective action of 130 million people within 3% of the final result is a colossal, unforgivable failure, if you think about it.
posted by empath at 3:19 AM on March 13, 2013 [6 favorites]


Here's the simple reason the polling numbers don't match the results.

The results are fabricated to deliver a predetermined result.
posted by mikelieman at 3:26 AM on March 13, 2013


The polls did match the results, except for Gallup, or at least a simple average of the polls did.
posted by empath at 3:27 AM on March 13, 2013


"You just wonder what went wrong."

It's also profit motive. You can become very rich in America by telling Republicans and their noise machine exactly what they want to hear.

So while FOX News, et al. started out as a GOP tool to help them win elections, it's sort of metastasized into the goal itself.

Basically, grifters gonna grift. Wing-nut welfare used to be merely tangential to getting Republicans elected. Given a soon-to-be-minority of older white males, the whole point now is simply to feed to machine.
posted by bardic at 3:37 AM on March 13, 2013 [8 favorites]


What really breaks my heart about Gallup's performance in the last election is that Gallup is the hero in the canonical explanation of why proper sampling is so important that is used in basically every Intro to Stats class ever. You just wonder what went wrong.

What went wrong is that Gallup went from being right because they chose the correct analysis to being wrong because they stuck with that analysis long past the point where it was appropriate. It's tough to remain nimble when you make the superiority of your method a part of your brand.
posted by belarius at 3:47 AM on March 13, 2013 [1 favorite]


It's also profit motive. You can become very rich in America by telling Republicans and their noise machine exactly what they want to hear.

This is it, exactly. We're at a point in American history when the management and business class has become a political block, and will relentlessly punish each other for breaking ranks politically with financial repercussions. If Gallup wants to keep making money, they will violate their integrity to keep the illusion of integrity with their customers - they won't remember that Gallup was wrong. They'll only remember that Gallup did as they were told; and then hire their performance-management consulting arm as a reward.
posted by Slap*Happy at 4:28 AM on March 13, 2013 [1 favorite]


One of the interesting statistics for me was that 15% of people rely only on a cell phone. Given that's been a trend for at least a decade, I'm surprised the number is still relatively small.
posted by CheeseDigestsAll at 4:54 AM on March 13, 2013 [2 favorites]


Two wonderful ironies here:

1) IIRC, Gallup made its original reputation on being ahead of the curve methodologically by using random sampling at a time when everyone else was using direct response and other biased methods.

2) We live in a world where an author can refer to 'data crunchers' as a group in opposition to pollsters, and it actually kind of makes sense.
posted by graphnerd at 5:15 AM on March 13, 2013


Public Policy Polling pretty much nailed all of the 2012 polls -- national and state. Their method? Call people and ask them who they're going to vote for (more or less). They realized that likely voter screens were skewing things too much, so they scrapped them.
posted by (Arsenio) Hall and (Warren) Oates at 5:15 AM on March 13, 2013 [1 favorite]


Crazy obsese "unskewed polls" guy!

Since when is it OK on Metafilter to gratuitously attack someone for being overweight?
posted by John Cohen at 5:16 AM on March 13, 2013 [5 favorites]


> Putting the thumb on the scale for the GOP is more likely to suppress Republican turnout and increase Democratic turnout.

The Romney campaign's Get Out The Vote system was a complete failure, both technolgically and logistically, and has been positively correlated to reduced GOP voter participation. That seems far more plausible than the Gallup poll's influence on voter turnout for either party. Especially since Gallup has been receding to pollster background noise, as merely one of dozens that get national media coverage every election cycle.
posted by ardgedee at 5:21 AM on March 13, 2013


Nate Silver had it right. Whats so complicated?
posted by sfts2 at 5:23 AM on March 13, 2013


If you think you're candidate is going to win there is less reason to leave the country club to go and stand in line with the plebes at the dingy local school.

Country Club? You've never met a real Republican voter, have you?
posted by rocket88 at 5:33 AM on March 13, 2013 [2 favorites]


Since when is it OK on Metafilter to gratuitously attack someone for being overweight?

At least since 2009.

You can become very rich in America by telling Republicans and their noise machine exactly what they want to hear.

Conversely....
posted by IndigoJones at 5:36 AM on March 13, 2013


Nate Silver had it right. Whats so complicated?

Without polls to aggregate, Nate Silver wouldn't have been able to do what he did. The aggregation balanced the house effects of various polls against each other, but he was still relying to a large degree on the soundness of the underlying polling methodologies.
posted by Area Man at 5:36 AM on March 13, 2013 [1 favorite]


Yeah, Nate Silver "had it right" by using data from polling companies that did actual, you know, polling, like North Carolina's Public Policy Polling. This article is a good addition to the one linked above:

How A Three-Man Polling Team Completely Nailed Their Election Prediction

While Nate Silver got a lot of attention for predicting the election through poll aggregation and statistics, Public Policy Polling nailed every state through first-hand polling.

It's strange the HuffPo piece doesn't mention them. The reason PPP doesn't show up on that 3rd graph as one of the "five other national pollsters" whose polls changed when they "narrowed from registered to likely voters" is that PPP doesn't fucking bother with anything other than what they consider to be likely voters in the first place. How do they screen them?

"We have a very simple likely voter screen," Jensen said, "'If you don't plan to vote in this fall's election, hang up now.'

Apparently, that works really, really well.

overleaf: There is so much more competition from awful robocalls

Public Policy Polling - again, the most accurate polling company in the US last cycle - uses nothing but robocalls.
posted by mediareport at 5:47 AM on March 13, 2013 [12 favorites]


Don't forget that part of the reason Gallup got it so wrong was the intervention of Hurricane Sandy shutting down polling operations for most of the week right before the election. That was when they were planning to slowly revert back to the mean, as they always do.
posted by gerryblog at 6:15 AM on March 13, 2013 [2 favorites]


Public Policy Polling - again, the most accurate polling company in the US last cycle - uses nothing but robocalls.

I was under the impression that robocalls (or I guess "pure" robocalls rather than predictive dialing) were banned from calling cell phones in the US. Is my insomniac brain just making things up? If not, how could PPP be accurate using just robocalls?
posted by Lemurrhea at 6:15 AM on March 13, 2013


This was a turnout election - if you just get your voters to cast their ballots, you'll win.

Every election is a turnout election if you're a Democrat, though.

How did I know we were going to lose the vote for unionisation last year? Forget the university's shenanigans. The 'election strategy' meeting called three weeks before was mostly about the importance of turnout. In other words, we had somehow gotten to this point, which had taken more than a year, and didn't know how you win elections.
posted by hoyland at 6:17 AM on March 13, 2013


I'm glad when polls go wrong. An unpredictable electorate is a less manipulable electorate.

I am always happy when the forces of data and statistics are proved ineffective in dealing with human beings.

The number of beans in a jar (or on a plate) don't have life-changing events, can't change their minds about being beans, nor cop an attitude about being counted.
 
posted by Herodios at 6:17 AM on March 13, 2013 [1 favorite]


I was under the impression that robocalls (or I guess "pure" robocalls rather than predictive dialing) were banned from calling cell phones in the US. Is my insomniac brain just making things up? If not, how could PPP be accurate using just robocalls?

It might be the case that people without land lines either aren't significantly different from people with them, or that people without landlines don't vote.
posted by empath at 6:19 AM on March 13, 2013


I am always happy when the forces of data and statistics are proved ineffective in dealing with human beings.

2012 was not one of those times. It was a deeply by-the-numbers election whose final results were almost exactly in line with average poll results near the election, and whose dramatic ups and downs only returned the election to almost exactly what longer-range forecasts predicted in summer/September.
posted by ROU_Xenophobe at 6:26 AM on March 13, 2013 [3 favorites]


Argh, folks, this is MetaFilter, you are better than this. The word "bias" doesn't have to mean "willful skewing". Bias creeps in in a number of ways, and Gallup's polling methodology simply failed to prevent the numbers from being skewed wrong.

Their methodology may subtly reflect the CEO's personal biases, or they might just be operating on wrong assumptions - in either case, I am willing to bet that Gallup wasn't trying to fudge the numbers.

TL;DR Never attribute to malice what is more easily explained by stupidity.
posted by Xoebe at 6:48 AM on March 13, 2013 [11 favorites]


I was under the impression that robocalls (or I guess "pure" robocalls rather than predictive dialing) were banned from calling cell phones in the US.

That's correct; you can't legally robocall a cell phone unless the person has given you prior consent.

how could PPP be accurate using just robocalls?

They use weighting to account for the differences between landline and cell users:

PPP doesn't call cell phones, meaning roughly 2-in-5 eligible voters have no chance of being contacted by their computers. PPP does weight their results by age, race and gender to account for these shortcomings, in addition to other sampling quirks. According to national trends, we can expect that not calling cell phones would require younger voters, minorities and males (who are less likely to own landline phones) to be weighted up, while older voters, whites and females (more likely to live in households with landlines) would be weighted down. "Weighting usually takes care of most of the problems associated with not calling cell phones," said PPP director Tom Jensen

PPP is very good at weighting, apparently. Bottom line is the results: they buck the conventional wisdom that you can't do good polling without cell phones, and they get better results than anyone else. *shrug* Hard to argue with that.
posted by mediareport at 6:56 AM on March 13, 2013 [1 favorite]


but he (Nate Silver) was still relying to a large degree on the soundness of the underlying polling methodologies.

As well as the unsound trends he identified and corrected for in his data sets.

(Where the real magic happens.)
posted by Skygazer at 7:04 AM on March 13, 2013 [1 favorite]


Bottom line is the results

Yeah fair enough. That is, indeed, impressive.
posted by Lemurrhea at 8:00 AM on March 13, 2013


I am always happy when the forces of data and statistics are proved ineffective in dealing with human beings.

There's a certain irony to this since there was a whole big blowup at the end about "NATE SILVER AND HIS 'NUMBERS' AND STATISTICS CAN'T COMPARE TO STORYLINES AND HORSE RACES!" from the punditocracy that went on right up until Silver's predictions absolutely crushed the pundits, leaving them muttering bitterly.
posted by Ghostride The Whip at 8:01 AM on March 13, 2013 [2 favorites]


I am always happy when the forces of data and statistics are proved ineffective in dealing with human beings.

So you are saying that you are rarely happy then.
posted by srboisvert at 8:19 AM on March 13, 2013 [1 favorite]


One of the interesting statistics for me was that 15% of people rely only on a cell phone. Given that's been a trend for at least a decade, I'm surprised the number is still relatively small.

It's sometimes hard to get DSL Internet access without a land line whether you want one or not. The thing is, I think the younger demographic is more likely to ignore land line calls (which are probably solicitors) or even turn their ringer off and have no answering machine.
posted by BrotherCaine at 8:35 AM on March 13, 2013


Out of curiosity, did anyone here ever get a "poll" call, and was it from a reputable/known polling company, or a partisan push-poll with leading questions designed to induce a desired response ?

I hang up on the lot of 'em. Poll me ? DIAF.
posted by k5.user at 8:43 AM on March 13, 2013 [1 favorite]


I have gotten both kinds (real and push poll). Probably half a dozen in 2012, presidential, Congressional, even local stuff.

Interestingly, I received one this time that clearly thought I still lived in Virginia. They never explicitly asked me if I did or if I intended to vote for one of the candidates, they just asked if I was following the race, etc., so I still answered it.
posted by Chrysostom at 8:53 AM on March 13, 2013


that went on right up until Silver's predictions absolutely crushed the pundits, leaving them muttering bitterly.

And, having learned nothing, ready to ride out for the next campaign, knowing that absolutely no one will care or ask them to explain themselves. The simple fact is that we like narratives and these people supply them.

Public Policy Polling - again, the most accurate polling company in the US last cycle -

Not according to Nate Silver. He has them lying very clearly in the middle of the table.
posted by yoink at 9:55 AM on March 13, 2013


"That's because you're thinking politics and not maths. In the primaries the impression of momentum would have helped Romney - GOP voters were lukewarm to him. But in the general election when voting is more partisan the only reason to get out to vote is to keep the other guy from winning. "

This is bullshit.

And it's frustrating, because you've trotted out bullshit about the 2012 election since, well, about 2010. We get it. You don't like Obama, because you're the lone capitalist in Sweden, or whatever. But the more you trot out bullshit, the dumber you look, and you've had at least two years of looking like a crank. Can you try to not make it three?
posted by klangklangston at 9:57 AM on March 13, 2013 [11 favorites]


Let's all remember that this HuffPo article just restates what Nate Silver said about Gallup last autumn. This is not news.
posted by yellowcandy at 10:11 AM on March 13, 2013


Well then it is an obvious liberal bias

So obvious, indeed, that the entire right wing media machine worked itself into a frenzy decrying it.

No...wait...that was decrying anyone who dared to question the all-knowing-wisdom of Gallup. This must be some new sort of usage of the word "obvious" that I'm not familiar with. Is it "street" slang like "bad" meaning "good" or something?
posted by yoink at 10:12 AM on March 13, 2013


Since Indiana was a lock for Romney and possibly because I use VoIP, I never got a single call related to politics. This saddens me because I was deprived of an opportunity to lie my ass off about how I intended to vote.
posted by double block and bleed at 10:56 AM on March 13, 2013


mediareport: "PPP is very good at weighting, apparently. Bottom line is the results: they buck the conventional wisdom that you can't do good polling without cell phones, and they get better results than anyone else. *shrug* Hard to argue with that."

I suspect that the massive volume of polls they do helps quite a bit. Probably also helpful that they do state-by-state polling rather than national polling like Gallup does. Regardless of their absolute accuracy, their polling picked the winner in more states than any other (single) polling outfit.

I can't say whether that's because they were just lucky enough to have their partisan leanings in sync with the overall electorate or if it's because they're just that good.
posted by wierdo at 11:30 AM on March 13, 2013


It's also a bit of a truism in polling that state polls are almost always more accurate than national polls — it's so much harder to get the sampling right when you're trying to talk to EVERYONE AT ONCE than doing it state by state.
posted by klangklangston at 11:37 AM on March 13, 2013


their polling picked the winner in more states than any other (single) polling outfit.

That's a bad measure of how good the polling is. "Picking winners" is not where the game is at: getting the percentages right is. If you pick that candidate A will get 90% and Candidate B 8% and then Candidate A wins by 49-47% you didn't do a better job than a polling firm that predicted that Candidate B would defeat Candidate A by 48-47%.
posted by yoink at 11:38 AM on March 13, 2013


> That's because you're thinking politics and not maths. In the primaries the impression of momentum would have helped Romney - GOP voters were lukewarm to him. But in the general election when voting is more partisan the only reason to get out to vote is to keep the other guy from winning.

Your explanation based on figuring out who will decide to vote and what motivates them sounds more like politics than math to me. I must have learned a different version of the Axiom of Choice than you.
posted by benito.strauss at 11:44 AM on March 13, 2013


Yeah, it's not even an accurate description of voter efficacy/preference under game theory. It's bullshit.
posted by klangklangston at 11:48 AM on March 13, 2013


yoink: "That's a bad measure of how good the polling is."

That's a value judgement on the use that polls should have. While I find it interesting to know relatively hard numbers regarding the popularity of various candidates, proposals, or what have you, what I'm most interested in immediately prior to an election is which candidate will win.
posted by wierdo at 12:38 PM on March 13, 2013


Who needs polls when you've got Intrade? Which is why Intrade had to go away.
posted by telstar at 2:17 PM on March 13, 2013


That's a value judgement on the use that polls should have. While I find it interesting to know relatively hard numbers regarding the popularity of various candidates, proposals, or what have you, what I'm most interested in immediately prior to an election is which candidate will win.

You're missing the point. If what you're interested in is "which candidate will win" then you want the most accurate prediction possible. The one that is consistently closest to the actual final numbers was the most accurate prediction--regardless of whether or not it happened to predict the wrong winner in a really close race.

Otherwise you're in the absurd position of saying that using a Magic 8 ball before an election is a "better" system than a scientific poll if the Magic 8 ball happens to luck into a correct answer and the scientific poll happens to get the winner wrong by a tiny statistical margin.
posted by yoink at 3:00 PM on March 13, 2013 [1 favorite]


Who needs polls when you've got Intrade? Which is why Intrade had to go away.

A) the polls did better than Intrade; B) there were more accurate betting markets than Intrade and they haven't gone away. So no, none of this has anything to do with Intrade's problems.
posted by yoink at 3:01 PM on March 13, 2013 [3 favorites]


I must have learned a different version of the Axiom of Choice than you.

I don't see how the Axiom of Choice is relevant here... (Actually, as there are only finitely many candidates, no matter what you won't need the Axiom of Choice. Unless we're picking something else, I guess.)
posted by hoyland at 3:07 PM on March 13, 2013


All I know is that I started looking at all the polls I could find in the summer and pretty much predicted the states that Obama would win. As election day got closer the results stayed the same (with the margin of victory/loss moving up and down). I was so confused at all the polls and people who were seemingly living in a different reality than what I was seeing.

After Obama won, I wondered why I wasn't on TV making big money, because somehow a schmuck like me was more accurate than a heard of pollsters. How is it that what was so plainly obvious to me was not seen by so many people PAID to predict what would happen?
posted by UseyurBrain at 4:08 PM on March 13, 2013


Not according to Nate Silver. He has them lying very clearly in the middle of the table.

That table is really apples to oranges, and Silver's methodology is needlessly complex. If you can tell me which of the companies above PPP in that table also nailed the result in all 50 states, I'd love to see it.
posted by mediareport at 4:09 PM on March 13, 2013


If you can tell me which of the companies above PPP in that table also nailed the result in all 50 states, I'd love to see it.

And, again, "nailing the results in all 50 states" is NOT a particularly sensitive measure of who did the best polling. And no, Silver's methodology is not 'needlessly complex'; it may well be too complex for you to understand, but there is a solidly reasoned out argument for each part of his methodology.
posted by yoink at 4:26 PM on March 13, 2013 [1 favorite]


Well, I'm being upfront about using "nailing the results in all 50 states" as my benchmark for best polling company. I doubt that some of the companies Silver mentions in that table even polled in all 50 states, thus my "apples and oranges" and "needlessly complex" comments. I think I get what he's done; I just don't think it's the most useful way to get a lay understanding of what happened.
posted by mediareport at 6:00 PM on March 13, 2013 [1 favorite]


yoink: "You're missing the point."

No, I'm not. I just disagree with your metric, as it applies to my use. It doesn't matter to me if a poll shows candidate A up by 10 points more than they really are as long as they get the W/L correct.

Yes, the argument can be made that statistically speaking, polls that have greater absolute errors are less likely to produce the correct outcome, but that's not what has happened recently (at least in PPP's case). Obviously, if we switched to proportional representation or something of that nature where the exact percentages are important, I'd have to change my metric for what makes a poll useful to me.

However, we don't live in that world, so that's not the metric I care about.
posted by wierdo at 11:58 AM on March 14, 2013


Oh, and it should be noted that other stats geeks who do polling aggregation also make the argument that Silver's methodology is more complex than it needs to be. In Sam Wang's case, that bias against complexity comes from experience in having said complexity backfire in past election cycles.
posted by wierdo at 12:06 PM on March 14, 2013


"No, I'm not. I just disagree with your metric, as it applies to my use. It doesn't matter to me if a poll shows candidate A up by 10 points more than they really are as long as they get the W/L correct."

You can't necessarily get the state right without that accuracy. The accuracy is more fundamental to the poll than the W/L.

"Well, I'm being upfront about using "nailing the results in all 50 states" as my benchmark for best polling company."

But… that's a bad benchmark. Most of the states could be guessed without polling. And the only way to evaluate that polling is after the fact — it has no predictive power.

I mean, seriously, Yoink is being a little strident, but you guys don't seem to be understanding either how polling works or how probability works.
posted by klangklangston at 12:19 PM on March 14, 2013 [1 favorite]


Win/loss in 50 states only gives you 1 data point. To know if the polling methodology was correct you'd have to keep it unchanged for multiple elections. No one is going to do that. So accuracy is the best metric. Especially if you have enough polling organizations trying that one is going to get it right just by a statistical fluke.
posted by BrotherCaine at 3:09 PM on March 14, 2013


They use weighting to account for the differences between landline and cell users:

This boggles my mind on several levels:

1. It's hard to believe weighting works that well. Is it really true that white, college-educated males 18-24 (or whatever demographic categories they use for weighting) who have a landline and are willing to submit to a political phone poll vote the same way as white, college-educated males 18-24 who don't have a landline and/or refuse to submit to polls?

2. It's hard to believe that anyone is willing to take a telephone poll about politics. I would have guessed that the people willing to do that would be so few and so unrepresentative of the population as a whole that no amount of weighting could allow you to predict an election from talking to the people willing to take a phone poll.
posted by straight at 8:43 AM on March 15, 2013


But it seems that there are enough people willing to take the polls, because by and large, pollsters do a pretty decent job of predicting results.
posted by Chrysostom at 9:20 AM on March 15, 2013


"2. It's hard to believe that anyone is willing to take a telephone poll about politics. I would have guessed that the people willing to do that would be so few and so unrepresentative of the population as a whole that no amount of weighting could allow you to predict an election from talking to the people willing to take a phone poll."

Heh. I love taking telephone polls. (Telephone poles, not as much.) I always think it's my duty to give an honest opinion to folks curious about how I vote, not least because I've never missed a single election since I turned 18.
posted by klangklangston at 9:46 AM on March 15, 2013




« Older The Art of Brian Sanders   |   You can make a leopard chase its spots Newer »


This thread has been archived and is closed to new comments