Understanding polls.
April 23, 2004 8:58 PM   Subscribe

Understanding polls. For those of us who slept through statistics.
posted by skallas (10 comments total)
That's pretty good, but it still makes my teeth ache in places. The discussion of "statistical ties" could use some work. It's right that newsreaders get this stuff wrong a lot.

Margins of error or confidence intervals aren't brick-wall filters. In the most basic sense, 51 for Kerry and 49 for Bush isn't a tie -- if you had to bet, you'd bet on Kerry that day... but you wouldn't bet the house. I'm too lazy to run the numbers now, but there'd be about 2/3 confidence that Kerry is actually ahead. Not the 95% of as-usually-reported CI's, but still you wouldn't bet that Bush is ahead.

At a more complex level, you need to think about what you're actually making inferences about: is it two things, or just one? In their toy example, the webpage shouldn't really be talking about a statistical tie between Bush and Kerry, because their numbers aren't independent. The proportion voting for Bush is one minus the proportion voting for Kerry. There is really only one proportion, and one confidence interval, that you're seeing here, and the real question is whether that proportion is more than 0.5*. Obviously in the real world it's more complicated than that because of third party candidates, but the numbers you see are still a linear combination of each other. They aren't two independent things that you're making an inference about, they're different sides of the same coin.

It also ignores the surrounding world. If four different polls with similarly-worded questions all turn up with Kerry ahead by 2 or 3 points, you can be more confident that Kerry is actually ahead. If they're asking identical questions, you could just treat it as one poll with a much larger N.

One thing that would have been useful to discuss is that the margin of error of a poll (the confidence interval around the estimation of a proportion) depends on the actual results. Margins of error are biggest with proportions around 0.50 and get smaller as you go out. This means that margins of error can be complicated to display in primary races, when one candidate is getting 30%+ of the responses and others are getting 5% or less.

*making this a one-sided inference, not a 2-sided
posted by ROU_Xenophobe at 11:00 PM on April 23, 2004

I think he did a good job explaining things without getting too deep into the math, which is difficult to explain well without resorting to formulas. It's clearly not meant to be a rigorous discussion; the target audience seems to be people who can't remember high-school algebra.

One of my favorite books in the same vein is How to Lie With Statistics, by Darrell Huff. A classic.
posted by Daddio at 8:58 AM on April 24, 2004

What an interesting link!

I did think though that it should at least have mentioned polling problems over the interpretion of differential voting and the sometime under-representation of certain parties due to shame at perceived 'selfish voting,' (Known by ICM as the Tory-shy factor.)

Perhaps more of a UK phenomenon it was much in evidence during the last dog days of the Thatcher administration; I’d cautiously posit that it’ll be evident to some extent on polling over the US Presidential election.

Also, the issue raised about response rates was well flagged but insufficiently explored in my view. While traditional telephone based polling is in terminal decline, online e-polling is exploding. Companies such as YouGov have claimed superior predictive power than the likes of MORI and ICM but there is still great controversy about the practice. John Curtice, professor of politics at Strathclyde University wrote about internet polling in The Independent here back in January of this year (reg required) and ICM has produced critical research on the subject. I’d be surprised if, following Dean’s adoption of online campaigning techniques, online polling isn’t used much more extensively in Presidential race.
posted by dmt at 9:23 AM on April 24, 2004

It's clearly not meant to be a rigorous discussion; the target audience seems to be people who can't remember high-school algebra.

True enough. It's just that I spend enough time dealing with people who want to treat .05 confidence levels as sacred brick walls -- something significant at 0.06 isn't significant at all -- that it sends up the red mist.

I'd have liked it better if the author had said something like "When the poll numbers are 2 margins of error apart, you can be about 95% confident that Kerry is actually ahead, but if they're only one and a half margins of error apart, your confidence drops to 80%"

What would probably be ideal would be to appeal to sports statistics: even highly innumerate people can have a nuanced understanding of something they give a shit about. But all my examples would be from F1 and CART, so nobody knows about them.
posted by ROU_Xenophobe at 10:31 AM on April 24, 2004

What would probably be ideal would be to appeal to sports statistics: even highly innumerate people can have a nuanced understanding of something they give a shit about. But all my examples would be from F1 and CART, so nobody knows about them.

this reminds me that Chomsky listens to sports programs on the road because they demonstrate they ability by the average man (and even the lower-than-average) to develop often very complex/subtle analysis of an issue.
posted by matteo at 11:17 AM on April 24, 2004

Chomsky, when he's driving, listens to sports programs etc

my bad
posted by matteo at 11:18 AM on April 24, 2004

Oh, I thought that said "Understanding pols."

posted by ZenMasterThis at 3:40 PM on April 24, 2004

I dont think the article was written to be stats 101 compressed into one page. It was written so people would stop believing the myth of the "one or two point change." That is as the author says you just can't say "Kerry lost 2 points, it much have been his lousy speech I wrote an op-ed about last week. See I told you!" When in reality, if the margin of error is around or more than 2 points, then it is safe to say nothing has changed.

This crap has been going on too long in the media and people are buying it left and right. The only safe assumption is that the US is steadily at 45-45 with the rest being swing voters.
posted by skallas at 3:47 PM on April 24, 2004

When in reality, if the margin of error is around or more than 2 points, then it is safe to say nothing has changed.

That's exactly what I mean. If you see a 2 point drop with a 3-point margin of error, the right inference isn't that nothing has changed. The CI for the drop is 5 to -1 points. If you had to bet on whether you'd see a 5, 4, 3, 2, or 1 point drop versus a 0 or -1 point drop, you shouldn't pick the latter. Especially because the probability is clustered in the 3-2-1 area.

The correct inference is that there's probably, but not clearly, a drop in support for Kerry (assuming the same question, etc).

As a rule of thumb, you'd only be safe-ish saying that nothing happened if the difference were less than about half the margin of error (if the difference were smaller than 1 standard error).

There's a big difference between not being safe in saying that there's a change, and being safe in saying there's no change.
posted by ROU_Xenophobe at 6:49 PM on April 24, 2004

The problem with the margin of error stated for a poll is that it gives the impression that the accuracy of the poll is much better than it actually is. The margin of error is only a theoretical calculation that represents a truly random sample from a population. As an example, suppose you wanted to find out how many green M&Ms are in a jar containing 1 million M&Ms. It would take too long to count them all, but statistics show that if you completely stir up the jar and then take a random sample of 500, you can estimate the actual percentage of greens within about plus or minus around 4% with a confidence of 95%. The 4% means that you could be off by that amount in your estimate. The 95% confidence level means that if you were to take your poll of 500 over and over 100 times, you would be within 4% 95 times and would be off by even more than 4% in 5 of those polls. This is know as the sampling error and is due just to random chance -- you just might be unlucky and draw an unrepresentative sample of M&Ms from the jar.

But notice there were two assumptions for these numbers to be true according to statistics: that the jar was randomly stirred up and the sample was completely random. Polling people's opinions violates those assumptions. For example people tend give an answer that they think will please the pollster. The question may be worded poorly. Some sorts of people refuse to answer the phone. For example, poor people may be more likely to be working and less likely to be at home so less likely to be sampled. We know from the infamous red vs. blue maps that people in each political party are not distributed evenly across the country so it is hard to get a representative sample. These types of errors are known as sampling bias. It is as if all the green M&Ms were lighter weight and clustered at the top of the jar so you don't get a random sample.

Each polling organization has formulas to try to correct for sampling bias. These are carefully guarded trade secrets. For example, one pollster may have data that leads him to believe that Republicans are more likely to go to vote and would be under-represented in a strictly random poll. So they will correct for this error by calling more phones in Republican neighborhoods. The accuracy of assumptions in these fudge factors affect the accuracy of the poll. Uncorrected sampling bias probably causes more variation in poll results than the quoted "margin of error" and there is no way of quantifying that error. This is why different polls frequently vary.

The most famous case of sampling bias was the 1936 election of FDR vs Alf Landon. The Literary Digest had a poll showing Landon winning 60% to 40%. The poll was based on 10 million postcards to subscribers, and a list from telephone directories and automobile registration. The response was over two million. This represented almost one-fourth of all voters at that time. It would seem that this large sample would give extreme accuracy, yet it did not. The election actually ended up the exact opposite with FDR having 60%. It turns out that at that period in history, those people having telephones and automobiles were over-represented by Republicans, resulting in a very biased sample.

You may think that taking large samples would improve the results, but not by as much as you think. The accuracy increases by only the square root of the sample size. For example the following table shows the margins of error associated with different sample sizes for an approximately 50/50 race at the 95% confidence level.

Sample size Margin of error
100 ------------ 9.8%
500 ------------ 4.4%
1000 ----------- 3.1%
10,000 -------- 1.0%

Increasing the sample size by a factor of 100 only decreased the error by a factor of 10. This leads to a point of diminishing returns that becomes very expensive. Most polls use a sample size of between 500 and 1000 to give a margin of error of 3 to 4%. Taking larger samples doesn't result in a much better result since the sampling bias is bigger than the sampling error.

The astonishing thing to most people is that the size of the population does not matter -- only the sample size. For example, the accuracy is almost the same if the population size is 10,000, 100,000, 1 million or 100 million. You don't need a larger sample size to poll a city of 10,000 than the entire nation. That is why you see so few local polls. It costs just as much money to poll Helena, MT as to poll the entire country.

This also provides the answer to the question that people always ask: "Why am I never polled?" Since a typical poll will only be 1000 people out of 100 million U.S. households, your chances of being called for any one poll are only about 1 in 100,000. You could go your whole life and never be called.

The thing to keep in mind is that there are two sources of error in polls and they only tell you about one of them, the statistical sampling error. The sampling bias is unknown. Also keep in mind that the margin of error applies to both opponents in a poll, so when you are estimating the lead of one over the other, the margin of error will be about 1.7 times greater than for each opponent alone. For example if the margin of error of a poll is 4% in which Bush has 50% and Kerry has 45%, then you would say that Bush leads Kerry by 5% with a margin of error of approximately 7%. In other words, there is a reasonable probability that Kerry could very well be leading Bush.
posted by JackFlash at 8:50 PM on April 24, 2004

« Older If the meek don't inherit the earth, they'll at...   |   Bi-Partisan Environmental Improvement? Newer »

This thread has been archived and is closed to new comments