Untrustworthy polls
February 5, 2010 11:41 AM Subscribe
2010 Top Ten "Dubious Polling" Awards from StinkyJournalism.org of the Art Science Research Laboratory's Media Ethics Project. Highlighting "the most risible and outrageous pronouncements by polling organizations". Fuzzy Math Award goes to Fox News Network, and the Fox in Sheep's Clothing Award to Scott Rasmussen ("Fox News’ favorite pollster"). Stonewalling/Coverup Award winners are Johns Hopkins Bloomberg School of Public Health and Gilbert Burnham, over the Lancet 2006 survey of Iraqi deaths.
more like projecta.
posted by boo_radley at 12:23 PM on February 5, 2010
posted by boo_radley at 12:23 PM on February 5, 2010
Yeah...Rasmussen didn't CAUSE Obama to win...he just correctly projected it. Not sure why that would make progressives love him.
Oh shit I fed the troll.
posted by Salvor Hardin at 12:27 PM on February 5, 2010
Oh shit I fed the troll.
posted by Salvor Hardin at 12:27 PM on February 5, 2010
Rasmussen regularly pushes the leaners when it suits him. 'nuff said.
posted by Ironmouth at 12:29 PM on February 5, 2010
posted by Ironmouth at 12:29 PM on February 5, 2010
92 out of 114 people (+/- 3) believe that Rasmussen sounds like the name of a long distant relative to Captain Haddock.
posted by Blazecock Pileon at 12:34 PM on February 5, 2010 [1 favorite]
posted by Blazecock Pileon at 12:34 PM on February 5, 2010 [1 favorite]
We've covered the Lancet study in some detail before -
Initial study finding 100 000 dead using cluster analysis
Follow up study (650 000 dead)
Roundtable of responses to follow up study
This American life bit
WHO survey with lower death count ("only" 150 000) where the term "neglicide" is coined to describe Bush's actions.
It's great that someone other than Nate Silver is scrutinizing these operations and polls, as some are pretty poor. Obviously I'm interested in the Lancet study - pretty interesting background there and when I get home I'll try and dig up a bit more information I've got about it.
posted by zenon at 12:38 PM on February 5, 2010
Initial study finding 100 000 dead using cluster analysis
Follow up study (650 000 dead)
Roundtable of responses to follow up study
This American life bit
WHO survey with lower death count ("only" 150 000) where the term "neglicide" is coined to describe Bush's actions.
It's great that someone other than Nate Silver is scrutinizing these operations and polls, as some are pretty poor. Obviously I'm interested in the Lancet study - pretty interesting background there and when I get home I'll try and dig up a bit more information I've got about it.
posted by zenon at 12:38 PM on February 5, 2010
I 120% positive there awards are all made up out of falsified data.
posted by cjorgensen at 12:40 PM on February 5, 2010
posted by cjorgensen at 12:40 PM on February 5, 2010
You may not like his results, but Rasmussen's methods seem to reflect election results pretty accurately. Here's his tracking results prior to the Massachusetts special election reflecting Brown clsoing the gap on Coakley. Simply because he's not projecting the results some desire, doesn't make him wrong. But whatever, believe what you like. So long!
posted by perfecta at 12:40 PM on February 5, 2010
posted by perfecta at 12:40 PM on February 5, 2010
If A + B = C, then C must = A, and there = these in my above sentence.
posted by cjorgensen at 12:41 PM on February 5, 2010
posted by cjorgensen at 12:41 PM on February 5, 2010
But what about the missing 'am' cjorgensen. What about the MISSING 'AM'?!
I question your methodology.
posted by Babblesort at 12:48 PM on February 5, 2010 [2 favorites]
I question your methodology.
posted by Babblesort at 12:48 PM on February 5, 2010 [2 favorites]
The background to the criticism of the Lancet study is the AAPOR investigation/censure mentioned in the article. AAPOR themselves weren't very forthcoming with details, but there's some useful background and filling in of the gaps here: http://www.mediahell.org/lancet_unspeak.htm
posted by internationalfeel at 12:57 PM on February 5, 2010
posted by internationalfeel at 12:57 PM on February 5, 2010
Long Rasmussen is long. -ish. According to current opinion data, based on the question "How long is Rasmussen." Results adjusted to the side of the +-3% margin of error favored by whomever's paying for the survey.
posted by ardgedee at 1:00 PM on February 5, 2010 [1 favorite]
posted by ardgedee at 1:00 PM on February 5, 2010 [1 favorite]
Man, I need a proopreader.
I printed out the Fox News Poll and put it up on my wall.
posted by cjorgensen at 1:27 PM on February 5, 2010 [1 favorite]
I printed out the Fox News Poll and put it up on my wall.
posted by cjorgensen at 1:27 PM on February 5, 2010 [1 favorite]
Comparing the Lancet study to Iraq Body Count is so far beyond apples and oranges it makes me question the entire article. Nobody thinks that because 44 000 Americans die annually due to lack of healthcare, that the administration is assassinating 44 000 Americans a year. IBC explicitly measures only deaths due to violence. Lancet measured total excess deaths as a result of the invasion. It was asking the broad question of "how many more died than would have?" - the same question we are answering when we say, "44 000 Americans annually."
This is basic reading comprehension, which is why the "Lancet scandal" infuriates me to this day. I'm not saying the Lancet study was perfect. But the ORB surveys backed it up. Even the Iraqi government asserts there are now 5 million orphans in Iraq, which makes the IBC absurdly (unless each victim was a single parent with 50 children!).
posted by mek at 1:49 PM on February 5, 2010 [3 favorites]
This is basic reading comprehension, which is why the "Lancet scandal" infuriates me to this day. I'm not saying the Lancet study was perfect. But the ORB surveys backed it up. Even the Iraqi government asserts there are now 5 million orphans in Iraq, which makes the IBC absurdly (unless each victim was a single parent with 50 children!).
posted by mek at 1:49 PM on February 5, 2010 [3 favorites]
Man, I need a proopreader.
ha. now WHICH typo is it this time? me? I'm saying it's /r.
posted by shmegegge at 2:31 PM on February 5, 2010
ha. now WHICH typo is it this time? me? I'm saying it's /r.
posted by shmegegge at 2:31 PM on February 5, 2010
proopreader
1. One who reads while he proops (defined as making a ROOOP! noise when defecating)
2. Greg Proops' personal phrenologist
posted by Skot at 2:37 PM on February 5, 2010
1. One who reads while he proops (defined as making a ROOOP! noise when defecating)
2. Greg Proops' personal phrenologist
posted by Skot at 2:37 PM on February 5, 2010
I have a habit of checking the RealClearPolitics rolling average of polls every few days to see what the general political mood is, and have always been struck by how Rasmussen is such a consistent and significant outlier. So last week I dropped all polling numbers from RCP's archives into an Excel file and graphed the results... make of it what you will. (The vertical axis is President Obama's net approval rating as shown by a given poll.)
posted by Rhaomi at 4:33 PM on February 5, 2010 [1 favorite]
posted by Rhaomi at 4:33 PM on February 5, 2010 [1 favorite]
make of it what you will
Unless the other sixteen signals are at fault, that's a pretty damning analysis.
posted by Blazecock Pileon at 5:06 PM on February 5, 2010
Unless the other sixteen signals are at fault, that's a pretty damning analysis.
posted by Blazecock Pileon at 5:06 PM on February 5, 2010
You may not like his results, but Rasmussen's methods seem to reflect election results pretty accurately.
There is a qualitative difference between election polls, which are often done under contract for a media outlet, and issue polls, which polling groups perform at their own initiative under their own quality controls, generally to keep attention on their brand.
Even Nate Silver liked his election polls in 2008, but in 2009 noted the problems with the issue polls.
posted by dhartung at 5:45 PM on February 5, 2010
There is a qualitative difference between election polls, which are often done under contract for a media outlet, and issue polls, which polling groups perform at their own initiative under their own quality controls, generally to keep attention on their brand.
Even Nate Silver liked his election polls in 2008, but in 2009 noted the problems with the issue polls.
posted by dhartung at 5:45 PM on February 5, 2010
Major election results also seem markedly more likely to involve polling people who have genuine opinions on something, versus polling a bunch of people who have no idea what you're talking about.
If I called you about the 2008 presidential election, it's very likely that no matter how I phrased the question, you knew that the answer was "Obama" or "McCain" depending on your personal politics. But if I call you about "maintenance budget reform" or something and you don't have the faintest idea what the current proposals actually are, and I talk about it in such a way that one looks good and the other looks bad, well, that's very easy to manipulate.
posted by larkspur at 7:08 PM on February 5, 2010
If I called you about the 2008 presidential election, it's very likely that no matter how I phrased the question, you knew that the answer was "Obama" or "McCain" depending on your personal politics. But if I call you about "maintenance budget reform" or something and you don't have the faintest idea what the current proposals actually are, and I talk about it in such a way that one looks good and the other looks bad, well, that's very easy to manipulate.
posted by larkspur at 7:08 PM on February 5, 2010
Comparing the Lancet study to Iraq Body Count is so far beyond apples and oranges it makes me question the entire article. Nobody thinks that because 44 000 Americans die annually due to lack of healthcare, that the administration is assassinating 44 000 Americans a year. IBC explicitly measures only deaths due to violence.
IBC Counts news reports about violent deaths. If you're name doesn't get into the paper, then from their perspective, you're not dead. It's absurd.
posted by delmoi at 7:45 PM on February 5, 2010
IBC Counts news reports about violent deaths. If you're name doesn't get into the paper, then from their perspective, you're not dead. It's absurd.
posted by delmoi at 7:45 PM on February 5, 2010
ha. now WHICH typo is it this time? me? I'm saying it's /r.
... if I am reading this right, I would not want to be one of those.
posted by JHarris at 7:50 PM on February 5, 2010 [1 favorite]
... if I am reading this right, I would not want to be one of those.
posted by JHarris at 7:50 PM on February 5, 2010 [1 favorite]
Hrm, "stinkyjournalism.org" thinks that Scott Rasmussen is a partisan pollster. If this becomes a trend, he might be facing tough times among various internet blogs and chat forums.
posted by Slap Factory at 10:08 PM on February 5, 2010
posted by Slap Factory at 10:08 PM on February 5, 2010
The worst news polling trick of all is broadcasting the results of public opinion polls (such as "Did scientists falsify research?") as if those numbers prove anything about the proposition inherent in the question. The story becomes a report on what people who are in no position to know the truth figure happened after watching reports on what people who are in no position to know the truth figure happened after watching reports on what people who are in no position to know the truth figure happened after watching...
posted by pracowity at 10:33 PM on February 5, 2010
posted by pracowity at 10:33 PM on February 5, 2010
'IBC Counts news reports about violent deaths. If you're name doesn't get into the paper, then from their perspective, you're not dead. It's absurd.'
In fact, IBC's count also includes tallies from morgues, NGOs etc. Corroborated news reports of mass deaths (with unidentified victims) are included in IBC's figure. It's not true that IBC's count is of reports. Their figure is of reported/documented deaths. The earlier comment about IBC and the Lancet study being like apples and oranges is correct. The article should have mentioned that. IBC counts actual deaths, Lancet provides a statistical estimate based on a sample (they recorded 300 violent deaths, extrapolated to 601,000). A more direct comparison is Lancet and IFHS (which estimated 151,000 violent deaths over the same period as the Lancet study). IFHS did provide the the kind of information that the Lancet authors are criticised for not providing (sampling methods, survey questions, etc).
posted by internationalfeel at 3:25 AM on February 6, 2010
In fact, IBC's count also includes tallies from morgues, NGOs etc. Corroborated news reports of mass deaths (with unidentified victims) are included in IBC's figure. It's not true that IBC's count is of reports. Their figure is of reported/documented deaths. The earlier comment about IBC and the Lancet study being like apples and oranges is correct. The article should have mentioned that. IBC counts actual deaths, Lancet provides a statistical estimate based on a sample (they recorded 300 violent deaths, extrapolated to 601,000). A more direct comparison is Lancet and IFHS (which estimated 151,000 violent deaths over the same period as the Lancet study). IFHS did provide the the kind of information that the Lancet authors are criticised for not providing (sampling methods, survey questions, etc).
posted by internationalfeel at 3:25 AM on February 6, 2010
Pretty misleading representation of the Lancet research into the number of death caused by the War in Iraq.
The review confirmed that the research methodology was sound and suspended the lead researcher for a breach of privacy (some data collection forms asked for the names of the respondents). It had nothing to do with the accuracy of the research, which used a standard method for estimating war deaths.
There have been two proper investigations into the death toll in Iraq, of which the Lancet report is one. The other estimates (like the Iraq Body Count) use statistically invalid methods. More here.
http://en.wikipedia.org/wiki/Casualties_of_the_Iraq_War#Various_estimates
I've no idea why IBC is so completely incorrect.
posted by bobbyelliott at 9:08 AM on February 6, 2010 [1 favorite]
The review confirmed that the research methodology was sound and suspended the lead researcher for a breach of privacy (some data collection forms asked for the names of the respondents). It had nothing to do with the accuracy of the research, which used a standard method for estimating war deaths.
There have been two proper investigations into the death toll in Iraq, of which the Lancet report is one. The other estimates (like the Iraq Body Count) use statistically invalid methods. More here.
http://en.wikipedia.org/wiki/Casualties_of_the_Iraq_War#Various_estimates
I've no idea why IBC is so completely incorrect.
posted by bobbyelliott at 9:08 AM on February 6, 2010 [1 favorite]
-- 'The review confirmed that the research methodology was sound' (bobbyelliott)
In fact, under the subheading 'Study Methodology', the review states that it "did not evaluate aspects of the sampling methodology or statistical approach of the study".
(Source: Johns Hopkins review)
-- 'There have been two proper investigations into the death toll in Iraq, of which the Lancet report is one.' (bobbyelliott)
I count at least five studies using the sample-survey approach: ILCS, Lancet 2004, Lancet 2006, ORB, IFHS. There have been other studies using a different approach, eg the one from CRED (Centre for Research on the Epidemiology of Disasters, Brussels) which estimated 125,000 deaths over the same period as Lancet 2006.
-- 'The other estimates (like the Iraq Body Count) use statistically invalid methods.' (bobbyelliott)
That's a curious statement since IBC doesn't provide an "estimate" in a statistical sense. It provides a straightforward count of documented deaths. No "statistical methods" required. The method of documenting and tallying deaths has a long pedigree, going back at least to the First World War (see this ZNet article).
-- '...the [Lancet study] research, which used a standard method for estimating war deaths.' (bobbyelliott)
In fact, the application of the cluster-sampling method (originating and "standard" in epidemiology) to deaths from violence in war zones is relatively new. The new Human Security Report states that "the appropriateness of using retrospective mortality surveys to estimate excess war deaths has never been validated, and the findings of the small number of surveys that have been used for this purpose show troubling inconsistencies." We can see such inconsistencies between the Lancet 2006 study and IFHS (their estimates for violent deaths over the same period differed by 450,000).
posted by internationalfeel at 4:02 AM on February 7, 2010
In fact, under the subheading 'Study Methodology', the review states that it "did not evaluate aspects of the sampling methodology or statistical approach of the study".
(Source: Johns Hopkins review)
-- 'There have been two proper investigations into the death toll in Iraq, of which the Lancet report is one.' (bobbyelliott)
I count at least five studies using the sample-survey approach: ILCS, Lancet 2004, Lancet 2006, ORB, IFHS. There have been other studies using a different approach, eg the one from CRED (Centre for Research on the Epidemiology of Disasters, Brussels) which estimated 125,000 deaths over the same period as Lancet 2006.
-- 'The other estimates (like the Iraq Body Count) use statistically invalid methods.' (bobbyelliott)
That's a curious statement since IBC doesn't provide an "estimate" in a statistical sense. It provides a straightforward count of documented deaths. No "statistical methods" required. The method of documenting and tallying deaths has a long pedigree, going back at least to the First World War (see this ZNet article).
-- '...the [Lancet study] research, which used a standard method for estimating war deaths.' (bobbyelliott)
In fact, the application of the cluster-sampling method (originating and "standard" in epidemiology) to deaths from violence in war zones is relatively new. The new Human Security Report states that "the appropriateness of using retrospective mortality surveys to estimate excess war deaths has never been validated, and the findings of the small number of surveys that have been used for this purpose show troubling inconsistencies." We can see such inconsistencies between the Lancet 2006 study and IFHS (their estimates for violent deaths over the same period differed by 450,000).
posted by internationalfeel at 4:02 AM on February 7, 2010
Thanks for that interesting contribution. But it it is important to distinguish between valid research approaches (you mention six) and other approaches, such as IBC, which have no research validity whatsoever. It isn't then helpful to conflate the two.
posted by bobbyelliott at 4:36 AM on February 7, 2010 [1 favorite]
posted by bobbyelliott at 4:36 AM on February 7, 2010 [1 favorite]
One of the world's most respected demographers, Beth Osborne Daponte (she produced authoritative death figures for the first Gulf War, and risked being fired by the government for doing so) wrote of IBC:
"While each approach has its drawbacks and advantages, this author puts the most credence on the work that the Iraq Body Count has done for a lowerbound estimate of the mortality impact of the war on civilians. The data base created by IBC seems exceptional in its transparency and timeliness. Creating such a data base carefully is an incredibly time-consuming exercise. The transparency of IBC’s work allows one to see whether incidents of mortality have been included. The constant updating of the data base allows one to have current figures."
Of course, it's difficult to weigh the expertise of someone like that against the knowledge demonstrated in your own posts.
posted by internationalfeel at 5:18 AM on February 7, 2010
"While each approach has its drawbacks and advantages, this author puts the most credence on the work that the Iraq Body Count has done for a lowerbound estimate of the mortality impact of the war on civilians. The data base created by IBC seems exceptional in its transparency and timeliness. Creating such a data base carefully is an incredibly time-consuming exercise. The transparency of IBC’s work allows one to see whether incidents of mortality have been included. The constant updating of the data base allows one to have current figures."
Of course, it's difficult to weigh the expertise of someone like that against the knowledge demonstrated in your own posts.
posted by internationalfeel at 5:18 AM on February 7, 2010
IBC Director John Sloboda has stated, "We've always said our work is an undercount, you can't possibly expect that a media-based analysis will get all the deaths."
Of course, it's difficult to weight the experise of someone like that against the knowledge demonstrated in your own posts.
posted by bobbyelliott at 12:59 PM on February 7, 2010
Of course, it's difficult to weight the experise of someone like that against the knowledge demonstrated in your own posts.
posted by bobbyelliott at 12:59 PM on February 7, 2010
« Older Roughly 1 in 4 Americans is employed to keep... | The Trauma Myth by Susan Clancy Newer »
This thread has been archived and is closed to new comments
posted by perfecta at 11:50 AM on February 5, 2010