Join 3,514 readers in helping fund MetaFilter (Hide)


Lies, Damn Lies, and Daily Kos polls
June 29, 2010 12:03 PM   Subscribe

For the past year and a half, Daily Kos has been running weekly polls from the respected polling firm, Research 2000. Earlier this month, former Daily Kos diarist Nate Silver of Five Thirty Eight published a rating of pollsters that placed R2k near the bottom, leading Markos to fire R2K. Today, Markos alleges that R2K committed fraud, publishing a study of their results by independent statisticians. He promises to sue.
posted by empath (91 comments total) 10 users marked this as a favorite

 
That sucks. From what I know of Markos, he seems a decent sort. At least he's probably well-equipped to deal with the problem, and his reputation will probably not suffer.
posted by kalessin at 12:11 PM on June 29, 2010


This is pretty serious. I thought I was cynical, but this fraud surprises me. Wow, I guess as the old Italian saying goes "we should trust everyone - except those with two nostrils" (which I got from watching Il Posto last night :)).
posted by VikingSword at 12:13 PM on June 29, 2010


Interesting. I subscribe to public policy polling's RSS feed and they rank pretty high up in 538's rankings. Also going for them is the fact that their twitter handle is ppppolls
posted by delmoi at 12:14 PM on June 29, 2010


Statisticians hoisted on their own petard.

It's still a bit shocking that a firm described as "respected" would even bother with this kind of fraud. What is the point? Were they making up numbers to save themselves the time/expense of having to actually do the polls? Did they do a micropoll and then extrapolate from there?

It certainly doesn't inspire confidence in any polling organization. I don't have the time to run this kind of analysis-over-time for every company whose results I see reported someplace. I guess there are watchers watching the watchmen, or questioners questioning the questioners, or something, but those need to be better publicized so an informed electorate can exercise true democracy.
posted by hippybear at 12:16 PM on June 29, 2010


serious question - does statistical significant constitute proof? Can you win a court case based on confidence intervals?
posted by JPD at 12:17 PM on June 29, 2010


Some ACORN-like fraudster probably came up to dailykos and said they would do some polling for them - and being liberals - the kossacks just trusted them to do a good job out of the goodness of their Obama-loving hearts. Now they link to a bunch of nerd number crap that I'm not going to read to try to show how smart they are for figuring out they were sold a bill of goods like the suckers they are for a whole year! See this is why liberals can't do anything. They can't govern, they can't run a business, all they can do is blog and try to tear down us John Galts of the world who would actually make damn sure their polling firm wasn't just making shit up before they hired them. Pathetic. When I want polling data I just ask Jesus what people think and he tells me. Give it a try liberals!
posted by ND¢ at 12:20 PM on June 29, 2010 [30 favorites]


Gotta hand it to Markos, he doesn't sugar coat it. If he thinks he's been publishing wrong data, he won't run from it.

Solid work.
posted by Ironmouth at 12:23 PM on June 29, 2010 [10 favorites]


Wow. Just wow.

A headline political blogger prepared to admit that his site has published grievous mistakes?

That's rarer than rocking horse shit.
posted by MuffinMan at 12:28 PM on June 29, 2010 [14 favorites]


Research 2000

Isn't it kind of obvious that this would happen when you use polling firm that uses data that's ten years old?
posted by MCMikeNamara at 12:30 PM on June 29, 2010 [12 favorites]


This is a sad story but Kos seems to be handling it well; despite one's political beliefs I think we can all agree that a victory for honesty and math embiggens us all.
posted by jtron at 12:32 PM on June 29, 2010 [2 favorites]


I will be interested to see when someone develops a more accurate polling method than random dialing to land-lines. Until then, I think we have to view even accurately reported polls with honest attempts at neutral questions with some suspicion.
They're still kind of fascinating, though. I'm a sucker, I'll admit it.
posted by Red Loop at 12:33 PM on June 29, 2010 [1 favorite]


Honestly, the severe Democratic house effect of R2000's results should have been enough for Kos to realize they were not getting what they want.

The reason for them to do polls is to get accurate data on races so they can figure out where to direct the grassroots readers of the site to spend their money.

It doesn't help if the polling firm is giving them data that makes the situation more rosy than it actually is or tells them what they want to hear.
posted by furiousxgeorge at 12:36 PM on June 29, 2010 [1 favorite]


Gotta hand it to Markos, he doesn't sugar coat it. If he thinks he's been publishing wrong data, he won't run from it.

And for this, the Great Orange Satan will come out ahead. With no coverup, it'll be a story about a media outlet getting screwed for a couple days, and will sink to the back pages over the course of a long, dreary court battle. Other outlets, especially oppositional ones, will have no traction to impugn GOS generally because they're being so open and honest about what happened and what they're doing about it. Ones that do anyway will be shooting themselves in the foot because every GOS defender can point to Kos's reaction as textbook transparency.

Why this lesson doesn't sink in with others, I don't know. It's not like "it's the coverup, not the scandal" is a new idea or anything.
posted by fatbird at 12:38 PM on June 29, 2010 [3 favorites]


It's war! Lawyer for DailyKos details lawsuit against Research 2000.
posted by ericb at 12:41 PM on June 29, 2010


Can you win a court case based on confidence intervals?

Depends on the case obviously, but in the right context, abso-f*ing-lutely. The statisticians would be called to testify as expert witnesses, and their testimony would be weighed by a jury (or judge, if a jury trial is waived), who would decide whether to credit their testimony or not.

Granted, statistics aren't preferred evidence most of the time, but here the focus is less on what the statistics themselves prove than whether these are legitimate statistics at all, which is a different question entirely.

It's also worth noting that if/when this thing goes to trial, both sides will have had ample opportunity to engage in discovery, which is likely to turn up more direct evidence of fraud if there is any.
posted by valkyryn at 12:42 PM on June 29, 2010 [2 favorites]


It's going to get worse for Markos: he used a Research2000 poll of self-identified Republicans to support points he made in his soon-to-be-published book, American Taliban.

Neocon schadenfreude for several months I guess.
posted by Eyebeams at 12:44 PM on June 29, 2010 [1 favorite]


I'm never going to be able to hang out as much at Daily Kos as I did in the beginning, because for better or for worse, I'll never be into politics the same way I was in 2003.

But, earlier thread joke aside, as someone who was completely unaware of this until this post showed up, I'm really impressed with how he's handled this, and so far, at least, it seems to be the perfect model of how to deal with a shit storm.

fatbird: It's not like "it's the coverup, not the scandal" is a new idea or anything.

No doubt. Which is pretty amusing to me (in a sad way) that the reason this even came to light is because they were so transparent in their data. I wish THAT would be the story/outcome from this. But alas...

On a related note, I'm going to read the "our findings for dummies" version that's written in this post after work today, but I look forward to hearing any of your (more learned statistical) opinion on the math/stats behind it.
posted by MCMikeNamara at 12:46 PM on June 29, 2010


Kos did post a comment regarding R2K data in the book:

Book was stripped of references to R2K...

But one wonders how big an impact stripping references could have if it's so close to press that pagination is written in stone.
posted by activitystory at 12:48 PM on June 29, 2010


Statisticians hoisted on their own petard.

It's still a bit shocking that a firm described as "respected" would even bother with this kind of fraud. What is the point? Were they making up numbers to save themselves the time/expense of having to actually do the polls? Did they do a micropoll and then extrapolate from there?


The funniest aspect to me is that they are so obviously bad at faking the numbers. I'm hardly a statistics expert but it would only take me a few lines of code to write some monte carlo simulations that would produce fake poll data with the desired results and not fail those statistical checks spectacularly like R2K's do. I would not be surprised at all if their entire "polling process" was just a guy tweaking numbers in an Excel spreadsheet every week.
posted by burnmp3s at 12:49 PM on June 29, 2010


from one of the links, emphasis mine:
Looking over the full 60-week set, the variance in the reports was only 9.947. To calculate how unlikely that is, we need to use a standard tool, a chi-squared distribution, in this case one with 59 degrees-of-freedom. The probability of getting such a low variance via regular polling is less than one in 10^16, i.e. one in ten million billion.
posted by jtron at 12:49 PM on June 29, 2010 [1 favorite]


Previously. Quite a bit of skepticism in that thread about the poll results. Evidently, the skeptics were onto something.
posted by Dojie at 12:50 PM on June 29, 2010


If the allegation is false, it should be trivial for R2K to document that the specified poll actually happened. They it's gotten here is a pretty bad sign.
posted by a robot made out of meat at 12:51 PM on June 29, 2010 [1 favorite]


"Tom I'm a veteran I didn't get a deferment in Vietnam because I was too depressed to fight".

Markos doesn't run from a fight (obv).
posted by Eyebeams at 12:52 PM on June 29, 2010 [4 favorites]


Can you win a court case based on confidence intervals?

From experience, yes, but not in front of a jury, usually. Forensic evidence is all about confidence intervals.
posted by bonehead at 12:55 PM on June 29, 2010


It's war! Lawyer for DailyKos details lawsuit against Research 2000

"It's war?"

Jesus fuck did all the grownups leave the Post with Weigel last week?
posted by XQUZYPHYR at 12:57 PM on June 29, 2010


In all fairness, they would have been worse off with Research Vista.
posted by zippy at 1:01 PM on June 29, 2010 [11 favorites]


From what I know of Markos, he seems a decent sort.

December 2009 - Insurance companies win. Time to kill this monstrosity coming out of the Senate.

March 2010 - The fact is, this does a heck of a lot for a lot of people. ... . And if somebody like Kucinich wants to block that, I find that completely reprehensible. ... What he’s doing, he’s undermining this reform. He’s making common cause with the Republicans. And I think that’s a perfect excuse and a rationale for a primary challenge.

Fuck Markos. Seriously.
posted by Joe Beese at 1:03 PM on June 29, 2010 [5 favorites]


Since the odds of getting a match each time are essentially 50%, the odds of getting 776/778 matches are just like those of getting 776 heads on 778 tosses of a fair coin. Results that extreme happen less than one time in 10228. That’s one followed by 228 zeros. (The number of atoms within our cosmic horizon is something like 1 followed by 80 zeros.)

*sigh* Was I the only one secretly hoping that this was leading up to a reference about being caught in a production of Rosencrantz and Guildenstern Are Dead?
posted by scody at 1:07 PM on June 29, 2010 [2 favorites]


burnmp3s: I would not be surprised at all if their entire "polling process" was just a guy tweaking numbers in an Excel spreadsheet every week.

I would be a bit surprised...if only because I think even my statistically challenged ass could figure out Excel to give you better random numbers last week.


XQUZYPHYR: It's war! Lawyer for DailyKos details lawsuit against Research 2000

"It's war?"

Jesus fuck did all the grownups leave the Post with Weigel last week?


No shit. And it gets even worse once you read the actual post from the Post.

Full disclosure: This blog, like Post blogs, regularly linked to Research 2000 polls, because the firm has been on the Post's "approved" list. Like I said above, this is likely to prompt a serious discussion about whether news orgs should be doing more to vet the polling they commission or publish
Emphasis mine. Likely? Really, that's the bold stand you want to take. Even if all of these allegations turn out to be 100% false, I'd really like to think that something like this would definitely prompt serious real journalists to have serious real discussions. Seriously.

I really hate that something made me almost consider using the phrase "lamestream media" non-ironically.
posted by MCMikeNamara at 1:08 PM on June 29, 2010 [1 favorite]


Standard (or at least, proper) PR options when you've made a mistake, knowlingly or otherwise, if you are an organization that requires credibility to operate:

1. 'fess up, in a detailed and factual way, with complete accountability for your responsibility (and, if it wasn't you directly as in this case, immediately seeking accountability for their responsibility.)

2. Hide it, but have a battle plan to do option 1 in case you are unsuccessful at hiding it.

Unfortunately, it's hard to tell which is going on in any given scenario -- an organization that is doing option 2 correctly will appear to be doing option 1, and an organization that is in the process of doing option 1 may be pre-emptively called out, making it appear they are ineptly doing option 2.

So, it comes down to who revealed it to the press and public initially. If the organization did it, it's option 1; if someone else did it, it's option 2; if the organization did it under threat from someone else doing it, it could still be either.

I say this not to suggest that option 2 is going on here, or confirm that option 1 is. Instead, I say this to point out that credibility-dependent organizations that take any other approach are being run without proper/competent PR support.
posted by davejay at 1:09 PM on June 29, 2010 [1 favorite]


Can you win a court case based on confidence intervals?

Well, civil cases do have an alpha way up at 0.4999999999.
posted by ROU_Xenophobe at 1:16 PM on June 29, 2010


Nate Silver had suspicions in February, but didn't follow them up.

It was hard enough to detect this from a firm that released all their crosstabs. Imagine how difficult it would be to detect it from the many that keep them as trade secrets. Half the polling industry could be manufacturing numbers. All of them should be made to release crosstabs or be shunned by the media.
posted by East Manitoba Regional Junior Kabaddi Champion '94 at 1:24 PM on June 29, 2010 [5 favorites]


It won't come down to statistics in the court case. It will come down to Daily Kos demanding that R2K produce phone records and the original survey records proving that they made the calls they did, and R2K either producing them or not.

R2K should also have audited a percentage of calls they made, and should have records of those, too.
posted by empath at 1:26 PM on June 29, 2010 [2 favorites]


I am heartened to hear you can win a court case based on solid statistical analysis.
posted by JPD at 1:33 PM on June 29, 2010


Between this and the 538/Strategic Vision flap, I'd like to see some serious efforts to improve transparency in the polling industry. If I were an honest pollster, I'd be seriously considering an outside audit of my numbers, just to give me something I could wave and say "Look, we're one of the good guys!"
posted by EarBucket at 1:58 PM on June 29, 2010 [2 favorites]



From what I know of Markos, he seems a decent sort.

December 2009 - Insurance companies win. Time to kill this monstrosity coming out of the Senate.

March 2010 - The fact is, this does a heck of a lot for a lot of people. ... . And if somebody like Kucinich wants to block that, I find that completely reprehensible. ... What he’s doing, he’s undermining this reform. He’s making common cause with the Republicans. And I think that’s a perfect excuse and a rationale for a primary challenge.

Fuck Markos. Seriously.
posted by Joe Beese at 3:03 PM on June 29 [1 favorite +] [!]
So it's "Fuck Markos" because he changed his mind on the health care bill, or because he didn't agree with you at one step of the process?

I find that remembering that the perfect is the enemy of the good to be a powerful tonic.
posted by jtron at 2:01 PM on June 29, 2010 [8 favorites]


A headline political blogger prepared to admit that his site has published grievous mistakes?

I've had the pleasure of spending a little time with both Markos and Nate Silver, on separate occasions (and not related to anything poltical). I can +1 that they're both the type of people who aren't afraid to be wrong if they learn something in the process.

The Internet (and America, and the world) could use a lot more of that type.
posted by rokusan at 2:19 PM on June 29, 2010


To think I defended how R2000 asked its questions while the issue was that they were "manipulating or inventing" data past any point of recognition. Blast. I certainly didn't anticipate that sort of nefarious behavior. I wonder how AAPOR will respond to this.
posted by zenon at 2:20 PM on June 29, 2010


In all fairness, they would have been worse off with Research Vista.

"Hey, it looks like you're trying to conclude that Republicans are stupid. Can I help with that?" -- Research Bob
posted by rokusan at 2:21 PM on June 29, 2010 [1 favorite]


I am a long-time dKos reader and occasional commenter (I've written a couple of diaries over the years, but not many). During the Bush years it was an oasis of opposition, and gave many of us a vital sense of a constituency for fighting back. But over the last two years, dKos has descended mightily, beginning with handling the democratic primaries terribly. Now it's become saturated with lifestyle bullshit that drowns out the once central mission, has too many celebrity diarists (including Olbermann, who made a might ass out of himself just recently at dKos), and is awash in insane, inane so-called "progressives" who never met a "perfect" they wouldn't use to beat the shit out of the "good," just as babyish as their FreeRepublic counterparts.

I still read it every day, but less and less closely and with more and more sense of boredom.
posted by fourcheesemac at 2:24 PM on June 29, 2010 [4 favorites]


I will be interested to see when someone develops a more accurate polling method than random dialing to land-lines.

If it's truly random-digit dialing, then it should not be limited to land lines, because you're just generating numbers--those could also be cell-phone numbers. If you are limiting your random generator to certain exchanges in order to target certain geographical areas, then that might exclude cell phones and introduce bias.

Cell phones actually introduce a related problem into sampling which is that with number portability and nationwide long distance, there is less motivation to get a local number. So if you're using the 901 area code to get Memphis voters, you won't get me, because I still have a Boston cell number, despite having been registered to vote in TN since 2007.

Poll methods will never be 100% accurate anyway, because you will always have non-responders. You can't actually force people to participate in public opinion polls.
posted by DiscourseMarker at 2:30 PM on June 29, 2010


I'm no fan of Kos, but I'll give him major points for going completely public on this and not trying to whitewash it, or sweep it under a rug.
posted by Chocolate Pickle at 2:36 PM on June 29, 2010 [1 favorite]


I find that remembering that the perfect is the enemy of the good to be a powerful tonic.

While I agree that there's not really anything wrong with changing your opinion. I find that phrase pretty annoying. Especially when it's used to justify things that, frankly, don't even qualify as "good" (Like failing to close gitmo). It's just a catchall excuse for inaction and incredibly intellectually lazy.
posted by delmoi at 2:58 PM on June 29, 2010 [2 favorites]


Kos was the one who failed to close Gitmo? Well dog my cats, I retract my earlier statement ;)

If I waited for a political party or elected representatives that 100% matched up with what I believe in order to gain my support, I would be waiting a considerably long time. Since I'm a hu-man and probably limited to somewhere around three score and ten, I do the best I can with what I've got, and put my votes and financial support where I think they'll do the most good. As to the Great Orange Satan, sure, they don't match up perfectly with my goals, methods, or priorities; at the same time, they seem to have done a lot of getting the word out, helping to organize from the bottom up, and at least making an honest attempt toward Keeping It Real (as in being very public with the utter failure of their pollster).

And thanks for backhandedly calling me 'incredibly intellectually lazy,' I needed that
posted by jtron at 3:21 PM on June 29, 2010


Research 2000's lawyer (Richard Beckler of Howrey, LLP) just sent Nate Silver at FiveThirtyEight.com a C&D.
posted by kipmanley at 3:22 PM on June 29, 2010 [1 favorite]


Kos was the one who failed to close Gitmo? Well dog my cats, I retract my earlier statement ;)

No, but that phrase has certainly been trotted out quite a bit to defend it. My problem is with the phrase not with Kos, or with you :)
posted by delmoi at 3:23 PM on June 29, 2010


Perfect is really more of a frenemy to good.
posted by Astro Zombie at 3:36 PM on June 29, 2010 [5 favorites]


It's just a catchall excuse for inaction

It's precisely the opposite.
posted by empath at 3:39 PM on June 29, 2010 [4 favorites]


Eyebeams: ""Tom I'm a veteran I didn't get a deferment in Vietnam because I was too depressed to fight" .

Markos doesn't run from a fight (obv).
"

As a Coloradoan, I'd like to apologize for Tom "Crazy Pants" Tancredo.
posted by boo_radley at 3:41 PM on June 29, 2010 [1 favorite]


I find that remembering that the perfect is the enemy of the good to be a powerful tonic.

Teh suck is also the enemy of the good. We'd be better off remembering that.
posted by ChurchHatesTucker at 3:46 PM on June 29, 2010 [2 favorites]


boo_radley: It's good you didn't say you were ashamed, else you'd have to go on Diane Sawyer's show and say you were sorry.
posted by Eyebeams at 4:00 PM on June 29, 2010


OK, I've read the report, and it's pretty compelling to me. But in the interest of pursuing the null hypothesis, is there any way -- any way at all, no matter how convoluted-- that this could be the result of bad computer programming or thickheaded assumptions or stupid freshman stats 101 errors or ANYTHING other than straight making the numbers up?
posted by KathrynT at 4:27 PM on June 29, 2010


...is there any way ... that this could be the result of bad computer programming or thickheaded assumptions or stupid freshman stats 101 errors or ANYTHING other than straight making the numbers up?


Not likely. See this post which graphs some of the problems with the results (techdirt discusses the same here.) This wasn't a rounding error or anything like that.
posted by ChurchHatesTucker at 4:35 PM on June 29, 2010 [1 favorite]


I am in no way sticking up for R2k as it seems beyond obvious that they are fraudulent asshats. But I can imagine how the male/female parity thing got messed up:
function MakeRandomFavorableStats() {
  int menApprove = Round(46 + GetRandomFluctuation());
  int menDisapprove = Round(43 + GetRandomFluctuation());
  int menUnknown = 100 - menApprove - menDisapprove;
  // ... same code for womenApprove, womenDisapprove, womenUknown
}

function Round(float numberToRound) {
  static bool roundUpThisTime = false; // Don't round in the same direction every time! Might be suspicious.
  roundUpThisTime = !roundUpThisTime; // Each time we get called, round in the opposite direction
  if(roundUpThisTime) {
    return floor(numberToRound);
  } else {
    return ceiling(numberToRound);
  }
}

posted by 0xFCAF at 4:45 PM on June 29, 2010


OK, I've read the report, and it's pretty compelling to me. But in the interest of pursuing the null hypothesis, is there any way -- any way at all, no matter how convoluted-- that this could be the result of bad computer programming or thickheaded assumptions or stupid freshman stats 101 errors or ANYTHING other than straight making the numbers up?

Basically, it looks like they're either totally fraudulent, or so staggeringly incompetent that nobody can tell the difference.
posted by Tomorrowful at 4:49 PM on June 29, 2010


Addendum: And if the latter, they've fucked up so catastrophically that nobody can even figure out how they did it.
posted by Tomorrowful at 4:50 PM on June 29, 2010 [2 favorites]


Well, that's why you've got to get the raw data, survey responses. Indeed, if they wanted to do this well they should have generated data by using a montecarlo processes with initial probabilities that lined up with their original data. Indeed, that's probably what they were trying to do all weekend, cooking up fake data, thus begging for more time.
posted by delmoi at 5:15 PM on June 29, 2010 [1 favorite]


0xFCAF: That's interesting, but there's no reason to think they weren't just using that wonky Round() function with real data. That would be an example of incompetence, not malice.
posted by delmoi at 5:23 PM on June 29, 2010


0xFCAF: That's interesting, but there's no reason to think they weren't just using that wonky Round() function with real data. That would be an example of incompetence, not malice.

That would be an example of incompetence being indistinguishable from malice.
posted by ChurchHatesTucker at 5:44 PM on June 29, 2010


Interesting. 2k's lawyer, Beckler,

claimed that Kos "wont even pay his goddamn bill. He owes [Ali] $50, $60, $70,000 dollars, something in that neighborhood."

It's a sad truth but money does dirty the water. The C&D obviously makes 2K look like a bunch of idiots, but the issue of dKos possibly owing them money may damage Markos' credibility in the public arena. I'd be very interested in what Markos has to say about the billing issue.
posted by DarlingBri at 5:58 PM on June 29, 2010


Previously. Quite a bit of skepticism in that thread about the poll results. Evidently, the skeptics were onto something.
posted by Dojie at 3:50 PM on June 29 [+] [!]

Props to the skeptics. Against that, the confirmation bias on display from the other (way more numerous) commenters who gobbled the whole poll down like crack ice cream with nary a critical quiver because everybody knows those people are just like that is awesome too, in its own way.
posted by jfuller at 6:33 PM on June 29, 2010


DalingBri, TPM quotes Markos on the billing issue in that very article. “By the way, that $50-70,000 number is nonsense. We've paid for all the polling we were supposed to have received.”
posted by kipmanley at 6:35 PM on June 29, 2010


Oh I saw that "we've paid" quote but it wasn't very.... fullsome. It just puts us in he said/he said territory.

Not that Kos's billables are any of my business, but really I would like him to be lily white here. Straight good over evil is a much better story.

I suspect this will turn out to be something like either a) Kos has paid his bills in full and the other side is full of shit, or b) Kos has paid for X week of polling service, but additional charges were incurred for Y poll format changes or Z data over the service level agreement, and those charges are in dispute.
posted by DarlingBri at 6:49 PM on June 29, 2010


A commenter at Slashdot writes:
I used to work a different major polling company, and I can assure you R2K is not alone in just making up numbers. Easily 80% of surveys that went through my region were completely falsified, and the remaining 20% rarely matched the demographic they were supposed to be answering for. Survey administrators have quotas, and then get paid extra for additional surveys past that, but there is basically nothing done to verify any of the surveys turned in, and everybody in the company knows it. Don't always trust what you read, especially not statistics.
posted by alms at 6:54 PM on June 29, 2010 [1 favorite]


I hope the fact that this happened not long after Strategic Vision was caught will lead to a wholesale examination of the polling industry. I do some public opinion polling in my research, and I've frequently felt a twinge of suspicion at poll results, but I'm not enough of statistician (or motivated enough, frankly) to have ever looked into it. Faking data is easy, and I assume it's more profitable than collecting data legitimately. Maybe the connection between Kos's demand for transparency and R2K being (presumably) caught doing bad will lead other media poll sponsors to do the same, though I'm not holding my breath.
posted by aaronetc at 7:04 PM on June 29, 2010


Oh I agree the Kos quote is a wee bit—studied, for my taste. But you asked for what he had to say on the billing issue. And so.
posted by kipmanley at 7:17 PM on June 29, 2010


DarlingBri, you're ignoring c) R2K's lawyer is flinging shit to obscure the issue and accomplish exactly what they seem to have accomplished with you: casting a baseless pall of suspicion on Kos.
posted by fatbird at 7:24 PM on June 29, 2010


A Continuum of Lies, Ranked From Least to Greatest:
4) Lies
3) Damned Lie
2) Statistics
1) Whatever it is that you call the drek produced by polling firms.
posted by kaibutsu at 12:34 AM on June 30, 2010


"He owes [Ali] $50, $60, $70,000 dollars, something in that neighborhood."

If they can't even keep track of their accounts receivable, how could they have the chops to do statistics?
posted by Jimmy Havok at 1:54 AM on June 30, 2010 [4 favorites]


Haha, good point Jimmy. You would think if they were trying to collect they would know what the bill was within like $5000 ballpark give or take. Or maybe the exact amount like everyone who tries to bill me.
posted by furiousxgeorge at 3:46 AM on June 30, 2010


I hope the fact that this happened not long after Strategic Vision was caught will lead to a wholesale examination of the polling industry.

Amen! I would love to see this happen. Given all the different ways polling has come to play such an indispensable role in our democratic processes--from shaping media narratives to influencing and reinforcing political trends, and potentially even influencing election outcomes--the polling industry should be tightly regulated and held accountable to much higher standards than is currently the case, if you ask me.
posted by saulgoodman at 7:58 AM on June 30, 2010


Research 2000's legal team sounds like they're unloading all barrels to try and get some kind of general public opinion against Markos, in that it seems to be the only way they won't immediately be losing most of their other clients. That doesn't sound too extraordinary in the typical legal fight. But between the CnD to Nate Silver and now claiming they have a criminal case against DailyKos, they sound completely bonkers.
posted by XQUZYPHYR at 8:12 AM on June 30, 2010


Jimmy Havok: "He owes [Ali] $50, $60, $70,000 dollars, something in that neighborhood."

If they can't even keep track of their accounts receivable, how could they have the chops to do statistics?


Maybe we're not familiar with how polling companies do invoicing?

"Total due: $60,000 with a margin of error ± $10,000"
posted by MCMikeNamara at 10:13 AM on June 30, 2010 [3 favorites]


...and now claiming they have a criminal case against DailyKos

From that TPM article is a link to the Lexington Herald-Leader:
"'I can tell you, we're fine. What we're going to reveal, that will be the end of the Daily Kos,' Ali said. 'I can say, it has to do with people owing money.'"
So, the alleged money issue seems to be the tack they're taking. Obfuscation, indeed!
posted by ericb at 10:56 AM on June 30, 2010


like a lot of "old school" political bloggers, i've known Markos personally and professionally in the 10 years i've been a blogger, so am going to sit back and wait for the shit to really hit the fan.

he wouldn't make this dispute public unless there was a bigger fish to fry and am going to put it out there --this is the beginning of a bigger effort to regulate the polling industry. given there's elections this year, i wouldn't be shocked if we saw congressional hearings on the matter of "how do we know if the poll is kosher if they keep their raw data secret".

it's an opportunity for something bigger, not am means to an end. if this were just about some company hoodwinking him, he wouldn't be making such a huge deal about it.


posted by liza at 11:03 AM on June 30, 2010 [2 favorites]


Jonah Goldberg | National Review:
"A longtime reader makes some good points:
Jonah,

I don't know about you, but I can't believe that so many people are giving Kos credit - even calling him laudible - in this situation. According to Nate Silver, Kos has known for most of the year that there were serious questions about his pollsters, yet he let that too-good-to-be-true (from his point of view) anti-Republican data sit out there unquestioned for months.

The only reason Kos acted when and how he did was because he was over a barrel - the independent report was going to come out one way or another, so he had to act quickly to retain a veneer of respectability for himself. If he was really a selfless man committed to the truth, shouldn't he ask OEN to take down the R2K-based op ed he wrote trumpeting the awfulness of all things and persons Republican? Shouldn't he submit a new op ed in order to reach the wider audience who has seen his bad polling but has not seen the polling retraction?

The fact of the matter is, Kos has been - and continues to be - content to let the negative assumptions based on his published data remain in the air for as long as possible. He knew almost as soon as he first published it that this R2K anit-GOP numbers would come into question. He could have retracted it at any time, but he chose to let that bad data sit out there for months, causing as much damage as possible until he had absolutely no choice but to act.

I think this entire deal makes him look even more odious than he ever has."
posted by ericb at 11:13 AM on June 30, 2010 [1 favorite]


And the scrutiny is broadening:
"Research 2000 came to our attention in 2008, when its pulse-taking in Missouri yielded curious results.

Missourians picked a new attorney general in '08. One month before the August primary, Research 2000 said Jeff Harris was the Democrat to beat. Harris led the next closest competitor, Chris Koster, by 10 percentage points, in a Research 2000 poll.

Harris ended up finishing third, behind Koster, the winner, and Margaret Donnelly. (Koster won the general election, as well.)

Asked about the poll in the wake of very different results, Research 2000 president Del Ali shrugged.

'To be honest, I'm not concerned, like, "Where did we go wrong,"' Ali told me in 2008. Ali said it was 'ludicrous' to expect precision from a poll that had asked about down-ballot candidates a month in advance of an election.

Working on behalf of the St. Louis Post-Dispatch and KMOV-TV, Research 2000 also sampled Missouri voters' preferences in the presidential race. A Research 2000 poll in July showed Barack Obama leading John McCain by five percentage points.

It was a startling finding. Other polls taken at around the same time showed McCain-Palin leading the race for Missouri's electoral votes."
posted by ericb at 11:18 AM on June 30, 2010


How did Research 2000 conduct its polls for Daily Kos? Here's one possibility.
posted by ericb at 11:23 AM on June 30, 2010


Regarding Ali and money: Polling firm accused of distorting data has history of financial trouble, court documents show.
posted by ericb at 11:25 AM on June 30, 2010


According to Nate Silver, Kos has known for most of the year that there were serious questions about his pollsters, yet he let that too-good-to-be-true (from his point of view) anti-Republican data sit out there unquestioned for months.

Now this is an accusation that might actually have legs (though I'm automatically a bit sceptical of whatever comes out of National Review). There was a period when Kos was using R2K, after 538 had started demonstrating that they weren't very good pollsters (although 538 wasn't examining R2K for fraud the way they did with Strategic Vision). If Kos' disclosure was demonstrably motivated by the imminent release of the statistician's report, then it really does take the sheen of his offensive against them.
posted by fatbird at 11:30 AM on June 30, 2010


R2K has nothing to lose by pushing back as hard as possible. That report is pretty damning, though. They have some serious explaining to do.
posted by Mental Wimp at 11:51 AM on June 30, 2010


I should also say, it is really, REALLY hard to make up data impervious to competent statistical forensics. You practically have to have real data on which to base your fake data, which sort of defeats the purpose of faking data. Distributions not only of the individual variables but also of their associations with every other variable need to look right. And even then, they can look "too right". You go out enough moments (the mean, the variance, skewness, kurtosis, etc.) and the data faker will have neglected something.

The chief way of telling whether a clinical center in a multicenter trial is fudging data is by examining the structure of the values in the suspected data compared to those from other centers. It is quite easy to spot that way, really.
posted by Mental Wimp at 12:03 PM on June 30, 2010


Jonah Goldberg!!! LOL!!! He thinks someone else is odious!!!
posted by Jimmy Havok at 12:43 PM on June 30, 2010 [2 favorites]


Looking at the fivethirtyeight archives, it looks like nate started doing the pollster comparisons in early March. So I'm not totally sure where the 'most of the year' claim comes from. It seems the timeline is that pollster rankings were made and R2k sucked and got fired by Kos. Then they did the analysis - taking their time because of the severity of the claims - and then released their findings.

The notion of this being the lead of a push to open up the data at polling firms is very interesting, and would be an excellent outcome, if it were to happen.
posted by kaibutsu at 1:01 PM on June 30, 2010


So I'm not totally sure where the 'most of the year' claim comes from<>

I know where it came from. The same place Jonah gets all his facts, straight out of his ass.

posted by Jimmy Havok at 2:00 PM on June 30, 2010


I don't know about you, but I can't believe that so many people are giving Kos credit - even calling him laudible - in this situation. According to Nate Silver, Kos has known for most of the year that there were serious questions about his pollsters, yet he let that too-good-to-be-true (from his point of view) anti-Republican data sit out there unquestioned for months.

Oh right sure. Most sites wouldn't even to their userbase's concerns that the results produced by their pollsters were suspect.

Now this is an accusation that might actually have legs

Naw, is not. It's just someone taking their shot where they can to confirm to the loyal followers that they're right like they've always been. It is a mechanical process; the shot is able to be taken, and so the shot is taken. Axles and gears.
posted by JHarris at 5:40 PM on June 30, 2010


Research 2000 and its president Del Ali are so screwed now. I found the statistical analysis compelling and even if they blame the problem on an employee nobody is going to trust them. They are probably stripping the company of assets in anticipation of law suits from people who have used them.

Del Ali is in a loosing battle for R2K's life (or maybe it is just a holding action) which reflects in this quote "What we're going to reveal, that will be the end of the Daily Kos." .

I guess one way to distract from your immanent destruction is talk about the destruction of your accuser.
posted by MonkeySaltedNuts at 11:18 PM on June 30, 2010


So I'm not totally sure where the 'most of the year' claim comes from.

Nate emailed Kos in February about doubts he had with R2K.
posted by empath at 5:40 AM on July 1, 2010


It's not like "it's the coverup, not the scandal" is a new idea or anything.

You print this as if you know for certain that every scandal that gets covered up eventually comes out. I'd posit it's the opposite that's true. I'm even cynical enough to believe that many low level scandals are red herrings to throw the sharks off the trail of the REAL scandal - that was the basic MO of the Bush administration.
posted by any major dude at 7:15 AM on July 1, 2010


Del Ali's must-read rant to TPM Muckraker. Not only did he apparently not clear this statement with legal counsel, it doesn't even look spell-checked.
posted by activitystory at 7:02 AM on July 2, 2010


Some more info from TPM about the background of Research 2000's incoherent-rant prone president here.

The really interesting part, for me, from the above link was this info:
Ali's first employment in the polling world was at the firm Mason-Dixon, where he was a vice president from 1990 to 1998. He says he did sales there as well as "a lot of the stuff I do now."
I've long suspected Mason-Dixon polls weren't entirely on the up-and-up. That outfit's past associations with this Ali character doesn't do anything to ease those misgivings, from where I sit.
posted by saulgoodman at 9:42 AM on July 2, 2010


« Older In the months preceding the release of The Empire ...  |  Google makes Picasa, YouTube, ... Newer »


This thread has been archived and is closed to new comments