Following polls has become our vision of what political participation is
September 30, 2024 12:11 PM   Subscribe

That error opens up onto the myriad conceptual fallacies built into the entire enterprise, if something so unavoidable can be called an “error.” Past performance is no guarantee of future results; but past performance is all a pollster has to go on. That’s why much of the process of choosing and weighting samples is … well, you can call it “more art than science.” Or you can call it “intuitive.” Or you can call it “trial and error.” But you can also call it “made up.” from The Polling Imperilment, an essay by Rick Perlstein built around Joseph Campbell's 2020 book Lost in a Gallup [The American Prospect; ungated]
posted by chavenet (40 comments total) 8 users marked this as a favorite
 
I'll add this brief piece by Jay Kaspian Kang into the mix.

I have long been quite critical about the amount of space the polls take up in our political discourse. The reasons are pretty simple: although polls certainly have a place in assessing the state of the election, they’ve inspired a type of sophistry in which the pundit or the politician flashes the results of some fallible poll and treats it as irrefutable proof of the will of the electorate. The result is a tower of bad takes, built upon a foundation of solid polls and good pollsters. The question is not whether we should “trust the polls.” It’s whether the onslaught of analysis and extrapolation that invariably follows them actually holds any predictive or explanatory power.
posted by kensington314 at 12:34 PM on September 30 [13 favorites]


I get so many terrible polls where I'm presented with a list of twenty issues and asked to pick five and rank them in the order of importance.

Maybe there's some ten-thousand foot view that makes the exercise worthwhile, but I honestly can't see that there's much meaning in determining whether "Combating Systemic Racism" is slightly more or less important than "Protecting Democracy" while measuring how either of those compare with "Ending Childhood Hunger" as a priority.

Fuck pollsters.
posted by RonButNotStupid at 12:40 PM on September 30 [7 favorites]


Atwood said. “Not every best candidate wins. That’s just how it is.”

Or maybe God answered your prayers, you ignorant *!($*#$&)%(%

I love that a literal fucking plague while their guy is an office couldn't possibly be a sign to these folks that maybe they're on the wrong path.
posted by Abehammerb Lincoln at 12:46 PM on September 30 [9 favorites]


There's the problem with polls and how politics/elections are covered in the media.

And then there's the difficulty of designing surveys that are clear, not unintentionally leading, and actually interesting. Regardless of topic! A lot of my students want to conduct surveys (Pew Research link) as part of their research and I always groan a little bit because survey design is SO HARD.
posted by spamandkimchi at 12:57 PM on September 30 [7 favorites]


Every poll that has my preferred candidate ahead is an infallible predictor of the election. Every poll that has them behind is partisan propaganda from some hack that's using bad data.
posted by fortitude25 at 12:58 PM on September 30 [2 favorites]


So what's the alternative? "Vibes?" Whatever the NYT decides it is?

Polls are deeply imperfect. But they beat the pants off of everything else.
posted by Frayed Knot at 1:01 PM on September 30 [6 favorites]


So what's the alternative? "Vibes?" Whatever the NYT decides it is?

Asking this with no snark, but alternative or "vibes" as regards what? What's the "it" in, "Whatever the NYT decides it is?" The election is the ultimate "it," but what are polls thought to be addressing, in this question? Just an overall need for political punditry? Is your question, "What is the alternative to polls for people who want political punditry and horse race coverage?" Or is it something else?
posted by kensington314 at 1:05 PM on September 30 [8 favorites]


You can’t ungate something that doesn’t have a paywall!
posted by Going To Maine at 1:06 PM on September 30 [1 favorite]


I actually bought a domain to post info about how off the polls have been in the past few years but I haven't had time to do anything with it. Argh.

I linked to some of the data I'd seen about polls being off in a comment back in June (I should just put THAT up at my incomplete website, it'd be better than nothing). There's also the way the polls were way off for the latest elections in France and the UK.

There are so many other indicators we should be folding in to the conversation:

* volunteers
* campaign contributions
* number of donors
* % and $ totals of donations from small donors

I wish we got more and better reporting on those indicators.

Thanks for this, chavenet. I appreciate the opportunity for us to talk about the many problems with polls these days.
posted by kristi at 1:09 PM on September 30 [5 favorites]


The article posted goes on at some length about how the polls are as inexact as they were 100 years ago, and should not be trusted to tell us how people will vote come election day.

So what will? Campaign's need to use something to decide where to put limited resources. I need to use something to decide where I ought to donate or volunteer.

And news outlets will inevitably use something to frame stories and develop narratives.

I asked it in a snarky way, because I'm frustrated by the article (I think it's basic assertion is dead wrong), but it's a serious question. These needs must be met. If not via polls, then what?
posted by Frayed Knot at 1:12 PM on September 30 [2 favorites]


I'd appreciate polls and pollsters much more if they stopped obfuscating voter sentiment behind vague concepts like "enthusiasm" and "double-haters" and just honestly tried to represent why voters support a given candidate. It frustrates me that I have to withhold honest criticism and always give five stars lest my preferred candidates chances be jeopardized because the polls show them to be "weak".

And news outlets will inevitably use something to frame stories and develop narratives.

Maybe they should try using candidates' policies to frame news stories? Like how one candidate is literally spreading malicious lies about immigrants?
posted by RonButNotStupid at 1:22 PM on September 30 [8 favorites]


Instead of polls (and especially candidate approval polls) being the media bread-and-butter for election season, I think getting a snapshot of what people want to know / worried about and then doing a series of actually reported articles on those topics would be amazing.

For example, taking the results from this Pew Research survey on voters' top issues where the options are already preset, and then actually digging into what aspects of the economy are stressful, and how that might differ for different age groups or economic classes etc, and if the candidates have addressed those in any meaningful way.

Issues will land differently because of ideology and (ahem) media coverage -- see crime/immigration/climate change. But also will land differently because how an individual is situated. Stock markets are up! I am, thanks to being in a public pension, one of those 62% of Americans who is categorized as an investor in the stock market. But on a gut level, record high stock markets give me no joy, only frustration in the hyperfinancialization of the economy and the dominance of the FIRE industries (finance, insurance, real estate) in economic coverage. Health care costs continue to rise. A concept of a plan isn't going to cut it.
posted by spamandkimchi at 1:23 PM on September 30 [10 favorites]


Maybe we should just go with hat sales (NYT).

“We saw virtually no Biden sales all year,” he said.
But when Ms. Harris became the nominee and Mr. Walz joined the ticket, enthusiasm for their campaign — and for the style of hat that came to be associated with it — led to a huge increase in Unionwear’s production.
The factory usually produces 2,500 hats a day, maxing out at around 4,000. Lately, Mr. Cahn said, it’s churning out about 5,000 Harris-Walz hats a day.Since 2008, he said, the number of hats ordered by a presidential campaign has foretold the winner of that year’s election.
In 2016, Mr. Cahn said, the Hillary Clinton campaign hadn’t moved many baseball hats.
“That was fascinating to us because she did not sell any hats,” Mr. Cahn said. “We were just trying to reconcile that with the presumption that she was going to win.”

posted by jenfullmoon at 1:25 PM on September 30 [20 favorites]


Surely the needs of campaigns to target resources must be met, and they rely heavily on internal polling that never gets into the punditry blender. Granted.

When 538 was a lot more popular, I always really resented the site because I felt it treated political coverage like sports betting. (Still, did I obsessively click through the page everyday? At times, I certainly did.) They modeled elections based on polling instead of, say, hiring a journalist to go cover what was going on with Michigan voters. (My ambivalence about this approach to political journalism showed up here in an AskMe, though my objection wasn't about polls, per se, and was more about election modeling a la 538.)

I use the Michigan example because in 2016 I remember going to the Clinton campaign office in East LA to call Michigan voters and having the creeping realization that the reason all the phone numbers were bad is because I was tapped into some list of people that was so economically vulnerable, they couldn't keep a steady phone number. That moment, days before the election, felt more telling than months of reading poll aggregations. I learned something, aside from the fact that polls with a 4% margin of error showed a tight race.

I think the reason this article resonated with me is that for the general public (as opposed to the campaign), I don't really understand the value of polling coverage. It fulfills a certain compulsive urge, and it allows many of us to feel informed. But what have we learned?

Anyway, I don't say this with much stridency. I look at polls. I read some polling punditry. I bank my hopes on what I take to be a polling trend toward Kamala Harris. I just don't think polling coverage is journalism and I always find myself wishing I could click through to rich information about the electorate in a given state. But I understand that as expensive as polling is, doing journalism well on a national scale is even more so.
posted by kensington314 at 1:27 PM on September 30 [6 favorites]


There's also this, from my current daily read, Electoral-vote.com:

"Who do you think will win?" may be the better question:
Going back 60 years and using state polling data, the leaders in the polls won the state 69% of the time. In contrast, the candidate the voters predicted would win actually won 81% of the time. In other words, asking people "Who do you think will win your state?" is a better predictor than asking people how they will vote and seeing who got the most votes.
posted by kristi at 1:36 PM on September 30 [14 favorites]


From the kristi link, I thought this was interesting:

The reason is not clear, but it could be that although a voter wants candidate X to win, he or she hears all the neighbors raving about candidate Y. The question about what the voter wants is just one data point, but the question about who will win the state collects information from many voters. That might actually be a better way to do forecasting.

Maybe tough to give this a very science-y valence, but it has a certain logic working in its favor!
posted by kensington314 at 1:39 PM on September 30 [4 favorites]


If you look back at the relatively modern era of U.S. presidential polling, starting in 1952 (after the famous "Dewey Defeats Truman" kerfluffle of 48), you see that national polling averages picked the popular vote winner every time, and in most cases to within a couple of percentage points.

Moderate outliers would be 1980, 1992 (shakes fist at Perot), and really 2020, when Biden underperformed the polling averages. 2016 was weird, but the polling predicted Clinton would win the popular vote--which she did.

Polling has had a couple of rough cycles, with polling orgs not adjusting for the Trump effect...and then overadjusting for it in 2022 in turn.

There's less polling being released to the public than ever, it seems like. Only about one in six U.S. House races have had any sort of polling released at all. Senate polling is thin outside of a few marquee races in swing states; it's been months since I've seen anything on the New Jersey race for Senate, as one example.

In 2022 there was a set of right-wing "polls" released that I'm pretty sure were flat-out fraudulent, made up out of thin air. Those got averaged into the polling trends with everything else, which polluted a lot of the state-by-state polling and made Republicans in those races look stronger than they really were.

Oh, and the NYT doesn't even really care about its own polls anyway. I remember at least one 2022 article where they released some Siena College polls that they had commissioned for Congressional races, the polls made Democrats look fairly good, the NYT was basically all "here's why you shouldn't believe these polls we just paid for".
posted by gimonca at 1:48 PM on September 30 [9 favorites]


The pollsters can likewise be arbitrary once the numbers come in. Lost in a Gallup notes a fascinating experiment carried out by Nate Cohn for The New York Times. He had four pollsters interpret the same raw data from a 2016 poll of Florida. Their choices in how to weight ranged from Clinton winning by four percentage points to Trump winning by one.

Is this supposed to be an argument against polling? It sounds like all the methods agree that the race is close and he variation induced by methodological choices is in a pretty narrow band.

They’re cocky about it; that’s a pattern, too.

I don't know what's meant by this. If you want to hear that poll numbers are probabilistic, fallible, and depend on a lot of unverifiable internal assumptions.... ask a pollster. They will also probably tell you that, for all this, polls are still a better indicator of where the race stands than the vibes Bret Stephens perceives in his neighborhood or what Thomas Friedman's cab driver told him. And they're right.
posted by escabeche at 1:56 PM on September 30 [8 favorites]


The article starts with "In 2016, I experienced the desolation of my candidate for president losing after the most respected polling experts told me she had a 71.4 percent, 85 percent, 98.2 percent, and even 99 percent chance of winning." That's not polling. That's overly clever people taking the second or third derivative of a tangent of a trend and presenting it as hard data. I remember those "99 percent" style predictions, they really didn't have any rationale for existing. And I don't see people are doing much of those since.
posted by gimonca at 2:16 PM on September 30 [7 favorites]


Two problems I see with polls are Nate Silver and Rasmussen.
posted by nofundy at 2:18 PM on September 30 [2 favorites]


I don't do polling but my work sometimes involves survey research. I've been in the field 25 years now and sampling design has gone from an interesting problem we must consider to a functional impossibility in a lot of our research. Phone numbers could be anywhere and people don't answer them anyway, panelists are fake, if you pay respondents you're swamped by bots which are now nearly undetectable, response rates are ridiculously low, etc. A lot of clients are moving to focus groups, which can't provide the same information but at least can tell you SOMETHING.
posted by metasarah at 3:26 PM on September 30 [9 favorites]


The media also capitalizes on the general public's ignorance of statistics. Tell someone that the polls show that Trump only has a 17% chance of winning and people figure he's as good as lost already. Tell them they have the same odds of getting a bullet playing Russian Roulette and all of a sudden no one wants to play (unless they've got a death wish).
posted by VTX at 3:29 PM on September 30 [2 favorites]


Now now now, that's not correct. Nate Cohn (NYT's lead polling entrails predictor) is a total POS tool as well!

Today he led with 'its been a quiet week on the campaign' story, you know, the week Trump went all in on immigrants as animals, the VP as having mental health problems, the magas vile Haitian immigrant hate-fest - now with bomb and death threats for anyone who also doesnt hate Haitians, and to top it off, the big orange rapist calling for a day of cop violence - hey remember with the NYT went bananas on the VP for saying Trump predicted a bloodbath is he didnt win?

In sum, the NYT politicos like Cohn are awful, and want Trump to 'win'.
posted by WatTylerJr at 3:32 PM on September 30 [1 favorite]


Rick Perlstein is a terrific historian and also a sweet person to boot.
posted by doctornemo at 3:53 PM on September 30 [1 favorite]


I'm a huge fan of Rick Perlstein's books, so I'm probably bringing a pretty generous reading ito this essay, but, for me, he doesn't quite make his case.

Like, he's criticizing polling, but he really ought to be criticizing journalism, or maybe he ought to be criticizing capitalism.

Past performance is no guarantee of future results; but past performance is all a pollster has to go on. That’s why much of the process of choosing and weighting samples is … well, you can call it “more art than science.” Or you can call it “intuitive.” Or you can call it “trial and error.” But you can also call it “made up.”

This sounds like someone complaining about meteorology.
posted by box at 3:55 PM on September 30 [4 favorites]


The preoccupation with polls seems like one facet of a larger tendency of our time, which is that the most politically sophisticated and engaged people I know spend a lot more time engaged in meta-political thinking about what the “average voter” wants than thinking about what they want. I worry that this constrains the political imagination severely.

On the other hand when I see the Nate Silver “sports betting” approach to politics contrasted with narrative exploration of issues and voter sentiment, there’s a part of me that wants to say - at least what Nate made his name doing is (sort of, if at unfortunately infrequent intervals) testable and accountable for its misses. The worst version of Nate Silver is clearly the version that strays from polls and odds into narrative punditry!
posted by atoxyl at 4:53 PM on September 30 [6 favorites]


I wouldn't have needed to read all the issues with polling he enumerated to get to where he ended up if he'd had atoxyl as his editor — "the most politically sophisticated and engaged people I know spend a lot more time engaged in meta-political thinking about what the 'average voter' wants than thinking about what they want" — which pulls forward what I thought the piece was promising going in.

The Clinton team coined "triangulation" in the '90s. I don't think we were meant to triangulate for them.
posted by mph at 6:03 PM on September 30 [4 favorites]


“Polls aren’t politics” - a very good point

“Polls aren’t accurate” - I dunno man, they are and they aren’t, that’s statistics
posted by atoxyl at 6:12 PM on September 30 [2 favorites]


I'm shocked people here have been polled! I don't know a single person who has been. Like...it really happens, I guess! Huh!
posted by tiny frying pan at 6:29 PM on September 30 [3 favorites]


I have been callled once by a national poll. It was when Terri Schiavo and the actions of Congress were in the news. I believe it was a polling firm associated with ABC news. I remember the pollster breaking script to plead with me that I should get a living will when I revealed that I didn't have one. This was 24+ years ago.
posted by mmascolino at 6:38 PM on September 30 [3 favorites]


But how do polls and horse-race coverage serve any public good? Like was said above, the ultimate poll is the election itself. In the meantime, news outlets should be providing the public with policy information. That kind of material can actually help people become better-informed voters.
posted by NotLost at 8:24 PM on September 30 [5 favorites]


Polls don't serve any public good whatsoever. If I was in charge, publication of polls about who will win an election would be banned. Or, at least, only be permitted as part of truthful, fact-checked and comprehensive analyses of policy positions from all sides. Any polls conducted by or on behalf of political entities would never, ever be permitted to see the light of day.

Actually, if you want to predict the outcome of elections fairly accurately, get bookmakers on the job. They make a good living from predicting outcomes that are influenced by so many variables it's practically witchcraft.
posted by dg at 8:48 PM on September 30 [3 favorites]


Atwood said. “Not every best candidate wins. That’s just how it is.”
I'm probably just being dense, but, if anyone else is in the same position as I am, of not being able to find this quote (which appeared in an early comment) in the early links, Googling found it in the story "After two years of far-right rule in a Michigan county, one chance to change it." (If I just missed a link, or if it's actually in the prominently linked articles and my skim just failed to notice it, then my apologies.)
posted by It is regrettable that at 9:27 PM on September 30 [1 favorite]


Oh, sorry, I hadn't looked at the rest of the front page. It was probably meant to be a comment on Ottawa Update. (Also, it's not the Atwater you think, at least not if you've been reading Reagan-era history like I was today.)
posted by It is regrettable that at 10:44 PM on September 30 [1 favorite]


the most politically sophisticated and engaged people I know spend a lot more time engaged in meta-political thinking about what the 'average voter' wants than thinking about what they want
What is so frustrating about American political discourse is that most non-engaged voters look to people who care more about politics to try and get a vibe of what's going on, so you get a poop ouroboros where regular people's confusion about politics becomes a signal about what the average voter feels, which said voters take as an indication for how they should feel, and so on

One of the reasons the far-right has been so successful is that they're the only ones talking about problems. Now, quite a lot of their 'problems' aren't problems, and their solutions are stupid, but journalists would be infinitely more useful if they would go and talk to some people and find out a problem. (The hollowing out of regional journalism is a huge factor here! A lot of the journalists who found out problems and wrote about them were local journalists, and now we're looking at the New York Times to do it when what they used to do was amplify the stories broken by people who live closer to where the stories are actually happening.)
posted by Merus at 12:45 AM on October 1 [3 favorites]


That kind of material can actually help people become better-informed voters.

Polling is a factor when I decide where to spend my attention and make donations to campaigns. My input is pretty tiny compared to the overall work and spend on those campaigns, but I'd at least like to send my contributions to close races where it will make a difference.

One frustration I have is that polling isn't being done on the vast majority of House races, or isn't being done publicly or independently. Absent that info, you can try to glean info from old-school pundit orgs like Cook Report or Sabato, who have their own partisan leans to be corrected for, or the mainstream media, who when they're not cheerleading their own candidate are pushing unhealthy bothsides-ism.

Or, you can look at financial reports for the candidates. Typically, any incumbent will have a big war chest, but with financial comparisons you can get an imperfect idea of what challengers might have a chance, who's getting support from the establishment, who might be a placeholder or an unserious candidate. But--you could also miss out on earnest, good candidates who deserve more attention. Or, you could donate to a no-chance candidate who's been soaking up donations because the incumbent is a notable media asshole.

None of that is a complete replacement from looking at policy to decide where to direct resources, of course.

More info is better than less, and polling is one of several items I like to look at. When it even happens!
posted by gimonca at 3:49 AM on October 1 [1 favorite]


Maybe the book covers territory the linked essay doesn't, but missing from this conversation (and most discussions of polling writ large in the year of our lord 2024) is sampling bias. Doesn't matter how big and representative and geographically disparate your cohort is; whatever your mechanism for contacting people is, it's going to skew your results.

In other words: most published poll data is claiming to represent "likely voters" (for whatever value of that term the poll's author dreamed up), but is actually representing "likely voters who answer calls from unknown numbers".
posted by Mayor West at 5:57 AM on October 1 [3 favorites]


Yup, that might be the problem nowadays.

Earlier versions of The Polling Problem were just about the demographics of the sample -- people who were willing to talk to pollsters were older and whiter and so on. *THAT* part isn't really a big problem -- you can just reweight the sample to count young people who respond as 1.5 people and old people as 0.5 people, or whatever reweighting gets you back to a nationally representative sample. But we can do that because we have gold-standard data from the Census Bureau about those demographic factors.

But that only works if the few 35 year old black women you get in the sample are very similar to the 35 year old black women who don't respond to polls. If that relationship breaks down -- and some of the crosstabs especially with respect to age are weird enough that either it's starting to break down or we're in the middle of a large-scale political realignment (but not in the polisci sense) -- then all bets are off. Polling doesn't really tell you anything about preferences or attitudes that are strongly correlated with the differences between responders and nonresponders.

Doesn't mean oooh everything is fine Harris is gonna roflstomp Trump. I just wouldn't be surprised to see the actual election results differ pretty notably, in either direction, from the endstage polling in the second half of October.
posted by GCU Sweet and Full of Grace at 8:06 AM on October 1 [3 favorites]


some of the crosstabs especially with respect to age are weird enough that either it's starting to break down or we're in the middle of a large-scale political realignment

All of the Nates will tell you to ignore the weird crosstabs because there’s no guarantee it’s a reasonable sample once it’s sliced like that. But people (including polling analysts when you put them on TV) love talking about these kinds of things.
posted by atoxyl at 8:39 AM on October 1 [1 favorite]


This reminded me of something from this recent UK Times article:
Labour’s lead was half the 20 points the industry said. It needs to admit it has a problem and try asking new questions...

This leads to a third consequence. We have to make sure that inaccurate polling does not become part of an inaccurate historical account. Neil Kinnock’s appearance at the Sheffield rally before the 1992 election is still described as a crucial event. Labour was winning until he overplayed his hand. Irritated by his triumphalism, voters turned against Labour at the last minute and Kinnock lost.

This common story about that election is almost certainly wrong. The polls were misleading from the beginning. Labour was always losing. Why does it matter that this is properly understood? The Sheffield rally version of events makes politicians think that political staging and gaffes are more important than policy positions and the economic fundamentals. They aren’t.
As the articles imply, I think a lot of mainstream politicians have interpreted volatile polling as indicating a volatile electorate, and run scared of saying or doing anything "controversial" in case it startles the voters.
posted by TheophileEscargot at 12:54 AM on October 2 [3 favorites]


« Older "A mic on the street that Shazams 24/7"   |   Playing Chicken Newer »


You are not currently logged in. Log in or create a new account to post comments.