Motivated Numeracy and Enlightened Self-Government
September 5, 2013 8:25 PM Subscribe
Motivated Numeracy and Enlightened Self-Government. "Why does public conflict over societal risks persist in the face of compelling and widely accessible scientific evidence? We conducted an experiment to probe two alternative answers: the “Science Comprehension Thesis” (SCT), which identifies defects in the public’s knowledge and reasoning capacities as the source of such controversies; and the “Identity-protective Cognition Thesis” (ICT) which treats cultural conflict as disabling the faculties that members of the public use to make sense of decision-relevant science.
In our experiment, we presented subjects with a difficult problem that turned on their ability to draw valid causal inferences from empirical data. As expected, subjects highest in Numeracy—a measure of the ability and disposition to make use of quantitative information—did substantially better than less numerate ones when the data were presented as results from a study of a new skin-rash treatment. Also as expected, subjects’ responses became politically polarized—and even less accurate—when the same data were presented as results from the study of a gun-control ban. But contrary to the prediction of SCT, such polarization did not abate among subjects highest in Numeracy; instead, it increased. This outcome supported ICT, which predicted that more Numerate subjects would use their quantitative-reasoning capacity selectively to conform their interpretation of the data to the result most consistent with their political outlooks. We discuss the theoretical and practical significance of these findings." Comments on the Kahan et al study by Chris Mooney at Mother Jones.
In our experiment, we presented subjects with a difficult problem that turned on their ability to draw valid causal inferences from empirical data. As expected, subjects highest in Numeracy—a measure of the ability and disposition to make use of quantitative information—did substantially better than less numerate ones when the data were presented as results from a study of a new skin-rash treatment. Also as expected, subjects’ responses became politically polarized—and even less accurate—when the same data were presented as results from the study of a gun-control ban. But contrary to the prediction of SCT, such polarization did not abate among subjects highest in Numeracy; instead, it increased. This outcome supported ICT, which predicted that more Numerate subjects would use their quantitative-reasoning capacity selectively to conform their interpretation of the data to the result most consistent with their political outlooks. We discuss the theoretical and practical significance of these findings." Comments on the Kahan et al study by Chris Mooney at Mother Jones.
does the "massive, longitudinal media campaign by obscenely wealthy industries" theory come up at all?
::adjusts tinfoil hat at a jaunty angle::
posted by Jon_Evil at 9:04 PM on September 5, 2013 [5 favorites]
::adjusts tinfoil hat at a jaunty angle::
posted by Jon_Evil at 9:04 PM on September 5, 2013 [5 favorites]
This is no easy problem for most people to solve: Across all conditions of the study, 59 percent of respondents got the answer wrong.
This is very disturbing. The problem was trivial and it should take barely a glance and less than 2 seconds to solve. So while this study does appear to show what they claim it shows it also shows that people are terrible at math, logic, and basic reasoning skills. Really terrible.
I was definitely predisposed to believe these results. It always amazes me how people will accept bad math and bad logic when it supports their ideological positions but come down on it like a ton of bricks when it attacks their ideological positions. Or start spouting random pseudo-intellectual noise like "correlation is not causation!!!11!1!1".
In any case, another relevant quote from Upton Sinclair:
posted by Justinian at 9:06 PM on September 5, 2013 [6 favorites]
This is very disturbing. The problem was trivial and it should take barely a glance and less than 2 seconds to solve. So while this study does appear to show what they claim it shows it also shows that people are terrible at math, logic, and basic reasoning skills. Really terrible.
I was definitely predisposed to believe these results. It always amazes me how people will accept bad math and bad logic when it supports their ideological positions but come down on it like a ton of bricks when it attacks their ideological positions. Or start spouting random pseudo-intellectual noise like "correlation is not causation!!!11!1!1".
In any case, another relevant quote from Upton Sinclair:
It is difficult to get a man to understand something, when his salary depends upon his not understanding it!"Substitute "worldview" for "salary" and it becomes even more difficult.
posted by Justinian at 9:06 PM on September 5, 2013 [6 favorites]
This just in: for most of human history, tribalism conferred more of a survival advantage than math skills.
I'd be glad to see the "deficit model" die off...oh but wait, if the "deficit model" of human consciousness agrees with someone's political beliefs, they'll probably ignore these numbers and continue right on believing in it anyways. Science Defeated By Irony, curses!
But it's true, I'd be really happy to see the end of the "oh those [people I disagree with] are so ignorant and unenlightened, if only they were better educated they'd stop [doing that bad thing, e.g., voting against their own interests] and [agree with me]" line of argumentation (which puts in plenty of appearances here on the Blue). They are very probably just as smart as you are, and quite possibly smarter, levels of formal education notwithstanding; however they identify with a different tribe than you do. Convincing a person to leave their tribe and join yours is a very different thing than convincing them they were incorrect about some facts.
posted by mstokes650 at 9:12 PM on September 5, 2013 [12 favorites]
I'd be glad to see the "deficit model" die off...oh but wait, if the "deficit model" of human consciousness agrees with someone's political beliefs, they'll probably ignore these numbers and continue right on believing in it anyways. Science Defeated By Irony, curses!
But it's true, I'd be really happy to see the end of the "oh those [people I disagree with] are so ignorant and unenlightened, if only they were better educated they'd stop [doing that bad thing, e.g., voting against their own interests] and [agree with me]" line of argumentation (which puts in plenty of appearances here on the Blue). They are very probably just as smart as you are, and quite possibly smarter, levels of formal education notwithstanding; however they identify with a different tribe than you do. Convincing a person to leave their tribe and join yours is a very different thing than convincing them they were incorrect about some facts.
posted by mstokes650 at 9:12 PM on September 5, 2013 [12 favorites]
The problem was trivial and it should take barely a glance and less than 2 seconds to solve.
I don't know, I spend a lot of my days looking at quantitative data but I still don't think I could orient myself correctly in a 2x2 contingency table, calculate the margins (223+75, 107+21), and then calculate the treatment success rates over those margins (223/298, 107/128) in my head in less than 2 seconds. The rates are within the same ballpark (75%, 84%) so it's not like the answer can be easily estimated at a glance. Maybe I'm missing something about how trivial this problem is supposed to be?
I mean, I totally understand the sentiment, since I've been in the position of trying to teach this kind of quantitative reasoning to undergraduates and was surprised at how many weeks it took for some of them to really catch on. But then, I was lucky enough to have bright, privileged, and relatively well-prepared students, which makes me wonder if there really is something about this type of reasoning that is inherently difficult.
posted by ootandaboot at 9:44 PM on September 5, 2013 [1 favorite]
I don't know, I spend a lot of my days looking at quantitative data but I still don't think I could orient myself correctly in a 2x2 contingency table, calculate the margins (223+75, 107+21), and then calculate the treatment success rates over those margins (223/298, 107/128) in my head in less than 2 seconds. The rates are within the same ballpark (75%, 84%) so it's not like the answer can be easily estimated at a glance. Maybe I'm missing something about how trivial this problem is supposed to be?
I mean, I totally understand the sentiment, since I've been in the position of trying to teach this kind of quantitative reasoning to undergraduates and was surprised at how many weeks it took for some of them to really catch on. But then, I was lucky enough to have bright, privileged, and relatively well-prepared students, which makes me wonder if there really is something about this type of reasoning that is inherently difficult.
posted by ootandaboot at 9:44 PM on September 5, 2013 [1 favorite]
Yeah i definitely used a calculator because fuck doing arithmetic in my head, but I got the right answer.
Anyway, the interesting part of the result isn't that people are stupid. The interesting thing is that the degree to which they answer math corrections correctly is correlated to how well the results match their ideological disposition. Very clever study, and matches my experience with arguing with people who are Wrong On The Internet and I'd love to see some people try to reproduce it with different sorts of questions.
posted by empath at 9:51 PM on September 5, 2013
Anyway, the interesting part of the result isn't that people are stupid. The interesting thing is that the degree to which they answer math corrections correctly is correlated to how well the results match their ideological disposition. Very clever study, and matches my experience with arguing with people who are Wrong On The Internet and I'd love to see some people try to reproduce it with different sorts of questions.
posted by empath at 9:51 PM on September 5, 2013
And the explanation for why makes sense to me, as well, and it's probably good to think about for anyone taking standardized tests, as well as for researchers -- getting a wrong answer that matches my first approximation from looking at the question is probably the #1 reason I've gotten stuff wrong on those sorts of things.
It might explain the results of that awful austerity study. They may not have intentionally done a bad study. They may have simply made a mistake that matched their ideological bias and then didn't bother to double check thoroughly.
posted by empath at 9:56 PM on September 5, 2013
It might explain the results of that awful austerity study. They may not have intentionally done a bad study. They may have simply made a mistake that matched their ideological bias and then didn't bother to double check thoroughly.
posted by empath at 9:56 PM on September 5, 2013
The thing I don't think I quite understand is why high-numeracy subjects perform so well on the skin-cream condition, which presumably doesn't threaten their identity in any meaningful way, if people are supposed to resort to the more effortful, deliberative form of reasoning only when they are unhappy with the results they come to with a simple heuristic (which, as I understand it, is what the authors claim in the middle of p. 23).
posted by ootandaboot at 10:02 PM on September 5, 2013 [1 favorite]
posted by ootandaboot at 10:02 PM on September 5, 2013 [1 favorite]
Because people who test as 'high-numeracy' are people who expend the effort to properly figure out all the questions they're presented with and have the intellectual tools to do it.
A significant percentage of people who aren't high numeracy will get the right answer that matches their bias either purely by guessing or will get the wrong answer initially and then fix it using either correct or incorrect reasoning.
posted by empath at 10:18 PM on September 5, 2013 [1 favorite]
A significant percentage of people who aren't high numeracy will get the right answer that matches their bias either purely by guessing or will get the wrong answer initially and then fix it using either correct or incorrect reasoning.
posted by empath at 10:18 PM on September 5, 2013 [1 favorite]
The rates are within the same ballpark (75%, 84%) so it's not like the answer can be easily estimated at a glance. Maybe I'm missing something about how trivial this problem is supposed to be?
You don't have to calculate the rates. You just have to note that the success rate was roughly 3x the fail rate (223 vs 75) in the first row and roughly 5x the fail rate (107 vs 21) in the second row. Therefore whichever contingency is used for the first row is "worse" than the second row. So if the cream is used for the first row it means the cream makes things worse. If the cream is used for the second row it means the cream makes things better.
Noticing the ratios are 3/1 and 5/1 takes barely a second.
posted by Justinian at 10:23 PM on September 5, 2013 [2 favorites]
You don't have to calculate the rates. You just have to note that the success rate was roughly 3x the fail rate (223 vs 75) in the first row and roughly 5x the fail rate (107 vs 21) in the second row. Therefore whichever contingency is used for the first row is "worse" than the second row. So if the cream is used for the first row it means the cream makes things worse. If the cream is used for the second row it means the cream makes things better.
Noticing the ratios are 3/1 and 5/1 takes barely a second.
posted by Justinian at 10:23 PM on September 5, 2013 [2 favorites]
Maybe the real trick for these sorts of problems is being able to quickly reduce it to an easy problem rather than being able to do the calculations?
posted by Justinian at 10:25 PM on September 5, 2013 [2 favorites]
posted by Justinian at 10:25 PM on September 5, 2013 [2 favorites]
I definitely know people who would not even recognize that you could determine that sort of question using math, rather than, I dunno, looking at which is more 'natural' or was made by a 'mom'.
posted by empath at 10:49 PM on September 5, 2013
posted by empath at 10:49 PM on September 5, 2013
The thing I don't think I quite understand is why high-numeracy subjects perform so well on the skin-cream condition, which presumably doesn't threaten their identity in any meaningful way, if people are supposed to resort to the more effortful, deliberative form of reasoning only when they are unhappy with the results they come to with a simple heuristic (which, as I understand it, is what the authors claim in the middle of p. 23).
The authors make some statements at the end of the paper, such that I interpreted it as 3 cases: for numerates, under a) neutral decision situation, their "numeracy" facilities are active by default, which lets them perform well; under b) antagonistic conditions, they use their numeracy to double-check and so arrive at the correct answer; and finally c) under group-think conditions, they disable their numeracy, succumbing to confirmation bias, and so get the wrong answer. You can search for the word "disable" in the paper.
posted by polymodus at 11:07 PM on September 5, 2013 [3 favorites]
The authors make some statements at the end of the paper, such that I interpreted it as 3 cases: for numerates, under a) neutral decision situation, their "numeracy" facilities are active by default, which lets them perform well; under b) antagonistic conditions, they use their numeracy to double-check and so arrive at the correct answer; and finally c) under group-think conditions, they disable their numeracy, succumbing to confirmation bias, and so get the wrong answer. You can search for the word "disable" in the paper.
posted by polymodus at 11:07 PM on September 5, 2013 [3 favorites]
A simpler explanation is that the higher-numeracy people are just lying.
I mean, think about what it would be like to receive this survey. If I received the one that asserts that gun bans increase crime, with just the 2x2 table, my immediate conclusion would be that this is a horseshit push-poll trying to fake some results showing that gun control is unpopular, or otherwise trying to hamper gun control efforts. Because that would be such a spectacularly stupid study that I could never possibly believe that it was real.
So, my answers switch to whatever option best approximates "Fuck you, asshole."
posted by ROU_Xenophobe at 11:23 PM on September 5, 2013 [3 favorites]
I mean, think about what it would be like to receive this survey. If I received the one that asserts that gun bans increase crime, with just the 2x2 table, my immediate conclusion would be that this is a horseshit push-poll trying to fake some results showing that gun control is unpopular, or otherwise trying to hamper gun control efforts. Because that would be such a spectacularly stupid study that I could never possibly believe that it was real.
So, my answers switch to whatever option best approximates "Fuck you, asshole."
posted by ROU_Xenophobe at 11:23 PM on September 5, 2013 [3 favorites]
I mean, think about what it would be like to receive this survey. If I received the one that asserts that gun bans increase crime, with just the 2x2 table, my immediate conclusion would be that this is a horseshit push-poll trying to fake some results showing that gun control is unpopular, or otherwise trying to hamper gun control efforts. Because that would be such a spectacularly stupid study that I could never possibly believe that it was real.
That's kind of not the way that academic studies are done. They'd be doing it under the aegis of the university and wouldn't prime you to believe it's a political survey. They'd also mix the ideological questions in with a bunch of completely unrelated math questions.
posted by empath at 11:38 PM on September 5, 2013 [1 favorite]
That's kind of not the way that academic studies are done. They'd be doing it under the aegis of the university and wouldn't prime you to believe it's a political survey. They'd also mix the ideological questions in with a bunch of completely unrelated math questions.
posted by empath at 11:38 PM on September 5, 2013 [1 favorite]
I'd love to see some people try to reproduce it with different sorts of questions.
You’re in a desert walking along in the sand when all of the sudden you look down, and you see a tortoise...
posted by titus-g at 3:35 AM on September 6, 2013 [2 favorites]
You’re in a desert walking along in the sand when all of the sudden you look down, and you see a tortoise...
posted by titus-g at 3:35 AM on September 6, 2013 [2 favorites]
Do you make up these questions, Mr. titus-g? Or do they write 'em down for you?
posted by DreamerFi at 4:04 AM on September 6, 2013 [2 favorites]
posted by DreamerFi at 4:04 AM on September 6, 2013 [2 favorites]
Do you make up these questions, Mr. titus-g? Or do they write 'em down for you?
Well, Hampton Fancher and David Peoples did. As you know.
posted by Gelatin at 5:47 AM on September 6, 2013
Well, Hampton Fancher and David Peoples did. As you know.
posted by Gelatin at 5:47 AM on September 6, 2013
A simpler explanation is that the higher-numeracy people are just lying.
You're a smart guy, I know from your posts. You are pro-science and rationality. I put it to you that this finding threatens your world view. And further I observe that you've come up with a very intelligent, plausible argument against it!
posted by alasdair at 7:04 AM on September 6, 2013 [2 favorites]
You're a smart guy, I know from your posts. You are pro-science and rationality. I put it to you that this finding threatens your world view. And further I observe that you've come up with a very intelligent, plausible argument against it!
posted by alasdair at 7:04 AM on September 6, 2013 [2 favorites]
If I see data about skin cream, I assume the data at least vaguely reflect something real according to some accepted standard of experimental design. And when a skin cream is associated with a rash getting better, I expect that to be causal.
If I see data about gun control, I assume they were cherry picked by somebody who's got an axe to grind and is probably abusing a metric, because that's what you get on both sides of that debate. And when a crime statistic is associated with gun control or the lack thereof, I have reason to question whether that's causal; there are tons of possible confounding factors and even a possibility that the arrow of causality is reversed.
In this particular case, the "gun control" example, as presented to the participants, seemed to use overall crime statistics as a proxy for the effectiveness of a legislative intervention. No matter how you collect such data, they're rarely that dispositive. It seems reasonable to expect a numerate person to notice that.
So if the subjects thought they were being asked to address the "real" meaning of "real" data, there are a lot of reasons they might be suspicious of the gun control data and not of the skin cream data. AND those reasons for suspicion are related to their prior subjective probabilities that the measurement is valid. And the researchers chose a metric that conservative republicans have heard before and are already primed to reject.
So maybe they're at least partly measuring people's priors, rather than their rationality. Or at least maybe that's part of it; it still wouldn't explain the paradoxical "hardening" effect unless they believed the source was likely to actively lie.
I'm not saying there isn't something real in this, but it seems to me that it's is being overinterpreted.
posted by Hizonner at 7:04 AM on September 6, 2013 [1 favorite]
If I see data about gun control, I assume they were cherry picked by somebody who's got an axe to grind and is probably abusing a metric, because that's what you get on both sides of that debate. And when a crime statistic is associated with gun control or the lack thereof, I have reason to question whether that's causal; there are tons of possible confounding factors and even a possibility that the arrow of causality is reversed.
In this particular case, the "gun control" example, as presented to the participants, seemed to use overall crime statistics as a proxy for the effectiveness of a legislative intervention. No matter how you collect such data, they're rarely that dispositive. It seems reasonable to expect a numerate person to notice that.
So if the subjects thought they were being asked to address the "real" meaning of "real" data, there are a lot of reasons they might be suspicious of the gun control data and not of the skin cream data. AND those reasons for suspicion are related to their prior subjective probabilities that the measurement is valid. And the researchers chose a metric that conservative republicans have heard before and are already primed to reject.
So maybe they're at least partly measuring people's priors, rather than their rationality. Or at least maybe that's part of it; it still wouldn't explain the paradoxical "hardening" effect unless they believed the source was likely to actively lie.
I'm not saying there isn't something real in this, but it seems to me that it's is being overinterpreted.
posted by Hizonner at 7:04 AM on September 6, 2013 [1 favorite]
« Older TIFs explained with sharpies | Found in Translation Newer »
This thread has been archived and is closed to new comments
posted by Zalzidrax at 8:30 PM on September 5, 2013 [11 favorites]