Rethinking the IAT
January 12, 2017 12:12 PM   Subscribe

The Implicit Association Test (IAT) has been a popular method in psychological studies to evaluate effects of implicit bias against women or minorities. The creation of an easy-to-use, online version of the test has led to many references in non-academic literature as well. However, the test as used for measuring racial bias may not measure what it's supporters purport and has been shown to have low test-retest reliability. In a lengthy article, Jesse Singal discusses the history of the test.
posted by demiurge (18 comments total) 25 users marked this as a favorite
 
If these criticisms are as serious as they seem, it's hard to understand how the IATs got this far. I have been skeptical about them as measures of what really underlies racism for a while. But I had just assumed that they were reasonably reliable between multiple administrations, and reasonably well validated against some other measures of racist attitudes. Apparently they aren't.

Pretty good Times magazine story on research suggesting that implicit bias is the wrong target for explaining and fighting racism.
At first he thought things were going pretty well. Some Protestant boys built what seemed like genuine friendships with some Catholic boys. But on the last day of the program — after three weeks of nature walks, impromptu dialogues and trust-building exercises — a fight broke out between two participants that quickly devolved into a full-scale, 250-child brawl: Catholics against Protestants. [...]

But for three subjects, the psychological and neurological tests contradicted each other. The psychological tests indicated that they held the same types of anti-Arab biases as the other Israeli subjects, but their brain scans, and their reasonableness ratings, indicated that they were able to identify with the Arab perspective nonetheless. All three of these outliers, it turned out, were Israeli peace activists. In a scatter plot of the study’s results, in which blue dots represented the Israeli subjects and red dots represented the Palestinian ones, the peace activists stood out: three specks of blue in a quadrant of red.

posted by grobstein at 12:24 PM on January 12, 2017 [1 favorite]


I'm finding the first part of the article hard to agree with on theoretical grounds, it's saying things about how different groups of scientists, media, etc., interpret and use the IAT, but all I really see is a deep confusion about the map looking like the territory. The IAT is obviously influenced from the behavioralist legacy of theories of psychology, i.e. the approach of "explaining" human behavioral phenomena at a low level in the individual mind, and any scientist knowledgeable about the issues there would be cognizant of how IAT fits in the bigger picture. Asking the question of whether the IAT measures "implicit bias" is conceptually backwards, and that's more a sign of the quality of scientific discourse in mainstream media and/or isolated/competing communities of scientists. Even if the IAT produced random noise, it is and was still something conceptually valuable in its own right.
posted by polymodus at 1:20 PM on January 12, 2017 [3 favorites]


I really have no background to comment on whether the IAT tells you something useful/interesting on an aggregate level - I can believe that it does but also that implicit bias is only one influence on behavior. What has always bugged me though is when the test is presented in a lay context as something individuals can and should take to scientifically reveal their personal innate prejudices. I am really doubtful that factors like the order the tasks are presented - which is of course randomized but again I'm talking about what you get when an individual takes the test once - don't have a significant impact, especially if it's your first time encountering the format. Not least because I've taken them a few times - including when online tests were posted here - and seen some fluctuation in the results.
posted by atoxyl at 1:20 PM on January 12, 2017 [5 favorites]


It's just so... this one weird trick will prove how racist you really are! (Probably not fair to blame on the academics but that wasn't my intention.)
posted by atoxyl at 1:30 PM on January 12, 2017 [2 favorites]


I took the online test last time it got linked here. It said I was significantly biased in favour of the test setup where questions you'd naturally answer "yes" to both get categorized in the same direction. Well, either that or I've somehow become subconsciously trained to like Arabs.
posted by sfenders at 1:58 PM on January 12, 2017


I can't comment on the merits of this particular article, but author Jesse Singal is well known in trans circles for writing an extremely flawed article about trans children. Very long rebuttal (author name unknown), medium rebuttal (Parker Molloy), medium rebuttal (Julia Serano). In short, I don't trust anything he writes.
posted by AFABulous at 2:36 PM on January 12, 2017 [6 favorites]


Perhaps, just as bravery is not the absence of fear but the ability to consciously overcome it, and goodness is not the absence of wicked impulses but the ability to resist them when they arise, wokeness is not the absence of accultured prejudice but the ability to consciously keep it in check...
posted by acb at 6:28 PM on January 12, 2017 [9 favorites]


Disclaimer: After reading 3/4 through I had to resort to skimming the rest (long article is loooong)

It seems like there are many indications that the IAT, especially in the case of examining race, is simply a poor tool as it currently exists. if it can't reliably reproduce results amongst individuals, and it doesn't appear to strongly predict behaviour in any specific direction, what exactly is it good for?

It's unfortunate, because it seems likely that implicit bias exists and plays just as much of a role in today's society as the race IAT says. But we shouldn't use a weak or faulty tool to demonstrate something just because it appears to agree with us.
posted by Nutri-Matic Drinks Synthesizer at 6:42 PM on January 12, 2017


Even if the IAT produced random noise, it is and was still something conceptually valuable in its own right.

I'm having trouble seeing what you mean by this. If the IAT produces random noise, then most of the claims made for it are false. It tells you nothing about a person's likely behavior towards other people, for example. What would it be good for?

I am on board with the program of coming up with instruments or operational measures for fuzzier concepts, like racial bias, that are hard to reduce to behavioral precision. But our instruments / operational measures have to pass some yardstick of realism to be worth adopting. In psychology, this is usually measured by their "reliability" and "validity." Reliability means, if you measure the same thing twice you should get similar results. Validity means, the thing you're measuring should have some statistical relationship to other constructs that intuitively measure the same thing. If you don't have those, you have very little.
posted by grobstein at 7:26 PM on January 12, 2017 [5 favorites]


Jesse Singal is pretty well known for getting on a hobbyhorse and ignoring actual contrary evidence in pursuit of his narrative. The trans community knows him really well for this; he's also one of the pundits who thinks political correctness on college campuses is What's Wrong With America, and very recently he went all-in on some stuff happening in the YA Fiction community, and ignored the actual people involved when they tried to connect his thirdhand story.

I'd take any "science says" article written by Singal with an entire barrel of salt, is what I'm saying.
posted by adrienneleigh at 8:31 PM on January 12, 2017 [6 favorites]


Asking the question of whether the IAT measures "implicit bias" is conceptually backwards, and that's more a sign of the quality of scientific discourse in mainstream media and/or isolated/competing communities of scientists. Even if the IAT produced random noise, it is and was still something conceptually valuable in its own right.

Perhaps I'm misunderstanding your point, but the article claims that some IAT proponents are making strong real-world claims about how IAT scores link to behavior. In a meaningful way, with imp measurable acts both by individuals and in aggregate.
posted by mark k at 11:17 PM on January 12, 2017


Sampling any real word data you get random noise, along with any statistically significant patterns. The presence of noise is no kind of criticism whatever.
posted by iotic at 1:02 AM on January 13, 2017 [2 favorites]


Sampling any real word data you get random noise, along with any statistically significant patterns. The presence of noise is no kind of criticism whatever.

The comment in question is suggesting that IAT measures would be valuable even if they only produced random noise.

The mere presence of noise is not at issue, and the "random noise" remark above makes no sense on that interpretation.
posted by grobstein at 10:47 AM on January 13, 2017 [1 favorite]


That depends on how you interpret the phrase "produced random noise". In any case the article is seeking to disparage the IAT on the basis of "reliability" and "validity". Reliability because the data is noisy, as you'd expect from any real world test involving human responses, and validity because they haven't found a link to racist behaviour - though testing that is even more fraught with issues, so again unsurprising. The article in the introductory paragraphs even seems to suggest the only reason people believe racist bias exists, helping to explain continuing disparities, is because of the acceptance of this test's usefulness(!) I'm taking issue with the article's emphasis on "reliability" and noise somehow reducing the importance of the findings (despite the author's admission that the trends are nevertheless clear and present), and agreeing with the commenter who seemed to me to be doing similar, in part.

I'm more interested with the validity or otherwise of aspersions the article seeks to throw on the IAT. Arguing about whether noise itself contains data seems off-topic
posted by iotic at 12:00 PM on January 13, 2017 [2 favorites]


It bothers me how reporting about IAT seems to conflate racism with a state of being rather than behaviours.
posted by mikek at 12:53 PM on January 13, 2017 [1 favorite]


Something can have noise, enough noise to limit test-retest reliability, and still tell you something useful with enough tests. However used on a small sample, with a small number of tests, the value of the results would be very dubious. This is what my comment was getting at - I (definitely not an expert in the field) am certainly not prepared to dismiss that the IAT does something worthwhile, but I have seen it presented in inappropriate ways. I think you're arguing (iotic) that Singal is making too much of a leap from "this test is too unreliable to use on individuals" to "this test is no good, period?" Yeah, maybe, but he does approach it from other angles as well.

I remember when it was posted here a while ago thinking even that version did a rather poor job explaining the limitations and presenting the results in context - you'd get "strong bias against," "weak bias against," "weak bias towards," etc., without indication of how scores are divided into those bins. And I think that may have been the Harvard thing mentioned?
posted by atoxyl at 2:06 PM on January 13, 2017 [1 favorite]


Yeah, I do get the feeling he's making that leap too much. Despite being careful with the statement of his case, the overall tone of the essay is one of detracting from the pretty evident power and significance of the test, which I'm guessing is fairly clear to anyone who has tried it, and is backed up by well-established results and trends. A more useful approach might be to admit the utility of the IAT, whilst analysing its shortcomings and perhaps even suggesting ways to improve such tests. At least, that would be a better approach if the aim is to better understand implicit bias. However, it often feels more like the author is trying to undermine the test's ability to show anything about implicit bias - and perhaps even to undermine the existence of such biases. If those are his aims, he fails to achieve them.
posted by iotic at 3:19 PM on January 13, 2017 [1 favorite]


I'm definitely sensing a high degree of implicit Jesse Singal bias here.
posted by Cogito at 7:19 PM on January 13, 2017 [2 favorites]


« Older Videos of cookery in an Indian village   |   I Grew Up In The Rust Belt, But I’m Not In Any Of... Newer »


This thread has been archived and is closed to new comments