Anti-Racist Graphics Research (SIGGRAPH 2021)
August 9, 2021 5:52 PM Subscribe
Theodore Kim's SIGGRAPH 2021 keynote on anti-racist graphics research: Computer graphics research has a race problem. (SLYT)
"What I mean is that our basic scientific formulations have insidious biases built into them. This is an extremely uncomfortable and disturbing version of this because many of us are used to thinking that math is math and physics is physics, and we resist the idea that science could contain any sort of bias. In fact this is what attracted many of us to the sciences to begin with. We get to look at these clean neutral problems all day and we don't have to get all tangled up in the ugly politics of the real world. Unfortunately it doesn't look like this is the case. [...] Historically racist assumptions have been baked into many of our everyday institutions whether we like it or not. [...] Computer graphics is not an exception to this."
Some books mentioned in the talk:
* Algorithms of Oppression, by Safiya Noble
* Race After Technology, by Ruha Benjamin
* Girl Head, by Genevieve Yue
* Captivating Technology, edited by Ruha Benjamin, Lorna Roth's chapter is "Making Skin Visible Through Liberatory Design"
The original Lorna Roth paper is "Looking at Shirley, the Ultimate Norm: Colour Balance, Image Technologies, and Cognitive Equity"
"What I mean is that our basic scientific formulations have insidious biases built into them. This is an extremely uncomfortable and disturbing version of this because many of us are used to thinking that math is math and physics is physics, and we resist the idea that science could contain any sort of bias. In fact this is what attracted many of us to the sciences to begin with. We get to look at these clean neutral problems all day and we don't have to get all tangled up in the ugly politics of the real world. Unfortunately it doesn't look like this is the case. [...] Historically racist assumptions have been baked into many of our everyday institutions whether we like it or not. [...] Computer graphics is not an exception to this."
Some books mentioned in the talk:
* Algorithms of Oppression, by Safiya Noble
* Race After Technology, by Ruha Benjamin
* Girl Head, by Genevieve Yue
* Captivating Technology, edited by Ruha Benjamin, Lorna Roth's chapter is "Making Skin Visible Through Liberatory Design"
The original Lorna Roth paper is "Looking at Shirley, the Ultimate Norm: Colour Balance, Image Technologies, and Cognitive Equity"
Physics, generally, is still "just" physics.
The problem is that, like film stock being optimized for lighter skin tones, computer graphics modeling is just straight up ignoring most of the actual physics of how light interacts with dark colored skin (specifically the degree to which subsurface scattering is hugely different between very light and very dark skin). It's the disregarding of the physical processes of how dark skin is captured with film stock back before, and now how the physics of light interaction with darker skin is just disregarded. By ignoring non-white skin, the researchers are just pretending that the physics of light's interactions with darker skin doesn't matter. And THAT is the problem.
If our systems didn't pretend that whiteness is the default and non-whiteness is an exception to that norm, but actually accounted for the physics of light interaction with all of the varying kinds and tones of human skin, we'd be actually accounting for the physics that's just waiting there.
posted by tclark at 6:36 PM on August 9, 2021 [15 favorites]
The problem is that, like film stock being optimized for lighter skin tones, computer graphics modeling is just straight up ignoring most of the actual physics of how light interacts with dark colored skin (specifically the degree to which subsurface scattering is hugely different between very light and very dark skin). It's the disregarding of the physical processes of how dark skin is captured with film stock back before, and now how the physics of light interaction with darker skin is just disregarded. By ignoring non-white skin, the researchers are just pretending that the physics of light's interactions with darker skin doesn't matter. And THAT is the problem.
If our systems didn't pretend that whiteness is the default and non-whiteness is an exception to that norm, but actually accounted for the physics of light interaction with all of the varying kinds and tones of human skin, we'd be actually accounting for the physics that's just waiting there.
posted by tclark at 6:36 PM on August 9, 2021 [15 favorites]
So Jean Luc Godard in 1977 said that Kodak film was racist, and that lesson still has NOT been learned?
By people who never use film, and were born years if not decades after that? No, that lesson was not learned, and must be taught repeatedly.
And I want to note that SIGGRAPH, as a CS research community, has many, many immigrant researchers who don't identify as white. It's a little daunting to think that the burden to fix things falls upon other, racially oppressed groups, and I think the 'well obviously they should have learned better by now' ignores who 'they' is.
posted by pwnguin at 7:01 PM on August 9, 2021 [1 favorite]
By people who never use film, and were born years if not decades after that? No, that lesson was not learned, and must be taught repeatedly.
And I want to note that SIGGRAPH, as a CS research community, has many, many immigrant researchers who don't identify as white. It's a little daunting to think that the burden to fix things falls upon other, racially oppressed groups, and I think the 'well obviously they should have learned better by now' ignores who 'they' is.
posted by pwnguin at 7:01 PM on August 9, 2021 [1 favorite]
That was a very interesting talk, thanks for posting this. I remember that when Disney’s Brave came out , there was a lot of emphasis on how hard it was to get Merida’s hair right.
posted by dhruva at 8:26 PM on August 9, 2021 [1 favorite]
posted by dhruva at 8:26 PM on August 9, 2021 [1 favorite]
About to embark on this (and maybe write about it.. thanks for the find, I've been trying but failing to keep up with SIGGRAPH) but it reminded me that Unreal's Metahumans show a better future for people of color in games and CG in general.
Also this reminds me of the paper The Whiteness of AI, which points out a similar problem in the conception, execution, and perception of AI.
posted by BlackLeotardFront at 8:51 PM on August 9, 2021
Also this reminds me of the paper The Whiteness of AI, which points out a similar problem in the conception, execution, and perception of AI.
posted by BlackLeotardFront at 8:51 PM on August 9, 2021
> that Unreal's Metahumans show a better future
The author has some interesting comments about that too, at the end. He shows how metahumans have a subsurface component (the glowy effect on skin) that is way too strong and pink for darker skin.
posted by haemanu at 9:02 PM on August 9, 2021
The author has some interesting comments about that too, at the end. He shows how metahumans have a subsurface component (the glowy effect on skin) that is way too strong and pink for darker skin.
posted by haemanu at 9:02 PM on August 9, 2021
Some big takeaways I got from it:
- A lot of the models weren't just white, they were actually just one white guy because that's just what was available for free use. It became a bit like the benchmark which the "China head"/"Roth head" was previously. It just what was there and inertia picked it up. This is the "systemic" park of systemic racism.
- Subsurface scattering is a relatively new development. Corridor Crew has an interesting video on it. Basically they used to more or less slap an image on a model. They found that different objects, not just people, had different subsurface scattering. Think how light shines through the thin skin of your ear lobe versus the solid color of an ear in most cartoons. Adding this added to realism. I'm going on a limb and saying most graphic artists used what tools were available to commercial rendering engines to make different face types and we found out through trial and error that high melatonin skin types scatter light fundamentally differently. It just isn't a darker white person.
- There are some great phrases for SIGGRAPH but in general for reviewers and commentators to look out for when reviewing papers. Don't just assume because it works for the reference image (usually a white person) that it will work elsewhere and that the paper needs a higher burden of proof.
Really great! I work a lot with targeted digital advertising and I wonder what biases I'm not meaning to introduce. I could see a world where a suburban white person gets incredibly targeted ads that an ML algorithm picks up on because they buy products where a white person is opening up the beer so the A/B testing wins out and that ad gets shown to them, reducing all diversity. Too easy to say it is just math or blame it on the algorithm.
posted by geoff. at 9:20 PM on August 9, 2021 [5 favorites]
- A lot of the models weren't just white, they were actually just one white guy because that's just what was available for free use. It became a bit like the benchmark which the "China head"/"Roth head" was previously. It just what was there and inertia picked it up. This is the "systemic" park of systemic racism.
- Subsurface scattering is a relatively new development. Corridor Crew has an interesting video on it. Basically they used to more or less slap an image on a model. They found that different objects, not just people, had different subsurface scattering. Think how light shines through the thin skin of your ear lobe versus the solid color of an ear in most cartoons. Adding this added to realism. I'm going on a limb and saying most graphic artists used what tools were available to commercial rendering engines to make different face types and we found out through trial and error that high melatonin skin types scatter light fundamentally differently. It just isn't a darker white person.
- There are some great phrases for SIGGRAPH but in general for reviewers and commentators to look out for when reviewing papers. Don't just assume because it works for the reference image (usually a white person) that it will work elsewhere and that the paper needs a higher burden of proof.
Really great! I work a lot with targeted digital advertising and I wonder what biases I'm not meaning to introduce. I could see a world where a suburban white person gets incredibly targeted ads that an ML algorithm picks up on because they buy products where a white person is opening up the beer so the A/B testing wins out and that ad gets shown to them, reducing all diversity. Too easy to say it is just math or blame it on the algorithm.
posted by geoff. at 9:20 PM on August 9, 2021 [5 favorites]
pwnguin: By people who never use film, and were born years if not decades after that?
I learned about the 1931 CIE colour space this year for work. It's amazing how many colour conversion matrices in virtually every piece of software that deals with colour are based on experimental results from 17 people who happened to be available to researchers 90 years ago.
posted by clawsoon at 9:41 PM on August 9, 2021 [5 favorites]
I learned about the 1931 CIE colour space this year for work. It's amazing how many colour conversion matrices in virtually every piece of software that deals with colour are based on experimental results from 17 people who happened to be available to researchers 90 years ago.
posted by clawsoon at 9:41 PM on August 9, 2021 [5 favorites]
Not a thing I knew I needed to be exposed to. Thank you.
posted by dangerousdan at 12:44 AM on August 10, 2021 [2 favorites]
posted by dangerousdan at 12:44 AM on August 10, 2021 [2 favorites]
It's amazing how many colour conversion matrices in virtually every piece of software that deals with colour are based on experimental results from 17 people who happened to be available to researchers 90 years ago.
This is true, but I would note that the colour matching functions captured in CIE 1931 have been reproduced and confirmed many times since. The 1931 model has persisted because it has proven accurate enough to remain useful, rather than because of a lack of wider sampling or later improvements. There are several improved models available, but they differ only very slightly.
Differences in human colour perception generally relate to specific permutations of the photoreceptor proteins (opsins) that occur in the cone cells of the eye. Most people have the same three opsins, roughly corresponding to red, green and blue. The CIE 1931 model describes the trichromatic colour perception available from this combination, and is remarkably accurate given the technology available when it was produced.
The various forms of "colourblindness" involve one or more of these three opsins being absent or nonfunctional, giving rise to dichromatic or monochromatic vision.
A small number of people with two X chromosones have a fourth opsin, and in a fraction of them it is functional, giving true tetrachromacy. However the fourth opsin is in the middle of the spectrum, essentially a different green - so the overall range of visible wavelengths is the same as with the common three opsins.
posted by automatronic at 2:23 AM on August 10, 2021 [5 favorites]
This is true, but I would note that the colour matching functions captured in CIE 1931 have been reproduced and confirmed many times since. The 1931 model has persisted because it has proven accurate enough to remain useful, rather than because of a lack of wider sampling or later improvements. There are several improved models available, but they differ only very slightly.
Differences in human colour perception generally relate to specific permutations of the photoreceptor proteins (opsins) that occur in the cone cells of the eye. Most people have the same three opsins, roughly corresponding to red, green and blue. The CIE 1931 model describes the trichromatic colour perception available from this combination, and is remarkably accurate given the technology available when it was produced.
The various forms of "colourblindness" involve one or more of these three opsins being absent or nonfunctional, giving rise to dichromatic or monochromatic vision.
A small number of people with two X chromosones have a fourth opsin, and in a fraction of them it is functional, giving true tetrachromacy. However the fourth opsin is in the middle of the spectrum, essentially a different green - so the overall range of visible wavelengths is the same as with the common three opsins.
posted by automatronic at 2:23 AM on August 10, 2021 [5 favorites]
automatronic: This is true, but I would note that the colour matching functions captured in CIE 1931 have been reproduced and confirmed many times since.
I'd be curious to know (and this is something I don't know the answer to): How many times have those confirmatory studies used non-Europeans? It's entirely possible that the operation of colour vision is virtually universal, and I can't think of any reason why it wouldn't be, but... do we know that for sure?
posted by clawsoon at 5:17 AM on August 10, 2021 [1 favorite]
I'd be curious to know (and this is something I don't know the answer to): How many times have those confirmatory studies used non-Europeans? It's entirely possible that the operation of colour vision is virtually universal, and I can't think of any reason why it wouldn't be, but... do we know that for sure?
posted by clawsoon at 5:17 AM on August 10, 2021 [1 favorite]
I'd be curious to know (and this is something I don't know the answer to): How many times have those confirmatory studies used non-Europeans? It's entirely possible that the operation of colour vision is virtually universal, and I can't think of any reason why it wouldn't be, but... do we know that for sure?
I'm not qualified to say definitively - this isn't my field, just a rabbit hole I went down after I got into colorimetry for photography purposes years ago. But I think we're pretty sure. Testing of colour perception is routinely done to diagnose the various forms of colourblindness. It's so widespread and well understood, that it seems practically impossible for there to be major functional variations that have gone undetected.
Colour vision is one of the aspects of human biology that seems relatively straightforward. We know what the opsin proteins are, we know their spectral sensitivities, we know the specific genes that code for them, which chromosones they're on and how they're inherited. It's hard to prove a negative but if there were further variations out there, I'd expect them to have been spotted by now - either from the functional side or the genetic side or both.
Although it's only recently that people with true functional tetrachromacy have been identified, it was theorised well in advance and the researchers that found it knew exactly where to look: at women with particular patterns of colourblindness in their children, which meant that the mother could have the right combination of X chromosones with both OPN1MW and OPN1MW2 genes.
There are some statistical differences along racial lines with regard to things like the frequencies of different forms of colourblindness, as one would expect for anything driven by genetics. And for instance this paper shows differences in the ratios of L and M ("red" and "green") cone cells between black & white men, and goes into the genetics behind that.
The known variations can influence individual ability to distinguish similar colours under certain conditions, but don't really factor into the overall model of the perceptible colour space set out in CIE 1931.
posted by automatronic at 6:28 AM on August 10, 2021 [7 favorites]
I'm not qualified to say definitively - this isn't my field, just a rabbit hole I went down after I got into colorimetry for photography purposes years ago. But I think we're pretty sure. Testing of colour perception is routinely done to diagnose the various forms of colourblindness. It's so widespread and well understood, that it seems practically impossible for there to be major functional variations that have gone undetected.
Colour vision is one of the aspects of human biology that seems relatively straightforward. We know what the opsin proteins are, we know their spectral sensitivities, we know the specific genes that code for them, which chromosones they're on and how they're inherited. It's hard to prove a negative but if there were further variations out there, I'd expect them to have been spotted by now - either from the functional side or the genetic side or both.
Although it's only recently that people with true functional tetrachromacy have been identified, it was theorised well in advance and the researchers that found it knew exactly where to look: at women with particular patterns of colourblindness in their children, which meant that the mother could have the right combination of X chromosones with both OPN1MW and OPN1MW2 genes.
There are some statistical differences along racial lines with regard to things like the frequencies of different forms of colourblindness, as one would expect for anything driven by genetics. And for instance this paper shows differences in the ratios of L and M ("red" and "green") cone cells between black & white men, and goes into the genetics behind that.
The known variations can influence individual ability to distinguish similar colours under certain conditions, but don't really factor into the overall model of the perceptible colour space set out in CIE 1931.
posted by automatronic at 6:28 AM on August 10, 2021 [7 favorites]
« Older Making Photography in a Surveillance State | Gonna Make You Sweat Newer »
This thread has been archived and is closed to new comments
All of this fabulous technology repeating the same mistakes we made without using the technology - definitely snafu.
posted by Barbara Spitzer at 6:34 PM on August 9, 2021 [8 favorites]