Equations can't be racist
July 13, 2015 1:11 PM   Subscribe

What does it mean for algorithms to be fair? Our lives are increasingly influenced by opaque algorithms. Thoughts on how our existing laws are handling this new environment.
posted by laptolain (31 comments total) 36 users marked this as a favorite
 
An algorithm inherits all the conscious and subconscious biases it's human creator(s) have. End of story.
posted by SansPoint at 2:07 PM on July 13, 2015 [4 favorites]


An algorithm inherits[...]

I don't think that's it at all. TFA says very few people understand machine-learning algorithms anyway -- it's not like there's a knob that a clever, racist programmer can turn to "make the algorithm discriminate against X people." It could be a matter of overfitting the data, or simply that the data have been collected through discriminatory activity, but I'm not at all sure what the solution should be.

Should we require cops to stop, arrest (and shoot in the back) more white people, just so that the data that's used to train the algorithms will be more balanced? Maybe if they did arrest more white people, they'd discover that they're committing as many crimes as the non-whites that are getting arrested now.

I wonder what you'd find if you did racial analyses of rich-people crime -- tax fraud, embezzlement, securities, stuff like that. I'd guess that whites are overrepresented, but I can't remember ever hearing that that shows institutional racism on the part of whatever regulators investigate those types of crimes.
posted by spacewrench at 2:21 PM on July 13, 2015 [7 favorites]


An algorithm inherits all the conscious and subconscious biases its human creators knowingly or unknowingly put into it. Not all the ones they possess, nor only the ones they possess.

I don't know whether advertising algorithms can ever be free of class discrimination which in the United States is largely inseparable from racial discrimination, but it is at least in theory possible for them to be totally void of underlying human tribal xenophobia motivations and accompanying identity-based hate.

So there's that.
posted by Ryvar at 2:21 PM on July 13, 2015 [2 favorites]


I think that focusing on the biases (or not) of the programmer misses the most important point of this article. In fact, in many cases, it's not that the algorithms are even written by the programmers in any meaningful sense.

Rather, it's that the data about likelihood, if followed without question, predicts unfair outcomes. If opportunities and advertisements are dictated by demographic likelihood, and past purchases and decisions, even correct ones, it can still be unfair and/or illegal.

It's those (statistically, but not determinatively) "correct" algorithms that are tricky. Because in situations where we are legally and ethically committed to equal opportunity, we need to ignore certain demographic things (like race, religion, or income level) but also other markers (like certain shopping choices, location, or other patterns) that are highly correlated.
posted by mercredi at 2:33 PM on July 13, 2015 [20 favorites]


SansPoint: "An algorithm inherits all the conscious and subconscious biases it's human creator(s) have. End of story."

I think the sentiment is right, but the framing here gets it exactly wrong. Support vector machines (for instance) don't inherit the unconscious biases of the programmer, they acquire the biases hidden in the training signal. Yes, they become racist because we are racist, but it's not the coder or even the machine learning theory on max-margin classifiers that is the source of the problem. The problem is that we have created a world in which racist signals are everywhere: every time you train up a sufficiently powerful classifier on a sufficiently rich data set it will learn to use racist, sexist, homophobic cues because they are in the broader social environment that produces the data set. Efforts to minimize this effect are laudable and should absolutely be encouraged, but you won't solve the problem that way. This isn't merely a machine learning problem, it's a cultural problem that machine learning has exacerbated.
posted by langtonsant at 2:37 PM on July 13, 2015 [34 favorites]


Google’s algorithm shows prestigious job ads to men, but not to women. Here’s why that should worry you. [WaPo]

White House Worried About Discrimination Through Analytics [Slashdot]
The review, led by Obama's senior counselor, John Podesta, will outline concerns about whether methods used for commercial applications may be inherently vulnerable to inadvertent discrimination. 'He described a program called "Street Bump" in Boston that detected pot-holes using sensors in smartphones of citizens who had downloaded an app. The program inadvertently directed repair crews to wealthier neighborhoods, where people were more likely to carry smartphones and download the app.' 'It's easy to imagine how big data technology, if used to cross legal lines we have been careful to set, could end up reinforcing existing inequities in housing, credit, employment, health and education,' he said."
posted by Little Dawn at 2:43 PM on July 13, 2015 [9 favorites]


From TFA:

“Advertisers can choose to target the audience they want to reach, and we have policies that guide the type of interest-based ads that are allowed,” reads a statement from Google.

The challenge with testing this on ads is that you don't know what explicit biases are built into the ad buy - an advertiser can purchase ads and only target male viewer (or whatever google predicts as being a male viewer). So it may be explicit as opposed to implicit bias.

The implicit bias question is still very real though.
posted by GuyZero at 2:59 PM on July 13, 2015


Garbage in, garbage out.

The interesting thing is that we now realise this is equaly true of these learning algorithms. The real trick would be to formulate an algorithm which recognises these biases and can show them to us and correct for them.

An interesting thing to try would be to add an explicit, complementary expectation function to the reward function which would signal discrepancies. These could either be biases in the training signal, or might be actual differences found in reality.

In effect, the algorithm would gain a 'hypothesis function'. Which is making these thing start to look very like the scientific method, even if the way it crunches the data, comes to it's conclusions/working theory and applies that is quite opaque to us (and that opacity is something we really have to work on understanding).
posted by MacD at 3:17 PM on July 13, 2015 [1 favorite]


The real trick would be to formulate an algorithm which recognises these biases and can show them to us and correct for them.

The challenge is that to do this you'd have to collect EVEN MORE data.

To correct for a racial bias - say black loan applicants are being denied at much higher rates than whites for an online, mechanically evaluated application - you'd have to get applicants to explicitly give you their race. Which itself is probably illegal to ask for.
posted by GuyZero at 3:42 PM on July 13, 2015


When people train and test an algorithm, they often forget about people different from themselves, or different from the target market, or different from the "norm" (white cis male). This is true even of pre-computer technology. Even the choices of how databases are structure for names will impose difficulties on some groups but not others.

Additionally, when an algorithm is build using data mining and machine learning, it can codify patterns that are real (and reflect structural racism), and in so doing help perpetuate and extend those patterns. This may be invisible to the developers.

The developers may not be aware of these risks. They may be aware but not care. For some products, these problems are detected once it is out and in public use. For other systems, it may never be detected unless there is research to look for differential effects across racial/ethnic backgrounds.
posted by neutralmojo at 3:53 PM on July 13, 2015 [1 favorite]


it's not like there's a knob that a clever, racist programmer can turn to "make the algorithm discriminate against X people."

The issue(s) of systemic bias have very little to do with purposeful evil and a whole lot more to do with programs and systems that seem fair on their face but because of historical structures shake out as anything but fair. To steal a comparison, if you have a broken leg and I have a sunburn, and only one ambulance shows up, is it fair to flip a coin to see who gets it?
posted by shakespeherian at 3:56 PM on July 13, 2015 [2 favorites]


When people train and test an algorithm, they often forget about people different from themselves, or different from the target market, or different from the "norm" (white cis male).

This is a factor but as langtonsant says there's an even more insidious possibility, which is that the systems built on black-box math that accurately matches patterns in the material world might feed back into the material world in a way that reinforces those patterns. No bias on the part of the designers is necessary, just a desire to optimize predictions - the structural biases are already ingrained. And the problem of designing an "anti-racist" algorithm is one that I think few people have any idea how to approach.
posted by atoxyl at 4:06 PM on July 13, 2015 [1 favorite]


An AI system could give "unfair" results based on connections people aren't even aware of.
posted by atoxyl at 4:09 PM on July 13, 2015


(I guess other people have already explained this idea pretty well. Sorry, jumped the gun on adding my version.)
posted by atoxyl at 4:12 PM on July 13, 2015


Also, guns don't kill people.
posted by acb at 5:56 PM on July 13, 2015


The faith the MBA's and management classes have placed in decision algorithms and deep learning has been shown again and again to be a path to failure. Suppose you have algorithm that can identify ideal job candidates with 99% accuracy. Now suppose that in a population of a million there are 1000 people who would be idea candidates. What is the likelyhood that a resume spit out by the machine is an ideal match. You might think it is 99%, but in fact it is about 10%. Your machine will crunch the million resumes and misidentify (1%) or 10,000 as ideal. It will also miss 10 of the actual ideal candidates. So
you end up with 10,990 resumes of which 990 are ideal and 10,000 are not. Unfortunately the boss man bought the HR recruiting software on its 99% accurate claim and because few understand what this means they get no better results in hiring than before. Often they make worse decisions because they have an unshakable faith in the machine.

Now what is even more horrifying is that these systems are often not subject to any kind of peer review. Their machine created model is entirely proprietary and secret. Too often I've seen these systems turn out to be nothing more than a kind of latter day alchemy backed by nothing more than faith, scientific sounding nonsense and a few overly optimistic unit tests.
posted by humanfont at 6:19 PM on July 13, 2015 [8 favorites]


In other words, the computer doesn't really ever screw up. It just magnifies the mistakes of whoever programmed it.

If someone screwed up programming the computer (which happens like 1000‰ of the time), then the computer can screw up. And then whoever put that screw-up of a computer in charge of important stuff without sufficient oversight also screwed up. There's plenty of blame to go around.

It's probably best to think of software as the CEO's idiot cousin. You might be able to teach it a little, but someone's gonna have to keep an eye on it all the time if it's supposed to make anything important decisions. But good luck getting rid of it.
posted by aubilenon at 6:27 PM on July 13, 2015 [4 favorites]


Technical paper by same co-author. It discusses the problem of predicting whether someone's salary is less than or greater than $50k from demographic characteristics, taking sample data from the USG Census. Women on average earning less than men, the "vanilla" algorithm learns to more often predict that a woman makes less than $50k (apparently without explicit use of the gender variable).

They discuss four methods of altering the algorithm so that women and men are equally likely to be predicted to earn >$50k. The first is to randomly flip a percentage of 'Female/<$50k' results to 'Female/>$50k'.

The second (which gave the best results) is to shift the decision threshold. The original algorithm produces a number between -1 and 1 with scores greater than 0 classified as a >$50k prediction. The modification is to test women against ~-0.3 while still testing men against 0.

The third is to "massage" the training data by randomly changing some women's incomes to >$50k, and the fourth is to train the algorithm to minimize the male-female difference as well as the usual minimization of incorrect predictions.
posted by save alive nothing that breatheth at 8:47 PM on July 13, 2015 [1 favorite]


In that situation I'm not sure why they just don't remove gender from the training data?
posted by GuyZero at 9:32 PM on July 13, 2015


"We note that simply removing the protected feature from the data does not reduce bias at all in this case since the classifier output by vanilla AdaBoost trained for 20 rounds on the full data doesn't explicitly use gender."
posted by save alive nothing that breatheth at 9:47 PM on July 13, 2015


Huh. I scanned the paper super fast but clearly not well enough.
posted by GuyZero at 10:14 PM on July 13, 2015


GuyZero: "In that situation I'm not sure why they just don't remove gender from the training data?"

Yeah, as save alive nothing that breatheth says, simply removing "gender" or "race" as explicit features doesn't work, because they're correlated with too many other things. All that would happen is that the algorithm would then shift to a bunch of proxy features. Don't want it to use "race" as a predictor? Sure, you can nominally do that, but don't act surprised when the algorithm starts redlining people on the basis of zip codes or something else that has racial correlates. Race, gender, sexual orientation etc, these are all things that are too deeply intertwined with everything we do. You can't remove them from the data. If you did, the classifier might superficially appear not to be racist, but it would still make racist decisions. Removing the "race" features from the data is the machine learning version of a human claiming that they "don't see race", and it's just as much of a lie as when humans do it.
posted by langtonsant at 11:45 PM on July 13, 2015 [3 favorites]


After an afternoon of reflection, I want to revisit my initial comment in this thread. My claim that machine learning algorithms capture and re-present the implicit racism in society is true as far as it goes, but it does not go far enough. There was a time, not too long past, when machine learning researchers could perhaps be forgiven for not thinking about these issues. A naive programmer could easily create an inadvertently racist decision procedure simply by taking an off-the-shelf learning method (AdaBoost, SVM, whatever) and applying it to a data set, and never even stop to consider the possibility that the machine would have learned to be racist. In the not too distant past I could call that a forgivable error: open any standard machine learning textbook or take any intro AI class and you probably won't find much of a discussion of the possibility that the expert system learns racist beliefs from racist data. An innocent error from the programmer, an unintentionally racist act.

But that argument won't fly any more, especially not when applied to companies like Google and Facebook. They employ some very talented machine learning researchers. They are - or should be - entirely aware that this happens. And if you know anything about the topic it does not take a lot of thought to come up with the remedies summarised in this comment (even I managed to independently think up methods 2, 3 and 4). To know that your classifier reproduces racist behavior and to do nothing about it is an act of racism.

As was commented on in the thread about trans folk in the military "the standard you walk past is the standard you accept". Companies that work with big data sets should neither walk past this nor accept it. If the bread-and-butter tools of machine learning are too dumb to remedy their own racism (and they are) it is the obligation of companies that profit from those tools to remedy it for them. So I stand corrected. SansPoint was right, in the very first comment in the thread:

"An algorithm inherits all the conscious and subconscious biases it's human creator(s) have. End of story."
posted by langtonsant at 1:53 AM on July 14, 2015 [5 favorites]


They discuss four methods of altering the algorithm so that women and men are equally likely to be predicted to earn >$50k.
Why would you want to do that?

I mean, I can understand why you wouldn't want to redline loans or something, but why would you want to reduce your chance of being correct about a matter of fact?

That sounds like the same kind of approach as trying to fix institutional racism by being "colorblind", and we definitely hear a lot about how badly that works. Next you're going to be saying there's no problem because, hey, your algorithm doesn't predict any wage disparity.

If people are acting unfairly on a set of facts, it seems to me you have to address the unfairness, not massage the facts. If a machine learning system learns (statistical versions of) those facts, that's not something you want to "fix". Instead, you may want to apply an explicit correction to the output, rather than trying to change the learning system itself. Even the fact that the output has a bias you don't want to put into action is information you may be able to use to fix something.
posted by Hizonner at 6:30 AM on July 14, 2015 [3 favorites]


I mean, I can understand why you wouldn't want to redline loans or something, but why would you want to reduce your chance of being correct about a matter of fact?

The worry is that action based on predictions optimized for unfair conditions might end up amplifying unfair conditions - for example failing to present women with high-level job offers - making the unadjusted case analogous to "colorblind" policies. I agree that simply "correcting" algorithms for certain biases isn't always that simple though since it falls back on a human estimation of what a "fair" result would be and whether it's situationally appropriate to apply a correction at all. But then nothing is simple.
posted by atoxyl at 11:01 AM on July 14, 2015 [2 favorites]


Hizonner: "If people are acting unfairly on a set of facts, it seems to me you have to address the unfairness, not massage the facts. If a machine learning system learns (statistical versions of) those facts, that's not something you want to "fix". Instead, you may want to apply an explicit correction to the output, rather than trying to change the learning system itself."

Well, if you look at the four listed solutions that's basically what method number 2 does, isn't it? Assume that I'm constructing a system that has to select users to be targeted with an ad for a high profile job. Each user is given a score, and the system will direct the ad at higher scoring people. However, because the training data faithfully reproduces the same structural sexism in society that causes women to be less successful in getting high paying jobs than men, female users get lower scores than male ones. So method number 2 adjusts the threshold used to select women: men might get targeted if they score above 0.9 (or whatever) and women if they score above 0.4. The different thresholds are chosen so as to ensure that 50% of people who see the ad are women. I think that's basically what you have in mind, right?

In method #2, no attempt is made to remove sexism from the data set itself, nor from the representation learned by the expert system: all of the "massaging" takes place at the final step. But honestly I don't think this is an ideological thing. Method #3, for instance, attempts to do the massaging at the level of the data set: the programmer tells the machine sweet lies about a nicer world, one where women earn the same as men and we don't have a problem with institutionalised sexism. The system should not learn any sexism at all, because it has never seen any sexism in the data.

For myself, I don't see any fundamental reason to care about whether you use #2 or #3. As it turns out, method #2 seemed to work better in this instance, so it should be preferred on a pragmatic basis. But the idea that there's some principled reason not to "massage" the facts strikes me as a little weak: statisticians and machine learning researchers do it all the time. They might call it data transformation or preprocessing, but it's the same idea: make the adjustments at the level of the data rather than incorporating it into the model. It's certainly less elegant than other methods, but it's done all the time. It think it would be a little weird to suddenly object to this very standard practice only when the reason for the preprocessing is to remove sexist or racist decision making.
posted by langtonsant at 2:24 PM on July 14, 2015


I think there needs to be a distinction between the output of a model and decisions made based on the output of a model. I don't believe a properly constructed model that predicts someones salary based on demographic data is inherently racists or sexist if it predicts women or POC earn less than white males. It just reflects a very unfortunate reality. What is troublesome is when the results of a model are improperly used which may or may not be the fault of the analyst who built the model. I'm an analyst myself (though I don't do much predictive/prescriptive analytics and don't deal at all with any sort of demographic data) and I spend a ton of time trying to understand exactly what executives are asking for which can be very difficult since most of the time they don't necessarily know what they want. I also spend a lot of time explaining all the nuances of the data and what it both means and doesn't mean. I can imagine there are corporate environments where analysts aren't really given access to the executives like I have and where they just churn out model after model without really questioning for what or how the models are being used. In that case, how much blame can be put on the analyst if a model is improperly used in a racist/sexist way? With all the hype around predictive and prescriptive analytics there are probably a lot of companies that are jumping on the analytics bandwagon without setting up the right environment and culture necessary to provide good models and good decisions based on those models.
posted by noneuclidean at 3:07 PM on July 14, 2015 [4 favorites]


With all the hype around predictive and prescriptive analytics there are probably a lot of companies that are jumping on the analytics bandwagon without setting up the right environment and culture necessary to provide good models and good decisions based on those models.

I'm not sure anyone here is blaming the people who do the analytics in particular - we're just talking about the overall implications. I think it's true that most of the concerns here are existing concerns with amoral capitalism or even well-intentioned social engineering. What's new and noteworthy is the possibility of automated decision-making producing patterns that are functionally equivalent to regular old human bias while being removed from human agency, hidden from human eyes, or even beyond human understanding.
posted by atoxyl at 4:30 PM on July 14, 2015 [1 favorite]


So of those those four methods to correct for bias, why isn't one method to weight the sampling of the data?

What I mean, using their example: "30.38% of men and 10.93% of women reported earnings of more than $50K" Say you sample those women who do earn more than 50K about 3 times more frequently. If a random woman in the training data is equally likely to earn over 50k as a man, then gender will have no predictive value.

And unlike, say, re-labeling low-income women as high-income, you aren't you going to obscure differences it can find between those two groups, and sacrifice the stated purpose of the algorithm. (To predict income.)

Or am I missing something?
posted by RobotHero at 12:13 PM on July 15, 2015


There could still be other factors correlated to gender that have predictive value.

Imagine training a model to predict height based on gender and weight. There's a strong correlation between height and weight, and there's a weaker correlation between height and gender, so the model will probably learn that taller people weigh more. If you ask it to predict the tallest people from a sample of genders and weights, it will pick more men than women, because on average, men have a larger body weight than women.

If you add more tall women or more short men to the training sample, it won't change the correlation between height and weight, but if you change the heights of part of the training sample, it disrupts the correlation.
posted by ectabo at 2:51 PM on July 15, 2015


I guess what I was suggesting would eliminate things that only had predictive value because they correlated with gender. Like it wouldn't discover it could predict height based on how much you spend on makeup or something. But it would leave things that still have predictive value within a gender, even if those things are not equally distributed between genders.

Going with what noneuclidean said, it matters a lot what this is being used for, whether that would be sufficient.
posted by RobotHero at 4:00 PM on July 15, 2015


« Older pxtobrx   |   TreeMail Newer »


This thread has been archived and is closed to new comments