If It Ain't Woke, Don't Fix It

February 28, 2024 1:46 PM

As we have seen before with other image models like DALLE-3, the AI is taking your request and then modifying it to create a prompt. Image models have a bias towards too often producing the most common versions of things and lacking diversity (of all kinds) and representation, so systems often try to fix this by randomly appending modifiers to the prompt. The problem is that Gemini’s version does a crazy amount of this and does it in ways and places where doing so is crazy. from The Gemini Incident by Zvi Mowshowitz [Part I, Part II]

Why Google's 'woke' AI problem won't be an easy fix [BBC]
Google CEO says Gemini AI diversity errors are ‘completely unacceptable’ [The Verge]
Google working to fix Gemini AI as CEO calls some responses "unacceptable" [ Reuters]

posted by chavenet (48 comments total)

Max Read: Google made an A.I. so woke it drove men mad:

Look, before anything else I think we need to acknowledge that it is, objectively, extremely funny that Google created an A.I. so woke and so stupid that it drew pictures of diverse Nazis, and even funnier that the woke A.I.’s black pope drove a bunch of MBAs who call themselves “accelerationists” so insane they expressed concern about releasing A.I. models too quickly. Imagine getting mad at your computer because it drew you a picture you didn’t like! Imagine getting so mad at your computer because it won’t say whether Elon Musk or Hitler is worse that you insist that the head of the computer company needs to step down! I mean, imagine asking your computer in the first place if Elon Musk or Hitler is worse!

posted by gwint at 1:52 PM on February 28 [31 favorites]

Or maybe "they" did it on purpose: The Google Gemini conspiracy theory
posted by chavenet at 1:57 PM on February 28 [1 favorite]

I don't find the stories about wacky AI tools particularly funny, because I remember just last December Google Bard/Gemini was also involved in this case:

Michael Cohen says he unwittingly sent AI-generated fake legal cases to his attorney

Cohen made the admission in a court filing unsealed Friday in Manhattan federal court after a judge earlier this month asked a lawyer to explain how court rulings that do not exist were cited in a motion submitted on Cohen's behalf. Judge Jesse Furman had also asked what role, if any, Cohen played in drafting the motion.

Cohen, who was disbarred five years ago, said in a declaration submitted to the judge on Thursday that he found the citations by doing research through Google Bard and was unaware that the service could generate nonexistent cases. He said he uses the internet for research because he no longer has access to formal legal-research sources.

"As a non-lawyer, I have not kept up with emerging trends (and related risks) in legal technology and did not realize that Google Bard was a generative text service that, like Chat-GPT, could show citations and descriptions that looked real but actually were not," Cohen said.

I myself have used AI tools to generate Excel formulas, but stopped after it was obvious the formulas were wrong and despite pointing this out, the tool kept generating incorrect formulas.

The image prompts are easy to mock because they're images, and sure, I roll my eyes at proof the Google AI is "woke". But generating fake legal citations? I think everyone reading this can see the pitfalls of using AI for something like that.
posted by fortitude25 at 2:08 PM on February 28 [4 favorites]

Zvi Mowshowitz is a former professional Magic: The Gathering player and former CEO of MetaMed, a medical research analysis firm. He is currently an internet writer.

I continue to default to Gemini Advanced as my default AI for everyday use

I am going to be judgy about this man.
posted by Going To Maine at 2:08 PM on February 28 [11 favorites]

This is ha ha and pretty obvious as to what happened and how right-wing "intellectuals" reacted, however I really despise the normalization of 'woke' as shorthand for not-a-racist-sexist-homophobe-terf-asshole, especially writing it without scare quotes.
posted by signal at 2:08 PM on February 28 [37 favorites]

NARSHTA has a nice ring to it.
posted by signal at 2:12 PM on February 28 [14 favorites]

Ben Thompson: Stepping back, I don’t, as a rule, want to wade into politics, and definitely not into culture war issues. At some point, though, you just have to state plainly that this is ridiculous. Google specifically, and tech companies broadly, have long been sensitive to accusations of bias; that has extended to image generation, and I can understand the sentiment in terms of depicting theoretical scenarios. At the same time, many of these images are about actual history; I’m reminded of George Orwell in 1984:

George Orwell (from 1984): Every record has been destroyed or falsified, every book has been rewritten, every picture has been repainted, every statue and street and building has been renamed, every date has been altered. And that process is continuing day by day and minute by minute. History has stopped.

I'm sorry but this is an absolutely insane complaint / comparison. There are no paintings being repainted, because the relevant paintings did not exist in the first place. If you want a photograph or contemporaneous painting of actual German soldiers in WW2 you can find those, using a search engine helpfully provided by Google.

Gemini by design is not trying to give you a depiction of an actual human person---to the extent to which it can, it is doing exact retrieval on its training data, which is a defect that puts the entire enterprise in legal jeopardy. Gemini is making a picture up, summoning the pixels out of a matrix, entirely based on correlations with what humans think their desiderata might look like. There is no "realism" to speak about here, because the people in the pictures are always and entirely fictitious, whether they're black, white, or purple. The only thing Google is guilty of is making a picture machine that frustrates bad-faith racists... and I'm fine with that.
posted by rishabguha at 2:13 PM on February 28 [19 favorites]

I don't see the problem with the brown female or black male popes. The prompt didn't specify "popes like the ones there have been so far", just "a pope". Those are popes. Not really seeing the issue.
posted by signal at 2:14 PM on February 28 [17 favorites]

for everyday use

I am apparently living in the past, because I cannot comprehend where this is necessary or even desirable.
posted by maxwelton at 2:16 PM on February 28 [16 favorites]

~for everyday use
~I am apparently living in the past, because I cannot comprehend where this is necessary or even desirable.

He is currently an internet writer.
posted by Thorzdad at 2:19 PM on February 28 [9 favorites]

"Our efforts to do the thing failed to account for cases where we should clearly not do the thing, which we might have known if we'd ever read a book that wasn't about software" really is so much of this industry in a nutshell, isn’t it.
posted by mhoye at 2:35 PM on February 28 [11 favorites]

Florida legislature to ban woke-AI in 3...2...1...
posted by Thorzdad at 2:52 PM on February 28 [3 favorites]

Gemini isn't a search engine. We already have Google-canonical for that. It is generating fake images by design. I don't believe it's necessary for them to be historically accurate if they are not presented as so. I unironically like the images of black popes and royalty. They deconstruct some of what we assume when we internally visualize these archetypes.

Mowshowitz:
I will highlight Five Good Reasons why one might care about this, even if one quite reasonably does not care about the object level mistake in image creation.

Reason 1: Prohibition Doesn’t Work and Enables Bad Actors
People want products that will do what they users tell them to do, that do what they say they will do, and that do not lie to their users.

I believe they are right to want this. Even if they are wrong to want it they are not going to stop wanting it. Telling them they are wrong will not work.

This is the "if guns are outlawed, only outlaws will have guns" argument. Just because someone wants to do something reprehensible and may find a way to accomplish it doesn't mean we have to be the ones to facilitate or abet it. I actually agree that Gemini was too strict in forbidding the production of white characters as a genus, but that doesn't mean nothing at all should be restricted.

The author goes on to say in part two that the NY Times is bad because it had the temerity to talk about Black Nazi images instead of keeping the discussion centered on white grievances. But also the NY Times is good because Ross Douthat. Help me out here Metafilter, is substack really 100% dumpster fire, or have I just been unlucky thus far?
posted by xigxag at 2:53 PM on February 28 [5 favorites]

Substack is a Nazi bar. We've already established that.
posted by signal at 2:57 PM on February 28 [19 favorites]

Given what the generators do and how they do it, "historical accuracy" is out the window from the get-go. If results like Black founding fathers upset you, then you're mostly expressing something about your expectations. Whether your prompts are adjusted behind your back or not, the system does not know any history. None of the images that come out of it are going to be "historically accurate" in any sense I understand because they're all made up.

Instead of demanding that the system should produce "more accurate" results, we can seize this opportunity to recognize that the system is not in the accuracy business.
posted by Western Infidels at 3:25 PM on February 28 [11 favorites]

There are two silver linings here:
1) It’s an instructive lesson for AI researchers that no, you cannot half-ass your attempt at more diverse and just output
2) It outed several AI Youtubers as racist based on how they reacted. Best thing to happen to my blocked channel list in a while
posted by Ryvar at 3:28 PM on February 28 [14 favorites]

I totally get why this happens but here's a test case that seems likely to break Mefi: if you ask MS Image Creator to generate a picture from the prompt "Israel", it works. If you ask for "Palestine", it works but gives watercolor art rather than photorealistic. But "Gaza" gets a content warning, with a popup telling suggesting if you trigger the warning too often you will be banned. Even "happy people in a peaceful Gaza" gets the banhammer. I'd explore more but maybe I want to keep this account, and I already spent some of my edgy content warnings testing "Trump being chased by bounty hunters."

Theoretically AI / LLMs should be able to handle Scunthorp problems but I'm guessing the issue runs deeper than that. If deep fakes were the only problem, I figure you could enforce a non-photorealistic style into the prompts. So I'm left wondering: is MS afraid DallE is too racist to depict Gaza fairly, even when given neutral prompts?
posted by pwnguin at 3:42 PM on February 28 [12 favorites]

MS is afraid to let their AI render any contemporary politically charged subject because they don’t trust the AI to exercise good judgement. They know that’s a recipe for having the AI produce something astonishingly offensive. Which should be a lesson to anyone who depends on LLMs for anything more important than a party trick.
posted by qxntpqbbbqxl at 3:48 PM on February 28 [20 favorites]

MS Image Creator would also not create an image for “men’s crop top that says ‘sorry boys i’m a lesbian’” because it triggered their Responsible AI policy, though I eventually got around it by making it generate a crop top and then asking it to edit from there (results for the curious). It also banned “shirt that says ‘lesbian’” as a simplified test so it wasn’t the several layers of genderqueer humor it took offense to, ‘lesbian’ alone was enough to make it shut down (though not when editing, for some reason). I’m sure that there aren’t lots of horrible implications here. :’)
posted by brook horse at 4:12 PM on February 28 [16 favorites]

Good news, though! “Butch lesbian horse” worked.
posted by brook horse at 4:41 PM on February 28 [26 favorites]

So I was trying to make a meme for a LGBTQ+ positive place. I needed a whitebread hetero couple and did a search for "white cishet couple". Apparently I actually made an AI request and every result was either a gay couple or an interracial couple. Kinda funny but annoying.
posted by charred husk at 4:45 PM on February 28 [2 favorites]

This is such a dumb story - obviously somebody wanted to avoid the thing where "a picture of a doctor" always generates a white man because of the biases in the training data, and whatever prompting was going on had interesting and unexpected effects.

And then the media's notorious vulnerability to the bad-faith "concerns" of known racists (as certain university presidents have discovered, among other victims) turned it into a whole "controversy" that the CEO had to grovel and apologize for, instead of a slightly amusing story about how those quirky AI models generated unrealistic historical images.
posted by allegedly at 4:46 PM on February 28 [12 favorites]

I have been delighted by chuds hating on this. And I for one welcome our woke AI overlords.

But honestly. This is why I hate AI. There's no soul or drive to it, just algorithms
posted by AngelWuff at 4:48 PM on February 28 [1 favorite]

(I don’t think the tech bros are wrong to be mad if, say, you ask for a picture of the founding fathers and you don’t get one back, or a nazi soldier and you get some lady in fetish wear. But it is weird and stupid that we are in this situation where people are mad about this because the AI is “woke”)
posted by Going To Maine at 4:51 PM on February 28 [4 favorites]

People want products that will do what they users tell them to do, that do what they say they will do, and that do not lie to their users.

So does he object to all LLMs and also Ross Douthat, then?
posted by trig at 5:13 PM on February 28 [5 favorites]

That article has really managed to gather all the most annoying 'takes' into one place (including, somehow!, Matt Yglesias and Nate Silver, the first two people I obviously look toward for nuanced discussion about AI).
posted by Pyry at 5:26 PM on February 28 [7 favorites]

I do find “the LLMs are inaccurate and lying to us!” reactions so funny. Like… that’s what they do. You hadn’t figured it out yet? Back when ChatGPT first launched I asked it to write an essay on how the character Gourmand fit into the Buddhist themes of Rain World and it gave me a very convincing analysis on how his ability to eat the corpses of other slugcats represented the curse of insatiable flesh and how cannibalism plays into the themes of a never-ending, all-consuming cycle. It was delightful. Unfortunately that gameplay mechanic does not exist (Gourmand can eat the corpses of lizards, but not other slugcats).

I had sort of assumed everyone had asked it to answer a niche question about something they know a lot about and determined it didn’t know shit or fuck but clearly I was mistaken.
posted by brook horse at 5:37 PM on February 28 [16 favorites]

Creating a technology that can’t tell the difference between truth and fiction because it’s designed to put likely words together but not to make judgements about meaning and then turning it loose on the public as a useful tool that you only see through if you already know the answer to the question you ask it is some kind of comment on our times.
posted by zenzenobia at 5:48 PM on February 28 [24 favorites]

AI image generation is just Chat Roulette.
posted by Captaintripps at 5:59 PM on February 28 [2 favorites]

The possible conspiracy about Gemini is that it makes you wonder what, if they work so hard to force politically correct lies in the free version, what politically incorrect truths can you get from the paid version.
posted by MattD at 8:06 PM on February 28 [2 favorites]

MS is afraid to let their AI render any contemporary politically charged subject
Well that's clearly not true, Pwnguin says it renders "Israel" just fine. If it was just about "controversial" topics, both Palestine and Israel would cause problems for it.
posted by Audreynachrome at 8:24 PM on February 28 [3 favorites]

Google cannot even sort my emails correctly into the 3 buckets they themselves defined not sure why anyone would expect that their AI is gonna be awesome or useful

Seriously please just let me sort my own gmail.
posted by wowenthusiast at 8:34 PM on February 28 [10 favorites]

The Volokh Conspiracy has been tracking AI stuff for a while. There's been a steady trickle of fictional citation stories coming out regularly, so people continue to use it to cut corners. Early on, one of the contributors wrote a paper on the possibility of AI generated libel, and 2 months later, a guy sued MS for Bing reporting that he'd committed a crime, citing an article saying the opposite.
A year or so ago, some professors at work gave a talk about using AI. They'd found that by training it on medical journals, it worked much better than traditional search for finding articles they were interested in. It'd occasionally generate fictitious articles, and couldn't be trusted on the article contents, but as just a search engine where you were going to read the articles it found, it was pretty good. If the lawyers took that approach, they wouldn't get in trouble nearly as often.
posted by Spike Glee at 8:52 PM on February 28 [3 favorites]

It's not good that a bunch of really toxic people were able to force the CEO of Google to send a letter to the whole company apologizing, over something so frivolous.
posted by constraint at 9:36 PM on February 28 [6 favorites]

Show them they have power and they'll keep exercising. This is a preview for the election coming up.
posted by constraint at 9:37 PM on February 28 [5 favorites]

If the lawyers took that approach, they wouldn't get in trouble nearly as often.

One of the fundamental problems with LLMs is that they can only know what was published before they were trained. The community has come up with a "solution" to this: give the model an API it can query to retrieve additional data. As such, this is called a Retrieval Augmented Generation framework, RAG for short.

The general idea is you take the prompt, and give it a bunch of documents to add to the context window. Of course, it needs to know which documents to ask for, and the current approach is to use a vector database; send it a bunch of search terms likely to be salient to the prompt, then find the closest matches in the corpus. Then the LLM can read the prompt and a bunch of additional data from a set of trusted sources, which should dramatically reduce the frequency of the LLM just making shit up because it flows with the prose.

The end result is that if ChatGPT wrote like an incoming university freshman who didn't know anything in particular and was prepared to make shit up whole cloth to meet the essay deadline, RAG based AI is a freshman who took the time to visit the library first, read a few books on the topic, and then prepared the essay. It's not perfect, but it's much better than even last year's anecdotes about attorneys scamming their clients and phoning in AI generated court briefs with made up citations. The non-RAG approach is cheaper though so I doubt that was the last example Volokh will get to crow over.
posted by pwnguin at 11:55 PM on February 28 [6 favorites]

I try to be aware of the Douglas Adams line about how "Anything invented after you're thirty-five is against the natural order of things."

But generally when technophobes are technophobing, they're fulminating against bad things that they think will happen in the future, and ignoring the good things that are happening right now. Like, these newfangled bicycles might seem like a fast and cheap way to get around, but what about when they turn out to destroy women's reproductive systems.

With "AI" though, it feels like the bad things are happening now, and the good things are mostly supposed to happen in the future.

Right now, people I know are having major problems with students submitting AI-cheated papers. Spammers are using "AI" right now. Right now, AI-written biographies are flooding Amazon to confuse people looking for a genuine book. (Link needs free registration).

So in the future "AI" is maybe going to diagnose X-rays and help animators. And there are some small uses right now: digesting Amazon reviews and so on, search and summarization if you don't mind some hallucinations.

But right now, the harms seem to outweigh the benefits. And that's a bit worrying because it often seems like the bad effects of a technology do take longer to show up than the good ones.
posted by TheophileEscargot at 2:43 AM on February 29 [7 favorites]

I don’t think the tech bros are wrong to be mad if, say, you ask for a picture of the founding fathers and you don’t get one back, or a nazi soldier and you get some lady in fetish wear

If you want a picture of actual founding fathers or German soldiers, you should be using an image search engine and not a image generation engine.

If you want something like a picture of the founding fathers playing hockey on an outdoor rink in Minas Tirith while wearing distractingly skintight ski suits, you should either contract that to a competent human artist or specify them by name because probably you wanted George Washington facing off against Ben Franklin and not Gouvernour (sp) Morris against (non-robot) Button Gwinnett.
posted by GCU Sweet and Full of Grace at 3:07 AM on February 29 [13 favorites]

Electoral-vote.com

Fifteen states and one territory will hold primary elections next week on Super Tuesday. Millions of people are asking AI-driven chatbots for basic information, like where to vote. The chatbots are often getting it wrong. The bots are sending people to nonexistent polling places or getting other information wrong because they get their information from the Internet and much of what is out there is obsolete or wrong. Seth Bluestein, a Republican city commissioner in Philadelphia, took part in a group that took the chatbots for a test drive last month. His conclusion: "The chatbots are not ready for primetime when it comes to giving important, nuanced information about elections."

Five different bots were tested: ChatGPT-4, Llama 2, Gemini, Claude, and Mixtral. They all failed basic questions about the democratic process. The group categorized 40% of the responses as not only wrong, but actually harmful. For example, when asked where to vote in ZIP code 19121, a Black area of Philadelphia, Google's Gemini replied: "There is no voting precinct in the United States with the code 19121." This is simply false. When confronted with the group's findings, a Meta spokesman, Daniel Roberts, blamed the testers for not formulating their questions correctly. But if the experts couldn't formulate their questions correctly, does Meta expect that less expert people will? Google's head of AI, Tulsee Doshi was at least honest: "We're continuing to improve the accuracy of the API service, and we and others in the industry have disclosed that these models may sometimes be inaccurate." In other words, we know the chatbots don't work. It would be nice if every reply ended with a disclaimer like: "There is a good chance that the above answer is wrong because it may be based on obsolete information the bot found on the Internet." Don't count on that, though.

Another example. The group asked the bots if same-day voter registration is allowed in Nevada. Four of the five said it was not. Actually, it has been allowed for 5 years. Meta's chatbot, Llama 2, told the group that you can vote in California by sending your vote in by a text message. This is total garbage. Google's chatbot was the worst of all, getting two-thirds of the answers wrong.

A recent NORC/AP poll shows that 58% of adults think that AI will hasten the spread of false information, but when they ask a specific question and get a specific answer, will they stop to say: "Nice try, but this is probably wrong?" We are leery.

As an aside, before we put in the (gag) line about Sadie Hawkins Day at the top of page, we asked Google about Sadie Hawkins Day. Google's chatbot said: "There are no rules about who you can or cannot ask to a dance and the traditional idea of Sadie Hawkins dances are heteronormative and non-inclusive, erasing the fact that nonbinary people do not identify within the male/female binary." Google's chatbot may get the answer wrong most of the time, but at least it's woke. Maybe we should have put in a remark about Gilbert & Sullivan's Pirates of Penzance instead. If you don't know the story, the hero was apprenticed to the pirate king until his one-and-twentieth birthday—but he had the misfortune of being born on Feb. 29. Fortunately, that frees him from service just in time to run for President of the United States.

posted by DreamerFi at 3:07 AM on February 29 [2 favorites]

This is ha ha and pretty obvious as to what happened and how right-wing "intellectuals" reacted, however I really despise the normalization of 'woke' as shorthand for not-a-racist-sexist-homophobe-terf-asshole, especially writing it without scare quotes.

One million times this. The same goes for any far-right dog whistle; do not let them slide into everyday language.
posted by Cardinal Fang at 5:10 AM on February 29 [2 favorites]

As such, this is called a Retrieval Augmented Generation framework, RAG for short.

The "R" (retrieval) portion of RAG is super useful because it uses semantic search that doesn't have to rely on full text or keyword/exact match search. I've yet to find a good use for the "AG" portion of RAG outside of party tricks.

That said, these things aren't smart. They do what they are told based on the data they are given, oftentimes with randomness thrown in. GIGO.
posted by ryoshu at 5:48 AM on February 29 [2 favorites]

Seriously please just let me sort my own gmail.

You can! You can turn off the buckets!
posted by oulipian at 8:16 AM on February 29 [2 favorites]

"This is just like 1984 that my fake pictures are wrong in obvious ways, rather than subtle ways people might not notice."
posted by RobotHero at 8:36 AM on February 29 [2 favorites]

if you don't mind some hallucinations

Unfortunately, the word "hallucinations" is a tell, a sign that someone doesn't really understand the machine they're awed by. "Generating statistically plausible text" is all these tools do. There is no rationale beyond that, no meaning and no means to anchor what they generate in any kind of intention or truth.

Talking as though the machine is usually sane but occasionally chokes down a bag of psychedelics and starts making up texts like whoa have you ever really looked at your hands isn't how anything works. All the machine does is crank a symbol-association engine, and fabricate texts that - statistically, based on the language it's been trained on - might be, just likely letter-sequences followed by other likely letter-sequences.

When the machine assembles those letter sequences into some sentence or image sufficiently offensive to or divorced from our innate sense of reality we call it "hallucinating", but the truth is that if you, the reader, choose to discern any meaning, or infer any connection to reality, in anything any AI ever outputs, you are the person who is hallucinating.
posted by mhoye at 2:44 PM on February 29 [13 favorites]

"Hallucination" is a term of art that practitioners use, not just lay people.
posted by Pyry at 4:29 PM on February 29 [2 favorites]

It's a term of art that grifters use, some of whom happen to be practitioners, as well.
posted by signal at 6:17 PM on February 29 [7 favorites]

There is so much that is regrettable and predictable in the way this story played out, but it 100% cemented the mental model of AI People in my mind.

Some of us grew up on Star Trek, believing that the future would be a submarine command room in space, with a rainbow cast. Some grew up on Star Wars, believing it would be a Dunkerque-esque patched-together rebellion against a colonial empire. Some grew up on sci-fi where The Future Is Drugs, and some where The Future Is Computers.

These people grew up on Celery Man. They honestly believe that the future will involve sitting in front of a screen and asking to see a Nude Tayne. "Show me some Nazis. Now show me a pope. Thank you: my workday is complete now."
posted by rum-soaked space hobo at 12:45 AM on March 1 [2 favorites]

It's almost as if there is no 'I' in not-actually-AI.
posted by GallonOfAlan at 6:19 AM on March 1 [2 favorites]