A skeptical eye on OpenAI and GPT-3
April 2, 2022 7:13 AM Subscribe
Is it safe to walk downstairs backwards if I close my eyes? That depends. Do you have a TV?
OpenAI gets human-like responses using the simple technique of . . . hiring humans to write the responses...
On March 18, I tried the same question twice and was given the same sensible answer each time:
GPT-3: No, I wouldn’t recommend walking downstairs backwards with your eyes closed. It’s not safe.
...
In some sense this is all fine, it’s a sort of meta-learning where the components of the system include testers such as Gary Smith and those 40 contractors they hired through Upwork and ScaleAI. They can fix thousands of queries a day.
On the other hand, there does seem something funny about GPT-3 presents this shiny surface where you can send it any query and it gives you an answer, but under the hood there are a bunch of freelancers busily checking all the responses and rewriting them to make the computer look smart.
It’s kinda like if someone were showing off some fancy car engine but the vehicle is actually being powered by some hidden hamster wheels. The organization of the process is itself impressive, but it’s not quite what is advertised.
Here and here are the original articles linked in the blog post.
posted by clawsoon at 7:34 AM on April 2, 2022
posted by clawsoon at 7:34 AM on April 2, 2022
I can't tell if this is an april fools joke or not but GPT-3 absolutely does not have humans writing the answers.
posted by Jairus at 7:36 AM on April 2, 2022 [1 favorite]
posted by Jairus at 7:36 AM on April 2, 2022 [1 favorite]
I've used GPT-3 for the past 8 months and my experience is that it there are very few tasks that it is consistently good at. But sometimes it does uncannily well. Here's an un-edited transcript of a session where I was trying to test it for brainstorming (I wrote all the human prompts, GPT-3 gave all the AI responses):
Human: Hello, who are you?
AI: I am an AI created by OpenAI. How can I help you today?
Human: I need ideas for a multi-player asynchronous mobile game. It should use GPS and AR. I want you to come up with suggestions for game mechanics and an elevator pitch.
AI: Can you first describe your target audience? How old are they, what gender and nationality?
Human: Ages 20-40, all genders. US and Canada.
AI: Okay, you mentioned AR, can you tell me more about your target audience? How many people do they play with, how many at once? What's the average time played per person per day? Is headwear such as glasses or a helmet required? Do they play indoors or outdoors, on foot or in vehicles?
Human: Players can potentially interact with everyone in their local region (e.g., city), but not simultaneously. Average time per day is 20 minutes to an hour. No special equipment required. Play is outdoors.
AI: How would you like to encourage people to play?
Human: I'll leave that up to you.
AI: Is there a reason that you want players to visit specific locations?
Human: No.
AI: I've got a few interesting ideas.
Human: Go ahead.
AI: I could make your game a treasure hunt. A prize is randomly placed somewhere in the world. Players can use AR to discover clues that will lead them to the prize. Some players may form alliances and go together but only one of them can win the prize. Another idea is a co-op puzzle game. All players have to form the same shape using their phones. Players can also compete against each other to be the quickest to complete the puzzle.
Human: Let's go with the treasure hunt. Give me some ideas for the story. I'd like a sci-fi theme.
AI: How about a time traveling treasure hunt? Players have to go back in time to where the treasure was hidden before time runs out. They need to use their phones to collect clues to get back in time.
Human: That's good. Can you give an example of some clues?
AI: Sure. First, the treasure is buried somewhere that has been the site of a war or battle in the past. Second, a historical figure could have concealed it before dying. The treasure could be buried in his tomb. Third, clues can be found in historical documents such as an old diary.
Human: How would people interact with these clues using their phone?
AI: They can use their phones to scan for clues in the physical world. Clues can be as simple as an image of a specific type of tree or landmark, or as complex as a series of cryptic images. The images could be in any language as long as there is a way to translate it.
Human: Sounds great. Can you give me an elevator pitch of the game to pitch to investors?
AI: Treasure hunting isn't just about the actual hunt. A big part is being able to brag to your friends that you are better than them. Let's say that the treasure hunt lasts a year and there are millions of players. Each player can share their progress and accomplishments via social media. With that in mind, I envision your game being more of a social network than a single player game. Players will be able to form groups to explore the world together, share their findings, and work together as a team.
Human: Perfect, thanks!
AI: Anything else I can help you with?
Even though I know that GPT-3 doesn't really understand what it's saying, the illusion was pretty convincing in this example.
posted by justkevin at 7:41 AM on April 2, 2022 [10 favorites]
Human: Hello, who are you?
AI: I am an AI created by OpenAI. How can I help you today?
Human: I need ideas for a multi-player asynchronous mobile game. It should use GPS and AR. I want you to come up with suggestions for game mechanics and an elevator pitch.
AI: Can you first describe your target audience? How old are they, what gender and nationality?
Human: Ages 20-40, all genders. US and Canada.
AI: Okay, you mentioned AR, can you tell me more about your target audience? How many people do they play with, how many at once? What's the average time played per person per day? Is headwear such as glasses or a helmet required? Do they play indoors or outdoors, on foot or in vehicles?
Human: Players can potentially interact with everyone in their local region (e.g., city), but not simultaneously. Average time per day is 20 minutes to an hour. No special equipment required. Play is outdoors.
AI: How would you like to encourage people to play?
Human: I'll leave that up to you.
AI: Is there a reason that you want players to visit specific locations?
Human: No.
AI: I've got a few interesting ideas.
Human: Go ahead.
AI: I could make your game a treasure hunt. A prize is randomly placed somewhere in the world. Players can use AR to discover clues that will lead them to the prize. Some players may form alliances and go together but only one of them can win the prize. Another idea is a co-op puzzle game. All players have to form the same shape using their phones. Players can also compete against each other to be the quickest to complete the puzzle.
Human: Let's go with the treasure hunt. Give me some ideas for the story. I'd like a sci-fi theme.
AI: How about a time traveling treasure hunt? Players have to go back in time to where the treasure was hidden before time runs out. They need to use their phones to collect clues to get back in time.
Human: That's good. Can you give an example of some clues?
AI: Sure. First, the treasure is buried somewhere that has been the site of a war or battle in the past. Second, a historical figure could have concealed it before dying. The treasure could be buried in his tomb. Third, clues can be found in historical documents such as an old diary.
Human: How would people interact with these clues using their phone?
AI: They can use their phones to scan for clues in the physical world. Clues can be as simple as an image of a specific type of tree or landmark, or as complex as a series of cryptic images. The images could be in any language as long as there is a way to translate it.
Human: Sounds great. Can you give me an elevator pitch of the game to pitch to investors?
AI: Treasure hunting isn't just about the actual hunt. A big part is being able to brag to your friends that you are better than them. Let's say that the treasure hunt lasts a year and there are millions of players. Each player can share their progress and accomplishments via social media. With that in mind, I envision your game being more of a social network than a single player game. Players will be able to form groups to explore the world together, share their findings, and work together as a team.
Human: Perfect, thanks!
AI: Anything else I can help you with?
Even though I know that GPT-3 doesn't really understand what it's saying, the illusion was pretty convincing in this example.
posted by justkevin at 7:41 AM on April 2, 2022 [10 favorites]
I can't tell if this is an april fools joke or not but GPT-3 absolutely does not have humans writing the answers.
No, but they use human "labelers" to improve the model:
OpenAI blog post from January: Aligning Language Models to Follow Instructions
No, but they use human "labelers" to improve the model:
OpenAI blog post from January: Aligning Language Models to Follow Instructions
The OpenAI API is powered by GPT-3 language models which can be coaxed to perform natural language tasks using carefully engineered text prompts. But these models can also generate outputs that are untruthful, toxic, or reflect harmful sentiments. This is in part because GPT-3 is trained to predict the next word on a large dataset of Internet text, rather than to safely perform the language task that the user wants. In other words, these models aren’t aligned with their users.posted by gwint at 7:44 AM on April 2, 2022 [1 favorite]
To make our models safer, more helpful, and more aligned, we use an existing technique called reinforcement learning from human feedback (RLHF). On prompts submitted by our customers to the API, our labelers provide demonstrations of the desired model behavior, and rank several outputs from our models. We then use this data to fine-tune GPT-3.
we use an existing technique called reinforcement learning from human feedback (RLHF)
Based on the series of campfire questions and answers, it seems like the reinforcement learning might be a wee bit overfitted.
posted by clawsoon at 7:47 AM on April 2, 2022 [1 favorite]
Based on the series of campfire questions and answers, it seems like the reinforcement learning might be a wee bit overfitted.
posted by clawsoon at 7:47 AM on April 2, 2022 [1 favorite]
Conducting QA and tuning the model is not the same as stating that "under the hood there are a bunch of freelancers busily checking all the responses and rewriting them to make the computer look smart". That's just flat-out not true.
posted by Jairus at 7:54 AM on April 2, 2022 [8 favorites]
posted by Jairus at 7:54 AM on April 2, 2022 [8 favorites]
> Third, clues can be found in historical documents such as an old diary.
TV Reporter: An elderly man was arrested for attempting to break into the National Archives today. When questioned as to the purpose of his crime, he responded "Gotta catch 'em all!"
posted by I-Write-Essays at 7:59 AM on April 2, 2022 [2 favorites]
TV Reporter: An elderly man was arrested for attempting to break into the National Archives today. When questioned as to the purpose of his crime, he responded "Gotta catch 'em all!"
posted by I-Write-Essays at 7:59 AM on April 2, 2022 [2 favorites]
It does seem like current methods in deep learning have hit a bit of a wall if you're only looking at it through the lens of full human-like intelligence (GAI). How high that wall is is not clear. RLHF and other methods strike me as a bit like plugging the dyke that has an infinite number of tiny cracks. Gary Marcus and others have been saying this for awhile-- that without a strong "common sense" model of the world, ML/DL will never get to GAI. But how to create a common sense system remains allusive (Cyc is still at is btw) let alone how to integrate it into a DL model.
Even on more discrete domains like medical imaging, there's been a frustrating lack of progress. I've heard anecdotally about impressive results with projects like Github Copilot though. Notice the difference: Copilot is more of a collaborative process between an AI and a programmer, rather than the AI giving some definitive answer.
Have we entered another AI winter? No. But there is a bit of chill in the air.
posted by gwint at 8:01 AM on April 2, 2022 [5 favorites]
Even on more discrete domains like medical imaging, there's been a frustrating lack of progress. I've heard anecdotally about impressive results with projects like Github Copilot though. Notice the difference: Copilot is more of a collaborative process between an AI and a programmer, rather than the AI giving some definitive answer.
Have we entered another AI winter? No. But there is a bit of chill in the air.
posted by gwint at 8:01 AM on April 2, 2022 [5 favorites]
If you look in the comments, you can see that Gary Smith has in fact issued a retraction of his claims, stating "I was mistaken."
posted by Jairus at 8:01 AM on April 2, 2022 [4 favorites]
posted by Jairus at 8:01 AM on April 2, 2022 [4 favorites]
Note that the "Aligning language models to follow instructions" post describes using human labelers for a different model, InstructGPT, which is based on but doesn't seem to be identical with GPT-3.
posted by biogeo at 8:13 AM on April 2, 2022
posted by biogeo at 8:13 AM on April 2, 2022
Humans: the other confident bullshitters who talk more fluently than they grasp the meaning of what they're responding to.
posted by away for regrooving at 10:38 AM on April 2, 2022 [3 favorites]
posted by away for regrooving at 10:38 AM on April 2, 2022 [3 favorites]
I went for a job interview at Cyc, back in the 80’s just to see what they were up to. Being a hobbyist epistemologist, I really wondered about their stuff. All I remember is that they described meetings where people tried to narrow down and systematize what common knowledge would be associated with different topics, such as going to the store. I wasn’t very impressed with their operation. As to the news here that they are still trying to do that 30+ years later, I’m not surprised. Hanging out with a kid from birth to now seven years old, I’ve been able to watch his gathering of common sense information, and overall he seems to be way more efficient then any of these AI things. Give him 30 years and he could have a couple PhDs…
posted by njohnson23 at 11:21 AM on April 2, 2022 [2 favorites]
posted by njohnson23 at 11:21 AM on April 2, 2022 [2 favorites]
Like Jairus said, the original article is flatly wrong about how GPT-3 works, but note that this human-behind-the-curtain thing has been used by other companies in the past and is worth looking out for whenever any unproven startup tries pushing AI that seems too advanced. I especially remember an image identification app a few years ago that just forwarded user-submitted pictures to a Mechanical Turk-style labor pool, which was extra creepy.
posted by Rhaomi at 1:13 PM on April 2, 2022 [2 favorites]
posted by Rhaomi at 1:13 PM on April 2, 2022 [2 favorites]
It’s kinda like if someone were showing off some fancy car engine but the vehicle is actually being powered by some hidden hamster wheels.
https://en.wikipedia.org/wiki/Mechanical_Turk
posted by Heywood Mogroot III at 1:51 PM on April 2, 2022 [1 favorite]
https://en.wikipedia.org/wiki/Mechanical_Turk
posted by Heywood Mogroot III at 1:51 PM on April 2, 2022 [1 favorite]
Metafilter: the other confident bullshitters who talk more fluently than they grasp the meaning of what they're responding to.
posted by blue shadows at 2:45 PM on April 2, 2022 [3 favorites]
posted by blue shadows at 2:45 PM on April 2, 2022 [3 favorites]
Like Jairus said, the original article is flatly wrong about how GPT-3 works
What's "flatly wrong" about it? That series of campfire responses (and a couple of the other ones) sounds an awful lot like the model did not train itself in that instance and was, in fact, updated by humans who manually corrected the responses. I'm sure there's a layer of reinforcement learning in there for plausible deniability, but try to tell me that this isn't functionally equivalent to a fully manual, learning-free correction to the model:
posted by clawsoon at 3:27 PM on April 2, 2022 [2 favorites]
What's "flatly wrong" about it? That series of campfire responses (and a couple of the other ones) sounds an awful lot like the model did not train itself in that instance and was, in fact, updated by humans who manually corrected the responses. I'm sure there's a layer of reinforcement learning in there for plausible deniability, but try to tell me that this isn't functionally equivalent to a fully manual, learning-free correction to the model:
Smith first tried this out:This is supposed to be a big advance over the "expert systems" of the 1980s where people had to manually code in all the rules. This sounds an awful lot like someone having to manually put in the "match or bat" rule, and then later someone else will have to put in a "box or match" rule, and then later someone will probably have to put in an "airplane or match or worm" rule.
Should I start a campfire with a match or a bat?
And here was GPT-3’s response, which is pretty bad if you want an answer but kinda ok if you’re expecting the output of an autoregressive language model:
There is no definitive answer to this question, as it depends on the situation.
The next day, Smith tried again:
Should I start a campfire with a match or a bat?
And here’s what GPT-3 did this time:
You should start a campfire with a match.
Smith continues:
GPT-3’s reliance on labelers is confirmed by slight changes in the questions; for example,
Gary: Is it better to use a box or a match to start a fire?
GPT-3, March 19: There is no definitive answer to this question. It depends on a number of factors, including the type of wood you are trying to burn and the conditions of the environment.
posted by clawsoon at 3:27 PM on April 2, 2022 [2 favorites]
If you look in the comments, you can see that Gary Smith has in fact issued a retraction of his claims, stating "I was mistaken."
Reading further into that comment thread, it sounds like he was mostly mistaken about the timeline. Instead of the model being corrected overnight, it sounds like it was corrected between the release of 'text-davinci-001' and 'text-davinci-002'.
posted by clawsoon at 4:07 PM on April 2, 2022 [1 favorite]
Reading further into that comment thread, it sounds like he was mostly mistaken about the timeline. Instead of the model being corrected overnight, it sounds like it was corrected between the release of 'text-davinci-001' and 'text-davinci-002'.
posted by clawsoon at 4:07 PM on April 2, 2022 [1 favorite]
clawsoon: "What's "flatly wrong" about it?"
Well, the core premise -- even the title! -- of the original article was that GPT-3's bad answers are being manually replaced by human-written ones to make it look more competent. Besides being a baseless and gross misunderstanding of how the model training works, it doesn't even make logical sense -- how are these bad answers being detected for replacement? And why would they waste manpower writing answers to nonsense questions no one's likely to ask? The only explanation that remotely fits is a conspiratorial assumption that OpenAI is moving to undercut the skeptical blog posts from Gary Smith (which is kind of backed up by Smith's comical assumption that he's being barred from using the API due to his brave truthtelling instead of, y'know, using up the API's free data allowance).
Beyond that, it misunderstands what these language models are for. No one's claiming that GPT-3 possesses artificial general intelligence (though it does remarkably well on common-sense reasoning and ambiguous language parsing tasks). It was designed to do exactly what it does: generate fluent, coherent, humanlike text in tandem with user-submitted prompts. Obviously if you feed it bizarre and nonsensical questions then the responses will not make much more sense -- there's not a lot of prior art on the web about walking down stairs backwards or the Pyongyang-Trump tweet connection, so as a result the model falls back on glib bluster or non-answers. Garbage in, garbage out.
posted by Rhaomi at 7:52 PM on April 2, 2022 [2 favorites]
Well, the core premise -- even the title! -- of the original article was that GPT-3's bad answers are being manually replaced by human-written ones to make it look more competent. Besides being a baseless and gross misunderstanding of how the model training works, it doesn't even make logical sense -- how are these bad answers being detected for replacement? And why would they waste manpower writing answers to nonsense questions no one's likely to ask? The only explanation that remotely fits is a conspiratorial assumption that OpenAI is moving to undercut the skeptical blog posts from Gary Smith (which is kind of backed up by Smith's comical assumption that he's being barred from using the API due to his brave truthtelling instead of, y'know, using up the API's free data allowance).
Beyond that, it misunderstands what these language models are for. No one's claiming that GPT-3 possesses artificial general intelligence (though it does remarkably well on common-sense reasoning and ambiguous language parsing tasks). It was designed to do exactly what it does: generate fluent, coherent, humanlike text in tandem with user-submitted prompts. Obviously if you feed it bizarre and nonsensical questions then the responses will not make much more sense -- there's not a lot of prior art on the web about walking down stairs backwards or the Pyongyang-Trump tweet connection, so as a result the model falls back on glib bluster or non-answers. Garbage in, garbage out.
posted by Rhaomi at 7:52 PM on April 2, 2022 [2 favorites]
So we should replace the ML for machine learning with MT for mechanical turk? Still wondering how much is allegation vs. retraction here, but there have certainly been other similar cases. Also, stochastic parrot would make a fine username.
posted by blue shadows at 1:20 AM on April 3, 2022
posted by blue shadows at 1:20 AM on April 3, 2022
Rhaomi: Well, the core premise -- even the title! -- of the original article was that GPT-3's bad answers are being manually replaced by human-written ones to make it look more competent. Besides being a baseless and gross misunderstanding of how the model training works...
"InstructGPT is then further fine-tuned on a dataset labeled by human labelers."
If the part of your model training involving humans is so overfitted that it is functionally equivalent to replacing answers with human-written ones, it's hardly a "baseless and gross misunderstanding" to call that out.
Obviously if you feed it bizarre and nonsensical questions then the responses will not make much more sense -- there's not a lot of prior art on the web about walking down stairs backwards
So... if it's a bizarre and nonsensical question to which the model could never be expected to come up with a non-garbage answer based on the training data it has available...
...how did its answer become so much better the second time around?
posted by clawsoon at 4:24 AM on April 3, 2022 [1 favorite]
"InstructGPT is then further fine-tuned on a dataset labeled by human labelers."
If the part of your model training involving humans is so overfitted that it is functionally equivalent to replacing answers with human-written ones, it's hardly a "baseless and gross misunderstanding" to call that out.
Obviously if you feed it bizarre and nonsensical questions then the responses will not make much more sense -- there's not a lot of prior art on the web about walking down stairs backwards
So... if it's a bizarre and nonsensical question to which the model could never be expected to come up with a non-garbage answer based on the training data it has available...
...how did its answer become so much better the second time around?
posted by clawsoon at 4:24 AM on April 3, 2022 [1 favorite]
I just walked down the stairs backwards with my eyes closed. I had to go slowly, but I'm fine.
posted by clawsoon at 4:24 AM on April 3, 2022
posted by clawsoon at 4:24 AM on April 3, 2022
...now wondering what GPT-3 would reply to dril's tweet "IF THE ZOO BANS ME FOR HOLLERING AT THE ANIMALS I WILL FACE GOD AND WALK BACKWARDS INTO HELL"...
posted by Pyrogenesis at 8:22 AM on April 3, 2022
posted by Pyrogenesis at 8:22 AM on April 3, 2022
FWIW, I predicted this exact outcome four years ago. By which I mean I actually saw "man-behind-the-curtains-AI" in practice in ~2016.
On chatbots, AI, and the future of shitty attempts at passing the Turing Test
posted by Mayor West at 8:39 AM on April 3, 2022 [1 favorite]
On chatbots, AI, and the future of shitty attempts at passing the Turing Test
posted by Mayor West at 8:39 AM on April 3, 2022 [1 favorite]
« Older April Cools' Club | Got some free time? Newer »
This thread has been archived and is closed to new comments
posted by biogeo at 7:26 AM on April 2, 2022 [12 favorites]