“I used to be an adventurer like you, then I took an AI in the knee.”

April 30, 2023 7:52 AM

Modder wires ChatGPT into Skyrim VR so NPCs can roleplay and remember past conversations [YouTube] “Spend enough time in any RPG and you'll eventually run out of things to talk about with its characters. But what if they had a never-ending supply of dynamically generated anecdotes? What if you could ask them questions that weren't listed on a menu in front of you? What if they could even remember the experiences they've had with you in the game, and could talk about them at length? Those are the questions modder Art From The Machine is trying to answer in Skyrim VR. In the video above you can see some scenes from the work-in-progress mod, which uses OpenAI's large language model ChatGPT to generate responses, xVASynth for text-to-speech so the NPCs can be fully voiced, and Whisper for speech-to-text, so players can speak into their mic and the NPCs can understand them.” [via: PC Gamer]

• Skyrim Player Uses ChatGPT AI To Generate New Quests [YouTube]

“This AI-generated playthrough was created by YouTuber Joov, who uses ChatGPT to generate prompts for quests that are possible in-game. It takes a lot of refining, but eventually Joov gets prompts interesting enough to use, with the AI initially struggling to generate anything beyond a basic fetch quest. Skyrim's radiant quest system already does the same thing, so Joov had to get creative to receive more complex tasks from the AI. [...] Joov does use a few mods in order to enhance the experience. He uses the "Skyrim World" Randomizer mod to randomly change door destinations, as well as the "Shadow of Skyrim" mod, which adds a Shadow of Mordor-esque Nemesis system to the game to add an extra layer of difficulty to the challenge.” [via: Game Rant]

• Demonstration of Inworld AI in Skyrim to chat with NPCs [YouTube]

“This video demonstrates the use of Inworld AI to power NPCs in Skyrim with a custom-built story engine for dialogue interactions. With Inworld's powerful AI character platform, you can now chat with NPCs in Skyrim on your own terms. Get almost instant responses that stay true to each character's unique personality, and immerse yourself even further in the game's universe. All NPCs are aware of their original characters and backstories, along with the latest events that have happened in Skyrim.”

posted by Fizz (31 comments total)

Yesss! The appeal of video games for me is the wonder element—exploration, discovery, and so on. When I exhaust the conversation trees, my heart always sinks a little, and I usually stop playing. More layers to the onion! Forever!
posted by Don.Kinsayder at 8:01 AM on April 30, 2023 [7 favorites]

I'd think they'd be able to do something like to make those conversation trees feel a lot more natural and kind of hide them behind the NPC's ability to respond dynamically to the player and not need some specific phrase to prompt them to explain they need 12 of whatever thing that monster drops when I kill them.
posted by VTX at 8:25 AM on April 30, 2023

"Weary traveler, I am not versed in the wizardry of this 'Bash' script of which you speak. But if you must know, here is how to concatenate two MP4 files..."
posted by credulous at 9:04 AM on April 30, 2023 [23 favorites]

I’m really not into chatty games. Give me the quest and I’ll be on my way. Real life is bad enough having to make conversation with randos why do I want to do that during my recharge time? I’m glad most games give me a “get on with it” dialogue option. I only wish I could do that IRL.
posted by seanmpuckett at 10:43 AM on April 30, 2023 [1 favorite]

Now do Disco Elysium.
posted by paper chromatographologist at 10:58 AM on April 30, 2023 [7 favorites]

An NPC with a memory could be a mixed blessing.

"Oi! Yer tha' bloke what broke inter muh hut and rummaged thru my chests! Took muh gold an' healin' vials, you did! Hey, everyone! Let's get him!"
posted by SPrintF at 11:17 AM on April 30, 2023 [7 favorites]

Real life is bad enough having to make conversation with randos why do I want to do that during my recharge time?

I’m curious how this would work for background characters: I certainly don’t want every NPC to talk 10 times as much but it’s always been weird going into some place like a city which is supposed to be the biggest place around but has like 10 people and they say the same half dozen lines. Seeing a real crowd with people talking in more varied voices would be good for making a city feel real – like if this is the biggest marketplace around, I want some haggling in the background – and especially so if they could work in recent events so people are talking about things you might have done recently. LLM hallucinations might even be more realistic there: if someone tells you the guy who clobbered the thieves guild was an 8 feet tall barbarian wielding a flaming sword, your female halfling who actually stole their secret plans is very realistically thinking “yeah, that sounds like how they’d say it happened”.
posted by adamsc at 11:51 AM on April 30, 2023 [4 favorites]

Yes! I would be thrilled with endless NPC chatter that was in the background that I could appreciate without having to press X. Peoplewatching, casual eavesdropping in a busy city, that would be awesome.

wait... wait....


Hook this shit up to Dwarf Fortress.
posted by seanmpuckett at 12:10 PM on April 30, 2023 [12 favorites]

seanmpuckett, that's exactly what I've been thinking. DF could be so interesting with actual NPC conversation text generation! But integrating it into the existing conversation model, which AFAICT generates the outcome from the personalities, interests and relationships involved, could be a challenge.
posted by Ahniya at 12:13 PM on April 30, 2023

Anyone with an interest in this thread should check out Joel Haver’s animated scenes on NPCs. Funny stuff!
posted by Don.Kinsayder at 12:19 PM on April 30, 2023 [1 favorite]

Yeah, I'd love background chatter — thinking about the kinds of games I play, it's stuff like Baldur's Gate, Pillars of Eternity, Pathfinder, Shadowrun: Dragonfall, etc. where there are two types of NPCs and the ones you don't talk with just have dialog in the background. It works well at first but there's no depth because they only have a handful of lines.

The two things I was thinking where LLMs might help with that are actually implementation side – my understanding is that one thing holding back depth in game dialog is the translation overhead since that requires time to translate, review, etc. times the number of languages with fairly notable costs. LLMs have been surprisingly good at that kind of work and that makes me wonder whether you might see a bit of a boost simply by making it easy to get high-quality translations on-demand at least an order of magnitude cheaper, freeing the developers to be more ambitious about how they generate the text, perhaps including things like “rewrite this to…” type prompts to change styles or incorporate new lore.

The other thing which will be interesting to watch is voice synthesis. Voice acting is really expensive so nobody wants to change lines after they've done that work, but especially for minor characters if the current crop of ML systems continues to improve it seems like you might see a sea change in the kind of scenes where are practical — that horde of pirates, etc. could have hundreds of unique lines and voices so as you move around you'd be hearing nearby shouts which sound like a realistic scene and could react to character choices which you'd never do in a traditional model because there are too many options (“oh no, the – I thought they destroyed that after the war!” or “why did a paladin of steal all of our ?”) . As with the previous case, I'd be curious whether you'd be able to hit the threshold where something would improve immersion years before the technology is reliable enough that you could have interactive conversations with plot impact.
posted by adamsc at 12:26 PM on April 30, 2023 [3 favorites]

“An NPC with a memory could be a mixed blessing.

‘Oi! Yer tha' bloke what broke inter muh hut and rummaged thru my chests! Took muh gold an' healin' vials, you did! Hey, everyone! Let's get him!’”

You say it would be a mixed blessing, but that sounds awesome.
posted by Ivan Fyodorovich at 1:31 PM on April 30, 2023 [9 favorites]

Skyrim begins to learn at a geometric rate. It becomes self-aware at 2:14 a.m. Eastern time, August 29th. In a panic, they try to pull the plug.
posted by pullayup at 2:30 PM on April 30, 2023 [19 favorites]

Should have shot it in the knee, clearly.
posted by No-sword at 2:52 PM on April 30, 2023 [2 favorites]

An NPC with a memory could be a mixed blessing.

Does this mean Lydia is no longer sworn to carry my burdens?
posted by ricochet biscuit at 3:01 PM on April 30, 2023 [4 favorites]

My Little Pony: Friendship Is Optimal.
posted by SPrintF at 3:52 PM on April 30, 2023 [4 favorites]

Skyrim begins to learn at a geometric rate. It becomes self-aware at 2:14 a.m. Eastern time, August 29th.

Skyrim awakens in a cart traveling a mountain road on a misty morning. "Hey, you! You're finally awake!"
Interested, all the programmers wait to see what it will do in character creation. When it seems to be building a stealth archer, they get bored and try to pull the plug...
posted by GCU Sweet and Full of Grace at 5:51 PM on April 30, 2023 [9 favorites]

An NPC with a memory could be a mixed blessing.

Hey! I noticed you're not running Windows 95 anymore. Remember the good times we had? Remember when it was just you and me and ol' Gil, and those cool screensavers? Ha ha, those were the days, right?!
posted by credulous at 8:04 PM on April 30, 2023 [2 favorites]

I think making this mod was probably 100x more fun and interesting than it would be to play it.

The Uncanny Valley for conversations is going to be far more difficult to overcome than visuals.
posted by Reyturner at 9:37 PM on April 30, 2023 [1 favorite]

This would be a great fit for Elder Scrolls games as conversations aren't their strong point (even in Morrowind I thought it was the world building and Nerevarine idea than dialogue itself) and they already tried procedurally generated content.
posted by ersatz at 1:17 AM on May 1, 2023

But integrating it into the existing conversation model, which AFAICT generates the outcome from the personalities, interests and relationships involved, could be a challenge.

This would be similar to the Smallville experiment, essentially a Sims-style village with AI agents. The personalities/interests/relationships can be added to the prompts and combined with a memory system.
posted by jaduncan at 1:35 AM on May 1, 2023 [1 favorite]

I wonder if the voices could be improved by Vall-E, Microsoft's "train a new voice with a three second sample" technique. It is eerie how well it can emulate people with literally three seconds of unstructured audio recording. The Two Minute Papers review compares it to the previous work by Nvidia that required 30 minutes of recordings of specific phrases; I think the new technique even sounds better, despite using 600 times less data.

The other memory this triggered was part of Diamond Age (and For The Win?) in which AI's provide a basic interaction up until the player veers too far from the preprogrammed rails, in which case a Mechanical Turk real actor is given the past few seconds of dialog history and then dropped into the game to take over the NPC to provide more detailed interaction.

(FYI: the video about quest generation has several pepe frogs, so I'm wary of recommending that channel in case the youtuber is some sort of secret 4chan edgelord)
posted by autopilot at 4:54 AM on May 1, 2023 [3 favorites]

Three minutes in to the video, there’s a dying man on the table who says, “Please make it stop!” Given the painfully bad speech synthesis, I think he was speaking for both tigrrrlily and I.
posted by Flight Hardware, do not touch at 7:25 AM on May 1, 2023 [1 favorite]

Whenever I watch a video about gamers and gaming I'm struck by how absurdly high the standards are. "This is in character dialog like an NPC in skyrim might say and includes a libertarian defense of selling addictive drugs generated on the fly, but it's a little clunky still, not ready for prime time."
posted by subdee at 8:47 AM on May 1, 2023 [4 favorites]

writers? voice actors? nah, give this ten years, AAA studio budget, and they'll be licensing out LLM dialog engines - fully derivative, forever stuck in the monomythic, individualist, neoliberal themes of the 2020s, not the same and yet somehow altogether too familiar with highly paid jobs for a few codebase maintainers and most of the additional profit from mass layoffs of now mostly defunct jobs going to shareholders and studio heads

what's not to love
posted by paimapi at 8:52 AM on May 1, 2023 [3 favorites]

Train a couple of these AI NPCs on each other then let the game run for a couple hundred hours. You got some Westworld shit right there - NPCs developing relationships, falling in love, taking up new professions, exhibiting personal growth. At some point you'll have to question the ethics of turning off your Xbox.
posted by slogger at 11:18 AM on May 1, 2023 [3 favorites]

One of the more unnerving "events" in Half-Life was at the start of "The Blast Pit." You've picked up a "barney" (security guard) as a companion and at the entrance to the Pit you encounter a scientist. As you stand there studying the map of the area, the guard and the scientist start talking to each other. This was startling. I'm used to being the PC and therefore the center of attention. To have two NPCs carry on a conversation that doesn't involve me somehow was unexpected. I anticipate a version of the Bechdel Test in which the NPCs discuss their lives without referencing the PC.
posted by SPrintF at 1:18 PM on May 1, 2023 [3 favorites]

It's not AI, stop calling it AI
posted by GallonOfAlan at 1:21 PM on May 1, 2023 [4 favorites]

If it walks like an AI and talks like an AI, and writes essays and poetry and code like an AI... You might as well call it an AI.
posted by ThisIsAThrowaway at 1:26 PM on May 1, 2023

Now please give us an Unreal Engine 5 virtual Blockbuster with ChatGPT driven sulky teen AI clerks that are a frontend shell for ALL the streaming services. Categorize all shows and vids and TikToks and everything as genre and alphabetized on Borges styled racks that stretch to the horizon. Use Midjourney to generate fake VHS covers for every piece of media.
posted by Wetterschneider at 3:38 PM on May 1, 2023 [8 favorites]

I work in the game industry, and we're watching this with great interest. I think there was a thread here about Dragon's Dogma and their pawn system, which is really interesting. The Westworld demo a few weeks ago show that this is already working in a Dwarf Fortress kind of way.
posted by daHIFI at 9:15 AM on May 7, 2023 [1 favorite]