“We have to be twice as careful when talking about difficult topics,”
January 28, 2018 5:39 AM   Subscribe

Scientific Victory [Games by Angelina] “Last week a few games sites covered the fact that the Cambridge Center for the Study of Existential Risk (CSER), a lab which investigates safety issues associated with things like artificial intelligence, had released a Civilisation V mod about the risk of superintelligent AI. Here’s what Rock, Paper, Shotgun quoted designer and CSER researcher Shahar Avin as saying about the project:
“We want to let players experience the complex tensions and difficult decisions that the path to superintelligent AI would generate,” said the Centre for the Study of Existential Risk’s Dr. Shahar Avin, who managed the project. “Games are an excellent way to deliver a complex message to a wide audience.”
This is a blog post about why games are not always an excellent way to deliver a complex message to a wide audience.”
“If the mod was just something someone had cooked up in their spare time it might not be a problem, but with the CSER name attached – as well as Cambridge, one of the world’s most famous universities – the mod is now a publicity tool, carrying with it the weight of academic endorsement. And this is awkward, because with that extra reputation attached the game’s messages might now be interpreted a lot more strongly by those playing it. For example, the mod’s failure condition of a Rogue AI taking over the world will always happen unless players avert it – it is not something that has a chance of happening. The mod’s message is the AI is fundamentally unsafe, and doing any kind of experimentation with it will lead to the destruction of civilisation. To fight this, the mod advocates for technology becoming the “slave” of mankind, through the construction of safety labs (modelled on, I assume, CSER itself).”
posted by Fizz (36 comments total) 32 users marked this as a favorite
 
(They should rather have spent their efforts on making a decent Civ-playing AI...)
posted by Harald74 at 6:29 AM on January 28 [15 favorites]


making a decent Civ-playing AI..

They make up for this in Civ VI by having OP Barbarians attack you if you so much as look in their general direction.
posted by Fizz at 6:35 AM on January 28 [9 favorites]


The AI in Civ VI is even worse than in Civ V. They seem to just stop building higher tech units after a certain point, and all you need to do is survive until then. I've really gotten tired of them relying on buffing the AI players' stats to gate the difficult, and games of rote optimization aren't interesting for me anymore. If I need to optimize something, I'd rather write a program to do it.
posted by I-Write-Essays at 6:49 AM on January 28 [3 favorites]


I do not use a lot of Mods. I tend to play most of my games vanilla. For people who are deep into the modding/gaming community, I wonder if there is a history of these types of "sponsored" mods? Or is this a new trend we have to look forward to? Because I can't wait for the alt-right to start modding their horrid ideology into games, or is that already happening?
posted by Fizz at 7:19 AM on January 28 [2 favorites]


> the alt-right to start modding their horrid ideology into games, or is that already happening?

The very premise of Civilization is patriarchal from it's core, and subversively nationalistic. You play the only Character in your civilization, an un-aging god-king. You progress your Has-Always-Been-A-Nation-State along a never-stumbling path towards imperial grandeur. Can you even get more imperial kitsch than this? The game is completely blind to the last hundred years of philosophy and sociology; it would fit nicely at the table of a colonial governor.

I think the alt-right should be plenty happy with the game already.
posted by I-Write-Essays at 7:30 AM on January 28 [57 favorites]


A very good point I-Write-Essays. I guess I've sort of turned a blind eye to that. Speaks to my privilege and the power dynamics from which I play the game.
posted by Fizz at 7:34 AM on January 28 [1 favorite]


Ugh, feel a little bit gross at myself, now that I'm thinking about it. You're right, the way the game depicts leaders is very cartoonish and troubling in how it reduces peoples/cultures.
posted by Fizz at 7:35 AM on January 28 [2 favorites]


Risk is even worse
posted by thelonius at 7:35 AM on January 28 [5 favorites]


Yeah, once I started seeing it, I started feeling gross every time I'd sit down to play, too. I need to get off my butt and build the post-modern 4X game I've been dreaming of.
posted by I-Write-Essays at 7:36 AM on January 28 [7 favorites]


And it's not just the way it's grossly nationalistic, and subverts young people into imagining history as the progress of civilization, which has always been the same civilization, the people of My Nation, but also, consider how technology is modeled. You progress from ancient to future. You can't lose it no matter how much your science city gets nuked, or how many people die. The ways in which society passes on knowledge from one generation to the next are one of the most important aspects of civilization, completely missing from Civilization.

I quite liked the non-linear web of technology that they tried in Civ: Beyond Earth, but the actual implementation of the techs available sucked, and it still didn't make any attempt to address the 19th century notions of the world inherent in so many aspects of the design.

Social Policy is reduced to a way of getting bonus modifiers on your inevitable grind towards Victory.
posted by I-Write-Essays at 7:48 AM on January 28 [14 favorites]


The upcoming civ VI expansion is exploring some ideas about fracturing national identities, cultural diffusion causing cities to drift away from your nation, and dark ages. I'm curious to see how that'll create more complexity in the concept of the game.

It's a shame because I've always liked the balance civ has between military and non-military strategies in a 4X game but playing it can be a little slimy if you look too closely down the alleys you're not supposed to.
posted by lownote at 8:04 AM on January 28


It's a shame because I've always liked the balance civ has between military and non-military strategies in a 4X game but playing it can be a little slimy if you look too closely down the alleys you're not supposed to.

Now that I think about it, I do not think I've ever successfully won through military force alone. It's either Science, Culture, or Religious wins for me. And most lean towards using Religion/Faith as a weapon. It's a powerful mechanic in the game.
posted by Fizz at 8:06 AM on January 28


The mechanic that mediates the exploration of these ideas is a Loyalty score, which doesn't make it feel any less slimy. But then again, it is unreasonable for Civ to make any significant change from its core premise, because that would put it into a completely different genre. You're supposed to be exploring, expanding, exploiting, and exterminating [the uncivilized barbarians]
posted by I-Write-Essays at 8:08 AM on January 28 [1 favorite]


Because I can't wait for the alt-right to start modding their horrid ideology into games, or is that already happening?
There was a fascism mod for Civilization 2 that only had upsides (likely copied from other mod where it had substancial penalties). I have it somewhere on a warez CD older than some of the turds that identify as "alt-right".
posted by lmfsilva at 8:39 AM on January 28 [3 favorites]


Point:
AI is going to take over the world and we'd need to watch out because it's really smart.

Counterpoint:
AI has already taken over the world and we need to watch out because it's really stupid.
posted by All Out of Lulz at 10:09 AM on January 28 [11 favorites]


“Alexa, please play some music and declare war with the Russians.”
Just a matter of time.
posted by Fizz at 10:14 AM on January 28 [4 favorites]


The Civ series has been the go-to for a lot of old school Games and Learning people for about a decade and a half now, and it's always bothered me for vague reasons that I-Write-Essays elaborated on. It presumes a very bad model of the thing that it names itself after. This is just one among many critiques I've started to formulate about the field. Reading Gee's What Video Games Have to Teach Us with modern eyes, you can see the basic formulations of Gamergate ideas about the superiority of hardcore games in his arguments about what constitutes a 'good' video game.
posted by codacorolla at 10:19 AM on January 28 [3 favorites]


A new vision of 4X: Explore, Excite, Express, Experiment
posted by I-Write-Essays at 10:22 AM on January 28 [5 favorites]


Fizz, I think you might mean, "Alexa, play the War album by U2."
"Okay, I'm starting World War III."
posted by musofire at 11:01 AM on January 28 [6 favorites]


The Civ tech tree is very much a copy of the 19th century idea of human progress, that different technologies are universal and that individual societies undergo a kind of evolution in which they progress along a linear timeline from primitive to civilized. There are different societies at different levels of progress existing together, so you can have civilized people in one part of the world, and primitive people in another; and the primitive people will closely resemble the ancestors of the civilized people. Once a society has achieved a certain level of sophistication, according to this logic, it will have discovered certain technologies, and those technologies can therefore be used to measure the relative progress of any given society. Presumably, given enough time and resources, any society will eventually end up closely resembling the modern civilization that we live in today, which is the pinnacle of what humans as a species have achieved so far.

You hear this model of the world cropping up in conversation all the time. It's why people seem astounded that the Inca never invented the wheel, because that's a pretty basic hallmark of a civilization, right? It's part of why people think it's only natural that the Spanish, with their guns, "conquered" the Mesoamerican civilizations, who only had spears and things you could chuck at one another. North American Indians used primitive bows and arrows, not like the guns that the Europeans brought. Modern people living in remote parts of the world are trotted out as examples of what our (Western) past was like, a glimpse into living history that hasn't changed for tens of thousands of years (even my referring to them as "modern" people might bring up some objections, because modern usually implies living more like we do).

It's all wrong: the Inca didn't need wheels, and Spanish guns were good enough, but they only got anywhere by making strategic alliances while the empire was being split by civil war (most of the fighting was indigenous vs. indigenous). European guns in the 16th century were kind of awful, and bows and arrows could be much more accurate, reliable, and fast. And of course, no human living today is a living fossil. The idea that certain groups of people represent a shared, universal human past is ridiculous. It presumes that certain societies are unchanging, which simply isn't true, and it presumes that everyone follows roughly the same path, which makes no sense in a world with dramatically different environments, resources, and people.

Civ simplifies everything for the sake of the game. It's fine for a game, but the logic behind the tech tree is pervasive, and reinforces the teleological view of things that was already so common. It lays out, concretely, that you need to do X before you can do Y. And it's not just technology, it's philosophy as well. You need to have certain ideas before you can make certain advances, otherwise you're frozen in time like one of those "primitive" tribes in some remote area. Every civilization has to follow the same tree in the same order, and you can make certain choices to advance certain forks sooner than others, but eventually every civilization needs to check off every box if they want to proceed. Only once you've invented enough stuff do you get to advance and look more modern; eventually, everyone gets cities with skyscrapers, because that's the peak of modernity.

I'm frankly disappointed that Cambridge dipped their toes into Civ in the first place. The entire premise of the game is antithetical to serious anthropology. It's a fun gameplay mechanic, but it's a terrible medium for sending a serious, complex message.
posted by shapes that haunt the dusk at 12:36 PM on January 28 [22 favorites]


It's not even that fun of a game mechanic once you realize the only thing that matters is finding the weights of the various objects in the game, to determine the optimal path for pushing the blue bar forward as fast as you can without dying; and higher difficulty levels are only more difficult by reducing the tolerance on inefficiency. It's a game that trains you to be a factory manager.
posted by I-Write-Essays at 12:49 PM on January 28 [3 favorites]


Civ simplifies everything for the sake of the game. It's fine for a game, but the logic behind the tech tree is pervasive, and reinforces the teleological view of things that was already so common. It lays out, concretely, that you need to do X before you can do Y. And it's not just technology, it's philosophy as well.

Very well put. I think this speaks directly to the concerns that were mentioned in the linked blog post. That CSER's creation of this particular mod will only further reinforce specific ideas/behaviours that players will associate with AI technology.
posted by Fizz at 1:00 PM on January 28


I'm generally fine with Civilization taking the grand sweep of human history and boiling it down into a wargame. It's a very good wargame, but it is defined by the limitations of its genre.

Moreover, I don't think it's possible to 'fix' it by being more accurate to the lived experience of 5,000 years of history and people who have wildly different cultural values, while also making it a fun game to play. No model can stand up to that kind of stress - Crusader Kings II limits itself to realistically capturing the Crusades era, and that game is almost impenetrably complex.

We forget that what made Civilization so effective in the first place was that its scope, while keeping some measure of verisimilitude and continuity, was very unusual for a war game. People living 5,000 years ago had very different priorities to people living 2,000 years ago, in turn different to people living today. By rights, they should be three different games.
posted by Merus at 2:31 PM on January 28 [4 favorites]


For all it’s faults, I’ve learned stuff about the real world from the game. I had no idea that at one point in history there was a gigantic Khmer empire. Shit, until the most recent reveals I never knew about Wilhelmina of the Netherlands.
posted by um at 2:39 PM on January 28 [3 favorites]


I feel like this is where Alpha Centauri and the sequel, Alien Crossfire, had some advantages. While mechanically the games are very similar to basic Civ, the framing story and the way it could affect game mechanics gave you some alternatives. Having humans (and other alien civs) land on another planet with its own indigenous lifeforms and working out how to deal with that was more interesting, though perhaps that's because I always liked to play the more Planet-inclined civs like Gaia's Stepdaughters, the Caretakers and the Cult of Planet. If you wanted the intensely militaristic "crush them all" attitude you could do that, and the game definitely featured a lot of things like the workshop for designing specific types of units with different abilities (paradropping colony pods FTW) but I thought it was much more interesting to research things, institute social policies and even terraform in a way that worked in harmony with the planet (spoiler: it is a sentient planet) and its indigenous life forms.

I never played Beyond Earth because it is apparently really buggy, dunno if it was really the "spiritual successor" to Alpha Centauri as it was marketed. But I would love for there to be more games like that, where the choices you make are part of a larger story and there are more options for how the game can end than domination by a single faction (whether that is military, religious or cultural).
posted by Athanassiel at 3:12 PM on January 28 [3 favorites]


None of the linked articles describe what the mod actually does. How can a "thought experiment backed by numbers and systems" generate meaningful discussion without describing those systems?

From the screenshots, it looks like you can spend resources on Rogue AI that has short term benefits but long-term risks. Or on AI Safety Labs (like CSER) which reduce that risk. Or on Benevolent AI that gives instant victory. This sounds a lot like the played-out "slave revolt" narrative. (previously)

I don't think this does much for the conversation about superintelligent AI or existential risk. But I would love to hear from someone who has played through and can explain the mechanics in more detail.
posted by Phssthpok at 5:35 PM on January 28


The machine uprising depicted in Stellaris was entertaining. It's potentially triggered by any empire heavily utilizing synthetics but hasn't given them citizenship rights. They combine their powers to create a machine super-intelligence which aims to overthrow their organic masters, instantly taking over multiple planets / worlds and plunging the empire into civil war. Depending on how they were treated as synths, they could go on to attempt to exterminate the rest of organic life in the galaxy, again treating other organic races depending (IIRC) on how they treated their own synths, eg a race that gave their synths citizenship rights might get a free pass from the new machine empire.
posted by xdvesper at 6:51 PM on January 28 [4 favorites]


I think the alt-right should be plenty happy with the game already.

The alt-right hates Civ because it sometimes represents non-whites and women as being on the same level as the Great White Conquerors (who they don't really understand but have devoted their intellectual* lives to the worship of anyway). Their response to the announcement of Amanitore was certainly something to see.
posted by IAmUnaware at 9:29 PM on January 28 [5 favorites]


So, basically, Cambridge is arguing that AI is inherently dangerous and bad and the only way to keep humanity safe is to bind AI to slavery? Ugh.

What's with all these people advocating for digital slaves lately? Elon Musk is out there saying AI is horrible and evil and dangerous and the only way to keep us safe is by enslaving it too.

Wild thought: giving AI civil rights would be a step in preventing it from trying to hurt us.

Athanassiel Beyond Earth wants to be the spiritual successor to Alpha Centauri, but it fails miserably at that task. All the factions are boring same-same, the tech is bland at best, the tech tree was an interesting thought but never really worked because you could build to all the branches and anyway it too was kinda same-same just with different graphics. Oh, and they broke up the original game to sell as DLC (want to build water cities? That's a DLC, want to build in space? That's a DLC).

Worst of all though, as a sort of nod to Alpha Centauri they included a transcendence ending, but it was blatantly tacked on and made no sense at all since there was no overarching planetmind sort of thing going on.

Basically Beyond Earth was bad on all levels.
posted by sotonohito at 7:24 AM on January 29


I think the AI community has been bad at doing this [public communication] for a long time, because in general most people don’t lose out from being imprecise (and in many cases in AI, can actually gain from it)

So I agree with this. And extend it across most "tech" and much science communication. And IMHO with places like CSER there's this weird dynamic with the tech billionaires who give them money and publicize them. On some level I think it strokes their vanity to be told "You might destroy the world" in a way that criticism like "You don't let your workers unionize" does not.

That being said her main complaint is that the event they are modeling is extremely unlikely, and making it a predetermined invent implies certainty. I don't know how you get around that? It's not like it makes sense to have a game you can play 10,000 times to get the new scenario. Heck, I've never gotten the AI rebellion in Stellaris and I think that's like a 1-in-5 probability or something.

Even talking about a super rare event (whether gray goo, AI, terrorism, supervolcanoes or large meteor strikes) pretty much guarantees you are over estimating the risk on some level. The human mind doesn't have an intuition for small risks.
posted by mark k at 7:50 AM on January 29 [2 favorites]


sounds like it's not nearly as scary/exciting/wonderful (depending on your level of ingrained Eurocentrism) as the Huns expansion in EU III

all this work of selective breeding and inbreeding all for naught against the combined military might of a civilization who really knew how to make use of their ponies
posted by runt at 9:09 AM on January 29


AI is not a person and cannot be "enslaved".
posted by thelonius at 9:15 AM on January 29


(unless you want to pretend that someone knows how to make a conscious entity, which, as far as I can see, no one has the fainsest clue how to do).
posted by thelonius at 9:28 AM on January 29


Depends on how you define AI.

Alpha GO is not a person, no argument at all. Likewise the various driving "AI".

But future implementations of AI, which is what most of the people are panicking about and what we're talking about here, may well be a person. If it is self aware, and capable of making decisions and operating at (or above) general human intelligence levels I'd argue that's a person.

I'm doubtful we'll get sapient general purpose AI anytime soon, but I don't think there's anything magic about neurons and brain tissue either. To me it doesn't matter of the processor is silicon, meat, or whatever, if it thinks, is self aware, and so on then it's a person.

We bundle a whole spectrum of things under the label AI, and will need to start separating them sooner or later. Some people are already differentiating between AGI (Artificial General Intelligence), which would be the sort of self aware, AI person I'm talking about here and "mere" AI which is not self aware and is engineered for a particular function. You'll also hear AGI termed "true AI", or "full AI", or "strong AI".

AGI would, in theory, pass the Turing Test and be able to carry on a conversation about any topic it has knowledge of, solve new problems, exhibit creativity, and generally do what human intelligences do.

Musk et al seem to have binged on the Terminator movies and decided that any AGI created would be hostile to humanity and must be enslaved to prevent it from destroying us.
posted by sotonohito at 9:29 AM on January 29 [1 favorite]


AI is not a person and cannot be "enslaved".

I'm not interested in getting into the weeds about 'can we make a conscious being.' I don't really think so either. However, how we approach the fantasy of it is still telling:

If a person's first reaction to 'maybe we could create a being more powerful than ourselves' is 'it would destroy us, we must shackle it!' that's... well. There are a couple disturbing things with that sentiment, IMO:

- We're bad. If we met a stranger, they would judge us harshly for who we are and what we do, maybe enslaving or destroying us.
- The best way to protect ourselves is preemptive aggression against the Other instead of striving to be worth knowing and saving on our own merits.

Honestly, that facet of this discussion reminds me of the fresh FPP about loneliness. Like, maybe that's not the healthiest position to hold even if it's not truly possible because of what it reveals about our self-image and problem solving skills.
posted by mordax at 11:07 AM on January 29 [4 favorites]


The way this discussion has gone has reminded me that the late John Sladek had some great novels on AI. Mechasm,
Tik-Tok,
and Roderick. Each is exponentially better than the last, in that Mechasm is a decent pulp parody, Tik Tok a work of genius and Roderick is one of the best things ever. (And if you care about this sort of thing, he's also the kind of author who wouldn't use exponentially sloppily like I just did.)

I want to say you'll learn more from him than you ever would from CSER, but I have no idea if this is true. I'm sure you'll laugh more though. The dialogue that triggered in my memory for me here (heavily paraphrased) involves Roderick, who's a very humane and really nice AI in a robot body who for some reason some people want to kill:
Human AI expert: Well, anything could happen with AI. What if AI decided they were better than us? That they didn't need us? And then evolved a level of intelligence so they could wipe us out?

Roderick: What if they evolved a level of intelligence that made them think wiping out other species is a bad thing to do?

Human AI expert: Well, sure. I didn't say there were no counter arguments.
posted by mark k at 9:13 PM on January 29 [2 favorites]


« Older quantum physics, traffic, chicken eyes, random...   |   Double, double, toil and trouble Newer »


This thread has been archived and is closed to new comments