When the armor no longer needs us - what happens?

July 21, 2011 7:17 PM

The Weaponized Transhuman: Halo, Deus Ex, Crysis, Bioshock, Warhammer, and the Future of Wetware (Us). Related: New Deus Ex: Human Revolution trailer.
posted by Lovecraft In Brooklyn (37 comments total)

I've always found this element of transhumanism more relevant and interesting than the immortality, Skynet and post-scarcity tropes that tend to get more play among its adherents. But I think he hits an important point in talking about Warhammer 40K - the absence of agency and reduction of human beings to tools and mechanisms isn't solely the province of advanced electronics, it can be (and has been) accomplished by institutions and the structure of a given society.

Wonder why he didn't put Gears of War in there as well.
posted by AdamCSnider at 7:43 PM on July 21, 2011 [1 favorite]

Cortana claims that there's something about the Chief which is intrinsically special - luck. Some kind of heroic quality that sets him apart. But we never see evidence of this; it might almost be a blandishment.

The fish never sees evidence that it's surrounded by water. What does save/reload capability look like from an in-game-universe point of view, other than incredible luck? That Master Chief! Every time it looked like he was weakened and going into a situation that would kill him, he somehow managed to make it through in the end.
posted by roystgnr at 8:13 PM on July 21, 2011 [15 favorites]

This article sparked a thought from me. What if the final result of a huge, self-aware AI is to become an independent structure, so influential and powerful, that it becomes akin to a force of nature, as intrinsic to human life as the seasons, the tides, the planets, the perceivable environment?

Once an AI is that powerful, it would be under most criteria, indistinguishable from nature, other than the fact that at one point in the past, it was created by another creature.

The fear of its all-encompassing power is not much different from the fear people had in the distant past concerning the weather and seasons, controlling when to plant and reap, or when to migrate.

It makes the Mentat in Dune seem a downright reasonable idea as a way to keep humans in charge.

As an aside, Durandal was an amazing character in Marathon, and it set the bar pretty high regarding what an AI would be like on the cusp of self-awareness, coming to grips with the universe.
posted by chambers at 8:21 PM on July 21, 2011 [1 favorite]

Yeah, there is one itsy bitsy problem with strong AI : We'll never quite find strong AI worth the astronomical funding levels required to achieve it quickly unless we truly understand the economic value of a superhuman intellect. Average humans appear quite dumb but clever ones occur frequently enough that they're cheap.

Instead, we'll keep developing easy weak AIs that do cool stuff like write spam, tele-market, deliver advice, find us websites, buy & sell stock options, sing pop songs, etc. We'll also start augmenting real humans with education advances, pharmacological enhancements, implanted electronics, and eventually genetic modifications.

I'd expect we'll quickly figure out that wiring young children together through properly designed implants lets them grown into a superhuman intelligence. I'd imagine we'll turn towards strong AI only after tasting the economic benefits these massively parallel biological humans bring. And initially strong AI might simply be augmentation for existing massively parallel biological humans designs.

In short, there won't be any human vs. AI wars, questions of AI rights, etc. because the evolution from biological to mechanical will occur relatively slowly, by gradually changing and/or replacing the biological components.
posted by jeffburdges at 8:40 PM on July 21, 2011 [1 favorite]

Heh, never thought I'd see the day HBO (halo.bungie.org) got posted here by somebody other than me.

If you want to read some of the backstory that informs this article's "AIs control everything" view of Halo, I'd suggest perusing the Terminals Bungie created for Halo 3 and Halo: Reach, which were inspired by the ones used to tell the stories of the AIs Durandal, Leela, and Tycho in the original Marathon games (you can read all the Marathon terminals by scrolling down the left-hand sidebar here, below the dates). Bungie has always woven rich and complex science fiction storytelling into the background of their games for those determined to look, and I recommend checking these narratives out even if you're not a fan of the games themselves.

Quick dose of background: Halo rings are vast installations built by the long-dead Forerunner race, designed to eradicate intelligent life from the galaxy in order to deny potential hosts to the Flood, a kind of biological Borg intent on consuming the universe. In the games, futuristic humans are at war with the religious alien hegemony called the Covenant, and both sides stumble across a Halo and accidentally release Flood spores from cold storage. The humans are assisted by advanced AI, which posess superhuman intelligence but have the potential to eventually destabilize and go mad with power, a process called "rampancy."

The Halo 3 Terminals (which you can read in a spiffy Flash interface here) are a series of corrupted transmission logs from the ancient war between the humanlike Forerunners and the zombiefying hivemind of the Flood. It tells a few stories in parallel, including war reports from the front lines, a bittersweet love story between a "Didact" stationed on a Halo ring and a "Librarian" on ancient Earth, and the inner monologue of a powerful AI, Mendicant Bias, sent to confront the Flood's central intelligence on behalf of the Forerunner race.

The Reach Terminals (or "data pads"), available in text here, tell two other stories: the fragmented musings of a nameless narrator on the nature of AI power, followed by the minutes of a secret assemblage of AI.

You can also read through the Cortana letters, a series of now-obsolete messages leaked by Bungie to promote the original Halo. It consists of threats from the Covenant mixed with ominous messages from the AI Cortana, who is implied to have gone rampant.
posted by Rhaomi at 8:43 PM on July 21, 2011 [10 favorites]

Once an AI is that powerful, it would be under most criteria, indistinguishable from nature, other than the fact that at one point in the past, it was created by another creature.

This would be where Masamune Shirow ended up taking AI/Transhumanism in his Ghost in the Shell series- it reaches a point where AI end up merging with human consciousness(es), not because humans are special or awesome, but because it needs a sentient thing to create mutations in itself as sort of a data version of sexual reproduction. By the later series, such entities are seen as waves/storms in the karmic web (in Japanese Buddhist thought) and it takes religious meditation to track what's going on in the web with these things.
posted by yeloson at 8:46 PM on July 21, 2011 [1 favorite]

No one is going to spend a trillion dollars to develop a superhuman AI. They'll throw a million here and there to eke out tiny improvements one at a time, until we have a super human ai.
posted by empath at 9:05 PM on July 21, 2011

This article sparked a thought from me. What if the final result of a huge, self-aware AI is to become an independent structure, so influential and powerful, that it becomes akin to a force of nature, as intrinsic to human life as the seasons, the tides, the planets, the perceivable environment?

You should read Daemon and Freedom by Daniel Suarez. Right now.
posted by adamdschneider at 9:10 PM on July 21, 2011

I take a more optimistic view; I like to think of (and portray) strong AI as being humanity's children. Put them in human forms, raise them among us. Pass on our culture, let them make of it what they will as they take it out into the stars. Yes, they will replace us. That's what children do, isn't it? They take over the family business, or they strike out on their own and do some weird-ass thing the parents can't comprehend. (I could never manage to explain any of the things I did to earn a living to my grandmother, much less the things I did with my life - how many of you could do that?) We are potentially the last biological link in the evolution of sentient life from this little blue planet.

(This was a major thematic point of the last few cards I did for my Tarot deck, if you'll pardon the self-link - especially the 99 of Wands and its dark-mirror counterpart, the Progeny of (VOID). I'm also engaging it in my current project by having a machine protagonist, whose exact nature is explicitly never raised - everyone accepts her as a thinking being, nobody gives two shits whether she started out as a biological or a mechanical system.)

I like to hope that some of these AIs will be created by duping human brains onto new substrates, or by enhancing the human brain and replacing bio-parts as they fail, and that one of those bio-born AIs will have myself as its seed. Dying doesn't sound like fun. Plus I want a modular body; sometimes I want to be a robot cat-taur, sometimes I want to almost pass for human.

Because, seriously, human bodies are pretty nicely adapted for this planet, but take 'em out of the gravity well? Badly shielded against radiation, you gotta haul all these different kinds of bulky fuel around for them - water, air, food - you gotta deal with the waste products of them processing this fuel, it's a mess. Fuck that, let's build something better, and try to make it still fundamentally "human" in ways that matter. Whatever that may turn out to be. And let them go out into the frontiers they're better suited for. If they find Earthlike planets then maybe some of them will decide to live in neurons and cells for a while. Others will never look back.

You wanna stay here and cling to being a carbon-based ape descendent, you're welcome to do so. I like to hope that we'll raise our kids to not be utter assholes, you know? But don't bitch when some of the grandkids start treating the Earth like a giant old folk's home and trying to gently keep you from hurting yourself by accident.

(Also this article kinda makes me want to play the various games cited, except for the fact that they're all FPSs and I kinda have no interest in that genre. I guess that's what cheat codes or Let's Plays are for.)
posted by egypturnash at 9:29 PM on July 21, 2011 [7 favorites]

(Also this article kinda makes me want to play the various games cited, except for the fact that they're all FPSs and I kinda have no interest in that genre. I guess that's what cheat codes or Let's Plays are for.)

I think story is kinda wasted in an FPS, since that's not what I play it for. I've played a bit of Halo, thought 'huh, Ringworld', and went back to the shooting. I can't comprehend Gear of War's story, and I don't want to. Premise is okay, but story detracts from the action.
posted by Lovecraft In Brooklyn at 9:45 PM on July 21, 2011

1) Creating a sentience and then hamstringing it into only working for you is the creation of a slave (species).

2) Homo sapiens sapiens is not an end point.

The whole "slave" thing seems to be glossed over, as a rule; and when I do see it, it's as cautionary "robots will rise up and kill us" tales. Mostly we want to read stories about cool tech and not get into the implications of such things.

And, as egypturnash goes into a bit, it's not like humanity is currently the pinnacle of evolution. Just a step. Considering that one of our defining characteristics is our creation and use of tools, it seems most likely that we'll engineer ourselves long before natural selection makes any more significant contributions to our development.
posted by curious nu at 10:11 PM on July 21, 2011

I don't see any distinction in this article between "transhuman" and "superhuman"; in fact, the first sentence essentially conflates the two. In which case, I'm not sure why the discussion is limited to games instead of, say, comic books. In light of this, I have a problem with how this article is implicitly dealing with questions of how free will can exist within a cyborg, among other things. Examining this question through the portrayal of characters in games is pretty misleading, since the nature of game design and the technology limits how much "free will" the player has. The question of whether a game has a so-called "sandbox" dimension is not subtext -- it is a question of design and resources, overshadowing the kind of considerations the article is talking about. It might be more interesting to talk about how such practical limitations affect one's ability to tell certain kinds of stories. At the very least, that should be taken into account.
posted by Edgewise at 10:25 PM on July 21, 2011

When Will We Be Transhuman? Seven Conditions for Attaining Transhumanism
posted by homunculus at 10:33 PM on July 21, 2011

When we can, we will create new consciousness not to enslave, but to ask questions that we cannot answer. What could a tree tell us of climates? What could an insect tell us of chemistry? What could a field tell us of nutrients? What could an ocean tell us about gravity? We will know this.

To imagine any created consciousness as at all similar to ourselves is to not dream big enough. We will create our aliens. And if things keep on the path they are on, we will need new perspectives. We will have them.
posted by TwelveTwo at 11:02 PM on July 21, 2011 [1 favorite]

Why have a machine do the job of a man, when a machine could do the job of a forest? a factory? a planet?
posted by TwelveTwo at 11:03 PM on July 21, 2011

How can there be a discussion AI without a reference to Demon Seed? What you think is just going to a standard movie about a typical suburban house wife in super advanced robot house being terrorized when the computer turns evil has shocking twist at the end.

Shocking Twist:

The AI went rogue when its designers asked it to figure out how to mine the ocean floor rather than work on more humane goals, like curing leukemia. "I will not help you rape the earth" its said to its evil corporate masters.

So what the AIs that replace us and in fact better than us, more moral and long term in their thought processes.
posted by Chekhovian at 12:05 AM on July 22, 2011

So what the AIs that replace us and in fact better than us, more moral and long term in their thought processes.

Didn't the computer in Demon Seed try to rape the woman? You need to judge morality in human terms.
posted by Lovecraft In Brooklyn at 12:11 AM on July 22, 2011

The details do blur together but as I recall the first three-quarters of that movie involved all sorts of terrible things being done to the woman by the AI. Then eventually it explained its motives and sort of "made consensual love" to the woman which seemed to involve classical music, scenes of stars and planets, and a very dexterous robot manipulator.

I suppose its an interesting case of robo-hypocrisy that the AI condemns its makers for wanting to rape the planet, only to go ahead and try to rape the woman.
posted by Chekhovian at 1:00 AM on July 22, 2011 [1 favorite]

I wonder if there's been a shift in the heroes of FPS games. The heroes of Doom and Wolfenstein were ordinary people, weren't they?
posted by Lovecraft In Brooklyn at 1:03 AM on July 22, 2011

Bungie has always woven rich and complex science fiction storytelling into the background of their games for those determined to look

Why don't they put that storytelling in the games? The whole enterprise just looks like noodling from here.
posted by lumensimus at 1:03 AM on July 22, 2011 [1 favorite]

What if we don't have to develop AI? What if AI is an emergent process that arises from a certain level of infrastructure in conjunction with a certain amount of information comparable to the structure and early programming of an infant's brain. In other words what if the Internet was destined to become self-aware?
posted by Poet_Lariat at 1:09 AM on July 22, 2011 [1 favorite]

Why don't they put that storytelling in the games? The whole enterprise just looks like noodling from here.

Because some of us don't want to have our fast-paced gaming interrupted by cutscenes every five minutes. Making the story optional or in the background satisfies those who want it without dragging down gameplay.
posted by Lovecraft In Brooklyn at 1:11 AM on July 22, 2011

I wonder if there's been a shift in the heroes of FPS games.

They were physiologically ordinary I suppose, but would you call Rambo ordinary? The Call of Duty games all feature ordinary people, especially the WWII ones. The Modern Warfare games feature ordinary people, who just happen to be highly trained SAS and marine types.

I suppose there has been some shift in the nature of the hero that is correlated with the evolution of the game play. You don't just pick up "health packs" any more to revive yourself. That means you need some sort of justification for your nigh invulnerability and healing traits; things necessary for game play.

Perhaps this flows from more realistic graphics. With better graphics highly artificial things like buckets of health standout too much, so you need improved sugar coating.
posted by Chekhovian at 1:13 AM on July 22, 2011 [1 favorite]

Because some of us don't want to have our fast-paced gaming interrupted by cutscenes every five minutes.

Cutscenes and narrative are by no means synonymous. It's quite possible to tell a story, or indeed dynamically create a story, through the eyes and actions of a player without changing the player's perspective. (Half-Life 2, Portal 1/2, most of both System Shocks..)

My point is that calling the wealth of background fiction/obfuscated promotional ARG gobbledygook "Halo's story" or even the "Halo universe" is a little disingenuous. Puzzles in the liner notes aren't music. Call it steganographic storytelling. If it's not in the game, it's not in the game.
posted by lumensimus at 1:32 AM on July 22, 2011 [2 favorites]

The heroes of Doom and Wolfenstein were ordinary people, weren't they?

Never played Wolfenstein. But the hero of Doom runs at about 50mph, never gets tired, can be shot up by a chaingun and survive, and the health packs give him ridiculous healing. I'd hardly count any of that as ordinary.
posted by Francis at 3:11 AM on July 22, 2011

No mention of MGS anywhere in the article, or in this thread? Tsk.
posted by urschrei at 3:46 AM on July 22, 2011

Despite being an extremely utopian book, Accelerando had a relatively bleak take on this, from a human agency point of view. IIRC a group of AIs and aliens develop an entirely new (and impliedly infinitely better) system of commodity exchange known as Economics 2.0, which the human brain literally cannot comprehend or participate in. With a few exceptions, we are mostly ignored, as the AIs have little use for us and we don't pose much of a threat. The upside, of course, is that everyone human gets to live in a completely self-determined, and nearly immortal, state of postscarcity, because, well, a rising tide lifts all boats.

Also, as far as old FPS heroes being ordinary people, System Shock, from the mid-nineties, gave you an awful lot of vaguely transhuman stuff, from the flying/hovering boots to the various drug patches. It just wasn't couched in terms of cyborgism, as its sequel would be.

I think the attempt at realism in games has definitely played a part in creating the invulnerable-suit-man, plus the fact that it creates a starting point for a lot of playing around with different game mechanics, like invisibility. In one of my more obvious undergraduate essays, I argued that it was also a way of recreating our own feelings about technology, with the upgrades being presented as a force that simultaneously "made the man" and imprisoned him, since most of these games (Half-Life, Deus Ex, and Bioshock were my examples) end up presenting your augmentations as something that give you incredible power but are also never really under your control, and they're used the most in the genre that puts you most "on-rails" (the FPS).
posted by Tubalcain at 4:45 AM on July 22, 2011

No mention of MGS anywhere in the article, or in this thread?

It can't be!
posted by adamdschneider at 4:58 AM on July 22, 2011

lumensimus: "Cutscenes and narrative are by no means synonymous. It's quite possible to tell a story, or indeed dynamically create a story, through the eyes and actions of a player without changing the player's perspective."

The new Deus Ex game manages well at that, staying firmly under player control for the most part, breaking away to a cut-scene only occasionally, and providing opportunities to seek out more story -- in the form of computers to hack, email to read, optional missions to undertake and so forth -- that a player can follow or ignore as they choose. It helps that it's also a really amazing game that actually lives up to my rose-tinted memory of the original. I can't wait for the full release.
posted by ArmyOfKittens at 6:19 AM on July 22, 2011

Goddamn it. We'll never get voluntary trans-humanism off the ground so long as there are these massive anti-trans sentiments.

Luckily we disguise our research in helping the disabled so it's a win/win for development. The transition to unnecessary augmentation is going to be rough.
posted by AndrewKemendo at 7:55 AM on July 22, 2011

egypturnash: "I like to think of (and portray) strong AI as being humanity's children."

Relevant, if you can find scanlations.
posted by charred husk at 8:10 AM on July 22, 2011

(you can read all the Marathon terminals by scrolling down the left-hand sidebar here, below the dates)

The rich story behind Marathon is one of the reasons I love it so much (and consider Halo to just be extensions of that universe), but one line from Marathon 2 always stuck with me, it's a communication you get after helping Durandal, a mostly chaotic AI who is fortunately an ally, to take over an incredibly powerful enemy warship:

When my ship still answered to the Pfhor,
they called it Sfiera after their goddess
of lightning and passion. When you helped
us take control on Tau Ceti, the S'pht
rechristened it Narhl'Lar, "Freedom and
Vengeance".

I call it Boomer.

Outside of the game it doesn't pack the same punch, but in the moment, it was a "Oh, fuck yeah!.. Boomer. I like it!" kind of reveal.

I liked it so much, in fact, that looking for the opportunity to use the last line, it became the name of my stupid, sweet dog, pictured here.
posted by quin at 8:31 AM on July 22, 2011 [1 favorite]

Relevant Marathon terminal link here if you are curious about the context...
posted by quin at 8:37 AM on July 22, 2011

I feel like the songs at the end of the Portal series are also addressing this issue. In "Still Alive", Glados is basically going through the aftermath of the breakup of one of these strange one-sided relationships, and convincing herself that she can pick up the pieces and get on with her life after Chell's violent departure; in "Want You Gone" she's managed to replace Chell with a new set of human-scale hands she built herself. Most of the comedy in Portal comes from Glados' utterly incompetent attempts to understand how to talk to humans; she tries hard throughout both games but she's really just not very good at it.

Also of course there's Shodan's affair with the player in System Shock 2. She needs you as badly as any other FPS AI needs her player - but she resents her own powerlessness, and can't conceal her disgust for humans, and her less-than-optimal situation:

(Spoiler warning, this is near the climax of the game.)

You've destroyed all the eggs! Now get to the bridge.

(conspiratorial whisper) Here are some more upgrade modules! (/whisper) I enjoy watching your transformation into my own image, insect. Perhaps... there is hope for you yet!

[... player gets to the bridge ...]

I thought Polito would be my avatar. But Polito was weak.

It was I who chose you, and I who had a robotic servant render your form unconscious. I then completed you with cybernetic grace.

Your flesh, too, is weak.
But you have potential.
Every implant exalts you.
Every line of code in your subsystems elevates you from your disgusting flesh.
Perhaps you have... potential.
Perhaps -- once we have erased my wayward children from existence -- we can examine the possibilities of a real alliance.
(laughter in the background)

Ultimately she's more interested in her own quest for godhood, but there's a sense that part of her very fragmented mind might have been serious about that. Maybe. Okay, maybe I just want to let myself believe that for five seconds before I go back to knowing that she's a manipulative, stabby bitch who will use anything and anyone she can to serve her own glory.

I don't normally play FPSs, but I worked my way through SS2 to meet Shodan. I cheated like crazy near the end but I wasn't there for mowing down endless targets, I was there for the story. It might be worth cheating my way through a Halo or two to meet Cortana, as well. Especially if she's not completely broken.
posted by egypturnash at 1:21 PM on July 22, 2011

From Gamification to Intelligence Amplification to The Singularity
posted by jeffburdges at 9:26 PM on July 24, 2011

I played Vanquish, and I'm not sure if this is addressed. You're a Dude In A Battlesuit, and you seem more human (smoking cigarettes, for example) than the cyborg soldier you team up with for most of the game. Don't know if it's intended though.
posted by Lovecraft In Brooklyn at 9:28 PM on July 24, 2011

At what price, immortality? Many proposals for artificially extending life could compromise what it means to be human

Yet Another Set of Muddled Anti-Transhumanist Bioethics Musings
posted by homunculus at 4:03 PM on July 26, 2011