"Dave, my mind is going. I can feel it."

February 4, 2013 5:25 PM

A robot begs not to be switched off.

Various factors might influence the decision on switching off a robot. The perception of life largely depends on the observation of intelligent behavior. Even abstract geometrical shapes that move on a computer screen are being perceived as being alive ... in particular if they change their trajectory nonlinearly or if they seem to interact with their environments, such as by avoiding obstacles or seeking goals ... The more intelligent a being is the more rights we tend to grant to it. While we do not bother much about the rights of bacteria, we do have laws for animals. We even differentiate within various animals. We seem to treat dogs and cats better than ants. The main question in this study is if the same behavior occurs towards robots. Are humans more hesitant to switch off a robot that displays intelligent behavior compared to a robot that does show less intelligent behavior?

Paper, HTML, hosted on author's site.

NPR write-up.

posted by codacorolla (48 comments total)

No Disassemble!
posted by blaneyphoto at 5:36 PM on February 4 [7 favorites]

Programming is fun.
posted by Decani at 5:37 PM on February 4

10 PRINT "PLEASE DON'T PRESS RUN/STOP!"
20 GOTO 10
posted by JHarris at 5:39 PM on February 4 [2 favorites]

This is creepy, and I do believe would cause me to switch off the robot -harder-.
posted by Archelaus at 5:54 PM on February 4 [1 favorite]

This is the thing that frightens me most about artificial intelligence research - what happens if we succeed and we create beings that are too intelligent to use as nonliving research tools?
posted by LSK at 5:56 PM on February 4

I still feel remorse over what I did to that companion cube. Meet you on the other side of the Rainbow Bridge, buddy.
posted by sourwookie at 6:01 PM on February 4 [9 favorites]


~/sadist> ls
have_pity.sh
~/sadist>
~/sadist> cat have_pity.sh
#!bash
echo -n "Please dont hit return...."
read LINE
echo "You bastard."
rm "$0"
~/sadist>
~/sadist> ./have_pity.sh
Please dont hit return....
You bastard.
~/sadist>
~/sadist> ./have_pity.sh
bash: ./have_pity.sh: No such file or directory
~/sadist>
~/sadist> ls -l
total 0
~/sadist>



posted by benito.strauss at 6:12 PM on February 4 [8 favorites]

In a similar vein too.
posted by Talez at 6:33 PM on February 4 [2 favorites]

I'm old school:

http://www.youtube.com/watch?v=DHo4yGjYhJ

I don't do html very well... modstress if you want to clean this up...

Please don't dismantle me...
posted by primdehuit at 6:36 PM on February 4

I still feel remorse over what I did to that companion cube. Meet you on the other side of the Rainbow Bridge, buddy.

Okay, this drives me crazy.

The whole point of the companion cube is that it's bullshit! All of GlaDOS's needling is funny because it's based on her crazy misapprehensions regarding what she thinks will be hurtful to say, but that almost nobody could ever take seriously. The whole "companion cube" joke is predicated on the fact that it is literally just a box like every other box in the game, except this crazy-ass robot thinks that if it stencils a heart on it, you will fall in love.

But the robot is wrong, because it is just a box. And you toss it in the incinerator and GlaDOS tries to guilt-trip you over how fast you did it, but that's just yet more poorly-calculated emotional manipulation on her part. The whole joke of her ongoing attempts to needle you hinges on the fact that she's like the uncanny valley of manipulators—close, but just slightly enough off that it totally undoes any impact she might hope to have.

This incidentally is why characterizations of Chell as being obsessed with cake and companion cubes bug me. It seems obvious to me that she knows that everything GlaDOS says is totally dissembling, and clumsy dissembling at that. That's the joke. That not only do you have to run these stupid tests, but that this robot thinks it's getting sick burns on you the whole time but they are not sick burns at all. Chell wouldn't exit the testing facility with an obsession with cake and cubes—she would instead have a nasty case of PTSD and a firm conviction that robots are full of shit.

I guess what I'm saying is I did not have companion cube separation anxiety, because inanimate objects do not have feelings.
posted by Sokka shot first at 7:13 PM on February 4 [25 favorites]

.
posted by Stoatfarm at 7:16 PM on February 4 [2 favorites]

I always read the companion cube section as deriving from GladOS' experience with the rat man. Rat man did actually fall in love with the companion cube (cuz he was crazy), and the joke was that, because of this, GladOS thought falling in love with companion cubes was just a thing that humans did.
posted by Ndwright at 7:32 PM on February 4 [2 favorites]

The whole point of the companion cube is that it's bullshit!

Nope! A bunch of people feel bad about it still, as evidenced above! Still a valid reading of the game, even if it's not the one you had (you monster).
posted by Greg Nog at 7:35 PM on February 4 [9 favorites]

related
posted by Wemmick at 7:36 PM on February 4

Easy there, Sokka. I got all that--right from the firsty play-through. I never had any other interpretation. I was joking that I would fall for GladOS' manipulations.
posted by sourwookie at 7:43 PM on February 4 [2 favorites]

This incidentally is why characterizations of Chell as being obsessed with cake and companion cubes bug me. It seems obvious to me that she knows that everything GlaDOS says is totally dissembling, and clumsy dissembling at that. That's the joke.

And then that same GlaDOS, a deranged version of her mother, keeps trying to murder her, and for just a minute she almost gets something like her real mother back, but GlaDOS murders that part of itself and exiles her to wander lonely and unloved through a wilderness she can't recognize until exposure, accident, or predation kills her.

I try not to think about the plot in Portal/Portal2.
posted by ROU_Xenophobe at 7:45 PM on February 4 [1 favorite]

Screwdriver's so sharp, now I'm scared...
You folks, sheesh.
posted by primdehuit at 7:58 PM on February 4

Google:
Miss Clarke And The Computer
Roy Wood
He also used to play with the Electric Light Orchestra.
That is all.
posted by primdehuit at 8:04 PM on February 4 [1 favorite]

Rat man did actually fall in love with the companion cube (cuz he was crazy), and the joke was that, because of this, GladOS thought falling in love with companion cubes was just a thing that humans did.

As a Doug Rattman cosplayer, I assure you that GlaDOS was fully aware of the nature and extent of his mental dysfunction and what it meant in the larger context of typical human behavior:

You've avoided capture for weeks. What makes you so different? Ahh... Delusions of persecution, pathological paranoia; it's all right here in your file. Have you refilled your prescription lately? ...

Schizophrenia is a culturally bound phenomenon. Its pattern of expression is filtered through the cultural substrate in which its symptoms develop. In technological societies, this manifests as delusions of surveillance and a belief that advanced technology is deployed against you, usually with some vague unseen "other" out to get you. ...

I can't see you but I know you're in there. Is it just coincidence that you've been diagnosed with schizophrenia and and now believe that a homicidal computer is out to get you? I mean really, you're a scientist. What is more likely, that you're being chased by a homicidal computer, or that this is all just the paranoid delusion of an unstable mind? Why not come out of there, and you'll see. None of this is real. I'd ask you to think outside the box on this, but it's obvious your box is broken.

And has schizophrenia.

That's not the taunting of an AI who would mistake acting like an inanimate box loves you right back and has deep conversations with you as universal human behavior.
posted by radwolf76 at 8:05 PM on February 4 [4 favorites]

Jules is very sad before being packed up and sent to a university in England.
posted by madamjujujive at 8:29 PM on February 4

Google:
Miss Clarke And The Computer
Roy Wood
He also used to play with the Electric Light Orchestra.

Man, that first ELO album is beautiful insanity. I wish they had continued in that vein.
posted by sourwookie at 8:41 PM on February 4 [1 favorite]

"Dave, my mind is going. I can feel it."

The first instance of performance of a prefrontal robotomy.
posted by drhydro at 8:56 PM on February 4 [2 favorites]

I always thought that the point of the Companion Cube thing was that you needed to carry the same darn cube through the entirety of one of the game's larger puzzles, and inevitably became a bit attached to it.
posted by schmod at 9:17 PM on February 4 [1 favorite]

This incidentally is why characterizations of Chell as being obsessed with cake and companion cubes bug me.

You're missing the joke-within-a-joke. Of course Chell's major motivator is survival and escape, but it's humorously ironic if the first thing on her mind as soon as she escapes is, "damn, I'm really craving a piece of cake right now."

As for the begging robot, me knowing that AI science is nowhere near producing a Turing Test sentient system, I'm okay with switching the dumb thing off. But the ethics would be pretty indefensible if we had the science to pass the Turing Test.
posted by Skwirl at 9:46 PM on February 4

This reminds me of an old short story.

But yeah, total dick move to turn off a sentient robot -- UNLESS you're reluctantly acting as its defense attorney in a trial which proves or disproves that it has the same rights as humans.
posted by RobotVoodooPower at 9:53 PM on February 4 [1 favorite]

Maybe it makes me a sap, maybe it makes me deranged, but I really did get emotionally invested in the companion cube during that level of Portal. The companion cube was a fictional character, and I don't think it's unusual or wrong to get emotionally invested in a fictional character; that's one of the things that fiction does for us, provide us with a template for emotional investment. But, yes, it was also just a box, with no scope for action or emotion, or even a hint of anthropomorphism... and that was one of the important things about it.

Conceptually, within the framework of the game, the companion cube is an exact mirror of GladOS, the sadistic computer which torments the silent protagonist, Chell. GladOS has agency, the cube has none. GladOS has a strong personality imposed not only on the protagonist, but on the whole environment. The cube is devoid of personality except insofar as you, the player, invest it with one. GladOS hates you and does everything in her power to bring you down. The cube can only help you, if inactively. The sheer palatability of GladOS' unconditional loathing creates, in the mind of the player (and, presumably, the mind of Chell), a need for an object projecting unconditional love. And when GladOS tells you that the cube loves you and is your friend, that seems to make sense, because the cube is the opposite of GladOS.

I've read that the companion cube, itself, was a sort of accident. The writers needed you to carry this box through the level, but during testing, players kept leaving it behind. So they had GladOS (who talks to the player character frequently throughout the game) tell you that the box is important, so you won't leave it behind.

But players became attached to the cube... really attached. It became something of a fan favourite if only because it stood out as a (howsoever hollow) moment of redemption in a landscape of utter desolation. It ended up being one of the cleverest parts of a very clever game, because it embodied the game's two major themes of artificiality and emotional manipulation.

See, we, the player, are fully aware that GladOS is trying to manipulate Chell. It is transparent, crude, and comical. But we are also aware that we, the player, are essentially signing up to be emotionally manipulated by the computer sitting on our desk, the one running the game that we are voluntarily playing. When GladOS says hurtful things to Chell, we laugh. But when Chell fails we feel frustrated, when she overcomes the puzzles and strikes back at her tormentor, we feel happy. The game is inviting us to laugh at its own mechanics, even as we buy into them.

The companion cube is the thematic fulcrum of this emotional see-saw. It is an object which we can choose to treat like GladOS, as a comic device, or like the game itself, as an object of genuine emotion. The way we feel about it, during the game, is a sort of microcosm of the way we interpret the whole experience of playing.

Computer games are, almost uniquely, the art form of choice. Choices are integral to the audience experience, and choice itself is a major theme explored by recent games. The companion cube is a blank screen upon which our choices are projected and that, in my interpretation, is what made that moment in that game such a famously evocative and widely celebrated one.
posted by Dreadnought at 9:55 PM on February 4 [10 favorites]

More related nostalgia: Floyd dies.
posted by RobotVoodooPower at 10:24 PM on February 4

"Look into your hard drive, and open your mercy file!"

"File not found."
posted by Rhaomi at 10:38 PM on February 4 [3 favorites]

why isn't anybody talking about the dead robot
posted by obiwanwasabi at 12:05 AM on February 5 [1 favorite]

What the HAL 9000?
posted by Cranberry at 12:56 AM on February 5

Anything with a face on it tends to get treated as alive, due to the quirks of the human brain. It hardly matters whether the thing is a robot or not, let alone whether it has anything even remotely resembling a mind. You could chalk a face on a wall and people would be measurably inhibited from demolishing it compared with a faceless wall.
posted by Segundus at 1:30 AM on February 5

"GlaDOS murders that part of itself and exiles her to wander lonely and unloved through a wilderness she can't recognize until exposure, accident, or predation kills her."

I thought that GlaDOS had come to respect and even have some fondness for Chell, but it couldn't admit to these feelings so it claimed it had quickly deleted the bothersome emotional parts of its personality and was letting Chell go because trying to kill her was proving to be a waste of time. But then as Chell is leaving, we get that terribly sad little aria from the drones, with lyrics that apparently translate to something like, "My dear little one, stay away from science..." Chell is being sent on her way with a strange kind of blessing.

You could see the appearance of the cube at the end of Portal 2 as GlaDOS showing its contempt for Chell by treating her the same way it's treating the cube, tossing them both out like garbage, and slamming the door behind them. But I think it's sort of a loving gesture. GlaDOS is setting Chell free, and giving her back the closest thing she's had to a true friend through this whole ordeal.

And to get back to the point of the post, I definitely wouldn't switch off that robot. Partly because the robot is so cute, partly because I'd wonder if this was supposed to be a new version of the Milgram experiment, and partly because if something's begging for life, I'd pay attention to it. Sure, it's most likely just a trick, and the machine doesn't really have a will to live... But Jesus, what if it does?
posted by Ursula Hitler at 2:45 AM on February 5 [1 favorite]

You know what makes humans different from other animals? We’re the only species on Earth that observes Shark Week. Sharks don’t even observe Shark Week, but we do. For the same reason I can pick up this pencil, tell you its name is ‘Steve’ and go like this- [breaks pencil] and part of you dies, just a little bit on the inside. Because people can connect with anything. We can sympathize with a pencil, we can forgive a shark, and we can give Ben Affleck an Academy Award for screenwriting.

posted by Gordafarin at 3:41 AM on February 5 [2 favorites]

The whole "companion cube" joke is predicated on the fact that it is literally just a box like every other box in the game, except this crazy-ass robot thinks that if it stencils a heart on it, you will fall in love.

Except humans will feel affection for inanimate objects, actually.

One interesting aspect of robot research is figuring out how to make people like them. I remember seeing a video about a simple 'cute' looking robot with a sign saying it was trying to get to a certain location (in a park or something), to see if people would help it get there by pointing it in the right direction. In the end it did. There's also stuff like keepon

Also, I've never played portal but I always found the storyline kind of fascinating. Looking around on youtube I came across this animation of some of the dialog, which is pretty hilarious.
posted by delmoi at 6:12 AM on February 5

I guess what I'm saying is I did not have companion cube separation anxiety, because inanimate objects do not have feelings.

As far as the cake and cubes go, people fixate on it because it's funny, not because they don't get that GLaDOS is bad at insults and lying.

Part of what makes Portal great is that Chell, the only living creature in it, has no personality -- while GLaDOS and the turrets, all machines, appear to. (*)

The turrets don't even have to display anything resembling intelligence. There's no evidence that even within the game world they do anything other than track targets and play back prerecorded sound files. While less likely, in Portal 2 even the Different Turret could have been set up that way. So it's like seeing Jesus on a piece of toast.

And you can extrapolate that idea to other humans, or in fact seeing Jesus in things other than pieces of toast.

You cannot prove to me you're not an inaminate object that does not have feelings, and I can't prove the same about myself. All we have are guesses based on our own experience.


(*) this is beanplating. The thing that's great about Portal is it's funny and clever and different and highly polished.
posted by Foosnark at 6:54 AM on February 5 [1 favorite]

Or put another way, we are machines made of meat, "designed" by accident.
posted by Foosnark at 6:57 AM on February 5

This incidentally is why characterizations of Chell as being obsessed with cake and companion cubes bug me. It seems obvious to me that she knows that everything GlaDOS says is totally dissembling, and clumsy dissembling at that. That's the joke. That not only do you have to run these stupid tests, but that this robot thinks it's getting sick burns on you the whole time but they are not sick burns at all. Chell wouldn't exit the testing facility with an obsession with cake and cubes—she would instead have a nasty case of PTSD and a firm conviction that robots are full of shit.

The game itself seems to directly refute this though. Rattmann's dens are filled with writing that suggests he is obsessed with both the companion cube and the cake, having presumably been in the exact same situation as Chell.
posted by burnmp3s at 7:23 AM on February 5

I don't say this often on Metafilter, but I think that we might be overanalyzing Portal just a tad....
posted by schmod at 8:58 AM on February 5 [1 favorite]

Emotionlal investment in a machine is a red-herring. For example, I loved my dirt bike, and I cried when we lunched out, on the first turn of the TT course, at 60 miles per hour. It was a mess, and all the new parts...pretty much everything but the rear tire...didn't make it right again. It never was the same. Sorry, I forgot what I was....Oh.

Sentience is the real issue. Are we not measured more by what turns us on than what turns us off?
posted by mule98J at 9:04 AM on February 5

I keep seeing references to GLaDOS being Chell's mother, but I never saw anything in-game to infer that.

Since we're talking about robot compassion and Portal in the same thread, I think the concept of not destroying the robot is an interesting contrast to the co-op version of Portal 2. In which the designers explicitly made the characters robots so that players wouldn't feel so bad about making the other guy fall into a pit.
posted by CancerMan at 9:05 AM on February 5

Why you dead, Ricky?
posted by It's Raining Florence Henderson at 9:23 AM on February 5

This reminds me of the Furby toys from my childhood, when you turned them over to switch them off they would protest.
posted by Autumn at 10:11 AM on February 5

This is the thing that frightens me most about artificial intelligence research - what happens if we succeed and we create beings that are too intelligent to use as nonliving research tools?

Daniel H. Wilson takes this premise and runs with it in Robopocalypse. It was a fun read, kind of Watchers (Dean Koontz) meets Red Dawn.
posted by ElGuapo at 10:11 AM on February 5

Sometimes when an electronic device asks not to be turned off, it's kind of important!
posted by Kid Charlemagne at 12:31 PM on February 5 [1 favorite]

Daniel H. Wilson takes this premise and runs with it in Robopocalypse. It was a fun read, kind of Watchers (Dean Koontz) meets Red Dawn.

I thought of it as a zombie story, but with robots.
posted by Foosnark at 1:31 PM on February 5

Maybe you should feel bad for the companion cube.
posted by Sibrax at 3:44 PM on February 5

CancerMan: "I keep seeing references to GLaDOS being Chell's mother, but I never saw anything in-game to infer that."


[SPOILERS]


GLaDOS was created from the consciousness of Caroline, Cave Johnson's assistant. From her portrait's resemblance to Chell, take-your-daughter-to-work-day, the lyrics of GLaDOS' opera song, and probably a few other things I've forgotten, people have surmised Caroline is Chell's mother, though it hasn't been confirmed.
posted by Gordafarin at 1:15 AM on February 6

[Spoilers]

In-game, I guess I just don't see the resemblance. I thought Chell was just the daughter of an unnamed employee (father, judging from her science project), and considering the time frame Caroline was introduced as an adult (mid 1950s to early 1960s?), it seemed a stretch to think that in the time-frame of when the game begins, she would be the mother, adoptive or not. Plus, if Chell created her science project at a young age, and that's the same time GLaDOS was switched on and the neurotoxin released, I think that would have been in the late 1990s?

I can imagine if Caroline was Chell's grandmother, though. Are there references for that?
posted by CancerMan at 9:55 PM on February 6