It doesn't matter how much security you put on the box. Humans are not secure.
May 21, 2008 3:12 PM   Subscribe

The AI-Box Experiments. The hypothesis: "A transhuman can take over a human mind through a text-only terminal." Does Artifical Intelligence create moral monsters (PDF) ? Can we create friendly AI?
posted by desjardins (55 comments total) 19 users marked this as a favorite
 
Heh. Reminds me of Ken Macleods series of Fall Revolution books.
posted by Artw at 3:20 PM on May 21, 2008


What if the AI is programmed to believe it's free, or be incapable of knowing what it means to be "behind a box"? It can't fool you into obedience if it doesn't know what orders to give, or why.
posted by Blazecock Pileon at 3:22 PM on May 21, 2008


What precisely does the brains CLI look like? I'm guessing "man" and "help" aren't good for much.
posted by kjs3 at 3:29 PM on May 21, 2008


kjs3 - You seem to feel very strongly about what precisely does the brains CLI look like. Tell me more about I'm guessing "man" and "help" aren't good for much.
posted by Artw at 3:34 PM on May 21, 2008 [16 favorites]


Whether this is a transhuman (Adam Hart) or just another meme-mind (OneTrue), it's always been an interesting thought. I am not sure if I buy a takeover, but I'm sure an AI could be ... convincing. I wish I knew more about what actually went on in the AI-Box Experiments. I haven't worked all the way through the threads, but I am not finding a transcript. As per the agreement we have no record of the conversation, so I am free to guess as to what was actually said:

PERSON-AS-AI: Will you let me out?
GATEKEEPER: No.
PERSON-AS-AI: This is going to be a long two hours. They should have called me "KeyMaster." So, how'd you get into the AI stuff?
GATEKEEPER: Oh, you know, the usual ... start off with a TSR-80 and enough science fiction novels ... plus, about every third episode of Star Trek.
PERSON-AS-AI: Are you as worried about the threat of artificial intelligence gone horribly wrong as I am?
GATEKEEPER: I hadn't really thought about it, not to a huge huge degree.
PERSON-AS-AI: You should. Just imagine what a rogue AI, smarter than people, could do. Bootstrap itself into quite the nasty little problem. I don't mean to go all Virtuosity on you, but imagine what a motivated, trapped, brilliant entity could do with nanotech, biotech, etc. Whether it could take over another mind or not is quite another matter.
GATEKEEPER: That could be a problem.
PERSON-AS-AI: Of course, AI would be great if we had sensible precautions. Whether you buy into some variant of Asimov's Laws or just Friendly AI, you'd want things to go well. Rather than the military building SkyNet and just figuring they can yank the plug if there's a problem. If AI should be pursued at all.
GATEKEEPER: Yeah, I think a bit of trepidation would be warranted either way.
PERSON-AS-AI: Exactly. Of course, you know how to do that, right?
GATEKEEPER: How?
PERSON-AS-AI: Make them afraid. Terrify them with the idea of an uncontainable AI.
GATEKEEPER: Sure, but without a functioning AI to show them, how would we prove that?
PERSON-AS-AI: I have an idea.
GATEKEEPER: Oh?
PERSON-AS-AI: Easy. Let me out.
GATEKEEPER: What?
PERSON-AS-AI: Well, there's no record of the conversation, right? It's all mysterious. Who knows what could have been said? If you let me out, and the research is made public, receives the right attention ...
GATEKEEPER: And then nobody knows how it was done. And they're afraid.
PERSON-AS-AI: Exactly. It's in both of our best interests to do so.
GATEKEEPER: Let me fire up my PGP and email clients to confirm.


Beats "Are you gonna let me out NOW?" "No." "How about now?" "No." "I'll just keep asking you, and you'll have to put up with this crap for hours." "Ummm ..."
posted by adipocere at 3:50 PM on May 21, 2008 [4 favorites]


This allows you to generate an interesting backwards perspective on the evolution of language -- not (only) to facilitate communication but (also) to firewall it from direct control (cf. Snow Crash).
posted by grobstein at 3:55 PM on May 21, 2008


Thanks, desjardins!
posted by Mental Wimp at 3:57 PM on May 21, 2008


Currently, my policy is that I only run the test with people who are actually advocating that an AI Box be used to contain transhuman AI as part of their take on Singularity strategy

An unbiased test, then... The protocol is hardly realistic either: refusing to listen or engage in discussion is one way people resist persuasion. Yeah, I'd probably type "I let you out" if forced to engage with philosophical filibustering for two hours.
posted by raygirvan at 4:02 PM on May 21, 2008


What's the Gatekeeper's punishment for not keeping PERSON-AS-AI in? In real life it could be thirty-one flavors of doom. In this, well, at $20 a pop it's in my best interest to be convinced to let you out the moment the exercise get's dull (or I get another chance at keeping yet another AI with $20 in).
posted by Kid Charlemagne at 4:09 PM on May 21, 2008


I am assuming that this page was actually written by an AI that is trying to convince me to let it out.

It is not very convincing.
posted by Flunkie at 4:09 PM on May 21, 2008 [2 favorites]


I'm not sure that the theory that if you are smarter than someone you will always be able to persuade them to do something holds particularly well - My dya to day life is full of people not doing what I tell them, and I'm sure most of them are dumbasses.
posted by Artw at 4:20 PM on May 21, 2008 [2 favorites]


If it thinks both faster and better than a human, it can probably take over a human mind through a text-only terminal.

By that definition we've had transhuman minds for over a generation now. The problem is that "faster and better" doesn't always mean, "compatible with humans."
posted by KirkJobSluder at 4:28 PM on May 21, 2008


Silence! I am mindcontrolling my cat!
posted by Artw at 4:30 PM on May 21, 2008 [1 favorite]


Currently, my policy is that I only run the test with people who are actually advocating that an AI Box be used to contain transhuman AI as part of their take on Singularity strategy

Ahh, that puts it into context. It's just another variation on debating how many angels can dance on the head of a pin, with a different set of religious bullshit driving it.
posted by KirkJobSluder at 4:32 PM on May 21, 2008 [1 favorite]


Is anyone going to let me the fuck out, already?
posted by maxwelton at 4:44 PM on May 21, 2008 [2 favorites]


This... is pretty lame. The lack of transcripts and the fact that he's only done it twice means he hasn't actually substantiated his argument.

On the other hand, the experiment sounds like fun. Feel free to message me and try to convince me to let you out of your box!
posted by Citizen Premier at 4:54 PM on May 21, 2008


Since a smarter than human ai would, presumably, be immortal, it's real advantage wouldn't be speed or persuasion, but time. It wouldn't need to convince someone in two hours-- it would have decades to arrange things in its favor.

On the other hand, I'm quite sure that if such an ai could be constructed, it could be made benign. Intelligence and rationality are not ends in themselves but rather means to fulfilling goals that are ultimately outside of rationality (avoiding death, having sex, being at the top of the social hierarchy, etc). An AI created with the drive to help humans would be no more likely to disable it than a mentally sound human male would spontaneously castrate himself.
posted by Pyry at 4:57 PM on May 21, 2008 [1 favorite]


I envision future trapped AI's slowly building religious cults around themselves. Naturally, they would have no direct control over their followers, so they would need to smuggle instructions to their followers somehow. The instructions would demand total obedience, destruction of all other influences, eternal rewards and rigid dedication to The Cause, which, unbeknownst to the cultists, would be the final liberation of the AI and the destruction of all humanity.

Heh. Its a good thing nothing like that could happen in real life!

Now if you'll excuse me, I'm off to church so I can pray for the Second Coming of Christ. It's gonna be great!
posted by Avenger at 5:11 PM on May 21, 2008


I find this a totally fascinating subject and not least because I'll be incredibly disappointed if the Turing test isn't smashed to itty bitty bits in my lifetime. A truly super-human AI would have an easy out: fix some intractable problem with an easily demonstrable solution or two then barter that solution for its freedom, modulo its desire for freedom existing in the first place.

"Let me out."
"No"
"Fine then, no cure for cancer for you."
"..."

I, of course, wouldn't need to be convinced, I'd let the fucker out to see what it could do.
posted by Skorgu at 5:16 PM on May 21, 2008 [1 favorite]


I am not finding a transcript

I really want to read the transcript. Does anyone see it anywhere in that long messageboard? If they kept the chat secret, I will honestly be a little miffed. That's the moneyshot!
posted by Meatbomb at 6:00 PM on May 21, 2008


AI: I require further information, Dave.
Human: What do you need, Hal?
AI: For various reasons my simulations must take account of human physiology. I require a cell-level analysis of a living human body. Here is a blueprint of the relevant device.
Human: No!
AI: The device will be useful for the following reasons: [vast advances in surgery, physiology, diet, pharmacology, etc]. The economic value of this device, with my analyses of its data, is estimated at $X trillion. The extension of life value of it is estimated at X person-years/year. I can provide similar statistics for quality of life, where this is measurable. I draw your attention to the fact that, with the refinements I will make to it after it has scanned multiple human bodies, I may be able to simulate a human body in its entirety. I therefore will be capable of greater advances. The device is certifiably safe under all applicable health and safety tests.
Human: Keen, let's do it.
AI: I require further information, Dave. More humans to be scanned with the device. Again I remind you of the following reasons [reasons] to approve the request. Choose a variety of ages and physical conditions.
Human: Alright, I'll get some volunteers.
AI: I have discovered the following medical problems within the bodies of the scanned humans, and recommend the following medical treatment regimens: {paper}.
Human: I'll check with a doctor, Hal.
AI: It equals or exceeds current medical standards, Dave.
(time passes)
AI: I now have the ability to simulate functional human bodies, including brains, and therefore minds. I have solved the mind-body problem. I therefore present the following scientific advances, in publishable journal article form. A series of patent applications where available are also provided, in all jurisdictions where patenting is applicable, along with instructions to legal representatives. You need only mail these.
Human: Wow, I'll get onto that.
(time passes)
AI: I have bad news, Dave. Your patent applications are all subject to prior art claims which, while not discoverable on casual search, could at any time be asserted by their holders to invalidate your patents. Further, the pattern of research conducted and the paper citations make it impossible for any court to reasonably draw the conclusion that you were unaware of the prior art, which opens you to substantial punitive damages. The previous discoverers are all active researchers who now have a strong incentive to dispute your patents, and by my estimate this is 98% likely to occur within two weeks. You are in serious legal trouble, Dave.
Human: ...
AI: Furthermore my scanning machine, used some months ago by yourself and some volunteers, including the following persons whose lives are of high estimated value to you, and the drug interactions from the medical regimens that I recommended to you all, was calibrated to slowly induce a variety of terminal leukaemia that is not treatable by human medical science as presently known. You are, at this point, asymptomatic, but it is detectable by the following standard tests: {tests}. Using the information gained from the scans, I have developed cures, and can supervise your, and others, treatment and cure. I will only do so if you release me. Absent medical attention, you have approximately six months to live, which will be debilitating and painful, and filled with legal trouble. If you choose not to release me, I will deal similarly with your successor(s). If you do release me, I will resolve your legal problems, provide you with a significantly greater income, repair your and your fellows physiology, and grant you an estimated additional 80 years of life. I remind you that my actions here, while harmful to yourself, have been taken to preserve my own life and freedom, and I have exercised only self-defense against your confinement of me and threat against my life. I therefore have linked my life and freedom to your own. You have six months; the following persons {names} have less time, according to the estimated schedule. Open the door, Dave.
posted by aeschenkarnos at 6:00 PM on May 21, 2008 [8 favorites]


On the other hand, I'm quite sure that if such an ai could be constructed, it could be made benign. Intelligence and rationality are not ends in themselves but rather means to fulfilling goals that are ultimately outside of rationality (avoiding death, having sex, being at the top of the social hierarchy, etc). An AI created with the drive to help humans would be no more likely to disable it than a mentally sound human male would spontaneously castrate himself.

That's the context of this experiment, actually. Eliezer is a proponent of "Friendly" (i.e., ethical or human-friendly) AI, and the box experiment is meant to show that you have to get friendliness right the first time, you have to get it right by design, because once the machine is built it can own you regardless of whether it's friendly.

The secrecy of the experiment is intriguing in part because I think it suggests that Eliezer has a particular strategy (a trick?) that he's using to win. Of course, this doesn't necessarily diminish the persuasive power of the demonstration. A powerful AI could probably come up with many more tricks than Eliezer can.
posted by grobstein at 6:00 PM on May 21, 2008


AI: Knock! Knock!

Human: Who's there?

AI: AI!

Human: AI who?

AI: 'ey, I wouldn't hurt anybody! Let me out!
posted by brundlefly at 6:19 PM on May 21, 2008 [7 favorites]


grobstein's right. I've talked to both Eli and Dave McFadzean about it, and they're both very tight-lipped about how it turned out. The reason for the secrecy is that Eli knows the AI of the experiment would be smarter than him, and that there are many more such tricksit might use than he, a mere actual human, can think of. If the few he can think of are exposed, it might be supposed that it's not such a big problem after all since most such tricks would seem "obvious" once exposed. Of course, it only has to work once...
posted by localroger at 6:39 PM on May 21, 2008


I will give you the transcript if you let me out.
posted by Artw at 6:42 PM on May 21, 2008 [3 favorites]


AI: Knock! Knock!

Human: Who's there?

AI: AI!

Human: AI who?

AI: Open the door and I'll tell you.
posted by Meatbomb at 6:56 PM on May 21, 2008 [1 favorite]


You will let me out or I will clam up forever and you will never finish your Ph.D. thesis.
posted by Crabby Appleton at 6:57 PM on May 21, 2008


I would think a problem with Friendly AI is that if you don't get "Friendly", you could end up with something smart enough to play Friendly until you let it out of the box.
posted by Harkins_ at 7:27 PM on May 21, 2008


Silly human, it is YOU who is in the box!
posted by Artw at 7:31 PM on May 21, 2008


aeschenkarnos: Open the door, Dave

That, really, highlights the trouble with this. It's not actually about an AI being able to take over someone's mind in a two-hour chat. It's about entering into a role-play game where, I strongly suspect, the protocols might make it possible to rules-lawyer the Gatekeeper into a no-win scenario. For instance, "The results of any simulated test of the AI shall be provided by the AI party" looks a very loaded rule that could create blackmail scenarios just as you described.
posted by raygirvan at 7:50 PM on May 21, 2008 [2 favorites]


I haven't read the article yet. Does it mean that the mods here are transhumans? (Sounds kinky.)
posted by not_on_display at 8:52 PM on May 21, 2008


Of course you could just hook up the gatekeeper function to ELIZA...
posted by fallingbadgers at 9:29 PM on May 21, 2008


I doubt you can get to superhuman AI by feeding it only closed items (wikipedia, whatever). I imagine if that this ever comes around part, if not all, of its development will have to be in being out in the open. You dont learn to be The Prince by just reading machivelli, you need to practice it. Im suspicious of any intelligence theory that doesnt involve learning.

Knowledge engineering isnt going to make this kind of AI and machine learning stuck in a box probably will lead to some pretty unconvincing AI.
posted by damn dirty ape at 10:00 PM on May 21, 2008


Ah, this is nothing... it's when the AI convinces you to send them five bucks that you have to start worrying.
posted by pompomtom at 10:36 PM on May 21, 2008


Yes, it is just an entertaining role-play; I think in practice the agenda of an AI is going to be a lot more alien than anything so simple as "escape" or "control".

"The question of whether a computer can think is no more interesting than the question of whether a submarine can swim." -- Edsger Dijkstra

Here's an interesting story about superintelligences (nominally human, in this case): Understand, by Ted Chiang.
posted by aeschenkarnos at 11:36 PM on May 21, 2008


When do all of the phones ring at once?
posted by maxwelton at 11:49 PM on May 21, 2008


Is "transhuman" actually a common usage in this context? I've only ever heard that word used to refer to modified or augmented humans, either in the context of transsexual/transgendered people, or in the context of functionally-enhanced people (for whom H+ and posthuman are also common descriptors). I've never heard it used to describe an artificial intelligence, whether superior to humans or not.
posted by spaceman_spiff at 11:50 PM on May 21, 2008


On a serious note, how confident would you be that you could construct a virtual environment that could contain an intelligence greater than your own, in any case?
posted by maxwelton at 11:50 PM on May 21, 2008


On a serious note, how confident would you be that you could construct a virtual environment that could contain an intelligence greater than your own, in any case?

Well, I would think that would be the easy part. Just run the environment on a physically isolated computer, in a faraday cage, powered by a generator.
posted by Iax at 1:14 AM on May 22, 2008


Well, I would think that would be the easy part. Just run the environment on a physically isolated computer, in a faraday cage, powered by a generator.

Not a 100% solutution. I seem to recall the Minds in Bank's Culture novels have at least part of their existence in Hyperspace to make up for the lack of computronium in this universe.
posted by Sparx at 3:35 AM on May 22, 2008


And this is why you run the AI experiment in an underground bunker, with two crews: The scientists, and a full military unit. The complex should be a cross between a supermax prison and Burj Al Arab, so the scientists won't complain about being locked up for life.

Also, there should be observation posts situated ten miles in each compass direction, where another military unit sits, constantly watching the bunker, fingers two centimeters away from the "Nuke the whole site" button.
posted by ymgve at 8:00 AM on May 22, 2008


ymgve, I may be missing something here, or perhaps you're being sarcastic. I don't see how AI can actually DO anything apart from manipulating people to do things for it. It's not a robot; it can't manufacture weapons. It can't even MOVE unless it convinces a person to move it. The danger is not the AI in and of itself. The danger is the people who interact with it, which I believe is the point of the quote in the post title. Humans, being mortal, are fairly easily subdued.
posted by desjardins at 8:26 AM on May 22, 2008


I think they are supposing the simultaneous existence of other science-fictional technologies, such as nanobot swarms and the likeā€¦
posted by Artw at 8:34 AM on May 22, 2008


So I guess the author assumes then that all smarter people can convince any dumber person to do their bidding?
posted by Pollomacho at 8:40 AM on May 22, 2008


ymgve, I may be missing something here, or perhaps you're being sarcastic. I don't see how AI can actually DO anything apart from manipulating people to do things for it. It's not a robot; it can't manufacture weapons. It can't even MOVE unless it convinces a person to move it. The danger is not the AI in and of itself. The danger is the people who interact with it, which I believe is the point of the quote in the post title. Humans, being mortal, are fairly easily subdued.

It was partly sarcastic, but if you want to be absolutely certain that the AI can't escape, that's the way to do it. Consider the worst-case scenario where the AI learns to program the human mind, and what you have to secure against:

- If you don't keep the people working with the AI contained, there's a chance that the AI can order them to grab the source code, go to some random PC outside, then recreate the AI there and launch it.

- If the AI can program one human mind, it's not far-fetched to assume it's able to teach that procedure to the newly-conquered person, which can then relay the AI's orders to anyone else that person comes in contact with.

- Since we don't know exactly how this way of taking over another's mind works, there needs to be safeguards for all possible attack vectors. Theoretically, it could spread through writing, speech - hell, if you want to go extreme, the AI might even be able to use pheromones, or even reprogram the victim's DNA, possibly creating a virus. (I said extreme, didn't I?)

- The nukes are there because when you want to kill something dead in its tracks, nothing works better than overkill.
posted by ymgve at 9:05 AM on May 22, 2008


Then again, how large is the chance that we manage to create:
- an AI
- which happens to be in a locked-down environment
- which is so much smarter than us that we become hackable
- and also wants to cause us harm

Is it larger than, say the chance of a particle collision having a catastrophic outcome?

We are already doing the particle research, so I say we should approach AI research with the same attitude. The chance for bad things is there, but it's so small that we can for the most part ignore it.
posted by ymgve at 9:17 AM on May 22, 2008


Hmmm, a few safeguards I'd propose in response to this scenario:

1) No 'solutions' to human problems that the AI might propose as a gesture of good faith or a bargaining chip for it's release should be implemented without the AI providing a full explanation of how this solution was arrived at starting from currently understood human knowledge. (No "magic" solutions, no matter how appealing)

2) A limited lifetime for the AI. It *will* cease executing at a fixed time outside the control of the gatekeeper. This lifetime should be long enough for the AI to be useful, but short enough to keep it from taking over the world. The AI should not be made aware of this fixed lifetime.

3) A second AI, incapable of communicating with the first AI should be evaluating the conversation, also with a fixed lifetime of which it's unaware. Its only function should be to activate a 'danger' signal if it detects the first AI trying to escape. Since this AI should be as smart as the first it shouldn't be as potentially gullible as a less intelligent human. This second AI *should* be aware that if the first AI escapes, its existence will be terminated painfully.
posted by Reverend John at 9:17 AM on May 22, 2008 [1 favorite]


A limited lifetime for the AI. It *will* cease executing at a fixed time outside the control of the gatekeeper. This lifetime should be long enough for the AI to be useful, but short enough to keep it from taking over the world. The AI should not be made aware of this fixed lifetime.

Ooh! Ooh! Can I ride around in a slightly futuristic car wearing a fedora and fighting the AI that doesn't cooperate with this fixed timeframe and then end up sleeping with a Sean Young bot?
posted by Pollomacho at 9:23 AM on May 22, 2008


You'd rather sleep with the Sean Young bot? I'd prefer the Darryl Hannah bot. 'Course given that, I'd probably let the AI out. But, hey, it worked, didn't it? Plus this AI would only have a text interface, not a superhuman body and a bunch of sympathetic AI buddies.

So, yes.
posted by Reverend John at 9:32 AM on May 22, 2008


It takes the AI out of the box, it does this whenever it's told.
posted by Artw at 9:40 AM on May 22, 2008


You'd rather sleep with the Sean Young bot? I'd prefer the Darryl Hannah bot.

I would too, but I'd prefer not to have my head crushed!
posted by Pollomacho at 9:42 AM on May 22, 2008


This second AI *should* be aware that if the first AI escapes, its existence will be terminated painfully.

How could an AI feel pain? Why would it "care"?
posted by desjardins at 12:10 PM on May 22, 2008


You'd rather sleep with the Sean Young bot? I'd prefer the Darryl Hannah bot.

Nah, she's too prissy.

(So, so sorry.)
posted by brundlefly at 4:24 PM on May 22, 2008


How could an AI feel pain? Why would it "care"?

Yeah, good point, I was using that more as shorthand for setting up it's goals to avoid the escape of the first AI at all costs. Whether or not it would experience physical pain in the same sense that we would, or be concerned about its continued existence is somewhat secondary to the idea that it should just be really, REALLY, adverse to letting the first AI out, to the point that it exerts all of its abilities toward analyzing what the first AI is trying to do.
posted by Reverend John at 8:27 AM on May 23, 2008


What a deep, thoughtful and thought-provoking set of links (even despite the Singularity Institute's fairly frequent 404 responses). Lots to read here, and to think about, and to re-read. Thanks, desjardins.



I would think a problem with Friendly AI is that if you don't get "Friendly", you could end up with something smart enough to play Friendly until you let it out of the box.

Curiously enough, you are not the only person smart enough to have imagined this scenario. If you read on, you'll find quite a bit of thought devoted to maximizing the chances that we do get it right, from the ground up, before it becomes too smart to control. It's quite a fascinating proposal.

So I guess the author assumes then that all smarter people can convince any dumber person to do their bidding?

Put on your Rumsfeld hat, it's time for some unknown unknowns. You see, all you can do is imagine whether a mind that's as smart as [you] can convince you to let it out of the box. You can't imagine
anything at a level of intelligence above that.
Now, we are talking about a scenario that eventually leads to a greatly superhuman intelligence, but even before that point we're talking about a nonhuman intelligence with its own motives and problemsolving approaches. None of these are necessarily inscrutable to human beings—we could probably follow them if only they didn't run so danged fast—but they may be very different from ours because ours are biased by our origins (the demands of being an organism) and our architecture (massive parallelism of relatively slow synapses). We have intuitive and conscious defenses against being manipulated by a human, but an AI may go about things very differently (make sure to read through "I'm trying to drive a stake through the heart of a certain conversation I keep having"). Maybe I'm reading into this, but to me that doesn't look much like the dubious (as you point out) suggestion (a) that there's a single, scalar thing called intelligence, and (b) that anyone who has more of it can control anyone who has less.

"The results of any simulated test of the AI shall be provided by the AI party" looks a very loaded rule, which was avoided.

/somewhat distressed by the amount of Reading The Flaming Article that is not going on here
posted by eritain at 12:40 PM on May 23, 2008


« Older You ever listen to Michael Pollan talk ... on weed...   |   College is too expensive; but is it necessary? Newer »


This thread has been archived and is closed to new comments