Jenning v. Rutter v. Watson

January 13, 2011 4:40 PM

The Watson Artificial Intelligence system will take on Jeopardy champions Ken Jennings and Brad Rutter in February (previously). Here's a sneak preview of a warm-up round between the contestants.

The video also shows Watson's "thought process": the three answers most likely to be correct, and their probabilities.

Below the contest video there's also a fascinating interview with one of the Watson developers, if you're into algorithms and context parsing.

posted by auto-correct (35 comments total)

Here's a more detailed article about the event if you don't want to watch the video.
posted by auto-correct at 4:44 PM on January 13, 2011

Epony... you know.
posted by Splunge at 4:50 PM on January 13, 2011 [1 favorite]

I am kind of unnaturally excited about this match. I love Ken Jennings, and Jeopardy, and I used to work at IBM, so I feel a tiny bit of pride about their whole involvement, even though I had nothing to do with it, and it's like nerdiness to the 10th power.
posted by jacquilynne at 4:55 PM on January 13, 2011

11:46AM IBM: Watson does not have emotions, but Watson knows that humans have emotions.

Tugs on collar nervously, wipes sweat off of brow, "Yeeesh, sounds like my ex-wife!"
posted by codacorolla at 5:10 PM on January 13, 2011 [7 favorites]

I'm not really fond of this whole man vs. machine thing. I mean, you've got a machine that doesn't do much besides play Jeopardy, and you've got a human like Ken Jennings, who does many other things besides play Jeopardy.

Ken Jennings is a Mormon who grew up in Seoul, South Korea and Singapore. He graduated with a degree in Computer Science and English and has two children. He lives just outside Seattle and is a comic and movie buff.

And besides all that, he's a trivia machine in his own right. That's way more interesting to me... that a person can do many things.

If I had to choose between having a beer with Watson or having a beer with Ken Jennings, I'd definitely choose Jennings.

Er, I forgot. He's a Mormon.
posted by twoleftfeet at 5:20 PM on January 13, 2011 [4 favorites]

And besides all that, he's a trivia machine in his own right. That's way more interesting to me... that a person can do many things.

If I had to choose between having a beer with Watson or having a beer with Ken Jennings, I'd definitely choose Jennings.

Er, I forgot. He's a Mormon.

This is a public showcase of technology, Natural Language Processing, that could literally change the way that we live life on Earth. In the same way that the microprocessor, the Internet changed life, and the printing press before them, but maybe even more so, because these inventions are a pooling of human intelligence, and AI is potentially a multiplication of it.

Maybe I've drunk the koolaid, but the fact that you can feed a computer a plain text file and have it understand it (so much as you can say that a computer "understands" anything) and then provide a correct answer... that's mind blowing, at least to me. The IBM guys are right, the health implications alone are staggering, much less what happens when they start leasing processor cycles to hedge funds.
posted by codacorolla at 5:26 PM on January 13, 2011 [4 favorites]

that's mind blowing

It's not that big. Jeopardy is a limited domain. Computers have won in limited domains for years... you won't beat a talented computer at checkers, for instance. For that matter, I won't compete against other humans in bar trivia if they're allowed to use their smart phones. The technological advantage, albeit minor, is still enough whoop my ass.

I propose the Metafilter Turing Test. It's not enough for a machine to give simple answers to common questions. The machine has to make enough posts and comments with enough nuanced opinion and humor to pass as a typical MeFite.

Then you'll have something. I'll even pony up the five bucks for Watson's membership fee.
posted by twoleftfeet at 5:33 PM on January 13, 2011 [4 favorites]

twoleftfeet: I propose the Metafilter Turing Test. It's not enough for a machine to give simple answers to common questions. The machine has to make enough posts and comments with enough nuanced opinion and humor to pass as a typical MeFite.

I've always assumed clavdivs was one of cortex's 'experiments' gone feral.
posted by Doofus Magoo at 5:50 PM on January 13, 2011 [7 favorites]

Health, investing, and tech support are limited domains too (to certain extents, and then you have human handlers and specialists when the human fails). What's the difference from a nuanced, punny question on Jeopary and a nuanced, individualized health diagnosis in a clinic?

Think of the middle level jobs (nurses, brokers, bankers, computer janitors) that widespread implementation like this could create. Think about a world where every lab in the world can give their local machine its dataset and then constantly pose questions to it, occasionally striking pay dirt within their limited domain.

AI being good doesn't cheapen the accomplishments of humanity, AI is the accomplishment of humanity. Maybe, anyway, we'll see what further uses this thing is put to.
posted by codacorolla at 5:53 PM on January 13, 2011 [2 favorites]

Think of the middle level jobs (nurses, brokers, bankers, computer janitors) that widespread implementation like this could create

Which is supposed to say "...that widespread implementation would destroy." I think the control team needs to look at my natural language quantum processo- I mean, not enough coffee.
posted by codacorolla at 5:54 PM on January 13, 2011

I thought you actually MEANT "create", and figured that, for instance, if you dropped a skilled medical "Dr. Greg House" AI in a remote town, now you have a need for a medical staff for it, where previously there was no doctor for 50 miles.
posted by scrowdid at 6:04 PM on January 13, 2011

I'm sort of disappointed to discover that it's being fed the answers in plain text and it's not deciphering the spoken question. Does anyone know if it's getting the plain text question at the same pace as the spoken question? Watching the avatar, it certainly seemed to grow more confident as the question completed, but if it's getting the whole thing up front while the Trebek-Clone is reading out the question it has a few extra seconds to cogitate that the humans don't.
posted by Kyol at 6:13 PM on January 13, 2011 [1 favorite]

I can't wait for the Burt Reynolds easter egg that will cause the computer to intermittently change its name to Turd Ferguson.
posted by dr_dank at 7:28 PM on January 13, 2011 [2 favorites]

dr_dank: From your mouth to god's ears.
posted by John of Michigan at 7:31 PM on January 13, 2011

I too am super excited for this match. I love Alex's promo's for it too, where he seems amused by albeit disinterested in the whole endeavor.
posted by milestogo at 7:36 PM on January 13, 2011 [1 favorite]

Ken is totally going to beat those two robots (just as long as there are no categories about firearms)! Go Ken!!!!
posted by Mael Oui at 7:49 PM on January 13, 2011

I recently started working for IBM, and we got to "meet" Watson during my "Welcome to IBM" day. I'm really pumped for seeing him in action.
posted by BZArcher at 8:15 PM on January 13, 2011

Ken Jennings was a lil baby
Sittin' on his mama's knee,
Said: "It's a stunt episode of Jeopardy!
Gonna cause the death of me.
Lord, Lord!
Gonna cause the death of me."

Alex says to Ken Jennings,
"Gonna bring me a 'puter 'round,
Gonna take that 'puter out on the job
Gonna whop that trivia down.
Lord, Lord!
Gonna whop that trivia down."

Ken Jennings told ol' Alex
(Lightnin' was in his eye)
"Alex, bet your last red cent on me,
For I'll beat it in the Final or die;
Lord, Lord!
I'll beat it in the Final or die."

Lights shine hot and burnin',
Wer'n't no AC a-tall,
Sweat ran down like water down a hill,
That day Ken Jennings let his answers fall.
Lord, Lord!
When Ken Jennings let his answers fall.

Ken Jennings started at the right hand
The 'puter started on the left.
"Before I let this 'puter beat me down
I'll answer my fool self to death.
Lord, Lord!
I'll answer my fool self to death."

IBM told Ken Jennings,
"Mormon, damn your soul!
You might beat this natural language processor o' mine
When the clue screens turn to gold.
Lord, Lord!
When those blue screens turn to gold."

Oh, Johnny Gilbert said to Ken Jennings,
"I believe the studio's cavin' in!"
Ken Jennings said to Johnny, oh my!
"Ain't nothin' but my answers blowin' wind.
Lord, Lord!
Ain't nothin' but my answers blowin' wind."

Ken Jennings told Brad Rutter
"Rutter, you better pray,
For if my thumb misses this here buzzer,
Tomorrow'll be your burying day,
Lord, Lord!
Tomorrow'll be your burying day."

Ken Jennings told IBM,
"Look yonder what I see-
Watson's choked, your 'puter's done broke,
And you can't answer trivia like me.
Lord, Lord!
You just can't answer trivia like me."

The man who programmed the 'puter
Though he was mighty fine.
Ken Jennings took in one-fifty grand
And the 'puter only made nine.
Lord, Lord!
That 'puter only made nine.

Ken Jennings was hammerin' on his buzzer,
And the button was strikin' fire.
He pressed so hard that he broke his poor heart,
And he lied down his buzzer and died,
Lord, Lord!
He lied down his buzzer and died.

They took Ken Jennings to the graveyard,
An' they buried him in the sand,
An' every school bus come roarin' by,
Says, "There lays a trivia-answerin' man,
Lord, Lord!
There lays a trivia-answerin' man."
posted by Iridic at 9:16 PM on January 13, 2011 [9 favorites]

The year before we married, my wife wrote for a mid-sized newspaper with a wonderful, brilliant nerd of a woman who qualified for Jeopardy. She was unfortunate enough to be scheduled to appear when Ken Jennings was mid-streak.

This woman now belongs to a Facebook group made up of people who lost to Ken Jennings.

I only bring this up to point out that there is a Facebook group of people who lost to Ken Jennings, which makes me smile.
posted by middleclasstool at 9:19 PM on January 13, 2011 [7 favorites]

Interesting that after missing a few questions to the two humans, Watson takes over again on the New York mayor one - early in the answer, '52nd Mayor of NYC' is mentioned, which is the kind of thing a computer would instantly be able to compute without probability issues.
posted by mannequito at 9:51 PM on January 13, 2011

Winning at Jeopardy seems to often come down to how fast one can press the buzzer. It would be hard to beat a machine with our meat reflexes.
posted by Thoughtcrime at 9:59 PM on January 13, 2011 [1 favorite]

Fascinating. On the one hand, I think this harbinges a vast coming wave of AI that will probably excel in wide swathes of previously human-only achievement.

On the other hand, it is ultimately still a game whose input is clearly given and whose rules and structure are defined tightly for the computer. There is a clear right answer. Deciding what's valuable, interesting, beautiful, or relevant seems to be the key human-only capacity. At least so far!
posted by shivohum at 10:21 PM on January 13, 2011

Eh. Call me when Watson wins at Plinko.

Why don't you take a spin at the Big Wheel, Watson? Just has to go around once, closest to a dollar gets a spot in the Showcase Showdown. C'mon, give 'er a spin. You might hit a dollar. No? Watson? What's the matter, Watson? NO ARMS?
posted by bicyclefish at 10:57 PM on January 13, 2011 [4 favorites]

Winning at Jeopardy seems to often come down to how fast one can press the buzzer. It would be hard to beat a machine with our meat reflexes.

Actually there's a guy behind the scenes that manually switches the buzzers on after Alex asks the question. Ken Jennings talks about it here. If Watson is getting the answers too fast, it might cripple its chances of winning.
posted by girih knot at 11:29 PM on January 13, 2011

There is something very poetic about an anthropomorphized machine asking "What is purple?" It strikes me as the kind of question a computer would never really be able to answer properly.

ok, it would be easy enough to talk about red+blue and specific energy wavelengths, but does that tell you, the average human, what purple is?
posted by Eumachia L F at 12:27 AM on January 14, 2011 [1 favorite]

I can't wait for the Burt Reynolds easter egg that will cause the computer to intermittently change its name to Turd Ferguson.

In this same vein, I'd also be very happy with a Sean Connery voice/personality plugin that would select "The Penis Mightier for $100" and call Alex Trebek's mother a whore.
posted by Strange Interlude at 2:00 AM on January 14, 2011 [1 favorite]

The video also shows Watson's "thought process": the three answers most likely to be correct, and their probabilities.

I saw the top correct answer and a Puffs tissue ad. Sigh.
posted by Lucinda at 5:34 AM on January 14, 2011 [1 favorite]

That was amazing. Although I have to wonder at the psychology of Watson's "face" being a swirly globe. Why not a happy smiley or something?
posted by DU at 5:37 AM on January 14, 2011

I suggest a big smiley face and the voice of Dave Foley.
posted by rokusan at 6:38 AM on January 14, 2011 [3 favorites]

I've always assumed clavdivs was one of cortex's 'experiments' gone feral.

Everything makes so much more sense now.
posted by thsmchnekllsfascists at 7:50 AM on January 14, 2011 [1 favorite]

Actually there's a guy behind the scenes that manually switches the buzzers on after Alex asks the question. Ken Jennings talks about it here. If Watson is getting the answers too fast, it might cripple its chances of winning.

Yeah, the board (in the studio) lights up when you're allowed to buzz in. If you buzz in too early, you're locked out for a short period. Though if you actually wait for the light to come on, you're usually too late, due to the time it takes for you to react and physically push the button. You have to try to time it to when Trebek finishes reading. Sadly, I sucked at it.

I know Watson is using a physical hand to buzz in, so I'm assuming it also has to do visual processing to see when it's allowed to buzz in and then buzz.
posted by kmz at 8:01 AM on January 14, 2011 [2 favorites]

It's not gonna pass the Turing Test until it's able to throw in a few "Suck it, Trebek"s.
posted by symbioid at 8:47 AM on January 14, 2011 [1 favorite]

Winning at Jeopardy seems to often come down to how fast one can press the buzzer. It would be hard to beat a machine with our meat reflexes.

I recall an interview with Ken where he stated that during his run he got to the point (or started there) where he assumed he knew every answer and would just start hitting the crap out of the button as soon as he could almost before he parsed the question. If Watson is "waiting" to get a solid answer it might even the playing field a bit.
posted by bitdamaged at 10:49 AM on January 14, 2011

I thought the Watson developers were being a little nice.

A common strategy in Jeopardy is to go through all of the questions in a given category before moving to the next category. This is because our minds have got "warmed up" and our connections for questions in that category are primed. Jumping all over the board from topic to topic is harder for the human mind. However, this is likely not true for Watson, which presumably is stateless in that the previous set of questions do not affect its ability to answer the current question. So if the developers wanted to be evil, they'd have Watson jump all over the board, constantly changing the topic and difficulty (question value).
posted by LoopyG at 1:31 PM on January 14, 2011 [1 favorite]

A contestant from the practice round posted about it on HuffPost yesterday.
posted by bearwife at 9:28 AM on February 10, 2011