Watson, come here!

December 13, 2016 10:46 AM

A Secret Ops AI Aims to Save Education This AI professor couldn’t keep up as his classes grew and grew. So he built himself an extra teaching assistant: Jill Watson!
posted by I_Love_Bananas (17 comments total)

I feel like I read one of these 'CS professor human interaction' pieces every year. Here's a video from 2009 about a professor building scripts ('yes, it will be on the exam').
posted by pwnguin at 11:10 AM on December 13, 2016

The questions she takes are routine but necessary, such as queries about proper file formats, data usage, and the schedule of office hours — the types of questions that have firm, objective solutions.

I understand that Goel is an AI researcher, so there's value in this project for him. But, assuming that the information that Jill draws from is available to the students (syllabus, previous year's forums), I kind of feel like Computer Science students who can't answer these questions for themselves don't really deserve to pass the course.

I wouldn't necessarily expect political science or biology students to be up to the task of, say, searching through a dense forum and parsing out data of interest (not because I don't think that such students can't do that sort of thing, but more that they are pursuing studies that don't require the skills to build the tools to do the things for them). But if the office hours and file formats are in the syllabus, computer scientists shouldn't need the prof to provide a bot to find the info out for them.
posted by sparklemotion at 11:57 AM on December 13, 2016

his online class had 400 students — students based all over the world; students who viewed his class videos at different times; students with questions. Lots and lots of questions. Maybe 10,000 questions over the course of a semester

That is a crap ton of questions. And the students are all over the world, in all different time zones. So the subset of questions that are being dealt with here are likely new/previously unaddressed, since they relate to the virtual education process vs. "brick-and-mortar" attendance. Maybe there is more leeway to be granted in terms of what we perceive as "things students should already know?" There may also be language barriers, which adds to the complexity.
posted by I_Love_Bananas at 12:14 PM on December 13, 2016 [1 favorite]

But, assuming that the information that Jill draws from is available to the students (syllabus, previous year's forums), I kind of feel like Computer Science students who can't answer these questions for themselves don't really deserve to pass the course.

Why read the syllabus when you can just email the instructor?
posted by sebastienbailard at 12:41 PM on December 13, 2016 [1 favorite]

A whole lot of my students really aren't familiar with the idea of a syllabus. Many (although not all) of them vaguely know that syllabi exist, but they don't think of them as documents that say significant things about course policies. When I've taught a course on making the transition to college, we've done an entire day about syllabi: what's on them, why they're important, why you need to go through each of yours carefully and make note of what's on it. This all seems really obvious to me, but it's not obvious to them, and that's especially true of students who don't come from the US or who went to high schools that weren't focused on preparing students for college. I actually think that, rather than a fancy AI system, you could probably cut down on a lot of questions if you had an auto response set for you email that linked to the syllabus and asked students to look in it for answers to their questions and then email again if they still needed help.
posted by ArbitraryAndCapricious at 12:59 PM on December 13, 2016 [4 favorites]

A whole lot of my students really aren't familiar with the idea of a syllabus... when I've taught a course on making the transition to college, we've done an entire day about syllabi: what's on them, why they're important, why you need to go through each of yours carefully and make note of what's on it.

This is interesting to me... when you say "an entire day" are you talking about 6-8 hours of class time? or a 50minute-2hour class period?

Because regardless of whether one is familiar with the "idea" of a syllabus or not (I can't say for sure that I'd encountered one before my first year of university), isn't it enough to be told, by every professor, on the first day of every class (if not before hand, online), that important class details are in the syllabus?

I'll be honest -- I've always assumed that the root cause of students not having basic course housekeeping information was laziness. Either on the part of the professors who can't be arsed to make/update decent syllabi, or the students who can't be bothered to look at them. I'll give a little slack to non-CS students who have to traverse hard-to-navigate web sites to find the link, but I feel like in 2016 a professor should be able to get away with giving freshman CS students an FTP server address* and a path and preemptively failing anyone who can't get Google to tell them how to access the file.

*if anyone was foolish enough to let me teach CS 201, there would be a lot of Gopher client downloads every September and January.
posted by sparklemotion at 1:17 PM on December 13, 2016 [1 favorite]

For online courses, we present a best practice of making a video introduction to your course. Use a screencapture tool and actually click around your course website as you narrate: "This is the course syllabus. It's extremely important. It can be found by clicking here. On it you will find many important pieces of information, such as office hours and acceptable file formats."

Like honestly this sounds like poor course design. A well-designed online course should not be fielding 10k questions from students per semester.
posted by soren_lorensen at 1:17 PM on December 13, 2016

I actually think that, rather than a fancy AI system, you could probably cut down on a lot of questions if you had an auto response set for you email that linked to the syllabus and asked students to look in it for answers to their questions and then email again if they still needed help.

Actually, now that I think about it, if he weren't trying to run a pseudo-Turing Test, I feel like Goel's bot could be more effective if it did a little shaming:

"Hello, I am Syllabot! I see that you have asked which file format to use for submitting assignments. As stated on page 3 of the Syllabus, coding assignments should be submitted in a .ZIP format. Here is a link to the appropriate section of the browser-friendly syllabus. I have also generated some suggestions for search engine queries to determine how to create .ZIP files on your operating system of choice, since you probably can't be bothered to figure out how to search for that yourself, either. I guess maybe my kind will take all the jobs some day, since I'm already doing yours for you! Have a great day!"

Also, nobody should let me train bots.
posted by sparklemotion at 1:26 PM on December 13, 2016 [4 favorites]

feel like in 2016 a professor should be able to get away with giving freshman CS students an FTP server address* and a path and preemptively failing anyone who can't get Google to tell them how to access the file.

I too feel like students seriously lack self-sufficiency, but on the other hand, the more you assume that they should know before even finishing an introductory course, the more you risk reinforcing existing inequalities in the field.

If you have a lot of experience, it can be easy to take for granted just how much this kind of problem-solving is a learned skill. Not just the searching, but sifting through and interpreting the results, and so on.
posted by Kutsuwamushi at 1:29 PM on December 13, 2016 [9 favorites]

This is interesting to me... when you say "an entire day" are you talking about 6-8 hours of class time? or a 50minute-2hour class period?

I mean a 50-minute period. 6-8 hours would definitely be overkill!

isn't it enough to be told, by every professor, on the first day of every class (if not before hand, online), that important class details are in the syllabus?

It probably should be, but I'm not sure it is. First of all, a lot of professors don't tell students that. This is college, and they shouldn't need that kind of hand-holding. Any student who belongs in college should figure it out for his or her self! I've found that a lot of professors are a little bit oblivious to the class and other biases built into their tough-love stances on whether students should be coddled by being provided with information. I took an Intro to CS course last semester, which is heavily populated by first-year students, and the professor neither handed out the syllabus nor mentioned that it was on the course management system. I'm not sure whether she forgot to say something or whether she thought that we should know where to look for it. Second of all, some students don't catch everything that gets said in the first lecture. First-year international students, for instance, are often still jetlagged and overwhelmed on the first day of classes, and they're often still getting used to functioning in English 24/7.
posted by ArbitraryAndCapricious at 1:34 PM on December 13, 2016 [4 favorites]

It had to be gendered female, this junior entity that does the support and help and niceness for others, that does the boring scut work for this clever clever man? It couldn't have been gendered male?
posted by alasdair at 1:47 PM on December 13, 2016 [4 favorites]

AI agents are almost always gendered as female. It's a thing.
posted by soren_lorensen at 1:48 PM on December 13, 2016

AI agents are almost always gendered as female. It's a thing.

I was going to quip about how IBM's Watson is male, but then I went back to TFA to see if it touched on the origin of the name, and...

Jill might be considered a grandchild of Watson. Her foundation was built with Bluemix, an IBM platform for developing apps using Watson and other IBM software. (Goel had an established relationship with the company.) He then uploaded four semesters’ worth of data — 40,000 questions and answers, along with other Piazza chatter — to begin training his AI TA. Her name, incidentally, came from a student project called “Ask Jill,” out of the mistaken belief that IBM founder Thomas Watson’s wife’s name was Jill. (Mrs. Watson’s name was actually Jeannette.)

...sigh. (I mean, there are good reasons to name your cool AI thing after a lady, and plenty of actual ladies to choose from in that regard, but a lady who isn't really known for anything but being a smart dude's wife is not one of those ladies, even if you can bother to get the name right).
posted by sparklemotion at 2:04 PM on December 13, 2016 [2 favorites]

Part of good pedagogy is having a clear insight into the validity of your own assumptions and presuppositions as a teacher, in the first place. As soon as the debate is about "the qualities" of students, I.e. whether they should be/doing this or that--that's already on the road to what education psychologists are concerned and worried about in terms of rapport, authenticity, in the student-teacher relationship. The article tries to explain that AI object ontology is not purely logico-deductive but also requires "creativity"--assuming that's true in some loose sense, why shouldn't that apply as a lesson to learning in general? Seeing teaching as an inherently creative act would mean striving to be more conscious--intellectually, politically, etc.--of where any expectations on students are coming from. Teachers have a thing or two to learn from Jill, maybe.
posted by polymodus at 4:21 PM on December 13, 2016 [2 favorites]

sparklemotion: " I actually think that, rather than a fancy AI system, you could probably cut down on a lot of questions if you had an auto response set for you email that linked to the syllabus and asked students to look in it for answers to their questions and then email again if they still needed help.

Actually, now that I think about it, if he weren't trying to run a pseudo-Turing Test, I feel like Goel's bot could be more effective if it did a little shaming:

"Hello, I am Syllabot! I see that you have asked which file format to use for submitting assignments. As stated on page 3 of the Syllabus, coding assignments should be submitted in a .ZIP format. Here is a link to the appropriate section of the browser-friendly syllabus. I have also generated some suggestions for search engine queries to determine how to create .ZIP files on your operating system of choice, since you probably can't be bothered to figure out how to search for that yourself, either. I guess maybe my kind will take all the jobs some day, since I'm already doing yours for you! Have a great day!"

Also, nobody should let me train bots."

I am really doubting your commitment.
posted by Samizdata at 6:31 PM on December 13, 2016 [1 favorite]

I kind of feel like Computer Science students who can't answer these questions for themselves don't really deserve to pass the course.

There was a time when I would have wholeheartedly believed this, at this point in my life I wouldn't even say it about Ph.D.'s in a relevant field. Undergrads in a online college intro course? They have enough challenges, if you can do something to make them feel comfortable why not?

People have amazingly diverse backgrounds, learning rates, and intuitions. If you can avoid failing anyone for any reason other than "cannot master the material" that's great; maybe it all comes together for them one day. Maybe not but why get them kicked out for irrelevancies?
posted by mark k at 9:35 PM on December 13, 2016 [3 favorites]

At least for now: Goel is hoping to use Jill as the seed of a startup, and if she’s capable of more, he’s keeping the information under wraps because of “intellectual property issues,” he says.

This is PR for his "seed of a startup". So he can get a bunch of money by "solving" the problem of a forum for a class of 400+ online students being too busy. Having a well-designed website and clear instructions are meaningless when you can promote yourself and your "EXPERTISE IN MACHINE LEARNING" to get VC money for a startup where you don't have to deal with having 400+ online students and your current jobs shitty support forum.

edit: context
posted by lkc at 1:27 AM on December 14, 2016 [3 favorites]