Using Game Theory to Teach Game Theory

May 6, 2013 11:34 AM

Peter Nonacs, a professor at UCLA, let his students cheat on an exam he called "impossibly difficult", saying, "Let’s see what you can accomplish when you have no restrictions."
posted by boo_radley (32 comments total)

It's not cheating if it's allowed, yeah?

All of my upper-level history classes had final exams that were just papers we had to turn in within 24 hours of picking up the exam. They were open-book, and while we weren't allowed to collaborate with anyone during that time period, it was expected that we would have been discussing the readings and lectures and our own papers with classmates throughout the term. One of my professors was explicit in her discussion of this - writing itself is a solitary activity, but everything leading up to it is (or should be) collaborative.
posted by rtha at 11:47 AM on May 6, 2013 [5 favorites]

How is something considered cheating if it's explicitly allowed?
posted by dfriedman at 11:53 AM on May 6, 2013 [2 favorites]

Where by "cheating" he apparently means "collaborating and using reference material"? I mean, it's great to make course evaluation less artificial, less rote, and more oriented toward the genuine intellectual substance of the discipline, but this really has nothing to do with "cheating" and everything to do with learning to teach better. And I don't really see why a guy giving a good assignment rather than a crappy one is headline-worthy when you subtract that deliberately distorted framing.
posted by RogerB at 11:53 AM on May 6, 2013 [2 favorites]

This is hardly a new concept. I TA'd a software engineering course at Ga Tech, and the midterm was 3 questions. The instructions were to place answers in a box after 90 minutes. In the meantime, anything was fair game, short of talking to any of the TAs or the prof. Leave the room, leave the building, bribe the Dean, whatever.
posted by Ardiril at 11:54 AM on May 6, 2013

n.b. the article is just part of the story.
posted by boo_radley at 11:57 AM on May 6, 2013 [3 favorites]

Seems quite possible that you'd get different results with the framing, "I'm going to let you cheat on this exam" vs. "This is a collaborative, open-book exam." It would be interesting to test the two different approaches with different classes.
posted by straight at 11:59 AM on May 6, 2013 [4 favorites]

This is a useful (and, as others point out, not at all unique) exercise, except for one thing: resources. Failure to account for scarcity (in the economic sense of being limited at all, not necessarily limited at a low level) means that people can afford to cooperate instead of compete. If Professor Nonacs had also told the class "10 percent of you will get an A; 20 percent, a B; 40 percent, a C; 20 percent, a D; and 10 percent of you will fail this exam," I suspect he would have seen markedly different behavior.
posted by Etrigan at 12:00 PM on May 6, 2013 [11 favorites]

This was an interesting post, thank you.
posted by the man of twists and turns at 12:02 PM on May 6, 2013 [2 favorites]

If your only reaction to this article is "well, cheating's only cheating if it's forbidden..." I think it's possible you missed the point completely.
posted by duffell at 12:08 PM on May 6, 2013 [8 favorites]

Failure to account for scarcity (in the economic sense of being limited at all, not necessarily limited at a low level) means that people can afford to cooperate instead of compete. If Professor Nonacs had also told the class "10 percent of you will get an A; 20 percent, a B; 40 percent, a C; 20 percent, a D; and 10 percent of you will fail this exam," I suspect he would have seen markedly different behavior.

Failure to impose, not account for. Just because A's are, in fact, scarce doesn't imply that the ambition to achieve an A isn't still held by every student. Everyone goes into an exam hoping to get an A and assuming everyone aces it, there is no shortage of As to be handed out. Same thing here and an obvious predictable result: "If we all turn in the same answer, he can't grade on a curve and he probably doesn't have the temerity to fail all of us."

Morally, of course, games can be tricky. Theory predicts that outcomes are often not to the betterment of the group or society. Nevertheless, this case had an interesting result. When the students got carte blanche to set the rules, altruism and cooperation won the day. How unlike a “normal” test where all students are solitary competitors and teachers guard against any cheating! What my class showed was a very “human” trait: the ability to align what is “good for me” with what is “good for all” within the evolutionary games of our choosing.

He seems to confuse altruism with an alignment of self-interest and the cold-blooded calculation of what's good for me is what's good for us. That property is not commutative.
posted by three blind mice at 12:17 PM on May 6, 2013 [2 favorites]

Oh hey - I co-taught an intro. course at UCLA with Peter in 2010. He has a reputation as the toughest instructor in the department. He's awesome. One of E.O. Wilson's last doctoral students, if I recall correctly.
posted by cnanderson at 12:19 PM on May 6, 2013 [6 favorites]

This was an episode of "The Paper Chase," the television show. Kingsfield gave a no rules test on the premise that the students would need to create contracts to protect the value of the answers transferred between themselves.
posted by rakish_yet_centered at 12:20 PM on May 6, 2013 [1 favorite]

Nope. He post-doc'd with E.O. Wilson.
posted by cnanderson at 12:22 PM on May 6, 2013

This guy seems all too much delighted with himself
posted by iotic at 12:28 PM on May 6, 2013 [3 favorites]

If your only reaction to this article is "well, cheating's only cheating if it's forbidden..." I think it's possible you missed the point completely.

And if the lecturer didn't want people to catch on the emotional and inflammatory but incorrect phrasing, then perhaps he shouldn't have used it as a hook in the headline.
posted by jacalata at 12:38 PM on May 6, 2013 [1 favorite]

He really needed follow up with a normed test of game theory learning to determine whether or not they actually learned game theory better than his previous classes. He has largely assumed the point he was trying to prove rather than actually tested it.
posted by srboisvert at 12:42 PM on May 6, 2013 [1 favorite]

There were 3 individuals who didn't participate, their grades were just above, just below, and about equal to the group exam. This seems to prove that any 2 out of 3 people can outhink the whole group. So it's a pretty poor experiment.

This reminds me of a section in one of Stanislaw Lem's stories, I think it's from when Pirx the Pilot takes his first exam to get into pilot school. He's in a roomfull of examinees. The test is long and impossible, full of mundane and useless questions like "list the first, middle, and last names of your grandparents and your greatgrandparents, including maiden names and birthdates." He notices some people quitting the test, but he persists a while longer, then gives up and hands in his test. On the way out, he is pulled aside by an examiner, and told he passed the test. He's shocked, there are people who are still in the room working hard on the exam, I'm one of the people who walked out and failed.
The examiner says, everyone quits this test eventually, but nobody fails. It's just a way to categorize people. The ones that gave up too easily are sent to lower management tasks. The ones that take too long to quit and try to complete every detail, are sent to become engineers. The ones who strike just the right balance, not too impatient with stupid useless facts, not too quick to rush to judgment, those are the ones we want, the rare ones that can become space pilots.
posted by charlie don't surf at 1:17 PM on May 6, 2013 [9 favorites]

I presented the class with one last evil wrinkle two days later, after the test was graded but not yet returned.

There were basically 4 total exams to grade here, right? The "group" submission and three "lone wolves". I think I've been doing this grading thing wrong the whole time.
posted by klausman at 1:40 PM on May 6, 2013 [8 favorites]

lol at that one person who scored worse than the shared answers. Nice work Galt-face.
posted by Potomac Avenue at 1:54 PM on May 6, 2013 [3 favorites]

Etrigan : I had a math teacher explain to me why he didn't grade on a curve once. He said it was because when he was a student he was graded on a curve and this resulted in all the students forming a pact that if any of them should be found studying, they were to be seized bodily by their fellows and tossed in the nearest fountain. So he may have still seen co-operative behaviour, just focused on resistance and sabotage.
posted by Grimgrin at 1:57 PM on May 6, 2013 [1 favorite]

Yeah, this is collaboration, not cheating. Cheating would allow you to kidnap the professor's cat and use it as hostage until he revealed the answers.
posted by dances_with_sneetches at 2:18 PM on May 6, 2013 [1 favorite]

@dances_with_sneetches: Have you ever considered teaching ethics? You have set a bar that I believe needs to be tested - and I like your moxie.

Also, do you have any kittens?
posted by Nanukthedog at 2:51 PM on May 6, 2013 [1 favorite]

Yeah, this is collaboration, not cheating. Cheating would allow you to kidnap the professor's cat and use it as hostage until he revealed the answers.

I believe that he would have permitted one student to just pay another student to take the test for him. That goes beyond collaboration.
posted by painquale at 3:26 PM on May 6, 2013 [2 favorites]

Real cheating would be:
* bribing/blackmailing someone else to take the test for you
* intercepting someone else's answers and copying them/changing the name/otherwise taking credit for their work without their knowledge or consent
* stealing the answer sheet and using it to complete the test
* sabotaging the test (fire alarm, bomb threat, copier accident)
* altering the official grade record
* bribing/blackmailing the professor, teaching assistant, or administrators
* forging a medical note or other excuse to be exempt from the test
* sabotaging the other student's work

Making a test open book is nowhere near "allowing cheating".
posted by ceribus peribus at 3:48 PM on May 6, 2013 [1 favorite]

If you want more hardcore allowed "cheating," you might find the Lessons of the Kobayashi Maru: Cheating is Fundamental lecture about a test at a military academy that required cheating to pass. The instructors told the cadets what the test would be (write the first 1,000 digits of pi) and that they could do anything they wanted to do to pass as long as they didn't get caught. There were some imaginative solutions.
posted by Candleman at 4:24 PM on May 6, 2013 [4 favorites]

* bribing/blackmailing someone else to take the test for you
* bribing/blackmailing the professor, teaching assistant, or administrators

Blackmailing someone was not permitted, given that it's illegal, but bribing someone was fine. You could pay someone else to take the test for you. You were even allowed to bribe the prof, though he said he'd turn down all bribes. (Though, if the price was right....)

* intercepting someone else's answers and copying them/changing the name/otherwise taking credit for their work without their knowledge or consent

If done non-violently, that was permitted.

* stealing the answer sheet and using it to complete the test
* sabotaging the test (fire alarm, bomb threat, copier accident)
* sabotaging the other student's work

If done legally: permitted.

* altering the official grade record

That runs afoul of the university rules, not the rules of the exam.

The kids in this class were too nice. I bet the prof was disappointed that everyone just buddied together into a big love-in. The smartest kid in the class should have announced that, for 50 dollars, he would submit a copy of his exam for anyone else. That alone probably would have caused a cascade of resentment against freeloaders, destroyed the whole-class camaraderie, and been a lot more fun.

And yeah, Kobayashi Maruing the exam somehow would have been brilliant. Replacing the stack of exams in the professor's car, hacking the scantron machine... you could get a whole Van Wilder-style movie out of this premise.
posted by painquale at 4:32 PM on May 6, 2013 [1 favorite]

If you want more hardcore allowed "cheating," you might find the Lessons of the Kobayashi Maru: Cheating is Fundamental lecture about a test at a military academy that required cheating to pass.

Meh. None of those cheats were anything like the Kobayashi Maru solution. There is a solution of that type, but more difficult. You give the answer as an arbitrary sequence like 3.111111111.. but you have already changed the professor's answer keys to 3.11.. all of them. Everywhere he could possibly look for the first hundred digits of pi, you change it to 3.111... etc. You look at his reference books, and if necessary, remove the page with Pi and rebind it with a substitute page. This would probably require hacking the entire campus's internet routers to look for strings beginning 3.14.. and substitute your answer. Essentially you would be changing the value of Pi in any internet communications.

I worked with someone who had memorized Pi out to 100 digits. I assume she would have flunked for not cheating.
posted by charlie don't surf at 5:53 PM on May 6, 2013 [1 favorite]

Is the take-home message, then, that cheating is good? Well…no. Although by conventional test-taking rules, the students were cheating, they actually weren’t in this case. Instead, they were changing their goal in the Education Game from “Get a higher grade than my classmates” to “Get to the best answer.” This also required them to make new rules for test taking. Obviously, when you make the rules there is no reason to cheat. Furthermore, being the rule-makers let students behave in a way that makes us a quintessentially unique species. We recognize when we are in a game, and more so than just playing along, we always try to bend the rules to our advantage.

This is interesting stuff.

In my sophomore year of college I sat in the back row of my 200 level Spanish class. I knew most of the stuff already thanks to high school classes and growing up in California, so the class was an easy A for me - and I didn't have many of those, quite frankly. As such, I had a pretty lackadaisical approach to the class in general and didn't pay attention much.

Which got me into trouble when Russel, the star running back of the school's football team, who sat next to me, completely copied every answer off of my midterm, without my knowledge. I blew through the test in 20 minutes or so and headed off for an early lunch, so I was quite surprised the next week when the teacher asked both Russel and I to stay behind after the tests were handed back (I had gotten a 97). She asked if there was anything either of us wanted to tell her about the test and before I could express my confusion Russel admitted the whole thing and that I had no part in it. So she excused me and Russel stayed behind.

I'm not certain what happened with his grade in the class, but I do know that he played in the football game that weekend, and every weekend after that, and that he went on to a brief stint in the NFL for the team in our nearest major city.

I was learning, although I probably didn't really realize it at the time, that life isn't fair, and that some people are going to cheat and get ahead. And some people are going to get ahead honestly. And some people are going to stay honest and get left behind.

I think I was also learning (and I did realize it consciously), that higher education (at least in the US, for undergrad), really was a game, which is why I agree in large part with the idea this prof is running with. I realized that to get ahead in class I didn't have to learn the material of the course as a primary objective, the primary objective was to, the way I saw it, give the professor what it is they wanted. This sometimes coincided with actually learning about business statistics et. al., but certainly not all of the time. It meant a lot of stupid little things that many students didn't seem to care about - I started sitting in the front of the classes, I started asking more questions, I started showing up to professor's offices during their hours there. Basically I started emulating the behaviour that I could ascertain the particular teacher in question would desire in all of their students. It wasn't hard, a lot of teachers would tell you flat out what that was, if you listened close enough.

For my last two years of college, it certainly became more of a game for me than an educational experience. I was learning material at the same time, but the real thing I was learning was how the world and the people in it really worked. Mainly in the university setting, but that would soon change too.

I gamed my way into my first job, and by that point I was so good at it that I didn't realize it when I did it. My grades had soared when I transferred from pre-med to business-marketing, and I was essentially de-facto added to the honorary society for kids studying in the various business tracks. I don't even remember what it was called - it had relatively less teacher-exposure than regular classes and such did, so it was naturally a lower game priority for me, and I think the other students in charge of the honorary could tell that. I didn't show up to the meetings or go on their field trips, and at one point I finally got a naughty-note in my campus mail that told me if I didn't go on the next field trip with the honorary, I would be booted out despite my grades and couldn't list it on my CV (we called them resume's at the time) any more.

Well, that last part didn't savvy with my job aspirations, so I threw a suit on (minus the tie), and took some of the older versions of my CV along because I didn't want to pay extra to print out the newer versions for this employer I wasn't very interested in in the first place. They weren't even printed on the nice stock paper. I could care less.

We ended up in the board room of one of the 7 divisions of the 2nd largest department store company in the world at the time. I was the only guy without a tie on, and I was bored. The meeting ended with a tour around their offices and their flagship store. I lagged towards the back of the group with one of the HR executives who I started chatting up with questions like "So who was responsible for deciding on this price point and making this particular sign?" or "And who decided which products to place at the front vs. the back of this department?"

Stupid stuff, really, just to make conversation. But at some point in my game education I had learned that asking questions shows (or imitates) that you're actually interested and engaged (even if you're not). So by that point I was doing it actively for the interviews for jobs I did want - preparing long lists of questions then prioritizing the good ones and asking as many as they gave me time for. And I was beginning to do it instinctively as well. It just happened to be on that particular day that I was doing it with the head of HR for the whole company, without my knowledge. And their particular recruitment philosophy happened to hone in on the candidates that asked the most questions of the recruiters.

I got back to my dorm room to a voicemail (well, we called them answering machine messages back then) basically begging me to come back for an interview. Which turned out to be not so much an interview as them trying to woo me into working for them (they did). Only 2 of the other 15 or so sycophants (as I haughtily considered them at the time of my pompous early 20's) even got an interview.

I wasn't particularly more qualified or even interested to work in the retail operations / fashion industry than my peers, but I won that particular game.

My first 13 years of employment hasn't been much different, its just been figuring out new games as you move between industries and new positions. You're given the game rules effectively in your standard working terms of reference - your Job Description, some of the answers to your questions in interviews, and ultimately in your performance objectives and appraisal. But its the same thing as a class, there's the surface material that everyone focuses on, and then there's the real game underneath. The office politics and the aligning yourself with the right projects and, let's face it, giving your management what they want, whether that coincides with the actual work on a particular day or not.

That's really what the students in this class were doing with the professor, they were giving him the desired outcome of his "flipping the test" - they were just doing it live. And it happened to coincide directly with the subject matter in this particular case.

I think the really interesting thing I'm learning now is that game theory (and I have precious little actual formal education on the topic) actually applies to most of life. For a lot of us, education and the work thereafter ARE most of life, but there's still relationships and family and personal business and finances and the like. There's a lot of game in all of them to an extent. A lot of give and take, a lot of different rules and objectives, to be sure. But a lot of the gaming approach can still be applied.

Hell, I'm gaming Metafilter right now by telling you a story.
posted by allkindsoftime at 7:54 PM on May 6, 2013 [11 favorites]

There were 3 individuals who didn't participate, their grades were just above, just below, and about equal to the group exam. This seems to prove that any 2 out of 3 people can outhink the whole group. So it's a pretty poor experiment.

Not quite. The author of the article clearly says that the "lone wolves" turned in variants of the group's answer, not completely original responses. If the high-scoring lone wolf had contributed his variants to the group, it's not unreasonable to think they all would have scored higher.
posted by duffell at 4:54 AM on May 7, 2013

Not necessarily, duffell, since that assumes the Lone Wolf answers would have been accepted into the Group answers without changes. Although maybe I'm just biased towards Lone Wolfery. The article doesn't give the exact format of the test, but it sounds like there was a great deal of group writing involved. My experiences with writing by committee have lead me to believe it sucks and I would be better off on my own.
posted by Panjandrum at 7:11 AM on May 7, 2013

The article doesn't give the exact format of the test, but it sounds like there was a great deal of group writing involved.

On the day of the hour-long test they faced a single question: “If evolution through natural selection is a game, what are the players, teams, rules, objectives and outcomes?”

posted by Etrigan at 7:25 AM on May 7, 2013

As there's a certain amount of subjectivity in grading essay-based exams, one has to wonder if Nonacs was subconsciously influenced by what he wanted the outcome to be. Doubly so considering he had apparently never asked this question of any class before. (The test, he told them, would be "far harder than any that had established my rep as a hard prof.")

To draw any conclusions at all from this, he should have given the same exam to two separate classes, one with the "anything (legal) goes" condition, as he did here, and one with traditional test-taking restrictions. No, scratch that, if you're trying to draw conclusions about cooperation specifically, the control group should be allowed everything the first is (open book, open internet, etc.) except the opportunity to work together. Then, the essays should be graded blind, by someone who doesn't know which essays come from the cooperative group and which from the control group. This could still be Nonacs himself, if there were some mechanism to hide from him which essays came from which class when he was grading them, and someone other than him proctored the exam (otherwise overhearing the group discussions could identify the group essay to him).
posted by DevilsAdvocate at 7:58 AM on May 7, 2013 [1 favorite]