Psychologist FTW
March 1, 2008 9:48 AM   Subscribe

This Psychologist Might Outsmart the Math Brains Competing for the Netflix Prize
posted by AceRock (32 comments total) 17 users marked this as a favorite
 
Great reading! Thanks so much for posting this. I was reading this out loud just now to some friends, and while I was thoroughly fascinated, the friends were looking at me like I was insane! I always cherish those moments.
posted by lonemantis at 10:10 AM on March 1, 2008 [4 favorites]


Thanks For The Link, AceRock, It's Great Reading.
posted by flatluigi at 10:24 AM on March 1, 2008


Yes, interesting.
posted by weapons-grade pandemonium at 10:28 AM on March 1, 2008


I wrote this article!

There's lots of great stuff that didn't make it into Wired for space reasons. Like Todd Holloway's beautiful visualizations of the Netflix dataset. Or Brandyn Webb, aka Simon Funk, explaining completely from scratch how to set up an SVD attack on the Netflix problem.
posted by escabeche at 11:11 AM on March 1, 2008 [20 favorites]


Oh, and here's "Just a Guy in a Garage"'s no-longer-anonymous blog.

By the way, despite the title of the piece (in the print magazine it's the more sober "The Netflix Challenge") I hope the article doesn't give the impression that psychology beats math. Much closer is to say that psychology can serve as a complement to math when analyzing giant datasets arising from human behavior.

Also, in an interesting development that took place after we went to press, the team from AT&T has been overtaken by "When Gravity and Dinosaurs Unite," a collaboration between a group of three recent Princeton grads and a team from the Budapest University of Technology and Economics.
posted by escabeche at 11:35 AM on March 1, 2008


Well done escabeche and ace rock. Interesting stuff!
posted by vronsky at 11:38 AM on March 1, 2008


This prize is very interesting, and despite the titling and hype so far Math is still beating Psychology.

Netflix should use some of either, though, to figure out that thanks to their pop up ads I will never ever give them any money.
posted by TheOnlyCoolTim at 11:47 AM on March 1, 2008 [1 favorite]


Looks like "When Gravity and Dinosaurs Unite" is a collaboration between team Gravity and team Dinosaur Planet. I wonder if we'll see any more of the top teams group up to eke out another percent.
posted by Justinian at 11:49 AM on March 1, 2008


Is there going to be some way to keep this out of the hands of the movie studios? I don't need complicated algorithms to predict what the studios will do with this technology.

Norbit 3: The Vampire Slayer

My theory is that the moviegoing public rates all movies on a simple scale with High School Musical on one end, and the Saw movies on the other. The perfect movie being a seamless combination of the two.

Thank you Netflix, as soon as i convert this theory to code, I would like my million dollars in small non-consecutive bills.
posted by billyfleetwood at 12:06 PM on March 1, 2008 [1 favorite]


I understand about Quebec, but why are residents of "Cuba, Iran, Syria, North Korea, Myanmar (formerly Burma) and Sudan" ineligible to participate in this contest?
posted by growabrain at 12:24 PM on March 1, 2008


Fantastic post. Very illuminating article, escabeche! It's fascinating stuff for a non-mathematician.

Posts like this is what makes this a great place to visit. Two thumbs up .
posted by PareidoliaticBoy at 12:26 PM on March 1, 2008


Nice article. But this should be easy. All I need to do is give completely accurate assessments of the objective goodness of each movie. Then see how stupid people are compared to me. If for instance, you liked Mr and Mrs Smith, then I'll know that you won't like 2001AD (it's too long and boring yeah?).
posted by leibniz at 12:32 PM on March 1, 2008


I understand about Quebec, but why are residents of "Cuba, Iran, Syria, North Korea, Myanmar (formerly Burma) and Sudan" ineligible to participate in this contest?

There are probably various embargoes in place that prevent American companies from sending money to those places.
posted by davey_darling at 12:32 PM on March 1, 2008


I understand about Quebec, but why are residents of "Cuba, Iran, Syria, North Korea, Myanmar (formerly Burma) and Sudan" ineligible to participate in this contest?

According to Netflix's Prize FAQ:

Why are some countries excluded from participation?

Most of those countries appear are on the U.S. Treasury Office of Foreign Assets Control’s list of embargoed counties for which we cannot provide economic assistance. If this list changes, we’ll post a change to the rules and let you know. Quebec has other reasons.

posted by Izner Myletze at 12:38 PM on March 1, 2008


Good show, escabeche! It's not often that I see science/tech writing which doesn't shy away from giving the reader some credit while at the same time giving enough technical background so that interested parties have good starting points to do further research. Bravo! And, quite frankly, I was rather surprised to see this published by Wired, of all places.
posted by ooga_booga at 1:15 PM on March 1, 2008


Yeah that was good pop science, I enjoyed the article.
posted by jcruelty at 1:34 PM on March 1, 2008


for instance, you liked Mr and Mrs Smith, then I'll know that you won't like 2001AD

I like both; the first being 3.5 stars, the latter, a solid 5.
posted by Mick at 1:43 PM on March 1, 2008


If for instance, you liked Mr and Mrs Smith, then I'll know that you won't like 2001AD

When being snobby it's good to get the titles correct.
posted by Justinian at 2:21 PM on March 1, 2008 [1 favorite]


This Psychologist Might Outsmart the Math Brains Competing for the Netflix Prize

If by "Psychologist" you mean "Amateur Mathematician who happens to have a psych degree."
posted by delmoi at 3:22 PM on March 1, 2008


This was very interesting. Oddly enough I had never even heard of the contest.

Thanks for posting this AceBase, and thanks for writing the article escabeche.
posted by tkolar at 3:22 PM on March 1, 2008


Yawn. When being a pedant, it's good to first ascertain whether your target was presenting a serious argument or not.

If pushed Mick, I suppose all I have to do is slightly adjust your insanity rating (they're supposed to get more accurate the more information you give right?) Not wishing to troll of course, but do you really want to start a discussion in which you have to defend the merits of Mr and Mrs Smith?
posted by leibniz at 3:45 PM on March 1, 2008


It met my needs (I was entertained).
posted by Mick at 4:14 PM on March 1, 2008 [1 favorite]


escabeche, that was an excellent article.

I was hoping that we'd get to see some more of the psychological insights that he's bringing to bear on the problem. The anchoring effect was mentioned, but nothing else. Just going down the list of cognitive biases at Wikipedia, I can see more than a few that might be relevant.
posted by painquale at 5:30 PM on March 1, 2008


studying psychology was very interesting in that it taught me to examine the ways i go about solving various problems, in a sort of round about way. i'm not smarter for it, but here and there i've improved my own algorithms and found solutions laterally. the list of cognitive biases, for example, is gold for anyone who enjoys a good argument.

great article, well written. good on ya :)
posted by Dillonlikescookies at 8:34 PM on March 1, 2008


fun
posted by LobsterMitten at 10:18 PM on March 1, 2008


As a statistician, I have two observations. First, is that the anchoring effect is pretty well known to statisticians working in the field of behavior. And the second is that I'm pretty sure by now that the algorithms are all overfit to the validation dataset, and therefore won't do as well in predicting new observations. That is all.
posted by Mental Wimp at 10:41 PM on March 1, 2008


Taking a look through the data, I note that despite having 100,000,000 ratings from 500,000 users, the median film only has 546 ratings.

My hat is off to anyone who can make predictions with data that sparse.
posted by tkolar at 11:10 PM on March 1, 2008


Ok Mick, you've forced me to criticise Mr and Mrs Smith. But first of all, I'm not even sure that 'it met my needs' would qualify for 3.5 stars. Eating a bag of crisps* meets your needs, but would you then describe it as 70% as good as the finest meal you've ever had?? Or a quick wank 70% as good as the best sex you've ever experienced?

Anyway lets suppose we accept the pretensions of Mr and Mrs Smith as an uncomplicated mainstream hollywood film. And forget this was a action-romantic-comedy, with boring action, no romantic warmth, and no laughs, wooden dialogue, generic soundtrack, unrealistic sets etc. Also forget about the moral bankruptcy of the characters (why am I supposed to be sympathetic to these smiling murderers again?). And the utterly predictable plot starting with the ridiculous contrivances needed just to get these brain dead characters not to realise the other is an assassin - finishing with an utterly dull last scene of endless repetitive shooting with a laughable timing of 'them finally helping each other' for no particular reason, but signalled by a cue in the sound track and a new camera angle as they wipe out some more anonymous bad guys - a million loose ends left hanging at the tacked on happy ending.

Suppose we ignore all that. Let's just think about so-called 'chemistry' between the leads. It had all the chemistry of an athletic porno fuck without the guts to make it sexually explicit. There was not a single moment of emotional warmth or tenderness between them. The only chemistry between these two actors was possibly that of the audience member ogling at these beautiful dolls. It had all the romantic charm of a bicep pumping body builder leering at you with his dead steriod glaze, forever.

The success of this film was the success of a marketing bubble and the fame/beauty of the leads, blinding you to its manifest rubbishness. I guarantee that history will consign this film to oblivion.

*chips in US right?
posted by leibniz at 2:31 AM on March 2, 2008 [1 favorite]


I wrote...
Taking a look through the data, I note that despite having 100,000,000 ratings from 500,000 users, the median film only has 546 ratings.

... and the median user has 96 ratings to their name. Which, frankly, is a lot more than I expected.
posted by tkolar at 8:10 AM on March 2, 2008


How do you give half ratings on Netflix? I can't seem to do that.
posted by philomathoholic at 12:21 AM on March 3, 2008


How do you give half ratings on Netflix? I can't seem to do that.

You can't. Your only choices are 1,2,3,4, and 5.
posted by tkolar at 7:24 AM on March 3, 2008


Besides inventing calculus, leibniz has beautifully deconstructed Mr. and Mrs. Smith. Bravo.
posted by Mental Wimp at 9:34 AM on March 3, 2008


« Older There's nothing on TV? Try channel 2 in Nicaragua.   |   Succumb to King Game. Newer »


This thread has been archived and is closed to new comments