Spontaneous Speech Cannot be Faked

April 28, 2011 10:12 AM

Achievement Metrics claims that a player's speech can predict how he'll perform in the NFL.
posted by beisny (37 comments total)

looking for traits such as "conceptual complexity," "need for power," and "deliberativeness."

I believe the strategery being used here is to build a nukelar core of a team.
posted by Mister Fabulous at 10:16 AM on April 28, 2011

Heh, just this morning I was listening to an (old) podcast about an (even older) company that claimed it had a computer program that could parse a movie script and predict blockbusters.

I'm just as skeptical of this as I was of that.
posted by DU at 10:18 AM on April 28, 2011

[throws challenge flag]
posted by preparat at 10:20 AM on April 28, 2011 [5 favorites]

My speech has long predicted how I'd perform in the NFL.
posted by StickyCarpet at 10:21 AM on April 28, 2011 [1 favorite]

Shades of the Wonderlic (try for yourself).
posted by box at 10:26 AM on April 28, 2011

um, ah, what?
posted by mccarty.tim at 10:29 AM on April 28, 2011

You know, I read things like this, and I realize that this is simply the sort of thing I can't ignore.

Let's leave aside the question of whether Achievement Metrics has actually been able to predict, even imperfectly, the outcomes of NFL players. I'm dubious, but hell, they might be right; I'm never going to care enough about the minutiae of sports statistics to determine the truth of any of this myself, certainly (although American football, with its propensity to leave its players brain-damaged, remains a guilty pleasure with me). But when Agger (at the Slate link) says "Achievement Metrics is trying to persuade NFL executives that it's not selling pointy-headed gibberish. It's presented its work at the premier sports stats conference at MIT and has been approached by curious teams," I suspect that there's a good chance that a positive feedback loop will be set up where players who score well on their metrics will be given more attention and end up doing better as a result.

And the part I can't ignore? I would not be surprised if this sort of thing filters into the workplace and this ends up being another tool to be potentially used to evaluate my potential. Ugh.
posted by UrineSoakedRube at 10:31 AM on April 28, 2011 [1 favorite]

At the very least, this speech research seems more persuasive than the Wonderlic, which has been shown in repeated studies to have no ability to predict future performance.

The only reason this bullshit set of voodoo metrics hasn't been shown to have zero ability to predict future performance is that it so new that it has never had a chance to fail at predicting future performance. Showing that your prediction algorithm correctly predicts events in the past that your prediction algorithm was developed to match is not proof of anything. And the fact that the company previously sold "terrorist speech" detecting software to the government doesn't exactly inspire confidence.

The beauty of what Achievement Metrics is trying to do is that spontaneous speech cannot be faked. (This is in contrast to the Wonderlic, which you can prep for like the SAT.)

Yeah right. The source data they are using for this is post-game athlete interviews, which almost always come across as completely canned and formulaic. Any athlete could easily game these metrics by throwing out all of the "Our team played great" and "I was really in the zone today" cliches without seeming at all suspect, because most of these interviews are a series of cliches anyway. Even if this metric worked, it would instantly be ruined because top prospects would all just waste time getting a "positive speech coach" or whatever to teach them what to say in interviews to get the right result.
posted by burnmp3s at 10:34 AM on April 28, 2011

With most of these things, regardless of whether or not the association between metric and performance is actually meaningful, there's always an underlying assumption that the association only holds so long as people aren't trying to game the system by optimizing the thing they're measuring.

IMO, this underlying assumption is present even (or maybe even especially) when the supporters are claiming that the metric can't be gamed.
posted by You Can't Tip a Buick at 10:36 AM on April 28, 2011 [2 favorites]

Joey Harrington and Matthew Stafford staring me in the face with their big ugly red circles...

WHY MUST I BE A LIONS FAN?!
posted by WinnipegDragon at 10:37 AM on April 28, 2011

I just ran my Metafilter comments through this thing and apparently I'm about to commit an unspeakable act of genocide.
posted by brain_drain at 10:39 AM on April 28, 2011 [1 favorite]

WHY MUST I BE A LIONS FAN?!

I'm in that boat with you, WinnipegDragon. But I am a hopeful person.
posted by grubi at 10:48 AM on April 28, 2011

I read this article yesterday, and was honestly fascinated. Sure, I have several grains of salt handy, but there is something to be said about speech patterns reflecting behavior. It really does say something about someone when they talking about "I" all the time instead of "we." This could also be counter-predictive, based on position -- for example, a skill position player that relies on individual achievement (e.g. a wide receiver) vs. another position that relies on coordinated work with teammates (e.g. an offensive lineman). The wide receiver constantly talking about his own performance and demands to get the ball may actually be what you want.

There is similar research in criminal behavior and interrogations. For example, under analysis, the sentence structure of O.J. Simpson's "suicide letter" points to his guilt, in that it fails to offer a true denial. There's a huge difference between "I have nothing to do with this" and "I didn't do it."
posted by Cool Papa Bell at 10:51 AM on April 28, 2011 [1 favorite]

My speech was a pretty good predictor of how I performed in the NFL

having made a dumb high school debate joke, he slinked back to work
posted by dismas at 10:54 AM on April 28, 2011 [1 favorite]

The only reason this bullshit set of voodoo metrics hasn't been shown to have zero ability to predict future performance is that it so new that it has never had a chance to fail at predicting future performance. Showing that your prediction algorithm correctly predicts events in the past that your prediction algorithm was developed to match is not proof of anything. And the fact that the company previously sold "terrorist speech" detecting software to the government doesn't exactly inspire confidence.

One way to approach this is to randomly divide your sample into a 'training' dataset and a 'verification' dataset- if a model developed on the training set predicts recorded outcomes in the verification dataset reasonably well, then the model is probably decent, at least for the population from which the sample was generated.

There's no evidence that they've done this here: I strongly suspect that the 'accuracy' statistics that they provide are based on the dataset that they used to generate the model. And as you say, making useless models is really easy- simply grab enough variables out of the subset that randomly vary in the same way your prediction variable does and you can make a model that holds to your training set well but will completely fall apart when new information is provided.
posted by monocyte at 10:57 AM on April 28, 2011 [5 favorites]

there is something to be said about speech patterns reflecting behavior.

Has that something been said in a peer-reviewed journal?

It really does say something about someone when they talking about "I" all the time instead of "we."^{[citation needed]}
posted by DU at 11:06 AM on April 28, 2011

I don't know about this...I mean this guy had a pretty successful NFL career.
posted by rocket88 at 11:07 AM on April 28, 2011

An illustration of monocyte's point
posted by hattifattener at 11:08 AM on April 28, 2011

I'm just happy to be here. Hope I can help the ballclub.

We gotta play 'em one day at a time.

I just want to give it my best shot, and -- the good Lord willing, things will work out.
posted by mr_crash_davis at 11:16 AM on April 28, 2011 [2 favorites]

Achievement Metrics could scam two birds with one stone if they claimed they could predict an sport star's performance based on how much he prayed.
posted by DU at 11:18 AM on April 28, 2011 [2 favorites]

I suspect that there's a good chance that a positive feedback loop will be set up where players who score well on their metrics will be given more attention and end up doing better as a result.

From the POV of someone putting together an NFL team, is that even a bad thing? Their goal is to put good players on their team, and if the way to do that is to find people willing to game a language test and then pay special attention to them so they do better, then the team still has a good player.


It would of course be interesting to do a study of players whose managers did/didn't use Achievement Metrics' numbers, to see how much that use actually does affect the players' performance in games, and to see whether different metrics are a better predictor of player performance in an AM-saturated context than without.
posted by hattifattener at 11:18 AM on April 28, 2011

Speech can make a difference in football. An anecdote from when I played high school football. My coach said, "Hill, I'm going to send you in on the next play." I said, "Why?" He sent in someone else.
posted by dances_with_sneetches at 11:24 AM on April 28, 2011 [3 favorites]

And the part I can't ignore? I would not be surprised if this sort of thing filters into the workplace and this ends up being another tool to be potentially used to evaluate my potential. Ugh.
posted by UrineSoakedRube at 1:31 PM on April 28 [+] [!]

Eponysterical?
posted by oneironaut at 11:32 AM on April 28, 2011

It really does say something about someone when they talking about "I" all the time instead of "we."[citation needed]

Here you go.
posted by Cool Papa Bell at 11:36 AM on April 28, 2011

I would not be surprised if this sort of thing filters into the workplace and this ends up being another tool to be potentially used to evaluate my potential. Ugh.

But this already happens every day in a typical job interview process. You go into an interview and answer questions, and the person doing the hiring performs an ad-hoc filter on your level of go-getter-ness.

All they're doing here is attempting to apply more scrutiny to word choices ... in addition to throwing some chicken bones on the floor and taking a guess.
posted by Cool Papa Bell at 11:39 AM on April 28, 2011 [2 favorites]

The comparison between terrorism detection and NFL performance brought something to my mind. False results. They claimed 90% accuracy, but what does that mean? Are they indicating players will do better than their results suggest, or worse? In other words, are those false positives, or false negatives?

In terrorism detection, you are screwed either way. False positives put innocent people in Gitmo forever. False negatives mean terrorists slip through, and people can die.

In personnel selection, false positives mean you simply don't select an otherwise good player. False negatives allow players predisposed to poor behavior on the team.

Now here's the moral hazard for the algorithm developers. False positives don't affect you. Nobody can prove that they "would have" been good. By keeping poor players off along with some good players, the team management sees a net drop in poor player incidents - but they have no way of knowing that there may be some great players they are missing. That's fine for the developers - they don't care as long as they get their money.

Now, if they allow false negatives, and some poor players get on the team, team management may look at the process, and say, "this process is crap, it's not working", or" it's not worth the money we are paying for it" - even though it is working "as well" as a model that errs in favor of false positives.

It's going to be prone to bias, and will add a layer of complexity - where ideally, management should be looking for clarity and simplicity. Nevertheless, in the "win at all costs" paradigm, people will be willing to shell out all kinds of money on questionable products.

Note that the same hazard exists for antiterrorism. Better for security consultants and contractors to put innocent people in jail than to permit terrorism. It's just good business.
posted by Xoebe at 11:42 AM on April 28, 2011 [3 favorites]

hattifattener> From the POV of someone putting together an NFL team, is that even a bad thing? Their goal is to put good players on their team, and if the way to do that is to find people willing to game a language test and then pay special attention to them so they do better, then the team still has a good player.

Well, that's assuming that the language test can be gamed. But if it can, then you end up with players with a specific skill: the ability to talk in AchievementMetrics-speak. I recall (but can't remember the name of) a player in the NBA who was borderline OCD, the sort of guy who had excellent free throw stats, because he practiced obsessively, but who would almost certainly fail at this sort of metric. A sane approach would be to cultivate his playing skills and ignore his quirks. A crazy approach would be to work on his public statements, as the benefits would be secondary to actually getting points, pulling down boards, making assists, and blocking shots.
posted by UrineSoakedRube at 11:43 AM on April 28, 2011

James Pennebaker at UT and colleagues have been studying the relationship of physical/mental health and language for a while. You can find citations for some of these studies at their Linguistic Inquiry and Word Count (LIWC) page.

I worked on a study in 2009 where we analyzed the (written) responses of participants in our mindfulness-based substance abuse relapse prevention program, looking to see if their use of mindfulness-related language at the end of the course predicted better outcomes after the course (ie, less substance use). We found greater use of mindfulness language predicted fewer substance use days during the 4-month follow-up. (of course, further research is needed on this!)

Here is the citation, if you have academic journal access:
Collins, S., Chawla, N., Hsu, S., Grow, J., Otto, J., and Marlatt, G. (2009). Language-based Measures of Mindfulness: Initial Validity and Clinical Utility. Psychology of Addictive Behaviors. 23(4): 743–749.
posted by oceansize at 12:01 PM on April 28, 2011

wow, nice predictions, but how this is possible
posted by vincentabry at 12:52 PM on April 28, 2011

I think some aspects of MetaFilter could be described as, you know, relatively interesting, maybe, to some people. And, I guess, others could probably find content along these lines, well, marginally less interesting to some degree. But that's just my opinion, YMMV.
posted by The Winsome Parker Lewis at 12:54 PM on April 28, 2011

This post sucked! FACT. Every FPP I ever made is demonstrably superior to this one! BAM!
posted by The Winsome Parker Lewis at 12:54 PM on April 28, 2011

Correlation, please to be meeting Causality. Though you two look the same, unfortunately, you are not equal.
posted by yankovic at 12:59 PM on April 28, 2011

There is similar research in criminal behavior and interrogations. For example, under analysis, the sentence structure of O.J. Simpson's "suicide letter" points to his guilt, in that it fails to offer a true denial. There's a huge difference between "I have nothing to do with this" and "I didn't do it."

That sounds like total BS.
posted by delmoi at 1:01 PM on April 28, 2011

With most of these things, regardless of whether or not the association between metric and performance is actually meaningful, there's always an underlying assumption that the association only holds so long as people aren't trying to game the system by optimizing the thing they're measuring.


In other words, grades are meaningful only as long as students don't care about them.
posted by madcaptenor at 1:07 PM on April 28, 2011

Also note that it's not a coincidence that they use NFL quarterbacks in their examples, because NFL quarterbacks are one of the most notoriously difficult athletes to predict the performance of. According the logic of Achievement Metrics, players with extremely bad scores on these metrics are still able to be the absolute best quarterbacks in high school and college. So whereas tracking the interception rate of a college player has basically no correlation to their NFL interception rate, somehow tracking their use of team oriented language would be able to predict it.

I guess it's still possible to say that somehow playing at the highest level in professional sports requires some sort of special personality traits not required at any of the lower levels. But that ignores the fact that quarterback performance varies wildly from year to year and across teams. It's not possible to predict how well a quarterback will play for a random team in 2013 if you watch them play in 2011 and 2012 and keep track of all of their stats, so the idea that you could somehow predict their 2013 performance on a random team based on quotes from interviews is preposterous.

In my opinion QBs are so hard to predict because most of their stats are heavily dependent on the performance of other players and the team as a whole, so when you are really measuring the team as much or more than the individual. This blog post goes through some analysis of that idea. Step one to coming up with a way to predict quarterback performance is to figure out a way to actually measure their performance properly, which is not exactly easy. Otherwise you are trying to predict inherently flawed metrics (QB ratings) using other completely unrelated metrics (Wonderlic scores and Achievement Metrics).
posted by burnmp3s at 1:18 PM on April 28, 2011

The paper discusses the technique of remote assessment, which involves gathering samples of speech and determining whether someone is likely to be a terrorist. [snip] John Kerry, for example, was classified as a terrorist. (The speech that the study sampled came from the months in the 2004 campaign when he went negative.)

This is Winston Smith, cowering in his room, surreptitiously jotting down negative thoughts in his unauthorized diary because if he said them out loud, he would surely be branded a terrorist.
posted by nzero at 1:59 PM on April 28, 2011

These people have never watched a Bill Belichick coached player give a press conference, have they?
posted by nathancaswell at 2:12 PM on April 28, 2011