"Usually I just do a Google search"

April 2, 2014 6:02 AM

Studying the "wisdom of the crowd" the Good Judgement Project has been asking average citizens to predict global events over the last 3 years. A weighted average shows these participants, who do not have access to classified materials, are more accurate than the Intelligence community. And the projects "elite" forecasters are 30% more accurate.
posted by fontophilic (30 comments total)

"How is it possible that a group of average citizens doing Google searches in their suburban town homes can outpredict members of the United States intelligence community with access to classified information?"

Maybe a better question is: what incentive does the United States intelligence community have to periodically be very wrong? The Department of Homeland Security has a $61 billion annual budget for 2013. It didn't exist at all until late December 2002.

That seems like a fair bit of money, and it's just a slice of the much larger US intelligence budget. With nobody being held accountable for how that money gets spent, I think you've got your answer.
posted by mhoye at 6:09 AM on April 2, 2014 [5 favorites]

The Department of Homeland Security has a $61 billion annual budget for 2013. It didn't exist at all until late December 2002.

Well yes and no. It's various parts all existed prior to 2002 and had a sizable budget. And it's not primarily an intelligence apparatus.
posted by empath at 6:17 AM on April 2, 2014 [7 favorites]

And did anyone think to look at how Google works? It remembers what you're looking for, and returns more of the same over time, all other things being held equal. Wouldn't that have an effect as well?
posted by ZeusHumms at 6:21 AM on April 2, 2014

Isn't groupthink pretty much ubiquitous across all large organizations? All the secrecy, financial incentives, etc. just makes it virtually impossible overcome the groupthink.

Yes, the Bush era yes-men legacy has removed anyone with even an inkling of correct analysis, but presumably intelligence analysts would fair worse even pre-9/11 too.

As noted in the CIA v Senate thread, the CIA has repeatedly lied about the fact that torture was useless.
posted by jeffburdges at 6:23 AM on April 2, 2014

what incentive does the United States intelligence community have to periodically be very wrong?

The intelligence people involved have no incentive to look dumb on this experiment. They have particular biases which this highlights, and some sections may not do effective quality control / improvement. Discussion of these results does not require your conspiracy theory.
posted by a robot made out of meat at 6:24 AM on April 2, 2014 [7 favorites]

"I'm just a pharmacist," she said. "Nobody cares about me, nobody knows my name, I don't have a professional reputation at stake. And it's this anonymity which actually gives me freedom to make true forecasts."

They do now! Thanks, NPR! You have just compromised an asset!
posted by GenjiandProust at 6:39 AM on April 2, 2014 [6 favorites]

I do this! And I'm terrible at it. Frankly I'm surprised that I haven't been kicked out for wrongness.
posted by troika at 6:40 AM on April 2, 2014 [1 favorite]

I wonder if the intelligence community has a bias toward interesting or edge case explanations and the lay-person, in their naivety, better applies occam's razor to these questions.
posted by dgran at 6:42 AM on April 2, 2014 [2 favorites]

We can predict the license plate numbers of all six cars outside. We can predict that our waitress is left-handed and the guy sitting up at the counter weighs two hundred fifteen pounds and knows how to handle himself. We can predict that the best place to look for a gun is the cab or the gray truck outside, and at this altitude, we can on average run flat out for a half mile before my hands start shaking. Now why would we know that (most of the time)?
posted by Think_Long at 6:46 AM on April 2, 2014 [3 favorites]

We know the CIA, NSA, etc., and their contractors, often lied, violate national security, etc. to boost and misuse their budget. There is really no conspiracy theory in saying that a secretive organization lies, cheats, etc. to bolster its budget. Yes, they'll tell the truth when the truth favors their interests, ala a stopped clock is right twice a day, but overall one should expect their conclusions carry a substantial self-serving bias. If anything, imagining that they're being particularly truthful is closer to a classical "conspiracy theory" because truth requires a specialized social enforcement mechanism while lies do not. Also the 2013 'black budget' was $53B, probably more form military than DHS.
posted by jeffburdges at 6:53 AM on April 2, 2014

And people wonder how I've been outpredicting the weatherman for almost 5 years. It just goes to show that if you know what you're looking at, nothing is hard to understand.
posted by deezil at 6:56 AM on April 2, 2014 [1 favorite]

See also: OSINT (Open Source Intelligence)

"I'm just a pharmacist," she said. "Nobody cares about me, nobody knows my name, I don't have a professional reputation at stake. And it's this anonymity which actually gives me freedom to make true forecasts."

They do now! Thanks, NPR! You have just compromised an asset!

I think that comment had more to do with jeffburdges' comment on groupthink. It's a whole lot easier to say something that goes against the "accepted" wisdom and internally agreed upon trends if you're not part of the intelligence agency. And if you're not part of the discussions where folks come to some agreement upon likelihood of trends or events, your opinions won't be biased/shaped by those discussions.

Also, I wonder how much of looking at broader world news helps inform the "elite" forecasters, compared to intelligence agents who live in a world of classified information and may be faced with information fatigue.
posted by filthy light thief at 6:59 AM on April 2, 2014 [1 favorite]

I'd suggest that intelligence agents over value classified information. That information is sexy and expensive, it must be important.
posted by bdc34 at 7:13 AM on April 2, 2014 [3 favorites]

And I'm sure because it is expensive, the folks whose task it is to rank and detail information by source are encouraged to value that information more highly than OSINT. Otherwise, why invade private conversations of US citizens evaluate metadata if you can just use Google better?
posted by filthy light thief at 7:21 AM on April 2, 2014

Deezil, its actually about the complete opposite of that. The pharmacist NPR interviewed does not have super powers because she "thought about it real hard". Likewise, this isn't a story that should prove the futility of "book-learning" and the triumph of amateurs.

It's about multiple sources converging on a signal, not one person being lucky. Lots of things are very hard to understand from one narrow individual experience—the parable of the blind men describing an elephant comes to mind. If you can look at the meta analysis of individual answers of a rope, a tree, a snake, you might get a picture of an elephant.

And likewise, if you ask the question of 3000 people from all over the country, vs 3000 people who all live in the same geographic area, went to one of a handful of schools, have middle class backgrounds, read the same sources, are trained to think in the same ways, and are cultured to give responses that are "acceptable", and must do so under incredible social pressure, you might get more people converging on "a tree" and less on "a snake" or "a rope".

But even the diverse amateurs, and the trained professionals don't individually have the right answers all of the time. Its the analysis of their answers that does.
posted by fontophilic at 7:23 AM on April 2, 2014 [4 favorites]

I think it's probably useful to note that this entire competition is sponsored by IARPA (an office within the Office of the Director of National Intelligence), indicating that the Intelligence Community isn't incapable of self-criticism.
posted by gsteff at 7:29 AM on April 2, 2014 [1 favorite]

is there really a story here? Just because she's consistently in the top 10% of predictors doesn't imply that she is somehow able to predict things better. In a group of 3000, you're always going to have 300 people who are in the top 10%.

What it reminds me the most of is this episode of darths and droids

Pete, being the highly logical, calculating person he is, rejects all of that as superstitious nonsense. He instead applies the scientific approach. Over the years, he's collected somewhere around a thousand twenty-sided dice. Every so often, he gathers them all together. He sits down at a table and carefully and individually rolls each of the thousand dice, once. Of course, roughly a twentieth of them will roll a one. He takes those fifty-odd dice and rolls them a second time. After about an hour of concentrated dice rolling, he'll end up with around two or three dice that have rolled two ones in a row. He takes those primed dice and places them in special custom-made padded containers where they can't roll around, and carries them to all the games he plays.

Then, when in the most dire circumstances, where a roll of one would be absolutely disastrous, he pulls out the prepared dice. He now has in his hand a die that has rolled two ones in a row. Pete knows the odds of a d20 rolling three ones in a row is a puny one in 8,000. He has effectively pre-rolled the ones out of the die, and can make his crucial roll with confidence. Furthermore, being scientific about it means he knows that it doesn't matter who rolls the die for the third time, so he has no qualms about sharing his primed dice with other players, if that's what it takes to avoid disaster.

posted by rebent at 7:34 AM on April 2, 2014 [7 favorites]

fontophilic: "And the projects "elite" forecasters are 30% more accurate."

The following excerpt has been going around in my head lately, and so it instantly came to mind upon seeing the above quote in the FPP -- and after reading the article I had to go look up the full passage.

[It might seem] that the discovery by the Culture's Minds that some humans were actually capable of matching and occasionally beating their record for accurately assessing a given set of facts would lead to machine indignation and blown circuits, but this was not the case. It fascinated those Minds that such a puny and chaotic collection of mental facilities could by sleight of neuron produce an answer to a problem which was as good as theirs. There was an explanation, of course, and it perhaps had something to do with patterns of cause and effect which even the almost god-like power of the Minds had difficulty trying to fathom; it also had quite a lot to do with sheer weight of numbers.

There were in excess of eighteen trillion people in the Culture, just about every one of them well nourished, extensively educated and mentally alert, and only thirty or forty of them had this unusual ability to forecast and assess on a par with a well-informed Mind (of which there were already many hundreds of thousands). It was not impossible that this was pure luck; toss eighteen trillion coins in the air for a while and a few of them are going to keep landing the same side up for a long, long time.

-Consider Phlebas, Iain M. Banks

posted by barnacles at 7:50 AM on April 2, 2014 [4 favorites]

An even better analogy is the scam where you send 512 letters to investors, 256 of which say, "Microsoft stock will go up next week" and 256 which say "Microsoft stock will go down next week." The next week, if Microsoft went up, you send 256 letters to the investors who got your correct projection, 128 of which say "Google stock will go up" and 128 which say "Google stock will go down." The next week 64 & 64 letters go out to the 128 who got correct prediction about Google.

After 5 weeks, you've got 16 investors who are convinced you're a genius and are dying to give you all their money to invest.
posted by straight at 7:51 AM on April 2, 2014 [10 favorites]

now i want a certification in elite forecasting.
posted by bruce at 7:51 AM on April 2, 2014

The Psychologist Gerd Gigerenzer (wikipedia) has a theory about why lay people sometimes outperform experts. A paper by Goldstein and Gigerenzer (pdf) gives a lot of good examples. Basically, if you're immersed in information it can be difficult to judge what's important - for example, if you watch 24 hour news channels it can become difficult to distinguish the really important news stories whereas if you just check the headlines occasionally that becomes more easy. And there's also the idea of overfitting - where a model is based on too much past data it becomes too specific to what has gone before and less good at generalising to novel situations.
posted by jamespake at 8:04 AM on April 2, 2014 [3 favorites]

I'd suggest that intelligence agents over value classified information. That information is sexy and expensive, it must be important.

This is what worries me about the special intelligence briefings that decision makers like the president & key members of Congress receive.

The information they get is probably no more important or accurate than if a normal person spent some time reading up a bit on the subject, but due to the classified and exclusive nature of the info and the way it is presented, it is **bound** to be way over-valued by those who receive it.
posted by flug at 8:20 AM on April 2, 2014

Barnacles, there was a similar thing, but institutionalized, somewhere in the Revelation Space universe, the Demarchy, I think. A near-continual polling system operated a distributed democracy, but those with proven better forecasting abilities slowly gained more and more "votes" to increase their weighting. I think there was a Vinge (Joan, not Vernor) book deep in the misty past, which did such a thing.
posted by adipocere at 8:22 AM on April 2, 2014 [1 favorite]

If ordinary people are this much better, imagine what using their MOMS would give us! It turns out, Putin just wanted attention.
posted by thelonius at 8:39 AM on April 2, 2014 [4 favorites]

I imagine these people as being like Fal from Bank's Consider Phlebas, a human who can predict future events better than hugely complex AIs.
posted by meta87 at 9:24 AM on April 2, 2014 [1 favorite]

And now I see barnacles has already brought it up hah!
posted by meta87 at 9:25 AM on April 2, 2014

They beat the unweighted average (wisdom-of-overall-crowd) by 65%; beat the best algorithms of four competitor institutions by 35-60%; and beat two prediction markets by 20-35%.

Without base rates this tells us next to nothing. It may be that one is useless and the other is just ever so slightly less useless.
posted by srboisvert at 9:43 AM on April 2, 2014

now i want a certification in elite forecasting.

You can register to be a part of their next season at the first link. Sounds fun.
posted by GalaxieFiveHundred at 2:31 PM on April 2, 2014

People still seem to entertain the silly notion that other people become employees for their competence. But those with a lot of competence, can just be a freelancer / contractor / entrepeneur.

Like Snowden. If you can get them as employees, you can limit the potential damage. You can promote them to positions of marginal competence. Know anyone like that?
posted by Twang at 9:58 PM on April 2, 2014

Basically, if you're immersed in information it can be difficult to judge what's important - for example, if you watch 24 hour news channels it can become difficult to distinguish the really important news stories whereas if you just check the headlines occasionally that becomes more easy.

I think you see that alot with consumer electronics, particularly w/rt apple products. The experts will all tell you that there's nothing new in every release, there are other products that have more features or are cheaper, etc. Then people buy them in droves.
posted by empath at 2:43 AM on April 3, 2014