The sharks are just jealous of our ice cream

September 3, 2008 10:50 AM

BBC News is running a weekly ongoing series of articles that describe and illustrate common misconceptions (and manipulations) of statistics using examples from the news and ads.
Lesson 1: surveys. Lesson 2: counting. Lesson 3: percentage. Lesson 4: averages. Lesson 5: causation.
posted by Tehanu (46 comments total)

You gotta love the BBC! Public money well spent.
posted by zouhair at 10:59 AM on September 3, 2008

Next up: a six-part expose on damned lies.
posted by cortex at 10:59 AM on September 3, 2008 [7 favorites]

Since people like myself enjoy this kind of stuff I was sure to find some enlightenment but now I'm about 78% sure I knew these things already. I also grew hungrier in the most recent 4.5 minutes, hence reading BBC consumes more calories than watching the Simpsons.
posted by Laotic at 11:01 AM on September 3, 2008

This is stuff any high-school grad should know and recognize, but sadly that's not true.
posted by MrBobaFett at 11:04 AM on September 3, 2008 [1 favorite]

ITT please post any research or analysis anyone has seen regarding sampling bias in surveys. The touted example, but still one on which I've seen very little real analysis, is that cell phone users are not reached, but there are certainly many more plausible and likely biases, e.g. from the article:

Who was surveyed? A cross-sample of the population. Good. How did they do the survey? By internet and text message. Not so good. How many replied? About half. Which half? Maybe those most likely already to have a big interest in self-harm?
posted by TheOnlyCoolTim at 11:09 AM on September 3, 2008

Statistics get a bad rap. People lie using statistics, people lie using words. A well-crafted set of statistics can elucidate an elegant truth. Great articles - but they should give examples of excellence in statistics as counterbalance.
posted by dances_with_sneetches at 11:12 AM on September 3, 2008 [1 favorite]

See also the startling correlation between the number of churches in a town and the number of bars.
posted by Skorgu at 11:15 AM on September 3, 2008

Judging by the comments on the first article: Everyone thinks this is a good idea. But, then Everyone say they already know this too. So, then it is useless to tell people what they already know and thus a bad idea.
posted by vacapinta at 11:17 AM on September 3, 2008

"Judging by the comments" should be a fallacy in itself.
posted by athenian at 11:20 AM on September 3, 2008 [1 favorite]

Nice, but it's journalists, not the public, who need this drilled into them. It's terrifying that people who can write things like "Surveys show most drivers think they're above average, which is statistically impossible...." are given space in national newspapers.

Also, this needs a Lesson 6: Counting Google hits and pretending it's news is bad and you should feel bad.
posted by enn at 11:28 AM on September 3, 2008

You've got to love the beeb, I just checked again now and "20 examples of grammar misuse" has climbed another notch from 4th most popular story to 3th.
posted by doobiedoo at 11:28 AM on September 3, 2008

Or maybe it's just zouhair at the mind control again.
posted by doobiedoo at 11:29 AM on September 3, 2008

Statistics get a bad rap. People lie using statistics, people lie using words. A well-crafted set of statistics can elucidate an elegant truth. Great articles - but they should give examples of excellence in statistics as counterbalance.

Yeah, I don't think it's meant to represent all of statistics, just to give news readers some basic tools for interpreting stats in the news using examples of stuff in the "bad" category that run the gamut from slight untruthfulness to outright misrepresentations. Most people reading news articles probably aren't too interested in examples of well-crafted statistics.
posted by Tehanu at 11:33 AM on September 3, 2008

I'm hoping it'll nudge up to 2th or (dare to dream) all the way to 1th.
posted by cortex at 11:33 AM on September 3, 2008 [6 favorites]

I gave my ice cream to a shark.
Now I've got nothing to eat.
posted by Dr-Baa at 11:43 AM on September 3, 2008

all the way to 1th

Do not be alarmed my child, your sins are forgiven.
posted by aramaic at 12:00 PM on September 3, 2008

Did anyone else follow the "Day in Pictures" link below the comments? I WANT GIGA PUDDING!
posted by owtytrof at 12:07 PM on September 3, 2008

I love to play poker with people who don't understand statistics. I take money from 88.8% of the people sitting at the poker table.
posted by Xoc at 12:16 PM on September 3, 2008 [1 favorite]

My favorite example of the nonintuitive nature of statistics:

A nuclear power plant goes into operation in Anytown, USA, providing most of the town's power. Ten years later, a rigorous statistical study finds a significant increase in cancer deaths that can be traced back to the same time and continues to this day. The margin of error is small, and all imaginable factors (health, wealth, etc.) have been controlled for in this study. The increase is most dramatic in the communities around the plant.

The natural conclusion is pretty clear: the nuclear plant is lethally irradiating the citizens and giving them cancer.

But the nuclear plant replaced a nearby coal-burning plant was decommissioned around the same time. The absence of coal-burning pollution meant that fewer people were dying from emphysema. Those folks were going to die of something else, obviously. And some of them died of cancer. Result: a rock solid correlation between nuclear power plants and cancer.
posted by Mapes at 12:32 PM on September 3, 2008 [18 favorites]

Articles are great. The comments, especially on the last one, are hysterical :-))
Thanks, Tehanu
posted by Goofyy at 12:40 PM on September 3, 2008

... all the way to 1th.

Finally, a word that rhymes with "month." Thanks, BBC!
posted by piers at 12:51 PM on September 3, 2008

84 % of these comments are utterly useless.

(Wow, I have ten toes! I'm almost average.)
posted by Dumsnill at 1:00 PM on September 3, 2008

49.3% of statistics are made up on the spot.
posted by kcds at 1:10 PM on September 3, 2008

8.5% of this comment is the figure on the left.
posted by cortex at 1:13 PM on September 3, 2008

It's terrifying that people who can write things like "Surveys show most drivers think they're above average, which is statistically impossible...." are given space in national newspapers.

If your head is in the oven and your feets are in the freezer, you should feel well on average :)
posted by elpapacito at 1:25 PM on September 3, 2008

A graphic designer and I were looking at an ad the other day, and I had some statistics for the product in question that showed an improvement from 3% to 9%.

"Great," he said, "I'll can make a graph that shows a 6% increase."

"No," I replied, "you can create a graph that shows a 300% improvement."

And I cackled madly as a cloud of pure evil coalesced over my head, squeezing my fingers, pretending to myself that I was choking math to death with my very hands.
posted by Shepherd at 1:39 PM on September 3, 2008 [3 favorites]

Mapes,
The margin of error is small, and all imaginable factors (health, wealth, etc.) have been controlled for in this study.

Doesn't the conclusion of your example violate this premise? If every imaginable factor has been controlled for, then surely atmospheric changes would be taken into account, to rule them out as a possible variable affecting the cancer rate. It seems odd to say that all variables have been controlled but then say, "Ah-ha, but there is one variable they didn't consider!"
posted by Sangermaine at 1:49 PM on September 3, 2008

"Because the 2004 increase in youth suicides coincided with strict government warnings about the use of antidepressants in teenagers and adolescents, some doctors suspect this may be the reason behind the spike."--from an article on an increase in teen suicides

I suspect it was the reelection of Bush.
posted by strangeleftydoublethink at 1:52 PM on September 3, 2008

It's terrifying that people who can write things like "Surveys show most drivers think they're above average, which is statistically impossible...."

I once wasted an entire day at work reading a fascinating article about how people can sometimes be shockingly bad at estimating how good they are at something, and as a result it should be no surprise that the average driver rates them self as above average...

link
posted by imh at 2:14 PM on September 3, 2008

Surveys show most drivers think they're above average, which is statistically impossible....

That's actually true in Lake Wobegon.
posted by kirkaracha at 2:30 PM on September 3, 2008

I did a survey of my myself, and it found that most quins favor BBC reporting over other sources.

I find that to be statistically significant.
posted by quin at 2:44 PM on September 3, 2008

And yet from the reader comments:

"Company A suffered a 100% increase in bad debt customers this year." This sounds terrible, but what if the company has 100m customers, had no customers with bad debt last year, and only one customer with bad debt this year? Still a 100% increase, but 1 in 100 million customers doesn't seem so bad after all!

FAIL.
posted by Durn Bronzefist at 3:11 PM on September 3, 2008

50% of Metafilter members are above fart jokes.
posted by cog_nate at 3:32 PM on September 3, 2008

45% are not above fart jokes. 3% are sockpuppets used entirely for fart jokes. 2% are robots who have no opinion on this issue.
posted by Tehanu at 3:50 PM on September 3, 2008

But, then Everyone say they already know this too.

Self-selected braggarts, obviously. Anyway, it's good to see a news source offering tools, so its readers can examine their news more critically.
posted by ersatz at 3:59 PM on September 3, 2008

"No," I replied, "you can create a graph that shows a 300% improvement."

And I cackled madly as a cloud of pure evil coalesced over my head, squeezing my fingers, pretending to myself that I was choking math to death with my very hands.

The difference between 3% and 9% shows a 200% improvement. I think math may have kicked you in the groin.
posted by oneirodynia at 4:16 PM on September 3, 2008 [3 favorites]

This isn't technically statistics-related but if I ever catch someone within kicking distance referring to the change between two discrete data points as an "exponential increase", I will - ooh, I can work in some probability here after all - I will with almost 100% certainty try to introduce another bifurcation at one of their primary branch points.
posted by Wolfdog at 4:28 PM on September 3, 2008 [3 favorites]

It seems odd to say that all variables have been controlled but then say, "Ah-ha, but there is one variable they didn't consider!"

Yes, it would seem odd. Except he said "all imaginable factors," not "all factors."

In other words, it's a failure of imagination to not recognize that any group of people will still die, no matter what happens, and just because they're dying as a result of Disease A instead of Disease B doesn't mean there's an unusual causative agent at work.

Look at it another way ... let's say I eradicated malaria in Africa by killing every single mosquito on the continent by hand, via armies of children armed with fly swatters. People will still die -- of cancer, of heart disease, of AIDS, etc -- but fly swatters will not have caused people to get cancer, heart disease and AIDS.
posted by Cool Papa Bell at 4:42 PM on September 3, 2008 [2 favorites]

45% are not above fart jokes

Of those, 79.6% are behind fart jokes.
posted by Sys Rq at 4:47 PM on September 3, 2008

I'm probably going to die.
posted by Dumsnill at 4:52 PM on September 3, 2008

Does anyone remember that website where you would enter everything you did, for all of time (color shirt worn, breakfast eaten, distance driven, etc), and then after a while, it would correlate the data with other user profiles, world news, and other sources? I could find out that a minor earthquake happened in Europe every time I had eggs for breakfast, or that my friend wore either blue shirts or yellow socks on any day that I woke up before 6am. All unrelated, of course, but obviously an entertaining site.
posted by niles at 11:39 PM on September 3, 2008

I'm going to make a fool of myself here, but what is actually wrong with the sentence "Surveys show most drivers think they're above average, which is statistically impossible."?
posted by Kiwi at 9:05 AM on September 4, 2008

The problem is that the statement assumes a perfectly normal distribution, where "normal" is a term of art describing a certain type of symmetrical data. Think bell curve: you've got a big bump in the middle and smooth tails running equally off to the left and the right.

And normal distributions—or, at least, nearly-normal distributions—are fairly common when you're looking at data, because a lot of systems tend to produce results along those lines. However, it's far from true that a normal distribution describes all system.

If you have a non-normal distribution, you likely have skew. Either gentle or extreme; the data might look like a drunken bell curve, or it might look like a great cliff on the left trailing off to the right, or it might look like a number of other things.

For example, if you have 1000 drivers, and 900 of them are reasonably competent and 100 of them are not, here's how it might break down. Assign a numeric competence to each driver; say a competent driver gets a 2, a sub-competent driver gets a 1. If you add up the competence scores of the thousand drivers (900*2 + 100*1 = 1900) and divide by the number of drivers (1900/1000 = 1.9), you have the average competence of a driver.

And, lo and behold, most drivers in our little study above are above average; they rate a 2 while the average is 1.9.

It's a contrived example, but hopefully it makes clear what's going on with that statement. There are simple measures such as median and mode that can help to better characterize skewed distributions where average falls on its face, as well as a slew of more advanced statistical techniques.

This butts up somewhat against the more colloquial notion of "average" as meaning just fairly typical, so there's a fair amount of maybe over-stated ire to be found re: people saying "average" when they don't mean to invoke any statistical truth; but there's also a lot of blatant misuse and misunderstanding in the context of folks actually trying to state truth or convey information, so it rankles to see e.g. journalists misuing the term as often as happens.
posted by cortex at 9:20 AM on September 4, 2008 [1 favorite]

Yes, of course. Thanks for that - my 15 year-old-self would have been pretty appalled by that question.
posted by Kiwi at 9:48 AM on September 4, 2008

Journalists and percentages mix like ball bearings in souffle.

LOVE IT
posted by DLWM at 2:22 PM on September 4, 2008

Part 6 has been posted.
posted by Eideteker at 5:13 AM on September 27, 2008