Fake Science Journalism
May 27, 2015 5:59 PM   Subscribe

“Slim by Chocolate!” the headlines blared. I got a call in December last year from a German television reporter named Peter Onneken. He and his collaborator Diana Löbl were working on a documentary film about the junk-science diet industry. They wanted me to help demonstrate just how easy it is to turn bad science into the big headlines behind diet fads. And Onneken wanted to do it gonzo style: Reveal the corruption of the diet research-media complex by taking part.
posted by contrarian (44 comments total) 31 users marked this as a favorite
 
Most in positions of authority in news media are imbeciles in relation to what they are reporting.
posted by Jessica Savitch's Coke Spoon at 6:19 PM on May 27, 2015 [2 favorites]


This is fantastic. At a recent family gathering I had to restrain myself from yelling VEGETABLES ARE NOT A MAGIC GLUCOSE NEUTRALIZER at an in-law promoting kale-mango smoothies to a Type 2 diabetic.

People need to learn that the diet and supplement industry are just as bad as the junk food industry, and usually the same companies are playing both sides of the field.
posted by benzenedream at 6:26 PM on May 27, 2015 [9 favorites]


When the Sokal hoax happened, it was funny because ha ha postmodern quantum phenomenology what even is that. Now that so many journals are "pay to play," the point Bohannon is making has (I hope) more and sharper teeth. But you don't even need the basic education in statistics that he's recommending - just do what academics have been doing forever and discount things published in journals with abysmal Impact Factors.

Of course, if riling up people's weight insecurities wasn't such big business, more than just hapless "science" "journalists" would be out of a job.
posted by katya.lysander at 6:29 PM on May 27, 2015 [1 favorite]


I totally agree with this guy's larger point, but . . . why was this a good thing to do? When messages catch in the real world, you can't just undo them by recanting. These ideas are sticky, stickier when they fit what people want to hear, and hard to overcome once we believe them. True, encouraging dieters to eat chocolate is far less harmful than telling people that vaccines cause autism, but it still seems mean. We KNOW it's possible to sell snake-oil, and we also know it's brutally hard to get snake-oil users to stop buying it once they're "hooked", even when trusted sources tell them to stop.

Even worse, it seems like the whole thing was done for a TV documentary. Were there ad revenues involved? How is that less obnoxious than the click-bait and inflammatory journalism he's supposedly fighting against?
posted by synapse at 7:06 PM on May 27, 2015 [13 favorites]


If you consider yourself a journalist, and you intentionally bring more lies into the world, you are doing it wrong.
posted by Horace Rumpole at 7:30 PM on May 27, 2015 [9 favorites]


What if your intent is to use your lies to illuminate an even greater truth?
posted by ZenMasterThis at 8:10 PM on May 27, 2015 [4 favorites]


I would bet the ratio of the people who read this debunking to those now walking around with "chocolate makes you skinny" in their heads is minuscule.
posted by Horace Rumpole at 8:18 PM on May 27, 2015 [1 favorite]


If you consider yourself a journalist, and you intentionally bring more lies into the world, you are doing it wrong.

If science journalism were better, there'd have been no need for this as a corrective action. It was ethical of them to expose that the process is partially broken.
posted by sebastienbailard at 8:27 PM on May 27, 2015 [7 favorites]


What if your intent by inventing the Gatling gun is to make war obsolete?

How many people have heard that you swallow eight spiders a year? How many people have heard that's an urban legend? How many people know it was deliberately spread to show how gullible people can be? People have a hard time remembering which part of a debunking was the bunk—not to mean we shouldn't try but that it's always perilous, and deliberately creating new bunk just to later pull the curtain back and say "ah ha!" to make a grand statement is playing with fire.
posted by traveler_ at 8:29 PM on May 27, 2015 [18 favorites]


Maybe these gonzo geniuses should've consulted the research on how difficult it is to correct misinformation before giving more of it a platform.
posted by aaronetc at 8:49 PM on May 27, 2015 [7 favorites]


I want to send this to my colleagues in my academic department but I am too scared that they will think I am advocating dietary advice and switch to a chocolate diet.

More seriously, I totally agree that it is at least irresponsible to pull a trick like this, given that we know how hard it is to correct misinformation. Apart from anything else, this is a well-written 'how to' guide for any malicious parties, in pretty much the same vein as cleaning up scripts to make them accessible for script kiddies.

Fascinating article. Real problem. Great point. Very bad idea to actually do.
posted by nfalkner at 9:21 PM on May 27, 2015 [3 favorites]


True story: The year I lost 50+ pounds in five months, I had a single-serving vending machine pack of M&Ms every blessed day (minus the blue ones). This was YEARS before this hoax came out.

I think having access to a small amount of indulgence per day genuinely helped me combat my normal urge to go after my usual, bigger calorie treats.
posted by mochapickle at 9:21 PM on May 27, 2015 [4 favorites]


I wonder if he would have been able to make his point if he hadn't actually done it. He notes that he himself believed science journalists would be too sharp to let it slide by.
posted by frumiousb at 9:27 PM on May 27, 2015 [1 favorite]


(Not that this cures the ethics concerns, I just think most reputable papers would have dismissed this as impossible if the only the vulnerability had been outlined.)
posted by frumiousb at 9:28 PM on May 27, 2015 [1 favorite]


At the very least, he can fall back on blogging for HuffPo under another pseudonym.
posted by Halloween Jack at 9:58 PM on May 27, 2015


Yeah, I would be a lot less conflicted about this whole thing if it hadn't been very well demonstrated through a number of studies that, when you tell someone "It is NOT TRUE that [interesting lie]," the long-term result is that all they remember is the "[interesting lie]" part. The end result is that, while the person behind this got to have their "aha! suckers!" moment, they are doing so by actively undermining their own long-term goals of societal scientific literacy.
posted by DoctorFedora at 10:14 PM on May 27, 2015 [3 favorites]


I assume the target audience the author intends is editors and media owners who may be embarrassed and (hopefully) impose some degree of fact checking on future stories, not someone who is already heavily into apple cider vinegar as a cure-all.

Societal scientific literacy, while an admirable goal, is a much larger long term project than preventing shit studies from getting parroted across all media.

Apart from anything else, this is a well-written 'how to' guide for any malicious parties,
...
I called a friend of a friend who works in scientific PR. She walked me through some of the dirty tricks for grabbing headlines.

The fact that there are people whose entire careers are devoted to this kind of crap indicates the malicious parties (i.e., the fitness and food industries) are not going to learn anything from this article. These are all old tricks for anyone involved in diet / supplement marketing.
posted by benzenedream at 12:07 AM on May 28, 2015


I suppose from a journalist's point of view it really doesn't matter if the claim is true so long as it's attributable.

You'll never (short of some terrible dictatorship) create an environment in which credulity is safe because all lies have been eliminated, so it's perhaps better to deluge people in falsity to help them learn to separate the wheat from the chaff (not a dietary message, chaff may have dietary benefits, etc)
posted by Segundus at 12:18 AM on May 28, 2015


I used to work for a publisher that produced several health and fitness magazines. Though none of the journalists who worked on those titles had science backgrounds, they all knew most of the studies they were printing were garbage -- they just didn't really care. They were overworked and underpaid and they had space to fill and a limited budget to do it with. True, the magazines were second-rate, but the number of newsrooms out there with large staffs and specialized reporters (with finance or science backgrounds, for instance) and fact-checkers are few and far between these days.

I think many of the writers who covered this fake story would be neither embarrassed nor surprised to learn the truth -- they see "chocolate is actually good for you!" articles as meaningless fluff like dubious "wacky" stories from overseas and anonymous celebrity gossip.
posted by retrograde at 12:20 AM on May 28, 2015


It's unfortunate, and interesting in a way, that this whole article is about bad nutrition science, and then the author quotes Attia, as if he is some kind of expert on the subject and not a big cause of misinformation about diet and health. The quote alone should have raised questions. “They spent $1 billion and couldn’t even prove that a low-fat diet is better or worse.” Because that was never the goal of the Women's Health Study! Why would it have to prove that? What is it supposed to say that it did not prove some random issue?

Attia is a surgeon (that is to say, he went to medical school, he quit medicine after his fellowship), who has not had any meaningful scientific training about nutrition at all (ironically, he learned all he knows about nutrition from a science journalist).

I now actually wonder if Bohannon included Attia on purpose and this whole debunking article is just part of the scam, and we will get a new article where he shows how nobody is critical about "experts" you cite as long as they have an MD or something similar that sounds expert-y
posted by blub at 1:12 AM on May 28, 2015 [1 favorite]


Here’s a dirty little science secret: If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result.

This not a "science secret." This is a basic misunderstanding of statistics and the importance of sample size.

"The sample size is chosen to maximise the chance of uncovering a specific mean difference, which is also statistically significant. Please note that specific difference and statistically significant are two quite different ideas.

The specific difference is chosen by the researcher in terms of the outcome measure of the experiment. For instance, 3kg mean weight change in a diet experiment, 10% mean improvement in a teaching method experiment.

Statistical significance is a probability statement telling us how likely it is that the observed difference was due to chance only.

The reason larger samples increase your chance of significance is because they more reliably reflect the population mean."
posted by three blind mice at 1:13 AM on May 28, 2015 [3 favorites]


I do not think that this exercise establishes what the author thinks it does.

Without question the journalists who fell for it should not have done so – and if they didn't even care one way or the other about its truth then that's even worse. But I've always thought (even with Sokal, to be honest) that you build the weakest possible case against someone's unprofessionalism when you deliberately, and with careful planning, set out to cause them to believe untrue things. So the author here changes his name, invents an affiliation, picks a conclusion tailor-made to lure his targets, and thinks of numerous other ways to achieve his goal of getting false conclusions into the media… and succeeds.

This doesn't show that there is a crisis of terrible science coverage (even though I am very open to the argument that there is such a crisis). It just shows that con tricks work on some people.
posted by oliverburkeman at 4:52 AM on May 28, 2015 [1 favorite]


The new publisher’s CEO, Carlos Vasquez, emailed Johannes to let him know that we had produced an “outstanding manuscript,” and that for just 600 Euros it “could be accepted directly in our premier journal.”

aye aye aye aye aye aye aye
posted by bukvich at 5:18 AM on May 28, 2015


> "This doesn't show that there is a crisis of terrible science coverage (even though I am very open to the argument that there is such a crisis). It just shows that con tricks work on some people."

I'd buy this more if it weren't a "con trick" that would have been exposed using just about the lowest possible standard of actual journalism.

Question 1: Was this study published in a reputable journal? Answer: No.
Question 2: Was this a large study? Medium sized? Small? Answer: Ludicrously tiny.

Red flags should have gone up before you even get to, "Read the study and, if you have an ounce of scientific literacy, notice massive methodological problems."
posted by kyrademon at 5:48 AM on May 28, 2015 [4 favorites]


The trouble with debunking is that it takes a lot more time and effort than to just run the press release, when you're done you don't have a story, and people will believe what they want to believe. Also, a lot of the scammers tend to fight dirty - and how much lawyerly fun do you want to have today?

So, those journals who do have an ethical approach to reporting science/tech/medical stories tend to just ignore the guff: there's just so much of it. If it's particularly dangerous (the anti-vax stuff) or particularly illustrative of real science (perpetual motion/free energy) then there may be good reasons to engage, but mostly - why give them the publicity?

I've seen a number of previously solid online sci-tech sites give up on being good and going for the clickbait, and that always saddens me, especially when they try to justify reprinting the press release by adding 'actual science' that just shows the writer doesn't know what they're talking about and hasn't had the time to do a five minute Google. But I know why they do it: if you don't, you risk falling off the edge of the world. I still feel most strongly that this is no good in the long term for anyone, but I've been fighting this war for decades now and if I were a fictional character, I'd be the Black Knight in Holy Grail.

Now, if Snopes had a newswire...
posted by Devonian at 5:55 AM on May 28, 2015 [1 favorite]


Red flags should have gone up before you even get to, "Read the study and, if you have an ounce of scientific literacy, notice massive methodological problems."

Agreed. I just mean: every profession has at least a few total idiots, and if you design your prank like a heatseeking missile to find and deceive exactly those idiots, you have not generated any evidence that idiocy is in fact unusually widespread in that profession.
posted by oliverburkeman at 5:56 AM on May 28, 2015 [2 favorites]


@three blind mice: I think you misread what he was saying. "Measuring a large number of things" means multiple endpoints. The small number of people gives you good odds that your samples are anomalous in some way--as you point out, they won't reliably reflect the population mean. It is a guarantee to find something somewhere. The samples just need to be different by one person to 'hit' on one of those many analyses around the 18 endpoints.

Your quote discusses the odds of finding a true effect when one exists, but in this case they were trying to do the opposite. They couldn't have teased out a weak signal--and might have missed a strong one due to rotten luck--but fake ones show up no problem.
posted by mark k at 7:15 AM on May 28, 2015 [1 favorite]


You'll never (short of some terrible dictatorship) create an environment in which credulity is safe because all lies have been eliminated

What would be so terrible about that?
posted by Faint of Butt at 7:15 AM on May 28, 2015


Oh my god what is the deal with people spelling "psych" "sike"?
posted by fiercecupcake at 7:17 AM on May 28, 2015 [6 favorites]


"This not a 'science secret.' This is a basic misunderstanding of statistics and the importance of sample size."

No, you misunderstood what he wrote. He wasn't talking about sample size, he was talking about p-hacking -- just fishing for a statistically significant result among many possibilities.

And p-hacking -- especially unconscious varieties -- is a big problem, as he says.

There's nothing going on in this hoax that isn't common everywhere. The high-impact journals have the same problems. From outright fraud, to p-hacking and widespread misunderstanding and misuse of statistical significance, to the academic PR machine and a credulous press, to the strong bias toward interesting, positive results and the widespread neglect of replication -- these are systemic issues that the nutrition industry exploits but which are pretty much universal.
posted by Ivan Fyodorovich at 8:03 AM on May 28, 2015 [1 favorite]


How many people have heard that you swallow eight spiders a year? How many people have heard that's an urban legend? How many people know it was deliberately spread to show how gullible people can be?

Well I see that somebody has bought into the lies being spread by Big Spider.
posted by Shepherd at 8:42 AM on May 28, 2015 [8 favorites]


The Washington Post article on this story mentioned "Only one journalist, a reporter from Ohio, questioned Bohannon about the institute and his sample size." I wish Bohannan had included this in the io9 article. Journalists and media outlets that are doing their job deserve to be credited.

> every profession has at least a few total idiots, and if you design your prank like a heatseeking missile to find and deceive exactly those idiots, you have not generated any evidence that idiocy is in fact unusually widespread in that profession.

I think most news outlets don't take science journalism very seriously, and that a lot of outlets took the bait uncritically does indicate that there's a problem. But you've definitely got a point: we don't know how many health news reporters silently passed on this story for obvious reasons. Bohannan mentions that most articles about this story only took information from the press release, not the journal article. It's quite possible most journalists who did take a look at the article decided not to run the story. (Unprofessionally written article that omits critical information included in standard research articles, published by an outfit that's not a real medical journal, press release from a sketchy "institute" that may not exist, and the lead author may not exist either – there are a lot of reasons for even sloppy science reporters to say "nope" on this one, without even considering the p values and the effect size. The documentary team only caught the lowest hanging fruit of bad science journalism here.)
posted by nangar at 9:31 AM on May 28, 2015


Oh my god what is the deal with people spelling "psych" "sike"?

IT IS THE WORST FUCKING THING

posted by poffin boffin at 12:51 PM on May 28, 2015 [5 favorites]


I thought most of the people who spelled "psych" "sike" were sharpie-wielding record store clerks who want to make the little section of the store where they put the Bardo Pond CDs look more enticing?
posted by Sonny Jim at 1:00 PM on May 28, 2015


guys does anyone want to join me on the "it is spelled 'whoa' and not 'woah'" hill to die on
posted by DoctorFedora at 4:06 PM on May 28, 2015 [9 favorites]


You have my blue pencil!
posted by Etrigan at 4:19 PM on May 28, 2015 [2 favorites]


And my axe! (Not my ax, because that spelling is terrible!)
posted by Faint of Butt at 5:05 PM on May 28, 2015 [2 favorites]


Whether this is Ioannidis for Dummies or actually about ethics in science journalism should be the subject of a complicated study with a veneer of rigour but a soft underbelly of wishful thinking.

I volunteer to spell-check and to do the margin doodles.
posted by Construction Concern at 6:41 PM on May 28, 2015


I've been on that hill for a while, DoctorFedora.
posted by bashos_frog at 10:35 PM on May 28, 2015


DoctorFedora: guys does anyone want to join me on the "it is spelled 'whoa' and not 'woah'" hill to die on

From what I can tell, since Middle English this word has been variously (and never consistently) spelled:

ho hoo hoa hoe wo woa woah way whoo who whoe whoh whoo whoa

There's a lot of hills full of corpses around yours I'm just sayin.
posted by traveler_ at 1:51 AM on May 29, 2015 [2 favorites]




If we really want to go and fix illusions of truth, making the letters in a Wikipedia entry proportionally darker when they've been there longer and lighter for shorter will go some way to defeating the total illusion that you are seeing a static and complete thing.

Yes, I realise that most of the people here are aware that it's a patchwork thing, but many others see it like a single white page of text, the illusory integrity of the printed page.

I had a student code up a browser plug-in that made controversial and repeatedly edited sections change colour. Some pages would have killed a chameleon at 50 paces.
posted by nfalkner at 4:24 AM on May 29, 2015 [3 favorites]


I was so furious when I read someone write "psych" as "sike" in a 1993 high school yearbook and guys that was over half my life ago so I'm not sure what to do with all this context
posted by rum-soaked space hobo at 5:11 AM on May 29, 2015 [1 favorite]


How many people have heard that you swallow eight spiders a year?

Yes, and I'm sure everyone thinks I'm soooo weird.
posted by krinklyfig at 10:49 PM on May 29, 2015


« Older Additional props are potato chips, pickles and...   |   Swords Into Plowshares: Military veterans turn to... Newer »


This thread has been archived and is closed to new comments