Retraction Isn’t Enough

May 9, 2024 7:04 AM

“the conclusions of this paper were disseminated to over 5 million people and less than 0.02% of them actually read the full text or the retraction notice. The result is roughly 5 million misinformed people”. What is ‘evidence-based’?

The unwritten context here is that Huberman Lab episode 114 discussed this study’s conclusions at length, without noting its obvious shortcomings, before its retraction.

posted by bq (28 comments total)

tl;dr

(too lucrative, didn't retract)
posted by lalochezia at 7:06 AM on May 9 [25 favorites]

The mass hordes of humanity will only remain misinformed until the next (perhaps equally poor) paper comes out. Spending time making sure that they get this retraction has a certain purity to it, but I don’t think it’s necessary.
posted by Tell Me No Lies at 7:14 AM on May 9 [1 favorite]

We need better scientific literacy. Like, never base any decision solely on the results of one paper.
posted by MisantropicPainforest at 7:24 AM on May 9 [3 favorites]

I don't understand the note about Huberman lab? I know that's a science podcast with a host who had some sort of being horrible to women scandal. Is it just mentioned here because it was a driving force popularizing this later retracted finding?
posted by Wretch729 at 7:25 AM on May 9 [2 favorites]

Is Huberman lab a big enough podcast that I should be as aware of it as I am of, I dunno, NPR's Science Friday? I'd never heard of it before googling it just now. This might just be a weird lacuna in my pop science knowledge.
posted by Wretch729 at 7:28 AM on May 9

So this study basically came to the conclusion that cold showers have meaningful health benefits, but it turns out the study can't actually prove that?
posted by grumpybear69 at 7:33 AM on May 9

This kind of thing is why I side-eye a ton of "science communicators". There are some very good journalists out there, but there are also a ton of folks who are just running social media accounts on "science is COOL" vibes, and who will cheerfully boost the results of any paper that excites them. Whether it passes any kind of reasonableness check or not.

Any single study is just a data point in a wider conversation within a scientific field. A ton of them will have minor issues because that's just how doing experimental work goes. Retractions are unfortunate, but they should be encouraged when mistakes are discovered.

In my Ideal World (TM), the results of any single study would never get publicized wider than that field's professional press. Journalists and others who write for the public should be reading meta-analyses -- or even synthesizing the results of multiple studies themselves! (Tons of good science journalists with PhDs who can do that work.) They shouldn't be broadcasting a claim supported by one research lab, even if it sounds neat.
posted by learning from frequent failure at 7:39 AM on May 9 [19 favorites]

The posted article tried to determine the extent of the exposure to the content of this paper. I remember hearing on the local news awhile back about the health benefits of cold showers etc. So what about all the people who heard this reportage who don’t even know about abstracts and journal articles? The news media frequently reports on “scientific studies.” But I rarely hear about retractions there.
posted by njohnson23 at 7:57 AM on May 9 [2 favorites]

For some, this is a feature, not a bug: The Daily Mail’s List Of Things That Give You Cancer: From A To Z
posted by slimepuppy at 8:24 AM on May 9 [3 favorites]

Huberman Lab is (AFAIK) currently the most popular fitness podcast and he trades heavily on his scientific credentials.

Previously:

https://www.metafilter.com/200530/Andrew-Huberman-Rockstar-Neuroscientist
https://www.metafilter.com/203071/We-may-live-in-chaos-but-there-are-mechanisms-of-control
posted by bq at 8:28 AM on May 9 [6 favorites]

I am still thinking about breathless coverage I saw of a couple small studies that were presented in what I understand to be responsible popular science outlets like Radiolab or the Nature magazine podcast.

Radiolab did two separate stories about one researcher's work on gamma frequency for Alzheimer's in a tone that really de-emphasized the context of how mouse studies have so frequently proved inapplicable to real human clinical outcomes in this field. I got really excited about this myself after the initial story (also covered widely in the popular science press) and learned ultimately that the researcher is hawking a product and there is now evidence that flashing lights are not likely to be an effective therapy).

Another example was a ton of coverage of some research on pheremone bedbug traps. This really caught my imagination after I initially heard about it on the Nature Magazine podcast.

Several years later this research seems to have resulted in a product that is not especially more effective than existing bedbug traps - useful for serveilance of your bedbug population but not eradication, but of course there is no news coverage of "this research did not pan out" or "was less effective in real life usage than in the lab".

Anyway, all this is to say that I agree with the need for better literacy on how we understand research, and popular science outlets like Radiolab should take an active role in that education!
posted by latkes at 8:54 AM on May 9 [5 favorites]

The Daily Mail’s List Of Things That Give You Cancer: From A To Z

Here, some music to listen to while you read that list. ("Music" is thankfully not on it)
posted by EmpressCallipygos at 8:57 AM on May 9 [1 favorite]

I appreciate the idea that, instead of evidence-based and not evidence-based, we should have a spectrum about how much evidence we've got.
posted by Nancy Lebovitz at 8:58 AM on May 9 [8 favorites]

I need a cold shower after reading this.
posted by chavenet at 8:59 AM on May 9 [4 favorites]

There's no way even an intelligent, highly educated reader can "do your own research" on more than a tiny fraction of topics. That's why we have to rely on others to not only do the raw research, but review and summarize it. The problem is that a lot of those reviewers are crappy - not only the ones you'd expect, like bro podcasters, but even organizations I used to consider reliable, like Consumer Reports, for fucks sake. (A couple of health sources I still trust: U.S. Preventive Services Task Force and Cochrane.)
posted by Mr.Know-it-some at 9:24 AM on May 9 [6 favorites]

god I love Trex and MASS and this whole ecosystem of science communicators

I feel like this is the only science-based podcast that my partner, MPH MD, is okay with me listening to and that's only because they occasionally will say anti-capitalist things and have also, on numerous occasions, spent more than hour talking about the hierarchy of evidence, the importance of methodology, trends in open vs private publishers, etc

RIP SBS of yesteryear
posted by paimapi at 10:21 AM on May 9 [1 favorite]

Sigh. I miss it. The new co-hosts aren’t the same, although i did appreciate the 6 hour deep dive into aspartame.

Sigma Nutrition is also excellent.
posted by bq at 10:38 AM on May 9

If you are a lifter AND sciencey some of the same folks (Erics Helms and Trexler) are on the Iron Culture podcast too. The MASS Office Hours podcast is fantastic.
posted by i_am_joe's_spleen at 12:35 PM on May 9

"Impact Of Cold Exposure On Life Satisfaction And Physical Composition Of Soldiers" is quite a title.
posted by doctornemo at 1:00 PM on May 9

Oog. My trainer at the gym has pushed a bit of Huberman Lab in my direction, and is himself really into the cold-water-immersion thing (maybe a causal relationship there, I dunno).

I couldn’t get into the podcast: Huberman embodies a certain reckless grandiosity that is kind of a stock character archetype in the Bay Area. I have encountered enough examples of that archetype that Huberman tripped all my bullshit alarms, and that the later emergence of his icky personal business is in the running for least shocking thing I’ve heard this year.

I told my trainer as much when the article about the gross behavior came out. Now I feel like it’d be irresponsible of me not to share this with him as well, except it’s sort of starting to feel like gratuitous piling on now.
posted by gelfin at 3:47 PM on May 9

Ever since the Wansink revelations I’ve trusted my bullshit detectors 100%. I knew his books sounded like bullshit but I conceived myself that I was wrong bc he was at an Ivy. Trust your BS detectors.
posted by bq at 4:09 PM on May 9 [1 favorite]

I put Huberman and Dr. Oz in the same class: They use their science credentials to build trust and to provide just enough plausible-sounding sciency-ness to make them highly effective at selling supplements (where the real money is).
posted by antinomia at 7:06 PM on May 9 [4 favorites]

Half were randomly assigned to the control condition (no intervention), while the other half were randomly assigned to the cold water intervention. The intervention began with a four-hour educational session about the benefits of cold exposure.

Not a good start for a study relying so heavily on self-report subjective outcome measures.
posted by Pouteria at 10:07 PM on May 9 [5 favorites]

17 months into using the Wim Hof method daily after irregularly dabbling with it for a few years, it is absolutely clear that it has improved both my physical and my mental health a great deal. Despite avoiding the broish "ice-bath culture" that is springing up with the ridiculously priced automated ice baths and similar crap, I am surrounded by quite a bit of anecdotal evidence as well. So not very evidence based, but it really works for me. Both Huberman and Hof have a tendency to overestimate their knowledge about human physiology, and I find it particularly funny that old Wim seems completely blind to the extent to which his constant cheer and being "high on his own supply" is a consequence of a few million people following him like a god - and yet the cold-immersion itself, coupled with the simple breathing exercise (of course, never at the same time!!), has helped me beat a bipolarity I struggled with for more than 30 years.

Here's a meta-study of 8 papers - they found that "the most prominent changes seen in people practising the Wim Hof method were an increase in the stress hormone adrenaline and anti-inflammatory chemicals called cytokines, and a reduction in pro-inflammatory cytokines."

Which of course takes nothing away from the moral of the FPP article... but failure to prove is not a disproof, either.
posted by holist at 1:02 AM on May 11 [1 favorite]

Casey Johnston has some interesting writing on the "it worked for me" situation.
posted by The corpse in the library at 10:01 PM on May 11 [1 favorite]

Thank you!
posted by holist at 2:37 AM on May 12

even organizations I used to consider reliable, like Consumer Reports, for fucks sake.

Classic Consumer Reports, as opposed to in house tests/research, always had the selection bias problem that the consumer reports were submitted by the kind of people that subscribe and submit to Consumer Reports. So it strongly selected for certain socio-economic classes and use cases.

Their recommendations always had to be viewed through that lenses.
posted by Mitheral at 6:33 AM on May 14

Consumer reports has a large staff of testers who subject items to stress tests and comparisons at a facility in Westchester ny. To my knowledge they do not accept reviews from the public. It’s for consumers, not by consumers.
posted by bq at 8:09 AM on May 14 [1 favorite]