Not reliably different from zero

October 26, 2018 7:23 AM

"The story of intranasal oxytocin (previously) and trust is more than sad; it’s a scientific tragedy. ... Some still dispute this, but it seems likely that the very large literature claiming behavioral effects of intranasal oxytocin on human behavior is completely and totally spurious. It’s been a colossal waste of money and time. It gave false hope to those with autism and needlessly harmed clinical trial participants. And the nightmare drags on as oxytocin->trust is *still* being cited and marketed as well-established science."

Kosfeld et al. (2005) [PDF] spurred a wave of research on intranasal oxytocin and human trust, but most of it was conducted with insufficient sample sizes. Despite this, nearly all published results were statistically significant, a highly implausible state of affairs. ...

As the published literature became more rife with seemingly positive results, it became nearly impossible to challenge the idea of oxytocin improving trust. One lab which had reported an initial set of ‘successful’ experiments had manuscripts reporting subsequent failures repeatedly rejected on the basis that the effect was now well established. This lab managed to publish just 39% of all the work it had conducted, all of it suggesting positive effects. Pooling over *all* their data, however, suggested little-to-no effect (see Lane et al., 2016 [PDF]).

From this manufactured certainty, numerous clinical trials have been launched trying to improve social function in children with autism through intranasal oxytocin. These trials have not yet yielded strong evidence of a benefit. Unfortunately, almost 40% of the 261 children so far treated with oxytocin have suffered an adverse event (DeMayo et al. 2017 [paywalled]; compared with only 12% of the 170 children assigned a placebo). Thankfully, most (93) of these adverse events were mild; but 6 were moderate, and 3 severe.

Here’s the kicker for this story: oxytocin delivered through the nose is probably inert in terms of brain function [PDF]. ...

I have a post-mortem of this field in press [embedded PDF] at The American Statistician.

The Lane et al. paper [PDF] is a remarkable document, part mea culpa and part indictment of a broken system:

Is there a file drawer problem in IN-OT research? If this is the case, it may also be the case in our laboratory. ... We submitted several studies yielding null-findings to different journals (from general interest in psychology to specialized in biological psychology and in psychoenodcrinolgy) but they were rejected time and time again. After realizing that our publication portfolio has become less and less representative of our actual findings, and because the non-publication of our null-findings might contribute to generating a publication bias in IN-OT research, we decided to get these studies out of our drawer, hoping that other laboratories will do the same. ...

As reported in the meta-analytic section, the aggregated effects are not reliably different from zero, regardless of how they have been pooled (by dependent variables, by theories or altogether). Our initial enthusiasm on IN-OT findings has slowly faded away over the years and the studies have turned us from “believers” into “skeptics”.

posted by clawsoon (9 comments total)

Great post. As an academic that works in a variety of fields, the one constant I can see is that people really do love just-so stories. I really can't stand them, even when boiled down into a simpler form that is ostensibly an empirical question.
posted by lilies.lilies at 7:32 AM on October 26, 2018 [7 favorites]

Also: the amount of research that continues to be done that aims to "fix" autistic people is utterly staggering.
posted by lilies.lilies at 7:34 AM on October 26, 2018 [14 favorites]

A quick glance at the 2017 meta-review paper (DeMayo et.al) says that the statement "...needlessly harmed clinical trial participants." is, perhaps, somewhat overblown. Adverse event reporting for clinical trials is ferociously strict, and should be, but none of the studies DeMayo and colleagues review seem to report meaningful numbers of significant adverse events versus placebo. So, waving the flag of "this research has harmed participants" seems at odds with the actual data.

What it suggests to me is that it's an ineffectual intervention, which, at a broad level, is why we do clinical trials. And why participants take part in them: there's no guarantee it'll work, but it might, and people are willing to try. A null result, which seems to be what we're really seeing in this literature, is, frankly, a pretty good outcome from a scientific and medical knowledge point of view. Far better a null result and no meaningful side effects (because it doesn't seem to do squat intranasally) than, say, intolerable side effects and lasting harm.
posted by Making You Bored For Science at 7:48 AM on October 26, 2018 [7 favorites]

I think the point is that the trials continued because of the bias to publish only "positive" results and despite the fact that researchers tried to report their null findings. It is problematic if ANY participant was indeed harmed during a trial that occurred after the evidence SHOULD have suggested that there was nothing more to see here. (I don't know if that is the case here.)
posted by eabomo at 8:35 AM on October 26, 2018 [1 favorite]

Making You Bored For Science: the statement "...needlessly harmed clinical trial participants." is, perhaps, somewhat overblown.

Agreed. I felt a bit uncomfortable with that, and perhaps shouldn't have included it in the top-line framing.

A null result, which seems to be what we're really seeing in this literature

What I found most interesting/important about this is that we didn't see null results in the literature for over a decade, despite scientists seeing null results in their labs. Clinical trials were mostly harmless, but they were also a needless waste given what the literature should have shown.
posted by clawsoon at 8:39 AM on October 26, 2018 [4 favorites]

Joined just to add to comments here since I’m in the field:

There is evidence that OT(nebulized at least) increases OT concentrations in Csf.

I wouldn’t completely lose hope in this field just because it is hard to find impact on high level behaviors. OT effects aren’t like taking mdma, it’s a weak neuromodulator. Most people in the field don’t try to sell it as an easy cure - but I know that has been in the media....

The story is more complicated than “wow it’s a miracle cure” or “wow it was all a hoax”
posted by jessicattiva at 11:47 AM on October 26, 2018 [2 favorites]

For what it's worth, back in my IRB days we had a researcher who ran these sort of intranasal oxytocin studies. His protocols usually looked good on paper, but were routinely plagued with deviations in enrollment and executions. Nothing major, but a general sloppiness which usually got pinned on whatever grad student was actually running the day to day activities, who would then be "retrained" as part of the corrective action plan, though in reality they would move on to another project, to be replaced by the next upcoming PhD candidate.

Regardless of the other issues brought up by the paper, it's easy to see how that kind of lack of rigor in carrying out study protocols can have an effect on skewing results. Particularly with something like oxytocin which may have only marginal effects, even small deviations could lead to garbage data.
posted by Panjandrum at 4:01 PM on October 26, 2018 [2 favorites]

Yeah, back in 2007 I read the literature available at the time, but it seemed off to me. I even went so far as to contact the old "liquid trust" company which claimed to be selling a product that used this effect, but it turned out they were just scammers, as their product was a "homeopathic" one, i.e. didn't actually contain any oxytocin (unsurprising as oxytocin is a prescription only drug, and I wondered how they could sell it without a prescription.) I was hoping to get some data from them, maybe even buy some and do a double-blind test, but you can't polish a turd. And since the reported effect is apparently bogus, it would have been a waste of time and money.
posted by Blackanvil at 9:50 PM on October 27, 2018

I found this bit interesting, too: "...most of it was conducted with insufficient sample sizes. Despite this, nearly all published results were statistically significant, a highly implausible state of affairs."

I've recently learned just enough about statistics to know what this means: If you're doing a whole bunch of n=30 studies where the real effect size is moderate, your p-values should jump around like crazy. If they aren't, it means that a whole bunch of studies aren't being published.
posted by clawsoon at 6:37 AM on October 28, 2018 [1 favorite]