The Ouroboros of Scientific Evidence
February 13, 2015 9:29 AM   Subscribe

"Do whatever it takes to not fool yourself, period, that's the scientific method" - Neil deGrasse Tyson. What if we can't do that?
posted by LiteS (35 comments total) 26 users marked this as a favorite
 
I think you meant "the ouroboros of evidence in psychology."
posted by bdc34 at 9:39 AM on February 13, 2015 [5 favorites]


How has this not been made into a buddy comedy yet

Seriously, I want to read this paper where they just accuse each other of sabotage.
posted by tofu_crouton at 9:49 AM on February 13, 2015 [5 favorites]


The article starts off talking about the problem of false positives, i.e. the issue that one will sometimes detect an effect that doesn't really exist, just by random chance. This is pretty old news in the sciences; there is a huge suite of statistical tools, starting with the humble p-value, designed to address exactly this problem. Their use is utterly routine in the sciences, and any publishable research is definitely going to use some of them. A researcher has not only to detect an effect, but also to quantify precisely what the odds are that this effect was a chimera. This is what is meant by whether a difference is "significant" or not.

Now, that doesn't address the meta-issue that not all published science actually stands up to all of the critiques and questions that one might want to ask of it. Experimental designs are often flawed, statistics may be used inappropriately, data may not be fully reported, and conclusions are sometimes oversold. That doesn't mean that scientific research itself is at fault, though it does point to the fact that scientists should expect better of each other as colleagues, and be diligent and rigorous in our work.

Crucially though, if one knows what to look for in a paper it is almost always possible to identify flaws in the research and weight one's confidence in the paper's conclusions accordingly. Scientists do this routinely when reading articles. Nobody reads papers and just accepts everything reported in them at face value. Rather, one always seeks both to understand the research presented and to evaluate how solid that research is. Scientists use published articles to build on and to support their own work, and it would be catastrophic to build from a foundation that later turns out to have been made of smoke.
posted by Anticipation Of A New Lover's Arrival, The at 9:50 AM on February 13, 2015 [3 favorites]


OK, "What if we can't do that?" is not the anthropomorphic theory - the universe' values are balanced "precariously" in such a way that our galaxy, sun, and planet exist, allowing life to develop, because it the universe was any other way we wouldn't, so there (and I'm really generalizing the idea, so don't get all "physics proves the Watchmaker exists!" on me).

But we've hit the idea before of "what if things can't be understood, and our approach is to lie to ourselves enough to believe things work." What's it called (technically)?
posted by IAmBroom at 9:50 AM on February 13, 2015


Ok, I skimmed it and it's pretty boring except for the fun of them referring to themselves in the third person.
posted by tofu_crouton at 9:52 AM on February 13, 2015


the idea that scientists follow something called "the scientific method" should probably be left behind in high school classes
posted by thelonius at 9:53 AM on February 13, 2015


Parapsychologists are able to produce experimental evidence for psychic phenomena about as easily as normal scientists are able to produce such evidence for normal, non-psychic phenomena.

Producing a manuscript ≠ Producing compelling evidence. I've spent a fair time leafing through parapsychological studies, and several characteristics pop out again and again:

1) Shoddy experimental design laden with bizarre caveats (as though the experimenter degrees of freedom are very large indeed).

2) No more than a journeyman's ability at statistical analysis.

3) Minuscule effect sizes.

I'll be the first to agree that an uncomfortable number of studies published in psychology generally fall prey to these same flaws, but their ubiquity in parapsychology is distinctive. The false equivalence between parapsychology and psychology generally is plain as day when one considers that the very best parapsychology has to offer barely reaches the level of a mediocre paper in other domains.

I think you meant "the ouroboros of evidence in psychology."

Speaking as a psychologist: I emphatically agree.
posted by belarius at 9:54 AM on February 13, 2015 [2 favorites]


Does anyone understand the reference here to "the (motherfucking) Sequences"? I'm learning about Bayesian analytics in kind of an ad hoc way and this is not something familiar to me.
posted by TypographicalError at 9:55 AM on February 13, 2015


Per your request: the Sequences
posted by just another scurvy brother at 9:58 AM on February 13, 2015 [2 favorites]


Oh, right. Part of my ad hoc learning of Bayesian analytics is trying to forget that those people exist.
posted by TypographicalError at 10:01 AM on February 13, 2015 [10 favorites]


From the rationalwiki article on LessWrong: "...the much-revered "Sequences," [are] a long series of essays by Yudkowsky that are considered essential reading by members of the [LessWrong] community."
posted by just another scurvy brother at 10:01 AM on February 13, 2015 [2 favorites]


Damn me for commenting before reading the whole article, that was a lot more interesting than I originally thought. Experimenter effect is a really interesting phenomenon, and one that seems not to be particularly well-understood or to be amenable to easy explanation. The article is a much more sophisticated take on errors in scientific research than I'd thought, and well worth reading. I apologize for being so dismissive up above.
posted by Anticipation Of A New Lover's Arrival, The at 10:07 AM on February 13, 2015 [4 favorites]


Their use is utterly routine in the sciences, and any publishable research is definitely going to use some of them. A researcher has not only to detect an effect, but also to quantify precisely what the odds are that this effect was a chimera.

THAT IS NOT WHAT P-VALUES TELL YOU! Even if you're not Bayesian.

P-values tell you the probability that you could observe at least as nonzero results by bad luck in the data you happened to draw IF the true data-generating process were some very specific kind of randomization and IF your data were a specific kind of random sample of all possible data generated by the true data-generating process.
posted by ROU_Xenophobe at 10:21 AM on February 13, 2015 [7 favorites]


... And there's a "data generating process" that is usefully described as producing some distribution.
posted by PMdixon at 10:40 AM on February 13, 2015


Sorry Xenophobe, you're absolutely right. I was trying to get my comment to fit into my phone's tiny screen and made a lazy generalization; I should do better, since I well know that p-values are pretty much the most abused statistical measure there is. I really regret that entire comment up above.

In my defense, I never meant to imply that p-values were the end-all-be-all of deciding whether an effect is real, just that they're the most basic and commonly-used tool—among many tools—for trying to quantify the uncertainty of one's results.
posted by Anticipation Of A New Lover's Arrival, The at 10:46 AM on February 13, 2015 [2 favorites]


How has this not been made into a buddy comedy yet

Seriously, I want to read this paper where they just accuse each other of sabotage.


Naturally the requested content is behind a paywall, but the abstract is readable.
posted by LiteS at 10:47 AM on February 13, 2015


Even danceable.
posted by Steely-eyed Missile Man at 11:03 AM on February 13, 2015


Heart Transplantion Study Halted Abruptly

"Transplant of placebo 'a disaster,' admits researcher"
posted by surazal at 11:12 AM on February 13, 2015 [4 favorites]


This was not a terrible post by Scott although he could really use an experienced editor, but I thought the post he did the other day on psych meds was actually pretty damn good.

The Efficacy Of Everything In Psychiatry In One Graph Plus Several Pages Of Dense But Necessary Explanation
posted by bukvich at 11:31 AM on February 13, 2015 [1 favorite]


Sorry Xenophobe, you're absolutely right.

FWIW, I meant to but forgot to include [morbo] and [/morbo] tags to make my SHOUTING less dick-ish.
posted by ROU_Xenophobe at 11:52 AM on February 13, 2015


You folk fell down a completely different rabbit hole than me. I am now curious about the logic of fitting mealsquares in my mouthcircle thanks to that sidebar ad.
posted by srboisvert at 11:56 AM on February 13, 2015 [3 favorites]


MataFilter: The logic of fitting mealsquares into your mouthcircles
posted by surazal at 12:22 PM on February 13, 2015 [3 favorites]


The whole soylent/mealsquares thing perfectly sums up the particular kind of derangement that most of the whole Rational Internet Libertarian Transhumanist thing represents to me. They just really kind of hate being biological. Scott seems like just about the least weird of the bunch.
posted by Steely-eyed Missile Man at 12:37 PM on February 13, 2015 [5 favorites]


I think they don't get that without the messy human stuff it's kinda both easy and uninteresting to be a good little rational Bayesian updater.
posted by PMdixon at 1:11 PM on February 13, 2015


But we've hit the idea before of "what if things can't be understood, and our approach is to lie to ourselves enough to believe things work." What's it called (technically)?

Isn't that "Scientific Pessimism"? Whoops, no, I guess it's called "Pessimistic induction" now, but I swear it used to be that first thing.
posted by saulgoodman at 2:16 PM on February 13, 2015


Well, pessimistic meta-induction is not the viewpoint, but rather one of the major arguments in favor of the viewpoint, which is some variety of scientific anti-realism. "Scientific pessimism" is a perfectly good label for it.
posted by vibratory manner of working at 2:35 PM on February 13, 2015


This was pretty informative and interesting, despite being associated with the whole crazy LessWrong/Yudkowski thingamajigger.

also 100% basilisk-free
posted by edheil at 4:22 PM on February 13, 2015


That was an interesting article that didn't go the way I expected. It's curious though that it uses parapsychology as the gold standard of something that's obviously not true and then shows the difficulty/impossibility of proving that it's not true experimentally... but never considers the possibility (which I don't believe, but who gives a shit what I or the author believes) that it is indeed true and there is indeed an effect, which would also explain all the results under consideration.

If you go back to general or special relativity, they were both concepts that were so radically out of the mainstream at the time that it was only experimental confirmation that gave them credibility. In the case of both, the confirmation was dramatic and clear-cut... but if it hadn't been?

Again, I have no belief in parapsychology, but the parapsychologists' complaint that they play by the rules but are still ridiculed/rejected is a valid one. There are FAR worse controlled results (and equally un-replicable) than theirs being published every day in the scientific mainstream. Skeptics need better arguments to refute these results than 'well, we don't know of a mechanism'.
posted by Sportswriters at 4:50 PM on February 13, 2015


(nevertheless the logic of the article is correct: either psi has been experimentally confirmed or our current tools for deciding the truth value of a hypothesis experimentally are broken)
posted by Sportswriters at 4:56 PM on February 13, 2015


If you go back to general or special relativity, they were both concepts that were so radically out of the mainstream at the time that it was only experimental confirmation that gave them credibility.

Not at all, the mathematical elegance and explanation of past results far outweighed any attempts to "test" relativity when it came to accepting the theory. Eddington's famous eclipse experiment is widely regarded as so tainted by experimental uncertainty as to be almost worthless in adjudicating between Newton and Einstein. Relativity was accepted despite the lack of experimental evidence, not because of overwhelming experimental support.
posted by Proofs and Refutations at 10:17 PM on February 13, 2015


Well, it was very important that the new theory would exactly generate all of the known regularities of celestial mechanics (by this point in history quite a lot), and Einstein didn't publish GR until he could guarantee this. So in that respect there was observational support from the very beginning.
posted by grobstein at 7:29 AM on February 14, 2015 [1 favorite]


What if we can't do that?

Then we have the wrong controls.

(Sometimes this is exciting because it means there's something new to discover about your system. But mostly it is infuriating because you have to check and double every constant in your system as if it were a variable and the answer is probably something like the length of a cord or an ingredient in the plastic bottle cap.)
posted by maryr at 9:52 AM on February 14, 2015 [1 favorite]


...I suppose I should now that the closest thing I have to a religion is a abiding love and respect for finding the right controls and maybe repaying free rides now that I have a car.
posted by maryr at 9:55 AM on February 14, 2015


(Really the actual problem is that constructing a null hypothesis is hard - it shouldn't be structureless, it should have all the structure you know is there and none of the structure you're inquiring about. But to say that it's hard doesn't mean that there's not better and worse ways to do it.)
posted by PMdixon at 11:34 AM on February 14, 2015


For those of you who do not know but might be interested, the Scott Alexander of the main link above and the Yvain of the post's link to the Less Wrong post from December 2010 Confidence levels inside and outside an argument are one in the same person.

Ouroboros indeed. This argument is fractally circular. The science that this fellow is mostly interested in is clinical trials for psych meds which is just about the last place to begin exploring these issues if one intended to make any headway. (The sequences are interesting in a something to do while surfing the net while bored at work kind of way but otherwise not so much.)
posted by bukvich at 6:10 AM on February 15, 2015


« Older TV’s Old Product-Placement Era Could Be Nearing...   |   Enter Franklin Newer »


This thread has been archived and is closed to new comments