The replication crisis comes for evolutionary psychology
October 25, 2018 6:16 AM   Subscribe

"A host of bewildering, bizarre results made their way into the journals. Fertile women are better at sussing out which men are gay, one study claimed. Their breasts become more symmetrical, said another. They wear skirts instead of pants. They have a better sense of smell. They’re more assertive. They seek out more variety in mini candy bars, but strive to lose more weight. They turn their backs on God, at least when they aren’t married, and they say they’ll vote for Barack Obama. ... Eventually, the underlying theory of this field of research grew so encrusted with garish findings, so brittle and baroque that it finally collapsed into a string of nonresults. ... The implications of this newer research have not escaped the founding fathers of the field. “In terms of overall effects, I don’t think there is anything,” admitted Gangestad in a recent interview, referring to his theory of women’s so-called dual sexuality." [SLSlate]
In other words, the man who literally co-wrote the book on human oestrus now believes that he and others in the field were the victims of pervasive problems in the way that the psychology was done—a bum steer that affected not just their research, but many others’ too. “When we wrote the book, we were drawing on a broad literature,” Gangestad told me, “but some of what we wrote was just garbage because we trusted all that work, including our own.”

...

“In many ways evolutionary psychology resembles religious belief,” wrote the evolutionary geneticist Jerry Coyne on his blog, in agreeing with the Slate barrage, “at least in the fervor of many of its advocates and their tendency to completely ignore data that don’t support their hypothesis.”
posted by clawsoon (67 comments total) 57 users marked this as a favorite
 
I felt a great disturbance in the MRA/PUA/incel axis...as if millions of voices suddenly cried out in outrage, and were suddenly silenced, except on social media, where they will doubtless never shut up about it. Also, hooray for Bubbles!
posted by Halloween Jack at 6:26 AM on October 25, 2018 [50 favorites]


I prefer your framing of the article! Slate seems more interested in peddling the same thrilling bullshit assuming, rightly, a click and a lazy reader who won’t get past the first paragraphs.

Having said that, this seems to ring true:

“In many ways evolutionary psychology resembles religious belief,” wrote the evolutionary geneticist Jerry Coyne on his blog,[...] “at least in the fervor of many of its advocates and their tendency to completely ignore data that don’t support their hypothesis.”
posted by amanda at 6:27 AM on October 25, 2018 [12 favorites]


Bubbles for president!
posted by amanda at 6:28 AM on October 25, 2018 [1 favorite]


I can't believe a bunch of men writing bizarre fanfic parables about their feelings about gender might turn out to have been nonsense instead of science! Shakes me to my core, truly.
posted by a fiendish thingy at 6:39 AM on October 25, 2018 [112 favorites]


About goddamn time. I've been consistently puzzled about how just so stories about the neolithic or paleolithic were remotely testable given the lack of surviving parallel species, broad evidence for hominid behavior from the period, or even the basic steps toward quantitative or molecular genetics for the trait in question.

"Human cognition is influenced by evolution," no shit Sherlock. Watson said that. Piaget said that. B.F. Fucking Skinner said that you contrarian fucks.
posted by GenderNullPointerException at 6:41 AM on October 25, 2018 [26 favorites]


Only one problem for me: I was filling my brain with knowledge during the heyday of poorly-done social psychology and evolutionary psychology research, and now it's filled with a bunch of half-remembered "facts" about humans which are mostly false.
posted by clawsoon at 6:49 AM on October 25, 2018 [25 favorites]


On the savannah, males could often improve their status by reporting dubious findings about the females' behavior during moon-cycles to the shaman.
posted by condour75 at 6:50 AM on October 25, 2018 [154 favorites]


The replication crisis comes for phrenology evolutionary psychology
posted by Kitty Stardust at 6:55 AM on October 25, 2018 [11 favorites]


Only one problem for me: I was filling my brain with knowledge during the heyday of poorly-done social psychology and evolutionary psychology research

I was a psych major in the late 90s. I'm like ... 60% sure? ... I took a class in evolutionary psychology (I was sort of adrift; my studies were very much on autopilot).

That this research is horseshit should be (and should have been) pretty self-evident -- dudes trying to invent black-and-white video-game logic for the mysteeeeeeerious ways of the feeeeemale? Come on.
posted by uncleozzy at 6:58 AM on October 25, 2018 [12 favorites]


and the parade of logical, rational men who were only following the logical, rational science to form their opinions of feeeeemales using this information to update their models of male-female behavior will begin... glances at clock when the sun burns out?
posted by Cozybee at 7:00 AM on October 25, 2018 [4 favorites]


This made me laugh a lot (at, not with). Though I did leave my husband not long after going off the Pill...lol
posted by wellred at 7:01 AM on October 25, 2018 [1 favorite]


There is no doubt a certain schadenfreude when we learn that things we want to not be true are in fact not true, as is the framing of this article.

But we should all be concerned in a much broader sense about how all scientific fields are currently operating and about how most of what we think we know we actually do not. The fields that laypeople understand less but are more enchanted with like biomed and neuroscience are facing the exact same crisis but get generally less attention than things like psych. Only 11% of biomed studies can be replicated. 11%! These are the studies your disease treatments are based on. It's as bad in neuroscience as well. This should cause everyone alarm. The real-world implications of terrible rigor and method in bioscience has life and death consequences.

It's fine to be outraged at these scientists, but like with many things this is systems level problem with roots in incentive, promotion, journal impact factors, all the way up to who the NIH and the NSF have to answer for. It's not a sexy political topic but we should be pushing on the political level for changes in government funded research.

Richard Harris's book 'Rigor Mortis' is a page-turner on this subject and approachable for anyone. The papers of Ionnaidis and others from Stanford's METRICS should be required reading for anyone interested in being critical about scientific research.
posted by Lutoslawski at 7:01 AM on October 25, 2018 [71 favorites]


Psych major in the mid-eighties, assuming that it's all pretty much useless by now. I mean, a strict regimen of corn flakes and ice baths is no longer assumed to cure the vapors, right?
posted by Halloween Jack at 7:03 AM on October 25, 2018 [19 favorites]


It’s always been a bunch of dubious Just So stories without backing, hasn’t it? It’s almosf a surprise to learn they had anything to back that up with that even resembles real science.
posted by Artw at 7:10 AM on October 25, 2018 [6 favorites]


.


(for the field of evolutionary psychology)
posted by darkstar at 7:10 AM on October 25, 2018 [1 favorite]


These studies were perhaps guided by a subconscious desire on the part of male researchers to justify taking women less seriously because they're "hormonal". Saying someone is hormonal is like saying they're irrational, unreliable, disobedient.
posted by mareli at 7:15 AM on October 25, 2018 [17 favorites]


I did learn most of my evo psych from Sarah Blaffer Hrdy, so I'm not completely screwed, but even she got a bit too enthusiastic about some of the non-replicated ovulation research.
posted by clawsoon at 7:17 AM on October 25, 2018 [2 favorites]


But we should all be concerned in a much broader sense about how all scientific fields are currently operating and about how most of what we think we know we actually do not.

Contrast this with the great glory that is 21st-century particle physics. 5 sigma (p = 3 x 10^{-7}) confidence level as the standard for discovery! Rigorous treatment of systematic uncertainties! Independent experiments competing to confirm/refute each others' findings! Full glory for publishing negative results exclusion limits!
posted by heatherlogan at 7:17 AM on October 25, 2018 [8 favorites]


I like how this post appears on the front page just a few ticks above a newspaper account of a psychology study with a suspiciously large effect size and alignment with things people are inclined to already believe.
posted by escabeche at 7:18 AM on October 25, 2018 [15 favorites]


The simple picture painted in the field’s early days—that menstrual shifts had women variously pursuing “cads” when fertile and hooking up with “dads” otherwise—was stretched and embellished in order to fit an increasingly disparate set of findings.

Aaaaand no one noticed how neatly this assumption mapped onto the belief in women's inherent Madonna-Whore behavioral binary that has basically under girded every form of Western patriarchal control since the Fertile Crescent? Lol. I guess all these logical men were just too objective to recognize they were making the same assumptions about women that an illiterate bronze-age shepherd would have made.
posted by Kitty Stardust at 7:18 AM on October 25, 2018 [68 favorites]


It's fine to be outraged at these scientists, but like with many things this is systems level problem with roots in incentive, promotion, journal impact factors, all the way up to who the NIH and the NSF have to answer for.

in certain fields and research topics we essentially have a system where scientists are in competition with each other to produce statistically significant positive results. the ones who lose the competition get fired.
posted by vogon_poet at 7:19 AM on October 25, 2018 [11 favorites]


As I was told in medical school - half of what you are taught is wrong. And, we don’t know which half.
posted by sudogeek at 7:19 AM on October 25, 2018 [17 favorites]


These studies were perhaps guided by a subconscious desire on the part of male researchers to justify taking women less seriously because they're "hormonal".

Of course. Even Aristotle knew that women were doomed to the brute life of animals, forever hostage to base, bodily impulses that crowded out all rational thoughts with either instinctual devotion to children or conniving plots to obtain sperm.
posted by Kitty Stardust at 7:24 AM on October 25, 2018 [6 favorites]


Research psychologist here. Evolutionary psychology is bullshit. all of it. my favorite piece of bullshit is the claim that people 'evolved' to remember things better when a savanna landscape is invoked. Evidence? you tell someone to read about a savanna, ask them to remember a list of words and their better at remembering them (by about 5%) compared to a non-savanna context. Evolved memory people! Thanks for posting this - the research is pretty equivalent to the social psychology findings that are the core of the replication crisis.
posted by bluesky43 at 7:25 AM on October 25, 2018 [31 favorites]


*squints* Interesting that this is the quote from Jerry Coyne, who has in fact defended [some forms of] evolutionary psychology as a field fairly sharply in the past.

Give me a sec, I need to read the actual article.
posted by sciatrix at 7:32 AM on October 25, 2018 [4 favorites]


sciatrix: Interesting that this is the quote from Jerry Coyne

Indeed. Posts on his blog tagged "evolutionary psychology"
posted by clawsoon at 7:43 AM on October 25, 2018 [2 favorites]


Kind of surprised it has taken this long. I was a psych undergrad in the late 80s/early 90s. Evo psych stuff came up several times in some of the classes; everyone had a really skeptical eye towards it. The methodological flaws were pretty evident in most of the studies we discussed. If a bunch of over-tired/hungover/whatever undergrads can pick apart the study in class, it's not generally a good sign.
posted by nubs at 7:46 AM on October 25, 2018 [2 favorites]


In conclusion, men evolved from Mars; woman evolved from Venus.
posted by benzenedream at 7:53 AM on October 25, 2018 [10 favorites]


I don't see this as quite parallel to social psychology. I don't think there's any question that people raised in different cultures practice different cultural norms, even if some of the specific claims may go deep into the woods.

Evolutionary psychology is a different matter because--barring molecular biology evidence that only exists for a handful of behaviors--its claims are untestable pseudoscience.
posted by GenderNullPointerException at 7:53 AM on October 25, 2018 [10 favorites]


Okay, so the article is citing a 2011 quote from Coyne written before the two pieces I pulled out and linked below, in which he bends over backwards to defend Steven Pinker in particular and the "real" contributions of evolutionary psychology in general. (See here this relevant criticism of Coyne and evolutionary psychology explanations from anthropologist Holly Dunsworth from 2016, for that matter.)

One thing I find is actually really interesting about this critique is that much of the work the article is pointing at and criticizing isn't actually necessarily evolutionarily mediated--after all, that's not relevant to any of the methodologies described. Instead, because we are largely looking at studies of the effects of ovulation on behavior, this is a body of literature with close ties to another area of psychology which also often attracts criticism, which is the effect of [sex-] hormonal shifts on behaviors and preferences. (I criticize it often myself!) My usual dry comment is that you can’t explain variation in the effects of hormonal signaling without also understanding variation in the distribution of receptors in the relevant tissues across both time and space, which is… usually not a feature of human studies, in part because it’s kind of difficult to accomplish while still abiding by basic ethical principles.

I'm still revising the first paper I'll have submitted since 2012, which is also a piece about the effects of hormonal manipulations on sexual behavior—and I’m ruminating about the things that worry me and make me wary of my own work, too. (Of course, I’m also not studying sex hormones—I work on leptin, which is involved in energy balance—but that doesn’t mean I’m not thinking about the possibility of not being able to replicate my own results, either. I have a study planned that should accomplish exactly that within a month or two, but by the time that one is done with data collection this paper will hopefully be toddling out into the world. If I fail to replicate my own work (I have no reason to think I will) I will be faced with a choice: should I pull the paper or leave it be?

I am seven years into my PhD, and I know I need to graduate soon, but I’ve been finding a really complicated story of order effects and social context that has muddied my data and taken years to understand. (Downside of working in a species with pretty minimal study before the last ten years, I guess. We didn’t understand anything about the order effects before I started working here.) If I publish a paper and then immediately pull it, that’s a hell of a thing to put on my CV, and I can more or less kiss my academic career goodbye. I understand the pressure to get something out, anything, to try and buy a little time before your career perishes underneath you. Most science is bound up intimately with the career prospects of folks who have very little security: how do we challenge replicability issues while people look at their jobs and paychecks and face the prospect of losing them if they can’t reliably return novel and interesting results?

The pressure is so fierce, that’s all. And there’s so little that I can control about the process. This is not a system that favors the careful—it’s a system that favors the flashy, and without a significant change in the way science and research is funded and trained I don’t see how anything will be changing any time soon.

What are we going to do about it?
posted by sciatrix at 8:02 AM on October 25, 2018 [43 favorites]


In conclusion, men evolved from Mars; woman evolved from Venus.

Hmm. Julius Caesar's family claimed descent from Venus. Via Aeneas, [mythic] Trojan Founder of Rome, no less.
posted by heatherlogan at 8:08 AM on October 25, 2018 [7 favorites]


Technically, Romulus and Remus were claimed children of Mars through a bloodline descended from Aeneas, though--thereby comingling the masculine godhead with a highly diluted personification of the long gone personification of female desire.

I'm not sure what that says about Roman self image, but I find it interesting.
posted by sciatrix at 8:14 AM on October 25, 2018 [12 favorites]


Contrast this with the great glory that is 21st-century particle physics. 5 sigma (p = 3 x 10^{-7}) confidence level as the standard for discovery! Rigorous treatment of systematic uncertainties! Independent experiments competing to confirm/refute each others' findings! Full glory for publishing negative results exclusion limits!

I had assumed it would be more difficult to conduct psychological studies in an evacuated tube surrounded by huge superconducting magnets, but maybe we should give it a shot or 10^6.
posted by Huffy Puffy at 8:14 AM on October 25, 2018 [13 favorites]


sciatrix: but that doesn’t mean I’m not thinking about the possibility of not being able to replicate my own results, either. I have a study planned that should accomplish exactly that within a month or two, but by the time that one is done with data collection this paper will hopefully be toddling out into the world. If I fail to replicate my own work (I have no reason to think I will) I will be faced with a choice: should I pull the paper or leave it be?

It's screwy that careful scientific work isn't publishable if it finds that there's no measurable effect. Let's say that you run the experiment again, and it doesn't replicate, and you don't publish. You've just created the conditions for half-a-dozen other young scientists to spend years also discovering that there's no measurable effect because they can't find your work in the literature.

If your work doesn't replicate, the system is set up so that the only result is for you to mislead other researchers, either by commission (you publish the original result) or omission (you don't publish). That's perverse.
posted by clawsoon at 8:15 AM on October 25, 2018 [21 favorites]


Yeah, it is perverse. But I find that the perspective of early career researchers in the position I'm in is not one that usually comes up in the context of discussions of reproducibility and scientific funding: the questions always seem to be about how to make science more and more competitive so that the errors are forced out, not about how the existing climate creates perverse incentives. That's particularly true when I see conversations among folks who aren't in academia.

I have noticed this about myself and my work: the more insecure I am about my career and its future prospects, the more I am acutely reminded about the competitive nature of my field, the more I find myself nervously insisting on grandiose findings and aggressive competitions instead of pausing and doing the best, most honest work I can possibly do.

I don't like that, either.
posted by sciatrix at 8:21 AM on October 25, 2018 [22 favorites]


The headline here is more exciting than the actual article.

It's about debunking claims about how ovulation affects women, not about debunking evolutionary psychology in general.
posted by Nancy Lebovitz at 8:31 AM on October 25, 2018


I like how this post appears on the front page just a few ticks above a newspaper account of a psychology study with a suspiciously large effect size and alignment with things people are inclined to already believe.

Heh, exactly what I was thinking, escabeche. I mean, pretty harmless really to encourage more contact, but the scale of the results did set off alarm bells.
posted by tavella at 8:35 AM on October 25, 2018


On the topic of publishing null results: yep, it's something of a mess. I've run into two different versions of it as a grad student and now postdoc. As a grad student, I abandoned my first-year project the summer between my first and second year of graduate school because I didn't want to try to publish a very boring (to me) null result that came down to "if you look for this visual effect with environmental references - like the monitor frame in a dark room - there's no effect to speak of." I had a conference poster on it, came back from the conference, thought about the feedback I'd gotten and decided to spin up a new project. Which, incidentally, didn't work, but the ideas there set me up for my first real project and what became my first paper in grad school... which came out in 2014, when I'd started the project in 2012.

At the moment, I've got a manuscript in revision that says, basically, "no, multitasking for naturalistic visual tasks isn't really an issue of cognitive load, it's an issue of where the information you need is in your visual field..." it's a null effect of cognitive load, which flies in the face of a couple of decades of more applied research. That I'm working on pushing through to publication, but the "nope, we don't believe you" edge is a significant one. There's extra fun here that the study doesn't fit neatly into anyone's box, but that's kinda my research world these days.

A few journals in my end of psychology have made real efforts to make null results and replications inherently publishable: about five years ago, Attention, Perception and Psychophysics spun up a new category of papers called Registered Replications or Reports. Quoting from AP&P's author guidelines: "Authors submit a proposed study. If it passes initial review, AP&P will commit to publishing the results, regardless of the outcome, if the final study conforms with the initially approved proposal." They mean it, too. I've reviewed a few of these as submissions, and it's a really nice way to either replicate or extend on previous work.
posted by Making You Bored For Science at 8:39 AM on October 25, 2018 [18 favorites]


I mean, a strict regimen of corn flakes and ice baths is no longer assumed to cure the vapors, right?

I do a strict regiment of ice for breakfast and corn flake baths...not one case of the vapors so far!
posted by Greg_Ace at 8:55 AM on October 25, 2018 [6 favorites]


Couldn't have happened to a better field.
posted by medusa at 8:56 AM on October 25, 2018 [4 favorites]


Maybe I should write a paper.
posted by Greg_Ace at 8:56 AM on October 25, 2018


a highly diluted personification of the long gone personification of female desire.

Derail: Venus was a war goddess in Rome. That's why Pompey built a giant temple to Venus Victor. She was also a goddess of love, and about a billion other things, including hair restoration in one specific incarnation.

On topic: I have always loathed this field, and I thoroughly enjoy calling it a pseudo-science whenever it comes up in class. It seems to be fading though, as I don't get it much from undergraduates even when you have to talk about anything prehistoric in myth.
posted by lesbiassparrow at 9:05 AM on October 25, 2018 [12 favorites]


Most science is bound up intimately with the career prospects of folks who have very little security: how do we challenge replicability issues while people look at their jobs and paychecks and face the prospect of losing them if they can’t reliably return novel and interesting results?

Publishing null (or complicated) findings has become even more of a minefield in our post-fact era. No longer is it just "journals might not be interested in publishing these", but multiple government agencies being hostile to science means that publishing null findings on programs that don't have an effect might result in Republicans using that as an excuse to defund programs wholesale.

Scientists know that null findings don't mean "nothing works, guess we shut down all programs on this subject and relegate it to the oubliette", but scientists are also aware that non-expert legislators are always looking to seize on excuses to stop all research and expenditures on subjects they hate.

It sucks! Null findings are so important, but between lack of interest in them by publications and readers of publications (HUGE drop off in reader engagement when covering null findings, which is understandable but still a problem), perverse incentives for academics to publish exciting findings or don't bother, and a hostile US govt eager to shut down all scientific inquiry and expenditure, they are more "dangerous" than you might think.
posted by a fiendish thingy at 9:18 AM on October 25, 2018 [16 favorites]


This is my asymmetrical shocked face.
posted by OmieWise at 9:24 AM on October 25, 2018 [5 favorites]


I do wonder what happened to the first result in the article, the one which led scientists to say, "Women can't possibly prefer men with feminine features! There must be a complicated evo-psych explanation for this!"

(Other than Justin Bieber, I mean.)
posted by clawsoon at 9:32 AM on October 25, 2018 [3 favorites]


In conclusion, men evolved from Mars; woman evolved from Venus.

Please, men evolved from clay; women evolved from ribs!
posted by GenjiandProust at 10:13 AM on October 25, 2018 [8 favorites]


in certain fields and research topics we essentially have a system where scientists are in competition with each other to produce statistically significant positive results. the ones who lose the competition get fired.

Capitalism strikes again.

We should really do something about that....
posted by eviemath at 11:06 AM on October 25, 2018 [5 favorites]


This is a good start. Now do MBA/"Management Science" case studies.
posted by sjswitzer at 11:43 AM on October 25, 2018 [4 favorites]


I had assumed it would be more difficult to conduct psychological studies in an evacuated tube surrounded by huge superconducting magnets, but maybe we should give it a shot or 10^6.

OK, now also do PET scans.
posted by sjswitzer at 11:51 AM on October 25, 2018 [1 favorite]


I will say that replication problems become suddenly rather more, ah, interesting when you're in one of two separate labs who even works on your species, working on a question of hormonal signaling and communication that has... one other guy following the same rabbit trail, I'd say. He works on anole lizards. I work on singing mice. They're pretty different organisms, I'd say.

In general, my experience is that there are just not many labs who are working on the same topics and systems; maybe that's just my thing over here in EEB where there are so many different species to pursue that it's common for people to know everyone working with a particular model, but how do you deal with replication when a) you'd probably need to do it yourself to get it done, and b) there's only one of you?

*grumbles* This is why I really, really wish that we'd publish our negative data sets, just to get it out there, but writing things up is also a whole lot of work, and like--how do you convince people to care about it enough to pay for it? I'd love to have access to null data sets in the literature to see what people have tried, but the thought of writing up my own null data sets on mercaptoacetate and methylmercaptoacetate in singing mice is, uh. Horrifying in concept, shall we say. And combine that with the tendency of scientists to run small pilots before committing to full experiments to see what works--how do you do a meta-analysis on many data sets each with slightly different methodologies or species?

I'm not so much arguing against replication as sketching on the reason it's so uncommon, but again: how the hell is anyone going to fix it? And as a fiendish thingy points out, it's not as if American scientists aren't under fire either way: it often feels like a damned-if-you-do, damned-if-you-don't position.

I got no answers.
posted by sciatrix at 12:47 PM on October 25, 2018 [10 favorites]


Wait, my breasts can become more symmetrical? Huh.
posted by emjaybee at 12:50 PM on October 25, 2018 [1 favorite]


Capitalism strikes again.

Huh?

Since I figure I must elaborate, putting people into competition with one another over scarce resources and rewarding those who fit a predetermined agenda, and not necessarily those who perform the best objectively, is not only NOT unique to capitalism, it's a feature of every hierarchical system ever invented by humanity.
posted by tclark at 1:01 PM on October 25, 2018 [1 favorite]


Sounds a hell of a lot like market bollocks to me.
posted by Artw at 1:02 PM on October 25, 2018 [1 favorite]


Tell that to comrade Lysenko. This is a people problem, not a capitalism problem, or a market problem.
posted by tclark at 1:04 PM on October 25, 2018 [5 favorites]


Lysenko was more of a genocidal dictator as patron problem.
posted by Artw at 1:11 PM on October 25, 2018 [2 favorites]


Hm. Almost like bad science and perverse incentivization has existed in every hierarchical system ever invented by humanity. Or is there an echo in here?
posted by tclark at 1:13 PM on October 25, 2018 [2 favorites]


I will say that replication problems become suddenly rather more, ah, interesting when you're in one of two separate labs who even works on your species, working on a question of hormonal signaling and communication that has... one other guy following the same rabbit trail, I'd say.

What factors determine which lab publishes their results? Perhaps dominant labs emit a chemical signal that suppresses the findings of junior laboratories.
posted by Joe in Australia at 1:13 PM on October 25, 2018 [9 favorites]


This is a people problem, not a capitalism problem, or a market problem.

No need to be so dismissive, and I disagree. If we had serious investments in funding for scientific research, and publishing (not the predatory kind like Elsevier), and teaching positions, people would be able to do exhaustive and ethical research knowing that the pressures of funding and employment are always about to run out or get shifted.

If funding wasn't tied to nebulous "outcomes" and "results" and "marketability" (yes even in science) and "impact" that are so much a part of the capitalist model of information as "content" and "product" rather than a good in and of itself, then a lot of these problems would be a lot less severe.
posted by a fiendish thingy at 1:14 PM on October 25, 2018 [13 favorites]


Research psychologist here. Evolutionary psychology is bullshit. all of it. my favorite piece of bullshit is the claim that people 'evolved' to remember things better when a savanna landscape is invoked. Evidence? you tell someone to read about a savanna, ask them to remember a list of words and their better at remembering them (by about 5%) compared to a non-savanna context. Evolved memory people! Thanks for posting this - the research is pretty equivalent to the social psychology findings that are the core of the replication crisis.

If you think the replication crisis is just for the sillier findings of evo or social psych you are probably going to be surprised in the near future. The problematic methods that underpinned the bogus claims of social pysch, the social priming lit in particular, and the emprical component of evo psych (ignoring the absurd theoretical part of evo psych) are used by almost all of psychology. Now their claims were obviously silly and it has attracted some scrutiny. So a lot of other areas of psych can giggle and tee-hee about it. But remember the Wansink nutrition debacle only got uncovered because he made an "I'm a jerk advisor" blog post that got people sniffing around. The new skeptical lens has not been applied to very much research at all.

p-hacking, harking and all those other little cheats are endemic to almost all science using inferential statistics.
posted by srboisvert at 1:16 PM on October 25, 2018 [4 favorites]


We both do! That's the good thing about working in different species--both of the pieces of work are novel and interesting based on being so phylogenetically divergent, so I'm not in competition with the other person working on the topic. Which is good, because I'm pretty sure that work is being driven by an undergraduate and a PI working at an undergrad-focused institution who doesn't take grad students or postdocs: I don't want to be in competition with them, I want them to generate interesting data that I can cite and use and in the case of the undergrad, go on to good jobs and careers.

The actual structural problems here are basically a byproduct of funding, scarcity, and willingness to commit to the issue. I don't think they're necessarily endemic to capitalism, but certainly the for-profit nature of many of the businesses that lurk adjacent to science (publishing, for one) combined with the increasingly scarce funding resources for scientists and the tendency to view training PhD students as a source of inexpensive labor rather than as long-term workers in the field exacerbate the precariousness.

I do have to wonder whether overthrowing capitalism is actually the solution here, basically because I wonder if it's realistically feasible. I'm all for overthrowing capitalism, but I'm also sitting here going "yeah, okay, what kinds of things that are achievable might fix this?" But the problem is so incredibly baked into the structure of how we do science and how scientists are paid that I don't think this is fixable without a massive structural change in the way we do things. So.

yaaaaaaay?
posted by sciatrix at 1:23 PM on October 25, 2018


Science in Germany has enjoyed a massive structural change in the opposite direction, and I only mention this because it means massive structural change is apparently something that countries can just do, for better or worse.
posted by Ashenmote at 3:23 PM on October 25, 2018 [4 favorites]


That sounds interesting, Ashenmote. Could you describe it more?
posted by clawsoon at 3:42 PM on October 25, 2018 [1 favorite]


Along the lines of what sciatrix said about wanting to see negative results, that's (probably) coming at a mandated level in the not-so-distant-future. NIH, at least, is quite grumpy about this, and we can expect a blanket policy (at least for human subjects work) of "we payed for it, so you get to report it." It's a start.

The thing is, NIH policy usually, over time, sets what amounts to national policy in the US, so I'd expect a registration/reporting requirement to grow from there. It won't encompass old work; it'd be a condition of funding for new grants going forward, but it'd begin training everyone that you don't file drawer things.
posted by Making You Bored For Science at 4:03 PM on October 25, 2018 [2 favorites]


Sorry, clawsoon, I know next to nothing about these things, but I meant that Germany had a system that limited the traction markets can get on science, with state funded universities and scholars who mostly didn't have to worry about money and/or career advancement. It kind of worked? Or at least it had a good call internationally? It was decided to drop that for the Brit and US way of doing science when Europe became a thing.
posted by Ashenmote at 4:16 PM on October 25, 2018 [2 favorites]


Ashenmote, I did find this from 2010 (paywalled):
UNTIL RECENTLY, AN ESSENTIAL INDICATOR in the evaluation of grant applicants by the Deutsche Forschungsgemeinschaft (DFG), Germany’s leading research foundation, was the quantity and impact of the applicant’s publications. This policy fit the increasing attention paid to Web of Science–listed publications, impact factors, and the h-index for competitive funding in science. The rationale is clear: On the basis of such variables, it is possible to compare performances and to provide a foundation for decisions.

However, the process overlooks one fundamental point: the content of research. The essence of the “Einsteins” of science history was surely not the quantity of their publications, but the quality of their research ideas. Ideas are hard to quantify—they are even harder to compare. But wise peer-referees can qualify them.

The DFG has recently taken an important step toward valuing content. The organization has changed its policy for evaluating research grants by restricting references in forthcoming applications to five of the authors’ most important publications and limiting reports of finished projects to the two most important publications per year. This helps reviewers appreciate the quality and the innovativeness of research. Of course, not every paper can introduce a Theory of Relativity. But we must focus on quality rather than quantity if we are to advance the world’s intellectual capital.
posted by clawsoon at 4:25 PM on October 25, 2018 [1 favorite]


I guess this is a younger massive structural change, clawsoon. This one is okay by me.
posted by Ashenmote at 4:58 PM on October 25, 2018


Almost like bad science and perverse incentivization has existed in every hierarchical system ever invented by humanity.

Of course. But the consequence of failure being loss of funding or getting fired from a job - as opposed to being put in front of a firing line, say - is a particular feature of the perverse incentives set out by capitalism. Nazi science, eg., was bad in different ways for different proximal reasons, that were not the ways described in the comment that I was responding to.

(* Noting that pseudoscience in support of white supremacy and using "scientific experimentation" as an excuse for genocidal crimes against humanity is incontrovertibly worse than a system that incentivizes junk results from experimental designs that at least pass an ethics review board; even if they are wholey unacquainted with anything resembling a statistics and experimental design review board or support insidious but not actively genocidally racist cultural assumptions. Which, sadly, is maybe a thing we have to worry about in this timeline. But that doesn't mean that we can't analyze the structural issues causing or perpetuating the less bad problems.)
posted by eviemath at 7:16 PM on October 26, 2018 [1 favorite]


« Older One way ticket   |   The Halibut Hook Revival Newer »


This thread has been archived and is closed to new comments