Cognitive Bias, Situations Matter, Pick a Noun, and other dead ends
October 8, 2023 7:35 PM Subscribe
Gino's work has been cited over 33,000 times, and Ariely's work has been cited over 66,000 times. They both got tenured professorships at elite universities. They wrote books, some of which became bestsellers. They gave big TED talks and lots of people watched them. By every conventional metric of success, these folks were killing it. Now let's imagine every allegation of fraud is true, and everything Ariely and Gino ever did gets removed from the scientific record, It's a Wonderful Life-style. What would change? Not much.I’m So Sorry for Psychology’s Loss, Whatever It Is: an essay by psychologist Adam Mastroianni on academic fraud, the replication crisis, and the questionable paradigms underlying a still-adolescent field
More essays in this series:
Science is a Strong-link Problem
In the long run, the best stuff is basically all that matters, and the bad stuff doesn’t matter at all. The history of science is littered with the skulls of dead theories. No more phlogiston nor phlegm, no more luminiferous ether, no more geocentrism, no more measuring someone’s character by the bumps on their head, no more barnacles magically turning into geese, no more invisible rays shooting out of people’s eyes, no more plum pudding, and, perhaps saddest of all, no more little dudes curled up inside sperm cellsOn the Importance of Staring Directly into the Sun
Okay, so an illusion of explanatory depth is extremely important to staying alive. It does, unfortunately, have a downside: it fools you into thinking the universe isn't full of mysteries. This, I think, explains the curious course of our scientific discovery. You might think that we discover things in order from most intuitive to least intuitive. No, thanks to the illusion of explanatory depth, it often goes the opposite way: we discover the least obvious things first, because those are things that we realize we don't understand.(See also: Reality has a surprising amount of detail)
Sorry Pal, This Woo is Irreducible
There isn’t an answer to those questions, of course. But there is a way to start, and it’s by believing that mysteries exist. I know that might sound dumb, but my three decades of formal education strongly implied to me that mysteries do not exist. There are things we don't know, sure, but these are temporary embarrassments, soon to be rectified. None of my textbooks included a chapter called “things we will never understand.” Even my religion classes were completely matter-of-fact about the mysterious: “an all-powerful, everlasting, omnipotent God sent his one son to earth to die for our sins because that had to happen, duh.”Psychology Might be a Big Stinkin’ Load of Hogwash and That’s Just Fine
And why would anyone claim otherwise? How could a high school principal get a school levy passed, and how could a university president justify charging you $60,000 per year for an education, if they had to admit there are things—important things, life-changing things—that they simply cannot teach you, because they cannot be taught at all?
We live in an age that’s less dark than ever, and yet we still know so little. If you don’t appreciate that, whenever you hear that another scientific finding has failed to replicate, you might feel like you’re being robbed of your birthright. But it was never yours to begin with. Or, if it was, it belongs to every human who ever lived as much as it belongs to you. Billions of humans lived and died in ignorance, and so will you, and so will I. All we can do is try to leave a little less ignorance than we found.
Oh no... I tried to get into a behavioral ecomonics program because of Dan Ariely. The economics of post-2008 put me on a different path. Now I see it would not have been a good path.
posted by rebent at 12:01 AM on October 9, 2023 [2 favorites]
posted by rebent at 12:01 AM on October 9, 2023 [2 favorites]
The whole article made me laugh, but this is probably the point:
Unfortunately, “humans are biased,” “situations matter,” and “pick a noun,” are unfalsifiable and inexhaustible. Nobody's ever going to prove that, actually, humans obey the laws of optimal decision making all the time. Nobody will show that situations don't matter at all. Nobody is going to demonstrate that leadership, creativity, or “social cryptomnesia” don’t exist. And we're never going to run out of biases, situations, or words. It's horrifying to think, but these proto-paradigms could be immortal.
The problem with psychology as a discipline is, as Fiasco da Gama notes, that it is a humanities discipline mascarading as hard science. AND there is a huge demand, too, both from public and private clients (think marketing, welfare management, healthcare). That combination of hazy method and theory and loads of money is a machine for fraud. Even people who aren't fraudsters to begin with can get sucked into the fraud vortex at the core of the discipline. And on the other hand, fraudsters are drawn to the field like moths to a candle. I'm not saying all psychologists are fraudsters, but that the profitable section of "psychology as hard science" has a strong draw towards fraud because there is so much nonsense in the field that it makes no difference what you are measuring, as long as the results fit your clients needs.
I think I like the above quote because of the words unfalsifiable and inexhaustible. The endlessness of it helps me make sense of the years of my life I spent fighting against this type of fraudulent research (mostly not within psychology, there are other, similar fields). I couldn't have done anything, not least because the people I were fighting didn't give a shit about results. They were interested in careers, in tenure and cash.
As the author writes:
But immortal does not mean invulnerable. Another way that paradigms die is people simply lose interest in them, so our best ally against these zombie paradigms is boredom. And we've got plenty. Psychologists already barely care about the findings in their own field...
(Yes, I am traumatized, and this is genuinely helpful).
posted by mumimor at 1:16 AM on October 9, 2023 [21 favorites]
Unfortunately, “humans are biased,” “situations matter,” and “pick a noun,” are unfalsifiable and inexhaustible. Nobody's ever going to prove that, actually, humans obey the laws of optimal decision making all the time. Nobody will show that situations don't matter at all. Nobody is going to demonstrate that leadership, creativity, or “social cryptomnesia” don’t exist. And we're never going to run out of biases, situations, or words. It's horrifying to think, but these proto-paradigms could be immortal.
The problem with psychology as a discipline is, as Fiasco da Gama notes, that it is a humanities discipline mascarading as hard science. AND there is a huge demand, too, both from public and private clients (think marketing, welfare management, healthcare). That combination of hazy method and theory and loads of money is a machine for fraud. Even people who aren't fraudsters to begin with can get sucked into the fraud vortex at the core of the discipline. And on the other hand, fraudsters are drawn to the field like moths to a candle. I'm not saying all psychologists are fraudsters, but that the profitable section of "psychology as hard science" has a strong draw towards fraud because there is so much nonsense in the field that it makes no difference what you are measuring, as long as the results fit your clients needs.
I think I like the above quote because of the words unfalsifiable and inexhaustible. The endlessness of it helps me make sense of the years of my life I spent fighting against this type of fraudulent research (mostly not within psychology, there are other, similar fields). I couldn't have done anything, not least because the people I were fighting didn't give a shit about results. They were interested in careers, in tenure and cash.
As the author writes:
But immortal does not mean invulnerable. Another way that paradigms die is people simply lose interest in them, so our best ally against these zombie paradigms is boredom. And we've got plenty. Psychologists already barely care about the findings in their own field...
(Yes, I am traumatized, and this is genuinely helpful).
posted by mumimor at 1:16 AM on October 9, 2023 [21 favorites]
It's ridiculous to look at this and minimize the impact of psychology as a science. It's actually a field that is honestly grappling with its epistemic issues head-on in a way that most other fields are not. Take a good look at biology and medicine and how many retractions are happening in those fields without any kind of existential examinations.
IMO the big problem with research psychology is that its methods and valid (and invalid) results are being deployed but mostly for very bad things. We are awash with technological behavioral manipulation and addiction thanks largely to research into psychological conditioning. Your shitty diet? That's psychology of taste. Your internet behavior? That's classical and operant conditioning. We are daily inundated with propaganda and misinfo thanks largely to research into attitudes and attitude change. There are huge political battles being fought over the smallest bureaucratic concerns of default options because of behavioral economics application of psychology research. When people want to control school boards that's because they understand developmental psychology. It goes on and on.
Psychology looks bad because being very humanities adjacent psychology is being looked at hard in a very multidisciplinary way. Most other sciences don't engage in this kind of scrutiny and when they do they blame any problems they find on .....human behavior!
An organism with all its built in desires, biases and shortcomings studying itself is an incredibly hard science problem! Probably the hardest.
posted by srboisvert at 1:35 AM on October 9, 2023 [40 favorites]
IMO the big problem with research psychology is that its methods and valid (and invalid) results are being deployed but mostly for very bad things. We are awash with technological behavioral manipulation and addiction thanks largely to research into psychological conditioning. Your shitty diet? That's psychology of taste. Your internet behavior? That's classical and operant conditioning. We are daily inundated with propaganda and misinfo thanks largely to research into attitudes and attitude change. There are huge political battles being fought over the smallest bureaucratic concerns of default options because of behavioral economics application of psychology research. When people want to control school boards that's because they understand developmental psychology. It goes on and on.
Psychology looks bad because being very humanities adjacent psychology is being looked at hard in a very multidisciplinary way. Most other sciences don't engage in this kind of scrutiny and when they do they blame any problems they find on .....human behavior!
An organism with all its built in desires, biases and shortcomings studying itself is an incredibly hard science problem! Probably the hardest.
posted by srboisvert at 1:35 AM on October 9, 2023 [40 favorites]
Who looks for great breakthroughs in history?
I think we do? Doesn't our understanding of the world change dramatically with new discoveries and analyses of history?
The way we view the lead-up to WW1, Egyptian & Mayan technological capabilities, the Holocaust, the European settling and indigenous people of the Americas...actually, just pick any genocide — these all depend on getting historical analysis right. It seems to me that it matters if history and psychology make things up or misuse techniques (e.g. statistics, lit review, sourcing of documents, translation). I mean, we literally had a "replicability crisis in history" thread recently, because there's strong evidence that a widely-disseminated breakthrough was maybe based on sloppy or deceitful work.
who ever heard of a replicability crisis in philosophy? Alan Sokal, I think. Chomsky. I think cognitive scientists make a good case as well.
posted by daveliepmann at 1:48 AM on October 9, 2023 [8 favorites]
I think we do? Doesn't our understanding of the world change dramatically with new discoveries and analyses of history?
The way we view the lead-up to WW1, Egyptian & Mayan technological capabilities, the Holocaust, the European settling and indigenous people of the Americas...actually, just pick any genocide — these all depend on getting historical analysis right. It seems to me that it matters if history and psychology make things up or misuse techniques (e.g. statistics, lit review, sourcing of documents, translation). I mean, we literally had a "replicability crisis in history" thread recently, because there's strong evidence that a widely-disseminated breakthrough was maybe based on sloppy or deceitful work.
who ever heard of a replicability crisis in philosophy? Alan Sokal, I think. Chomsky. I think cognitive scientists make a good case as well.
posted by daveliepmann at 1:48 AM on October 9, 2023 [8 favorites]
I have to hand a multi-volume report of a symposium organized by the American Society of Zoologists in 1989. As far as the internet is concerned, this event has sunk without trace but the title has been a watchword for my endeavours for 30 years: "Science as A way of knowing". You can get as much useful insight on how to successfully interact with other people from reading Jane Austen as reading Francesca Gino.
posted by BobTheScientist at 2:28 AM on October 9, 2023 [16 favorites]
posted by BobTheScientist at 2:28 AM on October 9, 2023 [16 favorites]
The strong link thing may be true big picture for the advancement of science, but when the garbage gets publicity and the public stop trusting scientists then we end up with the whole antivax thing.
posted by onya at 4:45 AM on October 9, 2023 [2 favorites]
posted by onya at 4:45 AM on October 9, 2023 [2 favorites]
when the garbage gets publicity and the public stop trusting scientists then we end up with the whole antivax thing.
Uh... no? The "whole antivax thing" dates back centuries, and while this latest thing may eventually work its way into their rationalization for being antivax, their movement hasn't depended on actual scientific fraud for justification.
posted by Halloween Jack at 5:37 AM on October 9, 2023 [7 favorites]
Uh... no? The "whole antivax thing" dates back centuries, and while this latest thing may eventually work its way into their rationalization for being antivax, their movement hasn't depended on actual scientific fraud for justification.
posted by Halloween Jack at 5:37 AM on October 9, 2023 [7 favorites]
plato’s socratic dialogues generally followed something like this structure:
- socrates meets someone who is certain they know something
- socrates asks them about that thing
- by asking a series of probing questions socrates helps the person realize that they don’t know the thing they thought they knew
- they together try to get back to a state of having knowledge — okay, so the person’s first idea wasn’t right, but can we find a better-grounded claim?
- they reach an impasse (an aporia).
- the dialogue ends with no one able to say they know anything and with everyone in a state of confusion.
which I bring up because a lot of people in this thread believe they are being scientific by opposing uncertainty and doubt about science. this is a position that’s not just anti-scientific but also, beyond that, irrational.
posted by bombastic lowercase pronouncements at 5:45 AM on October 9, 2023 [7 favorites]
posted by bombastic lowercase pronouncements at 5:45 AM on October 9, 2023 [7 favorites]
I confess, this is such an odd piece for me to read as a behavioral ecologist who has been sitting in a psychology department for the past two years while working in community with folks in neuroscience and behavior genetics. Behavior is my thing—good old fashioned ethology!—and the thing about being really into behavior and also being really quite ADHD is that you can wander from discipline to discipline quite easily, since there are, mm, at least half a dozen disciplines just in the intersection of behavior and genetics or evolution alone. So I have.
For one thing, I fundamentally disagree with the author that other fields are doing a better job of this. Take those behavior geneticists I mentioned--who, by the way, are residents of the Psychology department and appear to come out of the undergrad disciplines to match. They were incredibly familiar to me from my experiences in EEB listening to people crow about "omics" advances in grappling with Big Data from a data-first pipeline, or for that matter from my undergrad Genetics degree, listening to the proponents of GWAS studies bragging about the phenomenal new advances that were sure to materialize any day now. You'd be able to take a genetic test and learn meaningful things about health risks! Doctors would be able to meaningfully talk to their patients about their results to, um, somehow help the patients change their behaviors towards better health! Avoid risk factors! Meaningfully changing their individual behaviors and letting high risk patients know they're high risk!
I might not have made friends when I asked another postdoc at one point in an early seminar: what meaningful advances have these high-data-driven massive-input tests actually shown us in the past ten years? You know, since exactly the same exultant promises were being made when I was in undergrad? But I stand by the question. The problem is that there are very many genes that often have effects that are unpredictably strong in specific environments--either in terms of the individual's environmental experience or even in terms of the other allelic variants at other loci that an individual might encounter. When you try to get meaningful results out by dumping more and more and more data without controlling for environment, the "noise" of environmental variation gets lost completely. As the main FPP link notes: situations matter. And yet that's still a fundamental insight that many fields struggle with understanding!
For another thing, I'm mystified by the notion that all psychology is a uniform field, any more than all behavior is. Like I said, I spent my PhD in behavioral ecology, which rarely bothers to speak much to animal cognition, and which uses quite different techniques and approaches than the neuroscientists I work with even when trying to disentangle the secrets of the brain. I have impatiently asked so many baby neuroscience students what c-fos actually does, for example, because that matters when it comes to evaluating and understanding immediate early genes studies, something I was explicitly trained how to do. I have had to ask one of the behavior genetics students how heredity is measured (a precursor for discussing flaws in his experimental paradigm) and found him unable to tell me; in my program, that would have been grounds for failing a candidacy defense, which he was about to take.
Even within the disciplines that crowd cheek by jowl into psychology departments themselves, people in different subdisciplines don't always read and comprehend the implications of one another's work on their own systems of study. Social psychologists don't seem to talk much to neuroscientists, for example. Behavioral economics — the discipline of the two frauds in the OP — sure does talk to neuroscience, though! Bewilderingly, though, the animal work in behavioral economics seems increasingly conducted in contexts that artificially inflate the value of reward items in order to maximize the number of repetitive trials an animal will perform. I am amazed at how many people are moving to water restriction and rewards of small quantities of liquid in rodent work, and then saying that the results discovered are generalizable to situations in which individuals are far less desperate and physiologically stressed. Don't get me started on the evolutionary psychologists.
What is psychology, anyway? If a discipline is an extended conversation between researchers, there are certainly an awful lot of little subgroups within its umbrella who aren't actually in broad-scale dialogue with one another. Maybe that's a piece worth fixing, too.
posted by sciatrix at 5:46 AM on October 9, 2023 [32 favorites]
For one thing, I fundamentally disagree with the author that other fields are doing a better job of this. Take those behavior geneticists I mentioned--who, by the way, are residents of the Psychology department and appear to come out of the undergrad disciplines to match. They were incredibly familiar to me from my experiences in EEB listening to people crow about "omics" advances in grappling with Big Data from a data-first pipeline, or for that matter from my undergrad Genetics degree, listening to the proponents of GWAS studies bragging about the phenomenal new advances that were sure to materialize any day now. You'd be able to take a genetic test and learn meaningful things about health risks! Doctors would be able to meaningfully talk to their patients about their results to, um, somehow help the patients change their behaviors towards better health! Avoid risk factors! Meaningfully changing their individual behaviors and letting high risk patients know they're high risk!
I might not have made friends when I asked another postdoc at one point in an early seminar: what meaningful advances have these high-data-driven massive-input tests actually shown us in the past ten years? You know, since exactly the same exultant promises were being made when I was in undergrad? But I stand by the question. The problem is that there are very many genes that often have effects that are unpredictably strong in specific environments--either in terms of the individual's environmental experience or even in terms of the other allelic variants at other loci that an individual might encounter. When you try to get meaningful results out by dumping more and more and more data without controlling for environment, the "noise" of environmental variation gets lost completely. As the main FPP link notes: situations matter. And yet that's still a fundamental insight that many fields struggle with understanding!
For another thing, I'm mystified by the notion that all psychology is a uniform field, any more than all behavior is. Like I said, I spent my PhD in behavioral ecology, which rarely bothers to speak much to animal cognition, and which uses quite different techniques and approaches than the neuroscientists I work with even when trying to disentangle the secrets of the brain. I have impatiently asked so many baby neuroscience students what c-fos actually does, for example, because that matters when it comes to evaluating and understanding immediate early genes studies, something I was explicitly trained how to do. I have had to ask one of the behavior genetics students how heredity is measured (a precursor for discussing flaws in his experimental paradigm) and found him unable to tell me; in my program, that would have been grounds for failing a candidacy defense, which he was about to take.
Even within the disciplines that crowd cheek by jowl into psychology departments themselves, people in different subdisciplines don't always read and comprehend the implications of one another's work on their own systems of study. Social psychologists don't seem to talk much to neuroscientists, for example. Behavioral economics — the discipline of the two frauds in the OP — sure does talk to neuroscience, though! Bewilderingly, though, the animal work in behavioral economics seems increasingly conducted in contexts that artificially inflate the value of reward items in order to maximize the number of repetitive trials an animal will perform. I am amazed at how many people are moving to water restriction and rewards of small quantities of liquid in rodent work, and then saying that the results discovered are generalizable to situations in which individuals are far less desperate and physiologically stressed. Don't get me started on the evolutionary psychologists.
What is psychology, anyway? If a discipline is an extended conversation between researchers, there are certainly an awful lot of little subgroups within its umbrella who aren't actually in broad-scale dialogue with one another. Maybe that's a piece worth fixing, too.
posted by sciatrix at 5:46 AM on October 9, 2023 [32 favorites]
Not a small amount of the crap published in psychology (all fields, and it is a very diverse area where cross-disciplinary groups exist in one department) is driven by the contingencies. Publishing, grants and the PhD producing mill contribute to life time appointment tenure decisions in a way that I think is really destructive. The one ray of hope I see in the field is a readjustment on the way statistics are used in ways that really constrain the ability to p-hack. Nevertheless, the inability to generalize to real human behavior from the kinds of carefully controlled experiments the field requires is real. And the empty terminology in many of the subdisciplines of psychology is insane.
FWIW I never believed the findings of the stuff Ariely and his ilk published. Even if you can find such an effect (like the stuff on feeling old) the actual effect is so small that it seemed unlikely to matter much anyway.
Don't get me started on the evolutionary psychologists. OMG this. I got the sense that psychologists who went this route thought it was a way to be more science-y because EVOLUTION. Once you poke at any of the conclusions it's pretty clear that they authors could've made exactly the opposite conclusion.
posted by bluesky43 at 6:57 AM on October 9, 2023 [7 favorites]
FWIW I never believed the findings of the stuff Ariely and his ilk published. Even if you can find such an effect (like the stuff on feeling old) the actual effect is so small that it seemed unlikely to matter much anyway.
Don't get me started on the evolutionary psychologists. OMG this. I got the sense that psychologists who went this route thought it was a way to be more science-y because EVOLUTION. Once you poke at any of the conclusions it's pretty clear that they authors could've made exactly the opposite conclusion.
posted by bluesky43 at 6:57 AM on October 9, 2023 [7 favorites]
Here's an interesting reply from Paul Bloom, mentioned in the first link. I actually really like his list of important discoveries in psychology noting that none of them are from social or evolutionary psychology.
posted by bluesky43 at 7:16 AM on October 9, 2023 [7 favorites]
posted by bluesky43 at 7:16 AM on October 9, 2023 [7 favorites]
Much of my work in traffic safety falls on the statistical side of evaluating countermeasures and building crash prediction models, but I'm honestly grateful for the contributions of psychologists to my field. Sure, I do think a lot of studies are inappropriately applying the Theory of Planned Behavior, and decades of research into distraction and exhaustion still haven't given us much to work with, but there has long been a pernicious idea, caused by a misunderstanding of a NHTSA study, that claims that 94% of crashes are caused by human error. The psychologists are helping us push back on that idea (or at least its uncritical repetition) and reminding folks that countermeasures, roadway design, etc. need to take human behavior (and the role of the environment in "education") into account.
Incidentally, I think we invited Dan Ariely to speak at a conference a few years ago, but he gave us the cold shoulder.
posted by TheKaijuCommuter at 7:48 AM on October 9, 2023 [7 favorites]
Incidentally, I think we invited Dan Ariely to speak at a conference a few years ago, but he gave us the cold shoulder.
posted by TheKaijuCommuter at 7:48 AM on October 9, 2023 [7 favorites]
I think the author is correct that a lot of psychology is “flabby,” and that the replication crisis has revealed the nature of the flab. I also think the author is not trained in philosophy, and that he’s doing philosophy in a very unsophisticated way.
With respect to the history of ideas, psychology originated as a branch of philosophy. As psychology incorporated scientific notions about experimental evidence and measurement, it evolved into disciplinary independence.
What the author calls the “pick a noun” approach strikes me as remnants of philosophy, conjoined with bad science or no science.
But he really misunderstands what he calls “useful fictions”. (Although perhaps this error is not crucial to his analysis.) He is trying to compare a topic like “leadership” to “money” or “the Ford Motor Company”. Money and corporations are real and widely understood in a way that leadership is not, because the first two institutions create widely understood social duties and rights.
It is easy to describe how to buy a car or what happens if you don’t make a car payment. Describing how leadership works is not so easy. There is a trivial sense in which economics and law are sciences, but the psychology of leadership (or pop psychology’s next noun) is not.
posted by PaulVario at 7:52 AM on October 9, 2023 [5 favorites]
With respect to the history of ideas, psychology originated as a branch of philosophy. As psychology incorporated scientific notions about experimental evidence and measurement, it evolved into disciplinary independence.
What the author calls the “pick a noun” approach strikes me as remnants of philosophy, conjoined with bad science or no science.
But he really misunderstands what he calls “useful fictions”. (Although perhaps this error is not crucial to his analysis.) He is trying to compare a topic like “leadership” to “money” or “the Ford Motor Company”. Money and corporations are real and widely understood in a way that leadership is not, because the first two institutions create widely understood social duties and rights.
It is easy to describe how to buy a car or what happens if you don’t make a car payment. Describing how leadership works is not so easy. There is a trivial sense in which economics and law are sciences, but the psychology of leadership (or pop psychology’s next noun) is not.
posted by PaulVario at 7:52 AM on October 9, 2023 [5 favorites]
Nevertheless, the inability to generalize to real human behavior from the kinds of carefully controlled experiments the field requires is real. And the empty terminology in many of the subdisciplines of psychology is insane.
On the neuroscience end, I'm often weirdly aghast because the attempts to control animal behavior experiments erase huge chunks of variance that can create massive generalizability confounds. Findings "In Mice" aren't even the half of it--I was reading a piece the other day that proudly proclaimed that its results were extra validated because it had been tested in three genetic backgrounds across three or four labs.
The three strains were, of course, 50/50 129/C57BlJ, C57Bl/6J, and C57Bl/6N. The latter two are the same basic strain, kept in two populations (Jackson Labs and the NIH) which have accumulated a few differences due to drift despite attempts to maintain the same inbred strain as closely as possible over the past 50 years. The genotype in question is known to have dramatically different impacts on animals from the outcross background and on pure C57 strain; among other things, the pure C57 animals with the genotype of interest are deaf, obese, and have known problems with things like swimming, none of which appear phenotypically in the outcrossed animals. And not all these "strain controlled" replications of effect used those outcross background animals, either--animals that still share approximately half their genome with the C57s!
Here is an experiment, then, that does have high replicability because many of the inherently confounding potential factors have been controlled. Yes? The background genotype of the mouse, which we know can have big impacts, has been controlled and held steady so that no dastardly emergent interaction can fuck up that nice consistent finding and complicate the story. As bluesky43 points out, those tidy stories are absolutely crucial to success in grant applications and publications and all the other career metrics that allow scientists to remain in the field.
But the actual generalizability of those results is dog shit. You can't even generalize all those findings to other domestic strains of mice, because no one has ever bothered to test the impacts of the genes in other strains! To say nothing of wild mice, to say nothing of humans, etc etc etc. Hell, replicability is the main reason that until the funding agencies made them stop, most behavioral research was done in male mice only--no one knows how those pesky estrous cycles might affect results, so individuals that possess them must be excluded systematically from the dataset. Replicability, you know. It's not as easy to check whether the finding can be generalized.
So why in the hell aren't people taking the time to use literally any other genetic background? Why don't we test multiple inbred lines of mice to estimate the impact of other interacting loci or even try to map those loci?
That's hard! And expensive!And the field is competitive! And no one else has to! And so we maintain a status quo. But the reason that fields like biology and psychology can't maintain the same consistency of finding as chemistry or physics or mathematics isn't because people aren't trying: it's because the problems of the fields are inherently complicated by a veritable host of competing factors, each with their own emergent dynamics. You won't see that changing until and unless there are field-wide mandates to change that, as with the new NIH mandates that you must test animals of both sexes unless you're working with a truly sex specific issue like prostate cancer. And even then, you'll see people doing their best not to change their approaches or their thought patterns about variation. Maddening.
posted by sciatrix at 8:42 AM on October 9, 2023 [15 favorites]
On the neuroscience end, I'm often weirdly aghast because the attempts to control animal behavior experiments erase huge chunks of variance that can create massive generalizability confounds. Findings "In Mice" aren't even the half of it--I was reading a piece the other day that proudly proclaimed that its results were extra validated because it had been tested in three genetic backgrounds across three or four labs.
The three strains were, of course, 50/50 129/C57BlJ, C57Bl/6J, and C57Bl/6N. The latter two are the same basic strain, kept in two populations (Jackson Labs and the NIH) which have accumulated a few differences due to drift despite attempts to maintain the same inbred strain as closely as possible over the past 50 years. The genotype in question is known to have dramatically different impacts on animals from the outcross background and on pure C57 strain; among other things, the pure C57 animals with the genotype of interest are deaf, obese, and have known problems with things like swimming, none of which appear phenotypically in the outcrossed animals. And not all these "strain controlled" replications of effect used those outcross background animals, either--animals that still share approximately half their genome with the C57s!
Here is an experiment, then, that does have high replicability because many of the inherently confounding potential factors have been controlled. Yes? The background genotype of the mouse, which we know can have big impacts, has been controlled and held steady so that no dastardly emergent interaction can fuck up that nice consistent finding and complicate the story. As bluesky43 points out, those tidy stories are absolutely crucial to success in grant applications and publications and all the other career metrics that allow scientists to remain in the field.
But the actual generalizability of those results is dog shit. You can't even generalize all those findings to other domestic strains of mice, because no one has ever bothered to test the impacts of the genes in other strains! To say nothing of wild mice, to say nothing of humans, etc etc etc. Hell, replicability is the main reason that until the funding agencies made them stop, most behavioral research was done in male mice only--no one knows how those pesky estrous cycles might affect results, so individuals that possess them must be excluded systematically from the dataset. Replicability, you know. It's not as easy to check whether the finding can be generalized.
So why in the hell aren't people taking the time to use literally any other genetic background? Why don't we test multiple inbred lines of mice to estimate the impact of other interacting loci or even try to map those loci?
That's hard! And expensive!And the field is competitive! And no one else has to! And so we maintain a status quo. But the reason that fields like biology and psychology can't maintain the same consistency of finding as chemistry or physics or mathematics isn't because people aren't trying: it's because the problems of the fields are inherently complicated by a veritable host of competing factors, each with their own emergent dynamics. You won't see that changing until and unless there are field-wide mandates to change that, as with the new NIH mandates that you must test animals of both sexes unless you're working with a truly sex specific issue like prostate cancer. And even then, you'll see people doing their best not to change their approaches or their thought patterns about variation. Maddening.
posted by sciatrix at 8:42 AM on October 9, 2023 [15 favorites]
My experience with psychologists, both for myself and for people close to me, in a clinical psych setting, has never been scientific. For example, I was given Prozac, a drug that supposedly increases the presence of serotonin to treat depression. No one measured my serotonin levels. When after six months of increasing dosages and no effect other than xerostomia and weight gain I told the attending psychologist and the psychiatrist who prescribed the drug without ever seeing me or looking at my records I’m going to stop taking it. They said nothing. As to talk therapy, my own experience is that individual psychologists have pet theories that they apply uniformly across all their patients. Not science. I know some people who had good results from Prozac but I can’t say whether that was science, luck, or placebo effect. It just happened to work. Psychology’s efficacy seems to rest on the talents of individual psychologists and not on any uniform theory of how the human mind functions. People want to explore and discover. We are all walking around with a three pound organ of extreme complexity and ability and I have doubts whether we have the right tools yet in our own three pound organs to figure out how it works. That old college maxim - if you can’t dazzle them with brilliance, baffle them with bullshit - seems to apply here.
posted by njohnson23 at 9:10 AM on October 9, 2023 [3 favorites]
posted by njohnson23 at 9:10 AM on October 9, 2023 [3 favorites]
I had a fascinating conversation yesterday with someone doing a PhD on decision making, specifically why people fluent in more than one language might make different decisions depending on which language they thought in. She's working with Chinese immigrants in London, and apparently it is a thing.
posted by 43rdAnd9th at 9:23 AM on October 9, 2023 [6 favorites]
posted by 43rdAnd9th at 9:23 AM on October 9, 2023 [6 favorites]
Plenty of these findings are interesting and some are useful (especially if you are a rich, lonely monkey).
I almost choked on the pizza I'm eating. This close to knowing whether I could self-administer the Heimlich!
posted by pinothefrog at 9:26 AM on October 9, 2023 [1 favorite]
I almost choked on the pizza I'm eating. This close to knowing whether I could self-administer the Heimlich!
posted by pinothefrog at 9:26 AM on October 9, 2023 [1 favorite]
When I teach the history of autism, I point out that since 1980, the American Psychological Association has had four different definitions of autism. Sometimes, psychology is science by voting. It's not a hard science.
The podcast, This American Life lays this out in the episode, 81 Words.
posted by ITravelMontana at 9:33 AM on October 9, 2023 [4 favorites]
The podcast, This American Life lays this out in the episode, 81 Words.
posted by ITravelMontana at 9:33 AM on October 9, 2023 [4 favorites]
metafilter: especially if you are a rich, lonely monkey
posted by hearthpig at 9:48 AM on October 9, 2023 [8 favorites]
posted by hearthpig at 9:48 AM on October 9, 2023 [8 favorites]
Psychology is unique in that the subject and object are the same. It has no Archimedean point of reference. I had a two year analysis with a gifted analyst who was trained in Analytical Psychology, i.e. Jungian. He unfortunately died in 2017. One of my symptoms was loss of smell and taste that returned after two years almost to the day. In late 2000, I again lost my sense of smell and taste due to an ear infection. I am back in therapy and quite down. I did not mean for this to turn into a confessional, but I have to be truthful.
posted by DJZouke at 11:17 AM on October 9, 2023 [4 favorites]
posted by DJZouke at 11:17 AM on October 9, 2023 [4 favorites]
Love this post, many interesting ideas on science, thanks Rhaomi.
posted by blue shadows at 12:57 PM on October 9, 2023 [1 favorite]
posted by blue shadows at 12:57 PM on October 9, 2023 [1 favorite]
"The leading American professional group for psychologists secretly worked with the Bush administration to help justify the post-9/11 US detainee torture program, according to a watchdog analysis released on Thursday."
I quit recommending therapy after that. And meanwhile repeated experiences with my search for a therapist ended with "buy my emo-workbook to continue". This has left me with therapy as "single level marketing".
So to watch its clinical side devolve is somewhat of a slow blink for me.
posted by MonsieurPEB at 12:59 PM on October 9, 2023 [3 favorites]
I quit recommending therapy after that. And meanwhile repeated experiences with my search for a therapist ended with "buy my emo-workbook to continue". This has left me with therapy as "single level marketing".
So to watch its clinical side devolve is somewhat of a slow blink for me.
posted by MonsieurPEB at 12:59 PM on October 9, 2023 [3 favorites]
I had a fascinating conversation yesterday with someone doing a PhD on decision making, specifically why people fluent in more than one language might make different decisions depending on which language they thought in. She's working with Chinese immigrants in London, and apparently it is a thing.
Interesting, on the face it makes me think of the (discredited) Sapir-Whorf hypothesis, which contended that the words you use change the meaning of what you say (simplified). I'd be curious to know how they're prepared to get around that.
posted by rhizome at 1:12 PM on October 9, 2023
Interesting, on the face it makes me think of the (discredited) Sapir-Whorf hypothesis, which contended that the words you use change the meaning of what you say (simplified). I'd be curious to know how they're prepared to get around that.
posted by rhizome at 1:12 PM on October 9, 2023
You know, it’s ok for a field of human knowledge and work to be not formally ‘scientific’, and yet still offer true and even useful insights. That something isn’t ‘science’ makes it no less difficult, and no less a discipline.
posted by Fiasco da Gama at 1:35 PM on October 9, 2023 [10 favorites]
posted by Fiasco da Gama at 1:35 PM on October 9, 2023 [10 favorites]
> apparently it is a thing
There was something similar in the Guardian recently - 'I couldn’t believe the data': how thinking in a foreign language improves decision-making. That refers to second language learning - unclear if that is the same thing as in the work of the person you met. There is another account of similar research from a few years earlier: Bilingual people may make different choices.
posted by paduasoy at 4:40 PM on October 9, 2023 [5 favorites]
There was something similar in the Guardian recently - 'I couldn’t believe the data': how thinking in a foreign language improves decision-making. That refers to second language learning - unclear if that is the same thing as in the work of the person you met. There is another account of similar research from a few years earlier: Bilingual people may make different choices.
posted by paduasoy at 4:40 PM on October 9, 2023 [5 favorites]
Mr Show: How does a scientist know what's true?
posted by Pickman's Next Top Model at 9:55 PM on October 9, 2023 [1 favorite]
posted by Pickman's Next Top Model at 9:55 PM on October 9, 2023 [1 favorite]
My comment should have read "In late 2019-early 2020" rather than 2000.
posted by DJZouke at 8:54 AM on October 10, 2023
posted by DJZouke at 8:54 AM on October 10, 2023
bluesky, I really enjoyed the Paul Bloom link. Went down a bit of a rabbit hole following his links and reading more of his stuff. Thanks.
posted by Well I never at 9:42 AM on October 10, 2023 [2 favorites]
posted by Well I never at 9:42 AM on October 10, 2023 [2 favorites]
I listen to the Hidden Brain psychology podcast periodically. I just went back to an episode I had started before reading the first link in the post. I noticed that I am listening with a much greater amount of skepticism now. As it turns out, I seem to enjoy the debunking of psychology as much as I have enjoyed the pre-debunking phase. Still, I like pop psychology books, and I probably always will. Thanks for the great FPP, Rhaomi!
posted by Bella Donna at 1:23 PM on October 10, 2023
posted by Bella Donna at 1:23 PM on October 10, 2023
Interesting, on the face it makes me think of the (discredited) Sapir-Whorf hypothesis, which contended that the words you use change the meaning of what you say (simplified). I'd be curious to know how they're prepared to get around that.
posted by rhizome at 1:12 PM
Interestingly, the debunking of the strong version of Sapir-Whorf has been debunked. In the last 15 years or so, there has been quite a bit of (IMO) very good research in psycholinguistics on this question. Much of it from Boroditsky (link goes to a Tedtalk. I'm sorry).
posted by bluesky43 at 4:16 PM on October 11, 2023 [4 favorites]
posted by rhizome at 1:12 PM
Interestingly, the debunking of the strong version of Sapir-Whorf has been debunked. In the last 15 years or so, there has been quite a bit of (IMO) very good research in psycholinguistics on this question. Much of it from Boroditsky (link goes to a Tedtalk. I'm sorry).
posted by bluesky43 at 4:16 PM on October 11, 2023 [4 favorites]
I remember when they made a TV show based on behavioral economics, which included a few items I still remember:
- the idea that microexpressions can be translated to specific mental states
- the idea that you can tell how long a woman has gone since orgasm by the way she walks
I've had both those ideas bouncing around my head. I bet neither replicated.
posted by rebent at 2:04 PM on October 24, 2023
- the idea that microexpressions can be translated to specific mental states
- the idea that you can tell how long a woman has gone since orgasm by the way she walks
I've had both those ideas bouncing around my head. I bet neither replicated.
posted by rebent at 2:04 PM on October 24, 2023
« Older Scrap the ‘fiesta’ fonts. | The plastic eating bacteria that could change the... Newer »
This thread has been archived and is closed to new comments
Welcome, oh psych, to the humanities!
posted by Fiasco da Gama at 9:39 PM on October 8, 2023 [25 favorites]