Shoddy Experiments and the Newspapers Who Love Them

November 15, 2008 1:49 PM

An experiment published in Biology Letters has been interpretted by the press in different ways. The Daily Mail: "Women over the age of 50 may be less frisky, less nimble and less cute but, as if by way of compensation, they are also a lot less bitchy." The Telegraph: "So today’s research published in Biology Letters, that finds women become less bitchy when they are older, is frankly, unsurprising. The mistake however is that the researchers thought the decline in bitchiness came post-menopause, when they should have set the benchmark as post-puberty." Dr. Petra Boynton's take: "If we don’t take action we’re going to keep on seeing this carnival of poor sex science being promoted. With poor media coverage following as a result."
posted by Marisa Stole the Precious Thing (33 comments total)

carnival of poor sex science

Didn't they tour with the Jim Rose Circus Sideshow?

sorry
posted by krinklyfig at 1:53 PM on November 15, 2008 [1 favorite]

November Fools Day?
posted by facetious at 1:55 PM on November 15, 2008

Thanks for reminding me of the Jim Rose Circus, I just barely managed to forget seeing them 10 years ago and now it's all coming back.
posted by baphomet at 1:59 PM on November 15, 2008

Metafilter: Carnival of Poor Sex Science
posted by Korou at 2:01 PM on November 15, 2008

So is "bitchiness" now a form of scientific measurement? What?

You know, I'm not even going to read these articles, as I may experience the onset of pre-50 year old bitchiness.
posted by jokeefe at 2:16 PM on November 15, 2008

Okay, so the experiment is one of those legendarily stupid ones:

We investigated circum-menopausal women's preferences for femininity in the faces of young adult men and women. Post-menopausal women demonstrated stronger preferences for femininity in same-sex faces than pre-menopausal women did. This effect was independent of possible effects of participant's age and suggests that dislike of feminine (i.e. attractive) same-sex competitors decreases as fertility decreases.

However, I can attest-- and some of my friends have experienced this too-- that young women have never looked do beautiful to me as they do now, that I am nearly 50. It's not to do with sexual attraction, though; it could be nostalgia, or a powerful awareness of how fleeting youth is, or perhaps something almost maternal.

However, some of the experiment's premises (feminity = attractive) is problematic to begin with, not to mention the "dislike of same-sex competitors". FFS.
posted by jokeefe at 2:24 PM on November 15, 2008 [1 favorite]

Research dismissed or warped depending on political and social bias of comment-obsessed media. Bitter weeping at decline of British journalism at 11.
posted by athenian at 2:49 PM on November 15, 2008 [1 favorite]

No, it's when Mr. Dead shows up unannounced and parks his ass on the living room couch. You ask him to leave and he just looks at you and smiles and takes another sip outta whatever you have in the house and then looks back at you, still grinning, saying "I got time."

That's when you realize who the real enemy is and suddenly a night out with the girls sounds quite nice.
posted by Brandon Blatcher at 2:55 PM on November 15, 2008

So, some of this science reporting is genuinely bad.

On the other hand, the last-linked comment looks like PC flailing. The "I wouldn't have published this" paragraph smacks of applying more stringent standards to science that is more politically threatening, an approach that is mathematically guaranteed to bias published results further from the truth. The proposed cultural explanation doesn't work on its own terms, because its argument applies with equal force to older and younger women -- there's no reason to believe it explains the difference. Perhaps some more work should be done on that explanation, but in context it's just another terrible excuse for sanctimony.
posted by grobstein at 3:11 PM on November 15, 2008 [1 favorite]

I've heard that people become more 'androgynous' as they get older, it may be that older women are simply more physically attracted to young women the way men would be.
posted by delmoi at 3:36 PM on November 15, 2008

Research dismissed or warped depending on political and social bias of comment-obsessed media.
And commenting MetaFilterites.

Because the results of an experiment may conflict with what a person prefers to think is true, does not make the experiment "shoddy". This is what distinguishes science from magical thinking. There is nothing in the first page of the paper (I couldn't access the full text) that indicates to me that the method of the experiment deserves the description of "shoddy", nor is the conclusion unreasonable or unfounded on the data. They're measuring face preference, ie asking people "of X and Y, who is prettier/uglier to you?" That's a valid thing to measure.

Also, it's well-known that faces can be feminized and masculinized through the use of algorithms as described. The human species, despite the insistence of a number of people (the gender relations equivalent of creation scientists, IMO) that we are not, is sexually dimorphic both in physiology and psychology. It is reasonable to measure characteristic behaviors of both genders, and in the course of this, gather and properly account for inter-gender and cross-gender behaviors.

It is also reasonable to do this for races (and the smallest unit of race: families), for age, for socioeconomics, for average temperature during gestation (the likely origin of the "star signs" effect), and for a wide variety of other factors of varying political and social palatability. None of these have any inherent bearing on moral equality or human rights or opportunity: any given individual is an individual. To my mind the error of prejudice is not to say "Japanese males are on average this tall", which is a fact; it is to say "You, Hiro, could not possibly be 205cm tall because Japanese males are not so tall", which is factually and ethically wrong. Or "women are good with kids, therefore Alice is good with kids" and so on. On the other hand, it's reasonable for tall Hiro to mention his height when booking a sleeping car on a long train journey (it would be foolish of him not to do so), and it's reasonable, assuming a child is found in the wilderness by Bob, Fred, and Alice, for Alice to have the first option to take responsibility for the child. The error is failure to adapt to the facts, to adjust for the individual: if the train company says "Japanese name, ignore the rest, give him the short beds", or if Bob and Fred insist that Alice look after the child despite the fact that Alice, of the three, is the best orienteer.

I'm not saying "it's okay to be prejudiced as long as you fix it", I'm saying: in the absence of individual data, it doesn't matter whether it's "okay" to use aggregate data; there's no other choice, and to use no data at all usually isn't an option. That's what stereotyping, in the human cognitive process, is for.

So, paraphrasing the abstract, "on the average, postmenopausal women" (objectively establishable) "feel more emotional warmth" (a personal opinion, the variable being measured by the research) "towards beautiful" (measured by conglomeration of average opinions; while not objective as such, certainly possible to establish as a control variable) "young women" (close enough to objective) "than women of the same age as those depicted". Alice may like pretty-faced girls; Betty may not. But over 97 women, things even out. What makes that "shoddy"?

Research that examines stereotyping and related concepts--as seen here--is very frequently seized upon and misrepresented by idiots and liars for political purposes, or (and this is a different thing) somewhat misrepresented and aspects of it emphasised for the entertainment of the public. This is not a flaw in the science, it's a flaw in the politics.
posted by aeschenkarnos at 3:48 PM on November 15, 2008 [7 favorites]

"Women over the age of 50 may be less frisky, less nimble and less cute but, as if by way of compensation, they are also a lot less bitchy."

Heh, so I've got something to look forward to.

*ducks*
posted by nola at 4:18 PM on November 15, 2008

Certain relatives of mine disprove this theory handily.
posted by St. Alia of the Bunnies at 4:39 PM on November 15, 2008 [1 favorite]

aeschenkarnos, research like this gets sent out in sexy oversimplified press releases and eaten up by the media which simplify them even more. So, garbage presentation. But consider the value of the science on its own. Usually these studies have very small sample sizes or samples that are unrepresentative of a larger population (eg, all white women who were free on Tuesday mornings when the research was being done, living in a relatively well-off and educated area around a university, who voluntarily came in for the study) -- but the findings get reported by the researchers or the university press office as if they represent a discovery about all women, and eaten up by the press and public because they support stereotypes or give ammunition (science proves old broads are bitchy and jealous! I saw it in the paper) to watercooler jawing.

Research on heavily socially freighted norms like this has a *terrible* track record; we're just not good at doing solid science on areas like gender and race where we have so many prejudices etc. Research in areas like this has a huge initial bar to overcome, in my view - it should be viewed with the very highest kind of skepticism, whether it seems to support or undermine our political views.
posted by LobsterMitten at 5:12 PM on November 15, 2008 [2 favorites]

I like em bitchy.
posted by Astro Zombie at 5:32 PM on November 15, 2008 [1 favorite]

To be clearer, what makes the research shoddy is usually things like this:

-small sample sizes

-unrepresentative sample, given the population the articles try to generalize over (eg sample will include only women from the town where the research is done, but articles talk about "women")

-subjective judgments like beauty/femininity operationalized and treated as if they are more objective than they are (eg we'll get undergrads to rank faces by beauty, then use those measures as measures of objective beauty when comparing to the preferences of older women. then the results get reported as "older women prefer beautiful faces" rather than "older women in this town, of a certain demographic group, prefer faces that are also preferred by college students in this town, of a certain demographic group".)

There are assumptions being made here (eg that beauty rankings can be universalized) that are unwarranted. The assumptions are being made in order to sex up the story and make it more marketable. (Why are studies like this being done except to be marketable? They're marketed to the press as if they reveal truths about human nature generally, when really they are more suited to revealing things about very specific groups - useful for designing ads to appeal to that group, but not revealing about general human nature unless we make a lot of big assumptions)

-conclusions drawn about how all this relates to mating and evolutionary fitness when there is no independent reason to believe those conclusions. They are just-so stories added to make the whole business sound more scientific. Here the authors say they think the increased preference for feminine faces over time might be because younger women dislike other young women, regarding them as competitors for mates, but once they're past menopause they don't need to dislike young women because they're not competing anymore. Two problems with this kind of speculation:
-Given the sample, there's no reason to think the change in preferences is universal (eg maybe European women 100 years ago would have had different preferences but today's women are conditioned by advertising in a certain way; maybe women from different cultures would have different preferences, etc).
-Even supposing it were universal, why not say it is because post-menopausal women are nostalgic for their children and prefer to look at childlike faces? Or that post-menopausal women prefer feminine faces because they think feminine women would be pliant daughters-in-law or more easily overpowered in a struggle for resources? Or more fertile daughters-in-law? Or better caretakers of elderly women? Or because they just begin to hate their peers more as they age? Or because in the environment of evolutionary adaptation, the traits that code for femininity today (highly contingent) instead coded for something else? Who knows - you can spin dozens of different post-hoc evolutionary justifications. Adding the evolutionary justification doesn't make it more scientific, it's sheer speculation based on no evidence at all. (And it's often where the politics is inserted - hyper-individualistic assumptions are made to sound like they are the inevitable result of evolutionary theory, when they aren't and would need to be supported by independent evidence.)
posted by LobsterMitten at 5:35 PM on November 15, 2008 [10 favorites]

aeschenkarnos, what if you simply disagree with the fact that research that focuses strictly on gender differences (even more so than on race) is inherently suspect because the group under study is too large? Slightly over half the human race. What is that now, more than 3 billion individuals? Can you really make many sweeping statements about so large a group, who are of necessity scattered in such diverse situations that normal research tools like control groups (what would a control group for half of humanity look like, anyway?) are inadequate?

Certainly to the point that 97 individuals is laughably inadequate? This is one of my personal objections to the pursuit of these studies; they do not make clear the limits of their conclusions. And as LobsterMitten points out, the track record on such studies is not a good one, historically speaking.

Nothing is more freighted with cultural weight, in terms of training and expectations, than gender, in every human society; how do you possibly establish what behavior is or isn't a result of that training and expectations, without a large group of women (or men, for that matter) who were raised outside of them? We know already that changes in opportunities for women create changes in women's achievement, despite previous "scientific" assertions that women were incapable of certain tasks because of biological, gender-based differences.

And there *is* no reason, unless she's lactating and the child is an infant, for Alice in your example to be the one to take charge of a lost child. Women are not necessarily more nurturing (esp. to kids not their own), and childcare skills are learnt, not innate.
posted by emjaybee at 5:39 PM on November 15, 2008 [2 favorites]

So is "bitchiness" now a form of scientific measurement? What?

They measure the half-life of bitchitrons.
posted by jason's_planet at 5:42 PM on November 15, 2008

So is "bitchiness" now a form of scientific measurement?


I, for one, look forward to further findings in this fascinating new science of Bitchology. With this new information, we may hone and pefect our bitchiness. Someday, our bitchiness may be faster, stronger, more accurate, and more efficient. Groundbreaking studies like this, for instance, pave the way for a future wherein we can maintain peak bitchery well into our seventies, possibly beyond.

I say this as the member of a family whose female members have unusually robust and long lived bitchitrons, and have been known to keep bitching robustly well into their nineties. There may be a positive correlation with the relative smartness of our asses, but early studies have so far been inconclusive.
posted by louche mustachio at 6:01 PM on November 15, 2008

Women over the age of 50 may be less frisky, less nimble and less cute but, as if by way of compensation, they...

Oh man, I thought we were talking about something else altogether.
posted by Mr. President Dr. Steve Elvis America at 6:04 PM on November 15, 2008

I get less bitchy every year or so, and I'm not even a woman.
posted by StickyCarpet at 6:09 PM on November 15, 2008

The only thing I could possibly add to Lobstermitten's excellent comment is to note that the sample of 97 women in this study were not only self-selected, but did the quiz/experiment online. So there's not even a guarantee, apart from their own reporting, that this was not a sample of 97 bored 19 year old fratboys.
posted by jokeefe at 6:20 PM on November 15, 2008 [3 favorites]

You critics are relying too heavily on the small-sample objection (and related points). The problem: in order to justify a large, foolproof study, we need work like this to show that there's something to look for.

If you would object just as vehemently (or more) to a bigger, better-designed study of the same issues, it's disingenuous to object to the small size of the sample in this case.
posted by grobstein at 7:50 PM on November 15, 2008 [1 favorite]

In other scientific news, men's "old fartiness" got progressively worse with age...
posted by Alexandra Kitty at 8:21 PM on November 15, 2008

Dailymail's version, when compared with younger women, we’re a lovely bunch of sweethearts, can really be great for those ladies. But as it says, younger women see themselves as rivals, I can't quite agree. Being a woman myself, I have found that everyone has his/her own opinion on anything, and I never saw anyone with a rivalrous eye.
posted by susanharper at 8:46 PM on November 15, 2008

grobstein: I take it that the point of the post is the sloppiness of this stuff, and the public's eagerness to eat it up.

One, the popular media eat this kind of sloppy science up and report it in ways that feed into "battle of the sexes" or "battle among women" mythology, which is socially damaging. They overextend the conclusions the researchers themselves draw. This is the main point made by the original post.

Two, research like this is often scientifically sloppy (or worse) to begin with, and the researchers often draw conclusions that are unsupported by the evidence they have gathered. (In this case it sounds like it may be SO sloppy that it doesn't even underwrite the conclusion "that there's something to look for". For all the evidence this study gives us, there may be nothing to look for.)

Why does this happen? Strong incentives for all parties to get a result that seems sexy or exciting.

A result is sexier if:

a. it's described as revealing a universal commonality of all humans, or all women, etc. Not just "women in this town" or "women of this demographic in this town".

b. it's described as revealing a biological commonality, not something that's largely a product of socialization

c. it's described as being a having some evolutionary purpose. (Humans evolved, but it's silly to think that every human behavior is best understood as something that served a purpose in the evolutionary history of humans. Plenty of our behaviors did not serve an evolutionary purpose, but are instead products of culture etc. So even if you discover something that humans have in common, you can't just go off positing its evolutionary purpose with no further evidence.)

d. it seems to ratify certain kinds of stereotypes/conventional wisdom (science reveals that women really do love makeup and chocolate! gay men are genetically bred to love interior decorating! black folks are lazy and impulsive but they're great athletes! men are bad lovers!). The media love this because it makes good water-cooler stories.

Also, people are more likely to accept results that accord with their own prejudices. This is generally okay, we need plausibility checks in science -- but in cases where there are heavy social prejudices, and the goal of research is to discover whether those social prejudices are true (or what they rest on), our plausibility check is not a reliable guide to the truth.

So I'm skeptical about the value of research on questions of this sort (what do women or men find attractive? - especially when we're going to pitch this as a phony "universal"). It seems to me this is an area where the science is pretty much inevitably going to be bad, because there's no incentive to find the truth (no bridges will fall down if scientists screw it up), there are only incentives to get sexy results or accord with existing prejudices. Then those bad results get the stamp of "science" on them, and people say things like "oh, just because you disagree ideologically, doesn't mean you can dispute the cold hard objective findings of science!" Feh.
posted by LobsterMitten at 9:33 PM on November 15, 2008 [5 favorites]

With these attractiveness studies, there's also a strong impulse to stamp whatever our culture finds attractive as being "naturally" the most attractive - not merely contingent, and not merely one among many choices that are equally natural. I don't really get why this impulse is so strong, but it comes out again and again in these studies that are based on like 30 undergrads rating pictures, and lo and behold they rate others attractive if they fit some conventional present-day media definition of attractive, and then the conclusion is "all humans find x,y,z attractive; this served evolutionary purpose b".
posted by LobsterMitten at 9:39 PM on November 15, 2008

Yep, I saw this at New Scientist yesterday and had a good head-shake; somebody's going to put an eye out with all this reckless conclusion jumping, I tell you.

So among the pre-menopausal women tested (only ages 40-64, oddly), they seem to prefer more masculine features. Hmmmmmmmm. What could it mean? What? What? Ohhhh... right! The answer is right there staring us in the face - they're bitches!
posted by taz at 10:00 PM on November 15, 2008 [2 favorites]

I'm going to put Bitchologist on my business cards.
posted by sugarfish at 10:08 PM on November 15, 2008

So is "bitchiness" now a form of scientific measurement? What?
They measure the half-life of bitchitrons.

Magnitude, however, is measured on using the logarithmic Harridan scale.

In other scientific news, men's "old fartiness" got progressively worse with age...

Similarly, this trait is measured in Beanos. My personal best is 7.8 on the Beano scale, with aftershocks hitting 9.0 on occasion.
posted by Enron Hubbard at 5:16 AM on November 16, 2008 [1 favorite]

sugarfish: maybe you can go into business with this woman.
posted by msconduct at 6:55 AM on November 16, 2008

As well as the sampling issues, aren't there all sorts of unwarranted assumptions about normality in the underlying population distributions here? What if, for example, astrology turns out to be true, and Libras are over represented in the sample? Or, perhaps more realistically, if the underlying populations are havily skewed, so that the notion of an average *anything* is simply suspect?
posted by fcummins at 12:56 PM on November 16, 2008

Ever stood on an old woman's lawn?

I refute it thus!
posted by turgid dahlia at 2:27 PM on November 16, 2008