Join 3,551 readers in helping fund MetaFilter (Hide)


Menstrual cycles, biceps, and political attitudes
May 30, 2013 8:34 AM   Subscribe

Psychological Science, “the highest ranked empirical journal in psychology", recently published two controversial journal articles. One claims that ovulation might change women's political and religious views (PDF warning). The other tries to link physical strength with political conservatism. Some people disagree and find serious flaws in the methodology.

Andrew Gelman uses these examples to criticize the state of publishing in Psychology. Other's resorted straight to mocking.

CNN reported, with blowback.

Gelman has more.
posted by MisantropicPainforest (65 comments total) 9 users marked this as a favorite

 
he other tries to link physical strength with political conservatism. .

HAHAHAHA WHAT
posted by entropicamericana at 8:47 AM on May 30, 2013


Well, most men are stronger on the right.
posted by srboisvert at 8:50 AM on May 30, 2013 [12 favorites]


When I offered "two tickets to the gun show," I meant the actual gun show.
posted by Ghostride The Whip at 8:51 AM on May 30, 2013 [29 favorites]


Ovulates, has two tickets to the proverbial gun show, left of center: Theory debunked.
posted by The 10th Regiment of Foot at 8:55 AM on May 30, 2013 [3 favorites]


Huh. I was married when I voted for Obama (both times!). But it's a gay marriage so I reckon that probably doesn't count, and anyway as a butch dyke I'm probably not a "real" woman for the purposes of this study.
posted by rtha at 8:57 AM on May 30, 2013 [7 favorites]


I don't see any reason why the effects of ovulation shouldn't be studied, and deserve to be mocked. When someone resorts to mockery, I always feel like the other side must be just a little bit right.

But the link to strength and conservatism is an interesting one. Most conservative types I know believe themselves to be smarter and stronger than average. At least in a sense: "I pulled myself up by my bootstraps, why can't everyone else?" "I successfully fought my own lascivious, sweaty, homosexual urges and live a Christian life, why can't everyone else?" It's a sort of narcisism, that I think we are all susceptible to to some extent, where one is afraid to acknowledge just how weak they really are/were.
posted by gjc at 9:02 AM on May 30, 2013 [1 favorite]


Who said that the effects of ovulation shouldn't be studied?

I read all these links, perhaps I missed it.
posted by MisantropicPainforest at 9:05 AM on May 30, 2013


Most conservative types I know

Observation bias is not empirical data. Also, contrary to what many may think, lack of empathy for your fellow man is not strength.
posted by The 10th Regiment of Foot at 9:05 AM on May 30, 2013 [6 favorites]


You know how we all like to point and laugh at conservatives who cavalierly dismiss scientific findings that don't happen to accord with their worldviews?

This might be the shittiest study in the history of bogus psychology, but it would be nice if the criticism was "here's this specific thing wrong with their methodology" and not "that's not how I think the world works, therefore it's obviously a stupid study."
posted by yoink at 9:06 AM on May 30, 2013 [37 favorites]


The premise of the strength-conservativism paper isn't totally nutty. They reckon that men are still monkeys, and tend to stand up for themselves more when they have bigger muscles. They find a link between bicep size and willingness to express a strong preference for self-interested redistrisbution policies (that is, men with big biceps support low redistribution if they are wealthy, and high distribution if they are poor).

The authors' suppositions for why this is so (in our evolutionary past, strong monkeys could get what they wanted by force) are unproveable and maybe a bit offensive. But it's not too hard to come up with alternate explainations for the data.
posted by subdee at 9:08 AM on May 30, 2013


yoink,

you clearly did not read the Gelman links.
posted by MisantropicPainforest at 9:12 AM on May 30, 2013 [2 favorites]


The premise of the strength-conservativism paper isn't totally nutty.

I think what I find nutty isn't the premise (although I suspect it's wrong), but the piece's author's contention in the second paragraph that the premise seems obvious. I'd say that's nutty, extremely sloppy, and moderately offensive.
posted by penduluum at 9:14 AM on May 30, 2013


"to link physical strength with political conservatism"

yeah, cos Hitler was Buff as.
posted by marienbad at 9:15 AM on May 30, 2013


you clearly did not read the Gelman links.

I did; his criticism of the menstruation-cycle paper is about as hand-wavy as it is possible to be. His central contention is that it is impossible to draw inferences about the behavior of individual people from large-scale statistical studies (i.e., that the authors infer individual variances of political attitudes from their statistical studies of variance in larger groups). This is a highly dubious proposition which, if true, would invalidate almost all statistical research.
posted by yoink at 9:15 AM on May 30, 2013 [3 favorites]


yeah, cos Hitler was Buff as.

You do realize that you can't really refute a claim about general statistical correlations with individual counter-instances, right? I mean, that one devout Catholic you know who also supports abortion rights doesn't invalidate the proposition that devout Catholics tend to be more likely to oppose abortion than the average American.
posted by yoink at 9:17 AM on May 30, 2013 [1 favorite]


yes. it was a joke. Jeez.
posted by marienbad at 9:18 AM on May 30, 2013 [1 favorite]


I read a press release about that study elsewhere, and didn't it say that it wasn't just stronger means conservative but stronger rich men were conservative and stronger poor men were more liberal?
posted by Schmucko at 9:19 AM on May 30, 2013


You know who else couldn't take a joke?
posted by Schmucko at 9:19 AM on May 30, 2013 [1 favorite]


from the Gelman links:

'For example, R-sq for each of the three countries is very low. The regression line and 90% confidence intervals drawn for each of the three countries is devoid of the individual data points and thus, no visual sense of the variability. SES assessment was highly subjective and different in each country. In the U.S. and Argentina, the study relied on college students measuring college students’ biceps while in Denmark “a protocol was devised and presented to the subjects over the internet instructing them on how to measure their biceps correctly.” As to SES, the questions used were different in each country, supposedly justified by taking “into account country-specific factors regarding the political discussions on redistribution.”'

"Lots of options in this analysis. Again, these decisions may make perfect sense but they indicate the difficulty of taking these p-values at anything like face value. As always in such settings, the concern is not a simple “file-drawer effect” that a particular p-value was chosen out of some fixed number of options (so that, for example, a nominal p=0.003 should really be p=0.03) but that the data analysis can be altered at so many different points under the knowledge that low p-values are the goal. This can all be done in the context of completely reasonable scientific goals.

3. My first reaction when seeing a analysis of young men’s bicep size is that this could be a proxy for age. And, indeed, for the analyses from the two countries where the samples were college students, when age is thrown into the model, the coefficient for bicep size (or, as the authors put it, “upper-body strength”) goes away."

"But then comes the big problem. The key coefficient is the interaction between bicep size and socioeconomic status. But the analyses don’t adjust for the interaction between age and socioeconomic status. Now, it’s well known that political attitudes and political commitments change around that time: people start voting, and their attitudes become more partisan. I suppose Petersen et al. might argue that all this is simply a product of changing upper-body strength, but to me such a claim would be more than a bit of a stretch."


You read all that and concluded that it was "as hand-wavy as it is possible to be"?
posted by MisantropicPainforest at 9:21 AM on May 30, 2013 [4 favorites]


The biceps thing smelled a bit of p-value fishing to me. Was it really the initial hypothesis that men with big biceps support high redistribution if they're poor, and low redistribution if they're rich? Or did they start off looking to see if men with big biceps supported low redistribution; then when that failed, look to see if they supported high redistribution; then when that failed, look for any cross-break supporting biological determinism they could find?
posted by TheophileEscargot at 9:21 AM on May 30, 2013


...and, in other news, it turns out that there's a third type of damned lie.
posted by Mooski at 9:23 AM on May 30, 2013 [3 favorites]


I thought Gelman's points here were also pretty good:
1. Analyze all your data. For most of their analyses, the authors threw out all the data from participants who were PMS-ing or having their period. (“We also did not include women at the beginning of the ovulatory cycle (cycle days 1–6) or at the very end of the ovulatory cycle (cycle days 26–28) to avoid potential confounds due to premenstrual or menstrual symptoms.”) That’s a mistake. Instead of throwing out one-third of their data, they should’ve just included that other category in their analysis.

2. Present all your comparisons. The paper leads us through a hopscotch of comparisons and p-values. Better just to present everything. I have no idea if the researchers combed through everything and selected the best results, or if they simply made a bunch of somewhat arbitrary decisions throughout of what to look for.

For example, I would’ve liked to see a comparison of respondents in different parts of their cycle on variables such as birth year, party identification, marital status, etc etc. Just a whole damn table (even better would be a graph but, hey, I won’t get greedy here) showing these differences for every possible variable.

Instead, what do we get? Several pages full of averages, percentages, F tests, chi-squared tests, and p-values, all presented in paragraph form. Better to have all possible comparisons in one convenient table.

3. Make your data public. If the topic is worth studying, you should want others to be able to make rapid progress. If there’s some confidentiality restrictions, remove the respondents’ identifying information. Then post the data online.
posted by rtha at 9:23 AM on May 30, 2013 [6 favorites]


Yoink, Andrew Gelman is Professor of Statistics and Political Science and Director of the Applied Statistics Center at Columbia. He may be many things, but I seriously doubt he is in the business of "invalidat[ing] almost all statistical research." All things being equal, I trust him to comment seriously and correctly on methodological matters more than I trust any given psychologist to do their stats properly.
posted by col_pogo at 9:31 AM on May 30, 2013 [8 favorites]


Well, the Lancet published Andrew Wakefield's junk science, too. I'm just sayin'.
posted by Mental Wimp at 9:34 AM on May 30, 2013 [1 favorite]


Any scholastic attempt to relate Political Views with any Physical Condition will be used to discredit Psychology by those with the most psychotic political views.
posted by oneswellfoop at 9:39 AM on May 30, 2013


When I was younger, I would read goofy findings from psychology and somberly think: well, of course it sounds goofy, but we'll have to wait for more studies.

Having, in the intervening years, read the reports of innumerably goofy findings form psychology that turned out to be false, I now often give myself the freedom to thing: nope, that's idiotic. Of course I'll be wrong sometimes...but I do have a pretty well-tuned bullshit detector.

I think the real turning point was that ridiculous business about men "thinking like blonds"--i.e. getting dumber--after seeing pictures of attractive blond women. (Of course the dumbest part of that was the allegedly explanatory hypothesis.) Fer chrissake. What a mess.

I saw the biceps thing a few weeks ago. Yeah, it's possible. No, it's not likely to hold up. I don't know what to say other than that it has the ring of bullshit about it.

Sometimes goofy things turn out to be true. But most don't.
posted by Fists O'Fury at 9:44 AM on May 30, 2013 [3 favorites]


So, people that value athletic development over intellectual endeavors tend to be assholes? Who'd have thunk?!?
posted by The 10th Regiment of Foot at 9:48 AM on May 30, 2013 [1 favorite]


So, people that value athletic development over intellectual endeavors tend to be assholes? Who'd have thunk?!?

Looks like somebody got picked last in gym class.
posted by schroedinger at 9:55 AM on May 30, 2013 [5 favorites]


An important point, which Gelman made a number of times, is that the problem is not with the actual data collection of the studies themselves, it's with the conclusions the studies' authors, and in turn the news media, draw from the studies' results, conclusions that are completely outlandishly outside of the scope of their study.

The bicep study said nothing about all men everywhere and their political beliefs versus their "upper body strength"; rather, it found a sketchy correlation between bicep size and political beliefs as evaluated in a questionable way among a bunch of male college students in 3 random countries. That says absolutely nothing about our hominid ancestors. It doesn't even say anything about the correlation between the political beliefs of 40-year old Canadian men (a population not studied) and how much they can benchpress (not necessarily related to bicep size).

Similarly, with the relationship of ovulating women's marital status and their votes in the last election, the whole thing was done through Mechanical Turk, whose users undoubtedly are not representative of much of anything. And they didn't even attempt to find out if the women planned to vote for who they voted for before they ovulated, even though other surveys tells us that very few Americans (even American women! who might be ovulating!) changed their minds in the week before the 2012 election. And, of course, this doesn't tell us anything about the politics of, for instance, women with demanding full-time jobs who don't have time to use Mechanical Turk, women who are politically involved, or women who are on birth control and thus don't ovulate.
posted by hydropsyche at 9:56 AM on May 30, 2013 [1 favorite]


Looks like somebody got picked last in gym class.

In an EXTREMELY red state no less! Crimson in fact.
posted by The 10th Regiment of Foot at 10:00 AM on May 30, 2013 [1 favorite]


His central contention is that it is impossible to draw inferences about the behavior of individual people from large-scale statistical studies (i.e., that the authors infer individual variances of political attitudes from their statistical studies of variance in larger groups)

It's more like "If you try to infer within-person variance from between-person variance, and the results you get are simply not credible, then you need to look directly at within-person variance."

The results from the study were out to lunch: they found that among women in relationships, support for Romney almost doubled when they were ovulating. There is just no way that this could possibly be a real effect. If it were, it would have shown up in any of the nine billion panel studies of mass political behavior as women having massively chaotic political attitudes.
posted by ROU_Xenophobe at 10:03 AM on May 30, 2013 [7 favorites]


When I'm ovulating, it kicks off the worse PMDD ever. I want to rip off everyone's head and punt it down the hall

Yet my feelings against conceal carry has never changed.

You can all feel safe now.
posted by stormpooper at 10:07 AM on May 30, 2013 [2 favorites]


> There is just no way that this could possibly be a real effect. If it were, it would have shown up in any of the nine billion panel studies of mass political behavior as women having massively chaotic political attitudes.

I don't really believe the study myself - but I don't think your rebuttal is correct. The confidence levels of political polling just aren't that high, and the ovulation period just isn't that long (less than 10% of the monthly cycle).

I was going to do the math but caught myself in the nick of time - it's your responsibility to prove it to us! (Plus I need to do actual work. :-D)
posted by lupus_yonderboy at 10:12 AM on May 30, 2013


How the menstrual cycles affects competitive or risk-taking behavior in women has been a topic of study in experimental economics - try doing a search on Google Scholar on "menstrual cycle bidding"

Some examples:

Chen, Y., Katuščák, P., & Ozdenoren, E. (2012). Why canʼt a woman bid more like a man?. Games and Economic Behavior.
We investigate gender differences and menstrual cycle effects in first-price and second-price sealed-bid auctions with independent private values in a laboratory setting. We find that women bid significantly higher and earn significantly less than men do in the first-price auction, while we find no evidence of a gender difference in bidding or earnings in the second-price auction. Focusing on the first-price auction, we find that, while the gender gap in bidding and earnings persists over the entire course of the menstrual cycle, bidding of contraceptive pill users follows a sine-like pattern throughout the menstrual cycle, with higher than average bidding in the follicular phase and lower than average bidding in the luteal phase. In comparison, pill non-users have a flat bidding profile throughout the cycle.

Buser, T. (2012). The impact of the menstrual cycle and hormonal contraceptives on competitiveness. Journal of Economic Behavior & Organization, 83(1), 1-10.
We examine whether competitiveness in women is influenced by biological factors. Female participants in a laboratory experiment solve a simple arithmetics task first under a piece rate and then under a competitive tournament scheme. Participants can then choose which compensation scheme to apply in a third round. We find that the likelihood of selecting into the competitive environment varies strongly and significantly over the menstrual cycle and with the intake of hormonal contraceptives. The observed patterns are consistent with a negative impact of the sex hormone progesterone on competitiveness. We show that the effect of the menstrual cycle and hormonal contraceptives on competitiveness is due neither to an impact on performance, nor to an impact on risk aversion or overconfidence.

posted by needled at 10:13 AM on May 30, 2013


Metafilter: Others resort straight to mocking.

Q: Why does Maryland governor Martin O'Malley (D) wear sleeveless t-shirts?
A: Because Maryland law prohibits the carrying of concealed weapons.
posted by drlith at 10:16 AM on May 30, 2013


[Stop the personal attacks.]
posted by jessamyn at 10:18 AM on May 30, 2013


I'm probably not a "real" woman for the purposes of this study.

I think I laid my last egg about ten days ago; guess that makes me what? psst menopause hits us one and all
posted by infini at 10:28 AM on May 30, 2013


Their belief was that since bicep size as a proxy for upper body strength

I really doubt this, and also I really doubt that "bicep size" is a good stand-in for strength overall.

If you need me I'll be in the squat rack.
posted by the man of twists and turns at 10:38 AM on May 30, 2013 [1 favorite]


So don't obese people have really large biceps?
posted by The 10th Regiment of Foot at 10:47 AM on May 30, 2013 [1 favorite]


The 'science' of gender roles
The clip has to be seen to be believed, but for those who can't watch clips online, Dobbs said the Pew report is evidence of "society dissolving around us." Juan Williams said the more women become the "primary bread winner," the more we see "the disintegration of marriage." He added, "Left, right, I don't see how you can argue this."

Erick Erickson went even further:
"I am so used to liberals telling conservatives that they are anti-science. But I mean this is -- liberals who defend this and say it's not a bad thing are very anti-science. When you look at biology, look at the natural world, the roles of a male and female in society; in other animals the male typically is the dominant role, the female is not antithesis or is not competing; it's a complementary role."
Erickson added that "reality" tells us that "having mom as the primary breadwinner is bad for kids." This, Erickson concluded, is "a war on women."

The third male panelist -- there were no women guests on the program -- was Fox's Doug Schoen, who described this "a catastrophic issue" that "could undermine our social order."

Just so we're clear, this discussion, if we can call it that, took place yesterday, not in 1953.
posted by zombieflanders at 10:52 AM on May 30, 2013 [11 favorites]


Thanks for that, zombieflanders
posted by MisantropicPainforest at 11:04 AM on May 30, 2013


Always relevant in cases like these: The Science News Cycle
posted by lalochezia at 11:12 AM on May 30, 2013 [1 favorite]


Juan Williams said the more women become the "primary bread winner," the more we see "the disintegration of marriage."

If this is true, it's largely because of the second shift - that working women are still expect to be the primary care giver and home maker. Being married for a lot of women is just an extra burden without much gain for those households. In more egalitarian homes where child care and home keeping duties are equally divided, marriage isn't going to be a problem.

Yet, I wonder why that wasn't discussed?
posted by [insert clever name here] at 11:21 AM on May 30, 2013


When you look at biology, look at the natural world, the roles of a male and female in society; in other animals the male typically is the dominant role

I do not think this person has ever looked at biology or the natural world.
posted by elizardbits at 11:26 AM on May 30, 2013 [11 favorites]


Conservatives continue stepping on their own dicks. To serve their moneyed masters, they need to support low wages, which means multiple earners per household. But to serve their fundie masters, they need to be outraged that mothers are working. What's a disingenuous political movement to do?
posted by Mental Wimp at 11:29 AM on May 30, 2013 [3 favorites]


I'm reading the ovulation/voting one right now and one of the big problems I see is this:

We obtained from participants: (1) the start date of their last menstrual period and the previous menstrual period, (2) the expected start date of their next menstrual period, and (3) the typical length of their menstrual cycle.

And that just makes me go, "uggggh," because self-reported medical information is really, really unreliable a lot of the time. I mean, maybe they just happened to get women who tracked their cycles carefully and who also happened to be on Mechanical Turk? But I bet there is a huge margin of error for the dates here.

I have very little sympathy for the research itself, because the last thing we need is more media-generated "women are controlled by their hormones!" outcry, but I do kinda feel for the researchers in the sense that, man oh man, it is hard as balls to get this kinda info reliably unless you also have a medical lab that can track their cycle for them very carefully and who has funding for that?

Another major problem that I see is that they classified committed relationships only as living together, married, engaged. Women who were dating someone but did not meet those criteria were lumped in with single women. But if your whole argument is founded upon risk-taking and mate selection strategies in women, lumping women who may have an established relationship with their partner and simply don't live with them in the same category as women who have no social romantic ties at all seems like a very odd choice. Maybe the risk of taking on a new sexual partner during ovulation isn't as great if you don't live together, but the risk of taking a new sexual partner during ovulation if you're not partnered at all previously is, like, virtually nil. So it makes no sense to me to put everyone who isn't engaged, living together or married in the same category.

I also question the validity of the data gathered from Mechanical Turk, because I know at least 4 people who use it for small amounts of supplementary income-- which means they go through and answer all the questions on surveys as fast as they can with almost no actual thought put into it. Sometimes they choose answers blindly. Yeah, anecdata, I know, but the way Mechanical Turk is set up (very small payments for lots of tiny tasks), it seems like it encourages people to game the system by doing things as fast as possible. I'd like to see the survey replicated with a better random sample and recruited in a way that didn't facilitate just randomly clicking things.
posted by WidgetAlley at 11:30 AM on May 30, 2013 [2 favorites]


Just to say it again: it’s easy for me to stand on the sidelines taking potshots at other people’s work.

It's easy for me too. How cool is that!

In contrast, ovulation led married women to become more conservative, more religious, and more likely to vote for Mitt Romney.

Doesn't explain Wonder Woman.
posted by Smedleyman at 11:32 AM on May 30, 2013 [1 favorite]


Who said that the effects of ovulation shouldn't be studied?

There was hilarious work being done in the lab next to mine when I was in grad school where they looked at women’s ability to draw a horizontal line freehand on a wall during certain parts of their menstrual cycle. Apparently they found that there is a time of the month when this ability degrades. I have no idea if this result held up (not my area and I'm not that interested) but I do make sure there is a level handy whenever my wife and I hang pictures. Just in case.
posted by srboisvert at 11:44 AM on May 30, 2013


I also question the validity of the data gathered from Mechanical Turk, because I know at least 4 people who use it for small amounts of supplementary income-- which means they go through and answer all the questions on surveys as fast as they can with almost no actual thought put into it. Sometimes they choose answers blindly.

Recently I was talking to some researchers who were using Mechanical Turk for their research, which involved discrete tasks and not surveys, and they said a large chunk of time and effort in designing data collection went into putting checking mechanisms to guard against the behavior you describe. Mechanical Turk is a way to gather a lot of data fast, but a lot of garbage data is mixed in there.

In studies using Amazon Turk that I have read, I am used to seeing at least a paragraph describing what the researchers did to to filter out the garbage data (e.g. for survey studies, insert a number of trick questions to catch people just answering blindly). The linked Psychological Science paper has no such information, or any justification of why their survey data is valid.
posted by needled at 12:19 PM on May 30, 2013 [3 favorites]


yoink, you can't necessarily infer individual behavior from statistics taken from a population without further work to show that the data holds true at an individual level as well. This is the ecological inference fallacy.
posted by zug at 12:23 PM on May 30, 2013


"Women in serious relationships are likely to be especially invested in their relationships. They are likely to have been with their partner for a longer time, depend more on their partner for support (e.g., financial, emotional), have important aspects of their lives tied to the relationship, and have children together. Thus, women in invested relationships have considerably more to lose from the dissolution of the relationship."

Women in committed relationships are more committed to their relationships? You don't say.
posted by desuetude at 12:31 PM on May 30, 2013 [4 favorites]


Apparently they found that there is a time of the month when this ability degrades. I belive it. It's called "I don't give a shit because I"m moody as hell. Just hang up the @#$# picture already."
posted by stormpooper at 12:42 PM on May 30, 2013 [3 favorites]


needled that is way fascinating! Can you link me to any discussions on research methods using Mechanical Turk in those papers? It's a possible avenue for my own research in the distant future, so I'd be interested to know what the pitfalls are well in advance.
posted by WidgetAlley at 12:45 PM on May 30, 2013


Thank God I, a married hetero, voted last year for Obama when I was pregnant. Threw their curve off, didn't I?!
posted by Leezie at 3:11 PM on May 30, 2013 [1 favorite]


yoink, you can't necessarily infer individual behavior from statistics taken from a population without further work to show that the data holds true at an individual level as well. This is the ecological inference fallacy.

Sure: but absolutely no one involved in the study is claiming "this is the final and definitive word on this phenomenon which has now been proven beyond any possible doubt." This is what I find tediously predictable about the way studies like this get received (and, in fact, this is true of statistical studies in general) depending on whether they flatter or contradict our world view. If a paper making exactly the same kind of statistical inference produces a result that accords with our world view we say "hmm, yes, this is a very interesting and suggestive result that calls for further study" and we file it away under "studies that generally suggest my view of the world is correct." When one comes along whose results we don't like then we dismiss the whole thing out of hand: "oh pish, tosh: why, I can invent a thousand more-or-less plausible scenarios that could also account for these results." Well, yeah, you could. You could do that with pretty much any statistical result you care to name. But that is arguing with a straw man. The point of these statistical findings is not to say "we have proven that X causes Y" the point is simply to say "there is a strong enough correlation here and we have eliminated enough of the obvious potential confounders to suggest that we have found something that merits further investigation."
posted by yoink at 6:15 PM on May 30, 2013 [1 favorite]


But they haven't really eliminated any of the obvious potential confounders, or even considered many of them. And the authors themselves are the ones immediately linking these to our ape-like ancestor and extrapolating to the world's population. As Andrew Gelman says, these methods and conclusions are so scientifically flawed that it is really not clear what they are doing in a supposedly high profile journal like Psychological Science when they barely belong in PLOS-ONE. If the purpose of these studies is to provide preliminary data to raise questions and inspire further study, they should be published in a low-profile journal to be read by other psychologists who are capable of designing better studies, not published in Psychological Science and accompanied by press releases and interviews with CNN.

And yes, before you ask, I am exactly as critical of poor methods and conclusions in the equivalent journals in my field. I am a tough peer reviewer and proud of that fact.
posted by hydropsyche at 4:34 AM on May 31, 2013 [1 favorite]


When one comes along whose results we don't like then we dismiss the whole thing out of hand: "oh pish, tosh: why, I can invent a thousand more-or-less plausible scenarios that could also account for these results." Well, yeah, you could. You could do that with pretty much any statistical result you care to name. But that is arguing with a straw man. The point of these statistical findings is not to say "we have proven that X causes Y" the point is simply to say "there is a strong enough correlation here and we have eliminated enough of the obvious potential confounders to suggest that we have found something that merits further investigation."

The point is, that as Gelman and others have argued, is that these two studies don't even do that. They are very very fundamentally flawed studies that have very suspect methodologies and the conclusions should not be derivable from the evidenced offered. This is not a criticism that could be launched against other, more rigorous studies, ones that get published in top journals in other fields.

The major criticisms aren't "Well have they thought about this possible scenario???" (although they exist). The major criticisms are "there are fundamental problems with the statistical analysis and research design used that call the whole thing into question, and anyone with moderately advance statistical training would be able to spot these flaws, because they are so glaringly obvious."
posted by MisantropicPainforest at 5:51 AM on May 31, 2013 [2 favorites]


Related, amusing.
posted by a robot made out of meat at 7:50 AM on May 31, 2013 [3 favorites]


If a paper making exactly the same kind of statistical inference produces a result that accords with our world view we say "hmm, yes, this is a very interesting and suggestive result that calls for further study"...

There is some truth to what you say in general, but the cited studies are laughably badly done. They deserve to be ridiculed, as they should be whether they support or refute ones world view. I certainly do not believe the "fatal flaw" approach to studies, wherein finding one flaw implies sapping a study of all meaning, but when the study design doesn't even address the purported conclusions and the statistical approach is amateurishly wrong-headed, then, yes, the study is worthless for any practical purpose other than mockery.
posted by Mental Wimp at 10:34 AM on May 31, 2013 [1 favorite]


Irrespective of the massive statistical deficiencies, attempting to represent "physical strength" via "bicep size" is so laughably inaccurate it serves as further demonstration of the incompetence of the researchers involved.
posted by schroedinger at 12:02 PM on May 31, 2013 [1 favorite]


Irrespective of the massive statistical deficiencies, attempting to represent "physical strength" via "bicep size" is so laughably inaccurate it serves as further demonstration of the incompetence of the researchers involved.

I can't (and don't wish to) speak for the work or conclusions in the study itself, but both you and the OP have wrongly framed the paper. It does not link physical strength to conservatism. Silly evo-psych is silly, but bad science reporting doesn't help, either.

The paper is specifically about upper body strength in men, represented synecdochically by bicep size, and it does not conclude that greater bicep size in men leads to conservatism. Rather, it asserts that there is some correlation between men with greater bicep size and whether they assume what is dubbed "self-beneficial" economic approach: that is to say, large bicep men of lower socioeconomic status were more inclined to support redistribution than similarly situated men with smaller biceps, whereas large bicep men of higher socioeconomic status were more inclined to oppose redistribution than similarly situated men with smaller biceps. The interplay between bicep size and the socioeconomic status of the man in question is what takes the authors' conclusion away from a mere link from upper body strength to any particular worldview. It is also interesting to note that the authors could find no such link in the women that they had studied.

Again, I'm not saying that the study is any good, but it is factually incorrect to say that it links physical strength to conservatism. At best, that's a sloppy oversimplification.

Some of Gelman's criticisms are a little odd as well. He no doubt knows more about statistics than I do, but I do know a thing or two about reading comprehension. For example, in his critique of the paper, he says:
Of course we can’t criticize the APS from running a press release, given that the paper was accepted in their journal. But the press release itself is a mix of caveman speculation and a misleading description of the research. It’s misleading to speak of “wealthy men” when two of the surveys were of college students).
Note the quotes around "wealthy men". While the press release uses the phrase "wealthy men", the paper most certainly does not. Is Gelman only criticizing the press release? The paper talks about people of high and low socioeconomic status, which is a different concept. While there's plenty of room to criticize a study that relies heavily on college students, there are most certainly college students of high and low socioeconomic status.
posted by Sticherbeast at 2:21 PM on May 31, 2013


"Note the quotes around "wealthy men". While the press release uses the phrase "wealthy men", the paper most certainly does not. Is Gelman only criticizing the press release? "

In that particular blog post, yes.
posted by MisantropicPainforest at 2:26 PM on May 31, 2013 [1 favorite]


By the way, Andrew Gelman is a highly regarded statistician, with many highly cited publications which can be read here.
posted by Mental Wimp at 2:39 PM on May 31, 2013


"Note the quotes around "wealthy men". While the press release uses the phrase "wealthy men", the paper most certainly does not. Is Gelman only criticizing the press release? "

In that particular blog post, yes.


The title of the blog post is "More on those “Psychological Science” papers" [emph. mine], not "...press releases". He frequently criticizes the paper itself, and not just the press release as a press release. For example [once again emph. mine]:
Just to relive these for a moment, the papers are:

“The Ancestral Logic of Politics: Upper-Body Strength Regulates Men’s Assertion of Self-Interest Over Economic Redistribution”: 3 surveys, two of which were of college students, no actual direct measures of upper-body strength, regulation, assertion, or self-interest.
Indeed, Gelman has a more nuanced (but still critical!) review of the actual paper here, in which he explains how the bicep paper is "of a higher quality" and "much more sensible" than the menstruation paper. I'm not sure why you didn't link to that blog post, because it's actually very interesting. Gelman himself says that the data and purported correlations themselves aren't really all that bad ("...the correlations do not seem completely unreasonable..."), but the overreaching and conclusory evo-psych framing are deeply problematic.

Yes, in the Gelman blog post you did link to, he does criticize the press release for its "misleading description of the research". This is what makes it ironic and strange when the paper is once again referenced with a similarly misleading description, especially in the context of a blog post criticizing the press release for such misleading framing.
posted by Sticherbeast at 6:37 PM on May 31, 2013


The paper is specifically about upper body strength in men, represented synecdochically by bicep size

Yes, I know, and my argument is that is a piss-poor representation of upper body strength. It's akin to saying you're relating intelligence to politics, then measuring intelligence solely through ability to add up numbers in one's head.
posted by schroedinger at 11:45 PM on May 31, 2013


« Older I’m 23 And I Still Have A Babysitter! I Really Hop...  |  A pop-up feast based on dishes... Newer »


This thread has been archived and is closed to new comments