Does research assessment discriminate against female academics?
March 21, 2014 12:02 AM Subscribe
"professorial women were rounded up in a meeting room, offered unappealing sandwiches." Classics professor Barbara Graziosi in the Times Higher Education: "But I am not sure that these inequalities quite explain the low submission of women in the RAE and now, perhaps, the REF. Is there something that favours men in the way assessment exercises are set up? Or are women simply less good at research?"
Quick acronym index: REF is "Research Excellence Framework" and RAE is "Research Assessment Exercise"
Quick acronym index: REF is "Research Excellence Framework" and RAE is "Research Assessment Exercise"
While waiting for official reports on this round of assessment, I decided to ask my Facebook friends about women and the REF. Within hours I had a stream of useful comment.
A rigorous scientific inquiry iwould also take into account YouTube comments.
posted by three blind mice at 12:51 AM on March 21, 2014 [9 favorites]
A rigorous scientific inquiry iwould also take into account YouTube comments.
posted by three blind mice at 12:51 AM on March 21, 2014 [9 favorites]
I then thought about the importance of using networks and did something bold. I have been writing book reviews for Times Higher Education for years and considered that I could offer to write an article. I tentatively suggested this to my contact person, the books editor. It turns out that some nine in 10 academics who approach her with unsolicited offers to write for the magazine, or petition for their own books to be reviewed, are men. We both celebrated the fact that I was helping to balance out the figures.
The hierarchy of data (scientific experiments > archival court records > oral history > anecdata) is a constant conundrum. We ought to be concerned with the validity of data. But using Facebook as one data source in her triangulation of her research inquiry on research assessments seems fine to me. Or maybe I'm sarcasm-blind today and three blind mice is actually engaging in quantitative analyses of YouTube comments.
posted by spamandkimchi at 12:57 AM on March 21, 2014 [4 favorites]
The hierarchy of data (scientific experiments > archival court records > oral history > anecdata) is a constant conundrum. We ought to be concerned with the validity of data. But using Facebook as one data source in her triangulation of her research inquiry on research assessments seems fine to me. Or maybe I'm sarcasm-blind today and three blind mice is actually engaging in quantitative analyses of YouTube comments.
posted by spamandkimchi at 12:57 AM on March 21, 2014 [4 favorites]
In the sciences, multi-authored papers can be a problem: when researchers work in the same institution, they cannot all claim the output. A second or third author of a paper will not usually be able to submit it. (Again this is likely to affect women more acutely than men because there are proportionally more junior than senior women, and hence more second and third authors, and because women tend to encounter greater family obstacles if they want to move institutions or collaborate closely with long-distance colleagues.)
This? There is so much meaning packed into first-author and last-author status on papers. It looks like a system needing expiration.
Anecdatadly, I have witnessed that when an academic is at the top of the heap, the gender of that person matters very little regarding the treatment of women in publishing.
I did RTFA but maybe I missed it: the ultimate gatekeepers are the editors of journals, isn't that the prime location of the problem?
Anyway, it is ripe for change. I hope to see some ASAP.
posted by drowsy at 4:15 AM on March 21, 2014 [1 favorite]
This? There is so much meaning packed into first-author and last-author status on papers. It looks like a system needing expiration.
Anecdatadly, I have witnessed that when an academic is at the top of the heap, the gender of that person matters very little regarding the treatment of women in publishing.
I did RTFA but maybe I missed it: the ultimate gatekeepers are the editors of journals, isn't that the prime location of the problem?
Anyway, it is ripe for change. I hope to see some ASAP.
posted by drowsy at 4:15 AM on March 21, 2014 [1 favorite]
"But using Facebook as one data source in her triangulation of her research inquiry on research assessments seems fine to me. "
Since a PhD is likely to have other PhD's as Facebook friends, the results are not necessarily substantially different than the informal conversations one has in hallways and over lunch with colleagues (either in your department or at conferences). Sometimes those conversations do generate quite interesting insights into your current research.
posted by oddman at 5:22 AM on March 21, 2014 [1 favorite]
Since a PhD is likely to have other PhD's as Facebook friends, the results are not necessarily substantially different than the informal conversations one has in hallways and over lunch with colleagues (either in your department or at conferences). Sometimes those conversations do generate quite interesting insights into your current research.
posted by oddman at 5:22 AM on March 21, 2014 [1 favorite]
Another issue is that the REF (and universities' emphasis on new hires being already "REF-able") places particular stresses on early career researchers. If you're unemployed, struggling on part-time contracts post-PhD, or you've taken one of those fixed-term "early career posts" that turns out to be an under-appreciated, teaching-heavy nightmare, you'll struggle to produce the four 3* or 4* publications you need to get over the application hurdle for your first permanent academic position.
It's at this "early career" stage, of course, that a lot of the more ruthless gender and class "weeding out" takes place. I don't have proof that young female academics, or academics from less-privileged backgrounds, are statistically more likely to drop out of the market or, at best, be "offered" the kinds of exploitative early career gigs (fixed-term teaching fellowships, etc.) that turn out to be REF poison later on, but I have my suspicions. As with so many things in the UK, academia is controlled by a small, incestuous Oxbridge elite and gender and class warfare is how that elite rolls.
posted by Sonny Jim at 6:00 AM on March 21, 2014 [5 favorites]
It's at this "early career" stage, of course, that a lot of the more ruthless gender and class "weeding out" takes place. I don't have proof that young female academics, or academics from less-privileged backgrounds, are statistically more likely to drop out of the market or, at best, be "offered" the kinds of exploitative early career gigs (fixed-term teaching fellowships, etc.) that turn out to be REF poison later on, but I have my suspicions. As with so many things in the UK, academia is controlled by a small, incestuous Oxbridge elite and gender and class warfare is how that elite rolls.
posted by Sonny Jim at 6:00 AM on March 21, 2014 [5 favorites]
...apparently one of the issues here is social politeness. Textbook contributions, book reviews and articles promised for conference proceedings do not necessarily make good REF outputs, but still need to be delivered, unless we are prepared to say “no”.
Well, yes, you should be, have to be prepared to say no (sometimes). Every researcher and academic I know says no regularly to things that are distractions from what they need to focus on: teaching and their own research. This issue isn't about devaluing women's contributions, it's about being able to deal with competing needs for your valuable time.
posted by bonehead at 6:10 AM on March 21, 2014 [2 favorites]
Well, yes, you should be, have to be prepared to say no (sometimes). Every researcher and academic I know says no regularly to things that are distractions from what they need to focus on: teaching and their own research. This issue isn't about devaluing women's contributions, it's about being able to deal with competing needs for your valuable time.
posted by bonehead at 6:10 AM on March 21, 2014 [2 favorites]
the ultimate gatekeepers are the editors of journals, isn't that the prime location of the problem?
The actual gatekeepers are the universities. Their decision to outsource large components of the tenure and promotion criteria to journal editors and academic presses is a choice, and if there are gender equity outcomes of that choice they need to confront that.
posted by Dip Flash at 6:30 AM on March 21, 2014 [4 favorites]
The actual gatekeepers are the universities. Their decision to outsource large components of the tenure and promotion criteria to journal editors and academic presses is a choice, and if there are gender equity outcomes of that choice they need to confront that.
posted by Dip Flash at 6:30 AM on March 21, 2014 [4 favorites]
The REF and REA are simply dumb. It is a system that is setup to be gamed. Rather than a judge a university by its output they allow the university to pick and choose select researchers and research like an allotment gardener picking out which vegetables to show. Imagine if economists ranked countries only by 5 companies and about 20 products selected by the countries themselves or if investors judged a company by 5 employees and 20 memos and you get the idea of how valid this managerialist facepalm is.
The winners in these assessments tend to be people who know how to play the game. The top ranked colleges in the assessments surprisingly have people on their REF/REA teams who are current or former members of the REF/REA judging panel who know which levers to pull. And they pull them. My favourite is Just-In-Time faculty recruitment specifically to bolster results. The other is skewing your department balance to favour trendy high impact research over everything else (neuro all the things!) regardless of the impact on teaching ( like have no actual researchers at all in lower impact but major subfields).
The end goal of course is to 'reward excellence' and punish 'lack of excellence' by distributing resources and research funding to the winners. In effect creating an un-level playing field favouring established researchers and top schools and making it even more difficult for schools to compete - think of it as being the opposite of American sports' drafts.
REF/REA obsession/pressure is a big part of what drove my wife out of UK to a much better job in the U.S. even though she was always REF/REA returnable during her 8 years in the UK and very successful at getting grants
I'm not sure if they are going to measure UK academia right off a cliff or if it will survive the current obsession (the brits are pretty darn plucky about managing to overcome crap) but it wouldn't surprise me if they see an exodus of the very talent they supposedly want to reward.
posted by srboisvert at 6:57 AM on March 21, 2014
The winners in these assessments tend to be people who know how to play the game. The top ranked colleges in the assessments surprisingly have people on their REF/REA teams who are current or former members of the REF/REA judging panel who know which levers to pull. And they pull them. My favourite is Just-In-Time faculty recruitment specifically to bolster results. The other is skewing your department balance to favour trendy high impact research over everything else (neuro all the things!) regardless of the impact on teaching ( like have no actual researchers at all in lower impact but major subfields).
The end goal of course is to 'reward excellence' and punish 'lack of excellence' by distributing resources and research funding to the winners. In effect creating an un-level playing field favouring established researchers and top schools and making it even more difficult for schools to compete - think of it as being the opposite of American sports' drafts.
REF/REA obsession/pressure is a big part of what drove my wife out of UK to a much better job in the U.S. even though she was always REF/REA returnable during her 8 years in the UK and very successful at getting grants
I'm not sure if they are going to measure UK academia right off a cliff or if it will survive the current obsession (the brits are pretty darn plucky about managing to overcome crap) but it wouldn't surprise me if they see an exodus of the very talent they supposedly want to reward.
posted by srboisvert at 6:57 AM on March 21, 2014
you'll struggle to produce the four 3* or 4* publications you need to get over the application hurdle for your first permanent academic position
A small point of clarification, but Early Career Researchers are not required to submit four papers. You get to submit one paper less for every year after 1 August 2009 that you became an ECR, normally evidenced by when you won your first proper grant. ECRs are great! A ECR with one 4* paper is worth the same as a Professor with four 4* papers.
In the sciences, multi-authored papers can be a problem: when researchers work in the same institution, they cannot all claim the output. A second or third author of a paper will not usually be able to submit it.
This is also not true; for five or less authors, HEFCE doesn't care where the submitter sits in the list. For six or more and where the author is not lead or corresponding, they have to be able to say they made a significant contribution under section A and Section B below, and to be able to produce a statement from the lead or corresponding author saying this is the case if audited:
SECTION A
The author made a substantial contribution
• to the conception and design of the study;
• to the organisation of the conduct of the study;
• to carrying out the study (including acquisition of study data);
• to analysis and interpretation of study data.
SECTION B
The author helped
• draft the output;
• critique the output for important intellectual content.
Who gets to submit a paper can be a problem at an Institutional level, but it's not something created by the system per se. There's no way to track back the relative merits of an individual's submission, HEFCE only publishes the profile of what a whole institute's submission was awarded, i.e. what percentage of the papers were rated as 4*, 3*, 2*, 1* or unclassified. There no way to look at Prof Blog's submission to see how good or otherwise they are.
The major problem with the REF is the amount of work it entails; in my institution, a small school that's part of a much larger university, 2.5 people worked on it full time for over a year. A working party of about 10 senior academics were given weekly tasks in relation to it, and an external writer was hired to polish the narrative half of the submission. Totes ridic.
posted by fatfrank at 7:24 AM on March 21, 2014 [2 favorites]
A small point of clarification, but Early Career Researchers are not required to submit four papers. You get to submit one paper less for every year after 1 August 2009 that you became an ECR, normally evidenced by when you won your first proper grant. ECRs are great! A ECR with one 4* paper is worth the same as a Professor with four 4* papers.
In the sciences, multi-authored papers can be a problem: when researchers work in the same institution, they cannot all claim the output. A second or third author of a paper will not usually be able to submit it.
This is also not true; for five or less authors, HEFCE doesn't care where the submitter sits in the list. For six or more and where the author is not lead or corresponding, they have to be able to say they made a significant contribution under section A and Section B below, and to be able to produce a statement from the lead or corresponding author saying this is the case if audited:
SECTION A
The author made a substantial contribution
• to the conception and design of the study;
• to the organisation of the conduct of the study;
• to carrying out the study (including acquisition of study data);
• to analysis and interpretation of study data.
SECTION B
The author helped
• draft the output;
• critique the output for important intellectual content.
Who gets to submit a paper can be a problem at an Institutional level, but it's not something created by the system per se. There's no way to track back the relative merits of an individual's submission, HEFCE only publishes the profile of what a whole institute's submission was awarded, i.e. what percentage of the papers were rated as 4*, 3*, 2*, 1* or unclassified. There no way to look at Prof Blog's submission to see how good or otherwise they are.
The major problem with the REF is the amount of work it entails; in my institution, a small school that's part of a much larger university, 2.5 people worked on it full time for over a year. A working party of about 10 senior academics were given weekly tasks in relation to it, and an external writer was hired to polish the narrative half of the submission. Totes ridic.
posted by fatfrank at 7:24 AM on March 21, 2014 [2 favorites]
Sorry for a double post, but another point of clarification.
The REF and REA are simply dumb. It is a system that is setup to be gamed. Rather than a judge a university by its output they allow the university to pick and choose select researchers and research like an allotment gardener picking out which vegetables to show. Imagine if economists ranked countries only by 5 companies and about 20 products selected by the countries themselves or if investors judged a company by 5 employees and 20 memos and you get the idea of how valid this managerialist facepalm is.
This is where the money calculation comes into play; a university's HEFCE award is based on the number of submitted academics (FTE) multiplied by the quality profile of the papers submitted, with a 4* papers being worth 3 times a 3* paper and 2* or below not being worth anything.
(For reference, a 3* paper is one that is internationally recognised, whatever that may mean. From the last RAE, a reasonable approximation is that a paper with citations in the top 10% for it's field is probably a 3* paper)
So, if you only decide to submit your 10 best academics with four 4* papers each, you only get money for 10 people. An institution submitting 30 people with four 3* papers each will get exactly the same amount.
And, in a new twist this year, HEFCE will be publishing the percentage of eligible academics submitted by an institution. So, if it turns out that the Institute with the stellar 10 academics only submitted 5% of their total eligible population, not only are the missing out on money, but they're all going to take a big hit reputationally as well which, in academia, is nearly as important to your future plans.
Apologies for the derail away from the gender issue. Wanna talk about Athena Swan now? And how most major funders in STEM, i.e. the people who allow researchers to publish papers, won't fund any institutes without at least a silver award in the next year or two?
posted by fatfrank at 7:47 AM on March 21, 2014
The REF and REA are simply dumb. It is a system that is setup to be gamed. Rather than a judge a university by its output they allow the university to pick and choose select researchers and research like an allotment gardener picking out which vegetables to show. Imagine if economists ranked countries only by 5 companies and about 20 products selected by the countries themselves or if investors judged a company by 5 employees and 20 memos and you get the idea of how valid this managerialist facepalm is.
This is where the money calculation comes into play; a university's HEFCE award is based on the number of submitted academics (FTE) multiplied by the quality profile of the papers submitted, with a 4* papers being worth 3 times a 3* paper and 2* or below not being worth anything.
(For reference, a 3* paper is one that is internationally recognised, whatever that may mean. From the last RAE, a reasonable approximation is that a paper with citations in the top 10% for it's field is probably a 3* paper)
So, if you only decide to submit your 10 best academics with four 4* papers each, you only get money for 10 people. An institution submitting 30 people with four 3* papers each will get exactly the same amount.
And, in a new twist this year, HEFCE will be publishing the percentage of eligible academics submitted by an institution. So, if it turns out that the Institute with the stellar 10 academics only submitted 5% of their total eligible population, not only are the missing out on money, but they're all going to take a big hit reputationally as well which, in academia, is nearly as important to your future plans.
Apologies for the derail away from the gender issue. Wanna talk about Athena Swan now? And how most major funders in STEM, i.e. the people who allow researchers to publish papers, won't fund any institutes without at least a silver award in the next year or two?
posted by fatfrank at 7:47 AM on March 21, 2014
fatfrank: Sorry for the confusion over ECRs. I was referring to the linked article's point that job ads in the lead up to the REF would typically refer to applicants needing a full complement of 4 publications to be considered. (A total effacement of the ECR, in other words.)
I agree with your wider point that ECRs are great for universities' REF submissions. My own experience, and that of other ECRs, is that REF has been considerably less great for us, particularly those of us on fixed-term contracts ...
posted by Sonny Jim at 8:44 AM on March 21, 2014
I agree with your wider point that ECRs are great for universities' REF submissions. My own experience, and that of other ECRs, is that REF has been considerably less great for us, particularly those of us on fixed-term contracts ...
posted by Sonny Jim at 8:44 AM on March 21, 2014
the ultimate gatekeepers are the editors of journals, isn't that the prime location of the problem?
No, they don't choose the order in which journal author are listed. How could they?
A small point of clarification, but Early Career Researchers are not required to submit four papers
Great in theory but in the three years leading up to the REF is an ERC going to get a first lectureship with less than 4 papers already in the bag?
posted by biffa at 12:22 PM on March 21, 2014
No, they don't choose the order in which journal author are listed. How could they?
A small point of clarification, but Early Career Researchers are not required to submit four papers
Great in theory but in the three years leading up to the REF is an ERC going to get a first lectureship with less than 4 papers already in the bag?
posted by biffa at 12:22 PM on March 21, 2014
Imho, women are probably marginally better at actual research, which requires teamwork, foregoing ego bullshit, memory, etc. Conversely, the RAE and REF are just a big parading of academics up and down in front of parliament, except the waste countless hours and create broken incentives. I bet women aren't so great at such parades.
posted by jeffburdges at 2:14 PM on March 21, 2014
posted by jeffburdges at 2:14 PM on March 21, 2014
And Barbara Graziosi mentioned not parading oneself rather explicitly here with her comments about professional service, books, multi-author papers, etc. not making good REF outputs.
posted by jeffburdges at 2:22 PM on March 21, 2014
posted by jeffburdges at 2:22 PM on March 21, 2014
What do the unappealing sandwiches have to do with anything?
And just how unappealing were they — Baloney on Wonder bread? Head cheese and marshmallow fluf on a stale supermarket bagel? Cold roadkill panini with a turnip-caramel remoulade?
posted by El Mariachi at 7:39 PM on March 21, 2014
And just how unappealing were they — Baloney on Wonder bread? Head cheese and marshmallow fluf on a stale supermarket bagel? Cold roadkill panini with a turnip-caramel remoulade?
posted by El Mariachi at 7:39 PM on March 21, 2014
There is so much meaning packed into first-author and last-author status on papers. It looks like a system needing expiration.
And it's not even consistent. In my field last author means contributed the last to the paper, obviously it's different in other fields, one (minor really) problem with the REF is is quite possible for an academic from one side of this paradigm to assess work from the other side.
posted by biffa at 4:54 AM on March 22, 2014
And it's not even consistent. In my field last author means contributed the last to the paper, obviously it's different in other fields, one (minor really) problem with the REF is is quite possible for an academic from one side of this paradigm to assess work from the other side.
posted by biffa at 4:54 AM on March 22, 2014
STEM Hiring Managers Often Favor Male Candidates, Says Yet Another Study
posted by homunculus at 3:56 PM on March 30, 2014 [1 favorite]
posted by homunculus at 3:56 PM on March 30, 2014 [1 favorite]
« Older From the Merchants of Cool to Generation Like | Are we real fans yet? Newer »
This thread has been archived and is closed to new comments
Or is this a false dichotomy that “could equally be a result of deeply rooted inequalities in the research careers of men and women”? (—as declared and described throughout the essay.)
posted by esprit de l'escalier at 12:50 AM on March 21, 2014 [3 favorites]