Focus on the Science, Not the Scientist

December 14, 2018 11:54 AM

Astronomers gathered in Baltimore recently to complete the annual Telescope Allocation Review for the Hubble Space Telescope. Last year, despite efforts made to reduce bias, proposals for medium and large programs on the Hubble Space Telescope had an acceptance rate of 24% for programs led by men and 13% for programs led by women, an imbalance largely in keeping with the telescope's history. This year, in one of the most competitive cycles ever, that suddenly changed to a near-equivalent 8.7% acceptance rate for women and an 8.0% acceptance rate for men, reversing the trend seen over the past 15 cycles. What happened? Anonymized proposals.

For the first time ever, the Hubble Telescope Allocation Committee used a dual anonymous system, in which committee reviewers were not made aware of the identities of the scientists proposing programs. The stark difference brings to mind the changes to orchestras wrought by blind auditions. Of course, that didn't solve every gender bias problem faced by orchestras, and it should be noted that the 8.7% acceptance rate for women meant 12 out of 138 proposals submitted, while the 8.0% acceptance rate for men meant 28 out of 351 proposals submitted.

posted by kyrademon (18 comments total)

I'm interested in how well the blinding works. The most common criticism in my field is that removing names isn't effective enough to actually blind people to the writer's identity, given how small the fields sometimes are. Mind you, in grant committee reviews, when people are often less familiar with the work in question than peer reviewers for journals... that might be useful, yes.
posted by sciatrix at 11:59 AM on December 14, 2018 [5 favorites]

Why the huge drop in acceptance rates over a year?
posted by mr_roboto at 12:19 PM on December 14, 2018 [2 favorites]

removing names isn't effective enough to actually blind people to the writer's identity, given how small the fields sometimes are

That seems like an inadequate reason to not even try it. Even when a proposal refers to a piece of equipment or a research site controlled by one lab, the judges wouldn't know if it was for the PI or a postdoc or a grad student, and that adds up over time.

Alternative hypothesis: the claim that double-blinding doesn't work is made mostly because it makes more work for the judges. Maybe the log-rolling is an unconscious attempt to "not be wrong", like "no-one was ever fired for buying IBM funding a guy from Harvard", so taking it away makes judges anxious.
posted by clew at 12:22 PM on December 14, 2018 [4 favorites]

This is incredibly good. I’m just thinking to ARC and NHMRC grants here in Australia, and the incredibly convoluted maze you have to negotiate to quantify every aspect of a researcher’s career on a personal level - a lot of which is intended to adjust for things like gender bias (does this researcher have enough publications for this point in their career? But what if they had a career gap? Was that career gap for a legitimate reason? Did they start their career late? What is their teaching/research balance?). It’s symptomatic of the worst possible integration of scientific precision obsession and managerialism - we can measure and account for everything and get the right answer! Except we don’t, and gender, racial and other biases aren’t improving.

I would 100% agree with throwing away all that bullshit in the grant process and just as far as possible anonymise it and focus on the scientific ideas, not the personnel.
posted by Jimbob at 12:28 PM on December 14, 2018 [2 favorites]

Shocked, I am shocked, etc.

Good on them for trying to fix it, though.
posted by praemunire at 12:32 PM on December 14, 2018 [1 favorite]

> "Why the huge drop in acceptance rates over a year?"

I believe it is primarily because only medium and large programs were accepted this year (the reasons for that are complicated and tied up with the expected but then delayed launch of the James Webb Space Telescope.) So the quick version is:

1) A lot of the shorter programs that could be accepted in greater numbers either didn't get submitted at all or were expanded or combined to become medium or large sized programs; that is to say, only one big program could get chosen instead of ten little ones for every available slot, reducing the acceptance rate.

2) On top of that, probably because of the number of short programs that people tried to supersize for this, more hours of telescope time were proposed for this year than have generally been proposed previously, further reducing the acceptance rate.

My understanding is that the "larger programs only" things was in part an attempt to reduce the number of hours proposed for this year which backfired rather spectacularly. They won't be doing it again.
posted by kyrademon at 12:41 PM on December 14, 2018 [2 favorites]

Why the huge drop in acceptance rates over a year?

It looks like there was just a large increase in proposals: 438 this time versus 167 in the last round.
posted by HiddenInput at 12:42 PM on December 14, 2018

> "It looks like there was just a large increase in proposals: 438 this time versus 167 in the last round."

Those numbers are for medium and large proposals only, so there were a whole bunch of small proposals last year on top of the 167 number (I'd have to look up how many, not sure offhand), while there weren't any this year.

But yes, the fact that medium and large proposals swelled from 167 last year to 438 this year shows exactly how badly the "let's only take medium and large proposals and that'll reduce the number of hours applied for with the small programs out of the picture" thing went wrong.
posted by kyrademon at 12:50 PM on December 14, 2018 [1 favorite]

One of our main journals has double-blind reviews. Reviewers still try to guess the authors (someone erroneously assumed I belonged to a project with access to data that I am not part of and do not have access to, and they chastised me for not using it), and often I have a pretty good idea who is reviewing, but I think it does improve the quality of reviews and diversity of authors. In addition to gender, I think it gives non-native English speakers a fairer revision. These results are awesome and promising.
posted by ChuraChura at 12:55 PM on December 14, 2018 [12 favorites]

I'm too lazy to look it up right now, but IIRC it also changes acceptances to hide the institutional affiliations -- proposals from people at less prestigious schools suddenly do a *lot* better.
posted by clew at 1:08 PM on December 14, 2018 [7 favorites]

Scientists of 1780s: "to reduce bias in observing an experiment, we can "blind" participants and experimentors."

Meanwhile "I wonder why lady-scientistresses keep complaining about not getting resources.... must be hormones!"

230 years later!

2018 "strange, when we anonymize applicant ID, the disparities in grant assignment collapse!"

Anonymization to reduce bias should be a standard protocol for grading, granting, for trials etc.

Some portions of society are so committed to self-interestedly denying that bias exists that they cling tenaciously to the subjective opportunities to exact bias. These same people use the word meritocracy a lot.

Justice should be blind.... sad that its still headline worthy at this late date.
posted by Anchorite_of_Palgrave at 1:20 PM on December 14, 2018 [10 favorites]

I am involved in supporting these meetings, gonna weigh in here...

I'm interested in how well the blinding works.

It works pretty well. But people can still occasionally recognize writing styles, and when you are one of only 10 or so people in the world researching a particular thing, you can pretty easily figure out who is submitting the proposal. But overall it does what it is intended to do.

Why the huge drop in acceptance rates over a year?

This was not a typical cycle/meeting for allocating time on the Hubble, it had been adjusted to accommodate JWST coming online and all the work associated with that. The previous review meeting had been expanded to evaluate observations that fall into the upcoming observation cycle (HST Cycle 26) as well, so this meeting things was more targeted towards certain types of proposals. The overall goal always being to wring as much useful science out of the telescope as is possible.

The number of submitted proposals way outstripped expectations, there was a bit of a scramble to get additional reviewing resources. The usual oversubscription rate for submitted proposals is 5:1 or 6:1 (i.e., enough proposed work to keep 5 or 6 telescopes busy).
posted by CosmicRayCharles at 1:53 PM on December 14, 2018 [10 favorites]

That seems like an inadequate reason to not even try it.

Just for clarity, this opinion is one I generally see expressed from people who might benefit, not by people who might be inconvenienced by the effort.
posted by sciatrix at 4:18 PM on December 14, 2018

I like how this approach solves for many different types of bias all at once. For instance, with acceptance now hinging on a written proposal rather than a face-to-face interview, scientists of any gender who may have had trouble getting a proposal through due to public speaking or conversational skills (probably not vitally important for hard science) now have a fair shot.
posted by mantecol at 5:25 PM on December 14, 2018

Hrm. Building a telescope proposal system right now (just as a programmer). I'll kick this up the stack and see if there is any traction.
posted by cowcowgrasstree at 5:42 PM on December 14, 2018 [2 favorites]

I seem to recall hearing a couple of years ago of the Pentagon bequeathing an in-place, Hubble-comparable telescope to the scientific community.

Was that just cross talk from a better timeline?
posted by jamjam at 6:52 PM on December 14, 2018

jamjam: your memory is correct. It's the WFIRST mission. Here's an article about the donation.
posted by amk at 7:00 PM on December 14, 2018 [1 favorite]

There was also the time the military wanted to get rid of a ~6 foot mirror probably for a KH-9 that was chipped in production so they auctioned it off.
posted by GCU Sweet and Full of Grace at 7:24 PM on December 14, 2018