How can track record matter in double-blind grant reviews?
We should have double blind grant reviews. I made this argument a couple weeks ago, which was met with general agreement. Except for one thing, which I now address.
Some readers said that double-blind reviews can’t work, or are inadvisable, because of the need to evaluate the PI’s track record. I disagree with my whole heart. I think we can make it work. If our community is going to make progress on diversity and equity like we keep trying to do, then we have to make it work.
We can’t just put up our hands and say, “We need to keep it the same because the alternative won’t work” because the status quo is clearly biased in a way that continues to damage our community.
It’s instructive to note that nearly everybody who pointed to “track record” as a qualm fell into the same demographic group as myself: tenured white guy. I think it’s no accident that members of the group that stand to lose the most from having their identity blinded are the first to raise concerns that the process can’t work. The enfranchised want to keep their franchise. I get this is how things work.
I'd love to see anybody other than a tenured man find fault with the idea of double-blind grant reviews. I think I'm 4 for 4 so far.
— Terry McGlynn (@hormiga) June 7, 2016
I don’t want to dismiss these concerns of tenured white guys like myself. Because like it or not, these folks are still the decision makers. If white men weren’t running the show, I’d bet a well-made sandwich and a glass of unsweetened iced tea that we’d probably have a lot more double-blind review than we do at the moment. So these people — my people — are the folks that I need to convince. I’m taking this demographic — which is my demographic — seriously. I just don’t want to scream “bigot bias unfair privilege yadda yadda” because we’ve got to build a common vision for change. I hear you, I respect your opinion — heck in many ways I’m one of you — and I’m taking the time out to write this post so that we might find some common ground.
In our status quo, reviewers evaluate the track record of grant seekers. In the absence of other variables, this should result in a more fair and higher quality decision. However, the mechanism that allows this track record — knowing the identity of the PI — also facilitates conscious and unconscious bias. This has has a demonstrable, known, and problematic negative effect on members of our community: women and ethnic minorities. If you argue that the lack of double-blind review doesn’t result in unfairly biased decisions, then your argument has an impossible uphill climb.
Which is worse?
Keeping a grant review process with a bias against women and minority scientists that perpetuates a long history of harmful exclusion and discrimination
Working to find a grant review process that uses a double-blind method, so that a set of reviews are created without use of the name, institution, or track record of the PI
If you say the second thing is worse than the first thing, then, well, I just can’t even. That’s an attitude that I don’t want in the scientific community. That’s an attitude that doesn’t belong in this 21st century in these United States of America. I want to find common ground, but if you are so unconcerned about bias against women and underrepresented minorities that you aren’t willing to rethink about how we can review grants more fairly, then I can’t imagine where that common ground might even be, maybe as far as Titan, as close as Ceres, but probably as far as the Kuiper belt. Our common ground cannot be one that we currently occupy on this Earth. Equity is a foundational value for our community.
Once we can get our head around the fact that biases in the review process are causing unfair and adverse decisions for historically excluded subsets of our community, then I see two ways to process this challenge:
“Since we really can’t give a grant to someone without seeing their track record when we do reviews, then we’ll just have to live with the insidious effects of these biases that will continue to harm my colleagues.”
“I do think it’s useful to consider track record in grant reviews, but I also see that blinded reviews are important to remove bias. Rather than dismissing the idea of double-blind reviews, I’d like to seek alternative ways to conduct reviews that remove this bias.”
If you fall into the first category, and think that the biases caused by a lack of double-blind peer reviews are not a big problem, could you do me a favor before you share your opinion? Could you ask a few women and a few scientists from underrepresented groups what they think about this? To see if they share your opinion?
If you’re a member of a group that is the beneficiary of unconscious bias against one’s competition in the applicant pool, then you don’t have a disinterested stake in the maintenance of this bias against people competing against you for grants. When it comes to the importance of double-blind reviews, my opinion shouldn’t count as much because I’m in the demographic category that stands to benefit from having my identity known.
I see two ways to conduct double-blind reviews of grants that also let the funding agency take the track record of the PI into account:
First, track record can be factored in by the program officer (which I mentioned in my original post). I do think it’s important that a person who is awarded a grant not have a history of squandering prior support. If a project is worthy of funding, then in my own opinion, then a program director is quite capable of making the call that a PI isn’t qualified to do a project because of their track record. It’s been argued that this is not a binary issue, that ‘a track record worthy of receiving funds for a proposed project’ is a complex thing that panels can assess better than program officers. That’s a valid opinion, and think I might even agree with it, but that also opens a door for bias. I can understand that people with a strong track record, especially those from non-margainalized groups, would hate to lose an advantage over people who have not yet established a track record, and also would think that it’s unfair that people with a poor track record receive funds. Just as track record isn’t binary, neither is bias, as was pointed out yesterday: “Insider status isn’t binary, of course.” You don’t get the benefits of review as an insider if your name isn’t on the proposal.
Second, in the comments to my original post, Emilio Bruna pointed out that there could be a two-stage review process, in which track record is assessed after the panel reviews a proposal. When a project is considered, federal agencies make a point of emphasizing that it’s projects that are funded, not people. Of course, when research takes place, we need to make sure that the people conducting the project are well qualified to do it in an excellent manner. But first, we must recognize the value of the project itself. There is no reason that we can’t do this double-blind assessment of the project, and then leave it to a different panel to ensure that the most worthy projects are being conducted by highly qualified parties with an appropriate track record.
These two ideas about implementing double-blind review and including track record as a variable aren’t just mine. After I wrote those paragraphs, I found this document on the NSF site that discusses ideas for enhancements to the review process. Near the bottom of the second page are a clear “Version 1” and “Version 2” for double-blind review that match mine. That document is from 2011.
How much time, trouble, or expense is it worth to conduct procedures that protect the marginalized members of our community from bias? I’m sick of people saying it’s too hard, or too much work, to make things more fair. Once the people who are getting screwed over by bias start saying we can’t make the system more fair, then maybe we should stop trying. But when a person who benefits from the bias, albeit inadvertently, says that it can’t be done or it’s not worth the effort? Pffft.
Even if it were possible for panels to take track record into account by reviewers without gender or ethnicity bias (which is, of course, not possible), then there are three reasons why I am not so hot on using track record in panels anyway.
First, the discussion of the PI’s track record is an open door to introducing spurious issues into the review process. A lot of us work in small academic communities where we know one another moderately well, either from personal interactions or by reputation. I often get grants to review by colleagues of mine. It’s my job to separate out my personal thoughts about the person from my assessment of the project. I try, and I hope I’m doing well at it of course. But a system that is designed to rely on forthright behavior of all community members is bound to have insidious outcomes. We can’t list people with whom we have a formal conflict (collaborators, mentees, mentors, and so on) because once you’re in the game for a decade or two, you have a history with most of the players. I can’t guarantee I’m being unbiased, as hard as I try. And there must be plenty of scientists who are not even trying. Are you going to be the person to claim that you are capable of writing an unbiased review of a proposal from someone you have a history with (of any kind)? If so, I imagine there will be a long line of sociologists and psychologists ready to point out how you’re wrong (which is presumably why, in sociology and psychology journals, they usually have double-blind reviews).
Second, consider that the track record itself is reflective of the bias against the marginalized members of our community. Even if there were not unconscious bias in the process, then using track record as a measure of merit is still flawed, because of all the crap that most people have to deal with gets in the way of producing an equivalent track record as a white man. Women are less likely to publish first-authored articles because there is a bias against them in the peer-review process. This is just a fact. Track record itself is a manifestation of bias. If we compare people on the basis of their track records, then the scientists from marginalized groups will, on average, come out a little behind because of the systemic resistance against their efforts to do science.
Third, in my experience, panels don’t seem to be that good at sizing up whether or not a PI is qualified to do the work, and this part of the evaluation process appears to be particularly prone to bias based on institutional and personal affiliations. Have you had a reviewer tell you that you weren’t capable of doing something, even though you had clearly demonstrated in the proposal that you have already done that thing plenty of times? I’ve not only experienced this, but have heard this from many people. Bias against small institutions is egregious when single-blind review is implemented. How often have you seen in a grant review that “The applicant was trained in a good lab,” or “The academic pedigree of the PI is an indicator that the project will be successful,” — which clearly implies the converse assessment. The only way to get rid of this bias is to double-blind the process. It’s hard to not wonder about how these biases are actually operating on a day to day basis. (drugmonkey seems to have been wondering about them this weekend as well.)
I suspect that opponents to double-blind review may not be adequately attuned to the pervasive biases against people in other demographic categories.
As a white guy, if I want to do science, then I can just go ahead and do science. But other members of the community have roadblocks put up in front of them all of the time. I haven’t had a senior scientist hit on me at a meeting, or have a supervisor target me with inappropriate advances, or have anybody doubt my scientific ability because of my identity that I was born with. If we take the time to listen to scientists who say that our established structures are barriers to success, we should be prepared to create evaluations that take into account this uneven playing field. Because some members of our community need to be twice as good just to keep pace with other members of the community, then we need to be consistently intentional about identifying and implementing mechanisms to reduce bias.
Reviewers are routinely incapable of abstracting how different conditions result in different track records. I think most men (myself included) are not capable of understanding how the experience of being woman in science affects one’s professional trajectory. I regularly hear stories from women that still raise the (few remaining) hairs on my head, that make me wonder how it’s even possible to operate in such a hostile environment. But I won’t say “I don’t know how you do it” to women, because that just normalizes the unacceptable condition of our community. Instead, I’m saying, “You shouldn’t have to experience this bias, and here is what I’m doing to change it.”
I know a little bit about how reviewers fail to understand how the academic environment shapes one’s track record. I just went through a bunch of my reviews from the past 6 years. Of the reviews that remark on track record (and a bunch do not), I’d say about half say that I have a particularly strong track record. The other half say I have a mediocre-to-marginally-acceptable track record. And there rarely is any middle ground. What’s the difference between one set of reviews and the other? I don’t have any idea. I can take some guesses though. It might be the people who say I have a great record are people who personally know me and/or my work and think highly of it. It might be the people who say that I have a strong record have looked at my institutional background, taken into account my historic teaching load, and that I have an undergraduate-powered laboratory. (Maybe the difference is panelists vs. ad-hoc reviewers?) As for the people who think my record is weak? I guess they’re comparing me to themselves, or to other people who have PhD students and work in universities that don’t take teaching that seriously. Or maybe they know my role in the academic community and still think that I’ve underperformed. I have no idea, really.
So my grant reviews have a bimodal distribution with respect to the assessment of my track record, which may or may not be caused by biases or blindness to the experience of others. I don’t want to generalize from my own experiences, and I don’t want to put biases against researchers in primarily undergraduate institutions in the same category as biases against against women and underrepresented minorities. It’s just my only experience of being in a marginalized demographic, aside from being veg. And both are my choice.
A recent paper on biases in peer review concluded with:
Peer review is a flawed process, full of easily identified defects with little evidence that it works. Nevertheless, it is likely to remain central to science and journals because there is no obvious alternative, and scientists and editors have a continuing belief in peer review. How odd that science should be rooted in belief.
There is some chance that you’ve read this far and are thinking, “Why is it that for you and others, so many of these issues in doing science have to deal with gender, ethnicity, privilege and other socioeconomic sociology?” The answer to that question is really simple: It’s because scientists are people. If we don’t work to fix our individual and structural biases, then things are not going to get better. If the moral arc of the universe bends towards justice, that’s only because people like us need to keep pulling on it.