The power of manuscript reviews

How we craft reviews for early career researchers shapes our culture and community

There are a lot of moments when junior scientists doubt that they’re prepared for our line of work, and whether our community will accept them for who they are and what they have to offer. I think this is true for everybody except those guys who have unearned and unmoored levels of overconfidence. This prospect of the lack of belongingness is amplified for those of us with identities that have been, and still are, excluded from full membership.

It can be hard to fit in. Especially for folks who aren’t white, cis, straight, didn’t have a middle/upper class upbringing that allowed you to know how to make small talk with other well-off folks, if you never took a plane flight for vacations as a kid, if you didn’t go to a college that these folks consider to be an indicator of good breeding, and all that. While our efforts at DEI have been designed to eliminate homophily in the scientific community, we’re not there, and at this moment it might feel like we’re going backward.

There are hundreds of little forks in the road where we are presented with the opportunity to derail someone else’s progress into a career in science. The choice to say a kind word; the choice to hold back criticism in the wrong place, at the wrong time, or from the wrong person; notes we leave on student writing; who we invite for a meal or who we don’t; how much we talk and how much we listen; if we provide for the whole person or for a brain in a vat; if we adequately fund them in our budgets; if they know that we will respond when they contact us; how we define success for ourselves and for others; who we honor and who we dishonor.

There is one juncture for junior scientists that I see having a very strong effect (either positive or negative, depending on how it goes) is when a student submits the first piece of their own work for publication and the reviews come back. The contents of that review are critical, but not in the way I think some folks imagine. It doesn’t even matter what the paper says about the soundness and the relevance of the research. What matters for the professional trajectory of the junior research is what the review tacitly says about whether the person doing the work belongs in our research community.

These reviews have an impact far beyond other factors for so many reasons. Even if students are being advised by and working with people who are well established in the field, there’s a matter of familiarity and when reviews come back from manuscripts, this is an external form of validation that feels subjectively different than feedback from people who you know and work with. For grad students who have poured their minds and hearts into their first piece of work, this piece of external validation is ginormous. It’s one thing when your committee members tell you that your stuff is good and accept you for who you are, but when that review comes back and it makes you feel like you don’t belong? Wow. This is the worst. On the other hand, when a review might be really critical of the science but it is written with a tone and perspective that the person who led the work is a legit member of the community and that their contributions are valued, even if there are issues with this manuscript? That can be an affirming experience.

I’ve seen people steer away from research because of unkind peer reviews. The problem was not their talent, wisdom, or quality of their work. In fact, their steering away might be a reflection of having those traits in mass quantities! It’s because the review process told them that getting their work — and their own selves — accepted into the field would be an uphill struggle because of who they are, or because of who they are not.

To put it in a single word, some reviewers see the process of review as gatekeeping. To keep some folks out and some folks in.

For example, when manuscripts that are very well written but a reviewer suggest that it be edited by a “native speaker,” that’s tantamount to telling someone that they don’t belong. It’s exclusionary (and often, just factually incorrect). Even when it’s not as unwelcoming as that kind of harm, there’s a lot that can be said explictly or implicitly with a combination of tone, word choice, and things that you choose to say or not say.

Especially when you’re dealing with people from different places or who have different roles, expressing tone in writing can be difficult. This means, if you want to get a message across, the best way to do it is to be explicit when it comes to being kind and welcoming. We should never judge the value of a person on the basis of what we perceive to the the quality of their work. It’s okay to take the time to package a review so that the author will feel like they are a colleague rather than a subject in the kingdom in which you reign.

There are a few people and moments I reflect on where this mattered. There was someone I went to grad school with who was doing high quality work and breaking new ground in an exciting area. They did work of the highest quality, knew all the literature in the field, and as a junior student, learned a lot from them as a role model. But then, they started to have a little trouble publishing their work. The critiques were not so much about the validity or quality of the work, but who, where, and how it was being done. It’s very easy to look at the work of someone who is not an existing member of a research community, working in a system that hasn’t been worked on much before, or working in places or with people who are not well established, and then convert this to concerns about the validity of methodology, whether the work was contextualized in previous work adequately, whether the conclusions stretch too far beyond the hypothesis, whether the findings are more generally applicable or too narrowly construed. All of these things can be valid criticism, but in the pen of an expert, they can be phantoms designed to keep certain people in and certain people out. And here’s the sticky part: sometimes researchers themselves don’t even actively know or consider whether they’re the bad guy when they are. Once some folks have so deeply internalized the idea that their research community contains a certain set of people who do things a certain way, then to them it’s possible that other people doing other ways are just not valid. I’ve seen some wild stuff over the years while handling hundreds of papers as an editor. People have shown themselves (sometimes for their loveliness, sometimes for their ugliness) time and again.

In the case of this person who was in grad school with me, some of their papers eventaully got accepted, but the real harm was that they got the message that their work was unwelcome in the community for which it was designed to support. So they moved on to do amazing work outside this exclusionary community of researchers. Truly, our loss.

I experienced a flavor of this more than a little while ago when I took on some projects that involved ecosystem ecology. When I started to try to publish my initial experiments and findings, I got a huge amount of pushback in the reviews that I think can only be characterized as exclusionary and clubbish. More than one review said not so much in these words but definitely with this intent, “Who are you to publish this work? Who did you train with and who are your collaborators and why did you choose to do it this way when everybody knows doing it this other way is better?” (To be clear, I had good reasons for doing it my way, and those were clearly stated, but also easy for them to overlook.) At this point, I was a moderately established person and my professional success didn’t hinge on getting these papers out in higher-profile journals. But the upshot is that I got the message that I should Get out of Dodge City, this isn’t my town, I don’t belong there. So I stopped doing the ecosystem work, even though I was excited about it, because I didn’t want to have to continually fight to argue for my belonging with reviewers who thought I didn’t belong simply because of who my PhD advisor was not and because I chose to work in a rainforest with an N:P ratio different than what everybody else wanted to see. Again, their loss. To be clear, not everybody was like this, but enough of them were. It was a club, and I didn’t go through the hazing process required of the members, so I felt I wasn’t welcome. That feels like forever ago.

After a couple decades, I’m shifting into a new line of work. The people who will end up reviewing our manuscripts may be like: “Who is this person and why are they doing this and who did they train with and are they a member of our club?” They may or may not decide that I and my students belong. To be clear, I do not want to be a member of this or any exclusionary club. I want to build an open house, a party where people can come and go and we build stuff together, where everybody is welcome and tries to uplift one another as we do good work. In the next year or two, suppose I’ll find out what the culture is like, and whether I’m welcome to the party.

One of the features of gatekeeping reviews is when the reviewer laments that the authors did not the same paper that the reviewer would have written if they had collected those data. Reviewers are asked to evaluate the soundness of the science, and often also how interesting or exciting the results are. I don’t think I’ve ever received a review request asking me, “If you were the author, what would you do?” though I can’t count the number of reviews I’ve seen that look like this was in the instructions. It’s a cool feature of the scientific community that two people can look at the same set of information and decide to take them in different directions. It’s possible for folks to get it wrong, but oftentimes they’re not wrong, just different. I wish more scientists really understood this.