An introduction to writing a peer review

I recently had an exchange with a colleague, who had just written a review at my request. They hadn’t written many reviews before, and asked me something like, “Was this a good review?” I said it was a great review, and explained what was great about it.  Then they suggested, “You should write a post about how to write a good review.”

So, ta da.

I’m not keen on reinventing something, so let me share (again) what I think is a superb guide to peer review, produced by the British Ecological Society. It’s comprehensive and worth your time.

Everything below is just me editorializing with preferences and priorities as a reviewer and editor. This organized as an arbitrarily ordered set of observations and opinions. Please feel free to disagree, in comments.

-Read the instructions. Different journals ask different things of reviewers. Some journals explicitly ask you to recommend a specific editorial decision. Other journals ask you to not make a recommendation. Some more prestigious journals specifically ask you rank the novelty/importance/sexiness. Though I expect all journals want you to find errors that need to be fixed, and identify ways that the manuscript could be improved.

-Even if a paper is really bad, please find something you can genuinely praise. It’s just humane. Also, it increases the probability that the reviewer will listen to what you have to say. Don’t you hate it when an author just ignores your useful review that was part of a rejection, and they just resubmit to a new journal without making any of the changes you suggested? Could it be because your review was just so negative, they didn’t pay much attention? Humans are funny that way.

-If you’re wondering about how to handle an issue of any kind, don’t hesitate to email the editor directly with a question. It’s their job, and editors will prefer to deal with reviewer questions before the review is submitted.

-Oftentimes, reviews have three main sections. The first is a short 1-3 sentence summary of what the reviewer thinks the paper says, with an overall opinion about quality and shortcomings. The second section is a list of the important matters, including things that are praiseworthy as well as major problems with design, interpretation, analysis, argumentation, writing, and so on. The third section has minor remarks on typos, citation errors, little things that matter but aren’t fatal to the manuscript. This is often just a bulleted list, as opposed to a narrative in the first two sections. As an editor, this format works fine for me.

-If you have identified deal breakers that wholly invalidate the work and make it not ready for publication whatsoever — and can’t be readily fixed without a massive revision — then you can get right to it, explain what you think, and not bother to remark on all the tiny little things. I’m thinking about huge things, like the method as implemented was irreparably flawed and it came up with numbers that are useless.

-If you get a manuscript that is poorly written but you can understand it, then please review the science on its merits, and then in a couple sentences, describe the shortcomings in the writing. You don’t need to go on and on about how bad the writing is. You can communicate the importance of this by choosing your words carefully, not with a detailed rant. Odds are that the editor knows the paper has writing issues, but they’ve decided that it deserves a review. What both the editor and authors need is a brief remark on how the writing can be fixed. Sometimes these issues arise in part because the authors are not native English speakers, but then again, some of the people who are best at writing and speaking in English are not native speakers, either! This is a realm where cultural sensitivity matters and please think carefully to avoid making assumptions.

-If you want to do a massive copyedit to improve the writing and the structure of the paper, then, I guess, feel free! But you’re not obligated or expected to do so. If you wanted to do this as a favor to help out inexperienced authors, then this is generous of you.

-What really slows down the publishing process is when invited reviewers don’t respond. If you get a request, if you’re not sure you can do it, please for the love of whatever is holy, just click that button that says ‘no.’

-When you say no, listing other possible reviewers is huge. If it doesn’t lead you to a dialog box to enter names, you can just reply to the email — doctoral candidates and postdocs are often great suggestions. Editors handle papers in their fields, but often outside their subsubspeciality, so these leads help.

-If a manuscript is so poorly written that you honestly can’t understand it, you don’t need to go to the trouble to attempt a full review. Just submit your review back to the editor, as promptly as possible, saying that you had trouble understanding it and you weren’t able to provide a full review. Ideally, you shouldn’t ever receive a manuscript like this — this is a situation where editors should be doing a desk reject.

-For many society journals, reviewing manuscripts with major shortcomings is a service to members of the academic community. Some authors might not feel they have an opportunity to get quality feedback prior to submission (e.g., those in institutions without a strong network of peers publishing on this topic in international journals). Assembling a set of reviews that is constructive and provides specific feedback can be a valuable service to the academic community. Please take care to be friendly, and to use polite language, but you don’t want to give a manuscript a more positive review than you think it deserves. In short, if you get a manuscript to review that you think deserved a desk reject for something other than incomprehensibility, you’re doing a service by providing a collegial review that takes the authors seriously and does them the favor of being honest.

-There are usually two boxes where you write feedback – one that is confidentially shared with the editor, and another where the main review goes. If you were being particularly ginger with your words in the main review, and you have more unvarnished thoughts that you’d like to be sure the editor hears, please put them here! While some folks complain that reviews are unnecessarily acerbic, on the editorial end I’ve had bigger challenges parsing the criticisms of reviewers who are trying to be gentle with reviewers and I’m trying to understand exactly how important matters are to the reviewer. Especially if I have a reviewer who is expert in an esoteric realm of statistical analysis or the details of a particular laboratory method, I need them to really say what they think to me as editor, even if they want to phrase it really gently for the reviewers. That helps the editor decide what to prescribe in the decision letter, if a revision is requested.

-If you think the paper merits a better title, feel free to suggest one.

-Please don’t recommend additional data collection. The authors submitted the work that they submitted, not the one you wanted them to submit. If some missing information constrains what you can learn, then explain this, and leave it to the editor to decide what the authors need to do.

-If you find yourself in the uncommon position where you think the authors truly do need to cite you, it’s okay to say so. But in this situation, it’s probably a good idea to just sign the review with your own name. Because, well, the authors will probably think it’s you anyway. As a corollary, if you want to throw off the scent, suggest they cite someone else’s paper.

-I think it’s better to direct the review to the editor, not the author. Yes, the review is there for both the editor and the author, but there a few reasons why it should be directed to the editor. First, the editor solicited the review, and that’s who you’re giving it to. Second, I think most authors — especially when getting unpleasant news — would much prefer to receive feedback in the third person than the second person. Third, this keeps the reviewers from directly prescribing specific changes to the authors.

-Related to the previous point, some folks think this this is just a matter of semantics, but to me, it’s not: Please don’t prescribe specific changes in a review, unless you are specifically asked to so. Let me give an example. Let’s say you are going over the methods/results section, and you find a problem with the statistical analysis. Let’s say that the GLM includes a variable as a random factor, but in your expert assessment, you are confident it should not be a random factor. Here is what I what think you should not write: “The statistical analysis is appropriate, although in the GLM, the authors need to change the random factor to a fixed effect.” What should you say instead? “The statistical analysis is appropriate, but in my view the analysis is not valid when the including that variable as a random factor. Using the variable as a fixed effect will fix this problem. [Then, include a sentence or two explaining why one is valid and the other is not]” What’s the difference between the two? In both cases, you’re saying that one way is the right way, and the other way is the wrong way. But in the former, the reviewer is giving instructions to the authors, the latter is a suggestion. Why is this more than semantics? Because when the authors respond to revisions, they may be dealing with conflicting “orders” from reviewers. The editor should be specific about how these contradictions should be handled, but it’s harder for all parties when there is more than one person giving orders instead of making suggestions. If there is a problem that you think is really huge, and you have some reason to think that the authors might not want to make the changes you hope they’ll make, then you can simply say how important it is.

-If you already said yes to reviewing a manuscript, late is better than never. This sounds crazy, but there are people out there who accept reviews, dither forever, never submit, and when you bug them repeatedly, they ignore you, and in the end when pushed will say that they don’t have time. I remember who you are, and you are on the equivalent of what my mother called her “shit list.” If you do get a ‘late’ notice, please write back to the editor with a timeline, and please stick to it.

I’m sure I could go on and on, but this is enough for today.

I’d also like to share two other resources:

Brian McGill talks about the role of the reviewer as gatekeeper and/or as editor. If you’re wondering, “Here’s what I think about the manuscript, but what should I say and how should I say it?” then this is useful.

Stephen Heard and Timothée Poisot wrote about how we can get more early career researchers into writing peer reviews.