Prescriptive reviews are a scourge

How do you spot an academic? Listen for gripes about manuscript reviews.

We all get bad reviews. I’m not talking about critical reviews — we all get those too. I mean: We all get reviews of bad quality.

Some reviews are not written with the care, diligence or maturity that we expect. Even if you’re blessed to avoid these most of the time, you surely know what I mean. Bad quality reviews happen. They can happen a lot.

How do you write a good review? There is a lot of help out there! This guide from the British Ecological Society is top notch. Also it’s worth checking out what Dynamic Ecology and Arthropod Ecology have to say about writing good reviews. I don’t think I can add much to the general genre of “what to put in a good review.”

Instead, I want to point out what a good review shouldn’t have in it. People often have put in the hard to work assemble the elements of a good review, and then spoil it by adding in stuff that gets in the way. What do I mean? Well, some reviewers don’t really understand the distinct roles of reviewers and editors. Or, maybe they understand that role, but want to assume an editorial role when they’ve been merely asked to review a manuscript.

First things first: Reviewers do not decide whether papers are accepted or rejected. That’s what editors do.

I often hear people say something like, “That manuscript I reviewed was so horrible and I rejected it.” That might be rhetorical shorthand for, “I recommended rejection.” Yes, this might be merely semantic. But I happen to think that this reflects, at least to some small extent, how some reviewers approach their responsibility with the wrong mindset.

Should reviewers be recommending rejection, revision, or acceptance? It depends on what the journal asks for. Some instructions to reviewers specifically ask for this kind of recommendation, but other journals are clear that they do not want reviewers to make a recommendation. Others ask you to make confidential recommendation to the editor, but not to the authors. I’m surprised how often — both as an author and as an editor — that reviewers don’t follow these instructions.

Editors solicit reviews to find out if the science is sound, whether claims are supported by the results, and whether the finding is novel or exciting. There are other things that editors will want to know, too.

Here is one thing that editors and authors don’t want: reviews that explain how the authors didn’t do the experiments that the reviewers wish they had done.

It’s not the place of a reviewer to prescribe additional experiments.

It’s not the place of the reviewer to prescribe that the authors test a theory that the reviewer wanted to test instead of the theory the author wanted to test.

It’s not the place of the reviewer to say that the author needs to make a different argument in the discussion.

Reviewers that don’t like how a paper is structured shouldn’t be telling the authors to restructure the paper.

Reviewers who think the introduction is too long shouldn’t tell the authors to cut it, and reviewers who think the methods are missing some important details shouldn’t tell the authors they need to put it in.

If a reviewer disagrees with an argument, it’s not the job of the reviews to tell the author to change the argument. Instead, the reviewer should explain how they disagree in the review and leave it to the editor to tell the author whether to change the argument. Why does this matter? Because:

Reviewers and editors who make sure manuscripts don’t contain ideas they disagree with are slowing down science.

— Terry McGlynn (@hormiga) November 2, 2015

A review should be a review, not an instruction manual. Leave the directions for repairs and improvements to the editor. It’s the job of the reviewers to tell editors and authors what they think, not what they prescribe.

Is this just a semantics, claiming that instructions for changing a paper are radically different from a set of opinions and facts? Not at all! I think it really changes the quality of a review, and modifies editor’s latitude to make nuanced recommendations to the authors.

When reviewers fill their reviews with a list of prescribed changes, then this places the editor in an awkward position. Editors craft letters to the authors and tell the authors what to do. These usually are one of these:

• “Revise the manuscript in line with the concerns of the reviewer.”

• “Be sure to focus on changing X and Y, and explain if you choose to not make further modifications.”

• “Address the concerns of the reviewer in the cover letter and make changes where you feel they are necessary.”

That’s in editorial code. Let me translate those three sets of instructions:

• “Do everything the reviewer says.”

• “Do these one or two things the reviewer says, and some of the other changes might matter, and some might not, but I’m not telling you which.”

• “You and I know the review is off the mark, so don’t make the changes but be sure to explain it in the cover letter to keep everybody happy.”

You know how it’s hard to find the time to write reviews? It’s the same for editors, too. We recognize that editing is a major responsibility but also it’s a volunteer role as a service for the profession. We need to treat authors fairly and make quality decisions, but also face a number of professional tradeoffs.

Editors often make decisions that piss people off, but we can’t make a habit of soliciting reviews and then ignoring them. In most journals, it’s standard form to include the reviewer on the outcome of the editorial decision as a courtesy.

It’s the job of the editor to tell the authors what changes need to happen.

That job gets a lot harder when the reviewer starts doing that job instead of the editor.

When we receive reviews that have an unsolicited detailed set of prescriptive instructions from a reviewer, that can put the editor in an awkward position. We can’t just write back to the authors and say, “Ignore everything that Reviewer 2 said.” It’s rare to get a review that is so hideous. On the other hand, it’s common to get a review that raises some very important points that need to be addressed, but the same review also raises other points that don’t matter, or might even be off the mark a bit.

It’s not kosher to edit manuscript reviews before they get back to the author, unless they contain obvious bias. So when we return the manuscript to the authors for revision, then there’s the delicate task of letting the authors know how much we agree with the reviewer, without showing any obvious disrespect for the time and effort that the reviewer put in.

On the other hand, life is a lot easier for the editor when a reviewer submits a genuinely good review. You know, a review that doesn’t contain any prescriptions but instead provides a comprehensive evaluation of strengths, weaknesses, and a set of opinions about what would improve the manuscript. With reviews like these, the editor can craft a set of instructions to the authors that prescribes what revisions are necessary, and what ideas need to be addressed, for a revised submission.

When the reviews don’t have prescriptions for specific changes, then the editor can identify what matters in the reviews and pass them on to the authors, without having to dance around the fact that the authors can ignore some of the things in the review.

Of course, editors are supposed to take charge and be leaders — and not let the feelings of reviewers get in the way of appropriate decisions. And that’s what I do. I’ll say “Be sure to address all of the concerns of Reviewer 1 and address concerns X and Y of reviewer 2.” Which I realize might piss off Reviewer 2 when they see it. But that’s a lot better than making a bad decision.

Sometimes when I get a manuscript back, the editor is acting more like an administrative assistant than an editor. They get the reviews, look at them, and pass them along to the authors and say, “make these changes.” If the reviews are negative, the paper gets rejected. If the reviews are mostly positive, then the editor ask for revisions. This is not what good editorial decisions look like.

As an author, what I usually think is: “What changes does the editor really want me to make? Some of these changes are simple, and some are difficult. Some of these I’m fine with, but I think some of them harm the paper. Some of these sell the paper to a broader audience, buts change my ideas in ways I don’t want them changed.” And what do I get from the editor? “Make the changes, do as the reviewers say.”

That then leads to a guessing game with the authors. Authors go ahead and do the revisions. But not all of them. The ones that they don’t want to do, they explain and rebut in the cover letter. But the problem is, some of these revisions might be deal breakers for the editor and some are whims of the reviewer that the editor doesn’t care about. What the authors want is the editorial judgment of the editor. What authors too often get is a transferral of editorial responsibility from the editor to the reviewer, and then the author needs to guess about whether or not the reviewer is going to get the manuscript in review again. This is not the way to run a journal.

Reviewers who provide prescriptive reviews are enabling the transfer of the editorial’s responsibilities to the reviewer. Moreover, reviewers who write prescriptive reviews make it harder for editors to clearly specify which changes need to be made.

What does a well-constructed review look like?

Let’s say as a reviewer, you don’t like a table, and think the information belongs in the body of the text instead. A good review says, “I had trouble following the manuscript and think the information in the table is better presented in the body of the text.” A bad review says, “The authors should delete Table 1 and move the information into the body of the text.”

Let’s say you don’t like the name of a certain hypothesis because it overlooks a piece of literature that you appreciate (or maybe one that you’ve written). A good review says, “This manuscript describes a new Very Interesting Mechanism Hypothesis, though in many ways resembles the Mechanistically Intriguing Hypothesis found in Smith 2012. Readers may track the literature more readily if these hypotheses were given a consistent name.” A bad review says, “The authors need to rename the Very Interesting Mechanism Hypothesis to the Mechanistically Intriguing Hypothesis and also need to cite Smith 2012 (which of course I had nothing to do with).”

Let’s say you think a major argument in the conclusion isn’t supported by the evidence. Then that’s what you should say! And you should explain it. But you shouldn’t tell the authors to fix it. That’s the editor’s frickin’ job.

Some journals are really good about making sure that editors can most readily wield editorial judgment over the caprices of reviewers. But I’ve dealt with some other journals that seem to be happy with editors that have decisions that run just a couple sentences, instructing the authors to follow reviewer instructions. Those are journals I’m not planning to revisit, because when I make revisions, I’d like to know the opinion of the editor first.

What would make the review-and-revise process work better for all parties?

As reviewers, we can write actual reviews, and avoid writing instructions to authors. We can let editors know our frank opinions but give them the professional latitude to make their own mind which of these changes need to happen.

As editors, we can exercise a heavy hand with reviews so that authors do not fuss about changes that will not improve the manuscript, spare the authors the rhetorical gymnastics required to refute obviously useless comments from reviewers, and not be afraid to write decision letters that specifically tell authors to overlook prescriptive remarks from reviews that do not need further attention.

I have yet to serve as an Editor-In-Chief. I think it’s the role of the EIC to set the tone for the journal, revise instructions to reviewers as appropriate, and provide feedback to subject editors about how they are using reviews in their recommendations. Ultimately, the direction of a journal is steered by the EIC. If the editorial process consistently involves overly prescriptive reviews that are passed on to authors without guidance from editors, then the buck stops at the EIC. (I haven’t had a bad experience yet with my EICs, though I don’t have as much editorial board experience as some of you all.)

As you might imagine, this post was inspired by revisions I just finished. I suppose I should have the wisdom to wait to find out whether this third round of revisions is going back to the reviewer. And whether the editor will have any opinion about anything other than what the reviewer thinks.

By the way, if you have somehow have navigated the internet without seeing this relevant clip, this made the rounds a while ago: