Portable peer review and the manuscript cascade

I really hope portable peer review picks up speed.

It’s normal for people to shoot high with submissions. Start with a journal that feels like a little stretch, and then work one’s way down the tiers of impressiveness.

I do it myself, sometimes, though this game gets weary and seems rather wasteful of everybody’s time. But this is part of the standard approach among collaborators and coauthors. As an editor, I see the papers that I cannot accept work their way down the tiers. Sometimes, papers that I have already reviewed end up in my hands to handle.

As a reviewer, I commonly end up getting asked to review the same paper for two different journals. (I don’t accept most reviews that come my way, for if I did, I wouldn’t have time for anything else. I manage to do at least a couple reviews per submission of my own, so I don’t feel that I’m a parasite on the system.)

Some of the papers that come my way are so obviously up my alley, it’s no surprise that an editor thought of me as a reviewer.

But other times, a paper comes my way that is within my bounds, but a little from left field. This isn’t a surprise. When editors are looking for reviewers, the process is haphazard and potential reviewers emerge from a variety of avenues.

What is weird, though, is when the slightly-from-left-field papers come to me multiple times. I once got the same paper — essentially unchanged from its the initial submission — from four different journals! But twice is not uncommon, and thrice has happened once or twice (but this all gets blurry). So, how the heck is it that I keep getting these same papers from different editors and different journals, when they’re not closely tied to things I’ve published?

The most parsimonious explanation is that the authors have identified me as a potential reviewer. Editors do use recommended reviewers in the mix, if only because it’s so hard to find people to accept a review that hitting up the recommended ones is a valid part of the mix. (As an editor, I sometimes use a recommended reviewer, as long as I have another one that I select independently.)

This is where it gets awkward. Let’s say I take on a review. Regardless what my review says, this manuscript might get rejected. Then, let’s say I see the same manuscript come to me from a different journal. What to do?

What I typically have done in the past is to reply, letting the editor know that I’ve reviewed a prior version of this paper, and if they want me to furnish them a review, I’m available to do so. I can see how some journals would want a review independent of prior reviews. Pretty much everybody says “yes, send the review along.”, They’re glad to have it in the mix.

But here’s the thing: nowadays, most of the papers that I review for a highly-ranked journal, I end up seeing again somewhere in the lower tiers. Since highly-ranked journals also have high rejection rates (because rejection results in the perception of quality), then most papers submitted to highly ranked journals will be seen lower down.

At least some of these times, I’m betting that I’m one of the preferred reviewers. As a reviewer, I try hard to be constructive and as fair and positive as possible. (I don’t like to make a specific recommendation to the editor, I think that’s their job to come to a decision based on my qualitative opinion.) But I’m not always sunshine and rainbows.

When I get the same paper for review more than once, then there’s a decent chance that the authors are listing me as a preferred reviewer on a resubmission to a new journal without realizing that it might be my review that informed an editor’s decision to reject a paper. If the authors keep listing me as a reviewer, and they keep getting the same review, then odds are that they’ll do the math.

The upshot? Some while ago I decided that I don’t want to review manuscripts I have already reviewed for a different journal. I don’t want to run the risk of pissing off someone who listed me as a preferred reviewer, even when I think my review is one that would be seen as a positive one by the editor. I realize this results in even greater inefficiency in the system, and this does bother me.

One solution is portable peer review: to conduct a review process that is not tied down to a single journal.

A number of journals — including Biotropica — are accepting prior reviews from other journals. Ain’t that grand? I don’t think authors are making avail of this option much yet, but I think it’s a good one.

There are also organizations that conduct portable peer review, like the folks at Axios Review. (For more about Axios, check this out from Jeremy Fox.) I have a paper in review with them now, and I’m looking forward to seeing how the process works. There are a bunch of other startups doing this, like Peerage of Science and Rubriq. (The latter is involved in a payola scandal with Nature Scientific Reports.)

I do have a big misgiving about portable peer reviews. There’s pretty good evidence that reviews are, to a certain extent, a big crapshoot. Whether or not an academic work gets accepted has a huge element of randomness, and a lot of that is because of what happens in the review process. If reviews are portable, then a randomly unlucky draw will stick with a paper, which is a bummer.

But I think the bummer of portable peer reviews is when the role of peer review gets distorted by the scientific community. In principle — and sometimes in practice — peer review isn’t there to decide how important or sexy a manuscript is. It’s there to verify that the work is sound. In that case, as long as peer reviewers perform that role, then portability should be fine! The rub is that editors nowadays are using reviewers to predict whether a paper will be “impactful.” Whether it’ll help boost the impact factor of the journal. Maybe portable peer review might help us get away from that mindset, and the editors that want to play their selectivity games can rely on their own judgment as much as the reviewers? Probably not, but one can dream.

I realize that those who favor non-anonymous reviews will argue that this problem of non-portability of reviews is merely an artifact of anonymity. Okay, that’s a great point. But since anonymous is how I roll most of the time, and that’s true for a lot of us, then this remains an issue.