Changes to NSF peer review

I just attended the virtual office hour of NSF's Biological Sciences Directorate. The recording isn't available, and I thought I'd share a few key things that I learned.

Things are really, really different in Biology at NSF now. There are no longer divisions, and they'll have three big 'areas.' At the moment, we should be submitting to the solicitations that are posted. They took pains to reassure us that even if we aren't submitting Biology that is AI-infused tech-bro-quantum and all that, they're still interested in funding that stuff. (I'm not fully reassured, to be clear, but that won't stop me from submitting the same exciting science I've been planning to.)

They are making major changes to how peer review happens. It seems clear that this is driven by necessity because staffing levels are so much lower. They said that they have 40% fewer program officers than before the new Regime took over.

Here are the changes that stuck out to me.

• A minimum of two reviews will be required needed to make a decision on a proposal. It used to be three.
• It sounded like there will be fewer ad-hoc reviews, though I don't explicitly recall them saying this. [on edit: other attendees verified for me that they said there would be fewer ad-hocs]
• They say there will be more consistency in the number of reviews. It sounds like the days of some proposals getting cross-panel reviews and multiple ad-hocs are in the past. They're working for about three reviews proposals for all proposals across the entire directorate.
• They will be doing triage. Proposals that have uniformly strong or uniformly weak ratings won't get discussed in panel (much, or at all? it's unclear), so that panels and program officers can spend their limited time on proposals that "need" more discussion.

There was a lot more they said, but those are the things that caught my attention.

In my opinion, these changes are weakening NSF's extremely robust review process that they've been using successfully for decades. It's driven not by an interest in improving the quality of decisions made, but by a reduction in investment in staffing at NSF, because they don't have the personnel to handle the process the way they used to do it. Considering that NSF has consistently been one of the most efficient federal agencies, with very little overhead relative to the amount of funds distributed, this is extremely disheartening.

There is a thing that I've seen on panel many, many times: There is a proposal that has mostly weak reviews, but then when the discussion happens, the panelists compare their notes, and opinions, some others might chime in, and they realize that the science is super cool and amazing. Sometimes groundbreaking science that pushes at disciplinary boundaries can get lost in the wash because they might not attract an advocate for them in review. I've loved that since every proposal gets discussed, if only for a few minutes, the review process results in better decisions.

I've also seen things go the other way lots of times. Proposals come in full of Excellents and maybe one Very Good. But then once the discussion happens, some fatal flaws are identified, or the whizz-bang excitement of the approach gives way to the cold reality that the project has not chance of advancing knowledge. Or, there might be reviewers who are unprepared to evaluate broader impacts, and once there is a broader discussion of the BI in panel, this lowers the overall rating. I've seen this a lot. If this new approach to triage leads to funding recommendation based on written reviews without a panel discussion – and if there might only be two reviews! – this sounds quite specious.

I'm so disheartened.

I don't think this is the death of NSF, and I don't think their peer review process is now fatally flawed. But it used to be the bestest ever, and now it's weaker.

Were any of you tuning in as well? What key things caught your attention, or do you think my interpretation is off?

I would like to emphasize that, as always, I am grateful to all of my scientific peers who are doing their darndest to hold down the fort at NSF and work to keep the scientific enterprise rolling along in the United States. When asked to review, when asked to serve on panel, I still am. I trust and support my colleagues, and I realize that they are not responsible for the adverse conditions under which they must work. Let's all continue to band together to preserve what we have and keep sciencing.

Oh by the way, the photo of the NSF building associated with this post? Yeah this week they're moving out of that building, built bespoke to NSF specifications, for some other space because of the Regime.