Evaluation criteria for training programs are self-defeating

Programs are incentivized to support students who are already on their way to thriving instead of genuinely broadening participation.

Perverse incentives are exhausting. When the reward structure is designed to defeat the purpose of the system, you wonder whether it makes sense playing the game when it’s rigged.

I suppose this is true about most everything in academic science [gesticulates arms broadly at the world around us]. In this case, I’m thinking about undergraduate training programs like REUs, McNair, RISE, Meyerhoff, etc. These programs are supposed to broaden participation. But these programs are often thought to be most successful when they are doing their least to broaden participation.

For example, there’s a particular site of an NIH-funded training program that I’ve had a lot of experience with. The program is designed to support students planning to earn a PhD in a biomedical field and prepare for research career. For this particular site to get renewed, they of course need to show to the NIH that they’re using resources well and having success. One of the key reportables is, of course, the fraction of participants that move on to graduate programs. If you don’t have enough success, then you don’t get renewed. That’s rather straightforward dynamic.

What is the most strategic way to maximize success in outcomes? Well, you could select students who are already doing so well that you know they’re going to go to grad school whether or not they get into the training program. Moreover, it turns out that getting into competitive biomedical PhD programs is a heavy lift that takes a lot of student support and mentorship, and it might be more risky. But if you bring in trainees that are preparing to enroll into unfunded master’s programs in sociology and psychology in the same department as their undergrad university, that’s a much lower bar that also happens to meet the criteria for this program. Perhaps you can just overlook students doing biology, biochemistry, and other fields that involve biomedical benchwork, and keep your numbers up by recruiting students in social sciences who can gain admissions to master’s programs that are often tangentially related to biomedical research. This is an extreme case. And in the case of this program, it seems that their long-term con has mostly caught up with them.

A far more common phenomenon, perhaps the default, is for training programs to pick the most competitive applicants. What’s wrong with that, you say? Aren’t training programs supposed to give opportunities to the ones who have the most talent and potential to fulfill the goals of the program?

Well, I dunno. If the goal of a program is to increase representation by changing the professional trajectory of students , that means you need to recruit students who will experience a change in trajectory and whose identity will result an increase in representation. In other words: if you give a training opportunity to someone who already is well headed on to graduate school, then are you changing anything?

Take, for example, the famous Meyerhoff scholars program at UMBC, which folks have been excited to replicate. This program is very specific about choosing students who already are tremendously successful. If you’re looking for a program that will help a student who isn’t quite on the track for grad school and to help them onto that track, then this ain’t the program. They’re not moving the needle on representation. They’re just boosting the prospects of the exemplars. This isn’t a bad thing, but it’s not designed to principally increase representation. I think there is a lot to be said for supporting success of well-prepared trainees with minoritized identities because they still are dealing with a mountain of structural biases. And there’s a lot to be said for a program that aspires to train these students to become rockstar scientists. (Maybe that can be said at another time.)

A few years ago, I had a really useful correspondence with a colleague who operates an REU program based out of a field station. (We starting talking about this because of a post I had written earlier about how students are selected for REU programs.)

I complained how training programs often choose the students who have the highest levels of preparation, which results inequitable access to opportunities. My pal was explaining that in their program, they have been making a point to provide opportunities to students who aren’t already fully on the path towards graduate school. They have a partnership with a community college, and routinely offer slots to students from regional public universities that haven’t had prior research experience.

He then went on to say that it’s very hard to show to funding agencies that the program is successful, compared to similar programs. Because they’re giving opportunities to students who aren’t currently on a trajectory towards grad school. They are making a difference in the lives of these students but it’s not going to be a 100% success rate in converting them into academic researchers. However, when they do have success, they just succeeded at genuinely broadening representation because they brought someone into the field who otherwise would have gone in a different direction.

They explained that this is a hard thing to send to the National Science Foundation when they ask you to report on outcomes. You track your participants and when you have some fraction of them not head off to grad school, this could be interpreted as a failure. Especially when this REU program is being compared to other REU programs that have a 100% success rate of sending students to grad school. Which makes you wonder: how many of those students would have went off to grad school if they didn’t have that REU? What does success of an REU program look like?

How can we repair this perverse incentive? There are a lot of things we can consider. For starters, we can establish selection criteira that are more closely aligned with the mission of the program. We also need to allow more more nuance in the reporting of outcomes of training programs, so that programs that are more likely to take chances on students who are less prepared don’t experience any negative perceptions for doing this hard work. I suspect that there already is that room for nuance in reporting, but I think there are plenty of PIs who would rather simply choose participants who are out-of-the-box ready for grad school instead of identifying students who require more support and mentorship. It’s easier just not for reporting, but also in running the program. If you pick students who already have what it takes to move on to the next career stage, that’s asking less of the people running the program.

Stepping back to look at the bigger picture, we can ask, what can we do to

support the development of programs that are designed to boost the population of students that can benefit the most from that boost? And what can funding agencies do to incentivize structuring training programs so that the selection phase is conducted to make the biggest difference?

---

You might have noticed that it’s been a while since I’ve posted. That’s because I a took a week and a half of vacation (which was lovely), and then I was sick. But now I’m not sick, and I’m not on vacation. You can expect a more Science for Everyone until the holiday break hits.