A Tentative Venture into Replication at NIH

Today, the NIH released a new “Replication to Enhance Research Impact Initiative,” along with a funding announcement.

The pros:

• The NIH is funding replication studies, as required by appropriations for FY24.

The cons:

First, the funding is only $50k per study. This may work in many cases, but could be too low of a limitation.

Second, the funding is apparently only for applicants who want to replicate their own study. That seems backwards.

That is, the funding announcement says, “The applicant must be prepared to engage with the CRO immediately after the start of an award, and provide required research methods, protocols, and unique experimental materials as soon as possible.”

You can’t do all of that for anyone’s study but your own.

If we think of a spectrum from “most likely to need replication” to “least likely to need replication,” this announcement is selecting for studies at the latter end of the spectrum.

After all, researchers doing fraudulent or poor-quality research are the least likely to ask for replication funds for their own work! They just want to skate under the radar when it comes to replication.

The only people who would bother applying for extra funds to replicate their own work are the ones who are the most worried about research quality in the first place.

This is the paradox of replication: We should mainly replicate researchers who are nervous and want to avoid it, whereas if researchers are eager to replicate their own work, we can probably feel satisfied and move on.

Third, the NIH states that the results of the replication won't be made public without consent of the PI. But this defeats the point of providing public information about what's replicable or not!

The more that a researcher wants to keep secret the results of a good replication, the more that we should want the opposite.

I’ve never heard of a good reason why the original researchers should be given an automatic veto over whether anyone else gets to know the results of a replication.

Maybe the goal is to entice researchers with the promise that “we’ll only release the results if they are positive, but if your work isn’t replicable, that secret can die with you.”

I understand why some researchers might find that bargain appealing, but it’s not clear why those of us paying for the research would like it.

In Short

The NIH’s new initiative is a good start, but could be improved. The replications will be mostly aimed at the wrong studies and won’t be made public in the cases where we most want to know the results.