Do We Fund New Research or Not?

Looking back at the research already completed . . .

A not-well-kept secret is that a great way to get funding from NIH is to stagger your applications such that you:

1) do a round of experiments and pick the successful ones,

2) submit a proposal in which you claim to have “preliminary data” predicting success from those experiments,

3) get a grant,

4) report back to the NIH that your experiments were a success (no surprise!), but meanwhile,

5) use the grant money to do a new round of experiments in which you pick the successful ones to be the basis for your next NIH request.

This is in large part because of the demand for preliminary data. A routine observation about NIH peer review is that everyone expects to see that your research has enough “preliminary data” to show that your future research will be likely to produce positive results.

For example, one NIH webpage advises that “when you’re crafting your grant application, high-quality preliminary data can make all the difference.” The webpage continues: “You must also assess whether or not your preliminary data are sufficient to convince reviewers that your project has a high likelihood of success.”

[All research needs a high likelihood of success? Compare that to this view from Nobel winner Robert Lefkowitz: “Science is 99 percent failure, and that's an optimistic view.”]

Indeed, even for grants that do not ask for preliminary data, you should provide preliminary data anyway:

“Preliminary data are not required for the exploratory/developmental grant (R21) or small grant (R03) mechanisms. However, we’ve seen that most R21 applicants include preliminary data, and those who do typically enjoy greater success rates.”

Here is a compilation of statements I’ve heard or read as to the staggering of applications, such that the NIH is funding work that is already well in progress and much of which may be completed:

• The vice president for research at a major university told me, “The NIH doesn’t fund research, it REfunds research.”

• An NIH leader told me, “We all know that when you submit a grant, you’ve already done 2/3rds of the work.”

• A top scientist told me: “It’s an open secret that people write grants for stuff that they’ve already done – they know it’s feasible and has good preliminary data, and by the time it gets funded, they can take the money and do something else.”

• As a Swiss researcher told me when I asked about the effects of soft money on innovation: “In the end what people do in the US is write a grant where they’ve done 90% of the work, so they’re sure that they will deliver on the proposal. I remember the first time I heard about this in the US – 30 years ago – I was totally shocked. So you’re always one cycle ahead in your grants. The problem, of course, is that research is not the same as building a house. You can’t expect to predict exactly where research will turn out over the next few years.”

• Here's a prof at Penn: 

  Nobody ever proposes the really exciting thing they want to do, instead they just propose what reviewers will think is safe and reasonable. Not that these are some brilliant insights on my part; I think most applicants and reviewers are acutely aware of this, hence the maxim “An NIH grant should have 3 aims: 2 you’ve already done and 1 you’re never going to do”. So to the extent that everyone already knows all this, why do we bother with the whole charade?

• Check out this passage from a fascinating NY Times article:  

  The problem, scientists and researchers say, is that there can be yearslong waits for university institutional approvals to move forward with promising research. The process, aimed at sifting out unrealistic proposals and protecting safety, can involve writing long essays that can consume more than half of some scientists’ time. When funding does come through, the initial research idea is often already stale, setting off a new cycle of grant applications for projects sure to be outdated in their own time.

  Stuart Schreiber, a longtime Harvard-affiliated researcher who quit to be Arena’s lead scientist, said his more out-there ideas rarely received backing. “It got to the point where I realized the only way to get funding was to apply to study something that had already been done,” Dr. Schreiber said.

• To quote several prominent scientists from a great piece in Philanthropy Roundtable:

“When you apply for federal grants at a place like the National Institutes of Health the game is that you propose to do what you’ve just done. Everybody knows that, though they won’t say it,” states Rick Horwitz, one of the nation’s leading biological researchers, and the director of an important new lab just set up by philanthropist Paul Allen. . . .

Another eminent biologist, Leroy Hood, echoes Horwitz. “At the National Institutes of Health, if you haven’t completed two thirds of your research, you’re probably not going to get a grant, because everything is so competitive and so cautious.”

Top New York University researcher Charles Marmar says the same thing: “Government research is powerfully conservative. I’ve been an NIH researcher for decades, and to get an NIH grant today you essentially have to already have solved the problem in question.”

This has been happening for a long time across many disciplines. In 1978, the NIH issued a major report on how peer review works, and among other things it said: "Many commentators expressed doubt about the validity of predictions concerning future research . . . applicants, on occasion, had set forth as plans for future investigations, work which they had already partly or substantially completed." In 1980, Richard Muller wrote in Science:

It is well known in the research community that one cannot expect a proposal to be funded until a considerable amount of work has been done on the project. When I began research in 1965, our research group often received more than the minimum support necessary for our projects, and the excess money was used to seed new ideas. Only a small fraction of these ideas led to a formal proposal. If the proposal was funded, it could provided seed money for the next idea."

And I just stumbled across these quotes from researchers in the Netherlands--apparently this is a common scenario for funding agencies that have a lot of competition between individual researchers: 

• “Science is per definition not predictable. Competitive funding forces you to predict your science, i.e. first do experiments then write the grant. Afterwards claim success because all your ‘predictions’ turned out to be true. This is often termed ‘pilot’-data.”

• “You have to have 2/3 of the paper already written to get the grant for the project.”

***

Two unrelated observations about all of this:

• NIH is arguably doing something a little bit closer to “fund the person, not the project” than anyone has ever publicly claimed. I.e., grantees are often being chosen based on their past track record (although not explicitly deemed as such), rather than on their actual (and mostly unstated) plans for the next 4-5 years. That said, researchers are still quite constrained in what they can do with any given grant, because they still have to worry about setting themselves up for a renewal application that has to be filed fairly soon in order to spend another two years getting reviewed.

• It's fascinating that the official NIH Grants & Funding webpage says this: "By law, NIH cannot support a project already funded or pay for research that has already been done." Another NIH webpage says: “NIH will withdraw any application that overlaps with research that is underway, completed, or under peer review.”

Wait, really? The thing that folks have pointed out since the 1960s-70s might actually be illegal?

Sounds bad.

Technically, one could argue that if researchers submit applications based on research that has already been done, but then they use the grant support (in reality) to do new research, NIH hasn’t “paid for research that has already been done.” But if we’re going to be explicit about that, rather than sotto voce, are the hundreds of peer review panels at NIH even conducting their work on an honest basis?

Thus, maybe some rethinking of the application process and peer review is in order? The whole idea of staggered/cycled research funding never made much sense anyway.

As Muller wrote in 1980:

Referees frequently expect all potential problems to be identified and their solutions outlined. Unfortunately, it is not an exaggeration to say that the agencies expect a proposal to outline the anticipated discoveries.

We should not expect research proposals to read like engineering proposals. To require that the solutions to all problems be obvious before the research is begun discriminates strongly against innovative work. The process of solving such problems is often the substance of research.

We need to move away from a system that at best encourages dishonesty, and at worst makes it difficult or impossible to do breakthrough research.