We should set up a Clinical Trial Committee at NIH that would have primary responsibility for strategizing and governing NIH’s current funding of clinical trials (no matter at what institute or center).
I agree on the usefulness of (some) pilot trials, feasibility studies, Phase I trials, etc. But I didn't invent the term "small crappy trials"-- the term originated (as far as I know) with Janet Woodcock of the FDA, who used it again just last month:
"Only about 5 percent of Covid-19 clinical trials aimed at identifying or developing treatments were rigorous enough to produce quality evidence, FDA Principal Deputy Commissioner Janet Woodcock said during a Milken Summit panel discussion on improving the clinical trial enterprise. We hardly got any evidence … because it was every person for themselves setting up these little, as I call them, small crappy trials,” Woodcock said."
Rob Califf expressed similar thoughts on a web conference that I viewed just the other day.
"Califf later looped back to Baden’s call to preserve the role of smaller studies in a future emergency governance plan for trials. Agreeing that “there are small, good trials,” Califf added, “but if you actually look at the analysis of Clinicaltrials.gov, whether it’s during the pandemic or during normal times, a huge number of small trials are small crappy trials. They couldn’t have possibly answered the question they were designed to complete. They didn’t finish, and they’re not published. It’s an enormous waste of money” that otherwise could have gone to address issues of community engagement and revamping infrastructure, he said."
I don't necessarily disagree with your main point of streamlining the process for deciding which clinical trials are prioritized (though I'm a bit skeptical that adding ANOTHER NIH committee would be helpful), but I think your focus on "small crappy trials" is misguided. Often they are intentionally small because they're supposed to be exploratory in nature. It's an initial step to see if there's any signal to a potential intervention. As Mike added, often they're also focused on assessing feasibility and acceptability of an intervention and trial design, not outcomes.
Importantly, these small crappy trials are milestones for junior investigators. I'm an early career physician scientist hoping to work my way up to conducting large RCTs that test clinical interventions. I've been successful so far, but I've never run a clinical trial. Do you think it would be appropriate for my first clinical trial experience to be PI of 10 site cluster RCT? These small trials provide the experiences junior investigators need to learn how to manage personnel and resources, navigate the bureaucratic red tape for RCTs specifically, and boost our CV's so that we can compete for larger grants in the future.
If the goal is 10x more clinical trials, you need many more investigators. Those investigators need to gain experience somehow. These small trials provide that opportunity with lower upfront investment. Way more important is to reduce regulatory and bureacratic hurdles.
I take your point, but it seems to me that funding lots of small trials as a type of training grant seems inefficient in multiple ways. Why not try harder to fund large trials with multiple PIs, some of whom are more junior? Then they get training from participating in real RCTs on important issues, rather than figuring it all out for themselves by doing tiny studies (often non-randomized) on unimportant issues?
..."a huge number of small trials are small crappy trials. They couldn’t have possibly answered the question they were designed to complete. They didn’t finish, and they’re not published. It’s an enormous waste of money”
And to address this point specifically, the NIH uses these small trials as a way to "vet" junior investigators. Yes, many of these trials are poorly designed and the PI's are not able to run competently. But at least they are failing at a small scale. This weeds them from the pipeline so that investigators who are competing for larger RCTS are those who could be reasonably expected to succeed (i.e, run the trial well). What would your alternative be?
It feels like there is a confusion here to NIH and other public funders supporting what are essential Phase I trials, Clinical Improvement testing, etc. We need to be Bayesians here: Clinical research is always going to be hard and expensive (thought it should be much cheaper!) and ethically complex. You need to start with small trials to figure out i) if there is any signal, ii) what outcomes it lives in, iii) how to measure it, iv) the experimental parameters for eliciting it. If the NIH is funding a lot of early trials that do just that, then that's a good thing --- private entities can fundraise off of early results, public ones can spin out or make partnerships based on it.
The issue is much more that there's not public funding of meaningful evidence generation at large scale, like the post mentions. There, the suggestions are good. There's probably some elimination of silly Ph1 style trials for outcomes that cannot be measured that way (viz the Covid examples), but even then, you'd still want those to feed into a large, publicly sponsored basket trial or the like. Let's build those trials first and figure out how to scale them.
Meh, I clicked on two of the “examples” of bad science, the yoga convincing one and the R44 about the social robot. The R44 is a development trial—by design, small, because the primary purpose is development not outcomes. The yoga one is a MOST trial where, again, the focus is not on clinical outcomes but is a pilot trial to test procedure feasibility. This are totally acceptable and fit the purpose and scope of development and feasibility trials, I’m not going to waste my time looking at the others, seems like this blog post is doing the usual outrage clickbait about govt spending of money without the appropriate context or consideration of details. Generally speaking the difference between a mature individual and an immature individual is the ability to make nuanced distinctions.
I'm curious about a related issue: what are the best ways for gov to make clinical trials more efficient? (I'm thinking especially of the big trials that pharma companies run for new drugs, and of things I've heard about UK trials being efficient due to how their system works....not sure that's even true, so curious to know more)
If you know of good articles about this, please suggest!
If only one in six drug candidates entering phase 1 are successful, for every successful phase 3 (large, non-crappy) trial, one would expect to need 2 phase 3 trials, 4 phase 2 trials and 6 phase 1 trials. Given that most of the phase 3s are funded by the pharmaceutical developer, these large trials further drop out of the mix. What median trial size would you expect?
Newsflash: Less than 50% of all biomedical research in the US is funded by the NIH. It's being replaced by the pharmaceutical industry. And each institute already had lots of advisory groups, including on trials.
If you want to deal with the issue of small crappy trials, then you need to focus on that alone. An advisory committee (and I've served on them) isn't going to address that issue.
What would you suggest as the solution to: 1) small crappy trials, and/or 2) the arguable need for more cross-IC coordination so that we get more trials like Women's Health Initiative, Lung Cancer Screening Trial, CAST, etc.?
Change the criteria for funding. The current criteria, which are pretty conservative, reflects years of Proximire's Golden Fleece Awards.
I would also change the criteria for advancement in the intramural programs away from publications as they are now configured and back to what they were in the 1950s and 1960s when they were focused on "high risk" studies that extramural scientists were not in a position to conduct. I would also move up the review cycles from what is basically an annual turn-around to 6 mos and require/permit only a small amount of the documentation currently used for extramural grants.
I agree on the usefulness of (some) pilot trials, feasibility studies, Phase I trials, etc. But I didn't invent the term "small crappy trials"-- the term originated (as far as I know) with Janet Woodcock of the FDA, who used it again just last month:
"Only about 5 percent of Covid-19 clinical trials aimed at identifying or developing treatments were rigorous enough to produce quality evidence, FDA Principal Deputy Commissioner Janet Woodcock said during a Milken Summit panel discussion on improving the clinical trial enterprise. We hardly got any evidence … because it was every person for themselves setting up these little, as I call them, small crappy trials,” Woodcock said."
https://www.politico.com/newsletters/future-pulse/2022/12/07/crappy-trials-are-an-innovation-problem-00072714
Rob Califf expressed similar thoughts on a web conference that I viewed just the other day.
"Califf later looped back to Baden’s call to preserve the role of smaller studies in a future emergency governance plan for trials. Agreeing that “there are small, good trials,” Califf added, “but if you actually look at the analysis of Clinicaltrials.gov, whether it’s during the pandemic or during normal times, a huge number of small trials are small crappy trials. They couldn’t have possibly answered the question they were designed to complete. They didn’t finish, and they’re not published. It’s an enormous waste of money” that otherwise could have gone to address issues of community engagement and revamping infrastructure, he said."
https://www.agencyiq.com/blog/white-house-floats-a-national-clinical-trial-governance-committee-for-future-emergencies/
I don't necessarily disagree with your main point of streamlining the process for deciding which clinical trials are prioritized (though I'm a bit skeptical that adding ANOTHER NIH committee would be helpful), but I think your focus on "small crappy trials" is misguided. Often they are intentionally small because they're supposed to be exploratory in nature. It's an initial step to see if there's any signal to a potential intervention. As Mike added, often they're also focused on assessing feasibility and acceptability of an intervention and trial design, not outcomes.
Importantly, these small crappy trials are milestones for junior investigators. I'm an early career physician scientist hoping to work my way up to conducting large RCTs that test clinical interventions. I've been successful so far, but I've never run a clinical trial. Do you think it would be appropriate for my first clinical trial experience to be PI of 10 site cluster RCT? These small trials provide the experiences junior investigators need to learn how to manage personnel and resources, navigate the bureaucratic red tape for RCTs specifically, and boost our CV's so that we can compete for larger grants in the future.
If the goal is 10x more clinical trials, you need many more investigators. Those investigators need to gain experience somehow. These small trials provide that opportunity with lower upfront investment. Way more important is to reduce regulatory and bureacratic hurdles.
I take your point, but it seems to me that funding lots of small trials as a type of training grant seems inefficient in multiple ways. Why not try harder to fund large trials with multiple PIs, some of whom are more junior? Then they get training from participating in real RCTs on important issues, rather than figuring it all out for themselves by doing tiny studies (often non-randomized) on unimportant issues?
..."a huge number of small trials are small crappy trials. They couldn’t have possibly answered the question they were designed to complete. They didn’t finish, and they’re not published. It’s an enormous waste of money”
And to address this point specifically, the NIH uses these small trials as a way to "vet" junior investigators. Yes, many of these trials are poorly designed and the PI's are not able to run competently. But at least they are failing at a small scale. This weeds them from the pipeline so that investigators who are competing for larger RCTS are those who could be reasonably expected to succeed (i.e, run the trial well). What would your alternative be?
It feels like there is a confusion here to NIH and other public funders supporting what are essential Phase I trials, Clinical Improvement testing, etc. We need to be Bayesians here: Clinical research is always going to be hard and expensive (thought it should be much cheaper!) and ethically complex. You need to start with small trials to figure out i) if there is any signal, ii) what outcomes it lives in, iii) how to measure it, iv) the experimental parameters for eliciting it. If the NIH is funding a lot of early trials that do just that, then that's a good thing --- private entities can fundraise off of early results, public ones can spin out or make partnerships based on it.
The issue is much more that there's not public funding of meaningful evidence generation at large scale, like the post mentions. There, the suggestions are good. There's probably some elimination of silly Ph1 style trials for outcomes that cannot be measured that way (viz the Covid examples), but even then, you'd still want those to feed into a large, publicly sponsored basket trial or the like. Let's build those trials first and figure out how to scale them.
Meh, I clicked on two of the “examples” of bad science, the yoga convincing one and the R44 about the social robot. The R44 is a development trial—by design, small, because the primary purpose is development not outcomes. The yoga one is a MOST trial where, again, the focus is not on clinical outcomes but is a pilot trial to test procedure feasibility. This are totally acceptable and fit the purpose and scope of development and feasibility trials, I’m not going to waste my time looking at the others, seems like this blog post is doing the usual outrage clickbait about govt spending of money without the appropriate context or consideration of details. Generally speaking the difference between a mature individual and an immature individual is the ability to make nuanced distinctions.
https://goodscience.substack.com/p/the-nih-should-launch-a-clinical/comment/12044771
I'm curious about a related issue: what are the best ways for gov to make clinical trials more efficient? (I'm thinking especially of the big trials that pharma companies run for new drugs, and of things I've heard about UK trials being efficient due to how their system works....not sure that's even true, so curious to know more)
If you know of good articles about this, please suggest!
If only one in six drug candidates entering phase 1 are successful, for every successful phase 3 (large, non-crappy) trial, one would expect to need 2 phase 3 trials, 4 phase 2 trials and 6 phase 1 trials. Given that most of the phase 3s are funded by the pharmaceutical developer, these large trials further drop out of the mix. What median trial size would you expect?
Newsflash: Less than 50% of all biomedical research in the US is funded by the NIH. It's being replaced by the pharmaceutical industry. And each institute already had lots of advisory groups, including on trials.
If you want to deal with the issue of small crappy trials, then you need to focus on that alone. An advisory committee (and I've served on them) isn't going to address that issue.
What would you suggest as the solution to: 1) small crappy trials, and/or 2) the arguable need for more cross-IC coordination so that we get more trials like Women's Health Initiative, Lung Cancer Screening Trial, CAST, etc.?
Change the criteria for funding. The current criteria, which are pretty conservative, reflects years of Proximire's Golden Fleece Awards.
I would also change the criteria for advancement in the intramural programs away from publications as they are now configured and back to what they were in the 1950s and 1960s when they were focused on "high risk" studies that extramural scientists were not in a position to conduct. I would also move up the review cycles from what is basically an annual turn-around to 6 mos and require/permit only a small amount of the documentation currently used for extramural grants.
Would you be up for a chat sometime? Would love to hear more about your perspective. Email me at stuartbuck@gmail.com