The NIH Should Launch a Clinical Trial Committee
We should set up a Clinical Trial Committee at NIH that would have primary responsibility for strategizing and governing NIH’s current funding of clinical trials (no matter at what institute or center).
Why? Clinical trials are the final stage in biomedical research. It’s where the rubber hits the road: do we have a clinical treatment that actually improves human health or not?
But the clinical trial ecosystem has many systematic flaws. Trials are far too expensive, cumbersome, and slow. The result is that the main clinical trials in the US are: 1) big trials sponsored by pharmaceutical companies hoping to get FDA approval; 2) big trials sponsored by the NIH (far too rarely); and, 3) small, crappy trials (to quote FDA officials) that use inadequate methods to study questions that often aren’t very important anyway.
Consider what Collison, Cowen, and Hsu wrote last year about their work with the Fast Grants program to fund Covid work:
We funded a number of drug trials in the U.S. However, we were disappointed that very few trials actually happened. This was typically because of delays from university institutional review boards (IRBs) and similar internal administrative bodies that were consistently slow to approve trials even during the pandemic. . . . Institutional barriers aside, clinical trial enrollment, management, and execution remains highly inefficient and expensive. It is extremely difficult to quickly and cheaply run human clinical trials, even if the drug being tested is already known to be safe. As a result, more than a year into the pandemic, there are completed U.S. clinical trials for just 10 drugs designed to treat COVID-19, which is a meager haul given the potential value of effective treatments. Making human clinical trials cheaper and quicker is probably one of the biggest opportunities for improving U.S. healthcare both during and beyond pandemics.
The problem stretches far beyond Covid trials. Because our clinical trial system is so inefficient, we don’t actually know very much about medical practice! Numerous reviews have found that 80+% of standard medical guidelines don’t have high-quality evidence behind them. And there are many unanswered questions about which of several drugs is more effective, which off-label usages are effective, whether there are personalized responses (and if so, for whom?), how to combine and/or stage various cancer treatments, and much more.
It’s no wonder that Rob Califf once said to me that we need 10x the current number of clinical trials.
The NIH could solve part of this problem, but its funding of clinical trials can be scattershot. A common refrain is that the NIH funds too many “small crappy trials.” For example, I went to ClinicalTrials.gov and searched for all clinical trials funded by NIH that were completed in 2020. Out of 1,142 trials, the median sample size was a mere 57. There were slightly over 100 trials (less than 10%) that had a sample size of 500 or more, and only 4.8% had a sample size over 1,000.
Indeed, even the quickest skim of NIH-funded clinical trials will uncover examples like this:
A non-randomized study of 39 patients in an outpatient treatment program for meth abuse, who were given the chance to use an Internet-based depression program called MoodGym. There was no control group.
An 80-person randomized trial where patients with Alzheimer’s disease will be assigned to play games with “Ryan,” a “Social Robot” (whatever that is).
A 10-person non-randomized study that asked sarcoma survivors to participate in a 3-month weight-training program at home.[3] There was no control group.
A 20-person randomized trial to see whether people who have opioid use disorder might be persuaded to try yoga. Note: This wasn’t about whether yoga would help them; it was just about whether they would try yoga at all.
A 10-person non-randomized study to see whether patients find it easy to use “video chat on a computer tablet that is used for Pulmonary Rehabilitation.”
A 20-person non-randomized study providing a cognitive training program to teens with heart disease.
A 7-person non-randomized study asking older adults to do a short training session involving “values affirmation” with regard to taking their medicine more consistently.
An 8-person non-randomized study to see whether older adults will eat foods containing soy fiber. This trial is a pilot study before a larger trial on 40 people to see whether soy fiber helps reduce apathy (that is, the “lack of motivation and emotional detachment”).
A 10-person non-randomized study offering up to 8 psychotherapy sessions to Latino adults who are caregivers for a family member with dementia.
A 17-person randomized crossover trial to see whether people with irritable bowel syndrome are helped by a 6-week yoga class delivered on Zoom.
Hardly anyone would defend these trials as a rigorous way to study treatments that are likely to be of great importance.
What should be done?
In some cases, the NIH does create a coherent research agenda as to clinical trials in a particular area. As to Covid, NIH came up with the ACTIV program, that is, “Accelerating Covid-19 Therapeutic Interventions and Vaccines.” With this program, NIH joined with BARDA, FDA, CDC, DOD, and other agencies, to coordinate a research response to Covid. The clinical trial component alone has many large, well-designed trials to systematically investigate a wide range of drugs.
But the NIH should take that kind of approach to far more diseases and even portfolio-wide across different Institutes so that the overall funding makes sense in terms of the burden of disease.
To accomplish such prioritization, a new Clinical Trial Committee should work with the Cochrane Collaboration, FDA, and professional medical societies to make a list of high-priority unanswered questions about medical practice, including off-label usage and repurposed drugs, generics, alternative formulations of cancer therapies, comparative effectiveness trials, and pilot studies of promising treatments—in other words, important issues that aren’t usually the subject of expensive trials funded by pharmaceutical companies.
NIH should then solicit robust and high-quality clinical trials on those issues. And across the board, statistical experts at the Clinical Trial Committee should at least give a quick glance at proposals to ensure that trials have an adequate sample size and statistical design.
We don’t need NIH to fund small, low-quality clinical trials on questions like whether a handful of people will try out a yoga class, a depression app, or foods with soy fiber. We need for it to take a more coherent approach.
A Clinical Trial Committee would help. It could coordinate cross-NIH priorities, as well as provide oversight on statistical and experimental design so that the NIH is no longer criticized for funding “small crappy trials.”
I agree on the usefulness of (some) pilot trials, feasibility studies, Phase I trials, etc. But I didn't invent the term "small crappy trials"-- the term originated (as far as I know) with Janet Woodcock of the FDA, who used it again just last month:
"Only about 5 percent of Covid-19 clinical trials aimed at identifying or developing treatments were rigorous enough to produce quality evidence, FDA Principal Deputy Commissioner Janet Woodcock said during a Milken Summit panel discussion on improving the clinical trial enterprise. We hardly got any evidence … because it was every person for themselves setting up these little, as I call them, small crappy trials,” Woodcock said."
https://www.politico.com/newsletters/future-pulse/2022/12/07/crappy-trials-are-an-innovation-problem-00072714
Rob Califf expressed similar thoughts on a web conference that I viewed just the other day.
"Califf later looped back to Baden’s call to preserve the role of smaller studies in a future emergency governance plan for trials. Agreeing that “there are small, good trials,” Califf added, “but if you actually look at the analysis of Clinicaltrials.gov, whether it’s during the pandemic or during normal times, a huge number of small trials are small crappy trials. They couldn’t have possibly answered the question they were designed to complete. They didn’t finish, and they’re not published. It’s an enormous waste of money” that otherwise could have gone to address issues of community engagement and revamping infrastructure, he said."
https://www.agencyiq.com/blog/white-house-floats-a-national-clinical-trial-governance-committee-for-future-emergencies/
It feels like there is a confusion here to NIH and other public funders supporting what are essential Phase I trials, Clinical Improvement testing, etc. We need to be Bayesians here: Clinical research is always going to be hard and expensive (thought it should be much cheaper!) and ethically complex. You need to start with small trials to figure out i) if there is any signal, ii) what outcomes it lives in, iii) how to measure it, iv) the experimental parameters for eliciting it. If the NIH is funding a lot of early trials that do just that, then that's a good thing --- private entities can fundraise off of early results, public ones can spin out or make partnerships based on it.
The issue is much more that there's not public funding of meaningful evidence generation at large scale, like the post mentions. There, the suggestions are good. There's probably some elimination of silly Ph1 style trials for outcomes that cannot be measured that way (viz the Covid examples), but even then, you'd still want those to feed into a large, publicly sponsored basket trial or the like. Let's build those trials first and figure out how to scale them.