I agree on the usefulness of (some) pilot trials, feasibility studies, Phase I trials, etc. But I didn't invent the term "small crappy trials"-- the term originated (as far as I know) with Janet Woodcock of the FDA, who used it again just last month:

"Only about 5 percent of Covid-19 clinical trials aimed at identifying or developing treatments were rigorous enough to produce quality evidence, FDA Principal Deputy Commissioner Janet Woodcock said during a Milken Summit panel discussion on improving the clinical trial enterprise. We hardly got any evidence … because it was every person for themselves setting up these little, as I call them, small crappy trials,” Woodcock said."


Rob Califf expressed similar thoughts on a web conference that I viewed just the other day.

"Califf later looped back to Baden’s call to preserve the role of smaller studies in a future emergency governance plan for trials. Agreeing that “there are small, good trials,” Califf added, “but if you actually look at the analysis of Clinicaltrials.gov, whether it’s during the pandemic or during normal times, a huge number of small trials are small crappy trials. They couldn’t have possibly answered the question they were designed to complete. They didn’t finish, and they’re not published. It’s an enormous waste of money” that otherwise could have gone to address issues of community engagement and revamping infrastructure, he said."


Expand full comment

It feels like there is a confusion here to NIH and other public funders supporting what are essential Phase I trials, Clinical Improvement testing, etc. We need to be Bayesians here: Clinical research is always going to be hard and expensive (thought it should be much cheaper!) and ethically complex. You need to start with small trials to figure out i) if there is any signal, ii) what outcomes it lives in, iii) how to measure it, iv) the experimental parameters for eliciting it. If the NIH is funding a lot of early trials that do just that, then that's a good thing --- private entities can fundraise off of early results, public ones can spin out or make partnerships based on it.

The issue is much more that there's not public funding of meaningful evidence generation at large scale, like the post mentions. There, the suggestions are good. There's probably some elimination of silly Ph1 style trials for outcomes that cannot be measured that way (viz the Covid examples), but even then, you'd still want those to feed into a large, publicly sponsored basket trial or the like. Let's build those trials first and figure out how to scale them.

Expand full comment

Meh, I clicked on two of the “examples” of bad science, the yoga convincing one and the R44 about the social robot. The R44 is a development trial—by design, small, because the primary purpose is development not outcomes. The yoga one is a MOST trial where, again, the focus is not on clinical outcomes but is a pilot trial to test procedure feasibility. This are totally acceptable and fit the purpose and scope of development and feasibility trials, I’m not going to waste my time looking at the others, seems like this blog post is doing the usual outrage clickbait about govt spending of money without the appropriate context or consideration of details. Generally speaking the difference between a mature individual and an immature individual is the ability to make nuanced distinctions.

Expand full comment

I'm curious about a related issue: what are the best ways for gov to make clinical trials more efficient? (I'm thinking especially of the big trials that pharma companies run for new drugs, and of things I've heard about UK trials being efficient due to how their system works....not sure that's even true, so curious to know more)

If you know of good articles about this, please suggest!

Expand full comment

If only one in six drug candidates entering phase 1 are successful, for every successful phase 3 (large, non-crappy) trial, one would expect to need 2 phase 3 trials, 4 phase 2 trials and 6 phase 1 trials. Given that most of the phase 3s are funded by the pharmaceutical developer, these large trials further drop out of the mix. What median trial size would you expect?

Expand full comment

Newsflash: Less than 50% of all biomedical research in the US is funded by the NIH. It's being replaced by the pharmaceutical industry. And each institute already had lots of advisory groups, including on trials.

If you want to deal with the issue of small crappy trials, then you need to focus on that alone. An advisory committee (and I've served on them) isn't going to address that issue.

Expand full comment