Good Science Project Update for Year-End 2022
The Good Science Project officially launched in April 2022. What have we done since then?
Perhaps most importantly, we’ve met with dozens of policymakers and stakeholders, and have been asked numerous times for ideas as to NIH reform. And among others, we helped assemble an ARPA-H metascience working group, and participated in two days of meetings to come up with advice for ARPA-H. The future of scientific reform and metascience looks bright.
A few other highlights:
Personnel:
First, we brought on Eric Gilliam as a Fellow. Eric is a Stanford graduate who spent a few years working for Steve Levitt’s social impact incubator, where he worked on projects related to increasing kidney donations, overcoming barriers to innovation in life sciences research, and fixing inefficiencies in the innovation pipeline. As well, he co-taught a data science class with Levitt at the University of Chicago.
Gilliam has a longstanding interest in the overall structure of scientific institutions as seen throughout history. More on his writings below.
Second, we more recently brought on Betsy Ogburn as a Senior Fellow. Betsy has a PhD in biostatistics from Harvard, and is an Associate Professor in the Department of Biostatistics at Johns Hopkins University. She founded the COVID-19 Collaboration Platform, an effort to “find effective treatments for COVID-19 much faster by sharing protocols for randomized clinical trials.” She is eager to find ways to improve the clinical trial ecosystem in the United States, which is massively inefficient and incentivizes too much waste rather than high-quality trials on important medical questions.
Writings:
At our launch, we published a Good Science manifesto about how science funding ought to work. We got a number of scientists and policy folks to sign the manifesto, including Bruce Alberts (former editor of Science and former president of the National Academies), Phil Bourne (former Associate Director for Data Science at NIH), and Richard Nakamura (former director of the Center for Scientific Review at NIH). It’s available here, and we are always open to adding new signatories.
To date, we have published the following metascience writings on the website, and then circulated them on Twitter and to the mailing list (which has over 1,400 subscribers, including many folks at NIH, NSF, Congress, and the White House):
Comments on the Center for Scientific Review’s 5-year strategic plan. These were formally filed with NIH as per the instructions in the Federal Register. We argue that NIH should undertake bold new experiments with different models of peer review (such as lotteries or golden tickets), and that it should make more data available to independent researchers for evaluation.
CSR’s final strategic plan was almost verbatim identical to the initial draft, and did virtually nothing to acknowledge the comments submitted.
An analysis of the HHS Office of the Inspector General’s report that critiqued various NIH institutes for funding grants with a lower overall ranking by peer reviewers. We argue that far from frowning on such an approach, HHS and NIH should go out of their way to fund grants “out of order” (so to speak) when experimenting with different models of peer review.
An article arguing that ARPA-H should be as independent as possible from NIH, so as to avoid falling into the traditional NIH culture and approach.
A four-part series on the burden of bureaucracy (see here, here, here, and here), including a back-of-the-envelope estimate of how much extra science funding it would be worth to cut the burden in half, and a unique proposal for actually solving the problem.
An article arguing that NIH/NSF should establish a chemistry database.
An article arguing that we should think beyond the distinction between "basic" and "applied" research, and instead set up institutions and funding that can include both in a virtuous cycle.
An article on how the NIH and NSF should look to a National Cancer Institute program for a model of how to fund staff scientists.
An article arguing that new scientific funding agencies should be set up for future evaluations.
An article arguing that NIH should do more to control the size of the biomedical workforce.
An article about the need for NIH and HHS to do more to meet the evaluation requirements of the Evidence Act.
An article laying out the case for using probabilistic forecasting in peer review.
An article on why the NIH should reinvigorate an advisory board that has the power to recommend changes to NIH’s organizational structure.
An article arguing that NIH and NSF should spend 0.1% of their budgets on reproducibility.
Eric Gilliam has written several pieces delving deep into the history of scientific innovation and organization.
First, he outlined the thoughts of Karl Compton on how an optimal university research department/basic research affiliated organization should be run. The basic idea — “fund department heads with a distinct vision, not individual people/projects” — is a model that is quite close to many memorable research organizations. Even the early 1900s, physics departments at universities often had one or two great researchers at the helm who heavily influenced the direction of the department, hiring, and internal budget usage. The piece dives into what Compton described in detail and how an organization like young DeepMind is one of the closest modern equivalents.
Second, Gilliam did an exploration of GE’s Research Laboratory in its earliest days and how the applied research and engineering work taking place in the laboratory was the primary driver of the lab’s eventual creation of top-tier basic research work/ideas, including Nobel Prize-winning work. The pipeline of basic research work, to translational thinking, and then applied work does not only flow in one direction. This is something research organizations in the earlier 1900s understood all too well, while many modern organizations have largely forgotten/under-weight. This is a very exciting thing to keep in mind for those building new science organizations. (Gilliam writes about other (causal) accounts of basic researchers being most useful when working in applied contexts here and here).
Third, in “When do idea’s get easier to find?,” Gilliam outlined how possibly the most important driver of the early/mid-1900s explosion in scientific ideas was the scientific ecosystem’s propensity to create “new branches of knowledge.” The explosive knowledge gains in the early years of a new branch are often the majority of the gains that will ever come from that branch, so it is worth attempting to incentivize branch-creation at all costs — rather than continuing to develop branches for decades. The piece was received extremely well, but, at the time, did not have a set of concrete next steps that were easy to put into practice. Now, Eric and Stuart are finishing up a piece that does just that. It covers the early Rockefeller Foundation and how they adapted their org in the 1930s to go from a traditional scientific grant-funder to one extremely focused on branch-creation — to devastatingly successful effect.
And in a few other pieces:
Using evidence from Warren Weaver’s autobiography and the Feynman oral histories, Gilliam wrote about several accounts of golden-era physicists blaming growing conference sizes and their changing structure, not the growing size of the literature, for the increasing difficulty of keeping up with new scientific ideas (here and here). Gilliam is working on a longer piece exploring this hypothesis in-depth.
Gilliam briefly wrote about how long it took for one of America’s most flexible and fast-moving scientific departments, the National Research Council, to grow so large and bureaucratic that it couldn’t (effectively) do anything new here.
Gilliam writes about how stark the differences were in journal publication habits and free-form (and amicable) exploration of ideas in early and mid 1900s science here. The extent to which half-baked, “thinking out loud” work was done out in the open, even at conferences or in publications, was possibly an order of magnitude greater than today.
We have also published written interviews with the following scientists:
Robert Langer of MIT, who runs the largest biomedical engineering lab in the world.
Thomas Sudhof of Stanford, a 2013 Nobel winner for his work on synapses.
Liz Stuart of Johns Hopkins, a well-known biostatistician.
Russ Poldrack of Stanford, a respected neuroscientist who has done considerable work on open science as well.
Mark Rossi of Rutgers, an early-career researcher who specializes in brain circuits and obesity.
Konrad Körding, a professor of neuroscience and bioengineering at the University of Pennsylvania.
David Allison, one of the nation’s most-respected nutrition researchers.
Rachael Neve, one of the pioneers of Alzheimer’s research.
Mike Lewis, Director of the UK’s Invention for Innovation Program.
Olivia Rissland, Associate Professor at the University of Colorado School of Medicine, and former Rhodes Scholar.
Jim Olds, a neuroscientist at George Mason and a former director of the NSF’s biosciences directorate.
Eileen Jaffe, a long-time professor at the Fox Chase Cancer Center.
Roger Peng, a biostatistician at the University of Texas, formerly at Johns Hopkins.
Jason Shepherd, a neuroscientist at the University of Utah with a number of early career awards from NIH, CZI, etc.
The reason for conducting and publishing all of these interviews? Actual, practicing scientists are often a missing voice in the meta-science discussion. Their stories need to be told.
Indeed, Gilliam has begun work on the ground with organizations like the MIT Office of Innovation and Arcadia Science. In the same way that he has written in-depth pieces exploring the administrative history of organizations like the early GE Research Laboratory or 1920’s MIT and its applied research pipeline, he will be working on the ground with modern research organizations that are trying new organizational approaches. It is the natural extension of his previous work and we are looking forward to partnering with more scientific organizations to expand this work in the new year.
Publicity:
Within a few weeks of the public launch, I appeared on the Thoughts in Between podcast with Matt Clifford (who later became the inaugural chairman of the new research agency ARIA). The Good Science Project was the subject of a widely-read profile in a national news publication. Soon thereafter, I appeared on the Unsupervised Learning podcast and the Narratives podcast. James Pethokoukis of the American Enterprise Institute featured me in his “Faster, Please!” newsletter. And the UK RIOT Science Club did a long interview with me (“Reproducible, Interpretable, Open, and Transparent”=RIOT).
Ideas:
At a July 1 meeting at Stanford, I proposed that NIH needs the equivalent of the Center for Medicare and Medicaid Innovation, i.e., an internal unit that is devoted to experimentation and innovation. Afterwards, I met with congressional staffers who were interested in that idea, and introduced them to Rahul Rajkumar, who spent several years as the Deputy Director of CMMI. I then published a piece in Health Affairs for which I brought on Kushal Kadakia as a co-author (he has worked at CMS): “Investing in the Science of Science: What Medicare Can Teach the NIH About Experimentation.”
A lesson here: The federal government is so big and diverse that many ideas might have some precedent — it’s just a matter of figuring out where! A lot can be accomplished just by noticing what is already being done or what is already in the law.
Meetings:
First, as a bit of background, the NIH Reform Act of 2006 created, for the first time, a way of revamping NIH’s organizational structure on a regular basis without getting too bogged down in political lobbying. It did so by creating an advisory board (the so-called “Scientific Management Review Board” or SMRB) composed of the NIH Director, some Institute directors, and some independent outside experts. This advisory board was ordered to write a report to Congress at least once every seven years assessing NIH’s overall portfolio, and making recommendations as to whether existing institutes should be abolished, whether new institutes should be created, and whether to reorganize existing activities.
The Board has never written an NIH-wide report (instead confining its reports to narrower subjects), and it has done nothing at all since 2015—in direct violation of the law. As a way of drawing attention to these issues, and to encourage innovative thinking on the organization of NIH, we wanted to sponsor a number of scholars to write short papers on NIH reform, and then host a convening to present those papers.
As of October 2022, a workshop called “Building a Better NIH” came together — it was sponsored by the Good Science Project, the Institute for Progress, and the Brooking Institution, with event funding from Schmidt Futures. The short papers are still being edited as of late 2022, and will be released in early 2023. We hope that the workshop and papers will stimulate further discussion about NIH reform.
Second, for several years, I’ve participated in the National Academies’ Roundtable on Aligning Incentives for Open Science, and was one of a few co-authors on a publication last year. In Dec. 2022, I attended one of the periodic meetings at the National Academies, and after more than an hour of people asking “how do we get stakeholder engagement” or “how do we best offer advice on open data to scholars at the point of publication” and similar questions, I spoke about the value of meta-science to answer such questions.
That is, wherever there is a question about how best to achieve a particular goal, we could actually apply the scientific method by testing, evaluating, iterating, etc.
This comment seemed to resonate throughout the room (it was followed by a “Hell yes!” from the next speaker), and several other folks mentioned the metascience point throughout the rest of the meeting.
Moral: We should be thinking about how to apply metascience to open science. Lots of promising avenues to explore there.
***
In all, not a bad year for metascience and innovation. Looking forward to 2023!