Master Protocols: A Tool for Increasing the Reliability of Behavioral Evidence

A guest post

The following is a guest post from Anna Harvey, president of the Social Science Research Council (to which I’m an advisor).

The National Institutes of Health (NIH) Common Fund supports “bold scientific programs that catalyze discovery across all biomedical and behavioral research.” NIH programs supported by the Common Fund “bring together investigators and multiple NIH Institutes and Centers (ICs) to collaborate on innovative research. Common Fund programs are expected to address high priority challenges for the NIH as a whole and make a broader impact in the scientific community.”

A recent Common Fund Request for Information (RFI) solicited “ideas for biomedical and behavioral research challenges and opportunities that may be of interest to the Common Fund.” In response to this RFI, the Social Science Research Council submitted a proposal to strengthen the reliability and policy relevance of behavioral research through the deployment of “master protocol” designs.

The challenge: Increasing the reliability of behavioral research

Biomedical products like vaccines and therapeutic drugs are powerful tools to combat disease and promote public health. Research to develop such products has long been prioritized by the NIH.

Behavioral interventions that promote the uptake of proven biomedical products are a necessary complement to this work. Randomized controlled trials to find behavioral interventions that increase the uptake of proven biomedical products have been supported by many NIH ICs.

However, findings from randomized controlled trials of behavioral interventions may be less reproducible than findings from similarly sized clinical trials of biomedical products. If behavioral interventions produce smaller and/or more variable effects, if ensuring fidelity to behavioral intervention designs is more challenging, and/or if behavioral interventions are more likely to have interactive effects with contextual features, then findings from single-shot trials of behavioral interventions will be less informative than findings from comparably sized trials of biomedical products.

Empirically, we know that many single-shot trials of behavioral interventions fail to replicate. Yet replications may be harder to publish than original studies, potentially leading to publication bias in favor of single-shot trials of behavioral interventions reporting possibly unreliable estimated effects.

Recent efforts by private philanthropic organizations have sought to address this challenge by funding multiple simultaneous randomized controlled trials of behavioral health interventions. During 2021 and 2022, for example, a coalition of private philanthropies contributed a total of $12.5 million to enable the Social Science Research Council to organize the Mercury Project research consortium to evaluate behavioral interventions designed to increase vaccination uptake in the United States, Africa, Asia, and Latin America/the Caribbean. Contributing philanthropic organizations included the Rockefeller Foundation, the Robert Wood Johnson Foundation, the Bill and Melinda Gates Foundation, the Alfred P. Sloan Foundation, and Craig Newmark Philanthropies. The National Science Foundation committed up to $12.5 million in additional matching funds if incremental private funds could be raised. Currently 13 teams are working to evaluate a portfolio of behavioral interventions in a variety of settings, with five additional teams to be announced shortly.

The Mercury Project consortium is organized around a common research framework that groups interventions by type and defines common outcomes. For example, intervention types include those that reduce search, decision, and logistical costs, and those that increase the social/peer benefits of vaccination. Common outcomes include accurate knowledge of disease risks and vaccine safety and verified vaccination uptake.

However, the trials in the Mercury Project consortium are not closely coordinated replications of a single well-defined set of interventions. Instead, the Mercury Project trials are better understood as one-shot trials of different behavioral interventions in different settings.

In order to have reliable, credible, and reproducible scientific knowledge about behavioral interventions, with both internal and external validity, trials of behavioral interventions need to be rigorously replicated in multiple settings. Because of the difficulty of publishing ex post replications of trials of behavioral interventions, multiple-site simultaneous replication should be built into study designs ex ante.

Addressing the challenge: Master protocols for behavioral research

“Master protocol” trial designs provide for closely coordinated clinical trials and meta-analyses designed to efficiently evaluate the efficacy of interventions in multiple populations and settings. The use of master protocol designs has grown rapidly in the field of oncology, but the design is yet to be widely adopted in biomedical research fields outside of oncology.

Master protocol designs provide a transformative opportunity to dramatically increase the reliability of trials of behavioral interventions in health settings. Because behavioral interventions may produce relatively small and/or variable effects, findings from single-shot trials may be unreliable: True but small signals may be swamped by noise; large signals may be misleading. Master protocols ensuring the coordinated replication and meta-analysis of multiple trials of behavioral interventions in diverse contexts and settings could efficiently identify those behavioral interventions that consistently replicate (or fail to replicate).

Because some behavioral intervention designs may be challenging to implement with fidelity in diverse settings, positive findings from single-shot trials may lead to overconfidence in external validity. Master protocols could efficiently identify those interventions that fail to replicate because of infidelity in implementation across multiple sites.

Behavioral interventions may be more likely to have interactive effects with contextual features, but single-shot trials will fail to identify these interactive effects across diverse contexts. Master protocols could efficiently identify whether and how these interactive effects exist.

Finally, because master protocols build in simultaneous replication from a project’s inception, both meta-analyses and findings from the subtrials in master protocol designs could be published simultaneously, avoiding the publication bias affecting ex post replications.

A potential opportunity for the NIH Common Fund to pilot a master protocol design in behavioral research lies in randomized controlled trials of behavioral interventions designed to increase vaccination uptake. For example, findings from the first 18 SSRC Mercury Project trials of behavioral interventions to increase vaccination uptake will be published in 2023 and 2024. Other trials of behavioral interventions designed to increase vaccination uptake could be sourced from the research literature. The most promising interventions could be selected for inclusion in a master protocol design launched in 2024/2025, providing for simultaneous replication trials in multiple populations and settings followed by meta-analysis. The master protocol could be led by a coordinating organization that would ensure fidelity to intervention and study designs across settings and conduct meta-analysis of coordinated findings.

This investment would be transformative, with high potential to dramatically affect behavioral research over the next decade by establishing master protocols as a benchmark standard. The investment would be catalytic, resulting in multiple high-impact deliverables within five years, including validated protocols for designing, managing, and analyzing the results from master protocol designs in behavioral research, and replicated and policy-relevant findings for the behavioral interventions evaluated in the pilot master protocol. Because the behavioral drivers of vaccination uptake likely share commonalities across domains, the investment would be synergistic, with findings in one domain (e.g., HPV) leading to knowledge advances in other domains (e.g., RSV). The investment would be cross-cutting across the missions of multiple ICs and be relevant for multiple diseases or conditions. Finally, the investment would be unique, with no other organization likely to fund such a research agenda.

Timeliness

Climate change-induced disruption of ecosystems, human expansion into animal habitats, globalization, and aging populations are likely to increase the risk and impact of global disease in the coming years. This increased risk underscores the need to have a repertoire of validated science-based behavioral interventions that can cost-effectively support rapid uptake of both newly developed and continuing vaccine products.

Teams in the Mercury Project research consortium will be reporting findings through the second half of 2023 and into 2024. A master protocol to replicate the most promising interventions identified in the Mercury Project’s 18 trials and in the broader research literature could launch in 2024 or 2025. Within five years, the world’s health decision-makers could have access to reliable evidence about multiple cost-effective behavioral interventions to increase vaccination uptake, reducing the risk, duration, and severity of global disease. More generally, the behavioral research community would have access to a validated set of protocols for deploying master protocol designs in behavioral research, significantly enhancing the reliability, reproducibility, and policy relevance of behavioral research.

Anna Harvey

Anna Harvey is president of the Social Science Research Council; professor of politics, affiliated professor of data science and law, and director of the Public Safety Lab at New York University; and codirector of the Criminal Justice Expert Panel. The Public Safety Lab works with teams of social scientists and data scientists to support more effective and equitable criminal justice practices. Its projects include the Jail Data Initiative, a large-scale effort to collect and report daily individual-level jail records in over 1,300 county jails in the United States, and the Prosecutorial Reform Initiative, a collaborative effort with district attorney’s offices to develop more effective and equitable prosecutorial policies. Harvey is the author of two scholarly books and a coauthored casebook on judicial decision-making, in addition to numerous peer-reviewed articles.