A Statistical Saga at a "Top" Journal
Even the “best” scientific journals can struggle with basic statistical literacy, to the detriment of science. We’ve known this for a long time (consider David Allison’s work trying to get journals to correct elementary statistical errors).
A recent case that came to my attention:
Richard Addante is a neuroscientist at Florida Tech. He and his colleagues did a study trying to replicate a prior finding on episodic memory (preprint here; published version here). The exact details of the study aren’t important — what’s important is what happened when they submitted their study to Nature Communications, specifically to an ongoing series on replication.
During the initial round of peer review, Reviewer 3 made the following comment (see the first doc here):
In other words, the reviewer was asking for a calculation as to statistical power: the ability of the sample in question to detect the effect that was seen.
The editor’s initial revise-and-resubmit email directly mentioned Reviewer 3’s comment (while confusing a “sensitivity analysis” with a post-hoc power analysis!):
The problem? Post-hoc power analysis isn’t useful or informative. You can plan a statistical power analysis in advance of a study, but once the study has already occurred, it is meaningless to ask, “Did this study have a sufficient sample size to uncover the effect observed here?” This is fairly uncontroversial (see Heinsberg and Weeks 2022, Hoenig and Heisey 2012, and Zhang et al. 2019).
Why? After a statistically significant effect has already been observed, it isn’t meaningful to ask whether the sample size in question would have had a particular chance of detecting that effect. We now know that it *DID* detect that effect.
***
The authors revised their paper to include a post-hoc power analysis, despite believing it to be meaningless.
But on a second round of peer review, a fourth reviewer pointed out (correctly!) that doing a post-hoc power analysis wasn’t useful or informative (see the third file here).
Even though the post-hoc power analysis had been done at the demand of the Nature Communications editor and Reviewer 3, the Nature Communications editor now used this review as a reason to reject the study for publication!
Needless to say, it was frustrating and arguably unethical for a researcher to be told:
do an inappropriate post-hoc power analysis thanks to Reviewer 3; but then,
we’re rejecting your paper in part because Reviewer 4 pointed out that post-hoc power analyses are wrong.
Moral: Don’t count on journal editors to have a correct understanding of basic statistics. And more broadly, the scientific journal system needs a better way to handle statistical issues like this.
There is a lot of blind leading the blind in peer review on statistical issues. It is a challenging issue as many quantitatively savvy scientists sometimes advocate bad statistical ideas but they might generally be good on other quantitative issues in their discipline.
Great discussion on the problems with post-hoc power a few years ago when surgeons insisted that post-hoc power analysis was needed and vehemently defended it.
https://discourse.datamethods.org/t/observed-power-and-other-power-issues/731/13
Peer review made sense in the era of print journals when space limitations required selecting only a fraction of submitted articles. However, with the advent of the internet, its usefulness seems less clear to me. This example is just one among many that highlights its shortcomings.
Feedback and revisions are undoubtedly crucial, but relying on a small sample of five so-called experts to decide publication often results in arbitrary decisions. Moreover, the peer review process significantly slows scientific progress and tends to push researchers toward more conservative, non-controversial ideas and methodologies.
I’d advocate for a centralized platform where all work can be published, allowing the scientific community to assess its value through votes, comments, and ongoing feedback. This approach would foster transparency, encourage diverse perspectives, and accelerate the exchange of ideas.