My friend Vinay Prasad let me publish this as a guest piece in his newsletter today. Reprinting it here: **** We are living in a time of growing distrust in science and scientific institutions. According to a 2022 Pew survey, “Trust in scientists and medical scientists, once seemingly buoyed by their central role in addressing the coronavirus outbreak, is now below pre-pandemic levels. Overall, 29% of U.S. adults say they have a great deal of confidence in medical scientists to act in the best interests of the public, down from 40% who said this in November 2020.”
Just one anecdote about JPSP and Eliot Smith -- way back when (ca 2009), we were replicating work by Diederik Stapel. Of course, we were finding a whole bunch of nothing.
We know what transpired in 2011.
We then decided to collect all the replications we had done of the specific paper (we still had the data and materials of 12). We went back to JPSP, where the original was published.
Eliot Smith's response was, indeed, that JPSP did not accept replications (at the time). When we pointed out that it was technically not a replication, as the original never existed, his response was that the original was interesting, because the results were significant. So, we instead had to go to Acta Psychologica (https://psycnet.apa.org/record/2015-16144-004).
(now, "fun fact" the other author's former supervisor would often say that "she had shit on her hands" as she was unable to replicate Stapel. Such were the times if you could not get your effects to be significant)
Structurally it seems odd that journals are still organized as periodic papers stitched together. It would be better in a digital system for a study webpage to have an addendum underneath that includes links to replications under the same journal for better organization.
Who really is out there reading every single item published by a journal? And who will stop doing that if they occasionally publish a replication?
Journals That Ban Replications--Are They Serious Scholarly Outlets At All?
Another great post.
Just one anecdote about JPSP and Eliot Smith -- way back when (ca 2009), we were replicating work by Diederik Stapel. Of course, we were finding a whole bunch of nothing.
We know what transpired in 2011.
We then decided to collect all the replications we had done of the specific paper (we still had the data and materials of 12). We went back to JPSP, where the original was published.
Eliot Smith's response was, indeed, that JPSP did not accept replications (at the time). When we pointed out that it was technically not a replication, as the original never existed, his response was that the original was interesting, because the results were significant. So, we instead had to go to Acta Psychologica (https://psycnet.apa.org/record/2015-16144-004).
(now, "fun fact" the other author's former supervisor would often say that "she had shit on her hands" as she was unable to replicate Stapel. Such were the times if you could not get your effects to be significant)
Structurally it seems odd that journals are still organized as periodic papers stitched together. It would be better in a digital system for a study webpage to have an addendum underneath that includes links to replications under the same journal for better organization.
Who really is out there reading every single item published by a journal? And who will stop doing that if they occasionally publish a replication?