The Failure that is Peer Review
NEWS FLASH: New paper in Nature from December 2023 finally acknowledges corruption in research:
It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor of The New England Journal of Medicine.
— Marcia Angell, MD
When I publish something on climate, doomsday enthusiasts always say I’m not a climate scientist and my work isn’t peer-reviewed, therefore it’s rubbish. They never say that about what Al Gore, Bill Gates, David Wallace-Wells, or Greta Thunberg publish.
We have all heard of the “crisis of reproducibility,” but science is broken far beyond that. It’s mostly about incentives and rent-seeking. Rather than write an essay, I will give several starting points to do your own research. I’ll start with more general sources, then go into more detail on the failure of climate research.
Formal publishing is now almost entirely noise. There are more than 8 million papers published a year now, a number going up and up and up. Nobody reads them. Why should they? They are almost all useless. Nearly all exist because, and only because, academics must publish or perish. Must.
— William Briggs
Humans are Overconfident
Why Most Published Research Findings are False, the famous 2011 paper by John Ioannidis, and an excellent video explanation by Derek Muller:
David Freedman’s book, Wrong, goes into detail on the gamification of peer review, saying “For every PhD thesis, there is an equal and opposite PhD thesis.
There are serious biases in science, and it’s a very difficult problem to address.
The Rise and Fall of Peer Review — good article
End all Formal Science Journals and Papers — by William Briggs
The Half-Life of Facts, by Samuel Arbesman tells amazing stories of how poorly humans attribute facts to knowledge, especially those based on peer-reviewed science. Facts are temporary. Knowledge is mostly an illusion.
Truth: A Brief History of Total Bullshit, by Tom Phillips
The Absurdity of Peer Review, by Mark Humphries
Peer review: a flawed process at the heart of science and journals, Journal of the Royal Society of Medicine; Richard Smith
The Rise and Fall of Peer Review, by Adam Mastroianni
Can Science be Saved? by John Staddon
More than 500 peer-reviewed papers retracted by Wiley after discovering a “ring” of academic fraud.
Two decades of Alzheimer’s research was based on deliberate fraud by 2 scientists that has cost billions of dollars and millions of lives. More on Alzheimer’s fraud.
The Doctor Who Championed Handwashing — Ignatz Semmelweis
Investigators found plagiarism and data falsification in work from prominent cancer lab
Academic fraud
Wiley to retract over 500 papers that were robo-reviewed — RetractionWatch.
A group of grad students wrote a nonsense-paper generator and started submitting the papers to peer-reviewed journals. More than 200 have been accepted, reviewed, and published. Read the report in Nature Magazine.
The Climate Industrial Complex
This is an actual graph from a peer-reviewed paper on climate impact published by the Lancet, Fig 3. Note the units on the X axis:
The best science money can buy — the Lancet manipulating the X axis units to make hotter look roughly equal to colder weather. The only thing worse than this is when they keep the axes the same and just fudge the data.
This “correction” is from Science Magazine, one of the most respected peer-reviewed journals in the world. The authors used stock photography to pretend that polar bears are threatened.
To understand what’s going on with polar bears, see the polar-bear section in my list of peer-reviewed papers.
The most important thing to read: PEER REVIEW Why skepticism is essential, an ebook by Donna Laframboise; includes an introduction by Christopher Essex and Matt Ridley.
Patrick Michaels on why peer review in climate science fails its own test.
An excellent Grokinfullness blog post shows publication bias in climate-related journals.
Peer Review is F***ed Up, by Michael Eisen
‘Fact Checks’ by Non-Experts Are Shutting Down Genuine Scientific Inquiry, by Willie Soon
The Unstoppable Momentum of Outdated Science, by Roger Pielke, Jr. Here is the introductory paragraph:
A 2015 literature review found almost 900 peer-reviewed studies published on breast cancer using a cell line derived from a breast cancer patient in Texas in 1976. But in 2007 it was confirmed that the cell line that had long been the focus of this research was actually not a breast cancer line, but was instead a skin cancer line. Whoops.
It gets worse.
Fixing Peer Review
Like so much of the academic world, the peer-review process has become corrupt, ineffective, and dysfunctional.
— George Leef
Some people are trying to do something about this problem. Obviously, the elephant in the room is that scientific publishers are in it for the money, so they block access from both creators and consumers. A few items of note:
Mitigating Manipulation in Peer Review — a paper proposing a solution.
Principled Methods to Improve Peer Review — another paper.
The Price of Transparency in Peer Review — article.
An innovative solution from Crane and Martin — I happen to favor this one, please read it.
Using tokens to fix peer review — interesting
Robin Hanson says — Ignore peer review and use a betting market to fix science. Another must-read.