Fixing Science

My name is David Siegel. I want to go after trillion-dollar problems and fix them. One of those problems is the scientific establishment. Quality science is quickly being replaced by scientism and flat-out gaming the system. Today, it’s more about authority, signaling, and narrative control. The fundamentals dropped out long ago.

Richard Feynman called it Cargo-Cult Science.

Science needs a complete overhaul.

Following my work on peer review, I would like to start a new foundation dedicated to reforming science and promoting critical thinking. In this short essay, I’ll lay out the problem and propose my nonprofit solution, hoping someone will be interested to sponsor it.

Introduction

Let’s start with a finding from Roger Pielke, Jr:

A 2015 literature review found almost 900 peer-reviewed studies published on breast cancer using a cell line derived from a breast cancer patient in Texas in 1976. In 2007 it was confirmed that the cell line that had long been the focus of this research was actually not a breast cancer line but was instead a skin cancer line. Whoops.

Even worse, from 2008 to 2014 — after the mistaken cell line was conclusively identified — the review identified 247 peer-reviewed articles putatively on breast cancer that were published using the misidentified skin cancer cell line. A cursory search of Google Scholar indicates that studies continue to be published in 2020 mistakenly using the skin cell line in breast cancer research.

And this:

It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor of The New England Journal of Medicine.

 — Marcia Angell, MD

And this:

“The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness”

— Richard Horton, editor of The Lancet

Now, before you move on, there are two critical videos. The first explains why most published research is wrong:

Some data points:

• Institutions fund more than $1 trillion of research every year. The majority of that is wasted on things that, a-priori, have no chance of success, because the researchers aren’t using good statistical methods, they are just gaming the system to get money.

• Germany is leading Europe down the drain with suicidal energy policy based on very bad science. Putin knows this.

• A 2016 survey by Nature found that more than 70% of researchers have tried and failed to reproduce another scientist's experiment results, and more than half have failed to reproduce their own experiments.

• The survey found that fewer than 31% of researchers believe that failure to reproduce results means that the original result is probably wrong, although 52% agree that a significant replication crisis exists.

• An estimated 795,000 Americans die or are permanently disabled by diagnostic error each year.

• In the biomedical field alone, irreproducible research costs the industry an estimated $38 billion annually​​.

• Over 80% of scientific research data is lost within 20 years, which can be attributed to the complex nature of R&D processes and the lag in documentation of experiments.

• It is claimed that up to 85% of biomedical research could be wasted due to inadequate production and reporting of research, equating to potential waste of $228 billion.

• A 2021 project was able to replicate fewer than half of the early-stage cancer biology studies it looked at.

• Brain imaging studies are usually too weak, statistically speaking, to detect the things they claim to have found. And there are a lot of them.

• “A friend of mine who’s a doctor says that he doesn’t trust what he reads in his medical journals any more. He believes that they’ve been too corrupted by the drug companies. I also know of a psychiatrist who considers medical journals and professional education seminars so compromised by Big Pharma that he relies chiefly on the anecdotal accounts of his peers in prescribing drugs.” — from The Best Science Money can Buy, by David Bollier

My proposal — the big picture

Not everything in science is broken, but a lot is. I propose to use a small amount of money to influence a large amount of scientific activity. At the moment, it’s just me. I want to create a new tax-deductible foundation dedicated to establishing science and the scientific method as the basis for rational policy decisions at the country, state, city, organization, and personal level. I would dedicate it to the memory of Richard Feynman, who said “Science is the belief in the ignorance of experts.” It’s a think tank, a get-things-done tank, and a conduit for many groups and individuals who need support. In conservation, foundations like ReWild and World Conservation Network use this same model very successfully. Here are the components:

Core principles promote critical thinking, Bayesian reasoning, open debate, transparency, inclusiveness, statistical rigor, statistical power, real-world data, limitations of models, look for conflicts of interest, identify biases, red-team checks, antagonistic collaboration, and open communication. Research is an ongoing conversation about data. Be willing to entertain other views. Lone dissenting voices can be right, while institutional consensus can be wrong. No politics. We don’t care what the data says as long as it’s quality data interpreted well.

Move the Overton Window for science. Our primary goal is to improve policy and practice. Today, we have policy-driven data rather than data-driven policy. I don’t plan to create a lobbying firm. I want to help create a foundation on which policymakers can learn critical thinking and overcome their biases. We want to point out the bad incentives and work to get public opinion on the side of science, rather than on pre-determined outcomes and conclusions. Today, science takes a back seat to political agendas.

Research and publish. We will do research and publish reports on the state of what is known and not known on various contentious topics. Examples:

• How much plastic is in our oceans, where does it come from, and what is the effect?

• How effective are today’s mainstream cancer treatments?

• What is really causing the obesity epidemic?

• What can we say about Covid vaccines at this point?

• Who should take statins?

• Are wind farms beneficial?

• Which forestry practices are best?

• What’s happening with freshwater ecosystems?

• What is the cost/benefit of mineral extraction around the world?

• Is the UN’s SDG framework a smart way to allocate resources?

• How can we fix voting systems?

• Will AI kill us all?

• Which scientific publications and platforms are biased and how?

• Which datasets are reliable?

• Do government institutions fabricate data to suit a political agenda?

• How much of our medical-research budget is wasted?

I want to do cost/benefit reports in line with the work of the Copenhagen Consensus Center.

National and international scope. I’m most interested in the big global issues, where trillions of dollars are spent. Below $1 trillion, I don’t think we’ll have time for, though there could be exceptions. Many things influence each other, so focusing on small things can be a good way to change big things.

Our idea of plagiarism will need to change. Plagiarism is serious, but it’s the ideas that matter, not the text, and ideas should be spread — they should have sex, they should have many offspring, and markets should pick them up and use them. Sure, some ideas should be protected to provide research incentives, but the patent system broke decades ago. And of course fraud must be called out, dealt with, and punished. But in the academic world, authorship, gaming the system, and gatekeeping are disproportionally rewarded. In the future, we’ll have to break ideas into modules that can find other modules, interbreed, and create new modules. While plagiarism is a hot topic now — and rightly — once we create better publication mechanisms, this kind of career-limiting scrutiny over what amounts to punctuation and citation error will cease to be front-page news.

Events. I want to create events where people can discuss contentious topics. I don’t think science is ever “settled.” We are always learning more and may need to change our view. We’ll need sponsors for events, but we won’t let the sponsors dictate the content or influence.

Debates. There aren’t enough scientific debates. Can you think of a good debate for AI alignment, climate science, statins, genetically modified organisms, etc? Instead, scientists are defunded, detenured, and deplatformed. There is a loud call to shut down conversations that people don’t agree with, especially people who own and run large social-media platforms. I propose a long-form debate format that is well funded. I have one in mind, though it may not be the first one.

Curriculum development. I want our group to be part of the curriculum conversation. We want to be the opposite of the Gates Foundation, which funnels money into favorite projects and has outsized influence through budget, rather than data and logic.

Courses. I want to offer free online courses and lectures on complexity, systems, statistics, cost/benefit analyisis, scientific experimental procedures, data science, science journalism, neutrality, biases, Bayesian calibration, and more.

Alternatives to academic publication. I believe academic publishing is fundamentally broken. Getting rid of it would be a good start. But what should take its place? How can we help design a system that is fair, cumulative, collaborative, Bayesian, and can’t be gamed? We want a body of accurate scientific evidence that AI can use to help humanity. A big part of this is that the incentives reward bad, clubby behavior. This is far too important to leave up to academia. We want to create, join, and boost good experiments here.

Services. I want to help grantmakers, universities, think tanks, governments, and institutions make science-based decisions. We’d like to build frameworks for how they should allocate their resources. I would love to have a service team that can go help clients decide what to fund, how much to fund, how to insure, how to plan for the future, what bets to take, what programs are cost-effective and which should be killed. We’d love to work on portfolios for large organizations making science-based decisions.

Meta-analyses. I’d like to develop expertise, resources, and teach how to do quality meta-analyses. We could collaborate with the Society for Research Synthesis Methodology, the Cochrane Library, John Ioannidis, and others.

Software. We’d like to develop AI-based tools that help evaluate proposals, read papers, build models, evaluate models, and much more. An early example is Elicit.com, which analyzes research papers.

Market-based science. I want to develop and promote prediction markets for science. One project in this area is Citation Future. There are others. Some, like Metaculus, are easily manipulated, but there are good lessons here and promising experiments to try. I want to be part of this movement and help it develop.

Community. In the same way that Radical Markets, LessWrong, and Astral Codex Ten have online and in-person communities, we want to build a community of critical thinkers who are willing to participate in debate, join and create their own projects, and serve as resources for policymakers.

Media. I want to support blogs, channels, science communicators around the globe. I’d like to have a boot camp for science communicators to come learn for 6 months before they get jobs in the real world. I want to educate thousands of science writers in the scientific method and critical thinking. I want to serve as a resource for them and get them good, balanced sources for their stories.

Public relations. The lay public generally has no idea of the misalignment between science journalism, scientism, and reality. People crave certainty. Today, the majority of science writers are informed by liberal academic groups with a liberal bias. I don’t want to create a politically conservative group to counter that. I want to help people become critical thinkers at all levels. To do that, we need to get the message out. We will need a strong reputation that continues the work of Richard Feynman — to find the truth wherever it is. Or, at the very least, help the public be less wrong.

Prizes and incentives. I want competition of ideas and execution. Many prizes and grants are good, but most are very biased and favor pre-determined outcomes. I would want to research and point out the problems and biases of existing prizes, like the Nobel prizes and others. I don’t know how to get it right, but I believe prizes and recognition can be good incentives.

Whistleblower protection and reward. I want to support people who can show institutional biases, favoritism, fraud, and crime. This is a very promising area for discovery if I can get the money to support it and learn how to do it effectively. I want to help people blowing the whistle at the United Nations, the World Economic Forum, and other politically motivated organizations.

Legal support. Many policies and decisions today are based on bad science and politically motivated arguments. I want to help bring data quality, data science, and cost-benefit analysis to the legal process. This is in stark opposition to groups grooming judges on political objectives using dishonest methods.

Feynman fellowships. I’d love to endow a number of positions in universities around the world. If I can set aside money for this, I will approach Carl and Michelle Feynman and ask what their father would want.

Funding. I want to give grants. Most grantmaking is designed to make grantmakers feel better. We want to give grants that make us feel uncomfortable. We want to come up with a good formula for taking scientific risk — knowing when to put it on and when to take it off. The current fad is to give microgrants to individuals and small groups. We want to understand what works for science and encourage people to find good signals. This is not only true for science but for education and epistemology in general. We want to fund replication studies. We want to fund boring, unsensational work that helps get to the foundation of science, clears the rubble, and makes way for new approaches. We may want to support a few individual scientists, especially those shunned by academia. We’ll mostly give money to groups moving science forward. We will allocate money carefully, with an eye to results and institutional learning.

Fundraising. I want to create the group that donors feel good about giving money to. I want to aggregate money and give it out systematically.

My proposal — getting started

Rome wasn’t built in a day. I’m not hiring a team of specialists in specific areas — I want to give money to specialists doing their thing. In-house, I want to build a core team of statisticians, systems-thinkers, researchers, meta-analysts, teachers, and communicators. I also want a small but aggressive fundraising team that will help us grow. It won’t be a big group. We’ll give 80 percent of our annual income to outside groups. We’ll also form relationships with many of the groups I’ve mentioned.

Once I hire a core team, our initial effort will be to create a cost/benefit report of how best to use the world’s scientific resources to benefit mankind. This is an effort on the scale of the Copenhagen Consensus’s Best Things First project, with an eye toward producing better scientific outcomes. This will take the better part of a year. During this time, we’ll also give money to a few groups and start working on our big annual event. That event will mark the end of the first year and lay out our road map for the next several years.

Budget

I’m looking for $20 million per year for 5 years. This is a tiny drop in the bucket compared to the liberal billions being spread for scientific and green signaling. It’s an even smaller drop compared with government largesse for white-elephant projects. With this money we will hit the ground running, rack up some small wins, tell a good story, and attract more funding from donors aligned with our purpose.

Name

I’d call it the Giordano Bruno Institute, but there are plenty of other options. This is something to discuss with the initial sponsor.

Resources

The Research Excellence Framework in the UK
The Broken Science Initiative
Retraction Watch
Neuro-Nonsense — The Folly in Trying to Find the Location of Behavior
Have you ever been wrong?
William M Briggs
The failure that is peer review
Friends of Science, Canada
We don’t know how to fix science
The Kleio Group
Fixing Science using a new science of science, by Yaneer Bar-Yam
Scientifically sound
Peer review: a flawed process at the heart of science and journals
Two decades of Alzheimer’s research was based on deliberate fraud by 2 scientists that has cost billions of dollars and millions of lives
Arc BioTech Institute at Stanford

Potential funders

I’ve already built one Swiss nonprofit foundation. I can create a US-based nonprofit foundation quickly. I’m looking for partners, money, and introductions to people who want to help. I’m happy to talk with anyone who doesn’t want to influence the outcome. These people are capable and, I hope, interested to talk with me about this project:

Patrick Collison, founder of Stripe.

Peter Thiel, founder of PayPal

Mark Cuban, founder of Broadcast.com

Bill Ackman, founder of Pershing Square Capital

Vinod Khosla, founder of Sun Microsystems

Ken Griffin, founder of Citadel hedge fund

Timothy Mellon, banking heir

Steven Schwarzman, Blackstone

Patrick Ryan, Aon insurance

Paul Singer, Elliott management

Brian Armstrong, Coinbase

David Cheriton, Stanford

And you, maybe?

I’m asking for conversations

I’m not asking for money based on a blog post and some fun AI-generated images. I’m asking for a conversation with anyone who could potentially support this with $1 million per year for five years (or more). I’d love to get input and feedback on how it should get started and where it should go. If you’re interested, contact me and let’s set up a time to talk.