How do we know what is true and what is not? These days we are expected to believe what we hear on television, read in the newspaper and read on the web. Science is portrayed as being able to answer (eventually) all questions and provide some sort of ultimate truth. But how much can you believe that? How much should you believe?
“A lot of what is published is incorrect.” So began a letter in the Comment section in the medical journal the Lancet.1 The following is a sample of excerpts from that letter (my emphases added in bold).
Why the paranoid concern for secrecy and non-attribution? [in regards to government employees.] Because this symposium—on the reproducibility and reliability of biomedical research, held at the Wellcome Trust in London last week—touched on one of the most sensitive issues in science today: the idea that something has gone fundamentally wrong with one of our greatest human creations.
The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness. As one participant put it, “poor methods get results”.
The apparent endemicity of bad research behaviour is alarming. In their quest for telling a compelling story, scientists too often sculpt data to fit their preferred theory of the world. Or they retrofit hypotheses to fit their data.
Can bad scientific practices be fixed? Part of the problem is that no-one is incentivised to be right. Instead, scientists are incentivised to be productive and innovative.