Physicists have a good excuse for huddling under the streetlight when they are pushing at the limits of human understanding. But the effect also vexes medical research, where you might think great patient data is there for the tabulating. The story of the anti-arrhythmia drugs only hints at the extent of the problem. In 2005, John Ioannidis of the University of Ioannina in Greece examined the 45 most prominent studies published since 1990 in the top medical journals and found that about one-third of them were ultimately refuted. If one were to look at all medical studies, it would be more like two-thirds, he says. And for some kinds of leading-edge studies, like those linking a disease to a specific gene, wrongness infects 90 percent or more.
We should fully expect scientific theories to frequently butt heads and to wind up being disproved sometimes as researchers grope their way toward the truth. That is the scientific process: Generate ideas, test them, discard the flimsy, repeat. In fact, testing ideas is supposed to be the core competence of most scientists. But if tests of the exact same idea routinely generate differing, even opposite, results, then what are we humble nonscientists supposed to believe?
I have spent the past three years examining why expert pronouncements so often turn out to be exaggerated, misleading, or flat-out wrong. There are several very good reasons why that happens, and one of them is that scientists are not as good at making trustworthy measurements as we give them credit for. It’s not that they are mostly incompetents and cheats. Well, some of them are: In several confidential surveys spanning different fields, anywhere from 10 to 50 percent of scientists have confessed to perpetrating or being aware of some sort of research misbehavior. And numerous studies have highlighted remarkably lax supervision of research assistants and technicians. A bigger obstacle to reliable research, though, is that scientists often simply cannot get at the things they need to measure.
for complete article:http://discovermagazine.com/2010/jul-aug/29-why-scientific-studies-often-wrong-streetlight-effect
***********************************************************************************************************
Validity of Research Though it is often assumed that a study’s results are valid or conclusive just because the study is scientific, unfortunately, this is not the case. Researchers who conduct scientific studies are often motivated by external factors, such as the desire to get published, advance their careers, receive funding, or seek certain results. As a consequence, a significant number of scientific studies are biased and unreliable. Dr. Ioannidis, a meta-researcher who is one of the world’s experts on the credibility of medical research, found that the studies that tend to get published are those with eye-catching findings, which when studied rigorously collapse under the weight of contradictory data. “Imagine, though, that five different research teams test an interesting theory that’s making the rounds, and four of the groups correctly prove the idea false, while the one less cautious group incorrectly ‘proves’ it true through some combination of error, fluke, and clever selection of data. Guess whose findings your doctor ends up reading about in the journal, and you end up hearing about on the evening news?” (1) Ioannidis also found that “researchers headed into their studies wanting certain results- and, lo and behold, they were getting them. We think of the scientific process as being objective, rigorous, and even ruthless in separating out what is true from what we merely wish to be true, but in fact it’s easy to manipulate results, even unintentionally or unconsciously.” (2)
Due to the large influence that external factors have on researchers and their studies, Ioannidis has dedicated much of his career to exposing the unreliability of scientific studies. He declared that “much of what biomedical researchers conclude in published studies- conclusions that doctors keep in mind when they prescribe antibiotics or blood-pressure medication, or when they advise us to consume more fiber or less meat, or when they recommend surgery for heart disease or back pain- is misleading, exaggerated, and often flat-out wrong. He charges that as much as 90 % of the published medical information that doctors rely on is flawed.” (3) Ioannidis subsequently decided to create a mathematical proof to prove that many research studies are unreliable. His model predicted that “rates of wrongness roughly corresponded to the observed rates at which findings were later convincingly refuted: 80 % of non-randomized studies (by far the most common type) turn out to be wrong, as do 25 % of supposedly gold-standard randomized trials.” (4) To further prove his findings that researchers frequently manipulate data and chase career advancing findings rather than good science, Ioannidis focused on 49 of the most highly regarded research findings in medicine (those which had appeared in the journals most widely cited in research articles and those which themselves were most widely cited). He found that of the top 49 articles, 45 claimed to have uncovered effective interventions, however, 34 of these claims had been retested, and of those, 41% had been shown to be wrong. “If between a third and a half of the most acclaimed research in medicine was proving untrustworthy, the scope and impact of the problem were undeniable.” (5)
uniteforsight.org |
