One of my favorite journals is called PLOS ONE. This is a journal which supports open access. That means anyone can access any article in this Journal without paying a fee. Medical studies published in this journal are accessible to anyone.
Most of you probably don’t realize but when you see a medical study quoted in a newspaper article, you can’t actually access the original study on the Internet without paying a hefty fee, usually $20-$40! If you have access to a medical library then you may be able to access the article but for most people the original articles are off-limits without paying large fees.
Plos.org is an organization that supports open access publication of scientific articles.
That’s why I admire them.
Back to my main story. A recent study in PLOS ONE looked at how often medical research results are replicated, meaning does a second or third similar study show the same results.
The researchers in this study looked at 4723 studies that were included in 306 meta-analysis articles. (A meta-analysis is a study where you combine the results of many other research studies in order to get an overview of findings.) The researchers divided the studies into lifestyle related studies which looked at things like drinking coffee or smoking cigarettes and non-lifestyle studies such as genetic markers for Alzheimer’s. There were 639 lifestyle studies and 4084 non-lifestyle studies.
The question is of the studies that were picked up by newspapers, how many of them were replicated by subsequent studies? The answer is only about half of the studies held up when tested again in another study. The other thing that was interesting in this article was that when studies failed to replicate, newspapers never reported that failure. Interesting examples included studies that linked a specific gene to depression, schizophrenia, or autism. None of these studies replicated successfully, which you think would be big news and would be reported by many newspapers, but the truth is that not a single newspaper article reported these failures to replicate.
This shows that newspapers don’t have much genuine interest in good science reporting. Good science reporting always involves being skeptical of new and different results, as well as following up on attempts to replicate those results.
So, what does this mean about science results reported in popular media? What it probably means is that if the finding is new and exciting and different, you probably should be highly skeptical of it being true. And the more esoteric the finding is (such as genetic markers) the more skeptical you should be.
For instance, a recent study that was funded by drug companies looked at whether the statin class of medications have side effects or whether these side effects are just a placebo effect. I’ll write more extensively about this study later, but the study’s findings–that only when people knew they were taking statins did they experience side effects– should probably be viewed very skeptically since many other studies have shown side effects from statins and many clinical reports have confirmed the side effects. (And of course any study that is funded by the manufacturer of a drug should be viewed highly skeptically.)
The bottom line is this: finding the truth is hard, and science is no shortcut. Only findings that have been repeated and replicated in numerous studies should be believed.
Very interesting post! Whereas I agree with your general conclusion that science is hard (and no shortcut), I think that you may have not taking your own point enough to heart. Much as I’m a fan of PLoS ONE, it is considered by some scientists to be a bit of a dumping ground, where papers that couldn’t get published elsewhere get sent. I have this on very good authority, as I have four papers in PLoS ONE, and it was never my first choice of venue! So, just as one shouldn’t believe everything one reads in the newspaper, one shouldn’t believe everything that one reads ANYWHERE! In fact, productive skepticism is perhaps THE defining characteristic of science. That said, news papers (and journals!) have to published something, and if one waited for meta-analyses for every decision, one might be waiting a long time before deciding whether to eat that steak or not! (Recently we found that NSAIDs and even Flossing are evil!) So, whereas I’m a fan of healthy skepticism, I’m also a fan of making one’s best guess from imperfect information. After all, to quote a famous philosopher, it’s not the known unknowns that get you, it’s the unknown unknowns! 🙂
Cheers,
‘Jeff
ps. Unless the author is dead, or their email has changed, you can almost always get a free reprint of any paper by just emailed the corresponding author. So PLoS ONE isn’t that much of a revolution, and, as above, it has issues.
pps. I don’t think that an “overview” is how one should characterize a meta-analysis. Meta analyses are generally considered stronger evidence than the individual studies, although they have to be done right (as does any study!)
Reply