Selective outcome reporting in medical journals
Slate has an interesting article implicating selective outcomes reporting as a major contributor to pervasive off-label drug prescribing. This practice can be summarized in the following from this article:
According to a remarkable analysis of the Neurontin documents, published last month, many clinical trials of the drug took a shotgun approach. Study patients took the drug, and researchers measured tons of possible outcomes (like pain with touch, pain with cold, excessive pain with pinpricks, more than a dozen different scales for psychiatric symptoms, and so on). By random chance, if you measure enough outcomes, at least some of them will appear better after drug treatment. When the time came to report the findings, however, the researchers systematically omitted the outcomes on which the drug had no effect—and presented only the data showing benefit. That’s like dealing dozens of hands of poker to yourself but showing only the hand with good cards.
Such research practices can be characterized as sloppy–at best–or, more accurately, as unethical and fraudulent. In the case of Neurontin, the researchers seem to have clearly set out to circumvent the scientific method in their work. However, researchers conducting properly designed and ethical projects are also likely to omit at least some results in the presentation of their work. Space is limited in a journal article. Most major or original research articles published in leading medical journals are limited to around 3,000 words and generally three or four tables and figures. For a large study with dozens of secondary outcomes, this is not enough room to detail all results. Thus, outcomes are selectively omitted. Hopefully, only less meaningful or non-clinically relevant outcomes are omitted. More often than not, a result will be omitted if “no difference” or equivalence is shown. This seems like a sensible approach when space is so limited. However, as the Slate article points out, this can become very important when considered in terms of off-label prescribing.
So, what’s the solution? It would be nice if researchers weren’t limited by a word count when presenting their work in medical journals. Unfortunately, this is unrealistic. We would likely see 15,000 word articles. Obviously, the medical journals can’t publish book-sized journals each week. Paper is expensive. Bandwidth, however, is relatively cheap. Thanks to the internet, publishers can put supplemental materials online in conjunction with what is physically printed in the journal. A few medical journals already do this in certain instances. This practice needs to become more prevalent and should be encouraged by editors. Not only will this give more information to physicians when considering something like off-label prescribing, future research can be guided by more information from previous work. Medical journals (along with the rest of the medical world) have been very slow to adopt new technologies. This is one area where they can jump ahead and set a new precedent for peer-reviewed research.