Advertisement
  • Loading metrics

Bias, Spin, and Misreporting: Time for Full Access to Trial Protocols and Results

  • An-Wen Chan

Bias, Spin, and Misreporting: Time for Full Access to Trial Protocols and Results

  • An-Wen Chan
PLOS
x

Although randomized trials provide key guidance for how we practice medicine, trust in their published results has been eroded in recent years due to several high-profile cases of alleged data suppression, misrepresentation, and manipulation [1–5, 39]. While most publicized cases have involved pharmaceutical industry trials, accumulating empiric evidence has shown that selective reporting of results is a systemic problem afflicting all types of trials, including those with no commercial input [6]. These examples highlight the harmful potential impact of biased reporting on patient care, and the violation of ethical responsibilities of researchers and sponsors to disseminate results accurately and comprehensively.

Biased reporting arises when two main decisions are made based on the direction and statistical significance of the data—whether to publish the trial at all, and if so, which analyses and results to report in the publication. Strong evidence for the selective publication of positive trials has been available for decades [7,8]. More recent cohort studies have focused on the misreporting of trials within publications by comparing journal articles either with documents from regulatory agencies [9–12] or with trial protocols from research ethics committees [13–16], funding agencies [17], research groups [18,19], and journals [20]. These cohort studies identified major discrepancies—favorable results were often highlighted while unfavorable data were suppressed; definitions of primary outcomes were changed; and methods of statistical analysis were modified without explanation in the journal article.

Linked Research Article

This Perspective discusses the following new study published in PLoS Medicine:

Rising K, Bacchetti P, Bero L (2008) Reporting bias in drug trials submitted to the Food and Drug Administration: A review of publication and presentation. PLoS Med 5(11): e217. doi:10.1371/journal.pmed.0050217

Lisa Bero and colleagues review the publication status of all efficacy trials carried out in support of new drug approvals from 2001 and 2002, and find that a quarter of trials remain unpublished.

New Evidence

In a new study published in PLoS Medicine, Lisa Bero and colleagues make an important contribution to the growing body of evidence that the randomized trial literature is skewed towards reporting favorable results [9]. The researchers identified trials from 33 new drug applications (NDAs) for new molecular entities approved by the United States Food and Drug Administration (FDA) in 2001–2002, and compared information from FDA reviews with journal articles. By including all NDAs from a variety of specialty fields, their findings have broad generalizability to pharmaceutical trials.

Overall, a substantial amount of primary outcome data submitted to the FDA was found to be missing from the literature. One quarter of trials in their sample were unpublished—predominantly those with unfavorable results. Not only were data suppressed for the unpublished trials, but an additional quarter of primary outcomes were omitted from journal articles of published trials. These findings are consistent with two recent reviews of FDA documents and journal articles [10,21], one of which was published in PLoS Medicine in September 2008 [21].

Bero and colleagues also identified important discrepancies between the primary outcomes, statistical analyses, and conclusions presented in NDAs versus those reported in journal articles. The vast majority of discrepancies favored the sponsor's new drug, suggesting biased reporting. While it is possible that the FDA requested modifications to the sponsor's analyses, these amendments should be mentioned in the FDA's statistical review; should not involve altering primary outcomes without explanation in the publication; and would not be expected to favor the sponsor's drug as often as was found in this study.

Biased reporting of results from NDA trials is particularly concerning because these journal articles are the only peer-reviewed source of information on recently approved drugs for health care providers, who will have had limited clinical experience with these new treatments. There are also substantial cost implications if the efficacy is overestimated and the drugs overused, as new molecular entities are among the most expensive pharmaceuticals on the market [22].

The Need for Increased Transparency

Since the interests of patients are of utmost importance, it is difficult to justify why health care providers and policy makers should have access to only a biased subset of information that is substantially different from that which regulatory agencies have at their disposal. Bero and colleagues' study highlights the importance of public access to key documents that have traditionally been deemed confidential—regulatory agency submissions and trial protocols. Both types of documents have unique properties that complement each other.

Regulatory agency submissions represent the final description of how the trial was conducted and analyzed prior to journal publication. However, details from these submissions are not publicly available in most countries. Although summaries of FDA reviews are posted on the FDA Web site, their content and availability is variable, and sections are often redacted [9,21,23]. Furthermore, regulatory agency submissions are prepared by companies after data analysis and may themselves be subject to biased reporting. Finally, only devices, pharmaceuticals, and biological agents require regulatory approval in the United States and other countries, meaning that trials examining other types of interventions (e.g., surgery, education)—which constitute 20% of published randomized trials [24]—would be excluded from reviews of regulatory agency documents. Pharmaceutical trials conducted post-approval would also be missed.

On the other hand, protocols constitute the most comprehensive description of study design prior to trial inception. Their content therefore cannot be influenced by the study results. However, access to trial protocols is particularly difficult to obtain [25,26]. As with summaries of FDA reviews, their content is also highly variable and often lacks sufficient detail [13–18,20]. The SPIRIT initiative (Standard Protocol Items for Randomized Trials) aims to address these deficiencies by producing evidence-based recommendations for key information to include in a trial protocol [27].

Time for Action

It is clear that the trial literature is biased, facilitated in part by limited oversight and difficulty in accessing detailed trial documents. Ongoing progress in trial registration and results disclosure represents a key initial step towards ensuring public access to basic information on trial methods and results [28–33]. Several journals have also acted by publishing protocols and requiring their submission with manuscripts [34–36].

However, much remains to be done—not only to establish reliable, comprehensive registration and results disclosure processes worldwide, but also to start heeding the calls for increased access to full protocols and regulatory agency submissions [14,23,33,37,38]. As shown by recent examples and studies highlighted above, misreporting of trials can be difficult to detect without access to detailed documents beyond what is currently available on registries and results databases. Only with full transparency can the validity of a randomized trial be judged.

The time has come to tackle the challenge of making key trial documents public. It has taken decades for trial registration and results disclosure to be implemented; hopefully, for the sake of patients, public access to full protocols and regulatory agency submissions will come much sooner.

Acknowledgments

I am very grateful to Professor Doug Altman for his helpful comments on a previous version of this paper.

Note Added in Proof

Reference 39 was added after this article was already in proof.

References

  1. 1. McHenry LB, Jureidini JN (2008) Industry-sponsored ghostwriting in clinical trial reporting: A case study. Account Res 15: 152–167.LB McHenryJN Jureidini2008Industry-sponsored ghostwriting in clinical trial reporting: A case study.Account Res15152167
  2. 2. Jureidini JN, McHenry LB, Mansfield PR (2008) Clinical trials and drug promotion: Selective reporting of study 329. Int J Risk Safety Med 20: 73–81.JN JureidiniLB McHenryPR Mansfield2008Clinical trials and drug promotion: Selective reporting of study 329.Int J Risk Safety Med207381
  3. 3. Psaty BM, Kronmal RA (2008) Reporting mortality findings in trials of rofecoxib for Alzheimer disease or cognitive impairment: A case study based on documents from rofecoxib litigation. JAMA 299: 1813–1817.BM PsatyRA Kronmal2008Reporting mortality findings in trials of rofecoxib for Alzheimer disease or cognitive impairment: A case study based on documents from rofecoxib litigation.JAMA29918131817
  4. 4. Curfman GD, Morrissey S, Drazen JM (2006) Expression of concern reaffirmed. N Engl J Med 354: 1193.GD CurfmanS. MorrisseyJM Drazen2006Expression of concern reaffirmed.N Engl J Med3541193
  5. 5. Whittington CJ, Kendall T, Fonagy P, Cottrell D, Cotgrove A, et al. (2004) Selective serotonin reuptake inhibitors in childhood depression: Systematic review of published versus unpublished data. Lancet 363: 1341–1345.CJ WhittingtonT. KendallP. FonagyD. CottrellA. Cotgrove2004Selective serotonin reuptake inhibitors in childhood depression: Systematic review of published versus unpublished data.Lancet36313411345
  6. 6. Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan A-W, et al. (2008) Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE 3: e3081.K. DwanDG AltmanJA ArnaizJ. BloomA-W Chan2008Systematic review of the empirical evidence of study publication bias and outcome reporting bias.PLoS ONE3e3081
  7. 7. Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ (2000) Publication and related biases. Health Technol Assess 4: 1–115.F. SongAJ EastwoodS. GilbodyL. DuleyAJ Sutton2000Publication and related biases.Health Technol Assess41115
  8. 8. Dickersin K (1997) How important is publication bias? A synthesis of available data. AIDS Educ Prev 9: 15–21.K. Dickersin1997How important is publication bias? A synthesis of available data.AIDS Educ Prev91521
  9. 9. Rising K, Bacchetti P, Bero L (2008) Reporting bias in drug trials submitted to the Food and Drug Administration: A review of publication and presentation. PLoS Med 5: e217.K. RisingP. BacchettiL. Bero2008Reporting bias in drug trials submitted to the Food and Drug Administration: A review of publication and presentation.PLoS Med5e217
  10. 10. Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R (2008) Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 358: 252–260.EH TurnerAM MatthewsE. LinardatosRA TellR. Rosenthal2008Selective publication of antidepressant trials and its influence on apparent efficacy.N Engl J Med358252260
  11. 11. Melander H, Ahlqvist-Rastad J, Meijer G, Beermann B (2003) Evidence b(i)ased medicine—Selective reporting from studies sponsored by pharmaceutical industry: Review of studies in new drug applications. BMJ 326: 1171–1173.H. MelanderJ. Ahlqvist-RastadG. MeijerB. Beermann2003Evidence b(i)ased medicine—Selective reporting from studies sponsored by pharmaceutical industry: Review of studies in new drug applications.BMJ32611711173
  12. 12. Hemminki E (1980) Study of information submitted by drug companies to licensing authorities. BMJ 280: 833–836.E. Hemminki1980Study of information submitted by drug companies to licensing authorities.BMJ280833836
  13. 13. Chan A-W, Hróbjartsson A, Jørgensen KJ, Gøtzsche PC, Altman DG (2008) Discrepancies in sample size calculations and data analyses reported in randomized trials: Comparison of publications with protocols. BMJ. A-W ChanA. HróbjartssonKJ JørgensenPC GøtzscheDG Altman2008Discrepancies in sample size calculations and data analyses reported in randomized trials: Comparison of publications with protocols.BMJIn press. In press.
  14. 14. Chan A-W, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG (2004) Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. JAMA 291: 2457–2465.A-W ChanA. HróbjartssonMT HaahrPC GøtzscheDG Altman2004Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles.JAMA29124572465
  15. 15. Hahn S, Williamson PR, Hutton JL (2002) Investigation of within-study selective reporting in clinical research: Follow-up of applications submitted to a local research ethics committee. J Eval Clin Pract 8: 353–359.S. HahnPR WilliamsonJL Hutton2002Investigation of within-study selective reporting in clinical research: Follow-up of applications submitted to a local research ethics committee.J Eval Clin Pract8353359
  16. 16. Pildal J, Chan A-W, Hróbjartsson A, Forfang E, Altman DG, et al. (2005) Does unclear allocation concealment in trial publications reflect poor methods or poor reporting of adequate methods? Cohort study of trial protocols and corresponding published reports. BMJ 330: 1049–1052.J. PildalA-W ChanA. HróbjartssonE. ForfangDG Altman2005Does unclear allocation concealment in trial publications reflect poor methods or poor reporting of adequate methods? Cohort study of trial protocols and corresponding published reports.BMJ33010491052
  17. 17. Chan A-W, Krleža-Jeric K, Schmid I, Altman DG (2004) Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ 171: 735–740.A-W ChanK. Krleža-JericI. SchmidDG Altman2004Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research.CMAJ171735740
  18. 18. Scharf O, Colevas AD (2006) Adverse event reporting in publications compared with sponsor database for cancer clinical trials. J Clin Oncol 24: 3933–3938.O. ScharfAD Colevas2006Adverse event reporting in publications compared with sponsor database for cancer clinical trials.J Clin Oncol2439333938
  19. 19. Soares HP, Daniels S, Kumar A, Clarke M, Scott C, et al. (2004) Bad reporting does not mean bad methods for randomised trials: Observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ 328: 22–24.HP SoaresS. DanielsA. KumarM. ClarkeC. Scott2004Bad reporting does not mean bad methods for randomised trials: Observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group.BMJ3282224
  20. 20. Al-Marzouki S, Roberts I, Evans S, Marshall T (2008) Selective reporting in clinical trials: Analysis of trial protocols accepted by The Lancet. Lancet 372: 201.S. Al-MarzoukiI. RobertsS. EvansT. Marshall2008Selective reporting in clinical trials: Analysis of trial protocols accepted by The Lancet.Lancet372201
  21. 21. Lee K, Bacchetti P, Sim I (2008) Publication of clinical trials supporting successful new drug applications: A literature analysis. PLoS Med 5: e191.K. LeeP. BacchettiI. Sim2008Publication of clinical trials supporting successful new drug applications: A literature analysis.PLoS Med5e191
  22. 22. Morgan SG, Bassett KL, Wright JM, Evans RG, Barer ML, et al. (2005) “Breakthrough” drugs and growth in expenditure on prescription drugs in Canada. BMJ 331: 815–816.SG MorganKL BassettJM WrightRG EvansML Barer2005“Breakthrough” drugs and growth in expenditure on prescription drugs in Canada.BMJ331815816
  23. 23. Turner EH (2004) A taxpayer-funded clinical trials registry and results database. PLoS Med 1: e60.EH Turner2004A taxpayer-funded clinical trials registry and results database.PLoS Med1e60
  24. 24. Chan A-W, Altman DG (2005) Epidemiology and reporting of randomised trials published in PubMed journals. Lancet 365: 1159–1162.A-W ChanDG Altman2005Epidemiology and reporting of randomised trials published in PubMed journals.Lancet36511591162
  25. 25. Chan A-W, Upshur R, Singh JA, Ghersi D, Chapuis F, et al. (2006) Research protocols: Waiving confidentiality for the greater good. BMJ 332: 1086–1089.A-W ChanR. UpshurJA SinghD. GhersiF. Chapuis2006Research protocols: Waiving confidentiality for the greater good.BMJ33210861089
  26. 26. Lurie P, Zieve A (2008) Sometimes the silence can be like the thunder: Access to pharmaceutical data at the FDA. Law Contemporary Problems 69: 85–97.P. LurieA. Zieve2008Sometimes the silence can be like the thunder: Access to pharmaceutical data at the FDA.Law Contemporary Problems698597
  27. 27. Chan A-W, Tetzlaff J, Altman DG, Gøtzsche PC, Hróbjartsson A, et al. (2008) The SPIRIT initiative: Defining Standard Protocol Items for Randomized Trials [conference abstract]. German J Evid Quality Health Care (suppl) 102: S27.A-W ChanJ. TetzlaffDG AltmanPC GøtzscheA. Hróbjartsson2008The SPIRIT initiative: Defining Standard Protocol Items for Randomized Trials [conference abstract].German J Evid Quality Health Care (suppl)102S27
  28. 28. World Health Organization (2008) International Clinical Trials Registry Platform. World Health Organization2008International Clinical Trials Registry Platform.Available: http://www.who.int/ictrp/en/. Accessed 20 October 2008. Available: http://www.who.int/ictrp/en/. Accessed 20 October 2008.
  29. 29. The PLoS Medicine Editors (2008) Next stop, don't block the doors: Opening up access to clinical trials results. PLoS Med 5: e160.The PLoS Medicine Editors2008Next stop, don't block the doors: Opening up access to clinical trials results.PLoS Med5e160
  30. 30. United States Congress (2007) Food and Drug Administration Amendments Act of 2007, Title VIII, Section 801. Expanded clinical trial registry data bank. United States Congress2007Food and Drug Administration Amendments Act of 2007, Title VIII, Section 801. Expanded clinical trial registry data bank.Available: http://www.govtrack.us/congress/billtext.xpd?bill=h110-3580. Accessed 20 October 2008. Available: http://www.govtrack.us/congress/billtext.xpd?bill=h110-3580. Accessed 20 October 2008.
  31. 31. Sim I, Chan A-W, Gulmezoglu AM, Evans T, Pang T (2006) Clinical trial registration: Transparency is the watchword. Lancet 367: 1631–1633.I. SimA-W ChanAM GulmezogluT. EvansT. Pang2006Clinical trial registration: Transparency is the watchword.Lancet36716311633
  32. 32. DeAngelis CD, Drazen JM, Frizelle FA, Haug C, Hoey J, et al. (2005) Is this clinical trial fully registered?: A statement from the International Committee of Medical Journal Editors. JAMA 293: 2927–2929.CD DeAngelisJM DrazenFA FrizelleC. HaugJ. Hoey2005Is this clinical trial fully registered?: A statement from the International Committee of Medical Journal Editors.JAMA29329272929
  33. 33. Krleža-Jeric K, Chan A-W, Dickersin K, Sim I, Grimshaw J, et al. (2005) Principles for international registration of protocol information and results from human trials of health related interventions: Ottawa statement (part 1). BMJ 330: 956–958.K. Krleža-JericA-W ChanK. DickersinI. SimJ. Grimshaw2005Principles for international registration of protocol information and results from human trials of health related interventions: Ottawa statement (part 1).BMJ330956958
  34. 34. PLoS Medicine (2008) Guidelines for authors. PLoS Medicine2008Guidelines for authors.Available: http://journals.plos.org/plosmedicine/guidelines.php#supporting. Accessed 20 October 2008. Available: http://journals.plos.org/plosmedicine/guidelines.php#supporting. Accessed 20 October 2008.
  35. 35. McNamee D, James A, Kleinert S (2008) Protocol review at The Lancet. Lancet 372: 189–190.D. McNameeA. JamesS. Kleinert2008Protocol review at The Lancet.Lancet372189190
  36. 36. Jones G, Abbasi K (2004) Trial protocols at the BMJ. BMJ 329: 1360.G. JonesK. Abbasi2004Trial protocols at the BMJ.BMJ3291360
  37. 37. Lassere M, Johnson K (2002) The power of the protocol. Lancet 360: 1620–1622.M. LassereK. Johnson2002The power of the protocol.Lancet36016201622
  38. 38. Hawkey CJ (2001) Journals should see original protocols for clinical trials. BMJ 323: 1309.CJ Hawkey2001Journals should see original protocols for clinical trials.BMJ3231309
  39. 39. Mitka M (2008) Controversies surround heart drug study: Questions about Vytorin and trial sponsors' conduct. JAMA 299: 885–887.M. Mitka2008Controversies surround heart drug study: Questions about Vytorin and trial sponsors' conduct.JAMA299885887