Citation: Doshi P, Jefferson T, Del Mar C (2012) The Imperative to Share Clinical Study Reports: Recommendations from the Tamiflu Experience. PLoS Med 9(4): e1001201. https://doi.org/10.1371/journal.pmed.1001201
Published: April 10, 2012
Copyright: © 2012 Doshi et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: Peter Doshi is funded by an institutional training grant from the Agency for Healthcare Research and Quality #T32HS019488. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: All review authors have applied for and received competitive research grants. All review authors are co-recipients of a UK National Institute for Health Research grant to carry out a Cochrane review of neuraminidase inhibitors (http://www.hta.ac.uk/2352). In addition: TJ was an ad hoc consultant for F. Hoffman-La Roche Ltd in 1998–1999. TJ receives royalties from his books published by Blackwells and Il Pensiero Scientifico Editore, none of which are on neuraminidase inhibitors. TJ is occasionally interviewed by market research companies for anonymous interviews about Phase 1 or 2 products unrelated to neuraminidase inhibitors. Since submission of this article, TJ has been retained as an expert consultant in a legal case involving Tamiflu. CDM provided expert advice to GlaxoSmithKline about vaccination against acute otitis media in 2008–2009. CDM receives royalties from books published through Blackwells BMJ Books and Elsevier. PD declares no further conflicts of interest.
Abbreviations: ACIP, Advisory Committee on Immunization Practices; CDC, US Centers for Disease Control and Prevention; EMA, European Medicines Agency; FDA, US Food and Drug Administration; HHS, US Department of Health and Human Services; WHO, World Health Organization
Provenance: Not commissioned; externally peer reviewed.
- Systematic reviews of published randomized clinical trials (RCTs) are considered the gold standard source of synthesized evidence for interventions, but their conclusions are vulnerable to distortion when trial sponsors have strong interests that might benefit from suppressing or promoting selected data.
- More reliable evidence synthesis would result from systematic reviewing of clinical study reports—standardized documents representing the most complete record of the planning, execution, and results of clinical trials, which are submitted by industry to government drug regulators.
- Unfortunately, industry and regulators have historically treated clinical study reports as confidential documents, impeding additional scrutiny by independent researchers.
- We propose clinical study reports become available to such scrutiny, and describe one manufacturer's unconvincing reasons for refusing to provide us access to full clinical study reports. We challenge industry to either provide open access to clinical study reports or publically defend their current position of RCT data secrecy.
Regulatory approval of new drugs is assumed to reflect a judgment that a medication's benefits outweigh its harms. Despite this, controversy over approved drugs is common. If sales can be considered a proxy for utility, the controversies surrounding even the most successful drugs (such as blockbuster drugs) seem all the more paradoxical, and have revealed the extent to which the success of many drugs has been driven by sophisticated marketing rather than verifiable evidence ,. But even among institutions that aim to provide the least biased, objective assessments of a drug's effects, determining “the truth” can be extremely difficult.
Consider the case of the influenza antiviral Tamiflu (oseltamivir). Prior to the global outbreak of H1N1 influenza in 2009, the United States alone had stockpiled nearly US$1.5 billion dollars worth of the antiviral . As the only drug in its class (neuraminidase inhibitors) available in oral form, Tamiflu was heralded as the key pharmacologic intervention for use during the early days of an influenza pandemic when a vaccine was yet to be produced. It would cut hospitalizations and save lives, said the US Department of Health and Human Services (HHS) . The Advisory Committee on Immunization Practices (ACIP, the group the US Centers for Disease Control and Prevention [CDC] uses to form national influenza control policy) said it would reduce the chances of developing complications from influenza . So, too, did the Australian Therapeutic Goods Administration  and the European Medicines Agency (EMA) .
Most (perhaps all) of these claims can be traced back to a single source: a meta-analysis published in 2003 that combined ten randomized clinical trials conducted during the late 1990s by the manufacturer prior to US registration of the drug . This analysis, conducted by Kaiser and colleagues, proposed that oseltamivir treatment of influenza reduced both secondary complications and hospital admission. In contrast, the Food and Drug Administration (FDA), which approved Tamiflu in 1999 and was aware of these same clinical trials, concluded that Tamiflu had not been shown to reduce complications, and required an explicit statement in the drug's label to that effect . FDA even cited Roche, Tamiflu's manufacturer, for violation of the law for claims made to the contrary .
Nor did the FDA approve an indication for Tamiflu in the prevention of transmission of influenza ,. This assumption was at the heart of the World Health Organization's (WHO) proposed plan to suppress an emergent pandemic through mass prophylaxis . While the WHO recently added Tamiflu to its Essential Medicines list, if FDA is right, the drug's effectiveness may be no better than aspirin or acetaminophen (paracetemol). The FDA has never clarified the many discrepancies in claims made over the effects of Tamiflu. Although it may have limited approval indications accordingly, the FDA has never challenged the US HHS or the US CDC for making far more ambitious claims. This means that critical analysis by an independent group such as a Cochrane review group is essential.
But which data should be used? In updating our Cochrane review of neuraminidase inhibitors, we have become convinced that the answer lies in analyzing clinical study reports rather than the traditional published trials appearing in biomedical journals . Clinical study reports contain the same information as journal papers (usually in standardized sections, including an introduction, methods, results, and conclusion ), but have far more detail: the study protocol, analysis plan, numerous tables, listings, and figures, among others. They are far larger (hundreds or thousands of pages), and represent the most complete synthesis of the planning, execution, and results of a clinical trial. Journal publications of clinical trials may generate media attention , propel researchers' careers, and generate some journals a revenue stream . However, when regulators decide whether to register a new drug in a manufacturer's application, they review the trial's clinical study report.
In 2010, we began our Cochrane review update using clinical study reports rather than published papers . We obtained some sections of these clinical study reports for the ten trials appearing in the Kaiser 2003 meta-analysis from Tamiflu's manufacturer, Roche—around 3,200 pages in total. In 2011, we obtained additional sections of clinical study reports for Tamiflu through a Freedom of Information request to the EMA, amounting to tens of thousands of pages. While extensive and detailed, it is important to note that what we have obtained is just a subset of the full clinical study reports in Roche's possession. Nonetheless, Box 1 provides a list of details we have already discovered—and would have never discovered without access to these documents. This information has turned our understanding of the drug's effects on its head. Other drugs for which previously unpublished, detailed clinical trial data have radically changed public knowledge of safety and efficacy include Avandia, Neurontin, and Vioxx (Table 1).
Box 1. What Is Missed without Access to Tamiflu Clinical Study Reports
- Knowledge of the total denominator. (How many trials were conducted on this drug that might fit the systematic review inclusion criteria?) 
- Realization that serious adverse events (SAEs) occurred in trials for which SAEs were not reported in published papers .
- Understanding what happened in some trials that were published 10 years after completion .
- Vital details of trials (content and toxicity profile of placebos, mode of action of drug, description and temporality of adverse events) .
- Authorship is not consistent with published papers  (although, if a review's inclusion criteria include clinical study reports, authorship is not an issue, as the responsibility is clearly the manufacturer's).
- Rationale for alternatively classifying outcomes such as pneumonia as a complication or an adverse event .
- Ability to know whether key subgroup analysis (influenza-infected subjects) is valid .
- Assessment of validity of previously released information on the drug (articles, reviews, conferences, media, etc.).
- Realization that Roche's claim of Tamiflu's mode of action  appears inconsistent with the evidence from trials ,.
An Urgent Call for a Debate on the Ethics of Data Secrecy
Taken together, these experiences suggest that any attempt at reliable evidence synthesis must begin with clinical study reports. Yet we are aware of only three other groups of independent researchers conducting a systematic review based entirely (or mostly) on these and other regulatory documents –. We can think of two major reasons this might be. First, outside of regulatory agencies, few researchers have ever heard of clinical study reports. Second, clinical study reports are massive in size and difficult to obtain, traditionally shared only with regulators. The first problem seems tractable, but gaining better access to manufacturers' clinical study reports requires shifting the status quo from a default position of confidentiality to one of disclosure.
In the mid-20th century, regulatory agencies became increasingly responsible to the public for ensuring the safety and effectiveness of approved medicines. This rise paralleled an increase in the number and complexity of clinical trials. Both in the United States and Europe, manufacturers would come to submit trial data to regulators with the assurance that authorities would treat all trade secret data as confidential. Even following passage of the 1966 Freedom of Information Act (FOIA), the FDA maintained that safety and effectiveness test data were not subject to the FOIA ,. One sign things may be changing came in November 2010, when the EMA announced its intention to make all industry clinical study reports about a drug publicly available as a matter of standard practice after reaching a regulatory decision ,. The EMA has also already improved its handling of data under Freedom of Information requests. But as we learned was true for Tamiflu, regulators may not be in possession of full trial reports of all studies on a given intervention, implying the necessity of obtaining reports from manufacturers . But at present, industry seems extremely reluctant to make its clinical study reports freely available.
In addition to clinical study reports, we also need access to regulatory information. By the very nature of their professional mandate, regulators may conduct some of the most thorough evaluations of a trial program, and efforts like ours aimed at up-to-date evidence synthesis are seriously deprived without access to their reports. While the FDA has increasingly published its reviews, memos, and other correspondence on its Drugs@FDA website (http://www.accessdata.fda.gov/scripts/cder/drugsatfda/), and additional documents are accessible through FOIA requests, the process can be very time-consuming, taking months or even years. Moreover, regulatory agencies' lack of public inventories of their documentary holdings complicates the retrieval of information. Ideally, we would also have details of the regulators' deliberations, which can serve as signposts to important issues that need investigating.
In December 2009, after we voiced serious concerns in the BMJ about Tamiflu's alleged ability to reduce complications , Roche wrote that it was “very happy to have its data reviewed by appropriate authorities or individuals,” and publicly pledged to release “full study reports” for ten trials “within the coming days” . But despite extensive correspondence over the next year and a half, Roche refused to provide any more than portions of the clinical study reports for the ten Kaiser studies (Table 2) and no reports for any of the additional Tamiflu trials we had subsequently identified and requested (Table 3). Reasons for refusing to share the full reports on Tamiflu kept changing, and none seemed credible (Tables 2 and 3).
There are strong ethical arguments for ensuring that all clinical study reports are publicly accessible. It is the public who take and pay for approved drugs, and therefore the public should have access to complete information about those drugs. We should also not lose sight of the fact that clinical trials are experiments conducted on humans that carry an assumption of contributing to medical knowledge. Non-disclosure of complete trial results undermines the philanthropy of human participants and sets back the pursuit of knowledge.
Potentially valid reasons for restricting the full public release of clinical study reports include: ensuring patient confidentiality (although this could be remedied with redaction); commercial secrets (although these should be clear after drugs have been registered, and we found no commercially sensitive information in the clinical study reports received from EMA with minimal redactions); and finally, industry concerns over adversaries' malicious “cherry picking” over large datasets (which could be reduced by requiring the prospective registration of research protocols). None appear insurmountable.
With the EMA's stated intentions on far wider data disclosure, we hope the debate may soon shift from one of whether to release regulatory data to the specifics of doing so. But until these policies go into effect—and perhaps even after they do—most drugs on the market will remain those approved in an era in which regulators protected industry's data . It is therefore vital to know where industry stands. If drug companies have legitimate reasons for maintaining the status quo of treating all of their data as trade secret, we have yet to hear them. We are all ears.
Translation of the Summary Points into Russian by Vasiliy Vlassov.
Translation of the Summary Points into Italian by Tom Jefferson.
Translation of the Summary Points into Japanese by Yuko Hara and Yasuyuki Kato.
Translation of the Summary Points into Croatian by Mirjana Huić.
Translation of the Summary Points into Danish by Andreas Lundh.
Translation of the Summary Points into German by Gerd Antes.
Translation of the Summary Points into Spanish by Juan Andres Leon.
We thank our co-authors of the Cochrane review of neuraminidase inhibitors. We also thank Yuko Hara for helpful comments on the draft manuscript and Ted Postol for many valuable insights. We are also grateful to those who helped with translations.
Analyzed the data: PD TJ CDM. Wrote the first draft of the manuscript: PD TJ CDM. Contributed to the writing of the manuscript: PD TJ CDM. ICMJE criteria for authorship read and met: PD TJ CDM. Agree with manuscript results and conclusions: PD TJ CDM.
- 1. Brody H, Light DW (2011) The inverse benefit law: how drug marketing undermines patient safety and public health. Am J Public Health 101(3): 399–404.
- 2. Smith R (2005) Medical journals are an extension of the marketing arm of pharmaceutical companies. Plos Med 2(5): e138.
- 3. The New York Times (28 April 2009) Tamiflu (drug). The New York Times. Available: http://topics.nytimes.com/topics/news/health/diseasesconditionsandhealthtopics/tamiflu-drug/index.html. Accessed 6 March 2012.
- 4. U.S. Department of Health and Human Services (2005) HHS pandemic influenza plan. Available: http://www.hhs.gov/pandemicflu/plan/pdf/HHSPandemicInfluenzaPlan.pdf. Accessed 6 March 2012.
- 5. Harper SA, Fukuda K, Uyeki TM, Cox NJ, Bridges CB (2004) Prevention and control of influenza: recommendations of the Advisory Committee on Immunization Practices (ACIP). MMWR Recomm Rep 53(RR-6): 1–40.
- 6. Roche Products Pty Limited (2011) TAMIFLU® capsules: consumer medicine information. Available: http://www.roche-australia.com/fmfiles/re7229005/downloads/anti-virals/tamiflu-cmi.pdf. Accessed 6 March 2012.
- 7. European Medicines Agency (2011) Summary of product characteristics (Tamiflu 30 mg hard capsule). Available: http://www.ema.europa.eu/docs/en_GB/document_library/EPAR_-_Product_Information/human/000402/WC500033106.pdf. Accessed 5 February 2012.
- 8. Kaiser L, Wat C, Mills T, Mahoney P, Ward P, Hayden F (2003) Impact of oseltamivir treatment on influenza-related lower respiratory tract complications and hospitalizations. Arch Intern Med 163(14): 1667–1672.
- 9. Hoffman-La Roche, Ltd. (2011) Product label. Tamiflu (oseltamivir phosphate) capsules and for oral suspension. Available: http://www.accessdata.fda.gov/drugsatfda_docs/label/2011/021087s056_021246s039lbl.pdf. Accessed 6 March 2012.
- 10. U.S. Food and Drug Administration (14 April 2000) NDA 21-087 TAMIFLU (oseltamivir phosphate) MACMIS ID#8675. Available: http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/EnforcementActivitiesbyFDA/WarningLettersandNoticeofViolationLetterstoPharmaceuticalCompanies/UCM166329.pdf. Accessed 6 March 2012.
- 11. Jefferson T, Jones MA, Doshi P, Del Mar CB, Heneghan CJ, et al. (2012) Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database Syst Rev 2012(1): CD008965.
- 12. World Health Organization (2007) WHO interim protocol: rapid operations to contain the initial emergence of pandemic influenza. Available: http://www.who.int/influenza/resources/documents/RapidContProtOct15.pdf. Accessed 6 March 2012.
- 13. Jefferson T, Doshi P, Thompson M, Heneghan C (2011) Ensuring safe and effective drugs: who can do what it takes? BMJ 342: c7258.
- 14. International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use (1996) Guideline for industry: structure and content of Clinical Study Reports (ICH E3). Available: http://www.fda.gov/downloads/regulatoryinformation/guidances/ucm129456.pdf. Accessed 6 2012.
- 15. Lundh A, Barbateskovic M, Hróbjartsson A, Gøtzsche PC (2010) Conflicts of interest at medical journals: the influence of industry-supported randomised trials on journal impact factors and revenue – cohort study. PLoS Med 7(10): e1000354.
- 16. Jefferson T, Jones MA, Doshi P, Del Mar CB, Heneghan CJ, et al. (2011) Cochrane Database Syst Rev Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children - a review of unpublished data. 2011: CD008965. Available: http://doi.wiley.com/10.1002/14651858.
- 17. Gøtzsche PC, Jørgensen AW (2011) Opening up data at the European Medicines Agency. BMJ 342: d2686.
- 18. Yale School of Medicine (2011) Yale University Open Data Access Project (YODA Project). Available: http://medicine.yale.edu/core/projects/yodap/index.aspx. Accessed 6 March 2012.
- 19. Eyding D, Lelgemann M, Grouven U, Harter M, Kromp M, et al. (2010) Reboxetine for acute treatment of major depression: systematic review and meta-analysis of published and unpublished placebo and selective serotonin reuptake inhibitor controlled trials. BMJ 341: c4737.
- 20. Halperin RM (1979) FDA disclosure of safety and effectiveness data: a legal and policy analysis. Duke Law Journal 1979(1): 286–326.
- 21. Kesselheim AS, Mello MM (2007) Confidentiality laws and secrecy in medical research: improving public access to data on drug safety. Health Aff (Millwood) 26(2): 483–491.
- 22. European Medicines Agency (2010) Output of the European Medicines Agency policy on access to documents related to medicinal products for human and veterinary use. Available: http://www.ema.europa.eu/docs/en_GB/document_library/Regulatory_and_procedural_guideline/2010/11/WC500099472.pdf. Accessed 6 March 2012.
- 23. Pott A (2011) EMA's response to articles. BMJ 342: d3838.
- 24. Jefferson T, Jones M, Doshi P, Del Mar C (2009) Neuraminidase inhibitors for preventing and treating influenza in healthy adults: systematic review and meta-analysis. BMJ 339: b5106.
- 25. Smith J, on behalf of Roche (2009) Point-by-point response from Roche to BMJ questions. BMJ 339: b5374.
- 26. Abraham J (1995) Science, politics and the pharmaceutical industry: controversy and bias in drug regulation. First edition. Routledge.
- 27. Cohen D (2010) Rosiglitazone: what went wrong? BMJ 341: c4848.
- 28. Rosen CJ (2007) The rosiglitazone story—lessons from an FDA advisory committee meeting. N Engl J Med 357(9): 844–846.
- 29. Richter B, Bandeira-Echtler E, Bergerhoff K, Clar C, Ebrahim SH (2007) Rosiglitazone for type 2 diabetes mellitus. Cochrane Database Syst Rev 2007(3): CD006063.
- 30. Nissen SE, Wolski K (2007) Effect of rosiglitazone on the risk of myocardial infarction and death from cardiovascular causes. N Engl J Med 356(24): 2457–2471.
- 31. Lenzer J (2003) Whistleblower charges drug company with deceptive practices. BMJ 326(7390): 620.
- 32. Kesselheim AS, Mello MM, Studdert DM (2011) Strategies and practices in off-label marketing of pharmaceuticals: a retrospective analysis of whistleblower complaints. PLoS Med 8: e1000431.
- 33. Steinman MA, Bero LA, Chren M-M, Landefeld CS (2006) Narrative review: the promotion of gabapentin: an analysis of internal industry documents. Ann Intern Med 145(4): 284–293.
- 34. Lenzer J (2004) Pfizer pleads guilty, but drug sales continue to soar. BMJ 328(7450): 1217.
- 35. Steinman MA, Harper GM, Chren M-M, Landefeld CS, Bero LA (2007) Characteristics and impact of drug detailing for gabapentin. PLoS Med 4(4): e134.
- 36. Bombardier C, Laine L, Reicin A, Shapiro D, Burgos-Vargas R, et al. (2000) Comparison of upper gastrointestinal toxicity of rofecoxib and naproxen in patients with rheumatoid arthritis. N Engl J Med 343(21): 1520–1528.
- 37. Curfman GD, Morrissey S, Drazen JM (2005) Expression of concern: Bombardier et al., “Comparison of upper gastrointestinal toxicity of rofecoxib and naproxen in patients with rheumatoid arthritis,” N Engl J Med 2000; 343:1520–8. N Engl J Med 353(26): 2813–2814.
- 38. Wilson D (22 November 2011) Merck to pay $950 million over Vioxx]. The New York Times. Available: https://www.nytimes.com/2011/11/23/business/merck-agrees-to-pay-950-million-in-vioxx-case.html. Accessed 6 March 2012.
- 39. Topol EJ (2004) Failing the public health–rofecoxib, Merck, and the FDA. N Engl J Med 351(17): 1707–1709.
- 40. Doshi P (2009) Neuraminidase inhibitors–the story behind the Cochrane review. BMJ 339: b5164.
- 41. Godlee F (2009) We want raw data, now. BMJ 339: b5405.
- 42. Melville NA (20 January 2012) Tamiflu data called inconsistent, underreported. Available: http://www.medscape.com/viewarticle/757226 [requires free registration].
- 43. Dutkowski R, Smith JR, Davies BE (2010) Safety and pharmacokinetics of oseltamivir at standard and high dosages. Int J Antimicrob Agents 35(5): 461–467.
- 44. Cohen D (2009) Complications: tracking down the data on oseltamivir. BMJ 339: b5387.
- 45. Welliver R, Monto AS, Carewicz O, Schatteman E, Hassman M, et al. (2001) Effectiveness of oseltamivir in preventing influenza in household contacts: a randomized controlled trial. JAMA 285(6): 748–754.
- 46. Gilead Sciences (1999) Roche announces new data on recently approved Tamiflu™, first pill to treat the most common strains of influenza (A & B). Available: http://www.gilead.com/pr_942082531. Accessed 6 March 2012.