KL codesigned the study; analyzed the data; cowrote the first draft; contributed to the writing of the paper; and reviewed final paper. PB helped design the study; helped analyze the data; contributed to the writing of the paper; and reviewed final paper. IS originated the idea for study, designed the study; helped analyze the data; co-wrote the first draft; contributed to the writing of the paper; and reviewed final paper.
The authors have declared that no competing interests exist.
The United States (US) Food and Drug Administration (FDA) approves new drugs based on sponsor-submitted clinical trials. The publication status of these trials in the medical literature and factors associated with publication have not been evaluated. We sought to determine the proportion of trials submitted to the FDA in support of newly approved drugs that are published in biomedical journals that a typical clinician, consumer, or policy maker living in the US would reasonably search.
We conducted a cohort study of trials supporting new drugs approved between 1998 and 2000, as described in FDA medical and statistical review documents and the FDA approved drug label. We determined publication status and time from approval to full publication in the medical literature at 2 and 5 y by searching PubMed and other databases through 01 August 2006. We then evaluated trial characteristics associated with publication. We identified 909 trials supporting 90 approved drugs in the FDA reviews, of which 43% (394/909) were published. Among the subset of trials described in the FDA-approved drug label and classified as “pivotal trials” for our analysis, 76% (257/340) were published. In multivariable logistic regression for all trials 5 y postapproval, likelihood of publication correlated with statistically significant results (odds ratio [OR] 3.03, 95% confidence interval [CI] 1.78–5.17); larger sample sizes (OR 1.33 per 2-fold increase in sample size, 95% CI 1.17–1.52); and pivotal status (OR 5.31, 95% CI 3.30–8.55). In multivariable logistic regression for only the pivotal trials 5 y postapproval, likelihood of publication correlated with statistically significant results (OR 2.96, 95% CI 1.24–7.06) and larger sample sizes (OR 1.47 per 2-fold increase in sample size, 95% CI 1.15–1.88). Statistically significant results and larger sample sizes were also predictive of publication at 2 y postapproval and in multivariable Cox proportional models for all trials and the subset of pivotal trials.
Over half of all supporting trials for FDA-approved drugs remained unpublished ≥ 5 y after approval. Pivotal trials and trials with statistically significant results and larger sample sizes are more likely to be published. Selective reporting of trial results exists for commonly marketed drugs. Our data provide a baseline for evaluating publication bias as the new FDA Amendments Act comes into force mandating basic results reporting of clinical trials.
Ida Sim and colleagues investigate the publication status and publication bias of trials submitted to the US Food and Drug Administration (FDA) for a wide variety of approved drugs.
Before a new drug becomes available for the treatment of a specific human disease, its benefits and harms are carefully studied, first in the laboratory and in animals, and then in several types of clinical trials. In the most important of these trials—so-called “pivotal” clinical trials—the efficacy and safety of the new drug and of a standard treatment are compared by giving groups of patients the different treatments and measuring several predefined “outcomes.” These outcomes indicate whether the new drug is more effective than the standard treatment and whether it has any other effects on the patients' health and daily life. All this information is then submitted by the sponsor of the new drug (usually a pharmaceutical company) to the government body responsible for drug approval—in the US, this is the Food and Drug Administration (FDA).
After a drug receives FDA approval, information about the clinical trials supporting the FDA's decision are included in the FDA “Summary Basis of Approval” and/or on the drug label. In addition, some clinical trials are described in medical journals. Ideally, all the clinical information that leads to a drug's approval should be publicly available to help clinicians make informed decisions about how to treat their patients. A full-length publication in a medical journal is the primary way that clinical trial results are communicated to the scientific community and the public. Unfortunately, drug sponsors sometimes publish the results only of trials where their drug performed well; as a consequence, trials where the drug did no better than the standard treatment or where it had unwanted side effects remain unpublished. Publication bias like this provides an inaccurate picture of a drug's efficacy and safety relative to other therapies and may lead to excessive prescribing of newer, more expensive (but not necessarily more effective) treatments. In this study, the researchers investigate whether selective trial reporting is common by evaluating the publication status of trials submitted to the FDA for a wide variety of approved drugs. They also ask which factors affect a trial's chances of publication.
The researchers identified 90 drugs approved by the FDA between 1998 and 2000 by searching the FDA's Center for Drug Evaluation and Research Web site. From the Summary Basis of Approval for each drug, they identified 909 clinical trials undertaken to support these approvals. They then searched the published medical literature up to mid-2006 to determine if and when the results of each trial were published. Although 76% of the pivotal trials had appeared in medical journals, usually within 3 years of FDA approval, only 43% of all of the submitted trials had been published. Among all the trials, those with statistically significant results were nearly twice as likely to have been published as those without statistically significant results, and pivotal trials were three times more likely to have been published as nonpivotal trials, 5 years postapproval. In addition, a larger sample size increased the likelihood of publication. Having statistically significant results and larger sample sizes also increased the likelihood of publication of the pivotal trials.
Although the search methods used in this study may have missed some publications, these findings suggest that more than half the clinical trials undertaken to support drug approval remain unpublished 5 years or more after FDA approval. They also reveal selective reporting of results. For example, they show that a pivotal trial in which the new drug does no better than an old drug is less likely to be published than one where the new drug is more effective, a publication bias that could establish an inappropriately favorable record for the new drug in the medical literature. Importantly, these findings provide a baseline for monitoring the effects of the FDA Amendments Act 2007, which was introduced to improve the accuracy and completeness of drug trial reporting. Under this Act, all trials supporting FDA-approved drugs must be registered when they start, and the summary results of all the outcomes declared at trial registration as well as specific details about the trial protocol must be publicly posted within a year of drug approval on the US National Institutes of Health clinical trials site.
Please access these Web sites via the online version of this summary at
The US
The World Health Organization's
In the United States, the Food and Drug Administration (FDA) approves new drug products for
sale and marketing based on results from clinical investigations that demonstrate the safety
and efficacy of a drug for a proposed indication. Sponsors of a drug (e.g., companies,
research institutions, or government) seek approval by submitting a new drug application
(NDA) [
For drugs that receive FDA approval, public disclosure of trial results may occur through a
variety of sources. The FDA discloses a Summary Basis of Approval document that contains
summaries and evaluations of clinical data and statistical analyses performed by FDA medical
officers during the approval process [
A string of recent controversies concerning the suppression of safety risks of
rosiglitazone [
In response to these concerns, the US recently mandated in the FDA Amendments Act 2007 (Public Law 110–85) that all trials supporting FDA-approved drugs and devices must be registered at inception and have their “basic results” publicly posted by the National Institutes of Health. The basic results to be disclosed include the demographics of the study participants, the number of participants who dropped out or were excluded from analysis, and the numeric and statistical test results of all primary and secondary outcomes declared at initial trial registration.
For the foreseeable future, however, the detailed information needed for full appraisal of
a trial's evidence is likely to be available only in journal publications. This information
includes protocol, protocol deviation, and conflicts of interest information, as well as
additional analyses beyond the primary and secondary outcomes. The availability of basic
results on ClinicalTrials.gov (
Previous research has documented the problem of publication bias and incomplete or
selective reporting of trials submitted to licensing authorities in Sweden [
We identified all drugs approved by the FDA between January 1998 and December 2000 at the
Center for Drug Evaluation and Research Web site, available at
Phase I trials are often small studies designed to provide supporting information about a
drug's pharmacokinetic parameters, dosing schedule, common side effects, tolerability, and
toxicity, but are limited by design or other factors in their ability to demonstrate
efficacy. Phase II and III trials are often larger studies designed to provide evidence on
the overall risks and benefits of a drug. The phase of a trial was often not reported in
the FDA documents. Sponsors and the FDA frequently categorize certain trials as
“pivotal.” These are trials that demonstrate the efficacy and safety
of a drug for its proposed indication and provide the most useful information for clinical
decision-making. Pivotal trials are typically Phase II or III trials, but there is no
formal definition of a pivotal trial. In practice, trials that are reported in the
“clinical studies” or “clinical efficacy” section
of the FDA-approved drug label are considered pivotal. We used this scheme to categorize
trials as “pivotal” or “nonpivotal.” We obtained
the product label at the time of FDA approval for each new drug, or the next available
product label if the initial product label was not available, at
For each submitted trial, we recorded the following characteristics when available in the
FDA documents: drug name (generic and trade), the number and location of study sites, the
name of the principal investigator, the number of study participants, dosage and
evaluation schedules, sample size, statistical significance of the primary outcome
(
We systematically searched common databases of biomedical journals that a typical
clinician, consumer, or policy maker living in the US would reasonably search. These
databases were PubMed, Cochrane Library, and the Cumulative Index for Nursing and Allied
Health Literature (CINAHL)
Only original research reports in full journal articles were counted as matching
publications; abstracts or review articles were not considered matches, as these types of
articles by definition contain incomplete descriptions of a trial's methods and results.
For remaining trials that were not matched to a publication, we searched PubMed again
without publication type limits using a variety of keywords (e.g., generic drug name;
names of other drugs in the trial; disease/condition studied; outcomes measured; and trial
characteristics such as “cross over”, “randomize”,
“blind”, “washout”, “placebo”,
“pharmacokinetics”, and “bioavailability”). If
trials remained unmatched to a publication in PubMed after this more comprehensive search,
we searched the entire Cochrane Library and CINAHL databases without limits using the
similar keyword strategy described above. We also reviewed
The main outcome measures were time from FDA approval to publication of a full report, and whether a report was published by 2 or 5 y after approval. We analyzed publication at 2 y because pending Congressional legislation is considering mandating results reporting by 2 y after drug approval. Trials that were not published were censored as of 01 August 2006.
To control for multiple variables simultaneously, we carried out multivariate mixed
effects logistic regression analysis and calculated odds ratios at 2 and 5 y after
approval. All models were adjusted for clustering by drug (treated as a random effect).
Predictors assessed in both univariate and multivariable analyses included statistical
significance of the primary results, double blinding, randomization, sample size
(dichotomized at the median size of ≤ 135 or > 135, or log-transformed to
each 2-fold increase in sample size), study type (pivotal or not), and company size.
Companies with annual revenues greater than $3 billion, and/or annual research and
development expenditures greater than $500 million in 2004, were classified as large
companies, and generally represented the top 30 pharmaceutical and biotechnology companies
in the world [
Our primary analysis was logistic regression analyses on all supporting trials
(
We identified 90 FDA-approved new drugs between January 1998 and December 2000.
Eighty-nine (99%) of the applications were submitted by a pharmaceutical
company; one application was submitted by the US Army Medical Research and Material
Command. Eighty-eight drugs were available by prescription only and two had
over-the-counter marketing status. Seven prescription drug products were discontinued
after initial FDA approval. We were able to identify a total of 909 trials with sufficient
description in the FDA review documents supporting these 90 new drugs.
Characteristics and Publication Rates of Trials Submitted for FDA Approval in 1998–2000
1Clinical trials that are adequately designed to demonstrate efficacy of the drug for a proposed indication and reported in the “clinical studies” or “clinical efficacy” section of the FDA approved drug label.
2Time to publication in years counting from the month of FDA approval.
In univariate analyses of all supporting trials, trials with statistically significant
results, larger sample sizes, double blinding, randomization, and trials that were pivotal
were more likely to be published by 2 and 5 y after FDA approval (
Characteristics Associated with Publication of Trials Submitted for FDA Approval in 1998–2000: Univariate Logistic Regressiona
Characteristics Associated with Publication of Trials Submitted for FDA Approval in 1998–2000: Multivariable Logistic Regressiona
Trials from less than half of the cohort (43%) were published. Of the trials that were published, 92% were published within 3 y of FDA approval. Trials could be published prior to or following submission of data to the FDA.
Of the 909 trials, 340 (37%) were identified as pivotal, of which 257
(76%) were published (
Of the pivotal trials that were published, 95% were published within 3 y of FDA approval. Trials could be published prior to or following submission of data to the FDA.
Our study evaluated the publication of 909 clinical trials identified in FDA medical and
statistical review documents in support of 90 new drug products approved between 1998 and
2000. We found that after a minimum of 5.5 y of follow-up after FDA approval, we identified
publications from 43% of the trials in the medical literature. For pivotal
trials, which are more clinically informative than nonpivotal trials, we found publications
from 76% of the trials. For one of the 90 approved new drugs, we could not find
any published supporting trial. We also found strong evidence of publication bias: trials
with statistically significant results were more likely to be published than trials with
nonsignificant results, as were trials with larger sample sizes. There was a weak suggestion
that the effect of sample size might be less among trials with statistically significant
findings, but
We also found the reporting of clinical trials in the FDA review documents and drug labels
to be variable in detail and content, and not an adequate substitute for full publication in
the medical literature. For example, reporting ranged from detailed descriptions of a
trial's study design, intervention, patient population, statistical analyses, adverse
events, primary outcomes, and other results, to brief statements that only summarized a
trial's primary outcome. We also noted sections of redacted information in the FDA review
documents. Neither the FDA review documents nor the drug labels followed a standard format
for reporting a trial's methodology and results. Use of guidelines such as the revised
CONSORT (Consolidated Standards of Reporting Trials) [
Our study has several limitations. First, we may have misclassified some published trials as being unpublished because of difficulties in matching publications to incomplete trial descriptions in the FDA documents. Also, we did not search other databases such as the European EMBASE, nor did we contact investigators or sponsors to determine publication status or verify that a trial was not published or in press. Thus, we are likely to have underestimated the overall publication rate of these trials. However, we believe that for clinicians and policy makers, the most relevant publication rate is not the overall rate but the publication rate in journals that a typical clinician, consumer, or policy maker would have access to through a reasonable literature search. We believe our searches of PubMed, the Cochrane Library, and CINAHL reflect such a reasonable search. It would not be reasonable to expect a clinician, consumer, or policy maker to contact investigators or sponsors to determine a trial's publication status.
A second limitation of our study is our follow-up time of 5.5 to 8.5 y after new drug
approval may be inadequate. However, we found that publications occurred almost exclusively
within the first 3 y after approval, making it unlikely that longer follow-up would yield
many additional publications. Third, time-to-publication is ideally counted from the date of
trial completion, but we were unable to obtain these dates reliably. Moreover, we believe
the month of approval is the most relevant time point when trial results should be available
to the public. Fourth, our study focused on publications in the medical literature, but some
companies have started making their trial results publicly available directly on their own
Web sites. For example, the pharmaceutical industry's Clinical Study Results Database
contains summaries of “hypothesis-testing” trials completed since
October 2002 for many pharmaceutical products [
Despite these limitations, our study provides ample evidence that in the years immediately
following FDA approval that are most relevant to public health, there exists incomplete and
selective publication of trials supporting approved new drugs. Potential reasons for this
publication bias may include the tendency of investigators and sponsors to delay or not
submit trial reports [
As discussed above, the FDA Amendments Act of 2007 mandates basic public results
reporting for all trials supporting FDA-approved drugs and devices. Our study shows that
this legislation was necessary because current reporting is marked by pervasive
publication bias of positive over negative trials. Moreover, because published trial
reports are often incomplete [
We anticipate that the new law will also speed the dissemination of trial information.
Currently, according to our data, 40% of the trials that were eventually
published were published more than 1 y postapproval (34% of pivotal trials).
Under the new law, basic results for all trials must be posted by 1 y after trial
completion or approval of the drug or device. This suggests that for all trials that the
sponsor wishes to publish, the manuscripts will have to be submitted for peer review
before the 1 y postapproval mark if they hope to allay journal concerns about publishing
trials whose primary and secondary outcome results have already been publicly posted.
Thus, we would expect the time-to-publication curves in
Paradoxically, however, this new law may increase rather than decrease publication bias.
Might sponsors feel less compelled to publish equivocal trials because the basic results
will already be in the public domain? Might the time pressure to submit manuscripts by 1 y
postapproval focus sponsor efforts even more on submitting positive trials and trials of
greatest interest to journals? Might the journals, if they accept manuscripts of trials
with publicly posted results, change the criteria by which publication importance is
judged, and how might this affect acceptance rates [
Number of supporting trials and the proportion published, and the number of pivotal trials and the proportion published, for each of the 90 drugs analyzed.
(215 KB DOC)
We thank Sophia Jeng, MS and James Hamrick, MD for study design, data collection, and data analysis assistance; Chau Ong, PharmD and Roop Prabhu for assistance with data collection; and Erika Campbell for assistance with obtaining drug labels.
confidence interval
Cumulative Index for Nursing and Allied Health Literature
Food and Drug Administration
new drug application
odds ratio
United States