Timing and Completeness of Trial Results Posted at ClinicalTrials.gov and Published in Journals

Agnes Dechartres and colleagues searched ClinicalTrials.gov for completed drug RCTs with results reported and then searched for corresponding studies in PubMed to evaluate timeliness and completeness of reporting. Please see later in the article for the Editors' Summary


Introduction
Without accessible and usable reports, research fails to help patients and their clinicians [1]. Over the past decades, underreporting of trial results has been increasingly acknowledged as one of the main causes of waste of research [2][3][4][5], contributing to biased evidence, with serious consequences for clinical practice, research, and, ultimately, patients [1]. This waste of research can occur at different stages: (1) failure to publish results of some studies, particularly those with negative results, ''publication bias'' [6][7][8]; (2) delay in publishing results of negative studies [9], ''timelag bias'' [10]; and (3) failure to publish complete results for all prespecified outcomes, ''reporting bias'' [10][11][12][13][14]. Among published studies, some results may be incompletely reported and therefore cannot be included in a meta-analysis. This is the case, for example, when the difference in means between treatments is reported but not a measure of precision.
To overcome these issues, the 2007 US Food and Drug Administration Amendments Act (FDAAA) requires that the results from clinical trials of Food and Drug Administrationapproved drugs and devices conducted in the United States must be made publicly available at ClinicalTrials.gov within 1 y of the completion of the trial, whether the results are published or not [15][16][17][18]. This US public law requires a ''table of the demographic and baseline data collected overall and for each arm of the clinical trial to describe the patients who participated in the clinical trial … [and] a table of values for each of the primary and secondary outcome measures for each arm of the clinical trial, including the results of scientifically appropriate tests of the statistical significance of such outcome measures.'' Researchers of other trials registered in ClinicalTrials.gov are welcome to post trial results as well. In our study, we aimed to compare the timing and completeness (i.e., whether all relevant information was fully reported) of results publicly posted at ClinicalTrials.gov and in published articles for trials of drug interventions.

Methods
We identified trials with results posted on ClinicalTrials.gov and their corresponding full-text publications in journals.

Search for Trials with Results Posted at ClinicalTrials.gov
We searched ClinicalTrials.gov on March 27, 2012, using the following keywords: ''Closed study'' in the Recruitment field, ''With results'' for Study Results, ''Interventional studies'' for Study Type, and ''Phase III and IV'' for Phase. We selected completed phase III or IV randomized controlled trials listing only drugs as intervention type. We excluded trials comparing a drug to a device. We excluded phase II/III trials, considering them to be phase II trials. Of all eligible trials (n = 1,592), we selected a random sample of 600 trials for which to search for full-text publications.

Search for Publication of Results in Journals
Whenever possible, we used the link within ClinicalTrials.gov to identify the published article. We also systematically searched MEDLINE via PubMed by using the ClinicalTrials.gov identification number (NCT number). If no publication was identified, we searched MEDLINE via PubMed again by using keywords for drug names and the condition studied. The articles identified through the search had to match the corresponding trial in terms of the information registered at ClinicalTrials.gov (i.e., same objective, same sample size, same primary outcome, same location, same responsible party, same trial phase, and same sponsor) and had to present results for the primary outcome. A second reviewer checked the matching between ClinicalTrials.gov and the published article. All disagreements were resolved by discussion between the two reviewers.
In order to compare the reporting between the ClinicalTrials. gov report and the published article, we excluded trials for which results were published in a journal that could not be retrieved or were published in a language other than English, French, or Spanish. We also excluded single-arm studies because they lacked a control group, as well as studies with four or more arms, for practical reasons.

Data Extraction
We collected the following information from ClinicalTrials.gov for the random sample of 600 trials with results posted at ClinicalTrials.gov: 1. General characteristics of the trial: primary funding sources (extracted from Study Sponsor at ClinicalTrials.gov), medical specialty (extracted from Conditions at ClinicalTrials.gov), and countries where the trial was conducted (extracted from Location Countries at ClinicalTrials.gov). We also collected trial primary completion date (defined as the date of final collection of data for the primary outcome) and date when results were first publicly posted-this date was extracted from the archive record and differs from the date on which results were first received, which is available under Study Results at ClinicalTrials.gov. The difference between these two dates is related to production and the vetting of the results by the US National Institutes of Health. 2. Design of the trial: we noted whether the trial was a phase III or IV trial. We recorded whether it was a parallel or crossover trial. 3. Interventions: details concerning interventions for experimental and control groups were extracted from Study Arm(s) at ClinicalTrials.gov.
Then, for all trials with both results posted and published, we collected the following information independently from Clinical Trials.gov and from the published article: 1. Flow of participants in the trial: reporting of the flow of participants, including number of participants assessed for eligibility, number of participants randomized overall and per arm, number of participants who received the intervention per arm and whether the reasons for not receiving the intervention were specified, number of patients lost to follow-up and those who discontinued the intervention and whether the reasons for discontinuation were given, and number of participants analyzed per arm and whether the reasons for excluding participants from the analysis were reported. 2. Efficacy results: for the primary outcome posted at ClinicalTrials.gov, we extracted data from ClinicalTrials.gov for this outcome. If several primary outcomes were posted at ClinicalTrials.gov, we focused on the first registered. When we extracted efficacy results in the published article, we extracted data for this outcome whether it was reported as primary or secondary. If the outcome was not reported at all, we considered the data to be missing. For all types of outcomes, we collected whether numbers of patients analyzed per arm were reported. For binary outcomes, we collected whether the number of events per arm was reported. For continuous outcomes, we collected whether (1) mean (6 standard deviation [SD]) or (2) median (interquartile) was reported or (3) neither of these was reported. For time-to-event outcomes, we collected whether the results of the log-rank test or Cox proportional hazard model were reported. 3. Adverse events: we noted whether adverse events were reported and whether they were reported by number per arm. We collected whether all adverse events were reported or only common events were reported or those with statistically significant differences between arms. We noted whether reporting of adverse events concerned all randomized participants or only those who received at least one treatment dose. We also collected whether withdrawals due to adverse events were reported. 4. Serious adverse events: we collected whether serious adverse events were reported, were reported per arm, and were reported with numerical data.
All data were extracted in duplicate by two reviewers in data collection forms. All disagreements were resolved by discussion to reach a consensus, including intervention of a third reviewer in case of discrepancies.
We also extracted the characteristics of the publication (i.e., the journal in which the article was published, the date of online publication, the type of journal [general medical, medical specialty], and whether the NCT number was reported in the published article).

Assessment of the Completeness of Reporting
Three experts in clinical epidemiology reached consensus on the main elements that needed to be reported for each of the following domains: the flow of participants during the trial, efficacy results, adverse events, and serious adverse events, on the basis of data required to perform meta-analyses. The reporting was considered complete for each domain if all of the included elements in Box 1 were reported and incomplete if one or more elements were missing from the items listed in Box 1.
In a second step, we compared the number of elements reported at ClinicalTrials.gov and in the published article for the flow of participants, efficacy results, adverse events, and serious adverse events. We assessed the number of pairs (with percentage) for which ClinicalTrials.gov provided more information (higher number of elements reported), similar information (same number of elements reported), and less information (lower number of elements reported) as compared with the published article for the flow of participants, efficacy results, adverse events, and serious adverse events as defined above.

Statistical Analyses
Inter-rater agreement between the two reviewers was assessed by the Kappa coefficient (95% CI). Descriptive analyses of trial characteristics included numbers and percentages. Time from trial primary completion date to the date of posting of results at ClinicalTrials.gov or online publication in journals was described with the Kaplan and Meier method for trials with results both posted at ClinicalTrials.gov and published.
We compared results posted at ClinicalTrials.gov and in the published article for completeness of reporting using McNemar's test of equality of paired proportions. We tested the interaction between completeness of reporting at ClinicalTrials.gov and in the published article and the following trial characteristics: type of journal (general versus specialty), type of control (active versus placebo or no treatment), source of funding (academic, industry, both), and study design (parallel arms versus crossover) using a generalized estimation equation (GEE) model to take into account the correlation between the paired observations. All tests were two-tailed, and p,0.05 was considered statistically significant. Analyses were conducted using R version 2.15.1 [19]. Figure 1 describes the selection of trials. Briefly, from the 2,837 trials retrieved by a search of ClinicalTrials.gov on March 27, 2012, we identified 1,592 completed phase III or IV randomized drug trials with results posted at ClinicalTrials.gov. We selected a random sample of 600 trials to search for corresponding publications.

Time to Posting Results at ClinicalTrials.gov and Publication in Journals
Of the 600 trials with results posted at ClinicalTrials.gov that were randomly selected for a search for corresponding publications, we excluded five that were not randomized controlled trials, and one that was still enrolling patients. Mean or median per arm and SD or SE or 95% CI or Q1-Q3 or Effect size (difference in means or standardized mean difference) with 95% CI For time-to-event outcomes: Hazard ratio with 95% CI

Adverse events
Number of adverse events per arm without restriction to statistically significant differences between arms for all randomized participants or for those who received at least one treatment dose

Serious adverse events
Number of serious adverse events per arm preliminary results such as baseline results (n = 6), interim analyses (n = 3), long-term outcomes (n = 15), other outcomes (n = 19), and/ or other results (n = 1). None of the eliminated reports contained additional safety data for the same time frame as the selected report. Two of the eliminated reports contained additional safety results for longer-term follow-up. The median time between primary completion date and results first being publicly posted at ClinicalTrials.gov was 19 mo (Q1 = 14, Q3 = 30 mo); the median time between primary completion date and publication in journals was 21 mo (Q1 = 14, Q3 = 28 mo) ( Figure 2).

Characteristics of Trials with Both Posted and Published Results
Inter-rater agreement between the 2 reviewers was good overall, with median Kappa coefficient 0.80 (range 0.47-0.95) for qualitative variables and 0.98 (0.91-0.99) for quantitative variables.

Reporting of Efficacy Results
For trials with a binary outcome (n = 73), the number of patients analyzed was reported for 97% (71/73) of trials at ClinicalTrials. gov and for 89% (65/73) in the published article ( Table 3). The number of events was reported for 55% (40/73) of trials at ClinicalTrials.gov and for 66% (48/73) in the published article. For trials with a continuous outcome (n = 107), the mean or median was reported for 100% (107/107) of trials at ClinicalTrials.gov and for 90% (96/107) in the published article (Table 3). Dispersion was reported for 96% (103/107) of trials at Clinical Trials.gov and for 64% (69/107) in the published article. For trials with a time-to-event outcome (n = 22), the median time to event was reported for 50% (11/22) of trials at ClinicalTrials.gov and for 41% (9/22) in the published article ( Table 3). The number of events was reported for 32% (7/22) of trials at ClinicalTrials.gov and for 32% (7/22) in the published article. The hazard ratio with 95% CI was reported for 68% (15/22)

Reporting of Adverse Events
For the 202 pairs of trial reports, the population for analysis corresponded to all randomized participants for 57% (115/202) of trials at ClinicalTrials.gov and 36% (72/202) of published articles ( Table 4). The total number of adverse events was reported for 96% (194/202) of trials at ClinicalTrials.gov and for 63% (128/ 202) in the published article. All adverse events per arm were reported for 13% (26/202) of trials at ClinicalTrials.gov and for 5% (10/202) in the published article. Otherwise, reporting was restricted to the most common events for 99% (174/176) of trials at ClinicalTrials.gov and for 44% (85/192) in the published article, or to statistically significant events for 15% (29/192) of trials in the published article. Withdrawals due to adverse events were reported for 80% (161/202) of trials in ClinicalTrials.gov and for 76% (153/202) in the published article. There was some mention of serious adverse events for 99% (200/202) of trials at ClinicalTrials.gov and for 71% (144/202) in the published article, and all serious adverse events were reported per arm for 99% (199/202) and 63% (127/202), respectively.

Completeness of Reporting
For the 202 pairs of trial reports, the proportion of trials with complete reporting was significantly higher at ClinicalTrials.gov than in the published article for the flow of participants ( We found statistically significant interactions between completeness of reporting of adverse events and type of journal (74% and 38%, respectively, in ClinicalTrials.gov and in the published article for trials published in a specialty journal versus 70% and 59%, respectively, for trials published in a general journal, p for interaction = 0.015) as well as source of funding (75% and 5%, respectively, in ClinicalTrials.gov and in the published article for trials with academic funding; 73% and 50%, respectively, for trials with industry funding; and 56% and 33%, respectively, for trials with academic and industry funding, p for interaction = 0.01).

Discussion
To our knowledge, this is the first study comparing the timing and completeness of trial results publicly posted at ClinicalTrials. In particular, serious adverse events were almost always reported at ClinicalTrials.gov.
A previous study assessed trial publication for completed trials registered at ClinicalTrials.gov and showed that fewer than half were published [20]. Other studies evaluated the quality of reporting of the World Health Organization minimum dataset in ClinicalTrials.gov [21]. A recent study published in 2012 [22] compared the quality of reporting among registry reports, clinical study reports submitted to regulatory authorities, and journal publications. The authors identified only industry registry reports, with no trials being registered in a public registry. They concluded that industry registry reports and journal publications insufficiently reported the results of clinical trials but may supplement each other. With the FDAAA requiring mandatory posting of results within 1 y after the primary completion date [15][16][17] and standardized reporting of results [17], Clinical-Trials.gov has become an interesting source of data for assessing trial results.

Implications
Our results have important implications for several stakehold ers: patients and clinicians, authors, researchers performing systematic reviews and meta-analyses, methodologists, peer reviewers, developers of reporting guidelines, and journal editors.
For patients and their clinicians, our results outline the importance of registries to improve transparency in clinical research by making information about clinical trials, including results, publicly available, which is the basis for well-informed decision-making about patients' health.
Our results are important for authors because they point out inconsistencies in reporting and highlight the need for more rigorous adherence to reporting guidelines to ensure that all critical information is provided in study reports.
For researchers performing systematic reviews, our results emphasize the importance of registries [23][24][25][26][27] in reducing publication bias and time-lag bias. Actually, about half of the trials with results posted at ClinicalTrials.gov did not have published results.
Further, our results highlight the need to assess trial results systematically from both ClinicalTrials.gov and the published article when available. Based on our results, searching Clinical Trials.gov is necessary for all published and unpublished trials to obtain more complete data and to identify inconsistencies or discrepancies between the publicly posted results and the publication. As outlined by Zarin et al., ClinicalTrials.gov is designed to complement, not replace, journal publication [28]. Nevertheless, not all trials have their results posted at ClinicalTrials.gov. Some studies previously showed low compliance with the FDAAA regarding mandatory posting of results at ClinicalTrials.gov [29][30][31][32]. Moreover, this law concerns trials performed in the US, with no similar law in Europe or elsewhere.
Our results also highlight the role of trial registries for researchers and methodologists exploring publication bias and selective reporting of outcomes. For example, researchers could use trial registries to assess whether studies with a significant main outcome are more likely to be published or published more quickly than those with a negative outcome using data recorded in registers.
For peer reviewers, our results emphasize the important role of trial registration during the peer-review process. Actually, reviewers and academic editors could assess whether all safety events, especially serious adverse events, are fully reported in the submitted articles. In our study, serious adverse events were reported for 99% of trials at ClinicalTrials. gov but for only 62% in the published article. A study published in 2009 found that 73% of articles published in journals with a high impact factor reported serious adverse events [33]. Nevertheless, a more recent study showed that only 34% of reviewers examined information registered in a trial registry [34].
For developers of reporting guidelines, such as the CONSORT group, and for editors, our study questions the current way of reporting trials and the peer-review process. In ClinicalTrials.gov, results are posted in a standard tabular format without discussions or conclusions. Using templates with mandatory reporting of some  elements may facilitate the work of researchers by reminding them what they need to report and by standardizing their reporting. Including a template for reporting the results as part of the CONSORT guidelines could be useful to improve the completeness of trials' publication. Furthermore, after the data results are submitted, ClinicalTrials. gov staff members review the submissions before public posting [18]. Data providers may be asked to clarify items or make corrections. This systematic verification could also contribute to the completeness of data posted. These results may help convince publishers of the value of changes in the presentation of the results section of articles (standardized tabular format rather than narrative text) or of implementation of reporting guidelines. To improve the quality of reports of clinical trials, journals-even those endorsing the CONSORT statement-must move from their current position of passive endorsement (for the vast majority of them) to a more active implementation of CONSORT guidelines [35,36].
Although the reporting of results was more complete at ClinicalTrials.gov than in the published articles, reporting at  ClinicalTrials.gov is still suboptimal and could be further improved. Some elements, such as the number of patients assessed for eligibility, are nearly never reported at ClinicalTrials.gov. Other elements can be improved upon, such as the reporting of results for binary data. At ClinicalTrials.gov, the percentage of events rather than the number of events is frequently reported. Both the number of events and the number of patients analyzed per arm are needed to perform meta-analyses of binary data. Other elements raise some important issues, including the frequent reporting of nonserious adverse events observed in more than 5% of patients, 5% being the default frequency threshold for reporting nonserious adverse events at ClinicalTrials.gov according to the FDAAA [15][16][17]. The reporting of adverse events observed at a certain frequency or threshold rate has been previously outlined as poor reporting practice [37].

Limitations
Our study has several limitations. We focused on trials with both results posted and published. It is possible that unpublished trial results could be published at a future date; some trials are submitted for publication several years after completion. When there were several publications for the same trial, we did not include all reports resulting from the trial but only the report describing the results for the primary outcomes. We chose this strategy because, according to the CONSORT statement, safety results should be reported with the main results. Only 9% of trials had multiple publications. These reports included protocols or preliminary or long-term results. None of the eliminated reports contained additional safety data for the same time frame as the selected report. Two of the eliminated reports contained additional safety results, but for longer-term follow-up. When assessing the completeness of reporting of efficacy results, we focused on a single primary outcome. If several primary outcomes were registered at ClinicalTrials.gov, we assessed the completeness of reporting for only the first primary outcome registered. Data extraction was not blinded to the source of data (ClinicalTrials.gov or published article) because blinding would have been impossible to achieve. Completeness of reporting was assessed as a binary outcome (all elements reported versus not all elements reported), so other assessment approaches may result in different findings. Finally, we could not determine whether publication, time to publication, and completeness were associated with risk of bias in trial design or conduct because ClinicalTrials.gov contained insufficient methodological information for assessing risk of bias [21,38].

Conclusions
In conclusion, our results highlight the importance of extracting efficacy and safety data posted at ClinicalTrials.gov not only for trials whose results are not yet published, but also for those with published results, because we found that reporting was more complete at ClinicalTrials.gov. Use of templates allowing for standardized reporting of trial results in journals or broader mandatory registration of results for all trials may help further improve transparency.

Supporting Information
Alternative Language Abstract S1 French translation of the abstract by AD.  corresponding journal publication. The median time between the study completion date and the first results being publicly posted at ClinicalTrials.gov was 19 months, whereas the time between completion and publication in a journal was 21 months. The flow of participants through trials was completely reported in 64% of the ClinicalTrials.gov postings but in only 48% of the corresponding publications. Results for the primary outcome measure were completely reported in 79% and 69% of the ClinicalTrials.gov postings and corresponding publications, respectively. Finally, adverse events were completely reported in 73% of the ClinicalTrials. gov postings but in only 45% of the corresponding publications, and serious adverse events were reported in 99% and 63% of the ClinicalTrials.gov postings and corresponding publications, respectively.
What Do These Findings Mean? These findings suggest that the reporting of trial results is significantly more complete at ClinicalTrials.gov than in published journal articles reporting the main trial results. Certain aspects of this study may affect the accuracy of this conclusion. For example, the researchers compared the results posted at ClinicalTrials.gov only with the results in the publication that described the primary outcome of each trial, even though some trials had multiple publications. Importantly, these findings suggest that, to enable patients and physicians to make informed treatment decisions, experts undertaking assessments of drugs should consider seeking efficacy and safety data posted at ClinicalTrials.gov, both for trials whose results are not published yet and for trials whose results are published. Moreover, they suggest that the use of templates to guide standardized reporting of trial results in journals and broader mandatory posting of results may help to improve the reporting and transparency of clinical trials and, consequently, the evidence available to inform treatment of patients.