Introduction

It is axiomatic among the research and patient communities that medical progress is dependent on clinical trials. As a result, both communities and the regulatory authorities have been concerned with optimizing access of patients to clinical trials and experimental new treatments, streamlining clinical trial design, and shortening the time to new drug approvals. Relatively little attention has been paid to efficiency of the clinical trial publication process.

Many observers have reported that positive clinical trials are more likely to be published than negative trials [1–7]. Other factors also affect the odds of publication such as whether or not the trial was industry-sponsored [8–11], was randomized [10], or was considered a pivotal trial [12]. Overall, the publication rates of clinical trials have generally been reported between 60–70% (range 46–86%)[8, 10, 12–15]. However, these analyses included clinical trials from a wide range of medical fields; oncology trials represented only a minority in most of these analyses and the reasons for non-publication were not explored.

Among oncology trials specifically, little has been published about the overall rate of publication or reasons for trials not to be published. Chen reported on 598 randomized oncology trials world-wide and found a 66.6% cumulative probability of publication with only 32% of the trials having been published within 24 months of the trial completion date [4]. They found that publication was less likely in trials with blinded assignment to treatment groups and in trials considered to have a negative outcome.

We examined clinical trials that closed to accrual between 2009–2013 at our cancer center to explore how long it took to publish the results, how many had not been published, and the reasons for non-publication.

Methods

Data collection

Using an institutional database, we identified all trials at Memorial Sloan Kettering Cancer Center (MSKCC) with a date of accrual closure between Jan 1, 2009 and Dec. 31, 2013. All trials had been approved by the MSKCC Institutional Review Board. These dates were chosen to ensure a minimum of three years of follow-up. November 1, 2016 was the date of last follow-up for publication status. All trials opened to accrual at MSKCC were included in this analysis except for studies not intended for independent publication such as specimen collection or expanded access trials. From our institutional database, we obtained start and stop dates of study accrual at MSKCC, number of patients accrued, study protocols, ClinicalTrials.gov registry (NCT) numbers, study purpose (therapeutic vs. diagnostic), and study phase (pilot, phase I, II, or III). We also categorized the studies based on the study sponsor according to the NCI Cancer Center Support Grant guidelines based on funding and who controlled the design and implementation of the study: Industrial (pharmaceutical company), Institutional (MSKCC or another academic institution), national cooperative group, or peer-reviewed (funded by other organizations employing external peer review). This was done in order to analyze outcomes by study sponsor type.

The primary endpoint of our study was to estimate the proportion of clinical trials that were ultimately published. Secondary endpoints were time to publication with special attention to 2-yr publication rates and median time to publication. The 2-yr publication rate was selected as a measure of timely publication as noted by previous investigators [4, 13] Among unpublished trials, we collected data on the reason for non-publication. Time to publication was computed as the time elapsed between the date of accrual closure at MSKCC and the date of publication. We selected date of accrual closure rather than date of study closure as our time zero because final study closure can often occur many years after accrual has been completed and after an initial publication has appeared. We defined a publication as a research article communicating the primary outcomes of a trial in any peer-reviewed journal. Publications linked to NCT numbers in the MEDLINE and EMBASE databases were retrieved for 235 trials. For the remainder (574 studies without an easily-identifiable publication), a survey was sent to the principal investigator (PI) at MSKCC of each trial to retrieve peer-reviewed publications of trial results. All publications were independently reviewed by one of the authors (NJL) to ensure agreement between the published result and the primary aims of the trial as delineated in the study protocol. When available, the time of first publication online was recorded as the date of publication. If not provided, the issue date of the article was recorded. The day of publication within the month was assumed to be the 15^th if no specific day of the month was given.

If the trial was not published, the PI was asked to select one main reason for non-publication among several predetermined options (Table 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Response options in reporting reasons for non-publication of trials.

https://doi.org/10.1371/journal.pone.0184025.t001

Results

Protocols analyzed

Between 2009–2013, 886 trials closed to accrual at MSKCC. Fifty-seven trials designed primarily for specimen collection, treatment continuation, or expanded access were excluded from this analysis because they were not designed for independent publication. Another 20 trials were excluded because they had been published prior to the date of closure to accrual. This left 809 trials that formed the basis of this analysis (Table 2, Fig 1). The year of activation ranged from 1977 to 2013 (Fig 2). The median year was 2009 (25%-75%; 2007–2010). Six hundred twelve trials (76%) had therapeutic endpoints; 197 (24%) were non-therapeutic trials. Trials included 109 pilot studies (13%), 181 phase I trials (22%), 290 phase II trials (36%), and 121 phase III trials (15%). The remaining 108 trials (13%) did not fall into these categories and were considered “other”. Sponsorship was divided between 286 industrially-sponsored trials (35%), 119 national cooperative group studies (15%), 346 trials institutionally sponsored (43%), and 58 trials externally peer-reviewed grants (7%).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Consort diagram.

https://doi.org/10.1371/journal.pone.0184025.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Distribution of activation years.

The year of activation at MSKCC is shown on the x-axis for the 809 protocols analyzed. Protocols for the year 2000 (N = 6 protocols) include 3 protocols activated in earlier years (1977, 1998, 1999). These protocols were collapsed into 2000 for clarity of the histogram.

https://doi.org/10.1371/journal.pone.0184025.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Protocol characteristics, overall observed, and estimated 2-year publication rates.

https://doi.org/10.1371/journal.pone.0184025.t002

Of the 574 trials for which we queried the PI to identify unindexed publications, we found publications for 234 and confirmed that 337 had not been published. For 3 trials, we could not get any information and counted them as having not been published, reasons unknown. Overall, 469 (58%) of the trials had been published by the data cutoff date.

Cumulative probability of publication

At the time of data lock, 469 of the 809 protocols (58%) had been published. We measured from the time of completion of accrual at MSKCC. For all protocols, the estimated probability of publication 7 years after closure to accrual was 70.4% (Fig 3A). The estimated median time to publication was 47 months after closure to accrual.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Cumulative probability of publication.

A) All protocols. Tick marks indicated censored (non-published) trials. B) Therapeutic (red) and non-therapeutic (green) protocols were plotted separately. Log-rank p = 0.025. C) Cumulative probability of publication based on type of sponsor. There was no significant differences between trials sponsored by industry (red line), National cooperative group trials (green), institutional sponsors (broken blue line), or by peer-reviewed grants (broken purple line). Log rank p = 0.673. D) Cumulative probability of publication based on phase of the trial. There was no significant differences between pilot trials (red), phase I (broken green line), phase II (broken blue line), phase III (broken purple line), and other trials (yellow) were plotted separately. Log rank p = .234.

https://doi.org/10.1371/journal.pone.0184025.g003

Fig 3B shows that the cumulative probability of publication over the entire time period was significantly different for therapeutic compared to non-therapeutic trials (log-rank p = 0.025). Although the difference in the proportion of trials published by 2 years was negligible, therapeutic trials had a higher cumulative publication probability by 7 years (72.1%; 95%CI 67–77) compared to non-therapeutic trials (65.2%; 95%CI 55.5–74.7). One possibility is that the non-therapeutic trials became less relevant after several years and were not considered of interest (discussed below). There were no significant differences in the cumulative probability of publication by sponsor type (Fig 3C). National cooperative group trials had a very low publication proportion by year 2 (5.9%; 95%CI 2.8–11.9%), presumably because many of these trials require longer periods of time to collect data and follow up. By year 4, the proportion of protocols published had caught up with the other trials.

Surprisingly, there was also no difference by trial phase (pilot, phase I, II, III, or other) (Fig 3D). Given that pilot and phase I trials generally require little follow up after patients complete treatment, we had expected to see a shorter time to publication for these trials.

Conditional probabilities of publication

For trials not published by 2 years, we calculated the conditional probability for publication by year 3 and by year 7 (Table 3). The probability these trials were published by the next year (year 3) was 22% overall; the probability of these trials were published by year 7 was 64%, overall. Therefore, there was a 36% chance that trials not published by year 2 would remain unpublished by year 7. There were no obvious differences in the conditional probability of publication based on trial purpose, phase, or sponsor, although formal statistical analyses were not applied.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Conditional probabilities of publication for trials not published by year 2.

https://doi.org/10.1371/journal.pone.0184025.t003

Reasons for protocols not being published

We collected information from the principal investigators at MSKCC as to the reason why the unpublished trials had not been published. Among the 340 protocols that had not been published, 154 (45.3%) completed data collection and analyses. For 93 (60%) of these completed but unpublished trials, a manuscript was either in preparation or had been submitted for publication (Table 4). Of the remaining unpublished completed trials, 45 (29%) were not considered of sufficient interest by the investigators. Many of these were either “negative trials” or were no longer considered of interest to the field. Of note, there were only 11 completed trials that had not been published due to sponsor delays (3.2% of all unpublished trials; 1% of all trials). Sixty trials (17.6% of unpublished trials; 7% of all trials) were unpublished due to ongoing data collection and/or analyses accounted for. Overall survival is a common endpoint in oncology trials and may explain some of this delayed publication.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Reasons trials were not published.

https://doi.org/10.1371/journal.pone.0184025.t004

Almost a third (30.6%) of unpublished trials (104 trials) had closed before completing accrual, most commonly due to poor accrual or sponsor-related issues (such as a change of sponsor priorities or other financial issues). Uncommon reasons for discontinuing accrual included that the PI had left the institution, unacceptable drug toxicity, or lack of drug availability.

Discussion

Timely publication of clinical research is critical to developing new and effective treatments for cancer and yet, others have previously lamented the under-reporting of oncology trials [5, 6, 11, 18, 19]. Relatively few analyses have been reported on the incidence of publication of clinical oncology trials. Two reports of publication rates of oncology-specific randomized clinical trials found that 66.6% [4] and 62% [20] were ultimately published. However, our study had important differences in criteria for trials inclusion, the methodology used to analyze the rate of publication taking into account censoring and follow-up time, and the definition of the primary endpoints making direct comparisons difficult. Among oncology trials reported as abstracts at the American Society of Clinical Oncology, the percentage of trials eventually published ranged between 61–74% [6, 11, 18, 19] with little difference between phase I, II, or III trials.

Our study reports on 809 oncology trials of all phases from pilot studies to phase III trials from a single cancer center including therapeutic and non-therapeutic trials. We chose to calculate time to publication from the date when patient accrual ended at our center because the date of study closure is often years after completion of accrual and it is common to publish data prior to final closure of the study. Also, from a patient’s point of view, the date of completing accrual represents the completion of patient participation in the clinical research. Starting from that time point, patients (and society) look to the timely publication of results. The estimated probability of publication by 7 years from the date of accrual completion at our center was 70.4% although only 18.8% of trials had been published within 2 years of completing accrual. The median time to publication was 47 months–nearly 4 years. Of unpublished trials, most were either completed or in progress although 45 trials (13.2% of unpublished trials) were completed but not published because the results were not considered of sufficient interest. Up to 2 years after completing accrual, the probability of publication for non-therapeutic trials was the same as for therapeutic trials. After 2 years however, non-therapeutic trials were less likely to be published. In 18.9% of the unpublished non-therapeutic trials, the reason given was that the data were not considered of interest. This is almost twice as frequent as for therapeutic trials (11%) and may speak to a shorter “shelf life” in which non-therapeutic trials that are not published quickly are more likely than therapeutic trials to lose relevance.

Much has been written about outcome reporting bias of industry-sponsored trials but there has been little published regarding the actual likelihood of publication of industry-sponsored trials within oncology. One analysis of randomized oncology trials registered at ClinicalTrial.gov found that industry-sponsored trials were published less often (40%) than nonindustry/nongovernment sponsored trials (56%) [8]. However, another trial evaluating the likelihood of publication among abstracts presented at ASCO found no difference between industry-sponsored and nonindustry-sponsored trials [11]. In our analysis, we saw no overall difference in the probability of publication based on the type of sponsor.

The phase of the trial did not affect the time to publication. Pilot trials and phase I trials were not published any quicker or at higher proportion than phase III trials. We had expected that all pilot and phase I trials would have been published in a relatively short period of time since they are typically smaller trials that do not have survival endpoints. Consistent with this, Table 4 confirms that very few unpublished pilot and phase I trials were still in progress in contrast to 45% of the unpublished phase III trials. Unpublished phase I trials were more likely than the other trials to have been discontinued by the sponsor, and both unpublished pilot and phase I trials were more likely to have been completed but considered not interesting than were the other trials. We have previously reported that lack of time and relocation of authors were common factors affecting the publication of phase I trials [19]. This has also been seen for phase II trials along with lack of interesting results [18] as reasons for failure to publish.

It is often difficult to get manuscripts published reporting negative trial results [1, 2, 5–7]. The scientific community interest level can be low and journals may believe that negative reports adversely affect their impact scores. However, most clinical trial participants believe that they have a right to study result disclosure regardless of trial outcome [21]. Depositing the results on clinicaltrials.gov in a timely manner can make minimum data sets available to the research and patient communities. However, for interventional FDA-regulated drug studies, the FDA Amendments Act only requires reporting of phase II and III trial results [22]. Even for these trials, reporting is not required until a year after all data have been collected. As noted above, oncology trials are often closed to accrual but left open for various reasons such as further survival follow up; full closure is typically several years after completion of accrual. Compounding this problem is the finding that compliance with the requirement to post results on clinicaltrials.gov is not universal. One study found that results had not been reported on clinicaltrials.gov within 4 years of study closure in 29.5% of completed trials [10]. Further, since results of phase I trials are not required to be reported on clinicaltrials.gov, phase I trials that find a new drug treatment inactive or too toxic may not be published anywhere.

One hundred four (30.6%) of the unpublished trials in our analysis were discontinued before completing accrual. Almost half of these cases were due to poor accrual which may indicate a failure to assess resources and patient availability accurately. To minimize the drain on resources and to limit patient involvement in trials that will ultimately fail to accrue, we have implemented institutional mechanisms to identify trials within 18 months that are unlikely to complete accrual and to close them within 30 months if the recruitment issue cannot be corrected. One cause of patient recruitment failure was a result of the PI relocating to another institution in 13 trials (3.8%). Since trial participation by an institution implies sufficient importance of the question being addressed, institutions should establish mechanisms to assure that opened trials will be completed even if the original PI relocates.

This analysis has some shortcomings. The results are limited only to our institution over this 5-year time period. However, we hope these results stimulate other cancer centers to look at their experiences regarding time to publication. Second, for publications indexed to NCT numbers, and unindexed publications that we identified by PI questionnaires, we cannot formally rule out that there could have been an earlier, unindexed publication. Similarly, for the 340 unpublished trials, it is possible that some were published unbeknownst to the PI and despite our NCT number database search. However, we do not think the number of such publications, if any, would have affected our results. A third shortcoming is that, for the trials that were published, we did not study the causes of publication delays. This would be a fruitful exercise for future investigation. For example, it is unclear why the published pilot and phase I trials took as long or longer to publish than phase III trials. We need to consider how long do investigators take to write manuscripts once sufficient data are available? How long does peer review take? How many separate submissions are made before acceptance? How many journals make accepted manuscripts available on line immediately? Finally, where can “negative” data be published so that they are available to the community and the author receives appropriate academic credit for having completed and published the study, which presumably had clinical relevance at the time the trial was opened? How these observations made from oncology trials of all phases relate to the publication barriers for non-oncology clinical trials would seem a valuable area of inquiry, but beyond the scope of our study. We all have a responsibility and an interest in minimizing the time it takes to publish the results of our clinical research. Moreover, our responsibilities to our patients demand it.