https://orcid.org/0000-0002-8135-6265
Figures
Abstract
Background
Clinical trial registries allow assessment of deviations of published trials from their protocol, which may indicate a considerable risk of bias. However, since entries in many registries can be updated at any time, deviations may go unnoticed. We aimed to assess the frequency of changes to primary outcomes in different historical versions of registry entries, and how often they would go unnoticed if only deviations between published trial reports and the most recent registry entry are assessed.
Methods and findings
We analyzed the complete history of changes of registry entries in all 1746 randomized controlled trials completed at German university medical centers between 2009 and 2017, with published results up to 2022, that were registered in ClinicalTrials.gov or the German WHO primary registry (German Clinical Trials Register; DRKS). Data were retrieved on 24 January 2022. We assessed deviations between registry entries and publications in a random subsample of 292 trials. We determined changes of primary outcomes (1) between different versions of registry entries at key trial milestones, (2) between the latest registry entry version and the results publication, and (3) changes that occurred after trial start with no change between latest registry entry version and publication (so that assessing the full history of changes is required for detection of changes). We categorized changes as major if primary outcomes were added, dropped, changed to secondary outcomes, or secondary outcomes were turned into primary outcomes. We also assessed (4) the proportion of publications transparently reporting changes and (5) characteristics associated with changes. Of all 1746 trials, 23% (n = 393) had a primary outcome change between trial start and latest registry entry version, with 8% (n = 142) being major changes, that is, primary outcomes were added, dropped, changed to secondary outcomes, or secondary outcomes were turned into primary outcomes. Primary outcomes in publications were different from the latest registry entry version in 41% of trials (120 of the 292 sampled trials; 95% confidence interval (CI) [35%, 47%]), with major changes in 18% (54 of 292; 95% CI [14%, 23%]). Overall, 55% of trials (161 of 292; 95% CI [49%, 61%]) had primary outcome changes at any timepoint over the course of a trial, with 23% of trials (67 of 292; 95% CI [18%, 28%]) having major changes. Changes only within registry records, with no apparent discrepancy between latest registry entry version and publication, were observed in 14% of trials (41 of 292; 95% CI [10%, 19%]), with 4% (13 of 292; 95% CI [2%, 7%]) being major changes. One percent of trials with a change reported this in their publication (2 of 161 trials; 95% CI [0%, 4%]). An exploratory logistic regression analysis indicated that trials were less likely to have a discrepant registry entry if they were registered more recently (odds ratio (OR) 0.74; 95% CI [0.69, 0.80]; p<0.001), were not registered on ClinicalTrials.gov (OR 0.41; 95% CI [0.23, 0.70]; p = 0.002), or were not industry-sponsored (OR 0.29; 95% CI [0.21, 0.41]; p<0.001). Key limitations include some degree of subjectivity in the categorization of outcome changes and inclusion of a single geographic region.
Conclusions
In this study, we observed that changes to primary outcomes occur in 55% of trials, with 23% trials having major changes. They are rarely transparently reported in the results publication and often not visible in the latest registry entry version. More transparency is needed, supported by deeper analysis of registry entries to make these changes more easily recognizable.
Protocol registration: Open Science Framework (https://osf.io/t3qva; amendment in https://osf.io/qtd2b).
Author summary
Why was this study done?
- Clinical trial registries are a key tool to increase the trustworthiness of clinical trials. They allow assessment of how closely a published trial follows its original plan.
- However, registry entries can be updated at any time, which creates a trail of historical versions. If the latest registry entry version matches with the published trial report, important preceding changes might thus be unapparent to assessors at first glance.
- Our objective was to investigate how often primary outcomes are changed in the trial registry over the course of a trial, and how often outcome changes are unapparent if one compares only the latest registry entry version to the publication.
What did the researchers do and find?
- We assessed all 1746 randomized controlled trials completed at German university medical centers between 2009 and 2017, that have a results publication and that had been registered in either an international or a German clinical trial registry. We determined the frequency of outcome changes between different versions of a registry entry, as well as the latest registry entry and the results publication.
- We defined adding or dropping primary outcomes, changing them to secondary outcomes, or turning secondary outcomes into primary outcomes, as major changes.
- We found that approximately 55% of trials had primary outcome changes at any timepoint over the course of a trial; 23% of trials had major changes. We observed changes that can be easily identified by comparing the published results to the latest registry entry in 41% of trials. In 14% of trials, however, the changes would require an in-depth look within the historical versions of that trial’s registry entry.
- Only 1% of trials with changes (2 trials) reported this in the corresponding publications.
Citation: Holst M, Haslberger M, Yerunkar S, Strech D, Hemkens LG, Carlisle BG (2023) Frequency of multiple changes to prespecified primary outcomes of clinical trials completed between 2009 and 2017 in German university medical centers: A meta-research study. PLoS Med 20(10): e1004306. https://doi.org/10.1371/journal.pmed.1004306
Received: March 1, 2023; Accepted: October 3, 2023; Published: October 31, 2023
Copyright: © 2023 Holst et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The final dataset has been posted to our Open Science Framework page (https://osf.io/e2uct/). Analysis code and original datasets are available on Github (https://github.com/Martin-R-H/InvisibleOutcomeChanges).
Funding: This work was funded under a grant from the Federal Ministry of Education and Research of Germany (Bundesministerium fuer Bildung und Forschung – BMBF) [01PW18012], awarded to DS. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Clinical trial registries are tools to enhance the transparency and trustworthiness of clinical trial evidence by openly reporting key study parameters in a rapidly accessible manner [1]. They make it possible to detect selective reporting biases that pose a threat to validity, by allowing reviewers to assess possible deviations from an original protocol and analysis plan. One of the most critical deviations is a change to prespecified outcome measures (or ‘outcomes’, for short). Consequently, outcomes and their pre-specification are among the most important reporting items of clinical trials, with the CONSORT statement asking authors to describe ‘completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed’ as well as ‘any changes to trial outcomes after the trial commenced, with reasons’ [2]. If readers do not know whether there have been any changes to the outcomes, e.g., by adding or dropping primary outcomes, they cannot assess the risk of bias due to selective reporting of only statistically significant results or multiple hypothesis testing [3].
Many clinical trial registries, such as ClinicalTrials.gov, allow entries to be updated after initial registration. Updating a registry entry is a useful tool, as parts of a trial protocol often change and it is desirable for an entry to reflect the most current information. It still constitutes another possible source of reporting bias if entries are not properly maintained and larger changes go unreported. While a number of ethical and reporting guidance documents recommend prospective trial registration [2, 4, 5] and some recommend the transparent reporting of protocol changes [2, 5], this is only checked in a minority of cases by peer-reviewers [6]. But even if trial registries were considered during review, looking only at the latest registry entry version may lead to an incomplete assessment, as changes over the course of the trial would remain unapparent if the latest registry entry matches with the results publication. These changes would only be discovered by closely investigating the history of changes in trial registries, which some trial registry platforms maintain. These histories, however, are often not easily accessible.
We systematically searched studies that have assessed the frequency of outcome changes between registries and publications [7]. We identified several analyses which widely agree that this practice is common, with a median estimated prevalence of 31% (interquartile range (IQR): 17–45%) of clinical trials affected [8]. However, most studies did not report which registry entry version they used [9–16], or reported that they only assessed changes between the publication and one single registry entry version (i.e., the latest available entry [17–25], the latest entry before trial completion [26], the first available entry [27], or the first entry during the active phase [28]). We are aware of only two studies that examined changes to primary outcomes over the course of the entire registry version history. Both used relatively small and selective samples (either trials published in ICMJE journals or trials registered on ClinicalTrials.gov covered by the Food and Drug Administration Amendments Act) over a four-year time period [29, 30]. We are not aware of studies quantifying the frequency of changes that may go potentially unnoticed because the latest registry entry and the results publication match.
In this study, we aimed to provide an in-depth analysis of changes to primary outcomes using a comprehensive nationwide sample over 9 years that represents the current practice of trial registration in academic research. Specifically, we aimed to assess how often primary outcomes of clinical trials were changed (1) across all clinical trial registry entry versions and (2) between the latest registry entry and the results publication; how often (3) changes are not detectable when only the latest registry entry version would be inspected, and (4) how often they are transparently reported in the results publications; and (5) which trial characteristics are associated with these outcome changes.
Methods
Data sources and sample
We based our analyses on previous work of our group that provided the highly granular and large-scale data on both registry entries and accompanying results publications required to address our aims. Two large datasets [31, 32] of randomized controlled trials covering a period of 9 years of trial completion plus a period of 5 years of publication tracking were used for this project. The trials were completed at German university medical centers between 2009 and 2017 and had been registered in either ClinicalTrials.gov or the German World Health Organization primary registry (German Clinical Trials Register or Deutsches Register Klinischer Studien; DRKS). Both registries have an accessible history of changes. The corresponding results publications were identified in manual searches up to 2022, ensuring an at least five-year period after trial completion and complementing the often inconsistent references in the trial registry [33]. We retrieved a combined dataset for both projects from a GitHub repository [34] on 24 January 2022.
Eligibility criteria
We included any study that fulfilled all of the following criteria: 1) has a registry entry in either the ClinicalTrials.gov or the DRKS database, 2) was completed between 2009 and 2017 according to the registry trial status described as ’Completed’, ’Unknown status’, ’Terminated’, or ’Suspended’ (ClinicalTrials.gov), or ’Recruiting complete, follow-up complete’, ’Recruiting stopped after recruiting started’, or ’Recruiting suspended on temporary hold’ (DRKS), 3) reported in the registry that a German university medical center was involved (i.e., mentioned as responsible party, lead/primary sponsor, principal investigator, study chair, study director, facility, collaborator, or recruitment location; for definitions see [31, 32]), 4) has published results, i.e., a full-text publication of the trial results was found by the search methods described in the protocol for the underlying datasets [31, 32], 5) is registered as a randomized trial.
Data extraction and processing
We used a stepwise approach of automatic data extraction and processing followed by reviewer assessment of identified outcome changes. We first compared outcomes between different registry entry versions, followed by comparing the most recent registry entries to the accompanying results publications.
Data extraction and review procedure.
Trial identifiers (NCT or DRKS number) and basic information about the trials and their accompanying results publications were included in the dataset [31, 32]. We then retrieved all historical registry entry versions of these trials, using R 4.1.3 [35] and the cthist R package [36], on 24 January 2022 (DRKS) and 12 April 2022 (ClinicalTrials.gov). We extracted data from the fields listed in the codebook [37]. For our classification of within-registry changes, we extracted primary and secondary outcomes at four key trial milestones: (1) the first entry after study start (start date; inclusion of the first patient), (2) the latest entry before the end of active status (completion date; determined by ‘completed’ status in the registry), (3) the latest entry before results publication (publication date), and (4) latest available entry.
We classified trials into medical fields based on their journal’s category from the SCImago Journal & Country Rank database [38], which is based on the Scopus medical field classification. The journals’ categories were divided into 17 higher-order categories based on consensus (MH, MH, LGH).
Assessment of within-registry changes.
Two reviewers (MH, MH) assessed within-registry changes in duplicate and resolved conflicts through discussion, using the Numbat Systematic Review Manager [39]. Where necessary, a third reviewer (LGH) resolved conflicts. Reviewers assessed changes to the primary outcomes between the four key trial milestones within the registry (Fig 1), using a classification system that categorizes changes to primary outcomes as either major changes (based on an existing classification system [40]) or minor changes (adapted from [20, 41]). Major changes were defined as adding or dropping primary outcomes, changing them to secondary outcomes, or turning secondary outcomes into primary outcomes. Minor changes were defined as changes to the outcome’s type of measurement, metric or method of aggregation, or timing of assessment, or as either added specificity or omissions to these (for more detail see Table 1).
Registry entry versions at four subsequent key trial timepoints were assessed (blue box). Changes can occur between registry entry versions (within-registry changes; red arrows) and between the latest registry entry and results publication (blue arrow). In the lower part, examples of primary outcome descriptions are given (pink boxes), with corresponding categorizations of changes in grey next to the arrows (see Table 1 for more details on categorization).
Assessment of registry-publication changes.
For feasibility, we drew a random sample of 300 trials with accompanying results publications, from which three reviewers (MH, MH, SY) extracted primary outcomes. Eight articles were excluded from the sample because they were not the main results publications (i.e., the final sample size was 292). We considered outcomes primary if they were explicitly named as such using a list of keywords (‘primary’, ‘main’, ‘outcome’, ‘endpoint’, ‘end point’). Otherwise, we used the outcome used for sample size calculation, if reported. In all other cases, we used the first reported outcome in the abstract and results section (with priority given to the abstract). For each entry, reviewers compared the extracted primary outcome(s) to the latest available registry entry version using the developed rating system (Fig 1 and Table 1). Reviewers also recorded whether there were any mentions of outcome changes in the assessed results publications. Conflicts were resolved through discussion between reviewers, with one additional reviewer (LGH) resolving conflicts, where necessary.
Inter-rater agreement.
The overall inter-rater agreement before discussion, over all timepoints and items (measured by Cohen’s kappa), was κ = 0.81 for the within-registry change ratings and κ = 0.46 for the registry-publication change ratings. Disagreement resulted partly from cases without clear identification of the primary outcomes in results publications (if only the explicitly mentioned outcomes were used for the registry-publication change ratings, inter-rater agreement was κ = 0.51). In a post-hoc sensitivity analysis we restricted our analysis to primary outcomes that were explicitly named as such in the results publications or had been used for sample size calculation.
Statistical analyses and reporting
We report descriptive statistics for the frequency of different types of within-registry outcome changes (between key trial milestones) and registry-publication outcome changes. For percentages within the random sample of 292 trials with manual assessment of results publications, we report 95% confidence intervals (CI). For the analysis of determinants of within-registry outcome changes or registry-publication outcome changes, we used logistic regressions, with either any within-registry outcome change, or any registry-publication outcome change, as output variables, and a set of 9 candidate input variables (study phase, sponsor, publication year, registration year, medical field, registry, multicenter trial, enrollment, intervention). These variables were prespecified before the start of regression analyses (details in S1 and S2 Tables). We used the PRISMA statement [42] to structure our manuscript (S1 PRISMA Checklist).
Results
We included 1746 clinical trials completed at German university medical centers between 2009 and 2017 (Table 2 and Fig 2). The median sample size was 100 (IQR: 50–264), their median registration year was 2011 (IQR: 2009–2013) and median publication year was 2015 (IQR: 2013–2017). Most trials investigated topics in general medicine (34%) and internal medicine (19%). Industry was the lead sponsor for 26% of trials.
Prevalence of outcome changes
Primary outcome changes within trial registry entries.
Of 1746 trials, 393 (23%) had a primary outcome change between trial start and latest registry entry version, i.e., within registry entry versions between key trial milestones (Table 3 and S1 Fig). Minor changes were found in 318 (18%) trials. 142 trials (8%) had major changes, i.e., primary outcomes were added, dropped, changed to secondary outcomes, or secondary outcomes were turned into primary outcomes. Of these major changes, 66 (4%) happened between first patient inclusion (start date) and trial completion date, 49 (3%) between completion and publication date, and 36 (2%) between the publication date and the latest registry entry.
Primary outcome changes between latest registry entry and publication.
Primary outcomes in publications were different from the latest registry entry version in 41% of trials (120 of the 292 sampled trials; 95% CI [35%, 47%]), with major changes in 18% (54 of 292; 95% CI [14%, 23%]) and minor changes in 26% (75 of 292; 95% CI [21%, 31%]) (Table 3).
Apparent and unapparent changes of primary outcomes
Among the 292 trials, 41 trials had changes only within the registry histories while the latest registry entry matched the results publication (14%; 95% CI [10%, 19%]). Of these changes, 13 were major (4%; 95% CI [2%, 7%]) and 35 minor (12%; 95% CI [8%, 16%]). Changes between the latest registry entry and the results publication, but also within the registry were found in 25 trials (9%; 95% CI [6%, 12%]), of which 12 (4%; 95% CI [2%, 7%]) had major and 14 (5%; 95% CI [2%, 7%]) had minor changes. Overall, 161 trials had any change to their primary outcomes after study start (55%; 95% CI [49%, 61%]), with 67 having major changes (23%; 95% CI [18%, 28%]) and 110 having minor changes (38%; 95% CI [32%, 44%]) (Figs 3, S2, and S3).
Registry entry versions were assessed at four subsequent key trial timepoints (within-registry changes; red arrows) and between the latest registry entry and results publication (blue arrow). Rows A to K show each observed combination of outcome change at the different timepoints across the 292 trials. For example, Row A indicates that 131 of 292 trials had no outcome change (45%). Row B indicates that 95 of 292 trials only had a change between the latest registry entry version and the results publication (blue circle; 33%). Changes in rows C, D, H, I and J only occurred within registry entries (red arrow, red circle) while there was agreement of the latest registry entry and the publication (no blue circle), i.e., these changes would not be detected by comparing the results publication to the latest available registry entry (41 of 292 trials, 14%). Row E, F, G and K indicates trials that had a change between the latest entry and the published paper, but also a change at earlier timepoints.
Reporting of changes in the publications
Of all 161 trials with any change of a primary outcome (i.e., 120 trials with easily detectable changes between the latest registry entry and the publication and 41 trials with more unapparent changes before), 2 trials reported this change in the results publication (1%, 95% CI [0%, 4%]). One of the trials, which exhibited a major change both within the registry (between completion and publication date) and between the latest registry entry version and the results publication, stated that some of the primary outcomes were going to be reported in a separate manuscript [43]. The other publication, which had a major change between the latest registry entry version and the results publication, stated that the changed outcome was the ‘more sensitive and direct measurement’ [44].
Trial characteristics associated with outcome changes (within-registry and registry-publication)
Within-registry outcome changes were less likely in non-industry-sponsored trials (14% Non-Industry vs 47% Industry, OR 0.29, 95% CI [0.21, 0.41], p < 0.001; S1 Table), in trials registered in DRKS and not on ClinicalTrials.gov (5% DRKS vs 27% ClinicalTrials.gov, OR 0.41, 95% CI [0.23, 0.70], p = 0.002), and in trials that were registered later (per registration year; OR 0.74, 95% CI [0.69, 0.80], p < 0.001). No statistically significant association with any other trial characteristic was found.
For registry-publication outcome changes, we identified no statistically significant association with any of the prespecified input variables (S2 Table).
We detected no statistically significant association between within-registry outcome changes and registry-publication outcome changes (S3 Table).
Results were similar in the sensitivity analysis (S4 Table).
Discussion
Our assessment of all 1746 published clinical trials conducted at German university medical centers over a period of 9 years shows that changes to primary outcomes are common. About one in four trials (23%) have outcome changes within the registry after study start; a major change occurs within one in twelve trials (8%). Of the 292 assessed results publications, 41% have changes between the latest registry entry and the outcomes presented to readers, and 14% of the trials have no easily detectable change, as the changes occurred between different historical versions of the trials’ registry entries while the latest entries match the results publications. These changes were minor in 12% and major in 4% of trials. An inspection of only the latest registry entry version alongside the results publication, without assessment of the entire registration history, would miss a primary outcome change in these trials.
We used a comparatively large sample of German clinical trials registered between 2009 and 2017, with a validated set of accompanying results publications for registry entries, determined by an extensive manual screening process [31, 32]. Our methodology allowed us to detect changes to registered primary outcomes that might have gone unnoticed by other studies, by retrieving all historical versions for trials, and detecting changes using a streamlined workflow. This is different from previous studies, which instead used manual searches [29, 30]. Many trials in our sample had changes to the registry after study completion (9%) or after publication of the first results publication (8%). The changes, for which there may be very plausible reasons, were almost never reported in the results publications. We revealed potential indicators for these practices, with trials that were registered later, trials not registered on ClinicalTrials.gov, and non-industry-sponsored trials having lower odds of within-registry outcome changes. The observation that the frequency of outcome changes declined over time may be explained by recent discussion on clinical trial reporting practices. Regarding the type of registry, the association might be explained by better usability of the ClinicalTrials.gov platform, which might facilitate more frequent changes to registry entries. Regarding sponsorship, our results could indicate better reporting and higher transparency in industry-sponsored trials (which change their registry entries more often than non-industry-sponsored trials). We currently have no sufficient data to quantify the role of selective reporting bias in this sample.
Assessments of outcome changes that rely on a single version of registry entries might underestimate the true prevalence of changes. The ICMJE policy states that any changes to the registration should be explained by authors in a publication [5]. Both the observed lack of reporting on outcome changes in results publications, as well as survey results among manuscript reviewers [6], indicate that reviewers check the registry entry of a clinical trial only in the minority of cases. While registries make change histories available and offer tools to compare different registration versions, and journal editors and reviewers might in some cases use these tools to check the registration histories, the low number of reporting of changes to primary outcomes in the results publications suggests that this is not properly enforced. At the same time, in their current form, historical versions of registry entries are not easily accessible and readable, which makes checking them a time-consuming task. Journals could implement editorial policies that require specialized personnel to assess the historical versions of submitted clinical trials. Clinical trial registries, on the other hand, could implement solutions to identify and mark trials with major changes to their outcomes.
Still, some trials change their outcomes at two or even three different timepoints over the course of the trial. Interestingly, we found not only cases in which primary outcomes gained more detail, but also cases in which primary outcomes were described with less detail than before. Overall, we found 55% of trials to have some form of outcome change over the course of the study. This is somewhat higher compared to other studies, which exhibit a large range for the frequency, with a median of 31% (IQR: 17–45% [8]). This might be due to different methodology, samples, timepoints, or definition of outcome changes. Chan et al. [45], for example, used a more narrow definition of outcome changes (roughly corresponding to our ‘major’ category). We used a relatively broad definition of primary outcome changes, defining changes to the specificity of primary outcomes as ‘minor’ changes. Our broader definition and inclusion of more ‘minor’ outcome changes, some of which might not constitute a large threat to a trial’s validity, might have led to our higher estimates.
This study has several limitations. First, we rely on the reporting quality of published studies. In 48 out of 292 publications, the primary outcome was not explicitly described in the results publication, so we had to select the outcome used for sample size calculation or the first reported outcome instead. A sensitivity analysis excluding these 48 studies supported our main analysis. Second, we only assessed reporting of outcome changes in the main results publications of the respective trials. These changes could also have been published in a protocol or methods paper. We assessed the results publications as they have higher impact, and because current reporting guidelines such as CONSORT state that deviations from the protocol should be reported in the results publication. Third, while we have information about changes that occurred post-completion, we have no data regarding changes in relation to the clinical trial’s data lock, that is, the timepoint at which no further information is added to the dataset (which might lie after the completion date). Fourth, the categorization of outcome changes is to some degree subjective, and while the agreement was very high for the within-registry changes (κ = 0.81), it was moderate (κ = 0.46) for the registration-publication ratings due to the added complexity of identifying the correct outcome as ‘primary’ and different wording and descriptions between registry and publication. However, conflicting primary outcome definitions and conflicting ratings were resolved in discussions between the reviewers (and, where necessary, a third reviewer), and excluding unclear situations in the above-mentioned sensitivity analysis did not change the main findings. Still, this could become an issue in practice if, for example, different peer-reviewers disagree in their assessment of whether an outcome change has happened or not. Fifth, although we used high quality underlying data reflecting German clinical research output, we only assess a single geographic area, partly reflecting EU/German registration policies. Still, many registration policies are international, and our results point to the possibility that technical details of the registries rather determine whether outcome changes occur. Finally, we only looked at changes between registry entry versions but did not assess how many outcome measures remain underspecified (e.g., no information on the type of measurement, timing, or method of aggregation).
Our analysis also demonstrated the feasibility of an efficient workflow [46] that can be used for future projects and overcome previously described challenges to incorporate historical registry data [47].
To conclude, primary outcome changes are not uncommon in clinical research. Such changes are, in one of four cases, not directly visible upon first inspection of the trial registries and almost never reported in publications. While changes to primary outcomes are not unusual in clinical trials, may be well justified and sometimes desirable, undisclosed changes are a problem. While the problem seems to be ubiquitous, it may be more common in older trials, trials registered in ClinicalTrials.gov, and industry-sponsored trials. Even though trial registries can be an important tool to detect outcome changes, review procedures employed by journals seem to rarely make use of them. Our approach provides a feasible workflow to further investigate this issue. Overall, it is not sufficient to only assess the publication and the latest registry entry–a careful assessment of the full research conduct from conception to publication is needed to ensure trustworthy evidence for clinical decision-making.
Supporting information
S1 PRISMA Abstract Checklist. PRISMA 2020 for abstracts checklist.
https://doi.org/10.1371/journal.pmed.1004306.s001
(DOCX)
S1 Fig. Percentage of trials showing primary outcome changes, for each of the within-registry comparisons.
The nodes on the left show changes between the registry entry version at the start date and the registry entry version at the completion date. The nodes in the middle show changes between the registry entry at the completion date and the registry entry at the publication date. The nodes on the right show changes between the registry entry at the publication date and the latest registry entry. ‘No information’ means that the registry entries at the two timepoints were the same. This applies to circa 54% of comparisons in the right node, as the registry entry at publication is often the latest entry. The grey bars show the ‘flow’ of changes: For example, the largest grey bar between the column on the left and the column in the middle indicates that the majority of trials with no changes between the registry entry versions at the start date and completion date also have no changes between the registry entry version at completion date and publication date. For more detailed information, please see our interactive online plot: https://martin-r-h.shinyapps.io/invisibleoutcomechangesfigure/.
https://doi.org/10.1371/journal.pmed.1004306.s003
(TIF)
S2 Fig. Number of trials with major outcome changes between registry entry versions and/or publications in a random sample of 292 out of 1746 randomized controlled trials.
Registry entry versions were assessed at four subsequent key trial timepoints (within-registry changes; red arrows) and between the latest registry entry and results publication (blue arrow). Rows A to H show each observed combination of outcome change at the different timepoints across the 292 trials. For example, Row A indicates that 225 of 292 trials had no major outcome change (77%). Row B indicates that 46 of 292 trials only had a major change between the latest registry entry version and the results publication (blue circle; 16%). Changes in rows C and D only occurred within registry entries (red arrow, red circle) while there was agreement of the latest registry entry and the publication (no blue circle), i.e., these changes would not be detected by comparing the publication to the latest available registry entry (13 of 292 trials, 4%). Rows E, F, G and H indicate trials that had a major change between the latest entry and the published paper, but also a change at earlier timepoints.
https://doi.org/10.1371/journal.pmed.1004306.s004
(TIF)
S3 Fig. Number of trials with minor outcome changes between registry entry versions and/or publications in a random sample of 292 out of 1746 randomized controlled trials.
Registry entry versions were assessed at four subsequent key trial timepoints (within-registry changes; red arrows) and between the latest registry entry and results publication (blue arrow). Rows A to K show each observed combination of minor outcome change at the different timepoints across the 292 trials. For example, Row A indicates that 178 of 292 trials had no minor outcome change (61%). Row B indicates that 61 of 292 trials only had a minor change between the latest registry entry version and the results publication (blue circle; 21%). Changes in rows C, D, E, I and J only occurred within registry entries (red arrow, red circle) while there was agreement of the latest registry entry and the publication (no blue circle), i.e., these changes would not be detected by comparing the publication to the latest available registry entry (39 of 292 trials, 13%). Rows F, G, H and K indicate trials that had a minor change between the latest entry and the published paper, but also a change at earlier timepoints.
https://doi.org/10.1371/journal.pmed.1004306.s005
(TIF)
S1 Table. Frequencies, odds ratios (exponentiated regression coefficients) and accompanying p-values for the logistic regression model, with any within-registry outcome change as the output variable (N = 1746 trials).
None of the input variables had missing data, except medical field, where 74 trials could not properly be assigned.
https://doi.org/10.1371/journal.pmed.1004306.s006
(DOCX)
S2 Table. Frequencies, odds ratios (exponentiated regression coefficients) and accompanying p-values for the logistic regression model, with any registry-publication outcome change as the output variable (n = 292 trials).
None of the input variables had missing data, except medical field, where 9 trials could not properly be assigned.
https://doi.org/10.1371/journal.pmed.1004306.s007
(DOCX)
S3 Table. Frequencies, odds ratios (exponentiated regression coefficients) and accompanying p-values for the logistic regression model, with any within-registry outcome change as the output variable (n = 292 trials) and any within-registry outcome change as input variable.
The input variable did not have missing data.
https://doi.org/10.1371/journal.pmed.1004306.s008
(DOCX)
S4 Table. Primary outcome changes in 1746 randomized controlled trials completed between 2009 and 2017 in German university medical centers.
This table includes a sensitivity analysis for the registry-publication changes, including only publications with explicitly named primary outcomes (n = 243).
https://doi.org/10.1371/journal.pmed.1004306.s009
(DOCX)
References
- 1. Zarin DA, Keselman A. Registering a clinical trial in ClinicalTrials.gov. Chest. 2007 Mar;131(3):909–912. pmid:17303677
- 2. Moher D, Hopewell S, Schulz KF, Montori V, Gotzsche PC, Devereaux PJ, et al. CONSORT 2010 explanation and elaboration: Updated guidelines for reporting parallel group randomised trials. BMJ. 2010 Mar 23;340(mar23 1):c869–c869.
- 3. Simmons JP, Nelson LD, Simonsohn U. False-Positive Psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci. 2011 Nov;22(11):1359–1366. pmid:22006061
- 4. World Medical Association. Declaration of Helsinki: Ethical principles for medical research Involving human subjects. JAMA. 2013 Nov 27;310(20):2191.
- 5.
ICMJE. Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals. [accessed 2023 Sep 28]. Available from: https://www.icmje.org/icmje-recommendations.pdf.
- 6. Mathieu S, Chan AW, Ravaud P. Use of trial register information during the peer review process. PLoS ONE. 2013 Apr 10;8(4):e59910. pmid:23593154
- 7. Holst M, Haslberger M, Yerunkar S, Strech D, Hemkens LG, Carlisle BG. A comprehensive assessment of changes to prespecified trial outcomes, selective reporting, and spin in clinical trials (ASCERTAIN). 2022 Feb 2 [accessed 2023 Aug 31]. Available from: https://osf.io/za9rx/.
- 8. Jones CW, Keil LG, Holland WC, Caughey MC, Platts-Mills TF. Comparison of registered and published outcomes in randomized controlled trials: a systematic review. BMC Med. 2015 Dec;13(1):282. pmid:26581191
- 9. Smith SM, Wang AT, Pereira A, Chang DR, McKeown A, Greene K, et al. Discrepancies between registered and published primary outcome specifications in analgesic trials: ACTTION systematic review and recommendations. Pain. 2013 Dec;154(12):2769–2774. pmid:23962590
- 10. Bourgeois FT, Murthy S, Mandl KD. Outcome reporting among drug trials registered in ClinicalTrials.gov. Ann Intern Med. 2010 Aug 3;153(3):158. pmid:20679560
- 11. Gandhi R, Jan M, Smith HN, Mahomed NN, Bhandari M. Comparison of published orthopaedic trauma trials following registration in Clinicaltrials.gov. BMC Musculoskelet Disord. 2011 Dec;12(1):278. pmid:22151841
- 12. Li XQ, Yang GL, Tao KM, Zhang HQ, Zhou QH, Ling CQ. Comparison of registered and published primary outcomes in randomized controlled trials of gastroenterology and hepatology. Scand J Gastroenterol. 2013 Dec;48(12):1474–1483. pmid:24131272
- 13. Chahal J, Tomescu SS, Ravi B, Bach BR, Ogilvie-Harris D, Mohamed NN, et al. Publication of sports medicine-related randomized controlled trials registered in ClinicalTrials.gov. Am J Sports Med. 2012 Sep;40(9):1970–1977. pmid:22679295
- 14. Rosenthal R, Dwan K. Comparison of randomized controlled trial registry entries and content of reports in surgery journals. Ann Surg. 2013 Jun;257(6):1007–1015. pmid:23478520
- 15. Ross JS, Mulvey GK, Hines EM, Nissen SE, Krumholz HM. Trial publication after registration in ClinicalTrials.gov: A cross-sectional analysis. PLoS Med. 2009 Sep 8;6(9):e1000144. pmid:19901971
- 16. Killeen S, Sourallous P, Hunter IA, Hartley JE, Grady HLO. Registration rates, adequacy of registration, and a comparison of registered and published primary outcomes in randomized controlled trials published in surgery journals. Ann Surg. 2014 Jan;259(1):193–196. pmid:23732270
- 17. Becker JE, Krumholz HM, Ben-Josef G, Ross JS. Reporting of results in ClinicalTrials.gov and high-impact journals. JAMA. 2014 Mar 12;311(10):1063–1065. pmid:24618969
- 18. Hartung DM, Zarin DA, Guise JM, McDonagh M, Paynter R, Helfand M. Reporting discrepancies between the ClinicalTrials.gov results database and peer-reviewed publications. Ann Intern Med. 2014 Apr 1;160(7):477. pmid:24687070
- 19. Ewart R, Lausen H, Millian N. Undisclosed changes in outcomes in randomized controlled trials: An observational study. Ann Fam Med. 2009 Nov 1;7(6):542–546. pmid:19901314
- 20. Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA. 2009 Sep 2;302(9):977. pmid:19724045
- 21. Hannink G, Gooszen HG, Rovers MM. Comparison of registered and published primary outcomes in randomized clinical trials of surgical interventions. Ann Surg. 2013 May;257(5):818–823. pmid:23407296
- 22. Milette K, Roseman M, Thombs BD. Transparency of outcome reporting and trial registration of randomized controlled trials in top psychosomatic and behavioral health journals: A systematic review. J Psychosom Res. 2011 Mar;70(3):205–217. pmid:21334491
- 23. Walker KF, Stevenson G, Thornton JG. Discrepancies between registration and publication of randomised controlled trials: an observational study. JRSM Open. 2014 May 1;5(5):204253331351768. pmid:25057391
- 24. Pinto RZ, Elkins MR, Moseley AM, Sherrington C, Herbert RD, Maher CG, et al. Many randomized trials of physical therapy interventions are not adequately registered: A survey of 200 published trials. Phys Ther. 2013 Mar 1;93(3):299–309. pmid:23125281
- 25. Anand V, Scales DC, Parshuram CS, Kavanagh BP. Registration and design alterations of clinical trials in critical care: a cross-sectional observational study. Intensive Care Med. 2014 May;40(5):700–722. pmid:24737256
- 26. Nankervis H, Baibergenova A, Williams HC, Thomas KS. Prospective registration and outcome-reporting bias in randomized controlled trials of eczema treatments: A systematic review. J Invest Dermatol. 2012 Dec 1;132(12):2727–2734. pmid:22832489
- 27. Jones CW, Misemer BS, Platts-Mills TF, Ahn R, Woodbridge A, Abraham A, et al. Primary outcome switching among drug trials with and without principal investigator financial ties to industry: A cross-sectional study. BMJ Open. 2018 Feb 8;8(2):e019831. pmid:29439079
- 28. Jones CW, Platts-Mills TF. Quality of registration for clinical trials published in emergency medicine journals. Ann Emerg Med. 2012 Oct 1;60(4):458–464. pmid:22503374
- 29. Huić M, Marušić M, Marušić A. Completeness and changes in registered data and reporting bias of randomized controlled trials in ICMJE Journals after trial registration policy. PLoS ONE. 2011 Sep 21;6(9):e25258.
- 30. Pranić S, Marušić A. Changes to registration elements and results in a cohort of Clinicaltrials.gov trials were not reflected in published articles. J Clin Epidemiol. 2016 Feb;70:26–37. pmid:26226103
- 31. Riedel N, Wieschowski S, Bruckner T, Holst MR, Kahrass H, Nury E, et al. Results dissemination from completed clinical trials conducted at German university medical centers remained delayed and incomplete. The 2014–2017 cohort. J Clin Epidemiol. 2022 Apr;144:1–7.
- 32. Wieschowski S, Riedel N, Wollmann K, Kahrass H, Müller-Ohlraun S, Schürmann C, et al. Result dissemination from clinical trials conducted at German university medical centers was delayed and incomplete. J Clin Epidemiol. 2019 Nov;115:37–45. pmid:31195110
- 33. Salholz-Hillel M, Strech D, Carlisle BG. Results publications are inadequately linked to trial registrations: An automated pipeline and evaluation of German university medical centers. Clin Trials. 2022 Jun;19(3):337–346. pmid:35362331
- 34.
IntoValue Dataset. [accessed 2022 Jan 24]. Available from: https://github.com/maia-sh/intovalue-data.
- 35.
R Core Team. R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; 2022 [accessed 2023 Feb 10]. Available from: https://www.R-project.org/.
- 36. Carlisle BG. ClinicalTrials History Scraper. The Grey Literature; 2020 [accessed 2023 Feb 10]. Available from: https://codeberg.org/bgcarlisle/ClinicalTrialsHistoryScraper.
- 37. Haslberger M, Holst M, Strech D, Hemkens L, Carlisle B. A comprehensive assessment of changes to prespecified trial outcomes, selective reporting, and spin in clinical trials (ASCERTAIN). 2022 Aug 30 [accessed 2023 Aug 29]; Available from: https://osf.io/t3qva.
- 38. Journal Scimago & Rank Country. 2022 [accessed 2022 Oct 31]. Available from: https://www.scimagojr.com/journalrank.php.
- 39. Carlisle BG. Numbat Systematic Review Manager. The Grey Literature; 2014 [accessed 2023 Feb 10]. Available from: https://numbat.bgcarlisle.com.
- 40. Chan AW. Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. Can Med Assoc J. 2004 Sep 28;171(7):735–740. pmid:15451835
- 41. Zarin DA, Tse T, Williams RJ, Califf RM, Ide NC. The ClinicalTrials.gov Results Database—Update and Key Issues. N Engl J Med. 2011 Mar 3;364(9):852–860. pmid:21366476
- 42. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ. 2021 Mar 29;372:n71. pmid:33782057
- 43. Alberer M, Burchard G, Jelinek T, Reisinger E, Beran J, Meyer S, et al. Co-administration of a meningococcal glycoconjugate ACWY vaccine with travel vaccines: A randomized, open-label, multi-center study. Travel Med Infect Dis. 2014 Sep;12(5):485–493. pmid:24873986
- 44. Wiegand A, Sommer A, Nieratschker V, Plewnia C. Improvement of cognitive control and stabilization of affect by prefrontal transcranial direct current stimulation (tDCS). Sci Rep. 2019 Dec;9(1):6797. pmid:31043662
- 45. Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. JAMA. 2004 May 26;291(20):2457. pmid:15161896
- 46. Carlisle BG. Analysis of clinical trial registry entry histories using the novel R package cthist. Naudet F, editor. PLOS ONE. 2022 Jul 1;17(7):e0270909.
- 47. Al-Durra M, Nolan RP, Seto E, Cafazzo JA. Prospective registration and reporting of trial number in randomised clinical trials: global cross sectional study of the adoption of ICMJE and Declaration of Helsinki recommendations. BMJ. 2020 Apr;14:m982. pmid:32291261