Outcome Discrepancies and Selective Reporting: Impacting the Leading Journals?

Background Selective outcome reporting of either interesting or positive research findings is problematic, running the risk of poorly-informed treatment decisions. We aimed to assess the extent of outcome and other discrepancies and possible selective reporting between registry entries and published reports among leading medical journals. Methods Randomized controlled trials published over a 6-month period from July to December 31st, 2013, were identified in five high impact medical journals: The Lancet, British Medical Journal, New England Journal of Medicine, Annals of Internal Medicine and Journal of American Medical Association were obtained. Discrepancies between published studies and registry entries were identified and related to factors including registration timing, source of funding and presence of statistically significant results. Results Over the 6-month period, 137 RCTs were found. Of these, 18% (n = 25) had discrepancies related to primary outcomes with the primary outcome changed in 15% (n = 20). Moreover, differences relating to non-primary outcomes were found in 64% (n = 87) with both omission of pre-specified non-primary outcomes (39%) and introduction of new non-primary outcomes (44%) common. No relationship between primary or non-primary outcome change and registration timing (prospective or retrospective; P = 0.11), source of funding (P = 0.92) and presence of statistically significant results (P = 0.92) was found. Conclusions Discrepancies between registry entries and published articles for primary and non-primary outcomes were common among trials published in leading medical journals. Novel approaches are required to address this problem.


Introduction
Journal of American Medical Association (JAMA). Two authors (PSF, DK) identified RCTs for inclusion by independent searching of the electronic archives of these journals. Associated registry entries were identified within the abstract or methods section of the published manuscript.
Data from eligible studies were extracted independently and entered on pre-piloted standardized forms. Initial calibration was performed between the two researchers (PSF, DK) on 10 articles. Disagreements were resolved by discussion or, if necessary, with adjudication by a third researcher (NP). Information obtained included number and type of primary and nonprimary outcomes based on explicit definitions within both the registry entries and the final publication. Changes in outcomes including addition, removal or redefinition of outcomes were recorded. An example of definition change would be a registry indicating, in a hypothetical study of a hypoglycemic agent, that the primary outcome was glycemic control assessed at 1 month, while glycemic control assessed at 2 months was reported in the study. However, if the registry entry reported the outcome without the time-point and the time-point was added in the report, this was noted but was not recorded as a discrepancy overall as additional detail is typical within trial reports.
In addition, data relating to changes to selection criteria, randomization and blinding procedures, analysis (per-protocol or intention-to-treat), and timing of assessments, sample size, funding, ethics and sponsorship were obtained. Estimates and associated confidence intervals where available were extracted for the primary outcome. Other characteristics including the trial register used and whether the research involved a single or multiple centers were also extracted.
Descriptive statistics were calculated for all included studies for the following variables: journal of publication, number of research centers, significance of the primary outcome, and registry used. Data were collected in relation to possible discrepancies between registry entries and final reports both for primary and non-primary outcomes. Other discrepancies in relation to trial design, randomization procedures, blinding, approach to analysis (intention to treat or per-protocol), incorporation of subgroup analyses, sample size, funding and registration details, timing of registration (retrospective or prospective), and allocation ratio were identified. Cross-tabulations were undertaken to investigate associations between discrepancies and timing of registration, type of sponsorship, and presence of significant results. All statistical analyses were conducted with STATA version 12.1 software (Stata Corporation, College Station, Texas, USA).
Discrepancies related to primary outcomes were identified in 18% (n = 25) of the RCTs assessed. The definition of the primary outcome had changed in 15% (n = 20). In relation to non-primary outcomes, differences were found for 64% (n = 87) with similar frequency of both omission of pre-specified non-primary outcomes (39%) and introduction of new non-primary outcomes (44%) in published studies. No discrepancies related to either the intervention or design of the trial were identified. However, inclusion and exclusion criteria were altered in 24% (n = 33). Randomization procedures and approach to analysis were rarely outlined in the registries (96%). Subgroup analyses were described in the final report but not in the registry commonly (54%, n = 74). Certain discrepancies were found more commonly in individual journals including changes relating to randomization, blinding, subgroup analyses, funding and sponsorship (Table 1).
Regarding primary outcomes, no association was found for outcome change (report vs registry entry) or omission in the final report and registration timing (prospective or retrospective; P = 0.11), source of funding (P = 0.92) and presence of statistically significant results (P = 0.92). Similarly, no association was observed for these variables concerning changes in Discrepancy in randomization 0. Stated neither in registry, nor in final report Discrepancy in population analysis 0. 16  Stated in registry, not in final report Discrepancy in subgroup analysis 0.  non-primary outcomes or where the primary outcome was omitted from the final publication (Table 2). An association between change in inclusion criteria and source of funding was noted with company-sponsored trials more likely to show changes (P = 0.014). No other statistically significant findings were observed.

Discussion
The presence of inconsistency between clinical trial reports and registry entries has previously been confirmed in major medical and surgical journals and within drug trials [6][7][8][9][10][11]. This study confirms the pattern alluded to in these studies; however, an association between timing of registration, funding source and presence of significant outcomes and discrepancies relating to primary and non-primary outcomes was not found. Even in high impact journals, which all endorse trial registration in guidelines for authors, selective reporting remains problematic. A previous analysis of selective reporting within JAMA and BMJ highlighted a discrepancy rate of 19% and 47%, respectively [16]. Analogous rates of discrepancy were exposed in the present analysis suggesting that merely endorsing and promoting registration is not sufficient; further action is required to produce more meaningful improvements. It appears that primary outcomes are being handled with greater fidelity than is the case with non-primary outcomes. Nevertheless, a sizeable proportion (18%) of studies were exposed as having discrepancies between primary outcomes on registry entries and in the final report. While this figure is considerably lower than that identified in a recent review of surgical studies, where rates of up to 49% were found [9], rates ranging from 18% to 43% have been found in previous studies [6][7][8]10]. This problem stemmed relatively evenly from either omission of primary outcomes outlined on the registry, de novo introduction of previously unmentioned outcomes and downgrade of primary to non-primary outcomes. This distribution is in accordance with other studies reporting the previous two issues with a frequency of 8% to 18% [6,8] and the latter at between 4% and 22% [6,7]. This, therefore, confirms that trials published more recently in the most reputed journals are susceptible to analogous levels of selective reporting as was revealed up to 5 years ago [6][7][8].
In keeping with previous findings, the rate of disagreement between non-primary outcomes listed on registry entries and in final reports was 64%. The same figure was previously   identified for surgical journals [6]; however, even higher levels (70%) have been exposed [7]. The most pervasive issues in the present report related to either omission (39%) or de novo introduction (44%) of non-primary outcomes in the published reports. Upgrade of non-primary outcomes to primary was relatively rare (2%); this figure corresponds with previous research alluding to a rate of 3% for this discrepancy [7], although levels of up to 14% have been reported [6]. No association was found between discrepancies in outcome reporting and statistical significance of the results. This finding may reflect a lack of power to identify a true difference. It may, however, be possible that outcomes are changed within clinical trials following genuine post hoc decisions, for example, based on pilot studies, recruitment or due to advice from data monitoring committees. Notwithstanding this, it is important that post hoc changes of this nature are delineated within the published manuscript, ideally accompanied by an explanation of the reasons for the change.
In respect of sample size, a discrepancy was found in 45% of trials. This figure is in general keeping with previous studies, which have alluded to differences in the region of 48% to 73% of entries [6,17]. The veracity of the sample size calculations and the information contributing to the calculated values were not assessed in the present study; however, previous research has exposed problems in this respect [18,19]. Consequently, further efforts are required both in  relation to the clarity and conduct of sample size estimation in clinical trials. It is particularly important, however, that sample size estimation is delineated in trial reports due to the risk of biased post hoc decisions in an effort to demonstrate that clinical trials possessed the requisite power to demonstrate clinically relevant differences where they are identified [20]. A number of factors described in the final report were rarely delineated in the registry entry. Subgroup analyses were referred to in only 6% of registry entries, while analyses were reported in the final paper in 54% without previous mention in the register. Similarly, randomization procedures, ethical approval status (74%) and mode of population analysis (88%) were rarely outlined in registry entries. This pattern may be overcome by stipulating mandatory entries in relation to each of these areas on trial registers or by providing additional space to provide further relevant but non-essential information. It has been demonstrated that completion of mandatory components is almost universal on recognized databases (e.g. ClinicalTrials.gov) [21], while optional fields are completed with significantly lower frequency. Completion rates were just 66% and 56%, respectively, for primary and non-primary outcomes. While evidence pointing to improved outcomes favoring sponsors of published pharmaceutical studies and a recognized susceptibility to publication bias exists [22], in the present study no association between either primary or non-primary outcome change or omission and either registration timing or source of funding was found.
Despite the adoption, promotion and endorsement of reporting guidelines, there remain significant shortcomings [2]. The emphasis should, therefore, shift to implementing changes rather than allowing relatively passive submission, peer review and editorial consideration, while expecting improvements in reporting and conduct. While stipulation of mandatory elements on registry databases is effective, it may be possible to implement a registry modeled on CONSORT items completed prospectively to stimulate optimal trial design and clarity at the protocol stage. The Standard Protocol Items: Recommendations for Intervention (SPIRIT) checklist has recently been developed with the intention of clarifying the items required during protocol reporting. Within these 33 items relating to all aspects of trial design, including ethics and dissemination are mentioned [23]. If the SPIRIT guidelines become embraced as a standard technique for outlining protocols of clinical trials, it is likely that these omissions will be rectified, particularly if registry entries could be made to reflect more detailed protocols rather than being restricted to more limited fields. Biomedical journals are beginning to adopt the SPIRIT statement and as acceptance increases it is possible trial registries may be modeled more closely on SPIRIT guidance [24]. A further option might be to append the registry entry to final submissions for inspection during the peer review process with authors encouraged to outline discrepancies between them to promote greater clarity and candor.
Limitations of this study include its restriction to a six-month period and to a subset of just five journals. These journals were included as they are prominent, well regarded and have the highest impact factors. It was therefore hoped that these might represent best practice with more significant discrepancies arising in less prominent journals. While the latter appears to apply to methodological quality of both systematic reviews and RCTs, there is no conclusive evidence that this extends to selective outcome reporting [14,15]. It would be intuitive to expect, however, that journals not stipulating the need for trial registration are likely to present similar or higher rates of discrepancy. Moreover, registry entries do not at this stage correspond to trial protocols as protocols are not usually publically available. It is possible, therefore, that information on registries was truncated deliberately in an effort to comply with registry policies. The benefit of encouraging more complete registry entries, with an onus on mandatory completion is, therefore, clear. Finally, it had been hoped to investigate the association between selective reporting and observed estimates; this was not possible due to the relatively low number of studies examined. It may be interesting, however, to investigate whether selective reporting of trial outcomes is associated with distortion of effect estimates likely resulting in a greater preponderance of positive outcomes for de novo primary or non-primary outcomes, which were not outlined in the registry.

Conclusions
A high rate of discrepancies between registry entries and published articles for primary and non-primary outcomes were identified among a sample of trials published in leading medical journals. Selective outcome reporting, therefore, appears to be a persistent issue. Novel approaches are required to address this issue throughout biomedical research.