Extent of Non-Publication in Cohorts of Studies Approved by Research Ethics Committees or Included in Trial Registries

Background The synthesis of published research in systematic reviews is essential when providing evidence to inform clinical and health policy decision-making. However, the validity of systematic reviews is threatened if journal publications represent a biased selection of all studies that have been conducted (dissemination bias). To investigate the extent of dissemination bias we conducted a systematic review that determined the proportion of studies published as peer-reviewed journal articles and investigated factors associated with full publication in cohorts of studies (i) approved by research ethics committees (RECs) or (ii) included in trial registries. Methods and Findings Four bibliographic databases were searched for methodological research projects (MRPs) without limitations for publication year, language or study location. The searches were supplemented by handsearching the references of included MRPs. We estimated the proportion of studies published using prediction intervals (PI) and a random effects meta-analysis. Pooled odds ratios (OR) were used to express associations between study characteristics and journal publication. Seventeen MRPs (23 publications) evaluated cohorts of studies approved by RECs; the proportion of published studies had a PI between 22% and 72% and the weighted pooled proportion when combining estimates would be 46.2% (95% CI 40.2%–52.4%, I2 = 94.4%). Twenty-two MRPs (22 publications) evaluated cohorts of studies included in trial registries; the PI of the proportion published ranged from 13% to 90% and the weighted pooled proportion would be 54.2% (95% CI 42.0%–65.9%, I2 = 98.9%). REC-approved studies with statistically significant results (compared with those without statistically significant results) were more likely to be published (pooled OR 2.8; 95% CI 2.2–3.5). Phase-III trials were also more likely to be published than phase II trials (pooled OR 2.0; 95% CI 1.6–2.5). The probability of publication within two years after study completion ranged from 7% to 30%. Conclusions A substantial part of the studies approved by RECs or included in trial registries remains unpublished. Due to the large heterogeneity a prediction of the publication probability for a future study is very uncertain. Non-publication of research is not a random process, e.g., it is associated with the direction of study findings. Our findings suggest that the dissemination of research findings is biased.


Introduction
The synthesis of published research in systematic reviews is essential when providing evidence to inform both clinical and health policy decision making. However, its validity is threatened if publications represent a biased selection of all the studies that have been conducted. Publication bias occurs when some types of results (e.g., those that are statistically significant) are reported more frequently or more quickly than others. [1][2][3] Increasingly, the term dissemination bias is used. It reflects that research reporting is not limited to journal publication alone but also comprises other forms of dissemination such as posting results in a trial registry. [4] Dissemination bias, similar to publication bias, results from favoured dissemination of research findings depending on their statistical significance and direction. It may lead to preferential prescribing of newer and more expensive treatments while underestimating the potential harm of drugs that have been in use for only a limited time. Clinical decisions may, therefore, be based on erroneous information. [5] It is obvious that these selection mechanisms violate the fundamental scientific and ethical imperative that findings from all research on humans should be available to advance knowledge. Furthermore, nonpublication of studies implies considerable financial investment by funders without any return. Further down the road, indirect costs incurred due to nonpublication of studies include those by health care providers, health insurances, and patients who all continue to pay for treatments that may not be the most effective ones or may even be harmful.
In response to these concerns, the OPEN Project (To Overcome failure to Publish nEgative fiNdings; www.open-project.eu) was developed with the goal of elucidating the scope of non-publication of studies through a series of systematic reviews and to develop recommendations. [4,[6][7][8] The OPEN Project was funded by the European Commission and conducted by an international working group of methodologists and other experts (see S1 Fig.). Besides evaluating the extent of non-publication of studies, OPEN examined current publication practices of key groups in the field of biomedical research (e.g., funding agencies, pharmaceutical industry, research ethics committees [RECs], trial registries, biomedical journals and regulatory agencies) through surveys and analysis of current policies and guidelines.
Because unpublished studies are hidden from view it is challenging to study dissemination bias. [9] One opportunity for such research is that in virtually all research settings REC approval is required before clinical studies can start. In addition, an increasing number of journals require prospective trial registration as a pre-condition for acceptance of manuscript reporting on studies. Further, any clinical study conducted under FDA regulations in the United States needs to be registered in clinicaltrials.gov (http://clinicaltrials.gov/ct2/manage-recs/ fdaaa#WhichTrialsMustBeRegistered). Thus, study protocols submitted to RECs and study data accessible in trial registries are a resource to identify unpublished studies and evaluate the extent of non-publication of clinical research.
This systematic review investigated the extent to which studies approved by RECs or included in trial registries remained unpublished. To this effect, evidence from methodological research projects (MRPs) following such studies was evaluated and summarised. In addition, we assessed study characteristics that are potentially associated with publication (dissemination bias). The review was conducted according to a protocol published previously. [6] Methods

Systematic literature search
We searched the databases Medline (Ovid), Embase (Ovid), The Cochrane Library and Web of Science from their inception until February 2012. An update search was performed in November 2013. The search strategy was based on combinations of medical subject headings (MeSH) and keywords and was not restricted to specific languages or years of publication. The search strategy used in Medline (Ovid) is presented in S2 Fig. Search strategies for other databases were adapted to meet the requirements of each database while keeping the search algorithm. The searches were supplemented by checking the bibliographies of any eligible articles for additional references. In addition, several experts in the field were contacted and asked to identify additional studies.

Study selection and inclusion criteria
Titles and abstracts were reviewed using specific inclusion criteria (see below). All stages of study selection, data extraction and quality assessment were done independently by two reviewers (CS, LC, PO, LKS). Any disagreement during the selection, extraction, and assessment process was resolved by discussion and consensus.
We included MRPs which reported the proportion of studies published as journal publication after (i) REC approval or (ii) inclusion in trial registries. MRPs evaluating approved studies were included under the assumption that the majority of those studies were completed at the time of the search for peerreviewed journal publications. In the case of multiple publications we extracted data from the MRP with the largest sample size and/or most comprehensive information while using cross-referencing.

Outcomes
Our main outcomes were the overall proportion of studies published as journal articles and time to journal publication after study completion. Thereby, study completion was defined as the last day of follow-up of study participants. If the last day of follow-up was not given, time to publication was calculated based on the time reported in the MRP. To calculate the overall proportion of studies published, we set a minimum follow-up time of 24 months after study completion. In addition, we aimed to identify study characteristics that were associated with an increased likelihood of journal publication and time to publication. [6] We also collated information on costs or other resources which occurred by studies that were not published (if available). For the evaluation of study associations with publication a minimum follow-up time after study completion was not necessary. Outcomes were reported separately for both types of MRPs (RECs and trial registries).

Data extraction and risk of bias assessment
Information on main characteristics of studies were abstracted for each MRP. [6] Internal and external validity of the identified MRPs was evaluated according to pre-defined criteria which were developed considering relevant literature investigating dissemination bias [10] and internal discussion. [6] Criteria for internal validity were: (i) follow-up time between study completion and search for journal publication, (ii) methodology used to identify journal publications, (iii) matching between study protocol or trial registry entry and retrieved journal publication and (iv) adjustment for confounders. External validity was judged based on the status of the study sample (i.e., whether the reported proportion of studies published was calculated based on a sample of completed and/or approved studies) and the sampling method used (i.e., whether a random or selected study sample was considered).
For each criterion an MRP's risk of bias was categorized as high, low or unclear.

Statistical analysis/data synthesis
For both types of MRPs (RECs and trial registries) we separately estimated the proportion of studies published as journal articles using a random effects metaanalysis (DerSimonian-Laird method) based on logit-transformed proportions and their corresponding 95% confidence intervals (CI). Heterogeneity was assessed with the Chi 2 -test and calculation of the I 2 statistic. [11] Given the substantial heterogeneity found we also decided to calculate prediction intervals (PI) -which were not pre-specified in the published protocol -using the method suggested by Higgins et al. [12] Pooled odds ratios (OR) were used to express associations between study characteristics and the likelihood of journal publication. Multivariate analyses of study characteristics were not feasible due to the small number of studies providing this information.
To address potential bias due to approved rather than completed studies, the status of the study sample (completed, on-going and/or approved) was evaluated within a sensitivity analysis.
Time to publication was analysed in two ways: (i) Mean or median time was used only if the proportion of studies published as peer-reviewed journal articles was larger than 50%. Some MRPs calculated time to publication from approval of studies to journal publication, others from study completion to journal publication. Due to these differences in definitions we refrained from pooling time-to-publication estimates. (ii) The proportion of studies published up to fixed time points (e.g., 6, 12, 18, 24, 36 months) was extracted from included MRP publications (e.g., from published Kaplan Meier curves). For each time point we performed a random-effects meta-analysis using logit-transformed proportions.

Characteristics of included MRPs
The main characteristics of included MRPs of both types are presented in Table 1 and Table 2.

Risk of bias
Results of the methodological quality assessment are presented in Table 3 and Table 4, respectively.
MRPs following studies after REC approval All of the included MRPs fulfilled the 24-month follow-up criterion. However, twelve MRPs based their follow-up time on studies which were approved but not necessarily completed. [13-16, 18, 20-25, 27] Although these MRPs fulfilled the 24-month follow-up criterion, we judged them to have an unclear risk of bias. The methodology used to identify journal publications was adequate in all but two MRPs. [23,24] Three MRPs performed adequate matching between protocol and retrieved journal publications. [13,26,28] However, in most MRPs this criterion was not applicable because identification of journal publications relied solely on author contacts. None of the MRP publications reported on adjustment for confounding factors when calculating proportions of published studies in specific subgroups or calculating measures of association between likelihood of publication and subgroup characteristics.

MRPs following studies after inclusion in trial registries
Fourteen MRPs following studies included in trial registries had a follow-up time between study completion and search for full publication of 24 months or more. [

Proportion of studies published
After REC approval, the proportion of studies published ranged from 26% to 76% in 17 MRPs with a follow-up of 24 months or more, including 5112 studies (Fig. 2, Table 5). The prediction interval was 22% to 72%; the heterogeneity among individual estimates was substantial (I 2 594.4%, p,0.0001). If one combined the individual estimates even so the pooled estimate would be 46.2% (95% CI 40.2-52.4).
After trial registration, the proportion of studies published ranged from 23% to 76% in 14 MRPs with a follow-up of 24 months or more, including 12660 studies (Fig. 3, Table 5). The prediction interval was 13% to 90%; again the heterogeneity among individual estimates was substantial (I 2 598.9%, p,0.0001). If one combined the individual estimates even so the pooled estimate would be 54.2% (95% CI 42.0-65.9).
In a sensitivity analysis we excluded those MRPs that were based on a cohort of approved [13-16, 18, 20-25, 27] or initiated [45] studies. In the resulting sample of completed studies a pooled proportion of studies published would be similar: 46.3% (95% CI 41.0-51.6; I 2 581.1%, p,0.0001; based on five MRPs following     Extent of Non-Publication of Research Studies studies after trial registration), respectively. [44-46, 50, 51, 54, 55] It should be noted that three of these MRPs had insufficient follow-up time for searching full publications [44,51,54] or included on-going studies. [45] Factors associated with publication Table 6 summarizes factors associated with journal publication of studies. Four MRPs following studies approved by RECs compared studies with statistically significant results (p,0.05) and studies with non-significant results. [9,18,25,26]   The pooled OR for publication of studies with statistically significant results (vs. non-significant) was 2.8 (95% CI 2.2-3.5). Also studies with positive results defined as experimentally better or clinically relevant had higher (but statistically not significant) odds of journal publication than studies with negative results (pooled OR 3.1; 95% CI 0.9-11.0; two MRPs). [25,32] Two of the MRPs that followed studies after REC approval [20,30] and three of the MRPs that followed studies after registration [39,42,43] investigated the association of different study designs with publication (i.e., randomised controlled trials versus observational studies). In both types of MRPs, randomised controlled trials had a greater odds of publication than observational studies (OR 2.0; 95% CI 1.3-3.3 and 1.2; 95% CI 1.0-1.5, respectively). A post-hoc analysis including MRPs that followed studies after trial registration revealed that phase-III trials were more likely to be published than phase-II trials (pooled OR 2.0; 95% CI 1.6-2.5). [37, 38, 40-43, 47, 50, 52, 53] In MRPs that followed studies after REC approval, multicentre studies were more likely to be published than single centre studies (pooled OR 1.5; 95% CI 1.0-2.4; four MRPs). [18,28,30,34] We also found that research funded by governments was more frequently published than research funded by the industry (pooled OR 2.2; 95% CI 1.7-2.9; eight MRPs following studies after trial registration). [37,39,41,42,46,50,52,53] But no difference in the probability of publication between basic and human research was identified (pooled OR 1.1; 95% CI 0.6-2.1; two MRPs). [24,30] There were also no significant differences for Extent of Non-Publication of Research Studies national versus international studies (OR 1.3; 95% CI 0.5-3.8) in one MRP following studies after inclusion in trial registries [40] and for studies with sample sizes larger (versus smaller) than the cohort's median sample size (OR 1.2; 95% CI 0.8-1.6) in another such MRP. [42] Other potential factors associated with journal publications could not be derived from the included MRPs. In addition, none of the MRPs reported on costs or use of other resources due to studies that were not published.  [26] confirmed that the time to publication is significantly associated with the direction of the results. Median time to full publication was 25 months in studies with positive outcomes and 38.5 months in those with negative results.

Principal findings
Overall, only about half of clinical and preclinical studies approved by RECs or included in trial registries are published as full journal articles; however estimates vary largely resulting in wide prediction intervals. For randomised controlled trials a pooled overall proportion of studies published would be somewhat larger (60.3%; 95% CI 45.4-73.6). Accordingly, prediction of the probability of publication for a future study is very uncertain. We also found evidence for dissemination bias in that studies with statistically significant results were more likely to be published than those without (pooled OR 2.8; 95% CI 2.2-3.5). This association is consistent with the finding that studies with positive results defined as experimentally better or clinically relevant were more likely to be published than studies with negative results though not reaching statistical significance (pooled OR 3.1; 95% CI 0.9-11.0). In addition, phase-III trials -which might be more successful than early-phase trials -were more likely to be published than phase-II trials (pooled OR 2.0; 95% CI 1.6-2.5). Also, randomised controlled trials which are considered as the ''gold standard'' design for a clinical study are published more often than observational studies (pooled OR 2.0; 95% CI 1,3-3,3). The reason for this finding could be that medical journals prefer to publish randomised controlled trials. But there may also be a tendency by study authors not to not write up results of observational studies, in particular when they are negative.

Strengths and weaknesses of this review
The findings of our systematic review are based on a thorough and comprehensive literature search for the available evidence on dissemination bias. We considered two types of MRPs which tracked studies from time of inception, thus including 39 individual MRPs evaluating more than 20,000 studies. For both types, the evidence on dissemination bias was consistent suggesting that publications over the last 20 years are an incomplete and biased subset of research findings. We conducted our systematic review following a pre-specified protocol thus preventing that any substantial post-hoc changes remain undisclosed. [6] Because Extent of Non-Publication of Research Studies a registry for methodological reviews is not yet available, this protocol was not prospectively registered, but previously published in an open-access journal. [6] Our systematic review may have some limitations. We identified a large number of MRPs but associations between study characteristics and journal publication had not been reported in most of these publications. Therefore, not all pre-specified subgroup analyses stated in the protocol could be performed. For example, it was not possible to collate enough data on sex and rank of lead investigator or language of publication to investigate these factors in association with non-publication. We could not determine with certainty whether the MRP authors carried out additional analyses that ultimately were not reported (as authors were not contacted personally), thus selecting outcomes for publication. Furthermore, the aggregated data for publication probability over time refer to less than five studies at most time points. Accordingly, publication probabilities at given time points have to be interpreted cautiously. The reported estimates can only give a rough picture of the publication course after REC approval or trial Extent of Non-Publication of Research Studies registration. No standard methodology is available to assess study quality of the types of research projects we considered for our review. Therefore, we devised a tool to assess internal and external validity of the identified evidence. A sensitivity analysis for MRPs with high risk of bias was planned initially [6] but not performed due to the overall low quality of the identified MRPs. When we calculated the overall proportion of studies published as journal articles we only included studies with an arbitrarily defined minimum follow-up time of 24 months after study completion. Therefore, the proportion of studies published may be underestimated in some MRPs because journal publication may have occurred later. We included MRPs which investigated approved or on-going rather than completed studies. A sensitivity analysis excluding those MRPs showed that the proportion of studies published was similar. In addition, limited data on potential risk factors (e.g., follow-up time, language of included studies) made it impossible to further explore the large heterogeneity observed in our data. We also acknowledge shortcomings (like inaccurate estimation of heterogeneity) of random effects models meta-analysis -as carried out in our systematic reviewwith a small number of included studies.

Comparison with other systematic reviews
In a Cochrane Methodology Review full publication of results initially presented in abstracts was examined combining data from 79 MRPs; the weighted mean full publication proportion was 44.5% (95% CI 43.9-45.1). [10] In this review, survival analyses combining aggregated data resulted in an estimated publication rate at nine years of 52.6% for all types of studies, 63.1% for randomised controlled trials and 49.3% for other types of study designs. In addition, the review showed a significant association of positive study results (defined as any statistically significant result) with full publication. Other factors associated with full publication were randomised trial study design and funded research. Despite the different criteria for inclusion of MRPs (REC approval/trial registration versus meeting presentation) their findings were consistent with our results.
The extent of dissemination bias in different types of research projects was also investigated by Song et al. 2009. [57] The authors identified 12 MRPs that followed up research from inception (studies approved by RECs or registered by research funding bodies), four MRPs that included trials submitted to regulatory authorities, 28 MRPs that assessed the fate of studies presented as conference abstracts, and four MRPs that followed manuscripts submitted to journals. This review concluded that dissemination bias related to direction of study results mainly occurs before the presentation of findings at conferences and the submission of manuscripts to journals. [57] A recent systematic review of studies limited to randomised trials confirmed the existence of dissemination bias and outcome reporting bias, although meta-analysis was not conducted due to the differences between included studies. [58] In addition, a Cochrane Review concluded that trials with positive findings are published more often and more quickly than trials with negative findings. [59] Despite differences in types of study cohorts or MRPs included, all these reviews were consistent with our body of evidence in concluding that a study with positive findings is more likely to be published than a study with negative results. One might speculate that journals prefer publishing reports with positive rather than non-positive results or that investigators do not submit reports of studies with negative results.

Implications for policy makers and further research
Overall, the scientific literature represents an incomplete subset of research findings. Due to the large heterogeneity, prediction of the probability of publication for a single study is very uncertain. Our findings clearly confirm that (non-)publication is not a random process and the likelihood of publication is associated with the direction of study findings. When results are not published or are published selectively based on the direction or the strength of the findings, healthcare professionals and consumers of healthcare cannot base their decisions on the full body of current evidence. This ignorance can lead to the use of ineffective or harmful interventions and to waste of scarce health-care resources. For example, when unpublished studies were included in a meta-analysis, the antidepressant reboxetine was shown to have more adverse effects but no better efficacy than placebo for treatment of major depression -a different finding from that when only published studies were included. [60] The inability to make evidence-informed decisions impacts the healthcare system at various levels: First, dissemination bias is at odds with the ethical responsibility towards patients to use all research to advance medical knowledge and improve their care. Second, if treatment effects are overestimated this may result in patients receiving treatments that may be more harmful or less efficacious than previously believed. Finally, non-publication of studies results is deleterious because a considerable part of the funds available for research are spent without return. Additional costs include those incurred by health care systems and individual patients who continue to pay for treatments that may not be as effective or efficient as commonly thought. Although the full extent of financial impact of non-publication of studies is currently unknown, the waste of funds is likely to be high. [61,62] The creation of clinical trial registers and the prospective publication of detailed study protocols with explicit outcome descriptions and analysis plans should help to combat dissemination bias. The recent AllTrials campaign has proposed that ''all trials past and present should be registered, and the full methods and the results reported'' (http://www.alltrials.net/). In addition, researchers should be encouraged and supported to present their studies at conferences and to proceed until full publication.
Nevertheless, dissemination bias exists and is currently invalidating findings in systematic reviews and meta-analyses when only published studies are considered. There is no excuse for study results to go unpublished and there is a huge public health benefit from obtaining a complete picture of what has been found in all studies to-date.