Can We Depend on Investigators to Identify and Register Randomized Controlled Trials?

Purpose To reduce publication bias, systematic reviewers are advised to search conference abstracts to identify randomized controlled trials (RCTs) conducted in humans and not published in full. We assessed the information provided by authors to aid identification of RCTs for reviews. Methods We handsearched the Association for Research in Vision and Ophthalmology (ARVO) meeting abstracts for 2004 to 2009 to identify reports of RCTs. We compared our classification with that of authors (requested by ARVO 2004–2006), and authors’ report of trial registration (required by ARVO 2007–2009). Results Authors identified their study as a clinical trial for 169/191 (88%; 95% CI, 84–93) RCTs we identified for 2004, 174/212 (82%; 95% CI, 77–87) for 2005 and 162/215 (75%; 95% CI, 70–81) for 2006. Authors provided registration information for 107/172 (62%; 95% CI, 55–69) RCTs for 2007, 103/153 (67%; 95% CI, 60–75) for 2008, and 126/171 (74%; 95% CI, 67–80) for 2009. Most RCT authors providing a trial register name specified ClinicalTrials.gov (276/312; 88%; 95% CI, 85–92) and provided a valid ClinicalTrials.gov registration number (261/276; 95%; 95% CI, 92–97). Based on information provided by authors, trial registration information would be accessible for 48% (83/172) (95% CI, 41–56) of all ARVO abstracts describing RCTs in 2007, 63% (96/153) (95% CI, 55–70) in 2008, and 70% in 2009 (118/171) (95% CI, 62–76). Conclusions Authors of abstracts describing RCTs frequently did not classify them as clinical trials nor comply with reporting trial registration information, as required by the conference organizers. Systematic reviewers cannot rely on authors to identify relevant unpublished trials or report trial registration, if present.


Introduction
Randomized controlled trials (RCTs) that are used to inform systematic reviews are typically identified through searching electronic databases, such as PubMed, EMBASE, and the Cochrane Central Register of Controlled Clinical Trials. Because only 60% of trials initially reported as conference abstracts are ever published in full [1], systematic review authors who neglect to search the grey literature, especially conference abstracts, may fail to identify relevant unpublished trials. Further, because of publication bias, the tendency to publish results based on the strength or direction of findings, systematic reviews that focus only on trials reported in full may present a biased representation of existing evidence.
Clinical trial registration was proposed in the 70's and 80's, in part as a mechanism of combating reporting biases [2.3], and the notion slowly gained ground in the 90's and 2000's [4,5,6], especially following support from the International Committee of Medical Journal Editors (ICMJE) [7]. Today, registration has been resoundingly endorsed by the scientific community [8,9], and systematic review authors are examining the potential for making use of register information to complement what is available in the published literature [10].
With 100 to 200 abstracts describing clinical trials published annually, the Association for Research in Vision and Ophthalmology (ARVO) annual meeting is an important source of reports of RCTs for the Cochrane Eyes and Vision Group (CEVG). We have searched the ARVO meeting books or the ARVO online abstract repository from 1990 forward for reports of RCTs and controlled clinical trials, to contribute to CEVG's specialized database of trial reports. Because clinical trials represent only 2 to 3% of ARVO abstracts published each year, and completing the searches requires about 45 hours per year [11], we began a project in 2004 with the initial objective of determining whether it is possible to develop a valid and reliable system for author classification of clinical trials. If so, such a system would obviate the need for labor-intensive handsearching. In response to our request, ARVO conference organizers added a definition and check-off box to the 2004 abstract submission form to indicate whether the submitted abstract described a ''human clinical trial''. Our overall study objective was to assess the reliability of information provided by ARVO abstract authors that might be used to aid in identification of relevant abstracts for systematic reviews. We evaluated whether authors correctly classify abstracts describing RCTs as clinical trials, and as an adjunct to this, determined where investigators are registering their RCTs.

Identification of Randomized Controlled Clinical Trials
We searched the ARVO annual meeting abstract book (in 2004 we searched a CD-ROM; in 2005 through 2009 we searched online at http://www.arvo.org) to identify abstracts describing RCTs and controlled clinical trials for inclusion in the CEVG specialized trial register. For this study, we include only abstracts reporting RCT findings. Abstracts were classified as RCTs if ''individuals (or other units) followed in the trial were assigned prospectively to one of two (or more) alternative forms of health care using random allocation'' [13]. Because the original purpose of our handsearch of the abstracts was to identify RCTs for the CEVG trial register, and reviewing all abstracts for each year is labor intensive, a single individual (RWS) completed the initial handsearch; this was done in the year following each annual meeting. We downloaded and printed a hard copy of each abstract classified as an RCT.

Author Classification of Abstract as a Trial
Following each annual meeting, the meeting organizers provided us with an electronic list of abstract program numbers that were identified by the author(s) as a ''human clinical trial'' (2004) or ''human controlled clinical trial' ' (2005 and 2006) report (see Box for definitions). For the 2007 meeting, the meeting organizers provided an electronic list of the program number, trial register name, and registration number for all abstracts with trial registration information. For the 2008 and 2009 meetings, the organizers provided a hardcopy lists that included this information. We searched for and obtained paper copies of the abstracts on the organizers' lists that we had not already identified as RCTs (2004 through 2009).

Reference Standard
We compared the list of abstracts provided by the meeting organizers with the abstracts that we had previously classified as RCTs. Any abstract on the list that the handsearcher had not classified as an RCT was re-evaluated. If the handsearcher agreed that the report described an RCT, then that abstract was included in the reference standard for that year.
Thus, for each year, there were two groups of studies: Abstracts in the reference standard: $ Classified as a clinical trial by the author and as an RCT by the handsearcher; $ Not classified as a clinical trial by the author, but classified as an RCT by the handsearcher.
Abstracts not in the reference standard: $ Classified as a clinical trial by the author, but not classified as an RCT the handsearcher. $ Not classified as a clinical trial by the author, nor classified as an RCT by the handsearcher.
A second handsearcher (PCS) reviewed the classification of 100% of abstracts that were included in the reference standard, i.e., abstracts that were classified by the handsearcher as an RCT. A 10% sample of abstracts classified by the author as a clinical trial, but not as an RCT by the first handsearcher, and a 5% sample of abstracts not classified as a clinical trial by the author nor as an RCT by the handsearcher were reviewed by another handsearcher (KD or AE). Abstracts in question were re-reviewed by the lead author for a decision or by both readers to arrive at consensus.

Characteristics of Trial Registration Information
We classified the type of organization recorded in the trial registration field (trial register, ethics board/institutional review board, regulatory agency, local authority, or other) for conference abstracts presented in 2007, 2008, and 2009. For trials in the reference standard that were reported by the author as registered at ClinicalTrials.gov, we verified the information by entering the trial registration number provided in the ClinicalTrials.gov search webpage [http://ClinicalTrials.gov].

Data Analyses
We entered into an Access database the program number and year of presentation of each abstract describing an RCT that we identified by handsearching the conference books, and imported the program number and year of presentation received from the meeting organizers for the abstracts from the years 2004 to 2006 as well as the program number, year of presentation, register name, and registration number for abstracts presented in years 2007 to 2009. We calculated the proportion of RCTs in the total number of abstracts presented at the conference for 2005-2009, and the proportion of RCTs in our reference standard that were correctly identified by the author, performing separate comparison analyses by year. We tabulated information recorded in the trial register fields, including trial register ''name'' and valid trial register number (yes/no). Results are presented as point estimates with 95% confidence intervals (CI). Two individuals performed all calculations independently.

Classification of Abstracts
The first handsearcher classified 1,121 of the abstracts presented at ARVO from 2004 to 2009 as RCTs. A second person read all 1,121 abstracts and classified 1,101 as RCTs, 10 as questionable, and 10 as ''not RCT''. After re-review the 2 readers reached consensus on the 20 articles where there was initial disagreement;, resulting in 1,114 abstracts that were included in the reference standard as RCTs.
The first handsearcher classified 1529 abstracts, originally categorized by the author as a clinical trial, as ''not RCTs''. A second person read a 10% sample (153) of these abstracts and classified 3 as RCTs and 150 as ''not RCTs.'' The first handsearcher, re-reviewed the 3 abstracts classified as RCTs by the second reader, and decided to keep the original classification of ''not RCT.'' The first handsearcher classified 32,565 abstracts, which had not been classified as a clinical trial by the author, as ''not RCT.'' A second person read a 5% sample (1628 abstracts), and classified all as ''not RCT.'' , years requiring trial registration information rather than a checked box to designate a clinical trial (see Table 1). However, the majority of studies that authors identified were of other types of study design rather than RCTs; only 841/2372 (35%; 95% CI, 34-37) abstracts classified as a clinical trial by authors described an RCT. The remaining abstracts described non-randomized controlled clinical trials, uncontrolled clinical trials, studies not involving humans, or trials in which participants were assigned to a treatment group based on some participant characteristic (e.g., severity of disease) (see Table 2).

Registration of Clinical Trials
Authors provided trial registration information at abstract submission for 797 abstracts (see Table 3 Almost all RCT authors who provided a trial register name specified ClinicalTrials.gov (see Table 4). When we checked the trial registration number provided by the author for RCTs registered in ClincalTrials.gov, we found a valid number for 95% of RCTs (261/276; 95% CI, 92-97). Invalid numbers included reports that registration was pending (n = 7), or a number that yielded no results upon searching the ClinicalTrials.gov database or was clearly not a valid registration number (n = 8) (e.g., ''0.

Discussion
We were disappointed that asking authors to identify their abstracts as describing trials is not reliable, since it could have helped to avoid laborious handsearching of conference abstracts required to identify all RCTs for systematic reviews [14]. If we depended solely on author classification of abstracts, we would not have identified a large proportion of trials since authors correctly identified only 75% of RCTs in our reference standard.
One of the reasons for the development of trial registers is to provide information about all initiated trials, including those remaining unpublished. Although registration does inform the public about the existence of trials, most investigations to date have demonstrated that registration details (e.g., study design, protocol and contact information) are less than optimal for inclusion in systematic reviews [10,15,16,17]. Our findings might suggest that a significant proportion -about one third of RCTs (32% [160/496] [95% CI, 28-36]) -reported at ARVO 2007-2009 were not registered publicly. This may be because authors did not consider their study eligible for registration. One possible explanation for why RCT investigators did not register their study is they do not recognize study designs, including those that require registration. For example, a recent study in China found that a high proportion of authors stating in their article that their trial was randomized revealed that they did not fully understand the principles of randomization when queried directly [18]. Our own experience showed that authors of 9/40 ARVO and American Academy of Ophthalmology 1988-89 meeting conference abstracts, who reported use of randomization, responded that assignment to treatment groups was not random when asked directly [19]. Similarly, investigators of other study designs (e.g., non-randomized trial) may not recognize their study design or that it requires registration. Alternatively, authors may have registered their trials, but may not have entered the required trial registration information on the abstract submission form. Lack of compliance with abstract submission requirements related to trial registration would not be surprising, as compliance with required trial registration is also problematic for journal articles [20,21]. If including trial registration information is to be meaningful, it should be accurate and complete. Journal editors and conference organizers may wish to require that authors receive formal registration instruction and may need to monitor the submission process more closely.
The fact that authors do not always recognize their trials as RCTs and the apparent lack of trial registration has broad implications for identifying all the evidence for informing systematic reviews and healthcare decision making. If authors have no intention of publishing their findings and registration is not required by law [22] or a research ethics review board, or as a condition of funding by an agency such as the National Institutes of Health [23], there is little incentive to register a trial. Community and commercial research ethics review boards in the US may not require registration [24]. Consistency in requiring trial registration through legislation, across funding and regulatory agencies, and research ethics boards, would likely increase registration and benefit the public and systematic reviewers alike.

Limitations
Our findings are limited to abstracts submitted to a single conference from 2004 through 2009 and may not apply to other years or areas of clinical research. The relatively low rate of correct identification of RCTs across the years searched implies that there is an ongoing problem with author classification of RCTs. Although we observed a high ''false positive'' rate for trial registration, this result may be due to the fact that registration is required for many types of ''clinical trials'' not just RCTs. We did not expect all abstracts with trial registration to be RCTs, but we did expect that all RCTs would be registered. We had hoped that a set of abstracts with trial registration would provide us an enriched and comprehensive source of RCTs to reduce the time and effort required to search the conference abstracts.

Implications
Our findings lead us to be somewhat pessimistic about authors being able to identify their own studies as randomized clinical trials. In addition, RCT investigators may not be registering their trials or reporting trial registration. Thus, it is unlikely that systematic reviewers would be able to use the author-classification of study design to identify ARVO abstracts describing RCTs.