Assessing reporting quality of randomized controlled trial abstracts in psychiatry: Adherence to CONSORT for abstracts: A systematic review

Background Reporting quality of randomized controlled trial (RCT) abstracts is important as readers often make their first judgments based on the abstracts. This study aims to assess the reporting quality of psychiatry RCT abstracts published before and after the release of Consolidated Standards of Reporting Trials for Abstracts (CONSORT-A) guidelines. Methods MEDLINE/PubMed search was conducted to identify psychiatric RCTs published during 2005–2007 (pre-CONSORT) and 2012–2014 (post-CONSORT). Two independent reviewers assessed abstracts using a 18-point overall quality score (OQS) based on the CONSORT-A guidelines. Linear regression analysis was conducted to analyze factors associated with reporting quality. Results Among 1,927 relevant articles, 285 pre-CONSORT and 214 post-CONSORT psychiatric RCT abstracts were included for analysis. The mean OQS improved from 6.9 (range: 3–13; 95% confidence interval (CI): 6.7–7.2) to 8.2 (range: 4–16; 95% CI: 7.8–8.5) after the CONSORT-A guidelines. Despite improvement, methods of randomization, allocation concealment, and funding source remained to be insufficiently reported (<5%) even after the release of CONSORT-A. High-impact general medical journals, multicenter design, positive outcome, and structured abstracts were associated with better reporting quality. Conclusions The reporting quality in psychiatric RCT abstracts, although improved, remains suboptimal. To improve reporting quality of psychiatry RCT abstracts, greater efforts by both investigators and journal editors are required to enhance better adherence to the CONSORT-A guidelines.


Introduction
The reporting quality of RCT abstracts is important as the readers often make their initial assessment of articles based on the abstracts. In addition, due to the high volume of annual publications, as well as limited time and resources, clinicians may even make clinical decisions based solely on the information provided in the abstracts.
Recognizing the importance of well-informed abstracts and the need for improvements in the reporting of abstracts, the Consolidated Standards of Reporting Trials for Abstracts (CON-SORT-A) was developed and finalized in January 2008. Many health care journals endorse the use of the CONSORT-A to provide guidance to authors about the necessary details and clarity required for reporting in abstracts [1]. However, low endorsement rates of editorial policies, including CONSORT, have been found within psychiatry journals [2,3]. Han et al. (2009) study, which used the CONSORT statement 2001 intended for full articles, found that the reporting quality of psychiatry RCTs, although improved, remained suboptimal even after the release of the statement [4].
To our knowledge, no studies have assessed the reporting quality of RCT abstracts in the field of psychiatry. Thus, this study aims to assess the overall reporting quality of psychiatry RCT abstracts published before and after the release of the CONSORT-A, and to determine the trial characteristics associated with reporting quality.

Inclusion and exclusion criteria
All abstracts of RCTs published in English and conducted on human subjects were included. A study was defined as an RCT if the participants were assigned to interventions that were described as random, randomly allocated, randomized, or if randomization was mentioned, and if a control group was included. The control group could receive a placebo, usual care, or a comparator. All other study designs, such as non-randomized studies, follow-up studies of previously published trials, studies analyzing more than two trials, crossover studies, diagnostic tests and biomarker analyses, economic analyses, safety analyses, reviews, protocols, editorials, and letters, were excluded.

Data collection
Extracted data included the year of publication, target disease, intervention type ('pharmacological', 'psychological treatment', or 'other' e.g., electroconvulsive therapy), name of the intervention, name of the journal, impact factor of the journal, number of authors, funding source, region of publication, number of conducting centers (single or multicenter), trial outcomes (positive, negative, or unclear), abstract format (structured or unstructured), sample size, as well as journal CONSORT endorsement, and word count restriction. The target diseases were classified in accordance with the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) [5]. With regard to trial outcomes, a trial was defined as positive when the experimental arm was considered superior to the standard arm in superiority trials, not inferior in non-inferiority trials, or equivalent in equivalence trials. Beyond this, all other studies were defined as trials with unclear outcomes. The region of publication was determined from the address of the first authors' institution.
The CONSORT-A guideline was developed in 2008 and provides a list of essential items that authors should consider when reporting the abstracts of RCTs (http://www.consortstatement.org/extensions?ContentWidgetId=562). The CONSORT-A checklist items include identification of the study as randomized, trial design, participants, interventions, details of the trial's objectives, clearly defined primary outcome, methods of randomization, blinding, number of participants randomized and analyzed in each group, the effect of interventions on primary outcomes and harms, trial conclusions, trial registration, and funding. Two reviewers independently assessed the adherence to the CONSORT-A guidelines by assigning 'yes' or 'no' to each item on the checklist. Both reviewers underwent training in evaluating RCTs using the CONSORT checklist, and the definition of each checklist item was discussed before the study was conducted. A pilot study was performed with randomly selected abstracts to assess interobserver agreement using Cohen's kappa statistics, and to resolve any discrepancies in the data extraction process.

Rating of overall reporting quality
The overall quality score (OQS) was adopted from the methodology used by previous studies.
[ [6][7][8][9][10][11] The OQS consists of 18 items modified from the CONSORT-A guidelines. Each item was given equal weight and a score of one. OQS% was then calculated by dividing the number of items met by the number of total items to generate a percentage score for easier interpretation and comparison with the results of previously published studies in the field.

Statistical analysis
Descriptive statistics were performed on the characteristics of RCT articles in pre-COSORT-A and post-CONSORT-A periods. The overall number and proportion (%) of RCT abstracts that met each of the CONSORT for Abstracts checklist items were determined. Pearson chi-square analyses or Fisher exact tests, where applicable, were conducted for each of the CONSORT items to compare pre-and post-CONSORT-A RCT abstracts. A mean OQS was generated for each RCT abstract on a scale of 0 to 18 and a mean OQS% was calculated along with the 95% confidence interval (CI). A t-test was performed to compare the mean OQS% of article characteristics between pre-CONSORT-A and post-CONSORT-A periods. To analyze factors associated with higher OQS%, a linear regression analysis was performed. All tests for statistical significance were two-tailed, with the threshold set at 0Á05. All analyses were performed using SPSS software (version 23Á0; IBM Corporation, NY, USA).

Results
Our search strategy identified 1,136 pre-CONSORT-A and 791 post-CONSORT-A RCT abstracts. After exclusion, 285 pre-CONSORT-A and 214 post-CONSORT-A RCT abstracts were included for analysis (Fig 1). The majority of pre-CONSORT-A RCTs (23Á5%) addressed 'schizophrenia and psychotic disorders' as their target disease, while most post-CONSORT-A RCTs (19Á6%) addressed 'depressive disorders.' The highest number of both pre-and post-CONSORT-A RCTs was published in the Journal of Clinical Psychiatry (31Á2% and 29Á0%, respectively; Table 1). Although the top 20 high-impact psychiatry journals were included in the initial search, abstracts from 18 psychiatry journals were finally included.

Reporting of general items
The quality assessment of psychiatry RCT abstracts is shown in Fig 2. A significantly greater number of studies in the post-CONSORT-A period stated 'randomized' in the title compared to the pre-CONSORT-A period (56Á5% vs. 78Á5%; P<0.001). The trial design was described by only a small portion of the abstracts in both the pre-CONSORT-A (13Á3%) and post-CON-SORT-A (15Á0%) periods.

Reporting of trial methodology
Nearly all studies reported the eligibility criteria for participants, with 96.1% of studies reporting it in the pre-CONSORT-A period and 94Á4% in the post-CONSORT-A period. Reporting on the specific setting of data collection was relatively low; however, its reporting frequency increased significantly in the post-CONSORT-A period (25Á3% vs. 41Á1%; P<0.001). A majority of RCT abstracts reported on the interventions assigned for each group (pre-CONSORT-A, 93Á7%; post-CONSORT-A, 88Á8%), and defined a specific objective or hypothesis (pre-CONSORT-A, 86Á3%; post-CONSORT-A, 88Á8%). More than half of the abstracts defined the primary outcome, and reporting on this criterion significantly increased over each period (50Á5% vs. 62Á6%; P = 0Á007). Few abstracts reported on the method of randomization (pre-CONSORT-A, 0Á0%; post-CONSORT-A, 1Á9%), and there were no reports on allocation concealment. Blinding details regarding participants were also rarely reported (pre-CONSORT-A, 4Á2%; post-CONSORT-A, 8Á4%), and 59Á6% of pre-CONSORT-A and 46Á3% of post-CONSORT-A studies referred to blinding methods using words such as 'single' or 'double'.

Reporting of trial results
The number of participants randomized (pre-CONSORT-A, 35Á4%; post-CONSORT-A, 38Á8%) and the number analyzed (pre-CONSORT-A, 17Á2%; post-CONSORT-A, 18Á2%) were inadequately reported by studies from both periods. Reporting on trial outcomes significantly improved with the reporting of primary outcome results increasing from 50.5% pre-CON-SORT-A to 66Á8% post-CONSORT-A (P<0Á001), and details regarding effect size and precision also increased from 10Á2% to 27Á1% (P<0Á001). Interestingly, reporting on harm was found to be higher before the publication of the CONSORT statement (pre-CONSORT-A, 44Á2%; post-CONSORT-A, 33Á6%; P = 0Á017).

Reporting of trial conclusions
Almost all of the studies reported conclusions in both the pre-and post-CONSORT-A periods (pre-CONSORT-A, 99Á6%; post-CONSORT-A, 99Á5%). Reporting on trial registration significantly improved from 9Á1% pre-CONSORT-A to 50Á5% post-CONSORT-A (P<0Á001). Reporting on the funding source also improved from no studies in the pre-CONSORT-A period to 3Á3% of studies in the post-CONSORT-A period reporting funding (P = 0Á003).

OQS%
The mean OQS on a 0 to 18 scale was 6Á9 (range: 3-13; 95% CI: 6Á7-7Á2) for pre-CONSORT-A and 8Á2 (range: 4-16; 95% CI: 7Á8-8Á5) for post-CONSORT-A studies. The mean OQS% improved significantly from 38Á4% (95% CI: 37Á0-39Á8) to 45Á4% (95% CI: 43Á5-47Á3) after the publication of CONSORT for Abstracts ( Table 2). The mean OQS% was higher in the abstracts in each kind of interventions; however, significant improvements were not observed in those with both pharmacological and psychological interventions. The highest mean OQS% was observed in the high-impact general medical journals (69Á0%; 95% CI: 61Á5-76Á6). The mean OQS% for journals with an impact factor of five and over increased significantly (between 5 and 10: 45Á9%; 95% CI: 43Á8-47Á9; more than 10: 47Á0%; 95% CI: 42Á9-51Á2) while those with an impact factor of less than five did not show significant improvement (35Á1%; 95% CI: 29Á6-40Á6). In addition, structured abstracts indicated a significant improvement in mean OQS% (46Á9%; 95% CI: 44Á9-48Á9). The improvements in abstract reporting did not depend on CON-SORT endorsements, as significant improvement was observed both with endorsements (45Á9%; 95% CI: 43Á9-47Á9) and without (45Á4%; 95% CI: 40Á7-50Á1). The annual mean OQS% Factors associated with reporting quality Table 3 shows the results of the linear regression analysis. The univariate analysis showed that abstracts published in high-impact general medical journals, with an impact factor of 5 to 10, or greater than 10, and with CONSORT endorsement, were more likely to have better reporting quality. In addition, those with industry or both industry and non-industry funding, a multicenter design, positive or negative outcomes, a structured format, and large sample size were more likely to have a higher mean OQS%. In the multiple linear regression model, high-impact general medical journals, number of authors greater than 7, a multicenter design, positive or negative outcomes, a structured abstract, and word count limit greater than 250 or no word limit, were associated with better reporting quality. On the other hand, studies with psychological interventions were associated with lower reporting quality compared to studies with pharmacological intervention.

Discussion
Randomized controlled trials (RCTs) are the gold standard for clinical trials, providing the most credible evidence for intervention efficacy, and are major sources of evidence-based research. Accurate and complete reporting of trial results is essential for readers to understand how a clinical trial was conducted and judge its validity. Reporting quality of randomized controlled trial (RCT) is also important especially in light of new NIH guidelines on rigor,  reproducibility, and transparency [12]. As a field, psychiatry can only be taken more seriously with better reporting of outcomes and better transparency on trial design. Although reporting quality itself does not necessarily correlate with study quality, inadequate reporting has the potential to bias the estimates of treatment effects in RCTs [7,13]. The reporting quality of RCT abstracts is also of great importance because the readers often make decisions to read the full article based on the abstracts. The reporting quality of psychiatry RCT abstracts improved significantly after the publication of the CONSORT for Abstracts. However, despite improvement, the reporting quality of abstracts remains suboptimal with a post-CONSORT-A mean OQS% of 45Á4%. Similar studies. reported a mean OQS% of 58Á6-62Á5% in the field of dentistry, and an annual increase in mean OQS% to more than 50% after the publication of the CONSORT for Abstracts in oncology [6,14,15]. The main finding of our study is that articles published in medical journals and studies with pharmacological intervention have better adherence to CONSORT for Abstract guidelines than psychiatry-based journals and studies with non-pharmacological intervention. Thus, although significant increase in the reporting quality of psychiatry abstracts was noted, it is still not impressive. In our study, abstracts published in high-impact general medical   journals, including BMJ, JAMA, Lancet, and NEJM, showed a much higher mean OQS% of 69Á0% after the release of the CONSORT for Abstracts compared to those published in psychiatry journals, with OQS% of 43.8%. This may be explained by the higher endorsement of editorial policies including the CONSORT guidelines by journals, and the rigorous peer review process conducted before publication. Interestingly, impact factor was not a significant factor affecting abstract reporting quality, while high-impact general medical journal type was. This shows that more efforts are required to endorse and promote adherence to CONSORT guidelines within psychiatry journals. The endorsement rates for psychiatry journals are reported to be low, and of the 18 high-impact psychiatry journals included in our study, only 12 journals endorsed CONSORT, with a 66Á7% endorsement rate [2]. Overall, less than a quarter of the included abstracts reported on the trial design, method of randomization, allocation concealment, blinding, and sources of funding. Such results are consistent with previous findings [6,16,17]. Of note, CONSORT items, including method of randomization, allocation concealment, and source of funding, were not reported at all in the pre-CONSORT-A period, and even after the CONSORT-A statement they were reported in less than 5% of the abstracts. Prior studies have found inadequate reporting of allocation concealment to be associated with exaggerated treatment effects [18,19]. It must be noted that a lack of description of important methodological items could cause bias and influence the internal validity of the trial [14]. In addition, according to the CONSORT guidelines the funding source is an important piece of information for the reader and should be reported in abstracts [20]. Our study found very few psychiatry RCT abstracts that reported funding source information, although it should be clearly stated in the abstract.
The use of a structured abstract was associated with better abstract reporting quality, which is in agreement with the findings of previous studies [6,21]. Structured abstracts can improve readability, and facilitate an easy assessment of the information reported in the abstract. Unfortunately, over 30% of the psychiatry journals included in this analysis did not recommend a structured abstract format, even after the publication of the CONSORT for Abstracts. Therefore, psychiatry journals need to actively recommend the use of structured abstracts, as it can improve the quality of reporting in abstracts. Also, 250-300 words are considered sufficient to address all items of CONSORT for Abstracts [7,19]. Our results support this with abstracts published in journals with word count limit greater than 250 or no word limit at all, achieving higher adherence to the items. Similar to the study by Mbuagbaw et al., multicenter studies were found to have a better abstract reporting quality [22]. The exact reason behind such a phenomenon is unknown; however, it can be assumed that multicenter studies are of a larger scale and involve a greater number of researchers, possibly leading to better reporting of abstracts. This study has several limitations. First, our study is not fully representative of all published psychiatry RCTs. This is because we extracted only certain abstracts from psychiatry journals, and excluded studies with primary outcomes other than changes in clinical symptoms for psychiatric diseases, such as studies on genetic psychiatry and abnormal morphology of the brain. However, our results may sufficiently reflect the overall trends in the abstract reporting of psychiatry RCTs because our study included the top 20 psychiatry journals. Second, our study analyzed the adequacy of reporting based on the CONSORT checklist, without considering whether the content of the full article was accurately reflected in the abstract. This was beyond the scope of our study. Thus, further research that assesses the accuracy of reporting in abstracts is needed. Finally, the type of intervention could have affected our findings. In general, in studies with non-pharmacological treatments, it is difficult to provide a sham intervention and often impossible to blind patients and care providers [23]. Accordingly, extended abstract reporting guidelines for studies with non-pharmacological interventions are required to improve the quality of reporting.
In conclusion, this study demonstrated that the quality of reporting in psychiatry RCT abstracts, although improved, remains suboptimal after the publication of the CONSORT guidelines. In particular, the finding of inadequate reporting on important methodological items could result in biased interpretations. Based on our findings, health professionals and policymakers should be careful when interpreting the information reported in psychiatry RCT abstracts. Moreover, additional efforts from both researchers and editors in the field of psychiatry appear to be necessary for better adherence to the CONSORT for Abstract guidelines and the provision of informative abstracts for readers.