Evaluation of reporting quality of randomized controlled trials in patients with COVID-19 using the CONSORT statement

Objective To evaluate the reporting quality of randomized controlled trials (RCTs) regarding patients with COVID-19 and analyse the influence factors. Methods PubMed, Embase, Web of Science and the Cochrane Library databases were searched to collect RCTs regarding patients with COVID-19. The retrieval time was from the inception to December 1, 2020. The CONSORT 2010 statement was used to evaluate the overall reporting quality of these RCTs. Results 53 RCTs were included. The study showed that the average reporting rate for 37 items in CONSORT checklist was 53.85% with mean overall adherence score of 13.02±3.546 (ranged: 7 to 22). The multivariate linear regression analysis showed the overall adherence score to the CONSORT guideline was associated with journal impact factor (P = 0.006), and endorsement of CONSORT statement (P = 0.014). Conclusion Although many RCTs of COVID-19 have been published in different journals, the overall reporting quality of these articles was suboptimal, it can not provide valid evidence for clinical decision-making and systematic reviews. Therefore, more journals should endorse the CONSORT statement, authors should strictly follow the relevant provisions of the CONSORT guideline when reporting articles. Future RCTs should particularly focus on improvement of detailed reporting in allocation concealment, blinding and estimation of sample size.


Introduction
With the development of evidence-based medicine, randomized controlled trials (RCTs) became the gold standard to compare the effectiveness of different interventions [1]. It can avoid possible bias in clinical trial design, balance confounding factors and improve the effectiveness of statistical tests [2]. Therefore, a complete and accurate report will enable readers to fully assess the authenticity of results [2]. If the RCT report is unsatisfactory, the validity of trials will be reduced [3], which may adversely affect the results of meta-analyses and recommendations for clinical practice [4].
To improve the reporting quality of RCTs, the CONSORT (reporting standards of randomized controlled trials) statement was developed in 1996 [5], revised in 2001 [6] and 2010 [7]. The updated CONSORT statement includes 25 entries that provide specific guidance for RCT reports. Since many of the 25 items were subdivided into two subitems, the list actually consists of 37 items [7]. It is currently known to be endorsed by over 600 biomedical journals and endorsed by several prominent editorial organizations including the International Committee of Medical Journal Editors (ICMJE) and the World Association of Medical Editors (WAME) [8]. After update of CONSORT statement, the overall reporting quality of RCTs has improved, but there were still many RCT reports with various deficiencies [9,10]. A study by Yao et al. of 65 RCTs related to ophthalmic surgery showed the mean CONSORT score was 8.9 (range 3.0-14.7) and the reporting quality was quite low [11]. A study of 71 RCTs regarding herbal interventions showed that the compliance rates of CONSORT checklist in these RCTs ranged from 0% to 97.18% [12]. Poor reporting quality of RCTs is the major barrier to evidence-based practices, as it can distort the available evidences in the medical literature, and prevent clinical decision-makers from obtaining true results from trials [13]. Thus, it is critical for researchers to build wellreported standard of RCTs. On the one hand, it can ensure the validity of clinical trials and the authenticity and scientificity of research results [14]. On the other hand, it is conducive to conducting secondary studies, such as meta-analyses and systematic reviews [15].
With the outbreak of the COVID-19 in 2019, in order to quickly and effectively control the epidemic, a large number of RCTs regarding COVID-19 have been published. However, the reporting quality of these RCTs is unclear, to our knowledge, no study has specifically evaluated the reporting quality of RCTs regarding patients with COVID-19.
The primary objective of our study was to assess the reporting of RCTs regarding patients with COVID-19 and analyze possible related factors, so as to provide theoretical basis for subsequent studies and meta-analyses.

Ethical review
Ethical approval was not necessary for this study, as the study did not involve patients and included RCTs can be traced from databases.

Study selection
Studies meeting the following criteria were enrolled in the study: (1) Randomized controlled trial. (2) The confirmed or suspected patients of COVID-19 according to the diagnostic criteria of "the latest Clinical guidelines for novel coronavirus" issued by the World Health Organization (WHO). (3) Interventions related to patients or suspected patients.
Exclusion criteria included: (1) Animal experiments, reviews, systematic reviews, case reports. (2) Repeated publications. (3) The abstract or full text is not available.
The titles of the retrieved article were imported into the Endnote X9 and screened by 2 reviewers independently. We first reviewed the title and abstract of each article and decided to regard its appropriateness for inclusion. In case of doubt, we downloaded full texts to judge whether an article was RCT. Any disagreement was solved by consensus.

Data extraction
Two authors independently extracted the general characteristics and reporting data of included studies into Excel, any discrepancy was resolved through discussion. The general characteristics include continent of first-author, number of authors, sample size, age and type of participants, the type of interventions, journal impact factor and journals' endorsement of the CONSORT statement.

Assessment of reporting quality
The CONSORT statement was chosen as a tool to assess reporting quality of these RCTs [7]. We assessed the compliance of each RCT by 25 items of CONSORT statement, each checklist item and subitem was answered with "yes" or "no". According to the above items, the coincidence rate of each item of 53 studies was counted one by one. A point for the item being granted if all sub-sections were answered as yes, if one of two subsections was reported, a score of 0.5 was awarded, then total score of each study was calculated [1]. Items 3b, 6b, and 14b are not necessarily applicable, if relevant, the article will be graded according to the above guidelines, if the article did not apply to this item, no points was deducted [3].

Statistical analysis
Descriptive statistics was performed to describe general characteristics of 53 studies and the reporting rate of each checklist item/subitem. We used the k coefficient to determine the degree of agreement between reviewers. T-test and ANOVA were used for univariate analysis, multiple linear regression analyses were used to determine the association between potential predictors and reporting quality. All significant predictors in the univariable analyses were entered individually into a multivariable analysis. No significant violation of normality was found in assessments of the residuals. Chi-square tests and Fisher's exact tests were used to analyze the relationship between journals impact factor and endorsement of CONSORT guideline, T-test was used to analyze the differences between different journal submission requirements and CONSORT score. For all analyses, the statistical significance level was set at P<0.05. Statistical analysis was carried out using SPSS version 21.

Search results
Initially, 8700 articles were obtained, excluding duplicates, 6,922 studies was remained. After screening the titles and abstracts, 198 potentially eligible articles were identified. Subsequently full text of each article was retrieved, and 53 RCTs were confirmed for further assessment.

Agreement of reviewers
In the pilot study, inter-observer concordance for article selection had a kappa score of 0.78, which was 0.89 after resolving all disputed items by a discussion with the third reviewer (ZJX), suggesting that inter-observer reliability was almost perfect.

Characteristics of included studies
Among the 53 RCTs, the first author of most RCTs were from Asia and accounted for 62.26%, half of the studies had a sample size greater than 100, 50 studies (94.34%) used drug interventions. 22 (42.51%) studies had an impact factor of more than 10. Nearly half of the studies were published in journals that did not explicitly require authors to follow the CONSORT statement. As showed in Table 1.

The evaluation result of the CONSORT checklist
The study showed that the average reporting rate for 37 items in CONSORT checklist was 53.85%, with 31.35% for the methodological section. Items with the reporting rate of more than 80% were abstract, background, eligibility criteria for participants, outcomes, statistical methods, participant flow chart, baseline data, limitations, generalisability and funding. Except for non-essential items, the remaining items with the reporting rate of less than 20% were trial design, sample size, allocation concealment, implementation, blinding, ancillary analyses and protocol. Only 7 studies (13.21%) reported the methods of masking concealment and 9 studies (16.98%) reported the details of the blinding. Table 2 outlined the reporting frequency of each checklist item.

Factors of effecting overall reporting score
Univariate results showed that CONSORT score was associated with sample size (P<0.001), types of intervention (P = 0.017), journal impact factor (P<0.001), and endorsement of CON-SORT statement (P<0.001) ( Table 1). Table 3 displayed the results of multiple linear regression analysis. The four predictors were entered into a multivariable model (constant = 7.779, R 2 = 0.533, adjusted R 2 = 0.494, P<0.001). Among these, journal impact factor (P = 0.006), and journal with endorsement of CONSORT statement (P = 0.014) still persisted as noticeable predictors of reporting quality.
Journal impact factors less than 10 and more than 10 had statistically significant differences in endorsement of the CONSORT statement (P<0.001) ( Table 4). There was a statistically significant difference in reporting quality between journals that required submission of CON-SORT checklist and those that did not (P = 0.031) ( Table 5).

Discussion
To our knowledge, this is the first study to evaluate the reporting quality of RCTs regarding patients with COVID-19 by the CONSORT statement [7], this will provide important information for clinical decision makers. Our study showed that the overall reporting quality of RCTs regarding patients with COVID-19 was suboptimal. It is clear that the reporting quality of these studies needs to be improved, particularly in terms of methodology section. Although the CONSORT statement was established to ensure the completeness and accuracy of RCT reports, our study found that some authors still reported their data selectively in a biased way, with the average reporting rate of 53.85%, none of the RCTs provided complete information as required, similar results have been found in other studies [11,[16][17][18]. This could be due to the large number of patients with COVID-19 emerging in a short period of time, in order to present positive results of various treatment regimens to readers as soon as possible, researchers may have paid more attention to the results of study than report specifications.
The key items with lower reporting rates were mainly concentrated in trial design, sample size, allocation concealment, and blinding. The complete description of trial design can provide readers with accurate research ideas and enable readers to better evaluate the trial results [19], but our study showed that only 5 (9.43%) studies reported the type of trial design. The neglect of two most important items in the methodology section (allocation concealment and details of blinding) was particularly worrisome, as these items are important information to ensure the authenticity of results [20]. Only 7 (13.21%) studies described the methods of allocation concealment, 9 (16.98%) studies reported the details of blinding. We found that some authors tend to write "single" or "double" blind rather than specifying exactly who were unaware of treatment identities. The low reporting rates of these items may reflect the lack of relevant knowledge of researchers to some extent, because the report of sequence generation and concealment of allocation need to have certain knowledge of clinical research methodology [21]. There was evidence that trials with inadequate or unclear allocation concealment overestimated the treatment effects up to 7% [22], and a meta-epidemiological study of blinding showed that unblinded RCTs overestimated the outcome effect by 0.56 standard deviations [23]. In spite of this, some studies still reported these items poorly [11,17,24]. A previous study showed that only 12% and 21% of RCTs reported the details of allocation concealment and blinding in four high-impact general medical journals [25]. What's more, a study of acute herpes zoster showed that none of RCTs reported blinding and masking [16]. Therefore, it is urgent for researchers to pay attention to the design and implementation of allocation concealment and blinding, in order to minimize measurement bias and improve the reporting quality of RCTs. The estimation of sample size can avoid false negative results between the intervention and control groups [26]. Our study showed that only 5 studies (9.43%) reported the details of sample size, which was similar to the results in the field of vascular and endovascular surgery, herpes zoster and plastic surgery [9,16,27]. Lack of the reporting of sample size estimation can prevent readers to verify the validity of the trial results [27], researchers should attach great importance to the report of sample size estimation, so as to provide scientific evidence for future clinical studies.
Our study showed that reporting quality was associated with journal impact factor and endorsement of the CONSORT statement, similar results have been found in other areas [15,[27][28][29]. Studies have shown that 80% of RCTs published in journals that do not endorse CON-SORT guideline had defective reporting specifications [30]. We found that journals with impact factor of more than 10 have a higher endorsement of the CONSORT guideline. However, even though some journals have endorsed the CONSORT guideline, the reporting quality of published RCTs was still suboptimal, this may be due to a problem with the entry point for paper submission. We found that some journals only encourage researchers to follow the CONSORT guideline, but do not require authors to submit a CONSORT checklist, which was more common in journals with lower impact factor, this may be because journals with lower impact factor have less stringent policies for accepting and publishing papers. In contrast, journals with higher impact factor have stricter requirements for submission of articles, requiring authors to upload CONSORT checklist when submitting RCT, which forces researchers to write RCTs according to standard reporting specifications. Our study showed that articles that uploaded the CONSORT checklist had higher reporting quality than those that did not have strict submission requirements, therefore, strict submission requirements are the premise to improve the reporting quality of RCTs.
Academic journals are the main media for carrying and publishing papers, the relevant provisions in the manuscript contract will directly affect the quality of published papers [28,30]. Thus, we suggest that editors should carefully assess whether their journals' submission requirements are normative, journals should not only endorse the CONSORT guideline, but more importantly require authors to upload the CONSORT checklist as key material for the initial screening when submitting their RCTs. Similarly, peer reviewers should check the completeness and accuracy of CONSORT checklist when reviewing RCTs, editorial boards should also increase their oversight of the entire process from submission to publication, articles of lower reporting quality will not be published. However, in addition to problems in the whole process of submission requirements, review and publication, another potential reason for the poor reporting quality of RCTs is that the CONSORT statement was not publicized enough, researchers lack the awareness of report guideline [31]. A survey of the authors of 101 studies found that only 3% acknowledged the importance of RCT reports and followed the CONSORT guideline when writing papers [31], this suggests that improving researchers' awareness of the CONSORT guideline is critical to improve the reporting quality of RCTs. Therefore, on the one hand, journals should vigorously promote the CONSORT guideline and can add relevant knowledge of RCT reports to their subscription feeds. On the other hand, research institutions should also increase training in these problems to improve the reporting quality of RCTs.

Limitations of this study
There are some limitations in our study. Although the literature retrieval, screening and quality evaluation were carried out simultaneously by two researchers, there was still some subjectivity. In addition, we only included RCTs of patients with COVID-19 from 4 databases, it could not represent the overall reporting quality of RCTs of COVID-19.

Conclusions
The primary objective of our study was to provide readers a broad overview of the reporting characteristics of RCTs regarding patients with COVID-19. The overall reporting quality of these RCTs was suboptimal, thereby diminishing their potential usefulness, and it can not provide valid evidence for clinical decision-making and systematic reviews. Better reporting quality was associated with higher journal impact factor and endorsement of the CONSORT statement. More journals should endorse the CONSORT statement, authors should strictly follow the relevant provisions of the CONSORT guideline when writing the paper. Future RCTs should particularly focus on improvement of detailed reporting in allocation concealment, blinding and estimation of sample size.