Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Assessment of the Reporting Quality of Randomized Controlled Trials on the Treatment of Diabetes Mellitus with Traditional Chinese Medicine: A Systematic Review

  • Ping Wang,

    Affiliation The Affiliated Traditional Medical Hospital, Xinjiang Medical University, Urumqi, Xinjiang Uygur Autonomous Region, China

  • Qin Xu,

    Affiliation The Affiliated Traditional Medical Hospital, Xinjiang Medical University, Urumqi, Xinjiang Uygur Autonomous Region, China

  • Qi Sun ,

    Cums6140@Hotmail.com

    Affiliation The Affiliated Traditional Medical Hospital, Xinjiang Medical University, Urumqi, Xinjiang Uygur Autonomous Region, China

  • Fang-fang Fan,

    Affiliation The Affiliated Traditional Medical Hospital, Xinjiang Medical University, Urumqi, Xinjiang Uygur Autonomous Region, China

  • Xue-rui Guo,

    Affiliation The Affiliated Traditional Medical Hospital, Xinjiang Medical University, Urumqi, Xinjiang Uygur Autonomous Region, China

  • Fei Guo

    Affiliation The Affiliated Traditional Medical Hospital, Xinjiang Medical University, Urumqi, Xinjiang Uygur Autonomous Region, China

Abstract

Background

After the publication of the CONSORT 2010 statement, few studies have been conducted to assess the reporting quality of randomized clinical trials (RCTs) on treatment of diabetes mellitus with Traditional Chinese Medicine (TCM) published in Chinese journals.

Objective

To investigate the current situation of the reporting quality of RCTs in leading medical journals in China with the CONSORT 2010 statement as criteria.

Methods

The China National Knowledge Infrastructure (CNKI) electronic database was searched for RCTs on the treatment of diabetes mellitus with TCM published in the Journal of Traditional Chinese Medicine, Chinese Journal of Integrated Traditional & Western Medicine, and the China Journal of Chinese Materia Medica from January to December 2011. We excluded trials reported as “animal studies”, “in vitro studies”, “case studies”, or “systematic reviews”. The CONSORT checklist was applied by two independent raters to evaluate the reporting quality of all eligible trials after discussing and comprehending the items thoroughly. Each item in the checklist was graded as either “yes” or “no” depending on whether it had been reported by the authors.

Results

We identified 27 RCTs. According to the 37 items in the CONSORT checklist, the average reporting percentage was 45.0%, in which the average reporting percentage for the “title and abstract”, the “introduction”, the “methods”, the “results”, the “discussion” and the “other information” was 33.3%, 88.9%, 36.4%, 54.4%, 71.6% and 14.8%, respectively. In the Journal of Traditional Chinese Medicine, Chinese Journal of Integrated Traditional & Western Medicine, and the China Journal of Chinese Materia Medica the average reporting percentage was 42.2%, 56.8%, and 46.0%, respectively.

Conclusions

The reporting quality of RCTs in these three journals was insufficient to allow readers to assess the validity of the trials. We recommend that editors require authors to use the CONSORT statement when reporting their trial results as a condition of publication.

Introduction

Randomized controlled trials (RCTs), when appropriately designed, conducted, and reported, represent the gold standard for evaluating healthcare interventions. However, randomized trials can yield biased results if they lack methodological rigor. To accurately assess a trial, readers of a published report require complete, clear, and transparent information on its methodology and findings. Unfortunately, assessments frequently fail because authors neglect to provide clear and complete descriptions of such critical information [1].

Reporting quality assessment is therefore the first stage in a critical literature review. In 1996, the Consolidated Standards for Reporting of Trials (CONSORT) group produced the CONSORT statement, an evidence-based approach to help improve the quality of reporting RCTs. The group published a revised statement in 2001. The methodological factors included in the CONSORT statement were selected due to empirical evidence indicating their importance. The CONSORT statement has been supported by a growing number of medical and healthcare journals and editorial groups, including the International Committee of Medical Journal Editors (ICMJE), the Council of Science Editors (CSE), and the World Association of Medical Editors (WAME) [2]. Subsequently, the expanding body of methodological research reported the refinement of CONSORT 2010. Over the past 16 years, a number of CONSORT recommendations (including updates and extensions) for the publication of RCTs have been developed.

Since their introduction, the quality of published RCTs has improved significantly in journals endorsing the CONSORT criteria. For example, analyses of the cardiothoracic and general surgery literature indicate substantial improvements in the reporting of RCTs after the CONSORT criteria were endorsed by their principal journals [3].

Traditional Chinese medicine (TCM) plays an important role in maintaining the health of the Chinese population (particularly before the introduction of western medicine into China) and its benefits are gradually being recognized worldwide [4]. The rapid increase in diabetes mellitus is becoming a serious health threat in all parts of the world. With its distinctive traditional medical opinions and natural medicines mainly originating from herbs, TCM has shown promise in clinical practice for the treatment of diabetes mellitus and its complications. Based on a large number of chemical and pharmacological research studies, numerous bioactive compounds have been found in Chinese medicinal plants for the treatment of diabetes [5]. Traditional Chinese herbal medicines have been used for a long time to treat diabetes, and many controlled trials have been carried out to investigate their efficacy [6].

Although the quality of reporting in RCTs in the medical sciences has been discussed, the quality of reporting in RCTs on the treatment of diabetes mellitus with TCM published in the Chinese language has not yet been assessed following publication of the CONSORT statement (2010 version). The aim of this study was to assess the completeness of reporting RCTs evaluating TCM in the treatment of diabetes mellitus published in three core Chinese medical journals based on the CONSORT 2010 checklist [7], and to provide recommendations for improving them in the future.

Materials and Methods

Search Strategy

The China National Knowledge Infrastructure (CNKI) electronic database was used to search all articles published between 01 January and 31 December 2011 reporting an RCT (i.e., a trial in which the assignment of participants to interventions was described by the words random, randomly, randomized, or randomization) in the full text. We only obtained the full text of RCTs on the treatment of diabetes mellitus with TCM, which were published in 2011, took place in China and included Chinese citizens.

The following search equation was used: “random, randomly, randomized, or randomization” AND “diabetes mellitus”(in the full text) AND (“2011/01/01″[PDat]: “2011/12/31″[PDat])) OR “Journal of Traditional Chinese Medicine” OR “Chinese Journal of Integrated Traditional & Western Medicine” OR “China Journal of Chinese Materia Medica”.

Selection of Journals and RCTs

We selected three CSCD (Chinese Science Citation Database) (2011–2012) -indexed Chinese medical journals (Journal of Traditional Chinese Medicine, Chinese Journal of Integrated Traditional & Western Medicine, and the China Journal of Chinese Materia Medica) with a top ranking impact factor as our source to identify RCTs. We selected these three core journals as they are thought to publish the majority of RCTs with TCM and each trial is reported in the Chinese language.

All potentially eligible full text articles were collected. These articles were then evaluated using the eligibility criteria.

We chose this time period to better assess the reporting quality of RCTs on the treatment of diabetes mellitus with TCM, which was after the publication of the CONSORT 2010 statement.

Eligibility Criteria

Two reviewers searched the three journals and selected potentially relevant articles after screening the titles and abstracts independently. In the case of uncertain eligibility, the full text was screened.

The clinical studies included in this review met the following criteria: 1) the report was published between 01 January and 31 December 2011; 2) the report was published as an original article of a RCT; 3) adult participants (18 years or older) with diabetes mellitus; and 4) TCM intervention including Chinese herbal medicines and Chinese proprietary medicines.

We excluded the trials reported as “animal studies”, “in vitro studies”, “case studies”, or “systematic reviews”.

For all remaining articles, the full text of the article was obtained and reviewed.

Data Extraction

The quality of reporting in RCTs was evaluated using a 37-item modified CONSORT checklist. The revised 25-item CONSORT checklist contains some items with multiple parts. These were separated into individual items in our modified checklist.

For the purpose of this study, the reviewers underwent systematic training. Initially, they received the “Revised CONSORT Statement for Reporting Randomized Trials: Explanation and Elaboration” [8] document which provides the meaning and rationale for each checklist item and examples of good reporting practice. All authors received training in research methodology for three consecutive months, including the studied CONSORT Statement 2010.

Prior to data extraction, as a calibration exercise, both authors independently evaluated two reports on RCTs that were not included into this study. Where differences in the interpretation of an item occurred, discussion ensued and consensus among the investigators was reached allowing clarification on how to score the item.

After training, data from the included trial reports were retrieved, independently and in duplicate, by the two authors and added to tables containing the 37 items to be assessed (Table 1, Table 2, Table 3, Table 4, Table 5 and Table 6). Discrepancies were resolved by consensus between those in charge of the primary evaluation and then checked under the supervision of QS. Once we had ensured consistency in the interpretation of the data extraction form we carried out double data extraction on all the remaining trials. No attempts were made to contact trial authors as the clarity and completeness of reporting were being assessed specifically.

thumbnail
Table 1. The average reporting percentage for the title and abstract section of the CONSORT checklist in the 3 core journals.

https://doi.org/10.1371/journal.pone.0070586.t001

thumbnail
Table 2. The average reporting percentage for the introduction section of the CONSORT checklist in the 3 core journals.

https://doi.org/10.1371/journal.pone.0070586.t002

thumbnail
Table 3. The average reporting percentage for the methods section of the CONSORT checklist in the 3 core journals.

https://doi.org/10.1371/journal.pone.0070586.t003

thumbnail
Table 4. The average reporting percentage for the results section of the CONSORT checklist in the 3 core journals.

https://doi.org/10.1371/journal.pone.0070586.t004

thumbnail
Table 5. The average reporting percentage for the discussion section of the CONSORT checklist in the 3 core journals.

https://doi.org/10.1371/journal.pone.0070586.t005

thumbnail
Table 6. The average reporting percentage for the other information section of the CONSORT checklist in the 3 core journals.

https://doi.org/10.1371/journal.pone.0070586.t006

Assessment of Reporting Quality

Assessment of all included trials was carried out, and the results were entered directly into a preformatted Excel spreadsheet. Each item was assigned a yes (Y, scored as 1) or no (N, scored as 0) response depending on whether it was reported by the author and each item was weighted with equal importance. A total quality of reporting score, the CONSORT score, was calculated by simply summing the scores of the 37-item checklist, resulting in a possible range of 0–37. Thus, the maximum possible score was 37 points.

Each of the study articles was then independently scored by two investigators. A final score for each item on the checklist was recorded for each article after consensus was reached through discussion between the two or in some cases, after arbitration by a third investigator.

The articles were grouped by journal. The primary outcome of the study was the percentage of applicable items on the CONSORT checklist that were reported in each journal. The main secondary outcome was the percentage of articles that reported each applicable section on the checklist. For clarity, the table has been divided into six sections (title and abstract, introduction, methods, results, discussion and other information) representing where the respective items would be expected to be reported. We compared the total mean number and the percentages reported regarding the breakdown of scores for each section of the CONSORT checklist between the three journals.

PRISMA checklist is provided in Appendix 1(Checklist S1).

Statistical Analyses

We conducted a descriptive statistical analysis of all evaluated articles. Data were analyzed using Microsoft Excel 2003. Interrater reliability was not calculated by consensus, and agreement on interpretation of items was carried out in a continual manner, such that any reliability analyses would be meaningless and confounded by learning effects between assessors. All 27 relevant studies were checked for compliance with the statement by assessing the fulfillment of the 37 CONSORT items. In order to assess adherence to the CONSORT checklist items, we calculated the number and proportion of reports describing each of the 37 items. In addition, we calculated the number and proportion of these items by the RCTs published in a journal. The sum of the scores was converted to a percentage value for each trial, each journal, each item, each section, and the total of the CONSORT checklist.

For each article, the quality of its reporting was determined by the total number of items it included in the 37-item checklist. For example, a RCT reporting 20 of the 37 items on the checklist would score 54.1%. Each item on the checklist was also evaluated by tabulating the number of RCTs that reported the item. For example, if 23 of 27 RCTs reported item 2a on the checklist, that item would score an overall compliance score of 85.2%.

Results

A flow chart of the selected RCTs is shown in Figure 1. The search strategy was chosen to be highly sensitive and less specific, allowing a wide range of initial references which were later reduced by reviewing the titles and abstracts. The CNKI database search identified 165 RCTs published between Jan. 1, 2011 and Dec. 31, 2011 in 3 journals. After reviewing these articles, 27 trials fulfilled the eligibility criteria and were assessed for this study. For this analysis, the Journal of Traditional Chinese Medicine contributed the majority of articles which met the eligibility criteria (21)[9][29], followed by the Chinese Journal of Integrated Traditional & Western Medicine (5)[30][34], and the China Journal of Chinese Materia Medica (1) [35].

thumbnail
Figure 1. Study selection.

Figure 1 shows the flow diagram of the selection of RCTs on the treatment of diabetes mellitus with TCM published in 2011 of the three core journals.

https://doi.org/10.1371/journal.pone.0070586.g001

Table 7 shows the sources of the articles.

thumbnail
Table 7. Sources of the articles (% of articles, N = 27).

https://doi.org/10.1371/journal.pone.0070586.t007

There were no disagreements between investigators regarding identification of articles that did not meet these criteria.

Table 1, Table 2, Table 3, Table 4, Table5 and Table 6 show the total mean number and the percentages of the trials that reported the CONSORT items in the 6 different sections, respectively. Table 1, Table 2, Table 3, Table 4, Table 5 and Table 6 also show the mean number and percentages of CONSORT items reported by journal category and compare the extent to which RCTs published in journals reported each of the 37 items examined. The highest possible number for each item was the number of trials included in each journal or the total number of trials.

Table 8 shows the average reporting percentage for each section and the total of the CONSORT checklist in the 3 core journals. The report quality assessed using the CONSORT criteria varied from 0 to 37 (maximum score was 37).

thumbnail
Table 8. The average reporting percentage for each section and the total of the CONSORT checklist in the 3 core journals.

https://doi.org/10.1371/journal.pone.0070586.t008

As seen in Table 1, Table 2, Table 3, Table 4, Table 5 and Table 6, only ten items were reported by 90% or more of the included trials. These included specific objectives or hypotheses, description of the trial design, the interventions for each group, defined outcome measures, the number of participants in each group, baseline data, numbers analyzed, generalizability (external validity, applicability) of the trial findings, outcomes and estimation, and the interpretation.

In contrast, none of the trials reported important changes to methods after trial commencement (such as eligibility criteria), any changes to trial outcomes after the trial commenced, how sample size was determined, explanation of any interim analyses and stopping guidelines when applicable, who generated the random allocation sequence, who enrolled participants, and who assigned participants to interventions, allocation concealment mechanism, why the trial ended or was stopped, where the full trial protocol can be accessed (if available), and presentation of both absolute and relative effect sizes which is recommended for binary outcomes. In addition, none of the trials included a flow diagram.

In addition, a number of items identified by data extraction were reported by only a small percentage of the trials. For example, only 1 (3.7%) stated in the title that the trial was randomized, only 1 (3.7%) reported the type of randomization; details of any restrictions (such as blocking and block size), only 2 (7.4%) reported blinding, only 1 (3.7%) described the similarity of interventions (if relevant), only 1 (3.7%) reported the methods for additional analyses (such as subgroup analyses and adjusted analyses), only 6 (22.2%) reported the losses and exclusions after randomization for each group (together with reasons), only 1 (3.7%) reported the ancillary analyses, only 8 (29.6%) reported the harms, only 4 (14.8%) reported the trial limitations, only 1 (3.7%) reported the registration number and name of the trial registry, and only 11 (40.7%) reported the sources of funding and other support.

Table 8 summarizes the average reporting percentages of the breakdown of scores for each section of the CONSORT checklist for the included trials. As seen in Table 8, the average reporting percentage for the “title and abstract” section of the trials for the 3 core journals (the Journal of Traditional Chinese Medicine, the Chinese Journal of Integrated Traditional & Western Medicine, and the China Journal of Chinese Materia Medica) was 26.2%, 60.0%, and 50.0%, respectively, these values for the “introduction” section were 85.7%, 100.0%, and 100.0%, respectively, for the “methods” section they were 34.5%, 44.7%, and 35.3%, respectively, for the “results” section they were 51.4%, 66.0%, and 60.0%, respectively, for the “discussion” section they were 68.3%, 86.7%, and 66.7%, respectively, and for the “other information” section they were 11.1%, 33.3%, and 0.0%, respectively. The introduction, results and discussion sections showed the highest scores with respect to their maximum theoretical scores.

Discussion

Chinese herbal medicine is becoming increasingly popular in industrialized nations as a form of “alternative” or “complementary” medicine. In order that this type of medicine is fully integrated into conventional medical systems, thereby utilizing its considerable benefits, evidence of the safety and efficacy of herbs and herbal products is necessary. Both randomized controlled trials (RCTs) and systematic reviews are commonly thought to provide the strongest level of evidence regarding the treatment efficacy of competing therapeutic interventions. The credibility of the evidence to support a treatment approach such as Chinese herbal medicine therefore depends on the quality of RCTs [36].

China is a developing country and has many patients with diabetes mellitus, therefore it is necessary to carry out more research in this field. However, fewer studies in this field have been published in foreign journals, as funding sources and the opportunity of collaborative research with foreign researchers are still limited. While English-speaking countries have led this movement to include standardization, there is still a limited amount of information from non-English-speaking countries, including China.

Although surveys and studies similar to the present study have previously been conducted, to our knowledge, this research is the first to investigate the quality of reporting of RCTs with particular reference to CONSORT for the treatment of diabetes mellitus with TCM published in the Chinese language following the revised CONSORT Statement (2010 version). Zhao-Xiang BIAN et al [36] carried out a systematic review on type 2 diabetes mellitus reported in 66 RCTs, the largest number of RCTs on Chinese herbal medicine (CHM) in the Cochrane Library (up to July 2005), and recorded the quality of reporting in RCTs of CHM. All studies were published in English. This study which assessed the quality of the reporting of clinical trials in type 2 diabetes mellitus concluded that “the overall quality of reporting of RCTs of CHM evaluated with a revised CONSORT (2001 version) checklist was poor” [36].

Therefore, in the present study, we investigated the quality of reporting of RCTs published in three leading Chinese medical journals regarding the treatment of diabetes mellitus with TCM using the recent CONSORT (2010 version) statement. The relevant content of each reviewed study was matched to the CONSORT statement to assess its compliance, to determine which sections of the articles should be improved, and we attempted to provide guidelines on how to make these improvements.

There are many other tools available to evaluate the quality of RCTs. We chose the CONSORT checklist for several reasons. Firstly, CONSORT is officially supported by the World Association of Medical Authors and the International Committee of Medical Journal Editors. Secondly, several reports demonstrated that the quality of reporting could be improved if the CONSORT statement was followed more closely[37][40]. Thirdly, CONSORT has been applied in many other medical specialities[41][43]. We used the CONSORT checklist 2010 version as it allowed us to compare our results with those obtained in other medical specialities and therapeutic areas.

The CSCD (Chinese Science Citation Database) contains the most authoritative citation reports in China, it is updated every two years and has achieved a good reputation in China. The CSCD is divided into the core journals and the expansion of journals. Chinese authors consistently select these journals for publication of their research papers as well as degree papers.

In the present study, 3 core journals were selected from CSCD-indexed journals to investigate adherence to the CONSORT statement by these high-impact Chinese medical journals. To determine whether a journal was a CONSORT endorser, we searched the “instructions to authors” section on the journal’s website to assess whether the standard CONSORT guidelines were endorsed as part of the publication process. We found that none of the 3 journals mentioned the CONSORT statement in their “instructions to authors” even though the CONSORT Statement was first introduced into China in 1997. It was not stated in the instructions for authors that clinical trials should have trial registration numbers and that priority would be given to clinical trials which were registered.

The results of our study show that the reporting quality of RCTs on the treatment of diabetes mellitus with TCM published in these three journals was insufficient to allow readers to assess the validity of the trials. Overall compliance with CONSORT guidelines in our study was 45.0%, and very few papers complied with all the criteria. These results come as no surprise when looking at similar studies published in the medical literature[36][45]. He et al concluded that the quality of the current TCM RCTs as judged by their publications is generally poor, especially those published in Chinese journals [38].

In addition, Hopewell S et al. confirmed that the reports on many randomized trials, most of which are published in specialty and subspecialty journals, have crucial information missing or reported in such a way that the information cannot be understood and/or used by readers [39]. Francisco Dasí et al concluded that “even high-impact journals publish articles of unsuitable quality” [40].

David Moher et al concluded that “While improvements have been seen over time, the majority of trials still do not report essential information available from every trial such as full details of the interventions and outcomes, and the methods of allocation of interventions” [46].

Although the articles were published in high-impact index journals, the results of this study show that the title and abstract, methods, and other information sections showed lower scores with respect to their maximum theoretical scores, which indicated that some aspects of the articles require improvement. We found that improvements are particularly needed in the “methods” section. Authors should pay particular attention to providing a detailed description of randomization, blinding, allocation concealment, and conducting proper statistical tests, including intention-to-treat analysis where appropriate. The discussion section in most articles was strong. In particular, authors provided a good literature review and a perspective on how their study impacts on the area of research. However, few authors attempted to generalize their findings in order to help clinicians determine whether the findings applied to their patients. This issue is particularly important because it has been reported that studies with lower methodological quality are most likely to produce “positive results” and are therefore more likely to be published [47].

A striking observation was that the trials were infrequently identified as randomized in the title. The ability to identify a report of a randomized trial in an electronic database depends to a large extent on how it was indexed. Indexers may not classify a report as a randomized trial if the authors do not explicitly report this information. To help ensure that a study is appropriately indexed and easily identified, authors should use the word “randomized” in the title to indicate that the participants were randomly assigned to their comparison groups [8].

To improve the reporting quality of abstracts of RCTs, the CONSORT group previously suggested a 16-item list which should be addressed in the abstract of a given published RCT [8]. Good reporting in RCT abstracts is crucial because they may have a significant and immediate impact on patient care. They must provide appropriate information to allow readers to quickly assess the validity and applicability of the findings regarding the care of their individual patients and to decide whether they should seek more information regarding a trial. However, our analysis revealed that issues such as randomization, blinding, harms, trial registration and funding were still underreported in the abstracts of RCTs published in these three journals. However, limited available space should be considered in this regard. Depending on journal requirements, the word count is usually limited to 250 to 300 words for abstracts. Therefore, the abstract format is often strictly controlled in terms of objective, methods, results, and conclusions. Thus, investigators are influenced and guided by these requirements.

Approval of the RCT by a clinical research ethics committee (CREC) and obtaining informed consent from patients should be mandatory conditions in carrying out a RCT and could, therefore, be considered the main components of ethical research in humans. Therefore, both ethical and methodological aspects should be listed in detail in articles, which would help the reader to make a proper assessment of a RCT [48]. In our study, the reporting of ethical issues was inadequate. None of the RCTs reported having ethical committee approval, although the latter is a legal requirement in China. In addition, a few studies mentioned that the participants attended of “their own free will” but the remainder made no mention of consent.

Similarly, trial registration is stipulated by the CONSORT guidelines and the World Health Organization states that the registration of all interventional trials is a scientific, ethical and moral responsibility. In 2004, the International Committee of Medical Journal Editors (ICMJE) implemented a policy change to state that they would only consider trials for publication if they had been registered before the enrolment of the first participant. This resulted in a dramatic increase in the number of trials being registered, and an associated improvement in the quality of reporting for registered trials [49]. However, only 1 (3.7%) RCT in our sample reported a trial registered with a recognized database and stated the registration number. This may have been due to lack of compulsory policies that only registered trials can be published in journals in China. Now, the Chinese Periodicals Association has only just recommended that clinical trials which have been registered should be prioritized for publication.

Many studies have shown that RCTs not using randomization, allocation concealment or blinding exaggerate estimates of effect to varying degrees. Compared with RCTs using blinding, RCTs without blinding yield larger (17%) estimates of treatment effects and in trials with subjective outcomes, effect estimates are exaggerated by 25%. Compared with the RCTs using adequate allocation concealment, RCTs using unclear or inadequate concealment of allocation exaggerate estimates of effect by 30%–41% [50][54].

Our study also showed some disappointing results. For example, with regard to randomization, none of the trials reported detailed information on their allocation concealment mechanism and implementation. The process of randomization was poorly explained in most studies. None of the studies stated who generated the allocation sequence and whether the same person enrolled the participants. In our opinion (and that of others), a marked improvement in the items relating to randomization was found [55]. Just over half the studies (n = 15) fully described the methods used to generate the allocation sequence. It is well known that only an unpredictable and unknown allocation schedule could minimize selection and confounding biases. Therefore, the CONSORT Statement deemed that use of the term ‘randomized’ is not sufficient. These reports showed that compared with other “flaws”, unclear or inadequate allocation concealment will cause a larger bias, which highlights the importance of allocation concealment. He et al [38] reported that adequate allocation concealment was the smallest proportion (7%) of the aspects they assessed, and some investigations have shown that only 6.8% of the RCTs published in Chinese journals were deemed authentic randomized trials [56]. Thus, the quality of RCTs on TCM in this study may be overstated.

Blinding is an important safeguard against bias, particularly when assessing subjective outcomes [8]. Regardless of whether blinding is possible, authors can and should always state who was blinded (that is, participants, healthcare providers, data collectors, and outcome adjudicators). Unfortunately, in total, 25 papers (92.6%) provided no information on blinding of either participants or investigators. We acknowledge that blinding is particularly challenging in the treatment of diabetes mellitus with TCM, as blinding of participants and investigators is often difficult or impossible, however, blinding of data collectors and outcome adjudicators is often achievable. If blinding is not achieved, this should be mentioned and discussed as a potential bias.

Furthermore, although all articles reported on the number of patients, details on the method used to calculate the sample size (power calculation) were not described in any of the articles. Reporting of a priori sample size calculation is an indicator of adequate trial planning and indicates whether a trial ends earlier than planned. However, Herdan A. et al [55] reported that sample size calculation showed a sharp increase over time, i.e. the percentage of trials correctly reporting this item increased from 23% (three of 13 reports) to 71% (17 of 27 reports) in the post-CONSORT period. Another concern in all these studies was the lack of reported information on the flow of participants through each stage of the treatment protocol, including information regarding the number of care providers or centers performing the intervention. The original CONSORT statement strongly advocated a flow diagram in order to follow the flow of patients from recruitment to the end of the clinical trial [8].

In our review, only nine papers (33.3%) did not report which statistical methods had been used. A total of 2 papers (7.4%) presented p-values, and 3 papers (11.1%) presented a 95% CI of the treatment effect. The statistical methods used to compare groups were reasonably well described, but in most cases the general statistical approach was described rather than linking a specific test to an outcome. We did not determine whether the statistical test was appropriate, although many were deficient in describing confidence intervals and accounting for multiple testing. In addition, all papers merely reported an ordered list of results from which it was not possible to distinguish which outcome was the primary outcome. In the reviewed trials, most papers only described baseline characteristics of the participants using text in the baseline information, simply stating that the baseline characteristics matched in both arms instead of fully reporteing the baseline characteristics of the participants in a separate table.

Identification and discussion of the weaknesses of a study are particularly important and are needed for a balanced interpretation of the results. Only 14.8% of RCTs studies discussed the limitations of the study. Similarly, information on the harms as well as the benefits of interventions is stipulated in the CONSORT guidelines. However, details of harms and unintended effects were reported in just 29.6% of the RCTs in our sample. Reporting of adverse events should always be considered during the study design as data on adverse events are less susceptible to bias and confounders when they are collected prospectively rather than retrospectively. Editors should probably state that authors should be encouraged to report undesired effects. Proper definition and reporting of adverse events are crucial for critical appraisal of the study results and facilitates systematic reviews and meta-analyses.

A shortcoming in all of the included studies was the lack of dates defining the periods of follow-up. In total, 24 papers (88.9%) reported the date on which participants were recruited, however, there was no information provided on the length of time the participants were followed. The length of follow-up is not always a fixed period after randomization. In many RCTs in which the outcome is time to an event, follow-up of all participants is ended on a specific date. This date should be provided, and it is also useful to report the minimum, maximum and median duration of follow-up. In addition, another important area of RCT quality is related to loss to follow-up. The majority of the trials reported that none of the participants had dropped out. In the reviewed trials, 77.8% of studies failed to report drop-out rates, which revealed that Chinese researchers do not adequately consider this problem. In general, the higher the ratio of participants with missing data to participants with events, then the greater the potential for bias. Perhaps for this reason many studies were unwilling to report real drop-out rates. When studies have drop-out rates greater than 20%, all results are likely to lose their authenticity, and consequently, clinical value will be badly affected if not meaningless.

It is important that authors describe in detail the role of the funding bodies and their level of involvement and influence on design, conduct and analysis of the data. A total lack of information on economic studies was observed in our study. This economic information is particularly important in a medical system in which many clinical decisions rely on tools such as the economic evaluation and the intervention outcome. Of the 27 papers, only 11 (40.7%) reported their sources of funding. Funding was from provincial/municipal and national sources (including provincial/municipal sources) in 8 trials (29.6%) and 3 trials (11.1%), respectively. Although there was some improvement over time, this improvement was only moderate.

In compliance with the legal regulations, the promoter of a RCT is required to publish the results (both positive and negative) in scientific journals, with reference to the clinical research ethics committee that approved the study. However, in our study, none of the trials published negative results. It has long been known that only a small proportion of RCTs publish negative results in scientific journals, which seems to be a common problem as the publication of RCTs in different countries varies between 31% and 67% [57]. In a review article, von Elm et al [58] concluded that the main reason for this was lack of time, however, some researchers responded that negative results were the major cause of nonpublication. The selective publication of “statistically significant” results is well documented, and it has been observed that a study was more likely to be published if the results were positive [59], [60].

Therefore, there is considerable room for improvement in the publication of RCTs conducted in China. In particular, the publication of methods described in as much detail as possible and the titles and abstracts and other information regarding the study should be improved.

Limitations

Considering the rigor of this systematic evaluation, we are confident that it is a complete summary of all available evidence. However, several limitations of the study should be acknowledged. First, since we only assessed the RCTs published by the three CSCD-indexed, leading Chinese medical journals from January to December 2011, the results do not fully represent the reporting quality of all Chinese RCTs. Second, we only assessed the reporting quality of RCTs, not the trials themselves, and failure to report is not necessarily equivalent to failure to actually carry out the procedures. It is possible that a poorly reported study was well designed and executed, and a well-reported study may have had several shortcomings. However, transparent and clear reporting is likely to be the result of a well-designed trial allowing the reader to make a judgment. Third, we did not calculate the Kappa statistic to quantitatively measure the interobserver agreement. Fourth, it should be noted that some journals have more restrictions than others in terms of word count for example. These restrictions may be related detailed descriptions of eligibility criteria, abstract, process of consent acquisition, and blinding methods. Fifth, the time frame (from January to December 2011) chosen for this review was meant to provide an indication of CONSORT reporting for the selected year only, and may not be representative of the journals’ reporting in general. In addition, any improvements in CONSORT reporting may take some time to become apparent after official endorsement of the CONSORT statement. Hence, for this review, the one year interval may not provide a sufficient time frame to observe dramatic changes in the reporting of RCTs. Sixth, the CONSORT score, although based on the statement, requires to be validated by other investigators. Although the exact score a study received may not be reproduced if evaluated by others, this study indicates the wide variation in compliance with the CONSORT statement. For example, each item was assigned a yes (Y, scored as 1) or no (N, scored as 0) response depending on whether the item was reported by the author. It may be unfair if an item was scored as 1, if it included three conditions (appropriately and transparently presented, inadequately stated, unclear stated); while an item was scored as 0, if it included two conditions (not applicable, not mentioned). The CONSORT statement itself has been revised, thus future compliance of RCTs needs to be amended. Seventh, the journals included in this study were selected due to their high impact factor. However, there are many other journals which should be assessed for their reporting of CONSORT information if a more comprehensive evaluation is undertaken. Eighth, criteria for the CONSORT statement were not weighted for scoring purposes, based on the assumption that each item of the CONSORT statement had equal importance. We avoided weighting as it would be arbitrary and subjective and thus subject to criticism. Another potential limitation is that we assessed only publications in Chinese, which may have introduced publication bias. However, only 10% of the eligible articles were reported in other languages, and it is unlikely that their inclusion would have changed the overall results.

Conclusions

In summary, the reporting quality of Chinese RCTs on the treatment of diabetes mellitus with TCM published in these three leading medical journals is far from satisfactory, particularly with respect to the title and abstract, methods and other information sections. The methodological quality of RCTs in China did not show any apparent improvement since the Chinese State Food and Drug Administration (SFDA) recommended Good Clinical Practice (GCP) in 1998. More attention should be paid to the design and methodology of these trials. Specifically, items such as identification as a randomized trial in the title, allocation concealment, blinding, sample size calculation, a participant flow diagram, information on the harms or unintended effects, trial limitations, trial registration, ethical committee approval, the patient’s informed consent, the periods of follow-up, role of funders, and compliance of participants and investigators should be implemented in all RCTs to improve their overall quality and ensure the validity and usefulness of their results.

Editorial mechanisms can be very successful in improving the quality and completeness of the research evidence by enforcing the requirements for article submission. The results of this review should strongly encourage journal editors to change the instructions to authors to ensure that the issues that affect the understanding of a manuscript, by referees or readers, and how the study was undertaken are adequately described. There is good evidence in the literature that the adoption of CONSORT improves the quality of both the conduct and reporting of trials in journals that have taken the decision to make it a requirement for submission acceptance[39], [61][63]. Therefore, authors, editors and referees should adopt the CONSORT Statement to improve the reporting quality of Chinese RCTs and ensure truth and reliability of conclusions. In the interest of transparency, authors of trial reports should be encouraged to state why any item in the checklist was not reported, thus making the reader’s task of understanding conformity to CONSORT more straightforward.

Supporting Information

Checklist S1.

PRISMA checklist. PRISMA 2009 checklist of information to include when reporting a systematic review.

https://doi.org/10.1371/journal.pone.0070586.s001

(DOC)

Acknowledgments

All authors were involved in each stage of the research: conception and design; implementation, analysis and interpretation of the data; manuscript writing; and revision and approval of the manuscript.

Author Contributions

Conceived and designed the experiments: QX QS PW. Analyzed the data: PW FF XG FG. Wrote the paper: PW. Acquisition of data: PW FF XG.

References

  1. 1. Schulz KF, Altman DG, Moher D, for the CONSORT Group (2010) CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials for the CONSORT Group. BMJ 340 c332: 698–702
  2. 2. Halpern SH, Darani R, Douglas MJ, Wight W, Yee J (2004) Compliance with the CONSORT checklist in obstetric anaesthesia randomised controlled trials. Int J Obstet Anesth 13: 207–214
  3. 3. Erin NK, Robert MS, Nader P, Aaron SD (2011) Standards for reporting randomized controlled trials in neurosurgery. J Neurosurg 114: 280–285.
  4. 4. Wang L, Li YL, Li J, Zhang MM, Xu L, et al. (2010) Quality of reporting of trial abstracts needs to be improved: using the CONSORT for abstracts to assess the three leading Chinese medical journals of traditional Chinese medicine. Trials 11: 75.
  5. 5. Li WL, Zheng HC, Bukuru J, Kimpe ND (2004) Natural medicines used in the traditional Chinese medical system for therapy of diabetes mellitus. J Ethnopharmacol 92: 1–21
  6. 6. Liu JP, Zhang M, Wang WY, Grimsgaard S (2002) Chinese herbal medicines for type 2 diabetes mellitus. Cochrane Database Syst Rev 3. https://doi.org/10.1002/14651858 CD003642.
  7. 7. Schulz KF, Altman DG, Moher D (2010) the CONSORT Group (2010) CONSORT 2010 Statement: updated guidelines for reporting parallel-group randomized trials. BMJ 340: c869.
  8. 8. Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, et al. (2010) CONONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 340: c869
  9. 9. Guan JT, Hu YH (2011) Clinical Research on Buxin Tongmai Decoction in the Treatment of type 2 diabetes combining nonalcoholic fatty liver disease: A Report of 34 Cases. JTCM 52(2): 155–156.
  10. 10. Song FL, Jia RX, Li GY, Wang CJ, Ma HJ (2011) Comparative Study on the Effect of Huangqi Guizhi Wuwu Decoction in Different Dosages on Diabetics with Peripheral Neuropathy. JTCM 52(7): 570–572.
  11. 11. Zheng YP, Chen C, Xu XY, Li Y (2011) Influence of Shenqi Powder on Free Fatty Acid in the Blood of Type2Diabetes Mellitus Patients. JTCM 52(11): 949–951.
  12. 12. Ma YQ (2011) Clinical Observation on Huazhuo Qingre Decoction for the Treatment of 40 Diabetic Patients with Syndrome of Damp-heat Blocking the Spleen. JTCM 52(16): 1390–1392.
  13. 13. Liang JF, Wu XW, Ji LJ, Song JY, Rong J, et al. (2011) Clinical Observation on Huoxue Lishui Formula Combined with 532 Laser Therapy for Diabetic Macular Edema:A Report of 30 Cases. JTCM 52(8): 662–665.
  14. 14. Cheng FX, Chen ZQ, Fu QY (2011) Influence of Jiawei Bugan Decoction on the Neural Growth Factor of the Diabetic Peripheral Neuropathy. JTCM 52(15): 1297–1300.
  15. 15. Li Y, Fu SB, Yang R, Fan L, Bai XQ (2011) Clinical Observation on Jiawei Gehong Decoction for the Treatment of 42 diabetes combined with Cardiovascular Autonomic Neuropathy(syndrome of Qi deficiency and Blood stagnation). JTCM 52(13): 1146–1148.
  16. 16. Yang ZH, Xia JE, Zhang HY (2011) Clinical Observation on Treatment 34 Patients of Early Diabetic Nephropathy by Invigorating Qi and Tonifying Shen and Removing Stasis. JTCM 52(3): 243–244.
  17. 17. Jian XQ (2011) Clinical Observation on Treatment 30 Patients of Constipation Due to Senile Diabetes Mellitus with Self-Prepared Tiaoli decoction. JTCM 52(14): 1238–1240.
  18. 18. Zhu LM (2011) Clinical Observation on Treatment 120 Patients of Diabetic Coronary Heart Disease of the Phlegm-blood-stagnancy-type by Dissipating Phlegm and Removing Blood Stasis. JTCM 52(supplement): 69–70.
  19. 19. Su M, Guan Y (2011) Observation on Therapeutic Effect of Jiangtang Granules on 40 type 2 diabetic patients. JTCM 52(19): 1683–1685.
  20. 20. Wang LX, Zhang L, Yan SQ, Liu GB (2011) Clinical Observation on Jiedu Tongmai Powder Combined with Weinong Zhangrou Plaster for Diabetic Foot Ulcer: A 24 Cases Report. JTCM 52(15): 1283–1285.
  21. 21. Chao Y, Zhao JX, Liu JL (2011) Effect of Qingre Qushi Lishui Formula on tumor necrosis factor α of type 2 diabetic-insulin resistant Patients. JTCM 52(4): 338–340.
  22. 22. Zhao LM, Du Y, Kang Z, Du ZC, Song XW, et al. (2011) Effect of Tangxin Heji on Cardiac Function in Type II Diabetic Patients with Heart Failure. JTCM 52(19): 1658–1660.
  23. 23. Wei LL, Zhang MQ (2011) Clinical Research on Wenjing Huoxue Huatan Formula in the Treatment of Type II Diabetic Peripheral Neuropathy: A Report of 64 Cases. JTCM 52(16): 1383–1386.
  24. 24. Ma L, Wang XM, Chen Q, Li KL (2011) Clinical Observation on Painful Diabetic Neuropathy Treated by Wengtong Huoxue Cream plus Limb Air-pressure Therapy: A Report of 30 Cases. JTCM 52(19): 1665–1667.
  25. 25. Zuo L, Zhang L, Zhang JD (2011) Observation on Therapeutic Effect of mecobalamin with Acupoint Injection on 30 Patients with Diabetic Peripheral Neuropathy. JTCM 52(5): 420–422.
  26. 26. Li L (2011) Internal and external combination of traditional Chinese medicine for treatment of 47 cases of Diabetic Foot by Yiqihuoxue Jiedulishi therapy. JTCM 52(14): 1236–1237.
  27. 27. Wu LK, Shi LL, Song XL, Si SY, Li YM, et al. (2011) Clinical Observation on 46 Cases of Type 2 Diabetes Mellitus Complicated with Fatty Liver Treated by Yiqi Yangyin and Huoxue Herbal Medicines. JTCM 52(9): 751–755.
  28. 28. Liu X (2011) Acupuncture Treatment with Warming-yang Method for the Patients of Diabetic Peripheral Neuropathy :A Report of 32 Cases. JTCM 52(20): 1745–1748.
  29. 29. Zhao S, Yang CJ (2011) Clinical Efficacy of External Medical Herbs Combining Acupoint Application on the Patients of Diabetic Peripheral Neuropathy. JTCM 52(21): 1860–1861.
  30. 30. Li ZH, Yang SQ, Liu JP, Chang Q, Liao X, et al. (2011) A Multi-center, Randomized Control Trial on Clinical Research Education to Diabetes Patients. CJITWM 31(10): 1359–1363.
  31. 31. Jiang ZT, Liang QL, Wang YM, Li P, Li J, et al. (2011) Effects of Tangshen Recipe on the Homocysteine Metabolism of Patients with Diabetic Nephropathy. CJITWM 31(8): 1057–1061.
  32. 32. Huang PJ, Mao XH, Wang YP, Pan XD, Liu LB, et al. (2011) Electrophysiological Changes in Diabetic Peripheral Neuropathy Patients of Different Chinese Medicine Syndrome Types Intervened by Naoxintong and Mecobalamin. CJITWM 31(8): 1051–1056.
  33. 33. Li SF, Zhao JY, Liu JP, Xiang F, Liang Y, et al. (2011) Effect of Tangzu Yuyang Ointmenton the Outcome Event of Patients with Chronic Diabetic Foot Ulcers. CJITWM 31(6): 775–779.
  34. 34. Jiang T, Wen Z, Jiang M, Lu GY (2011) Effect of Integrative Medical Therapy on Post-angioplasty Arterial Restenosis in Lower Lmibs of Diabetic Patients. CJITWM 31(2): 182–187.
  35. 35. Xie XM, Meng XJ, Zhou XJ, Shu XC, Kong HJ (2011) Research on therapeutic effect and hemorrheology change of berberine in new diagnosed patients with type 2 diabetes combining nonalcoholic fatty liver disease. CJCMM 36(21): 3032–3035.
  36. 36. Bian ZX, Li YP, Moher D, Dagenais S, Liu L, et al. (2006) Improving the quality of randomized controlled trials in Chinese herbal medicine, partI: clinical trial design and methodology. JCIM 4 (2): 120–129.
  37. 37. Marshman Z, Farid F (2010) The quality of reporting of randomised controlled trials in dental public health. Community Dent Health 27(4): 253–256.
  38. 38. He J, Du L, Liu GJ, Fu J, He XY, et al. (2011) Quality assessment of reporting of randomization, allocation concealment, and blinding in traditional chinese medicine RCTs: A review of 3159 RCTs identified from 260 systematic reviews. Trials 12: 122.
  39. 39. Hopewell S, Dutton S, Yu LM, Chan AW, Altman DG (2010) The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed. BMJ 340: c723
  40. 40. Francisco D, María MNG, Mercedes JH, Jose M, Juan RV, et al. (2012) Evaluation of the Quality of Publications on Randomized Clinical Trials Using the Consolidated Standards of Reporting Trials (CONSORT) Statement Guidelines in a Spanish Tertiary Hospital. J Clin Pharmacol 52: 1106–1114.
  41. 41. Ma B, Ke FY, Chen ZM, Qi GQ, Li HM, et al. (2012) Does the reporting of randomized clinical trials published in Chinese pediatrics journals improve after the CONSORT Statement is adopted? Contemp Clin Trials 33: 889–894
  42. 42. Pandis N, Polychronopoulou A, Eliades T (2010) Evaluation of the quality of randomised clinical trials published in leading dental specialty journals. J Dent 38: 713–721.
  43. 43. Daniel LK, Paul HD, Mark DG, Michael G (2011) Assessing clinical trials of Internet addiction treatment: A systematic review and CONSORT evaluation. Clin Psychol Rev 31: 1110–1116
  44. 44. Karsten K, Uzung Y, Hans OR, Peter MV (2011) Quality of reporting according to the CONSORT, STROBE and Timmer instrument at the American Burn Association (ABA) annual meetings 2000 and 2008. BMC Med Res Methodol 11 161: 1–10.
  45. 45. Lu LM, Zeng JC, Chen Y (2011) Quality of reporting in randomized controlled trials conducted in China on the treatment of cancer pain. Expert Rev. Anticancer Ther 11(6): 871–877.
  46. 46. Moher D, Hopewell S, Schulz KF, Altman DG (2011) Resources for authors of reports of randomized trials: harnessing the wisdom of authors, editors, and readers. Trials 12: 98.
  47. 47. Juni P, Altman DG, Egger M (2001) Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ 323: 42–46.
  48. 48. Ruiz-Canela M, Aguinaga I, Guillen F, Martínez-González MA (1998) Publicación de ensayos clínicos: método y ética. Med Clín (Barc) 110: 238.
  49. 49. Parsons NR, Hiskens R, Price CL, Achten J, Costa ML (2011) A systematic survey of the quality of research reporting in general orthopaedic journals. J Bone Joint Surg Br 93-B: 1154–1159.
  50. 50. Schulz KF, Chalmers I, Hayes RJ, Altman DG (1995) Empirical evidence of bias dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 273(5): 408–412.
  51. 51. Moher D, Pham B, Jones A, Cook DJ, Jadad AR, et al. (1998) Does quality of reports of randomized trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 352: 609–613.
  52. 52. Wood L, Egger M, Gluud LL, Schulz KF, Juni P, et al. (2008) Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 336(7644): 601–605
  53. 53. Kunz R, Vist GE, Oxman AD (2007) Randomisation to protect against selection bias in healthcare trials. Cochrane Database Syst Rev 2. https://doi.org/10.1002/14651858 MR000012.
  54. 54. Kunz R, Oxman AD (1998) The unpredictability paradox: rovicw of empirical comparisons of randomized and non-randomized clinical trials. BMJ 317 1185–1190.
  55. 55. Herdan A, Roth R, Grass D, Klimek M, Will S, et al. (2011) Improvement of quality of reporting in randomized controlled trials to prevent hypotension after spinal anaesthesia for caesarean section. Gynecol Surg 8: 121–127.
  56. 56. Wu TX, Li YP, Bian ZX, Liu GJ, Moher D (2009) Randomized trials published in some Chinese journals: how many are randomized? Trials 10: 46
  57. 57. Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan A-W, et al. (2008) Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One 3(8): e3081
  58. 58. von Elm E, Costanza MC, Walder B, Tramer MR (2003) More insight into the fate of biomedical meeting abstracts: a systematic review. BMC Med Res Methodol 3: 12.
  59. 59. Ioannidis JP (1998) Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA 279: 281–286.
  60. 60. Johnson RT, Dickersin K (2007) Publication bias against negative results from clinical trials: three of the seven deadly sins. Nat Clin Pract Neurol 11: 590–591.
  61. 61. Plint AC, Moher D, Morrison A, Schulz KF, Altman DG, et al. (2006) Does the CONSORT checklist improve the quality of reports of randomized controlled trials? A systematic review. Med J Aust 185: 263–267.
  62. 62. Karin P, Christophe D, Johan V (2008) Systematic review of symptom control and quality of life in studies on chemotherapy for advanced non-small cell lung cancer: How CONSORTed are the data? Lung Cancer 62, 126–138. https://doi.org/10.1016/j.lungcan2008.02.018.
  63. 63. Uetani K, Nakayama T, Ikai H, Yonemoto N, Moher D (2009) Quality of Reports on Randomized Controlled Trials Conducted in Japan: Evaluation of Adherence to the CONSORT Statement. Inter Med 48: 307–313.