Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Assessment of the Quality of Reporting in Abstracts of Randomized Controlled Trials Published in Five Leading Chinese Medical Journals

  • Yaolong Chen ,

    Contributed equally to this work with: Yaolong Chen, Jing Li

    Affiliations Chinese Cochrane Centre, West China Hospital, Sichuan University, Chengdu, Sichuan, China, Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, Gansu, China

  • Jing Li ,

    Contributed equally to this work with: Yaolong Chen, Jing Li

    Affiliation Chinese Cochrane Centre, West China Hospital, Sichuan University, Chengdu, Sichuan, China

  • Changlin Ai,

    Affiliation Chinese Cochrane Centre, West China Hospital, Sichuan University, Chengdu, Sichuan, China

  • Yurong Duan,

    Affiliation Chinese Cochrane Centre, West China Hospital, Sichuan University, Chengdu, Sichuan, China

  • Ling Wang,

    Affiliation Chinese Cochrane Centre, West China Hospital, Sichuan University, Chengdu, Sichuan, China

  • Mingming Zhang ,

    chicc86@hotmail.com (MMZ); shopewell@cochrane.co.uk (SH)

    Affiliation Chinese Cochrane Centre, West China Hospital, Sichuan University, Chengdu, Sichuan, China

  • Sally Hopewell

    chicc86@hotmail.com (MMZ); shopewell@cochrane.co.uk (SH)

    Affiliation UK Cochrane Centre, Oxford, United Kingdom

Assessment of the Quality of Reporting in Abstracts of Randomized Controlled Trials Published in Five Leading Chinese Medical Journals

  • Yaolong Chen, 
  • Jing Li, 
  • Changlin Ai, 
  • Yurong Duan, 
  • Ling Wang, 
  • Mingming Zhang, 
  • Sally Hopewell
PLOS
x

Abstract

Background

Clear, transparent and sufficiently detailed abstracts of randomized trials (RCTs), published in journal articles are important because readers will often base their initial assessment of a trial on such information. However, little is known about the quality of reporting in abstracts of RCTs published in medical journals in China.

Methods

We identified RCTs abstracts from 5 five leading Chinese medical journals published between 1998 and 2007 and indexed in MEDLINE. We assessed the quality of reporting of these abstracts based on the Consolidated Standards of Reporting Trials (CONSORT) abstract checklist. We also sought to identify whether any differences exist in reporting between the Chinese and English language version of the same abstract.

Results

We identified 332 RCT abstracts eligible for examination. Overall, the abstracts we examined reported 0–8 items as designated in the CONSORT checklist. On average, three items were reported per abstract. Details of the interventions (288/332; 87%), the number of participants randomized (216/332; 65%) and study objectives (109/332; 33%) were the top three items reported. Only two RCT abstracts reported details of trial registration, no abstracts reported the method of allocation concealment and only one mentioned specifically who was blinded. In terms of the proportion of RCT abstracts fulfilling a criterion, the absolute difference (percentage points) between the Chinese and English abstracts was 10% (ranging from 0 to 25%) on average, per item.

Conclusions

The quality of reporting in abstracts of RCTs published in Chinese medical journals needs to be improved. We hope that the introduction and endorsement of the CONSORT for Abstracts guidelines by journals reporting RCTs will lead to improvements in the quality of reporting.

Introduction

There are more than 1200 biomedical journals in China [1], which publish thousands of randomized controlled trials (RCTs) each year. Clear, transparent, and sufficiently detailed abstracts of RCTs are more important particularly in China and other developing countries where researchers and health professionals often use an abstract to decide whether to seek more information about a trial; or may have access to the abstracts only [2]. Previous studies have attempted to assess the quality of reporting in abstracts of clinical studies [3] or to assess the quality of reporting in abstracts of RCTs published in English language journals [4]. The aim of our study is to assess the quality of reporting in abstracts of RCTs published in five leading Chinese medical journals using the recent CONSORT (Consolidated Standards of Reporting Trials) checklist for reporting abstracts of RCTs.

Materials and Methods

1. Selection of journals and sample of abstracts of RCTs

We selected five leading Chinese medical journals indexed in MEDLINE from 1998 to 2007 with top ranking impact factor (IF) in each field based on the “2007 data for Sci-tech Journal Citation Reports of China”(Table 1) and with both English and Chinese abstracts: the National Medical Journal of China, Chinese Journal of Internal Medicine, Chinese Journal of Obstetrics and Gynecology, Chinese Journal of Pediatrics, and the Chinese Journal of Surgery.

thumbnail
Table 1. Number of abstracts identified and included from five Chinese medical journals.

https://doi.org/10.1371/journal.pone.0011926.t001

The CBM (Chinese Biomedical Database) disk database was searched using an extended version of the Cochrane Highly Sensitive Search Strategy [5] (Table 2) to identify RCTs (searched April, 2008) for Chinese abstracts in each of the five leading journals. Based on the citation of each RCT, we hand searched these five printed journals for the English abstracts. We included all publications reporting RCTs,which was defined as a trial where the allocation of participants to interventions was described by the words random, randomly, randomized, or randomization in all disease areas and all types of interventions, dealing with patients or volunteers.

thumbnail
Table 2. Search strategy to identify RCTs in five leading Chinese medical journals.

https://doi.org/10.1371/journal.pone.0011926.t002

2. Evaluation Method

We assessed the quality of reporting in abstracts of RCTs based on the CONSORT for Abstracts checklist [6]. Refined items from the CONSORT for Abstracts are presented in the Table 3. We assessed each item as “reported” or “not reported” according to whether the author reported all the contents listed in the refined items or not. We checked for any differences in reporting between the Chinese and English language version of the same abstract.

thumbnail
Table 3. Adapted CONSORT for Abstracts checklist items reported in the 332 journal abstracts of RCTs.

https://doi.org/10.1371/journal.pone.0011926.t003

We determined the number and proportion (%) of RCT abstracts that reported each of the CONSORT for Abstracts checklist items. Four assessors (YL Chen, CL Ai, L Wang and YR Duan) independently rated each RCT abstract and inter-rater agreement for each checklist item was determined using the Kappa statistic. Overall, this produced good agreement between the reviewers (0.7). Discrepancies were resolved by consensus and or discussed. Data analysis was carried out using Excel 2003, Epidata 3.02 and Stata version 9.0.

Results

1. Quality of reporting in Chinese abstracts

We identified 692 abstracts of RCTs during our initial search and subsequently determined that 332 pairs were eligible for analysis based on our criteria of findings being reported both Chinese and English (Figure 1). We have presented the results of our findings in Table 3. Overall only 10% (34/332) of the abstracts could be identified as an RCT based on the title. No abstracts mentioned the method of allocation concealment and, while 49 (15%) RCT abstracts reported on blinding, only one indentified who was blinded. Although 65 (20%) of the abstracts mentioned adverse events or side effects, less than 10% reported specific symptoms (n = 20) or rates (n = 26). Lastly, 246 (75%) abstracts did not report their sources of funding.

thumbnail
Figure 1. Identification of RCT abstracts from the Chinese Biomedical Database.

https://doi.org/10.1371/journal.pone.0011926.g001

2. Comparison the quality of reporting between the Chinese with English abstracts

The results of the English language version of the same RCT abstracts are also presented in Table 3. In terms of the proportion of RCT abstracts fulfilling a criterion, the absolute difference (percentage points) between the Chinese and English abstracts was 10% (ranging from 0 to 25%) on average, per item. Some items such as details of the primary outcome, were only reported in the English version of the abstract. Some items were complementary between the Chinese and English abstracts, for example, some Chinese abstracts only reported the “numbers randomized” while their corresponding English abstracts only reported the “numbers analyzed”.

Discussion

1. General quality of reporting in abstracts and its possible reasons

Our study indicates that the quality of reporting in abstracts of RCTs published in these five leading Chinese medical journals is insufficient. Overall, the average number of reported items in each RCT abstract, based on the CONSORT for Abstracts checklist, was three. None of the abstracts adequately reported the key features of the trial design, including the method of generating the allocation sequence, concealment of the allocation, or the estimated effect size and its precision for the primary outcome. Less than half of the RCT abstracts reported the study as randomized in the title, the number of participants analysed, details of harms, trial registration and source of funding.

Berwanger and colleagues [7] evaluated the quality of reporting in 227 abstracts describing RCTs published in the New England Journal of Medicine, JAMA, the BMJ and the Lancet in 2006. In their paper, the authors identified that details regarding methodological quality were also poorly reported as only one RCT abstract described the method of allocation concealment; 21 (9%) clearly specified who was blinded; 51 (23%) described intention-to-treat analysis; and 32 (14%) outlined losses to follow-up.

In the mid-1990s, Anrong [8] and Bili [9] reported that the quality of abstracts in Chinese medical journals was poor. They suggested that poor abstract reporting may indicate suboptimal study design, inadequate reporting, or both. Previous work has shown that abstracts frequently under report key features of study design [10][11] and omit important results. It is suggested that the poor quality of reporting of Chinese abstracts may also be an indication of poor reporting in full-text RCTs in China [12][18]. When an item isn't reported in the full publication, it is likely to also be missing from the abstract. A lack of detailed requirement for abstracts in a journal's ‘Instruction for Authors’ is also a potential problem. Although most of the journals included in our study have adopted structured abstracts, the structured format alone is not sufficient to guide authors regarding their content. Thus, some of the most salient information might be missed if no detailed instructions for each heading are given. Finally, a lack of sufficient knowledge of research methodology might also attribute to the poor quality of reporting.

Abstracts should contain the most important information and accurate study results relevant to clinical practice. While we did find that most abstracts reported the purported “study benefits,” we also observed that the primary outcome, the estimated effect size, and the precision and harms of the study were rarely reported. Cumulatively, this lack of information hampers readers in making clinical decisions [19][20] and presents clinicians many difficulties when trying to evaluate the benefits and harms of any intervention. Unfortunately, only two RCT abstracts reported details of trial registration.

While, not included in our study, the Chinese Journal of Evidence-Based Medicine recently introduced the CONSORT for Abstracts guidelines into its ‘Instructions for Authors’ [21]. The application of the CONSORT Statement has demonstrated benefits in improving the quality of reporting in abstracts [22][24], and we hope that it will do the same with the introduction of CONSORT for Abstracts. We very much hope that the five journals included in our study will also consider including such guideline to their ‘Instructions for Authors’.

2. Limitations

This study has several limitations, firstly, we did not take a random sample of all RCT abstracts published in Chinese medical journals and therefore our findings may not reflect all abstracts of RCTs published in China or in other countries. Secondly, we did not compare abstracts with their corresponding full articles, However, previous studies have shown that the quality of reporting of the full-text is also poor in the same medical journals [12][18]. We further wish to point out that the WHO International Clinical Trial Register Platform only approved The Chinese Clinical Trial Register (ChiCTR) in 2007. Thus, it may well be that many researchers do not know the importance of prospective trial registration [25]. While not included in our current study, the Chinese Journal of Evidence-Based Medicine recently introduced the CONSORT for Abstracts guidelines into its ‘Instructions for Authors’ [21]. This adoption has improved the quality of reporting in other journals [22][24] and it is our sincere hope that the five journals included in our study, as well as other Chinese journals, adopt the CONSORT statement for Abstracts into their ‘Instructions for Authors.’ We believe that this adoption will markedly improve both the interpretation and practice of medicine in China in the future.

Conclusions

In summary, the quality of reporting in abstracts of RCTs published in these five leading Chinese medical journals requires substantial improvement to meet the recommendations set out in CONSORT for Abstracts guidelines. With the publication of CONSORT for Abstracts checklist, we suggest that Chinese medical journals should adopt these recommendations and do more to ensure that authors apply to meet internationally agreed standards, thereby allowing the conduct of their studies to be monitored and improved.

Author Contributions

Conceived and designed the experiments: YC JL CA YD MZ SH. Performed the experiments: YC JL CA YD LW MZ SH. Analyzed the data: YC JL CA YD LW MZ SH. Contributed reagents/materials/analysis tools: YC JL CA YD LW MZ SH. Wrote the paper: YC JL CA YD LW MZ SH.

References

  1. 1. Du YX, Lei CB, Chen JQ, Zhang XM, Zhang Y, et al. (2006) A comparative study of the literature included in the five authoritative domestic databases of Chinese biomedical journals. J Prev Med Inf 22: 162–165.YX DuCB LeiJQ ChenXM ZhangY. Zhang2006A comparative study of the literature included in the five authoritative domestic databases of Chinese biomedical journals.J Prev Med Inf22162165
  2. 2. Hopewell S, Eisinga A, Clarke M (2008) Better reporting of randomized trials in biomedical journal and conference abstracts. J Inf Sci 34: 162–173.S. HopewellA. EisingaM. Clarke2008Better reporting of randomized trials in biomedical journal and conference abstracts.J Inf Sci34162173
  3. 3. Dupuy A, Khosrotehrani K, Lebbe C, Rybojad M, Morel P (2003) Quality of abstracts in 3 clinical dermatology journals. Arch Dermatol 139: 589–593.A. DupuyK. KhosrotehraniC. LebbeM. RybojadP. Morel2003Quality of abstracts in 3 clinical dermatology journals.Arch Dermatol139589593
  4. 4. Burns KE, Adhikari NK, Kho M, Meade MO, Patel RV, et al. (2005) Abstract reporting in randomized clinical trials of acute lung injury: an audit and assessment of a quality of reporting score. Crit Care Med 33: 1937–1945.KE BurnsNK AdhikariM. KhoMO MeadeRV Patel2005Abstract reporting in randomized clinical trials of acute lung injury: an audit and assessment of a quality of reporting score.Crit Care Med3319371945
  5. 5. Robinson KA, Dickersin K (2002) Development of a highly sensitive search strategy for the retrieval of reports of controlled trials using PubMed. Int J Epidemiol 31: 150–153.KA RobinsonK. Dickersin2002Development of a highly sensitive search strategy for the retrieval of reports of controlled trials using PubMed.Int J Epidemiol31150153
  6. 6. Hopewell S, Clarke M, Moher D, Wager E, Middleton P, et al. (2008) CONSORT for reporting randomized controlled trials in journal and conference abstracts: explanation and elaboration. PLoS Med 5(1): e20.S. HopewellM. ClarkeD. MoherE. WagerP. Middleton2008CONSORT for reporting randomized controlled trials in journal and conference abstracts: explanation and elaboration.PLoS Med5(1)e20
  7. 7. Berwanger O, Ribeiro RA, Finkelsztejn A, Watanabe M, Suzumura EA, et al. (2009) The quality of reporting of trial abstracts is suboptimal: survey of major general medical journals. J Clin Epidemiol 4: 387–392.O. BerwangerRA RibeiroA. FinkelsztejnM. WatanabeEA Suzumura2009The quality of reporting of trial abstracts is suboptimal: survey of major general medical journals.J Clin Epidemiol4387392
  8. 8. Li AR (1995) Problems of abstracts published in 1993 in Chinese Journal of Internal Medicine. Zhonghua Nei Ke Za Zhi 34: 65–66.AR Li1995Problems of abstracts published in 1993 in Chinese Journal of Internal Medicine.Zhonghua Nei Ke Za Zhi346566
  9. 9. Hu BL (1997) Quality of abstracts analysis of the Chinese Journal of Arteriosclerosis from 1994 to 1996. Chin J Arteriosclerosis 5: 85–90.BL Hu1997Quality of abstracts analysis of the Chinese Journal of Arteriosclerosis from 1994 to 1996. ChinJ Arteriosclerosis58590
  10. 10. Trakas K, Addis A, Kruk D, Buczek Y, Iskedjian M, et al. (1997) Quality assessment of pharmacoeconomic abstracts of original research articles in selected journals. Pharmacoeconomics 31: 423–428.K. TrakasA. AddisD. KrukY. BuczekM. Iskedjian1997Quality assessment of pharmacoeconomic abstracts of original research articles in selected journals.Pharmacoeconomics31423428
  11. 11. Bhandari M, Devereaux PJ, Guyatt G, Cook DJ, Swiontkowski MF, et al. (2002) An observational study of orthopaedic abstracts and subsequent full-text publications. J Bone Joint Surg 84: 615–621.M. BhandariPJ DevereauxG. GuyattDJ CookMF Swiontkowski2002An observational study of orthopaedic abstracts and subsequent full-text publications.J Bone Joint Surg84615621
  12. 12. Xu L, Li J, Zhang M, Ai C, Wang L (2008) Chinese authors do need CONSORT: reporting quality assessment for five leading Chinese medical journals. Contemp Clin Trials 29: 727–731.L. XuJ. LiM. ZhangC. AiL. Wang2008Chinese authors do need CONSORT: reporting quality assessment for five leading Chinese medical journals.Contemp Clin Trials29727731
  13. 13. Liu YL, Yang KH, Wu TX, Zhang PZ, Tian JH, et al. (2006) Quality evaluation of randomize controlled trials involving microwave therapy for cervical erosion. Chin J Evid Based Med 6: 768–774.YL LiuKH YangTX WuPZ ZhangJH Tian2006Quality evaluation of randomize controlled trials involving microwave therapy for cervical erosion.Chin J Evid Based Med6768774
  14. 14. Chang J, Li TQ, Wan MH, Zhang Y (2006) Quality Assessment for Randomized Controlled Trials Published in Four Acta of Traditional Chinese Medicine. Chin J Evid Based Med 6: 171–178.J. ChangTQ LiMH WanY. Zhang2006Quality Assessment for Randomized Controlled Trials Published in Four Acta of Traditional Chinese Medicine.Chin J Evid Based Med6171178
  15. 15. Zang XL, Li J, Zhang MM, Yuan WM (2006) Assessing the reporting quality of randomized controlled trials on acupuncture for acute ischemic stroke using the CONSORT Statement and STRICTA. Chin J Evid Based Med 6: 586–590.XL ZangJ. LiMM ZhangWM Yuan2006Assessing the reporting quality of randomized controlled trials on acupuncture for acute ischemic stroke using the CONSORT Statement and STRICTA.Chin J Evid Based Med6586590
  16. 16. Shen J, Tang X, Zou K (2007) Quality assessment of the reporting of randomized controlled trials of traditional chinese medicine for chronic fatigue syndrome. Chin J Evid Based Med 7: 385–391.J. ShenX. TangK. Zou2007Quality assessment of the reporting of randomized controlled trials of traditional chinese medicine for chronic fatigue syndrome.Chin J Evid Based Med7385391
  17. 17. Wang G, Mao B, Xiong ZY, Fan T, Chen XD, et al. (2007) The quality of reporting of randomized controlled trials of traditional Chinese medicine: a survey of 13 randomly selected journals from mainland China. Clin Ther 29: 1456–1467.G. WangB. MaoZY XiongT. FanXD Chen2007The quality of reporting of randomized controlled trials of traditional Chinese medicine: a survey of 13 randomly selected journals from mainland China.Clin Ther2914561467
  18. 18. Zhang D, Yin P, Freemantle N, Jordan R, Zhong N, et al. (2008) An assessment of the quality of randomised controlled trials conducted in China. Trials 9: 22.D. ZhangP. YinN. FreemantleR. JordanN. Zhong2008An assessment of the quality of randomised controlled trials conducted in China.Trials922
  19. 19. Chen YL, Du L, He DD, Zhang QW, Wang MS, et al. (2008) Reading Habits of Authors in the Chinese Journal of Evidence-Based Medicine: A Questionnaire Survey. Chinese Journal of Evidence-Based Medicine 8: 312–314.YL ChenL. DuDD HeQW ZhangMS Wang2008Reading Habits of Authors in the Chinese Journal of Evidence-Based Medicine: A Questionnaire Survey.Chinese Journal of Evidence-Based Medicine8312314
  20. 20. Barry HC, Ebell MH, Shaughnessy AF, Slawson DC, Nietzke F (2001) Family physicians' use of medical abstracts to guide decision making: style or substance? Journal of the American Board of Family Practice 14: 437–442.HC BarryMH EbellAF ShaughnessyDC SlawsonF. Nietzke2001Family physicians' use of medical abstracts to guide decision making: style or substance?Journal of the American Board of Family Practice14437442
  21. 21. Instructions for Authors (2003) Chin J Evid Based Med 3: 254–258.Instructions for Authors2003Chin J Evid Based Med3254258
  22. 22. Moher D, Jones A, LePage L (2001) Use of the CONSORT Statement and quality of reports of randomized trials: a comparative before-and after evaluation. JAMA 285: 1992–1995.D. MoherA. JonesL. LePage2001Use of the CONSORT Statement and quality of reports of randomized trials: a comparative before-and after evaluation.JAMA28519921995
  23. 23. Mills EJ, Wu P, Gagnier J, Devereaux PJ (2005) The quality of randomized trial reporting in leading medical journals since the revised CONSORT Statement. Contemp Clin Trials 26: 480–487.EJ MillsP. WuJ. GagnierPJ Devereaux2005The quality of randomized trial reporting in leading medical journals since the revised CONSORT Statement.Contemp Clin Trials26480487
  24. 24. Klassen TP, Pham B, Lawson ML, Moher D (2005) For randomized controlled trials, the quality of reports of complementary and alternative medicine was as good as reports of conventional medicine. J Clin Epidemiol 58: 763–768.TP KlassenB. PhamML LawsonD. Moher2005For randomized controlled trials, the quality of reports of complementary and alternative medicine was as good as reports of conventional medicine.J Clin Epidemiol58763768
  25. 25. China and India join WHO clinical trial registry platform. China and India join WHO clinical trial registry platform. www.who.int/mediacentre/news/releases/2007/pr41/en/index.html. www.who.int/mediacentre/news/releases/2007/pr41/en/index.html.