The quality of reporting in systematic reviews (SRs)/meta-analyses (MAs) of diagnostic tests published by authors in China has not been evaluated. The aims of present study are to evaluate the quality of reporting in diagnostic SRs/MAs using the PRISMA statement and determine the changes in the quality of reporting over time.
According to the inclusion and exclusion criteria, we searched five databases including Chinese Biomedical Literature Database, PubMed, EMBASE, the Cochrane Library, and Web of knowledge, to identify SRs/MAs on diagnostic tests. The searches were conducted on July 14, 2012 and the cut off for inclusion of the SRs/MAs was December 31st 2011. The PRISMA statement was used to assess the quality of reporting. Analysis was performed using Excel 2003, RevMan 5.
A total of 312 studies were included. Fifteen diseases systems were covered. According to the PRISMA checklist, there had been serious reporting flaws in following items: structured summary (item 2, 22.4%), objectives (item 4, 18.9%), protocol and registration (item 5, 2.6%), risk of bias across studies (item 15, 26.3%), funding (item 27, 28.8%). The subgroup analysis showed that there had been some statistically significant improvement in total compliance for 9 PRISMA items after the PRISMA was released, 6 items were statistically improved regarding funded articles, 3 items were statistically improved for CSCD articles, and there was a statistically significant increase in the proportion of reviews reporting on 22 items for SCI articles (P<0.050).
The numbers of diagnostic SRs/MAs is increasing annually. The quality of reporting has measurably been improved over the previous years. Unfortunately, there are still many deficiencies in the reporting including protocol and registration, search, risk of bias across studies, and funding. Future Chinese reviewers should address issues on these aspects.
Citation: Ge L, Wang J-c, Li J-l, Liang L, An N, Shi X-t, et al. (2014) The Assessment of the Quality of Reporting of Systematic Reviews/Meta-Analyses in Diagnostic Tests Published by Authors in China. PLoS ONE 9(1): e85908. https://doi.org/10.1371/journal.pone.0085908
Editor: Brett Thombs, McGill University, Canada
Received: August 3, 2013; Accepted: December 6, 2013; Published: January 21, 2014
Copyright: © 2014 Ge et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: The authors have no support or funding to report.
Competing interests: The authors have declared that no competing interests exist.
Systematic reviews (SRs) and meta-analyses (MAs) of diagnostic tests will potentially have an increasing role in healthcare as decision makers check the evidence before implementing new diagnostic technologies . However, only high quality diagnostic SRs can provide the best evidence for clinical decision makers. Low quality SRs may mislead clinical practice, which may arise at a primary study level, due to flaws in the design, execution and reporting of the component studies , . Like all other types of SRs, diagnostic SRs are also prone to a number of shortcomings .
In order to improve the quality of diagnostic SRs, quality evaluation of the primary studies has become an important element of the review process. The publication of the Quality Assessment of Diagnostic Accuracy Studies (QUADAS) tool  provides investigators with a means of assessing the major domains that affect the validity of diagnostic study. As a matter of fact, some experiences, reports, feedbacks from users or academic organizations have suggested the potential to improve this tool. Therefore, the QUADAS Group revised it and then launched QUADAS-2 in 2011 . The Standards for the Reporting of Diagnostic accuracy studies (STARD) initiative  has given investigators a reporting framework that undoubtedly improves the quality of primary studies.
SRs of diagnostic tests have been developing for nearly two decades abroad . The first SR of diagnostic tests was published in Chinese journal in 2001 . Since then, an increasing numbers of SRs of diagnostic tests have been published in China, however, the methodological and reporting quality of these reviews varies widely. Therefore, it is vital to assess the quality of diagnostic SRs before being used for healthcare policy or clinical decision making.
Methodological quality considers how well the SR is conducted (including literature searching, pooling of data, etc.). Reporting quality considers how well systematic reviewers have reported their methodology and findings . There is no specialized tool to evaluate the quality of SRs of diagnostic research. As a newer standard of reporting SR, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) was released to replace the Quality of Reporting of Meta-analyses (QUOROM) for guiding the review reporting . The updated PRISMA statement was based on the conceptual and practical advances that had been made in the science of SRs. The complete PRISMA statement consists of a 27-item checklist, along with a flow diagram . Most of the checklist items are relevant when reporting SRs of non-randomized studies assessing the benefits and drawbacks of interventions, however, the need to modify or incorporate additional items for diagnostic accuracy studies should be recognized .
Willis BH et al. have investigated the quality of reporting of MAs in diagnostic researches published in English prior to December 31st, 2008, and found that the quality of reporting had many flaws . It is unclear whether the quality of reporting of SRs/MAs on diagnostic researches published by authors in China have a similar result, and further validation is necessary. Thus, the objective of this study is to examine the quality of reporting of published SRs/MAs in diagnostic tests by authors in China, according to their compliance with the PRISMA statement.
Data Sources and Searches
Two independent reviewers systematically searched the following electronic databases: Chinese Biomedical Literature Database, PubMed, EMBASE, the Cochrane Library, and Web of science. Searches were conducted using a combination of the following terms: “systematic reviews”, “systematic review”, “meta analysis”, “meta analyses”, “meta-analysis”, “meta-analyses”, “sensitivity”, “specificity”, “China”, “Chinese”. The searches were implemented on July 14, 2012 and the cut off for inclusion of the SRs/MAs was December 31st 2011. The syntaxes were adjusted corresponding to different database. The detailed search algorithms for each database were listed in Text S1.
Two independent reviewers selected articles according to the inclusion and exclusion criteria designed in advance. Disagreements were resolved in consultation with Dr. Tian. After discarding duplication studies by Endnote X3 (The Thomson Reuters, Britain), we reviewed all the abstracts, identified potentially eligible articles and citations for which a decision could not be made from the abstract. We then managed to retrieve the full - text of these articles to determine whether they were eligible.
The inclusion criteria were as follows: (a) The SRs/MAs on diagnostic tests were published by authors in China. The diagnostic test was defined as a technology to distinguish between patients with disease (or more generally, a specified target disorder) and those without disease. Target disorders were considered to be pathological processes and not related to a success or failure of an intervention, such as successful placing of stents. In such a test accuracy study, the results of ‘index test’ were compared with those of the reference standard determined in the same patient. The reference standard should be the best available method for identifying patients that have the target disorders. (b) One of the objectives of the included studies was to estimate a measure of the performance of the diagnostic test; articles that reported relevant data would be extracted; (c) The qualitative and quantitative comprehensive analyses were performed for all included studies; (d) The search terms should be explicitly stated and include sensitivity or specificity.
The exclusion criteria were as follows: (a) Repeated literature, review and methodological literature, original research, animal research and case reports. (b) Conference abstracts and letters to the journal editors; (c) The SRs/MAs on other fields.
The PRISMA statement  which consists of a twenty seven-point checklist was used to evaluate the overall quality of reporting of meta analyses. To indicate the degree of compliance, each checklist item was assigned one of three responses: ‘Yes’ for total compliance; ‘partial’ for partial compliance; and ‘No’ for non-compliance.
Data Collection and Analysis
Data collection was carried out independently by two reviewers using standard data extraction forms (Table 1). All disagreements were resolved by discussion or the third researcher.
Extraction of data included the following items: title; publication year; publication journals; publication language; number of authors; affiliations of authors; number of affiliations; index test; reference standard; funding sources; categories of disease; method used to assess the quality for original studies including QUADAS, and responses to the PRISMA statement.
According to An N et al. , we performed subgroup analysis on the quality of reporting for total compliance in each items of PRISMA by t-test. The factors of subgroup analysis were presented as following: the year of publication (≤2008 vs. ≥2009), the affiliations (hospital vs. university), funding sources (funded vs. non-funded), Chinese Science Citation Database (CSCD) vs. non-CSCD, Science Citation Index (SCI) vs. non-SCI. The odds ratio (OR) value and 95% CI was used as the summary statistic for subgroup comparisons through Mantel-Haenszel and Std, by using a fixed-effect model. If the OR was undefined, then Fisher’s exact test was used. The ORs for each item represent responses of “Yes” for group 1 versus responses of “Yes” for group 2; OR>1 represents that the quality of group 1 is better than group 2. Statistical significance was set at P≤0.050. Analysis was performed using Excel 2003 and RevMan 5.0 software.
3,214 records were retrieved as the result of searches. 2,767 articles were excluded due to duplication, non diagnostic tests, conference abstracts, animal researches, non Chinese authors, or not being a SR/MA. After examination of the full texts of 355 articles, a further 43 reviews were excluded because they were original researches, reviews, or duplication. A total of 312 SRs and MAs were included, 235 in Chinese and 77 in English (Figure 1).
Characteristics of the Included SRs/MAs (Table 2)
312 SRs/MAs were written by authors in China and published in 169 different journals. Only 134 (43.0%) were published in the journals cited by Chinese Science Citation Database, with 74 (23.7%) being published in Science Citation Index Journals. In which, the impact factor (2011) was between 0.9 and 11.7, and the impact factor for 12 reviews was higher than 5.0, dispersed throughout eight journals as following: Hepatology (impact factor (IF): 11.7), Archives of Internal Medicine (IF:11.5), Thorax (IF: 6.8), Radiotherapy and Oncology (IF: 5.6), European Journal of Cancer (IF: 5.5), American Journal of Kidney Diseases (IF: 5.4), Intensive care medicine (IF:5.4), CHEST (IF:5.3). Figure 2 illustrated the number of SRs/MAs per publication year in the included set. The first diagnostic SR was published in 2001, nearly 89.0% (278 reviews) were published after the year 2007. According to ICD-10 , 312 studies involved 15 system diseases. The most common disorders were neoplasms (42.3%), diseases of the digestive system (10.3%), and certain infectious and parasitic diseases (10.3%). More than half of (66.4%) the SRs/MAs were written by clinicians. A wide range of diagnostic tests were featured in the reviews, with laboratory technologies (51.0%) and imaging technologies (45.2%) being the most common category of tests evaluated. 28.9% (90 reviews) reported funding sources. 88.1% (275 reviews) studies were composed by more than three authors. None of the SRs/MAs had been updated from a previous review. Characteristics of the included reviews were shown in Table 2.
Tools of Quality Assessment of Primary Studies
All reviews published before 2003 reported no formal quality assessment of the primary studies. However, most of (69.6%) reviews used the QUADAS tool for quality assessment in 2011, although other methods of quality assessment were still being used (see Figure 3).
PRISMA Checklist Assessment (Table 3)
Generally compliance with PRISMA was poor: none of included 312 SRs/MAs fulfilled all 27 items of PRISMA. 70.0% of PRISMA criteria were complied within the following items: title (item 1, 82.4%), rational (item 3, 89.7%), eligibility criteria (item 6, 88.1%), information sources (item 7, 87.8%), summary measures (item 13, 74.4%), synthesis of results (item 14, 85.3%), study characteristics (item 18, 78.8%), results of individual studies (item 20, 84%), summary of evidence (item 24, 85.6%). Total compliance with PRISMA items was less than 30.0% in structured summary (item 2, 22.4%), objective (item 4, 18.9%), protocol and registration (item 5, 2.6%), risk of bias across studies (item 15, 26.3%), funding (item 27, 28.8%). Detailed information was presented in Table 3.
Comparison of the quality of reporting of included SRs/MAs based on the publication time (≤2008 vs. ≥2009).
To determine whether there had been an improvement in the quality of reporting after the PRISMA statement was released. The full compliance in each of the PRISMA criteria were compared between the period in or before 2008 and the period in or after 2009 (Figure 4). The results showed that there was an improvement in the following items after the PRISMA statement was released, which was significant difference (P<0.050): structured summary (item 2), rational (item 3), objective (item 4), data items (item 11), risk of bias in individual studies (item 12), study selection (item 17), study characteristics (item 18), risk of bias with studies (item 19), limitations (item 25). However, other items didn’t show significant difference or there was no improvement after the PRISMA statement was released.
Comparison of the quality of reporting of included SRs/MAs based on the affiliations (Hospital vs. University).
To investigate whether there had been a difference in the quality of reporting in different affiliations. SRs/MAs published by hospital were compared with that published by university for full compliance in each of the PRISMA criteria (Figure 5). The studies published by hospital had an improvement, and there had been a significant difference in limitations (item 25).
Comparison of the quality of reporting of included SRs/MAs based on the funding sources (Funded vs Non-funded).
The funded articles were compared with non-funded articles for full compliance in all the PRISMA criteria (Figure 6). We found that funded articles had an improvement and there was a significant difference (P<0.050) in objective (item 4), protocol and registration (item 5), search (item 8), risk of bias with studies (item 15), study selection (item 17), conclusions (item 26), funding (item 27). However, there was no statistical difference in other items.
Comparison of the quality of reporting of included SRs/MAs from CSCD vs non CSCD.
Chinese Science Citation Database (CSCD) has a collection of all core and excellent journals in China. By comparing the quality of reporting of SRs/MAs from CSCD and from non-CSCD, using the PRISMA criteria (Figure 7), we found that there was a statistically significant difference (P<0.050) in study selection (item 9), data collection process (item 10), summary measures (item 13). However, there was not statistical difference in other items.
Comparison of the quality of reporting of included SRs/MAs from SCI vs non SCI.
We included all SRs/MAs published in Chinese and English by authors in China. A comparison was conducted on the quality of reporting between SCI articles and non-SCI articles according to full compliance (Figure 8), there was statistical difference in 22 items (81.5%), and the quality of SCI articles was better compared with non-SCI articles.
Characteristics of Included Diagnostic SRs or MAs
The number of diagnostic SRs/MAs written by authors in China is increasing annually, 75.3% of SRs/MAs were published in China. Nearly 89.0% were published after the year 2007, a large part of this probably resulted from the introduction QUADAS checklist  in 2007. The diagnostic SRs/MAs published by hospitals were 66.3%, indicating that clinicians also devoted to evidence production, which would be beneficial to evidence transforming clinical practice. Only 11.5% research were conducted by 1–2 authors, indicating that the result of data collection and quality assessment were reliable for most of diagnostic SRs/MAs. Most of these articles were published in evidence-based medical journals, and they should be an important evidence sources for diagnostic tests. The impact factor was higher than 5.0 for 12 SRs/MAs. 312 studies involved 15 symptom systems, and neoplasms (42.3%) was a hot topic for researchers. The imaging technologies and laboratory technologies were main diagnostic mode for included studies, which probably due to the fact that it is easy to conform the quantitative index and the testing threshold in imaging technologies and laboratory technologies.
The Quality of Reporting of Included Diagnostic SRs/MAs
Our PRISMA results showed that the quality of reporting still needs further improvement, the main problems were identified in following four areas: The compliance with PRISMA was not good in structured summary and objectives. The reporting flaws were found in most of (50.0%) included reviews in methods section: protocol and registration, search, study selection, data items, risk of bias across studies, and additional analyses including subgroup analysis and sensitivity analysis). These flaws would affect the quality of diagnostic SRs/MAs. The reporting rate of study selection, risk of bias across studies and additional analyses was less than 50.0% in results section. Even though some of SRs/MAs reported risk of bias across studies and additional analyses in methods section, the analytic results were not presented in results section. Similarly, some of SRs/MAs reported risk of bias across studies and additional analyses in results section, but the analytic results were not reported in methods section. It would affect the integrity and accuracy of researches. The reporting of conclusions was incomplete, most of studies only provided a general interpretation of the results, however, the reporting of implications for future research was poorly provided.
Comparisons of the Quality of Reporting of SRs/MAs Published by Authors in China and Authors Abroad
Willis BH et al have investigated the quality of reporting of meta-analyses in diagnostic tests published by international authors prior to December 31st, 2008 . We compared the total compliance (Yes) in each item of PRISMA by t-test in regarding to SRs/MAs published by authors in China and authors abroad (Text S2). The results showed that the quality of reporting of diagnostic SRs/MAs published by authors in China had more serious reporting flaws in the following items when compared with SRs/MAs published by authors abroad, and the difference was statistically significant (the compliance for PRISMA was less than 50.0%): structured summary (22.4% vs. 34.8%, P = 0.002), objectives (18.9% vs. 66%, P<0.001), risk of bias across studies (methods section, 26.3% vs. 64.4%, P<0.001), risk of bias across studies (results section, 34.4% vs. 57.6%, P<0.001), funding (28.9% vs. 48.3%, P<0.001). It is worth mentioning that the reporting quality of diagnostic SRs/MAs published by authors in China was higher statistically in following items when compared with SRs/MAs published by authors abroad: search (methods section, 38.5% vs. 25.0%, P<0.001), risk of bias in individual studies (methods section, 65.4% vs. 43.6%, P<0.001), study characteristics (results section, 78.9% vs. 55.9%, P<0.001), results of individual studies (results section, 85.0% vs. 68.6%, P = 0.020).
Although both Chinese and English databases were searched, and study selection, data extraction and quality assessment were conducted independently by two reviewers, there are still some limitations to our study due to the different level of understanding PRISMA statement between different researchers. Second, our study only included diagnostic SRs/MAs, however, the SRs/MAs on pathogenesis, prognosis, genetic polymorphism, and intervention were not included. Third, in our search, we searched the term “systematic review” and “meta-analysis” in titles, abstracts or keywords. Some potentially eligible systematic reviews may, however, not use these terms in their titles or abstracts. PRISMA statement is generic checklist aimed at improving the reporting of all types of SRs and does not contain some of the more specific nuances of diagnostic test reviews. However, most of items of PRISMA are applicative regarding to diagnostic SRs/MAs, especially eligibility criteria, search, risk of bias with studies, risk of bias across studies, and additional analysis etc. The quality of reporting for these items was presented in Table 2. Finally, the included reviews evaluated a range of diagnostic tests involving 15 symptom systems and different diagnostic technologies. It is unlikely that the quality of reporting will be completely independent of variation in these factors, therefore, the effect of heterogeneity on the results needs to be considered.
We have investigated the quality of reporting of 312 SRs/MAs related to diagnostic tests published by authors in China. The results demonstrate that the number of diagnostic SRs/MAs is increasing annually. The quality of reporting has measurably improved over the previous years. Unfortunately, there are still many flaws in the areas including protocol and registration, search, risk of bias across studies, additional analyses. We propose following strategies for future research: Since a protocol can pre-specify the objectives and methods of the systematic review , it is important to provide registration information including the registration number. There is great demand to establish a registry platform for SRs/MAs in China. The instructions to authors should include related reporting items, and the submission of PRISMA checklist. The researchers should update timely knowledge on SRs/MAs. The reviewers writing SRs/MAs, especially Chinese reviewers, should pay more attention to area such as search, risk of bias across studies, additional analyses.
The detailed search algorithms for each database.
Comparisons of the quality of reporting of Systematic reviews/Meta-analyses published between authors in China and authors abroad.
Theoretical support was given by Evidence Based Medical Center at Lanzhou University. We thank teacher Jin-Hui Tian, Ya-rui Li, and Jun-feng Xu for their help on this work.
Conceived and designed the experiments: LG JcW JhT. Analyzed the data: LG LL NA XtS YcL. Contributed reagents/materials/analysis tools: LG JcW JhT. Wrote the paper: LG JhT. Extracted the data: JlL LL NA XtS YcL. Final approval of manuscript: LG JcW JlL JhT.
- 1. Willis BH, Quigley M (2011) The assessment of the quality of reporting of meta-analyses in diagnostic research: a systematic review. BMC Med Res Methodol. 11: 163.
- 2. Devillé WL, Buntinx F, Bouter LM, Montori VM, de Vet HC, et al. (2002) Conducting systematic reviews of diagnostic studies: didactic guidelines. BMC Medical Research Methodology 2: 9.
- 3. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, et al. (2003) Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Standards for Reporting of Diagnostic Accuracy. Clin Chem 49(1): 1–6.
- 4. Whiting P, Rutjes AW, Dinnes J, Reitsma J, Bossuyt PM, et al.. (2004) Development and validation of methods for assessing the quality of diagnostic accuracy studies. Health Technol Assess 8(25):ii,1–234.
- 5. Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, et al. (2011) QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 155(8): 529–536.
- 6. Wu L, Zhang Y, Zeng XT (2013) The QUADAS-2 Tool for the Quality Assessment of Diagnostic Accuracy Study: an Introduction. Journal of Hubei University of Medicine 32(3): 201–208.
- 7. Liu J, Yang Q, Lan YJ (2001) Expression of human telomerase in lung cancer: A meta-analysis of diagnostic test. Chin J Lung Cancer 4(4): 241–244.
- 8. Shea BJ, Bouter LM, Peterson J, Boers M, Andersson N, et al. (2007) External Validation of a Measurement Tool to Assess Systematic Reviews (AMSTAR). PLOS ONE 2(12): e135.
- 9. Moher D, Liberati A, Tetzlaff J, Altman DG (2010) PRISMA Group (2010) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg 8: 336–341.
- 10. Swartz MK (2011) The PRISMA Statement: A Guideline for Systematic Reviews and Meta-Analyses.J Pediatr Health Care. 25: 1–2.
- 11. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, et al. (2009) The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol 62(10): e1–34.
- 12. An N, Xu JF, Ge L, Liang L, Shi XT, et al. (2013) Report quality assessment of systematic reviews or meta-analyses of interventions published in Chinese Journal of Evidence-Based Pediatrics. Chin J Evid Bases Pediatr 8(2): 87–92.
- 13. Dong JW (2008) International Statistical Classification of Diseases and Related Health Problems. People’s Health Publishing House.
- 14. Whiting PF, Weswood ME, Rutjes AW, Reitsma JB, Bossuyt PN, et al. (2006) Evaluation of QUADAS, a tool for the quality assessment of diagnostic accuracy studies. BMC Med Res Methodol 6: 9.