The Assessment of the Quality of Reporting of Systematic Reviews/Meta-Analyses in Diagnostic Tests Published by Authors in China

Background The quality of reporting in systematic reviews (SRs)/meta-analyses (MAs) of diagnostic tests published by authors in China has not been evaluated. The aims of present study are to evaluate the quality of reporting in diagnostic SRs/MAs using the PRISMA statement and determine the changes in the quality of reporting over time. Methods According to the inclusion and exclusion criteria, we searched five databases including Chinese Biomedical Literature Database, PubMed, EMBASE, the Cochrane Library, and Web of knowledge, to identify SRs/MAs on diagnostic tests. The searches were conducted on July 14, 2012 and the cut off for inclusion of the SRs/MAs was December 31st 2011. The PRISMA statement was used to assess the quality of reporting. Analysis was performed using Excel 2003, RevMan 5. Results A total of 312 studies were included. Fifteen diseases systems were covered. According to the PRISMA checklist, there had been serious reporting flaws in following items: structured summary (item 2, 22.4%), objectives (item 4, 18.9%), protocol and registration (item 5, 2.6%), risk of bias across studies (item 15, 26.3%), funding (item 27, 28.8%). The subgroup analysis showed that there had been some statistically significant improvement in total compliance for 9 PRISMA items after the PRISMA was released, 6 items were statistically improved regarding funded articles, 3 items were statistically improved for CSCD articles, and there was a statistically significant increase in the proportion of reviews reporting on 22 items for SCI articles (P<0.050). Conclusion The numbers of diagnostic SRs/MAs is increasing annually. The quality of reporting has measurably been improved over the previous years. Unfortunately, there are still many deficiencies in the reporting including protocol and registration, search, risk of bias across studies, and funding. Future Chinese reviewers should address issues on these aspects.


Introduction
Systematic reviews (SRs) and meta-analyses (MAs) of diagnostic tests will potentially have an increasing role in healthcare as decision makers check the evidence before implementing new diagnostic technologies [1]. However, only high quality diagnostic SRs can provide the best evidence for clinical decision makers. Low quality SRs may mislead clinical practice, which may arise at a primary study level, due to flaws in the design, execution and reporting of the component studies [2,3]. Like all other types of SRs, diagnostic SRs are also prone to a number of shortcomings [1].
In order to improve the quality of diagnostic SRs, quality evaluation of the primary studies has become an important element of the review process. The publication of the Quality Assessment of Diagnostic Accuracy Studies (QUADAS) tool [4] provides investigators with a means of assessing the major domains that affect the validity of diagnostic study. As a matter of fact, some experiences, reports, feedbacks from users or academic organizations have suggested the potential to improve this tool. Therefore, the QUADAS Group revised it and then launched QUADAS-2 [5]in 2011 [6]. The Standards for the Reporting of Diagnostic accuracy studies (STARD) initiative [3] has given investigators a reporting framework that undoubtedly improves the quality of primary studies.
SRs of diagnostic tests have been developing for nearly two decades abroad [1]. The first SR of diagnostic tests was published in Chinese journal in 2001 [7]. Since then, an increasing numbers of SRs of diagnostic tests have been published in China, however, the methodological and reporting quality of these reviews varies widely. Therefore, it is vital to assess the quality of diagnostic SRs before being used for healthcare policy or clinical decision making.

Items Interpretation
Methodological quality considers how well the SR is conducted (including literature searching, pooling of data, etc.). Reporting quality considers how well systematic reviewers have reported their methodology and findings [8]. There is no specialized tool to evaluate the quality of SRs of diagnostic research. As a newer standard of reporting SR, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) was released to replace the Quality of Reporting of Meta-analyses (QUOROM) for guiding the review reporting [9]. The updated PRISMA statement was based on the conceptual and practical advances that had been made in the science of SRs. The complete PRISMA statement consists of a 27-item checklist, along with a flow diagram [10]. Most of the checklist items are relevant when reporting SRs of non-randomized studies assessing the benefits and drawbacks of interventions, however, the need to modify or incorporate additional items for diagnostic accuracy studies should be recognized [11].
Willis BH et al. have investigated the quality of reporting of MAs in diagnostic researches published in English prior to December 31 st , 2008, and found that the quality of reporting had many flaws [1]. It is unclear whether the quality of reporting of SRs/MAs on diagnostic researches published by authors in China have a similar result, and further validation is necessary. Thus, the objective of this study is to examine the quality of reporting of published SRs/MAs in diagnostic tests by authors in China, according to their compliance with the PRISMA statement.

Data Sources and Searches
Two independent reviewers systematically searched the following electronic databases: Chinese Biomedical Literature Database, PubMed, EMBASE, the Cochrane Library, and Web of science. Searches were conducted using a combination of the following terms: ''systematic reviews'', ''systematic review'', ''meta analysis'', ''meta analyses'', ''meta-analysis'', ''meta-analyses'', ''sensitivity'', ''specificity'', ''China'', ''Chinese''. The searches were implemented on July 14, 2012 and the cut off for inclusion of the SRs/MAs was December 31 st 2011. The syntaxes were adjusted corresponding to different database. The detailed search algorithms for each database were listed in Text S1.

Selection
Two independent reviewers selected articles according to the inclusion and exclusion criteria designed in advance. Disagreements were resolved in consultation with Dr. Tian. After discarding duplication studies by Endnote X3 (The Thomson Reuters, Britain), we reviewed all the abstracts, identified potentially eligible articles and citations for which a decision could not be made from the abstract. We then managed to retrieve the full -text of these articles to determine whether they were eligible.
The inclusion criteria were as follows: (a) The SRs/MAs on diagnostic tests were published by authors in China. The diagnostic test was defined as a technology to distinguish between patients with disease (or more generally, a specified target disorder) and those without disease. Target disorders were considered to be pathological processes and not related to a success or failure of an intervention, such as successful placing of stents. In such a test accuracy study, the results of 'index test' were compared with those of the reference standard determined in the same patient. The reference standard should be the best available method for identifying patients that have the target disorders. (b) One of the objectives of the included studies was to estimate a measure of the performance of the diagnostic test; articles that reported relevant data would be extracted; (c) The qualitative and quantitative comprehensive analyses were performed for all included studies; (d) The search terms should be explicitly stated and include sensitivity or specificity.
The exclusion criteria were as follows:

Quality Assessment
The PRISMA statement [10] which consists of a twenty sevenpoint checklist was used to evaluate the overall quality of reporting of meta analyses. To indicate the degree of compliance, each checklist item was assigned one of three responses: 'Yes' for total compliance; 'partial' for partial compliance; and 'No' for noncompliance.

Data Collection and Analysis
Data collection was carried out independently by two reviewers using standard data extraction forms ( Table 1). All disagreements were resolved by discussion or the third researcher. Extraction of data included the following items: title; publication year; publication journals; publication language; number of authors; affiliations of authors; number of affiliations; index test; reference standard; funding sources; categories of disease; method used to assess the quality for original studies including QUADAS, and responses to the PRISMA statement.
According to An N et al. [12], we performed subgroup analysis on the quality of reporting for total compliance in each items of PRISMA by t-test. The factors of subgroup analysis were presented as following: the year of publication (#2008 vs. $2009), the affiliations (hospital vs. university), funding sources (funded vs. non-funded), Chinese Science Citation Database (CSCD) vs. non-CSCD, Science Citation Index (SCI) vs. non-SCI. The odds ratio (OR) value and 95% CI was used as the summary statistic for subgroup comparisons through Mantel-Haenszel and Std, by using a fixed-effect model. If the OR was

Search
3,214 records were retrieved as the result of searches. 2,767 articles were excluded due to duplication, non diagnostic tests, conference abstracts, animal researches, non Chinese authors, or not being a SR/MA. After examination of the full texts of 355 articles, a further 43 reviews were excluded because they were original researches, reviews, or duplication. A total of 312 SRs and MAs were included, 235 in Chinese and 77 in English (Figure 1). (Table 2) 312 SRs/MAs were written by authors in China and published in 169 different journals. Only 134 (43.0%) were published in the journals cited by Chinese Science Citation Database, with 74 (23.7%) being published in Science Citation Index Journals. In which, the impact factor (2011) was between 0.9 and 11.7, and the impact factor for 12 reviews was higher than 5.0, dispersed throughout eight journals as following: Hepatology (impact factor According to ICD-10 [13], 312 studies involved 15 system diseases. The most common disorders were neoplasms (42.3%), diseases of the digestive system (10.3%), and certain infectious and parasitic diseases (10.3%). More than half of (66.4%) the SRs/ MAs were written by clinicians. A wide range of diagnostic tests were featured in the reviews, with laboratory technologies (51.0%) and imaging technologies (45.2%) being the most common category of tests evaluated. 28.9% (90 reviews) reported funding sources. 88.1% (275 reviews) studies were composed by more than three authors. None of the SRs/MAs had been updated from a previous review. Characteristics of the included reviews were shown in Table 2.
Subgroup Analysis on the Quality of Reporting (Figures 4-8) Comparison of the quality of reporting of included SRs/ MAs based on the publication time (#2008 vs. $2009). To determine whether there had been an improvement in the quality of reporting after the PRISMA statement was released. The full compliance in each of the PRISMA criteria were compared between the period in or before 2008 and the period in or after 2009 ( Figure 4). The results showed that there was an improvement in the following items after the PRISMA statement was released, which was significant difference (P,0.050): structured summary (item 2), rational (item 3), objective (item 4), data items (item 11), risk of bias in individual studies (item 12), study selection (item 17), study characteristics (item 18), risk of bias with studies (item 19), limitations (item 25). However, other items didn't show significant difference or there was no improvement after the PRISMA statement was released.
Comparison of the quality of reporting of included SRs/ MAs based on the affiliations (Hospital vs. University). To investigate whether there had been a difference in the quality of reporting in different affiliations. SRs/MAs published by hospital were compared with that published by university for full compliance in each of the PRISMA criteria ( Figure 5). The studies published by hospital had an improvement, and there had been a significant difference in limitations (item 25).
Comparison of the quality of reporting of included SRs/ MAs based on the funding sources (Funded vs Nonfunded). The funded articles were compared with non-funded articles for full compliance in all the PRISMA criteria ( Figure 6). We found that funded articles had an improvement and there was Comparison of the quality of reporting of included SRs/ MAs from CSCD vs non CSCD. Chinese Science Citation Database (CSCD) has a collection of all core and excellent journals in China. By comparing the quality of reporting of SRs/MAs from CSCD and from non-CSCD, using the PRISMA criteria (Figure 7), we found that there was a statistically significant difference (P,0.050) in study selection (item 9), data collection process (item 10), summary measures (item 13). However, there was not statistical difference in other items.
Comparison of the quality of reporting of included SRs/ MAs from SCI vs non SCI. We included all SRs/MAs published in Chinese and English by authors in China. A comparison was conducted on the quality of reporting between SCI articles and non-SCI articles according to full compliance (Figure 8), there was statistical difference in 22 items (81.5%), and the quality of SCI articles was better compared with non-SCI articles.

Characteristics of Included Diagnostic SRs or MAs
The number of diagnostic SRs/MAs written by authors in China is increasing annually, 75.3% of SRs/MAs were published in China. Nearly 89.0% were published after the year 2007, a large part of this probably resulted from the introduction QUADAS checklist [14] in 2007. The diagnostic SRs/MAs published by hospitals were 66.3%, indicating that clinicians also devoted to evidence production, which would be beneficial to evidence transforming clinical practice. Only 11.5% research were conducted by 1-2 authors, indicating that the result of data collection and quality assessment were reliable for most of diagnostic SRs/MAs. Most of these articles were published in evidence-based medical journals, and they should be an important evidence sources for diagnostic tests. The impact factor was higher than 5.0 for 12 SRs/MAs. 312 studies involved 15 symptom systems, and neoplasms (42.3%) was a hot topic for researchers. The imaging technologies and laboratory technologies were main diagnostic mode for included studies, which probably due to the fact that it is easy to conform the quantitative index and the testing threshold in imaging technologies and laboratory technologies. The Quality of Reporting of Included Diagnostic SRs/MAs Our PRISMA results showed that the quality of reporting still needs further improvement, the main problems were identified in following four areas: The compliance with PRISMA was not good in structured summary and objectives. The reporting flaws were found in most of (50.0%) included reviews in methods section: protocol and registration, search, study selection, data items, risk of bias across studies, and additional analyses including subgroup analysis and sensitivity analysis). These flaws would affect the quality of diagnostic SRs/MAs. The reporting rate of study selection, risk of bias across studies and additional analyses was less than 50.0% in results section. Even though some of SRs/MAs reported risk of bias across studies and additional analyses in methods section, the analytic results were not presented in results section. Similarly, some of SRs/MAs reported risk of bias across studies and additional analyses in results section, but the analytic results were not reported in methods section. It would affect the integrity and accuracy of researches. The reporting of conclusions was incomplete, most of studies only provided a general interpretation of the results, however, the reporting of implications for future research was poorly provided.

Limitations
Although both Chinese and English databases were searched, and study selection, data extraction and quality assessment were conducted independently by two reviewers, there are still some limitations to our study due to the different level of understanding PRISMA statement between different researchers. Second, our study only included diagnostic SRs/MAs, however, the SRs/MAs on pathogenesis, prognosis, genetic polymorphism, and intervention were not included. Third, in our search, we searched the term ''systematic review'' and ''meta-analysis'' in titles, abstracts or keywords. Some potentially eligible systematic reviews may, however, not use these terms in their titles or abstracts. PRISMA statement is generic checklist aimed at improving the reporting of all types of SRs and does not contain some of the more specific nuances of diagnostic test reviews. However, most of items of PRISMA are applicative regarding to diagnostic SRs/MAs, especially eligibility criteria, search, risk of bias with studies, risk of bias across studies, and additional analysis etc. The quality of reporting for these items was presented in Table 2. Finally, the included reviews evaluated a range of diagnostic tests involving 15 symptom systems and different diagnostic technologies. It is unlikely that the quality of reporting will be completely independent of variation in these factors, therefore, the effect of heterogeneity on the results needs to be considered.

Conclusion
We have investigated the quality of reporting of 312 SRs/MAs related to diagnostic tests published by authors in China. The results demonstrate that the number of diagnostic SRs/MAs is increasing annually. The quality of reporting has measurably improved over the previous years. Unfortunately, there are still many flaws in the areas including protocol and registration, search, risk of bias across studies, additional analyses. We propose following strategies for future research: Since a protocol can prespecify the objectives and methods of the systematic review [11], it is important to provide registration information including the registration number. There is great demand to establish a registry platform for SRs/MAs in China. The instructions to authors should include related reporting items, and the submission of PRISMA checklist. The researchers should update timely knowledge on SRs/MAs. The reviewers writing SRs/MAs, especially Chinese reviewers, should pay more attention to area such as search, risk of bias across studies, additional analyses.