Reporting Quality of Systematic Reviews/Meta-Analyses of Acupuncture

Background The QUOROM and PRISMA statements were published in 1999 and 2009, respectively, to improve the consistency of reporting systematic reviews (SRs)/meta-analyses (MAs) of clinical trials. However, not all SRs/MAs adhere completely to these important standards. In particular, it is not clear how well SRs/MAs of acupuncture studies adhere to reporting standards and which reporting criteria are generally ignored in these analyses. Objectives To evaluate reporting quality in SRs/MAs of acupuncture studies. Methods We performed a literature search for studies published prior to 2014 using the following public archives: PubMed, EMBASE, Web of Science, the Cochrane Database of Systematic Reviews (CDSR), the Chinese Biomedical Literature Database (CBM), the Traditional Chinese Medicine (TCM) database, the Chinese Journal Full-text Database (CJFD), the Chinese Scientific Journal Full-text Database (CSJD), and the Wanfang database. Data were extracted into pre-prepared Excel data-extraction forms. Reporting quality was assessed based on the PRISMA checklist (27 items). Results Of 476 appropriate SRs/MAs identified in our search, 203, 227, and 46 were published in Chinese journals, international journals, and the Cochrane Database, respectively. In 476 SRs/MAs, only 3 reported the information completely. By contrast, approximately 4.93% (1/203), 8.81% (2/227) and 0.00% (0/46) SRs/Mas reported less than 10 items in Chinese journals, international journals and CDSR, respectively. In general, the least frequently reported items (reported≤50%) in SRs/MAs were “protocol and registration”, “risk of bias across studies”, and “additional analyses” in both methods and results sections. Conclusions SRs/MAs of acupuncture studies have not comprehensively reported information recommended in the PRISMA statement. Our study underscores that, in addition to focusing on careful study design and performance, attention should be paid to comprehensive reporting standards in SRs/MAs on acupuncture studies.


Introduction
Systematic reviews (SRs) and meta-analyses (MAs) summarize large amounts of evidence and are a valuable tool for keeping clinicians up to date within their specialty [1,2]. As with all research, however, the value of SRs/MAs depends on how the analyses are performed, the actual findings, and the clarity of reporting [3]. If key information is reported poorly, the potential usefulness of the SRs/MAs is diminished.
Since 1987, numerous researchers have recognized the need to evaluate the quality of these types of reviews. For example, in 1987 Sacks and colleagues [4] evaluated reporting in SRs/MAs and found that it was inadequate. The Consolidated Standards of Reporting Trials (CONSORT) Group subsequently developed the Quality of Reporting of Meta-Analyses (QUOROM) statement to address suboptimal MA reporting. Ten years later, an updated QUOROM statement-entitled Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statementwas developed and published [3]. The PRISMA statement consists of a checklist of 27 study reporting items such as title, abstract, methods, results, discussion, and funding sources. The checklist is intended to guide authors of SRs/MAs to improve the consistency and quality of reporting.
Acupuncture, a traditional medicine technique, has been widely used in clinical practice for thousands of years in China and many western countries. The number of published SRs/MAs of acupuncture studies has increased considerably in recent years. As the transparency and completeness of SRs/MAs in many fields is still not optimal [5][6][7][8], we examined how well SR/MA reporting standards have been followed in the field of acupuncture and compared adherence to these standards in acupuncture SRs/MAs published in three different types of journals/databases.

Methods
The protocol for this study was written in Chinese and has not been published. The study was not a classical systematic review, but we tried to report it according to PRISMA Checklist [3] (Text S1).

Inclusion/exclusion criteria
We included all SRs or MAs of acupuncture published in Chinese journals, international journals, and the Cochrane Database of Systematic Reviews (CDSR) prior to 2014. The experimental group of SRs/MAs of acupuncture studies was also compared with a control group of SRs/MAs of studies of other interventions, such as herbal medicine, massage and western medicine. Participants: human in any conditions, not animal; Intervention: acupuncture; Comparisons: sham acupuncture or other interventions, such as herbal medicine, massage, western medicine, etc; Outcomes: no limitations; Study design: SRs/MAs. We excluded SRs/MAs that focused primarily on traditional Chinese medicine (TCM) other than acupuncture (e.g., herbal medicine, massage).

Screening
The titles and abstracts of the studies were independently screened by at least two reviewers (Jiao Huang, Xu Zhao, or Rui Zhang) based on inclusion and exclusion criteria, and the full text of potentially suitable articles was retrieved for further assessment (Text S2).

Data extraction and analysis
Data were extracted independently by at least two reviewers (Rui Zhang, Jiao Huang, Xu Zhao, or Danlu Liu) in accordance with the PRISMA checklist and the assessment checklist for SRs/ MAs of acupuncture studies developed for this study. Inconsistencies were subsequently resolved by discussion between the two reviewers or final decisions were made by the third principal investigator (Yali Liu). Data input utilized a standardized form and was done by trained data extractors (Wanting Sun, Pen Zhang, and Hua Cao). The form consisted of a general characteristics section (title, first author, funding source, study design, disease(s) examined, diagnostic criteria, intervention, and outcome) and a 27-item PRISMA information section (including title, abstract, introduction, method, results, discussion, and funding). Each item was assessed as ''yes'' if it was described in the paper or ''no'' if it was not (Text S3). Data were summarized with descriptive statistical analysis. For continuous data, the means 6 SD was provided and one-way ANOVA was used. Data that followed a normal distribution were compared using the LSD-t test. Dichotomous data were summarized with descriptive statistical analysis (frequency, percentage). Pearson's x 2 test and/or Fisher's exact test were used to assess differences in reporting among groups. P values less than 0.05 were considered significant. All statistical analyses were performed using Microsoft Excel (version 2007) and SPSS (version 13.0) software.

Search
Our initial literature search identified 3993 potential SRs/MAs of acupuncture-related studies. After closer examination, 476 were chosen for inclusion in our analysis (Text S4). Of these, 203, 227, and 46 were published in Chinese journals, international journals, and CDSR, respectively. A flow chart of the literature search is shown in Figure 1.

General characteristics
General characteristics of the SRs/MAs analyzed are summarized in Table 1 Acupuncture SRs/MAs in Chinese journals were conducted entirely by Chinese authors, whereas those published in international journals tended to be multinational collaborations, with Chinese first authors being most prevalent (27.31%, 62/227). Chinese authors were also most prevalent first authors in CDSR studies (34.78%, 16/46). The percentage of published acupuncture SRs/MAs in Chinese journals, international journals, and the CDSR that reported at least one funding source was 54.19% (110/ 203), 50.66% (115/227), and 89.13% (41/46), respectively, and the maximum number of funding sources reported was 5, 3, and 7, respectively.
The majority of SRs/MAs (95.37%, 454/476) included at least one randomized controlled trial (RCT). Nervous system diseases, musculoskeletal system diseases, and mental illness were most frequently examined (23.11, 18.91 and 13.44%, respectively). Approximately 38% (180/476) of the SRs/MAs reported western diseases or TCM syndromes in their diagnostic criteria. All acupuncture SRs/MAs examined described the interventions in detail. 39.92% (190/476) and 19.54% (93/476) SRs/MAs included adverse events and quality-of-life in the outcome which were reported as primary and/or secondary outcomes.

Discussion
Over the last decade, numerous studies have assessed the quality of reporting in SRs/MAs by their compliance with assessment instruments such as the QUOROM and PRISMA statements [9][10][11][12]. These studies focused predominantly on SRs/MAs covering diagnostic research and critical care. Although some quality assessment studies have looked at acupuncture SRs/MAs [13][14][15], they have focused mainly on methodological diversity in database searching, risk of bias, and heterogeneity in search strategies among CDSR. Our study compared reporting quality and PRISMA compliance in acupuncture SRs/MAs between different journal types.
We found that the five PRISMA items, namely ''Protocol and registration'', ''Risk of bias across studies'' (both in the methods and results), and ''Additional analyses'' (both in the methods and results) in the methods and results, are not frequently reported, indicating that the overall quality of reporting in acupuncture SRs/MAs is far from adequate. Compared with SRs/MAs published in CDSR, those in Chinese and international journals were of inferior reporting quality. One possible explanation for the limited compliance may be that journals have failed to incorporate the PRISMA statement into their instructions to authors about submitting SRs/MAs [16]. We also found that SR/MA reporting was more complete in SCI journals than in non-SCI journals but that both require improvement in adherence to PRISMA standards.
Several studies have focused on the reporting quality of SRs/ MAs covering the fields of TCM [17,18], physical therapy [19], orthopaedics [20], and oral implantology [21] field, which showed that the reporting quality was indeed poor. Although differences exist between these results and those we repot here, the reporting of major items in the PRISMA statement was similar to what we found in our present study. Additional, Fleming PS et al. [22] found that the quality of reporting was considerably better in reviews published in CDSR (P,0.001) than in non-CDSR.
Both the QUOROM and PRISMA statements encourage the use of specific terms in the titles of SRs/MAs, which help to identify these studies. Because of the special title format requirements of the CDSR, however, SRs/MAs published in this database cannot conform to the QUOROM/PRISMA recommendation.
Unequivocal descriptions of the scientific background and rationale for using acupuncture in the treatment of both western diseases and TCM syndromes provide the reader with a better understanding of the research context and rationale of SRs/MAs. In this respect, SRs/MAs in the CDSR were more explicit in their descriptions than those in international or Chinese journals.
The importance of protocol consistency and registration of SRs/MAs to the transparency of reporting is underscored by the fact that they are considered key aspects of the ''reporting guidelines for systematic review protocols'' in the international prospective register of systematic reviews (PROSPERO) [23,24]. We found that only SRs/MAs published in the CDSR provided protocol and registration details.
The PRISMA standards suggest that methodological details such as eligibility criteria, information sources, search strategies, study selection criteria, and data collection processes are necessary to judge the quality and accuracy of SRs/MAs. The majority of the SRs/MAs published in the CDSR adequately reported these items, whereas those published in Chinese and international journals did not. Eligibility criteria are an aspect of the PICOS criteria (participants, interventions, comparisons, outcomes, and study design) central to the PRISMA approach. We propose that it is equally important that search strategies be uniformly reported. Many international journals require information about search strategies in at least one database, and the flexibility of the CDSR layout allows reporting of search strategies for multiple databases. Chinese journals, however, rarely request search strategy information. There is also considerable need for more consistency in the databases obtain acupuncture studies. We propose that, AcuBriefs (www.acubriefs.com), AcuBase (www.acubase.fr), Acu-doc2 RCT (www.acubriefs.com/), and the TCM database are the most systematic and comprehensive sources for acupuncture information. Chinese RCTs make up the highest proportion of primary studies included in acupuncture SRs/MAs. If methods for sequence generation, allocation concealment, and study blindness are not adequately described, low-quality studies [25] may mislead reviewers.  We found that there is also considerable inconsistency in reporting of study selection criteria. For example, many primary studies on acupuncture report a random allocation design but are not specific enough for the reader to determine if they are actual RCTs. We propose that these uncertainties should be clarified by contacting the primary authors to determine the appropriateness of including the studies in the SRs/MAs. Because it has been suggested that only 6.8% of acupuncture efficacy studies published in Chinese journals are based on actual RCTs [25], we strongly propose that authors of SRs/MAs verify this information prior to inclusion of studies.
Acupuncture is considered an alternative or complementary treatment to western medical interventions such as drugs and surgery, and it can be considered a separate specialty. Thus, SRs/ MAs on acupuncture require not only compliance with general PRISMA reporting standards but also accurate reporting of acupuncture information. As a result, it is necessary to develop an extension of the PRISMA statement for acupuncture.
There are several limitations to our study. First, our analyses were limited to acupuncture-specific SRs/MAs and therefore may not be applicable to SRs/MAs in other fields. Second, our assessment process was not blinded, and therefore the outcomes may be influenced by publication date and other factors. Third, our assessment criteria (yes or no) did not allow partial information to be used. Fourth, our study focused primarily on acupuncture rather than other TCM. We failed to distinguish acupuncture from herbal medicine massage, or western medicine because individual SRs/MAs we included in our analysis often contained several control groups rather than one group.
In summary, SRs/MAs of acupuncture studies have not comprehensively reported the information recommended in the PRISMA statement. Our study underscores that, in addition to focusing on careful study design and performance, attention should be paid to comprehensive reporting standards when publishing SRs/MAs of acupuncture studies.

Supporting Information
Checklist S1 PRISMA Checklist.

(DOC)
Text S1 The English and Chinese databases search strategy.