16S Ribosomal Ribonucleic Acid Gene Polymerase Chain Reaction in the Diagnosis of Bloodstream Infections: A Systematic Review and Meta-Analysis

Objective We aim to evaluate the accuracy of the 16S ribosomal ribonucleic acid (rRNA) gene polymerase chain reaction (PCR) test in the diagnosis of bloodstream infections through a systematic review and meta-analysis. Methods A computerized literature search was conducted to identify studies that assessed the diagnostic value of 16S rRNA gene PCR test for bloodstream infections. Study quality was assessed using the revised Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool. We calculated the sensitivity, specificity, positive likelihood ratio (PLR), negative likelihood ratio (NLR), diagnostic odds ratio (DOR) and their 95% confidence intervals (95% CI) for each study. Summary receiver operating characteristic (SROC) curve was used to summarize overall test performance. Statistical analysis was performed in Meta-DiSc 1.4 and Stata/SE 12.0 software. Results Twenty-eight studies were included in our meta-analysis. Using random-effect model analysis, the pooled sensitivity, specificity, PLR, NLR, and DOR were 0.87 (95% CI, 0.85–0.89), 0.94 (95% CI, 0.93–0.95), 12.65 (95% CI, 8.04–19.90), 0.14 (95% CI, 0.08–0.24), and 116.76 (95% CI, 52.02–262.05), respectively. The SROC curve indicated that the area under the curve (AUC) was 0.9690 and the maximum joint sensitivity and specificity (Q*) was 0.9183. In addition, heterogeneity was statistically significant but was not caused by the threshold effect. Conclusion Existing data suggest that 16S rRNA gene PCR test is a practical tool for the rapid screening of sepsis. Further prospective studies are needed to assess the diagnostic value of PCR amplification and DNA microarray hybridization of 16S rRNA gene in the future.


Introduction
Bloodstream infections (BSIs) remain a major cause of morbidity and mortality especially in the Intensive Care Unit [1][2][3][4][5]. Moreover, inadequate antibiotic therapy is associated with higher mortality rates. Early microbiological diagnosis is of paramount importance for appropriate antibiotic treatment which increases the survival rate of patients [4]. Therefore, it is evident that a rapid, sensitive and specific diagnosis of BSIs is urgently needed.
Conventional identification methods have several limitations such as lack of rapidity and sensitivity. Blood cultures followed by conventional identification methods are currently the reference method for the detection of pathogens in blood. This well-established method can detect a wide range of microorganisms. However, disadvantages do exist, as the time to detection is often too long. After the detection of bacteria by conventional blood culture, identification and assessment of antibiotic sensitivity take at least a further 24 h [6,7]. The sensitivity is also unacceptably low in the detection of pathogenic bacteria in cases of low-grade bacteremia, in cases where blood cultures are inoculated without adequate sample volume, and in cases where antibiotics are used before blood samples are taken [8,9]. If blood cultures were negative, repetition of sampling would been required, while positive cases need further identification of the isolated microorganism using different culture media and biochemical tests [9]. For these reasons, development of detection methods that provide more rapid results and higher sensitivity is expected to optimize use of antibiotics.
An ideal diagnostic tool for BSIs should be rapid, sensitive and unaffected by antibiotic therapy. In recent years, polymerase chain reaction (PCR) assay using the 16S ribosomal ribonucleic acid (rRNA) gene has been used as a diagnostic tool in many setting [1,9,10]. This test is based on the rationale that 16S rRNA gene of bacteria comprises both conserved and variable regions [11]-the conserved regions are targeted by universal primers for identification of bacterial infection and the variable regions by genus or species-specific assays [2]. Amplified target regions may then be subjected to downstream applications such as sequence analysis and microarray hybridization [5,9,11,12]. In addition, the PCR test has the advantages of amplifying minute amounts of DNA, even from nonviable bacteria [13], and costing less money than blood cultures in both negative and positive cases [9].
However, the results of these studies were variable although inspiring. Some studies revealed the diagnosis of BSIs by 16S rRNA gene PCR test with no less than 95% sensitivity and 95% specificity [1,14,15], whereas others reported low sensitivity values ranging from 41% to 90% [9,[16][17][18] and low specificity values ranging from 32% to 90% [2,3,19,20]. Therefore, we performed a systematic review and meta-analysis to evaluate the accuracy of 16S rRNA gene PCR test compared with conventional blood culture in the diagnosis of BSIs.

Search strategy
This systematic review was performed according to the PRISMA Statement [21] (S1 PRISMA Checklist) and Cochrane Collaboration guidelines (http://handbook.cochrane.org/). A systematic literature search was performed for studies that assessed the diagnostic value of 16S rRNA gene PCR test for BSIs. We searched PubMed, Embase, the Cochrane Library, Clinical-Trials.gov (www.clinicaltrials.gov) and the World Health Organization International Trials Registry Platform search portal (http://apps.who.int/trialsearch/Default.aspx) up to March, 2015. The search terms were "RNA, Ribosomal, 16S", "16S ribosomal ribonucleic acid gene", "16S rRNA gene", "16S rDNA", "sepsis", "bloodstream infections", "bacteremia", and "septicemia". We gave the detailed search strategies in S1 Table. Our searches were not limited by publication date, country or language. The databases search was conducted independently by two authors (Guoming Su and Yueying Wang). To ensure comprehensive acquisition of literature, we also manually searched for any additional studies in the reference lists of retrieved studies and recent reviews.

Inclusion and exclusion criteria
Studies were then included if they met the following inclusion criteria: (1) studies assessed diagnostic value of 16S rRNA gene PCR test for BSIs such as neonatal sepsis and bacteremia; (2) Blood cultures followed by conventional identification methods were used as the reference standard; (3)each study contained no less than ten specimens; and (4) Studies provided sufficient data to allow construction of two-by-two tables. If the same experimental results were repeatedly or multiply published, only the most informative publication was included. Relevant articles were excluded if they were review articles, meta-analysis, commentaries, letters, or case reports. Two reviewers (Guoming Su and Zhuqing Fu) independently screened studies according to eligibility criteria. Disagreements were resolved by consensus.

Data extraction and quality assessment
Two reviewers (Guoming Su and Zhuqing Fu) independently extracted information from eligible studies using a predefined data extraction form, and then another reviewer (Yueying Wang) verified them. Disagreements between reviewers were resolved through discussion. The following information was subtracted from the studies: the first author, publication year, country, disease type, participant characteristics, specimen type, test methods, cut-off of index test, and data for a two-by-two table (true positive, false positive, false negative, and true negative information), respectively.
The quality of each study was assessed using the revised Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool (http://www.bris.ac.uk/quadas/). This tool comprises 4 domains that discuss patient selection, index test, reference standard, and flow and timing [22]. Each domain is assessed in terms of risk of bias by using signaling questions which are answered with"yes," "no," or "unclear". And risk of bias is judged as "low", "high", or "unclear". In addition, the first 3 domains are simultaneously assessed in terms of concerns regarding applicability which are also rated as ''low", ''high", or ''unclear" with the similar criteria.

Definitions of amplification methods
Amplification methods are defined as following: (1)PCR is defined as a conventional PCR amplification strategy that targets 16S rRNA gene in microorganisms. (2)Real-time PCR is defined that is an amplification method with a real-time monitoring system. (3)PCR-hybridization is defined as an amplification method followed by reverse hybridization or DNA microarray hybridization. (4)PCR-sequencing is defined that is an amplification method followed by sequence analysis. (5) RT-qPCR is a reverse transcription-quantitative PCR method for the determination of copy number of PCR templates such as RNA or cDNA in a PCR reaction. (6)FQ-PCR is a fluorescent quantitative PCR method for the determination of copy number of amplification cycles.

Data analysis
All analyses were undertaken using the Meta-DiSc 1.4 [23] and Stata/SE 12.0 software [24]. The Spearman model was applied to assess heterogeneity caused by different cut-off threshold effects, while the heterogeneity that was caused by other factors was checked using Cochran-Q value and I 2 test [25,26] for diagnostic odds ratio (DOR). If heterogeneity (ρ <0.05 or I 2 >50%) was statistically significant among studies, the random-effect model [27] was performed for the meta-analysis; otherwise, the fixed-effect model [28] was chosen. For each study, the following indexes of test accuracy were calculated: sensitivity, specificity, positive likelihood ratio (PLR), negative likelihood ratio (NLR), diagnostic odds ratio (DOR), and their 95% confidence intervals (95% CI). Summary receiver operating characteristic (SROC) curve was used to summarize overall test performance [29]. The area under the curve (AUC) and Q point value (Q Ã ) were also counted to assess the overall performance of the diagnostic test accuracy [30]. In addition, subgroup and meta-regression analysis [31,32] were conducted to explore the possible sources of heterogeneity among studies. Publication bias was inspected using Deeks' funnel plot asymmetry test [33]. A two-sided p-value of <0.05 was considered to be statistically significant.

Search results
A total of 1,715 titles and abstracts were found from initial searches of the electronic database. Four records were identified through reviewing the references of the other meta-analysis and reviews. Firstly, 367 records were excluded using EndNote X6 due to duplication. We applied the inclusion and exclusion criteria to filter out 1,286 records, because they were considered as review articles, meta-analysis, commentaries, letters, case reports, or records about apparently irrelevant to study question. Leaving 66 articles were eligible for further full-text review. Subsequently, additional 38 articles were further excluded after a full-text review. The list of full-text excluded articles, along with detailed reasons for exclusion, is presented in the supporting information (S2 Table). Finally, a total of 28 studies that met inclusion criteria were included in the present meta-analysis [1-5, 9, 10, 12, 14-20, 34-46]. The details of study selection flow are summarized in Fig 1.

Characteristics and quality of the studies
The main characteristics of the included studies are shown in Table 1. Our meta-analysis included 28 studies which were published between 1997 and 2014. A total of 7,378 specimens were taken from infants to adults. Of 7,378 specimens, 219 were venous access ports [10], 10 were cerebrospinal fluids [38,40,41], and the rest were blood samples. All specimens were confirmed by conventional identification methods which were currently the reference standard in the diagnosis of bacterial infections.
The detailed quality information of the included studies is shown in S3 Table. According to QUADAS-2 tool, 24 (85.7%) studies were at low risk of patient selection bias. A similar situation was observed in the flow and timing. As for index test and reference standard, the overwhelming majority (82.1% and 92.9%, respectively) studies were at high or unclear risk due to insufficient information to judge whether their test results were interpreted blind. Only 6 studies reported blinded interpretation of index test [1,10,12,18,19,36], and 2 studies reported the blinded interpretation of reference standard [35,37]. From an overall perspective, the qualities of the reported studies all turned out to be moderate to high concerns about applicability. Risk of bias and applicability concerns graph is presented in Fig 2. Risk of bias and applicability concerns summary is presented in

Heterogeneity analysis and diagnostic accuracy
Spearman correlation coefficient of sensitivity and 1-specificity was found to be -0.177 with a ρ value of 0.367, indicating that there was no heterogeneity caused by the threshold effect. As was depicted in the Fig 4, statistically significant heterogeneity was observed when we pooled DOR of included studies. The result suggested that there should be other factors rather than threshold effect resulting in variations in accuracy estimates. We performed the univariable meta-regression analysis based on the publication year, sample size, disease type (sepsis or bacteremia), population characteristics (neonates or adult), and PCR test (qualitative or quantitative). Meta-regression analysis indicated that disease type and PCR test were significantly (p<0.05) associated with specificity and that population characteristics were significantly (p<0.05) related to the sensitivity. The detailed results of meta-regression analysis are presented in S4 Table and S1 Fig. Therefore, we decided that a random-effects model was used to eliminate some heterogeneity.

Publication bias
The Deeks' test did not indicate any strong statistical evidence of publication bias, with ρ-value of 0.24 for the overall analysis. The shape of the funnel plot of the pooled DOR of 16S rRNA gene PCR in the diagnosis of BSIs also did not show any evidence of obvious asymmetry (Fig  7), indicating that there was no potential publication bias.

Discussion
A previous meta-analysis had demonstrated that 16S rRNA gene PCR test had excellent sensitivity and specificity in the diagnosis of bacterial meningitis [47]. However, we did not know whether the test had a similar effect to BSIs. Studies focusing on the diagnostic value of 16S rRNA gene PCR test have been conducted in recent years. Thus, in this study we evaluated the accuracy of 16S rRNA gene PCR test in the diagnosis of BSIs. Our results showed that the pooled sensitivity of the 16S rRNA gene PCR test was 0.87 and the pooled specificity was 0.94. This test may be a valid tool for confirming the diagnosis of BSIs, although not perfect. To illustrate the overall performance of 16S rRNA gene PCR test, we also counted the AUC and Q Ã of the SROC curve. A SROC curve is usually used to summarize overall test performance, while the AUC under the SROC curve is a measure of the overall performance of a diagnostic test to accurately differentiate those with and those without the condition of interest [30]. Q Ã is defined by the intercept of the SROC, which is closest to the ideal top-left corner of the SROC space and which corresponds to the highest value of sensitivity and specificity for the test [30,48]. In present meta-analysis, the data showed that the AUC and Q Ã were 0.9690 and 0.9183, indicating very good ability to diagnose BSIs. The DOR reflects the relationship between the result of the diagnostic test and the disease, the value of which ranges from 0 to infinity-higher values indicating better discriminatory test performance [49]. Our meta-analysis showed that the pooled DOR was 116.76, suggesting a high level of overall accuracy. Compared with the DOR and SROC curve, the likelihood ratio (PLR and NLR) is considered to be more clinically meaningful for our measures of diagnostic accuracy [50]. The PLR represents the value by which the odds of the disease increase when a test is  positive. Whereas NLR shows the value by which the odds of the disease decrease when a test is negative. The PLR value was 12.65 in the overall analysis, which suggested that patients with a positive PCR result had a about 13-fold chance of being diagnosed with BSIs rather than non-BSIs. On the other hand, the NLR was 0.14, which suggested that if a PCR result was negative, the probability rate of the individual having BSIs was 14% in theory.  Heterogeneity is a potential problem in interpreting the results of any meta-analysis. The threshold effect arises when differences in sensitivities and specificities occur due to different cut-offs or thresholds used in different studies to define a positive or negative test result [23]. We took the threshold effect as the first factor in our meta-analysis. We used the Spearman correlation coefficient to analyze the threshold effect. The result showed that Spearman value was found to be -0.177 (ρ = 0.367) using Meta-Disc analysis, suggesting that the heterogeneity was not caused by the threshold effect. However, the Cochran-Q value and I 2 test showed that the heterogeneity among studies was too obvious to be ignored. To find the possible reasons  for heterogeneity, we undertook a univariable meta-regression analysis based on the publication year, sample size, disease type (sepsis or bacteremia), population characteristics (neonates or adult), and PCR test (qualitative or quantitative). Unexpectedly, we found that disease type, population characteristics and PCR test were attributable to the sources of heterogeneity. Thus, we performed the subgroup analyses according to population characteristics, disease type and test methods. Compared with the previous meta-analysis conducted by Pammi et al [51], we found that there was similar specificity (0.96, 95% CI: 0.94-0.97) and sensitivity (0.90, 95% CI: 0.78-0.95) to our subgroup analysis based on sepsis. It was noteworthy that PCR-hybridization test was more accurate in distinguishing patients with BSIs from non-BSIs people than other amplification methods, whereas the result should be interpreted with caution due to limited data and heterogeneity. But we did not conduct subgroup analyses based on the fluorescent quantitative PCR (FQ-PCR) and reverse transcription-quantitative PCR (RT-qPCR) owing to limited original data.
Similar to other meta-analyses, several limitations should be acknowledged. Firstly, While the methodological quality of studies was assessed according to the QUADAS-2 tool, most studies were at unclear risk bias in index test and reference standard due to lacking of blinding results interpreted. Secondly, only published English and Chinese language studies were included in this meta-analysis, so the language bias might influence the results. Thirdly, the cutoff values varied widely, which made it difficult to determine the optimized cut-off value. Finally, Only 902 (12.2%) of 7,378 specimens had positive blood culture results, which could lead to broad variance about sensitivity.
In conclusion, our study is the first comprehensive meta-analysis to date that has assessed the accuracy of 16S rRNA gene PCR test in the diagnosis of BSIs. Despite the limitations mentioned above, the current evidence suggests that the 16S rRNA gene PCR test is a rapid, practical and valid tool for confirming the diagnosis of BSIs, especially sepsis. However, there is insufficient data to fully confirm diagnostic accuracy of PCR-hybridization test. Further metaanalysis involving more prospective studies with analysis of subgroups by amplification methods should be performed in the future.