Evaluation of a Commercial Multiplex PCR Assay for Detection of Pathogen DNA in Blood from Patients with Suspected Sepsis

The Magicplex Sepsis Real-time Test (MST) is a commercial multiplex PCR that can detect more than 90 different pathogens in blood, with an analysis time of six hours. The aim of the present study was to evaluate this method for the detection of bloodstream infection (BSI). An EDTA whole blood sample for MST was collected together with blood cultures (BC) from patients with suspected sepsis at the Emergency Department of a university hospital. Among 696 study patients, 322 (46%) patients were positive with at least one method; 128 (18%) were BC positive and 268 (38%) were MST positive. Considering BC to be the gold standard, MST had an overall sensitivity of 47%, specificity of 66%, positive predictive value (PPV) of 23%, and a negative predictive value of 87%. Among the MST positive samples with a negative BC, coagulase-negative staphylococci (CoNS) and species that rarely cause community-acquired BSI were frequently noted. However, the quantification cycle (Cq) values of the MST+/BC- results were often high. We thus hypothesized that the performance of the MST test could be improved if the Cq cut-off level was adjusted downwards. With a lower Cq cut-off value, i.e. 6.0 for Staphylococcus species and 9.0 for all other species, the number of MST positive cases decreased to 83 (12%) and the overall sensitivity decreased to 38%. However, the PPV increased to 59% and the specificity increased to 96%, as many MST positive results for CoNS and bacteria that rarely cause community-acquired BSI turned MST negative. In conclusion, our study shows that with a lower Cq cut-off value, the MST will detect less contaminants and findings with unclear relevance, but to the cost of a lower sensitivity. Consequently, we consider that a positive MST results with a Cq value above the adjusted cut-off should be interpreted with caution, as the result might be clinically irrelevant. In a correspondent way, quantitative results could probably be useful in the interpretation of positive results from other molecular assays for the detection of BSI.


Introduction
Sepsis is a major cause of morbidity and mortality [1,2]. Early, accurate administration of antibiotics is the most important intervention for successful outcome in this serious condition [3,4]. In international guidelines [5] it is recommended that effective antibiotic treatment should be given as soon as sepsis is suspected, preferably within one hour. Broad-spectrum antibiotics are often recommended as empiric treatment. However, with the emerging problem of antibiotic resistance, it is essential to administer narrow-spectrum antibiotics, more specifically targeting the causative pathogen, as soon as possible.
Blood culture (BC) is the current 'gold standard' for diagnosing bloodstream infections (BSI), but the method has several limitations. Sensitivity decreases greatly when antibiotics have been given prior to culture [6,7], and slow-growing, fastidious organisms can be difficult to detect by BC [8]. It has been shown that a positive BC is found in around only half of patients with sepsis and septic shock [9,10]. Furthermore, it takes about 24-72 hours before the pathogen is identified and information about antibiotic susceptibility is available [8]. Consequently, changes in empirical antibiotic therapy are more frequently guided by clinical response than culture results [11]. With new techniques, such as matrix-assisted laser desorption ionization-time of flight (MALDI-TOF) or the FilmArray Blood Culture Identification Panel, the time to identification can be significantly shortened, and it has recently been described that this reduction can lead to a significantly shorter time to accurately targeted antibiotic treatment [12,13].
For optimal empirically selected antibiotic therapy in sepsis, techniques on whole blood for the direct and rapid detection of causative agents are essential. Molecular methods based on polymerase chain reaction (PCR) technology have been developed for this purpose. By using PCR methods on blood samples, it is possible to detect pathogen DNA in blood within hours.
There are a few commercial tests available for the direct molecular identification of pathogens in blood samples [14][15][16]. Out of these tests, only the Light Cycler SeptiFast Test (Roche Diagnostics) has, to date, been well investigated in clinical studies. In a meta-analysis upon the SeptiFast test, Dark et al [17] concluded that the method appeared to have higher specificity than sensitivity, with the reservation of deficiencies in study quality.
More recently, the promising IRIDICA PCR/electrospray ionization-mass spectrometry (PCR/ESI-MS) technology (Abbot), has been launched, but so far this method has only been evaluated in a few study-cohorts [18,19].
The Magicplex Sepsis Real-time PCR (SeeGene, Seoul, Korea) is one of the most recent commercial devices for this purpose. This multiplex PCR is able to detect 73 gram-positive and twelve gram-negative bacteria, three drug resistance markers (mecA, vanA and vanB) and six fungi. So far, clinical validation has been sparse, with only a few studies published on the Magicplex Sepsis Real-time test (MST) [20][21][22]. All these studies, including one study with 375 patients, conducted by Ljungström et al [22], one with 267 patients, conducted by Carrara et al [20] and one with 125 patients, conducted by Loonen et al [21], found many MST positive cases that were not supported by other traditional microbiological tests, and where the clinical picture was doubtful. Based on these findings, we hypothesized that the MST assay may detect low levels of bacterial DNA without clinical relevance, and that the method could possibly be more useful after adjustment downwards of the detection cut-off value.
The aim of the present study was to compare BC with the MST assay using different quantification cycle (Cq) cut-offs in a large study cohort including unselected patients with suspected sepsis. To evaluate the accuracy of positive PCR results without BC-support, such samples were analyzed with another, species-specific PCR method.

Materials and Methods Patients
This study was performed at the Emergency Department of Ö rebro University Hospital, Sweden, between February 2011 and June 2012. During the study period, an EDTA whole blood sample was collected from the same venous puncture as BC samples in consecutive patients subjected to blood culturing. The EDTA blood samples were stored in a Biobank. The Regional Medical Ethics Review Board of Uppsala, Sweden, approved the use of the blood samples for this study, provided that they were made anonymous, and that no patient data were available other than information on gender, age and BC result. No informed consent was thus required from the patients.

Blood culture
Two pairs of BC samples, i.e. four BC bottles, were collected per patient, according to routine practice. For each BC, a volume of 8-10 mL of venous blood was inoculated in one Bactec Aerobic/F bottle and the same volume in one Bactec Plus Anaerobic /F bottle. The BC bottles were incubated in a Bactec blood culturing system (Becton Dickinson and Company, Sparks, MD, USA) for up to seven days.

Magicplex sepsis real-time Test (MST)
Within three days after sample collection, bacterial DNA was extracted from 1 mL EDTA whole blood, using the Blood Pathogen Kit (Molzym, Bremen, Germany) on an Arrow instrument (Diasorin, Solna, Sweden), following the manufacturer's instructions, and frozen at -70˚C pending the PCR run. All runs included a positive and a negative control.
Pathogen detection was performed using the Magicplex Sepsis Real-time Test. According to the manufacturer, the MST has a limit of detection (LOD) of 30 colony-forming units (CFU)/mL. The LOD was tested using blood samples spiked with Staphylococcus aureus, Streptococcus pneumoniae, Pseudomonas aeruginosa and Candida albicans respectively, at known concentrations. All pathogens with concentrations ! 30 CFU/mL were detected, but below 30 CFU/mL the detection rate declined.
The MST process is summarized in Table 1.
Briefly, MST is a three-step procedure with an initial PCR amplification of target nucleic acids using 20 μL of extracted DNA in two different reactions, one for Gram-positive bacteria/ drug resistance genes (GPB/DR) and one for Gram-negative bacteria/Fungi (GNB/Fungi). In the second step, screening for detection of more than 90 different pathogens at the genus level and resistance markers is performed using real-time PCR. This is done in three different reactions where 2 μL of the GPB/DR amplicon are used for screening of GPB /DR, and 2 μL of the GNB/Fungi amplicon are used for screening of GNB/Fungi. Species identification (without additional amplification) is accomplished in the third step where 2 μL of the initial amplicons are used with different Magicplex Sepsis ID (1-9) Real-time Detection kits, depending on the results of the previous screening PCR. Twenty-seven pathogens can be identified at the species level.
The amplification was performed on a GeneAmp 9700 (Applied Biosystems, Foster City, CA, USA) and all real-time PCR analyses were performed on the CFX96 Real-time PCR System (Bio-Rad Laboratories, Hercules, CA, USA).
is generated. A sample is positive if the amount of fluorescence produced rises above a defined threshold level. This point is called the quantification cycle (Cq). Thus, the more target-DNA present in a sample, the lower the Cq value will be. According to the MST manufacturer, a Cq of <10 cycles is needed for positivity for Staphylococcus species, and a Cq of <15 cycles is needed for positivity for all other pathogens.
With the MST, a Cq value is generated both in the second (screening) step and in the third (identification) step. In the present study, we included Cq values from the identification step only, as they are specific for the species identified. For example, a sample containing DNA from both Staphylococcus aureus and Coagulase negative staphylococci (CoNS) will give two separate Cq values in the identification step, but just one in the screening step (Staphylococcus species).
In-house real-time PCR BC negative samples, with positive MST results for S. aureus, Escherichia coli or S. pneumoniae, were further analyzed using species-specific in-house real-time PCR protocols with the same DNA extractions as for MST. The S. aureus and E. coli PCR: s were performed on Rotorgene Q (Qiagen, Hilden, Germany) in a volume of 20 μL including 5μL template DNA.
The S. aureus PCR was run as described previously [23], however, adopted on a Rotorgene Q (Qiagen) using 0.3μM of each nuc primer. The E. coli PCR mix contained 1× Rotor-Gene Probe PCR Master Mix (Qiagen), 0.5 μM respective 0.3 μM of the primers uidA F and uidA R and 0.3μM of a probe uidA P2 [24]. The PCR was run at 95˚C for 3 minutes followed by 40 cycles of 95˚C for 3 seconds and 60˚C for 10 seconds.

Statistics
Medians and ranges were used for descriptive statistics of continuous variables, and percentages for categorical variables. Comparison of groups was performed using the Mann-Whitney

Results
The MST test was run on 703 blood samples from patients with suspected sepsis. Seven samples turned out to be invalid and were excluded. Consequently, samples from 696 patients were included in the study. Ten samples were MST positive only in the screening step (six positive for Staphylococcus species, two positive for Streptococcus species and two positive for the mecA-gene). However, in the present study, these samples were considered to be negative, as we required complete pathogen detection in the third step for a positive MST result. In Fig 1, the inclusion process and overall results are illustrated in a flow-chart. Table 2 shows the distributions of age and gender in the study population.

Blood culture positivity in relation to Cq values
Among MST positive cases, those with concordant positive BC had significantly lower MST Cq values than cases with a negative BC (Fig 2). This pattern was also noted for the three largest pathogen groups, i.e. E. coli, S. aureus, and S. pneumoniae.
Based on these findings, we evaluated the performance of MST at different cut-off values. ROC-curves were generated for MST detection of BSI with S. aureus, E. coli and S. pneumoniae at different MST Cq values. The AUC:s for MST detection were 0.860 for S. aureus BSI, 0.754 for E. coli BSI, and 0.723 for S. pneumoniae BSI. Table 4 shows MST sensitivities, specificities and predictive values for BSI detection of S. aureus, E. coli, and S. pneumoniae, at different cut-off values.
When the lower MST cut-off values were used for all pathogens, we found that overall specificity and PPV clearly increased (Table 5). Cq cut-off value 6.0 for Staphylococcus species and 9.0 for all other pathogens At the cut-off value 6.0 for Staphylococcus species and 9.0 for all other pathogens, the overall sensitivity was 38% (49/128), specificity 96% (546/568), PPV 59% (49/83), and NPV 89% (546/615). Table 3 shows the distribution of BC and MST results after lowering the cut-off value. With the lower cut-off value, the number of MST positive results fell from 362 to 88, and the number of MST positive samples from 268 to 83. The rate of concordant positive BC in cases with a positive MST tests increased from 18% (66/362) to 59% (51/87). Among the 61 patients with concordant positive BC and MST results, 49 (80%) had a Cq value below the lower cut-off   limit. The number of samples positive for CoNS was reduced from 138 to 13 with the lower cut-off. Polymicrobial findings were dramatically reduced, from 78 to 5 samples. All 5 were BC-positive for at least one of the pathogens detected, and in 3 both bacteria were grown in BC. Table 6 shows all cases with polymicrobial findings in BC and in MST with the lower cutoff value.

Discussion
In this study on MST performance in a large study-cohort of patients with suspected sepsis, we found that the MST-positive rate was notably high, with a high frequency of polymicrobial results, CoNS, and other findings of uncertain clinical relevance. A lower Cq cut-off value significantly improved the specificity and the positive predictive value, but to the cost of a lower sensitivity.
The clinical interpretation of a positive MST with a negative BC is difficult, as there are several situations where bacterial DNA can be found in culture-negative blood. First, such a result might represent a true BSI that BC fails to detect. It is known that a positive BC is found in around only half the patients with sepsis and septic shock [9,10]. PCR positivity, even when BC remains negative, was recently found to correlate to disease severity [26]. Second, a positive MST might be due to DNA from dead bacteria, a finding with unclear clinical significance. Third, it might represent a transient bacteremia without clinical relevance. It has been shown that bacteremia is common after dental procedures and tooth-brushing [27,28]. Molecular pathogen detection by 16S RNA sequencing has been performed on blood at repeated intervals after dental extraction, resulting in massive findings of diverse pathogens at low concentrations     [29,30]. Fourth, it can be caused by contamination. Contamination can occur during blood sampling, resulting in positive tests for normal human skin flora such as CoNS. It can also occur in the laboratory; all manual work at the laboratory represents a risk for contamination, and PCR or DNA isolation reagents can be contaminated with bacterial DNA.
Based on the results of the present study, we suspect that results with positive MST and negative BC were often due to contamination or clinical irrelevant DNA-aemia. This assumption is not based on clinical evidence though, but on sepsis epidemiology. A large proportion of the MST+/BC-findings were pathogens that rarely cause community-acquired sepsis, including CoNS (n = 129) Streptococcus agalactiae (n = 26) Stenotrophomona maltophilia (n = 24) and Acinetobacter baumanii (n = 10). Since S.maltophilia and A. baumanii are not part of the normal human flora, these findings were probably due to laboratory contamination despite the use of DNA-free reagents and stringent hygiene routines. The MST method is a procedure in many steps, which might be an advantage to achieve better sensitivities, but it is also a laboratory contamination risk.
Regarding CoNS, most findings in the study were probably due to contamination, but not necessarily all of them. Among the samples with concordant CoNS positivity in MST and BC, two had growth in all four BC bottles, indicating clinical relevance. The Ct values of those samples were 4.3 resp. 6.3, which means that one of them would have been missed with the lower cut-off. MST+/BC-results for common sepsis-causing pathogens such as S. aureus, E. coli, and S. pneumoniae could not be judged as clinically relevant or not with the present study design, as clinical data was not available.
The MST has been described in only a few previous publications, by Carrara et al [20], Loonen et al [21], and Ljungström et al [22].
Carrara and colleagues compared MST and BC results from 267 patients in the ICU, and at emergency and haematology departments. The sensitivities and specificities reported in that study were calculated on a reference standard where clinical data and other cultures were added to BC results in order to see if a positive MST represented a true BSI or not. The overall sensitivity of MST was then 65%. When compared to BC, 29 of 69 BC positive specimens were also MST positive, and consequently the sensitivity was 41%.
The study cohort in the work by Loonen et al [21] was similar to ours, i.e. patients presenting at the emergency unit with suspected sepsis. The study included 125 patients, of whom 12 were positive in both MST and BC, and a further 23, 15 of which being CoNS, by MST only. The sensitivity and PPV were 37% and 30%, respectively. As in our study, the low PPV resulted from the fact that many of samples were MST-positive for CoNS.
The study by Ljungström et al [22] was performed on 375 patients with suspected sepsis at a Swedish emergency unit. The reference standard used for calculations was a combination of BC findings, other relevant culture results, and clinical evaluation. Findings of suspected contaminants such as CoNS were not included in the analyses. They then reached a sensitivity of 64% and a specificity of 96%. If all findings from both MST and BC had been included in the results, the sensitivity in Ljungström's study would have been 38%, the PPV 17% and the specificity 65%. Consequently, this study, as well as ours, shows that the MST certainly detects an important proportion of sepsis pathogens, but also a lot of bacterial DNA in blood that has doubtful clinical significance.
With the lower Cq cut-off value, i.e. 6.0 for Staphylococcus species and 9.0 for all other pathogens, almost all findings with uncertain clinical relevance disappeared. In contrast, the majority of MST-positive samples supported by BC findings remained positive. Recently, we showed a similar pattern for the SeptiFast test [31], where a SeptiFast test with a Cp (equivalent to Cq value) below 17.5 indicated BC positivity.
The purpose of the lower Cq cut-off value in this study was to reduce the frequency of false positive results. However, as a consequence, some true positive MST results became negative, and the sensitivity decreased. The number of BC positive samples with a concordant positive MST fell from 61/128 (48%) to 49/128 (38%) ( Table 3). In addition, some MST+/BC-results that became MST-/BC-with a lower Cq cut-off, might have been clinically relevant, as we know that BC is not an optimal reference standard. Thus, we do not consider that a positive MST test with a Ct value above the lower cut-off limit should be ruled out as clinically irrelevant, but should be interpreted with caution.
A large proportion of the BC positive samples was PCR negative. MST has this problem in common with SeptiFast and other commercial PCR assays for pathogen detection in blood [14,[32][33][34]. One possible explanation for a negative PCR despite a positive BC might be that the sample volume of 1 mL is insufficient. The fact that BC positive samples showing a negative MST were frequently positive in only one or two bottles, whilst the majority of MST positive samples were positive in all 4 bottles, supports the fact that sample volume is an important factor. Using a larger sample volume in the MST assay would almost certainly improve sensitivity. Another explanation to some of the MST-/BC+ results is that BC positivity due to suspected contaminants such as CoNS are included in the analyses. Thirteen samples were BC positive for pathogens not included in the MST PCR panel, and were thus MST-/BC+.
The limited PCR panel of the assay is also a disadvantage that MST has in common with most other commercial PCR assays for pathogen detection in blood. The panel did not cover pathogens found in 10% (13/128) of the BC positive samples. In addition, there are fastidious organisms, such as Mycobacterium tuberculosis, Bartonella species, and Legionella pneumophila, that seldom grow in BC and that cannot be detected by MST. Another limitation of the MST is that among the more than 90 pathogens that are included in the screening step, only 27 can be identified to the species level. In our study 3% (10/372) of all pathogens detected by MST were positive only in the screening step and not further identified.
When evaluating the MST, it is relevant to compare it with the LightCycler SeptiFast test (Roche), which is, to date, the most established commercial PCR test for diagnosing BSI. In 2007-2008, our group [32] evaluated the SeptiFast test on emergency room samples with a similar study design and study population including 1,093 patients, at the same hospital. Compared to BC, the SeptiFast test showed a positive result in 9.8% (107/1,093), and sensitivities and PPV of 67% and 43% for S. aureus, 12% and 67% for S. pneumoniae, and 53% and 56% for E. coli. In the present study, MST with the lower Ct cut-off value showed a positive result in 12% (83/696) and sensitivities and PPV of 63% and 60% for S. aureus, 39% and 71% for S. pneumoniae, and 43% and 73% for E. coli. Consequently, the MST test with the lower Cq cutoff value showed at least as good performance as the SeptiFast test, and was clearly superior to the SeptiFast test for detection of S. pneumoniae BSI.
Other commercially available molecular assays include SepsiTest (Molzym) and VYOO (Analytik Jena), with reported sensitivities of 37%-87% [15,35] for SepsiTest and 38%-60% [14,36] for VYOO. Thus, these assays and MST appear to have similar sensitivities. However, comparison between studies is difficult due to factors such as differences in the clinical condition of the patients, the number of patients receiving antibiotic treatment before taking the blood samples, the number of blood cultures taken, and whether or not findings of CoNS and other suspected contaminants in BC were included. The specificities are even more difficult to compare between studies. In some studies, all suspected contaminants have been excluded before performing analyses, and in many studies the reference standard used for the calculations has included more data than just BC positivity.
The latest commercial molecular assay, the IRIDICA PCR/electrospray ionization-mass spectrometry assay (Abbott), has shown promising sensitivity in BSI pathogen detection; 73%-81% [18,19]. The method uses a large blood sample volume, 5 mL, which is probably the main reason for its increased sensitivity. This method uses mass-spectrometry that allows a broadrange detection, which is an advantage compared to methods, such as MST, with limited PCR panels. The specificity of the IRIDICA test is unclear, as it has been found to detect many pathogens that have not been supported by BC, in many cases with unknown significance [18,19].
The present study has some strengths and limitations. The major strength is that the MST was run on consecutive patients in a large, defined study population.
The main limitation is that we did not have access to clinical data other than gender, age and BC results for the study cohort, according to the decision of the Regional Medical Ethics Review Board. However, in order to improve the relevance of this evaluation, we included species-specific PCR for the most frequently detected pathogens. It was our assumption that if the pathogen was detected by two different PCR methods, it was likely to be a relevant finding.

Conclusions
In conclusion, our study showed that the Cq value can be used as a tool in the interpretation of a positive MST result, but we do not suggest a strict application of the lower cut-off, as it would affect the sensitivity. However, even with the lower cut-off the sensitivity of the MST was comparable with other commercial molecular methods in the same field of application. Similar diagnostic strategies, taking quantitative DNA data into account, could probably be used in the interpretation of other molecular methods as well.
Future studies on commercial tests for microbiological sepsis diagnosis, should further establish the role and clinical importance of DNA quantification.