Evaluation of Four Commercial Multiplex Molecular Tests for the Diagnosis of Acute Respiratory Infections.

Acute Respiratory Infections (ARIs) are responsible for considerable morbidity and mortality worldwide. Documentation of respiratory specimens can help for an appropriate clinical management with a significant effect on the disease progress in patient, the antimicrobial therapy used and the risk of secondary spread of infection. Here, we compared the performances of four commercial multiplex kits used in French University Hospital diagnostic microbiology laboratories for the detection of ARI pathogens (i.e., the xTAG Respiratory Viral Panel Fast, RespiFinder SMART 22, CLART PneumoVir and Fast Track Diagnostics Respiratory Pathogen 33 kits). We used a standardised nucleic acids extraction protocol and a comprehensive comparative approach that mixed reference to well established real-time PCR detection techniques and analysis of convergent positive results. We tested 166 respiratory clinical samples and identified a global high degree of correlation for at least three of the techniques (xTAG, RespiFinder and FTD33). For these techniques, the highest Youden’s index (YI), positive predictive (PPV) and specificity (Sp) values were observed for Core tests (e.g., influenza A [YI:0.86–1.00; PPV:78.95–100.00; Sp:97.32–100.00] & B [YI:0.44–1.00; PPV:100.00; Sp:100.00], hRSV [YI:0.50–0.99; PPV:85.71–100.00; Sp:99.38–100.00], hMPV [YI:0.71–1.00; PPV:83.33–100.00; Sp:99.37–100.00], EV/hRV [YI:0.62–0.82; PPV:93.33–100.00; Sp:94.48–100.00], AdV [YI:1.00; PPV:100.00; Sp:100.00] and hBoV [YI:0.20–0.80; PPV:57.14–100.00; Sp:98.14–100.00]). The present study completed an overview of the multiplex techniques available for the diagnosis of acute respiratory infections.


Introduction
Acute Respiratory Infections (ARIs) are responsible for considerable morbidity and mortality worldwide [1]. The precise and rapid documentation of respiratory specimens can lead to a significant effect on the disease progress in patients [2] when identification of the causative agents leads to providing an appropriate therapy administration, but it is also useful for decreasing the use of unnecessary antimicrobial therapy [3,4] and limiting the risk of secondary spread of infection [5]. However, ARIs are most often associated with poorly specific clinical presentations [6] and therefore, their aetiological diagnosis mainly relies on laboratory testing.
Conventional laboratory diagnostic methods have several limitations for routine detection of respiratory pathogens. For example, culture isolation of viruses of bacteria is poorly sensitive, time consuming and usually too slow to provide a biological result at the acute phase of the disease [7]; direct immunofluorescence assays and immunochromatographic antigen testing can provide results very rapidly but represent time-and labor-consuming tasks for the former, and are poorly sensitive for the latter [8,9,10]. Molecular biology testing allows the detection of a variety of viral and bacterial pathogens within hours, with excellent sensitivity and specificity and may represent a credible alternative to the aforementioned biological assays. However, given the number of aetiological agents potentially implicated in ARIs, conventional monoplex PCR assays are tedious, expensive and require large amounts of biological samples [10,11]. Consequently, multiplex molecular detection tests have been increasingly developed in recent years and a number of commercial kits have been proposed to diagnostic microbiology laboratories, which allow the detection of 12 to 33 different pathogens. This includes the xTAG Respiratory Viral Panel, Fast or v1, assays (Luminex Molecular Diagnostics, Toronto, Canada); the FilmArray respiratory viral panel assay (BioFire Diagnostics, Salt Lake City, UT); the Fast Track Diagnostics Respiratory Pathogen, 21 or 33, assays (Fast Track Diagnostics, Luxembourg); the CLART PneumoVir (Genomica, Coslada, Spain); the RespiFinder, 19 or SMART 22, assay (Pathofinder, Maastricht, Netherlands); the eSensor Respiratory Viral Panel (Gen-Mark Dx, Carlsbad, CA); the MultiCode-PLx (EraGen Biosciences); the Easyplex respiratory pathogen 12 kit (Ausdiagnostics, Sydney, Australia); the Seeplex RV15 OneStep ACE Detection and Pneumobacter ACE Detection (Seegene Inc., Seoul, South Korea); the Magicplex RV Panel Real-time Test (Seegene Inc); and the ResPlex II Panel v2.0 (Qiagen, Hilden, Germany). Precisely evaluating such multiplex kits is difficult in the absence of undisputable golden standard reference techniques for a large number of the pathogens tested. A number of studies have compared the detection of the molecular multiplex platforms with in-house conventional molecular methods [11,12,13,14] but since the reference methods vary from study to study [15], estimating the actual performance of the assays remains difficult. More recently, direct comparison of commercialised multiplex kits was undertaken, with reference to in-house multiplex real-time PCR systems [16], or to commercial duplex PCR tests [17].
Here, as a follow-up to these pioneer studies, we compared the performances of four commercial multiplex kits used in French University Hospital diagnostic microbiology laboratories for the detection of ARI pathogens, i.e., the xTAG Respiratory Viral Panel Fast, RespiFinder SMART 22, CLART PneumoVir and Fast Track Diagnostics Respiratory Pathogen 33 kits. Our analysis mainly focussed on viral pathogens that represent the common target of all kits, the number of bacteria potentially detected varying from 4 (RespiFinder SMART 22) to 12 (Fast Track Diagnostics Respiratory Pathogen 33), making any sound comparison impossible. We combined normalised extraction of nucleic acids, "reference" monoplex real-time PCR detection and a consensus interpretation algorithm to estimate the intrinsic capability of each kit to detection of the various pathogens tested.

Nasopharyngeal swabs
Out of the 1,465 patients enrolled in the CoPanFlu-France cohort study [18], 166 presented with ARI during the 2010 and 2011 influenza epidemic seasons. Nasopharyngeal samples were available for all these patients and were selected for this study.
The patients' age ranged from 1 month to 79 years (median: 31.2yo).

Broad range multiplex Kits
Four commercial kits for Multiplex detection of respiratory viruses were tested: -The CLART PneumoVir (PneumoVir) assay is based on PCR-DNA micro-array detection. This assay allows the detection of 17 respiratory viruses.
-The Fast Track Diagnostics Respiratory Pathogen 33 (FTD-RP33) version 3 assay is based on multiplex one-step reverse transcription polymerase chain reactions (RT-PCR) with probes for detecting 33 respiratory pathogens (21 viruses and 12 bacteria). It was used with the Bio-Rad CFX96 thermocycler and The SuperScript III Platinum One-Step Quantitative RT-PCR System without ROX (Invitrogen).
-The xTAG Respiratory Viral Panel Fast (xTagRVP-F) is based on a multiplex RT-PCR reaction in which the target-specific primers are chimeric, including a terminal Universal Tag sequence. This allows the detection of 19 different respiratory viruses [4,19] by sorting on a Luminex xMAP instrument [20].
-The RespiFinder SMART 22 (RespiFinder) assay is based on the multiplex ligationdependent probe amplification (MLPA) technology, preceded by a pre-amplification step [20]. Results are analysed by capillary electrophoresis [9]. This assay differentiates 18 respiratory viruses and 4 bacteria in one reaction by melt curve analysis.
The different protocols were performed according to manufacturers' instructions but the extraction of nucleic acids was standardised to use a similar molecular substrate in each assay. One extraction dedicated to each test (each multiplex assay and for the "confirmation tests") was performed from one of the identical 100μL aliquots prepared from nasopharyngeal swabs. The internal controls of each test were included in the dedicated extraction, and their final concentration adapted to manufacturers' recommendations. Nucleic acids were extracted using the EZ1 Virus Mini Kit v2 on the EZ1 advanced XL Biorobot workstation (Qiagen). Elution was performed in a 90μL volume and nucleic acids were subsequently stored at -80°C before analysis.
The combination of all systems potentially allowed the detection of 41 respiratory patho- The spectrum of detection of the different kits is reported in Table 1. Tests allowing the detection of respiratory pathogens were classified in two different classes: the "Core tests" and the "Extra tests", the "Core tests" being defined as those corresponding to pathogens detected by all four kits.

Processing of samples and interpretation of results
Firstly, all samples were tested with the four broad range multiplex kits (Fig 1). In addition, they were systematically tested for Inf A and Inf B using in-house standard real-time RT-PCR assays [21], and for Core tests (Inf A, Inf B, hRSV, hMPV, EV + hRV, AdV and hBoV) using the Argene (Respiratory Multi Well System r-gene) low range kit, consisting of 4 Taqmanbased duplex PCR or RT-PCR reactions (Inf A+B, hRSV+hMPV, hRV&EV+cellular control, AdV+hBoV). Samples testing negative for a given pathogen with all detection methods were proposed negative for this pathogen. Samples testing positive for a pathogen with at least 2 different techniques were proposed positive. Samples with discrepant results (positive for one pathogen with only one detection method) were further processed with "confirmation" assays, consisting of one or several real-time or nested amplification protocols (Table A in S1 File) and subsequently declared positive or negative according to the result obtained.
Positive predictive value (PPV), negative predictive value (NPV), Sensitivity, Specificity and agreement (as evaluated by the Youden's index) values were calculated for the Core tests of the kits tested (see below).
Secondly, all samples were analysed according to the results obtained by the four broad range multiplex kits for Extra tests (S1 Fig Only results relating to viral pathogens were analysed in depth since a majority of bacterial pathogens were detected only by the FTD-RP33 test. A decision tree similar to that used for Core tests was designed and confirmation techniques were used to confirm or invalidate discrepant results (Table B in S1 File). PPV, Sensitivity, NPV, and Specificity values were calculated for Extra tests (see below).

Performances of the kits
To compare the performances of the kits, Sensitivity, Specificity and Positive and Negative Predictive values were calculated. In accordance with the interpretation rules decreed in the previous section, calculations relied on the following definitions: (i) True Positives were Positive results confirmed by another kit or a confirmation test; (ii) False Positives were Positive results that were not confirmed by another kit or a confirmation test; (iii) False Negatives were Negative results invalidated by at least 2 other kits or 1 kit and 1 confirmation test; (iv) True Negatives were Negative results that were not invalidated by at least 2 other kits or 1 kit and 1 confirmation test. Nucleic acids were extracted as described above but using an initial sample volume of 400μL and an elution volume of 120μL. Tenfold serial dilutions of nucleic acids were performed and tested using the relevant in-house "confirmation" real time PCR or RT-PCR assays (S1 File). The last three dilutions allowing unambiguous detection (i.e., the lowest dilution had a Ct at 33 and the highest dilution had a Ct between 37 and 40) were used to create a panel, which was subsequently used to evaluate the limit of detection of the 4 broad range multiplex kits.

Ethics statement
The CoPanFlu-France cohort protocol was approved by the French ethics committee ("Comité de Protection des Personnes Ile-de-France I") on 8 September 2009. All study participants provided written informed consent [18].

Results
The analysis of 166 Nasopharyngeal samples using a combination of four commercial broad range multiplex kits and of confirmation tests provided confirmed aetiological viral diagnosis for 87 samples (52%), including 8 samples (4.8%) with confirmed dual infection (see details in Table 2).

Performances of the four commercial broad range multiplex kits
The performances of the kits for the detection of specific pathogens are presented in Table 3. Regarding the detection of Influenza A viruse(s), the sensitivity and PP values ranged from 88.2% and 79% respectively for the PneumoVir kit to 100% for the FTD-RP33 kit. Regarding Influenza B, the best sensitivity was observed for the RespiFinder kit (100%). It was under 80% for the PneumoVir and FTD-RP33 kits and under 50% for the xTagRVP-F kit. PP values were at 100% for all kits except the PneumoVir kit (87.5%).
Globally, the performances of the PneumoVir kit were hindered by the existence of false positive results for a proportion of the pathogens tested, but also of false negative results for the most common pathogens and in particular human Rhinoviruses/Enteroviruses. The performances of the xTagRVP-F kit were hampered by its low sensitivity for the detection of Influenza B, human bocaviruses and human coronaviruses, but the kit was associated with high PP values (very few false positives). This profile is somewhat similar to that of the FTD-RP33 kit that exhibited high PP values and perfectible sensitivity for the detection of Influenza B, Parainfluenzaviruses, human Respiratory Syncytial virus and human Rhinoviruses/ Enteroviruses. The RespiFinder Kit displayed the highest overall sensitivity values, associated with high PP values. Its lowest performances were for human Metapneumoviruses (sensitivity: 71.4%) and Parainfluenzaviruses (66.6%, similar with other kits).
Regarding the global performances of the kits, the Positive Predictive values were >90% for three of the four kits (xTagRVP-F, FTD-RP33 and RespiFinder) ( Table 4). The highest overall specificity was observed for the FTD-RP33 kit (94.4%), with a 100% score for five common pathogens (Influenzaviruses A and B, Parainfluenzaviruses, human Respiratory Syncytial viruses, human Rhinoviruses/Enteroviruses). The most important differences between the kits tested were identified for the overall sensitivity (ranging from 63.9% for the PneumoVir Kit to 88.4% for the RespiFinder Kit) and for the Negative Predictive Value (ranging from 70% for the PneumoVir Kit to 85.3% for the RespiFinder Kit).

Limits of detection of the four commercial broad range multiplex kits
Results are presented in Fig 2 and show that, at the highest dilution proposed, the RespiFinder kit detected 8 pathogens (out of the 9 tested), whilst the PneumoVir kits detected 6 (out of 10 tested), the xTagRVP-F kits detected 5 (out of 9 tested) and the FTD-RP33 detected 4 (out of 11 tested).

Discussion
We have compared 4 commercialised multiplex PCR techniques for the detection of respiratory pathogens using standardised nucleic acid extracts from French patients presenting with ARIs in 2010 and 2011. Our study mainly focussed on viral pathogens. There is previous significant information relating to the comparison of the performances of the kits studied with conventional diagnostic technique. This includes: (i) direct immunofluorescence, viral culture, and home-brew PCR in the case of the CLART PneumoVir assay [22];  [17]. Globally, the literature indicates that multiplex molecular techniques compare favourably with traditional diagnostic technique. They can be (at least partially) automated, are highly reproducible and are generally more sensitive.
However, a first difficult issue for comparing respiratory pathogens multiplex techniques is the absence of gold standard for the detection of respiratory pathogens. If we refer only to the ability of a technique to detect a pathogen or some of its molecular components in biological samples, some newly developed molecular techniques may be more sensitive than the reference techniques, (i.e., some "false positives", referring to the reference technique, may be true positives that have not been detected by this reference method). This situation may be met when the new technique is able to detect viral variants that could not be detected by previous methods and, in that case, corresponds to an actual improvement of the technique if the variants detected are true pathogens that can be implicated in ARIs. However, it can also be met when the new technique is able to detect a lower viral load. The actual clinical impact of an improved detection of low viral loads in respiratory samples has not been accurately evaluated and its medical significance at the acute phase of ARIs obviously requires to be further documented. Altogether, this indicates that the interpretation of evaluation scores should remain careful and open-minded. Accordingly, we have tried to build a comprehensive approach of the comparison of diagnostic kits that mixed reference to well established-and clinically evaluated-real-time PCR detection techniques (this included in the current study the Argene Respiratory Multi Well System r-gene low range kit for Core tests, which has been used for years for routine detection in medical virology laboratories; and a series of published home-brewed techniques) and analysis of convergent positive results. For the latter, we considered that, given (i) the intrinsic high specificity of molecular amplification techniques, and (ii) the fact that the techniques compared used different technologies and amplification primers, the probability of obtaining false positive results with two different techniques for the same sample was negligible for an evaluation purpose. Consequently, samples in which a given pathogen was detected independently by two different techniques were declared positive for this pathogen, otherwise it was declared negative. We cannot exclude that a few samples were wrongly assigned a negative result (if a specific technique performed better than all others for a given pathogen), but the high degree of convergence between the different techniques suggests that the number of "false negative" results that could have been potentially generated is limited.
A second difficult issue for comparing respiratory pathogens multiplex techniques is the choice of the panel of samples tested. The nature and frequency of pathogens present in the samples tested are highly dependent on the epidemiological circumstances and on the populations tested, and have an obvious influence on the results of the analysis, e.g., the low number or absence of positive samples for some viruses represents in most of the studies an obvious obstacle for accurate comparisons. In the current study, we tested patients with ARIs from all age groups recruited in a household cohort that aimed at providing a picture of the French general population. This means that, in comparison with previous hospital-based studies, the paediatric population was under-represented and the clinical presentations were milder. In addition, the specific period studied (2010-2011) was associated with the predominant circulation of the pandemic H1N1 variant of the influenza A virus. Accordingly, the low number or absence of parainfluenza virus, human coronavirus type 229E, H3N2 influenza A virus, influenza C virus, human adenovirus and human parechovirus infections did not allow a significant comparison of kits for these pathogens. This is important to notice since in previous studies the low number of positive samples for some pathogens may have concealed the poor performances of some kits for detecting these pathogens (e.g., the case of Adenoviruses for the xTAG Respiratory Viral Panel fast assay [9,17,25]).
A third difficult issue for comparing respiratory pathogens multiplex techniques relies in the different spectra of detection and precision of identification offered by the various commercialised kits. To take this aspect into account, we have identified "Core tests", i.e., a series of common pathogens detected by all kits (Inf A, Inf B, hRSV, hMPV, EV + hRV, AdV and hBoV). We have also limited our analysis to viral pathogens since bacterium detection was absent or very limited in 3 of the 4 kits tested. The simultaneous detection of bacteria and viruses represents a theoretical great advantage in terms of clinical management (e.g., for determining the appropriateness of starting an antibiotic treatment). However, it is noticeable that, in the current study, bacteria detection provided a large number of positive results, potentially shifting the rate of aetiological elucidation from 52% to >77% in the panel studied. Obviously, this deserves further in depth investigations, which should include samples from both ARIs and paired asymptomatic individuals. Determining the actual imputability of bacterial detection in the occurrence of ARIs may be difficult to achieve from qualitative results and clinically relevant interpretation algorithms remain to be established. As noted above, the detection of low pathogen loads may be an issue for viruses, but the case of bacteria is specifically difficult due to the frequent asymptomatic carriage of potentially pathogenic bacteria.
For Core tests, we identified an important degree of correlation between the performance of the xTAG Respiratory Viral Panel fast, the Fast Track Diagnostics Respiratory Pathogen 33 and the RespiFinder SMART 22 assays (Youden's index: 0.67; 0.73 and 0.83, respectively) ( Table 5). The CLART PneumoVir assay exhibited the most divergent performance (global Core tests Youden's index value: 0.51; global Core tests sensitivity and specificity values: 63.3% and 87.4%, respectively), despite previous favourable performance by comparison with homebrew PCR techniques [22]. When examined in more details (Table 3), the sensitivity values for Core tests compared with those of the other kits, except for human Rhinoviruses/Enteroviruses (sensitivity: 41.2%) and human Bocaviruses (40%). There was also a trend for the existence of false positive results for Influenza A and Metapneumovirus tests, which altered the global positive predictive value of the kit (81.9% vs >90% for all other kits). False positive results may hypothetically be related to the fact that the technique includes an open-tube post amplification step, potentially associated with an increased risk of PCR contamination. In serial dilution tests, 4 (Inf A H3N2, Inf B, Inf C and hMPV A) out of the 10 pathogens detected in the most concentrated dilution tested negative at dilutions 10 −1 and 10 −2 (Fig 2).
The Fast Track Diagnostics Respiratory pathogens 33 kit performed well for Core tests (global sensitivity and specificity values were 75.9% and 96.5%) ( Table 5). The excellent specificity values (the highest in the panel tested) may be related in part to the simple and safe realtime PCR format of the technique, which does not require any post-PCR manipulation. Amongst Core tests, and in agreement with previous findings [13], the lowest sensitivity values were observed for human Respiratory Syncytial Virus and Enterovirus/Rhinovirus (50% and 61.8%, respectively). In serial dilution tests, out of the 11 pathogens detected in the most concentrated dilution, 3 (Inf B and C, hRSV B) tested negative at dilution 10 −1 and 7 (Inf B and C, hRSV B, EV Cox B3, hPeV, hAdV B and hMPV A) at dilution 10 −2 , in coherence with previous observations [13].
The RespiFinder SMART 22 assay, provided for Core tests both the highest Youden's index performance values (global Core tests Youden's index value: 0.83) and the best global sensitivity (87.3%). It also was associated with an excellent specificity (95.4%), in agreement with previous evaluations [2,9,26,27]. Amongst Core tests, the lowest sensitivity value was observed for human Metapneumovirus (71.4%). The kit also provided the best results in serial dilution tests: of the 9 pathogens detected in the most concentrated dilution, all tested positive at dilution 10 −1 and 1 (hMPV A) tested negative at dilution 10 −2 .

Conclusion
The present study identified a high degree of correlation for at least three of the techniques and completed an overview of the multiplex techniques available for the diagnosis of acute respiratory infections. All of the kits evaluated performed properly for Core tests and Youden's index, positive predictive and specificity values were systematically higher for Core tests than for Extra tests. Significant differences in the ability of the kits to detect low viral loads were identified, but the medical implications of this observation remain to be evaluated. The range of detected pathogens, the technology used for PCR product revelation, the laboratory organization and other issues, such as sample through-put, staffing levels and staff expertise, clinicians' Supporting Information S1 File. PCR confirmation assays for Core tests (Table A) and extra-tests (Table B).