A comparison of host response strategies to distinguish bacterial and viral infection

Objectives Compare three host response strategies to distinguish bacterial and viral etiologies of acute respiratory illness (ARI). Methods In this observational cohort study, procalcitonin, a 3-protein panel (CRP, IP-10, TRAIL), and a host gene expression mRNA panel were measured in 286 subjects with ARI from four emergency departments. Multinomial logistic regression and leave-one-out cross validation were used to evaluate the protein and mRNA tests. Results The mRNA panel performed better than alternative strategies to identify bacterial infection: AUC 0.93 vs. 0.83 for the protein panel and 0.84 for procalcitonin (P<0.02 for each comparison). This corresponded to a sensitivity and specificity of 92% and 83% for the mRNA panel, 81% and 73% for the protein panel, and 68% and 87% for procalcitonin, respectively. A model utilizing all three strategies was the same as mRNA alone. For the diagnosis of viral infection, the AUC was 0.93 for mRNA and 0.84 for the protein panel (p<0.05). This corresponded to a sensitivity and specificity of 89% and 82% for the mRNA panel, and 85% and 62% for the protein panel, respectively. Conclusions A gene expression signature was the most accurate host response strategy for classifying subjects with bacterial, viral, or non-infectious ARI.

Introduction non-infectious illness and who had samples available to support all biomarker testing [23]. The study protocol and analysis plan for this study were not pre-specified at the time the RAD-ICAL study was designed and implemented. Subjects with a low-confidence reference standard were excluded, consisting of those without confirmatory microbiology or bacterial/viral co-infection. Enrollment occurred in the emergency departments of Duke University Medical Center, Durham VA Health Care System, Henry Ford Hospital, and University of North Carolina Medical Center from 2005 through 2016. They were enrolled as part of CAPSOD (Community-Acquired Pneumonia and Sepsis Outcome Diagnostics; ClinicalTrials.gov NCT00258869) or CAPSS (Community-Acquired Pneumonia and Sepsis Study). Patients �6-years were eligible if they had a known or suspected infection and at least two Systemic Inflammatory Response Syndrome (SIRS) criteria [24]. Additional subjects were enrolled at Duke and the Durham VA as part of RADICAL (Rapid Diagnostics in Categorizing Acute Lung Infections). RADICAL enrolled subjects �2-years with <28-days duration of suspected bacterial, viral, or non-infectious ARI.

Reference standard and case definitions
Since there is no gold standard to define whether ARI is bacterial, viral, or non-infectious, panel adjudication served as the reference standard [18,23,25,26]. Adjudications were performed by specialists in emergency medicine, infectious diseases, pulmonary medicine, or hospital medicine based on chart reviews performed >28 days after enrollment, and before measuring procalcitonin, the protein panel, or the mRNA panel. Due to the retrospective nature of the adjudications, the diagnostic accuracy of this reference standard is expectedly higher than what might be achieved in real-time by clinical providers. Adjudicators had access to medical records including but not limited to admission notes, consultant notes, discharge summaries, patient-reported symptoms, and laboratory and radiographic results (when clinically indicated). In addition, we performed supplemental microbiological testing including a multiplex viral respiratory pathogen panel (ResPlex V2.0, Qiagen; Respiratory Viral Panel, Luminex; or Respiratory Pathogen Panel, Luminex). Each case was independently adjudicated by two reviewers. If there was discordance with respect to the primary etiology, the case was reviewed by a panel of at least three reviewers. To be categorized as having a viral or bacterial ARI, a subject must have had a compatible clinical syndrome and an identified pathogen. The determination of "noninfectious illness" required an alternative non-infectious diagnosis and negative microbiological testing. Sepsis was defined based on the Sepsis-2 criteria, which requires the presence of systemic inflammatory response syndrome (SIRS) due to infection [24].

Procalcitonin measurement
Procalcitonin was measured for study purposes and was not available to adjudicators, limiting any potential incorporation bias. Serum measurements were made on a Roche Elecsys 2010 analyzer (Roche Diagnostics) by electrochemiluminescence immunoassay or the miniVIDAS immunoassay (bioMérieux). Measurements in plasma-EDTA were made by the Phadia Immunology Reference Laboratory by immunofluorescence using the B�R�A�H�M�S PCT sensitive KRYPTOR (Thermo Fisher Scientific). Since all three platforms are approved for this indication, results were treated equivalently regardless of test platform. Subjects were classified as bacterial (�0.25 ng/mL) or non-bacterial (<0.25 ng/mL) (Fig 1).

Protein panel measurement
Concentrations of plasma IP-10, TRAIL, and CRP were measured by sandwich immunoassay with electrochemiluminescent detection using the UPLEX Human IP-10 Assay, the UPLEX Human TRAIL Assay, and the VPLEX Human CRP Assay with imaging using the QuickPlex SQ 120 Imager (Meso Scale Discovery, Rockville, MD). Plasma samples stored at -80˚C were thawed on ice and processed according to the manufacturer's recommended protocol, similar to published methods [12]. Calibration curves were generated using 4-fold serial dilutions of the calibrator. Generating probabilities of bacterial vs. viral infection was performed as previously described: protein concentrations were used to build a multinomial logistic regression model, establishing a bacterial likelihood score from zero to one [12].
Subjects were classified as bacterial or viral using two different thresholds (Fig 1). The first used published thresholds: subjects with a score �0.35 were classified as viral or non-infectious, subjects with a score �0.65 were classified as bacterial, and scores from 0.35-0.65 were considered equivocal [12]. The second method identified a single ideal threshold, above which subjects were classified as bacterial and below which subjects were classified as viral. We used the Youden Index to identify a threshold that maximized sensitivity and specificity. Values �0.30 were considered bacterial and values <0.30 were considered non-bacterial.

mRNA panel measurement
The composition and development of the mRNA panel measured in this study was previously described [21]. In brief, the gene signature test is composed of two unique host response signatures: a bacterial host response signature that assigns a probability of having a bacterial vs. non-bacterial etiology and a viral host response signature that assigns a probability of having a viral vs. non-viral etiology (Fig 1). The two probabilities are independent and do not sum to 1, which allows for the identification of co-infection (i.e., both positive) or no infection (i.e., both negative). In order to generate these probabilities, the BioFire System measured transcript abundance, which was then used to build a logistic regression model trained on subjects with known phenotype. Gene expression was measured via research-use-only FilmArray pouches as previously described [21]. At initial clinical presentation, subjects were enrolled and whole blood was collected directly into PAXgene Blood RNA tubes (PreAnalytix) and stored at -80˚C until use. For testing, 100μL was used to measure the relative abundance of 45 target host mRNAs as defined by the PCR cycle at which the amplified target was detected using FilmArray chemistry and processing. Normalized target expression values were used to build two independent sparse logistic regression models: one to discriminate viral from non-viral etiologies and the other to discriminate bacterial from non-bacterial etiologies. The regularization parameter of the model and performance metrics were estimated using nested Leave-One-Out Cross-Validation (LOOCV), where the internal LOOCV was used for the regularization parameter and the outer LOOCV for performance estimates. This approach was selected to match the model training and validation process used for the protein panel. Thresholds for the bacterial (0.263) and viral (0.361) tests were chosen to optimize the average weighted accuracy [27].

Statistical analysis
Each of the three tests was evaluated for its ability to distinguish bacterial from non-bacterial subjects. The protein panel and gene signature were also evaluated for their ability to distinguish viral from non-viral subjects. Procalcitonin was not evaluated for viral vs. non-viral discrimination since it does not discriminate these two possibilities. Further, a combined model was created using a multinomial logistic regression model, treating each of the aforementioned tests as an individual input. Test performance was evaluated by comparing the area under the receiver operating characteristic curve (AUC), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). Biomarker strategies were compared using the Chi-squared test except for AUC, which was compared using the Delong test. Data is available in S1 Table.

Demographics
Samples from 286 subjects were analyzed. Subjects had ARI due to clinically adjudicated bacterial (n = 47), viral (n = 162), or non-infectious (n = 77) etiologies. Demographics were similar across different disease states. Subjects with bacterial infection had a mean age of 52 years, were 58% male, 43% African American, and 2% Hispanic. Subjects with viral infection had a mean age of 43 years, were 40% male, 58% African American, and 4% Hispanic. Subjects with non-infectious illness had a mean age of 54 years, were 52% male, and 41% African American ( Table 1). The most common bacterial pathogens were Streptococcus pyogenes, Streptococcus pneumoniae, and Staphylococcus aureus ( Table 2). The most common viral pathogens were influenza A, rhinovirus/enterovirus, respiratory syncytial virus (RSV), and human metapneumovirus ( Table 2). The most common non-infectious etiologies included asthma and congestive heart failure ( Table 2).

Procalcitonin
Procalcitonin was used as a binary marker of bacterial infection, with a concentration �0.25 ng/ml denoting a bacterial infection and values <0.25 ng/ml indicating a non-bacterial process. In this cohort, procalcitonin differentiated bacterial from non-bacterial etiologies with 68.1% sensitivity (95% confidence interval [CI], 52.9%-80.9%), 86.6% specificity (95% CI, 81.6%-90.7%), and AUC of 0.837 (95% CI, 0.773 to 0.901) (Fig 2). These and additional test characteristics are shown in Table 3. Procalcitonin does not differentiate viral from non-infectious etiologies. As such, we did not apply it to the question of viral vs. non-viral (which includes bacterial and non-infectious etiologies).
When equivocal results apply to such a large number of patients, generalizability becomes limited. We therefore used the previously established threshold of �0.65 to indicate the presence of a bacterial infection and any values <0.65 to indicate a non-bacterial etiology (viral infection or non-infectious illness). This resulted in a test sensitivity of 42.6% (95% CI 28.3%-57.8%) and specificity of 91.6% (95% CI 87.4%-94.8%). Since these protein panel thresholds were established in other clinical cohorts, they may not be well-calibrated to this study population. We therefore selected a new threshold that optimized test performance in this cohort. Using a threshold of 0. 30 Fig 2).
Viral vs non-viral. Using the previously published threshold of �0.35, the protein panel had 86.4% sensitivity (95% CI, 80.2%-91.3%) but only 57.3% specificity (95% CI, 48.1%-66.1%) for the classification of viral infection. Most misclassifications were non-infectious subjects being classified as viral, which may have smaller clinical consequences as both groups would be spared antibacterial use.  Fig 2).  Fig 2). Specificity was significantly better than the protein panel (p <0.01), and sensitivity was significantly better than both procalcitonin and the protein panel (p<0.001) ( Table 3). A direct comparison of the predicted probabilities of bacterial infection using the mRNA and protein panel tests is shown in Fig 4. A model combining the protein panel, mRNA panel, and procalcitonin did not improve performance beyond that of the mRNA panel alone (Table 3, Fig 2). Viral vs non-viral. Any subject with a viral probability of �0.361 was classified as having a viral infection. Using this threshold, the gene signature distinguished viral vs. non-viral subjects with a sensitivity of 88.9% (95% CI, 83.0%-93.3%), specificity of 82.3% (95% CI, 74.4%-88.5%), and AUC of 0.928 (95% CI, 0.899-0.956) ( Table 3, Fig 2). Specificity was significantly better than the protein panel (p <0.0001) ( Table 3). A direct comparison of the predicted probabilities of viral infection using the mRNA and protein panel tests is shown in (Fig 4). Comparison to procalcitonin was not performed since it does not discriminate viral from noninfectious etiologies. A model combining the protein panel and mRNA panel did not improve performance beyond that of the mRNA panel alone (Table 3, Fig 2).

Discordant classifications
We hypothesized there would be scenarios in which all three host response tests agreed with each other but disagreed with clinical adjudication. This occurred in nine cases. The only pattern we could discern among them were three cases adjudicated as having Group A Streptococcal pharyngitis, but all three host response tests identified a viral infection. In the entire cohort, there were 11 cases of Group A Streptococcal pharyngitis corresponding to a 27% (3/ 11) error rate.
We also stratified the cohort by the presence or absence of SIRS criteria, which in the case of bacterial and viral infections, would denote sepsis. SIRS was present in 155 subjects and absent in 131. We observed no differences in the sensitivity for any biomarker strategy, though this was limited by having only six bacterial infections without sepsis. In contrast, the specificity to diagnose a bacterial infection was lower for all three biomarker strategies among subjects with SIRS compared to those without: 78.1% vs. 94.4% for procalcitonin (p<0.001), 62.3% vs. 83.2% for the protein panel (p<0.001), and 71.9% vs. 92.8% for the mRNA panel (p<0.001).
There were no other identifiable patterns with respect to illness or pathogens among the other discordant predictions. Similarly, there was no clear pattern among cases where the different biomarker panels were discordant with each other.

Discussion
ARI is among the most common acute care presentations worldwide. This syndrome is commonly due to viral, bacterial, or non-infectious etiologies. The greatest challenge in clinical management is identifying when a bacterial infection is present and subsequently, which patients need antibacterial therapy. Pathogen-detection strategies are an important component of this evaluation but have limited utility. A complementary strategy measures the host's molecular response to the infection, which can discriminate between etiologies. This study evaluated three different host-based diagnostic approaches in a common cohort of patients with bacterial, viral, and non-infectious ARI. We showed that all three strategies discriminate between bacterial and viral etiologies, although an mRNA panel test performed best, in part because of its ability to discriminate patients with non-infectious etiologies.
Currently, procalcitonin is the most widely used biomarker to aid in the differentiation of bacterial and non-bacterial etiologies of ARI. Procalcitonin has been cleared by the FDA to guide the management of antibiotic therapy in patients with lower respiratory tract infection based on previously published trial data [28]. However, a recent study failed to show an impact by procalcitonin on antibacterial use, the reasons for which are likely multifactorial [29]. Other recent publications have demonstrated a poor ability to discriminate bacterial and viral etiologies [29,30]. Further, high false positive rates for procalcitonin have been observed for patients with a variety of inflammatory and chronic disease conditions [31][32][33][34][35][36][37][38]. The limitations of procalcitonin may be inherent to the use of a single biomarker approach. Consequently, integrating multiple biomarkers from different biological pathways into a single test could improve ARI diagnostics.
One example is the use of CRP, IP-10, and TRAIL, which were identified in a proteomic study based on their ability to differentiate bacterial and viral infections [12]. This panel has been validated in several studies, demonstrating an ability to distinguish bacterial from viral infection with a sensitivity of 86%-94% and specificity of 90%-94% [13,14,39,40]. Equivocal test results were obtained in 11-17% of these cohorts compared to 19% in this study. Such a high rate of non-informative results limits the utility of such a test. Furthermore, these prior publications excluded subjects with equivocal results when calculating sensitivity and specificity. Removing subjects from the denominator (all subjects tested who had a valid test result) will artificially inflate performance estimates. When performance characteristics are calculated with all tested subjects accounted for, results are expectedly lower. This was observed in our study as well as another independent validation performed by van der Does et al [41]. In that study, sensitivity was 78% and specificity was 73% for the diagnosis of bacterial infection, which is similar to our observations (81% and 73%, respectively).
Transcriptomic approaches present another opportunity for disease classification. Many transcriptomic-derived classifiers for distinguishing between patients with bacterial or viral infections have been described [15][16][17][18][19][42][43][44][45][46][47][48][49][50]. Despite these discoveries, the means to measure these signatures in a clinically meaningful time was not previously available. To address that, we translated the gene expression signature using BioFire technology, taking advantage of the sample-to-answer format [23]. Using this research-use-only test, the mRNA panel test evaluated here was better than the protein panel and procalcitonin for classifying subjects with ARI. One potential reason is that a 45-gene signature captures more biology due to the increased number of analytes measured. Perhaps similar improvements could be achieved with a larger panel of proteins although adding procalcitonin to the protein panel did not improve results. Furthermore, protein biomarkers for measurement in serum or plasma will be restricted to those that are secreted, representing a biologically limited view of the underlying host response. Furthermore, the mRNA panel was discovered in a cohort of patients that had a higher proportion of non-infectious etiologies as compared to the alternative biomarker strategies. As such, it is expected to perform better in an undifferentiated and more heterogeneous patient population.
This study identified Group A streptococcal pharyngitis as one diagnostically challenging area. Specifically, all three biomarker strategies were discordant with clinical adjudication in 27% of subjects with streptococcal pharyngitis. Rates of pharyngeal S. pyogenes colonization are as high as 40% depending on how the carrier state is defined [51]. This suggests that the presence of Group A Streptococcus in and of itself is a poor indication of the presence of bacterial infection. Host response biomarkers may therefore offer additional and perhaps superior diagnostic information for such infections.
More broadly, all three biomarker strategies offered high NPVs for bacterial infection, ranging from 93% for procalcitonin to 98% for the mRNA test. These values highlight the potential utility for such biomarkers in mitigating inappropriate antibacterial use. The PPV for viral infection was lower: 75% for the protein panel and 87% for the mRNA panel. We note, however, that tests to positively identify viral infection are not currently available. As such, these tests, particularly the mRNA strategy, address an important diagnostic gap.
One study limitation includes the use of LOOCV rather than using independent training and validation cohorts. This was necessary since the protein panel demonstrated poor calibration using previously published model parameters. Although this may overestimate performance as compared to a truly independent validation, it enabled a head-to-head comparison between biomarker strategies. Another limitation is that we measured the protein panel using standardized research assays and not the proprietary ImmunoXpert™ platform. That could affect results, although we optimized the panel's performance in this study cohort, which should mitigate technical differences. Most subjects in this study were adults. If distinguishing bacterial and viral etiologies in children is easier, then the differences observed between the three host response approaches may diminish. Children and adults may have distinct biological responses to infection. However, studies have shown that procalcitonin, the protein panel, and the mRNA panel evaluated here perform similarly in children and adults [12,18,21,40,52]. Three of the four enrollment sites were in the same geographic area. To ensure generalizability of these results, further investigation must be done in a more heterogeneous population. We note that the indication for use for procalcitonin focuses on lower respiratory tract infection and therefore, its application to cases of upper respiratory tract ARI including pharyngitis may be more limited. Nevertheless, inclusion of a heterogeneous cohort strengthens the study's generalizability. Lastly, all tests were performed on banked samples although the biomarkers evaluated here are not known to be affected by a single freeze-thaw cycle.
This study demonstrated a clear improvement in discriminating ARI etiologies using a gene expression approach, although it is unknown whether this will translate into improved patient management. Clinical trials are currently underway to investigate all three biomarker strategies and how best to apply the test results. Host response-based diagnostics offer new opportunities to characterize patients in ways not previously possible. Gene expression signatures, coupled with technical advances that bring these tests to clinical practice, potentially offer the best approach for host response-based infectious disease diagnosis.
Supporting information S1 Checklist. STROBE statement checklist for cohort studies. (DOC) S1