https://orcid.org/0000-0002-0132-0001
Figures
Abstract
Candidemia often poses a diagnostic challenge due to the lack of specific clinical features, and delayed antifungal therapy can significantly increase mortality rates, particularly in the intensive care unit (ICU). This study aims to develop a machine learning predictive model for early candidemia diagnosis in ICU patients, leveraging their clinical information and findings. We conducted this study with a cohort of 334 patients admitted to the ICU unit at Ji Ning NO.1 people’s hospital in China from Jan. 2015 to Dec. 2022. To ensure the model’s reliability, we validated this model with an external group consisting of 77 patients from other sources. The candidemia to bacteremia ratio is 1:1. We collected relevant clinical procedures and eighteen key examinations or tests features to support the recursive feature elimination (RFE) algorithm. These features included total bilirubin, age, platelet count, hemoglobin, CVC, lymphocyte, Duration of stay in ICU and so on. To construct the candidemia diagnosis model, we employed random forest (RF) algorithm alongside other machine learning methods and conducted internal and external validation with training and testing sets allocated in a 7:3 ratio. The RF model demonstrated the highest area under the receiver operating characteristic (AUC) with values of 0.87 and 0.83 for internal and external validation, respectively. To evaluate the importance of features in predicting candidemia, Shapley additive explanation (SHAP) values were calculated and results revealed that total bilirubin and age were the most important factors in the prediction model. This advancement in candidemia prediction holds significant promise for early intervention and improved patient outcomes in the ICU setting, where timely diagnosis is of paramount crucial.
Citation: Meng Q, Chen B, Xu Y, Zhang Q, Ding R, Ma Z, et al. (2024) A machine learning model for early candidemia prediction in the intensive care unit: Clinical application. PLoS ONE 19(9): e0309748. https://doi.org/10.1371/journal.pone.0309748
Editor: Giovanni Giordano, Sapienza University of Rome: Universita degli Studi di Roma La Sapienza, ITALY
Received: May 30, 2024; Accepted: August 17, 2024; Published: September 9, 2024
Copyright: © 2024 Meng et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: Key R&D Program of Jining, grant number 2023YXNS151. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Fever is a common symptom among patients admitted to the intensive care unit (ICU), often attributed to infections caused by variable microorganisms. Candida spp. is the leading cause of invasive fungal infections, ranking as the fourth most common microorganisms responsible for bloodstream infections (BSIs) [1]. Both invasive fungal infections and candidemia predominantly affect critically ill patients, with up to one-third episodes of candidemia cases occurring within the ICU [2]. Candidemia usually causes life-threatening conditions, resulting in increased mortality rate, healthcare costs, and prolonged ICU stays [3,4]. However, distinguishing candidemia from bacteremia is challenging due to the lack of specific clinical symptoms and presentations [5,6]. The current gold standard for candidemia diagnosis is a positive blood culture test [7]. However, blood culture tests have low sensitivity and require extended incubation times, potentially delaying antifungal therapy [8]. Therefore, physicians usually need to decide whether to wait for blood culture result which may delay initiation of antifungal therapy and prescribe empirical antifungal drugs when they suspect candidemia infection based on their experience. Retrospective studies have showed that such delays can substantially enhance mortality rates, with percentages rising from 35% to 80% [9,10].Conversely, initiating empirical antifungal treatment too early may induce antifungal resistance which has become a significant global public health threat.
New diagnostic biomarkers such as (1, 3)-β-D-glucan (BGD) and galactomannan (GM), glycan antigens derived from the fungal cell wall, or antibody recognizing mannan, have been introduced in clinical practice [6]. However, these biomarker tests are often costly and require complex infrastructure, limiting hospitals’ ability to conduct such around-the-clock tests [11]. So, they may not provide early diagnosis of candidemia in ICU patients, leaving clinicians without timely information when suspecting a candidemia infection [12]. Furthermore, biomarkers could not accurately predict candidemia in many conditions. BGD yields a high percentage of false-positive results due to transfusion of human blood products, hemodialysis, infection by some Gram-positive bacteria, use of certain beta-lactam antibiotics, cellulose dressings or enteral nutrition, and disruptions of gastrointestinal (GI) tract integrity [13,14].Therefore, the demand for the development of more rapid and accurate approaches to differentiate candidemia from bacterial infections is not only evident but also urgent, such approaches could aid ICU physicians in making informed decision regarding the initiation of antifungal therapy [15,16].
Recently, clinical decision support based on machine learning (ML) has received increasing attention for early prediction of candidemia. ML techniques possess a unique capacity to process vast quantities of data, enabling them to employ advanced mathematical models to grasp intricate patterns within clinical datasets. This capability equips ML to excel in tasks such as pattern detection and differentiation [17]. Previous studies have unveiled the potential of ML in early candidemia diagnosis. Bhavani SV et al. [16] trained a gradient boosting machine (GBM) model for early detection of candidemia in high and low risk groups. Their model outperformed the traditional logistic regression model, demonstrating the effectiveness of ML approaches with a higher area under the receiver operating characteristic (AUC). Similarly, Yoo J et al. [18] developed the CanDETEC model, utilizing random forest (RF) algorithms for candidemia risk prediction in patients with malignancy, and validates its efficacy through a single-center retrospective stud applied random forest (RF) models for the early prediction of candidemia in cancer patients. Atamna A et al. [19] and Gao Y et al. [20] constructed a logistic regression (LR) model to pinpoint risk factors of candidemia in internal medicine wards, offering valuable insights for the clinical management of patients with candidemia and bacteremia. Besides, Hu WH et al. [21] developed a two-level stacked generalisation model using multiple machines learning algorithms, including Naïve Bayesian (NB), k-Nearest Neighbour (KNN), logistic regression (LR), and random forest (RF) as base classifiers, with a support vector machine (SVM) as the meta-classifier, to predict 14-day mortality in candidemia patients. However, the aforementioned studies primarily focused on general populations of patients outside the ICU setting. Risk factors for Candida bloodstream infections in adult patients have been extensively described and identified, often observed in hospitalized patients. Nevertheless, critically ill ICU patients often present with distinct risk factors that are less applicable to distinguishing from bacteremia patients. Furthermore, the majority of studies have relied on traditional linear models to construct their predictive models, which offer only moderate predictive accuracy even in the best performing models. The demand for a more accurate and rapid approach to differentiate candidemia from bacteremia in ICU patients has grown, which will guide timely and effective therapeutic interventions in clinical settings. This study focuses on developing an early prediction model for fungal bloodstream infections in ICU patients, employing various machine learning algorithms. Through extensive retrospective analysis and external validation, the study validates the effectiveness and generalizability of the models. Additionally, the research elucidates the contribution of individual features in the model using SHAP values and constructs a predictive nomogram for diagnosing candidemia, providing clinicians with a valuable decision-making tool.
Materials and methods
Human ethics
This study adhered to the guidelines set forth in the TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) [22] for the construction of the multivariable prediction model and its subsequent validation. The study protocol received approval from the Human Ethics Review Committee of Ji Ning NO.1 people’s Hospital, assigned the reference number KYLL-202308-134. The ethics review board at Ji Ning No.1 people’s Hospital granted an exemption from the acquisition of informed consent, given the retrospective nature of the study. All patients’ information was handled confidentially during the data collection and manuscript preparation.
Patient inclusion and exclusion criteria
Clinical data for model development and internal validation were retrospectively collected from the electronic medical record system at Ji Ning No. 1 People’s Hospital, spanning from Jan. 2015 to Dec. 2022. The clinical data for this study were accessed from September 2023 to October 2023. Patients in the candidemia group were included if they met the following criteria: (1) aged ≥ 18 years; (2) requirement for intensive care; (3) expected duration of stay in ICU >48 hours; and (4) at least one blood peripheral positive for Candida spp. Candidemia group excluded criteria: (1) patients who co-infected with bacteremia within 7 days before and after the detection of candidemia; (2) patients with fungal infections other than those caused by Candida species, or (iv) patients who died within 48 hours. (3) patients who were pregnant or lactating. To ensure a comparable periods of risk exposure for both groups, each control patient had a hospitalization duration similar to the time at risk of cases, defined as the number of days from hospital admission to the occurrence of candidemia. To ensure comparable risk exposure periods between candidemia cases and control patients, we implemented a meticulous matching process. Candidemia cases were retrospectively identified using the microbiology laboratory database, and only the first episode was considered if a patient experienced multiple episodes. Control patients diagnosed with bacteremia were selected in a 1:1 ratio and matched to candidemia cases based on several criteria, including age (±5 years), sex, date of hospital admission, and duration of hospitalization at the time of the first positive blood culture. The hospitalization duration for candidemia cases was defined as the number of days from hospital admission to the occurrence of candidemia. Similarly, for control patients, hospitalization duration was defined as the time from admission to the onset of bacteremia, ensuring alignment with the candidemia cases’ risk period. This matching strategy aims to create a balanced and representative dataset for the analysis [23].
For external validation, clinical data were obtained from Teng Zhou Central People’s Hospital, covering the period from Jan. 2020 to Jun. 2023. The external validation data for this study were accessed from January 2024 to February 2024.
Data collection
We determined the potential risk factors for Candida infection through reviewing the past literature and discussions among the experts in the research team. BDG and GM tests, as new biomarkers, have been rarely used in the past, so we did not include those as risk factors in our study. Data collected at baseline encompassed various characteristics, including the duration of stay in ICU, antibiotic usage within the previous two weeks, presence of solid cancer, diabetes, history of chemotherapy, recent surgical procedures, use of central venous catheter (CVC), receipt of total parenteral nutrition (TPN), immunosuppressant treatments and other clinical laboratory data. Prior to analysis, we applied data processing techniques. Cases with missing data excessing 50% of features were excluded from the dataset. To ensure the quality and completeness of the dataset for subsequent analysis, missing values in the remaining cases were imputed using the mean value of the training set and testing set, respectively.
Microbiological test
Blood culture was processed using the automated blood culture system (Bio-Merieux SA, Marcy l’etoile, France). Fungal isolates and bacterial isolates were cultured at 35°C for 48–72 hours. Simultaneously, we conducted gram staining and microscopic examination. Strain identification was carried out using the VITEK2 Compact system (Bio-Merieux SA, Marcy l’etoile, France) for both fungal and bacterial isolates.
Descriptive analysis
Continuous data were presented as mean± standard deviation (SD) or median (interquartile range, IQR), while categorical data were reported as frequencies and percentages. Univariate analyses were conducted, with continues data analyzed using independent t-test or non-parametric tests, and categorical data analyzed using the Chi-square or Fisher’s exact test. The descriptive analyses were performed using IBM SPSS Statistics for Windows, Version 25.0. A significance level of p <0.05 was considered statistically significant.
Feature selection techniques
Feature selection is a crucial process aimed at identifying the most relevant subset of variables within a dataset. This practice serves to reduce the number of features, enhance the model’s generalization ability, and mitigate the risk of overfitting [24]. While machine learning algorithms in our study can automatically rank features of importance, they may occasionally introduce a small number of redundant features due to the inherent randomness in feature selection [25]. To address this problem, we employed recursive feature elimination (RFE) on the training set for feature selection. RFE involves the iterative removing of the least important features, allowing us to identify the optimal feature combination that can improve overall generalization performance [26].
Model development and external validation
We used a range of machine learning algorithms, including extreme gradient boosting (XGB), logistic regression (LR), support vector machines (SVM), recurrent neural network (RNN) and random forest (RF), to establish candidemia prediction models with the selected variables. Extreme gradient boosting (XGBoost) operates by sequentially constructing an ensemble of decision trees, where each tree corrects the errors of its predecessors. This method uses gradient descent to minimize the loss function and enhance model performance [27]. Logistic Regression (LR) is a form of linear regression aimed at establishing the relationship between a binary response variable (indicating the occurrence of an event) and a set of explanatory variables. It achieves this by using a logistic function to model the probability of the binary outcome [28]. Support Vector Machine (SVM) works by constructing an optimal hyperplane that maximizes the margin between different classes. For nonlinear problems, it employs kernel functions to project data into higher-dimensional spaces for effective classification [29]. Recurrent Neural Networks (RNN) simulate the interconnected structure of neurons. Input data passes through multiple layers of nodes (neurons), which apply weighted sums and activation functions to learn complex patterns and make predictions [30]. Random Forest (RF) is an ensemble method that trains multiple regression trees on different random subsets of the data. At each node, a randomly selected subset of predictors is used, which enhances robustness to predictor correlations [31].
The entire dataset was randomly divided into ten groups. In each iteration, seven groups of patients were assigned to the training cohort, while the remaining three groups were assigned to the testing cohort. These models (XGB, LR, SVM, RNN and RF) underwent ten-fold cross-validation on the training set to determine the optimal hyperparameters. Using the optimal hyperparameters for each model, we trained the models on the training set. Subsequently, we applied these optimized machine learning models to the testing cohort and external cohort for internal and external evaluation.
External validation was performed using an independent set of electronic records of healthy individuals obtained from Teng Zhou Central People’s Hospital spanning from 2020 to 2023. Through machine learning algorithm of RFE, we have obtained features for constructing an early stage candidemia predict model. Then, we collected data of the selected features from Teng Zhou Central People’s Hospital for external validation of candidemia predict model.
Evaluating performance and interpretation of models
We use receiver operating characteristic curve (ROC) to predict power of all the models for candidemia diagnosis. The area under the ROC curve (AUCROC) served as a metric to assess their capacity to distinguish candidemia from bacteremia. The best predict model was defined as the one with the highest AUC value. Furthermore, we assessed performance using metrics such as accuracy, precision, recall, and F1-score to comprehensively evaluate the predictive model’s effectiveness [30]. Score and accuracy were calculated using a threshold of 0.5. Model calibration was visualized through the calibration curve [32]. These curves plot the mean decile prediction probability for patients in each group. Calibration results were evaluated by examining the proximity of the calibration curve to the identity line (y = x), representing the standard calibration. In addition, we conducted a decision curve analysis to evaluate the potential benefits of clinical decision- making based on RF model predictions.
Given that machine learning model are often considered as black-boxes, understanding the impact of each risk variable on predict model can be challenging. To address this problem, we used Shapley additive explanation (SHAP) value [33] to explain the contribution of each feature to specific prediction made by the optimal model. SHAP value was conducted using the “create_explainer” and “shap” function in “shapper” R package (version 4.3.2). Finally, we used nomograms to estimate probability of individual candidemia infection using the “nomogram “function in the “rms” R package (version 4.3.2)[34]. The nomogram analysis considered multiple variables while offers a more accurate prediction and enhances decision-making for candidemia prediction.
Results
Significant differences in patient parameters
In this study, we included a total of 334 patients, comprising 167 patients with candidemia and 167 with bacteremia, to construct a risk prediction model for early candidemia diagnosis in ICU setting. The patient cohort was divided into a training set that consisted of 70% of the patients (234 patients), and a testing set that comprised the remaining 30% (100 patients). Additionally, data from 77 patients, with 38 diagnosed with candidemia and 39 with bacteremia, were recruited from Teng Zhou Central People’s Hospital to serve as an external validation set. The procedure for patient’s selection is detailed in Fig 1.
(A)Patients select from Jining NO. 1 people’s hospital for model development which including the training (n = 234) and internal validation sets (n = 100). (B) Patients select from Teng Zhou People’s Hospital for external validation set (n = 77).
Comparisons of the baseline features between the candidemia and bacteremia group were provided in Table 1. As shown in Table 1, age is significantly difference between candidemia and bacteremia patients. There is no statistically significant difference in comorbidities between candidemia and bacteremia patients. The key characteristics of the patients in the training and testing dataset are presented in Table 2. As shown in Table 2, several features, including age, presence of central venous catheter (CVC), platelet count, use of total parenteral nutrition (TPN), and duration of stay in the ICU, exhibited significant differences between patients with candidemia and those with bacteremia patients, both in training and test sets. However, there was no difference in gender, presence of fever, alanine transaminase (ALT), aspartate transaminase (AST), albumin, glutamyl transpeptidase (GGT), total bilirubin, creatinine, diabetes, history of solid cancer, chemotherapy, use of immunosuppressive drugs, antibiotic therapy, prior abdominal surgery, white blood count (WBC), neutrophils count, lymphocyte count, C-reactive protein (CRP), procalcitonin (PCT) levels, and hemoglobin level between candidemia and bacteremia patients. A detailed breakdown of the selected features for candidemia and bacteremia in external validation can be found in S1 Table. The comparison of selected features between internal and external validation can be found in S2 Table, WBC and CRP exhibited significant difference.
Model development and selected features
To identify the most relevant features for our prediction model, the Recursive Feature Elimination (RFE) algorithm, a tree-based method, was utilized to rank the best feature combination on the training set [35]. Finally, the most significant 18 features were selected according to their importance, which include total bilirubin, age, platelet count, hemoglobin, CVC, lymphocyte, duration of stay in ICU, neutrophils, monocyte, antibiotic therapy, PCT, CRP, WBC, immunosuppressive drugs, TPN, abdominal surgery, chemotherapy and solid cancer (Fig 2). These selected features were used for the subsequent model development.
The dotted line shows the highest accuracy is achieved with 18 features. The horizontal axis is the number of feature selections, and the vertical axis is the prediction accuracy.
Model performance for predicting candidemia
We employed multiple prediction models, including Random Forest (RF), Logistic Regression (LR), Support Vector Machine (SVM), extreme gradient boosting (XGB), and Recurrent Neural Network (RNN), to forecast the recurrence of candidemia with the 18 features which were selected. In Table 3, we present a comparative analysis of these models, demonstrating their performance metrics regarding accuracy, precision, recall, F1-score and Area Under the Curve (AUC). The ROC curves were constructed based on the AUC value (Fig 3). For the ML models, we conduct repeated 10-folding cross validation based on the optimization of AUC to identify the optimal model parameters. Our analysis demonstrated the effectiveness of all models, as each exhibited strong predictive performance with an AUC greater than 0.80. Notably, the RF model outperformed the others with an AUC of 0.87. The highest accuracy attained for predicting candidemia was 0.79 achieved by both the RF and RNN models.
ROC: Receiver operating characteristic curve; auc: Area under the curve; LR: Logistic regression; SVM: Support vector machine; RF: Random forest; XGB: Extreme gradient boosting. RNN: Recurrent neural network.
The calibration curves for our predictive models were presented in Fig 4. The Random Forest and Logistic Regression models showed excellent calibration, with their curves closely aligning with the diagonal line. The mean absolute errors (MAE) were 0.088 and 0.094, respectively. Likewise, the Recurrent Neural Network and Support Vector Machine models exhibit strong calibration performance. However, the calibration curve for the Extreme Gradient Boosting model noticeably deviates from diagonal line, suggesting a potential inaccuracy in comparison to the other models. In addition, we also provide the decision curves that assess the clinical usefulness of the RF model on the test set in Fig 5. These curves show that the RF model offers substantial benefit when the threshold probability of candidemia falls within the range of 55% to 85%.
The calibration curve plotted means predicted probabilities for each model against the observed event frequencies on the testing set (n = 100), using decile-binned data.
X-axis indicates the threshold probability for candidemia diagnosis and Y-axis indicates the net benefit. The red line represents the random forest model.
The Random Forest model continued to perform well again in our external validation cohort from Teng Zhou Central People’s Hospital. As shown in Table 4, the RF model achieved a good predictive performance with an AUC of 0.83, slightly lower than the testing set. The RF model exhibited a precision of 0.83 and accuracy of 0.81 in the external validation cohort. These results confirm the model’s reliability in correctly discriminating true positives and true negatives.
Interpretation of random forest predictive model
We used SHAP values to elucidate the contribution of individual features in the Random Forest model. As shown in Fig 6, all features are ranked by their importance. The most influential features in red are positioned at the top, while less significant ones in blue are at the bottom. There are a few key insights from SHAP analysis. First, total bilirubin emerges as the most crucial predictor for candidemia in the RF model, appearing that a higher level of total bilirubin is associated with an elevated risk of candidemia. Second, several other factors positively correlate with the risk of candidemia, including age, antimicrobial therapy, neutrophil count, use of immunosuppressive drugs, total parenteral nutrition, abdominal surgery, chemotherapy, the presence of a central venous catheter, a history of solid cancer, and duration of ICU stay. Third, lower levels of platelets, hemoglobin, lymphocytes, white blood cells, C-reactive protein, procalcitonin, and monocytes are negatively associated with the risk of candidemia.
The SHAP values were calculated using the training data (n = 234). The plot sorts features by the sum of SHAP value magnitude over all samples. The blue to red color represents the feature value (red high, blue low). The x-axis measures the impacts on the model output (right positive, left negative).
The predictive nomogram for candidemia probability
In this study, we constructed a predictive nomogram based on 18 selected features to estimate the probability of candidemia. The primary goal of this analysis is to provide clinicians with a valuable resource for assessing and quantifying the likelihood of candidemia based on specifical clinical characteristics. As shown in Fig 7, each feature is assigned a probability score within a range from 0 to 100. The cumulative scores for all the features are then summed to generate a total score. This total score can then be projected onto the lower risk axis of the nomogram, enabling the prediction of the risk of candidemia diagnosis in ICU patients. To further facilitate clinical use, we plan to convert the content of Fig 5 into a list of point values and include it in the S3 Table.
The nomogram was developed using the training data (n = 234). To use the nomogram, we first score each patient’s variables to the score axis, added points of all the variables, draw a line from the total score axis to determine candidemia predicting probability on the lower line of the nomogram.
Discussion
In this study, we aimed to develop a predicative model for candidemia in intensive care unit patients using machine learning techniques. Our study cohort included 167 patients with candidemia infection and 167 patients with bacteremia infection admitted to the ICU at Ji Ning NO.1 people’s hospital from Jan. 2015 to Dec. 2022. Our approach distinguishes itself from prior researches that mainly focused on identifying risk factor for candidemia susceptibility, where healthy people or non-infected patients in medical ward served as the control group. Here, we not only identified risk factors but also constructed a diagnostic model, with bacteremia patients serving as the control model.
When compared our study with Li, et al.’s predictive model for candidemia in cancer patients [36], it is worth noting that while they achieved a slightly higher AUC value, variance in patient selection criteria plays a significant influence. Our study exclusively focused on ICU patients, a group with widely distributed risk factors. By using bacteremia patients within the same ICU as the control group in our model development, we aimed in order to better predicate the candidemia especially in ICU setting. Li et al. also employed bacteremia patients as the control group in their predictive model for invasive candidiasis [37], constituting 81.63% of candidemia and 64.79% of bacteremia cases from ICU, which yielded an AUC value of 0. 92. The slightly lower AUC values in this study may be attributed to the differences in the selected patient populations. An important distinction in our study is the use of data from an external hospital for validation, a practice not commonly observed in previous candidemia prediction study. This external validation provides additional credibility to the generalizability of our model.
We employed a range of machine learning algorithms, including LR, RF, SVM, XGB, and RNN, to construct diagnostic models using routine clinical data. These models displayed robust performance in distinguishing candidemia from bacteremia among ICU patients. Among these models, the RF model stood out as the most promising candidate. It achieved impressive precision, recall, and F1 scores of 0.85, 0.69, and 0.76, respectively. These metrics were either equal to or greater than the corresponding scores obtained by the RNN model (0.78). When subjected to external validation, the RF model maintained its excellence, yielding an accuracy of 0.81, precision of 0.83, recall of 0.63, and F1 score of 0.71 reflective of its exceptional diagnostic performance. The decision to select the Random Forest (RF) model as the final machine learning model was driven by its remarkable diagnostic performance, highlighted by a high AUC value of 0.87 and these outstanding evaluation metrics. In addition, RF algorithm displays an advantage in handling missing data points, whether they are numerical or factor variables. This capability is particularly valuable in the context of clinical studies, where missing data can be a common occurrence.
The recurrent neural network (RNN) model also showed strong diagnostic performance, with an AUC of 0.84 and the highest accuracy (0.79) among the models tested. RNNs are recognized for their ability to model sequential data and complex temporal patterns, suggesting potential improvements with larger datasets and time-series features. Although not outperforming the RF model, the RNN’s capacity to capture intricate patterns remains a noteworthy advantage, particularly in scenarios involving larger, more dynamic datasets. Notably, prior studies like Cleophas et al. [38] have highlighted the benefits of RNNs in capturing complex relationships, though performance can vary significantly with data quality and quantity.
Logistic regression, traditionally favored for its interpretability, exhibited an AUC of 0.82, which is slightly lower than the more complex models like RF and RNN. This result is consistent with existing literature indicating that logistic regression often provides a baseline level of performance but may not fully capture the complexities of clinical data, especially when interactions between variables are nonlinear and multifaceted. Studies such as Holmes et al. [39] have found logistic regression effective but limited when dealing with complex, high-dimensional data. Nevertheless, its simplicity and ease of interpretation remain beneficial for understanding individual predictors’ influence.
Extreme gradient boosting (XGB) achieved an AUC of 0.83, showcasing its ability to handle structured data and select relevant features effectively. While its performance was competitive, the RF model consistently outperformed it across key metrics, highlighting the importance of model selection based on dataset characteristics and clinical needs. Similar studies, such as those conducted by Baxter et al. [40], have demonstrated the strength of XGB in various clinical contexts, although the choice between XGB and RF often depends on specific data characteristics and computational considerations.
The support vector machine (SVM) achieved an AUC of 0.82, similar to logistic regression. Despite SVMs’ effectiveness in high-dimensional spaces, their performance was limited, likely due to kernel selection and clinical data complexity. The results are consistent with other studies in the field that report SVM as a powerful classifier, particularly when paired with appropriate kernel functions and parameter tuning [36].
SHAP values were used to identify the crucial predictors of candidemia. Notably, the following factors emerged as positively associated with candidemia risk: high levels of total bilirubin, patient age, total parenteral nutrition (TPN), duration of ICU stay, the presence of a central venous catheter (CVC), and the influence of glucocorticoids. Among these factors, high level of total bilirubin is of most valuable feature. Serum bilirubin is a stable and well-established marker of liver impairment [41,42]. The level of bilirubin is negatively correlated with antimicrobial immunity, rendering individuals more susceptible to candidiasis [43]. Albumin act as an antioxidant and free radical scavenger which can offer protection from inflammatory processes [44]. Patients with hypoalbuminemia are more susceptible to infect opportunistic pathogens like candidemia. However, hypoalbuminemia was not identified as a predict feature for distinguishing between candidemia and bacteremia in this model. In inflammatory states, capillary escape and breakdown of albumin increased which can lead to hypoalbuminemia [44]. Besides, server inflammation often combined with liver failure, which could decrease serum albumin mass because albumin synthesis rate is mainly decreased in liver failure [45]. Low serum albumin levels have been associated with an increased risk of mortality in patients with bloodstream infections because of bacterial or fungal infection [46]. Several studies have demonstrated a correlation between hypoalbuminemia and higher mortality rates in these patients. For example, a study by Lao et al. [47] found that low serum albumin levels were independently associated with increased mortality in patients with candidemia. Similarly, a study by A. Capelastegui et al. [48] showed that hypoalbuminemia was a significant predictor of mortality in patients with bacteremia.
The remaining positive predicators identified in our study have been supported by previous research in various contexts. For example, both TPN and CVC presence have been recognized as an independent risk factor for candidemia [2]. TPN induced candidemia may arise from alternations in gut microbiota [49], leading to changes in epithelial barrier function and subsequent intestinal inflammation [50]. The strong association of CVC presence with candidemia, even after adjusting for TPN, underscores its significance for candidemia [19].
Duration of stay in the ICU has also been proposed as an essential indicator of candidemia [51]. This metric is influenced by various factors, including patient demographics, treatment complexity and comorbidities. Moreover, glucocorticoids, which are used in the management of various immune diseases, carry several side effects, such as insulin resistance, increased susceptibility to severe infections, Cushing’s syndrome, and peptic ulcers [52]. Prolonged treatment with high-dose systemic glucocorticoids elevates the risk of candidemia [53].
In addition, our study highlighted the association of recent abdominal surgery, chemotherapy, solid cancer and antimicrobial therapy with candidemia. Surgery may compromise the natural body barrier [54], cytotoxic chemotherapy can disrupt the mucosal integrity [55], and antimicrobial therapy may induce alterations in intestinal flora [2]. All these factors collectively create an environment conducive to Candida invasion through mesenteric circulation [2,55].
Platelets are widely recognized as integral components of the host’s defense system against invasive infections [56]. In our study, low platelet count was positively associated with candidemia prediction. This finding is significant since platelets paly a protective role in integrating vascular endothelial cells against Candida–induced damage [56]. The decreased presence of platelets may then increase the risk of candidemia. PCT and CRP are well-known for their predictive value in assessing the prognosis of patients with candidiasis and bacterial infections. Some studies have even attempted to use these two markers to distinguish candidemia from bacteremia [57]. Like PCT and CRP, lymphocytes, WBC and monocytes were negatively correlated with candidemia in this study. These findings are particularly intriguing, as white blood play a pivotal role in initiation and maintenance of phagocyte-dependent fungal eliminations [58]. Furthermore, lymphocyte subtyping has even been used as a good marker in the early diagnosis of candidemia [59], which we did not included in this study. Interestingly, our study uncovers an association between low hemoglobin level and high level of bilirubin with an increased incidence of candidemia. This observation might be attributed to hemolytic factors produced by Candida, which can lead to rupture of red blood cells upon binding to the cell membrane [60].
This study has several clinical application prospects. Firstly, the developed machine learning models provide a reliable early prediction method for candidemia in intensive care unit (ICU) patients, which could assist clinicians in making timely decisions regarding the initiation of antifungal therapy. By identifying high-risk patients in advance, hospitals can arrange treatment and monitoring equipment in advance, thereby avoiding additional costs and adverse outcomes resulting from delayed diagnosis. Accurately identifying candidemia patients helps hospitals allocate resources more effectively and can reduce unnecessary resource wastage. Secondly, the identification of key risk factors through SHAP analysis reveals potential mechanisms underlying the development of candidemia, providing important insights for implementing personalized interventions for high-risk patients. Additionally, the predictive nomogram offers a convenient tool for clinicians to estimate the probability of candidemia based on patients’ clinical characteristics, aiding in the optimization of individualized treatment plans. Overall, this study provides new insights and approaches for improving patient outcomes, optimizing resource utilization, and enhancing the management strategies for fungal bloodstream infections.
There are several limitations in our study. First, the sample size was constrained by the fact that the data was collected from a single medical center. The relatively small sample size may have implications for the generalizability of our findings. Second, our study design was retrospective in nature. This choice was necessitated by the low incidence of candidemia, which limited our ability to incorporate the latest biomarkers into our predictive model. Third, although our model demonstrated good performance and achieved acceptable results during external validation, it is essential to acknowledge that retrospective studies inherently carry the potential bias. Therefore, there is a need for multiple-center prospective cohorts to expand our understanding and refine the predicative model.
Conclusions
In this study, we have demonstrated that the machine learning model on the RF algorithm outperforms other algorithms in predicting candidemia. Our findings emphasize the critical importance of considering total bilirubin levels as a key predictor for candidemia, particularly among ICU patients. The RF model’s predictive capabilities could serve as a valuable tool for clinicians in the early diagnosis and treatment of candidemia. The results of this study suggest the potential utility of AI-assisted diagnosis for enhancing the clinical management of candidemia, representing a promising avenue for improving patient care in this context.
Supporting information
S1 Table. Characteristics of selected features in the external validation group.
Definition of abbreviations: y: Years; TPN: Total parenteral nutrition; CVC: Central venous catheter; WBC: White blood cell; PCT: Procalcitonin; CRP: C-reactive protein; ICU: Intensive care unit; IQR: Interquartile range; SD: Standard deviation; *The t-test for metric variable if data are normally distributed; **the Chi-square test (big sample size) and Fisher’s exact test (small sample size) for categorical variables; ***the Mann–Whitney U-test for metric variables if data are not normally distributed.
https://doi.org/10.1371/journal.pone.0309748.s001
(DOCX)
S2 Table. Comparison of selected features between internal and external validation.
Definition of abbreviations: y: Years; TPN: Total parenteral nutrition; CVC: Central venous catheter; WBC: White blood cell; PCT: Procalcitonin; CRP: C-reactive protein; ICU: Intensive care unit; IQR: Interquartile range; SD: Standard deviation; *The t-test for metric variable if data are normally distributed; **the Chi-square test (big sample size) and Fisher’s exact test (small sample size) for categorical variables; ***the Mann–Whitney U-test for metric variables if data are not normally distributed.
https://doi.org/10.1371/journal.pone.0309748.s002
(DOCX)
S3 Table. Nomogram points system for clinical variables.
Definition of abbreviations: TPN: Total parenteral nutrition; CRP: C-reactive protein; WBC: White blood cell; CVC: Central venous catheter.
https://doi.org/10.1371/journal.pone.0309748.s003
(DOCX)
References
- 1. Wisplinghoff H, Bischoff T, Tallent SM, Seifert H, Wenzel RP, Edmond MB. Nosocomial bloodstream infections in US hospitals: analysis of 24,179 cases from a prospective nationwide surveillance study. Clin Infect Dis. 2004;39(3):309–17. Epub 2004/08/13. pmid:15306996.
- 2. Poissy J, Damonti L, Bignon A, Khanna N, Von Kietzell M, Boggian K, et al. Risk factors for candidemia: a prospective matched case-control study. Crit Care. 2020;24(1):109. Epub 2020/03/20. pmid:32188500; PubMed Central PMCID: PMC7081522.
- 3. Zaoutis TE, Argon J, Chu J, Berlin JA, Walsh TJ, Feudtner C. The epidemiology and attributable outcomes of candidemia in adults and children hospitalized in the United States: a propensity analysis. Clin Infect Dis. 2005;41(9):1232–9. Epub 2005/10/06. pmid:16206095.
- 4. Fisher BT, Boge CLK, Xiao R, Shuster S, Chin-Quee D, Allen J, et al. Multicenter Prospective Study of Biomarkers for Diagnosis of Invasive Candidiasis in Children and Adolescents. Clin Infect Dis. 2022;75(2):248–59. Epub 2022/02/09. pmid:35134165; PubMed Central PMCID: PMC9890499.
- 5. Pappas PG, Lionakis MS, Arendrup MC, Ostrosky-Zeichner L, Kullberg BJ. Invasive candidiasis. Nat Rev Dis Primers. 2018;4:18026. Epub 2018/05/12. pmid:29749387.
- 6. Bassetti M, Giacobbe DR, Vena A, Wolff M. Diagnosis and Treatment of Candidemia in the Intensive Care Unit. Semin Respir Crit Care Med. 2019;40(4):524–39. Epub 2019/10/05. pmid:31585478.
- 7. Magadia RR, Weinstein MP. Laboratory diagnosis of bacteremia and fungemia. Infect Dis Clin North Am. 2001;15(4):1009–24. Epub 2002/01/10. pmid:11780265.
- 8. Playford EG, Lipman J, Jones M, Lau AF, Kabir M, Chen SC, et al. Problematic Dichotomization of Risk for Intensive Care Unit (ICU)-Acquired Invasive Candidiasis: Results Using a Risk-Predictive Model to Categorize 3 Levels of Risk From a Multicenter Prospective Cohort of Australian ICU Patients. Clin Infect Dis. 2016;63(11):1463–9. Epub 2016/09/08. pmid:27601224.
- 9. Morrell M, Fraser VJ, Kollef MH. Delaying the empiric treatment of candida bloodstream infection until positive blood culture results are obtained: a potential risk factor for hospital mortality. Antimicrobial agents and chemotherapy. 2005;49(9):3640–5. Epub 2005/08/30. pmid:16127033; PubMed Central PMCID: PMC1195428.
- 10. Zilberberg MD, Kollef MH, Arnold H, Labelle A, Micek ST, Kothari S, et al. Inappropriate empiric antifungal therapy for candidemia in the ICU and hospital resource utilization: a retrospective cohort study. BMC Infect Dis. 2010;10:150. Epub 2010/06/08. pmid:20525301; PubMed Central PMCID: PMC2890008.
- 11. Liesenfeld O, Lehman L, Hunfeld KP, Kost G. Molecular diagnosis of sepsis: New aspects and recent developments. Eur J Microbiol Immunol (Bp). 2014;4(1):1–25. Epub 2014/03/29. pmid:24678402; PubMed Central PMCID: PMC3955828.
- 12. Zhang J, Cheng W, Li D, Chen J, Zhao G, Wang H, et al. Development and Validation of a Risk Score for Predicting Invasive Candidiasis in Intensive Care Unit Patients by Incorporating Clinical Risk Factors and Lymphocyte Subtyping. Front Cell Infect Microbiol. 2022;12:829066. Epub 2022/05/17. pmid:35573797; PubMed Central PMCID: PMC9091371.
- 13. Paiva JA, Charles PE. Biomarker-guided antifungal therapy in patients with suspected invasive candidiasis: Ready for prime time? Intensive Care Med. 2017;43(12):1889–91. Epub 2017/11/10. pmid:29119220.
- 14. Posteraro B, Tumbarello M, De Pascale G, Liberto E, Vallecoccia MS, De Carolis E, et al. (1,3)-β-d-Glucan-based antifungal treatment in critically ill adults at high risk of candidaemia: an observational study. J Antimicrob Chemother. 2016;71(8):2262–9. Epub 2016/04/30. pmid:27125554.
- 15. Chertoff J, Ataya A. The Timing of Early Antibiotics and Hospital Mortality in Sepsis: Playing Devil’s Advocate. Am J Respir Crit Care Med. 2017;196(7):934–5. Epub 2017/05/10. pmid:28485616.
- 16. Bhavani SV, Lonjers Z, Carey KA, Afshar M, Gilbert ER, Shah NS, et al. The Development and Validation of a Machine Learning Model to Predict Bacteremia and Fungemia in Hospitalized Patients Using Electronic Health Record Data. Crit Care Med. 2020;48(11):e1020–e8. Epub 2020/08/17. pmid:32796184; PubMed Central PMCID: PMC7554143.
- 17. Leiner T, Rueckert D, Suinesiaputra A, Baessler B, Nezafat R, Isgum I, et al. Machine learning in cardiovascular magnetic resonance: basic concepts and applications. J Cardiovasc Magn Reson. 2019;21(1):61. Epub 2019/10/09. pmid:31590664; PubMed Central PMCID: PMC6778980.
- 18. Yoo J, Kim SH, Hur S, Ha J, Huh K, Cha WC. Candidemia Risk Prediction (CanDETEC) Model for Patients With Malignancy: Model Development and Validation in a Single-Center Retrospective Study. JMIR Med Inform. 2021;9(7):e24651. Epub 2021/07/27. pmid:34309570; PubMed Central PMCID: PMC8367162.
- 19. Atamna A, Eliakim-Raz N, Mohana J, Ben-Zvi H, Sorek N, Shochat T, et al. Predicting candidemia in the internal medicine wards: a comparison with gram-negative bacteremia-a retrospectives study. Diagnostic microbiology and infectious disease. 2019;95(1):80–3. Epub 2019/05/28. pmid:31129007.
- 20. Gao Y, Tang M, Li Y, Niu X, Li J, Fu C, et al. Machine-learning based prediction and analysis of prognostic risk factors in patients with candidemia and bacteraemia: a 5-year analysis. PeerJ. 2022;10. pmid:35726257
- 21. Hu WH, Lin SY, Hu YJ, Huang HY, Lu PL. Application of machine learning for mortality prediction in patients with candidemia: Feasibility verification and comparison with clinical severity scores. Mycoses. 2024;67(1):e13667. Epub 2023/11/02. pmid:37914666.
- 22. Moons KG, Altman DG, Reitsma JB, Ioannidis JP, Macaskill P, Steyerberg EW, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162(1):W1–73. Epub 2015/01/07. pmid:25560730.
- 23. Ripoli A, Sozio E, Sbrana F, Bertolino G, Pallotto C, Cardinali G, et al. Personalized machine learning approach to predict candidemia in medical wards. Infection. 2020;48(5):749–59. Epub 2020/08/03. pmid:32740866.
- 24. Remeseiro B, Bolon-Canedo V. A review of feature selection methods in medical applications. Comput Biol Med. 2019;112:103375. Epub 2019/08/06. pmid:31382212.
- 25. Lin H, Xue Y, Chen K, Zhong S, Chen L. Acute coronary syndrome risk prediction based on gradient boosted tree feature selection and recursive feature elimination: A dataset-specific modeling study. PLoS One. 2022;17(11):e0278217. Epub 2022/11/30. pmid:36445881; PubMed Central PMCID: PMC9707772.
- 26. Guyon I, Elisseeff A. An Introduction to Variable and Feature Selection. Journal of machine learning research. 2003;(3):1157–82.
- 27. Lv CX, An SY, Qiao BJ, Wu W. Time series analysis of hemorrhagic fever with renal syndrome in mainland China by using an XGBoost forecasting model. BMC Infect Dis. 2021;21(1):839. Epub 2021/08/21. pmid:34412581; PubMed Central PMCID: PMC8377883.
- 28. Cox DRJJotRSSSBSM. The regression analysis of binary sequences. 1958;20(2):215–32.
- 29. Cortes C, Vapnik VJMl. Support-vector networks. Machine Learning. 1995;20:273–97.
- 30. Zeng Z, Tang X, Liu Y, He Z, Gong X. Interpretable recurrent neural network models for dynamic prediction of the extubation failure risk in patients with invasive mechanical ventilation in the intensive care unit. BioData Min. 2022;15(1):21. Epub 2022/09/27. pmid:36163063; PubMed Central PMCID: PMC9513908.
- 31. Breiman L. Random forests. Machine learning. 2001;45:5–32.
- 32. Alba AC, Agoritsas T, Walsh M, Hanna S, Iorio A, Devereaux PJ, et al. Discrimination and Calibration of Clinical Prediction Models: Users’ Guides to the Medical Literature. JAMA. 2017;318(14):1377–84. Epub 2017/10/20. pmid:29049590.
- 33. Li J, Gong M, Joshi Y, Sun L, Huang L, Fan R, et al. Machine Learning Prediction Model for Acute Renal Failure After Acute Aortic Syndrome Surgery. Frontiers in medicine. 2021;8:728521. Epub 2022/02/04. pmid:35111767; PubMed Central PMCID: PMC8801502.
- 34. Giorgi FM, Ceraolo C, Mercatelli D. The R Language: An Engine for Bioinformatics and Data Science. Life (Basel). 2022;12(5). Epub 2022/05/29. pmid:35629316; PubMed Central PMCID: PMC9148156.
- 35. Deberneh HM, Kim I. Prediction of Type 2 Diabetes Based on Machine Learning Algorithm. Int J Environ Res Public Health. 2021;18(6). Epub 2021/04/04. pmid:33806973; PubMed Central PMCID: PMC8004981.
- 36. Li J, Li Y, Gao Y, Niu X, Tang M, Fu C, et al. Prediction of Prognostic Risk Factors in Patients with Invasive Candidiasis and Cancer: A Single-Centre Retrospective Study. BioMed Research International. 2022;2022:1–9. pmid:35692595
- 37. Li Y, Wu Y, Gao Y, Niu X, Li J, Tang M, et al. Machine-learning based prediction of prognostic risk factors in patients with invasive candidiasis infection and bacterial bloodstream infection: a singled centered retrospective study. BMC Infect Dis. 2022;22(1):150. Epub 2022/02/15. pmid:35152879; PubMed Central PMCID: PMC8841094.
- 38. Cleophas TJ, Cleophas TF. Artificial intelligence for diagnostic purposes: principles, procedures and limitations. cclm. 2010;48(2):159–65. pmid:20001439
- 39. Holmes JH, Sacchi L, Bellazzi R, Peek N. Artificial Intelligence in Medicine AIME 2015. Artif Intell Med. 2017;81:1–2. Epub 2017/07/25. pmid:28733119.
- 40.
<embc44109.2020.9175287.pdf>.
- 41. Perez Ruiz de Garibay A, Kortgen A, Leonhardt J, Zipprich A, Bauer M. Critical care hepatology: definitions, incidence, prognosis and role of liver failure in critically ill patients. Critical Care. 2022;26(1). pmid:36163253
- 42. Hadem J, Stiefel P, Bahr MJ, Tillmann HL, Rifai K, Klempnauer J, et al. Prognostic implications of lactate, bilirubin, and etiology in German patients with acute liver failure. Clin Gastroenterol Hepatol. 2008;6(3):339–45. Epub 2008/03/11. pmid:18328438.
- 43. Chang Y-C, Chen J-S, Yin C-H, Shin-Jung Lee S, Chen W-C. Candidemia in hospitalized cirrhotic patients with bloodstream infection: A retrospective analysis and brief summary of published studies. Journal of the Chinese Medical Association. 2022;85(3):295–303. pmid:35259132
- 44. Soeters PB, Wolfe RR, Shenkin A. Hypoalbuminemia: Pathogenesis and Clinical Significance. JPEN J Parenter Enteral Nutr. 2019;43(2):181–93. Epub 2018/10/06. pmid:30288759; PubMed Central PMCID: PMC7379941.
- 45. Rafferty MJ, Mcmillan DC, Preston TC, Hamid R, Small AC, Joshi N, et al. Reprioritisation of Liver Export Protein Synthesis in Patients with Decompensated Alcoholic Liver Disease. 2016;(3).
- 46. Wiedermann CJ. Hypoalbuminemia as Surrogate and Culprit of Infections. Int J Mol Sci. 2021;22(9). Epub 2021/05/01. pmid:33925831; PubMed Central PMCID: PMC8123513.
- 47. Lao M, Li C, Li J, Chen D, Ding M, Gong Y. Opportunistic invasive fungal disease in patients with type 2 diabetes mellitus from Southern China: Clinical features and associated factors. Journal of diabetes investigation. 2020;11(3):731–44. Epub 2019/11/24. pmid:31758642; PubMed Central PMCID: PMC7232281.
- 48. Capelastegui A, Zalacain R, Bilbao A, Egurrola M, Iturriaga LA, Quintana JM, et al. Pneumococcal pneumonia: differences according to blood culture results. BMC Pulm Med. 2014;14:128. Epub 2014/08/07. pmid:25096919; PubMed Central PMCID: PMC4127038.
- 49. Demehri FR, Barrett M, Ralls MW, Miyasaka EA, Feng Y, Teitelbaum DH. Intestinal epithelial cell apoptosis and loss of barrier function in the setting of altered microbiota with enteral nutrient deprivation. Front Cell Infect Microbiol. 2013;3:105. Epub 2014/01/07. pmid:24392360; PubMed Central PMCID: PMC3870295.
- 50. Harris JK, El Kasmi KC, Anderson AL, Devereaux MW, Fillon SA, Robertson CE, et al. Specific microbiome changes in a mouse model of parenteral nutrition associated liver injury and intestinal inflammation. PLoS One. 2014;9(10):e110396. Epub 2014/10/21. pmid:25329595; PubMed Central PMCID: PMC4203793.
- 51. Zhang Z, Zhu R, Luan Z, Ma X. Risk of invasive candidiasis with prolonged duration of ICU stay: a systematic review and meta-analysis. BMJ Open. 2020;10(7):e036452. Epub 2020/07/15. pmid:32660950; PubMed Central PMCID: PMC7359383.
- 52. Yang R, Yu Y. Glucocorticoids are double-edged sword in the treatment of COVID-19 and cancers. Int J Biol Sci. 2021;17(6):1530–7. Epub 2021/04/29. pmid:33907516; PubMed Central PMCID: PMC8071771.
- 53. Lionakis MS, Kontoyiannis DP. Glucocorticoids and invasive fungal infections. Lancet. 2003;362(9398):1828–38. Epub 2003/12/05. pmid:14654323.
- 54. Teoh F, Pavelka N. How Chemotherapy Increases the Risk of Systemic Candidiasis in Cancer Patients: Current Paradigm and Future Directions. Pathogens. 2016;5(1). Epub 2016/01/20. pmid:26784236; PubMed Central PMCID: PMC4810127.
- 55. Nucci M, Anaissie E. Should vascular catheters be removed from all patients with candidemia? An evidence-based review. Clin Infect Dis. 2002;34(5):591–9. Epub 2002/01/26. pmid:11810600.
- 56. Yeaman MR, Cheng D, Desai B, Kupferwasser LI, Xiong YQ, Gank KD, et al. Susceptibility to thrombin-induced platelet microbicidal protein is associated with increased fluconazole efficacy against experimental endocarditis due to Candida albicans. Antimicrobial agents and chemotherapy. 2004;48(8):3051–6. Epub 2004/07/27. pmid:15273120; PubMed Central PMCID: PMC478484.
- 57. Wang Q, Wang C, Yang M, Li X, Cui J, Wang C. Diagnostic efficacy of serum cytokines and chemokines in patients with candidemia and bacteremia. Cytokine. 2020;130:155081. Epub 2020/04/05. pmid:32247169.
- 58. Hatinguais R, Willment JA, Brown GD. PAMPs of the Fungal Cell Wall and Mammalian PRRs. Curr Top Microbiol Immunol. 2020;425:187–223. Epub 2020/03/18. pmid:32180018.
- 59. Zhang J, Cui N, Wang H, Han W, Li Y, Xiao M, et al. Invasive Fungal Disease in Critically Ill Patients at High Risk: Usefulness of Lymphocyte Subtyping. J Intensive Care Med. 2020;35(9):909–18. Epub 2018/09/21. pmid:30231674.
- 60. Roy U, Kornitzer D. Heme-iron acquisition in fungi. Current opinion in microbiology. 2019;52:77–83. Epub 2019/07/03. pmid:31265986.