2.2.1. Handling missing values and applying feature importance.

Our initial dataset comprised 32 columns (31 independent features and 1 dependent feature), as detailed in Table 2. The presence of null values does not necessarily indicate data missingness; for instance, null values in binary columns often represent the value 0. To ensure accurate interpretation, we consulted a medical expert familiar with the data collection process. Several features were identified as redundant or non-essential and were subsequently removed to refine the dataset. The ’Code’ feature, representing patient IDs, was removed as it did not contribute to our analysis. Similarly, the ’Valve’ feature was removed as it duplicated information found in the ’CABG_Valve’ feature, and the ’Off_pump’ feature, having values opposite to those of ’On_pump’, was also eliminated.

Furthermore, certain features exhibited a high percentage of missing values, prompting specific actions: ’DURATION’ and ’CHF’, with more than 95% missing values, were removed to preserve the validity and reliability of our model. For features ’PH’, ’MG’, and ’MAP’, which had more than 15% but less than 50% missing values, we employed the K-Nearest Neighbors (KNN) imputation method. This method was crucial in preserving the underlying data structure and maintaining feature integrity by leveraging the similarity between data points. Additionally, six samples that lacked any values were removed.

Upon addressing these issues, 26 of the initial 31 independent features and 1092 of the 1098 samples were retained for further analysis. We also conducted a feature importance analysis using both RF and GBM models, as illustrated in Fig 2. This analysis was instrumental in identifying and retaining the most informative features, using a threshold importance score of 0.001. Only features with significant predictive power were included in further analyses. The intersection of important features identified by these models, as shown in Fig 2, included ’Arrythmia’, ’Number_of_Grafts’, ’Addiction’, ’Arrest’, ’Hight’, ’Weight’, ’MAP’, ’MG’, ’Grade_of_Age’, ’Age’, ’CABG_Valve’, ’EF’, ’IABP’, ’Cross_Clamp’, ’Smoker’, ’MI’, ’Albumin’. These features were carefully considered to ensure robustness in our feature set.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Feature importance with Random Forest and Gradient Boosting.

https://doi.org/10.1371/journal.pone.0311250.g002

2.2.2. Utilizing one-hot encoding.

To incorporate categorical variables into ML models, it’s advisable to transform them into binary variables using one-hot encoding [10, 22]. As an instance, a categorical feature like sex, which can only be female or male, cannot be directly incorporated into a ML model; This technique transforms the Sex column into a binary column, where male is represented by 1 and female by 0.

2.2.3. Partitioning the data into train, validation, and test sets.

To mitigate the risk of overfitting in ML models, we split the dataset into separate training, validation and testing sets [7]. The primary objective of this data partitioning is to evaluate the model’s performance on unseen data, thereby ensuring its generalizability. The training set is used initially to develop the models, while the validation set is applied for fine-tuning the hyperparameters of these models using metaheuristic algorithms. Finally, we evaluate their performance on the test set, which provides insights into their general applicability. To determine the most effective model, we evaluated their performance on various metrics. To achieve this, we divided the data into a 60-20-20 split, with 60% dedicated to training, 20% reserved for validation, and 20% for testing. Specifically, of the 1092 total records, 655 were utilized to train the models, 218 for validation, and 219 to test their performance. This methodical approach allows for a comprehensive assessment of each model’s ability to generalize beyond the training data.

2.2.4. Improving imbalanced data by using SMOTE method.

To address potential biases introduced by imbalanced data in our study, we employed the Synthetic Minority Over-sampling Technique (SMOTE). This method is particularly effective in balancing datasets by synthesizing new samples for the under-represented class [1]. By using SMOTE, we enhanced the dataset’s diversity and improved the generalizability of our models. SMOTE works by selecting data points that are close in the feature space, drawing a line between the points in the feature space, and creating a new point along that line. This approach is crucial for creating a balanced dataset and avoiding model bias towards the majority class.

SMOTE was applied exclusively to the training data to balance it. Initially, the training data had an imbalance between the two classes: class 0 (patients who do not need a ventilator) and class 1 (patients who need a ventilator). After applying SMOTE to the training data, we achieved a balanced distribution with 414 samples in each class, totaling 828 samples. In contrast, the validation set, which did not undergo SMOTE, contains 142 samples that do not need a ventilator and 76 that do, reflecting the original distribution of the dataset. Similarly, the test set has 154 samples that do not need a ventilator and 65 that do. This strategic application of SMOTE ensures that our model training is robust and fair while allowing us to assess model performance against the more naturally imbalanced conditions seen in the validation and test data.

2.2.5. Using the Z-score to standardize the data.

Feature scaling or standardization, a critical step in data preprocessing, is essential when dealing with datasets where features have varying scales or are measured in distinct units. The discrepancies in feature ranges can hinder the effectiveness of many ML models. For instance, in algorithms that use distance-based calculations, a feature with a significantly larger range will exert an undue influence on the distance calculations.

The Z-score is a popular approach for standardizing data. It entails subtracting the mean of each feature and then dividing by its standard deviation [1].

(1)

The Z-scores for each sample are calculated in Eq (1) as you can see above.

Upon completing the standardization procedure, all features will be centered at zero and have a standard deviation of one, ensuring a consistent scale. This standardization is performed separately for the train, validation, and test sets. Standardizing these sets separately safeguards against data leakage, a phenomenon where information from the validation and test data infiltrates the modeling process.

2.3.1. Machine learning & metaheuristic models.

In the first stage of our ensemble model, we implemented four classification models to predict the necessity of ventilators for patients in the cardiac surgery ICU. This involved the use of supervised ML and DL models, specifically LDA, CatBoost, ANN, and XGBoost, as provided in the Scikit-learn, CatBoost, Keras, and xgboost libraries of Python. Each of these classifiers was trained using the training dataset to assess their performance differences.

We chose CatBoost for its ability to handle categorical features without extensive preprocessing and its efficiency in both regression and classification tasks [33]. LDA was selected for its strength in dimensionality reduction and class distinction based on linear combinations of features, which is vital for medical datasets [34]. ANN was chosen for its capability to model non-linear relationships in medical data through deep learning architectures [35]. XGBoost was included due to its proven efficacy and recommendation by the TPOT AutoML algorithm [36], enhancing the ensemble’s performance. These models were validated against empirical evidence, ensuring their superior performance in metrics such as Accuracy, Precision, Sensitivity, Specificity, and F1-score, as documented in [26] study. We reviewed related studies, as presented in Table 1, to support our model selection.

After this, we need to do hyperparameter tuning to determine which parameters of our classifiers are good to be used in the ensemble model to either improve evaluation metrics or avoid overfitting. Hyperparameters in ML are parameters that values are predetermined before the learning process starts [12]. To optimize these parameters, we employed the SA metaheuristic algorithm, a widely recognized method for its efficiency in navigating and optimizing complex parameter spaces. This approach was chosen for its ability to effectively balance the exploration and exploitation of the parameter space, thereby improving evaluation metrics and reducing the risk of overfitting.

The ANN model underwent parameter adjustments including the number of layers, units per layer, activation functions, optimizer type, loss function, batch size, and epochs. For the LDA model, parameters like solver type, number of components, store covariance option, tolerance level, and shrinkage were fine-tuned. The CatBoost model’s tuning focused on parameters such as learning rate, tree depth, L2 leaf regularization, and the number of iterations. The tuning of the XGBoost model included hyperparameters such as C, dual, loss, penalty, tol, learning rate, max depth, max features, min samples leaf, min samples split, n_estimators, and subsample. Each of these hyperparameters was carefully selected based on recommendations from developer documentation and insights from related studies, leading to different numbers of parameters being optimized for each model. This variance is inevitable due to the unique characteristics and capabilities of each model, ensuring each is optimized to enhance performance and robustness against overfitting.

To effectively manage potential overfitting in predictive models, we employed a range of regularization techniques tailored to each model’s architecture. For the LDA model, we considered a shrinkage parameter to be optimized through SA and GA, effectively balancing model complexity and generalization. In CatBoost model, instead of traditional L1 or L2 regularization, we adjusted the l2_leaf_reg parameter to 3, based on preliminary tests which suggested this setting optimally prevents overfitting while maintaining model flexibility. For XGBoost, we applied L1 regularization with an alpha value of 0.01 to encourage feature selection and L2 regularization with a lambda value of 1.0 to penalize larger weights, thus enhancing model generalization across various datasets. Lastly, in the ANN, both L1 and L2 regularizations were set at 0.01 directly in the layers, and a combined L1_L2 regularization was employed to ensure a balance between reducing model complexity and retaining predictive accuracy.

These optimizations were captured in detailed logs, showing the best accuracy, precision, recall (Sensitivity), F1 score, specificity.

The hyperparameters set for each ML and DL model are summarized in Table 3. This table provides a comprehensive overview of the hyperparameters utilized during the tuning phase. The ranges for these hyperparameters were determined based on the developer documentation for each model, expert feedback tailored to our problem characteristics, and insights from other studies, which are referenced in the last column of Table 3. This approach ensures that the tuning is grounded in both theoretical best practices and practical insights relevant to our specific research context. Also, the below pseudo code (Algorithm 1) represents the process of hyperparameter tuning using SA:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. The hyperparameters set for each ML and DL model.

https://doi.org/10.1371/journal.pone.0311250.t003

1. Algorithm 1. Pseudocode for Hyperparameter Selection using Simulated Annealing (SA)
2.  1. Define the hyperparameter space for each model:
3.   ANN_hyperparameters = [number_of_layers, units_per_layer, activation_functions, optimizer, loss_function, batch_size, epochs]
4.   LDA_hyperparameters = [solver, number_of_components, store_covariance, tolerance, shrinkage]
5.   CatBoost_hyperparameters = [learning_rate, depth, l2_leaf_reg, iterations]
6.   XGBoost = [C, dual, loss, penalty, tol, learning_rate, max_depth, max_features, min_samples_leaf, min_samples_split, n_estimators, and subsample]
7.  2. Initialize the SA algorithm parameters:
8.   -initial_temperature (set initial temperature for SA) = 100
9.   -cooling_rate (set cooling rate for the SA process) = 0.95
10.   -max_iterations (set maximum number of iterations for SA) = 250
11.  3. For each model (ANN, LDA, CatBoost, XGBoost):
12.   -Initialize best_solution and best_score to None or initial values
13.   -Set current_solution to a random selection from the model’s hyperparameter space
14.   -Evaluate current_solution using the model’s evaluation function (e.g., evaluate_ann for ANN)
15.   -Set best_solution to current_solution if it’s the first iteration or better than the existing best_solution
16.   For iteration in range(max_iterations):
17.    -Generate new_solution by slightly modifying current_solution’s parameters
18.    -Evaluate new_solution using the model’s evaluation function
19.    -Calculate acceptance_probability using SA algorithm’s acceptance criteria
20.    -If acceptance_probability > random value between 0 and 1:
21.    -Update current_solution to new_solution
22.    -Update best_solution if new_solution is better than current_solution
23.    -Reduce temperature based on cooling_rate
24.   -Return final best_solution for the model
25.  4. After completing SA for all models, select the best hyperparameters for each model.

To evaluate the predictive performance of the models, we need to measure their performance on the validation set for further tuning and the test set for final evaluation and comparisons. The ensemble model, constructed from the individually optimized models, is then tested to choose the best setup based on various evaluation metrics. We will discuss on ensemble model and various evaluation metrics in the next subsection.

2.3.2. Ensemble model construction.

In our study, we have developed an ensemble model that synthesizes the predictive power of four distinct base classifiers—CatBoost, LDA, ANN and XGBoost—each individually fine-tuned using metaheuristic optimization algorithms. This tuning process was essential for adjusting the hyperparameters specific to our data, maximizing the effectiveness of each model within the ensemble framework.

Our ensemble employs a weighted voting mechanism to integrate outputs from each base model. The weighting system is designed to allocate more influence to models that demonstrate superior performance on the validation set, enhancing the predictive accuracy and reliability of the ensemble’s overall output. This methodological choice is particularly effective in leveraging the distinct strengths of each classifier, thereby improving the ensemble’s capability to predict ventilator needs accurately.

The model training was conducted on 60% of our dataset, with the remainder split equally between validation and testing. The validation phase was crucial not only for hyperparameter tuning but also for determining the appropriate weights for each model within the ensemble. This setup ensures that our model is robust and well-adapted to the nuances of our dataset, minimizing the risk of overfitting.

Following the training and validation, we evaluated the ensemble model on the test set. This evaluation helped us assess the practical effectiveness of the ensemble in a controlled, yet realistic setting. We calculated key performance metrics such as accuracy, precision, sensitivity, specificity, and F1-score for each component model as well as for the ensemble as a whole. These metrics provided a comprehensive view of how each model contributes to the ensemble and how effectively the ensemble performs as a unit. The final prediction can be represented as: (2)

Where the parameter n represents the number of classifiers integrated within the ensemble. Each classifier, indexed by i, contributes to the final prediction with a speci fic weight w_i​, which is indicative of its importance or performance relative to the ensemble. The function f_i(x) denotes the prediction output by the i-th classifier for the input x. Additionally, the function sign() is utilized to convert the aggregated weighted sum of predictions into a definitive class label. This conversion is dependent on a threshold, 0.5, which is commonly used in binary classification scenarios to determine the class labels. This ensemble not only capitalizes on the individual strengths of each model but also significantly enhances our ability to make accurate predictions about ventilator needs, which is critical for efficient ICU management and patient care.

2.3.3. Comparison.

In this study, we implemented the Genetic Algorithm (GA) alongside Simulated Annealing (SA) to conduct an in-depth comparison of these hyperparameter optimization techniques. Our goal was to evaluate their efficacy in tuning the hyperparameters of our base classifiers—LDA, CatBoost, Artificial Neural Network (ANN), and XGBoost. We utilized GA, with hyperparameters defined in Table 3, to find optimal parameter sets for each base classifier. This approach allowed us to directly compare the effectiveness of GA and SA in enhancing the performance of these models by adjusting their hyperparameters to achieve the best possible outcomes. The below is pseudo code (Algorithm 2) of hyperparameter tuning using GA:

1. Algorithm 2. Pseudocode for Hyperparameter Selection using Genetic Algorithm (GA)
2. 1. Define the hyperparameter space for each model:
3.  ANN_hyperparameters = [number_of_layers, units_per_layer, activation_functions, optimizer, loss_function, batch_size, epochs]
4.  LDA_hyperparameters = [solver, number_of_components, store_covariance, tolerance, shrinkage]
5.  CatBoost_hyperparameters = [learning_rate, depth, l2_leaf_reg, iterations]
6.  XGBoost = [C, dual, loss, penalty, tol, learning_rate, max_depth, max_features, min_samples_leaf, min_samples_split, n_estimators, and subsample]
7. 2. For each model (ANN, LDA, CatBoost, XGBoost):
8.  - Generate an initial population of 20 individuals.
9.  - For each individual in the population:
10.   - Calculate the model’s performance (F1-score) on validation set.
11.  - For 250 generations:
12.   - Selection:
13.   - Select the top 10 performing individuals from the population to serve as parents for the next generation.
14.   - Crossover:
15.   - For each pair of parents, perform a two-point crossover to produce offspring. This involves selecting two random crossover points in the hyperparameter list and swapping the segments between these points to create new offspring.
16.   - Mutation:
17.   - Apply mutations with a 0.1 mutation rate to the offspring. Randomly alter one or more hyperparameters within their defined ranges. The standard mutation involves randomly selecting a hyperparameter and changing it to another valid value from its range.
18.   - Evaluate new generation:
19.   - Assess the fitness of each new individual in the population using F1-score.
20.   - Select the best individual:
21.   - After all generations have been processed, identify the individual with the highest fitness score as possessing the optimal set of hyperparameters.
22. 3. Output the best hyperparameter set:
23.  - Return the best hyperparameters and their associated performance metrics.

We assessed the performance of each base classifier when their hyperparameters were not tuned using SA and GA, as well as the performance of an ensemble model composed of these untuned classifiers. These initial results are presented in Table 4. Further, we analyzed the performance of each base classifier after their hyperparameters were tuned with both GA and SA. Additionally, we evaluated an ensemble model that was comprised of classifiers tuned with these methods. This allowed us to directly compare the impact of hyperparameter tuning on the performance of individual classifiers and their collective performance within an ensemble framework. The results of this analysis are detailed in Table 6, showcasing the effectiveness of each tuning method.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Evaluation metrics of ML and DL models before hyperparameter tuning using SA algorithm.

https://doi.org/10.1371/journal.pone.0311250.t004

To provide a broader perspective and benchmark our results against current automated techniques, we incorporated AutoML that named TPOT into our study [36]. This AutoML simplifies the selection of ML models and hyperparameter tuning but typically limits this tuning to a narrow range of parameters. We conducted a detailed comparison between the performance of the AutoML-selected model and our ensemble models, which were tuned using SA and GA, and also evaluated them in their untuned states. This analysis, displayed in Table 7, highlights the performance distinctions and demonstrates how our tuning approaches potentially offer more robust customization compared to AutoML’s more generic methodology.

While AutoML requires less technical knowledge and provides a streamlined, efficient process suitable for general applications, it lacks the granular control over hyperparameter settings that SA and GA provide. This control is crucial for addressing the specific needs of complex datasets, like ours, where flexibility in the tuning process is essential. The limited hyperparameter tuning range of AutoML compared to our extensive range may also impact the depth of model optimization achievable. Our study discusses these differences, emphasizing the trade-offs between the ease of AutoML and the detailed control offered by SA and GA, which can significantly influence model performance and transparency in research settings.

This comprehensive evaluation strategy not only highlighted the strengths and weaknesses of hyperparameter tuning methods like GA and SA but also underscored the potential of AutoML as a viable alternative in scenarios where manual tuning may be impractical. By comparing these methods across a range of performance metrics, we gained valuable insights into their applicability and effectiveness in a critical healthcare setting, providing essential guidance for future implementations of ML technologies where prediction accuracy and model reliability are crucial.

2.3.4. Evaluation.

We utilized five classification metrics to evaluate the predictive capabilities of the developed ensemble, ML and DL models: accuracy, precision, recall (sensitivity), specificity, and F1-score. To ensure the robustness and reliability of our evaluation, we implemented 10-fold cross-validation at each step of performance assessment. This methodological approach allows us to generate more stable and generalizable performance estimates by averaging results across different subsets of the data.

Accuracy, the ratio of the number of correct predictions to the total number of predictions [8]. This assessment utilizes True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN) to calculate the model’s accuracy, which is defined as follows: (3)

The formula for precision represents the proportion of positive predictions that were indeed correct.

(4)

The recall (sensitivity) formula is given below. It represents the percentage of actual positive cases that were correctly identified.

(5)

The specificity formula is provided below. It indicates the proportion of actual negative cases that were correctly classified.

(6)

The trade-off relationship between precision and recall can result in a model performing well on one metric but poorly on the other. The F1-score addresses this issue by considering both metrics simultaneously [8]. It is presented as below: (7)

The sensitivity index stands out as a key evaluation tool in our research due to its emphasis on the FN error, which is a critical factor given the importance of ensuring that patients who need ventilators are accurately identified. To avoid putting patients’ lives at risk due to inaccurate predictions, we aim to minimize the FN error and, consequently, maximize the sensitivity of our model.

Evaluation metrics of ML and DL models before and after using SA and GA algorithms are represented in Tables 4 and 6 respectively, all calculated using 10-fold cross-validation. As you can see in these tables, the best model based on its metrics is ensemble model tuned with SA.

So, based on our comprehensive evaluation of different models, we determined that ensemble with SA achieved the best performance in predicting ventilator requirements for patients in cardiac surgery ICU.