Structured data

Structured data from sources like Electronic Health Records (EHRs) and administrative claims have played a key role in enhancing machine learning’s predictive capabilities in healthcare. Due to its fixed structure, this data is relatively straightforward to analyze, making it suitable for rule-based methods, traditional statistical models, and machine learning techniques.

Traditional statistical models

Historically, hospital readmission prediction models have mainly used methods like logistic regression, which are simple, interpretable, and effective for binary classification [15,16]. One example is the HOSPITAL score, a clinical rule-based tool designed to predict 30-day readmissions using patient and administrative data collected prior to discharge. The score incorporates seven independent factors to identify high-risk patients [11]. Similarly, the LACE index, introduced in 2010 [10], is another clinical rule-based developed to quantify the risk of readmission or death following discharge. It achieved a C-statistic of 0.684 after external validation. The LACE index uses four variables: Length of Stay, Acuity of Admission, Patient Comorbidity, and Emergency Department use, to predict readmission risk. Despite its widespread use, the LACE index has been criticized for its moderate discriminative ability [17]. Like the LACE index, the HOSPITAL score also demonstrates only moderate discriminative power and does not fully utilize the rich patient data available in Electronic Health Records.

Another widely used predictive model based on logistic regression is the Patient at Risk of Readmission within 30 Days (PARR-30) model, developed by [12]. This model, which achieved a C-statistic of 0.70, utilizes a broader range of variables, including patient demographics, previous hospital admissions, and clinical data. However, it still heavily relies on administrative data, which limits its predictive accuracy. Despite these limitations, the PARR-30 model is actively used by the NHS for policy-making, demonstrating its practical value.

Logistic Regression (LR) continues to be one of the most commonly used methods for predicting hospital readmissions. A review by [18]found that 68% of the studies employed LR or other regression techniques as their primary approach. These models typically depend on manually derived features from patient data, such as demographics, comorbidities, and healthcare utilization patterns. However, such models often exhibit only moderate discriminative ability in predicting hospital readmissions [19].

In a 2019 study, [20] applied the HOSPITAL score and LACE index to a dataset containing medical claims data from over 100,000 patients in the Geisinger Health System. Both methods achieved an area under the receiver operating characteristic curve (AUC) score of 0.60, with the HOSPITAL score performing slightly better. In another study, [21], developed a logistic regression model using health insurance claims data from 138,222 hospitalized adults in Switzerland. This model incorporated variables such as pharmacy-based cost groups, emergency visits, and outpatient costs. This model also achieved an AUC score of 0.60, indicating limited discriminative ability, only slightly better than chance (0.5). hese findings further suggest that predictive models developed using Electronic Health/Medical Records data tend to offer better performance than those based on insurance claims or other administrative data [20,22].

Machine learning models

On the other hand, Machine Learning (ML) models have shown promise in improving the accuracy of readmission predictions. The types of machine learning (supervised, unsupervised, and reinforcement learning), along with their various applicable algorithms, are discussed and presented by Pandey et al. [23]. In a review of 9 studies, [16] evaluated the performance of logistic regression and ML models and found that ML models generally outperformed logistic regression in predicting 30-day all-cause hospital readmissions, with deep learning models performing the best.However, they observed that tree-based and kernel-based methods did not offer significant improvements. The authors concluded that while ML models outperform traditional methods in predicting hospital readmissions, challenges such as model interpretability and integration into clinical workflows remain unresolved.

Conversely, Min et al. [20] compared traditional ML models (e.g., logistic regression, random forest) with deep learning techniques (e.g., CNN, RNN) for predicting readmission risk, using both knowledge-driven and data-driven features. Despite the complexity of deep learning models, the Gradient Boosting Decision Tree (GBDT) achieved the highest AUC (0.653), while deep learning methods did not show significant improvement. This raises questions about the added value of deep learning in certain clinical prediction tasks, especially when simpler models provide similar performance.

A range of studies have applied machine learning models to structured data but very few have achieved consistently high metrics. Typically, when a model achieves high recall, t is often accompanied by low precision, and vice versa. For instance, Lo et al. [24] developed predictive models for 14-day unplanned readmissions, and among these, CatBoost delivered the best performance, achieving an AUROC of 0.9909 after feature selection.However, the model also exhibited a moderate sensitivity of 0.56. While not necessarily poor, this suggests a potential class imbalance, with the Receiver Operating Characteristic (ROC) curve potentially biased towards the positive class. A similar issue was observed in the study by [25], which aimed to improve early Intensive Care Unit (ICU) readmission prediction. Using Extreme Gradient Boosting (XGBoost) and structured data from the MIMIC-III dataset, they achieved an AUROC of 0.92 ± 0.03, significantly outperforming previous state-of-the-art models (AUROC ranging from 0.66 to 0.78). However, their model reported a specificity of 0.99 and a recall of only 0.40, again highlighting the challenge of class imbalance.

To address the issue of class imbalance, Yu et al. [26] employed a modified weight-boosting algorithm combined with a stacking method to predict hospital readmissions. The authors used advanced feature engineering techniques to manage the high dimensionality and sparsity of medical codes. Their model was trained and validated on a large nationwide healthcare dataset from China, comprising inpatient administrative data, and achieved a high recall of 0.891, outperforming benchmark models. Despite these promising results, the proposed approach may face practical limitations due to its high computational demands, particularly when applied to large datasets. Furthermore, successful implementation requires careful feature engineering and parameter tuning, demanding a high level of expertise in machine learning.

Magboo et al. [27] compared the performance of three models: Random Forest, Adaptive Boosting, and K-Nearest Neighbors for predicting hospital readmissions among diabetic patients. The study incorporated Local Interpretable Model-agnostic Explanations (LIME) method to provide visual interpretations of the models’ predictions. All three models achieved high accuracy, exceeding 92%, and the LIME outputs offered clinically meaningful insights into the factors driving predictions. However, the study was limited in scope, focusing exclusively on diabetic patients and relying on simulated rather than real-world hospital data, which may limit its generalisability and practical applicability.

Liu et al. [28] introduced an Optimal Variational Bayesian Logistic Regression (OVBLR) model enhanced with a Salient Feature Estimation (SFE) strategy, referred to as OVBLR-SFE, to address the limitations in existing predictive methods by prioritizing accuracy over interpretability.he model demonstrated strong performance and stability across four benchmark medical datasets from the UCI repository, as well as in a real-world application predicting intensive care unit (ICU) readmissions for liver transplant patients. It achieved an average classification accuracy of 90.10% on the UCI datasets and 88.11% on the ICU readmission task. Despite these promising results, the model encountered scalability issues due to its high computational requirements when applied to larger datasets.

Recent review studies by [29] found that, although traditional statistical models are still widely used, machine learning (ML) and deep learning (DL) techniques have demonstrated promising performance in predictive healthcare tasks.

Deep learning

Jamei et al. [30] investigated the use of artificial neural networks (ANNs) to predict 30-day hospital readmissions, utilizing data from over 300,000 hospital stays in California. The ANN was trained on features derived from electronic health records (EHRs) and social determinants of health, and it outperformed traditional models, including the LACE index. Specifically, it achieved a precision of 0.24 in identifying high-risk patients, representing a 20% improvement over LACE’s precision of 0.20. However, the model faced limitations in generalizability to other patient populations and encountered implementation challenges due to its lack of interpretability.

Barbieri et al. [31] compared several deep learning architectures including attention mechanisms, recurrent layers, neural ordinary differential equations (ODEs), and medical concept embeddings using the MIMIC-III dataset [32]. Among the models evaluated, recurrent neural networks (RNNs) with time-dynamic code embeddings generated by neural ODEs achieved the highest average precision of 0.331, with an AUROC of 0.739 and an F1-score of 0.372. The study found that attention-based models offer improved interpretability with only a minimal reduction in accuracy. Interpretation of the attention model indicated that patients at higher risk of readmission often presented with infectious complications, chronic or progressive conditions, or required non-standard medical care. However, the study did not incorporate unstructured clinical notes, limiting the scope of the analysis. Machine Learning models have come a long way in using structured data for hospital readmission prediction, however, structured data alone is not a complete representation of a patient’s condition.

Unstructured data

The evolution of predictive models has increasingly highlighted the value of clinical notes, such as radiology reports and discharge summaries [29]. Natural Language Processing (NLP) techniques are capable of analyzing clinical notes, admission and discharge summaries, and other text-based data to identify risk factors and patterns that may not be captured by structured data alone. Sheikhalishahi et al. [33] reviewed the application of various NLP methods in clinical research, emphasizing the need for NLP to evolve beyond text extraction to an understanding of clinical concepts. Their study identified gaps in the use of NLP, particularly regarding the limitations of text extraction, entity recognition in isolation, and reliance on shallow classification methods.

Wu et al. [34] reviewed the growing use of deep learning (DL) in clinical Natural Language Processing (NLP), noting that studies in this area have been increasing rapidly, with Recurrent Neural Networks (RNNs) and word2vec embeddings being among the most commonly employed methods. The primary tasks addressed include text classification, named entity recognition, and relation extraction. However, they also observed that deep learning models do not always outperform simpler models, as demonstrated by Christodoulou et al. [35]. While RNNs are effective, they struggle to capture long-range dependencies in sequences [13,36], particularly with longer sequences. As an alternative, the transformer architecture, introduced by [37], relies entirely on attention mechanisms and eschews recurrent layers. By using multi-headed self-attention, transformers can process all positions in the input and output sequences in parallel, offering improved computational efficiency.

Large language models

Large Language Models (LLMs) have revolutionized NLP by significantly enhancing the understanding and generation of textual data. Trained on vast amounts of text, these models are capable of performing a wide range of downstream tasks. Based on the transformers architecture, Bidirectional Encoder Representations from Transformers (BERT), developed by Delin J et al. [38], was introduced to capture bidirectional context, which is essential for understanding the relationships between words in a sentence.

Despite the advancements of Large Language Models (LLMs) in processing large text corpora, domain-specific models are still more beneficial in many contexts. Alsentzer et al. [39] applied BERT to clinical notes and released publically available clinically trained BERT models. Similarly, Huang et al. [40] introduced ClinicalBERT, adapted for readmission prediction and pre-trained on longer sequence lengths. ClinicalBERT demonstrated superior predictive performance compared to baseline models, including Bidirectional Long Short-Term Memory (Bi-LSTM) and BERT, when using notes from early patient admissions (AUROC: 0.674) and discharge summaries (AUROC: 0.714). Nazyrova et al. [41] further demonstrated the potential of domain-specific LLMs to enhance predictive accuracy for hospital readmissions. Their study explored BERT variants such as BioBERT [42], SciBERT [43] and ClinicalBERT [40] in predicting 30-day readmissions for elderly patients. Their findings revealed that domain-specific models, particularly SciBERT, outperformed general LLMs in medical contexts. Using the MIMIC-IV dataset, SciBERT improved the AUROC for readmission prediction from 0.714 to 0.735. Although LLMs excel at processing and understanding clinical notes, their integration with structured data remains a largely unexplored area.

Combining structured and unstructured data

While both structured and unstructured data offer distinct advantages in clinical predictive modeling, emerging evidence suggests that the most effective approach combines both data types. Studies have consistently demonstrated that models integrating both structured (e.g., lab results, medications) and unstructured (e.g., clinical notes, imaging reports) data from electronic health records (EHRs) outperform those relying on a single data source. This approach significantly enhances the accuracy of predicting various clinical outcomes, ultimately leading to more informed and personalized patient care [44]. For instance, Zang et al. [45] proposed neural networks that combine sequential unstructured clinical notes with structured data for predicting multiple outcomes, including 30-day hospital readmissions. Their fusion models, which utilize document embeddings alongside convolutional neural networks (CNN) or long short-term memory (LSTM) networks, demonstrated improved prediction accuracy over baseline models, with an AUROC of 0.674. However, upon closer inspection, the performance of these models in predicting hospital readmissions was less impressive compared to other tasks, such as in-hospital mortality prediction. This highlights the inherent complexity of modeling readmission risk.

Lin et al. [46] employed a Recurrent Neural Network (RNN) architecture with LSTM layers to capture temporal dependencies and fluctuations in patient data. Their LSTM-based model outperformed traditional machine learning and convolutional neural network (CNN) models in predicting ICU readmissions, achieving a sensitivity of 0.742 and an AUROC of 0.791. The model effectively captured high volatility and unstable physiological states—key indicators of readmission risk—and identified key predictive features, including glucose levels, heart rate, body temperature, Glasgow Coma Scale, and oxygen saturation. Similarly, Rajkomar et al. [47] developed a deep learning framework using patients’ complete raw EHR data, including free-text clinical notes, from the University of California, San Francisco (UCSF) and the University of Chicago Medicine (UCM). Their model, trained on data from 2012–2016 (UCSF) and 2009–2016 (UCM), predicted multiple outcomes, including 30-day unplanned readmissions, with an AUROC of 0.75–0.76.

Johnson et al. [48] investigated the use of unstructured clinical notes for predictive modelling using deep learning architectures, including RNNs, attention-based time-aware neural networks (TANNs), and boosted time-based decision stumps. Their approach demonstrated how incorporating clinical text can enhance model explainability by highlighting note features relevant to predictions. However, the method had limitations in transferability and required substantial computational resources.

A study conducted in Alberta, Canada [49] compared the performance of the traditional LACE model with a Gradient Boosting Machine (GBM) model that incorporated both manually derived features from population-level linked administrative hospital data and machine-learned features. These features included longitudinal patient health records encoded using Word2Vec, a NLP technique. The target outcome was all-cause 30-day hospital readmission. The GBM model significantly outperformed the LACE model, achieving an AUC of 0.83 compared to 0.66.

Summary of literature review

The literature reveals a clear transition from traditional statistical models to advanced machine learning (ML) and natural language processing (NLP) techniques for hospital readmission prediction. Transformer-based large language models (LLMs), such as BERT, have demonstrated improved predictive performance by effectively leveraging unstructured clinical notes. However, the integration of LLMs with structured data remains underexplored. This study proposes a framework that combines structured electronic health record (EHR) data with unstructured clinical notes, using embeddings from a fine-tuned ClinicalT5 model for both clinical narratives and diagnostic information. This approach aims to investigate the added value of integrating LLM-derived representations with structured features to enhance 30-day hospital readmission prediction. A summary of key studies, including model descriptions, performance metrics, and their respective strengths and limitations, is presented in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Authors, Description & Metric Scores.

https://doi.org/10.1371/journal.pone.0328848.t001

Proposed framework

We propose a method that constructs a comprehensive patient representation by combining data modalities. The overall framework of the proposed approach is illustrated in Fig 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Proposed framework.

https://doi.org/10.1371/journal.pone.0328848.g002

The proposed framework comprises two main components: (1) LLM Embedding Extraction using Clinical-T5, a large language model trained on MIMIC clinical notes [14], and (2) Integration with Structured Data. Text representations are derived from the decoder’s hidden states of Clinical-T5. Structured data undergoes preprocessing, where numerical features are standardized and categorical features are encoded. The resulting embeddings from both modalities are concatenated to form a unified patient representation, which is then input into an ensemble classifier to predict 30-day hospital readmissions.

LLM embeddings extraction

To perform this task, we selected ClinicalT5, a variant of the T5 model fine-tuned on clinical notes. ClinicalT5 builds upon the original Text-to-Text Transfer Transformer (T5) architecture, an encoder-decoder framework designed for text generation tasks. It retains the core components of the original Transformer model [37], with modifications including changes to the LayerNorm bias, layer normalization, and positional embedding scheme. For detailed architectural insights, readers are referred to the original T5 paper [50].

The version of Clinical T5 used in this study is Clinical-T5-Scratch, which follows the same architecture as T5-Base with 220 million parameters. Clinical-T5-Scratch builds on T5-Base by undergoing additional training on MIMIC-III [32] and MIMIC-IV [51] data with random initialized weights.

To obtain the embeddings, the clinical texts are split into chunks and tokenized to generate input tokens: , X , A and fed to ClinicalT5 in batches. The model processes the input and produces a sequence of hidden states, , corresponding to each token.

Where,

The text representation is obtained by averaging the decoder hidden states through mean pooling across the decoder layers, resulting in a single embedding for the entire input sequence.

Where,

This process was performed in batches of 32 input sequences, with the resulting embeddings stored in a list for future use. These embeddings were then concatenated with pre-processed numerical and categorical data, along with the readmission outcome, to create a comprehensive representation of each patient’s condition during their admission or hospital stay.

Data source and selection

This research uses the MIMIC-IV dataset, derived from the electronic health records (EHRs) of Beth Israel Deaconess Medical Center (BIDMC) in Boston, MA, USA [51]. Access was granted through PhysioNet [52] upon completion of CITI training.The dataset includes two primary modules: hosp (our focus) and icu, covering both hospital and ICU stays. The hosp module mainly contains data from hospital admissions, with some from the emergency department. Clinical notes, including 331,794 discharge summaries and 2,321,355 radiology reports, were obtained from MIMIC-IV-Note [51]. For this study, we extracted the admissions, patients, DRG codes, and transfers tables from hosp, the radiology notes table from MIMIC-IV-Note, and the vital signs table from the Emergency module. These tables provide demographic, admission, vital signs, diagnostic, and radiological data essential for our research. The dataset has been de-identified to ensure that no personally identifiable information is included. The dataset was accessed on 04/07/2024. The final dataset integrated the extracted tables into a single comprehensive table, as shown in Fig 3, which represents unique hospital stays. Prior to analysis, the data underwent a thorough cleaning and transformation process Fig 4 to ensure its quality and appropriateness for the study.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. MIMIC-IV database schema and No of records.

https://doi.org/10.1371/journal.pone.0328848.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Data pre-processing framework.

https://doi.org/10.1371/journal.pone.0328848.g004

Handling missing value

The missing values in the dataset were addressed on a case-by-case basis using a function, as illustrated in the following Table 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Handling missing value.

https://doi.org/10.1371/journal.pone.0328848.t002

Feature engineering

To enhance the quality of the dataset, several feature engineering procedures were undertaken. A new feature, termed duration, was created to quantify the length of patient hospitalization. The resulting values were rounded up to whole days, as length of stay is usually calculated in whole days. The admission type feature contained multiple synonyms. For example, “EW EMER.”, “URGENT”, and “DIRECT EMER.” can all be represented as “Emergency”. Furthermore, the race column had 33 values which were consolidated into 7 main values: “BLACK”, “WHITE”, “ASIAN”, “HISPANIC”, “HAWAIIAN”, “AMERINDIAN” and “OTHER” for unspecified races. Additionally, data types were appropriately adjusted for the numerical columns from object to “int” or “float” where relevant. These refinements were implemented to improve data uniformity and facilitate more robust subsequent analyses.

Dropping features

1. The following columns, which did not provide any further benefit for the readmission task, were dropped:
   • Admittime
   • Dischtime
   • edregtime
   • edouttime
   • Next admittime
   • subject id
   • admittime c
2. The columns storing “Next admission type” and “Days until next admission” were dropped, as they are strong predictors of a future readmission, which could bias the model’s performance.

Data subsampling

The dataset exhibited class imbalance, with 268,105 instances of the negative class (class “0”) and 48,743 instances of the positive class (class “1”), as shown in Fig 5. This imbalance could lead to a biased model that favours the negative class. To address this, the negative class was undersampled to match the size of the positive class, resulting in a more balanced dataset, as shown in Fig 6. Undersampling was selected for its simplicity and to prevent the generation of artificial data.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Imbalance datasets (counts of patients).

https://doi.org/10.1371/journal.pone.0328848.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Balance dataset and distribution of readmission outcome.

https://doi.org/10.1371/journal.pone.0328848.g006

Feature analysis

The subsampled training set contains 79,731 rows and 21 features, including 10 numerical, 8 categorical, and 2 textual features.

Numerical features observations.

The distribution of numerical features is presented in Fig 7.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Distribution of numerical features.

https://doi.org/10.1371/journal.pone.0328848.g007

• Age: The distribution is approximately normal, with a peak around 60–70 years, indicating an older population.
• Heart Rate: The distribution is close to normal, with most heart rates ranging from 70 to 100 bpm, which is typical for adults.
• Systolic Blood Pressure: The distribution centers around 120–150 mmHg, which falls within the normal adult range.
• Length of Stay: The distribution is strongly right-skewed, indicating that most patients had relatively short stays.

Categorical features.

The distribution of categorical features is shown in Fig 8 (correlation matrix), Fig 9 (demographics), and Fig 10 (socioeconomic features). The key observations for these categorical features are as follows:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Correlation matrix of features.

https://doi.org/10.1371/journal.pone.0328848.g008

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Distribution of demographic features.

https://doi.org/10.1371/journal.pone.0328848.g009

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Distribution of socio economic features.

https://doi.org/10.1371/journal.pone.0328848.g010

• Admission Location: Patients admitted from the emergency room are more likely to have another emergency readmission.
• Gender: Slightly more females are associated with negative outcomes, while slightly more males are linked to positive outcomes.
• Race: The dataset predominantly consists of white individuals, with a balanced distribution across outcomes but a slightly higher count in the negative class. Black individuals show a balanced distribution, and other races have smaller counts with a fairly balanced distribution.
• Insurance Type: The ’Other’ insurance type has a higher count in the negative class. Medicare has an almost even distribution across both classes, while Medicaid has a smaller overall count, with a higher proportion in the positive class.
• Marital Status: More married individuals are associated with the negative outcome.

Textual data.

The text length distribution is left-skewed, with most entries ranging from 1300 to 1500 characters, as shown in Fig 11.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. Combined text length distribution.

https://doi.org/10.1371/journal.pone.0328848.g011

Proposed method

Hospital readmission prediction was performed using tabular data and text embeddings extracted via ClinicalT5.

Tabular data + ClinicalT5 text embedding + classifier.

In this study, we developed a model that combines structured patient data with unstructured clinical notes. We fine-tuned ClinicalT5 for 5 epochs using the Hugging Face Trainer API [62]. Further training was not conducted due to computational constraints. The training parameters for ClinicalT5 are provided in the Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. ClinicalT5 parameters.

https://doi.org/10.1371/journal.pone.0328848.t003

Neural network.

We constructed a feed-forward neural network comprising an input layer, three hidden layers, and an output layer. The network’s architecture and parameters are shown in Table 4 and Table 5, respectively. The training process was managed by a train_model method, which oversees the entire training pipeline.

Initialization: The model is set to training mode using self.train(). A history dictionary is initialized to store loss values, and variables for early stopping are prepared.

Epoch Loop: The model is trained for a specified number of epochs, iterating over the dataset multiple times.

• For each epoch, the training dataset is divided into batches. For each batch:
  1. – A forward pass is performed, where the input data is passed through the model to generate predictions.
  2. – The loss is computed by comparing the predictions to the true labels using a loss function (criterion).
  3. – The gradients of the loss with respect to the model’s weights are calculated via backpropagation (loss.backward()), and the optimizer updates the model’s weights accordingly
• The average training loss across all batches is recorded at the end of each epoch.

Validation: After each epoch, the model is evaluated on the validation dataset, and the corresponding loss is computed.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Neural network model description.

https://doi.org/10.1371/journal.pone.0328848.t004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Model training parameters.

https://doi.org/10.1371/journal.pone.0328848.t005

Comparison method: PubMedBERT, a Large Language Model (LLM), was fine-tuned exclusively on free-text data for the task of predicting hospital readmission, as detailed in the Comparison Method section.

Comparison method

In this study, we fine-tuned the PubMedBERT transformer model [53] on the dataset’s text and diagnosis description variables to enable a comparative analysis. The dataset used in this experiment differs slightly from others, primarily due to its larger number of text entries, which allowed for more extensive model training. However, the dataset underwent pre-processing and subsampling following the same procedures described in previous sections. The specific variant of PubMedBERT used in this experiment is the PubMedBERTMNLI-MedNLI model, introduced by [54]. This model was initially fine-tuned on the Multi-Genre Natural Language Inference (MNLI) dataset [55], and subsequently further fine-tuned on the MedNLI dataset [56].

Fine-tuning.

The model was fine-tuned for the hospital readmission task using the radiology notes and diagnosis texts. Various hyperparameters were explored, and we ultimately selected a learning rate of 5e⁻⁵, a batch size of 32, and 5 epochs. Validation was performed every 3,000 steps, with results recorded and model checkpoints saved. At the conclusion of training, the best model was chosen based on the lowest training loss.

The training parameters for PubMedBERT are presented in Table 6.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. PubMedBERT training parameters.

https://doi.org/10.1371/journal.pone.0328848.t006

Evaluation matrix

To assess the performance of the models across the different experiments, the following evaluation metrics were employed:

Accuracy: is defined as the proportion of correctly predicted outcomes relative to the total number of predictions, and is calculated as follows:

Recall: Also referred to as sensitivity or the true positive rate, recall assesses the model’s ability to correctly identify actual positive cases. It is calculated as:

Precision: This metric evaluates how accurately the model predicts the positive class among all instances classified as positive. It focuses on minimizing false positives. Precision is calculated as:

Specificity: Also referred to as the true negative rate, specificity measures how effectively the model identifies negative cases. It is calculated as:

Area Under the Receiver Operating Characteristic Curve (AUROC): The AUROC curve reflects the likelihood that a randomly chosen positive instance is ranked higher than a negative one. It provides an aggregate measure of a model’s ability to distinguish between positive and negative classes across all classification thresholds. A higher AUROC indicates better model performance [59].

F1 Score: The F1 Score is the harmonic mean of Precision and Recall, offering a balanced measure between the two metrics. It is especially useful in cases with imbalanced class distributions, where both false positives and false negatives are significant. The F1 Score is calculated as:

Matthews Correlation Coefficient (MCC): The MCC is a correlation coefficient between actual and predicted binary classifications, considering true positives, true negatives, false positives, and false negatives. It is a balanced metric, particularly useful for imbalanced datasets, as it offers a more informative measure of model performance than accuracy. The MCC is calculated as: [60].

PubMedBERT with text only

The results of applying PubMedBERT to the clinical notes, including the AUROC curve (left) and Precision-Recall curve (right), are shown in Fig 12. With an AUROC score of 0.64, the model performs only marginally better than random chance (0.5). As seen in Fig 12 (left), the AUROC curve demonstrates the model’s performance in comparison to random guessing. Although the results are moderate, there is significant potential for improvement.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 12. PubMedBERT AUROC Curve (left) and PubMedBERT Precision-Recall Curve (right).

https://doi.org/10.1371/journal.pone.0328848.g012

The confusion matrix in Fig 13 visually confirms the model’s imbalanced performance, showing a high number of true positives, consistent with the relatively high recall observed.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 13. PubMedBERT confusion matrix.

https://doi.org/10.1371/journal.pone.0328848.g013

Neural network.

The Neural Network’s AUROC score, shown in Fig 14 (left) demonstrates moderate discriminative ability, slightly outperforming PubMedBERT. Similarly, Fig 14 (right) visualizes the trade-off between precision and recall at various thresholds.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 14. Neural network performance evaluation curves.

https://doi.org/10.1371/journal.pone.0328848.g014

The confusion matrix in Fig 15 confirms the model’s imbalanced performance, with a high number of true positives and false positives.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 15. Neural network confusion matrix.

https://doi.org/10.1371/journal.pone.0328848.g015

Extreme Gradient Boosting (XGBoost) classifier.

The performance of the XGBoost Classifier is shown in Fig 16, with the AUROC curve (left) and Precision-Recall curve (right). This model demonstrates improved accuracy compared to the Neural Network Classifier and PubMedBERT. The recall and precision are closely aligned, indicating a more balanced approach to predicting both positive and negative cases.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 16. Extreme gradient boosting classifier performance evaluation curves.

https://doi.org/10.1371/journal.pone.0328848.g016

The confusion matrix in Fig 17 shows an improved balance across the quadrants, with the model successfully identifying more true negatives and true positives.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 17. Extreme gradient boosting classifier confusion matrix.

https://doi.org/10.1371/journal.pone.0328848.g017

Light Gradient Boosting (LGBM) classifier.

The performance of the LGBM Classifier is shown in Fig 18, with the AUROC curve (left) and Precision-Recall curve (right). The AUROC curve in Fig 18 (left) shifts more to the left, indicating a better balance between sensitivity and specificity. The precision-recall curve in Fig 18 (right) demonstrates a higher optimal threshold for the precision-recall trade-off.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 18. Light gradient boosting classifier performance evaluation curves.

https://doi.org/10.1371/journal.pone.0328848.g018

The confusion matrix in Fig 19 shows a more balanced distribution across the quadrants, with the model successfully identifying a greater number of true negatives and true positives.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 19. Light gradient boosting confusion matrix.

https://doi.org/10.1371/journal.pone.0328848.g019

Voting classifier.

The performance of the Voting Classifier is shown in Fig 20, with the AUROC curve (left) and Precision-Recall curve (right). The AUROC curve in Fig 20 (left) indicates a better optimal threshold, with an AUROC score of 0.68, matching the LGBM Classifier and reflecting a moderate ability to distinguish between classes. Fig 20 (right) shows a lower optimal threshold for the precision-recall trade-off when compared to the LGBM Classifier.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 20. Voting classifier performance evaluation curves.

https://doi.org/10.1371/journal.pone.0328848.g020

The confusion matrix in Fig 21 shows a higher number of true positives, consistent with the high recall recorded, along with an increased number of false positives.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 21. Voting classifier confusion matrix.

https://doi.org/10.1371/journal.pone.0328848.g021

To summarize

PubMedBERT and ClinicalT5+NeuralNetwork prioritize recall but are affected by a high number of false positives, which leads to lower precision and specificity. Consequently, these models also exhibit the lowest accuracy among the five models considered.

ClinicalT5+LGBM and ClinicalT5+VotingClassifier outperform PubMedBERT across most metrics, particularly in terms of precision-recall balance and AUROC. This makes them more reliable for predicting hospital readmissions.

Comparison with other studies

The results of this study are consistent with broader trends in hospital readmission prediction research, where models that incorporate structured data tend to outperform those that rely exclusively on unstructured text. For instance, studies by [45] and [47] demonstrated that integrating electronic health record (EHR) data with clinical notes significantly enhances model performance in predicting patient outcomes, including readmissions. Notably, the performance of our model exceeds that of the study by [45].

Key differences and improvements

Text Segmentation: In this study, clinical notes were processed using ClinicalT5 in manageable chunks of 200 words, allowing the model to better retain the richness of the clinical text without being overwhelmed.

LLM Usage: Unlike the approach taken by [47], which utilized recurrent models that often struggle with long clinical notes, this study integrated structured data with a Large Language Model (LLM). While this method offers improved performance, it comes with a trade-off: LLMs are computationally intensive, which may limit their practicality in smaller healthcare settings with limited resources.

Strengths and drawbacks