Dataset

The Cincinnati Children’s Hospital Institutional Review Board approved this retrospective study (STUDY# 2020−0942). The study analyzed existing pediatric electronic health records (EHRs) from 1.3 million patients collected between January 1, 2009, and March 31, 2022. The data were extracted and processed by authorized staff at Cincinnati Children’s Hospital Medical Center (CCHMC) to create a static database, which was subsequently transferred to Oak Ridge National Laboratory for secure hosting. The Institutional Review Board waived the requirement to obtain informed consent from adult participants, parental permission from parents or guardians of child participants, and assent from children. The IRB also granted a waiver from the requirement to obtain authorization for the use and/or disclosure of protected health information (PHI).

In this study, we used a retrospective case-control study design, where each anxiety case (anxiety group) was matched to a control (non-anxiety group) by age at the time of the case’s diagnosis and sex assigned at birth. Anxiety cases were identified using ICD codes described in S1 Table. The date of anxiety onset was determined using the first instance of an anxiety ICD code. This resulted in the selection of 53,728 anxiety patients diagnosed between the ages of 2 and 21 between 2009–2022. We included patients aged 2–21 to encompass the full developmental span from early childhood through young adulthood, consistent with recent psychiatric epidemiology studies that examine youth mental health across broad developmental windows [3,36]. At least one visit in the 18 months prior to the diagnosis date was required for inclusion in the anxiety or non-anxiety group. For the non-anxiety group selection, the patient was required to be the same sex assigned at birth as the case, born within 30 days of the case, have not developed anxiety at the time of the case’s anxiety diagnosis record, and have had at least one encounter in the EHRs in the 18 months preceding the case’s anxiety diagnosis date. The matched case-control dataset was then stratified by single-year age groups from ages 2–21 based on age of diagnosis of the case. The descriptive statistics for each age group are shown in Table 1.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Statistics of the Structured EHR Data.

https://doi.org/10.1371/journal.pone.0324673.t001

Data preprocessing and feature engineering

Comprehensive patient histories were extracted from the CCHMC EHR. We then followed a series of preprocessing and feature engineering steps to create the final analytic tables to train the ML and DL models: 1) We extracted the static features that describe patient characteristics that do not change over time. 2) We collapsed time-dependent features into 30-day time-bins to capture the relevant EHR events during each time-bin. 3) We created an analytic file for the ML analyses that only included 30-day information prior to a recent 30-day blackout period to the time of diagnosis. 4) We created a time-dependent dataset for the DL analyses that included 30-day time-bins from birth to the time of the case’s diagnosis with a 30-day blackout period. 5) We appended the ABMH data to the analytic files created in steps three and four. 6) We split the analytic files into age-specific datasets for analysis. 7) We created train/test splits using patient ID.

We selected 30-day time bins to reflect clinically meaningful intervals commonly used in pediatric monitoring (e.g., follow-up visits, readmission risk). A 30-day blackout period prior to diagnosis was applied to reduce information leakage from diagnostic encounters themselves, ensuring that models learned predictive rather than diagnostic signals. Temporally bounded windows are widely used in predictive modeling with EHR data [37–39]. Our approach extends this precedent by discretizing the full patient history into 30-day intervals, enabling deep learning models to capture longitudinal developmental patterns while maintaining clinical interpretability.

Structured EHR data

The structured data includes information from two types of features: time-dependent (features that change over time) and static (features that do not change over time). The time-dependent features consist of diagnosis codes, procedure codes, medication codes, visit metadata (encounter type, provider type, place of service, care site, hospitalization), and measures (BMI, height, weight, blood pressure, heart rate). The static features include information set at birth, such as allergies, family mental health history, and patient demographics. The categorization of allergies (food, medications, and environment) and family histories (psychiatric disorders, substance abuse, sexual/verbal abuse, autism, attention-deficit/hyperactivity disorder, and developmental disorder). For the structured EHR data, we use frequency encoding and replace missing values with −1. Detailed information on time-dependent and static features is discussed in S1 Appendix.

Generation of time-dependent features

Fig 1 illustrates the utilization of time-dependent features. For these experiments, we used 30-day time-bins. The 30-day time-bin consolidates all EHR and ABMH data within each time-bin during the 30-day period. Time-bins extend from the time of birth to the time of the case’s diagnosis and include relevant EHR events that occurred during each 30-day window. Using the 30-day time-bins results in approximately 12 time-bins per year. We utilize this sequence of time-bins as the input feature for the DL-based models, while for the ML-based models, we use the final time-bin (x_T) as the input feature. Table 1 summarizes the feature size and the number of time-bins of each age group.

Area-based Measures of Health (ABMH) data

Measuring ABMH across time using EHR data is a challenging task. This information is not consistently stored in EHR records and must be reconstructed using residential history information for each patient. In order to do this effectively, each residential address must be geocoded, assigned a relevant time window, and spatially joined to external sources of data that describe a patient’s community environment. We use the DeGAUSS package [40] to define a community environment at a single time point and then develop ABMH trajectories using a patient’s residential history.

We sought to capture the dynamic ABMH a patient might be exposed to throughout their life course. Therefore, ABMH features were considered to be time-dependent. This was done by constructing patient residential histories from residential address information captured in the CCHMC enterprise data warehouse (EDW). We extracted all known residential locations for each patient and assigned a start and stop date for each unique location. The date of visit corresponding to the residential location was used to construct the residential history of the patient from birth to the time of anxiety diagnosis. This required several assumptions: (1) the patient’s first observed address in the EHR record was also their address at the time of birth, and (2) the patient resided at their last known or current location for the full length of time between visits. DeGauss was used to geocode and link each patient’s residential location to a corresponding United States Census Tract (CT) at a single point in time. We accounted for changes in US Census Tract boundaries over time to ensure a correct characterization of the community a patient resided in at each point in time. The CT level measures used to characterize a patient’s ABMH over time can be found in S2 Table. Using a modified version of the time-bin code, we created environment measures for every 30-day time-bin used in the EHR data construction. If an individual moved during a 30-day time-bin, we created a weighted average of each environmental measure, with weights determined by the proportion of time spent at each location during the bin to select the most common address during the time. In addition to the structured features (EHR), 17 features were added to each time-bin as ABMH features (EHR+ABMH) to train the model.

Classification models

ML models were implemented as cross-sectional classifiers using 30-day feature windows to evaluate short-term patterns preceding diagnosis, while DL models were designed as sequence models that utilized full longitudinal patient histories to capture temporal dependencies. These approaches were intended to provide complementary perspectives on predictive modeling rather than to serve as direct comparators.

Machine learning-based models

For each age group, we apply ML models such as Logistic Regression (LR), Decision Tree (DT), Random Forest (RL) [32], K-Nearest Neighbor (KNN), and Extreme Gradient Boosting (XGBoost) [33]. All models are widely used in classification tasks in various domains such as image recognition [41,42], natural language processing [43,44], and recommender systems [45,46]. For all ML-based models, we use the last 30-day time-bin, x_T, from Fig 1, as an input feature to prevent the feature size from becoming too large. The increment of the feature size can cause a curse of dimensionality [47], which results in performance degradation by losing the meaning of the distances of the data points. Detailed information on each model is introduced in S2 Appendix.

Deep learning-based models

For the DL-based models, we apply Long Short-Term Memory (LSTM) [35], Gated Recurrent Unit (GRU) [34], Reverse Time Attention (RETAIN) [18], and Diagnosis Prediction Model (Dipole) [19]. Unlike ML-based models, all time-bins ( from the Fig 1) are used as input feature matrix. We use Recurrent Neural Network (RNN)-based models that are effective, especially with sequential information, to handle such a long sequence of encounters. Table 1 shows the statistics of the features and time-bins used for each age group. Note that the feature size in Table 1 is the feature size of EHR. Detailed information on each model is introduced in S2 Appendix.

Experimental setting

Model training and hyperparameter tuning are conducted to generate accurate classifications of patients at risk of developing anxiety. For hyperparameter tuning, we split the dataset into 64% training, 16% validation, and 20% test sets. All models use the same setting and are compared with the same split across the age group. All ML models are implemented using the scikit-learn package [48], except XGBoost, which has its dedicated library [33]. The implementation of LSTM and GRU uses the PyTorch library [49]. We use the source code provided by the papers for RETAIN (implementation available at PyHealth RETAIN) and Dipole (implementation available at Dipole GitHub repository). We optimize the ML-based models through a grid search for the most effective hyperparameters using the scikit-learn package [48]. For the DL-based models, we perform fine-tuning using the Adam optimizer with a grid search. S3 Appendix summarizes the hyperparameters used for tuning and shows the selected hyperparameters for each model. Hyperparameters are selected using the EHR features, and the same hyperparameter settings are used throughout the experiment.

This study was reported in accordance with the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) statement to ensure transparent and complete reporting of predictive modeling studies [50].

Evaluation metrics

We present six metrics for the pediatric anxiety classification task in this study: (1) accuracy, (2) area under the receiver operating characteristics curve (AUROC), (3) area under the precision and recall curve (AUPRC), (4) positive predictive values (PPV), (5) negative predictive values (NPV), and (6) F1 score. The AUROC is a common evaluation tool that quantifies overall classification performance. The AUROC scale is from 0 to 1, with a value of 0.5 indicating an uninformative classifier. Higher AUROC scores typically signify better performance. While AUROC measures the area under the true positive and false positive rate, AUPRC evaluates the performance on the precision and recall graph. AUROC is a metric independent of class imbalance, and AUPRC is a metric for imbalanced datasets. The F1 score is the harmonic mean of the precision and recall of a classification model. It ranges between 0 and 1, and models with values closer to 1 are better models. PPV and NPV represent the accuracy of a diagnostic test in identifying true positive and true negative results, respectively. Predictive values indicate the likelihood that a specific diagnosis given by a test is accurate for a subject. The following equation computes PPV:

(1)

where TP is the true positive (the number of cases correctly identified as anxiety), and FP is the false positive (the number of cases incorrectly identified as anxiety). NPV can be computed with:

(2)

TN is the true negative (the number of cases correctly identified as non-anxiety), and FN is the false negative (the number of cases incorrectly identified as non-anxiety).

Machine learning-based models

Fig 2 displays the outcomes of ML models in terms of Area Under the Receiver Operating Characteristic (AUROC) score and Positive Predictive Value (PPV) for all age groups. Fig 2(a) and (c) employ the EHR features, while Fig 2(b) and (d) showcase outcomes utilizing EHR+ABMH features. Fig 2(a) and (b) display the AUROC score, whereas Fig 2(c) and (d) display the PPV for each model across the age groups.

According to the results shown in Fig 2(a) and (b), XGBoost consistently beats other ML models in terms of AUROC score in most of the age groups. RF demonstrates the second-best performance, while KNN performs worse than all other models. RF surpasses XGBoost in AUROC score in age groups 2 and 21, with the lowest number of patients. Nevertheless, Fig 2(c) and (d) demonstrate that LR and DT exhibit superior PPV compared to RF. This indicates that LR and DT have a higher precision but a lower predictive performance than the other two models. The findings generally indicate that XGBoost performs better than other models in terms of AUROC score and PPV, while KNN performs worse than other models.

Deep learning-based models

Fig 3 displays the outcomes of DL models in terms of AUROC score and PPV for all age groups. Fig 3(a) and (c) employ the EHR features, while Fig 3(b) and (d) display the outcomes utilizing EHR+ABMH features in AUROC score and PPV, respectively.

Fig 3(a) demonstrates that either RETAIN or Dipole achieve higher AUROC scores than other DL models in most of the age groups. However, in the case of the results utilizing EHR+ABMH features, unlike the ML results, which exhibit identical outcomes to those with EHR features, the results utilizing EHR+ABMH features display a distinct pattern. Although most models show a consistent pattern when using EHR features, EHR+ABMH features experience declines in certain age groups (namely, the GRU results for age group 10 and the RETAIN results for age group 17). Furthermore, although RETAIN demonstrates high performance in AUROC score when utilizing EHR features, EHR+ABMH features do not exhibit the same outcome. Fig 3(b) demonstrates that LSTM yields more consistent results but does not surpass other models. In both datasets, the RETAIN and Dipole models have superior AUROC scores, surpassing all other models in most age groups.

The PPV results exhibit greater complexity, as depicted in Fig 3(c) and (d). According to the data, Dipole does not demonstrate the highest PPV across the age groups. On the contrary, RETAIN demonstrates superior PPV in about 50% of the age groups, although the AUROC score is lower than Dipole in most age groups. From this, we infer that the findings of RETAIN exhibit higher precision but a poorer predictive performance than Dipole. For some age groups, specifically 2, 3, 20, and 21, the 95% confidence interval (CI) is substantial. This means the data exhibits significant variability, which could impact the analysis. It also shows that the patient population needs to be increased to train the model adequately. For instance, for the results utilizing EHR features (Fig 3(c)), age group 2 consists of 930 patients, resulting in a CI of 0.0026 for the PPV of the RETAIN model. On the other hand, age group 16 includes 11,150 patients and a CI of 0.00088. This indicates that the size of the dataset is also essential to train a more effective model.

Developmental subgroup analyses

To contextualize age-specific results, we summarized model performance across three developmental periods: preschool (ages 2–5), school-age (6–12), and adolescence (13–21), aggregating the single-year estimates reported in Figs 2 and 3. For ML models, XGBoost generally achieved the highest AUROC across subgroups, with RF competitive at the youngest and oldest edges (e.g., age 21 AUROC 0.845 with EHR and 0.848 with EHR+ABMH). For DL models, RETAIN and Dipole consistently yielded the strongest AUROC across all developmental periods under EHR, with Dipole exhibiting relatively stable performance into adolescence (e.g., ages 13–17 with EHR: 0.848, 0.854, 0.829, 0.845, 0.819). PPV was more variable: RETAIN often showed higher precision in preschool and school-age (e.g., EHR PPV from 0.834 to 0.838 at ages 4–8), whereas Dipole was frequently competitive in adolescence under EHR+ABMH. Comparing EHR+ABMH to EHR, ABMH features produced modest and non-uniform changes across ages and models (e.g., Dipole age 11 AUROC increased from 0.842 to 0.856, age 17 decreased from 0.819 to 0.813; RETAIN age 2 decreased from 0.833 to 0.765, age 8 increased from 0.851 to 0.853). Overall, these patterns indicate that developmental stage influences both discrimination and precision, with DL architectures (RETAIN/Dipole) generally outperforming other models and XGBoost providing the strongest ML baseline.