Ethics statement

All the data were fully anonymized before we accessed them. Our study was approved by the Institutional Review Board at the Washington University School of Medicine in St. Louis. We obtained a written acknowledgement of proprietary rights and non-disclosure and data use agreement from the American Heart Association (The Washington University_NDA_DUA_CONTRACTID 158065_2019.04.26_K).

Established in 2011, The Guideline Advantage (TGA) was a clinical data registry jointly operated by the American Cancer Society, the American Diabetes Association, and the American Heart Association [20]. The program collects EHR data to track and monitor outpatient prevention and disease management. Briefly, the data collected through TGA from over 70 clinics provide a unique platform for longitudinal research on the impact of preventative care. The program’s research strategy is focused on identifying patient-, provider-, and practice-level factors associated with guideline adherence and assessing the effectiveness of quality improvement interventions in increasing guideline adherence. Here we used TGA data to predict the diagnosis of CD among 362,533 unique patients in the data set.

Our data set represented patients seen in the outpatient setting over a 10-year period (2007 to 2016). We defined our study outcome by classifying 19,597 unique ICD-9 and ICD-10 codes to a smaller number of clinically meaningful categories using Clinical Classifications Software (CCS) [21]. After the codes were converted to the appropriate CCS category, we identified 34,511 patients with a diagnosis of CD (single level CCS code = 106). Among them, the majority (55%) were female patients, and 66% of patients were white. If a patient had multiple CD diagnoses in the data set, only the earliest one was considered.

Next, we extracted all measurements of CVH prior to the diagnosis of CD. We utilized measures of CVH as follows: smoking status, body mass index (BMI), blood pressure, hemoglobin A1c, and cholesterol, which were defined and classified by the AHA into three categories: ideal, intermediate, or poor according to Table 1. To classify patients as intermediate health or treated-to-goal for selected CVH submetrics (Table 1), we converted the drug names to their drug classes by comparing the drug names in our dataset with the Multum drug database [22]. One string match technique–Levenshtein distance algorithm [23]–was applied and we considered the distance between the two matched strings as less than five to be matched and included these in subsequent analyses.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Measures of CVH which are available in the EHR (adapted from: Lloyd-Jones, 2011) [24].

https://doi.org/10.1371/journal.pone.0239007.t001

We studied patients with CD who had four or more outpatient CVH measures in the data set (n = 5,271). Using the same criteria, we randomly selected 5,784 patients from the dataset who did not have a diagnosis of CD. In sensitivity analyses, we tested the robustness of our strategy by changing the number of outpatient CVH measures from zero to three, respectively. Ultimately, our data set comprised 11,055 patients who had four or more encounters over the 10-year study period.

Statistical analysis

To prepare the CVH measures for analysis, we combined the submetric with its classification according to Table 1. For example, if a patient had a measurement of “ideal” cholesterol, then we combined the submetric and its value as cholesterolideal. The resulting features were mapped to a 32-dimensional vector by word embeddings [25] in our model. The Genism Word2Vec model was configured the hyperparameters as following: size (embedding dimension) as 32, window (the maximum distance between a target word and all words around it) as 5, min_count (the minimum number of words counted when training the model) as 1, sg (the training algorithm) as CBOW (The continues bag of words). The input of Word2Vec model was all above combined measurements of all 11,055 patients. We also added time information for all measurements as time steps. Each feature was associated with a time point which was calculated by the difference in days between the corresponding visit time and the latest measurement time. For example, if the most recent visit date was February 11, 2019, and measurement was conducted on January 11, 2019, then the time point value is: 31. Thus, each individual patient had its own vector to represent their measurements of CVH.

The embedded vectors of patients were the inputs for our long short-term memory (LSTM) model. We applied an LSTM algorithm to investigate the association between time-series CVH measurements and the outcome of CD. We also investigated other machine learning and deep learning algorithms such as DNN [26], LR [27], RF [28] and NB [29] to study the same association between CVH and CD. All of the CVH measurements for each patient were sorted in chronological order. We padded the patients with virtual events as the same length (311) in the form of [0₁, … 0_k, event_1, event_(311-k)] if they had less events than the maximum number of measures (311), where k was the difference of 311 and number of records that patients had.

To investigate the effects by continuous vectors obtained from Word2Vec algorithm, we conducted the same predictions by using categorical variables. These categorical variables were sorted in a time order, and each categorical event concatenated with the same time points (e.g., difference in days between the corresponding visit time and the latest measurement time) were the inputs of the models of LSTM, DNN, RF, LR, and NB. We did the same padding approach as above for patients had less events than 311.

For each model of predictions, we utilized 5-fold cross validation by dividing dataset into 5 folds with each fold serving as a testing dataset and the remaining 4 folds as a training dataset. Criteria of the area under the receiver operator curve (AUROC) and other metrics, i.e., accuracy, sensitivity, precision, f1 score, and specificity were calculated to evaluate the performance of the models.

LSTM unit

A common LSTM unit is composed of a cell and three gates: input gate, output gate and forget gate. The cell remembers information at each time step and these gates control the flow of information pass on to and forget/discard to the next time step [30]. We illustrated the basic structure of an LSTM unit as in Fig 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Graph illustration of LSTM unit.

https://doi.org/10.1371/journal.pone.0239007.g001

Mathematically, the equations for forward pass to update an LSTM unit with a forget gate at a time t are:

Where * denotes the element-wise product and X_t is the input vector (i.e., embedding vector in our case) at time t. The weight matrices W_f, W_i, W_c, W_o for hidden state h_t, U_f, U_i, U_c, U_o matrices for input X_t, and bias vector parameters b_f, b_i, b_c, b_o are learned during the training stage and h_t is the hidden layer output vector. Activation function σ is the sigmoid function and tanh is the hyperbolic tangent function.

Our LSTM model comprised an input layer, one hidden layer (100 dimensions) and a scalar output layer. A binary cross-entropy loss function was employed as the output layer and a sigmoid function was used as the activation function for the hidden layer. Adam optimizer [31] was used to optimize the model with a mini-batch size of 64 samples. The DNN was comprised of an input layer, 5 hidden layer (with 256, 256, 128, 64 and 32 dimensions respectively). and a scalar output layer. We used the Sigmoid function [32] at the output layer and ReLu function at each hidden layer. Binary cross-entropy was used as loss function and Adam optimizer was used to optimize the models with a mini-batch size of 64 samples. The LR, RF and NB models were configured by default options in the package of Scikit-learn in Python 3.

We then investigated which features were the most important in CD prediction. To obtain this goal, we iterated the model 15 times by setting constant value for one feature each time. For each feature, we first manually set it as a constant (not informative for the predictive models), then tested the prediction performance of trained models using the manually changed features to evaluate the discriminative importance of the given feature. The resulting performance then was compared its prediction accuracy and AUROC with the full model. If there was a large change between these two values, it indicated that this feature was important and discriminative to the prediction. Analyses were conducted by using the libraries of Scikit-learn, Scipy, Matplotlib with Python, version 3.6.5 in 2019.

Limitations

We encountered some limitations to using EHR data for these analyses. First, patients had different times for visits of CVH measurements as some patients visited more frequently and had high numbers of visits and some just had a few visits. To address this, we created virtual events for patients with fewer visits in order to conduct our analyses. Second, the prediction accuracy might be further improved with additional demographic and clinical factors in addition to the regular medical visits and measurements used in this study (e.g., health data collected from wearable devices). Our findings would be much more generalizable if we had greater representation from more clinics across the U.S.