Introduction

Globally, cardiovascular diseases (CVDs) are the number one cause of all death [1, 2]. In 2016, 17.9 million people died of CVDs alone, accounting for 31% of all global deaths [1]. The direct costs of CVDs in the US for 2010 were $272.5b whereas indirect costs were $171.7b and are expected to increase to $818.1b and $275.8b in 2030 respectively [3, 4]. Atherosclerosis alone is responsible for 1.3% of all hospital stays with costs of $9b per year, while all atherosclerosis-related diseases amount to $43.5b of hospital costs annually [5]. Individually, patients with CVD incur more than twice the medical costs of age- and sex-matched patients without CVD, largely because of the increased likelihood of subsequent hospitalizations. The greatest differences in total CVD costs usually occur when comparing patients with and without a secondary CVD hospitalization [6].

All current guidelines on the prevention of CVD in clinical practice recommend the assessment of total CVD risk since atherosclerosis is usually the product of a number of risk factors [7, 8] and in recent years these guidelines have evolved to focus on the absolute risk of disease as opposed to relative risk [7–10]. Clinician tools for CVD risk estimation must enable rapid and accurate estimation of an individual patient’s absolute CVD risk [7], or for opportunistic screening of high-risk patients from relevant populations [11]. Screening is the identification of unrecognized disease or risk of disease in individuals without symptoms. In addition to opportunistic screening, which is carried out without a predefined strategy (e.g. when the individual is consulting a general practitioner (GP) for some other reason), tools can be used for systematic screening, which is centrally organized strategic screening in the general population or in targeted subpopulations, such as subjects with a family history of premature CVD or familial hyperlipidaemia [7]. There is ongoing debate on the role of systematic centralized population based screening in CVD [10, 12] because of burdensome diagnostic testing following the use of risk based screening tools [13]. A relatively new area of screening is self-screening, carried out by proactive individuals using screening tools on mobile devices such as smartphones or smartwatches, which may use built in app-linked sensors or screening chat-bots [14–16]. There is public demand for reliable, actionable, explainable and usable health information tools [17], including for disease screening.

The risk to build up atherosclerotic plaque varies and is determined by multiple factors such as genetics, environment and lifestyle [11, 18–21]. The risk of developing atherosclerotic plaque can be reduced based on an individual’s behavioral risk factors, such as smoking, physical activity and nutrition [1, 11, 19, 20].

Most diseases, including atherosclerotic CVDs, have a complex pathophysiology that involves multiple interacting molecular systems, making it insufficient to look only at an isolated biological pathway or a subset of markers to predict disease risk [22]. A precision medicine based approach is required, where multiple biological layers are considered (i.e., ‘multi-omics’), alongside clinical and lifestyle data [22]. Such an approach has the potential to capture all important interactions or correlations detected between molecules in different biological layers, providing a holistic understanding of an individual’s current health status and enabling the quantification of an individual’s absolute risk of atherosclerotic CVDs [23, 24].

Previous studies in this area use a limited set of risk factors and outcomes for their analyses [7, 25, 26]. In recent years, the knowledge of behavioral risk factors and of the pathophysiology of atherosclerotic CVDs have advanced tremendously [11, 25]. Current absolute risk prediction models have limited predictive capability as they have not been trained on all possible atherosclerotic CVD outcomes [27–29], or they include outcomes which are unmodifiable such as those related to pregnancy, accidents, or congenital factors [29].

Both SCORE (Systematic COronary Risk Evaluation) and SCORE2 [30, 31], are models for predicting relative CVD risk, whereas we focus on predicting absolute CVD risk, which is why we chose to omit those models from our analysis. Another related investigation, which also used the UK Biobank (UKB) dataset, developed multiple Cox Proportional Hazard models for 10-year CVD risk prediction, with a reduced version requiring 47 risk factors and another version disregarding all cholesterol risk factors as well as systolic blood pressure, in order to provide a simple approach for risk prediction in remote settings with limited testing resources [32]. However, survival models such as the proportional hazard model are not designed to provide absolute risk estimates for individual patients.

Machine learning (ML) based approaches have many advantages compared to humans or standard statistical algorithms, such as superior performance, being able to identify complex non-linear patterns, the ability to encode diverse and high dimensional data types, being more stable to outliers, allowing continuous model updates, versatility for different domains and scalability [33–36].

However, classic disadvantages of ML based approaches are their lack of interpretability, risk for inherent bias due to the used data, difficulty to acquire physician adoption, explaining to physicians why a new risk model might be superior to existing ones, with all of these hindering widespread adoption of ML based risk prediction models [36, 37]. One example for ML based CVD risk prediction is the AutoPrognosis based approach, where an ensemble of multiple ML pipelines has also been applied on the UK Biobank dataset for 5-year CVD risk prediction [29]. Further, using a purely ML-driven approach can lead to a model that requires too many risk factors to compute risk, which is infeasible for routine clinical check-ups. Another disadvantage of purely data-driven approaches is the inclusion of risk factors which might show strong correlations but are unrelated to the pathophysiology of CVDs or are not actionable, making them inapplicable in a clinical setting or as an actionable self-management tool [29].

The aim of this study was to use a large-data ML approach to develop an actionable absolute risk prediction tool which considers the holistic health of an individual. Uniquely, we focused on behavioral risk factors relating to all atherosclerotic CVD outcomes. Our goal was to have a holistic understanding of an individual’s current health status, to better quantify their risk of atherosclerotic CVDs, and to provide actionable advice. Our approach is novel in that we employ a highly holistic understanding of an individual’s current health status, to better quantify their risk of all atherosclerotic CVDs. By utilizing a comprehensive set of lifestyle factors, we enable the subsequent suggestion of personalized and actionable advice relating to unhealthy risk factors. Instead of using only a limited set of risk factors, we aimed to achieve this by taking multiple biological layers into account, which include: (i) multi-omics data from blood samples (e.g. lipidome and proteome); (ii) family history (e.g. genome), (iii) lifestyle data, (iv) clinical data and (v) environmental data; along with (vi) an extensive set of risk factors and outcomes.

We used data from 464,547 participants of the UK Biobank study who did not have atherosclerotic CVD at their baseline visit. We created an automated pipeline to benchmark risk prediction classifier algorithms against each other, then evaluated their predictive performances in the overall population and tested the generalizability of the top-performing classifiers through retraining and testing on different sub-populations. We explored the clinical implications of the proposed classifiers, with a focus on the top-performing models. This study does not focus on the algorithmic aspects of the utilized classifiers.

Methodological details on the utilized classifiers can be found in the open-source documentation of the respective algorithms of the scikit-learn [38] and xgboost [39] libraries and in the supporting information (S4 Table).

Materials and methods

Baseline data from the UK Biobank was utilized to extract an extensive set of risk factors and outcomes associated with the pathophysiology of atherosclerotic CVDs. A benchmarking pipeline was used to train and evaluate different standard and ML algorithms for the task of 10-year atherosclerotic CVD risk prediction. The performance was measured using AUROC and compared against the baseline models Framingham and QRisk3, which are widely used and recommended models. We evaluated our best performing models further by analyzing the most informative features and assessed model generalizability and created a reduced model.

Model development and benchmarking using pipeline

We built a benchmarking pipeline for automated and reproducible data extraction, normalization, imputation, model training, tuning of model hyperparameters, classification, documentation and reporting.

We implemented all models using their respective scikit-learn library or xgboost library implementation using the Python programming language [38, 39]. Details on the used Python libraries, methods and parameters are provided in the supplementary data (S3 and S4 Tables).

Categorical values were one-hot encoded. Data normalization was performed by removing the mean and scaling to unit variance. Data imputation was performed for all models using a simple mean imputation. The models’ hyper-parameters were determined using grid search and stratified k-fold cross validation using 3 folds was employed to avoid overfitting.

Finally, we assessed model performance mainly using the AUROC. Fig 1 visualizes an overview of all performed steps of our experimental setup.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Overview of experimental setup of proposed approach.

https://doi.org/10.1371/journal.pone.0263940.g001

Iterative feature elimination.

We employed an iterative feature elimination procedure based on the regularized logistic regression for finding the best trade-off between predictive performance and number of risk factors, with the aim of creating a risk prediction algorithm that is applicable in the clinical context. We used the standard L1 regularization (also known as Lasso) proposed by [57]; it implements a strong penalty on non-zero feature weights of our logistic regression model, resulting in a sparse feature set for prediction.

A logistic regression coefficient value β can be interpreted as the expected change in log odds of having the outcome per unit change in the feature x_β. Therefore, increasing the feature by one unit multiplies the odds of having the outcome by e^β. This means that we can interpret the coefficients as feature importance values in the sense that the feature with the smallest coefficient has the least importance on model predictions. Importantly, this holds only true in the context of the parameters contained in the current model. Thus, we re-estimate the model after each feature elimination round.

In each iteration, we re-estimated the logistic regression model on the remaining parameters, and then discarded all parameters that were set to zero by the L1 regularization; finally, we also discarded the parameter with the lowest non-zero absolute value.

As an additional step, we created a ranking of the relative feature importance value of each feature by dividing its absolute coefficient weight by the sum of all absolute coefficient weights.

Statistical analysis.

To reduce overfitting, we evaluated the classification performance of all our benchmarked algorithms by using 3-fold stratified cross-validation and measuring the Area Under the Receiver Operating Characteristic Curve. For the cross-validation, we used a training set with 325,182 participants to train and derive our standard linear and ML models and then assessed the AUROC performance on the held-out test set with 139,365 participants using 203 risk factors respectively. We reported the AUROC and the 95% confidence intervals (Wilson score intervals) for all models and performed a sensitivity analysis using Shapley Additive Explanations (SHAP values) for the best performing linear model.

Generalizability.

With 442,620 out of the 502,551 participants in the UK Biobank, the cohort has a high proportion (88.1%) of participants with British White ethnicity. In an effort to estimate a proxy for out-of-sample generalizability, we re-trained the two best models, XGB and logistic regression with L1 regularization, only on Whites and tested their performance on a non-White test set. The white-only training set consists of 378,836 participants (81.5%). The non-White test set consists of 85,711 participants (18.5%).

Results

Characteristics of the training and test populations

Of 502,551 patients in the UK Biobank, we filtered out 7.6% who already experienced a relevant CVD outcome (during or before baseline) and the participants being lost or who withdrew from the biobank. This resulted in 464,547 participants who met the inclusion criteria. 28,561 (6.1%) of those participants developed at least one of the relevant CVD outcomes during their 10-year follow-up period. We used a common 70% of the data as a training set and 30% as a hold-out test set. Table 1 shows the overlap of our atherosclerotic CVD outcome definition with the CVD outcome definition used in the related work approach by Alaa et al. [29]:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. CVD outcomes statistics according to definition in current study and the comparator study definition by Alaa et al. [29].

https://doi.org/10.1371/journal.pone.0263940.t001

Prediction accuracy

The resulting prediction accuracy of the benchmarked models is depicted in Table 2. We used both Framingham 10-year CVD risk versions, with and without lipids, as well as QRisk3 as baseline models to assess the performance of predicting someone’s 10-year risk of developing an atherosclerotic cardiovascular disease based on a holistic set of risk factors, with a focus on actionable risk factors and outcomes. The best performing model was XGB with an AUROC of 75.73%, only marginally higher than the logistic regression model with L1 regularization (75.44%) and substantially better than the Random Forest model (66.90%).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Performance of all tested classifiers including baseline models.

https://doi.org/10.1371/journal.pone.0263940.t002

Fig 2 shows the AUROCs of the best performing models XGB and from logistic regression with L1 regularization, which is the simplest model tested and amongst the top two best performing models. Logistic regression comes with the advantages of being interpretable by providing reasoning for its classifications, and being a simple and robust method [36].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. AUROC of logistic regression with L1 regularization and XGBoost.

https://doi.org/10.1371/journal.pone.0263940.g002

In order to better evaluate the clinical implications and significance of our results, we compared the results of our benchmarked models with our baseline models Framingham and QRisk3. Table 2 shows that both our XGB and logistic regression classifiers achieved superior performance compared to the baseline models. Apart from the Random Forest model, all tested models had a higher AUROC than both baseline Framingham (68.0% and 68.1%) and QRisk3 (72.5%) models.

The difference in AUROC performance of the Framingham score in our experiments in Fig 2 compared to Alaa et al. [29] is explainable by their use of an older UK Biobank version with 40,000 fewer baseline patients with their last available date of participant follow-up being February 17, 2016. The UK Biobank version we used includes biochemistry data which was released May 1, 2019 including cholesterol and additional questionnaires data. Additionally, more diagnosis data was made available over time. These dataset differences may help explain the difference in AUROC.

Figs 3 and 4 show the AUROCs of all baseline models on imputed and unimputed data respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. AUROC curves of baseline models on imputed data.

https://doi.org/10.1371/journal.pone.0263940.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. AUROC curves of baseline models on unimputed data.

https://doi.org/10.1371/journal.pone.0263940.g004

Both Framingham versions perform nearly identically on imputed and unimputed data whereas QRisk3 performs worse on unimputed data.

Feature elimination vs. predictive performance

Fig 5 shows how the performance of the best logistic regression model depends on the number of risk factors used. Discarding the risk factors stepwise leads to a relatively unchanged and stable model performance until around 170 iterations of feature elimination. This indicates that for predicting an individual’s 10-year atherosclerotic CVD risk, many features provide only marginal value and a small subset of features provides substantial informative value. After around 170 iterations, there was a marked decline in model performance associated with further reductions in utilized features.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Performance of best logistic regression model depending on number of features.

AUROC performance of best performing logistic regression model with L1 regularization (continuous blue line) compared to number of features utilized in each iterative feature elimination step (orange line), dotted blue horizontal line showing intersection of 25 features with iterative feature elimination step, allowing for extrapolation to model performance.

https://doi.org/10.1371/journal.pone.0263940.g005

Table 3 shows in more detail the dependence of the model performance on the number of features. Utilizing only 25 (88%) out of the 203 total risk factors still leads to a reasonable AUROC performance, with a high reduction in utilized features. Compared to the model performance with an AUROC of 75.44% when using all 203 risk factors, the model still achieves 74.15% (95% CI: 0.7392–0.7438) with the 25 most informative risk factors.

We also assessed the performance for fewer features. To reach the same performance as QRisk3 of 72.5% AUROC, 16 features would be necessary. The two most informative features were age and biological sex. To reach a similar performance as Framingham (68.0%), just two features were necessary (68.98%). It is worth noting, however, that both Framingham and QRisk3 were trained and tuned on other datasets and have different CVD definitions and objectives.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Performance of best logistic regression model depending on number of features.

https://doi.org/10.1371/journal.pone.0263940.t003

Generalizability of results

We assessed the generalizability of our models by re-training the two previously best performing models only on a White cohort and then testing them on a non-White cohort. Table 4 and Fig 6 show the results for logistic regression and XGB. The logistic regression model has an AUROC of 75.86% in the generalizability experiment, compared with an AUROC of 75.44% in the previous experiment. XGB has an AUROC of 76.26% in the generalizability experiment and 75.73% in the previous experiment. These results show only marginal differences to the results of the previous experiments.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. AUROC of logistic regression with L1 regularization and XGBoost when trained on Whites and tested on non-Whites.

https://doi.org/10.1371/journal.pone.0263940.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Model performance when trained on Whites and tested on non-Whites.

https://doi.org/10.1371/journal.pone.0263940.t004

Predictive ability of individual variables in UK Biobank

Table 5 shows the relative regression feature weights of the 25 most informative risk factors in descending order. A full list is provided in the supplementary materials (S5 Table). Based on our previous manual curation of risk factors and outcomes, we can see that the most informative risk factors are distributed across 5 categories (Table 6), with the lifestyle category contributing the most risk factors. The two most informative features were age and biological sex. We provided a sensitivity analysis using SHAP values of the best performing logistic regression model for all risk factors in the supplementary materials (S1 Fig).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Relative regression feature weights of 25 most informative risk factors from best logistic regression model.

https://doi.org/10.1371/journal.pone.0263940.t005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Categorization of the 25 most informative risk factors into categories from the best logistic regression model.

https://doi.org/10.1371/journal.pone.0263940.t006

Discussion

Using data gathered from the large longitudinal cohort UK Biobank study, we developed a pipeline to benchmark several classification models for predicting a subject’s 10-year absolute risk of developing an atherosclerotic CVD. We used an extensive set of physician curated risk factors and outcomes methodology, employing a holistic view of the subject’s current health status rooted in a precision medicine approach. The models were trained and evaluated using data from 464,547 UK Biobank participants, spanning 203 CVD risk factors for each subject. Using a simple logistic regression model with a holistic set of risk factors significantly improved the accuracy of atherosclerotic CVD risk prediction compared to currently available, widely used and recommended models such as Framingham and QRisk3. Both of these existing models rely on a limited set of risk factors and outcomes and do not focus on modifiable lifestyle factors. Further, our best performing logistic regression model utilizes new CVD risk predictors showing high predictive power, namely: social visits, walking pace and overall health rating. The frequency of social visits could be indicative of someone’s current mental health status, which has been shown to be a relevant CVD risk factor [58, 59]. These and other non-laboratory risk factors could be collected by means of a questionnaire or passively deduced using data analytics from data sources such as GPS, calendar and sensors [26, 60] from e.g. smartphones, smartwatches and fitness trackers.

Additionally, our best performing models, XGBoost and logistic regression, showed marginal differences when trained and tested on particular sub-populations, which is indicative of good generalizability to other ethnicities.

As there was little performance difference between the best performing models, we primarily discuss the simplest model, logistic regression with L1 regularization. This model has the inherent benefit of offering reasoning for its predictions through analyzing the learned coefficients for every risk factor and having feature selection performed by the L1 regularization. With L1 regularization, less important risk factors’ coefficients are minimized and also set to zero, which then leads to entire removal of these features from the model, and fewer risk factors needed for an accurate prediction.

Using iterative feature elimination, we identified a subset of the 25 most relevant risk factors providing a similar performance compared to using all 203 risk factors. The 25 most relevant risk factors are distributed across five different categories, suggesting that different biological layers contribute to the risk of atherosclerotic CVD. This result confirms that it is insufficient to assess only one biological layer for accurate risk prediction, supporting our initial model development approach [61]. Our approach takes into account multiple biological layers by using multi-omics as well as clinical and lifestyle data with the aim to capture all potential interactions or correlations detected between molecules in different biological layers [22]. Multi-omics data generated for the same set of samples can provide useful insights into the interaction of biological information at multiple layers and thus can help in understanding the mechanisms underlying the complex biological condition of interest.

In our model, the lifestyle category contributed the most risk factors, suggesting that accurate prediction relies upon continuous daily lifestyle data and not just periodic snapshots of clinical data. The causal relationships between the risk factors considered in our model and atherosclerotic CVDs have been demonstrated by other studies [11, 19, 21, 25].

Innovative approaches are needed in order to tackle the increasing prevalence and mortality of CVD-related diseases [2], and the associated healthcare systems’ financial burdens. This is particularly true in low and middle income countries where CVD prevalence has also been increasing and is expected to increase as a consequence of an aging and growing population [2]. Our atherosclerotic CVD prediction model has the potential to support healthcare systems by identifying more people at risk earlier and more accurately than currently available models and intervening with personalized behavior change programs. Currently available models, like Framingham and QRisk3, have limited predictive capability for atherosclerotic CVDs as they were not trained on all of them and do not provide actionable results.

There is potential for novel disruptive approaches to affordably improve CVD outcomes. Areas where this may have an impact is in novel approaches to screening, lifestyle coaching and prevention [2]. Screening will become more accessible and widespread by more (near-)medical-grade sensors being integrated into smartphones and smartwatches, enabling continuous monitoring of relevant behavioral CVD risk factors, as well as biomarkers such as heart rate, blood pressure and blood glucose. By gathering a wider spectrum of relevant risk factors for cardiovascular disease automatically and continuously, an ongoing and personalized cardiovascular disease risk prediction could be enabled. Through linking personalized information on an individual’s CVD risk with app-based programs for sustained behavioral modification, it may be possible to lower the incidence and mortality of CVDs [62]. Combined with a companion smartphone-based app, an AI or healthcare provider-generated personalized intervention program could be provided and targeted at those people who need it the most.

A system and method gathering personal health data and predicting an individual’s atherosclerotic CVD risk is handling sensitive health data (e.g. laboratory values) and must adhere to local regulations and best practices in data transfer, processing and storage to ensure data privacy and security.

Many studies have shown that digital health interventions are cost effective for managing CVD (for a review see [63]). One report found that a community-based prevention program could have a mean return on investment (ROI) on medical cost savings of $5.60 for every $1 spent within a 5 year timeframe by improving physical activity and nutrition and reducing tobacco usage [64]. A review of 11 in-home cardiac rehabilitation programs for the secondary prevention of CVD found that social support, goal setting, monitoring, credible instructions and literature resources are all effective behavior change techniques to reduce behavioral risk factors for CVD [65].

The improvement achieved by our models might be partially attributed to being trained and assessed on the UK Biobank dataset, whereas the baseline Framingham model was derived from a different population. The population and many of the data sources used in the QRisk3 model are similar, being the general UK population and using their GP, hospital and mortality records. However, our risk model generation approach and QRisk3’s approach were designed with different aims and objectives and the modelling strategy was different. For these reasons, direct comparison between the models is limited. Notable differences between the approaches include a more limited set of risk factors included in Framingham and QRisk3’s and a focused and wider range of atherosclerotic CVDs included in our approach.

The results from our generalizability sub-analysis indicate that our XGB and logistic regression models might generalize well to other ethnicities and do not overfit to our cohort, however, this needs to be further evaluated with more data from diverse ethnicities.

Our results show that our models have improved performance over the baseline models Framingham and QRisk3 (Table 2). This is because the selection of the appropriate disease modelling approach, classifiers and careful tuning of the model’s hyperparameters are crucial steps for realizing the potential benefits of ML. Our pipeline automates some of these steps which makes the tuning and discovery of new disease risk models easily accessible for clinical research. Our prospective cohort modelling approach, which is rooted in precision medicine, is the first to generate an atherosclerotic CVD absolute risk prediction tool based upon a complete definition of atherosclerotic CVD outcomes and a holistic set of risk factors.

Conclusions

We benchmarked multiple classifiers to predict an individual’s 10-year risk of developing an atherosclerotic CVD, using a holistic set of risk factors and a specific definition of atherosclerotic CVDs. Our reduced logistic regression with L1 regularization classifier, a simple and interpretable model, is amongst our best prediction models, includes actionable lifestyle factors, has great predictive power and requires 13 unique features. Our experiments showed that a two feature-questionnaire is as accurate as the Framingham models and a 16 feature-questionnaire is as accurate as QRisk3 for 10-year atherosclerotic CVD risk prediction. Both prediction models, XGBoost and logistic regression, generalize well to non-White people, which might indicate that our models generalize well to other (western) countries. Framingham and QRisk3, which are well established and validated absolute risk prediction models, do not perform as well on predicting individuals’ 10-year risk of developing an atherosclerotic CVD. With our logistic regression model, we created a promising new interpretable, actionable and accurate risk prediction tool that could assist individuals and public health in CVD risk reduction.

Supporting information