Introduction

Reduction of cardiovascular disease (CVD) burden, i.e. at population level, is commonly accomplished using preventive strategies (like lifestyle and dietary advice or preemptive drug treatment) in individuals with marked elevations in risk factors, e.g. low-density lipoprotein (LDL), or a high predicted CVD risk based on a combination of risk factors [1]. Identification of high risk individuals is often achieved using CVD risk prediction models of which over 360 different variants have been published as of 2016 [2]. However, different models may predict multiple and often different CVD outcomes or sets of outcomes (as is the case in model with composite endpoints) [2–4]. These differences in predicted outcomes may result in large variation in CVD risk estimates. Consequently, it is unclear to what extent the predicted CVD risks obtained from different prediction models are comparable and can be interpreted similarly in clinical practice [4–7].

The large variation in CVD risk estimates combined with different recommended risk thresholds for each prediction model, may lead to different definitions of high-risk individuals. For example, the Pooled Cohort Equation stratifies individuals with a > 7.5% 10-year CVD risk as high-risk whereas the recommended threshold for the Framingham risk equation is 10% [8, 9]. Different definitions of high-risk individuals may, in turn, lead to different treatment recommendations. Furthermore, the expected health benefits of treatment may also be different since the impact on quality of life differs per CVD event type and severity. For example, the expected health loss due to a stroke is expected to be higher than the health loss due to a myocardial infarction [10].

Since the implication of different treatment recommendations could be large, the aim of this paper is to assess if the use of different prediction models leads to different treatment recommendations in clinical practice. Therefore, four widely used CVD risk prediction models were investigated regarding their comparability and interpretation, after applying them to a large population cohort. Additionally, we discuss the usefulness of such models based on the comprehensiveness of their composite endpoint and provide a recommendation for the development of new prediction models in order to enhance their usefulness in clinical practice. This paper does not focus specifically on Dutch clinical practice and does not provide guidance on preferred prediction models for the Dutch context.

Methods

Adult Treatment Panel III (ATP), Framingham Global Risk Score (FRS), Pooled Cohort Equations (PCE), and SCORE-low (SCORE) are four widely used CVD risk prediction models for primary prevention [11–14]. All are derived from general population cohort data. Hence, they include (often similar) predictors that are easy to measure in everyday clinical practice, such as gender, age and systolic blood pressure. The exact definition of the included risk factors in the risk equation can be found in the original publication [11–14]. Furthermore, the probability estimate of each model reflects the absolute risk of the composite endpoint occurring within 10 years. In order to compare these four models, we first identified the exact definition of each composite endpoint from the original publication describing the development of the model [11–14]. We then, standardized the composite endpoints using ICD-10 codes. This was necessary since the published articles often only described the outcomes in words, e.g. “coronary heart disease” or “ischemic heart disease”.

To compare the composite endpoints, we used the MORGEN cohort. The MORGEN cohort is a large Dutch general population cohort which includes men and women aged 20 to 74 years at baseline, recruited from the general population between 1993 and 1997 [15]. Participant information on vital status, cause of death and comorbidity was obtained from Statistics Netherlands and the National Medical Registry (NMR). The follow-up period of the MORGEN cohort was 10 to 15 years with a mean follow-up time of 12 years. To apply the prediction models, information both from baseline and from follow-up was required, leaving 19,484 (72%) individuals with adequate data for the analysis from the original cohort [16, 17]. To further investigate the constitution of the composite endpoint, we determined the observed rates and distributions of the individual components, i.e. included CVD event type according to the associated ICD-10 code(s), for each model separately.

As the indication for statin therapy is also LDL-dependent and we aim to illustrate the complexity of CVD risk predictions by comparing results of different prediction models, individuals with an elevated level of LDL and/or diabetes were excluded for further analysis. We focused on individuals in whom preventive intervention was indicated based on predicted CVD risk rather than on elevated LDL levels and/or diabetes. After excluding 231 individuals with diabetes, 1,141 individuals with elevated LDL levels and 25 individuals with both risk factors at baseline, this resulted in a cohort size of 18,137 individuals (mean age = 42.4 years, range 20.1–73.7 years, and 45% men).The MORGEN cohort was also used to compare the predicted CVD risks by estimating every individual’s 10-year CVD risk with each of the four prediction models. Implementation of a prediction model typically follows updating or recalibration of the model in the target setting, as the target cohort may differ from the original development cohort [18]. Therefore, we first recalibrated the four prediction models using the MORGEN cohort to ensure that the models provide accurate risk estimates in this cohort. For the survival data (time-to-event data) considered in this study, recalibrating a prediction model typically involves updating the baseline hazard and centering each predictor around the mean value of all patient characteristics in our empirical cohort, correcting for men and women separately [19, 20]. Furthermore, we incorporated an additional correction factor to ensure that the updated baseline hazards actually reflect the observed probability of survival after 10 years.

Many clinical guidelines advocate the use of prediction models to select individuals with a predicted risk above a certain threshold for preemptive lipid or blood pressure lowering drug treatment. Different recommendations for absolute 10-year risk thresholds were identified for each model: 10% (ATP), 10% (FRS), 7.5% (PCE), and 5% (SCORE) [9, 12, 21]. By doing this, we were able to further explore and compare the varying treatment decisions according to the four models. Finally, we first assigned individuals to treatment based on their FRS risk and the FRS risk threshold and then reassigned individuals according to their ATP, PCE, and SCORE risks, and the corresponding thresholds.

The aim of this paper was to illustrate the complexity of comparing predicted risks. This paper does not focus specifically on Dutch clinical practice and does not provide guidance on preferred prediction models for the Dutch context.

Results

Although the predictors of the four prediction models are similar, the composite endpoints vary widely and include different CVD event types (Table 1, column 1–6). Myocardial infarction (MI) is included in all four composite endpoints, either alone or in combination with other CVD event types. The endpoint defined for FRS includes the largest range of fatal and non-fatal CVD event types, whereas the endpoint defined for SCORE only includes fatal event types.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Constitution of the composite endpoints according to ATP, FRS, PCE, and SCORE and incidence of CVD events in MORGEN cohort.

https://doi.org/10.1371/journal.pone.0209314.t001

Table 1 (column 4, 6, 8, and 10) shows the incidence of each CVD event type as observed in the MORGEN cohort for the four different prediction models. Due to different composite endpoints, individuals with an earlier CVD event which was not included in the considered endpoint were not censored. Therefore, the observed event rates for a specific CVD event vary per prediction model. Definition of a first and secondary event within individuals thus depends on whether the observed CVD events for individuals are included in the composite endpoint of each prediction model. Due to the different definitions of the composite endpoint of the four prediction models, the total number of observed events for SCORE (n = 105) is almost nine times smaller than for FRS (n = 928). These differences in composite endpoints also affect the absolute number of observed events per prediction model, due to different censoring mechanisms. For example, the absolute number of fatal MIs varies per prediction model because secondary fatal MIs may be censored due to occurrence of another primary event present in the composite endpoint. To illustrate: a fatal MI following a non-fatal stroke event would be accounted for (not censored) in the SCORE and ATP model and not accounted for (censored) in the FRS and PCE model. The relative incidence of CVD event types within composite endpoints also varies substantially. For example, of the 105 events observed according to SCORE, 48 (46%) were fatal MIs, whereas the relative incidence of fatal MIs is 17%, 4%, and 11%, for ATP, FRS, and PCE, respectively. This means that the burden, or health loss, associated with the incidence of each composite endpoint varies with a) the included event types, and b) the relative incidence of these event types.

The performance of the recalibrated prediction models was good and quite similar; the c-index was 0.81, 0.78, 0.78, and 0.81 for ATP, FRS, PCE, and SCORE respectively. S2—Table 1 shows an overview of the observed and predicted number of CVD events for each of the four models. Following from this table, it is apparent that the events are well captured by the models and that the number of predicted events closely matches the observed number of events.

Fig 1 shows that the dissimilarities in composite endpoints lead to substantial variation in predicted 10-year CVD risks. Since predicted risks increase with the inclusion of more CVD event types in the composite endpoint, to an extent that depends on their absolute incidence. For example, in our cohort the broad composite endpoint used in the FRS model, covering a large range of CVD event types, yields higher risk predictions than the ATP, PCE, and SCORE models. Similarly, the narrow composite endpoint of SCORE (only fatal events), and its inherent low incidence of included event types yields the lowest risk predictions of all models considered. The average predicted risks in the MORGEN cohort are 1.2% (ATP), 5.1% (FRS), 1.9% (PCE), and 0.6% (SCORE). Hence, the differences in composite endpoints between prediction models, shown in Table 1, result in large variation in predicted CVD risks across prediction models.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1.

Predicted (absolute) CVD risk according to FRS and A) ATP, B) PCE, and C) SCORE. The red marker is the estimate of the mean predicted risk according to FRS and ATP, PCE, or SCORE. The grey lines (raster lines) represent the different risk thresholds and reveal the fraction of individuals eligible for treatment.

https://doi.org/10.1371/journal.pone.0209314.g001

Considering that the largest set of CVD event types was included by the FRS composite endpoint, we compared FRS risk estimates with risk estimates from the other three models using more narrow composite endpoints. Fig 1 shows the comparison in CVD risks and reveals an association between these risk estimates. This association indicates that, in this cohort, individuals who have the highest risk according to FRS typically also have the highest risk according to ATP, PCE, and SCORE. However, while the relative risks are similar the absolute risks are clearly different. Furthermore, the vertical spread of points in Fig 1 shows how individuals with a certain FRS risk estimate may have varying risk estimates according to the other models, due to the effect of different risk factor combinations in each model. For example, the group of individuals with an average predicted FRS risk of 10% had an average PCE risk of 3.9%, with a 95% percentile range of 2.2%-5.1% (Fig 1, plot B).

Given the variation in composite endpoints and the subsequent variations in risk predictions from the four models, selecting high risk individuals based on the corresponding recommended risk thresholds results in highly different high risk groups, identified per model. Unfortunately, the fact that each prediction model has its own associated risk threshold further complicates the interpretation and comparison of absolute predicted risks between prediction models. Consequently, treatment decisions may vary with the prediction model that is used. For example, in the MORGEN cohort these thresholds would possibly lead to a seventy-fold difference in prescription of preventive drug treatment in 0.2%, 14.4%, 4.3%, and 1.4% of all individuals, when using ATP, FRS, PCE and SCORE, respectively. To illustrate the implications of these differences, we determined the CVD risks and the consequences on treatment decisions according to the four prediction models for one individual in our cohort. Indeed, using FRS for this individual implies both a greater necessity to consider preventive drug treatment and a larger potential benefit of such treatment, compared to ATP, PCE, and SCORE (see S1 Appendix—Clinical example).

The treatment decisions based on the four risk thresholds are shown in Table 2. We found that the treatment decisions based on the different models vary widely, which is undesirable from a public health point of view. When using FRS, 2618 individuals have an estimated risk exceeding the FRS threshold and would thus be eligible for medical treatment. Of these individuals, only 32 (1.2%), 725 (27.7%), and 56 (2.1%) individuals would be considered eligible for medical treatment using the estimated risk and corresponding threshold when applying ATP, PCE, and SCORE respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Reclassification table where all individuals are classified and considered for treatment, according to the FRS risk threshold (10%), and reassigned for treatment according to the thresholds according to ATP (10%), PCE (7.5%), and SCORE (5%).

https://doi.org/10.1371/journal.pone.0209314.t002

These different decisions may be due to either the different estimated risks or due to the use of different risk thresholds for classifying individuals as high risk and thus eligible for medical treatment. In our cohort, we observed that mostly the same individuals were assigned a relatively high risk according to each of the four prediction models (Fig 1). For example, of the individuals with the highest 20% predicted risks according to FRS (n = 3621), 3106 (85.8%), 3131 (86.5%), and 861 (23.8%) of individuals were also classified as relatively high risk (top 20%) according to ATP, PCE, and SCORE, respectively. This relatively high risk group had an average CVD risk of 14.2% according to FRS, and average risks of 3.9%, 5.6%, and 0.7% according to ATP, PCE, and SCORE, respectively. None of the individuals within the top 20% risk group according to FRS had a relatively low risk (bottom 20%) according to the other models. Hence, the expected differences in treatment decisions across prediction models is mainly due to the different corresponding treatment thresholds, and their relation to predicted risks, and not due to the different classification of individuals.

Discussion

CVD risk prediction is key in providing preventive medication to large groups of individuals at intermediate or high risk of future CVD events, despite absence of specific elevated risk factors. Although PCE is often used, contemporary decision making and CVD management in the US, FRS is also applied, for example to guide pharmacotherapy for LDL-C lowering in women [9].

This paper illustrates the complexities of interpreting and comparing predicted 10-year CVD risks from four widely used CVD risk prediction models. We showed that the models vary substantially regarding their composite endpoints, and therefore also regarding their predicted absolute risks. As a result, absolute predicted 10-year risks from different prediction models cannot be compared directly and treatment decisions depend on the applied prediction model and its associated risk threshold. For example, of the high-risk individuals considered for preventive treatment according to FRS, only 1%, 28%, and 2% were eligible according to ATP, PCE, and SCORE, respectively (Table 2). Hence, the choice for a specific prediction model is very likely to impact treatment decisions in a large group of assessed individuals. Fortunately, the variation in relative predicted CVD risks is limited, implying that these prediction models rank individuals similarly regarding their CVD risk.

Conclusion

Current CVD risk prediction models vary widely in predicted outcomes, which directly impact their usefulness in clinical practice. Furthermore, this renders estimates of the population burden of CVD, and of the impact of risk-based CVD intervention strategies that highly depend on the prediction model used. Physicians, patients and health policy makers may benefit from a broader and more standardized method of defining outcomes and classification thresholds in prediction model studies.

Supporting information