To construct and validate a prediction model for individual combined benefit and harm outcomes (stroke with no major bleeding, major bleeding with no stroke, neither event, or both) in patients with atrial fibrillation (AF) with and without warfarin therapy.
Using the Kaiser Permanente Colorado databases, we included patients newly diagnosed with AF between January 1, 2005 and December 31, 2012 for model construction and validation. The primary outcome was a prediction model of composite of stroke or major bleeding using polytomous logistic regression (PLR) modelling. The secondary outcome was a prediction model of all-cause mortality using the Cox regression modelling.
We included 9074 patients with 4537 and 4537 warfarin users and non-users, respectively. In the derivation cohort (n = 4632), there were 136 strokes (2.94%), 280 major bleedings (6.04%) and 1194 deaths (25.78%) occurred. In the prediction models, warfarin use was not significantly associated with risk of stroke, but increased the risk of major bleeding and decreased the risk of death. Both the PLR and Cox models were robust, internally and externally validated, and with acceptable model performances.
In this study, we introduce a new methodology for predicting individual combined benefit and harm outcomes associated with warfarin therapy for patients with AF. Should this approach be validated in other patient populations, it has potential advantages over existing risk stratification approaches as a patient-physician aid for shared decision-making
Citation: Li G, Thabane L, Delate T, Witt DM, Levine MAH, Cheng J, et al. (2016) Can We Predict Individual Combined Benefit and Harm of Therapy? Warfarin Therapy for Atrial Fibrillation as a Test Case. PLoS ONE 11(8): e0160713. https://doi.org/10.1371/journal.pone.0160713
Editor: Johannes Boltze, Fraunhofer Research Institution of Marine Biotechnology, GERMANY
Received: March 9, 2016; Accepted: July 22, 2016; Published: August 11, 2016
Copyright: © 2016 Li et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Most of the data underlying the findings in this study is included in the manuscript and Supporting Information files clearly. However, because the dataset includes information that may compromise patient privacy, the whole dataset cannot be made publicly available. The rest of the data is available upon request from the author Thomas Delate (Kaiser Permanente Colorado Clinical Pharmacy Research Team, Aurora, CO, USA; Tom.Delate@kp.org).
Funding: This work was supported by the Father Sean O'Sullivan Research Award to GL, the Research Institute of St. Joseph’s Healthcare Hamilton, and a doctoral award from the China Scholarship Council (CSC). DMW and TD received research funding from the Kaiser Permanente Pharmacy Department. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Atrial fibrillation (AF) is a common, age-related, chronic arrhythmia that is a major risk factor for stroke and mortality [1,2]. The presence of AF increases the risk of stroke five-fold independently , and doubles the risk of death from AF-related stroke . At present, oral anticoagulants are the mainstay for stroke prophylaxis in patients with AF . Despite the growth in use of newer oral anticoagulants, warfarin remains a frequently used antithrombotic therapy for AF, where it lowers rates of stroke as well as mortality [2,4–6]. However, the use of anticoagulants also is associated with an increased risk of major bleeding including intracranial hemorrhage (ICH) . Thus, this combination of potential life-saving benefit and life-threatening harm may dissuade clinicians from prescribing warfarin for eligible patients [7–11].
Clinical prediction rules such as the CHADS2 (Congestive heart failure, Hypertension, Age > 75 years, Diabetes, Previous stroke [2 points]) and the CHA2DS2-VASc (Congestive heart failure; Hypertension; Age ≥ 75 years [2 points]; Diabetes mellitus; Stroke [2 points], Vascular disease, Age 65–74 years, and Sex category [female]) scores have been developed and widely used to predict stroke risk in AF patients [2,5,12,13]. Likewise, the HAS-BLED score (Hypertension; Abnormal renal/liver function; Stroke history; Bleeding history or predisposition; Labile international normalized ratio [INR], Elderly [>65 years]; Drugs/alcohol concomitantly) has been validated to predict risk of major bleeding with warfarin therapy [2,5,14–17]. Unfortunately, the CHADS2, CHA2DS2-VASc and HAS-BLED scores were not derived from the same patients or populations. Specifically, the CHADS2 used data from 1733 patients in the US National Registry of AF , while the CHA2DS2-VASc and HAS-BLED scores were both developed from the Euro Heart Survey on AF population but used data on 1084 and 3978 patients respectively [12,14]. Thus these scores are unable to assess simultaneously a patient’s potential for benefit and/or harm with warfarin therapy, yet this is exactly what each patient wants to know .
While the CHADS2, CHA2DS2-VASc and HAS-BLED scores help estimate an individual’s chance of benefit and harm separately, a more sophisticated methodology is needed. The ‘net benefit’ approach involves calculating the main benefit of warfarin therapy (reduced risk of stroke or systemic embolism) then deducting the main harm (weight*increased risk of ICH, weight = 1.5) in the same population [19–21]. However, this approach does not take into account gastrointestinal (GI) bleeding risk, and the weighting for ICH is chosen arbitrarily.
In general, treatment effects of warfarin therapy for individual patients can be divided into four quadrants: 1) benefit without harm; 2) harm without benefit; 3) neither benefit or harm; and 4) both benefit and harm simultaneously (Table 1). A method for predicting the probabilities of the four outcome quadrants (i.e., individualized combined benefit and harm outcomes) for each patient is needed. The polytomous logistic regression (PLR) modelling can be used for predictions due to the four multinomial levels of outcomes [22,23]. Therefore, the objective of this study was to use the PLR modelling to construct and externally validate a prediction model for patients’ individual combined benefit and harm outcomes (stroke with no major bleeding, major bleeding with no stroke, neither event, or both stroke and major bleeding) with and without warfarin therapy for AF. In real-world clinical settings, the prediction of individualized combined benefit and harm outcomes related to warfarin therapy could assist with the patient-physician shared decision-making process.
Study design and setting
The methods have been described in detail previously . Briefly, Kaiser Permanente Colorado (KPCO), a non-profit, integrated health care delivery system in the U.S. Denver-Boulder metropolitan area, utilizes a centralized anticoagulation service that provides anticoagulation services for KPCO patients with AF [24,25]. KPCO maintains extensive medical, pharmacy, laboratory, utilization, mortality, and membership electronic, integrated administrative datasets. Data were extracted for KPCO patients diagnosed with AF who were and were not prescribed warfarin therapy and analyzed at St. Joseph’s Healthcare Hamilton in Hamilton, ON. The KPCO Institutional Review Board and the Hamilton Integrated Research Ethics Board approved this study with a waiver for informed consent.
Patients newly diagnosed with AF between January 1, 2005 and December 31, 2012 were included. Newly diagnosed status was defined by absence of AF diagnosis in the previous 180 days. Patients were followed for up to 180 days after AF diagnosis to assess if warfarin therapy was initiated. Patients who had at least one warfarin purchase or no warfarin purchases were grouped as warfarin users and non-users, respectively. Warfarin non-users were randomly matched 1:1 to warfarin users on year of AF diagnosis . Patients with AF diagnosed between January 1, 2005 and December 31, 2008 comprised the derivation cohort (KPCO-I), while patients with AF diagnosed between January 1, 2009 and December 31, 2012 comprised the validation cohort (KPCO-II). Compared with internal validation by randomly splitting the entire dataset, separating derivation and validation cohorts by AF diagnosis dates enabled a external validation of the model independent of the original data and development process . In addition, the separation by dates of AF diagnoses could also account for changes in standards of care and management for patients over time.
The date of AF diagnosis for each patient was defined as study start date. To include as many outcomes as possible, study outcome end date was defined as June 30, 2009 and June 30, 2013 for the derivation and validation cohorts, respectively. To control the potential of immortal time bias, the study index date for warfarin users was defined as the first warfarin purchase date after start date [28,29]. Warfarin non-users were assigned an index date corresponding to the length of time from study start date to the index date of their randomly-matched warfarin user . Warfarin non-users who died prior to their assigned index date were excluded from the analyses, because they were unable to be chosen to enter the cohort . Patients were followed from index date until KPCO plan disenrollment, death, or study outcome end date, whichever came first .
The primary outcome was a prediction model of composite of stroke or major bleeding. The secondary outcome was a prediction model of all-cause death. All of the outcomes were assessed from the index date to outcome end date. For the prediction model of stroke or major bleeding, we categorized patients into one of the four outcome groups based on their survival time to first event: stroke with no major bleeding, major bleeding with no stroke, neither event, or both stroke and major bleeding. For the prediction model of all-cause mortality, patients were categorized into survival or non-survival groups.
Stroke and major bleeding events were identified during an ambulatory KPCO medical office visit, emergency department (ED) visit, or inpatient stay using International Classification of Disease, Ninth Revision, Clinical Modification (ICD-9-CM) codes in the primary position. Major bleeding was defined as bleeding that led to a hospital admission or an ED visit requiring a transfusion . However, bleeding that caused a drop in hemoglobin of ≥ 20g/L but did not necessitate a transfusion  was not included as major bleeding since no inpatient or ED hemoglobin laboratory values were available. ICH was categorized as major bleeding, rather than stroke. Stroke or major bleeding occurring before the index date was categorized as a risk factor (i.e., prior stroke, prior major bleeding) rather than a study outcome .
Potential predictors of benefit and/or harm
The potential predictors used in this study included patients’ demographic characteristics (i.e., sex, age), laboratory measures, baseline comorbidities, warfarin use, and concurrent use of medications that interact with warfarin. Laboratory measurements included INR, hemoglobin, serum creatinine and albumin recorded most proximal but prior to the index date. Comorbidities were from ambulatory KPCO medical office visits in the 180 days prior to the index date. Comorbidities were components of the CHA2DS2-VASc and HAS-BLED schemes, as well as components included in the Charlson Comorbidity Index . Data on warfarin use included the length of time from study start date to the first purchase date, the length of time for each dispensed warfarin prescription from index date, and days of warfarin supplied. Concurrent use of other medications included purchases for medications made during the 90 days after index date. We included concurrent medications for which there was evidence of an interaction that potentiated or inhibited the effect of warfarin. The list of included medications was from two systematic reviews that investigated warfarin interactions with other drugs [33,34].
All tests were two-sided with a significance level of 0.05, unless otherwise specified. We described continuous variables as means (+/- standard deviations [SDs]), and frequencies and percentages for categorical variables. Student’s t-tests were used to compare continuous variables and chi-square tests of associations were applied for categorical variables. In the derivation and validation cohort, we assessed the stroke and major bleeding incidence rate trends stratified by the CHA2DS2-VASc score and HAS-BLED score, respectively.
PLR modeling was used to develop a prediction model for the four individual benefit and harm outcomes using the neither event group as the referent category. Odds ratios (ORs) with 95% confidence intervals (CIs) were used to quantify the relationship between outcomes and predictors. We employed Cox proportional hazards regression analysis to build a prediction model for all-cause mortality, using hazard ratios (HRs) to quantify the associations between predictors and mortality. All of the analyses were adjusted for matching of warfarin users and non-users.
Both the PLR and Cox regression models followed the same procedures for model construction. First, the effect of multicollinearity was evaluated using the criterion of a variance inflation factor ≥ 4 to prune candidate predictors. Subsequently, we performed univariate analyses to select all possible predictors with a p-value ≤ 0.20 to enter the multivariable analyses. And then the predictors with a p-value < 0.05 in the multivariable analyses were retained in the prediction models. Lastly we identified significant two-way interactions to finalize our prediction models .
For the primary outcome, three sensitivity analyses were performed by: 1) using multiple imputations if missing data were ≥ 10%; 2) treating the use of warfarin as a time-dependent covariate to evaluate the effect of warfarin on stroke and major bleeding, using a gap of > 30 days to indicate warfarin discontinuation ; and 3) employing a competing risk analysis using the Fine and Gray method to take into account all-cause mortality as a competing risk of stroke and major bleeding .
Model performance and validation
Comparison between the predicted and observed risks in deciles was used to evaluate calibration of the prediction models. Discrimination was measured by the area under the receiver operating characteristic curves (AUCs) for the PLR model and Harrell's C index for the Cox model. Goodness-of-fit was assessed by a Hosmer-Lemeshow statistic  and Gronnesby and Borgan test  with ten groups based on the predicted risk scores for the PLR and Cox models, respectively.
Two internal validations were performed for the PLR model by using 10-fold cross-validation  and bootstrap analysis . We also used bootstrap analysis to internally validate the Cox model for all-cause mortality. For the external validation, because the incidence rates of outcomes were different from the derivation and validation cohorts and there was evidence that the original models were not a good fit to the validation cohort, we updated the models’ intercepts as well as the regression coefficients by using the calibration intercepts and calibration slopes [23,42,43]. The evaluation of goodness-of-fit, calibration, and discrimination was repeated in the validation cohort.
Analyses were performed with the software packages SAS Version 9.3 (SAS Institute, Inc., Cary, NC) and STATA Version 12 (Stata Corp., College Station, TX, USA). For the calibration plots of the PLR model, we used the software R version 3.2.1 (R Foundation for Statistical Computing, Vienna, Austria) with the Design library.
We included 9074 patients diagnosed with AF with 4537 and 4537 warfarin users and non-users, respectively (see S1 Fig for patient dispositions). Overall mean age was 71.7 years (SD: 13.0) and 46% were female (Table 2). Overall mean CHA2DS2-VASc and HAS-BLED scores were 2.99 (SD: 1.56) and 1.73 (SD: 0.88), respectively.
The derivation cohort (KPCO-I) included 4632 patients with a median follow-up of 652 days, while the validation cohort (KPCO-II) included 4442 patients with a median follow-up of 628 days (Table 2). In the KPCO-I cohort, warfarin users were significantly older and had higher proportions of patients with congestive heart failure, hypertension, renal disease, prior major bleeding, anemia, and alcohol abuse than non-users (all p < 0.05). The CHA2DS2-VASc (mean 3.09 versus 2.73) and HAS-BLED (mean 1.80 versus 1.54) scores were higher in warfarin users. A higher proportion of warfarin users had purchased concurrently an NSAID, antibiotic, cardiac drug, GI drug, and other drug (tramadol) than non-users. However, a lower percentage of antiplatelet use was observed in warfarin users compared with non-users (p = 0.001). Similar characteristics and comparison between warfarin users and non-users were found in the KPCO-II cohort (Table 2). S1 Table presents the comparison between warfarin users and non-users in the whole cohort (i.e., KPCO-I combined with KPCO-II), with similar results to findings as those from the KPCO-I cohort alone.
Twenty-eight patients (12 and 16 in the KPCO-I and KPCO- II cohorts, respectively) had a stroke and major bleeding outcome on the same date; thus, their time to first event could not be identified. Because of the low frequency, these patients were randomly allocated into either stroke with no major bleeding (n = 14) or major bleeding with no stroke (n = 14). Therefore, in the combined cohort there were 278 strokes (3.06%), 453 major bleedings (4.99%) and 2186 deaths (24.09%) occurred during follow-up. Of these, 136 strokes (2.94%), 280 major bleedings (6.04%) and 1194 deaths (25.78%) occurred in the KPCO-I cohort. In both the KPCO-I and KPCO- II cohorts, the rates of major bleeding and death, but not stroke, differed between warfarin users and non-users (Table 3). Also, as shown in S2 Fig, there was a significant difference in all-cause mortality (log-rank p-value = 0.001) between the KPCO-I cohort and KPCO-II cohort.
Significant trends for increasing stroke and major bleeding rates with higher CHA2DS2-VASc and HAS-BLED scores were found (p < 0.001) for both the KPCO-I and KPCO-II cohorts (S2 Table).
The PLR model included age, female sex, warfarin use, CHF, other cerebrovascular disease, hypertension, diabetes, prior major bleeding, prior stroke, renal disease, and concurrent use of antibiotics, antiplatelets, and GI drugs (Table 4). Warfarin use was not associated with stroke (OR = 0.94, 95% CI: 0.66–1.34) but was associated with increased risk of major bleeding (OR = 1.71, 95% CI: 1.32–2.22). All other predictors in the model were associated with an increased risk of outcomes, except hypertension (OR = 0.88, 95% CI: 0.56–1.39) and antibiotic use (OR = 0.98, 95% CI: 0.64–1.51) for stroke, and female sex (OR = 0.73, 95% CI: 0.56–0.94), hypertension (OR = 0.94, 95% CI: 0.66–1.33), and prior stroke (OR = 0.73, 95% CI: 0.40–1.34) for major bleeding.
The all-cause mortality model included age, warfarin, anemia, other cerebrovascular disease, CHF, diabetes, hypertension, prior major bleeding, malignancy, and concurrent use of antifungals and antidepressants (Table 5). Warfarin use was associated with a decreased risk of death (HR = 0.55, 95% CI: 0.49–0.62). All other predictors were associated with increased risk of death except hypertension (HR = 0.76, 95% CI: 0.66–0.85).
When warfarin use was treated as a time-dependent covariate, similar associations between warfarin and outcomes were found as in the PLR model for stroke and major bleeding and the Cox model for all-cause mortality (S3 Table). Results from the competing risk sensitivity analysis for stroke and major bleeding identified similar coefficients for all the predictors included in the PLR model, indicating the robustness of the prediction model (Table 6).
Model performance and validation
The prediction models had a good fit to the data in the derivation cohort (p > 0.05) (Table 7). The discrimination of the models (AUC = 0.71 and 0.72 for stroke and major bleeding, respectively, and C index = 0.75 for all-cause mortality) were acceptable. The overall calibration of the PLR model (S3 Fig and S4 Fig) and the Cox model (S5 Fig) was satisfactory. Bootstrap analyses for the PLR model and the Cox model yielded the same predictors and similar coefficients as the original models, indicating internal model validation (Tables 4 and 5). Findings from 10-fold cross-validation also produced similar AUCs to the original PLR model: 0.69 for stroke and 0.71 for major bleeding (Table 7). For external validation in the KPCO-II cohort, the models’ intercepts and the regression coefficients were updated (S4 Table). Results of the model goodness-of-fit test (Table 7), discrimination (Table 7) and calibration (S6, S7 and S8 Figs supported external validation for the PLR and Cox models.
In this study of patients diagnosed with AF who were and were not initiated on warfarin therapy, we present a new methodology to predict individual combined benefit and harm outcomes of warfarin therapy. We utilized a PLR model to predict the individual benefit and harm outcomes due to its simplicity and flexibility, especially in predictor selection [22,23]. The PLR modelling can incorporate individual baseline characteristics of patients to estimate individual probabilities of the combined benefit and harm outcomes. Compared with the decision tree model which is another commonly-used method for prediction building, the PLR models have shown greater discrimination and predictive accuracy [44–49].
We found that warfarin use, age, female sex, CHF, other cerebrovascular disease, hypertension, diabetes, prior major bleeding, prior stroke, renal disease, and concurrent use of antibiotics, antiplatelets, and GI drugs were included in the PLR model for stroke and major bleeding. Our model performance was acceptable and robust. Using the predictors we identified, the estimated probabilities of the potential outcomes can be computed. For example, if an 82 year-old woman taking warfarin had CHF, diabetes, renal disease and prior major bleeding, and used GI medications concurrently with warfarin, then her log(stroke/neither event) would be -0.85, and log(major bleeding/neither event) would be -0.33, respectively. Subsequently, her estimated 3-year probability of stroke would be: = 19.9%, her probability of major bleeding would be: = 33.6%, and her probability of neither event would be: = 46.5% . By contrast if she did not start warfarin therapy but all other factors were the same, her estimated probability of stroke, major bleeding and neither event would be 24.3%, 22.5% and 53.2%, respectively. Likewise, her estimated 3-year probability of all-cause mortality with and without warfarin therapy initiation would be 6.9% and 24.4% respectively, using the Cox model.
In our prediction models, warfarin was associated with an increased risk of major bleeding and decreased risk of death, which is in accordance with previous findings [50,51]. However, we did not identify an association between warfarin use and decreased risk of stroke. A possible explanation for this unexpected observation might include lack of INR control measures, such as time in therapeutic range (TTR), in our prediction models. Prior research indicates that the full benefit of stroke risk reduction may require an individual TTR of at least 70% in warfarin users . However, individual TTR results for patients in our cohorts could not be included in the models since warfarin non-users were unmeasured on this factor. Another possible explanation relates to our use of ICD-9-CM codes alone to identify stroke and bleeding outcomes without confirmatory chart review. The positive predictive values of ICD-9-CM codes for bleeding have been shown to be higher than those for stroke [53,54]; thus, the use of ICD-9-CM codes alone may have provided a high rate of stroke false positives. In addition, a stroke history may have increased the likelihood that a given patient received warfarin to prevent further stroke risk and concurrently increased the likelihood that false positive stroke ICD-9-CM codes were identified during administrative data acquisition.
The CHADS2/CHA2DS2-VASc, and HAS-BLED scores are used worldwide in patients with AF to stratify the risk of stroke and major bleeding, respectively. However, these risk-stratification tools cannot provide the individual combined benefit and harm assessments needed by patients and physicians at inception of warfarin therapy or when concerns arise during ongoing use. Moreover, concerns have been expressed about their scoring algorithms and poor discrimination [55–59]. For instance, in one study compared with their peers with a CHA2DS2-VASc score of 0 and 1 for men and women, respectively, the unequal risk of stroke for the additional risk factors resulted in different weighting in the scoring algorithm. This corresponded to a HR of from 1.68 with vascular disease to 3.09 with an age of 65–74 years for men and a HR of from 1.71 with hypertension to 3.03 with an age of 65–74 years for women . Therefore given the potential different weighting for individual components of the scores as well as more detailed information provided by the individual components, we used individual risk factors, rather than gross risk scores, in our model construction.
Other studies have used the ‘net benefit’ approach of considering stroke and major bleeding outcomes simultaneously [19–21]. Unfortunately, GI bleeding risk was not considered, and the weighting factor reflecting the importance of ICH was chosen subjectively and arbitrarily in these studies. Additionally, while some studies have combined stroke and bleeding risk-stratification scores to calculate overall clinical outcome risks including stroke and major bleeding [60,61], they did not improve prediction of stroke and major bleeding beyond the individual stroke (CHADS2, CHA2DS2-VASc) or bleeding scores (HAS-BLED) . In contrast, our study may provide insights into using a new methodology to take into account individual benefit-harm outcomes with warfarin therapy. Our PLR model calculates the specific probabilities of stroke and major bleeding at the same time, which may be more practical and acceptable in real-world clinical practice compared with using separate stroke and bleeding risk-stratification scores. Moreover, because our model produces individualized risk estimates for each patient based on various characteristics, it offers more personalized and detailed information for patients with AF rather than the population-level estimates associated with CHADS2, CHA2DS2-VASc, and HAS-BLED scores . Thus the PLR model may better facilitate patient-physician shared decision-making with regard to warfarin therapy initiation.
In our study, an unexpected inverse association between comorbid hypertension and stroke, major bleeding, and all-cause death was observed. During the model construction, we used either the ICD-9-CM codes or the antihypertensive drug surrogates including angiotensin-converting enzyme inhibitors, angiotensin II receptor blockers, thiazides, beta-blockers, calcium channel blockers, and other antihypertensive purchases, to identify hypertension comorbidity (S5 Table). Additionally, we ran two post-hoc sensitivity analyses using different methods to imply hypertension diagnosis: ICD-9-CM codes only, and both ICD-9-CM codes and antihypertensive drug purchases. These two methods yielded the same predictors included in the PLR and Cox model with extremely similar coefficients (S6 Table). Moreover, removing hypertension from the model entirely also yielded similar results (S7 Table for the PLR model; S8 Table for the Cox model). Therefore, the unexpected relationship between hypertension and outcomes requires further exploration.
The strengths of our study include the use of a large sample of patients with AF to construct and validate the prediction model. Moreover, model building, assessment, and validation included rigorous and detailed statistical analyses. Another strength is the efforts in controlling bias in study design and data analyses to preclude misleading predictors from being included into the models. Nevertheless, our study also has several limitations. The majority of the data used in this study were from ICD-9-CM codes only without confirmatory chart review of the diagnosis. Thus data accuracy for baseline comorbidities may be less than optimal. Likewise, the incidence rates of stroke and major bleeding may be over- or underestimated. This could lead to false positive/negative values and weaken the findings based on the data. Additionally, we intended to predict four outcome quadrants (Table 1). However, the number of patients experiencing simultaneous stroke and major bleeding (n = 28) was insufficient for model construction. Another limitation is lack of data from contemporary non-KPCO cohorts for model validation; thereby, potentially limiting the generalizability of the prediction model .
In this study, we introduce a new methodology for predicting individual combined benefit and harm outcomes associated with warfarin therapy for patients with AF. Should this approach be validated in other patient populations, it has potential advantages over existing risk stratification approaches as a patient-physician aid for shared decision-making.
S1 Fig. Flow diagram of selecting patients for analyses.
S2 Fig. Kaplan-Meier survival curves for death in the derivation and validation cohorts.
S3 Fig. Calibration curve in the PLR model for stroke in the derivation cohort.
S4 Fig. Calibration curve in the PLR model for major bleeding in the derivation cohort.
S5 Fig. Calibration curve in the Cox model for death in the derivation cohort.
S6 Fig. Calibration curve in the PLR model for stroke in the validation cohort.
S7 Fig. Calibration curve in the PLR model for major bleeding in the validation cohort.
S8 Fig. Calibration curve in the Cox model for death in the validation cohort.
S1 Table. Characteristics of study patients stratified by taking versus not taking warfarin for the whole cohort.
S2 Table. Rates of stroke and major bleeding in the KPCO cohorts stratified by CHA2DS2-VASc and HAS-BLED scores.
S3 Table. Sensitivity analysis results from multivariable model to assess time-varying effect of warfarin on stroke, major bleeding and death.
S4 Table. Updates of the models’ intercepts and the regression coefficients for external validation in the KPCO-Ⅱ cohort.
S5 Table. Hypertensive drugs as surrogates for hypertension.
S6 Table. Results for effect of hypertension in the PLR and Cox model using different data on hypertension in the KPCO-I cohort.
S7 Table. Sensitivity analysis leaving hypertension out of the PLR model for stroke and major bleeding in the KPCO-I cohort.
We thank Dr. Jennifer A. Pereira for her substantial preparatory contribution in her unpublished doctoral thesis  to this study.
- Conceived and designed the experiments: GL LT AH.
- Analyzed the data: GL JC.
- Contributed reagents/materials/analysis tools: TD.
- Wrote the paper: GL LT TD DMW AH.
- Critical revisions of the manuscript: TD DMW MAHL.
- 1. Rockson SG, Albers GW (2004) Comparing the guidelines: anticoagulation therapy to optimize stroke prevention in patients with atrial fibrillation. J Am Coll Cardiol 43: 929–935. pmid:15028346
- 2. Camm AJ, Kirchhof P, Lip GY, Schotten U, Savelieva I, Ernst S, et al. (2010) Guidelines for the management of atrial fibrillation: the Task Force for the Management of Atrial Fibrillation of the European Society of Cardiology (ESC). Eur Heart J 31: 2369–2429. pmid:20802247
- 3. Lip GY, Tse HF, Lane DA (2012) Atrial fibrillation. Lancet 379: 648–661. pmid:22166900
- 4. Hart RG, Halperin JL, Pearce LA, Anderson DC, Kronmal RA, McBride R, et al. (2003) Lessons from the Stroke Prevention in Atrial Fibrillation trials. Ann Intern Med 138: 831–838. pmid:12755555
- 5. Cairns JA, Connolly S, McMurtry S, Stephenson M, Talajic M (2011) Canadian Cardiovascular Society atrial fibrillation guidelines 2010: prevention of stroke and systemic thromboembolism in atrial fibrillation and flutter. Can J Cardiol 27: 74–90. pmid:21329865
- 6. Wells G, Coyle D, Cameron C, Steiner S, Coyle K, Kelly S, et al. Safety, effectiveness, and cost-effectiveness of new oral anticoagulants compared with warfarin in preventing stroke and other cardiovascular events in patients with atrial fibrillation [Internet]. Available from: http://www.cadth.ca/media/pdf/NOAC_Therapeutic_Review_final_report.pdf. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2012 Apr 9 173 p (CADTH Therapeutic Review).
- 7. Holbrook A, Labiris R, Goldsmith CH, Ota K, Harb S, Sebaldt RJ (2007) Influence of decision aids on patient preferences for anticoagulant therapy: a randomized trial. CMAJ 176: 1583–1587. pmid:17515584
- 8. Ansell J (2010) Warfarin versus new agents: interpreting the data. Hematology Am Soc Hematol Educ Program 2010: 221–228. pmid:21239798
- 9. Arepally G, Bauer KA, Bhatt DL, Merli GJ, Naccarelli GV, Carter RD, et al. (2010) The use of antithrombotic therapies in the prevention and treatment of arterial and venous thrombosis: a survey of current knowledge and practice supporting the need for clinical education. Crit Pathw Cardiol 9: 41–48. pmid:20215910
- 10. McCrory DC, Matchar DB, Samsa G, Sanders LL, Pritchett EL (1995) Physician attitudes about anticoagulation for nonvalvular atrial fibrillation in the elderly. Arch Intern Med 155: 277–281. pmid:7832599
- 11. Holbrook A, Pullenayegum E, Troyan S, Nikitovic M, Crowther M (2013) Personalized Benefit-Harm Information Influences Patient Decisions Regarding Warfarin. J Popul Ther Clin Pharmacol 20: e406–e415.
- 12. Lip GY, Nieuwlaat R, Pisters R, Lane DA, Crijns HJ (2010) Refining clinical risk stratification for predicting stroke and thromboembolism in atrial fibrillation using a novel risk factor-based approach: the euro heart survey on atrial fibrillation. Chest 137: 263–272. pmid:19762550
- 13. Gage BF, Waterman AD, Shannon W, Boechler M, Rich MW, Radford MJ (2001) Validation of clinical classification schemes for predicting stroke: results from the National Registry of Atrial Fibrillation. JAMA 285: 2864–2870. pmid:11401607
- 14. Pisters R, Lane DA, Nieuwlaat R, de Vos CB, Crijns HJ, Lip GY (2010) A novel user-friendly score (HAS-BLED) to assess 1-year risk of major bleeding in patients with atrial fibrillation: the Euro Heart Survey. Chest 138: 1093–1100. pmid:20299623
- 15. Lip GY (2011) Implications of the CHA(2)DS(2)-VASc and HAS-BLED Scores for thromboprophylaxis in atrial fibrillation. Am J Med 124: 111–114. pmid:20887966
- 16. Olesen JB, Lip GY, Hansen PR, Lindhardsen J, Ahlehoff O, Andersson C, et al. (2011) Bleeding risk in 'real world' patients with atrial fibrillation: comparison of two established bleeding prediction schemes in a nationwide cohort. J Thromb Haemost 9: 1460–1467. pmid:21624047
- 17. Lip GY, Frison L, Halperin JL, Lane DA (2011) Comparative Validation of a Novel Risk Score for Predicting Bleeding Risk in Anticoagulated Patients With Atrial FibrillationThe HAS-BLED (Hypertension, Abnormal Renal/Liver Function, Stroke, Bleeding History or Predisposition, Labile INR, Elderly, Drugs/Alcohol Concomitantly) Score. Journal of the American College of Cardiology 57: 173–180. pmid:21111555
- 18. Li G, Holbrook A, Delate T, Witt DM, Levine MA, Thabane L (2015) Prediction of individual combined benefit and harm for patients with atrial fibrillation considering warfarin therapy: a study protocol. BMJ Open 5: e009518. pmid:26546146
- 19. Singer DE, Chang Y, Fang MC, Borowsky LH, Pomernacki NK, Udaltsova N, et al. (2009) The net clinical benefit of warfarin anticoagulation in atrial fibrillation. Ann Intern Med 151: 297–305. pmid:19721017
- 20. Friberg L, Rosenqvist M, Lip GY (2012) Net clinical benefit of warfarin in patients with atrial fibrillation: a report from the Swedish atrial fibrillation cohort study. Circulation 125: 2298–2307. pmid:22514252
- 21. Olesen JB, Lip GY, Lindhardsen J, Lane DA, Ahlehoff O, Hansen ML, et al. (2011) Risks of thromboembolism and bleeding with thromboprophylaxis in patients with atrial fibrillation: A net clinical benefit analysis using a 'real world' nationwide cohort study. Thromb Haemost 106: 739–749. pmid:21789337
- 22. BECG CB, Gray R (1984) Calculation of polychotomous logistic regression parameters using individualized regressions. Biometrika 71: 11–18.
- 23. Wijesinha A, Begg CB, Funkenstein HH, McNeil BJ (1982) Methodology for the differential diagnosis of a complex data set. A case study using data from routine CT scan examinations. Medical decision making: an international journal of the Society for Medical Decision Making 3: 133–154.
- 24. Pereira JA (2008 (unpublished doctoral dissertation)) Methods to predict individualized combined benefit/harm patient profiles for warfarin [Unpublished doctoral thesis]. Toronto, Canada: University of Toronto.
- 25. Witt DM, Sadler MA, Shanahan RL, Mazzoli G, Tillman DJ (2005) Effect of a centralized clinical pharmacy anticoagulation service on the outcomes of anticoagulation therapy. Chest 127: 1515–1522. pmid:15888822
- 26. Zhou Z, Rahme E, Abrahamowicz M, Pilote L (2005) Survival bias associated with time-to-treatment initiation in drug effectiveness evaluation: a comparison of methods. American journal of epidemiology 162: 1016–1023. pmid:16192344
- 27. Altman DG, Vergouwe Y, Royston P, Moons KG (2009) Prognosis and prognostic research: validating a prognostic model. BMJ: British Medical Journal 338: 1432–1435.
- 28. Suissa S (2008) Immortal time bias in pharmacoepidemiology. American journal of epidemiology 167: 492–499. pmid:18056625
- 29. Lévesque LE, Hanley JA, Kezouh A, Suissa S (2010) Problem of immortal time bias in cohort studies: example using statins for preventing progression of diabetes. Bmj 340: 907–911.
- 30. Witt DM, Delate T, Clark NP, Martell C, Tran T, Crowther M, et al. (2009) Outcomes and predictors of very stable INR control during chronic anticoagulation therapy. Blood 114: 952–956. pmid:19439733
- 31. Schulman S, Kearon C (2005) Definition of major bleeding in clinical investigations of antihemostatic medicinal products in non-surgical patients. J Thromb Haemost 3: 692–694. pmid:15842354
- 32. Charlson ME, Pompei P, Ales KL, MacKenzie CR (1987) A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis 40: 373–383. pmid:3558716
- 33. Holbrook AM, Pereira JA, Labiris R, McDonald H, Douketis JD, Crowther M, et al. (2005) Systematic overview of warfarin and its drug and food interactions. Arch Intern Med 165: 1095–1106. pmid:15911722
- 34. Holbrook A, Schulman S, Witt DM, Vandvik PO, Fish J, Kovacs MJ, et al. (2012) Evidence-based management of anticoagulant therapy: Antithrombotic Therapy and Prevention of Thrombosis: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines. Chest 141: e152S. pmid:22315259
- 35. McWilliams A, Tammemagi MC, Mayo JR, Roberts H, Liu G, Soghrati K, et al. (2013) Probability of Cancer in Pulmonary Nodules Detected on First Screening CT. New England Journal of Medicine 369: 910–919. pmid:24004118
- 36. Fisher LD, Lin DY (1999) Time-dependent covariates in the Cox proportional-hazards regression model. Annual review of public health 20: 145–157. pmid:10352854
- 37. Fine JP, Gray RJ (1999) A proportional hazards model for the subdistribution of a competing risk. Journal of the American Statistical Association 94: 496–509.
- 38. Hosmer DW, Lemesbow S (1980) Goodness of fit tests for the multiple logistic regression model. Communications in statistics-Theory and Methods 9: 1043–1069.
- 39. Gronnesby JK, Borgan O (1996) A method for checking regression models in survival analysis based on the risk score. Lifetime Data Anal 2: 315–328. pmid:9384628
- 40. Kohavi R. A study of cross-validation and bootstrap for accuracy estimation and model selection; 1995. pp. 1137–1145.
- 41. Efron B, Tibshirani RJ (1994) An introduction to the bootstrap: CRC press.
- 42. Moons KG, Kengne AP, Grobbee DE, Royston P, Vergouwe Y, Altman DG, et al. (2012) Risk prediction models: II. External validation, model updating, and impact assessment. Heart 98: 691–698. pmid:22397946
- 43. Janssen KJ, Moons KG, Kalkman CJ, Grobbee DE, Vergouwe Y (2008) Updating methods improved the performance of a clinical prediction model in new patients. J Clin Epidemiol 61: 76–86. pmid:18083464
- 44. Ho WH, Lee KT, Chen HY, Ho TW, Chiu HC (2012) Disease-free survival after hepatic resection in hepatocellular carcinoma patients: a prediction approach using artificial neural network. PLoS One 7: e29179. pmid:22235270
- 45. Bilen K, Ponzer S, Ottosson C, Castren M, Pettersson H (2013) Deliberate self-harm patients in the emergency department: who will repeat and who will not? Validation and development of clinical decision rules. Emerg Med J 30: 650–656. pmid:22962049
- 46. Brasier AR, Ju H, Garcia J, Spratt HM, Victor SS, Forshey BM, et al. (2012) A three-component biomarker panel for prediction of dengue hemorrhagic fever. Am J Trop Med Hyg 86: 341–348. pmid:22302872
- 47. Ferreira D, Oliveira A, Freitas A (2012) Applying data mining techniques to improve diagnosis in neonatal jaundice. BMC Med Inform Decis Mak 12: 143. pmid:23216895
- 48. Cheong JY, Um SH, Seo YS, Shin SS, Park RW, Kim DJ, et al. (2012) A practical scoring system for predicting cirrhosis in patients with chronic viral hepatitis. Hepatogastroenterology 59: 2592–2597. pmid:23178625
- 49. Minne L, Eslami S, de Keizer N, de Jonge E, de Rooij SE, Abu-Hanna A (2012) Statistical process control for monitoring standardized mortality ratios of a classification tree model. Methods Inf Med 51: 353–358. pmid:22773038
- 50. Hart RG, Pearce LA, Aguilar MI (2007) Meta-analysis: antithrombotic therapy to prevent stroke in patients who have nonvalvular atrial fibrillation. Ann Intern Med 146: 857–867. pmid:17577005
- 51. Wysowski DK, Nourjah P, Swartz L (2007) Bleeding complications with warfarin use: a prevalent adverse effect resulting in regulatory action. Arch Intern Med 167: 1414–1419. pmid:17620536
- 52. Morgan CL, McEwan P, Tukiendorf A, Robinson PA, Clemens A, Plumb JM (2009) Warfarin treatment in patients with atrial fibrillation: observing outcomes associated with varying levels of INR control. Thromb Res 124: 37–41. pmid:19062079
- 53. Olson KL, Wood MD, Delate T, Lash LJ, Rasmussen J, Denham AM, et al. (2014) Positive predictive values of ICD-9 codes to identify patients with stroke or TIA. Am J Manag Care 20: e27–34. pmid:24738552
- 54. Cunningham A, Stein CM, Chung CP, Daugherty JR, Smalley WE, Ray WA (2011) An automated database case definition for serious bleeding related to oral anticoagulant use. Pharmacoepidemiol Drug Saf 20: 560–566. pmid:21387461
- 55. Ince B, Benbir G, Gozubatik-Celik G (2014) Should HAS-BLED scoring be revised for better risk estimation in patients with intracerebral hemorrhage? Expert Rev Cardiovasc Ther 12: 929–931. pmid:25017498
- 56. Poli D, Antonucci E, Testa S, Cosmi B, Palareti G, Ageno W (2013) The predictive ability of bleeding risk stratification models in very old patients on vitamin K antagonist treatment for venous thromboembolism: results of the prospective collaborative EPICA study. J Thromb Haemost 11: 1053–1058. pmid:23578305
- 57. Chao TF, Liu CJ, Wang KL, Lin YJ, Chang SL, Lo LW, et al. (2015) Should atrial fibrillation patients with 1 additional risk factor of the CHA2DS2-VASc score (beyond sex) receive oral anticoagulation? J Am Coll Cardiol 65: 635–642. pmid:25677422
- 58. Donze J, Rodondi N, Waeber G, Monney P, Cornuz J, Aujesky D, et al. (2012) Scores to predict major bleeding risk during oral anticoagulation therapy: a prospective validation study. Am J Med 125: 1095–1102. pmid:22939362
- 59. Olesen JB, Lip GY, Hansen ML, Hansen PR, Tolstrup JS, et al. (2011) Validation of risk stratification schemes for predicting stroke and thromboembolism in patients with atrial fibrillation: nationwide cohort study. BMJ 342: d124. pmid:21282258
- 60. Lip GY, Lane DA, Buller H, Apostolakis S (2013) Development of a novel composite stroke and bleeding risk score in patients with atrial fibrillation: the AMADEUS Study. Chest 144: 1839–1847. pmid:24009027
- 61. Banerjee A, Fauchier L, Bernard-Brunet A, Clementy N, Lip GY (2014) Composite risk scores and composite endpoints in the risk prediction of outcomes in anticoagulated patients with atrial fibrillation. The Loire Valley Atrial Fibrillation Project. Thromb Haemost 111: 549–556. pmid:24452108
- 62. Dzeshka MS, Lane DA, Lip GY (2014) Stroke and bleeding risk in atrial fibrillation: navigating the alphabet soup of risk-score acronyms (CHADS2, CHA2 DS2 -VASc, R2 CHADS2, HAS-BLED, ATRIA, and more). Clin Cardiol 37: 634–644. pmid:25168181