Study design and cohorts

For the creation of a predictive model, we retrieved data from all patients undergoing non-cardiac surgery between January 2008 and August 2019 at The First Medical Center of Chinese PLA General Hospital (n =  376,933 total qualified patients). For external validation, we used data of non-cardiac surgical inpatients from Nanfang Hospital (between January 2019 and October 2021, n =  62,407 total qualified patients) and Henan Provincial People’s Hospital (between December 2014 and June 2021, n =  412,043 total qualified patients). The current study included all the non-cardiac surgeries during the research period, such as ear, nose and throat, obstetrics and gynecology, abdominal surgery, orthopedics, stomatology, urology, general surgery, neurosurgery, thoracic surgery, vascular surgery, and other surgeries (trauma surgery and plastic surgery). To avoid repeated data use, for patients with multiple surgeries during one hospitalization period, only the first surgery was included in our analysis. Exclusion criteria were patients under the age of 18 years old, with surgery duration of less than one hour, those who underwent regional anesthesia, American Society of Anesthesiologists (ASA) classification ≥ V, or high levels (55% or above) of missing variables. The external cohorts followed the same exclusion criteria as the development cohort.

Outcome

The outcome of interest was a perioperative ischemic stroke, which was defined as a new-onset brain infarction during their hospital stay within 30 days following surgery (excluding the day of surgery). Neuroimaging was performed on patients suspected with cerebral stroke when typical symptoms and signs appeared after surgery. Stroke diagnoses were confirmed by a combination of neuroimaging and clinical evidence of cerebrovascular ischemia as identified through International Classification of Diseases 9 (ICD9)/ICD10 diagnosis codes (S2 Table). The outcome variable was validated through a medical record review by the neurologists.

Predictor variables

We specified candidate predictor variables based on published studies and established clinical knowledge including baseline characteristics, comorbidities, pre-operative medication, pre-operative laboratory tests, and surgery-related variables. Pre-operative laboratory tests were performed by the central laboratory. Preoperative mean arterial pressure (MAP) was calculated based on the pressure value measured at the time of preoperative evaluation. Variables with ≥20% missing data were excluded. We used multiple imputations to impute missing values for variables with <20% missing data by the classification and regression tree (cart) method.

Model development and validation

The prediction model was constructed based on the data from The First Medical Center of Chinese PLA General Hospital. Patients were randomly divided into a training dataset and an internal validation dataset with a split ratio of 70% and 30% of total patients. The training dataset was used to develop the prediction model in the final logistic regression, whereas the validation dataset was used for internal validation. A univariate analysis was performed, and candidate variables with a P-value less than 0.05 were included in the multi-variable model.

Dependent variables including age, ASA classification, body mass index (BMI), hypertension, diabetes mellitus, myocardial infarction, coronary heart disease, heart failure, atrial fibrillation, previous stroke, valvular heart disease, angina pectoris, peripheral vascular disease, malignant tumor, preoperative serum albumin, preoperative fasting plasma glucose (FPG), preoperative angiotensin-converting enzyme inhibitors (ACEIs), preoperative angiotensin II receptor blockers (ARB), preoperative steroids, preoperative β-blockers, preoperative calcium channel blockers, preoperative MAP, emergency surgery, surgery type, surgery length, preoperative neutrophil-lymphocyte ratio (NLR), preoperative platelet-to-lymphocyte ratio (PLR), preoperative fibrinogen to albumin ratio (FAR), blood product usage, crystals volume, and opioids dosage were collected for the multi-variate analysis.

Stepwise elimination was performed with P < 0.05 for retaining a given predictor in the model. But, for preoperative β-blockers (P = 0.093) and preoperative steroids (P = 0.057), we included the two variables in the final model, considering the clinical significance. We also assessed potential multi-collinearity amongst all predictors in the final model using variance inflation factors (VIFs), and VIF less than 4 was considered to represent no significant collinearity. Model discrimination was measured using the area under the receiver operating characteristic curve (AUC). Model calibration was assessed using the Hosmer–Lemeshow goodness of fit test. Decision curve analysis (DCA) was used to demonstrate the net benefits (NBs) with each threshold probability.

External validation was performed using patients’ data from Nanfang Hospital and Henan Provincial People’s Hospital. The training dataset was also used to validate the existing prognostic model proposed by Mashour and colleagues [5], the ATRIA stroke risk score [14], the model proposed by Lip and colleagues [15], the model proposed by Wolf and colleagues [16], and the model proposed by Woo and colleagues [10]. The prediction model was only compared with the ATRIA stroke risk score and the model proposed by Lip and colleagues using the data of Nanfang Hospital and Henan Provincial People’s Hospital, as the prediction variables—current smoker, chronic obstructive pulmonary disease, hematocrit, and preoperative serum sodium in the other three models were absent in the two external cohorts.

Statistical analyses

The risk prediction model was built based on the data set from The First Medical Center of Chinese PLA General Hospital (n = 223,415). The data from Nanfang Hospital (n = 37,568) and Henan Provincial People’s Hospital (n = 48,719) were used to validate the model. Predictor variables were compared using χ² (n ≥ 5) or Fisher’s exact tests (n < 5) for categorical variables. Normally distributed continuous variables were compared using analysis of variance, and skewed continuous variables were compared using Kruskal–Wallis tests. Categorical data were reported as frequencies (percentages), and continuous variables were reported as medians (quartiles). We constructed the logistic regression model with perioperative ischemic stroke as the dependent variable. Data analyses were performed in R version 4.1.2 with R studio, along with the stringr, mice, car, rms, MASS, pROC, nomogramFormula, verification, ResourceSelection, and rmda packages.

Study cohorts and perioperative stroke rate

We screened a total of 376,933, 62,304, and 412,043 patients at The First Medical Center of Chinese PLA General Hospital, Nanfang Hospital, and Henan Provincial People’s Hospital, which yielded 223,415, 37,568, and 48,719 eligible patients respectively, aged from 18 to 99 years old (S1 Fig). A total of 525 (0.23% of 223,415), 122 (0.32% of 37,568), and 173 (0.36% of 48,719) patients experienced perioperative ischemic stroke, respectively. Patients that developed perioperative ischemic stroke were generally older and experienced longer surgery duration in comparison to those who did not suffer perioperative ischemic stroke (64-years old versus 52-years old, P < 0.001; 195 min versus 148 min, P < 0.001, respectively). The prevalence of hypertension, diabetes, and previous ischemic stroke and peripheral vascular disease was also higher in patients experiencing perioperative stroke (P < 0.001 for each comparison). The baseline characteristics of the participants from The First Medical Center of Chinese PLA General Hospital are shown in Table 1. The baseline characteristics of the participants from Nanfang Hospital and Henan Provincial People’s Hospital are shown in S3 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Patient characteristics in the development cohorts.

https://doi.org/10.1371/journal.pmed.1004539.t001

Analysis of risk factors

Univariate logistic regression analysis demonstrated that risk factors for perioperative ischemic stroke included patient age, ASA classification, BMI, hypertension, diabetes mellitus, myocardial infarction, coronary heart disease, heart failure, atrial fibrillation, previous stroke, valvular heart disease, angina pectoris, peripheral vascular disease, malignant tumor, preoperative serum albumin, preoperative FPG, preoperative ACEIs, preoperative ARB, preoperative steroids, preoperative β-blockers, preoperative calcium channel blockers, preoperative MAP, emergency surgery, surgery type, surgery length, preoperative NLR, preoperative PLR, preoperative FAR, blood product usage, crystals volume, and opioids dosage (S4 Table).

Prediction model for perioperative stroke

A total of 13 factors were classified as independent predictors of perioperative ischemic stroke in the final model. These factors were: patient age, ASA classification, hypertension, previous stroke, valvular heart disease, preoperative steroid hormones, preoperative β-blockers, preoperative MAP, preoperative FAR, preoperative FPG, emergency surgery, surgery type, and surgery length (Table 2). As analyses were adjusted in the final model, we detailed the unadjusted analyses in S5 Table. Furthermore, the rejected variables are detailed in S6 Table. The perioperative ischemic stroke predictors were incorporated into the predictive nomogram (S2 Fig). For each variable, there is a corresponding score on the top line, and the total points could be calculated to indicate the probability of perioperative ischemic stroke (S7 Table). The best cutoff value was determined by the maximum value of Youden index, of which the sensitivity was 0.874 and the specificity was 0.544. The total points above the cutoff value were classified as high risk, while the results below the cut-off value were deemed low risk. The cutoff value of high-risk probability was 144.2 points, and the corresponding percentage chance of developing stroke was 0.2%.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Variables for perioperative stroke in final multi-variable logistic regression model.

https://doi.org/10.1371/journal.pmed.1004539.t002

Model efficiency and internal validation

The AUC in the developmental cohort was 0.893 (95% confidence interval (CI) [0.879, 0.908]; P < 0.001) (Fig 1A). The model demonstrates high performance in a Hosmer–Lemeshow test between the expected and observed risks, with overestimation at the extremes (P = 0.059) (Fig 1B), owing to the small sample size of individuals with higher stroke probability. The clinical utility of the risk model was evaluated by DCA, which mainly focused on the NBs. NB combines the number of true positives and false positives into a single “net” number and is obtained by dividing the net true positives by the sample size. It reflects the balance between the benefit of a true positive and the harm of a false positive. In the context of this study, “treated” refers to patients receiving interventions based on the prediction model. The “treat all” strategy assumes all patients receive interventions regardless of risk, while the “treat none” strategy assumes no patient receives interventions. The decision curve evaluates the NB of using the model to selectively treat patients based on predicted risk thresholds compared to these two baseline strategies. In the current model, the NB was 0.002 if all the subjects were treated, and the NB was 0 if none of them were treated. The DCA showed the satisfactory NB that an affected patient could receive from the predictive nomogram, with the predicted probability threshold between 0% and 25% (Fig 1C). The validation dataset (n = 67,025) was used to evaluate the model’s predictive performance. Internal validation demonstrated consistently strong discrimination with an AUC of 0.878 (95% CI [0.848, 0.909]; P < 0.001) (Fig 1D).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Development and internal validation of the prediction model.

(A) ROC curve for the training dataset. (B) Calibration curve for the training dataset. P-value was determined using Hosmer and Lemeshow test. (C) Decision curve analysis for the training dataset. (D) ROC curve for the internal validation model. AUC, area under the receiver operating characteristic curve. CI, confidence interval. ROC, receiver operating characteristic curve.

https://doi.org/10.1371/journal.pmed.1004539.g001

External validation of prediction model

The external validation cohort was comprised of 37,568 patients from Nanfang Hospital and 48,719 patients from Henan Provincial People’s Hospital. The model demonstrated strong discrimination in the external validation cohorts, with an AUC of 0.897 (95% CI [0.872, 0.922]; P < 0.001) from the Nanfang Hospital data (Fig 2A) and 0.895 (95% CI [0.876, 0.914]; P < 0.001) in the Henan Provincial People’s Hospital data (Fig 2B). The model exhibited a well-calibrated performance in the cohorts from Nanfang Hospital (Fig 2C) and Henan Provincial People’s Hospital (Fig 2D), with overestimation at the extremes in Hosmer and Lemeshow tests, indicating poor calibration in those with higher predicted stroke probability.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. External validation of the prediction model.

(A) ROC curve for the external validation dataset from Nanfang Hospital. (B) ROC curve for the external validation dataset from Henan Provincial People’s Hospital. (C) Calibration curve for the external validation dataset from Nanfang Hospital. P-values were determined using Hosmer and Lemeshow tests. (D) Calibration curve for the external validation dataset from Henan Provincial People’s Hospital. P-value was determined using Hosmer and Lemeshow test. AUC, area under the receiver operating characteristic curve. CI, confidence interval. ROC, receiver operating characteristic curve.

https://doi.org/10.1371/journal.pmed.1004539.g002

Comparison with existing stroke models

The prediction model herein exhibited improved discrimination with a higher AUC (AUC 0.893, 95% CI [0.879, 0.908]; P < 0.001) when compared to the model reported by Mashour and colleagues (AUC 0.777, 95% CI [0.751, 0.803]; P < 0.001), the ATRIA stroke risk score (AUC 0.740, 95% CI [0.713, 0.767]; P < 0.001), the model of Lip and colleagues (AUC 0.804, 95% CI [0.779, 0.829]; P < 0.001), the model of Wolf and colleagues (AUC 0.769, 95% CI [0.744, 0.794]; P < 0.001), and the model of Woo and colleagues (AUC 0.883, 95% CI [0.868, 0.899]; P < 0.001) (Fig 3A), with a positive NB superior to the five alternative models (Fig 3B). When compared using data from Nanfang Hospital, the prediction model exhibited improved discrimination (AUC 0.897, 95% CI [0.872, 0.922]; P < 0.001) than the ATRIA stroke risk score (AUC 0.763, 95% CI [0.719, 0.807]; P < 0.001) and the model proposed by Lip and colleagues (AUC 0.810, 95% CI [0.769, 0.851]; P < 0.001), with a positive NB superior to the two models (S3 Fig). When compared using data from Henan Provincial People’s Hospital, the prediction model also exhibited improved discrimination (AUC 0.895, 95% CI [0.876, 0.914]; P < 0.001) than the ATRIA stroke risk score (AUC 0.797, 95% CI [0.765, 0.830]; P < 0.001) and the model proposed by Lip and colleagues (AUC 0.822, 95% CI [0.792, 0.853]; P < 0.001), with a positive NB superior to the two models (S4 Fig).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Comparison with existing models for stroke.

(A) The 301PSRC showed higher AUC when compared to existing models reported by Mashour [5], Lip [15], Wolf [16], Woo [10], and the ATRIA stroke risk score [14]. (B) The 301PSRC exhibited a positive net benefit for predicted probability thresholds between 0% and 14%, superior over existing models reported by Mashour, Lip, Wolf, Woo, and the ATRIA stroke risk score.

https://doi.org/10.1371/journal.pmed.1004539.g003

Model application

According to the prediction model, we developed an evaluation software named 301PSRC, which can be accessed and applied at the following link: http://139.196.204.68:8087/.

Here, we present two examples of clinical application of our stroke risk calculator. An 82-year-old man with an ASA status of III and a history of stroke was admitted to the hospital for elective abdominal surgery. The preoperative FPG was 7.64 mmol/L, the preoperative MAP was 90, the preoperative fibrinogen was 2.78 g/L, the preoperative serum albumin was 31.5 g/L, and the derived FAR was 0.0882. The surgery duration was 167 min. The patient was identified as having a high risk of perioperative stroke with an estimated absolute risk of 0.6%. A 26-year-old man with an ASA status of II was admitted to the hospital for elective orthopedic surgery. The preoperative FPG was 4.59 mmol/L, the preoperative MAP was 96, the preoperative fibrinogen was 2.67 g/L, the preoperative serum albumin was 38.4 g/L, and the derived FAR was 0.0695. The surgery length was 420 min. The patient was identified as having a low risk of perioperative stroke with an estimated absolute risk of 0.04%.