Performance of SAPS II and SAPS 3 in Intermediate Care

Objective The efficacy and reliability of prognostic scores has been described extensively for intensive care, but their role for predicting mortality in intermediate care patients is uncertain. To provide more information in this field, we have analyzed the performance of the Simplified Acute Physiology Score (SAPS) II and SAPS 3 in a single center intermediate care unit (ImCU). Materials and Methods Cohort study with prospectively collected data from all patients admitted to a single center ImCU in Pamplona, Spain, from April 2006 to April 2012. The SAPS II and SAPS 3 scores with respective predicted mortality rates were calculated according to standard coefficients. Discrimination was evaluated by calculating the area under receiver operating characteristic curve (AUROC) and calibration with the Hosmer-Lemeshow goodness of fit test. Standardized mortality ratios (SMR) with 95% confidence interval (95% CI) were calculated for each model. Results The study included 607 patients. The observed in-hospital mortality was 20.1% resulting in a SMR of 0.87 (95% CI 0.73-1.04) for SAPS II and 0.56 (95% CI 0.47-0.67) for SAPS 3. Both scores showed acceptable discrimination, with an AUROC of 0.76 (95% CI 0.71-0.80) for SAPS II and 0.75 (95% CI 0.71- 0.80) for SAPS 3. Calibration curves showed similar performance based on Hosmer-Lemeshow goodness of fit C-test: (X2=12.9, p=0.113) for SAPS II and (X2=4.07, p=0.851) for SAPS 3. Conclusions Although both scores overpredicted mortality, SAPS II showed better discrimination for patients admitted to ImCU in terms of SMR.


Introduction
Worldwide health care institutions try to give care based on best-practice models with cost-effectiveness. In this scenario, intermediate care units (ImCU) provide a rational and proportional treatment between the intensive care and the general ward.
Previous studies suggested that around 35% of intensive care unit (ICU) admissions were for low risk patients that were admitted mainly for monitoring purposes [1,2], while, in contrast, some patients were treated on general wards when they should receive more intensive care and monitoring [3]. As a result, the development of intermediate care showed encouraging results in terms of cost-containment and ICU utilization, triage flexibility for acute patients and mortality rates for hospital wards [3][4][5][6][7][8][9][10].
Nevertheless, the characteristics of the ImCUs and the type and amount of services provided, varies depending on resource availability, institutional infrastructure and the parent health care system. Accordingly, the case mix population of intermediate care may show great heterogeneity.
Characterization of these patients relies on the assessment of their illness severity, using severity scores. Although the performance of severity scores has been widely described in ICU patients, the information in the setting of ImCU is very limited [11][12][13]. The purpose of this study was to analyze the performance of both, the Simplified Acute Physiology Score (SAPS) II and SAPS 3 in a single center ImCU.

Materials and Methods
This study was performed at the Clínica Universidad de Navarra, an academic medical center of 300 beds, in Pamplona, Spain. Its ImCU is a 9-bed multi-purpose unit adjacent to, but independent from, the mixed ICU. Each bed is equipped with continuous telemetry, pulse oxymetry, noninvasive arterial blood pressure, central venous pressure monitoring, and noninvasive pressure support ventilation. The signals are relayed to a central monitoring station and the nurse-to-patient ratio is 1:3. The ImCU infrastructure (beds, technical resources and nursing staff) is shared by the Stroke Unit and the Coronary Care Unit. These units also have independent admission criteria and medical staff, and the patients from these units were not taken into account for the present study. The ImCU rounding team involves a nurse, the hospital pharmacist, the ImCU resident, the specialist or surgeon and the attending hospitalist. The hospitalist was responsible for admission and discharge of all ImCU patients. Admission and discharge criteria for the ImCU were set according to previous guidelines defined by The American College of Critical Care Medicine [14], and also served as inclusion criteria for the present study. Exclusion criteria included: age less than 18 years old, severe respiratory failure at imminent risk of requiring intubation, status epilepticus, and catastrophic brain illness. ImCU readmissions and patients admitted for drug administration and desensitization, were excluded from data analysis. Patients came from medical and surgical wards, ICU, the operating room, and the emergency room.
From April 2006 to April 2012, every consecutive patient admitted to the ImCU was evaluated. Demographics, past medical history, reasons for admission, physiological parameters at the time of admission and during the first 24 hours of ImCU stay, laboratory variables and survival to hospital discharge were prospectively recorded by the authors. Patients with missing variables were also excluded from data analysis.
The SAPS II and SAPS 3 scores with their respective predicted mortality rates were calculated according to standard coefficients [15,16]. We did not use the SAPS 3 customized equation for Southern Europe, since previous studies showed similar results using the general equation [17]. Data considered for the calculation of SAPS 3 and SAPS II scores were recorded within 1 hour and during the first 24 hours of ImCU admission, respectively. In-hospital mortality was the end-point of the study.

Ethics Statement
The study protocol was approved by the Institutional Review Board (IRB) at the Clínica Universidad de Navarra (ref. 129/2010). The IRB waived the need for informed consent, because it is an observational non-interventional cohort study with prospectively collected data and retrospective analysis, and also because it did not interfere with decisions related to patient´s care. The study has been performed in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki and its later amendments.

Statistical Analysis
Data were entered into a computer data base by the authors. Statistical analysis was performed using SPSS for Windows, version 15.0 (SPSS Inc, Chicago, IL). Continuous variables were reported as mean ± standard deviation or median (25%-75% interquartile range). For nonparametric measure of statistical dependence of quantitative variables, we used Spearman´s correlation coefficient. Categorical variables were reported as absolute numbers (frequency percentages) and analyzed by chi-square test. Validation of the prognostic scores was assessed by standard tests to measure discrimination and calibration. Discrimination was evaluated by calculating the area under receiver operating characteristic curve (AUROC) [18] and calibration with the Hosmer-Lemeshow goodness-of-fit C test [19]. A high p value (> 0.05) would indicate a good fit for the model. Calibration curves were constructed by deciles of predicted mortality (x-axis) against observed mortality (y-axis). Standardized mortality ratios (SMR) (ratio of observed to expected deaths) with 95% confidence interval (95% CI) were calculated for each model.

Results
During the study period, 938 patients were admitted in the ImCU. Of these, 331 were excluded: 66 low risk patients (drug administration and desensitization), 170 readmissions, and 95 patients for missing variables. Six hundred and seven (607) patients were included for data analysis.
Patient characteristics, location prior to admission, surgical status, coomorbidities and discharge location are described in Table 1.
The mean SAPS II and SAPS 3 of the cohort were 36.6±11.9 and 58.4±15.4 respectively, and the expected mortality derived from these scores were 22.9 % and 35.6%. The observed inhospital mortality was 20.1% (122/607) resulting in a SMR of 0.87 (95% CI 0.73-1.04) for SAPS II and 0.56 (95% CI 0.47-0.67) for SAPS 3. Performances of the models are presented in Table 3. Both scores showed acceptable discrimination, with an AUROC of 0.76 (95% CI 0.71-0.80) for SAPS II and 0.75 (95% CI 0.71-0.80) for SAPS3 ( Figure 1). Although both scores overpredicted mortality, SAPS II showed better discrimination with results closer to the observed mortality based on SMR. Nonetheless, the discrimination performance assessed by AUROC did not show any meaningful difference. Calibration curves are presented in Figures 2 and 3. Both scores showed similar calibration performance based on Hosmer-Lemeshow goodness of fit test: (X 2 =12.9, p=0.113) for SAPS 2 and (X 2 =4.07, p=0.851) for SAPS 3. Nonetheless, there is a non-significant trend for better calibration performance for SAPS 3. We also established a significant correlation between SAPS II and SAPS 3 in predicting mortality (Rho 0.66 p < 0.01).  [12].

Discussion
To the best of our knowledge, this is the first external validation study of the SAPS 3 score in ImCU population. In our cohort, this relatively new score was not superior to its previous version. Furthermore, the performance of SAPS II showed better discrimination with results closer to the observed mortality.
It is noteworthy the high risk of the patients in our study, based on elevated SAPS II (36.6±11.9) and SAPS 3 (58.4±15.4) scores, with considerable expected mortality rates 22.9% and 35.6% respectively. The contribution of the subgroup of oncologic patients (187/607), with advanced disease (75%), and elevated SAPS II (42.1±12.2) and SAPS 3 (67.2±14.1) scores, could explain in part these findings. The admission of these patients was required because their acute disease needed close monitoring and was not due to ICU bed restriction. However, in some cases, they were not candidates for ICU admission due to the underlying disease, chronic coomorbidities or advanced age. Observed to expected mortality ratios (SMRs) have become standard tools for assessing the impact of ICU-related outcomes, including unit organization and management [20]. Therefore, it is very important to evaluate the performance of    prognostic models that are used to generate these data and finally detecting potential biasses. Poor calibration and mortality underestimation could prevent relevant comparisons of data extracted from SMRs. In addition, external validation studies are required before applying new scores to other casemixes. Soares and Salluh [21], found excellent discrimination for SAPS II and SAPS 3 in a cohort of 952 oncologic patients admitted to the ICU. The calibration of SAPS 3 and its  customized equation for Central and South American countries in this cohort was appropriate, but the calibration of SAPS II was poor. These results were in agreement with other reports on the performance of SAPS II in ICU patients from European countries, in which the model failed to adjust adequately for differences in the case-mix profiles [22][23][24].
In the present study, we found similar calibration performance of the models, with a non-significant trend for better performance for SAPS 3, based on the Hosmer Lemeshow goodness of fit C-test. However, it is surprising the significant mortality overprediction based on the original SAPS 3 (SMR 0.56, 95% CI 0.47-0.67), and compared to SAPS II (SMR 0.87, 95% CI 0.73-1.04, closer to unit).
Changes in case mix in critical care, has been one of the reasons related to the poor calibration of SAPS II over time and one of the arguments for the development of new generation scores. It is possible that the oldest score avoids the potential overfitting bias of SAPS 3, when the validation case mix cohort is substantially different from the original study. Additionally, recent external validation studies in ICU population, showed similar results to ours, with both scores overpredicting mortality, but SAPS 3 more than SAPS II [25,26].
These conflicting results emphasize the need to find more reliable and accurate scores for intermediate care patients, based on larger, prospective and well designed studies.
The present study provides an external validation of the SAPS 3 score in intermediate care. However, several limitations must be addressed. The case-mix of our cohort may differ from that in other ImCUs, limiting the extrapolation of the results to other populations. Case-mix differences between the development cohort of SAPS 3 and our population could deteriorate calibration and the ratios of observed to expected mortality [27]. The small sample size derived from a single center study, and even more restricted samples of various subgroups of the population, could interfere with the evaluation of the uniformity of fit among different expected mortality subgroups. Moreover, the traditional calibration methods, such as Hosmer-Lemeshow, only describe the deviations between observed and expected mortality, without any description of the direction, extent and risk subgroups affected by these deviations. In the same way, they average the risk of patients in each decile and do not use the whole information carried by the individual patient [28,29].

Conclusions
This study highlights the importance of external validation of prognostic scores in intermediate care, before the routine application of new severity indexes. SAPS II and SAPS 3 showed similar calibration performance in ImCU, but SAPS II showed better discrimination in terms of SMR.