The association between outcome-based quality indicators for intensive care units

Ilona W. M. Verburg; Evert de Jonge; Niels Peek; Nicolette F. de Keizer

doi:10.1371/journal.pone.0198522

Abstract

Purpose

To assess and improve the effectiveness of ICU care, in-hospital mortality rates are often used as principal quality indicator for benchmarking purposes. Two other often used, easily quantifiable, quality indicators to assess the efficiency of ICU care are based on readmission to the ICU and ICU length of stay. Our aim was to examine whether there is an association between case-mix adjusted outcome-based quality indicators in the general ICU population as well as within specific subgroups.

Materials and methods

We included patients admitted in 2015 of all Dutch ICUs. We derived the standardized in-hospital mortality ratio (SMR); the standardized readmission ratio (SRR); and the standardized length of stay ratio (SLOSR). We expressed association through Pearson’s correlation coefficients.

Results

The SMR ranged from 0.6 to 1.5; the SRR ranged from 0.7 to 2.1; and the SLOSR ranged from 0.7 to 1.3. For the total ICU population we found no significant associations. We found a positive, non-significant, association between SMR and SLOSR for admissions with low-mortality risk, (r = 0.25; p = 0.024), and a negative association between these indicators for admissions with high-mortality risk (r = -0.49; p<0.001).

Conclusion

Overall, we found no association at ICU population level. Differential associations were found between performance on mortality and length of stay within different risk strata. We recommend users of quality information to take these three outcome indicators into account when benchmarking ICUs as they capture different aspects of ICU performance. Furthermore, we suggest to report quality indicators for patient subgroups.

Citation: Verburg IWM, de Jonge E, Peek N, de Keizer NF (2018) The association between outcome-based quality indicators for intensive care units. PLoS ONE 13(6): e0198522. https://doi.org/10.1371/journal.pone.0198522

Editor: Pierre Moine, University of Colorado Denver, UNITED STATES

Received: November 21, 2017; Accepted: May 21, 2018; Published: June 13, 2018

Copyright: © 2018 Verburg et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: The data used in this study belongs to the individual ICUs participating in the Dutch National Intensive Care Evaluation (NICE) registry. We obtained permission from the secretary of the NICE board, Dr. D.W. de Lange, email: info@stichting-nice.nl, to use data from the NICE registry for this study. Requests for the data used in this study can be provided to the NICE board which will judge requests on feasibility of the analysis and whether or not the confidentiality of patients and ICUs is protected. We confirm that others would be able to access these data in the same manner as the authors. The authors did not have any special access privileges that others would not have.

Funding: Drs. Verburg and De Keizer’s institutions received grant support and support for participation in review activities from the National Intensive Care Evaluation (NICE) Foundation (The NICE Foundation pays the department of Medical Informatics for maintaining the national database, providing feedback reports, and doing analyses; Drs. Verburg and De Keizer are employees of the Department of Medical Informatics). Drs. de Keizer and De Jonge are members of the board of the NICE Foundation.

Competing interests: The authors have declared that no competing interests exist.

Introduction

In recent years attention to quality of healthcare as expressed by quality indicators has increased [1, 2]. Treatment at the intensive care unit (ICU) is very complex and delivered in a highly technical and labor-intensive environment. Along with these developments the cost of intensive care has increased substantially resulting in a high proportion of the health care expenditure accountable to ICUs [3]. This all makes ICUs a particularly interesting part of the hospital to assess and improve performance.

Within the ICU setting several outcome, structure and process quality indicators are presented [4, 5]. This study focusses on three outcome indicators for ICU care. To assess and improve the effectiveness of ICU care, in-hospital mortality rates are often used as principal quality indicator. Two other often used, easily quantifiable, quality indicators to assess the efficiency of ICU care are based on readmission to the ICU and ICU length of stay. ICU length of stay is highly associated with the costs of ICU care and an easily quantifiable outcome indicator for efficiency of ICU care [6, 7]. ICU discharge is often based on subjective criteria and discharge policy could be influenced by the availability of bed and staff resources [8–10].

Premature discharge may lead to an increase in patients readmitted to the ICU or higher post-ICU mortality rates [11–15]. Improving discharge decision making may reduce resource consumption, prevent readmissions to the ICU, shorten ICU length of stay, and hence reduce costs [16].

Several studies have shown patient level associations. Patients having an ICU readmission are more severely ill, show longer ICU length of stay and higher in-hospital mortality rates compared to patients not having an ICU readmission [17, 18]. Furthermore, ICU length of stay differs for survivors compared to non-survivors [19, 20]. This study focuses on the association between these quality indicators on unit-level in the context of benchmarking the performance of ICUs.

To get a more complete picture of these quality indicators when comparing ICUs, presenting several quality indicators simultaneously may have added value. Quality indicators could be positively correlated, that is, ICUs performing well on one of the quality indicators may also perform well on other quality indicators. This means that the quality indicator does not only provide information about the related process, on efficiency or effectiveness, but it also provides information about quality in general. Quality indicators could also be negatively correlated, meaning that performing well on one of the quality indicators often comes at the expense of performance on the others. It could be the case that these quality indicators are not correlated because the quality indicators capture independent aspects of performance. Finally, it is relevant to know whether ICUs perform similarly across their entire patient population or differently in specific subgroups.

The aim of this study was to examine whether there is an association between unit-level, case-mix adjusted quality indicators based on in-hospital mortality; readmission to the ICU; and ICU length of stay in the general Dutch ICU population as well as within specific subgroups of ICU patients.

Materials and methods

The Dutch National Intensive Care Evaluation registry

The NICE registry has been active since 1996 and, in 2015, all 83 Dutch ICUs delivered data to the registry. The NICE registry [21] collects demographical, physiological and diagnostic data from the first 24 hours of all patients admitted to participating ICUs, including all variables used in the Acute Physiology and Chronic Health Evaluation (APACHE) IV model to predict probabilities of in-hospital mortality [22]. Collected data is checked by registry staff for internal consistency, by performing onsite data quality audits and training local data collectors [23].

The data collected by the registry are officially registered and collected in accordance with the Dutch Personal Data Protection Act. The medical ethics committee of the Academic Medical Center waived the need for medical ethics for this study (registration number W17_162).

Study data

We included consecutive ICU admissions with initial ICU admission between January 1^st 2015 and January 1^st 2016. Fig 1 presents a flow chart of patient inclusion. We excluded cardiac surgery patients, as patients admitted for cardiac surgery in general have a short, generally fixed, ICU length of stay. We applied the APACHE IV exclusion criteria [22]. Since patient data are anonymized, we can only match different ICU admissions of a patient within the same hospitalization period. For this reason we excluded patients who were discharged to another ICU, as their observed ICU length of stay was truncated, and we excluded patients with missing values for model covariates.

Download:

Fig 1. Flowchart of admissions included for the quality indicators for in-hospital mortality, readmission to the ICU and ICU length of stay.

https://doi.org/10.1371/journal.pone.0198522.g001

A readmission to the ICU was defined as a second ICU admission within 48 hours of ICU discharge from the initial ICU admission, in the same hospital within the same hospital stay. A time period of 48 hours was chosen, since readmissions within this period have a stronger relationship with ICU interventions, such as mechanical ventilation, and discharge circumstances, than later readmissions [24, 25]. When considering readmissions to the ICU we excluded patients who were not at risk for readmission to the ICU during the same hospitalization period: patients who died during their initial ICU admission and patients who were discharged from the ICU to a location outside the hospital (e.g. to home, nursery home or another hospital) [24].

The NICE board has established subgroups of ICU admissions, of which the NICE registration publishes quality indicators [26]. Associations were examined for the entire cohort and for these subgroups: medical admission, urgent surgery or elective surgery; groups based on severity of illness defined using the calibrated APACHE IV probability of mortality (low: smaller than 0.3, medium: larger or equal to 0.3 and smaller than 0.7, or high: larger or equal to 0.7); and several homogeneous diagnostic subgroups, i.e. patients admitted for community acquired pneumonia (CAP), sepsis, or out of hospital cardiac arrest (OHCA). The precise definitions of all subgroups are presented in S1 Table.

Quality indicators

Quality indicator scores may be influenced by differences in patient characteristics. Therefore, we performed case-mix correction to adjust for these differences. For case-mix correction we used patient characteristics available at admission time or within the first 24 hours of ICU admission. For all outcome measures case-mix correction models and model performances are presented in S1 File. Furthermore, the included ICUs differ in size. For subgroups of ICU patients with a small number of admissions, the estimators of quality indicators scores for some small ICUs would be less reliable than for ICUs with a larger number of admissions. For this reason we adjusted the scores for reliability using empirical Bayes estimators, by effectively shrinking unreliable scores (i.e. of small ICUs) towards 1 [27, 28]. The empirical Bayes estimators were implemented by fitting mixed effects regression models with random intercept for ICU [27, 28]. To avoid that subgroup analyses with small numbers of admissions would lead to non-convergence of the mixed effects regression analyses, we applied a two-step approach. First we fitted fixed effect models to calculate predicted values for each of the three outcome measures at patient level. These models were used for case-mix correction in the entire cohort and in subgroups. In a second step, the (log odds of the) predicted outcome was included as fixed covariate in mixed effect models for each of the three outcome measures and each subgroup.

Mortality: As quality indicator for in-hospital mortality after or during ICU admission we used the standardized mortality ratio (SMR). To estimate the reliability adjusted SMR, we first recalibrated the APACHE IV probability of mortality [29] to our dataset using logistic regression analysis. Secondly, for each subgroup we performed logistic regression with the log-odds of the probability of mortality as fixed covariates and ICU as random intercept. To estimate the reliability adjusted risk adjusted rates we subsequently added the log-odds of the overall mortality rate to the random intercept per ICU and applied the inverse logit of the result in order to calculate the reliability adjusted risk adjusted rates for each ICU. We divided the risk adjusted rates by the overall mortality rate to obtain the reliability adjusted SMR values.

Readmission to the ICU: As quality indicator for readmission to the ICU we used the standardized readmission ratio (SRR). To estimate the reliability adjusted SRR, we first estimated the expected probability of readmission by developing a logistic regression model on the entire cohort, in a similar way as described in a previous study [24]. Secondly, we estimated the SRR using random intercepts per ICU as is done for the SMR.

ICU length of stay: As quality indicator for ICU length of stay we used the standardized length of stay ratio (SLOSR). To estimate the reliability adjusted SLOSR, we first estimated the expected ICU length of stay, in fractional days, using ordinary least square regression with a log-link function, as we did in a previous study [19]. Secondly, we applied ordinary least square regression with the expected ICU length of stay as fixed covariate and ICU as random intercept. We added the random intercept per ICU to the average length of stay for each subgroup to obtain the reliability adjusted average ICU length of stay and divided this value by the average length of stay for the subgroup to get the reliability adjusted SLOSR.

Statistical analysis

The overall performance for the models for in-hospital mortality and readmission to the ICU was assessed by the scaled Brier score [30] and the discriminative ability by the concordance index (C-statistic). The overall performance of the model for ICU length of stay was assessed by squared Pearson correlation coefficient (R²). Model calibration was assessed by creating 50 equally-sized subgroups of predicted values.

To present associations between quality indicators, we plotted pairs of quality indicator values (measured at unit level) against each other. We examined the association between pairs of quality indicator values by Pearson’s correlation coefficients and calculated corresponding p-values using the Student’s t-distribution. Because small numbers of patients per ICU may result in unreliable estimates of quality indicator values, we also performed a sensitivity analysis using Spearman’s rank correlation coefficient which is less sensitive to outliers. We considered p-values smaller than 0.01 as statistically significant.

We performed all analyses using the R statistical software, version 3.3.2 [31]. We used linear models (lm) and general linear models (GLM) for regression analysis [32] and the 'splines' package for calculating restricted cubic splines [33]. We used the lme4 package [34] to fit linear mixed-effects models (LMM) and generalized linear mixed-fixed models (GLMM) by respectively the functions 'lmer' and 'glmer' and estimated the random effects by the function 'ranef'. The 'cor.test' function was used to examine Pearson’s and Spearman’s correlation coefficients and corresponding p-values [35].

Results

Study data and quality indicators

In total 72,797 patients were admitted to 83 Dutch ICUs, for other reasons than admission after cardiac surgery. S2 Table presents the number of admissions per subgroup. For in-hospital mortality and ICU length of stay as outcome measure, after applying the exclusion criteria 59,809 (82.2%) patients, ranging from 205 to 2,282 patients per ICU were included. From these patients 48,496 (81%) were included for readmission as outcome measure, see Fig 1.

Table 1 presents crude percentages of the outcomes and case-mix adjusted quality indicators, for the entire cohort and each subgroup. Crude in-hospital mortality was 14.3% ranging from 2.3% to 26.4% and the recalibrated SMR ranged from 0.6 to 1.7. The crude percentage of ICU readmissions was 2.4% ranging from 0.0% to 5.5% and the SRR ranged from 0.7 to 2.1. The overall mean ICU length of stay was 3.3 ranging from 1.7 to 6.1 and the SLOSR ranged from 0.7 to 1.3. Associations based on the Pearson correlation coefficient were r = 0.40 (p<0.001) between the crude in-hospital mortality rate and SMR; r = 0.89 (p<0.001) between the crude readmission rate and SRR; and r = 0.48 (p<0.001) between the mean ICU length of stay and SLOSR.

Download:

Table 1. Values for the crude outcomes and for the quality indicators standardized mortality ratio (SMR); standardized readmission ratio (SRR) and standardized ICU length of stay ratio (SLOSR).

Overall values are shown with minimum and maximum ranges over the ICUs.

https://doi.org/10.1371/journal.pone.0198522.t001

Statistical analysis

Results on the performance of the prediction models are shown in Table D and Figs A- C of S1 File. Fig 2 presents the pairs of SMR, SRR and SLOSR plotted against each other for the entire cohort. Table 2 presents Pearson’s correlation coefficients for the entire cohort as well as for subgroups. No significant associations were identified between SRR and SMR or SRR and SLOSR. This means that ICUs with lower than expected readmission rates did not have lower or higher mortality, or shorter or longer length of stays, than expected. No significant associations were identified at cohort level between SMR and SLOSR. However, a negative significant association (r = -0.49; p<0.001) was found for the high mortality risk subgroup (i.e., recalibrated APACHE IV probability of mortality larger than 0.7), meaning better performance on SMR (lower mortality) was associated with worse performance on SLOSR (longer ICU length of stay). Conversely, a positive association was found for the low mortality risk subgroup (i.e., recalibrated APACHE IV probability of mortality smaller than 0.3). This association was not significant using Pearson’s correlation coefficient (r = 0.25; p = 0.024), but was significant using the Spearman’s rank correlation coefficient (ρ = 0.29; p = 0.009). Fig 3 presents the pairs of SMR and SLOSR for these subgroups based on probability of mortality.

Download:

Fig 2. Pairs of quality indicators plotted against each other on ICU level.

Fig 2A: standardized in-hospital mortality ratio (SMR) against standardized ICU readmission within 48 hours ratio (SRR). Fig 2B: standardized length of stay ratio (SLOSR) against SMR. Fig 2C: SLOSR against SRR.

https://doi.org/10.1371/journal.pone.0198522.g002

Download:

Fig 3. Quality indicators SMR and SLOSR plotted against each other on ICU level for patients grouped based on severity of illness defined using the probability of mortality.

Fig 3A: probability of mortality smaller than 0.3. Fig 3B: probability of mortality larger or equal to 0.3 and smaller than 0.7. Fig 3C: probability of mortality larger or equal to 0.7.

https://doi.org/10.1371/journal.pone.0198522.g003

Download:

Table 2. Pearson’s correlation coefficients with corresponding p-values for the pairs of quality indicators standardized mortality ratio (SMR); standardized readmission ratio (SRR) and standardized ICU length of stay ratio (SLOSR).

https://doi.org/10.1371/journal.pone.0198522.t002

S3 Table presents Spearman’s rank correlation coefficients for the entire cohort as well as for subgroups. These results are compatible with the results from the primary analysis.

Discussion

In this study we examined the association between SMR, SLOSR and SRR as individual outcome quality indicators for benchmarking ICU quality of care. We did not find significant correlations between SRR and SMR or SRR and SLOSR. This means that ICUs with lower than expected readmission rates did not have lower or higher mortality, or shorter or longer length of stays, than expected. We also did not find a significant correlation between SMR and SLOSR for the general ICU population. However, subgroup analyses based on probability of mortality showed associations in opposite directions.

Among the high-risk patient population the association between SMR and SLOSR was negative, i.e. better performance on SMR (lower mortality) was associated with worse performance on SLOSR (longer ICU length of stay). One of the explanations could be the process of end-of-life decisions, which might differ between ICUs. Among the low-risk patient population there was a non-significant positive association between SMR and SLOSR. One explanation might be that problems in the care process resulting in more complications during ICU stay may for example cause higher mortality and longer ICU length of stay.

Although results from subgroups should interpreted cautiously, such different directions in the associations between SMR and SLOSR depending on mortality-risk could be plausible. High quality of care could result in both lower mortality and earlier ICU discharge, explaining the positive association in low-risk patients. However, patients who die in a specific ICU may die later in another ICU. Thus, if mortality is high, this may paradoxically decrease the mean ICU length of stay, explaining the negative association between SMR and SLOSR in high-risk populations.

On patient level several studies found that patients that survived their initial ICU stay and are readmitted to the ICU showed higher mortality rates and longer length of stay [13, 16, 36] compared to patients not readmitted to the ICU. However, studies using quality indicators after correction for case-mix reported similar results as ours. A recent study on Dutch hospitals found that hospitals with many patients with long length of stay also had higher in-hospital mortality rates, but they found no correlation for readmissions [37]. However, for Dutch ICU patients in-hospital mortality predictions seems to be more reliable using clinical data compared to administrative data [38]. Two earlier studies found no association between quality indicators for ICU patients after case-mix correction [13, 36]. For all hospitalized patients one study found no correlation between SMR and readmission rate, but length of stay was positively correlated with SMR [39]. These results may differ since they are not specific for ICU patients, hospital length of stay was used and no-case mix correction was applied for length of stay. Two other studies on hospitalized patients also found that a decrease in length of stay may not necessarily lead to increases in hospital readmission rates [14] and that hospitals with different levels of performance on mortality showed similar results on hospital readmission rates [40].

Interestingly, in our pre-defined subgroup analyses, an association between SMR and SLOSR was identified. In patients with high mortality risk, a high SMR was associated with a shorter case-mix corrected ICU length of stay. In contrast a non-significant opposite association was found in low-risk patients. Although results from subgroups should interpreted cautiously, such different directions in the associations between SMR and SLOSR depending on mortality-risk could be plausible. High quality of care could result in both lower mortality and earlier ICU discharge, explaining the positive association in low-risk patients. However, patients who die in a specific ICU may die later in another ICU. Thus, if mortality is high, this may paradoxically decrease the mean ICU length of stay, explaining the negative association between SMR and SLOSR in high-risk populations.

In this study, we focused on three quality indicators related to the outcomes of ICU patients. Within the ICU domain many other quality indicators, such as structure and process indicators are also used [4, 41]. For example, within the NICE registry new actionable quality indicators for pain management, blood use, antibiotics and mechanical ventilation have been recently developed [42]. Their aim is to provide tools to directly improve specific care processes. This is in contrast to the three quality indicators examined in this study, which provide a more general view on overall quality of ICU care. For the total cohort of ICU patients the risk of bias in our study was low and the generalizability was high, since many patient level data were available and because all Dutch ICUs participated in this study. Although we do not know the level of residual confounding, we believe that it is small because case-mix correction was applied using a well-established mortality prediction model (APACHE IV) and the models for predicting readmissions and length of stay were developed on our data using a broad set of predictive variables, all measured during the first 24 hours of ICU admission.

ICU discharge decisions often do not only depend on a patient's recovery, but on organizational circumstances such as availability of beds on the general ward and the need to free up ICU beds for other patients. Nevertheless, we have deliberately chosen not to include ICU and hospital level covariates in our prediction model for ICU length of stay, since our study has been performed in the context of benchmarking where one does not want to adjust for but detect organizational differences. Furthermore, in a previous study, we found that after correcting for patient characteristics including ICU characteristics did not significantly improve ICU LoS predictions [43].

Calibration of all three models was satisfactory, but we did find low values of the scaled Brier score and C-statistic for readmission predictions, Table C of S1 File. This indicates that there was variation in readmission rates that could not be explained by case-mix factors; we do not believe that it has influenced our findings. Our model to predict the probability of being readmitted within the first 48 hours after ICU discharge relies on patient data reflecting severity of illness available from the first 24 hours of ICU admission [24]. It may be better to take severity of illness upon ICU discharge into account [16, 17], but these data are not available in our registry.

A limitation of our approach is that some quality indicator values were estimated from small subgroups. For subgroups of ICU patients with a small number of admissions, estimators of quality indicator scores for these small ICUs would be less reliable than for ICUs with a larger number of admissions, which may have resulted in unstable estimates (i.e. of small ICUs) towards 1 [27, 28]. As a solution, for each subgroup we decided to apply reliability adjustment to shrink the observed indicator values to the overall average. A sensitivity analysis using the more robust rank correlation coefficient and analysis without adjustment for reliability showed similar results as the main analysis. However, for benchmarking purposes we think that it is not convenient to present results on ICU-level based on very few patients.

Our criteria for patient inclusion lead to several limitations. Firstly, the exclusion of patients that were transferred to another ICU or are discharged from the hospital in order to get palliative care elsewhere may have biased the SMR values. These patients are known to have higher mortality rates [44]. Using long term mortality as outcome measure compared to in-hospital mortality may be useful [45–47]. We found that the number of transferals in the Dutch ICU system was 1.2% after applying all other patient exclusion criteria. Secondary analyses showed that using SMR based on three months mortality instead of in-hospital mortality did not change the results regarding associations between SMR, SRR and SLOSR (results not shown). Secondly, no information was available regarding readmission to the ICU for patients discharged from the hospital, which may have biased our estimates of SRR values. Thirdly, death or hospital discharge within 48 hours could be competing risks with readmission to the ICU and was not taken into account. Patients who die or are discharged from the hospital within 48 hours after ICU discharge may have a lower probability to be readmitted within 48 hours after ICU discharge compared to patients who are not discharged from the hospital within 48 hours after ICU discharge.

Conclusion

We identified no significant association between quality indicators for in-hospital mortality, readmission to the ICU within 48 hours after ICU discharge and ICU length of stay at ICU population level. Differential associations were found between performance on mortality and length of stay within different risk strata. We recommend users of quality information for benchmarking purposes to take these three outcome indicators into account when judging or monitoring ICU quality of care as they capture different aspects of ICU performance. Furthermore, we suggest that users of quality information also receive quality data for patient subgroups, especially low-risk and high-risk patients groups.

Supporting information

S1 Table. Definitions of patients subgroups included in the study.

https://doi.org/10.1371/journal.pone.0198522.s001

(PDF)

S2 Table. Number of included ICUs and number of included admissions for each subgroup and quality indicator.

Overall values are shown accompanied with minimum and maximum ranges over the ICUs.

https://doi.org/10.1371/journal.pone.0198522.s002

(PDF)

S3 Table. Spearman rank correlation coefficients with corresponding p-values for the pairs of quality indicators standardized mortality ratio (SMR); standardized readmission ratio (SRR) and standardized ICU length of stay ratio (SLOSR).

https://doi.org/10.1371/journal.pone.0198522.s003

(PDF)

S1 File. Results of recalibration and regression analysis to calculate predicted outcome.

https://doi.org/10.1371/journal.pone.0198522.s004

(PDF)

Acknowledgments

We acknowledge all participating ICUs in the National Intensive Care Registry for their participation and hard work to collect and improve their processes based on these data. We would like to thank Rebecca Holman for statistical advise during the process of writing this paper.

References

1. Kolfschoten NE, Kievit J, Gooiker GA, van Leersum NJ, Snijders HS, Eddes EH, et al. Focusing on desired outcomes of care after colon cancer resections; hospital variations in 'textbook outcome'. Eur J Surg Oncol. 2013;39(2):156–163. pmid:23102705
- View Article
- PubMed/NCBI
- Google Scholar
2. Goldstein H, Spiegelhalter DJ. League Tables and Their Limitations: Statiatical Issues in Comparisons of Institutional Performance. Journal of the Royal Statistical Society 1996;159(3):358–433.
- View Article
- Google Scholar
3. Halpern NA, Pastores SM. Critical care medicine in the United States 2000–2005: an analysis of bed numbers, occupancy rates, payer mix, and costs. Critical care medicine. 2010;38(1):65–71. pmid:19730257
- View Article
- PubMed/NCBI
- Google Scholar
4. Berenholtz SM, Dorman T, Ngo K, Pronovost PJ. Qualitative review of intensive care unit quality indicators. J Crit Care. 2002;17(1):1–12. pmid:12040543
- View Article
- PubMed/NCBI
- Google Scholar
5. Flaatten H. The present use of quality indicators in the intensive care unit. Acta Anaesthesiol Scand. 2012;56(9):1078–1083. pmid:22339772
- View Article
- PubMed/NCBI
- Google Scholar
6. Kahn JM, Rubenfeld GD, Rohrbach J, Fuchs BD. Cost savings attributable to reductions in intensive care unit length of stay for mechanically ventilated patients. Medical care. 2008;46(12):1226–1233. pmid:19300312
- View Article
- PubMed/NCBI
- Google Scholar
7. Moran JL, Solomon PJ, Outcome ACf, Resource Evaluation of the A, New Zealand Intensive Care S. A review of statistical estimators for risk-adjusted length of stay: analysis of the Australian and new Zealand Intensive Care Adult Patient Data-Base, 2008–2009. BMC medical research methodology. 2012;12:68. pmid:22591115
- View Article
- PubMed/NCBI
- Google Scholar
8. Dara SI, Afessa B. Intensivist-to-bed ratio: association with outcomes in the medical ICU. Chest. 2005;128(2):567–572. pmid:16100139
- View Article
- PubMed/NCBI
- Google Scholar
9. Williams T, Leslie G. Delayed discharges from an adult intensive care unit. Aust Health Rev. 2004;28(1):87–96. pmid:15525255
- View Article
- PubMed/NCBI
- Google Scholar
10. Russell S. Reducing readmissions to the intensive care unit. Heart & lung: the journal of critical care. 1999;28(5):365–372. pmid:10486454
- View Article
- PubMed/NCBI
- Google Scholar
11. Beck DH, McQuillan P, Smith GB. Waiting for the break of dawn? The effects of discharge time, discharge TISS scores and discharge facility on hospital mortality after intensive care. Intensive care medicine. 2002;28(9):1287–1293. pmid:12209279
- View Article
- PubMed/NCBI
- Google Scholar
12. Goldfrad C, Rowan K. Consequences of discharges from intensive care at night. Lancet. 2000;355(9210):1138–1142. pmid:10791376
- View Article
- PubMed/NCBI
- Google Scholar
13. Cooper GS, Sirio CA, Rotondi AJ, Shepardson LB, Rosenthal GE. Are readmissions to the intensive care unit a useful measure of hospital performance? Medical care. 1999;37(4):399–408. pmid:10213020
- View Article
- PubMed/NCBI
- Google Scholar
14. Kaboli PJ, Go JT, Hockenberry J, Glasgow JM, Johnson SR, Rosenthal GE, et al. Associations between reduced hospital length of stay and 30-day readmission rate and mortality: 14-year experience in 129 Veterans Affairs hospitals. Annals of internal medicine. 2012;157(12):837–845. pmid:23247937
- View Article
- PubMed/NCBI
- Google Scholar
15. Benbassat J, Taragin M. Hospital readmissions as a measure of quality of health care: advantages and limitations. Archives of internal medicine. 2000;160(8):1074–1081. pmid:10789599
- View Article
- PubMed/NCBI
- Google Scholar
16. Rosenberg AL, Hofer TP, Hayward RA, Strachan C, Watts CM. Who bounces back? Physiologic and other predictors of intensive care unit readmission. Critical care medicine. 2001;29(3):511–518. pmid:11373413
- View Article
- PubMed/NCBI
- Google Scholar
17. Rosenberg AL, Watts C. Patients readmitted to ICUs*: a systematic review of risk factors and outcomes. Chest. 2000;118(2):492–502. pmid:10936146
- View Article
- PubMed/NCBI
- Google Scholar
18. Metnitz PG, Fieux F, Jordan B, Lang T, Moreno R, Le Gall JR. Critically ill patients readmitted to intensive care units—lessons to learn? Intensive care medicine. 2003;29(2):241–248. pmid:12594586
- View Article
- PubMed/NCBI
- Google Scholar
19. Verburg IW, de Keizer NF, de Jonge E, Peek N. Comparison of regression methods for modeling intensive care length of stay. PLoS One. 2014;9(10):e109684. pmid:25360612
- View Article
- PubMed/NCBI
- Google Scholar
20. Strand K, Walther SM, Reinikainen M, Ala-Kokko T, Nolin T, Martner J, et al. Variations in the length of stay of intensive care unit nonsurvivors in three Scandinavian countries. Critical care. 2010;14(5):R175. pmid:20920348
- View Article
- PubMed/NCBI
- Google Scholar
21. Van de Klundert N, Holman R, Dongelmans DA, de Keizer NF. Data Resource Profile: the Dutch National Intensive Care Evaluation (NICE) Registry of Admissions to Adult Intensive Care Units. Int J Epidemiol. 2015;44(6):1850–1850h. pmid:26613713
- View Article
- PubMed/NCBI
- Google Scholar
22. Zimmerman JE, Kramer AA, McNair DS, Malila FM. Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today's critically ill patients. Crit Care Med. 2006;34(5):1297–1310. pmid:16540951
- View Article
- PubMed/NCBI
- Google Scholar
23. Koetsier A, Peek N, de Jonge E, Dongelmans D, van Berkel G, de Keizer N. Reliability of in-hospital mortality as a quality indicator in clinical quality registries. A case study in an intensive care quality register. Methods Inf Med. 2013;52(5):432–440. pmid:23807704
- View Article
- PubMed/NCBI
- Google Scholar
24. van Sluisveld N, Bakhshi-Raiez F, de Keizer N, Holman R, Wester G, Wollersheim H, et al. Variation in rates of ICU readmissions and post-ICU in-hospital mortality and their association with ICU discharge practices. BMC Health Serv Res. 2017;17(1):281–289. pmid:28416016
- View Article
- PubMed/NCBI
- Google Scholar
25. Brown SE, Ratcliffe SJ, Kahn JM, Halpern SD. The epidemiology of intensive care unit readmissions in the United States. Am J Respir Crit Care Med. 2012;185(9):955–964. pmid:22281829
- View Article
- PubMed/NCBI
- Google Scholar
26. Dutch National Intensive Care Evaluation (NICE) foundation. Data in beeld. [Website]. 2014. Available from: http://www.stichting-nice.nl/datainbeeld/public [in Dutch].
27. Dimick JB, Staiger DO, Birkmeyer JD. Ranking hospitals on surgical mortality: the importance of reliability adjustment. Health services research. 2010;45(6 Pt 1):1614–1629. pmid:20722747
- View Article
- PubMed/NCBI
- Google Scholar
28. Hashmi ZG, Dimick JB, Efron DT, Haut ER, Schneider EB, Zafar SN, et al. Reliability adjustment: a necessity for trauma center ranking and benchmarking. The journal of trauma and acute care surgery. 2013;75(1):166–172. pmid:23940864
- View Article
- PubMed/NCBI
- Google Scholar
29. Verburg IW, Holman R, Peek N, Abu-Hanna A, de Keizer NF. Guidelines on constructing funnel plots for quality indicators: A case study on mortality in intensive care unit patients. Stat Methods Med Res. 2017:962280217700169. pmid:28330409
- View Article
- PubMed/NCBI
- Google Scholar
30. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128–138. pmid:20010215
- View Article
- PubMed/NCBI
- Google Scholar
31. Development Core Team R. A Language and Environment for Statistical Computing. 2.15.1 ed. Vienna, Austria: R Foundation for Statistical Computing; 2005.
32. Davies S. Fitting Generalized Linear Models 1992 [2017-09-26]. Available from: https://stat.ethz.ch/R-manual/R-devel/library/stats/html/glm.html.
33. Harell FE. Package rms 2017-09-26 [2017-07-18]. 5.1–0:[Available from: https://cran.r-project.org/web/packages/rms/rms.pdf.
34. Bates D, Maechler M, Bolker B, Walker S, Christensen R, Singmann H, et al. Package lme4. 1.1–12 ed2016.
35. Savicky P. Test for Association/Correlation Between Paired Samples 2014-03-20 [2017-09-26]. 0.3–0:[Available from: https://cran.r-project.org/web/packages/pspearman/pspearman.pdf.
36. Kramer AA, Higgins TL, Zimmerman JE. The association between ICU readmission rate and patient outcomes. Critical care medicine. 2013;41(1):24–33. pmid:23128381
- View Article
- PubMed/NCBI
- Google Scholar
37. Hofstede SN, van Bodegom-Vos L, Kringos DS, Steyerberg E, Marang-van de Mheen PJ. Mortality, readmission and length of stay have different relationships using hospital-level versus patient-level data: an example of the ecological fallacy affecting hospital performance indicators. BMJ Qual Saf. 2017 pmid:28986516
- View Article
- PubMed/NCBI
- Google Scholar
38. Brinkman S, Abu-Hanna A, van der Veen A, de Jonge E, de Keizer NF. A comparison of the performance of a model based on administrative data and a model based on clinical data: effect of severity of illness on standardized mortality ratios of intensive care units. Critical care medicine. 2012;40(2):373–378. pmid:21983367
- View Article
- PubMed/NCBI
- Google Scholar
39. Pouw ME, Peelen LM, Moons KG, Kalkman CJ, Lingsma HF. Including post-discharge mortality in calculation of hospital standardised mortality ratios: retrospective analysis of hospital episode statistics. BMJ. 2013;347:f5913. pmid:24144869
- View Article
- PubMed/NCBI
- Google Scholar
40. Mulvey GK, Wang Y, Lin Z, Wang OJ, Chen J, Keenan PS, et al. Mortality and readmission for patients with heart failure among U.S. News & World Report's top heart hospitals. Circ Cardiovasc Qual Outcomes. 2009;2(6):558–565. pmid:20031893
- View Article
- PubMed/NCBI
- Google Scholar
41. Mainz J. Defining and classifying clinical indicators for quality improvement. International journal for quality in health care: journal of the International Society for Quality in Health Care. 2003;15(6):523–530.
- View Article
- Google Scholar
42. Roos-Blom MJ, Dongelmans D, Arbous MS, de Jonge E, de Keizer N. How to Assist Intensive Care Units in Improving Healthcare Quality. Development of Actionable Quality Indicators on Blood use. Studies in health technology and informatics. 2015;210:429–433. pmid:25991180
- View Article
- PubMed/NCBI
- Google Scholar
43. Verburg IWM, Holman R, Dongelmans D, de Jonge E, de Keizer NF. Is patient length of stay associated with intensive care unit characteristics? J Crit Care. 2017;43:114–121. pmid:28865340
- View Article
- PubMed/NCBI
- Google Scholar
44. Vasilevskis EE, Kuzniewicz MW, Dean ML, Clay T, Vittinghoff E, Rennie DJ, et al. Relationship between discharge practices and intensive care unit in-hospital mortality performance: evidence of a discharge bias. Medical care. 2009;47(7):803–812. pmid:19536006
- View Article
- PubMed/NCBI
- Google Scholar
45. Brinkman S, Abu-Hanna A, de Jonge E, de Keizer NF. Prediction of long-term mortality in ICU patients: model validation and assessing the effect of using in-hospital versus long-term mortality on benchmarking. Intensive care medicine. 2013;39(11):1925–1931. pmid:23921978
- View Article
- PubMed/NCBI
- Google Scholar
46. Rydenfelt K, Engerstrom L, Walther S, Sjoberg F, Stromberg U, Samuelsson C. In-hospital vs. 30-day mortality in the critically ill—a 2-year Swedish intensive care cohort analysis. Acta Anaesthesiol Scand. 2015;59(7):846–858. pmid:26041018
- View Article
- PubMed/NCBI
- Google Scholar
47. Engerstrom L, Kramer AA, Nolin T, Sjoberg F, Karlstrom G, Fredrikson M, et al. Comparing Time-Fixed Mortality Prediction Models and Their Effect on ICU Performance Metrics Using the Simplified Acute Physiology Score 3. Critical care medicine. 2016;44(11):e1038–e1044. pmid:27513546
- View Article
- PubMed/NCBI
- Google Scholar

[ref1] 1. Kolfschoten NE, Kievit J, Gooiker GA, van Leersum NJ, Snijders HS, Eddes EH, et al. Focusing on desired outcomes of care after colon cancer resections; hospital variations in 'textbook outcome'. Eur J Surg Oncol. 2013;39(2):156–163. pmid:23102705
View Article
PubMed/NCBI
Google Scholar

[2] View Article

[3] PubMed/NCBI

[4] Google Scholar

[ref2] 2. Goldstein H, Spiegelhalter DJ. League Tables and Their Limitations: Statiatical Issues in Comparisons of Institutional Performance. Journal of the Royal Statistical Society 1996;159(3):358–433.
View Article
Google Scholar

[6] View Article

[7] Google Scholar

[ref3] 3. Halpern NA, Pastores SM. Critical care medicine in the United States 2000–2005: an analysis of bed numbers, occupancy rates, payer mix, and costs. Critical care medicine. 2010;38(1):65–71. pmid:19730257
View Article
PubMed/NCBI
Google Scholar

[9] View Article

[10] PubMed/NCBI

[11] Google Scholar

[ref4] 4. Berenholtz SM, Dorman T, Ngo K, Pronovost PJ. Qualitative review of intensive care unit quality indicators. J Crit Care. 2002;17(1):1–12. pmid:12040543
View Article
PubMed/NCBI
Google Scholar

[13] View Article

[14] PubMed/NCBI

[15] Google Scholar

[ref5] 5. Flaatten H. The present use of quality indicators in the intensive care unit. Acta Anaesthesiol Scand. 2012;56(9):1078–1083. pmid:22339772
View Article
PubMed/NCBI
Google Scholar

[17] View Article

[18] PubMed/NCBI

[19] Google Scholar

[ref6] 6. Kahn JM, Rubenfeld GD, Rohrbach J, Fuchs BD. Cost savings attributable to reductions in intensive care unit length of stay for mechanically ventilated patients. Medical care. 2008;46(12):1226–1233. pmid:19300312
View Article
PubMed/NCBI
Google Scholar

[21] View Article

[22] PubMed/NCBI

[23] Google Scholar

[ref7] 7. Moran JL, Solomon PJ, Outcome ACf, Resource Evaluation of the A, New Zealand Intensive Care S. A review of statistical estimators for risk-adjusted length of stay: analysis of the Australian and new Zealand Intensive Care Adult Patient Data-Base, 2008–2009. BMC medical research methodology. 2012;12:68. pmid:22591115
View Article
PubMed/NCBI
Google Scholar

[25] View Article

[26] PubMed/NCBI

[27] Google Scholar

[ref8] 8. Dara SI, Afessa B. Intensivist-to-bed ratio: association with outcomes in the medical ICU. Chest. 2005;128(2):567–572. pmid:16100139
View Article
PubMed/NCBI
Google Scholar

[29] View Article

[30] PubMed/NCBI

[31] Google Scholar

[ref9] 9. Williams T, Leslie G. Delayed discharges from an adult intensive care unit. Aust Health Rev. 2004;28(1):87–96. pmid:15525255
View Article
PubMed/NCBI
Google Scholar

[33] View Article

[34] PubMed/NCBI

[35] Google Scholar

[ref10] 10. Russell S. Reducing readmissions to the intensive care unit. Heart & lung: the journal of critical care. 1999;28(5):365–372. pmid:10486454
View Article
PubMed/NCBI
Google Scholar

[37] View Article

[38] PubMed/NCBI

[39] Google Scholar

[ref11] 11. Beck DH, McQuillan P, Smith GB. Waiting for the break of dawn? The effects of discharge time, discharge TISS scores and discharge facility on hospital mortality after intensive care. Intensive care medicine. 2002;28(9):1287–1293. pmid:12209279
View Article
PubMed/NCBI
Google Scholar

[41] View Article

[42] PubMed/NCBI

[43] Google Scholar

[ref12] 12. Goldfrad C, Rowan K. Consequences of discharges from intensive care at night. Lancet. 2000;355(9210):1138–1142. pmid:10791376
View Article
PubMed/NCBI
Google Scholar

[45] View Article

[46] PubMed/NCBI

[47] Google Scholar

[ref13] 13. Cooper GS, Sirio CA, Rotondi AJ, Shepardson LB, Rosenthal GE. Are readmissions to the intensive care unit a useful measure of hospital performance? Medical care. 1999;37(4):399–408. pmid:10213020
View Article
PubMed/NCBI
Google Scholar

[49] View Article

[50] PubMed/NCBI

[51] Google Scholar

[ref14] 14. Kaboli PJ, Go JT, Hockenberry J, Glasgow JM, Johnson SR, Rosenthal GE, et al. Associations between reduced hospital length of stay and 30-day readmission rate and mortality: 14-year experience in 129 Veterans Affairs hospitals. Annals of internal medicine. 2012;157(12):837–845. pmid:23247937
View Article
PubMed/NCBI
Google Scholar

[53] View Article

[54] PubMed/NCBI

[55] Google Scholar

[ref15] 15. Benbassat J, Taragin M. Hospital readmissions as a measure of quality of health care: advantages and limitations. Archives of internal medicine. 2000;160(8):1074–1081. pmid:10789599
View Article
PubMed/NCBI
Google Scholar

[57] View Article

[58] PubMed/NCBI

[59] Google Scholar

[ref16] 16. Rosenberg AL, Hofer TP, Hayward RA, Strachan C, Watts CM. Who bounces back? Physiologic and other predictors of intensive care unit readmission. Critical care medicine. 2001;29(3):511–518. pmid:11373413
View Article
PubMed/NCBI
Google Scholar

[61] View Article

[62] PubMed/NCBI

[63] Google Scholar

[ref17] 17. Rosenberg AL, Watts C. Patients readmitted to ICUs*: a systematic review of risk factors and outcomes. Chest. 2000;118(2):492–502. pmid:10936146
View Article
PubMed/NCBI
Google Scholar

[65] View Article

[66] PubMed/NCBI

[67] Google Scholar

[ref18] 18. Metnitz PG, Fieux F, Jordan B, Lang T, Moreno R, Le Gall JR. Critically ill patients readmitted to intensive care units—lessons to learn? Intensive care medicine. 2003;29(2):241–248. pmid:12594586
View Article
PubMed/NCBI
Google Scholar

[69] View Article

[70] PubMed/NCBI

[71] Google Scholar

[ref19] 19. Verburg IW, de Keizer NF, de Jonge E, Peek N. Comparison of regression methods for modeling intensive care length of stay. PLoS One. 2014;9(10):e109684. pmid:25360612
View Article
PubMed/NCBI
Google Scholar

[73] View Article

[74] PubMed/NCBI

[75] Google Scholar

[ref20] 20. Strand K, Walther SM, Reinikainen M, Ala-Kokko T, Nolin T, Martner J, et al. Variations in the length of stay of intensive care unit nonsurvivors in three Scandinavian countries. Critical care. 2010;14(5):R175. pmid:20920348
View Article
PubMed/NCBI
Google Scholar

[77] View Article

[78] PubMed/NCBI

[79] Google Scholar

[ref21] 21. Van de Klundert N, Holman R, Dongelmans DA, de Keizer NF. Data Resource Profile: the Dutch National Intensive Care Evaluation (NICE) Registry of Admissions to Adult Intensive Care Units. Int J Epidemiol. 2015;44(6):1850–1850h. pmid:26613713
View Article
PubMed/NCBI
Google Scholar

[81] View Article

[82] PubMed/NCBI

[83] Google Scholar

[ref22] 22. Zimmerman JE, Kramer AA, McNair DS, Malila FM. Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today's critically ill patients. Crit Care Med. 2006;34(5):1297–1310. pmid:16540951
View Article
PubMed/NCBI
Google Scholar

[85] View Article

[86] PubMed/NCBI

[87] Google Scholar

[ref23] 23. Koetsier A, Peek N, de Jonge E, Dongelmans D, van Berkel G, de Keizer N. Reliability of in-hospital mortality as a quality indicator in clinical quality registries. A case study in an intensive care quality register. Methods Inf Med. 2013;52(5):432–440. pmid:23807704
View Article
PubMed/NCBI
Google Scholar

[89] View Article

[90] PubMed/NCBI

[91] Google Scholar

[ref24] 24. van Sluisveld N, Bakhshi-Raiez F, de Keizer N, Holman R, Wester G, Wollersheim H, et al. Variation in rates of ICU readmissions and post-ICU in-hospital mortality and their association with ICU discharge practices. BMC Health Serv Res. 2017;17(1):281–289. pmid:28416016
View Article
PubMed/NCBI
Google Scholar

[93] View Article

[94] PubMed/NCBI

[95] Google Scholar

[ref25] 25. Brown SE, Ratcliffe SJ, Kahn JM, Halpern SD. The epidemiology of intensive care unit readmissions in the United States. Am J Respir Crit Care Med. 2012;185(9):955–964. pmid:22281829
View Article
PubMed/NCBI
Google Scholar

[97] View Article

[98] PubMed/NCBI

[99] Google Scholar

[ref26] 26. Dutch National Intensive Care Evaluation (NICE) foundation. Data in beeld. [Website]. 2014. Available from: http://www.stichting-nice.nl/datainbeeld/public [in Dutch].

[ref27] 27. Dimick JB, Staiger DO, Birkmeyer JD. Ranking hospitals on surgical mortality: the importance of reliability adjustment. Health services research. 2010;45(6 Pt 1):1614–1629. pmid:20722747
View Article
PubMed/NCBI
Google Scholar

[102] View Article

[103] PubMed/NCBI

[104] Google Scholar

[ref28] 28. Hashmi ZG, Dimick JB, Efron DT, Haut ER, Schneider EB, Zafar SN, et al. Reliability adjustment: a necessity for trauma center ranking and benchmarking. The journal of trauma and acute care surgery. 2013;75(1):166–172. pmid:23940864
View Article
PubMed/NCBI
Google Scholar

[106] View Article

[107] PubMed/NCBI

[108] Google Scholar

[ref29] 29. Verburg IW, Holman R, Peek N, Abu-Hanna A, de Keizer NF. Guidelines on constructing funnel plots for quality indicators: A case study on mortality in intensive care unit patients. Stat Methods Med Res. 2017:962280217700169. pmid:28330409
View Article
PubMed/NCBI
Google Scholar

[110] View Article

[111] PubMed/NCBI

[112] Google Scholar

[ref30] 30. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128–138. pmid:20010215
View Article
PubMed/NCBI
Google Scholar

[114] View Article

[115] PubMed/NCBI

[116] Google Scholar

[ref31] 31. Development Core Team R. A Language and Environment for Statistical Computing. 2.15.1 ed. Vienna, Austria: R Foundation for Statistical Computing; 2005.

[ref32] 32. Davies S. Fitting Generalized Linear Models 1992 [2017-09-26]. Available from: https://stat.ethz.ch/R-manual/R-devel/library/stats/html/glm.html.

[ref33] 33. Harell FE. Package rms 2017-09-26 [2017-07-18]. 5.1–0:[Available from: https://cran.r-project.org/web/packages/rms/rms.pdf.

[ref34] 34. Bates D, Maechler M, Bolker B, Walker S, Christensen R, Singmann H, et al. Package lme4. 1.1–12 ed2016.

[ref35] 35. Savicky P. Test for Association/Correlation Between Paired Samples 2014-03-20 [2017-09-26]. 0.3–0:[Available from: https://cran.r-project.org/web/packages/pspearman/pspearman.pdf.

[ref36] 36. Kramer AA, Higgins TL, Zimmerman JE. The association between ICU readmission rate and patient outcomes. Critical care medicine. 2013;41(1):24–33. pmid:23128381
View Article
PubMed/NCBI
Google Scholar

[123] View Article

[124] PubMed/NCBI

[125] Google Scholar

[ref37] 37. Hofstede SN, van Bodegom-Vos L, Kringos DS, Steyerberg E, Marang-van de Mheen PJ. Mortality, readmission and length of stay have different relationships using hospital-level versus patient-level data: an example of the ecological fallacy affecting hospital performance indicators. BMJ Qual Saf. 2017 pmid:28986516
View Article
PubMed/NCBI
Google Scholar

[127] View Article

[128] PubMed/NCBI

[129] Google Scholar

[ref38] 38. Brinkman S, Abu-Hanna A, van der Veen A, de Jonge E, de Keizer NF. A comparison of the performance of a model based on administrative data and a model based on clinical data: effect of severity of illness on standardized mortality ratios of intensive care units. Critical care medicine. 2012;40(2):373–378. pmid:21983367
View Article
PubMed/NCBI
Google Scholar

[131] View Article

[132] PubMed/NCBI

[133] Google Scholar

[ref39] 39. Pouw ME, Peelen LM, Moons KG, Kalkman CJ, Lingsma HF. Including post-discharge mortality in calculation of hospital standardised mortality ratios: retrospective analysis of hospital episode statistics. BMJ. 2013;347:f5913. pmid:24144869
View Article
PubMed/NCBI
Google Scholar

[135] View Article

[136] PubMed/NCBI

[137] Google Scholar

[ref40] 40. Mulvey GK, Wang Y, Lin Z, Wang OJ, Chen J, Keenan PS, et al. Mortality and readmission for patients with heart failure among U.S. News & World Report's top heart hospitals. Circ Cardiovasc Qual Outcomes. 2009;2(6):558–565. pmid:20031893
View Article
PubMed/NCBI
Google Scholar

[139] View Article

[140] PubMed/NCBI

[141] Google Scholar

[ref41] 41. Mainz J. Defining and classifying clinical indicators for quality improvement. International journal for quality in health care: journal of the International Society for Quality in Health Care. 2003;15(6):523–530.
View Article
Google Scholar

[143] View Article

[144] Google Scholar

[ref42] 42. Roos-Blom MJ, Dongelmans D, Arbous MS, de Jonge E, de Keizer N. How to Assist Intensive Care Units in Improving Healthcare Quality. Development of Actionable Quality Indicators on Blood use. Studies in health technology and informatics. 2015;210:429–433. pmid:25991180
View Article
PubMed/NCBI
Google Scholar

[146] View Article

[147] PubMed/NCBI

[148] Google Scholar

[ref43] 43. Verburg IWM, Holman R, Dongelmans D, de Jonge E, de Keizer NF. Is patient length of stay associated with intensive care unit characteristics? J Crit Care. 2017;43:114–121. pmid:28865340
View Article
PubMed/NCBI
Google Scholar

[150] View Article

[151] PubMed/NCBI

[152] Google Scholar

[ref44] 44. Vasilevskis EE, Kuzniewicz MW, Dean ML, Clay T, Vittinghoff E, Rennie DJ, et al. Relationship between discharge practices and intensive care unit in-hospital mortality performance: evidence of a discharge bias. Medical care. 2009;47(7):803–812. pmid:19536006
View Article
PubMed/NCBI
Google Scholar

[154] View Article

[155] PubMed/NCBI

[156] Google Scholar

[ref45] 45. Brinkman S, Abu-Hanna A, de Jonge E, de Keizer NF. Prediction of long-term mortality in ICU patients: model validation and assessing the effect of using in-hospital versus long-term mortality on benchmarking. Intensive care medicine. 2013;39(11):1925–1931. pmid:23921978
View Article
PubMed/NCBI
Google Scholar

[158] View Article

[159] PubMed/NCBI

[160] Google Scholar

[ref46] 46. Rydenfelt K, Engerstrom L, Walther S, Sjoberg F, Stromberg U, Samuelsson C. In-hospital vs. 30-day mortality in the critically ill—a 2-year Swedish intensive care cohort analysis. Acta Anaesthesiol Scand. 2015;59(7):846–858. pmid:26041018
View Article
PubMed/NCBI
Google Scholar

[162] View Article

[163] PubMed/NCBI

[164] Google Scholar

[ref47] 47. Engerstrom L, Kramer AA, Nolin T, Sjoberg F, Karlstrom G, Fredrikson M, et al. Comparing Time-Fixed Mortality Prediction Models and Their Effect on ICU Performance Metrics Using the Simplified Acute Physiology Score 3. Critical care medicine. 2016;44(11):e1038–e1044. pmid:27513546
View Article
PubMed/NCBI
Google Scholar

[166] View Article

[167] PubMed/NCBI

[168] Google Scholar

Figures

Abstract

Purpose

Materials and methods

Results

Conclusion

Introduction

Materials and methods

The Dutch National Intensive Care Evaluation registry

Study data

Quality indicators

Statistical analysis

Results

Study data and quality indicators

Statistical analysis

Discussion

Conclusion

Supporting information

S1 Table. Definitions of patients subgroups included in the study.

S2 Table. Number of included ICUs and number of included admissions for each subgroup and quality indicator.

S3 Table. Spearman rank correlation coefficients with corresponding p-values for the pairs of quality indicators standardized mortality ratio (SMR); standardized readmission ratio (SRR) and standardized ICU length of stay ratio (SLOSR).

S1 File. Results of recalibration and regression analysis to calculate predicted outcome.

Acknowledgments

References