https://orcid.org/0000-0001-8680-2028
Figures
Abstract
Purpose
This study aims to determine if the primary care provider (PCP) assessment of readmission risk is comparable to the validated LACE tool at predicting readmission to hospital.
Methods
A prospective observational study of recently discharged adult patients clustered by PCPs in the primary care setting. Physician readmission risk assessment was determined via a questionnaire after the PCP reviewed the hospital discharge summary. LACE scores were calculated using administrative data and the discharge summary. The sensitivity and specificity of the physician assessment and the LACE tool in predicting readmission risk, agreement between the 2 assessments and the area under receiver operating characteristic (AUROC) curves were calculated.
Results
217 patient readmission encounters were included in this study from September 2017 till June 2018. The rate of readmission within 30 days was 14.7%, and 217 discharge summaries were used for analysis. The weighted kappa coefficient was 0.41 (95% CI: 0.30–0.51) demonstrating a moderate level of agreement. Sensitivity of physician assessment was 0.31 (95% CI: 0.22–0.40) and specificity was 0.80 (95% CI: 0.77–0.83). The sensitivity of the LACE assessment was 0.42 (95% CI: 0.25–0.59) and specificity was 0.79 (95% CI: 0.73–0.85). The AUROC for the LACE readmission risk was 0.65 (95% C.I. 0.55–0.76) demonstrating modest predictive power and was 0.57 (95% C.I. 0.46–0.68) for physician assessment, demonstrating low predictive power.
Citation: Walji S, McIsaac W, Moineddin R, Kalia S, Levy M, Tu K, et al. (2021) Predicting hospital readmission risk: A prospective observational study to compare primary care providers’ assessments with the LACE readmission risk index. PLoS ONE 16(12): e0260943. https://doi.org/10.1371/journal.pone.0260943
Editor: Tamer Hudali, University of Alabama at Birmingham, UNITED STATES
Received: February 25, 2021; Accepted: November 20, 2021; Published: December 15, 2021
Copyright: © 2021 Walji et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All data underlying the findings described in the manuscript are not fully available/without restriction. The data used within this study was obtained through the Institute for Clinical and Evaluative Sciences (ICES), and as such there are legal restrictions to sharing the dataset. As well, given that the data includes patient health information, there are ethical restrictions in regard to the sharing of data, whereby the lead study institution, Sinai Health System, would require a data transfer agreement to be approved and in place prior to transferring or sharing the data. Requests for access to the data should be directed to Dr. Sakina Walji, at sakina.walji@sinaihealth.ca, and Stefana Jovanovska, at stefana.jovanovska@ices.on.ca.
Funding: This study received grant funding from the University of Toronto’s UTOPIAN Ideas to Proposal program. The funder provided support to cover the cost of conducting the study but did not have any additional role in the study design, data collection and analysis, decision to publish or preparation of the manuscript.
Competing interests: Chaim Bell, a co-author on this project, is a medical consultant to the Ontario Ministry of Health, however the Ministry of Health had no role in this study. This commercial affiliation does not alter our adherence to PLOS ONE policies on sharing data and materials. There are no additional competing interests to declare.
Abbreviations: PCP, Primary care provider; OHIP, Ontario Health Insurance Plan; AUROC, area under receiver operating characteristic curves; DAS, Data and Analytic Services; SPOR, Strategy for Patient-Oriented Research; AUC, area under the curve
Introduction
Readmission after hospitalization is common and costly [1, 2]. It affects almost one in ten of all hospitalizations and up to one-third may be avoidable [1, 3, 4]. Many jurisdictions have identified 30-day hospital readmissions as a key quality indicator [5]. Similarly, strategies to reduce rates of readmissions have become a priority in many countries [6, 7].
Several studies provide support for the effectiveness of early follow-up post-hospital discharge in the primary care setting to reduce readmission [8–11]. Patients aged 65 and over who do not have physician follow-up within 30 days of discharge are three times more likely to be re-admitted [8]. Follow-up post-discharge was associated with a 19% lower chance of readmission for patients with congestive heart failure [9], and a 15% reduction in readmission to hospital after high-risk surgery associated with complications [10]. Follow-up within 7 days post-discharge from hospital in the primary care setting has been used in primary care quality frameworks [11].
A number of readmission risk assessment tools have been developed to identify high-risk patients who may benefit from early post-discharge follow-up [12–15]. The LACE index is a validated score that assesses the risk of death or unplanned readmission after discharge from hospital [12]. This tool has been suggested to help identify patients needing post-discharge interventions [16]. However, some information needed to calculate the score, such as emergency room visits in the previous 6 months, may not be readily available to primary care providers (PCPs) at the time an assessment regarding the need for early post-discharge follow-up is being made.
Primary care providers who possess detailed clinical and social knowledge of patients in the community gained over time may be able to predict which patients are at higher risk for hospital readmission. This study aims to determine if the primary care provider assessment of readmission risk is comparable to the validated LACE tool at predicting readmission to hospital within 30 days, for patients who have been discharged from hospital within the last 14 days.
Methods
Study setting and design
This study took place in Toronto, Canada between September 2017 and June 2018. Three family practice clinics of varying size participated. The sites included 31 physician faculty and a total of approximately 27,500 rostered patients in the sampling frame of the study. This study received ethics approval from the Mount Sinai Hospital Research Ethics Board (MSH REB #17-0173-E). Written consent was obtained for all participating clinicians.
This was a prospective observational study of recently discharged adult patients clustered by PCPs to observe agreement between PCP risk assessment of readmission compared to the LACE tool against a criterion standard of observed hospital readmission within 30 days in any hospital in Ontario, Canada.
Determination of PCP and LACE risk estimates for readmission
To determine PCP estimate of the risk for an individual patient of being readmitted, hospital discharge summaries from the patient’s electronic medical record or faxed to the clinic from the hospital were identified from the period of September 2017 to June 2018. Only discharge summaries for adults 18 years of age or older cared for by a consenting study physician, and received within 14 days of hospital discharge, were eligible.
A survey was attached to each hospital discharge summary that asked the PCP to rate their risk of readmission for that patient within 30 days as low, moderate or high, as well as the factors that led him/her to that decision (S1 File).
The LACE tool provides a risk estimate of readmission based on clinical and administrative data elements to provide a risk score from 0–19. The variables comprising the score include length of stay (“L”); acuity of the admission (“A”); comorbidity of the patient (measured with the Charlson comorbidity index score) (“C”) [17]; and emergency department use (measured as the number of visits in the six months before admission) (“E”) [12]. The Charlson comorbidity index is determined by assigning a score for each comorbid condition depending on the risk of dying associated with each condition. In the original study, the LACE score for each patient was classified into (i) low risk (with LACE score 0–8) corresponding to < 10% risk of readmission, (ii) medium risk (with LACE score 9–13) corresponding to 10–20% risk of readmission and (iii) high risk (with LACE score 14–19) corresponding to >20% risk of readmission [12].
Ontario has universal health care coverage for visits to doctors in emergency departments, clinics, and for hospital admissions under the Ontario Health Insurance Plan (OHIP). These visits are captured using a unique identifier for each eligible resident of the province and all analyses are considered population based. This administrative data is available for research upon request through ICES (www.ices.on.ca, n.d.). Discharge summaries were linked to the administrative data through a combination of patient identifiers which included hospital number, patient name, hospital of admission and date of admission. Administrative data was also used to determine the number of emergency department visits in the prior 6 months, as well as whether the patient was readmitted within 30 days. This was then linked to the data from the hospital discharge summary and physician risk estimate. Anonymized data was sent back to the research team through a secure portal. Data on length of stay, acuity and co-morbidities was retrieved manually from the discharge summary.
Statistical analysis
Agreement between the LACE and physician risk assessment was quantified using the weighted kappa statistic to account for differences on ordinal scale of predicting 30-days readmission (low, medium, high). The weighted kappa statistic can be interpreted as indicating slight agreement (0.01–0.20), fair agreement (0.21–0.40), moderate agreement (0.41–0.60), substantial agreement (0.61–0.80), and almost perfect agreement (0.80–0.99) [18]. The association between the readmission rate derived from physician risk assessment and LACE tool was assessed using two-sided Cochrane-Armitage trend test.
Sensitivity and specificity for predicting a high risk for hospital readmission was estimated by collapsing the “low risk” and “medium risk” categories into a single group for the PCP assessment, as indicated by the treating PCP. This was done to be consistent with the LACE categorization of high-risk readmissions as per the original paper. We felt it would be most prudent to identify and closely follow-up those patients deemed to be high-risk for readmission. These diagnostic measures were estimated for the LACE tool and physician assessment.
The area under receiver operating characteristic curves (AUROC) was used to estimate the predictive power of 30-days readmission for LACE score and physician assessment. An AUROC value of 0.5 suggests no discriminatory predictive power, 0.7 to 0.8 as acceptable discriminatory predictive power and greater than 0.8 as excellent discriminatory predictive power [19]. We compared the AUROC of LACE tool and physician assessment tool using the non-parametric approach, as further described by DeLong, Delong and Clarke-Pearson [20].
Multiple logistic regression models were fitted to describe the relationship for LACE score and physician assessment with respect to the 30-days readmission, while adjusting for patient’s age and gender [21]. We conducted complete case analysis when fitting the multivariable logistic regressions.
All analyses were conducted using SAS v9.4.
This study contracted ICES Data and Analytic Services (DAS) and used de-identified data from the ICES Data Repository, which is managed by ICES with support from its funders and partners: Canada’s Strategy for Patient-Oriented Research (SPOR), the Ontario SPOR Support Unit, the Canadian Institute of Health Research and the Government of Ontario. The opinions, results and conclusions reported are those of the authors. No endorsement by ICES or any of its funders or partners is intended or should be inferred.
Results
Twenty-one of 31 eligible physicians (67.7%) agreed to participate in the study. A total of 257 discharge summaries were collected across the 3 sites during the study period. Eight summaries were excluded as they included pediatric or obstetric cases. Physicians completed survey questionnaires for 238 (96.6%), of which 21 were excluded as the provider was aware of the readmission prior to completing the survey leaving 217 on which analysis was completed. The LACE score was able to be calculated for 208 patients and physician assessments were available for 202 patients (Fig 1).
The overall rate of readmission within 30 days was 32/217 (14.7%). There was no statistically significant difference in proportion of readmission when comparing patients less than 64 years of age with more than 65 years of age, males with females or low-income group with high income group (Table 1).
Both the LACE tool and physician assessment identified a higher proportion of older patients (age 65+) than younger patients, a higher proportion of male than female patients and a higher proportion of low income group than high income group to be at high risk of readmission.
Patients with at least one emergency visit in the previous 6 months had 18.6% readmission rate while patients with no emergency visit had 9.1% readmission rate (p = 0.05) (Table 2).
The LACE tool categorised 97/208 (46.6%) as low risk, 60/208 (28.8%) as medium risk and 51/208 (24.5%) as high risk. Physicians estimated the risk of readmission low for 101/202 (50.0%), medium for 58/202 (28.7%) and high for 43/202 (21.3%).
Agreement between physician risk assessment and LACE instrument estimates
The weighted kappa coefficient was 0.41 (95% CI: 0.30–0.51) demonstrating the presence of statistically significant agreement between the LACE readmission risk tool and physician assessment demonstrating a statistically significant moderate level of agreement.
When LACE score deemed patient to be low risk of readmission, there was 76% agreement with physician assessment. In contrast, when LACE score deemed patient to be high risk, there was 43% agreement with physician assessment. Overall, there was 57.8% agreement (i.e. (71+20+21)/194) between physician risk assessment and LACE risk groups (Table 3).
The odds of being readmitted within 30 days increased by 1.16 with every one point increase in LACE score (95% CI: 1.04–1.28; p-value = 0.004). The AUROC estimated for the LACE readmission risk estimate was 0.65 (95% C.I. 0.55–0.76) demonstrating modest predictive power. The AUROC estimate for the physician risk for readmission suggested a lower level of discrimination with a value of 0.57 (95% C.I. 0.46–0.68) (Fig 2). However, the AUROC of LACE readmission was not statistically significant at the nominal level of 5% to the AUROC of physician assessment (Chi-square = 3.11, p = 0.08).
When adjusting for age range and gender, the odds of 30-days readmission among patients labelled in high risk group by LACE score was 3.61 times higher than the odds of readmission among patients labelled in low risk group by LACE score (95% CI: 1.37–9.55; p-value = 0.009).
Accuracy of physician and LACE estimates of risk of readmission
There was a 9% rate of readmission within 30 days when LACE score was low risk, 15% risk of readmission when LACE score was medium risk and 25% rate of readmission when LACE score was high risk, indicating statistically significant trend in the increasing rate of re-admission with respect to LACE score (P-value = 0.009) (Table 4). Similarly with physician assessment, there was a 12% readmission rate within 30 days when physician assessment was deemed to be low risk, 14% risk of readmission when physician assessment was medium risk and 21% risk of readmission when physician assessment was high risk, indicating statistically insignificant trend in the increasing rate of re-admission with respect to physician assessment (P-value = 0.18) (Table 4).
The diagnostic estimates for the physician risk assessment were: (1) sensitivity = 0.31 (95% CI: 0.22–0.40) and (2) specificity = 0.80 (95% CI: 0.77–0.83). The diagnostic estimates for LACE group were: (1) sensitivity = 0.42 (95% CI: 0.25–0.59) and (2) specificity = 0.79 (95% CI: 0.73–0.85).
Discussion
Our study of 21 PCPs at three sites evaluated 194 patients recently discharged from hospital. We found a moderate level of agreement between the LACE tool and physician assessment for predicting the risk of hospital readmission within 30 days after being discharged. The LACE index performed slightly better than physician assessment with moderate predictive power, although this difference was not statistically significant.
The overall rate of readmission in our study was 14.7%. The original LACE study had a readmission rate of 8% [12] which was comparable to population-based data [1]. This may be because the clinics involved in this study were mostly affiliated with tertiary care hospitals where patients may be more complex or because of secular changes over time.
A number of studies assessing the LACE tool have been performed, but these are often in certain population groups e.g. cardiovascular disease [22], chronic obstructive pulmonary disease [23] or in older people [24] and they show conflicting results. The results of this study compare with the Damery study [25] where increasing LACE score and certain components of the LACE index were independent predictors of readmission. A trade-off between sensitivity and specificity can be observed with increasing sensitivity and decreasing specificity as the threshold for LACE score to classify patients as high risk for 30 days readmission is decreased.
Both methods performed sub-optimally; the PCP assessment of readmission risk may not be as effective as the LACE tool. There was overlap of the 95% C.I. for both AUROC deeming the difference not statistically significant. Studies looking at clinician assessment of readmission risk have shown varied results with one demonstrating an AUC derived from ROC of 0.689 for the risk assessment completed by the discharging attending on a hospital team vs. 0.620 for the LACE tool as a predictor [26]. Allaudeen et al. found poor ability of inpatient teams (physician, nurse and case manager) to discriminate between readmissions vs. non-readmissions (AUC from ROC of 0.5786) [27]. However, PCPs have long term patient relationships that may affect their assessment differently as a result of their knowledge of a patient’s social conditions and overall medical status. To the best of our knowledge there are no studies looking at the PCPs’ judgment of readmission, but the results of our study are comparable to the Allaudeen study which showed poor discriminatory power of clinicians in assessing readmission risk. This may indicate that the relationship and contextual knowledge a primary care provider has with a patient does not improve assessment of readmission risk and that the information in the EMR is not readily accessible to the PCP to allow for pertinent decision making.
If the LACE score can easily be calculated and provided to the PCP by the discharging physician it may be helpful in conjunction with physician assessment in deciding who requires more intensive follow-up after discharge. Future directions may include calculating the sensitivity and specificity of a score that combines the LACE tool with physician assessment and to observe whether implementation of such a tool can reduce readmission rates.
Our study has limitations that merit emphasis. ICES data was used to collect data on emergency department use in order to calculate the LACE score. Currently in our healthcare system emergency room visits are not comprehensively reported to PCPs. As a result, it may be more practical for the hospital discharge team to provide the PCPs with a readmission risk score. The analysis was performed without taking into account the unique structure of patient-physician hierarchy (i.e. clustering) due to small sample size of this study. As there are no significant results, further adjustments for clustering would not change these results. During the study period there was a change in the process with regards to how discharge summaries were received resulting in some physician assessments being lost. Furthermore, only readmissions where discharge summaries were received by the PCP were included, and there is a possibility that some discharges were not included. This may have weakened or underestimated the sensitivity and specificity; however this would apply to both the LACE estimates as well as the PCP estimates. We did not assess the quality of the discharge summary, the instructions provided for follow-up care or if this impacted their decision making. However physician response to the survey indicates the factors most likely to influence their assessment were familiarity with the patient and the hospital diagnosis. Finally, this study did not observe effects of implementing an intervention using the LACE tool or physician risk assessment on readmission rates but could be an opportunity for future direction.
Conclusion
The LACE index shows moderate discriminatory power in patients at increased risk for readmission after discharge, and may be superior when comparing to PCP’s clinical judgement of readmission risk. Obtaining all information needed to calculate the LACE score such as emergency department visits may be difficult to obtain in primary care. Consideration should be given to providing the LACE score in the hospital discharge summary sent to the PCP to facilitate identification and early follow up of patients at high risk for hospital readmission. Future studies can include the effectiveness of combining physician assessment with elements of the LACE tool in predicting hospital readmission.
Supporting information
S1 File. Re-admission risk assessment survey.
https://doi.org/10.1371/journal.pone.0260943.s001
(DOCX)
References
- 1.
Canadian Institute for Health Information. All-cause readmission to acute care and return to the emergency department [Internet] Ottawa, Ont: CIHI; 2012 [cited 2019 Nov 25]. Available from: https://secure.cihi.ca/free_products/Readmission_to_acutecare_en.pdf
- 2. Jencks SF, Williams MV, Coleman EA. Rehospitalizations among patients in the Medicare fee-for-service program. N Engl J Med. 2009 Apr 2;360(14):1418–1428. pmid:19339721
- 3. Anderson GF, Steinberg EP. Hospital readmissions in the Medicare population. N Engl J Med. 1984 Nov 22;311(21):1349–53. pmid:6436703
- 4.
Medicare Payment Advisory Commission. Report to the Congress: Promoting Greater Efficiency in Medicare [Internet]. Washington: MedPAC; 2007 Jun. Available from http://www.medpac.gov/docs/default-source/reports/Jun07_EntireReport.pdf
- 5. Baker GR, Abrams HB, Bell C, Blackstien-Hirsch P, Brown G, Cochrane PA, et al. Enhancing the Continuum of Care—Report of the Avoidable Hospitalization Advisory Panel [Internet]. Ministry of Health and Long-Term Care; 2011 Nov [cited 2015 Oct]. Available from: http://www.health.gov.on.ca/en/common/ministry/publications/reports/baker_2011/baker_2011.pdf
- 6. Conroy S, Dowsing T. What should we do about hospital readmissions? Age Ageing. 2012 Nov;41(6):702–4. pmid:23045361
- 7. Epstein AM. Revisiting readmissions–changing the incentives for shared accountability. N Engl J Med. 2009;360:1457–9. pmid:19339727
- 8. Ham C, Imison C, Jennings M. Avoiding hospital admissions—Lessons from evidence and experience—Seminar Highlights [Internet]. The King’s Fund; 2010. Available from: https://www.kingsfund.org.uk/sites/default/files/field/field_publication_file/avoiding-hospital-admissions-lessons-from-evidence-experience-ham-imison-jennings-oct10.pdf
- 9. Lee KK, Yang J, Hernandez AF, Steimle AE, Go AS. Post-discharge follow-up characteristics associated with 30-day readmission after heart failure hospitalization. Med Care. 2016 Apr;54(4):365–372. pmid:26978568
- 10. Brooke BS, Stone DH, Cronewett JL, Nolan B, DeMartino RR, MacKenzie TA, et al. Early primary care provider follow-up and readmission after high-risk surgery. JAMA Surg. 2014 Aug;149(8):821–8. pmid:25074237
- 11.
Health Quality Ontario. Primary Care Quality Improvement Planning for 2015/2016 [Internet] 2014 Dec 2 [cited 2015 Oct 18]. Available from: http://www.hqontario.ca/portals/0/Documents/qi/qip-webinar-pc-1501-en.pdf
- 12. van Walraven C, Dhalla IA, Bell C, Etchells E, Stiell IG, Zarnke K, et al. Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. CMAJ. 2010 Apr 6;182(6):551–557. pmid:20194559
- 13. Amarsingham R, Moore BJ, Tabak YP, Drazner MH, Clark CA, Zhang S, et al. An automated model to identify health failure patients at risk for 30-day readmission or death using electronic medical record data. Med Care. 2010;48(11):981–8. pmid:20940649
- 14. Donze J, Aujesky D, Williams D, Schnipper JL. Potentially avoidable 30-day hospital admissions in medical patients: derivation and validation of a prediction model. JAMA Intern Med. 2013 Apr 22;173(8):632–8. pmid:23529115
- 15. Kansagara D, Englander H, Salanitro A, Kage D, Theobald C, Freeman M, et al. Risk prediction models for hospital readmission: a systematic review. JAMA. 2011;306(15):1688–1698. pmid:22009101
- 16. Gruneir A, Dhalla IA, van Walraven C, Fischer HD, Camacho X, Rochon PA, et al. Unplanned readmissions after hospital discharge among patients identified as being at high risk for readmission using a validated predictive algorithm. Open Med. 2011 May 31;5(2):e104–111. pmid:21915234
- 17. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40(5):373–83. pmid:3558716
- 18. Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med. 2005 May;37(5):360–3. pmid:15883903
- 19. Altman DG. Diagnostic tests 3: receiver operating characterstic plots. BMJ. 1994 Jul 16;309(6948):188. pmid:8044101
- 20. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach. Biometrics. 1988 Sept; 44(3):837–845. pmid:3203132
- 21.
Hosmer DWJ, Lemeshow S, Sturdivant RX. Applied logisitc regression. 3rd ed. New Jersey: John Wiley & Sons; 2013.
- 22. Yazdan-Ashoori P, Lee SF, Ibrahim Q, Van Spall HGC. Utility of the LACE index at the bedside in predicting 30-day readmission or death in patients hospitalized with heart failure. Am Heart J. 2016 Sep 1;179:51–58. pmid:27595679
- 23. Bashir B, Schneider D, Naglak MC, Churilla TM, Adelsberger M. Evaluation of prediction strategy and care coordination for COPD readmissions. Hosp Pract (1995). 2016 Aug;44(3):123–8. pmid:27391991
- 24. Cotter PE, Bhalla VK, Wallis SJ, Biram RW. Predicting readmissions: poor performance of the LACE index in an older UK population. Age Ageing. 2012 Nov;14(6):784–9.
- 25. Damery S, Combes G. Evaluating the predictive strength of the LACE index in identifying patients at high risk of hospital readmission following an inpatient episode: a retrospective cohort study. BMJ Open. 2017 Jul 13;7(7):e016921. pmid:28710226
- 26. Miller WD, Nguyen K, Vangala S, Dowling E. Clinicians can independently predict 30-day hospital readmissions as well as the LACE index. BMC Health Serv Res. 2018 Jan 22;18: 32. pmid:29357864
- 27. Allaudeen N, Schnipper JL, Orav EJ, Wachter RM, Vidyarthi AR. Inability of providers to predict unplanned readmissions. J Gen Intern Med. 2011;26(7):771–6. pmid:21399994