http://orcid.org/0000-0002-6427-1964
Figures
Abstract
Methods
Construct validity of acute morbidity was established by comparison to other outcomes using linear and logistic regression models.
Results
Data of 4608 patients were analysed. 1869 patients (40.6%) fulfilled the criteria for acute morbidity. Using multivariate analyses, acute morbidity was associated with outcomes such as hospitalisation (OR: 11, 95%-CI 9–13), mortality (OR 15, 95%-CI 6–49), and ICU admission (OR: 46, 95%-CI 25–96). Reliability of the construct “acute morbidity” was estimated using Cohens Kappa, which was 0.96 for intra-rater and 0.94 for inter-rater reliability.
Citation: Schmid F, Malinovska A, Weigel K, Bosia T, Nickel CH, Bingisser R (2019) Construct validity of acute morbidity as a novel outcome for emergency patients. PLoS ONE 14(1): e0207906. https://doi.org/10.1371/journal.pone.0207906
Editor: Bala Subramaniam, Beth Israel Deaconess Medical Center, UNITED STATES
Received: August 30, 2017; Accepted: November 8, 2018; Published: January 2, 2019
Copyright: © 2019 Schmid et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its supporting information files.
Funding: The authors received no specific funding for this work.
Competing interests: Roland Bingisser is editor, Christian H. Nickel is co-editor of medstandards.com, a symptom-based decision-support tool licensed to 150 European hospitals, owned by the University Hospital of Basel. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
Introduction
Emergency department (ED) visits occurring independently of the acuteness of the health status [1–3] are one of the drivers of crowding [2, 4, 5]. Length of stay is prolonged due to crowding, resulting in adverse outcomes [2, 4]. Emergency physicians (EP) focus on serious, life-threatening illness, and the potential need of invasive procedure or intensive care (ICU) [6, 7]. Outcome research focuses on hospitalisation, length of stay, readmission, morbidity, and mortality [8–12]. Certain outcomes, such as hospitalisation and length of stay, are influenced by non-medical factors [12, 13] others, such as ICU admission, by differences in healthcare systems [10], and mortality—being the hardest of outcomes—is infrequent in ED all-comer studies. Therefore, emergency care research should report additional outcomes, such as the acuteness of health status, or “acute morbidity” [6, 14]. Some tools assessing acute morbidity are available, but have no reported applicability [6, 15]. Acute morbidity can be defined as any potentially life-threatening condition, or any condition requiring early intervention to prevent disability, deterioration, or death [16–19]. While there is an association between acute morbidity and mortality, construct validity and reliability has not yet been formally addressed. We therefore adapted a framework of acute morbidity [18] and assessed its reliability and construct validity, using well-described outcomes as comparators.
Methods
Study design and setting
This study was performed at the ED of the University Hospital Basel, Switzerland, a tertiary 700-bed hospital with an annual ED census of over 50‘000. Obstetric and paediatric patients are treated nearby. For seasonal reasons, the study was conducted from October 21st to November 11th 2013 and from February 1st to February 23rd 2015, and was approved by the ethics committee northwest- and central Switzerland (236/13, www.eknz.ch) and Swissethics (BASEC = Business Administration System for Ethics Committees).
Selection of participants
All adult patients presenting to the ED were screened at presentation and prospectively included, if they consented, were capable to answer questions, and did not withdraw their consent before submission of the manuscript. Informed consent was obtained verbally. The ethics committee waived written consent due to the argument that inclusion could decrease, which would impede validity of an all-comer study. Capacity to consent was determined by a short interview conducted by the study personnel at presentation. Patients had to be able to understand the purpose of the study and to be able to answer questions regarding their complaints.
Study protocol
The study team consisted of medically trained personnel, instructed to collect online data from the hospital’s electronic health records (EHR), and to interview patients and physicians. Twenty-one days of 3 shifts were covered in each period. Findings were documented in a standardized questionnaire, which is shown in S1 Table. The data were double-checked and digitalized by an external institution (Health Care Research Institute AG, Zürich, Switzerland).
Data collection
Patients presenting were electronically registered and an EHR was created. Study personnel recorded all data between presentation and discharge/disposition. Patients were initially triaged by a nurse or an EP using the German version of ESI (Emergency Severity Index). [20], and the physician’s first impression of disease severity was recorded on a numeric rating scale from 0–10 (0: not ill, 10: extremely ill). All external examinations were recorded. End of ED workup was defined by discharge, disposition, or death.
Patient characteristics (i.e. age, sex, comorbidities, and ICD-10 diagnoses), dispositions (i.e admissions to wards or ICU, length of stay), follow-up presentations, readmissions, and mortality were taken from the patients’ EHR. One year after presentation, survival was recorded with the help of the EHR, phone-calls (with patients, proxies, or family physicians), or the registry office of Basel.
Medical chart review methods
Medical chart review methods were based on the twelve criteria of Worster [21], which are based on the eight criteria of Gilbert [22]. Eleven of twelve criteria were fulfilled, namely training of abstractors, selection of case criteria, definition of variables, abstraction of forms, monitoring of performance, mentioning and testing of interrater reliability (IRR), identification of medical records, use of sampling methods, planning of missing data management, and institutional review board approval.
Acute morbidity [19] was defined for non-trauma patients and revised for all-comers [18]. After a predefined chart review of 100 patients, Rohacek’s definition was refined. The chart review was independently performed by two physicians, in case of disagreement, a referee (i.e. senior physician) was included. A data abstraction form was used to document results [18].
The refined framework consists of fourteen rationales for the composite endpoint “acute morbidity”: administration of antibiotics, virostatics, antifungals, immunosuppressives, diuretics, anticoagulants, antihypertensives, and procoagulants; the need for invasive interventions, or prolonged monitoring; new neurological deficits, or seizures; fractures, or self-harm. One of the fourteen rationales was attributed—if several rationales were attributable, the one needing the most resources was chosen according to rules of ICD-10 and Swiss Diagnosis Related Groups [23, 24]. A detailed description of the original [18] and refined framework for the classification of acute morbidity is shown in S2 Table.
Reliability
For intra-rater reliability, 100 charts were extracted by random sampling. After a month, these charts were independently re-assessed by the original rater. For inter-rater reliability, another 100 charts were extracted for comparison of the two ratings before the referee’s decision.
Definitions
ESI score was defined as the urgency level a patient has to be treated according to the ESI handbook, German version. [20]
For physician’s disease severity rating (PDSR) the question “how ill does this patient look right now” was asked by the study personnel. The answer was given using a numeric scale from 0 to 10 (0: not ill at all, 10: extremely ill).
Resources were defined as the sum of all external resources used during the ED workup as defined by the ESI.
Emergency department length of stay (ED LOS) was defined as the time in minutes from presentation to discharge, disposition, or death.
Hospitalisation was defined according to Swiss law (LOS ≥24h, including immediate disposition to other hospitals).
In-hospital length of stay (LOS) was defined as the days spent in hospital during index hospitalisation.
ICU-admission was defined as any stay on a medical, surgical, or neurosurgical intensive care unit, or on a stroke or intermediate care unit during index hospitalisation.
In-hospital mortality was defined as death between presentation and discharge.
One-year mortality was defined as death within 365 days after the index presentation to the ED.
Follow-up presentation was defined as visit to the ED or the hospital within 365 days after the index presentation.
Readmission was defined as admission within one year after discharge from the index admission.
Charlson Comorbidity Index (CCI) was calculated for all hospitalised patients. The CCI was calculated using the ICD-10 based Halfon-Version coding system [25, 26]. All ICD-10 coding was performed by a team of professional certified coders of the University Hospital Basel.
Statistical analysis
Overall sample characteristics were determined using univariate analyses. Unadjusted comparison between patients with acute morbidity and categorical or binary variables was performed using Chi-square test or exact Fisher test; T-test or Kruskall-Wallis test were performed for continuous variables. Results are expressed as mean value with corresponding standard deviation (SD), median with corresponding interquartile range (IQR), or as percentage of the study population.
Univariate and multivariate (adjusted for age and sex) logistic regression models were performed to calculate the association between acute morbidity and the binary outcomes (I) hospitalisation, (II) ICU admission, (III) in-hospital mortality, and (IV) one-year mortality. Results are shown as odds ratio (OR) with 95% confidence interval (95%-CI). For the metric variables (I) ED-LOS, (II) LOS, (III) readmission, and (IV) CCI-Halfon, uni- and multivariable linear regression models were performed. Results are expressed as differences of means with 95%-CI. In order to detect the association between acute morbidity and (I) resources and (II) follow-up presentations, uni-and multivariable Poisson regression models were performed (using a quasi-poisson distribution). Results are expressed as estimates (factor of the mean) with 95%-CI.
The inter-rater and intra-rater reliability of the variable acute morbidity was estimated with weighted Cohens Kappa.
A p-value of <0.05 was considered significant for all tests. All calculations were performed with the statistical softwares SPSS (BM SPSS Statistics 22) and R (Version 3.3.1).
Results
During the study periods 4703 (83.5%) of all patients presenting to the ED were screened. After exclusion of 95 patients, 4608 patients (81.1%) were included (see Fig 1).
Table 1 shows the patient characteristics and classifications. The median age was 51 years (IQR 33;72). A total of 1869 (40.6%) patients suffered from acute morbidity. Of all patients with acute morbidity, 1203 (64.4%) were hospitalised, 290 (15.5%) were admitted to an ICU, 69 (3.69%) died during hospitalisation, and 208 (11.6%) died within one year after presentation. 267 patients (5.8%) were lost to follow up, 154 (57.7%) being men and 71 (26.6%) being classified with acute morbidity. The median ED LOS for patients with acute morbidity was 236 minutes (IQR 140;362), compared to 160 minutes (IQR 91;264) without acute morbidity. No difference was found for sex. In the group classified without acute morbidity, nine patients were triaged with an ESI of 1, nine patients were admitted to an ICU, and four patients died during hospitalisation. A detailed description of the suspected diagnosis, the discharge diagnosis and the explanatory statement for these cases is given in S3 and S4 Tables.
The rationales for acute morbidity are shown in Table 2, the top ten being antibiotics, invasive procedure, fracture, prolonged monitoring, new neurological deficit, immunosuppressives, antihypertensives, anticoagulants, diuretics and seizure. These rationales account for 1806 (96.6%) of all acute morbidity cases. The subgroup “others” includes virostatics, antifungals, pro-coagulants, and risk of self-harm.
The inter-rater reliability was estimated by weighted Cohens Kappa, which was 0.94. Weighted Cohens Kappa for the intra-rater reliability was 0.96.
A significant positive association between acute morbidity and ED LOS, hospitalisation, ICU admission, LOS, in-hospital mortality, one-year mortality, resources and follow-up presentations was found (see Table 3). No significant positive association was found for CCI-Halfon (p = 0.27) and readmissions (p = 0.17). To demonstrate the independent association between acute morbidity and the different variables logistic, linear and Poisson regression models adjusted for age and sex, were used. The results are comparable to the results from the unadjusted analyses (see Table 3).
Discussion
The main findings of this study are the excellent reliability of the novel outcome acute morbidity, and a good construct validity. All traditional outcomes (i.e. mortality, ICU-admission, hospitalisation, LOS, etc.) have their disadvantages, such as a low incidence or a high interaction with social and resource-associated factors, taking into account that simple survival is not the ultimate goal for every patient, and most studies in emergency medicine cannot be powered for mortality, as it fortunately is infrequent. Thus, our novel construct “acute morbidity” was associated with other outcomes in order to validate it: First, the association with the outcome hospitalisation is excellent, the age-adjusted OR being 11. As hospitalisation is not only dependent on acuteness of the health status, but also on availability of care (e.g. by proxies or caregivers), other outcomes needed to be addressed. Second, the association between acute morbidity and in-hospital mortality is excellent, the age-adjusted OR being 15. Four patients without acute morbidity died during hospitalisation—all for reasons unrelated to presentation. Further, long-term mortality, though being lightly associated (odds ratio of 2), was much less related with acute morbidity than short-term mortality. This supports the interpretation that acute morbidity and acute mortality are highly associated, not only by coincidence. Third, the association between acute morbidity and intensive care is highly associated—the limiting factor being the interdependence of both outcomes via the rationale “prolonged monitoring”. Therefore, this association must be interpreted with care (see limitations). Nine patients without acute morbidity were admitted to ICU—all nine with conditions unrelated to presentation, e.g. due to postoperative monitoring for non-urgent operations in the course of hospitalisation.
Other outcomes were also highly associated with acute morbidity: hospital length of stay with OR of 3, and ED length of stay with odds of 54. ED-LOS may be a measure of acuteness or severity, as patients with minor illness tend to be discharged early. As of now, all these associations have not been reported in the literature.
Furthermore, patients with acute morbidity had higher use of resources and more follow-up presentations, another possible relation to severity of disease. Of course, even if the associations were statistically adjusted for age and sex, age and co-morbidity must still be taken into account. Patients with acute morbidity were 20 years older (64 vs. 44 y. Interestingly, CCI did not show any difference between morbidity and no acute morbidity (data not shown).
Other studies have brought forward different constructs for acute morbidity, but they focused on acuteness of disease as a predictor rather than an outcome [15, 27]. Relying on disabling disease codes, such measures have advantages predicting e.g. readmission rates with an OR >5, without associations between predictors and outcomes [28]. However, it seems difficult in clinical practice to use such scores on a daily basis as they depend on lists of specific diseases, typically needing information technology (IT) based decision support.
Is the novel outcome of acute morbidity of any practical use? First, there is increasing interest in clinical practice, quality management, and research in outcomes other than survival [6]. Second, this novel outcome is assessable immediately after work-up (after four hours in our population). Therefore, it is the first outcome available in ED patients. According to this definition, the likelihood of further deterioration in patients without acute morbidity seems exceedingly low: Both in-hospital mortality and intensive care occurred in 0.2% of these patients. As one-year mortality is below 3% in these patients, a truly benign outcome may be predicted. It can be argued that identifying 59% of patients without acute morbidity is the focus of emergency medicine anyway—with or without novel constructs. This may indeed be the case, but some arguments merit consideration: First, hospitalisation is one of the most expensive decisions EPs can take. General rules for this decision do not exist, but individual reasoning is most commonly used. Second, teaching junior physicians how to decide—often under pressure—whom to hospitalise, remains a challenge. Constructs, such as acute morbidity, may support rational decisions. Third, further outcome research in emergency medicine is in need of such constructs. Fourth, all rationales of the composite outcome can be operationalised. Therefore, with the aid of computers, all rules can be implemented in EHRs in order to establish the outcome acute morbidity online during ED work-up.
Limitations
The main limitations are the single centre approach, the lack of an independent validation cohort, and the interdependence between acute morbidity and ICU admission. First, the framework was initially developed and later refined in our hospital with our patients therefore, external validity may be compromised. Additionally, the 6 weeks study period did not cover all seasons, only fall and winter. However, the ED of the University Hospital Basel is comparable to other urban European EDs, as the high number of ongoing multi-centre studies allows for comparisons. Second, an external independent validation cohort could increase external validity. Third, due to one of the predefined rationales (need for prolonged monitoring), the odds ratio for ICU admission is particularly high. We have decided to report these extremely high odds, because only 12.4% of all patients with acute morbidity were attributed to this subgroup of the composite endpoint. Still, a clear bias must be admitted due to this interdependence of the associations between the two outcomes. Generally, in validation of related outcomes, there is often a “separation phenomenon” involved; namely, most events may be associated with a predictor category. E.g. practically no ICU admissions were found in the group without acute morbidity. Therefore, all models tested tend to an infinite OR (high OR with broad confidence intervals). This, however, is a generic problem in related outcomes. Furthermore, 16.5% of all patients presenting were not screened. This was most likely due to crowding, when patients with minor health problems were rapidly discharged or transferred. Hence, these patients could be underrepresented. Yet another bias is due to the missing data for ambulatory patients concerning readmission and CCI. Moreover, 6% of patients were lost to follow up. If a bias could have occurred, it is most likely an overestimation of 1-year mortality, as patients lost seemed less ill (26% acute morbidity vs. 41% acute morbidity in all-comers).
Conclusion
Acute morbidity is a novel outcome in emergency care research. It was shown to have a good applicability, as reliability is high, and construct validity can be shown by excellent associations with traditional outcomes, such as hospitalisation, mortality, and—with limitations—intensive care. Further studies are needed for external validation.
Supporting information
S4 Table. Mortality without definition of “acute morbidity” at presentation.
https://doi.org/10.1371/journal.pone.0207906.s004
(PDF)
Acknowledgments
We thank the study personnel, HCRI for quality control, Christian Müller for professional statistical analyses, and Heiner Bucher for helpful discussions.
The authors had full access to all data used for the study. The authors take responsibility for the integrity of the data and the correctness of the data analysis.
References
- 1. Graham CA. The changing face of emergency medicine. LWW; 2015.
- 2. Bardelli P, Kaplan V. Non-urgent encounters in a Swiss medical emergency unit. Swiss Med Wkly. 2013;143:w13760. pmid:24018701
- 3. Schull MJ, Kiss A, Szalai J-P. The effect of low-complexity patients on emergency department waiting times. Ann Emerg Med. 2007;49(3):257–64. e1. pmid:17049408
- 4. George F, Evridiki K. The effect of emergency department crowding on patient outcomes. Health Science Journal. 2015.
- 5. Anneveld M, Van Der Linden C, Grootendorst D, Galli-Leslie M. Measuring emergency department crowding in an inner city hospital in The Netherlands. Int J Emerg Med. 2013;6(1):1.
- 6. Hung KKC, Lo RSL, Graham CA. Outcomes in emergency care research. Eur J Emerg Med. 2017;24(1):1. pmid:27984368
- 7. Kong G, Yin X, Wang T, Body R, Chen Y-W, Wang J, et al. Current state of trauma care in China, tools to predict death and ICU admission after arrival to hospital. Injury. 2015;46(9):1784–9. pmid:26093963
- 8. Becker JB, Lopes MCBT, Pinto MF, Campanharo CRV, Barbosa DA, Batista REA. Triage at the Emergency Department: association between triage levels and patient outcome. Rev Esc Enferm USP. 2015;49(5):783–9. pmid:26516748
- 9. DeWalt DA, Berkman ND, Sheridan S, Lohr KN, Pignone MP. Literacy and health outcomes. J Gen Intern Med. 2004;19(12):1228–39. pmid:15610334
- 10. Ibrahim I, Jacobs IG. Can the characteristics of emergency department attendances predict poor hospital outcomes in patients with sepsis? Singapore Med J. 2013;54(11):634–8. pmid:24276100
- 11. LaMantia MA, Platts-Mills TF, Biese K, Khandelwal C, Forbach C, Cairns CB, et al. Predicting hospital admission and returns to the emergency department for elderly patients. Acad Emerg Med. 2010;17(3):252–9. pmid:20370757
- 12. Lewis Hunter AE, Spatz ES, Bernstein SL, Rosenthal MS. Factors Influencing Hospital Admission of Non-critically Ill Patients Presenting to the Emergency Department: a Cross-sectional Study. J Gen Intern Med. 2016;31(1):37–44. pmid:26084975
- 13. Mushlin AI, Appel FA. Extramedical factors in the decision to hospitalize medical patients. Am J Public Health. 1976;66(2):170–2. pmid:1251951
- 14. Kellett J. Death is not the only healthcare outcome important to patients. Eur J Intern Med. 2016;32:e11–e2. pmid:27062683
- 15. O’Sullivan E, Callely E, O’riordan D, Bennett K, Silke B. Predicting outcomes in emergency medical admissions—role of laboratory data and co-morbidity. Acute Med. 2012;2:59–65.
- 16. Misch F, Messmer AS, Nickel CH, Gujan M, Graber A, Blume K, et al. Impact of observation on disposition of elderly patients presenting to emergency departments with non-specific complaints. PloS One. 2014;9(5):e98097. pmid:24871340
- 17. Wiswell J, Tsao K, Bellolio MF, Hess EP, Cabrera D. "Sick" or "not-sick": accuracy of System 1 diagnostic reasoning for the prediction of disposition and acuity in patients presenting to an academic ED. Am J Emerg Med. 2013;31(10):1448–52. pmid:23972480
- 18. Rohacek M, Nickel CH, Dietrich M, Bingisser R. Clinical intuition ratings are associated with morbidity and hospitalisation. Int J Clin Pract. 2015;69(6):710–7. pmid:25689155
- 19. Nemec M, Koller MT, Nickel CH, Maile S, Winterhalder C, Karrer C, et al. Patients presenting to the emergency department with non-specific complaints: the Basel Non-specific Complaints (BANC) study. Acad Emerg Med. 2010;17(3):284–92. pmid:20370761
- 20.
Gilboy N, Tanabe P, Travers D, Rosenau AM. Emergency Severity Index (ESI): a triage tool for emergency department care, version 4. Implementation handbook. 2012:12–0014.
- 21. Worster A, Bledsoe RD, Cleve P, Fernandes CM, Upadhye S, Eva K. Reassessing the methods of medical record review studies in emergency medicine research. Ann Emerg Med. 2005;45(4):448–51. pmid:15795729
- 22. Gilbert EH, Lowenstein SR, Koziol-McLain J, Barta DC, Steiner J. Chart reviews in emergency medicine research: where are the methods? Ann Emerg Med. 1996;27(3):305–8. pmid:8599488
- 23. Grossmann FF, Nickel CH, Christ M, Schneider K, Spirig R, Bingisser R. Transporting clinical tools to new settings: cultural adaptation and validation of the Emergency Severity Index in German. Ann Emerg Med. 2011;57(3):257–64. pmid:20952097
- 24. Mehra T, Müller CTB, Volbracht J, Seifert B, Moos R. Predictors of high profit and high deficit outliers under SwissDRG of a tertiary care center. PloS One. 2015;10(10):e0140874. pmid:26517545
- 25. Charlson M, Szatrowski TP, Peterson J, Gold J. Validation of a combined comorbidity index. J Clin Epidemiol. 1994;47(11):1245–51. pmid:7722560
- 26. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40(5):373–83. pmid:3558716
- 27. Chotirmall S, Picardo S, Lyons J, D’alton M, O’riordan D, Silke B. Disabling disease codes predict worse outcomes for acute medical admissions. Intern Med J. 2014;44(6):546–53. pmid:24690304
- 28. Conway R, Byrne D, O’Riordan D, Silke B. Emergency readmissions are substantially determined by acute illness severity and chronic debilitating illness: a single centre cohort study. Eur J Intern Med. 2015;26(1):12–7. pmid:25582075