A clinical prediction model to identify children at risk for revisits with serious illness to the emergency department: A prospective multicentre observational study

Background To develop a clinical prediction model to identify children at risk for revisits with serious illness to the emergency department. Methods and findings A secondary analysis of a prospective multicentre observational study in five European EDs (the TRIAGE study), including consecutive children aged <16 years who were discharged following their initial ED visit (‘index’ visit), in 2012–2015. Standardised data on patient characteristics, Manchester Triage System urgency classification, vital signs, clinical interventions and procedures were collected. The outcome measure was serious illness defined as hospital admission or PICU admission or death in ED after an unplanned revisit within 7 days of the index visit. Prediction models were developed using multivariable logistic regression using characteristics of the index visit to predict the likelihood of a revisit with a serious illness. The clinical model included day and time of presentation, season, age, gender, presenting problem, triage urgency, and vital signs. An extended model added laboratory investigations, imaging, and intravenous medications. Cross validation between the five sites was performed, and discrimination and calibration were assessed using random effects models. A digital calculator was constructed for clinical implementation. 7,891 children out of 98,561 children had a revisit to the ED (8.0%), of whom 1,026 children (1.0%) returned to the ED with a serious illness. Rates of revisits with serious illness varied between the hospitals (range 0.7–2.2%). The clinical model had a summary Area under the operating curve (AUC) of 0.70 (95% CI 0.65–0.74) and summary calibration slope of 0.83 (95% CI 0.67–0.99). 4,433 children (5%) had a risk of > = 3%, which was useful for ruling in a revisit with serious illness, with positive likelihood ratio 4.41 (95% CI 3.87–5.01) and specificity 0.96 (95% CI 0.95–0.96). 37,546 (39%) had a risk <0.5%, which was useful for ruling out a revisit with serious illness (negative likelihood ratio 0.30 (95% CI 0.25–0.35), sensitivity 0.88 (95% CI 0.86–0.90)). The extended model had an improved summary AUC of 0.71 (95% CI 0.68–0.75) and summary calibration slope of 0.84 (95% CI 0.71–0.97). As study limitations, variables on ethnicity and social deprivation could not be included, and only return visits to the original hospital and not to those of surrounding hospitals were recorded. Conclusion We developed a prediction model and a digital calculator which can aid physicians identifying those children at highest and lowest risks for developing a serious illness after initial discharge from the ED, allowing for more targeted safety netting advice and follow-up.


Background
To develop a clinical prediction model to identify children at risk for revisits with serious illness to the emergency department.

Methods and findings
A secondary analysis of a prospective multicentre observational study in five European EDs (the TRIAGE study), including consecutive children aged <16 years who were discharged following their initial ED visit ('index' visit), in 2012-2015. Standardised data on patient characteristics, Manchester Triage System urgency classification, vital signs, clinical interventions and procedures were collected. The outcome measure was serious illness defined as hospital admission or PICU admission or death in ED after an unplanned revisit within 7 days of the index visit. Prediction models were developed using multivariable logistic regression using characteristics of the index visit to predict the likelihood of a revisit with a serious illness. The clinical model included day and time of presentation, season, age, gender, presenting problem, triage urgency, and vital signs. An extended model added laboratory investigations, imaging, and intravenous medications. Cross validation between the five sites was performed, and discrimination and calibration were assessed using random effects models. A digital calculator was constructed for clinical implementation. 7,891 children out a1111111111 a1111111111 a1111111111 a1111111111 a1111111111 Introduction A small, but not negligible proportion of children (2.2% to 7%) will return to the emergency department (ED) within the course of their illness after an initial discharge from the ED [1][2][3][4][5][6]. Rates of children with a revisit to the ED increased in recent years, partly contributing to the overcrowding of emergency departments [5]. Until now, most interest has been in understanding revisits which do not require emergency care, and avoiding those revisits that could be managed safely in the community or with appropriate safety netting information. Several factors at the time of an initial ED visit, such as young age, nighttime presentation and high acuity level, have been linked to an increased risk of having a revisit to the ED [3,4,[7][8][9][10]. Yet, less is known about children returning to the ED with a serious illness, seen in 13 to 42% of the children with an ED revisit [3,4,7,9,11].
At present, no prospective study has looked at determinants associated with return visits in children in EDs across Europe, let alone with a focus on return visits with serious illness. Increasing our understanding of determinants associated with return visits to EDs with serious illness will aid the identification of patients that can be discharged safely with a low risk for clinical deterioration, allowing for improved ED discharge flow and alleviating ED crowding. It will also create opportunities for more targeted safety netting and follow up for those patients at a higher risk. Hence, we aimed to develop a clinical prediction model and digital calculator to support clinicians identifying children at high and low risks of a return visit with serious illness.

Methods
We first looked at the number of return visits with a serious illness amongst children who had previously presented to the ED in five hospitals across Europe. We then looked at characteristics of the initial ('index') ED visit and the association with a return visit with a serious illness. Finally, we developed and validated a clinical prediction to estimate the risk of an individual patient to return to the ED within seven days of initial ED discharge, following TRIPOD guidelines for the development and validation of clinical prediction models.

Design, participants and setting
The study was embedded in the TrIAGE (TRiage Improvements Across General Emergency departments) project, aiming to improve the early recognition of seriously ill children at the ED [12,13]. We performed a secondary analysis of data collected as part of a prospective observational multicentre cohort study in five European EDs (Erasmus MC, Rotterdam, the Netherlands (EMC); Maasstad Hospital, Rotterdam, the Netherlands (MHR); St Mary's Hospital, London, United Kingdom (SMH); Hospital Fernando da Fonseca, Lisbon, Portugal (HFF); Medizinische Universitaet Wien, Vienna, Austria (MUW)), with data collected between 2012 and 2015 (Table 1). Data analyses were planned and executed following completion of the original data collection. Consecutive children aged 0 to 16 years who presented to EDs for unscheduled healthcare were included in the study. Universal health care coverage was available across the participating sites, with similar and free of charge access to urgent and emergency care, and primary care. Primary care with either general practitioners and/or primary care paediatricians was available in all the countries in our study. Patients who were redirected towards a primary care provider or urgent care centre after presenting to the ED at the time of their index visit were not eligible for this study, as we did not record complete clinical data for these children. Patients who left against medical advice or who left before being seen were only eligible if sufficient baseline data were available for that visit. Children who were admitted following their index visit were subsequently excluded from analysis, as they have a different risk profile for a revisit with serious illness [5,14]. A waiver for the need of patient informed consent was obtained at all participating hospitals.

Data collection and predictor variables
In all hospitals data on: patient characteristics, triage urgency, presenting problem, vital signs, ED interventions and procedures, treatment, and outcomes were recorded in the electronic health records in a standardised manner as part of routine clinical practice. Predictor variables were chosen based on their known association with return visits, patient urgency and severity scores [15,16]. Data were extracted from the hospital's electronic systems and checked for completeness, validity and outliers. Season, time and day of presentation were derived from the patient's arrival date and time. Seasons were divided in Winter (21/12-20/03), Spring (21/ 03-20/06), Summer (21/06-20/09), Autumn (21/09-20/12). Date and time of presentation were grouped into: Weekday evenings (Monday-Friday; 18.00-23.00 hrs), Weekday nights (Monday-Friday; 23.00-07.00 hrs), Weekend days (Saturday-Sunday; 08.00-18.00 hrs), Weekend evenings (Saturday-Sunday; 18.00-23.00 hrs), Weekend nights (Saturday-Sunday; 23.00-07.00 hrs), and Weekdays (Monday-Friday; 08.00-18.00 hrs). Triage urgency was determined using the third edition of the Manchester Triage System [17]. This is a validated 5-level urgency classification system that allocates an urgency category using specific discriminators for 52 flowcharts detailing varying presenting problems [18]. These flowcharts were re-classified into 11 main groups of presenting symptoms (S1 Table). Low urgency levels standard (level 4) and non-urgent (level 5), as well as emergent (level 1) and very urgent (level 2) were combined in the multivariable regression analyses due to low number of cases in the highest and lowest category. Vital signs were measured at the time of triage by the triage nurse, following advanced paediatric life support standards [19]. Heart rate (beats per minute) and respiratory rate (breaths per minute) were categorised using the age-specific Advanced Paediatric Life Support threshold values. Oxygen saturations were dichotomised at <94%; body temperature at > = 38.0 degrees Celsius. For level of consciousness, Verbal, Pain, and Unconsciousness on the Alert-Verbal-Pain-Unresponsive (AVPU) scale were combined into 'reduced level of consciousness'. Interventions and procedures in the ED included: any laboratory tests performed (ie. blood, urine, cerebral spinal fluid, other), any imaging done (ie. Xray, ultrasound, CT, MRI), and any intravenous medications and/or fluids administered. Immediate lifesaving interventions included airway and breathing support, haemodynamic support, emergency procedures, and emergency medications (S1 Table). Blood pressure, capillary refill and lifesaving interventions were not used for prediction modelling owing to low number of values measured in children across all institutions (blood pressure available for 8%; capillary refill for 47%); and/or <10 cases with abnormal values for children with serious illness at return visit ( Table 2). In three hospitals (EMC, SMH, MUW), co-morbidity was coded into complex and non-complex by trained medical students blinded for the outcome of this study according to the definitions by Simon et al. [20].

Outcome measures
Serious illness was defined as the need for hospital admission or paediatric intensive care unit (PICU) admission or death in ED after an unplanned return visit within 7 days of the index visit. Characteristics of children admitted to PICU at the time of their revisit were described in detail separately (S2 Table).

Statistical analysis
Firstly, we performed uni-and multi-covariate logistic regression to look at the association between the characteristics of the index visit and a return visit with a serious illness, reporting crude unadjusted and adjusted odds ratios (OR) with 95% confidence intervals. For the three hospitals (Erasmus cohort, London cohort, Vienna cohort) with available data on co-morbidity we performed an additional analysis to determine the role of co-morbidity on return visits with serious illness in these institutions. Our cohort provided sufficient power for both the derivation and cross-validation studies, estimating the required sample using the recommended approach by Riley et al. (S1 Appendix) [21][22][23].
Secondly, we developed two clinical prediction models using logistic regression. A first model included clinical characteristics of the index visit (ie: 'clinical model'), and then the use of laboratory investigations, intravenous medications or fluids, and imaging were added to the model (ie.: 'extended model') as a second step. We included the five hospitals as a categorical variable to account for case mix differences between the cohorts. All variables were preselected, and variables were included independent from their statistical contribution to the models based on previous literature. No variables were excluded from the final prediction models based on statistical contribution or when adding the variables of the extended model [15]. The overall performance of the extended model was compared with the clinical model using the likelihood ratio test. Missing data for vital signs ( Table 2) were imputed using a multiple imputation model including a hospital variable, age, available vital signs, triage urgency, presenting problem, and discharge destination. This resulted in twenty-five iterative databases, including data from all five cohorts, on which we performed statistical analysis after which results were pooled [24]. Imputation was performed using the multivariate imputation by chained equations. available data, n (%). � definition of life saving interventions in S1 Table; airway and breathing support, n = 7; emergency procedures and haemodynamic support, n = 16; emergency medications, n = 23; electrical therapy, n = 0. + : crude unadjusted ORs based on imputed datasets; Blood pressure not shown because of high % missing values; Capillary refill not shown because of % missing values and <10 cases of serious illness on revisit; immediate lifesaving interventions not included as n<10 for serious illness on revisit.
CI confidence interval; ED: emergency department; MTS: Manchester Triage System; IQR: interquartile range; iv: intravenous. https://doi.org/10.1371/journal.pone.0254366.t002 We initially derived the prediction model in four of the five cohorts, and then cross-validated the model in the fifth cohort that was left out. We repeated this "leave-one-out"approach as a method for cross-validating the prediction model four times with each of the five cohorts serving as an independent validation cohort once [25]. Each time, we adjusted the intercept by estimating the linear predictors for the validation cohort without using the hospital variable and then recalculating an intercept, thus acknowledging differences between the hospitals. External validity of the model was determined by estimating the discrimination by means of the Area Under the operating characteristic Curve (AUC), and model calibration by means of calibration plots, the calibration coefficient, or calibration slope, and change in intercept of the model. The AUC is used to estimate the ability to discriminate between patients with and without a return visit with serious illness, with values ranging between 0.5 (indiscriminate test) and 1 (optimal test). Calibration refers to the level of agreement between predicted risks and observed outcome; the ideal calibration coefficient, or slope, being one, indicating perfect agreement between observed and predicted risks. The calibration intercept reflects the extent that predictions are systematically too low or too high ('calibration-in-the-large'), e.g. due to differences in the underlying incidence between cohorts, with an optimal value of zero [26]. The five estimates of AUCs and calibration coefficients of each of the five validation cohorts were then pooled by means of random effects models to provide summary estimates. Finally, we developed a clinical prediction model combining all available data from the five cohorts and with a refitted intercept to allow for hospital differences. An apparent optimism corrected AUC was calculated by means of bootstrapping (500 iterations). We performed bias corrected calibration with 200 bootstrapping iterations to demonstrate goodness of fit for the final overall prediction models. Variance inflation factors were calculated to assess collinearity between predictor variables. Diagnostic performance measures, including sensitivity, specificity and likelihood ratios (LRs) were calculated for different risk thresholds of both the clinical and the extended model. We used R statistical software version 4.0.0 for all our analyses, including use of the rms, metaphor, MICE, pmsampsize, pROC and epiR packages.

Rates of revisits
109,482 children with an ED visit were included, of whom 98,561 children (90%) were discharged from ED (Fig 1). A total of 1,026 children (1.0%) returned to the ED with serious illness out of a total of 7,891 children who represented to the ED. Rates of revisits with serious illness varied between the hospitals (range 0.7-2.2%); unadjusted odds ratios for variables of interest and their risk of a return visit with serious illness are shown in Table 2. Twenty-three children (0.02%) were admitted to PICU and one child died at revisit (characteristics of PICU patients presented in S2 Table).

Characteristics of return visits with serious illness
Several clinical characteristics of the index visit were associated with a revisit with serious illness. These included young age, female gender, time of presentation during the weekend daytime and weekday night-time, and higher triage urgency (Table 3). Presenting problems such as shortness of breath, gastro-intestinal problems, neurological problems and soft tissue infections were also linked to an increased risk of a return visit with serious illness, as were any laboratory tests performed, any imaging done, and any intravenous medications and/or fluids administered. Presentations for trauma and wounds made a revisit with serious illness less likely.
Comorbidity was significantly associated with a return visit with serious illness in the three hospitals for which we had these data, both for children with non-complex (aOR 2.36, 95% CI 1.82-3.05)) and children with complex co-morbidities (aOR 3.39, 95% CI 2.67-4.29) (S3 Table). Table 4 presents the overall discrimination and calibration of the "leave-one-out"cross-validation of the clinical and the extended model. Intercept coefficients ranged from -0.91 to 0.

Derivation of the final clinical and extended prediction models
Next, we combined all cohorts and derived a clinical model predicting the risk of a revisit with serious illness based on clinical characteristics (Table 3; model coefficients are presented in S4  Table). 4,433 children (5%) had a risk of > = 3%, which was useful for ruling in a revisit with serious illness, with positive likelihood ratio 4.41 (95% CI 3.87-5.011) and specificity 0.96 ). An extended model had similar overall diagnostic performance for the low risk threshold of 0.5% and high-risk threshold of 4% (Table 5). The extended model improved risk classification for a modest proportion of children (Fig 2).  Table).

Principal findings
Young age under two years, high urgency, and type of presenting problem were highly associated with a return visit to the emergency department with a serious illness. Rates of revisits with serious illness varied between the five hospitals, ranging from 0.7 to 2.2%, and were in line with the lower end of previously reported rates [3,4,7,9,11]. A clinical prediction model, including the variables season, time and day of presentation, age, gender, presenting problem, triage urgency, and vital signs, was able to select a high number of children with a low risk of return visit with serious illness, as well as identify those with the highest risks. Initial derivation and cross-validation of the model showed the robustness of the model in all settings. We developed a digital calculator to enable clinical implementation. For illustration, in our case example shown in Fig 3, a 3 year boy with shortness of breath had a 3.1% predicted risk of having a return visit with a serious illness, which was between the >70 th -≦80 th centiles (S5 Table). The predicted probabilities for a revisit with serious illness remained relatively low, even for children with an increased risk, mostly as a result of the overall low prevalence (1.0%). Performing laboratory tests and imaging, as well as administration of intravenous medications were all associated with an increased risk of a revisit with serious illness and improved overall risk prediction, even as the diagnostic performance for low and high-risk threshold were similar for the clinical and extended model.

Comparison to previous literature
Reasons for children to represent with a serious illness after initial ED visit are diverse. One might be concerned about medical errors in the decision making process during the first visit [27,28]. However, in one study, return visits resulting in hospital admission were rarely the result of poor quality of ED care at first visit [29]. Another study also suggested that only a minority of return visits with serious illness were the result of a failure to make an appropriate diagnosis, a failure to instigate appropriate treatment, or a failure to investigate fully, and that the large majority of these children with a revisit resembled disease progression [30]. For example, in meningococcal disease is it well known that children might present with non-specific signs and symptoms at first, with the severity of the meningococcal septicaemia only becoming manifest at a later stage of the disease [31]. There might also be parental factors involved, such as a refusal to be admitted in first instance, non-adherence to prescribed  therapy at home, or a failure to attend a follow up appointment [30]. This all contributes to the importance of empowering parents to provide informed monitoring at home, and providing them with clear instructions on when to return to the ED [32]. More advanced modelling techniques have been used to provide clinicians with clinical tools to predict return visits to the ED [1,33,34]. However, to our knowledge, none focused on the risk of return visits with serious illness, and none presented a tool ready for clinical implementation. From the outset, we felt it was more important to understand the group of children at risk for clinical deterioration, rather than preventing avoidable revisits, even though both have significant implications for appropriate use of scarce resources. One of the difficulties was capturing the vast spectrum of problems clinicians are confronted with in the ED in a single tool [16]. For certain conditions, such as children at risk of dehydration or children at risk for severe infection, specific clinical features were identified predictive of clinical deterioration [35,36]. Although those predictors might be important, they only apply to a select proportion of the children visiting the ED, and this will need to be positioned against the convenience of a 'one-fits-all' tool. Our model assigned higher risks of serious illness at revisit to children presenting with specific problems, such as shortness of breath, gastro-intestinal problems, and neurological problems.
Tachycardia has been shown to be associated with an increased risk of a revisit, as well as with revisits with serious illness, yet we did not find evidence of this [11,37]. We did not include heart rate at the time of discharge; however, Daymont et al. looked at this and only found limited additional value for different ways of including heart rate as a predictor of return visits with serious illness [37]. The role of other vital signs as predictors of serious illness needs further exploring.
Some variables, such as social economic status, ethnicity, language fluency, and health literacy were previously shown to be associated with revisits as well as revisits with serious illness, yet were not included in the model [2-4, 7, 10]. Studies had inconsistent and sometimes contradicting risk profiles for these variables [36]. Crowding of EDs has previously been associated with both an increased as well as a reduced risk [2][3][4]. Moreover, Akenroye et al. found that ED crowding was an increased risk for revisits, but not for a revisit with admission to hospital  [3], highlighting that risk factors for return visits differ from risk factors for revisits with serious illness. Notably, ethnicity and variables derived from the use of postal codes, such as deprivation scores, cannot routinely be used in most European countries without explicit informed consent and hence were not included in our models.

Implications for clinical practice and future research
The limited variability in calibration slopes testifies that this model can be used in a diverse range of clinical settings [25]. The differing underlying incidence of serious revisits, as also reflected in the intercept values of the cross-validation studies ('calibration-in-the-large', Table 4), indicate that the local incidence should be taken into account when interpreting the risk predictions. If the local incidence is known, one could adjust our model for local use by adjusting the intercept of the model. Knowing which children are at increased risk for clinical deterioration following discharge from ED can contribute to more tailored, patient-specific follow-up [15]. How this follow-up should look will be highly setting dependent, with some institutions having the capacity to organise planned face to face review appointments in ED or by an ambulatory care team, and in other settings this might be done via telephonic consultation or with the involvement of primary care. Many factors contribute to the development of serious illness, including natural disease progression and treatment compliance. By implementing tailored safety netting and follow-up, this will help early detection of children with progressing disease, preventing delayed presentations after an initial health care visit, as well as ensuring treatments are continued at home, and are given in a timely and appropriate manner. Examples of the latter could include antibiotic tolerance in young children, appropriate inhaler technique of bronchodilators, and reinforcement of hydration strategies. More senior clinicians will likely have an experienced feel for those patients at highest risk for deterioration. Our models enable this for more junior physicians, who see the large majority of unwell children in acute care facilities, and for settings with no dedicated paediatric emergency care team. Also, our prediction model can complement decision making in acute paediatric conditions where a watchful waiting approach is justified and a return visit can be interpreted as a marker of good clinical care, such as in children with non-specific abdominal pain deemed to be at low risk for appendicitis at first assessment. Importantly, any intervention with individualised follow-up or targeted safety netting will need to be studied carefully, in combination with costeffectiveness analyses, as it might inadvertently have unwanted effects. One study showed an increase in the rate of return visits following the introduction of telephone calls after ED discharge [38]. In future, studies should also look into the reasons for revisiting the ED, such as parental factors, potential of medical errors, and natural disease course, as they will be useful in fine-tuning the specific interventions for safety netting and follow-up. In addition, the potential for machine learning techniques to improve our prediction model should be explored.

Strengths
This is the first study including a large cohort of children using a prospective multicentre design with data from five different European EDs. The design allowed for standardised, near complete data on predictors known to be related with urgency systems and severity scores. We opted for including clinical variables with minimal interuser variability that were all readily available in routine clinical practice. For example, the reliability and validity of the MTS triage urgency classification was shown previously [39]. We also converted our understanding of determinants of revisits with serious illness into a clinical tool that can be used in clinical practice. We included the clinical variables independent of their statistical contribution to the model to ensure validity of the model and enhance clinical uptake. Although one might argue that the discriminative ability of the model is only moderate, the tool is able to identify a large number of children who can be discharged safely, and also identifies a select and well-defined group of patients that can be targeted for more tailored safety netting advice and follow-up strategies.

Limitations
One of the limitations was the over-representation of patients from the larger of the five hospitals, and we adjusted for this by including the hospitals as a categorical variable whilst developing and validating the prediction models. Generally, findings across the sites were similar (data not presented), as was also confirmed by the cross-validation analyses. Furthermore, one can argue the accuracy of the outcome definition for serious illness; however, defining serious illness by the need for hospitalisation at revisit was least subjective to a doctor's impression, and with limited variability between hospitals [40]. We performed a secondary analysis with a broader outcome definition of serious illness, including those who underwent lifesaving interventions or had triage urgency 'emergent' at revisit. This yielded only 16 additional cases, which did not affect our overall findings, supporting the validity of our outcome definition. Despite our large sample size, it did not allow for a separate model for children admitted to PICU at revisit (S2 Table). In addition, we lack data to show the impact of revisits to other EDs than the one originally visited, and future studies will need to provide detailed insight into where children with a serious illness re-present during an episode of acute illness. As our study hospitals have an important regional referral function with limited admission capacity, hospital admission was warranted only to provide therapeutic of supportive treatment that would not have been able in the community or by other health care providers in the large majority of cases. The role of other types of admission, such as for social reasons, will need to be studied in future. Also, only a relatively small number of variables were included resulting in a low explained variance of our models, meaning future studies should explore the incremental predictive value of additional variables, such as social economic status and language. We did not consider language fluency, as reliably determining parental or carer's language proficiency was not feasible. Another important candidate predictor will be co-morbidity, which was an important predictor of return visits with serious illness in three out of five cohorts with data on co-morbidity available. Furthermore, the validity and applicability of our prediction models still needs to be proven outside Europe and in different health systems., specifically in those without primary care or universal health care coverage. Lastly, the data used for this study are from the period prior to the COVID-19 pandemic. The COVID-19 pandemic has impacted local healthcare delivery systems globally, and future validation studies will need to study long term impact of the current COVID-19 pandemic.

Conclusion
We developed a prediction model and a digital clinical calculator that can aid physicians identifying those children at highest and lowest risks for developing a serious illness after initial discharge from the ED, allowing for more targeted safety netting advice and follow-up.