A novel early risk assessment tool for detecting clinical outcomes in patients with heat-related illness (J-ERATO score): Development and validation in independent cohorts in Japan

Background We sought to develop a novel risk assessment tool to predict the clinical outcomes after heat-related illness. Methods Prospective, multicenter observational study. Patients who transferred to emergency hospitals in Japan with heat-related illness were registered. The sample was divided into two parts: 60% to construct the score and 40% to validate it. A binary logistic regression model was used to predict hospital admission as a primary outcome. The resulting model was transformed into a scoring system. Results A total of 3,001 eligible patients were analyzed. There was no difference in variables between development and validation cohorts. Based on the result of a logistic regression model in the development phase (n = 1,805), the J-ERATO score was defined as the sum of the six binary components in the prehospital setting (respiratory rate≥22 /min, Glasgow coma scale<15, systolic blood pressure≤100 mmHg, heart rate≥100 bpm, body temperature≥38°C, and age≥65 y), for a total score ranging from 0 to 6. In the validation phase (n = 1,196), the score had excellent discrimination (C-statistic 0.84; 95% CI 0.79–0.89, p<0.0001) and calibration (P>0.2 by Hosmer-Lemeshow test). The observed proportion of hospital admission increased with increasing J-ERATO score (score = 0, 5.0%; score = 1, 15.0%; score = 2, 24.6%; score = 3, 38.6%; score = 4, 68.0%; score = 5, 85.2%; score = 6, 96.4%). Multivariate analyses showed that the J-ERATO score was an independent positive predictor of hospital admission (adjusted OR, 2.43; 95% CI, 2.06–2.87; P<0.001), intensive care unit (ICU) admission (3.73; 2.95–4.72; P<0.001) and in-hospital mortality (1.65; 1.18–2.32; P = 0.004). Conclusions The J-ERATO score is simply assessed and can facilitate the identification of patients with higher risk of heat-related hospitalization. This scoring system is also significantly associated with the higher likelihood of ICU admission and in-hospital mortality after heat-related hospitalization.


Introduction
Exposure to high ambient temperatures can be deadly, making extreme summer heat a serious global public health threat [1,2,3]. The association of extreme summer heat with excess mortality and morbidity has been well documented in the last decades. For example, the unprecedented heat waves resulted in 700 excess death in Chicago in 1995 [4] and 14,800 excess deaths in France in 2003 [5,6]. While evidence suggests the adverse impacts of heat waves on human health in many regions [7], the morbidity and mortality related to heat condition are reduced by appropriate preventions and treatments [8].
Heat-related illnesses occur when high ambient temperatures overcome the body's ability to dissipate heat. Sepsis and heatstroke have similar mechanisms and include the production and release of several pro-inflammatory cytokines in association with the systemic inflammatory response syndrome [9,10,11,12]. With the rapidly aging population in Japan, the heatrelated hospitalization and mortality have emerged as a serious social problem in the recent years [13,14]. Concurrently, an increasing number of heat-related ambulance dispatches is reported every year [15]. Indeed, the growing incidence of heat-related disorders warrants early recognition of clinical risks in patients with heat illness. Thus, developing a useful assessment tool to assess the negative health impacts of heat waves is an unmet medical need. However, clinical studies with a large cohort, which investigated an early warning system to assess the outcomes of heat illness, are extremely limited.
We sought to establish a simple scoring system that can be useful for predicting the clinical outcomes of heat illness in the prehospital setting. The aim of this study to develop the novel prediction system for clinical outcomes after heat illness using the database from a large, multicenter observational registry of patients with heat illness in Japan. The outcomes of interest were hospital admission, ICU admission, and in-hospital mortality. Here, we present a novel scoring system, called the Early Risk Assessment Tool for Detecting Clinical Outcomes in Patients with Heat-related Illness (J-ERATO score). It is assessed using the prehospital six binary components to identify the risk of clinical deterioration in patients with heat-related illness.

Patient selection
All adult patients enrolled in the Heatstroke STUDY 2010 and 2012 were included in this study. Patients who suffered out-of-hospital cardiac arrest were excluded arrest (systolic blood pressure [SBP] = 0 or 10 mmHg, heart rate = 0 bpm, respiratory rate = 0). In addition, patients with missing data on hospital admission were excluded.
Before to any modeling, we randomly selected 60% of the eligible patients for model development. The remaining 40% of patients were reserved as a validation cohort after model development. Thus, 1,805 and 1,196 patients were assigned in the development and validation cohorts, respectively.

Data collection
The patients' demographics, prehospital information collected by the EMS, and in-hospital information were prospectively recorded; these included weather of the day, incident location, preexisting functional dependency, age, sex, prehospital vital signs, past medical history (preexisting hypertension, heart disease, psychological disorder, diabetes mellitus, cerebrovascular disease, dementia, Parkinson's disease, chronic kidney disease), symptoms assessed by the EMS at the scene (if patients had any abdominal conditions, such as abdominal cramps, nausea, vomiting and diarrhea, those patients were defined as having abdominal symptoms. Similarly, if patients had any muscular conditions, such as muscle cramps, spasms and pain, those patients were defined as having muscular symptoms), vital signs on ED arrival, laboratory data on arrival (white blood cell count, hematocrit, platelet count, blood urea nitrogen [

Definition of outcomes
The primary outcome was hospital admission. Secondary outcomes were ICU admission and in-hospital mortality. Outcomes were assessed by the patient's attending physicians.

Sample size calculation
As we used a logistic regression model to construct a predictive score, the sample size had to be based on the events-per-variable ratio. This ratio had to be greater than 10. We had 710 and 459 events (patients with hospital admission) in the development and validation cohort, respectively. Therefore, we could construct a predictive model with 71 and 45 explanatory variables in the development and validation cohort, respectively [16].

Score development
In the development phase, the scoring system was created. Considering the abundant evidences that the pathophysiology of heat illness has many similarities with the sepsis syndrome [9,10,11,12], we referred to the quick sequential organ failure assessment (qSOFA) score which is a method to estimate the risk of inpatient mortality in patients with suspected infection outside the ICU [17,18] and in the ED [19,20]. However, previous researches reported the poor accuracy of qSOFA for prehospital identification of severe sepsis [21] and some components of this score do not consider specific features of heat illness. Patients were divided into two categories with the following variables: age (defined as !65 years based on the previous evidence [22,23,24]), body temperature and prehospital HR (based on the median values [38˚C and 100 bpm, respectively]), and three components of qSOFA (prehospital SBP, RR and mental status) [19]. Multivariate logistic regression model was fit using the primary endpoint as a dependent variable. A set of potential confounders included the binary categories of elderly age (< 65 or ! 65 y), prehospital body temperature (< 38 or ! 38˚C), prehospital HR (< 100 or ! 100 bpm), prehospital RR (< 22 or ! 22 /min) [19], prehospital SBP (> 100 or 100 mmHg) [19], and prehospital mental status (GCS = 15 or <15) [19], year of incidence, weather of the day, incident location, preexisting functional dependency, sex, past medical histories, and whether or not patients had abdominal or muscular symptoms at prehospital setting. Thereafter, based on the results, biological plausibility [21,25], and the previous knowledge [22,23,24], the J-ERATO score was defined. Favoring simplicity over accuracy, a point score of 1 was assigned to each variable in the final model, irrespective of the regression coefficients [18].

Score validation
In the validation phase, logistic regression was used to analyze the prediction model. Discrimination and calibration were assessed by the C-statics and Hosmer-Lemeshow tests to indicate risk score performance. On the other hand, no calculations were made of the classical indicators of a diagnostic test, such as sensitivity, specificity, predictive values and likelihood ratios, as the test constructed does not indicate a single value (positive or negative) but rather a probability of hospital admission associated with each score. Accordingly, differences were studied between the expected (given by the predictive model) and the observed events to determine whether the reality corresponded to what was indicated by the model. A similar methodology has been used in other studies [26].

Statistical analysis
Categorical variables are presented as the number (frequency), while continuous variables are presented as the median (interquartile range) because the duration of all continuous variables in our data showed non-normal distribution. The differences between the groups were tested with Kruskal-Wallis test. The chi-square or Fischer's exact test was used to compare binary variables. To improve the quality of analyses, a multiple imputation was performed to replace each non-outcome missing value with a set of substituted plausible values by creating five filling-in copies to reduce bias caused by incomplete data [27,28]. Multivariate logistic regression models were constructed in each imputed copy, and the results of the five imputed copies were combined into one model, from which the statistical inference was taken [28,29].
All P values are two-tailed, and a P < 0.05 was considered significant. The outcome odds ratio was reported along with the P-values and 95% CIs. Statistical analyses were performed using IBM SPSS Statistics for Windows, Version 23.0 (IBM Corp., Armonk, NY).

General characteristics
In total, 3,910 patients who were transferred to the 172 hospitals were consecutively enrolled in the Heatstroke STUDY 2010 (between June 1, 2010, and August 31, 2010) and 2012 (between July 1, 2012, and September 30, 2012). After the exclusion of patients based on the predetermined criteria, 3,001 eligible patients were included. Of those, 1,805 and 1,196 patients were assigned in the development and validation cohorts, respectively (Fig 1). Early risk assessment tool for heat illness Table 1 shows the information for each cohort. There were 710 (39.3%) and 459 (38.4%) cases of hospital admission in the development and validation cohorts, respectively. There was no significant difference in variables between each group.

Development of the J-ERATO score
A multivariate analysis showed that an increased likelihood of hospital admission was associated with prehospital GCS, prehospital body temperature, prehospital SBP, prehospital heart rate, age, male sex, and psychological disorder as past medical history (Fig 2). Based on the results and clinical plausibility, the J-ERATO score was defined as the total of the six binary components in the prehospital setting, including respiratory rate ! 22/min, altered mentation (Glasgow Coma Scale < 15), SBP 100 mmHg, prehospital HR ! 100 bpm, body temperature ! 38˚C, and elderly age (! 65 years), for a total score ranging from 0 to 6.

Validation of the J-ERATO score
Patients' clinical findings and laboratory data at ED arrival according to J-ERATO score in the validation cohort were shown in Table 2. The J-ERATO score had excellent discrimination (C-statistic 0.84; 95% CI 0.79-0.89, p<0.0001). The Hosmer-Lemeshow goodness-of-fit statistics were P>0.20 in all imputed copies of validation cohort, indicating good calibration. Calibration plot comparing the distribution of patients in relation to observed and estimated probability using logistic regression is presented in Fig 3. In addition, proportions for hospital admission in the development and validation cohorts are presented in Fig 4. The observed proportion of hospital admission increased with increasing J-ERATO score (score = 0, 5.0%; score = 1, 15.0%; score = 2, 24.6%; score = 3, 38.6%; score = 4, 68.0%; score = 5, 85.2%; score = 6, 96.4%).

Discussion
Although heat-related hospitalization and mortality became a serious social problem worldwide, studies for the risk assessment tools predicting the occurrence of these problems after heat illness have been limited. The current study demonstrated that the novel J-ERATO score, which included only information regarding prehospital vital signs and age of patient, was independently associated with the increased likelihood of hospital admission from ED among adult patients with heat-related illness. The associations of higher J-ERATO scores with increased ICU admission rate and mortality were also observed. Because most of heat-illness patients access acute emergency care through 1-1-9 EMS in Japan, optimizing field triage has been a crucial aspect of concentrating high-need patients in the hospitals most capable of caring for them. Thus, J-ERATO score can be helpful for EMS personnel to triage patients triage and choose an appropriate hospital.
Although several studies have identified the risk factors for mortality in patients with heatstroke admitted at the ICU [6,30,31], the studies focusing on early screening tools predicting the clinical outcomes in patients with heat-related illness in the prehospital setting do not exist. The J-ERATO score was developed based on the results of multivariate logistic model in the developmental cohort. The score included the six binary components such as heart rate, Early risk assessment tool for heat illness body temperature and age in addition to the three components of qSOFA score in the prehospital setting. Previous evidence showed that older adults are more prone to heat stress due to an existing chronic medical condition that changes their normal body responses to heat and takes prescription medicines that affect the body's ability to control its temperature or sweat [21,22,23,24,25]. A study in the USA reported that extreme heat wave was associated with a 3% increase in all-cause hospitalization in older people [22]. The qSOFA score was recently derived by a panel of experts as a scoring system for patients with suspected sepsis [17]. The utility of the qSOFA to the estimate risk of inpatient mortality was validated in patients with suspected infection outside the ICU [18] and in adult ED patients likely to be admitted both with and without suspected infection [20], whereas qSOFA had poor sensitivity for prehospital identification of severe sepsis and septic shock [32]. In addition, since our data showed that the higher J-ERATO score was significantly associated with the poorer vital sings at ED arrival and worsened laboratory data ( Table 2), the score in the prehospital field can predict the physical condition. Taken together, the components included in the J-ERATO score can be considered to be well reflected the physiological aspects of patients with acute illness. Using the prehospital binary parameters, patients with hospital admission were well differentiated from those with ED discharge to home, patients with ICU admission from those admitted to general ward, and non-survivors from survivors after heat illness. Our findings assist the ED in the risk stratification of patients with heat illness. Moreover, the advantage of prognostication using J-ERATO is that it can be readily performed at scene by EMS personnel, which has a significant clinical benefit. Thus, it is conceivable that clinical usage of the J-ERATO score by EMS in the field can improve the outcomes among those patients.
Evidence has shown that having a pre-existing psychiatric illness can more than triple the risk of death during a heat wave [33]. Previous studies have demonstrated that several mental illness and/or behavioral disorder hospitalizations are associated with concurrent diagnoses of hospitalizations for a heat-related illness [34,35]. Recent study also suggested an increased risk of hospitalization among diagnoses of psychoses with concurrent heat-related illness, including dementia and schizophrenia, which may have similar symptoms of cognitive impairment as those with psychoactive substance abuse [35]. A contributing factor for hospitalizations after a heat-related illness in psychotic patients may be due to medications that are used for several mental illnesses and other chronic conditions. Many medications used in psychotic patients increase vulnerability to heat-related morbidity by altering the body's ability to thermoregulate. These types of medications have been suggested as one of the causes for heatrelated hospitalizations and mortality [36]. Consistent with these, the logistic regression model  in the development phase revealed that pre-existing psychotic disorder was associated with hospitalization. Considering that it is challenging for EMS personnel to obtain the certain information regarding past medical history from patients with altered mentation, we did not include the past medical history as a component of the predictive score. Of note, the J-ERATO score was an independent positive predictor of the outcomes after heat illness after accounting for the past medical histories including psychotic disorder.
Our study had several limitations. First, since this is an observational study, the positive association between J-ERATO scores and worsened clinical outcomes does not necessarily prove the causality and can be confounded by unmeasured factors. Other variables may have contributed, which were unable to control or that were not collected a priori. Second, a multivariate analysis to assess the association between J-ERATO score and mortality was not conducted because of the small numbers of events. Thus, further investigations to determine the role of J-ERATO on mortality after heat illness are warranted. Third, the treatments patients received during ED visits and after admission were not controlled in this study. Examples of potential confounding variables include intravenous fluid therapy, cooling methods, time interval from ED arrival to ED discharge, and post-admission care such as ventilation, renal replacement therapy, and active cooling device for heat stroke. Finally, this large, multicenter cohort study describes that in Japan only. In fact, Japan has the highest proportion of elderly Early risk assessment tool for heat illness persons who are more prone to heat-related health problems. Thus, similar studies with data from other countries may result in different findings.  Early risk assessment tool for heat illness

Conclusion
The J-ERATO score is easily and quickly calculated before ED arrival. Higher J-ERATO score was associated with higher likelihood of hospital admission, ICU admission, and increased mortality and was independent positive predictor of hospitalization among patients with the heatrelated illness. Further clinical studies are warranted to validate J-ERATO score as a useful tool in the prehospital setting to rapidly assess the potential for adverse outcomes in heat illness. Early recognition of risk likelihood using the J-ERATO score helps in the early allocation of resources, for example, the need for intensive treatments for patients with higher J-ERATO scores.
Supporting information S1