Predicting Mortality in Low-Income Country ICUs: The Rwanda Mortality Probability Model (R-MPM)

Introduction Intensive Care Unit (ICU) risk prediction models are used to compare outcomes for quality improvement initiatives, benchmarking, and research. While such models provide robust tools in high-income countries, an ICU risk prediction model has not been validated in a low-income country where ICU population characteristics are different from those in high-income countries, and where laboratory-based patient data are often unavailable. We sought to validate the Mortality Probability Admission Model, version III (MPM0-III) in two public ICUs in Rwanda and to develop a new Rwanda Mortality Probability Model (R-MPM) for use in low-income countries. Methods We prospectively collected data on all adult patients admitted to Rwanda’s two public ICUs between August 19, 2013 and October 6, 2014. We described demographic and presenting characteristics and outcomes. We assessed the discrimination and calibration of the MPM0-III model. Using stepwise selection, we developed a new logistic model for risk prediction, the R-MPM, and used bootstrapping techniques to test for optimism in the model. Results Among 427 consecutive adults, the median age was 34 (IQR 25–47) years and mortality was 48.7%. Mechanical ventilation was initiated for 85.3%, and 41.9% received vasopressors. The MPM0-III predicted mortality with area under the receiver operating characteristic curve of 0.72 and Hosmer-Lemeshow chi-square statistic p = 0.024. We developed a new model using five variables: age, suspected or confirmed infection within 24 hours of ICU admission, hypotension or shock as a reason for ICU admission, Glasgow Coma Scale score at ICU admission, and heart rate at ICU admission. Using these five variables, the R-MPM predicted outcomes with area under the ROC curve of 0.81 with 95% confidence interval of (0.77, 0.86), and Hosmer-Lemeshow chi-square statistic p = 0.154. Conclusions The MPM0-III has modest ability to predict mortality in a population of Rwandan ICU patients. The R-MPM is an alternative risk prediction model with fewer variables and better predictive power. If validated in other critically ill patients in a broad range of settings, the model has the potential to improve the reliability of comparisons used for critical care research and quality improvement initiatives in low-income countries.


Introduction
Intensive Care Unit (ICU) risk prediction models estimate expected hospital mortality based on patient characteristics. The models facilitate case-mix adjustment in research, across-institution benchmarking, and individual ICU quality improvement evaluations. [1] The first adult ICU risk prediction model to gain broad use, the Acute Physiology and Chronic Health Evaluation (APACHE) model, was developed in 1981 using a sample of 805 patients from two hospitals. [2] Since then, numerous other ICU models have been developed, validated, and modified to improve goodness-of-fit. [1,[3][4][5][6][7][8][9][10][11][12][13][14] While the large majority of resource-intensive critical care occurs in middle-and highincome countries, critical care does occur in low-income countries, and critical illness disproportionately affects people in low-income countries. [15][16][17] Attempts to improve the research methods and quality of critical care in low-income countries are hindered by the dearth of context calibrated risk prediction models with feasible data collection requirements. Data collection burden is an important barrier to the use of current risk prediction models in low-income countries. Even in the United States, only 10-15% of ICUs regularly use predictive models for quality improvement, and the burden of data collection is an oft-cited reason. [18] In addition, the characteristics of ICU patients in low-income countries may be quite different from those in high-income countries, [19] thus raising the question of whether models developed from populations in high-income countries will be accurate for low-income countries. [7,20] To our knowledge, the only ICU model developed in a low-income country is the "Clinical Sickness Score" (CSS), created by Watters et al in 1989 based on 624 ICU admissions to a university teaching hospital in Zambia. [21] This model did not use current statistical methods for assessing discrimination and calibration, and it was never validated in another population.
In order to facilitate ICU research and quality improvement efforts in low-income countries, a risk prediction model that is calibrated for the population and relies on a parsimonious, easily collected set of variables is needed. In our study, we first assess the performance of the Mortality Probability Admission Model, version III (MPM 0 -III) in patients admitted to two ICUs in Rwanda; this model was chosen because it is the only ICU risk prediction that has been validated in a large cohort and is not dependent on laboratory values. We then sought to develop a new model with better predictive ability in this population and with less data collection burden.

Study oversight
The study was conducted at the University Teaching Hospital of Kigali and the University Teaching Hospital of Butare, both affiliated with the University of Rwanda. Ethical and scientific committees at the University of Rwanda approved the study, as did the Committee on Clinical Investigations at Beth Israel Deaconess Medical Center in Boston, USA. Requirement for individual patient-level consent was waived given the determination of minimal risk to patients.

Study population and setting
Consecutive patients admitted to the ICU at the University Teaching Hospital of Kigali (6 ICU beds) and the ICU at the University Teaching Hospital of Butare (5 ICU beds) were enrolled from August 19, 2013 to October 6, 2014. These hospitals are both public referral academic teaching hospitals, and contain all public adult ICU beds in Rwanda. For purposes of this analysis, we excluded all patients younger than 15 years old.

Definitions
Sepsis was defined using criteria outlined by the most recent international consensus groups at the time of the study: at least two of four Systemic Inflammatory Response Syndrome (SIRS) criteria and suspected infection. [22][23][24] Severe sepsis was defined as these criteria plus evidence of organ hypoperfusion or dysfunction. Septic shock was defined by the presence of all of these, with mean arterial pressure<60 mmHg or systolic blood pressure <90 mmHg despite adequate fluid resuscitation. The Acute Respiratory Distress Syndrome (ARDS) was defined based on the Berlin definition: bilateral opacities on chest radiograph consistent with edema; lack of evidence of left atrial hypertension; positive end expiratory pressure (PEEP) of at least 5 cm H 2 O, and occurrence within one week of a clinical insult. [25] Since arterial blood gases were not available throughout the duration of the study, we used a hypoxia cutoff of S p O 2 / F i O 2 , 315 (excluding any S p O 2 >97%), based on a study that derived and validated the correlation between P a O 2 /F i O 2 300 and S p O 2 /F i O 2 , 315. [25,26] Operational definitions for the MPM 0 -III variables were based on those from the initial MPM 0 -III development and validation study. [6] MPM 0 -III consists of 16 variables assessed within one hour of ICU admission. As in the initial development and validation of MPM 0 -III, missing values were treated as normal. Discharge and death diagnoses were determined using information in the chart. Diagnoses were classified using the Agency for Healthcare Research and Quality's (AHRQ) Clinical Classifications Software (CCS). This maps 17 broad diagnostic categories and a total of 260 specific diagnoses to ICD-10 codes. [27] Data collection and quality assurance One physician and one nurse at each site prospectively recorded data during each ICU admission. Data were collected on paper forms, and then entered into a web-based Research Electronic Data Capture (REDCap) electronic data capture tool. [28] Data collected included demographic information, insurance status, hospital admission data, ICU admission data, MPM 0 -III variables (acute and chronic organ-specific conditions), laboratory values when available, variables to determine the presence of sepsis, severe sepsis, septic shock within 24 hours of ICU admission, and ARDS at any time during ICU admission, interventions performed during admission, discharge diagnoses, and in-hospital vital outcomes. Training in data collection, including detailed operational definitions for each data element, was provided to the data collectors prior to beginning data collection. Study investigators randomly selected and reviewed 3 charts each week for eight weeks (approximately 40% of all charts in those weeks) to assess for inter-rater agreement in interpretation of data elements. Throughout the study, data validation reports were produced to check data accuracy and to identify any areas of systematic error. Any inter-rater disagreement was corrected for the chart in question, and other prior charts with potential for similar disagreement were reviewed. Biweekly meetings addressed inconsistencies, and study investigators were available every day for data collector questions.

Statistical Analysis
The primary outcome was in-hospital mortality. For descriptive variables, we calculated proportions for categorical variables and medians with interquartile ranges (IQRs) for continuous variables. We performed univariate analyses using Student's t-tests and chi-square tests with significance set at p<0.05.
Multivariate logistic regression models were used for in-hospital mortality prediction. We chose variables from the univariate analyses, based on their predictive power (as determined by a p value< 0.05) as well as their ease of capture based on our experience, the proportion of missing values in our dataset, and their clinical significance. We tested each of the 16 independent variables in the MPM 0 -III model, as well as additional variables: HIV status (positive/negative), HIV treatment status (on/off antiretroviral medications), age, insurance status (national public, private, or none), time prior to receiving care (in days), admission and ICU vital signs (systolic and diastolic blood pressure, pulse, temperature, respiratory rate, oxygen saturation, Glasgow Coma Scale score), reasons for ICU admission, presence or absence of sepsis, severe sepsis, or septic shock, presence or absence of ARDS, and blood laboratory values (sodium, potassium, creatinine, urea, white blood cell count, hemoglobin, platelets, aspartate transaminase, and alanine transaminase).
The final parsimonious model became the Rwanda Mortality Probability Model (R-MPM). We used area under the receiver operating characteristic curve (AUC, or c-statistic) to assess model discrimination (how effectively a model assigns a higher probability of death to a nonsurvivor than a survivor). [3] Generally, an AUC of 0.70-0.80 is acceptable, 0.80-0.90 good, and greater than 0.90 excellent. We calculated a Hosmer-Lemeshow statistic and p-value to assess calibration (how well the model predicts outcomes across the entire spectrum of risk, assessing predicted versus actual outcomes in each decile of risk.) [6] Acceptable calibration is generally defined as a non-significant Hosmer-Lemeshow value (p>0.05). Since we did not have a separate validation cohort, we also performed internal validation with bootstrapping in order to estimate the optimism in our model, expressed as a confidence interval (CI) around the AUC. We compared the model performance to the MPM 0 -III. We completed all analyses using SAS software, version 9.3.

Patient characteristics, interventions, and outcomes
There were 427 patients admitted to the ICUs during the study period; we were unable to locate discharge vital status on two patients after extensive searching, so we had outcomes data on 425 patients. Patient characteristics are presented in Table 1 and S1 Table. About half of patients were male, and median age was 34 (IQR 25-47) years. Patients were insured in 93.3% of cases, with the vast majority insured by the national community-based medical insurance. [29] Admissions largely originated from district hospital transfers [72.3%), with an additional 10.2% coming directly from an accident site. The median time spent sick at home before seeking healthcare was one day, and the median time spent at a district hospital prior to referral was one day.
The most common reason for ICU admission was respiratory failure or need for endotracheal intubation (72.8%) ( Table 2). Cardiopulmonary resuscitation (CPR) was performed within 24 hours before ICU admission for 7.5% (S2 Table). Within 24 hours of ICU admission, 42.2% had a diagnosis of sepsis, 33.0% severe sepsis, and 20.8% septic shock. ARDS criteria were met in 12.9% of all patients at any time during their ICU stay. In-hospital mortality was 48.7% (Table 3).
Surgical interventions were performed for 69.3% of all patients (Table 3). Mechanical ventilation was initiated for 85.3%, for a median of 2 (IQR 1-7) days. Blood products were given to 37.8%, and 41.9% received vasopressors. Renal replacement therapy was given to 7.5%. Median   (Table 4). Based on the odds ratios, the incremental change in mortality expected for a change in each independent variable is as follows: an increase in 10 years in age translates to an increase in the odds of death of 23%; patients with a suspected or confirmed infection within 24 hours of ICU admission have a 214% increase in the odds of death as compared to patients without; patients admitted to the ICU for shock or hypotension have a 155% increase in the odds of death as compared to those without shock or hypotension on admission; each increase in the GCS score by one point decreases the odds of death by 21%; and each 10-point increase in the heart rate translates to an increase in the odds of death of 19%. Fig 2 demonstrates the predicted versus actual mortality rates of the MPM 0 -III and R-MPM models by quartile. While the GCS is documented on most patients in our dataset and in our clinical practice (63.7% have a score, 31.4% were sedated so had no numerical score, and 4.9% had a missing value in our dataset), we recognize that it may not be documented in all settings, and there is frequent uncertainty about scoring of verbal domains for endotracheally intubated and sedated patients. We therefore also examined a model using altered mental status on ICU admission (present versus not present) in place of the GCS score, the simplified R-MPM. This model gave an area under the ROC curve of 0.76 (Hosmer-Lemeshow chi-square statistic of 11.46, p = 0.177) ( Table 4). Other than GCS, we had very few missing values for either the MPMo-III or R-MPM variables; all variables in the models had missing values <8% of the total  Table 4. Simplified R-MPM = the Rwanda Mortality Probability Model as detailed in Table 4 except that the variable altered mental status replaces the variable Glasgow Coma Scale score. The number in parentheses after each model name in the legend is the area under the ROC curve for that model.  (S3 Table). Just as in the original MPMo-III, a missing value was assumed to be normal for a given variable.

Discussion
In this observational study of consecutive critically ill patients admitted to two ICUs in the low-income country of Rwanda we found that a series of five easily obtained bedside clinical variables have reasonable predictive ability, discrimination, and calibration. Our model performance compares favorably to other more complicated and less accessible clinical prediction systems (APACHE IV area under the ROC curve 0.88 and 0.86, Simplified Acute Physiology Score (SAPS3) 0.85 and 0.80, and Mortality Probability Admission Model (MPM 0 -III) 0.82 and 0.72 in a large validation cohort and single-center US study of 2596 patients, respectively. [1,30]) The development of a risk prediction model in a low-income country is important because: 1) predictive models are essential for interpreting research and assessing quality of care for critically ill patients; [1] 2) models need to be calibrated to specific populations and contexts; Expected and actual mortality rates by prediction model quartiles. Each bar represents the actual mortality rate for that quartile, with quartiles determined by the specified risk prediction model. Each diamond represents the expected average mortality per quartile. [7,20,31] 3) models need to have reasonable data collection burden; [18] and 4) a model has not been developed or validated in a low-income country since 1989. [21] While controversy exists about the value of risk prediction models for comparing ICUs to each other (ranking) in high-income countries, there is widespread agreement that the modeling of critical care outcomes is necessary for quality improvement efforts. [32] Even among high-income countries, the recalibration of models is deemed necessary to reflect differences in epidemiology and practices across countries. [7,20,31] It is not surprising that MPM 0 -III does not discriminate or calibrate well in our Rwandan ICU population. Even in gross comparisons, the Rwandan ICU population is markedly younger and more often has surgical diagnoses than the United States cohort on which MPM 0 -III was based. This is a common difference between low-and high-income settings, and is consistent with cohorts from other African ICUs. [5,19] The issue of data collection burden, already prohibitive in many high-income countries [18], is even more salient in low-income countries, where ICUs do not have funding for data collection. The R-MPM model is more feasible for ongoing use due to its small number of variables and its focus on acute physiologic variables that can be easily assessed at the bedside. Finally, the last predictive model to be developed in a low-income country was Watters' "Clinical Sickness Score" (CSS) in 1989 based on 624 ICU admissions to a university teaching hospital in Zambia, which yielded a sensitivity of only 36.4% and specificity of 93% in predicting survival. Contemporary statistical methods of developing and testing a model were not employed, and the model has not been updated to reflect changes in epidemiology or practice. [21] Our model is simple to use and performs well in our development population with over one year of data collection to minimize seasonal bias. The model has several limitations. First, its relative simplicity will presumably translate to lower discrimination and calibration statistics when it is validated in a separate population. The tension between data collection burden and performance is not specific to low-income countries settings. One study applied the most common risk prediction models to 11,300 patients in 35 Californian ICUs and found excellent discrimination for all models (0.892, 0.873, 0.809 for APACHE IV, SAPS II, and MPM 0 -III respectively), but with time required to abstract the data in inverse relationship to model accuracy (37.3 minutes, 19.6 minutes, and 11.1 minutes, respectively). [33] Given the acceptable performance in all three models, using a model with lesser performance but lower data collection burden is thought to be a reasonable choice. [33] Second, our model is based on a small sample size in two ICUs in a single country, and we were only able to perform internal validation using our development sample. The current validated models used cohorts ranging from 16,784 admissions for SAPS 3, [13] to 110,558 for APACHE IV, [4] to 124,855 for MPM 0 -III. [6] However, the first APACHE model was developed with only 805 admissions, [2] and the first MPM with 755. [34] Our model represents a foundation for ongoing work, which must include validation in future patients in the same ICUs as well as validation in other low-income-country ICUs. Third, our model contains one variable, "suspected or confirmed infection within 24 hours of admission" that cannot be assessed at time of ICU admission. This is inconvenient in that it requires waiting 24 hours for full assessment. Its value may also be impacted by processes of care within the first 24 hours of ICU admission. While this is potentially problematic, ICUacquired infections are generally not recognized within 24 hours, making this variable a marker of infections that began prior to ICU admission. Nonetheless, it may be worthwhile in future validations of the model to assess suspected or confirmed infection at the time of ICU admission. Fourth, our model suffers from the same challenges as other risk prediction models: leadtime bias, the impact of pre-ICU and post-ICU care on outcomes, and the need for ongoing recalibration. [18] These are issues inherent to ICU risk prediction models, which can be partially mitigated by ongoing recalibration, but cannot be fully overcome. They are perhaps magnified in the mismatch of demand for and capacity of ICU beds, meaning that decisionmaking about who gets an ICU bed will have a large impact on the characteristics of ICU patients over time. Our model is based on patients admitted to two ICUs, a clinically-selected population that may not reflect the characteristics of all critically ill patients in the hospital. This is in fact one of the reasons that having a risk prediction model is so important in this setting-tracking outcomes over time may be impacted by changes in the ICU population over time, and a risk prediction model helps to control for such changes.

Conclusions
Our study is the first performance-test of an ICU risk prediction model in a low-income setting and the first development of a new context-specific model in over twenty-five years. Our model is both better fit to our population and has a lower data collection burden than models developed in high-income countries. Our small sample size requires external validation in our own ICUs, other ICUs in low-income countries and in critically ill patients outside of ICUs. We are already aware of ICUs in three African countries that could apply this model with currently collected data. Just as large cohort databases have developed in the United States and Europe, so too a network of ICUs and hospitals from low-income countries could provide the necessary cohort to validate and reassess the model over time. While this presents a formidable challenge, we believe that both low-and high-income countries need ICU predictive models in order to effectively pursue quality improvement and research. [18] Our model is a starting place, and its choice of few and accessible variables means it can be readily assessed by other sites caring for critically ill patients in low-income countries.
Supporting Information S1 Supporting Information. Reason for ICU admission. (DOCX) S1 Table. Additional patient characteristics at hospital admission. n = number of patients. IQR = interquartile range. HIV = human immunodeficiency virus. Ã Totals vary depending upon missing data for some patients. ÃÃ We could not locate in-hospital vital outcomes for two patients after extensive searching, so the number of patients in the survivor and non-survivor columns add to 425. (DOCX) S2 Table. Additional patient characteristics at ICU admission and during ICU stay. ICU = intensive care unit. n = number of patients. IQR = interquartile range. GCS = Glasgow Coma Scale. CPR = Cardiopulmonary resuscitation. Ã Totals vary depending upon missing data for some patients. ÃÃ We could not locate in-hospital vital outcomes for two patients after extensive searching, so the number of patients in the survivor and non-survivor columns add to 425. (DOCX) S3 Table. Missing values for variables in the MPMo-III and R-MPM models. GCS = Glasgow Coma Scale. CPR = Cardiopulmonary resuscitation. Ã The high proportion of missing values for GCS is driven by the many patients who had received sedating medications and could therefore not have GCS assessed. Of the 155 participants with missing GCS value, 134 were missing due to receiving sedating medications and 21 were missing due to having no GCS recorded. (DOCX)