Using Highly Detailed Administrative Data to Predict Pneumonia Mortality

Background Mortality prediction models generally require clinical data or are derived from information coded at discharge, limiting adjustment for presenting severity of illness in observational studies using administrative data. Objectives To develop and validate a mortality prediction model using administrative data available in the first 2 hospital days. Research Design After dividing the dataset into derivation and validation sets, we created a hierarchical generalized linear mortality model that included patient demographics, comorbidities, medications, therapies, and diagnostic tests administered in the first 2 hospital days. We then applied the model to the validation set. Subjects Patients aged ≥18 years admitted with pneumonia between July 2007 and June 2010 to 347 hospitals in Premier, Inc.’s Perspective database. Measures In hospital mortality. Results The derivation cohort included 200,870 patients and the validation cohort had 50,037. Mortality was 7.2%. In the multivariable model, 3 demographic factors, 25 comorbidities, 41 medications, 7 diagnostic tests, and 9 treatments were associated with mortality. Factors that were most strongly associated with mortality included receipt of vasopressors, non-invasive ventilation, and bicarbonate. The model had a c-statistic of 0.85 in both cohorts. In the validation cohort, deciles of predicted risk ranged from 0.3% to 34.3% with observed risk over the same deciles from 0.1% to 33.7%. Conclusions A mortality model based on detailed administrative data available in the first 2 hospital days had good discrimination and calibration. The model compares favorably to clinically based prediction models and may be useful in observational studies when clinical data are not available.


Introduction
Bacterial pneumonia is a leading cause of morbidity and mortality in the United States. Every year, more than 8 million patients are admitted to US hospitals with pneumonia; 8.8% of them will die. [1] Despite the common nature of this condition, there are large gaps in our knowledge regarding how best to care for pneumonia patients. Most recommendations in national treatment guidelines are not based on randomized trials, and there is a paucity of comparative effectiveness research.
Administrative databases derived from billing records are attractive candidates for health services research, as well as for use in hospital profiling initiatives, because the number of patient records is large and the acquisition cost is low. Observational studies using administrative data can be used to assess comparative effectiveness in real world settings, and findings from such studies are sometimes confirmed in randomized trials. One concern, however, is that such studies are often biased by confounding by indication, in which the choice of treatment is influenced by a patient's severity of illness. This threat can be limited through the use of validated risk prediction instruments that are capable of adjusting for pre-treatment severity of illness, as well as comorbidities.
There exist a number of validated pneumonia mortality prediction instruments for use in clinical care. [2,3] All of these require clinical data, such as respiratory rate or blood urea nitrogen, which are not generally available in administrative data sets. Others have attempted to construct predictive mortality models from administrative data. International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes assigned at discharge are highly predictive of mortality, in great part because they include complications of hospitalization which often precede death. [4] Such models are not useful for severity adjustment because they incorporate the results of treatment (e.g., complications) as predictors. Models restricted to demographics and comorbidities at the time of admission have much lower predictive accuracy [5].
Highly detailed administrative datasets include a date-stamped record for each item administered during a hospitalization; this allows for differentiation between factors present at the time of hospitalization and those arising during the stay. We used one such dataset to create and validate a mortality risk prediction model that included only tests and treatments administered in the first 2 hospital days along with patient demographics and comorbidities.

Setting and Patients
We identified patients discharged between July 1, 2007 and June 30, 2010 from 347 US hospitals that participated in Premier, Inc.'s Perspective, a database developed for measuring quality and healthcare utilization that has been described previously. [6][7][8] Member hospitals represent all regions of the US, and are generally reflective of US hospitals; although larger hospitals, hospitals in the South and those in urban areas are over represented. Perspective contains all data elements found in the uniform billing 04 form, such as sociodemographic information, ICD-9-CM diagnosis and procedure codes, as well as hospital and physician information. It also includes a date-stamped log of all billed items and services, including diagnostic tests, medications, and other treatments. Because the data do not contain identifiable information, the Institutional Review Board at Baystate Medical Center determined that this study did not constitute human subjects research.
We included all patients aged $18 years with a principal diagnosis of pneumonia, or a secondary diagnosis of pneumonia paired with a principal diagnosis of respiratory failure, ARDS, respiratory arrest, sepsis or influenza ( Figure S1). Diagnoses were assessed using International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes. We excluded all patients transferred in from or out to other acute care facilities, because we either could not assess initial severity or could not assess outcomes; those with a length of stay of 1 day or less; patients with cystic fibrosis; those whose attending physician of record was in a specialty that would not be expected to treat pneumonia (e.g., psychiatry); those with a diagnosis related grouping (DRG) inconsistent with pneumonia (e.g., Prostate OR procedure); those with a code indicating that the pneumonia was not present on admission; and any patient who did not have a chest radiograph and did not begin antibiotics on hospital day 1 or 2. For patients with multiple eligible admissions in the study period, 1 admission was randomly selected for inclusion.

Markers of Comorbid Illness and Pneumonia Severity
For each patient, we extracted age, gender, race/ethnicity, insurance status, principal diagnosis, comorbidities, and specialty of the attending physician. Comorbidities were identified from ICD-9-CM secondary diagnosis codes and DRGs using Healthcare Cost and Utilization Project Comorbidity Software, version 3.1, based on the work of Elixhauser. [9] We identified a group of medications, tests, and services that are typically associated with chronic medical conditions (e.g., spironolactone, warfarin, need for a special bed to reduce pressure ulcers), as well as acute medications that may indicate severe illness (e.g., vasopressors, intravenous steroids). We also identified early use of diagnostic tests (e.g., arterial blood gas, serum lactate) and therapies (e.g., mechanical ventilation, blood transfusion, restraints) that are associated with more severe presentations of pneumonia. The complete list of medications, tests, and treatments appears in Table S1. To avoid conflating initial severity with complications of treatment, we limited our analysis to those markers received in the first 2 hospital days. We used the first 2 days because hospital days are demarcated at midnight and the first day often represents only a few hours.

Statistical Analysis
Individual predictors of mortality were assessed using Chisquare tests using the full study cohort. Stratifying by hospital, 80% of the eligible admissions were randomly assigned to a derivation and 20% to a validation cohort, and the two cohorts were compared for differences in potential predictors. Using the derivation cohort, we developed a series of multivariable logistic regression models to predict in-hospital death. Hierarchical generalized linear models (HGLM) with a logit link (SAS PROC GLIMMIX) were used to account for the clustering of patients within hospitals. We grouped predictors into the following categories: demographics, comorbid conditions, and severity markers. We developed separate mortality models for each of these categories, including main effects and significant pairwise interactions. Factors significant at p,0.05 were retained. For each model we calculated the area under the receiver operating characteristic (AUROC) curve, together with 95% confidence intervals. [10] The final model was developed by sequentially adding effects retained in individual category models and evaluating pairwise interaction terms. Main effects that were dropped at earlier stages were re-evaluated for inclusion in the final model.
The purpose of the model was accurate prediction of mortality and risk stratification. We did not attempt to determine which individual factors were associated with mortality or to imply causality. Therefore, we did not require a priori information about the association of the various risk factors or interaction terms with the outcome. Although such an approach may result in spurious associations of individual risk factors, it need not necessarily detract from the model's accuracy of prediction, which was our primary concern [11].
In order to guard against the possibility of overfitting our model, parameter estimates derived from the model were used to compute individual mortality risk in the remaining 20% of the admissions (the validation cohort). Discrimination of the final model in the validation set was assessed by the c-statistic as well as the expected/observed ratio. Both cohorts were categorized by decile of risk based on the probability distribution in the derivation cohort, and observed mortality was compared to that predicted by the model. We also used the integrated discrimination improvement (IDI) index [12] to measure the improvement of the final model over a basic model including only demographics and ICD-9-CM comorbidities. We next evaluated model performance in subpopulations of the entire cohort based on hospital and patient characteristics. Specifically, we assessed model performance in strata defined by hospital size, teaching status, patient age, ICU and non-ICU admissions, and pneumonia type [healthcare-associated (HCAP) vs. community-acquired (CAP)]. All analyses were performed using the Statistical Analysis System (version 9.

Results
The dataset included 200,870 patients in the derivation cohort and 50,037 patients in the validation cohort. Patient characteristics of the full study cohort appear in Table 1. Most patients were over age 65, 53.3% were female and 68.0% were white. The most common comorbidities were hypertension (46.5%), diabetes (23.8%), chronic pulmonary disease (48.6%), and anemia (22.2%). Patients in the validation cohort were similar (Table S2).

Model Development
Overall in-hospital mortality in the derivation cohort was 7.2%. A large number of patient and hospital factors were associated with mortality (Table 1). Due to the large sample size, even weak associations appear highly statistically significant. Figure 1 shows the model discrimination, as measured by the area under the ROC curve, when subgroups of factors were used to model mortality. Including only patient demographics produced a model with poor discrimination (AUROC = 0.66). Using traditional ICD-9-CM based measures of comorbidity showed greater discrimination (AUROC = 0.71), as did a model that used admission to the ICU in day 1 or 2 as the only predictor (AUROC = 0.73). As an alternative measure of comorbidity, chronic medications were superior to ICD-9-CM codes in predicting mortality (AUROC 0.74 vs. 0.71, p,.001). Combining demographics, comorbidities, and markers of severity of illness on presentation (other infections present-on-admission, admission to ICU, the ability to take oral medications, and acute medications, tests and therapies used in first 2 days) offered excellent discrimination in the derivation cohort (AUROC = 0.85). We also assessed model discrimination using the IDI. Compared to the model including only demographics and ICD-9-CM comorbidities, the full model had an IDI which was 12 percentage points higher (16.6% vs. 4.6%, p,.001).

Validation
Model discrimination measured by the c-statistic in the validation set was 0.85 (95%CI: 0.844-0.856). Deciles of predicted risk ranged from 0.3% to 34.3% with observed risk over the same deciles from 0.1% to 33.7% (Figure 2). The expected mortality rate according to the model was 7.1% (expected/observed ratio: 1.00 [95% CI 0.97-1.03]). Performance of the model in subpopulations of the entire cohort is shown in Table 2. The model performed well in all subpopulations tested, but discrimination was poorest among patients in intensive care (c-statistic 0.78) and best among patients aged 18 to 64 years (c-statistic 0.89). In all subgroups the range of predicted mortality extended from #0.3% to .90%. Model calibration was also good in all subgroups. The model tended to underestimate the risk of mortality among patients with healthcare-associated pneumonia, and to a lesser extent among patients in teaching hospitals and those outside of the ICU. At the same time, it overestimated the risk of mortality among patients with community-acquired pneumonia.

Discussion
In this retrospective cohort study, we used highly detailed administrative data to derive and validate a pneumonia mortality prediction model for use in observational studies. The model had discriminatory ability comparable to those derived from clinical data, but unlike most other administrative models, it included information on illness severity that would be available in the first 2 hospital days. The model also had excellent calibration and successfully divided patients into mortality deciles ranging from ,0.5% to .33%. Interestingly, the 30% of patients with the lowest predicted mortality had an observed mortality of ,1%.
At least two clinical prediction tools have been developed for the purposes of risk stratifying patients with community acquired pneumonia-the CURB-65, [3] modified from earlier work by the British Thoracic Society, and the Pneumonia Severity Index (PSI).
[2] The CURB-65 consists entirely of exam findings and laboratory values, while the PSI incorporates some historical information as well. At least 3 studies have prospectively compared the predictive abilities of these two measures. [13][14][15] Perhaps due to differences in study population, c-statistics for predicting 30-day mortality ranged from 0.73 to 0.89 across studies; however, within any given study, there were no statistically significant differences between the two scales.
Because the clinical information required for these tools is not available in administrative databases, others have attempted to create models based solely on administrative claims. In general, such models have modest discriminatory ability, unless they are combined with laboratory data. For example, one administrative claims model developed for profiling hospitals' pneumonia mortality rates, and containing age, sex, and 29 comorbidities (based on ICD-9-CM codes from the index hospitalization and the prior year's outpatient visits) had a c-statistic of 0.72. [16] Addition of laboratory values to administrative data can substantially enhance discrimination. Tabak et al. demonstrated that laboratory values alone contributed 3.6 times as much explanatory power as ICD-9-CM codes and 2.5 times as much as vital signs to mortality prediction. [17] For example, the c-statistic for a model that only includes laboratory values and age was 0.80. Adding ICD-9-CM codes and vital signs increased the c-statistic to 0.82. [17] Pine et al. also found that ICD-9-CM codes alone produced a c-statistic of 0.78, whereas addition of laboratory values increased the c-statistic to 0.87. [4] Addition of chart-based data (e.g., vital signs) had a small marginal effect on the model's predictive ability [4,18].
Our study takes a different approach to overcoming the limitations of administrative data. In brief, our results suggest that it is possible to tell a lot about patients by the tests, medications and treatments they are prescribed. Although others have utilized    ambulatory medications to predict outpatient costs and mortality, these generally do not perform better than comorbidity models based on ICD-9-CM codes. [19,20] In contrast, by assessing medications, tests and treatments administered in the first 2 hospital days, we were able to identify chronic comorbid conditions, as well as factors indicative of the severity of illness on presentation. Indeed, use of chronic medications alone predicted mortality better than ICD-9-CM codes. This could be because billing codes are more sensitive than ICD-9-CM codes, but also because medication use can identify not just the presence of disease, but also provide information about disease severity. For example, among patients with heart failure, spironolactone often signifies severe systolic dysfunction, and nadolol in the presence of liver disease likely indicates portal hypertension. Medications, however, did not capture all the information present in ICD-9-CM codes, and the combination of the two was a more powerful predictor than either one alone. This is likely because some chronic conditions, such as metastatic cancer, may not be associated with any routine medications, but are nonetheless potent predictors of mortality. The inclusion of certain initial tests and therapies also allowed us to estimate the severity of illness at the time of admission in the absence of laboratory or clinical data. Although it would be helpful to know the results of a blood gas, the simple presence of that test is indirect evidence that the treating physicians were concerned about a patient's respiratory condition. Similarly, a patient receiving vasopressors is almost certainly hypotensive. More importantly, our model's predictive ability was comparable to that seen with other administrative models that include laboratory data, as well as those that are based on physiological information obtained from review of medical records. An analogous model, designed for use in sepsis patients, demonstrated that highly detailed administrative data can achieve discrimination and calibration similar to clinical mortality prediction models, [21] with the majority of the additional explanatory power of the model arising from the inclusion of initial treatments [22].
Our study has several limitations. First, our main outcome was in-hospital mortality. Others have modeled 30-day mortality and the factors that are predictive of in-hospital mortality may be different than those which predict 30-day mortality. [23] Second, our study was conducted retrospectively and the model, therefore, may perform differently in a prospective cohort. It would certainly be premature to base treatment decisions on our model, but that is not its intended purpose. Third, our definition of pneumonia was based on diagnosis and charge codes. Some patients may not have had pneumonia and some cases of pneumonia may have been missed. These numbers are likely to be small, as the positive predictive value of an ICD-9 diagnosis paired with an antibiotic description is .95%. [24] Fourth, we excluded patients with pneumonia not present on admission, as well as transfer patients, so our model is not applicable to these groups. Finally, our model derives much of its power from physician assessments of patients' disease, as represented by physician ordering. To the extent that prescribing thresholds vary by institution, the model may be more or less accurate in certain hospitals, and therefore could not be used for benchmarking purposes. The fact that model discrimi-nation was good across various subgroups of hospitals is reassuring in this regard. This model could be used in various ways. It could be used for adjustment in observational trials, including comparative effectiveness or epidemiologic studies. Although such studies might also be performed using clinical data, many institutions do not  currently have the ability to automatically extract clinical data from electronic medical records and many administrative datasets do not yet contain laboratory data. Our model represents a lowcost yet accurate alternative. In addition, unlike existing clinical models, our model was validated in several different subpopulations, with excellent performance in small and large hospitals, and in teaching and non-teaching institutions.
The model could, for example, be used for severity-adjustment in a study to compare effectiveness of guideline recommended therapies to alternative treatment options in community acquired pneumonia. It could also be used to study the severity of an illness such as healthcare associated pneumonia, in which multiple comorbid illnesses might contribute to poor outcomes. It could have application for studying the methods of hospital profiling for public reporting (e.g., testing alternative definitions of diagnosis), but may not be useful for profiling hospitals per se, because thresholds for treatment might vary across hospitals making patients appear more or less sick. Finally, some aspects of the model-specifically the chronic medications-could be incorporated into clinical prediction rules such as the PSI, in order to improve their accuracy. To avoid showering clinicians with unnecessary complexity, these could be embedded in clinical information systems to provide prognostic information at the point of care. However, prospective validation of such a hybrid model is required before it can be applied in clinical care.
In conclusion, we have created a mortality prediction model based on highly detailed administrative data available in the first 2 days of hospitalization. The performance of the model was comparable to that of models based on clinical data, and the performance was consistent across different patient subpopulations. The model should be useful for comparative effectiveness research using large, administrative databases.