30-Day Survival Probabilities as a Quality Indicator for Norwegian Hospitals: Data Management and Analysis

Background The Norwegian Knowledge Centre for the Health Services (NOKC) reports 30-day survival as a quality indicator for Norwegian hospitals. The indicators have been published annually since 2011 on the website of the Norwegian Directorate of Health (www.helsenorge.no), as part of the Norwegian Quality Indicator System authorized by the Ministry of Health. Openness regarding calculation of quality indicators is important, as it provides the opportunity to critically review and discuss the method. The purpose of this article is to describe the data collection, data pre-processing, and data analyses, as carried out by NOKC, for the calculation of 30-day risk-adjusted survival probability as a quality indicator. Methods and Findings Three diagnosis-specific 30-day survival indicators (first time acute myocardial infarction (AMI), stroke and hip fracture) are estimated based on all-cause deaths, occurring in-hospital or out-of-hospital, within 30 days counting from the first day of hospitalization. Furthermore, a hospital-wide (i.e. overall) 30-day survival indicator is calculated. Patient administrative data from all Norwegian hospitals and information from the Norwegian Population Register are retrieved annually, and linked to datasets for previous years. The outcome (alive/death within 30 days) is attributed to every hospital by the fraction of time spent in each hospital. A logistic regression followed by a hierarchical Bayesian analysis is used for the estimation of risk-adjusted survival probabilities. A multiple testing procedure with a false discovery rate of 5% is used to identify hospitals, hospital trusts and regional health authorities with significantly higher/lower survival than the reference. In addition, estimated risk-adjusted survival probabilities are published per hospital, hospital trust and regional health authority. The variation in risk-adjusted survival probabilities across hospitals for AMI shows a decreasing trend over time: estimated survival probabilities for AMI in 2011 varied from 80.6% (in the hospital with lowest estimated survival) to 91.7% (in the hospital with highest estimated survival), whereas it ranged from 83.8% to 91.2% in 2013. Conclusions Since 2011, several hospitals and hospital trusts have initiated quality improvement projects, and some of the hospitals have improved the survival over these years. Public reporting of survival/mortality indicators are increasingly being used as quality measures of health care systems. Openness regarding the methods used to calculate the indicators are important, as it provides the opportunity of critically reviewing and discussing the methods in the literature. In this way, the methods employed for establishing the indicators may be improved.


Background
The Norwegian Knowledge Centre for the Health Services (NOKC) reports 30-day survival as a quality indicator for Norwegian hospitals. The indicators have been published annually since 2011 on the website of the Norwegian Directorate of Health (www.helsenorge.no), as part of the Norwegian Quality Indicator System authorized by the Ministry of Health. Openness regarding calculation of quality indicators is important, as it provides the opportunity to critically review and discuss the method. The purpose of this article is to describe the data collection, data pre-processing, and data analyses, as carried out by NOKC, for the calculation of 30-day risk-adjusted survival probability as a quality indicator.

Methods and Findings
Three diagnosis-specific 30-day survival indicators (first time acute myocardial infarction (AMI), stroke and hip fracture) are estimated based on all-cause deaths, occurring in-hospital or out-of-hospital, within 30 days counting from the first day of hospitalization. Furthermore, a hospital-wide (i.e. overall) 30-day survival indicator is calculated. Patient administrative data from all Norwegian hospitals and information from the Norwegian Population Register are retrieved annually, and linked to datasets for previous years. The outcome (alive/death within 30 days) is attributed to every hospital by the fraction of time spent in each hospital. A logistic regression followed by a hierarchical Bayesian analysis is used for the estimation of risk-adjusted survival probabilities. A multiple testing procedure with a false discovery rate of 5% is used to identify hospitals, hospital trusts and regional health authorities with significantly higher/lower survival than the reference. In addition, estimated risk-adjusted survival probabilities are published per hospital, hospital trust and regional health authority. The variation in risk-adjusted survival probabilities across hospitals for AMI shows a decreasing trend over time: estimated survival probabilities for AMI in 2011 varied

Introduction
Several different quality indicators, with the aim of monitoring the quality of care in the health services, have been developed, studied and reported in the literature during the past decades [1][2][3][4][5]. Quality indicators are measured with the aim to provide information regarding how well a hospital performs its services [6]. However, such measurements alone does not assess the quality of care at a hospital [7]. By considering several indicators that describe the relevant aspects of a hospital, one may get a picture of the overall performance of the hospital.
Mortality within a fixed-time period (commonly 30 days) after admission is among the reported indicators and is considered to be an effective means for measuring the outcome of hospital care [8]. As mortality is perceived a negative framing, the Norwegian Knowledge Centre for the Health Services (NOKC) reports the estimated 30-day survival probabilities as routinely reported quality indicators for Norwegian hospitals [9][10][11][12]. This is in contrast to the vast majority of other quality indicator systems which report mortality or standardized mortality ratio (SMR).
Three disease-specific and one hospital-wide survival indicators are estimated. The diseasespecific conditions are first time acute myocardial infarction (AMI), stroke and hip fracture. The hospital-wide survival is calculated for a subset of diagnoses, which accounts for approximately 80% of the total number of deaths within 30 days after admission to Norwegian hospitals. The patient administrative data used for calculation of the hospital-wide indicator correspond to the data commonly used for the hospital standardized mortality ratios.
The four quality indicators from NOKC are estimated and reported at hospital level, hospital trust level and regional health authority level. We employ information from all admissions for a patient when calculating the indicators i.e. for transferred patients, the outcome (of alive or death within 30 days) is attributed to every hospital by the fraction of time spent in them (weighted admissions) [13].
Several studies have highlighted the growing need for methodological peer-reviewed publications and guidelines that can be employed for studying and producing health care indicators [14,15]. This technical article aims to give a detailed description of the methods we employed for data collection, data processing and statistical analysis of the 30-day survival indicators.

Material and Methods Data
Data Sources. Patient Administrative System (PAS) Data Each record in PAS data contains information from a single ward admission. The same patient might have several records as the patient is transferred between wards and even hospitals. PAS data comprise admission category (i.e. elective or emergency), diagnosis codes (both primary and secondary), codes for medical procedures, age, gender, date and time for ward admission/discharge. PAS data are collected from two sources: 1. Data from 2009 and earlier: PAS data extracted directly from the hospitals using an inhouse developed system for the purpose of quality monitoring [16]. At each hospital the same patient has a unique patient record key for the whole period.
2. Data from 2010 and onward: PAS data are provided by the Norwegian Patient Register (NPR); it is mandatory for all Norwegian hospitals to submit data to NPR. The records from each patient have the same unique patient record key throughout the dataset period.
The National Registry Data from the National Registry are provided by Statistics Norway [17]. All permanent residents in Norway have a Personal Identification Number (PIN). Statistics Norway have yearly received files from the hospitals (for data up to the year 2009) or from NPR (for data for 2010 and onward), containing the PIN and the unique patient record key (mentioned above). An encrypted PIN is generated by Statistics Norway, enabling us to track all patients between hospitals and link the patients to their hospitalizations throughout the data collection period. Each year, updated information for the respective patients; age, gender, date of death (in-hospital and out-of-hospital) and patients' status (e.g. habitant or emigrated) is retrieved from the National Registry.
Data Pre-processing. Constructing episodes of care We define an episode of care as the continuous course of care of a patient. This may include a sequence from admission to a hospital, through within-hospital transfers and transfers between different hospitals, to the discharge of the patient.
An overview of how the episodes of care are constructed is given in the following, while the detailed description of the algorithm is shown in a flowchart (see Fig 1).
• Firstly, ward admissions that lack time or date for admission or discharge are excluded.
• Thereafter, ward admissions are ordered in time sequence.
• Ward admissions are then aggregated into a single episode of care, if the time difference between the discharge of the first one and admission of the next one is less than eight hours.
• In case of parallel ward admissions: If the parallel ward admissions start at the same time, the shorter ones are excluded, while their diagnosis and procedure codes are still taken into account.
If the parallel ward admissions start at different time points, the ones that start later are excluded, while their diagnosis and procedure codes are still taken into account. to D). Thereafter, the patient is transferred to Hospital II, and back to Hospital I, followed by a stay at Hospital III, and finally discharged from Hospital I. If the time difference is more than eight hours, a new episode of care is generated. The time difference between ward admissions I and J are more than eight hours, and a new episode of care is therefore generated. Two episodes of care are constructed for the patient based on these admissions (illustrated in Fig 2b).
Calculating hospital weights In the analysis, episodes spanning more than one hospital must be attributed to these hospitals in a meaningful way. For this purpose, episodes are assigned hospital weights. For the episodes of care containing more than one hospital (i.e. for transferred patients), we establish the hospital weights proportional to the fraction of time (within the episode of care) spent in each hospital [13]. Weights are restricted to the interval [0.05, 1.00]. One single episode of care has a total weight of 1. For the episodes of care including only one hospital, the hospital weight will be 1.
The amount of hours spent in each hospital in the previous example (in Fig 2) are shown in Fig 2c and  Weight calculated based on episode 2: The total weight of a hospital is defined as the summed weights for all of the episodes of care belonging to that hospital.
Total weights calculated based on both episodes: Hospital II 0 s weight ¼ 0:2 Case selection. The case selection criteria are different for the hospital-wide data set and the three diagnosis-specific data sets. Diagnosis codes are the ICD-10 diagnosis codes from the Norwegian version of ICD-10 [18]. In this technical article, however, the corresponding English description of the codes are taken from the original WHO (World Health Organisation) version of ICD-10 [19]. Moreover, wherever "x" is written in a diagnosis code, it may be replaced by nothing or anything i.e. an empty string or any number may replace "x".
Hospital-wide data set Different admissions may have different sets of diagnosis and procedure codes. The first non-vague primary diagnosis [20] is used to identify the patients in the hospital-wide data set, and is defined as the first occurring (according to the time sequence of admissions) primary diagnosis that is not on the list of vague diagnoses (see S1 Appendix). If no such diagnosis is found, the last occurring diagnosis is used.
Inclusion Criteria Episodes of care are included if 1. The episode discharge date is between January 1 and December 31 of a given year i.e. a oneyear period hospital-wide data set is selected.
2. The first non-vague primary diagnosis code belongs to a subset of Clinical Classifications Software (CCS) [21,22]  2. The diagnosis codes correspond to the criteria specified in Table 1. For transferred patients, the diagnoses must occur at the first hospital in the episode of care.
3. Admission date is recorded (not missing).
4. Admission category is recorded as emergency.
5. The patient has a valid encrypted PIN and information regarding the patient's status (alive or dead) 30 days from the admission date is recorded.
The admission date and admission category, however, refer to the first ward admission in the episode of care.
Exclusion Criteria Episodes of care are excluded if the criteria specified in Table 2 are met.

Statistical Analysis
Definition. By the 30-day survival indicator, we refer to the risk-adjusted probability of survival within 30 days of the (episode) admission date.
Modelling. The survival probabilities are estimated in three steps: Step 1: Generalised Linear Model (GLM) Regression Analysis A GLM regression model is fitted to the set of episode data from all of the hospitals, where the explanatory variables are case-mix variables as well as hospital weights and (for the threeyear diagnosis-specific data sets) a time trend, linear in admission year. The response variable is death within 30 days from the admission date.
The case-mix variables used for the hospital-wide and for the diagnosis specific data sets are slightly different and are, therefore, described separately in the following. Age is modelled by natural splines [25] while pre-admissions and CCI are modelled by fractional polynomials [26].
Step 2: Identifying outliers by Multiple Significance Testing The regression coefficient of each hospital is compared to a reference value. The reference value is the 10%-trimmed mean of the regression coefficients, i.e. the lowest 10% and the highest 10% of the regression coefficients are excluded prior to calculation of the mean value. Multiple significance testing is then used to identify hospitals (hospital trusts or regional health authorities, depending on the level of the analysis) deviating from the reference (either lower or higher survival). Multiple testing is performed using the Benjamini-Hochberg method [27] with a False Discovery Rate (FDR) of 5%. Two-sided tests are used. Step

3: Survival Probability Calculation by Shrinkage Estimation
The estimated regression coefficients for each hospital are shrunk towards the reference value by a Hierarchical Bayesian Model [28,29]. The prior distribution is as follows: the true hospital regression coefficients are assumed to follow a Student's t-distribution with five degrees of freedom, multiplied by a scale factor σ. The precision σ -2 has a gamma distribution with both shape and rate parameters equal to 0.01.
Hypothetical survival probabilities for each patient are thereafter calculated using the adjusted GLM with the shrunk hospital coefficients, assigning all patients to each of the hospitals in turn. The survival probabilities for each hospital are then computed as the mean survival probabilities for all the patients when they are assigned to each hospital.
Hospital and hospital trust inclusion/exclusion. Several Norwegian hospitals are small and have a limited number of relevant cases. We thus do not consider hospitals (or hospital trusts, depending on the level of the analysis) in the analysis for hospital-wide data, if their total weight (according to the section Calculating the hospital weights) calculated for the oneyear period (for the patients belonging to the hospital-wide data), is less than 400.
Similarly, hospitals (or hospital trusts) are not considered in the analysis for diagnosis-specific data, if their weight calculated for the last year (for the patients belonging to the respective diagnosis-specific data) is less than 20 or their weight calculated for the whole three-year period (for the patients belonging to the respective diagnosis-specific data) is less than 100.

Ethics
The Norwegian Data Inspectorate and the Ministry of Health approved the data collection for 30-day mortality. Ethical approval was not necessary according to the Regional Ethics Committee, as existing data were employed for quality improvement purposes.
The Norwegian Knowledge Centre for the Health Services (NOKC) receives an encrypted PIN, generated by Statistics Norway, which does not enable us to have access to patient identifying information. The Norwegian Knowledge Centre does not have any direct interaction with patients.

Results
The Norwegian Knowledge Centre for the Health Services has published survival indicators annually since 2011 at hospital, hospital trust and regional health authority level [10][11][12]. The results are summarized in Tables 3-5. Total number of hospitals, hospital trusts and regional health authorities as well as the number of those with significantly higher/lower survival (compare to the reference) for each indicator category are given in the above mentioned tables. The highest, lowest and the reference for the risk-adjusted estimated survival probabilities calculated for the hospitals over the last three years are given in Table 6. Interestingly, it may be noticed that variation in estimated risk-adjusted survival indicators i.e. the difference between the highest and lowest estimated survival probabilities for each year, has shown a decreasing trend over time (although not statistically tested). This observed decreasing trend is visualised in Figs 3 and 4, where the distribution of estimated risk-adjusted 30-day survival probabilities for AMI (as an example of the diagnosis specific indicators) and hospital-wide indicator are plotted. The reference values for these two survival indicators show an increasing trend over time (see Figs 3 and 4), indicating increased survival over time.

Discussion
A quality indicator, as the name implies, only gives an indication of quality, it is not a direct measure of quality. Small hospitals that do not deviate significantly might have a too small sample size to be identified in the statistical testing. Another important issue to have in mind is the fact that hospitals that receive patients in the most acute and critical stage may have lower survival rates than hospitals that receive patients who are transferred or are in a more stable phase. This may especially be considered for AMI, where about 50% of patients are transferred. Estimation of the survival indicators fulfils its purpose if the indicators are actively monitored by the hospital trusts and regional health authorities in order to improve their quality of health care. Since the first public reporting in 2011, several hospitals and hospital trusts in Norway have initiated quality improvement projects, and some of the hospitals have improved the survival over the last four years. This is described in a separate paper, where a before-after study is reported [30].
Several methods for calculating 30-day mortality as a quality indicator have been suggested [29,31]. A commonly used approach is to calculate the hospital standardized mortality ratio (HSMR), i.e. the ratio of observed versus expected number of deaths [26]. Estimates for the case-mix effects are obtained from a reference population and therefore hospitals may be compared indirectly. Our choice of method compares the hospitals within the same logistic regression model and provides direct comparison of the hospitals [31]. Comparing a few hospitals where some hospitals have very low or very high mortality, the outlier hospitals may highly influence the hospital mean. Therefore, we believe that the trimmed mean is preferable to the common deviance from mean testing, by using a direct comparison approach.
On the other hand, our in-house developed script is specifically designed to handle both PAS data and data from the National Registry. Strengths of our script compared to the other quality indicator software available today, includes but is not limited to: computing survival rather than mortality, assigning hospital weights, pre-processing of the data received, constructing episodes of care, updating the database annually, generating patient population for each indicator with respect to inclusion/exclusion criteria, and running the statistical analyses.

Limitations
The estimated indicators may only be as accurate as the underlying PAS data. Hospital trusts in Norway are required to document the treatment given to their patients according to a defined coding system for activity based financing based on Diagnosis Related Groups. The Norwegian version of ICD-10 defines the primary diagnosis as the diagnosis most responsible for the hospital care given. This will often represent the greatest length of stay or the greatest consumption of hospital resources and may not necessarily be the reason for the patient's admission. For example, a hip fracture case may be coded with pneumonia as the main diagnosis, if this complication occurs and requires considerable resources and prolongation of the patient's length of stay.
For AMI and stroke, severity of the diseases could also affect the estimated indicators. Information regarding the severity currently does not exist in the PAS data and therefore we were not able to consider it in our data modelling. We are planning to retrieve this information and include it in the modelling as an adjustment factor (the National Cardiovascular Disease registry collects this information).

Conclusions
The methods that NOKC employs for the calculation of four 30-day survival indicators is described in this technical article. Our methods have been tested, employed and revised over 30-Day Survival Quality Indicators for Norwegian Hospitals the past 10 years [16]. The methods described in this article are the ones that are currently in use. NOKC aims to evaluate alternative approaches for calculating the hospital weights, among other methodological aspects. As our methods develop, the plan is to update this technical article accordingly.

Softwares
Implementation of the protocol described in this paper was done by means of in-house-written and standard R routines. All data pre-processing and statistical modelling are performed in R version 3.0.3. Baysian shrinkage analysis is performed using JAGS.