Improving medication safety: Development and impact of a multivariate model-based strategy to target high-risk patients

Background Preventive strategies to reduce clinically significant medication errors (MEs), such as medication review, are often limited by human resources. Identifying high-risk patients to allow for appropriate resource allocation is of the utmost importance. To this end, we developed a predictive model to identify high-risk patients and assessed its impact on clinical decision-making. Methods From March 1st to April 31st 2014, we conducted a prospective cohort study on adult inpatients of a 1,644-bed University Hospital Centre. After a clinical evaluation of identified MEs, we fitted and internally validated a multivariate logistic model predicting their occurrence. Through 5,000 simulated randomized controlled trials, we compared two clinical decision pathways for intervention: one supported by our model and one based on the criterion of age. Results Among 1,408 patients, 365 (25.9%) experienced at least one clinically significant ME. Eleven variables were identified using multivariable logistic regression and used to build a predictive model which demonstrated fair performance (c-statistic: 0.72). Major predictors were age and number of prescribed drugs. When compared with a decision to treat based on the criterion of age, our model enhanced the interception of potential adverse drug events by 17.5%, with a number needed to treat of 6 patients. Conclusion We developed and tested a model predicting the occurrence of clinically significant MEs. Preliminary results suggest that its implementation into clinical practice could be used to focus interventions on high-risk patients. This must be confirmed on an independent set of patients and evaluated through a real clinical impact study.


Introduction
Drug-related problems result in substantial medical and economic burden, with an estimated cost of $177.4 billion per year [1]. They are commonly defined as "events or circumstances involving drug therapy that actually or potentially interfere with desired health outcomes" [2]. These include both adverse drug events, which result in actual harm, and clinically significant medication errors (called "potential adverse drug events"), which have the potential to cause harm in absence of correction. Review of the literature suggests that medication errors (MEs) are significantly common, occurring in up to 67% of hospitalized patients [3]. Given their preventability, frequency, and potential association with adverse patient outcomes [4][5][6], strategies for reducing clinically significant MEs are of the utmost importance.
Several studies have highlighted the role of pharmacist-led medication review for preventing potential adverse drug events [6][7][8][9][10][11][12][13][14][15][16]. By closely collaborating with each other, physician and pharmacist complement one another and attempt to address patient needs. This inter-professional intervention, done in association with the medical team, consists of optimization of medication regimens after home medication reconciliation at admission, pharmaceutical analysis and review of currently prescribed drugs [6][7][8][9][10][11][12][13][14][15][16]. The process is very time consuming and, because of competing clinical demands, review of all inpatients by a clinical pharmacist is usually not possible. As a consequence, those at increased risk of potential adverse drug events need to be identified and considered a priority for evaluation.
Targeting "high-risk" patients might be a key aspect of successful interventions [6,16]. Nevertheless, commonly chosen criteria, such as age or number of medications [16], often fail to identify these patients [6]. Instead of considering a single predictor strategy, we propose a multivariate model-based strategy to detect these high-risk patients. Several approaches to the prediction of actual or potential adverse drug events are described in the literature [17][18][19][20][21], though they vary in terms of definition. While some focus on a subcategory [17][18][19][20], others are specific to certain age groups or diagnostic populations [17,19,20]. Rather than focus on a specific subgroup, our approach aims to develop a generalizable tool for daily practice that includes non-invasive predictors of avoidable outcomes that are readily available at the time of hospital admission.
We report the derivation of a model predicting in-hospital significant medication errors (PRISMOR), according to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement [22,23] and assess its clinical impact on decision-making compared to a strategy based on age.

Methods
Data collection for the study was approved by the French National Commission of Information Technology and Civil Liberties (CNIL, Commission Nationale de l'Informatique et des Libertés; agreement No. 1738666) and the Institutional Review Board of the Nîmes University Hospital Centre, which waived the need for written informed consent (No. IRB 14/01-03). Patient information were de-identified prior to analysis.

Design
We conducted a prospective cohort study in the Nimes University Hospital Centre from March 1 st to April 31 st , 2014, targeting a sample size that would result in 20 events per variable [24,25]. Based on a recent systematic review reporting 45% of patients with at least one clinically significant ME [6], we estimated a required sample size of 1,000 patients. All patients over 17 years old admitted to hospital during the study period were included. Patients were followed from admission to discharge. During admission, all prescribed medications were reviewed by one of 15 clinical pharmacists, all of whom were trained in standardized medication reconciliation, prescription analysis and ME reporting.

Endpoint and predictors
The primary outcome was the occurrence of at least one clinically significant ME at any time during in-hospital stay. MEs were defined as either unintentional medication discrepancies (i. e. involuntary divergence between admission prescription and best possible medication history obtained from a minimum of 3 sources of information) or prescribing errors (i.e. non-adherence to recognized clinical prescribing guidelines). Given their more imprecise nature, two clinicians (a physician and a pharmacist) independently reviewed all MEs using the method described by Quélennec et al. [26], a European adaptation of the National Coordinating Council for Medication Error Reporting and Prevention (NCC MERP) Index [27]. Potential clinical impact of MEs was rated regarding patient's clinical characteristics, medications, biological data and number of MEs. MEs were classified into 3 categories: "without potential harm (NCC MERP category C)", "requiring potentially monitoring or intervention to preclude harm (NCC MERP category D)" and "with potential harm (NCC MERP category E and above)". Disagreements were resolved by a third evaluator. MEs without potential harm were not considered as events of interest in the primary analysis.
For risk prediction, we extracted patient-, hospitalisation-, and prescription-related covariates from the electronic medical record. Specifically, age, sex, admission details, admission type (medical or surgical), hospital admission within the preceding 30 days, hour (day or night) and day of admission (weekday or week-end), number of prescribed drugs and drug classes (according to the Anatomical Therapeutic Chemical (ATC) classification system). We also documented whether a current treatment had been initiated before admission and whether a best possible medication history had been obtained before the admission medication order.

Statistical analysis
Univariate analyses were performed using Wilcoxon-Mann-Whitney test for continuous variables and Chi-square test for binary variables. Relations between drug ATC classes and the endpoint were explored using association rules mining. This method is useful to detect cooccurrence of an endpoint with multiple binary covariates in the absence of an a priori hypothesis [28]. The algorithm was applied assuming a minimal support (i.e. minimal probability of co-occurrence) of 5% and a minimal confidence (i.e. minimal probability of endpoint occurrence given the covariates) of 10%. ATC classes found to be significant using association rules were then included in logistic regression. Variables with a P less than 0.50 were considered eligible for inclusion in the multivariate model. For the multivariate model development, a logistic regression was performed. Because a strict selection criterion would lead to discarding informative variables, the alpha risk threshold (P <0.30) was intentionally set as large in order to provide reliable estimates and performances [25,29,30]. Multivariable fractional polynomials analysis was conducted to take into account non-linearity of continuous variables. The apparent performance of the model was described using c-statistic and a calibration curve. Corrected performances were reported after bootstrapping (b = 500) for internal validation [24,31,32].
In order to evaluate the impact of implementing our predictive model into clinical practice we compared two strategies for pharmaceutical intervention. The first involved clinical decision-making based on age, whereas the second was based on the PRISMOR model. To this end, we conducted a series of simulated randomized controlled trials (RCTs), which allowed us to test our model under the conditions of variable pharmacist staff coverage (Fig 1). The endpoints of these trials were the interception of at least one harmful ME during the intervention and the failure to identify at least one harmful ME in the absence of intervention. We simulated the situation, in which such a model (training model) would be applied on an external sample (testing sample) [31,32]. After fitting a similar predictive model on a bootstrap sample (training model), we randomly assigned patients from our initial sample (testing sample) into one of two groups in a 1:1 ratio. In group A (control), older patients were identified as a priority for pharmacist review whereas, in group B, patient priority was determined by the predicted probability of ME, provided by the training model. This simulation was repeated 1,000 times within five scenarios, within each the pharmacy intervention coverage was set to 10%, 30%, 50%, 70% or 90% of patients.
Analyses were conducted with R 2.15.2 (R Development Core Team. R Foundation for Statistical Computing, Vienna, Austria).

Fig 1. Simulated randomized controlled trial comparing two decision-making strategies for intervention to reduce medication errors.
The analysis was reiterated 1,000 times within 5 scenarios, within each the intervention coverage k was fixed at 10%, 30%, 50%, 70% and 90%.
During the study period, pharmacists identified 609 MEs. Of these, 475 were considered as events of interest due to their association with harm or potential harm (Table 2). Most occurred on the first day of hospitalization (Fig 2), with a majority of unintentional discrepancies resulting from inaccurate information about current medications, which were corrected by physician after notification. Three MEs actually led to adverse drug events, due to a failure of correction at time ( Table 2). During their stay, 365 (25.9%) of patients included experienced at least one clinically significant ME. Occurrence of ME was associated with 5 ATC classes: antithrombotic agents (B01), antibacterial agents for systemic use (J01), psycholeptics (N05), blood substitutes and perfusion solutions (B05) and analgesics (N02).
RCT simulations (n = 5,000) demonstrated the effect of the PRISMOR model on the identification and interception of MEs when compared to identification based on age (Fig 3). In all clinical coverage scenarios, clinical pharmacists were more likely to identify MEs when the PRISMOR model was used to target high-risk patients. This effect was strongest in the 10% coverage scenario, with 17.5% mean improvement over age-based selection (statistical significance P <0.05 in 83.2% of simulations) and an absolute probability of ME interception of 50.6%. This corresponds to a number needed to treat of 6 to avoid at least one harmful ME and, when compared to the absence of clinical pharmacist evaluation, a number needed to

Respiratory depression after Midazolam overdose (requiring Flumazenil)
Bleeding and hematoma secondary to non-adaptation of antivitamin K therapy Cardiogenic pulmonary edema due to the unintentional omission of diuretic and beta-blocker medications (patient with chronic heart failure) treat of 2. Conversely, the simulation demonstrated that by using our prediction model, clinicians would have missed fewer MEs, with an effect that increased with clinical coverage. Similar analysis was conducted to compare the decision-making supported by our model versus the one based on the number of medications (S3 Fig). Although the enhancement in intercepting potential adverse drug events is more limited (mean improvement of 5.4% in the 10% coverage scenario), using our model as decision-support reduced the risk to miss clinical significant MEs in absence of intervention (absolute risk difference of -9.9%, 1 ME avoided every 10 decisions in the 90% coverage scenario).

Discussion
The incidence [3,4,6,13,33,34] and timing [3,4,34,35] of MEs in our sample is consistent with that seen in previous studies. This literature emphasizes the importance of implementing a strategy to prevent MEs early in hospitalization, ideally prior to admission medication prescription. Using predictors identified within our sample, we developed and tested a model to inform such a strategy. As expected, age and number of medications were significant predictors of MEs. Additionally, the availability of accurate information about current medications and doses prior to prescription substantially decreased the risk of MEs. In particular, a best possible medication history could have been obtained by other clinicians. As it is usually the same as that listed in the discharge summary from any recent hospitalization, hospitalization within the previous 30 days was associated with lower risk of MEs. To inform clinical triage of patients, we evaluated the potential of broadly-defined MEs to cause harm. We chose to develop a generalizable predictive model that could be implemented in everyday practice to intercept avoidable outcomes. This model was reported using international standards [22,23] and assessed for robustness using resampling methods.
The randomized experiments showed that targeting high-risk patients with our model had the potential to substantially improve the efficiency of pharmaceutical intervention. Implementing our model in practice could enhance the identification and interception of potentially adverse drug events, a result that was consistent across levels of clinical pharmacist coverage. In all scenarios, results suggest that our model may assist clinicians to select patients who are more likely to benefit from pharmaceutical intervention. In situations of limited pharmacy coverage, our model would theoretically enable clinicians to intercept a greater number of clinically significant MEs by intervening whereas, in high coverage scenarios, fewer MEs would be missed in those patients not undergoing pharmacy review. This trend is not surprising, since greater clinical pharmacy coverage would result in a greater proportion of patients who receive intervention, including higher-risk subjects. When complete coverage is provided, targeting high-risk patients would not improve ME interception, as pharmacists already review all subjects. In contrast, focusing interventions on high-risk patients might be essential in the setting of resource limitations. Using our model-based strategy, a new clinical pharmacy service covering 10% of inpatients would avoid a potential adverse drug event every two interventions. A small number of studies have previously reported statistical models developed to predict medication errors [17][18][19][20][21]. Though apparent discriminations were comparable to ours [17][18][19][20][21], our study improves on these through its transparent reporting of calibration and optimism [22,23]. More importantly, we assessed the potential impact of our tool on clinical practice. In addition, because we did not restrict our study to a specific age range or field of medicine [17,19,20], it can be generalized to both medical and surgical inpatient settings.
Previous studies have restricted their studies to actual adverse events [17,19,20]; we chose to be more inclusive, including both actual and potential adverse events. As our study was based on observational (i.e., "real-life") data, medication reviews were being conducted by clinical pharmacists as part of routine clinical practice. As such, when they were identified, mitigating action was taken to prevent medication errors which could cause harm. Had we not taken them into account, our model would have underestimated the incidence of the outcome. Several limitations to our study need to be considered. Our model does not take into account biological markers [19] or diagnostic category [21]. Given that data consistently demonstrates that the highest risk of potential adverse drug events is on the first day admitted to hospital, we chose to exclude these potential predictors, as we felt that the most clinically valuable model would be limited to predictors that are readily available at the time of admission to hospital. Though patient comorbidities are potential predictors, the size of our derivation set limited the inclusion of these variables in the model. Having added them as independent predictors would almost certainly have resulted in over-fitting, a recognized problem to which models derived from small cohorts are prone [36][37][38]. Summarizing indices, such as Charlson's index or Elixhauser's index, would have been inappropriate given that they were developed to predict mortality, an outcome which is very different from ours [39,40]. Moreover, recent studies have suggested that these indices have relatively poor discrimination and calibration in their prediction of nonfatal outcomes [41,42]. Diagnoses could have been grouped in other ways; we felt it was impractical with regard to our model, however. Indeed, comorbidities were recorded in only 723 (51.3%) patient discharge summaries, which raised a real concern for the usability of a model including these predictors. In case of no recorded information (i.e. no input value), the patient's probability of outcome would no longer be computable using the model formula.
In spite of including an appropriate number of variables for the number of outcomes [24,25], we had to shrink initial estimates to address over-fitting [38]. That being said, bootstrapcorrected models developed based on values of 20 events per variable tend to demonstrate similar performance in validation studies [24]. Finally, our impact analysis is limited by the assumption that reducing potential adverse events would result in reduction of actual outcomes. Our preliminary results must therefore be confirmed on an independent set of patients, ideally from other hospital centers, and the usability of our predictive tool must be evaluated through a clinical impact study.
In conclusion, we developed and internally validated a model to predict potential adverse drug events as a strategy to improve pharmacist human resource allocation and subsequent patient safety. Once external validation and evaluation of concrete clinical outcomes takes place, its incorporation into clinical practice could potentially allow institutions to identify atrisk patients at their time of arrival to hospital, thus allowing for efficient, patient-specific allocation of clinical pharmacy services.