The Effects of Clinical Decision Support Systems on Medication Safety: An Overview

Background The clinical decision support system(CDSS) has potential to improving medication safety. However, the effects of the intervention were conflicting and uncertain. Meanwhile, the reporting and methodological quality of this field were unknown. Objective The aim of this overview is to evaluate the effects of CDSS on medication safety and to examine the methodological and reporting quality. Methods PubMed, Embase and Cochrane Library were searched to August 2015. Systematic reviews (SRs) investigating the effects of CDSS on medication safety were included. Outcomes were determined in advance and assessed separately for process of care and patient outcomes. The methodological quality was assessed by Assessment of Multiple Systematic Reviews (AMSTAR) and the reporting quality was examined by Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA). Results Twenty systematic reviews, consisting of 237 unique randomized controlled trials(RCTs) and 176 non-RCTs were included. Evidence that CDSS significantly impacted process of care was found in 108 out of 143 unique studies of the 16 SRs examining this effect (75%). Only 18 out of 90 unique studies of the 13 SRs reported significantly evidence that CDSS positively impacted patient outcomes (20%). Ratings for the overall scores of AMSTAR resulted in a mean score of 8.3 with a range of scores from 7.5 to 10.5. The reporting quality was varied. Some contents were particularly strong. However, some contents were poor. Conclusions CDSS reduces medication error by obviously improving process of care and inconsistently improving patient outcomes. Larger samples and longer-term studies are required to ensure more reliable evidence base on the effects of CDSS on patient outcomes. The methodological and reporting quality were varied and some realms need to be improved.


Introduction
Various interventions using information technology (IT) have been developed to improve medication safety, and IT-based interventions such as clinical decision support system(CDSS) plays an integral role in this field [1]. According to the definition of the US Food and Drug Administration, medication safety means risk management, medication errors, and surveillance for adverse drug reactions [2]. Medication errors are recognized as the single most preventable cause of patient harm, and their reduction is of increasing importance [1]. CDSS used to promote medication safety by facilitating evidence-informed medication use [3], reducing the incidence of harmful medication errors [4], and improving healthcare system efficiency [5]. A CDSS has been defined as a computerized system that uses case-based reasoning to assist clinicians in assessing disease status, in making a diagnosis, in selecting appropriate therapy or in making other clinical decisions [6].Characteristics of individual patients are matched to a computerized knowledge base, and software algorithms generate patient-specific information in the form of assessments or recommendations [7]. CDSS can improve medication safety and reduce medication-related expenditures because it encompasses a wide range of computerized tools directed at improving patient care, including computerized reminders and advice regarding drug selection, dosage, interactions, allergies, and the need for subsequent orders [8].
Medication errors and adverse drug events (ADEs) are common costly and clinical important problems [8]. More than half a million patients are injured or die each year in hospital from adverse drug events, which may cost up to USD 5.6 million annually per hospital in America [9]. Medication errors could occur at any stage of the medication management process, including prescription, transcription, preparation and administration [10]. A review identifying that prescribing errors occur in up to 11% of all prescriptions [11]. Two inpatient studies found that medication errors occurred at rates of more than 5% and nearly half of all medication errors occurred at the stage of drug ordering [8,12]. Analysis of medication error suggests that prevention strategies targeting systems rather than individuals are most effective in reducing errors [13]. CDSS has been widely promoted as the most promising approaches that target the ordering stage of medications, where most medication errors and preventable ADEs occur [8].
Recently, several systematic reviews (SRs) have summarized the effects of CDSS on practitioner performance and patient outcome [7,[14][15][16][17] and the evidence has been synthesized into an overview [18].The overview demonstrated modest benefits on practitioner performance and patient outcomes [10]. Meanwhile, concerns have arisen regarding the impact of CDSS on medication safety and the evidence has been synthesized into several SRs, though findings from various SRs are conflicting. For example, a CDSS team conducted three SRs to evaluate the impact of CDSS on drug prescribing and management [4], medication dosing assistants [19] and therapeutic drug monitoring and dosing [20]; Two SRs summarized the effects of CDSS on ADEs [21] and medication safety [8]. Given the growing awareness of the importance of CDSS on medication safety and the numerous SRs on this topic, it is surprising that the overviews of the effects of CDSS in this field are lacking. As reported by Brown2014 [22], overview brings SRs together into one coherent document and serve as a user-friendly "digest" by evaluating and synthetizing current evidence which can be used by clinicians and policy makers in making decisions about optimal treatment, so for the first time, we set out to conduct an overview to examine CDSS interventions on medication safety in health care settings. The purpose of our study is: to evaluate the effects of CDSS on medication safety; to examine the methodological and reporting quality and to highlight areas where more research is needed.

Inclusion criteria
Types of participants. The participants should be health care professionals such as: physicians, nurses, pharmacists, and other practitioners with responsibility for patient care. We excluded practitioners who are indirectly involved in patient care at ancillary clinical departments such as radiology, pathology departments.
Types of interventions. The CDSS combining clinical knowledge with patient characteristics and can provide either basic (e.g., drug-allergy checking) or advanced (e.g., drug dosing support for renal insufficiency) guidance to the participants.
Outcome measures. Within the SRs, at least one outcome relating to medication safety should be measured, such as: medication error, adverse drug events, prescribing errors, dosing errors and medication/drug related outcomes.
Selection of studies. Only SRs were considered for inclusion. To assess whether a study was a SR, we used the checklist of Cochrane Handbook for Systematic Reviews of Interventions [23].Two reviewers independently scanned titles and abstracts to exclude obviously irrelevant studies and potentially relevant studies were investigated as full text. Disagreements were resolved by discussion with a third reviewer.

Search strategy
To identify relevant SRs, we searched the PubMed, Embase and Cochrane Library up to Aug. 2015 and elected English-only publications. Multiple keywords and Medical Subject Headings terms for CDSS were used. The search terms were customized for different databases (see S1 File: Search Strategy). No limitations were made on the outcomes. Additionally, we also handsearched reference lists and relevant reviews to identify SRs.

Data extraction and management
Two reviewers independently extracted the data. We developed a data extraction sheet, pilottested it on 5 randomly-selected included studies, and refined it accordingly. The following data were extracted: general information, study design, study population, intervention and main study outcomes. Separate summaries were made for the outcomes. Within these summaries, the outcomes were determined in advance and assessed separately for process of care and patient outcomes (Table 1). A process of care outcome represents quality of care, such as practitioner performance and clinical intermediate outcome. A patient outcome is directly measured patient's health and always be endpoint outcome, such as the number of symptomatic hypoglycaemic episodes, death or bleeding [20]. Results on process of care and patient outcomes were aggregated by grading them on the strength of evidence for improvement. The evidence strength is based on the included randomized controlled trials (RCTs) in the SRs [18]: strong evidence: results based on RCTs and effect in 50% of more of the studies; limited evidence: results based on RCTs and effect in 40-50% of the studies; results based on both RCTs and non-randomized studies and effect in 50% of the studies.
insufficient evidence: results based on non-randomized studies or effects in less than 40% of the studies.

Quality assessment
We used two criteria sets for evaluating SRs because they have different foci. In the first stage, Assessment of Multiple Systematic Reviews (AMSTAR) was used to evaluate the methodological quality. AMSTAR is an 11-item measurement tool for the assessment of multiple systematic reviews that have good reliability and validity [24]. It was scored as "Yes", "Partially yes", "No", "Can't answer" or "Not applicable". A criterion was defined as "Partially yes" if it was half met. For example, the fifth criterion, "A list of included and excluded studies should be provided", was scored as "Partially yes" if either the included or the excluded studies were listed [25]. Each item is given a score of 1 if the specific criterion is met, a score of 0.5 if the criterion is partially met, or a score of 0 if the criterion is not met, is unclear, or is not applicable [24]. We summarized the final scores at three levels based on the criterion used by Beverley and Jeremy: 9 to 11 is high quality; 5 to 8 is medium quality and 0 to 4 is low quality [26].
In the second stage, the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) was used to evaluate the reporting quality. The checklist consists of 27 items and focuses on ways in which authors can ensure the transparent and complete reporting of systematic reviews and meta-analyses [27]. To indicate the degree of compliance, each checklist item was assigned one of four responses: 'Yes' for total compliance; 'partial' for partial compliance; 'No' for noncompliance and 'NA' for not applicable [28].

Data analysis
We planned to conduct a meta-analysis if the treatment outcomes considered were comparable. Continuous measures were reported as mean differences and standard deviations or as standardized mean differences. Dichotomous outcomes were reported as odds ratio or rate ratio. Where studies provided sufficient data for meta-analysis, the Cochrane Collaboration's RevMan software was used to perform a meta-analysis [23]. In studies in which outcome data were not suitable for meta-analysis, the data were described narratively.

Search results
The initial search provided a pool of 9,002 SRs published in English. After eliminating based on duplicates, title and abstract, 42 SRs remained for full texts sifting and 29 finally proved to fulfil the inclusion criteria. Then two reviewers independently assessed the 29 SRs using the checklist of handbook to determine whether a publication was a SR ( Table 2, Table 3). Finally, 20 SRs were included (Fig 1).

General study characteristics
The 20 SRs were published between 1994 and 2014 and conducted in diverse and mixed study settings, such as: inpatient settings[16 17 29-31]

Quality assessment
Methodological quality. Ratings for the overall scores of AMSTAR resulted in a mean score of 8.3 with a range of scores from 7.5 to 10.5. Based on the criterion used by Beverley and Jeremy [26]: four studies is high quality [20 30 36 38]; Sixteen is medium quality [4 10 14-17 19 29 31-35 37 39 40] and 0 to 4 no study is low quality (Table 4). Meanwhile, the 20 SRs varied in the tools used to assess the methodological quality of their including studies: 10-point scale was used by six studies[4 16 19 20 32 34]; Two SRs conducted the processes based on the criteria developed by the Cochrane Effective Practice and Organization of Care (EPOC) Group [30 39]; The Cochrane Collaboration's tool for assessing risk of bias was used by Bennett2003 [33] and Alldred2013 [38] and one study used the Downs and Black tool [37] (Table 5).
Reporting quality. Overall, none of the included 20 SRs fulfilled all 27 items of PRISMA. The "rationale for the review, study selection, definition of data items, synthesis of results, presentation of study characteristics, summary of evidence and conclusions" were well described across all SRs; The item "Identify the report as a systematic review, meta-analysis, or both" were adequately reported in 75% of the SRs. Search strategy for at least one major database was reported by 15 SRs; "Risk of bias of individual studies" were evaluated completely in 17 SRs. Total compliance with PRISMA items was less than 30.0% in structured summary (item 2, 5%), objectives (item 4, 15%), data collection process (item 10, 5%), summary measures (item 13, 25%), results of individual studies (item 20, 25%); No study described additional analyses in the methods, such as meta-regression or sub-group analyses ( Table 6).

Synthesis of evidence
We planned to conduct a meta-analysis if the treatment outcomes considered were comparable. However, the variability in methods and the ways outcomes measured and presented made the generation of pooled estimates impossible. We presented the results in narrative and tabular form. The 20 SRs included 629 references, 413 of which represented unique studies, with 237 RCTs. Of these unique 237 RCTs, 178 RCTs studied process of care or patient Note: A publication was determined as a SR if meets the items (1,2,4,5,6,7,8,9).  (Table 4). Thirteen out of the 20 SRs studied the impact of CDSS on patient outcomes. Evidence that CDSS significantly impacted patient outcomes was found in 18 out of 90 unique studies of the 13 SRs that examined this effect (20%).Only two of the 13 SRs found strong evidence that CDSS impacted patient outcomes: computer support for determining drug dose [36]and computerized medication dosing assistants [19] Three found limited evidence [16 17 29] and the remaining 8 SRs found insufficient evidence: computerized drug-lab alerts [31], computerized drug dose [15 20 30], computerized clinical decision support systems for drug prescribing[4 32 37 38] (Table 4).

Discussion
Currently, an overview focuses on the effects of CDSS on practitioner performance and patient outcome has been published [18] and AMSTAR was used to evaluated the quality of the   included SRs. To the best of our knowledge, this is the first overview focus on examining the effects of CDSS on medication error which is more important and direct on patient safety, specifically the drug-related injuries [21]. It is, furthermore, the first overview assessed the quality of SRs on the effectiveness of CDSS interventions using AMSTAR and PRISMA. AMSTAR focuses on methodological quality, while PRISMA incorporates items deemed important for the transparent reporting of a SR. Both the methodological and reporting quality are important for judging the overall strength of evidence on given research questions and are considered in Clinical Decision Support Systems and Medication Safety assessment of the quality of a research design, implementation and reporting rather than the intervention's true effect in the process of research [41]. The study by Spyridon demonstrated that the evaluation of the reporting quality of published SRs is very useful, as it is directly related to the study's methodology and conclusions [25]. Our recent initial search yielded 9,002 SRs, of which only 20 SRs met inclusion criteria. We planned to examine the effects of CDSS on medication safety, four studies directly targeting medication error [10 39]or adverse drug events [37 38]. The majority of our studies (16 studies) focused on the medication/drug related outcome (i.e. INR time in therapeutic range, serum concentration, time to achieve stabilization, physiological parameter change, drug dose/prescribing). This situation was true in our study, but changes in medication/drug related outcomes are also important. A study by Lainer2013 showed that the CDSS interventions consistently improve medication/drug related outcome that may provide indirect evidence for the improvement of medication safety [1]. So the influence of CDSS on medication safety may be direct, or indirect. Although a surrogate measure, changes in accordance with best practice guidelines and underpinned by evidence from high-quality would be expected to deliver improved medication safety, even if the evidence was not captured in these studies [17].

Quality of Included Studies
Methodological quality. Our study included 20 SRs over a 20-year span and 16 of the studies demonstrated medium quality, only four studies found high quality. No increase in SR quality, with regard to fulfillment of the AMSTAR criteria, was visible over the years, a similar 1.Bayoumi2014 [31]  finding was convinced by Monique2011 [18] and Seo2012 [41], but Dereck [15]and Amit [16] indicated that the number and methodological quality of trials have improved over time.
Three studies [4 19 20] used a "vote-counting" method to establish general conclusions. "Votecounting" is a method that the overall results of the trials were reported by taking the number of trials with statistically significant results and dividing them by the total number of trials [19]. This method does not consider the magnitude of effects and may have underestimated the overall efficacy [20]. Hedge concluded that aggregated outcome relied upon vote counting increasing the risk of type 2 (false negative) error and formal assessment for publication bias using funnel plots was not possible with the vote-counting technique [4]. Meanwhile, a common bias, as reported by Gillaizeau2013 [30] and Walton1999 [36], of the included studies was that when studies were randomized by participant, the same healthcare professional may have given treatment both to intervention and control groups. It is possible that the CDSS influenced the treatment of the control groups. Reporting quality. The reporting quality of 20 SRs varied significantly. Some contents were particularly strong, such as the reporting of search of studies, selection of articles, and  Clinical Decision Support Systems and Medication Safety synthesis of results. However, there are also some realms need to be improved. One of the issues may be the insufficient reporting of publication bias. Sahota2011 reported that publication bias exists in this field and the presence of publication bias may be a significant confounding factor when authors are trying to aggregate results [19]. This problem results in studies with small sample sizes and non-significant effects being left out of the aggregated pool [42]. The second significant issue is the lack of quantitative analysis, such as meta-analysis, sensitivity or subgroup analyses, meta-regression. One of the possible reason may be the bulk of the qualitative literature [42]. Only 3 of the 20 SRs attempted to compute effect sizes, and all of those were in the domain of medication dose. The majority of included SRs (85%) used narrative descriptions or a "vote-counting" method to establish general conclusions. Another possible factor is the heterogeneity. In our study, the variation in the study design, clinical setting, study population, software specifications, and CDSS workflow integration led the results were not consistent among the 20 SRs, even when evaluating the same drug or the same disease [20]. The reporting flaws would affect the method quality, the integrity and accuracy of researches, so the quality of reporting still needs further improvement [28].

Synthesis of the systematic reviews results
It is clear from our synthesis that most SRs (95%) measured process of care outcomes and the results indicated that the application of CDSS would improve the outcomes. Similar findings were reported by a number of published studies [4 19 20 30 31 34 36]. In addition, our study found that CDSS that alert or reminder participants' adherence to medication were most likely to impact the medication errors and patient care by improving the process of care outcomes. The degree to which participants adherence to the computerized medication advice will vary depending on the specificity level of computer-generated advice. As reported by Monique 2011 [18], the specificity level of computer-generated advice is known to highly influence the chance that physicians adhere to the advice, with low specificity resulting in computer-advice fatigue and in situations where physicians ignore the advice. Simultaneously, consistent with previous studies[17 29 32], the results of this overview suggests that CDSS has a positive effect in changing prescribing outcomes. Alldred2013 illustrated that CDSS improved the prescribing by discontinuing inappropriate medication; commencing beneficial medicines; and ensuring appropriate monitoring of long-term conditions and medicines [38]. Consequently, this process may lead to a reduction in medication error and the improvement of medication safety. However, while 65% of the included studies measured a patient outcome, only a small proportion demonstrated positive findings. Regarding lack of positive findings on patient outcomes, the related study by Monique 2011 showed that it was likely attributed to the small sample sizes in the original studies that consequently were underpowered [18]. Furthermore, follow-up periods in some studies were always too short to assess long-term differences on patient outcomes associated with the computerized interventions [43]. Outcomes such as hospital length of stay, death and bleeding complications will be influenced by factors other than better medication management [17]. Our study also found that CDSS for medication dosing assistants were most likely to impact the medication errors and patient care by improving the related patient outcome. An explanation for the findings may be that computers help doctors to tailor drug doses more accurately to individual patients, bringing benefits for patients and reducing the time that they spent in hospital [36]. This was also elucidated in the study by Kuperman who found that computer-generated orders are more legible than those written by hand [44]. A knowledge-based CDSS can assure that the order is safe and compliant with guidelines [45] because it introduces automation at the time of ordering, a key process in health care [44].

Strengths and Limitations
Our study has several strengths. We innovatively assessed the quality of SRs on the effectiveness of CDSS interventions using AMSTAR and PRISMA. Secondly, the literature search was comprehensive and detailed inclusion and exclusion criteria were developed to ensure transparency and reproducibility in the judgments. Thirdly, we manually searched the reference lists of the selected SRs to identify SRs that we could have missed in our literature search. In addition, the utilization of two independent reviewers for preselection of SRs, the assessment of SRs' quality and the final data extraction would be a great help to avoid mistakes and subjective judgments.
Meanwhile, our study also has some limitations. Firstly, we were unable to use meta-analysis to pool effect sizes, given the substantial differences among the outcomes evaluated. Secondly, we defined improvement in 50% or more of the RCTs as strong evidence, in 40-50% of the RCTs as limited evidence, and in less than 40% of the RCTs or non-randomized studies as insufficient evidence respectively. The methods, combined with the strict inclusion criteria, may have underestimated the effects of CDSS intervention. Thirdly, we tried to interpret the overall effects of CDSS by aggregating all included studies, the results of high quality studies are given the same weight as low-quality studies. Fourthly, there may be some overlaps of our included studies which may have led to an overestimation the effects of CDSS. We therefore analyzed the number of unique studies from the total number included in all SRs and provided overall estimates of the evidence that CDSS significantly impacted process of care outcomes and patient outcomes [18]. Lastly, we classified the methodological quality at three levels based on the final scores of each studies, but scoring systems are controversial [46].

Future Directions
First, larger samples and longer-term studies are required to ensure more reliable evidence base on the effects of CDSS on medication error. Decision makers should balance the effects of CDSS against its burden on costs and workflow. However, the increasing number and quality of CDSS trials in the past few years and the rapid assimilation of technological information into clinical settings bode well for the future of improving the effectiveness and efficiency of clinical care [15].

Conclusions
CDSS reduces medication error by obviously improving process of care, but inconsistently improving patient outcomes. There is significant evidence that CDSS for medication alerts or reminders can positively impact process of care outcomes and CDSS related to medication dosing assistants were most likely to impact patient outcomes. The methodological and reporting quality of the included studies were varied. The changes of process care may be the most immediate effect of CDSS, nevertheless, like other health care interventions, CDSS must demonstrate benefit on patient outcomes related to drug efficacy and safety before extensively recommended into clinical care, particularly given the cost of implementing and maintaining computerization.