Adherence to guidelines pertaining to stroke prevention in patients with atrial fibrillation is poor. Decision support systems have shown promise in increasing guideline adherence.
To improve guideline adherence with a non-obtrusive clinical decision support system integrated in the workflow. Secondly, we seek to capture reasons for guideline non-adherence.
A decision support system was developed that implemented properties positively associated with effectiveness: real-time, non-interruptive and based on data from electronic health records. Recommendations were based on the Dutch general practitioners guideline for atrial fibrillation that uses the CHA2DS2-VAsc for stroke risk stratification. Usage data and responses to the recommendations were logged. Effectiveness was measured as adherence to the guideline. We used a chi square to test for group differences and a mixed effects model to correct for clustering and baseline adherence.
Our analyses included 781 patients. Usage of the system was low (5%) and declined over time. In total, 76 notifications received a response: 58% dismissal and 42% acceptance. At the end of the study, both groups had improved, by 8% and 5% respectively. There was no statistically significant difference between groups (Control: 50%, Intervention: 55% P = 0.23). Clustered analysis revealed similar results. Only one usable reasons for non-adherence was captured.
Citation: Arts DL, Abu-Hanna A, Medlock SK, van Weert HCPM (2017) Effectiveness and usage of a decision support system to improve stroke prevention in general practice: A cluster randomized controlled trial. PLoS ONE 12(2): e0170974. https://doi.org/10.1371/journal.pone.0170974
Editor: Terence J. Quinn, University of Glasgow, UNITED KINGDOM
Received: June 26, 2016; Accepted: December 20, 2016; Published: February 28, 2017
Copyright: © 2017 Arts et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All data that were used for the primary outcomes (Group differences post study) are included in the supporting files.
Funding: DA was Funded by Boehringer Ingelheim with an unrestricted grant through the ZORRO foundation (no website available).
Competing interests: The authors have declared that no competing interests exist.
The burden of atrial fibrillation (AF) increases year by year in societies with aging populations, such as Western Europe and the United States [1–3]. This burden is largely due to the five-fold increased risk for stroke that is associated with AF . There is ample evidence that the efficacy of oral anticoagulation (OAC) in stroke prevention is excellent [5–7]. Nonetheless overtreatment should be avoided, as this can result in hemorrhaging, mainly intracerebral and gastrointestinal. Therefore, treatment strategies for stroke prevention have been optimized over recent years, and clinical practice guidelines describing these treatment strategies are publicly available . However, adherence to these guidelines remains poor [9–11]. This non-adherence is mainly related to underuse of OAC in patients with medium to high stroke risk. Two important reasons for OAC underuse are 1) the complexity of the decision rules used and 2) physicians’ concerns with the bleeding risk associated with OAC. However, the benefit of stroke prevention greatly outweighs the risk of bleeding due to OAC, and OAC should therefore not be withheld when indicated [6, 12].
Studies have shown that clinical decision support systems (CDSSs) have the potential to improve guideline adherence [13–15], as these systems can make physicians aware of the guidelines when they lack the appropriate knowledge and eliminate inappropriate use of clinical decision rules. The overall effectiveness of CDSS on mortality has not been established, but a recent review did find a moderate improvement (RR = 0.82; 95% CI = [0.68, 0.99]) in morbidity outcomes . CDSSs therefore present a possible solution for the issue of guideline non-adherence in stroke prevention among AF patients.
CDSS effectiveness varies by study and domain, and several studies have tried to determine what factors predict success or failure,[17, 18]  although it is hard to pinpoint these .We have developed a CDSS that implements properties that appeared to have a positive effect on CDSS effectiveness, most notably: “implementing decision support as part of the workflow”, “during the time of decision making” and “optimized human-computer interface”[18, 21]. Too many alerts will tend to result in all alerts being ignored, a phenomenon known as “alert fatigue” . Given the possible adverse effects of “alert fatigue” and interruption , we considered the optimal interface to be one which minimized these effects.
In this study we investigate the effectiveness of our CDSS as measured by general practitioners’ adherence to the Dutch GP guideline for patients with atrial fibrillation. A secondary objective of this study is to gain insight into reasons for deviations from the guideline and to adjudicate these reasons with peers.
We hypothesized that the implementation of a non-obtrusive CDSS, integrated in the GP’s workflow, will increase guideline adherence. Secondly, we expect GPs will often have valid reasons for guideline non-adherence.
This trial is registered with the Dutch Trial Register under registration number 3570. http://www.trialregister.nl/trialreg/admin/rctview.asp?TC=3570
Materials and methods
The trial protocol for this study, which was dubbed the “Expert-AF” trial, has been published elsewhere . Below, we will provide a short overview of the most important aspects of the methodology used.
A cluster-randomized controlled design was used. Randomization was done at the GP practice level to reduce contamination bias. GPs were allocated into one of the following three groups:
- Received no messages (control group)
- Received messages that could be declined without documenting justification (intervention 1)
- Received messages that could only be declined if justification was documented (intervention 2)
The planned allocation ratio was 2:1:1 respectively (Fig 1). Randomization was performed in the statistical environment R  using the ‘sample’ function to create a random sequence from the list of GP practices by DA. GPs were aware that they were allocated to a variant of the system, but were blinded as to their allocation and how the variants differed.
The medical ethics committee of the AMC deemed this study exempt from ethical approval because the intervention is a way of providing valid medical information to GPs and does not change medical treatment or use invasive procedures. Furthermore, only anonymized patient data was used for analysis. All GPs consented to participation in the trial.
We planned to include GP practices in the Netherlands that used the electronic health record (EHR) for which we developed a decision support plugin (an extra piece of software providing feedback that is not an integral part of the main system). All patients with AF (both incident and pre-existing) who had been in contact with their GPs during the length of the trial were evaluated. The GPs use ICPC-coded diagnoses for both care and reimbursement, thus we considered a patient to have AF if a coded diagnosis (‘K78’) for AF was documented . This method was chosen to exclude care avoiders (patients who actively avoid contact with their GP despite their condition) that could bias results.
We developed a CDSS for a single EHR system. This CDSS was automatically activated using event-based triggers, and responded to actions of the GP in real time . Our study ran concurrently with the ICOVE trial that contained 15 decision rules per randomization group . For each decision rule that was not satisfied, notifications were shown in a floating notification window. The window could contain multiple notifications if multiple rules applied to the patient. Notifications contained a short (1- to 3-word) title (Fig 2). By clicking on a notification, a popup appeared that contained: background information, an actionable recommendation and two response buttons that allowed the GP to either accept or decline the advice (Fig 3). Accepting or declining was logged by the plugin but had no effect on the EHR. Due to the fact that there were other notifications present in the system, GPs were likely unaware of their allocation. The Expert-AF notification was positioned at the top of the notification window. For intervention group 2 there was a small free text box that required the GP to enter a reason for deviating from the recommendation. The plugin was only active for GPs.
Training and support
GPs were trained using three methods: a live demonstration, a PDF manual and a video demonstration. One live demonstration per GP practice was provided. Additionally, a detailed manual of the system was distributed in PDF format among all participants. Lastly, GPs were made aware per email of a 10-minute video demonstration of the system available online. Support was available during office hours via email and phone.
The computerized decision rule used in the Expert-AF system was based on the Revised Dutch GP guideline for atrial fibrillation, authored by the Dutch College of General Practitioners . This guideline uses the CHA2DS2-VASc score, which consists of 7 variables to predict risk of stroke . Patients can score from 0 to 9 points. All patients with a score >1 have an indication for OAC according to the current GP guideline. These same recommendations are made in the 2012 European Society of Cardiology guideline on atrial fibrillation . The HAS-BLED bleeding risk score  was not yet formally introduced in the Dutch GP guideline for AF, and thus not included in the decision rule. However previous large bleeds, untreated hypertension, kidney failure and thrombotic disorders were included as contra-indications for VKA.
Baseline data collection
Baseline measurements were performed to determine the proportion of patients with AF who were treated in accordance with the guideline. Guideline adherence was determined by automatically evaluating the computerized decision rule on all patients in an anonymized copy of the GP’s EHR databases.
Data collection during and after the study
Every action that related to the plugin was stored. Anonymized patient files were saved to confirm the system functioned properly and to determine what triggered the notifications. Guideline adherence by the participating GPs was evaluated using an anonymized copy of the GP EHR databases, which was made after the trial was completed. The trial was conducted from October 1st, 2013 to September 1st, 2014, resulting in 240 active workdays.
We defined the primary outcome as the difference in the proportion of patients with AF treated in accordance with the guideline between the intervention and control groups.
A secondary outcome of the study was the reasons GPs provided for deviating from the guideline and the manner in which they responded to required justification (intervention 2).
A power analysis accounting for clustering resulted in a required sample size of 500 patients for an effective sample size of 300 .
A chi-square test was used to compare between-group differences at baseline and differences in adherence rates post-study. Additionally, a mixed effects logistic regression model was used with the GP practice as random effect to correct for clustering at the practice level and baseline adherence.
Intervention 2 was an extension of intervention 1; the only difference between the two was the addition of a free text box that required a reason for ignoring recommendation. For the all quantitative analyses and plots, these groups were merged to increase statistical power. Analyses were performed using R .
For the baseline analyses 731 patients with AF were included (Table 1). In total, 781 patients with AF were in contact with the GP practice during the trial and included in the analysis. These patients were seen by 39 GPs in 19 GP clusters.
Table 2 contains usage statistics. The notification was shown 3848 times, and clicked 188 times to open the pop-up with the information and advice (5%). Usage over time declined, with the most activity in the first month, and a steady decline of usage over the following 8 months (Fig 4). In total, GPs actively responded to 76 notifications by either dismissing them (N = 44, 58%) or indicating they would follow the advice (N = 32, 42%). Some GPs did not click on the notification at all, while others clicked nearly all of them (Fig 5).
Blue dots indicate clicked notifications, red dots indicate responses to recommendations (clicking accept or decline), orange dots indicate the number of times the GP hovered over the notification window with their mouse regardless of whether the notification was opened.
Average usage (responses / notification) per general practitioner.
In the second intervention group, GPs clicked on twenty-two notifications and accepted two. Five notifications were declined. Four reasons for declining were related to the plugin not being able to detect Warfarin as active medication or not related to the guideline, thus only one reason for guideline non-adherence was presented, which was insufficient for further analysis.
Guideline adherence, defined as the percentage of patients treated in accordance with the guideline, differed between the control and intervention groups at baseline (Control: 42%, Intervention: 50% chisq P = 0.04). At the end of the study, both groups had improved, by 8% and 5% respectively. There was no statistically significant difference between groups (Control: 50%, Intervention: 55% chisq P = 0.23) (Table 1). Clustered analysis revealed similar results (P = 0.21); correcting for baseline adherence did not alter these results (P = 0.25).
We evaluated the effectiveness of a real-time CDSS in general practice, specifically for increasing adherence to antithrombotic guidelines for atrial fibrillation. We did not find a significant difference in guideline adherence between the intervention and control groups.
Strengths and limitations
Strengths of this study included the decision support system feedback characteristics that built on evidence pertaining to effectiveness of decision support systems [17, 18]. We implemented a real-time CDSS that supported the GP during their contacts with patients by showing alerts in a non-obtrusive way. Our trial implemented many of the features that we should expect from decision support in the future. We reduced contamination by using clustered randomization. Lastly we studied a topic (prevention of stroke), that is considered highly relevant by the participating GPs.
However, this study was hampered in several ways. First, the vendor of the GP system that implemented the CDSS plugin stopped supporting it, thereby preventing us from enrolling more GPs in our trial. As initially, more GPs were included in the intervention arm, this lead to asymmetric randomization groups, reducing potential power of the study. Nonetheless the calculated sample size (n = 500) was reached, with more than 250 inclusions in each group post study. We would have preferred to develop a generic system that worked on any platform but were due to funding and organizational restrictions we were limited to working with a single vendor.
Lastly, the new guideline for the management of AF by the Dutch College of General Practitioners was introduced during the start of the trial. Combined with the fact that all GPs were trained in use of the system, it is likely there was a teaching effect, or at least a substantial increase in awareness of OAC under prescribing in patients with AF. This might have resulted in increased adherence in both groups. We selected guideline adherence as primary outcome, but perfect guideline adherence is undesirable as most guidelines do not account for complex patients with multi-morbidity and physicians often have good reasons for deviating from a guideline . Thus, using guideline adherence as outcome measure might have decreased the potential room for improvement of our intervention.
Comparison with existing literature
Guideline adherence was poor both before and after the study, which is in line with other studies that specifically investigated this topic [9, 11, 32]. Guideline non-adherence consisted of both under- and overtreatment, which was also present in other studies investigating OAC use [9, 11, 33]. Nonetheless, the increased OAC use in both groups could indicate awareness of OAC underuse is increasing.
We attribute the lack of effectiveness mainly to low usage (measured by interaction with the notifications), which had many reasons. These will be discussed in detail in a separate qualitative system evaluation. Briefly, barriers were mainly related to lack of time, too many alert notifications and limitations of the system’s functionality. Participants in acknowledged the potential of CDSSs for the future of healthcare, but implementing these systems in daily practice for multiple domains remains challenging. Alert prioritization, user customization, tight EHR integration and strict selection of alerts might improve CDSS effectiveness. Eccles et. al. found similar low usage rates and attributed the limited effectiveness of their system to this lack of use . Lugtenberg et. al. recently published results on a process evaluation of ‘NHGDoc’, an EHR-integrated decision support system that was implemented in 65% of GP practices in the Netherlands. Usage of the system was as low as 0.24%, and it also did not lead to changes in outcomes . A recent trial by Cook et. al. that attempted to improve prescription in AF by way of a CDSS also failed to show effectiveness of their intervention . While Cook et al did not report on usage, low usage may have also hampered their study. All four studies (including ours) attempted to provide the user with a non-obtrusive system, integrated in daily workflow. To provide more insight on when CDSS might be more effective, we performed a sub-group analysis on patients with incident AF. Although this sample was too small to be adequately powered, there was a trend towards effectiveness in this sub-group (P = 0.09 vs P = 0.23), which could indicate that GPs might be more inclined to follow CDSS based recommendations when first treating a patient. However, this trend could also be due to the difference between the quality of old and new data or other biases, and should be investigated in detail in future studies.
It is still unclear what can be considered ‘optimal timing’ in the busy daily practice. Its non-obtrusive nature may have been disadvantageous in a setting where patients, staff, e-mails and other matters compete for the GP’s attention. Recent studies have indicated that systems which force the user to respond to the alert are more effective , although clearly only a limited number of alerts can be presented in this way. Furthermore, physicians do not always agree with the guidelines these systems are based on,3 or have (valid) reasons for non-adherence [31, 37, 38]. The number of declined recommendations (N = 44, 54%) we found in our study suggests that this is at least part of the problem. ‘In terms of the recently-described “Two Stream Model”, the lack of use can be attributed mainly to data quality problems and sub-optimal presentation.
Implications for research
Guideline adherence in both the intervention and control groups increased during this trial investigating the effectiveness of a CDSS for stroke prevention. Our study could not demonstrate the effectiveness of our intervention, which was likely due to lack of use. The implementation of multi-domain CDSSs in clinical practice is challenging and future studies should investigate further improvements to facilitate effectiveness in a real-world setting. Stroke prevention in patients with AF is a field where much can be gained by following guidelines, and we therefore urge other researchers to dedicate their efforts to investigate how to effectively increase system use and guideline adherence.
S1 Checklist. The Consort checklist for this study.
S1 Dataset. The cleaned dataset used for this paper.
S1 Protocol. The study protocol for the ExpertAF study.
- Conceptualization: AA, HW, DA.
- Data curation: DA, SK.
- Formal analysis: DA.
- Funding acquisition: HW, AA.
- Investigation: DA, SK.
- Project administration: AA, HW, DA.
- Software: DA.
- Supervision: HW, AA.
- Writing – original draft: DA.
- Writing – review & editing: AA, HW, SK.
- 1. Chugh SS, Havmoeller R, Narayanan K, Singh D, Rienstra M, Benjamin EJ, et al. Worldwide Epidemiology of Atrial Fibrillation A Global Burden of Disease 2010 Study. Circulation. 2014;129(8):837–47. pmid:24345399
- 2. Zoni-Berisso M, Lercari F, Carazza T, Domenicucci S. Epidemiology of atrial fibrillation: European perspective. Clin Epidemiol. 2014;6:213–20. PubMed Central PMCID: PMC4064952. pmid:24966695
- 3. Lip GY, Brechin CM, Lane DA. The global burden of atrial fibrillation and stroke: a systematic review of the epidemiology of atrial fibrillation in regions outside North America and Europe. Chest. 2012;142(6):1489–98. Epub 2012/03/31. pmid:22459778
- 4. Wolf PA, Abbott RD, Kannel WB. Atrial fibrillation as an independent risk factor for stroke: the Framingham Study. Stroke. 1991;22(8):983–8. pmid:1866765
- 5. Hart RG, Pearce LA, Aguilar MI. Meta-analysis: Antithrombotic therapy to prevent stroke in patients who have nonvalvular atrial fibrillation. Ann Intern Med. 2007;146(12):857–67. pmid:17577005
- 6. Ruff CT, Giugliano RP, Braunwald E, Hoffman EB, Deenadayalu N, Ezekowitz MD, et al. Comparison of the efficacy and safety of new oral anticoagulants with warfarin in patients with atrial fibrillation: a meta-analysis of randomised trials. Lancet. 2014;383(9921):955–62. pmid:24315724
- 7. Nieuwlaat R, Olsson SB, Lip GY, Camm AJ, Breithardt G, Capucci A, et al. Guideline-adherent antithrombotic treatment is associated with improved outcomes compared with undertreatment in high-risk patients with atrial fibrillation. The Euro Heart Survey on Atrial Fibrillation. Am Heart J. 2007;153(6):1006–12. Epub 2007/06/02. pmid:17540203
- 8. Camm AJ, Lip GYH, De Caterina R, Savelieva I, Atar D, Hohnloser SH, et al. 2012 focused update of the ESC Guidelines for the management of atrial fibrillation. Eur Heart J. 2012;33(21):2719–47.
- 9. Dinh T, Nieuwlaat R, Tieleman RG, Buller HR, van Charante NA, Prins MH, et al. Antithrombotic drug prescription in atrial fibrillation and its rationale among general practitioners, internists and cardiologists in The Netherlands—The EXAMINE-AF study. A questionnaire survey. Int J Clin Pract. 2007;61(1):24–31. Epub 2007/01/19. pmid:17229177
- 10. Lip GYH, Lane DA. Stroke Prevention in Atrial Fibrillation A Systematic Review. Jama-Journal of the American Medical Association. 2015;313(19):1950–62.
- 11. Arts DL, Visscher S, Opstelten W, Korevaar JC, Abu-Hanna A, van Weert HCPM. Frequency and Risk Factors for Under- and Over-Treatment in Stroke Prevention for Patients with Non-Valvular Atrial Fibrillation in General Practice. PLoS One. 2013;8(7).
- 12. Eckman MH, Wise RE, Speer B, Sullivan M, Walker N, Lip GY, et al. Integrating real-time clinical information to provide estimates of net clinical benefit of antithrombotic therapy for patients with atrial fibrillation. Circ Cardiovasc Qual Outcomes. 2014;7(5):680–6. Epub 2014/09/11. pmid:25205788
- 13. Goud R, van Engen-Verheul M, de Keizer NF, Bal R, Hasman A, Hellemans IM, et al. The effect of computerized decision support on barriers to guideline implementation: a qualitative study in outpatient cardiac rehabilitation. Int J Med Inform. 2010;79(6):430–7. Epub 2010/04/10. pmid:20378396
- 14. Wilson FP, Shashaty M, Testani J, Aqeel I, Borovskiy Y, Ellenberg SS, et al. Automated, electronic alerts for acute kidney injury: a single-blind, parallel-group, randomised controlled trial. Lancet. 2015;385(9981):1966–74. pmid:25726515
- 15. Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med. 2012;157(1):29–43. Epub 2012/07/04. pmid:22751758
- 16. Moja L, Kwag KH, Lytras T, Bertizzolo L, Brandt L, Pecoraro V, et al. Effectiveness of Computerized Decision Support Systems Linked to Electronic Health Records: A Systematic Review and Meta-Analysis. Am J Public Health. 2014;104(12):E12–E22. pmid:25322302
- 17. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330(7494):765. Epub 2005/03/16. PubMed Central PMCID: PMC555881. pmid:15767266
- 18. Sittig DF, Wright A, Osheroff JA, Middleton B, Teich JM, Ash JS, et al. Grand challenges in clinical decision support. Journal of Biomedical Informatics. 2008;41(2):387–92. pmid:18029232
- 19. Roshanov PS, Fernandes N, Wilczynski JM, Hemens BJ, You JJ, Handler SM, et al. Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials. BMJ. 2013;346:f657. pmid:23412440
- 20. Medlock S, Wyatt JC, Patel VL, Shortliffe EH, Abu-Hanna A. Modeling information flows in clinical decision support: key insights for enhancing system effectiveness. J Am Med Inform Assoc. 2016.
- 21. Kawamoto K, Lobach DF, Willard HF, Ginsburg GS. A national clinical decision support infrastructure to enable the widespread and consistent practice of genomic and personalized medicine. BMC Med Inform Decis Mak. 2009;9:17. Epub 2009/03/25. PubMed Central PMCID: PMC2666673. pmid:19309514
- 22. McCoy AB, Waitman LR, Lewis JB, Wright JA, Choma DP, Miller RA, et al. A framework for evaluating the appropriateness of clinical decision support alerts and responses. J Am Med Inform Assoc. 2012;19(3):346–52. PubMed Central PMCID: PMCPMC3341775. pmid:21849334
- 23. Ash JS, Sittig DF, Campbell EM, Guappone KP, Dykstra RH. Some unintended consequences of clinical decision support systems. AMIA Annu Symp Proc. 2007:26–30. PubMed Central PMCID: PMCPMC2813668. pmid:18693791
- 24. Arts DL, Abu-Hanna A, Buller HR, Peters RJG, Eslami S, van Weert HCPM. Improving stroke prevention in patients with atrial fibrillation. Trials. 2013;14.
- 25. Team RDC. R: A Language and Environment for Statistical Computing. 2011.
- 26. An international glossary for primary care. Report of the Classification Committee of the World Organization of National Colleges, Academies and Academic Associations of General Practitioners/Family Physicians (WONCA). J Fam Pract. 1981;13(5):671–81. Epub 1981/10/01. pmid:7024465
- 27. Eslami S, Askari M, Medlock S, Arts DL, Wyatt JC, van Weert HCPM, et al. From assessment to improvement of elderly care in general practice using decision support to increase adherence to ACOVE quality indicators: study protocol for randomized control trial. Trials. 2014;15.
- 28. fibrillation DCoGPGDGfA. Guideline Atrial fibrillation (second partial revision). Huisarts & Wetenschap. 2013;56(8):392–401.
- 29. Lip GY, Nieuwlaat R, Pisters R, Lane DA, Crijns HJ. Refining clinical risk stratification for predicting stroke and thromboembolism in atrial fibrillation using a novel risk factor-based approach: the euro heart survey on atrial fibrillation. Chest. 2010;137(2):263–72. Epub 2009/09/19. pmid:19762550
- 30. Pisters R, Lane DA, Nieuwlaat R, de Vos CB, Crijns HJ, Lip GY. A novel user-friendly score (HAS-BLED) to assess 1-year risk of major bleeding in patients with atrial fibrillation: the Euro Heart Survey. Chest. 2010;138(5):1093–100. Epub 2010/03/20. pmid:20299623
- 31. Arts DL, Voncken AG, Medlock S, Abu-Hanna A, van Weert HC. Reasons for intentional guideline non-adherence: A systematic review. Int J Med Inform. 2016;89:55–62. pmid:26980359
- 32. Ogilvie IM, Newton N, Welner SA, Cowell W, Lip GYH. Underuse of Oral Anticoagulants in Atrial Fibrillation: A Systematic Review. Am J Med. 2010;123(7):638–U86. pmid:20609686
- 33. Palomaki A, Mustonen P, Hartikainen JE, Nuotio I, Kiviniemi T, Ylitalo A, et al. Underuse of anticoagulation in stroke patients with atrial fibrillation—the FibStroke Study. Eur J Neurol. 2015.
- 34. Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. Br Med J. 2002;325(7370):941–4.
- 35. Lugtenberg M, Weenink JW, van der Weijden T, Westert GP, Kool RB. Implementation of multiple-domain covering computerized decision support systems in primary care: a focus group study on perceived barriers. BMC Med Inform Decis Mak. 2015;15:82. PubMed Central PMCID: PMCPMC4603732. pmid:26459233
- 36. Cook DA, Enders F, Caraballo PJ, Nishimura RA, Lloyd FJ. An automated clinical alert system for newly-diagnosed atrial fibrillation. PLoS One. 2015;10(4):e0122153. PubMed Central PMCID: PMCPMC4388495. pmid:25849969
- 37. Cabana, Rand CS, Powe NR, Wu AW, Wilson MH. Why Don ‘ t Physicians Follow guidlines, A Framework for Improvement. October. 1999;0718.
- 38. Lugtenberg M, Zegers-van Schaick J, Westert G, Burgers J. Why don't physicians adhere to guideline recommendations in practice? An analysis of barriers among Dutch general practitioners. Implementation Science. 2009;4(1):54.