Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Implementation of a clinical decision support tool to improve the adequate prescription of low-molecular-weight heparins in non-surgical patients

  • Froucke van Gosliga ,

    Roles Conceptualization, Formal analysis, Investigation, Methodology, Project administration, Resources, Validation, Visualization, Writing – original draft

    froucke.van.gosliga@frisiusmc.nl

    Affiliation Center of Pharmacy, Frisius Medical Center, Leeuwarden, The Netherlands

  • Dagmar Pals,

    Roles Formal analysis, Investigation, Validation, Visualization, Writing – original draft

    Affiliation Faculty of Science and Engineering, University of Groningen, Groningen, The Netherlands

  • Annette van Ojik,

    Roles Conceptualization, Methodology, Supervision, Writing – review & editing

    Affiliation Center of Pharmacy, Frisius Medical Center, Leeuwarden, The Netherlands

  • Eric van Roon

    Roles Conceptualization, Methodology, Supervision, Writing – review & editing

    Affiliation Center of Pharmacy, Frisius Medical Center, Leeuwarden, The Netherlands

Abstract

Background

Non-surgical hospitalized patients have an increased risk of developing venous thromboembolism (VTE). This risk can be reduced by thromboprophylaxis with low-molecular-weight heparins (LMWHs), but adherence to thromboprophylaxis guidelines is generally low.

Objectives

To study the effect of implementing a clinical decision support (CDS) tool in the electronic health record on the percentage of adequately prescribed thromboprophylaxis with an LMWH in non-surgical patients at high risk for VTE.

Methods

Data on the Padua Prediction Score (PPS) and thromboprophylaxis in high-risk non-surgical patients (≥18 years) were collected at different time points before and after the implementation of the intervention consisting of a validated CDS tool. The percentage of adequately prescribed LMWH thromboprophylaxis was described pre- and post-intervention in a stepwise approach, in which both crude and adjusted pre- and post-intervention differences were assessed using an interrupted time series analysis, accounting for potential time trends and autocorrelation.

Results

In 400 patients included, 200 pre- and 200 post-intervention, the percentage of adequately prescribed LMWH thromboprophylaxis increased from 78% to 91%, an increase of 13% (95% CI: 6%−20%). This effect diminished to 8% (95% CI: −4%−19%) after adjustment for the pre-intervention slope and autocorrelation.

Conclusion

In our study, the already high percentage of patients with adequately prescribed LMWH-thromboprophylaxis could potentially be increased further with the implementation of a CDS tool. With increasing amounts of data available in electronic health records, CDS tools might be an efficient and sustainable intervention to improve healthcare quality.

Introduction

Venous thromboembolism (VTE), including deep vein thrombosis (DVT) and pulmonary embolism (PE), is the third most common cardiovascular diagnosis and a leading cause of increased healthcare costs and preventable death in hospitalized patients [13]. About 60% of all cases of VTE are hospital-acquired, occurring during or after a recent hospital admission [4]. Without adequate thromboprophylaxis, approximately 3% of hospitalized non-surgical patients develop VTE, increasing to 11% among patients at elevated risk for developing VTE [5]. The appropriate use of primary thromboprophylaxis in the form of a low-molecular-weight heparin (LMWH) in hospitalized non-surgical patients at increased risk for VTE has proven to be safe, effective, and cost-effective in reducing DVT and PE [57].

Although there is some discussion about which risk assessment model works best for identifying (subtypes of) patients at increased risk of VTE [813], the Padua Prediction Score (PPS), shown in Table 1, is currently considered the best available model to stratify between non-surgical patients with a high and low risk of VTE. Therefore, the PPS is included in international and national thromboprophylaxis guidelines to determine risk of VTE [5,14,15]. According to these guidelines, thromboprophylaxis, preferably in the form of an LMWH, should be started if a patient is at high risk for VTE.

thumbnail
Table 1. Padua Prediction Score risk assessment tool.

https://doi.org/10.1371/journal.pone.0350017.t001

However, adherence to thromboprophylaxis guidelines is low, and studies show that thromboprophylaxis continues to be used inadequately [1620]. According to a Dutch report published in 2017, 49% of the non-surgical hospitalized patients with a high risk of VTE received adequate thromboprophylaxis according to the Dutch guideline [21], which was confirmed by a more recent Dutch study performed in 2021 [6].

A recent review assessing the effects of interventions to increase the adequate use of thromboprophylaxis in hospitalized patients found that alerts and multifaceted interventions were associated with increased use of thromboprophylaxis, with computer alerts suggested as the most effective intervention [22]. Computerized clinical decision support (CDS) systems are associated with substantial improvements in the use of adequate prophylaxis and reductions in VTE events, particularly in non-surgical patients [23,24]. The multifaceted intervention from Jaspers et al, increased thromboprophylaxis guideline adherence from 49% to 82%, which they believe can mostly be attributed to the implementation of a highly specific and user-friendly CDS tool integrated into the electronic health record (EHR) [6].

We aimed to study the effect of a single intervention, the implementation of a CDS tool in the electronic health record, on the percentage of adequately prescribed thromboprophylaxis in non-surgical patients with a high VTE risk. We looked at both the crude effect and the adjusted pre-post intervention effect that was corrected for potential underlying pre-intervention time trends and autocorrelation using an interrupted time series analysis.

Methods

Study design

This study was conducted at the Medical Centre Leeuwarden, a large teaching hospital in the Netherlands. The study has a prospective time series design with data collected at different time points pre- and post-intervention.

Patient selection

Non-surgical patients ≥18 years of age who were admitted to a medical ward with a hospital stay of ≥48 hours were eligible for inclusion in the study. Patients with a high risk of VTE, defined as a PPS of ≥4 (Table 1), and no anticoagulant in use were included. Patients who had recently (<1 week) undergone a percutaneous coronary intervention and palliative patients were excluded.

Intervention

The intervention consisted of the implementation of a CDS tool involving a Best Practice Advisory (BPA) in Epic® (Epic Systems, Verona, USA). This tool provides an automated recommendation to the prescriber of the appropriate specialization when there is an indication for thromboprophylaxis according to the PPS (score ≥4). The CDS tool collects data from the patient’s EHR on patient characteristics (sex, age, BMI), from the patient’s problem list (e.g., active cancer, previous VTE, thrombophilic condition), the active medication list (including oncology treatment plans),and the nursing assessment (mobility). The nursing assessment is carried out several times a day, and information about mobility (e.g., bedbound, sitting in chair, walking) is documented. When there is no anticoagulant in use at 36 hours after start of admission in the active medication list the recommendation is shown to the prescriber (physician, physician assistant or certified nurse specialist) of the main specialty as a pop-up when opening the patient file. The CDS tool pop-up displays the PPS and associated risk factors and an order for nadroparin 2,850 IU, the standard prophylactic dose according to local protocol, can be initiated immediately. The bleeding risk is not taken into account in the tool and has to be assessed by the prescriber. An action has to be taken and a reason has to be provided if the advice is not adopted by the prescriber, choosing from: I am not the responsible physician, the calculated score is not correct, there is a contraindication, or other (with explanation). Before live implementation, we tested the CDS tool in a test environment and evaluated its performance. The CDS tool achieved a sensitivity of 92% and a specificity of 68%. Prescribers were informed about the implementation of the new CDS tool before going live.

Data collection

Data were collected between July 2022 and July 2023 and the CDS tool became operational in March 2023. Data were collected at ten different time points, with each time point representing approximately one month. Data were collected in Microsoft Excel 2021 from the first working day of the month until 40 patients per month were included.

Statistical analysis

Patient characteristics were described before and after the intervention with median (interquartile range [IQR]) for continuous variables and numbers (%) for categorical variables. Patient characteristics were compared using the Mann-Whitney U test for continuous variables and the Fisher’s exact test for categorical variables.

The primary endpoint, the percentage of adequately prescribed thromboprophylaxis, was first assessed for the combined time points before and after the intervention with a Fisher’s exact test and the pre-post difference with the 95% confidence interval (CI) was reported. Next the potential pre-implementation time trend was determined and corrected for by using an interrupted time series (ITS) analysis using an autoregressive integrated moving average (ARIMA) model, both unadjusted and adjusted for autocorrelation.

At the time the study was designed, no standard methodological guidance was available for power or sample size considerations within an ITS framework. Therefore, no formal sample size calculation was performed. Given the exploratory nature of the study, p-values are not provided and estimates are presented with appropriate indicators of uncertainty. Statistical analysis was performed using SPSS Statistics 28.0 (IBM, New York, USA).

Ethical approval

Local feasibility of the study was assessed by the local research committee (COV) of the Medical Centre Leeuwarden and a written waiver was obtained as the study was not within the scope of the Medical Research Involving Human Subjects Act (WMO 15-03-2022).

Results

Patient characteristics

In total 400 patients were included: 200 pre- and 200 post-intervention. Patient characteristics are shown in Table 2. There was no difference between patient characteristics before and after the implementation of the intervention, with the exception of the distribution across the different departments. Most patients (> 70%) in both groups were admitted to the department of internal medicine, neurology, or pulmonary medicine.

Effect of the CDS tool

Before implementation of the CDS tool, the overall percentage of adequately prescribed thromboprophylaxis was 78%, after implementation of the CDS tool this percentage increased by 13% (95% CI: 6%−20%) to 91%. Although no clear pre-intervention time trend was observed (Fig 1), the intervention effect diminished when accounting for the pre-intervention slope and autocorrelation between time points to 8% (95% CI: −4%−19%) (Table 3). After implementing the CDS tool, besides the direct level effect, the percentage of adequately prescribed thromboprophylaxis did not increase further over time.

thumbnail
Table 3. The adjusted effect of the CDS tool on the percentage of adequately prescribed thromboprophylaxis.

https://doi.org/10.1371/journal.pone.0350017.t003

thumbnail
Fig 1. Adequately prescribed thromboprophylaxis pre- and post-intervention.

The grey line shows the average percentage pre- and post-intervention with the 95% CI in the grey transparent frame, the solid blue line shows the pre- and post-intervention slopes and the dashed post-intervention line shows the counterfactual trend line. Y-axis starts at 70% to illustrate detailed trends.

https://doi.org/10.1371/journal.pone.0350017.g001

Discussion

The implementation of a clinical decision support (CDS) tool integrated into the electronic health record (EHR) was associated with an increase in the percentage of adequately prescribed low-molecular-weight heparin (LMWH) thromboprophylaxis in non-surgical patients at high risk of venous thromboembolism (VTE). While a crude increase of 13% was observed, the effect estimate decreased to 8% after adjustment for pre-intervention trends and autocorrelation.

In this study, before implementation of the CDS tool, an average percentage of 78% adequately prescribed thromboprophylaxis was found, which was higher than expected based on previous literature [6,16,2528]. In a recent Dutch study, adherence to the thromboprophylaxis guideline of 55% was observed for non-surgical patients with a high VTE risk. An international cross-sectional study performed in 32 countries reported that on average 40% of non-surgical high-risk patients received adequate thromboprophylaxis. The reason for the high percentage in our hospital is not exactly known, but we hypothesize it might be due to increased awareness because of several reported local incidents of VTE in the years prior to performing this study due to inadequate thromboprophylaxis and because of the COVID pandemic when there was a lot of attention for thromboprophylaxis [29]. As a consequence, the potential for further improvement was limited, which likely contributed to the modest effect size observed in this study, consistent with the findings reported by Spyropoulos et al [30].

Several studies have evaluated the impact of CDS tools integrated into EHR systems to improve thromboprophylaxis prescribing. Early work by Kucher et al. demonstrated that electronic alerts to physicians significantly increased thromboprophylaxis use and reduced symptomatic VTE events, although this study was conducted in a setting with substantially lower baseline adherence and did not account for underlying temporal trends [25]. More recent studies have confirmed that alert-based CDS interventions are associated with improved guideline adherence but reported effect sizes vary widely depending on study design, baseline adherence, and whether the intervention was part of a multifaceted quality improvement strategy [6,2224,28,31]. Systemic reviews and meta-analyses emphasize substantial heterogeneity across studies and highlight that improvements in prescribing behavior do not necessarily translate into measurable reductions in clinical outcomes such as VTE or bleeding events [2224]. In line with this literature, our study focused on prescribing behavior as a process measure and did not assess clinical outcomes.

With regard to the performance the implemented CDS tool had a sensitivity of 92% and a specificity of 68%. Sensitivity is the ability of a tool to alert prescribers adequately when patients are at risk of an event, where specificity is the ability to correctly identify patients that are not at risk where no alert has to be generated. The ideal CDS tool should have high specificity and sensitivity, however CDS systems tend to have a high sensitivity and lower specificity as we saw with our CDS tool [3234]. To obtain a high sensitivity an increased number of alerts should be accepted, this can however lead to alert fatigue and overriding of advices generated by the CDS system or patients unnecessarily receiving thromboprophylaxis with higher risk of bleeding as a consequence [3537]. The ability of a tool to identify as many patients that are at risk against the risk of false positives and overalerting should be carefully weighted. We observed that often prescribers do not immediately follow-up on the advice, which is believed being due to the fact that the pop-up appears when opening the patient file before the prescriber has assessed the patient. We know that appropriate timing of alerts within the workflow is important, alerting as early as possible and at that time having complete information [3335,3740].

A limitation of the implemented CDS tool is that bleeding risk was not incorporated and had to be assessed separately by the prescriber. Although no safety signals or reported cases of misprescription attributable to the CDS tool were identified during the study period, bleeding outcomes were not systematically assessed. Balancing thrombotic and bleeding risks remains a key challenge in thromboprophylaxis decision-making, and future versions of the CDS tool may benefit from integrating validated bleeding risk assessment models to improve specificity and safety.

A strength of this study is that a single intervention, a CDS tool, was studied rather than a multifaceted intervention. Because data were collected at different a priori set time points over a longer period, potential underlying trends could be detected and accounted for. However, because an interrupted time series of aggregated level data was used (i.e., percentage of adequately prescribed thromboprophylaxis per month), this analysis has much less statistical power than a straightforward pre-post comparison. Due to the small number of time points some influential periods resulted in a slight positive slope before the intervention, which diminished the direct intervention effect. Moreover, the percentage of adequately prescribed thromboprophylaxis before the intervention was much higher than expected, resulting in less room for improvement. It is known that ITS designs are often underpowered and when the expected effect size is small or the number of time points is small this analysis should be used with caution [41,42].

Further, the optimal performance of the CDS tool is highly dependent on an updated patient file. Risk factors of the PPS have to be documented and kept up at the right locations, for example in the so-called problem list, for the tool to automatically abstract them to be able to give a reliable advice. When risk factors are not documented properly they may be missed by the tool and a patient can incorrectly be seen as a low risk patient. Several studies have found that completeness of the problem list is an issue across healthcare facilities, influencing the performance of CDS tools [43,44]. In our experience, correct definition and registration of reduced mobility, one of the highest scoring risk factors of the PPS (score of 3), is the most challenging and should be given attention.

Future studies should look into the generalizability, safety and effectiveness of this type of intervention across different hospital settings and assessing their impact on clinical outcomes. We aim to further optimize our CDS tool by improving specificity to prevent alert fatigue and over-treatment, and determine the appropriate timing of the alert.

Conclusion

Our CDS tool integrated in the EHR may increase the percentage of high-risk patients adequately treated with thromboprophylaxis. With increasing amounts of data available in electronic health records, fine-tuned CDS tools might be an efficient and sustainable intervention to improve healthcare quality without interrupting workflow. Future research is needed to further optimize these types of tools to reach their full potential.

Supporting information

References

  1. 1. Heit JA, Spencer FA, White RH. The epidemiology of venous thromboembolism. J Thromb Thrombolysis. 2016;41(1):3–14. pmid:26780736
  2. 2. H T. Thromboembolism: an under appreciated cause of death. Lancet Haematol. 2015;2(10):e393.
  3. 3. Wendelboe AM, Raskob GE. Global burden of thrombosis: epidemiologic aspects. Circ Res. 2016;118(9):1340–7.
  4. 4. Heit JA, O’Fallon WM, Petterson TM, Lohse CM, Silverstein MD, Mohr DN. Relative impact of risk factors for deep vein thrombosis and pulmonary embolism: a population-based study. Arch Internal Med. 2002;162(11):1245–8.
  5. 5. Barbar S, Noventa F, Rossetto V, Ferrari A, Brandolin B, Perlati M, et al. A risk assessment model for the identification of hospitalized medical patients at risk for venous thromboembolism: the Padua Prediction Score. J Thromb Haemost. 2010;8(11):2450–7. pmid:20738765
  6. 6. Jaspers T, Duisenberg-van Essenberg M, Maat B, Durian M, van den Berg R, van den Bemt P. A multifaceted clinical decision support intervention to improve adherence to thromboprophylaxis guidelines. Int J Clin Pharm. 2021;43(5):1327–36. pmid:33709383
  7. 7. Streiff MB, Lau BD. Thromboprophylaxis in nonsurgical patients. Hematology Am Soc Hematol Educ Program. 2012;2012:631–7. pmid:23233645
  8. 8. Stuck AK, Spirk D, Schaudt J, Kucher N. Risk assessment models for venous thromboembolism in acutely ill medical patients. A systematic review. Thromb Haemost. 2017;117(4):801–8. pmid:28150851
  9. 9. Zhou H, Hu Y, Li X, Wang L, Wang M, Xiao J, et al. Assessment of the risk of venous thromboembolism in medical inpatients using the padua prediction score and caprini risk assessment model. J Atheroscler Thromb. 2018;25(11):1091–104. pmid:29540637
  10. 10. Pandor A, Tonkins M, Goodacre S, Sworn K, Clowes M, Griffin XL, et al. Risk assessment models for venous thromboembolism in hospitalised adult patients: a systematic review. BMJ Open. 2021;11(7):e045672. pmid:34326045
  11. 11. Lavon O, Tamir T. Evaluation of the Padua Prediction Score ability to predict venous thromboembolism in Israeli non-surgical hospitalized patients using electronic medical records. Sci Rep. 2022;12(1):6121. pmid:35414101
  12. 12. Zhou C, Guang Y, Luo Y, Ge H, Wei H, Liu H, et al. Superior predictive value of D-dimer to the padua prediction score for venous thromboembolism in inpatients with AECOPD: a multicenter cohort study. Int J Chron Obstruct Pulmon Dis. 2022;17:2711–22. pmid:36304969
  13. 13. Yang S, Zhang Y, Jiao X, Liu J, Wang W, Kuang T, et al. Padua prediction score may be inappropriate for VTE risk assessment in hospitalized patients with acute respiratory conditions: a Chinese single-center cohort study. Int J Cardiol Heart Vasc. 2023;49:101301. pmid:38035260
  14. 14. Nederlandse Internisten V. Richtlijn antitrombotisch beleid.
  15. 15. Kahn SR, Lim W, Dunn AS, Cushman M, Dentali F, Akl EA, et al. Prevention of VTE in nonsurgical patients: antithrombotic therapy and prevention of thrombosis, 9th ed: American college of chest physicians evidence-based clinical practice guidelines. Chest. 2012;141(2 Suppl):e195S-e226S. pmid:22315261
  16. 16. Cohen AT, Tapson VF, Bergmann J-F, Goldhaber SZ, Kakkar AK, Deslandes B, et al. Venous thromboembolism risk and prophylaxis in the acute hospital care setting (ENDORSE study): a multinational cross-sectional study. Lancet. 2008;371(9610):387–94. pmid:18242412
  17. 17. Goldhaber SZ, Dunn K, MacDougall RC. New onset of venous thromboembolism among hospitalized patients at Brigham and Women’s Hospital is caused more often by prophylaxis failure than by withholding treatment. Chest. 2000;118(6):1680–4. pmid:11115458
  18. 18. Tapson VF, Decousus H, Pini M, Chong BH, Froehlich JB, Monreal M, et al. Venous thromboembolism prophylaxis in acutely ill hospitalized medical patients: findings from the International Medical Prevention Registry on Venous Thromboembolism. Chest. 2007;132(3):936–45. pmid:17573514
  19. 19. Kucher N, Spirk D, Baumgartner I, Mazzolai L, Korte W, Nobel D, et al. Lack of prophylaxis before the onset of acute venous thromboembolism among hospitalized cancer patients: the SWIss Venous ThromboEmbolism Registry (SWIVTER). Ann Oncol. 2010;21(5):931–5. pmid:19828560
  20. 20. Chopard P, Dörffler-Melly J, Hess U, Wuillemin WA, Hayoz D, Gallino A, et al. Venous thromboembolism prophylaxis in acutely ill medical patients: definite need for improvement. J Intern Med. 2005;257(4):352–7. pmid:15788005
  21. 21. Moesker MJ, Damen NL, Groot JF d, Bruijne MC d, Wagner C. Antistollingszorg in Nederlandse ziekenhuizen: evaluatie van tromboseprofylaxe en perioperatief antistollingsbeleid in vergelijking met geldende richtlijnen. NIVEL; 2017.
  22. 22. Kahn SR, Morrison DR, Diendéré G, Piché A, Filion KB, Klil-Drori AJ, et al. Interventions for implementation of thromboprophylaxis in hospitalized patients at risk for venous thromboembolism. Cochrane Database Syst Rev. 2018;4(4):CD008201. pmid:29687454
  23. 23. Adams P, Riggio JM, Thomson L, Brandell-Marino R, Merli G. Clinical decision support systems to improve utilization of thromboprophylaxis: a review of the literature and experience with implementation of a computerized physician order entry program. Hosp Pract (1995). 2012;40(3):27–39. pmid:23086092
  24. 24. Karajizadeh M, Hassanipour S, Sharifian R, Tajbakhsh F, Saeidnia HR. The effect of information technology intervention on using appropriate VTE prophylaxis in non-surgical patients: a systematic review and meta-analysis. Digit Health. 2022;8:20552076221118828. pmid:36003314
  25. 25. Kucher N, Koo S, Quiroz R, Cooper JM, Paterno MD, Soukonnikov B, et al. Electronic alerts to prevent venous thromboembolism among hospitalized patients. N Engl J Med. 2005;352(10):969–77. pmid:15758007
  26. 26. Labarere J, Bosson J-L, Sevestre M-A, Sellier E, Richaud C, Legagneux A. Intervention targeted at nurses to improve venous thromboprophylaxis. Int J Qual Health Care. 2007;19(5):301–8. pmid:17726037
  27. 27. Pai M, Lloyd NS, Cheng J, Thabane L, Spencer FA, Cook DJ, et al. Strategies to enhance venous thromboprophylaxis in hospitalized medical patients (SENTRY): a pilot cluster randomized trial. Implement Sci. 2013;8:1. pmid:23279972
  28. 28. Roy P-M, Rachas A, Meyer G, Le Gal G, Durieux P, El Kouri D, et al. Multifaceted intervention to prevent venous thromboembolism in patients hospitalized for acute medical illness: a multicenter cluster-randomized trial. PLoS One. 2016;11(5):e0154832. pmid:27227406
  29. 29. Marchandot B, Trimaille A, Curtiaud A, Matsushita K, Jesel L, Morel O. Thromboprophylaxis: balancing evidence and experience during the COVID-19 pandemic. J Thromb Thrombolysis. 2020;50(4):799–808.
  30. 30. Spyropoulos AC, Goldin M, Koulas I, Solomon J, Qiu M, Ngu S, et al. Universal EHRs clinical decision support for thromboprophylaxis in medical inpatients: a cluster randomized trial. JACC Adv. 2023;2(8):100597.
  31. 31. Tsaftaridis N, Goldin M, Spyropoulos AC. System-wide thromboprophylaxis interventions for hospitalized patients at risk of venous thromboembolism: focus on cross-platform clinical decision support. J Clin Med. 2024;13(7):2133. pmid:38610898
  32. 32. Weingart SN, Simchowitz B, Padolsky H, Isaac T, Seger AC, Massagli M, et al. An empirical model to estimate the potential impact of medication safety alerts on patient safety, health care utilization, and cost in ambulatory care. Arch Intern Med. 2009;169(16):1465–73. pmid:19752403
  33. 33. Olakotan OO, Mohd Yusof M. The appropriateness of clinical decision support systems alerts in supporting clinical workflows: a systematic review. Health Informatics J. 2021;27(2):14604582211007536. pmid:33853395
  34. 34. Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med. 2020;3:17. pmid:32047862
  35. 35. Coleman JJ, van der Sijs H, Haefeli WE, Slight SP, McDowell SE, Seidling HM, et al. On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop. BMC Med Inform Decis Mak. 2013;13:111. pmid:24083548
  36. 36. Khalifa M, Zabani I. Improving utilization of clinical decision support systems by reducing alert fatigue: strategies and recommendations. Stud Health Technol Inform. 2016;226:51–4. pmid:27350464
  37. 37. Alexiuk M, Elgubtan H, Tangri N. Clinical decision support tools in the electronic medical record. Kidney Int Rep. 2023;9(1):29–38. pmid:38312784
  38. 38. Shi Y, Amill-Rosario A, Rudin RS, Fischer SH, Shekelle P, Scanlon DP, et al. Barriers to using clinical decision support in ambulatory care: do clinics in health systems fare better?. J Am Med Inform Assoc. 2021;28(8):1667–75. pmid:33895828
  39. 39. Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc. 2003;10(6):523–30. pmid:12925543
  40. 40. Meunier P-Y, Raynaud C, Guimaraes E, Gueyffier F, Letrilliart L. Barriers and facilitators to the use of clinical decision support systems in primary care: a mixed-methods systematic review. Ann Fam Med. 2023;21(1):57–69. pmid:36690490
  41. 41. Ramsay CR, Matowe L, Grilli R, Grimshaw JM, Thomas RE. Interrupted time series designs in health technology assessment: lessons from two systematic reviews of behavior change strategies. Int J Technol Assess Health Care. 2003;19(4):613–23. pmid:15095767
  42. 42. Zhang F, Wagner AK, Ross-Degnan D. Simulation-based power calculation for designing interrupted time series analyses of health policy interventions. J Clin Epidemiol. 2011;64(11):1252–61. pmid:21640554
  43. 43. Wang EC-H, Wright A. Characterizing outpatient problem list completeness and duplications in the electronic health record. J Am Med Inform Assoc. 2020;27(8):1190–7. pmid:32620950
  44. 44. Wright A, McCoy AB, Hickman T-TT, Hilaire DS, Borbolla D, Bowes WA 3rd, et al. Problem list completeness in electronic health records: a multi-site study and assessment of success factors. Int J Med Inform. 2015;84(10):784–90. pmid:26228650