We analysed potential drug-drug interactions (DDI) in ambulatory care in Switzerland based on claims data from three large health insurers in 2010 to identify physicians with peculiar prescription behaviour differing from peers of the same specialty.
We analysed contraindicated or potentially contraindicated DDI from the national drug formulary and calculated for each physician the ratios of the number of patients with a potential DDI divided by the number of patients at risk and used a zero inflated binomial distribution to correct for the inflated number of observations with no DDI. We then calculated the probability that the number of caused potential DDI of physicians was unlikely (p-value < 0.05 and ≥0.01) and very unlikely (p-value <0.01) to be due to chance.
Of 1'607'233 females and 1'525'307 males 1.3% and 1.2% were exposed to at least one potential DDI during 12 months. When analysing the 40 most common DDI, 598 and 416 of 18,297 physicians (3.3% and 2.3%) were causing potential DDI in a frequency unlikely (p<0.05 and p≥0.01) and very unlikely (p<0.01) to be explained by chance. Patients cared by general practitioners and cardiologists had the lowest probability (0.20 and 0.26) for not being exposed to DDI.
Citation: Bucher HC, Achermann R, Stohler N, Meier CR (2016) Surveillance of Physicians Causing Potential Drug-Drug Interactions in Ambulatory Care: A Pilot Study in Switzerland. PLoS ONE 11(1): e0147606. https://doi.org/10.1371/journal.pone.0147606
Editor: Kent E. Vrana, Penn State College of Medicine, UNITED STATES
Received: April 15, 2015; Accepted: January 6, 2016; Published: January 25, 2016
Copyright: © 2016 Bucher et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data for this analysis was provided by the Swiss health insurers Helsana, CSS and Sanitas and is owned by these companies. The data may be available upon request from Helsana. Request for the file pilot study for quality indicators of drug-drug-interactions in Switzerland in 2010 should be addressed to: Dr. Oliver Reich, Helsana, Zurich, Switzerland, Email: firstname.lastname@example.org.
Funding: This study was supported by a grant from the Swiss Medical Association (FMH) and Santésuisse. The Basel Institute for Clinical Epidemiology has been supported by unrestricted research grants by Santésuisse and the Gottfried and Julia Bangerter-Rhyner-Foundation. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The study was financed by Santésuisse (the trustee of Swiss health insurers) and the Swiss Medical Association (FMH). This does not alter the authors' adherence to PLOS ONE policies on sharing data and materials.
Potential drug-drug interactions (DDI) are a common drug-related problem in ambulatory care and may lead to adverse drug reactions with serious consequences . In epidemiological surveys, between 0.1% and 2.6% of hospitalisations are related to adverse drug reactions from DDI [2, 3]. Rates for elderly patients are higher, ranging from 0.7% to 6.4% . Potential DDI may be inevitable in patients with comorbidities in need of multiple drugs, and the risk of adverse clinical consequences may be reduced by appropriate dose adjustment and monitoring. Many potential DDI may not be clinically relevant or can be handled if monitored adequately; some DDI, however, can be deleterious, and concomitant use of such drugs has to be strongly discouraged. For example, co-administration of rifampicin, a bactericidal antibiotic against tuberculosis with potent cytochrome P450 enzyme induction properties, should not be combined with anti-HIV protease inhibitors or with new oral anticoagulants, as plasma levels of the latter drugs may substantially decrease and lead to antiretroviral therapy failure with serious consequences for the patient. Potential DDI are an explicit indicator for the evaluation of the appropriateness of medication in ambulatory care and for measuring physician performance . Explicit indicators such as DDI can be applied to large data, but generally cannot address other factors such as comorbidities which likely influence drug prescribing decision leading to DDI. Quantifying potential DDI may not necessarily be suitable as marker for the quality of therapy of physicians for particular patients; clinical judgement and implicit criteria are needed to achieve this detailing. However, this approach is time consuming, depends on users’ knowledge and attitude, and may lack reliability .
In Switzerland no population-based data on the prevalence of potential DDI in ambulatory care exist, and no indicators for the quality of ambulatory care are routinely used. The goal of this pilot study was to investigate the frequency of DDI in the Swiss population using health insurer claims data and to explore whether individual physicians causing potential DDI at higher frequency than their peers of the same specialty can be identified. In theory this would open the possibility to establish feedback or benchmark systems for physicians with peculiar drug prescription behaviour to better understand and to potentially improve their prescription behaviour.
We assessed the frequency of potential DDI in patients of all age using claim data from three large health insurance companies in Switzerland in 2010. Reimbursement claims from pharmacists and self-dispensing physicians and hospital outpatients are electronically processed by these insurers. The system provides information on the date the prescription drug was issued, the delivery date, the active ingredient, the drug formulation, the amount of the active ingredient, and the number of dispensed units of a drug. In the reimbursement system prescribed drugs can be identified by a specific ‘pharmacode’ (www.e-mediat.ch), and prescribing physicians by a unique registration number (‘Zahlstellenregisternummer’). For the purpose of this study, the provided data files did not contain information potentially allowing the identification of individual patients. According to Swiss law ethical approval for routinely gathered claim data is not mandatory if data files are entirely anonymized.
For the reporting of all potential DDI we included drug combinations of single compounds or drug classes prescribed by physicians or outpatient clinics classified as contraindicated (grade 1) or potentially contraindicated (grade 2) based on the Pharmavista database version 2011. For the comparison of physicians we restricted the number of contraindicated and potentially contraindicated drug combinations to 40. We excluded all DDI we encountered in our database with less than 50 patients for contraindicated and less than 250 patients for potentially contraindicated drugs. We also disregarded all clinically not relevant DDI (own judgement) and DDI that can be avoided with appropriately scheduled drug intake. Contraceptives are not reimbursed by Swiss health insurers and therefore are not recorded. The Pharmavista database is tailored to the Swiss market and allows for the identification of DDI with the use of ATC codes from the Anatomical Therapeutic Chemical Classification system, an internationally accepted classification system for drug consumption studies provided by WHO (http://www.whocc.no). The database covers 86% of drug interaction information provided by Stockley’s Drug Interaction compendium . A list of these DDI combinations is provided in the S1 Table.
To identify potential DDI the overlapping drug use periods had to be calculated. We first determined the treatment duration of each drug because this information is not contained in the claims database. We made an assumption that the number of days of intake is on average equal to the number of defined daily doses (DDD) per active substance prescribed. In case of a subsequent prescription of the same active substance, the next prescription date was taken to determine when an intake period ended; otherwise, the length of the intake period was set to the number of DDD. The information on DDD was taken from Galdat, a database provided by e-mediat (e-mediat AG Schönbühl); in case of missing information for DDD we consulted a database provided by WHO (http://www.whocc.no/atc_ddd_index/). For the determination of all overlapping drug intake time periods, we assumed that patients were actually taking both drugs for the entire period. Differences between the estimated and actually drug intake period can obviously lead to a biased determination of potential DDI. Such differences are more likely to occur if a patient takes two or more drugs for shorter periods than if one drug has to be taken on a regular basis (e.g. oral anticoagulant in a patient with atrial fibrillation), with a second drug with a potential DDI being added. Therefore we checked the lag time periods between delivery dates of potentially interacting drugs.
Patients may have multiple exposures to the same potential DDI caused by one or more physicians caring for the same patient. In this project we aimed to identify the physician responsible for a potential DDI by issuing a prescription on top of another pre-existing treatment. The prescription date does not always reflect the date when a physician had contact with a patient. Some prescriptions can be renewed by the pharmacists without seeing the physician, but these were included in the analyses as well. For each prescription we assigned the preceding consultation date with the physician who issued the prescription. If a conflicting drug was already prescribed at this time point, the physician was considered responsible for the potential DDI. If a physician caused in the same patient multiple potential DDI with the same drugs, we counted only the first potential DDI. If the same DDI in the same patient was caused by a second physician, this was counted as a new potential DDI and assigned to the second physician.
The likelihood for potential DDI depends on the patient population under study. Physicians caring for patients with multiple comorbidities are more likely to prescribe several drugs and to cause potential DDI. We lacked comorbidity parameters to adjust for patient case mix; we therefore restricted for each physician the patient collective to those patients who were at risk for a potential DDI. We defined being at risk as taking at least one of the two drugs of a DDI from our predefined list. In case of a DDI, typically one drug is usually taken over a longer period and the other only for a short period. Therefore we considered only patients at risk who were chronically exposed to a long-term treatment with a drug. For example, to explore the potential DDI for statins and macrolide antibiotics, only patients having received statins were included in the patient collective at risk. The drugs used to identify the patient collective at risk are marked in S1 Table (drug class A and B). Because the clinical relevance of the relatively prevalent DDI of clopidogrel and proton pump inhibitors is controversially discussed we present our result with the inclusion and exclusion of this DDI.
For each physician we calculated the ratios (rP) of the number of patients with a potential DDI (nP) divided by the number of patients at risk (ΩP). Thus, we formed for each physician a reference population, that allowed to calculate the proportion of patients with DDI at the physician level. Our model is based on the assumption that populations at risk for drug interactions (ΩP) taken care by a given physician are comparable. Given the variety of ratios for DDI across specialities we decided to compare DDI only within specialties. The distribution of the ratios rP was modelled as a zero inflated binomial distribution (with the two coefficients πzero, πbin) to correct for the inflated number of observations with zero DDI in our dataset. The underlying assumption is that with probability πzero, the ratio rP is 0, and with probability 1 – πzero, a binomial random variable (πbin) is observed. Thus, the coefficients πzero, corrects for the overrepresentation of physicians with no patients exposed to DDI and is expressed as probability between 0 and 1. The higher the coefficient (πzero) the higher is the probability that zero patients of physicians from a given specialty have been exposed to a potential DDI. The coefficient πbin represents the proportions of patients per physician exposed to potential DDI if the outcome variable rP follows with probability 1 – πzero a binomial distribution.
For each physician, we calculated the probability of being at the upper tail of the zero inflated binomial distribution. If the probability was <0.05 and ≥0.01 we classified the prescription behaviour of a physician to cause a potential DDI as ‘unlikely to be due to chance’, and if the probability was <0.01 we regarded the prescription behaviour to cause a potential DDI as ‘very unlikely to be due to chance’.
In a pre-processing step, we imported data from the insurance companies into a MySQL database. All analyses were carried out using R version 3.0.1 (R Foundation for Statistical Computing, Vienna, Austria). The parameters for the zero inflated binomial distribution were estimated with the R-library Vector Generalized Linear and Additive Models VGAM .
The analysis was based on 282,780,644 claim data entries and 32,817,874 prescriptions from three major health insurers covering 3,132,540 individuals (40% of the Swiss population) with known Swiss residence in 2010. Of claims processed by these insurers 5.4% contained insufficient information of prescribed drugs and therefore could not be included. Among 20,710 registered physicians 19,630 prescribed at least one drug. The median number of patients per physician in the study population was 290, [Interquartile range (IQR) 23 to 418]. Sixty percent of all drugs were prescribed by 7845 general practitioners to 1’567’000 patients; of these, 48% of prescribed drugs were self-dispensed by the doctor without involving a pharmacy. In the study population the number of individuals aged 70 years and older was overrepresented (14.7%) when compared with the general Swiss population (11.9% aged 70 years or older), as were individuals from the French speaking part of Switzerland (16.4% study population, 18.8% Swiss population).
Of 1'607'233 females and 1'525'307 males 1.3% of females and 1.2% of males received a drug combination which is contraindicated or potentially contraindicated, and 0.4% of females and 0.5% of males received a drug combination which is contraindicated (Table 1). The rate for exposure to at least one contraindicated drug combination was slightly higher for females than for males, and for individuals aged 70 or older of both sexes (Table 1).
Based on our probability definitions, 416 of 18,297 (2.3%) physicians prescribed drugs leading to potential DDI in a frequency that is very unlikely (p<0.01) to be explained by chance and 598 physicians (3.3%) prescribed drugs leading to potential DDI in a frequency unlikely (p<0.05 p≥0.01) to be explained by chance (Table 2). The probability to cause DDI differed substantially between specialties. For example the coefficient to account for the probability of no DDI (πzero) from the binominal distribution was high for ophthalmologist (0.72) indicating that a high proportion of ophthalmologists did not cause any DDI whereas the coefficient in general practitioners (0.20) indicated that the proportion of general practitioners causing no DDI was substantially lower. General practitioners (0.20), cardiologist (0.26), physicians in group practices (0.30) and oncologists (0.37) showed the lowest coefficient of πzero indicating that in these specialities there were the lowest number of patients with no DDI. When we excluded DDI by clopidogrel and proton pump inhibitors the coefficient increased in particular for cardiologist (0.34) indicating that a considerable proportion of DDIs caused by these specialists is due to this drug combination.
The probabilities for DDI were highest for general practitioners (πbin 0.03), oncologists (πbin 0.02), other physicians (πbin 0.02), and psychiatrists (πbin 0.02). Although the coefficients πbin for general practitioners and oncologists are not considerably different, the coefficients of πzero for these two disciplines indicate that probability of patients cared by an oncologist not to be exposed to a DDI is about 50% higher than of patients served by a general practitioner.
The delivery date of interacting drugs occurred for 28.7% of events on the same day, and in 78.9% within 30 days. Overall the median number of patients at risk (ΩP) encompassed 30% [IQR 26%-46%] of the total number of patients of a physician. Most common contraindicated DDI (expressed per 10,000 person years) were prescriptions of macrolide antibiotics and HMG CoA reductase inhibitors, potassium sparing diuretics and potassium salt, and antiarrhythmic drug combinations of class I and class III (Fig 1).
Most common potentially contraindicated DDIs were proton pump inhibitors with clopidogrel, cytochrome 3A4 inhibitors (such as protease inhibitors, azole antimycotics, erythromycin, clarithromycin or nefazodone) with salmeterol, and St. John's wort and serotonin reuptake inhibitors (Fig 2).
Fig 3 depicts common DDI by specialists. The most common potential DDI involved clopidogrel with proton pump inhibitors, HMG CoA reductase inhibitors (statins) with macrolide antibiotics, serotonin reuptake inhibitors and St. John's wort, clopidogrel with cytochrome P450-2C19 inhibitors, salmeterol with cytochrome P450-3A4 inhibitors, and potassium salt together with potassium saving diuretics. For each of these DDI the rate of general practitioners with drug prescriptions leading to potential DDI unlikely (p<0.05 & p ≥0.01) and very likely to be explained by chance was higher than in specialists.
This study investigated for the first time the crude prevalence of potential DDI in a large sample of approximately 40% of the Swiss population. The findings are consistent with results from population-based studies from Scandinavia which also reported a higher frequency of DDI in females and in older patients [9–11]. However, comparisons across studies are somewhat difficult as definitions of DDI differed across various observational investigations. In two Swedish population-wide studies the prevalence of DDI with serious clinical consequences during the course of one year was 1.4%, thus closely similar to our estimates, but the prevalence in female elderly individuals was 50% higher than in our study [9, 10]. In a population-based study in one county in Denmark the prevalence of major DDI was 1.9% .
The goal of this study was to investigate whether monitoring of DDI and tracking of potential DDI is possible on the individual physician level. Due to the large sample we were able to identify in each specialty group and for all clinically relevant contraindicated DDI physicians with an unusual prescription behaviour unlikely to be explained by chance. The detailed information in the database further allowed us to analyse for all patients the entire sequence of all prescribed drugs and to exactly determine which physician was responsible for causing a particular potential DDI. This may be seen as a particular strength of the study.
The odds for DDI vary by specialty because the type of drugs that may interact with other drugs may be related to the speciality of a physician and to the comorbidities of the served patient population. Social health insurers in Switzerland do not receive detailed diagnosis-related information from ambulatory care. For this reason we lacked relevant data on diagnoses and comorbidities which can only indirectly and rather rudimentary be assessed from pharmaceutical drug groups, for example from drugs used to specifically treat diabetes, specific neurological diseases, or COPD. For this reason we chose to restrict for each physician the study population to patients who were at least once prescribed a base drug or a counteracting drug from the predefined list of 40 DDI. We were also missing information on treatment indication and duration, dosing and refill. Therefore we had to approximate drug use periods with fixed assumptions of dosage and refill length using DDD. This approach has several disadvantages and does not consider different dosing in particular for children. The refill length was assigned as the typical duration for a single purchase which may not reflect shorter drug prescription periods. We could not use a more refined method to estimate drug use periods from historical purchase data because such models depend on the number of purchases and longer observation periods.
Our intention was to investigate whether Swiss health insurer claim data can be used to monitor and track individual physicians’ prescription behaviour. Therefore it was not our intention to use a hierarchical logistic regression model including, patient age, gender and co-medication that would have allowed to adjust for risk factors known to be associated with DDI. From an epidemiological perspective this might be seen as a disadvantage. However, from a health system surveillance perspective our approach is sensitive because it allows for the identification of physicians with odd prescription behaviours that distinguishes them from their peers within a specialty. Given the lack of any information on diagnosis and comorbidity in Swiss claim data our approach seems sensitive. For these reasons we formed a reference population for each physician, which allowed to determine the accumulation of DDI at the physician level unlikely to be explained by chance. Our model is based on the assumption that populations at risk for a given drug interactions (ΩP) and being served by a given physician are comparable. This assumption might not reflect the heterogeneity of patient populations served by physicians. In addition, due to variation of odds across specialities DDI should only be compared within specialties which from a health system perspective might be seen as a disadvantage.
Our study has several additional limitations. The proportion of elderly patients in the study population was higher than in the general population. Due to the preponderance of these age groups we probably overestimated the prevalence of DDI in the general population to some degree. Patients from the south-east and French speaking part of Switzerland were underrepresented. In addition, patients with high deductibles can pay their drugs themselves and may not hand in drug prescriptions to their health insurance. However, internal analyses from health insurer prescription data indicate that less than 2–3% of prescriptions get thereby lost to the analyses, and therefore it is highly unlikely that many DDI have been missed . In addition, patients with potential DDI are more likely to have annual costs above the deductible and will hand in their invoices. In this study we cannot account for over-the-counter drugs potentially leading to DDI. Furthermore, our analyses are based on the assumption that patients truly took all prescribed drugs, which is certainly not always the case. Our estimates relate to a 12 month period, however, large variations over time in prescription behaviour leading to DDI have been found by others. This might dilute our power to detect unusual prescription behaviour compared to studies with longer monitoring time periods .
Many of the contraindicated and potential DDI from our list may not be clinically relevant and can be justified with appropriate monitoring or dose adjustment. Claims data from Swiss health insurers do not encompass the necessary details which would allow to investigate or to verify the appropriateness of committed DDI. Likewise it is not possible to investigate from Swiss claims data whether potential DDI actually led to an adverse drug reaction with serious consequences and need for hospitalisation. At present no linkage with the hospitalisation statistics of Swiss DRG is possible. Given the expected low number of such events caused by an individual physician it is unlikely that adverse drug reactions might form a suitable parameter for judging the quality of a physicians’ prescription behaviour. Given the high prevalence of potential DDI in particular in the elderly population and incidence of hospital admission due to adverse drug events, intervention studies to lower potential DDI in the elderly population at large might appear useful from the public health and system perspective.
Evidence from randomized controlled intervention trials on the effectiveness of computerized decision support systems for the reduction of DDI in ambulatory care is very limited . In a Canadian trial 107 general practitioners from Montreal were assigned to a computerized decisions support system versus a control group . During the one year intervention period the number of discontinued drugs due to DDI nearly doubled in the intervention group compared to controls (165 versus 76 drug discontinuations per 1000 visits, relative risk 2.15, 95%CI 0.98–4.70). Preliminary evidence from a large pre- and post-intervention study with an automated DDI alert system for electronic prescribing indicated that providers stopped in a substantial proportion the prescription of 18 high-volume and high risk DDI medications (5.9% versus 10.9%), leading to a statistically non-significant decrease of definitive or probable adverse drug reactions as assessed with the Naranjo scoring system . Estimates from a simulation study in Massachusetts, which was based on roughly 280,000 prescription alerts of over 2000 clinicians in ambulatory care using the PocketScript e-prescribing system, indicated that 402 adverse drug events during one year could have been averted if clinicians followed the recommendation of the e-alert system . Of those, 49 were estimated to be serious and even life threatening events, and 3 leading to death. According to the model, 2715 and 44,350 alerts would have had to be checked to avert one serious adverse event and one death, respectively.
The number of physicians in ambulatory care in Switzerland with an unusual level of DDI would be sufficiently large to test whether the provision of an electronic alert system could reduce the number of DDI. Health insurer claims data would allow to identify individual physicians through the unique physician identification number and to contact and motivate them to participate in such a study. The obvious advantage of such a study would be the high external validity of findings. Future linkage to outcome data would allow assessing the impact of such an intervention on the prevention of serious adverse drug reactions, the most relevant patient important outcome.
This study was supported by a grant from the Swiss Medical Association (FMH) and Santésuisse. The Basel Institute for Clinical Epidemiology has been supported by unrestricted research grants by Santésuisse and the Gottfried and Julia Bangerter-Rhyner-Foundation. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Conceived and designed the experiments: HCB RA NS CRM. Analyzed the data: RA. Contributed reagents/materials/analysis tools: HCB RA NS CRM. Wrote the paper: HCB RA NS CRM.
- 1. Hammerlein A, Griese N, Schulz M. Survey of drug-related problems identified by community pharmacies. Ann Pharmacother 2007 Nov; 41(11):1825–32. pmid:17925500
- 2. Becker ML, Kallewaard M, Caspers PW, Visser LE, Leufkens HG, Stricker BH. Hospitalisations and emergency department visits due to drug-drug interactions: a literature review. Pharmacoepidemiol Drug Saf 2007 Jun; 16(6):641–51. pmid:17154346
- 3. Juurlink DN, Mamdani M, Kopp A, Laupacis A, Redelmeier DA. Drug-drug interactions among elderly patients hospitalized for drug toxicity. JAMA 2003 Apr 2; 289(13):1652–8. pmid:12672733
- 4. Hanlon JT, Schmader KE, Samsa GP, et al. A method for assessing drug therapy appropriateness. J Clin Epidemiol 1992 Oct; 45(10):1045–51. pmid:1474400
- 5. Spinewine A, Schmader KE, Barber N, et al. Appropriate prescribing in elderly people: how well can it be measured and optimised? Lancet 2007 Jul 14; 370(9582):173–84. pmid:17630041
- 6. Vonbach P. Drug-Drug Interactions in the Hospital (Thesis) University of Basel; 2007.
- 7. Suzuki Y, Suzuki H, Umetsu R, et al. Analysis of the Interaction between Clopidogrel, Aspirin, and Proton Pump Inhibitors Using the FDA Adverse Event Reporting System Database. Biol Pharm Bull 2015; 38(5):680–6. pmid:25947914
- 8. VGAM: Vector Generalized Linear and Additive Models. R package version 0.8–5. [computer program]. 2014.
- 9. Merlo J, Liedholm H, Lindblad U, et al. Prescriptions with potential drug interactions dispensed at Swedish pharmacies in January 1999: cross sectional study. BMJ 2001 Aug 25; 323(7310):427–8. pmid:11520839
- 10. Johnell K, Fastbom J. Multi-dose drug dispensing and inappropriate drug use: A nationwide register-based study of over 700,000 elderly. Scand J Prim Health Care 2008; 26(2):86–91. pmid:18570006
- 11. Bjerrum L, Andersen M, Petersen G, Kragstrup J. Exposure to potential drug interactions in primary health care. Scand J Prim Health Care 2003 Sep; 21(3):153–8. pmid:14531506
- 12. Tanskanen A, Taipale H, Koponen M, et al. From prescription drug purchases to drug use periods—a second generation method (PRE2DUP). BMC Med Inform Decis Mak 2015; 15(1):21.
- 13. Achermann R. Abrechnungseffekte bei der Umstellung von Tiers Garant (TG) zu Tiers Payant (TP). In: Helsana IS, ed. 2006.
- 14. Solberg LI, Hurley JS, Roberts MH, et al. Measuring patient safety in ambulatory care: potential for identifying medical group drug-drug interaction rates using claims data. Am J Manag Care 2004 Nov; 10(11 Pt 1):753–9. pmid:15623265
- 15. Wolfstadt JI, Gurwitz JH, Field TS, et al. The effect of computerized physician order entry with clinical decision support on the rates of adverse drug events: a systematic review. J Gen Intern Med 2008 Apr; 23(4):451–8. pmid:18373144
- 16. Tamblyn R, Huang A, Perreault R, et al. The medical office of the 21st century (MOXXI): effectiveness of computerized decision-making support in reducing inappropriate prescribing in primary care. CMAJ 2003 Sep 16; 169(6):549–56. pmid:12975221
- 17. Steele AW, Eisert S, Witter J, et al. The effect of automated alerts on provider ordering behavior in an outpatient setting. PLoS Med 2005 Sep; 2(9):e255. pmid:16128621
- 18. Weingart SN, Simchowitz B, Padolsky H, et al. An empirical model to estimate the potential impact of medication safety alerts on patient safety, health care utilization, and cost in ambulatory care. Arch Intern Med 2009 Sep 14; 169(16):1465–73. pmid:19752403