Impact of a Prospective Audit and Feedback Antimicrobial Stewardship Program at a Veterans Affairs Medical Center: A Six-Point Assessment

Background Prospective audit and feedback is a core antimicrobial stewardship program (ASP) strategy; however its impact is difficult to measure. Methods Our quasi-experimental study measured the effect of an ASP on clinical outcomes, antimicrobial use, resistance, costs, patient safety (adverse drug events [ADE] and Clostridium difficile infection [CDI]), and process metrics pre- (9/10–10/11) and post-ASP (9/12–10/13) using propensity adjusted and matched Cox proportional-hazards regression models and interrupted time series (ITS) methods. Results Among our 2,696 patients, median length of stay was 1 day shorter post-ASP (5, interquartile range [IQR] 3–8 vs. 4, IQR 2–7 days, p<0.001). Mortality was similar in both periods. Mean broad-spectrum (-11.3%), fluoroquinolone (-27.0%), and anti-pseudomonal (-15.6%) use decreased significantly (p<0.05). ITS analyses demonstrated a significant increase in monthly carbapenem use post-ASP (trend: +1.5 days of therapy/1,000 patient days [1000PD] per month; 95% CI 0.1–3.0). Total antimicrobial costs decreased 14%. Resistance rates did not change in the one-year post-ASP period. Mean CDI rates/10,000PD were low pre- and post-ASP (14.2 ± 10.4 vs. 13.8 ± 10.0, p = 0.94). Fewer patients experienced ADEs post-ASP (6.0% vs. 4.4%, p = 0.06). Conclusions Prospective audit and feedback has the potential to improve antimicrobial use and outcomes, and contain bacterial resistance. Our program demonstrated a trend towards decreased length of stay, broad-spectrum antimicrobial use, antimicrobial costs, and adverse events.


Introduction
Antimicrobial resistance is one of the greatest public health threats worldwide. [1] In the United States (US), the Obama Administration recently identified antimicrobial resistance as a national security issue. [2] Infections with antimicrobial-resistant bacteria and Clostridium difficile lead to increased morbidity, mortality, longer hospital stays, and dramatically increased healthcare costs. [3][4][5] The Centers for Disease Control and Prevention estimated that in 2013, antimicrobial-resistant organisms caused two million infections and 23,000 deaths in the US, with an additional 14,000 deaths due to C. difficile infection (CDI). [1] In the US, resistant infections are responsible for $20-35 billion in excess healthcare costs each year. [1] The driving forces that select for antimicrobial-resistant bacteria and promote CDI are antimicrobial use and suboptimal infection control practices. While some cases of CDI are not associated with prior antibiotic use and many other risk factors for CDI exist, including advanced age and protein pump inhibitor use, antibiotic use remains the most important risk factor for the development of CDI. [6] Given that over 50% of antimicrobial use in hospitals may be inappropriate, antimicrobial stewardship interventions (coordinated strategies to improve antimicrobial use) are critically important. [7] The 2007 Infectious Diseases Society of America and Society for Healthcare Epidemiology of America guidelines for developing an antimicrobial stewardship program (ASP) have recognized prospective audit of antimicrobial use and feedback back to the prescriber and formulary restriction as two core strategies that provide the foundation of an ASP. [7] Literature supports the effectiveness of prospective audit and feedback, however measuring the impact of these programs has been difficult. [8][9][10][11][12][13][14][15][16][17] Therefore, the purpose of this study was to conduct a broad evaluation of a prospective audit and feedback ASP on the following six measures: 1) clinical outcomes, 2) antimicrobial utilization, 3) costs, 4) resistance, 5) patient safety (adverse drug events [ADE] and CDI), and 6) process metrics. To our knowledge, our study is one of the first published studies to provide a comprehensive six-point assessment on the impact of an ASP.

Methods
We conducted a single-center quasi-experimental study. Study metrics were compared pre-(9/ 2010-10/2011) and post-ASP (9/2012-10/2013). The study protocol was approved by the Institutional Review Board and the Research (IRB) and Development Committee of the Providence Veterans Affairs Medical Center (PVAMC). The PVAMC IRB specifically waived the need for written informed consent for this retrospective study as it met the requirements of 38 CFR 16.116(d).

Intervention
The PVAMC is a Veterans Affairs (VA) teaching hospital licensed for 119 beds. In September 2012, the PVAMC invested in and implemented a formal ASP. Prior to formal introduction, the program was pilot tested for~18 weeks between 10/2011-4/2012. After that time the PVAMC funded a new ID fellowship position for pharmacists focusing in ASP. The new ID pharmacist fellow began in July 2012 and spent the next two months writing the policy and getting it approved by hospital administration. The ID pharmacist fellow began prospective audit and feedback in September 2012. A second new ID pharmacist fellow joined the team in July 2013. The core members of the program included the co-directors (a board certified infectious diseases [ID] attending physician and a clinical pharmacist with formal ID fellowship training), two other board certified ID attending physicians, two ID pharmacist fellows, and when on rotation, ID physician fellows (~6 months), PGY-1 pharmacy practice residents and APPE students (~9 months). Infection control practitioners, microbiology laboratory personnel, and an epidemiologist supported the core team. The main strategy implemented by the ASP was prospective audit and feedback. Since formal introduction, core team members have provided prospective audit and feedback for every patient admitted with active antimicrobial orders (Monday-Friday).
The on-service ID pharmacist fellow manually reviewed a list of all active antimicrobial orders daily. The list was generated in the morning and all active orders were reviewed with no restrictions for how long the patient was on the antibiotic before review. Each antimicrobial order was reviewed for appropriateness. Appropriateness was determined by the ID pharmacist fellow, who reviewed each order to make sure the correct drug, dose, duration, and/or route were used. The ID pharmacist fellow also ensured there was an indication for the antimicrobial order. No single definition for appropriateness was instituted, however the ID pharmacist fellow utilized institutional guidelines (PVAMC Antimicrobial Treatment Guidebook) professional society guidelines, expert opinion of the ASP core members (pre-rounding with an ID physician and/or the senior clinical pharmacist), and local and regional resistance patterns to determine appropriateness. [7] The PVAMC has published an annually updated Antimicrobial Treatment Guidebook since 2004, which contains empiric treatment guidelines, dosing recommendations, infection control policies, and an antibiogram of antimicrobial resistance rates. Additionally, a pre-designed decision-support template was used to collect and organize pertinent clinical data for ASP interventions (Fig 1). Other antimicrobial stewardship principles such as intravenous (IV) to oral (PO) conversion, de-escalation of empiric therapy based on culture results, and antimicrobial optimization were used to make recommendations to improve "appropriateness". [7] Antimicrobial optimization involved recommendations to improve the drug, dose, or duration of the antimicrobial based on patient characteristics, causative organism, site/type of infection, and pharmacokinetic/pharmacodynamics characteristics. Potential interventions were then relayed to the on-service ID physician and/or the senior clinical pharmacist. These "ASP rounds" were conducted daily and generally ranged from 15-60 minutes.
After discussing patients and interventions, verbal communication (telephone and in-person) and/or written notes in the electronic medical record (EMR) were used to relay interventions to the provider. The mode of communication (verbal or written by physician, pharmacist, or pharmacy resident/student) depended on the type of intervention that was needed. The specific intervention also dictated who made the intervention (physician, pharmacist, or pharmacy resident/student). For example, for a simple IV to PO antimicrobial conversion (e.g. IV to PO ciprofloxacin), a pharmacy student or resident may have written a draft note. However, discontinuation of an antimicrobial in a complex patient may have necessitated a phone call to the primary team by the on-service ID physician. To alert the provider (usually the medical resident) of the note, they were added as co-signers. The ID pharmacy fellow fully reviewed and signed-off on all notes written by residents and students before they were incorporated into the EMR. Additionally, the ID pharmacy fellow alerted the on-service ID physician to all written notes for review and co-signature.

Process Metrics
During the post-study period, the on-service ID pharmacy fellow documented all patients that were reviewed by the ASP in an excel database. Variables collected included admission date, treating specialty, antimicrobial indication, time-spent, and whether an intervention was made. If an intervention was made, the pharmacy fellow documented the type of intervention made, the stewardship team member who made the intervention, intervention acceptance or nonacceptance, and reasons for non-acceptance. Acceptance or non-acceptance was qualified as a dichotomous variable for each recommendation made. Interventions were categorized as follows: vancomycin dosing or therapeutic drug monitoring, antimicrobial discontinuation, IV to PO conversion, de-escalation, antimicrobial optimization (i.e. change to optimize the antimicrobial drug, dose, or duration), antimicrobial discontinuation, or other.

Clinical Outcomes
Clinical outcomes were compared between patients pre-and post-ASP. We identified all hospital inpatients with antimicrobials administered during the pre-and post-ASP periods. [18] Patients with a long-term stay ( 90 days) were excluded. Inpatient antimicrobial administrations were captured using patient barcode medication administration (BCMA) data. Outcomes included time to hospital discharge (length of stay [LOS]), 7-, 14-, and 30-day all-cause mortality, inpatient all-cause mortality, and 30-day readmission. The index date for hospital discharge, 7-, 14-and 30-day mortality, and inpatient mortality was the date of antimicrobial initiation and for 30-day readmission was the date of hospital discharge. We calculated the time from the index date to the date of event for each outcome. Patients were censored on their date of death.
We determined demographics, comorbid conditions, and health-care exposures from the national VA standardized databases which contain ICD-9 diagnostic and procedure codes, vital status, microbiology results, barcode medication administration, and laboratory results.

Antimicrobial Utilization
Antimicrobial utilization was compared pre-and post-ASP. The antimicrobial utilization metric used was days of therapy per 1,000 patient days (DOT/1000PD) based on inpatient medication administration data. [19,20] We assessed overall antimicrobial use, as well as specific categories of use by route, agent, class, and spectrum. [21] Antimicrobial Costs Antimicrobial costs were estimated using the Average Wholesale Price. The cost metric used was cost per 1,000 patient-days. Overall costs and costs for specific antimicrobial categories described above were compared pre-and post-ASP.

Patient Safety
Monthly episodes of CDI per 10,000 patient-days were compared pre-and post-ASP. CDI episodes were obtained from VA Inpatient Evaluation Center (IPEC) data. [22,23] Rates of ADEs among hospital inpatients with antimicrobial administrations were compared pre-and post-ASP. ADEs were identified using ICD-9 codes for adverse effects of drugs.

Statistical Analysis
All analyses were performed using SAS (SAS Institute Inc., Cary, NC, Version 9.2).
Process metrics. We used descriptive statistics, including means and percentages, to summarize the data.
Clinical outcomes. Baseline differences between patients in the pre-and post-ASP periods were assessed using Fisher's exact or χ 2 tests (categorical data), and a t-test or Wilcoxon Rank Sum test (continuous data), as appropriate. Propensity score adjustment and matching was implemented to balance differences between patients in the pre and post-ASP periods. [24,25] Propensity scores were developed from an unconditional logistic regression model (manual backward elimination). Hazards ratios comparing clinical outcomes in post-ASP patients to pre-ASP patients were calculated from Cox proportional-hazards regression models.
Antimicrobial utilization. T-tests were used to compare mean DOT/1000PD pre-and post-ASP. We utilized interrupted time series (ITS) methods to assess the impact of ASP on monthly antimicrobial utilization. Segmented linear regression models were used because they can tolerate fewer time points than autoregressive integrated moving average models. [26,27] We tested for autocorrelation using the Durbin-Watson statistic, and for seasonality/stationarity using the Dickey-Fuller unit root test. [27,28] Estimates for regression coefficients corresponding to the effect sizes of a change in level and a change in trend for post-to pre-ASP were obtained. A change in level was defined as the difference between the observed level immediately post-ASP and the predicted level by the pre-ASP trend. A change in trend was defined as the difference between the pre and post-ASP slopes.
Antimicrobial costs. T-tests were used to compare mean costs/1000PD pre-and post-ASP. Segmented linear regression models were utilized to model temporal trends in monthly antimicrobial costs.
Antimicrobial resistance. We used Fisher's exact or χ 2 tests, as appropriate, to compare the number of resistant and susceptible isolates for select organism-antimicrobial combinations pre-and post-ASP.
Patient safety. A t-test was used to compare mean CDI rate/10,000PD and the χ 2 test was used to compare ADEs pre-and post-ASP. Segmented linear regression models were utilized to model temporal trends in monthly CDI rates.

Process Metrics
During the post-ASP period, we reviewed 1,049 patient charts. Interventions were made in 36.7% of patients reviewed. The most common interventions made were antimicrobial optimizations, IV to PO conversions, and discontinuations (Fig 2). Among the patients with an intervention, interventions were most often (88.3%) made through a written note in the patients' EMR. The on-service pharmacy fellow made the intervention in almost half of the patients who needed an intervention (47.8%). Overall, 522 interventions were made with an overall acceptance rate of 77.2%. The most common reasons for non-acceptance, were that the primary team never viewed the recommendation (29%) or that the antimicrobial was changed/ discontinued (14%) and therefore the recommendation was no longer applicable.

Clinical Outcomes
We identified 2,696 patients treated with antimicrobials in the pre-(49.0%, n = 1,321) and post-ASP (51.0%, n = 1,375) periods. The median patient age was~70 years in both groups ( Table 1). The median Charlson (2 vs. 4) and Elixhauser (3 vs. 5) scores were higher for patients post-ASP (p<0.001). History, in the year prior to the antibiotic-related admission, of diabetes, congestive heart failure, myocardial infarction, chronic respiratory disease, and chronic renal disease were more common among post-ASP patients. Diagnoses of cellulitis, osteomyelitis, and influenza during the current admission were more common for patients post-ASP (Table 1). More post-ASP patients were hospitalized in the 90 days prior to admission than pre-ASP (Table 2). Despite these differences between pre-and post-ASP patients, we were able to balance significantly different baseline characteristics using propensity scores.

Antimicrobial Costs
Total antimicrobial costs decreased 14% pre-to post-ASP, with a non-significant 5.3% decrease in mean antimicrobial costs/1000PD (p = 0.5). The cost for fluoroquinolones decreased 29% pre-to post-ASP (p<0.05). IV (-4.2%), digestive (-7.6%), and broad-spectrum (-9.5%) costs all decreased non-significantly.  ITS demonstrated several significant increases in antimicrobial costs immediately following the implementation of ASP. While the level of anti-CDI, anti-anaerobic, and broad-spectrum costs increased, this increase was not sustained during the post-ASP period.

Antimicrobial Resistance
No significant changes in antimicrobial resistance were observed for any of the Gram-positive or Gram-negative organism-antimicrobial combinations assessed (Table 6), except for Klebsiella pneumoniae, in which several significant (p<0.05) increases in resistance were observed.

Patient Safety
The mean rate of CDI/10,000PD was 14.

Discussion
Currently, there is no consensus on which metrics are the most optimal to adequately assess the impact of an ASP. [29] Our study provides a detailed assessment of the impact of an ASP on clinical outcomes, antimicrobial utilization, costs, resistance, patient safety, and process  Streptococcus species culture, current rheumatoid arthritis, current valvular disease, current Elixhauser score, creatinine, days of antimicrobial therapy, ethnicity, current beta-lactam/ beta-lactamase inhibitor exposure, current anti-influenza drug exposure, current fluoroquinolone exposure, current metrics. Due to the challenges associated with outcomes assessment, most studies to date have focused on measuring the impact of an ASP on just one or two metrics, most commonly antimicrobial utilization and costs. While median LOS was 1 day shorter post-ASP, this difference was not statistically significant in propensity matched or adjusted analyses. Despite patients being generally sicker post-ASP (higher Charlson and Elixhauser scores and higher prevalence of several comorbidities), ASP interventions may have led to improved quality of care, enabling patients to be discharged sooner. Nonetheless, in general, ASP implementation had a limited impact on the clinical outcomes assessed. These findings are similar to most studies, which have demonstrated little to no impact of prospective audit and feedback ASPs on clinical outcomes, including LOS, [8][9][10][11][12][13][14][15][16][17] mortality, [8][9][10][12][13][14][15][16][17] and 30-day readmission. [10,14] This may be because, a large number of factors affect clinical response and outcomes, and therefore the independent effect of ASP Data are mean ± standard deviation or % change. The DOT represents the sum of the days for which a single antimicrobial was administered, regardless interventions on these outcomes may be negligible. [29] Additionally, while in adjusted analyses 30-day mortality was higher post-ASP, this included deaths due to all-causes. The Centers for Medicare and Medicaid Services (CMS) 30-day risk standardized mortality rates for congestive heart failure at the Providence VA Medical Center were higher during the post-ASP period than the pre-ASP period. [30]. Also, antimicrobial stewardship interventions are likely to have    a greater impact on 7-and 14-day mortality and inpatient mortality, which did not differ between periods. We also measured the effect of our ASP on antimicrobial resistance. In another study, reduction of broad-spectrum antimicrobial use was not associated with improvements in the hospital antibiogram. [31] As with clinical outcomes, the factors associated with antimicrobial resistance are complex and involve many factors such as infection control, antimicrobial use within and outside the hospital, and patient colonization and immune status. Therefore, it can be challenging for an ASP to demonstrate a favorable impact on antimicrobial resistance. [32] Moreover, it can take years before a program has an effect on antimicrobial resistance.
In our assessment of antimicrobial use, we did not observe a decrease in overall mean antimicrobial use, which may be related to the appropriateness of antimicrobial utilization prior to implementation of our ASP. It is estimated that 50% of antimicrobial use in hospitals is inappropriate. [7] However, in our study, only 37% of patient records reviewed were deemed to require intervention. Since 2004, a clinical pharmacist with formal training in infectious diseases has provided the PVAMC expert consultation, an antimicrobial guide with empiric treatment recommendations and an antibiogram, and educational programs. Additionally, several broad-spectrum antimicrobials have been restricted since before the implementation of our ASP. Therefore, at baseline appropriate antimicrobial use at the PVAMC may have been relatively high. Though overall use did not decrease, we did see significant reductions in broad-spectrum, fluoroquinolone, and anti-pseudomonal use post-ASP. Our ASP improved the use of these broad-spectrum antimicrobials, through appropriate antimicrobial de-escalation and optimization. We also observed a reduction in mean carbapenem use post-ASP, however ITS demonstrated an increasing trend in carbapenem use. This highlights the importance of conducting ITS analysis to uncover immediate and sustained changes in outcome measures over time. This increasing trend in carbapenem use may be due in part to rotating medical residents. At the PVAMC, residents are the primary antimicrobial prescribers, and they rotate out of the PVAMC to other local hospitals every month. At the time of this study, the PVAMC had the only comprehensive multidisciplinary ASP in the area. Moreover, there was no formal ASP at the flagship hospital that the residents rotate through. Therefore residents may have not been used to the ASP service. In a recent study, investigators demonstrated an improvement in the level of audited antimicrobials but no change in the trend, which was also likely due to residents changing to different departments or institutions frequently. [15] Monthly introductions of the house-staff and new medical residents to our ASP and other educational material such as newsletters or posters, may increase residents' awareness and connection to our service, and improve the ASP culture at the PVAMC. The increasing trend in carbapenem use may also be related, in part, to the significant increases in resistance observed for Klebsiella pneumoniae.
Measuring the impact of ASPs on patient safety is also important. Rates of CDI were similar pre-and post-ASP. This is not surprising, as CDI rates were already low prior to ASP implementation, likely due to strong infection control practices. Infection control has had guidelines for the prevention and control of CDI since before the pre-ASP period. Guidelines include barrier methods, contact precautions, hand hygiene, and environmental infection control methods. Additionally, we observed a trend towards decreased ADEs post-ASP. Due to the difficulties in obtaining accurate data, very few studies have assessed the impact of ASPs on ADEs. [33] Our ASP did not have a significant impact on the clinical outcome measures assessed. This may be due in part to the outcomes metrics chosen. As previously mentioned, the most optimal metrics to demonstrate the value of an ASP are largely unknown. [29] As we continue to strengthen our program, we look to assess additional metrics such as infection-related clinical outcomes and total costs of care, not just drug costs, and to assess the impact of these outcomes over a longer follow-up period. Additionally, almost 90% of our recommendations were made through written notes. Our feedback may have had a greater impact if it was provided through face-to-face communication or phone calls directly to the provider. Notes left in the chart are unlikely to be seen in a timely manner. [34] Busy providers may miss or ignore notes. Moreover, the impact of our program may be limited by the timeliness of final culture results. At the PVAMC, traditional microbiologic testing (culture and susceptibility) is primarily utilized, which is suboptimal in providing rapid organism identification and susceptibility results. [35] Previous research by our group has demonstrated that the median time to final culture results ranged from 3-5 days at our facility and regionally. Therefore, incorporation of rapid diagnostic testing (RDT) could significantly enhance the impact of our ASP. [35] RDT has the potential to improve clinical outcomes, costs, and resistance rates by decreasing the time to appropriate therapy and quickly stopping unnecessary therapy.
There are several limitations to our study. The quasi-experimental design is associated with a number of inherent limitations, including the potential for confounding bias. However, we did our best to control for differences between patients in the pre-and post-ASP periods through propensity score adjustment and matching. Still, differences in unmeasured factors may exist between the groups. We may not have been able to capture all residual confounding, and having a generally sicker population in the post-period may bias estimates of differences in clinical outcomes towards the null. Of note, while there were no outbreaks at the PVAMC in either period, the 2012-2013 influenza season started earlier in Rhode Island and was more severe than previous years (including the 2010-2011 season). [36,37] Significantly more patients in the post-ASP period had a diagnosis of influenza than in the pre-ASP period.
As with any study that utilizes secondary data sources, this study may be limited by the accuracy of the data contained within the various data sources. While we attempted to develop accurate definitions for outcomes and potential confounders, misclassification bias may still affect our results. However, the VA has used an electronic medical record for over 15 years, from which the VA research databases are extracted, and the accuracy and completeness of several VA datasets has been verified in previous studies. [38][39][40][41] It is unclear how long it takes for changes in antimicrobial utilization to subsequently impact resistance rates and clinical outcomes. Our study only assessed the first year postimplementation. Therefore, it is possible that we did not allow enough time to observe an effect, as it may take several years of follow-up. However, we utilized interrupted time series analysis which is the strongest approach to quantify the effects of an intervention over time for quasi-experimental studies. [27] Additionally, since few deaths occurred, we may not have been able to detect a difference between groups. Finally, we conducted a single center VA study and the generalizability of our study may be limited to the VA setting. VA patients tend to differ from the general population in terms of patient demographics and comorbidities, and the VA has unique resources, which may assist with ASP efforts. Nonetheless, our study could serve as an example to other burgeoning stewardship programs that are interested in analyzing the potential effectiveness of their interventions.

Conclusions
Our prospective audit and feedback program was associated with improvements in broad-spectrum antimicrobial use. While median LOS was shorter post-ASP, clinical outcomes were similar pre-and post-ASP. Resistance, costs and patient safety indicators did not significantly change, but these changes may have a positive impact long term. Further measures, such as increased use of RDT, increased direct verbal feedback, and additional outcomes metrics, may be necessary moving forward. Moreover, as our ASP has now been in effect for over three years, we look to continue to measure the sustained impact of our program over time.
Overall, prospective audit and feedback has the potential to improve antimicrobial use and outcomes, and contain bacterial resistance. Our program demonstrated a trend towards decreased length of stay, broad-spectrum antimicrobial use, antimicrobial costs, and adverse drug events. While these results were not statistically significant, we believe that these findings have important clinical impact to the care of our patients.