Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Predicting Antimicrobial Resistance Prevalence and Incidence from Indicators of Antimicrobial Use: What Is the Most Accurate Indicator for Surveillance in Intensive Care Units?

  • Élise Fortin,

    Affiliations Department of Epidemiology, Biostatistics and Occupational Health, McGill University, Montréal, Québec, Canada, Direction des risques biologiques et de la santé au travail, Institut national de santé publique du Québec, Québec and Montréal, Québec, Canada

  • Robert W. Platt,

    Affiliation Department of Epidemiology, Biostatistics and Occupational Health, McGill University, Montréal, Québec, Canada

  • Patricia S. Fontela,

    Affiliations Department of Epidemiology, Biostatistics and Occupational Health, McGill University, Montréal, Québec, Canada, Department of Pediatrics, The Montréal Children's Hospital, McGill University, Montréal, Québec, Canada

  • David L. Buckeridge,

    Affiliation Department of Epidemiology, Biostatistics and Occupational Health, McGill University, Montréal, Québec, Canada

  • Caroline Quach

    caroline.quach@mcgill.ca

    Affiliations Department of Epidemiology, Biostatistics and Occupational Health, McGill University, Montréal, Québec, Canada, Direction des risques biologiques et de la santé au travail, Institut national de santé publique du Québec, Québec and Montréal, Québec, Canada, Department of Pediatrics, The Montréal Children's Hospital, McGill University, Montréal, Québec, Canada

Abstract

Objective

The optimal way to measure antimicrobial use in hospital populations, as a complement to surveillance of resistance is still unclear. Using respiratory isolates and antimicrobial prescriptions of nine intensive care units (ICUs), this study aimed to identify the indicator of antimicrobial use that predicted prevalence and incidence rates of resistance with the best accuracy.

Methods

Retrospective cohort study including all patients admitted to three neonatal (NICU), two pediatric (PICU) and four adult ICUs between April 2006 and March 2010. Ten different resistance / antimicrobial use combinations were studied. After adjustment for ICU type, indicators of antimicrobial use were successively tested in regression models, to predict resistance prevalence and incidence rates, per 4-week time period, per ICU. Binomial regression and Poisson regression were used to model prevalence and incidence rates, respectively. Multiplicative and additive models were tested, as well as no time lag and a one 4-week-period time lag. For each model, the mean absolute error (MAE) in prediction of resistance was computed. The most accurate indicator was compared to other indicators using t-tests.

Results

Results for all indicators were equivalent, except for 1/20 scenarios studied. In this scenario, where prevalence of carbapenem-resistant Pseudomonas sp. was predicted with carbapenem use, recommended daily doses per 100 admissions were less accurate than courses per 100 patient-days (p = 0.0006).

Conclusions

A single best indicator to predict antimicrobial resistance might not exist. Feasibility considerations such as ease of computation or potential external comparisons could be decisive in the choice of an indicator for surveillance of healthcare antimicrobial use.

Introduction

Although the causal relationship between antimicrobial use and antimicrobial resistance is difficult to quantify due to the various settings and measures studied and to related biases, this relationship is generally accepted.[13] The European Surveillance of Antimicrobial Comsumption (ESAC) has shown that countries using antimicrobials more intensively tend to also present higher levels of resistance.[4] Considering that antimicrobial use is modifiable, surveillance of antimicrobial use is often recommended as a complement to surveillance of antimicrobial resistance in hospitals.[58]

In practice however, methodologies vary between networks and research teams. For surveillance of antimicrobial use, the World Health Organization recommends the use of defined daily doses (DDDs) per patient-days.[9] ESAC also measures hospital antimicrobial use in point prevalence surveys (proportion of patients receiving treatment), while the American National Healthcare Safety Network prefers agent-days (days of therapy [DOT]) per patient-days, among others.[6, 10, 11] Authors have suggested that the solution might reside in the monitoring of sets of indicators, but composition of these sets also varies: DDD and locally defined daily doses per patient-days[12]; daily doses per admissions and per patient-days[13, 14]; DOT, length of therapy (LOT) and the DOT:LOT ratio[15]. Although many authors have exposed either the limitations of different indicators, their own choice of indicator, or the necessity for more research to identify the most appropriate indicator(s) for surveillance of antimicrobial use, ultimately, very few published studies have actually compared these indicators’ ability to predict resistance levels in hospital settings.[1, 11, 1518]

Public health authorities or hospital epidemiologists wishing to develop a coordinated program devoted to the surveillance of hospital antimicrobial use have to identify one or a few of these indicators for their surveillance. However, the optimal way to measure antimicrobial use in hospital populations, to complement surveillance of resistance, is still unclear. Using respiratory isolates and antimicrobial prescriptions of nine intensive care units, and assuming a causal association between antimicrobial use and resistance, this study thus aimed to identify the indicator of antimicrobial use that predicted prevalence and incidence rates of resistance in ICUs with the best accuracy. Specifically, the objective was not, however, to demonstrate the existence of a causal association between antimicrobial use and resistance, nor was it to quantify such an association without bias.

Methods

This study was approved by the Research Ethics Boards of McGill University (study number A10-M105-10B) and of the Centre Hospitalier Universitaire Sainte-Justine (study number 3305). No consent from patients was necessary as the data was analyzed anonymously.

Study design and population

This was a retrospective cohort study on all patients admitted to ICUs of four hospitals located in Montreal, Canada, between April 1st, 2006 and March 31st, 2010. Participating ICUs included three neonatal ICUs (NICU), two pediatric ICUs (PICU) and four adult ICUs. The study design focused on recreating a surveillance context to identify the best indicator of antimicrobial use for surveillance activities, in this case, an ICU-based surveillance.

Admission–Discharge–Transfer (ADT) data were extracted for all ICU patients. Our data included unique identifying numbers, ICU identification and dates of admission and discharge to and from hospitals and ICUs. After a careful review, these data were used to compute numbers of ICU admissions (including transfers from other wards), numbers of patients present in the ICU and ICU patient-days, per 4-week period, for each ICU.[19]

Antimicrobial resistance

Using microbiology laboratory information systems, a database of bacteria isolated from ICU patients was built. Variables extracted were: unique identifying number, ICU identification, sampling site, sample collection date, identified microorganism, antimicrobial tested and resistance profile (susceptible, intermediate or resistant). This database was merged with ADT data, to link cultures with specific care episodes. Susceptibility tests performed on positive respiratory tract cultures were selected. We assumed that a large proportion of ICU patients were intubated at some point during their ICU stay and that respiratory cultures were done for intubated patients as part of the investigation for unstable ICU patients. This was thus an attempt to describe the respiratory microbiota, regardless of the presence of an infection. Intermediate strains were counted with susceptible as non-resistant strains. For the measurement of resistance prevalence and incidence, clinically relevant antimicrobial resistances (microorganism / antimicrobial combinations) were selected. Coliforms were analyzed as a group and included the following microorganisms: Enterobacter sp., Escherichia coli, Hafnia alvei, Klebsiella sp., Morganella morganii, Providencia rettgeri, Raoultella sp., Serratia sp. and microorganisms coded as “Coliforms”.

Based on the Society for Healthcare Epidemiology of America (SHEA) and Healthcare Infection Control Practices Advisory Committee (HICPAC) recommendations for metrics for multidrug-resistant organisms in healthcare settings, prevalence of resistance per 100 ICU admissions was measured to estimate exposure burden and incidence of resistance per 10,000 patient-days was also measured to quantify healthcare acquisition.[20] Prevalence of resistance was measured by counting the number of ICU admissions where a resistant strain of a given microorganism was isolated. Resistance was considered incident when a resistant microorganism was detected in a patient with a previously susceptible organism or with no positive culture at least 2 days after admission to ICU; patient-days were computed excluding the first 2 days after ICU admission, based on dates, as these patient-days had an event probability equal to zero. Incidence rates and prevalence were computed per 4-week period, for each ICU.

Antimicrobial use

Hospital pharmacy databases provided information on all prescriptions for antimicrobials issued for patients included in the study: unique identifying number, age, weight, antimicrobial prescribed, posology and prescription start and stop dates. These databases described prescribed drugs and not necessarily drugs administered to a patient. Only agents belonging to class J01 of the Anatomical Therapeutic Chemical (ATC) classification system (anti-infectives for systemic use) were kept for analysis.[21] Doses and days of treatment prescribed for use before or after ICU admission were excluded (as these would not be included in an ICU-based surveillance), but we included those used on the ICU admission or discharge dates, or in between; in contrast with resistance computations, the first two days of each stay were kept in measurements of antimicrobial use).

Population antimicrobial use was measured using fifteen different indicators. These indicators were obtained by combining five numerators (defined daily doses [DDDs], recommended daily doses [RDDs], agent-days, exposed patients, and number of courses) with three denominators (ICU patient-days, ICU admissions and ICU patients), all previously identified in a systematic review of indicators used for populations that included pediatric patients.[11] DDDs were computed dividing quantities prescribed by the standard values specified on the website of the ATC/DDD system.[21] RDD were computed similarly, but standard values for the computation of RDDs were based on doses recommended in the 2008 Sanford Guide, the 2012 Red Book, The Montreal Children’s Hospital drug formulary and Nelson 2012; pediatric patients’ weight was accounted for in RDD computations (RDDs in mg/kg).[2225] A table of standard values is provided as supporting information (S1 Table). Agent-days were the numbers of days when each specific antimicrobial was prescribed; in a combined therapy, each agent prescribed for a day counts for one agent-day. Exposure was the number of patients prescribed an antimicrobial agent, regardless of quantity or duration. Courses were the number of distinct periods of consecutive days when a patient was prescribed a specific antimicrobial. All ICU patient-days were included in the computation of denominators, as antimicrobial exposure was measured for the entire ICU stay; ICU admission and discharge day each counted for half a day. For a given 4-week period, “ICU admissions” only include patients admitted to the ICU during the period, while “ICU patients” include all patients present in the ICU at some point in time during the period. Indicators of antimicrobial use were computed per 4-week period, for each ICU.

Ten different resistance / antimicrobial use combinations were studied and are listed in the first column of Tables 1 and 2, for a total of 20 scenarios: 10 for prediction of prevalence and 10 for prediction of incidence rates. These combinations were selected based on the frequency of resistant strains in the Province of Quebec, and on their clinical relevance. In two combinations, use of three classes of antimicrobials was taken into account, as Staphylococcus aureus resistant to methicillin and Escherichia coli, Klebsiella sp. and Proteus sp. resistant to carbapenems can simultaneously present other resistances. All of these additional antimicrobials can thus also indirectly contribute to selection of targeted resistances.

thumbnail
Table 1. Most accurate, second most accurate and least accurate indicators in predicting resistance prevalence, for selected resistance /antimicrobial combinations.

https://doi.org/10.1371/journal.pone.0145088.t001

thumbnail
Table 2. Most accurate, second most accurate and least accurate indicators in predicting resistance incidence rates, for selected resistance /antimicrobial combinations.

https://doi.org/10.1371/journal.pone.0145088.t002

Statistical analyses

For each combination, scatterplots of resistance and antimicrobial use according to the different indicators were produced to visualize aggregation of data at the ICU level and time series were produced to see trends. After adjustment for ICU type, indicators of antimicrobial use were successively tested in regression models, to predict resistance prevalence and incidence rates, per 4-week time period, per ICU (data available in S2 Table). Binomial regression was used to model prevalence and Poisson regression, to model incidence rates. Multiplicative (log link) and additive (identity link) models were tested, [26] as well as no time lag and a one 4-week-period time lag; in total, for each scenario, 60 models were compared (15 indicators x 2 regression links x 2 time lags). As there were repeated measurements for every ICU (51 measurements with a 1-period time lag, 52 measurements without a time lag), generalized estimating equations were used to account for correlated values at the ICU level.

For each model, the mean absolute error (MAE) was computed. The MAE is a statistic used in the analysis of time series, to quantify the difference (or error) between observed values (prevalence or incidence rates) and values predicted by a model.[27] Predictive accuracy of different regression models can be compared using t-tests, to determine whether differences observed in predictions are statistically significant. Absolute values of these errors were computed per 4-week period and per ICU, and were then averaged, to produce the MAE of each model. The most accurate indicators were the ones with the smallest MAEs. MAEs were then compared using t-tests. In a given scenario, the most accurate model was compared to all other models (59 t-tests), beginning with the least accurate model. A Holm correction was applied to account for multiple comparisons, to keep an overall α of 0.05.[28] As an indication, when comparing the smallest MAE to the largest one, the t-test p-value had to be smaller than 0.0008 to reject the null hypothesis. Analyses were performed using SAS 9.3.

Results

MAEs for the most, second most and least accurate indicators, for each combination, are presented in Table 1 (prevalence) and Table 2 (incidence rates). As regression models were adjusted for ICU type, MAEs stratified per ICU type are also provided in S3 and S4 Tables. The most and least accurate indicators were usually not statistically different, except in the prediction of resistance prevalence in Pseudomonas sp. (cut-off value of 0.0008, given the Holm correction). When using carbapenem use to predict prevalence of carbapenem-resistant Pseudomonas sp., the indicator with the smallest MAE was courses per 100 patient-days (no time lag, using an identity link), which was significantly more accurate (p = 0.0006) than the least accurate model, RDD per 100 admissions (with a 1-period time lag, using a log link).

In regression models, additive models (identity link) frequently failed at producing coefficients and predicting prevalence or incidence. This was the case for 40 / 600 models for prediction of resistance prevalence and for 99 / 600 models for prediction of incidence rates, while all multiplicative models (log link) converged and produced coefficients. This problem was due to the fact that predicted values below 0 or above 1 in the case of binomial regression, or simply below 0 for Poisson regression, were obtained with additive models.

Examples of descriptive graphs produced are presented in Figs 1 and 2. Fig 1 presents a scatterplot of prevalence of carbapenem-resistant Pseudomonas sp. per 100 admissions against carbapenem use in courses per 100 patient-days, the most accurate indicator for this combination. Data was aggregated per ICU; it was also aggregated per year rather than per period, to make the graph clearer. This is representative of most scatterplots produced, showing an apparent clustering of antimicrobial use at the ICU level. Fig 2 presents time series of piperacillin-tazobactam, quinolone and carbapenem use per 4-week period, all ICUs combined. Each graph presents the most accurate and the least accurate indicators for the prediction of resistance in Pseudomonas sp.. As observed in MAE comparisons, indicators are visually more similar for piperacillin-tazobactam use and quinolone use than with carbapenem use, for which the only difference in predictive accuracy of resistance prevalence was observed.

thumbnail
Fig 1. Scatterplot of prevalence of carbapenem-resistant Pseudomonas sp. per 100 admissions and carbapenem use in courses per 100 patient-days, per year and per intensive care unit (ICU).

https://doi.org/10.1371/journal.pone.0145088.g001

thumbnail
Fig 2. Time series of piperacillin-tazobactam, quinolone and carbapenem use per 4-week period, all ICUs combined.

Part A: quinolone use in courses per 100 admissions and exposed per 100 admissions; part B: carbapenem use in courses per 100 patient-days and in RDD per 100 admissions; part C: piperacillin-tazobactam use in DDD per 100 admissions and agent-days per 100 admissions.

https://doi.org/10.1371/journal.pone.0145088.g002

Discussion

To our knowledge, this comparison of population antimicrobial use indicators’ ability to predict resistance is novel, even though this knowledge gap had been highlighted in the scientific literature previously. Using respiratory tract isolates and antimicrobial prescriptions from nine intensive care units, this study compared the accuracy of fifteen indicators of population antimicrobial use in predicting prevalence and incidence rates of different resistances in the respiratory microbiota. A statistically significant difference between MAEs was observed for only 1 of the 20 scenarios studied: carbapenem use to predict prevalence of carbapenem-resistant Pseudomonas sp. This difference identified one indicator that did not perform as well; however, no single indicator (or no set of indicators) stood out as better than the others.

Identifying the most accurate indicator

The absence of difference between indicators that was observed for most scenarios could be explained by different factors. These are not limitations, but rather reflect the reality of ICUs and of their use of antimicrobials. First, as described in Fig 1, throughout the four years of the study, levels of antimicrobial use and resistance tended to correlate at the ICU level; after adjusting for ICU type, there was thus less variation that could be explained with indicators of antimicrobial use. In addition, all indicators were attempting to measure similar variations in time: no exposure is equal to a value of zero for all numerators, which will tend to increase or decrease together with different magnitudes. Also, ICUs’ median lengths of stay are very short (5 days in neonatal ICUs and 2 days in other ICUs, data not published); for an ICU admission of two days, the difference between the number of agent-days, the number of courses and the simple exposure cannot be as large as for a longer admission. Of note, the situation would not necessarily have been dramatically different using hospital-wide data, as the median length of stay in Québec acute-care hospitals was 4 days.[29] Finally, although resistance / antimicrobial use combinations studied included entire antimicrobial classes (sometimes even three), a single agent can sometimes constitute most of an antimicrobial class usage. For instance ampicillin-days represented 67% of all penicillin agent-days; therefore, even if the standard DDD and RDD for ampicillin are very different, indicators of ampicillin use using DDDs, RDDs or agent-days will tend to follow the same time trends, and respective indicators of penicillin use will be driven by ampicillin use. In this study, rather than the actual values of indicators, what mattered were variations in time and across ICUs, and correlation of indicators with resistance measures; most indicators did not differ at these levels. Whenever a difference was found, it was between the most accurate and the least accurate indicators, but most indicators’ MAEs were not different from the indicator with the smallest MAE.

Actually, statistically different MAEs were observed only for one scenario, involving prevalence of resistance to carbapenems in Pseudomonas sp.. Pseudomonas sp. are prone to the development of resistance, especially to imipenem: they might react more swiftly to an exposure to antimicrobials, amplifying the possibility to detect differences between indicators of antimicrobial use.[30] Another interesting observation regarding this scenario is that, although a large proportion of prevalent cases are also incident cases, no statistically significant difference between indicators was detected in their prediction of incidence rates. As admissions last longer in neonatal ICUs than in other ICUs, prevalence and incidence rates do not follow the same trends, despite their similar numerators.

Limitations

In this study, different indicators of antimicrobial use usually had similar accuracy in the prediction of resistance prevalence or incidence in the respiratory microbiota. Interpretation of results is however limited by assumptions made in the study design. In this study, we assumed that surveillance would ideally include pediatric populations, which have been frequently excluded from antimicrobial use surveillance.[15, 17, 31, 32] We also assumed that ICUs would perform surveillance on a 4-week period basis, accounting for no time lag or for a 1-period time lag, without information on antimicrobial use before ICU admission, and that neonatal, pediatric and adult ICUs would be considered different enough to be treated separately. We also limited our cohort to ICU patients and to respiratory tract cultures performed for these patients, in an attempt to include colonizing microorganisms rather than only microorganisms infecting patients. All this was done to be as representative and similar as possible to a real surveillance setting, but results could differ if other assumptions were made. Second, our results describe prescribed antimicrobials rather than administered or dispensed antimicrobials, which probably lead to some degree of overestimation of antimicrobial use in our ICUs. As we were interested in prediction of resistance time trends, this might not be as critical as for a study estimating association between antimicrobial use and resistance. Also, a study with more participating ICUs would have allowed the use of hierarchical models with random intercepts, rather than population average models using generalized estimating equations. More participating ICUs would have also allowed us to compare ICUs and perform benchmarking, but this was not our objective. Finally, with nine ICUs followed during four years, the study population was large enough to detect significant associations between some indicators and resistance levels, but a lack of power to detect differences between indicators was still possible (not enough ICU-4-week-periods observed). This possibility was however rejected after a simulation study replicating this cohort study in two much larger simulated networks of ICUs failed to demonstrate differences between indicators in a majority of studied scenarios.

Conclusion

Indicators of population antimicrobial use have been developed, used and discussed for many decades now, but the identification of the best indicator is still an object of debate. We believe that the purpose of measurement, surveillance of antimicrobial use as a complement to surveillance of antimicrobial resistance, has to be taken into consideration. Our study has shown that, at least in our context, indicators are equivalent. Had an indicator been more accurate than others, it would have allowed a closer monitoring of variations in antimicrobial resistance frequency, and an increased ability to detect the impact on resistance of interventions targeting antimicrobial use. These first results indicate that a single best indicator might not exist and that feasibility considerations, such as ease of computation or potential external comparisons could be more decisive in the choice of an indicator for surveillance of healthcare antimicrobial use.

Supporting Information

S1 Table. Standard values used in the computation of defined daily doses (DDD) and recommended daily doses RDD).

https://doi.org/10.1371/journal.pone.0145088.s001

(PDF)

S3 Table. Most accurate, second most accurate and least accurate indicators in predicting prevalence of antimicrobial resistance, for different scenarios, with their regression link and their mean absolute error, stratified per ICU type as adjusted for in regression models.

https://doi.org/10.1371/journal.pone.0145088.s003

(PDF)

S4 Table. Most accurate, second most accurate and least accurate indicators in predicting incidence rates of antimicrobial resistance, for different scenarios, with their regression link and their mean absolute error, stratified per ICU type as adjusted for in regression models.

https://doi.org/10.1371/journal.pone.0145088.s004

(PDF)

Acknowledgments

We thank Ms. Milagros Gonzales for her work on data extractions.

Author Contributions

Conceived and designed the experiments: EF RWP DLB CQ. Performed the experiments: EF CQ. Analyzed the data: EF. Wrote the paper: EF RWP PSF DLB CQ.

References

  1. 1. Schechner V, Temkin E, Harbarth S, Carmeli Y, Schwaber MJ. Epidemiological interpretation of studies examining the effect of antibiotic usage on resistance. Clin Microbiol Rev. 2013;26:289–307. pmid:23554418
  2. 2. Steinke D, Davey P. Association between antibiotic resistance and community prescribing: a critical review of bias and confounding in published studies. Clin Infect Dis. 2001;33 Suppl 3:S193–205. pmid:11524719
  3. 3. Lipsitch M. Measuring and interpreting associations between antibiotic use and penicillin resistance in Streptococcus pneumoniae. Clin Infect Dis. 2001;32:1044–54. pmid:11264033
  4. 4. Goossens H, Ferech M, Vander Stichele R, Elseviers M. Outpatient antibiotic use in Europe and association with resistance: a cross-national database study. The Lancet. 2005;365:579–87.
  5. 5. World Health Organization. Anti-Infective Drug Resistance Surveillance and Containment Team. WHO global strategy for containment of antimicrobial resistance. Geneva: World Health Organization; 2001. 99 p. p.
  6. 6. NHSN. Antimicrobial Use and Resistance (AUR) Module. 2014. Available from: http://www.cdc.gov/nhsn/PDFs/pscManual/11pscAURcurrent.pdf.
  7. 7. Dellit TH, Owens RC, McGowan JE Jr., Gerding DN, Weinstein RA, Burke JP, et al. Infectious Diseases Society of America and the Society for Healthcare Epidemiology of America guidelines for developing an institutional program to enhance antimicrobial stewardship. Clin Infect Dis. 2007;44:159–77. pmid:17173212
  8. 8. Transatlantic Taskforce on Antimicrobial Resistance. Transatlantic Taskforce on Antimicrobial Resistance Progress Report—Recommendations for future collaboration between the US and EU. 2014. 85 p.]. Available from: http://www.cdc.gov/drugresistance/pdf/TATFAR-Progress_report_2014.pdf.
  9. 9. WHO Collaborating Centre for Drug Statistics Methodology; Norwegian Institute of Public Health. Guidelines for ATC classification and DDD assignment. 4th ed. Oslo: WHO Collaborating Centre for Drug Statistics Methodology; 2011. 286 p. p.
  10. 10. Zarb P, Amadeo B, Muller A, Drapier N, Vankerckhoven V, Davey P, et al. Identification of targets for quality improvement in antimicrobial prescribing: the web-based ESAC Point Prevalence Survey 2009. J Antimicrob Chemother. 2011;66:443–9. pmid:21084362
  11. 11. Fortin E, Fontela PS, Manges AR, Platt RW, Buckeridge DL, Quach C. Measuring antimicrobial use in hospitalized patients: a systematic review of available measures applicable to paediatrics. J Antimicrob Chemother. 2014;69:1447–56. pmid:24481320
  12. 12. Haug JB, Reikvam A. WHO defined daily doses versus hospital-adjusted defined daily doses: impact on results of antibiotic use surveillance. J Antimicrob Chemother. 2013;68:2940–7. pmid:23838948
  13. 13. Ruef C. What's the best way to measure antibiotic use in hospitals? Infection. 2006;34:53–4. pmid:16703292
  14. 14. de With K, Maier L, Steib-Bauert M, Kern P, Kern WV. Trends in antibiotic use at a university hospital: defined or prescribed daily doses? Patient days or admissions as denominator? Infection. 2006;34:91–4. pmid:16703299
  15. 15. Ibrahim OM, Polk RE. Benchmarking antimicrobial drug use in hospitals. Expert Rev Anti Infect Ther. 2012;10:445–57. pmid:22512754
  16. 16. Monnet DL. Measuring antimicrobial use: the way forward. Clin Infect Dis. 2007;44:671–3. pmid:17278057
  17. 17. Berrington A. Antimicrobial prescribing in hospitals: be careful what you measure. J Antimicrob Chemother. 2010;65:163–8. pmid:19884120
  18. 18. Filius PM, Liem TB, van der Linden PD, Janknegt R, Natsch S, Vulto AG, et al. An additional measure for quantifying antibiotic use in hospitals. J Antimicrob Chemother. 2005;55:805–8. pmid:15814603
  19. 19. Fortin E, Gonzales M, Fontela PS, Platt RW, Buckeridge DL, Quach C. Improving quality of data extractions for the computation of patient-days and admissions. Am J Infect Control. 2015;43:174–6. pmid:25530553
  20. 20. Cohen AL, Calfee D, Fridkin SK, Huang SS, Jernigan JA, Lautenbach E, et al. Recommendations for metrics for multidrug-resistant organisms in healthcare settings: SHEA/HICPAC Position paper. Infect Control Hosp Epidemiol. 2008;29:901–13. pmid:18808340
  21. 21. WHO Collaborating Centre for Drug Statistics Methodology; Norwegian Institute of Public Health. ATC/DDD Index 2014. 2014. Available from: http://www.whocc.no/atc_ddd_index/.
  22. 22. Gilbert DN, Robert C. Moellering JMD, Eliopoulos GM, Sande MA, Chamber HF. The Sanford Guide to Antimicrobial Therapy: Antimicrobial Therapy; 2008.
  23. 23. La Salle M, Moore D, Quach C, McDonald J, Rubin E, Noya F. Montreal Children's Hospital Pediatric Drug Formulary—Antimicrobial Agents. Moore D, editor. Montreal2004.
  24. 24. Pickering LK, Long SS. Red Book: 2012 Report of the Committee on Infectious Diseases: American Academy of Pediatrics; 2012. 1058 p. p.
  25. 25. Bradley J, Nelson JD, Kimberlin DW, Leake JAD, Palumbo PE, Sanchez PJ, Sauberan J, Steinbach WJ. 2012–2013 Nelson's Pediatric Antimicrobial Therapy, 19th Edition: American Academy of Pediatrics; 2012.
  26. 26. Maldonado G, Greenland S. A comparison of the performance of model-based confidence intervals when the correct model form is unknown: coverage of asymptotic means. Epidemiology. 1994;5:171–82. pmid:8172992
  27. 27. Hyndman RJ. Another look at forecast-accuracy metrics for intermittent demand. Foresight. 2006;8:43–6.
  28. 28. Holm S. A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics. 1979;6:65–70.
  29. 29. Gilca R, Hubert B, Fortin E, Gaulin C, Dionne M. Epidemiological patterns and hospital characteristics associated with increased incidence of Clostridium difficile infection in Quebec, Canada, 1998–2006. Infect Control Hosp Epidemiol. 2010;31:939–47. pmid:20677973
  30. 30. Li H, Luo YF, Williams BJ, Blackwell TS, Xie CM. Structure and function of OprD protein in Pseudomonas aeruginosa: from antibiotic resistance to novel therapies. Int J Med Microbiol. 2012;302:63–8. pmid:22226846
  31. 31. Kern WV, de With K, Steib-Bauert M, Fellhauer M, Plangger A, Probst W. Antibiotic use in non-university regional acute care general hospitals in southwestern Germany, 2001–2002. Infection. 2005;33:333–9. pmid:16258863
  32. 32. Polk RE, Fox C, Mahoney A, Letcavage J, MacDougall C. Measurement of adult antibacterial drug use in 130 US hospitals: comparison of defined daily dose and days of therapy. Clin Infect Dis. 2007;44:664–70. pmid:17278056