Figures
Abstract
Background
Surgical patients are at risk for preventable adverse drug events (ADEs) during hospitalization. Usually, preventable ADEs are measured as an outcome parameter of quality of pharmaceutical care. However, process measures such as QIs are more efficient to assess the quality of care and provide more information about potential quality improvements.
Objective
To assess the quality of pharmaceutical care of medication-related processes in surgical wards with quality indicators, in order to detect targets for quality improvements.
Methods
For this observational cohort study, quality indicators were composed, validated, tested, and applied on a surgical cohort. Three surgical wards of an academic hospital in the Netherlands (Academic Medical Centre, Amsterdam) participated. Consecutive elective surgical patients with a hospital stay longer than 48 hours were included from April until June 2009. To assess the quality of pharmaceutical care, the set of quality indicators was applied to 252 medical records of surgical patients.
Results
Thirty-four quality indicators were composed and tested on acceptability and content- and face-validity. The selected 28 candidate quality indicators were tested for feasibility and ‘sensitivity to change’. This resulted in a final set of 27 quality indicators, of which inter-rater agreements were calculated (kappa 0.92 for eligibility, 0.74 for pass-rate). The quality of pharmaceutical care was assessed in 252 surgical patients. Nearly half of the surgical patients passed the quality indicators for pharmaceutical care (overall pass rate 49.8%). Improvements should be predominantly targeted to medication care related processes in surgical patients with gastro-intestinal problems (domain pass rate 29.4%).
Citation: de Boer M, Ramrattan MA, Boeker EB, Kuks PFM, Boermeester MA, Lie-A-Huen L (2014) Quality of Pharmaceutical Care in Surgical Patients. PLoS ONE 9(7): e101573. https://doi.org/10.1371/journal.pone.0101573
Editor: Darwin Ang, University of South Florida, United States of America
Received: February 25, 2014; Accepted: June 7, 2014; Published: July 9, 2014
Copyright: © 2014 de Boer et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This work was supported by funding of ZonMw, The Dutch Organization for Health Research and Development [grant number 170882706]. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
The quality of health care systems is an important area of concern. In hospitals, one of every 150 admitted patients dies as a consequence of adverse events. [1] Almost 60% of these in-hospital events are associated with surgical care. [1] In the past years, there has been an increased awareness in patient safety in surgery. Therefore, several effective strategies are developed to measure and improve patient safety in the surgical processes, such as the use of Surgical Safety checklists. [2]–[5]
Medication-related adverse events also occur in surgical patients. [6], [7] These events are known as adverse drug events (ADEs). [8] The incidence of ADEs in surgical patients varies between 2.1–27.7 ADEs per 100 admissions of which 15.3%–53.6% of the ADEs are preventable. [6] In a recent study, the occurrence of preventable ADEs was measured to be 4.2 per 100 admissions in the surgical population in hospitals using computerized physician order entry (CPOE) systems with clinical decision support. [7] This indicates that medication-related events are still a problem in hospitals.
Preventable ADEs are often used as an outcome measure for improvements in the quality of in-hospital pharmaceutical care. [9]–[11] These ADEs are usually measured retrospectively by screening patient records on potential ADEs using, for example, a so-called trigger tool method. [8], An expert panel then assesses the causality of found triggers with medications to determine the actual presence of ADEs. This approach seems effective [14] and widely accepted in several patient populations. [12], [13], [15]–[18] However, it is a time consuming method and depending on the substantial contribution of the expert panel.
Another approach to assess improvements in the quality of in-hospital care is the use of process measures, such as quality indicators (QIs). [19], [20] Process measures elucidate suboptimal care in the process, and are influenced directly by implementations of quality improvements. [21], [22] Furthermore, these measures are easier to recognize than outcome measures and are more efficient in assessing the quality of care. [21], [22] QIs are explicitly defined and measurable items that refer to processes of care, but can also refer to structures or outcomes of care. [20] Focusing on the processes of care, a commonly used example of these measures is the quality indicator set to assess the quality of pharmacologic care of vulnerable elderly patients (ACOVE). [23], [24] Also, a set of quality indicators has been developed to assess the care of elderly surgical patients. [25] However, only few of these indicators are directly related to the medication processes on the surgical ward.
Due to the above mentioned reasons, we composed and tested a set of quality indicators specifically for the medication use in the surgical population in addition to the outcome assessment of ADEs. [7] Furthermore, we assessed the quality of pharmaceutical care of several medication-related processes using QIs on surgical wards in order to detect targets improvements.
Methods
Setting and population
In this observational cohort study three surgical wards of an academic hospital in the Netherlands (Academic Medical Centre, Amsterdam) participated. Consecutive elective surgical patients with a hospital stay longer than 48 hours were included from April until June 2009. Patients were excluded when transferred from another ward within the same hospital or transferred between hospitals. Only the first elective admission of each patient was assessed during the observation period. The surgical wards contained mainly gastro-intestinal and vascular surgery patients.
Selection and composition of QI set
To compose a representative set of QIs for medication-related processes on surgical wards, several frequently occurring problems in the surgical population for which medication is used (medication care related problems), were selected by an expert panel. This expert panel consisted of a consultant surgeon (MB) and hospital pharmacist (PK). The problems were defined into domains in which the QIs were to be classified: pain, infection, thrombosis, gastro-intestinal problem, delirium and ‘other’ problem. The documentation and handovers of medication-related information between care providers (communication related problem) was also selected as a frequently occurring problem in the surgical population.
Several inclusion criteria were noted to select eligible QIs for the surgical QI set:[26]
- aimed at medication-related processes on surgical wards;
- applicable to a surgical population and administrative data; and
- able to measure changes in medication-related processes (due to interventions by physicians or hospital pharmacists).
The selection of QIs was predominantly based on existing sets of QIs. [23]–[25], [27] After selecting the QIs, these were adapted to the surgical population incorporating national and local surgical care guidelines. Expert opinion of the consultant surgeon and hospital pharmacist was used during this consensus based selection and composition process of QIs. We choose to define the quality indicators according to the ‘IF…THEN’-principle from ACOVE [28] with explicitly defined denominator and numerator.
Testing of QI set
Several steps were performed to determine if the composed QIs were eligible to apply on the surgical population. Firstly, the expert panel individually reviewed the developed QIs on acceptability, i.e. relevance and appropriateness for the surgical population, and on content validity and face validity, such as phrasing and comprehensibility. [20], [27], [29], [30] They scored these aspects on a 9-point Likert scale. If a score of five or below was assessed by the expert panel, we indicated this test as fail and excluded the quality indicator. If disagreement occurred between the experts, consensus based conclusions were made on their testing results leading to inclusion or exclusion of the indicator.
Next, the feasibility of the remaining QIs was tested in 50 randomly selected test patients admitted to the surgical wards between December 2008 and February 2009 to assess the overall usability. Different aspects of feasibility were scored by two individual reviewers, i.e. the availability of data and the clinical burden to collect data on a 9-point Likert scale, the occurrence of eligibility (e.g. the number of patients eligible for the indicator; >1%), and the timeframe necessary to apply the QI set to a patient. [20], [29] A timeframe of one hour is considered acceptable to apply a QI set on a single patient. [23], [25], [31] Furthermore, the ‘sensitivity to change’ was calculated using the indicators pass rates (100% minus pass rate, e.g. the percentage of eligible patients meeting the QI). [20] Since no literature was available on a cutoff point, we considered a quality indicator to be ‘sensitive to change’ when a pass rate of 90% or less was reached. In the last step, the inter-rater agreement was assessed to obtain information about the reliability of the composed QI set. [20], [27], [31] Inter-rater reliability was assessed by two independent reviewers on 50 test patients. The agreement between reviewers on eligibility and on pass rates of the quality indicators on the eligible patients, was calculated by use of Cohen's kappa (κ) statistic [32] in SPSS version 18.0 (SPSS Inc., Chicago, IL). Values for κ between 0.41 and 0.60 were considered to reflect moderate agreement, values between 0.61 and 0.80 substantial agreement, and values >0.81 almost perfect agreement. [33] Also, percentages of agreement on eligibility and pass rates of the quality indicators in this patient population were calculated between the reviewers.
Assessment of quality of pharmaceutical care
The composed and tested QI set was retrospectively applied on the medical records of 252 surgical patients, admitted between March and June 2009. The number of eligible patients was determined, as well as the pass rates of the quality indicator in the total patient population. The pass rate of an indicator represents the percentage of eligible patients that met that particular quality indicator. [23], [24], [31] If the QI could not be evaluated due to insufficient data, this was scored as unknown. Mean pass rates were calculated for each domain of QIs. Evaluation of the pass rates will detect targets for quality improvements in this population.
Comparison of QIs with ADEs
ADEs are outcome measures and QIs are process measures. A preventable ADE can be the result of a failing medication process. Depending on the content of QIs, ADEs can therefore have a direct relationship with the QIs. However, if no direct relationship is present, then QIs and ADEs can have a complementary value in measuring the quality of pharmaceutical care. To study and describe the interaction and complementary value of the QIs and ADEs, each ADE and preventable ADE was linked to the corresponding QI domain, if possible. The method to detect and assess ADEs and preventable ADEs in this population is previously described by De Boer et al. [13]
Ethics and informed consent
After evaluation by the Medical Ethics Committee at the Academic Medical Centre Amsterdam, it was decided that this study was exempt from ethical approval, because this study did not meet the criteria for Medical Scientific Research with Humans under the Dutch Law. Informed consent was therefore not required. Patient records and information was anonymized prior to analysis.
Results
Selection and composition of QI set
The QIs were divided in domains representing medication care related problems, e.g. pain, infection, thrombosis, gastrointestinal problem, delirium, and ‘other’ problem, and in communication related problems, e.g. the domain documentation and discharge. Based on QIs from existing sets and national guidelines, 34 quality indicators were primarily composed for the pharmaceutical care in the surgical population. A flowchart of the composition and testing of the surgical QI set is depicted in figure 1.
Testing of QI set
The expert panel tested the QIs on acceptability, content validity and face validity. Six QI were excluded based on these tests (table 1). The 28 candidate QIs were tested for feasibility and sensitivity to change on 50 test patients (table 2). The availability of data and the clinical burden was scored 7–9 on the 9-point Likert scale for 27 indicators, demonstrating that these QIs were acceptable for application in the surgical population. One indicator scored poor on feasibility (3 for availability of data, 6 for clinical burden). All candidate indicators occurred more than 1% in the test population. Furthermore, the complete set of 28 QIs was scored on patients’ records within one hour with an average time of 49 minutes per patient. In addition, table 1 shows the average times necessary to score each indicator. The ‘sensitivity to change’ was scored more than 10% for 22 indicators and 0–8% for the six remaining indicators. In four of these QIs the number of eligible patients was very low (2–6% of 50 patients), indicating an unreliable result on ‘sensitivity to change’. In the remaining two QIs the sensitivity to change was 4% and 8%, indicating a high pass rate score, thus hardly any improvement of the quality of care was possible. Based on the test population of 50 patients, it cannot be definitively concluded that these QIs will not be useful to assess the quality of pharmaceutical care. Furthermore, the feasibility was good for these six QIs. As a result, it was decided to exclude only one indicator (no. 28 in table 2) from the set due to poor feasibility. This QI was defined as: “If a specification of the discharge medication of the patient is included in the discharge letter, THEN there should be none of the following discrepancies with the medication used as an outpatient UNLESS it is noted in the patients' records: differences in names of used medication, dosages, route of administration, formulation”. The 27 selected QIs are displayed in table 3.
The inter-rater reliability was tested between two reviewers applying the set of 27 QIs on 50 test patients. On eligibility, the κ-value was 0.92 (95% CI 0.90–0.94) indicating almost perfect agreement between the reviewers. The percentage of agreement was 98% on eligibility. On pass rates, the reviewers had a κ-value of 0.74 (95% CI 0.69–0.79), indicating substantial agreement. There was 96.9% agreement between the reviewers.
Assessment of quality of pharmaceutical care
In total, 262 patients were evaluated for inclusion in this study. The patient characteristics are displayed in table 4.Ten patients were not evaluated with the QI set since their medical records were not completely available.
The results on eligibility and pass rates per QI are shown in table 3. The overall pass rate of pharmaceutical care in surgical patients was 49.8% (range 0%–100%). Mean pass rates were calculated per domain: 65.5% for pain, 71% for infection, 48.6% for thrombosis, 29.4% for gastrointestinal problem, 63% for delirium 31.4% for other problem and 40% for documentation and discharge. Three QIs (12, 20, 27) show less than 1% eligibility indicating that only very few patients met the inclusion criteria for this indicator. In five QIs (14, 15, 19, 20 and 24) a pass rate of less than 20% was found (range 0–13.8%). This indicates that much improvement of pharmaceutical care is possible, especially in the patients on whom QIs 14, 15 and 24 were applicable because large numbers of patients (range 204–246) were eligible for these QIs.
Comparison of QIs with ADEs
In the population of 252 surgical patients, 85 ADEs (33.7%) and 5 preventable ADEs (2.0%) were found. The ADEs and preventable ADEs were divided, if possible, in the different domains of the QIs (table 5). Most ADEs (70.6%) were caused by medication used for pain. These ADEs were mainly gastrointestinal problems such as nausea/vomiting due to opioids. Other medication (16.5%) accountable for ADEs were for example anesthetics or diuretics, causing mainly hallucinations (due to Ketamine) or electrolyte (potassium) disorders respectively. The preventable ADEs were allergic reactions, hyperkalemia, excessive anticoagulation and electrolyte disorders caused by medication for pain, infections, thrombosis, and ‘other problems’ as categorized in table 5.
Compared to the QI pass rates, for example the domain of gastrointestinal problems scored a mean pass rate of only 29.4%, indicating poor quality of pharmaceutical care. However, only one ADE was found in this domain of medication processes, meaning that medication used to treat gastrointestinal problems do not frequently result in harm for the patient. On the other hand, many ADEs (70.6%) occurred due to analgesics (domain pain) and these ADEs were mainly related to gastrointestinal problems. This means that the pharmaceutical care to manage pain and analgesic adverse effects, such as gastrointestinal ADEs, was poor.
Discussion
In this study, a representative set of valid quality indicators was composed to assess the quality of pharmaceutical care in surgical patients. Based on our results, half of the medication-related processes on the surgical wards need quality improvement.
The QIs set was composed by defining and classifying frequently occurring problems in surgical patients with several criteria, by use of existing QI sets and local surgical guidelines, and by using expert opinion. Several methods have been described to develop indicators, such as the Delphi technique and the RAND/UCLA appropriateness method. [20], [26] Since we only adapted existing QIs, we solely used an iterated consensus method [20] with two experts from the field. They tested acceptability, content validity and face validity for the appropriateness of the developed QIs. Furthermore, feasibility, ‘sensitivity to change’ and reliability were assessed. [20] Until now, McGory et al. [25] were the first to describe quality indicators in elderly patients undergoing surgery. They developed QIs for several perioperative processes, but only 14 of the 91 QIs were related to medication use. These QIs formed the basis for the development of present QI set for surgical patients.
When our QI set was applied to 50 medical records, the inter-rater reliability between two reviewers was almost perfect (κ = 0.92) with regard to eligibility and substantial (κ = 0.72) between two reviewers with regard to pass rate. Similar reliability rates were found by Wierenga et al. [27] when they applied their 87-item QI set on 10 test patients. Good inter-rater reliability results show that the QIs were explicitly defined. Therefore, personal interpretation of the reviewers in applying the QIs was minimal.
Using the QI set in 252 surgical patients resulted in an overall pass rate of 49.8% (range 0%–100%), indicating that the quality of the selected medication-related processes on surgical wards of a Dutch academic centre can be improved substantially. Only Higashi et al. described pass rates while assessing the pharmacological care in elderly using the ACOVE quality indicators. [24] They found an overall pass rate of 74% in several domains; prescribing indicated medications (50%, CI 45–55%), avoiding inappropriate medication (97%, CI 96–98%), education, continuity and documentation (81%, CI 79–84%), and monitoring (64%, CI 60–68%). These QIs are specifically developed for the elderly population, also the defined domains used for the ACOVE QIs as described above, were different compared to our QI set. Therefore, a comparison with our QI pass rates in the surgical population is not valid. Furthermore, no data is available on the quality of the pharmaceutical care assessed with quality indicators in surgical patients. We are, to our knowledge, the first to describe applied QIs for specifically pharmaceutical care in surgical patients.
The mean pass rates of the domains in our population show that much more attention is needed for the medication processes to prevent gastro-intestinal problems (mean pass rate 29.4%) and other problems (mean pass rate 31.4%), such as administering controlled-release medication through a (naso)gastric or post-pyloric tube. The domain documentation and discharge(mean pass rate 40%) shows that registration of the patients' actual medication is liable to many medication errors. For example, 204 patients were eligible for indicator 24 in which, on average, 7.2 discrepancies in medication per patient were found. In 60% of all cases, dosages were registered incorrectly. Such registration errors can lead to medication-related harm and are an important issue of concern in medication safety. [34]
For quality improvements, the QI pass rates should be carefully examined individually. Based on the number of eligible patients, problems can be evaluated and prioritized, and improvements can be implemented. The QIs can be applied again retrospectively after implementation of improvements to assess the actual benefit in the quality of the medication processes or can be applied prospectively during pharmacotherapeutic care. For example, it appeared that adding a laxative to the prescribed opioid therapy to prevent obstipation was not daily practice in surgery or the physicians involved, did not register the reasons for omitting the laxative (pass rate 7.2%). Exploring the reasons for errors in this medication process might lead to improvements that prevent patient harm.
Outcome assessments, such as ADE assessment, constitute a valuable source of information in addition to QIs pass rates. For example, many patients in our surgical population experienced nausea and/or vomiting while being treated with opioids. Although, nausea is a well-known intrinsic side effect of opioids, obstipation due to opioids may lead to nausea as well. ADE assessment helps to distinguish between possible causes of nausea. In our population, the benefit of routine combination of antiemetics (in addition to laxatives) with opioids may very well exceed the risk of side effects from the antiemetics. If this were confirmed in a prospective study, then ADE assessment would have contributed to the development of a new QI.
This study has a number of limitations. First, the present QI set contains a selection of medication processes on surgical wards which were believed to be amenable to quality improvements. However, it appears that three QIs (12, 20, 27) only had 0.8% eligible patients in the total population, so the impact of any quality improvement on the whole population would be small. For these QIs, a larger sample size is necessary to draw conclusions on the quality of pharmaceutical care of these medication processes in surgical patients. It is possible that these medication processes are not applicable for this specific surgical population to assess for quality improvements in clinical practice. Second, our method of assessing the quality of care is retrospective and, therefore, dependent on registration quality and completeness in the patients' medical records. This registration bias may have influenced our findings if physicians and nurses under- or overreport specific clinical information used in our assessment. Third, our QIs were based on current guidelines at the time of development. Standards of care change over time and QIs need to be updated regularly.
Here, a representative and valid QI set was composed to assess quality of pharmaceutical care in surgical patients. On average, half of the medication processes on the surgical wards of a Dutch academic hospital need quality improvement. Also, ADEs are common in this population. Since the QIs are based on literature and guidelines, this QI set can also be used in other surgical settings. Future improvements in the surgical medication processes will lead to increased quality of pharmaceutical care and, therefore, increased medication safety and overall patient safety in surgery.
Acknowledgments
Other contributions: We would like to thank research students Liza Utama, Josephine F. Stutterheim and Nitika Chouhan, Pharmaceutical Sciences of the University of Utrecht, The Netherlands, for their contribution in this study.
Author Contributions
Conceived and designed the experiments: MdB MR PK MB LL. Performed the experiments: MdB. Analyzed the data: MdB EB. Contributed reagents/materials/analysis tools: MdB PK MB. Wrote the paper: MdB MR EB PK MB LL.
References
- 1. De Vries EN, Ramrattan MA, Smorenburg SM, Gouma DJ, Boermeester MA (2008) The incidence and nature of in-hospital adverse events: a systematic review. Qual Saf Health Care 17: 216–23.
- 2. De Vries EN, Prins HA, Crolla RMPH, den Outer AJ, van Andel G, et al. (2010) Effect of a comprehensive surgical safety system on patient outcomes. N Engl J Med 363: 1928–37.
- 3. Verdaasdonk EGG, Stassen LPS, Widhiasmara PP, Dankelman J (2009) Requirements for the design and implementation of checklists for surgical processes. Surg Endosc 23: 715–26.
- 4.
WHO (2008) Surgical Safety Checklist. Available: http://www.who.int/patientsafety/safesurgery/ss_checklist/en/. Accessed 2013 Mar 16.
- 5. Haynes AB, Weiser TG, Berry WR, Lipsitz SR, Breizat AH, et al. (2009) A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med 360: 491–9.
- 6. Boeker EB, De Boer M, Kiewiet JJS, Lie-A-Huen LL, Dijkgraaf MG, et al. (2013) Occurrence and preventability of adverse drug events in surgical patients: a systematic review of literature. BMC Health Serv Res 13: 364.
- 7. De Boer M, Boeker EB, Ramrattan MA, Kiewiet JJS, Dijkgraaf MG, et al. (2013) Adverse drug events in surgical patients. An observational multicentre study. Int J Clin Pharm 35: 744–52.
- 8. Morimoto T, Gandhi TK, Seger AC, Hsieh TC, Bates DW (2004) Adverse drug events and medication errors: detection and classification methods. Qual Saf Health Care 13: 306–14.
- 9. Leape LL, Cullen DJ, Clapp MD, Burdick E, Demonaco HJ, et al. (1999) Pharmacist participation on physician rounds and adverse drug events in the intensive care unit. JAMA 282: 267–70.
- 10. Kaboli PJ, Hoth AB, McClimon BJ, Schnipper JL (2006) Clinical pharmacists and inpatient medical care: a systematic review. Arch Intern Med 166: 955–64.
- 11. Kucukarslan SN, Peters M, Mlynarek M, Nafziger DA (2003) Pharmacists on rounding teams reduce preventable adverse drug events in hospital general medicine units. Arch Intern Med 163: 2014–8.
- 12. Rozich JD, Haraden CR, Resar RK (2003) Adverse drug event trigger tool: a practical methodology for measuring medication related harm. Qual Saf Health Care 12: 194–200.
- 13. De Boer M, Kiewiet JJS, Boeker EB, Ramrattan MA, Dijkgraaf MG, et al. (2013) A targeted method for standardized assessment of adverse drug events in surgical patients. J Eval Clin Pract 19: 1073–82.
- 14. Meyer-Massetti C, Cheng CM, Schwappach DLB, Paulsen L, Ide B, et al. (2011) Systematic review of medication safety assessment methods. Am J Health Syst Pharm 68: 227–40.
- 15. Singh R, McLean-Plunckett EA, Kee R, Wisniewski A, Cadzow R, et al. (2009) Experience with a trigger tool for identifying adverse drug events among older adults in ambulatory primary care. Qual Saf Health Care 18: 199–204.
- 16. Takata GS, Mason W, Taketomo C, Logsdon T, Sharek PJ (2008) Development, testing, and findings of a pediatric-focused trigger tool to identify medication-related harm in US children's hospitals. Pediatrics 121: 927–35.
- 17. Handler SM, Hanlon JT (2010) Detecting Adverse Drug Events Using a Nursing Home Specific Trigger Tool. Ann Longterm Care 18: 17–22.
- 18. Seynaeve S, Verbrugghe W, Claes B, Vandenplas D, Reyntiens D, et al. (2011) Adverse drug events in intensive care units: a cross-sectional study of prevalence and risk factors. Am J Crit Care 20: 131–40.
- 19. De Vos M, Graafmans W, Kooistra M, Meijboom B, van der Voort P, et al. (2009) Using quality indicators to improve hospital care: a review of the literature. Int J Qual Health Care 21: 119–29.
- 20. Campbell SM, Braspenning J, Hutchinson A, Marshall MN (2003) Research methods used in developing and applying quality indicators in primary care. BMJ 326: 816–9.
- 21. Rubin HR, Pronovost P, Diette GB (2001) The advantages and disadvantages of process-based measures of health care quality. Int J Qual Health Care 13: 469–74.
- 22. Lawrence M, Olesen F (1997) Indicators of Quality in Health Care. Eur J Gen Pract 3: 103–8.
- 23. Wenger NS, Shekelle PG (2001) Assessing care of vulnerable elders: ACOVE project overview. Ann Intern Med 135: 642–6.
- 24. Higashi T, Shekelle PG, Solomon DH, Knight EL, Roth C, et al. (2004) The quality of pharmacologic care for vulnerable older patients. Ann Intern Med 140: 714–20.
- 25. McGory ML, Kao KK, Shekelle PG, Rubenstein LZ, Leonardi MJ, et al. (2009) Developing quality indicators for elderly surgical patients. Ann Surg 250: 338–47.
- 26. Evans SM, Lowinger JS, Sprivulis PC, Copnell B, Cameron PA (2009) Prioritizing quality indicator development across the healthcare system: identifying what to measure. Intern Med J 39: 648–54.
- 27. Wierenga PC, Klopotowska JE, Smorenburg SM, van Kan HJ, Bijleveld YA, et al. (2011) Quality indicators for in-hospital pharmaceutical care of Dutch elderly patients: development and validation of an ACOVE-based quality indicator set. Drugs Aging 28: 295–304.
- 28. Shekelle PG, MacLean CH, Morton SC, Wenger NS (2001) Assessing care of vulnerable elders: methods for developing quality indicators. Ann Intern Med 135: 647–52.
- 29. Kristensen S, Mainz J, Bartels P (2009) Selection of indicators for continuous monitoring of patient safety: recommendations of the project “safety improvement for patients in Europe”. Int J Qual Health Care 21: 169–75.
- 30. Rubin HR, Pronovost P, Diette GB (2001) From a process of care to a measure: the development and testing of a quality indicator. Int J Qual Health Care 13: 489–96.
- 31. Arora VM, Johnson M, Olson J, Podrazik PM, Levine S, et al. (2007) Using assessing care of vulnerable elders quality indicators to measure quality of hospital care for vulnerable elders. J Am Geriatr Soc 55: 1705–11.
- 32. Cohen J (1960) A Coefficient of Agreement for Nominal Scales. Educ Psychol Meas 20: 37–46.
- 33. Landis JR, Koch GG (1977) An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics 33: 363–74.
- 34. Van Doormaal JE, van den Bemt PM, Mol PG, Zaal RJ, Egberts AC, et al. (2009) Medication errors: the impact of prescribing and transcribing errors on preventable harm in hospitalised patients. Qual Saf Health Care 18: 22–7.