Using Quality Measures for Quality Improvement: The Perspective of Hospital Staff

Research objective This study examines the perspectives of a range of key hospital staff on the use, importance, scientific background, availability of data, feasibility of data collection, cost benefit aspects and availability of professional personnel for measurement of quality indicators among Iranian hospitals. The study aims to facilitate the use of quality indicators to improve quality of care in hospitals. Study design A cross-sectional study was conducted over the period 2009 to 2010. Staff at Iranian hospitals completed a self-administered questionnaire eliciting their views on organizational, clinical process, and outcome (clinical effectiveness, patient safety and patient centeredness) indicators. Population studied 93 hospital frontline staff including hospital/nursing managers, medical doctors, nurses, and quality improvement/medical records officers in 48 general and specialized hospitals in Iran. Principal findings On average, only 69% of respondents reported using quality indicators in practice at their affiliated hospitals. Respondents varied significantly in their reported use of organizational, clinical process and outcome quality indicators. Overall, clinical process and effectiveness indicators were reported to be least used. The reported use of indicators corresponded with their perceived level of importance. Quality indicators were reported to be used among clinical staff significantly more than among managerial staff. In total, 74% of the respondents reported to use obligatory indicators, while this was 68% for voluntary indicators (p<0.05). Conclusions There is a general awareness of the importance and usability of quality indicators among hospital staff in Iran, but their use is currently mostly directed towards external accountability purposes. To increase the formative use of quality indicators, creation of a common culture and feeling of shared ownership, alongside an increased uptake of clinical process and effectiveness indicators is needed to support internal quality improvement processes at hospital level.


Introduction
Measuring quality of care is a powerful mechanism to drive health system performance improvement. The availability of reliable quality information empowers patients to make informed decisions about where to seek health care, and supports health care providers to provide better health care. Recently, measuring quality of care in hospitals has become a main policy priority in many countries. Numerous health systems are in the process of developing and instituting requirements for routine measurement and reporting of quality data at both national and hospital levels for external and internal purposes [1][2][3][4]. Quality indicators can be used for summative (external) purposes to increase the account-ability of hospitals towards different stakeholders (such as; government, patients or health insurers). They can also be used for formative (internal) purposes by health care organizations and providers to measure, monitor and improve the provided levels of quality of care [5,6].
Although quality indicators represent a promising strategy for improving quality of care and accountability, the implementation process in hospitals is often faced with problems and challenges at the ground level [7][8][9][10][11][12][13]. Some of the challenges in the implementation process may originate from the absence of a role for hospital staff in the development, selection and execution of quality indicators. Successful development and implementation of quality indicators in hospitals strongly depends on frontline staff's involvement and awareness [14,15]. In addition, the consultation with key hospital staff and their involvement in quality improvement (QI) activities may increase the effectiveness and likelihood of success of such efforts. It can provide an avenue for better communication on relevant insights into the workflow and potential challenges. It can also provide an increased sense of accountability of staff, which is positively correlated with the success of such plans in practice [10,[16][17][18][19][20].
In response to the increasing worldwide interest in quality measures, the Iranian Ministry of Health and Medical Education (MOHME) developed a set of obligatory quality indicators. These indicators have been statutorily included in the Iranian National Hospital Evaluation Program in 2002. The plan aimed to realize improvement of the quality of hospital care, support informed decision making by providing health-related information, and increasing accountability and regulation in hospitals [21]. The quality indicators in Iran were primarily designed for summative purposes to increase the accountability of hospitals for the quality of care they provide. All hospitals were obliged to implement the quality indicators applying a top-down approach steered by the MOHME. As a result, the development of the quality indicators was conducted with little consultation of hospital staff [22].
Despite nationwide efforts to develop and implement the quality indicators, at present there is no study available about the extent of application and perspective of hospital staff on such measures. This study therefore aimed to assess the perspectives of hospital frontline staff on the seven themes in regards to the selected 27 obligatory and voluntary quality indicators. The indicators are categorized in organizational, clinical process and outcome groups. The outcome indicators included clinical effectiveness, patient safety and patient centeredness issues. The themes that were assessed in this study, included: use, importance, scientific soundness, availability of data, feasibility of data collection, cost benefit aspects and availability of professional personnel for measurement of quality indicators.

Methods
This is a cross-sectional study conducted over the period 2009 to 2010. Data was collected using a self-administered questionnaire. The survey was focused on twenty-seven hospital quality indicators, in each case exploring the perspectives of hospital frontline staff on seven themes. The indicators and themes included in the questionnaire were derived from a two-stage expert panel process. The panel of ten health experts consisted of two health services academics, two health services researchers, two hospital managers, a physician, two nurses, and an epidemiologist. To obtain the views of hospital staff on quality indicators for hospital care, it was important not to limit the selection of indicators to the set of obligatory indicators used by the MOHME. The expert panel was therefore presented with an initial list of obligatory and voluntary indicators and seven themes. The indicators and themes were selected from the current Iranian hospitals' annual evaluation program [23] and the World Health Organization Performance Assessment Tool for quality improvement in Hospitals (PATH) project [24]. In addition, the results of a literature search [25] and expert opinions on quality indicators were used. The panel was tasked with evaluating the indicators and themes, and given the opportunity to change, add or remove indicators and themes from the list.
As a result, twenty-seven quality indicators were selected by the panel. The indicators consisted of a combination of obligatory (developed by the MOHME) and voluntary indicators. Implementation of seven indicators was obligatory under the Iranian hospitals' annual evaluation program and the 20 remaining indicators recommended by the expert panel were voluntary indicators (see Table 1). The indicators were grouped based on a triad method in organizational, clinical process, and outcome indicators related to the clinical effectiveness, patient safety, and patient centeredness as shown in Table 1. The resulting questionnaire consisted of 9 organizational, 6 clinical process, and 12 outcome indicators relating to the clinical effectiveness (6), patient safety (4) and patient centeredness (2) of hospital care. The themes that were selected by the panel, included; i) the (areas of) use, ii) perceived importance of the indicator, iii) assessment of scientific background, iv) availability of data, v) feasibility of data collection, vi) cost benefit aspects, and vii) availability of professional personnel for measurement.
The questionnaire was in Farsi. It was piloted among five health services experts and ten frontline staff members (including two hospital managers, a medical doctor, two nurses, two nursing managers, two QI officers, and one head of medical records department). A purposive sample of five hospitals (three public governmental, one Social Security Organization (SSO), and one private hospital) was selected for piloting, which were not included in the sample of final survey. The questionnaire was assessed for face validity and clarity based on the responses received.
The questionnaire was self-administered with the exception of a few respondents in Tehran, capital city of Iran, whose questionnaires were administered face-to-face. The questionnaires were sent out and returned by mail with support provided for the respondents by email and telephone. Participation in the survey was voluntary and periodic reminders were sent out. Questionnaires received were assessed for completeness and consistency. Respondents were contacted to address any observed issues. Questionnaires were excluded if these issues could not be resolved.
In the final questionnaire, the perspectives of hospital frontline staff were asked on the organizational, clinical process and outcome quality indicators in relation to the seven themes. We analyzed the heterogeneity in the responses of staff to the questions by conducting Pearson's chi-squared test using SPSS. We explored whether hospital staff (both total study population, and different types of professionals) had statistically significant different perspectives on the seven themes related to the implementation of specific quality indicators. The average rates of positive answers to questions on the use, level of importance, scientific background, data and personnel availability, feasibility of data collection and cost benefit aspects of quality indicators were used in the analysis (i.e. whether there is agreement in the responses). Our criterion for the significance level was p,0.05.

Study population
The study population consisted of 160 respondents in three groups; managerial staff (active hospital managers, quality improvement officers, the heads of medical records department), clinical staff (medical doctors, nurses and nursing managers), and other health professionals (e.g. health economists and health services researchers). They were working in 75 hospitals across nine regions in Iran during the study period. The 75 hospitals were selected using a purposive sampling method, representing (around) 10% of hospitals in Iran. The studied hospitals were largely representative in terms of type (public, private, SSO, and othermilitary and charity -hospitals) and geographical distribution.
The response rate of hospital staff was 71.3% (114 respondents out of 160 sampled in the study) and covered 64% (48) of the (75) sampled hospitals in the study. Twenty-one questionnaires were excluded for reasons of incompleteness or inconsistency, resulting in a final population of 93 respondents. Table 2 shows the characteristics of the study population, which is representative both across hospitals and type of professionals.
The study was approved by the Deputy of Research and Technology of the Tehran (Iran) University of Medical Sciences (Code: 958/1635996).

Perspectives of hospital staff on quality indicators
Quality indicators in general. Hospital staff had significantly different (heterogeneous) perspectives on the use, level of importance, scientific background, feasibility of data collection, and cost benefit aspects of measurement of quality indicators. In contrast, there was more agreement (homogeneity) in the perspectives of hospital staff on the availability of data and professional personnel to measure the indicators. Figure 1 summarizes the overall perspectives of respondents. Organizational, clinical process, clinical effectiveness, patient safety, and patient centeredness indicators were reported to be used by 75%, 57%, 66%, 72%, and 86% of the respondents respectively (p,0.05). Clinical process indicators had the lowest reported use. This was also the indicator group with the lowest perceived importance by hospital staff. The importance given to indicators corresponded with their use, which was particularly high for outcome indicators compared to the other type of quality indicators (p,0.05). Despite the sufficient scientific background reported by respondents, on average, only 68% of respondents reported using quality indicators at their affiliated hospitals. Data were thought to be available by 83% of respondents for organizational indicators, and only 75% and 79% of respondents thought data was available for the clinical process and outcome indicators respectively. Data collection was judged to be feasible by 96%, 89%, and 94% of respondents for organizational, clinical process, and outcome indicators respectively. However, the availability of personnel and cost benefit aspects of organizational, clinical process, and outcome indicators was judged to be sufficient by 86%, 80%, 84%, and 92%, 84%, 91% of respondents respectively.
Organizational indicators. Table 3 shows the responses to the questions related to the organizational indicators. Respondents had significantly different perspectives on the level of importance, scientific background, data and professional personnel availability of the organizational indicators. However, they did not have significant different perspectives on the use, feasibility of data collection and cost benefit aspects of these indicators. The most commonly used measures were the bed occupancy rate, length of stay and bed turnover (p,0.05). The operating room (OR) efficiency indicator had a significantly lower reported use in this group, followed by a measure of the duration of quality assurance training. Medium scoring (around 73% reported use) indicators in this group were related to waiting times. Lower use of organizational indicators seemed to be mostly related to the perceived importance, availability of data, and available human resources to collect the data.
Clinical process indicators. Respondents assessed the use, level of importance, scientific background, data and professional personnel availability of the clinical process indicators significantly different. More agreement existed on the feasibility of data collection and cost benefit aspects of these indicators. Table 4 shows that clinical process indicators were reported to be significantly less used compared to the organizational and outcome indicators. In addition, there was also a relatively high level of unawareness among staff (19%) whether or not the clinical process indicators were applied in their affiliated hospitals, compared to the organizational (9%) and outcome (11%) indicators. On the extreme, only 38% of respondents reported to apply cross match rates for transfusions (and 26% of respondents were unaware of the use of this indicator). The indicators were mostly reported to be used for external assessments and internal audits purposes.
Outcome indicators. Table 5 shows the results for the questions on outcome indicators related to clinical effectiveness, patient safety and patient centeredness. Hospital staff had significantly different perspectives on the use, level of importance, data and professional personnel availability of the clinical effectiveness outcome indicators. However, they had relatively similar opinions on the scientific background, feasibility of data collection and cost benefit aspect of these indicators. Significant heterogeneity existed in the responses of hospital staff on the use,  level of importance, data and professional personnel availability, and feasibility of data collection of the patient safety outcome indicators. More homogenous responses were reported regarding the scientific background and cost benefit aspect of these indicators. Hospital staff also significantly differed in their perspectives on the availability of data and professional personnel with regard to the patient centeredness outcome indicators. In contrast, such heterogeneity was not seen regarding the use, level of importance, scientific background, feasibility of data collection, and cost benefit aspect of these indicators.
Perspectives of managerial, clinical and other hospital staff on different types of quality indicators Figure 2 shows the perspectives of managerial, clinical and other staff on organizational, clinical process and outcome indicators. The corresponding p-values indicate the statistical differences in perspectives of staff on each indicator. Clinical staff reported significantly higher utilization rates than managerial staff, i.e.: organizational indicators (80% versus 71%), clinical process indicators (64% versus 51%), clinical effectiveness indicators (75% versus 58%) and patient safety indicators (81% versus 66%). Both groups reported more or less equal use of patient centeredness indicators. The differences in reported use were (significantly) higher in case of clinical process, clinical effectiveness and patient safety indicators, compared to organizational indicators. There were about equally large (significant) differences among managerial and clinical staff in the level of perceived importance of indicators. However, both groups rated organizational and clinical outcome indicators as most important. Both clinical and managerial staff rated the scientific background of all indicators relatively high.
Clinical staff rated data availability significantly higher than managerial staff, i.e.: organizational indicators (87% versus 83%), clinical process indicators (83% versus 70%), and outcome indicators (85% versus 77%). The same was true for the perspectives on the feasibility of data collection and availability of professional personnel.

Perspectives on obligatory and voluntary indicators
Hospital staff had significantly different perspectives on the use, level of importance, scientific background, data availability, feasibility of data collection and professional personnel availability on the obligatory and voluntary indicators. However, there was no significant difference in the perspectives of staff on the cost benefit aspect of the obligatory and voluntary indicators. In total, 74% of respondents reported using obligatory indicators for different processes in their hospital. Sixty-eight percent of the respondents reported use of the remaining twenty voluntary indicators. Although the implementation of seven indicators in our study was obligatory as part of external evaluation program, only 44% of respondents reported using these indicators in the hospitals.

Discussion
We studied the perspective of frontline staff on the seven themes related to the implementation of 27 quality indicators in Iranian hospitals. This is to our knowledge the first study providing insight into the perspectives of frontline hospital staff on important aspects related to the use and implementation of quality indicators in Iranian hospitals. The comprehensive scope of the included (consensus-based) quality indicators, representative sample of all types of hospitals, and breadth of hospital professionals are strong features of the study design. However, the study has some limitations. First, the non-response of 28.7% of hospital staff and exclusion of around 20% of received questionnaires from our study population due to incompleteness of surveys resulted in a lower response rate than planned. It is possible that the nonrepresented hospitals had a lower awareness of the importance and usability of quality indicators, leading to a potential (over-)reporting bias. Furthermore, there was no clear standardized classification of quality indicators available for our study. This may be explained by the various definitions of quality indicators used by different users. However, the classification of indicators studied in this survey was based on relevant literature, knowledge and the experience of the authors.

(4)
Yes  (2) 14 (11) 12 (9) 3 (2) 14 (10) 16 (11) 12 ( (7) 7 (5) 15 (9) 7 (5) 16 (11) 13 (10) Is it feasible to collect the data for this indicator? 10 (8) 4 (3) 12 (8) 10 (7) 12 (8) 13 (10) Yes Ensuring availability of dedicated staff for measuring quality indicators Despite the acknowledgement of the importance and scientific background of quality indicators, just around two-third of hospital staff in total reported using quality indicators at practice level in their hospitals. The overall application rate shows an interest in the use of quality indicators in Iranian hospitals. However, we found significant differences between the rate of use in daily practice and the perceived importance and scientific background of different quality indicators by hospital staff. The results indicate a gap between theory and practice in the utilization of quality indicators by hospital frontline staff. Hospital staff believe that quality indicators are sufficiently grounded in science and should play an important role for monitoring quality and quality improvement (also reported by other studies [26,27]) in hospitals. However, they experienced a number of problems in the implementation of quality indicators in practice. The results of our study show that the availability of dedicated staff for measuring indicators is one of the major concerns for using quality indicators from the perspective of hospital frontline staff. This was also confirmed by related studies in this field [10,16]. For health care professionals to actually use quality indicators it is essential to involve hospital staff in the development and implementation of indicators. Other studies also identified that the involvement and participation of staff is associated with the success of quality improvement initiatives and better values of quality indicators in hospitals [10,28,29].

More attention on clinical process and effectiveness indicators
Our comparison of the overall utilization of different quality indicators revealed that there is significant difference between organizational, outcome and clinical process indicators. Although more than two-thirds of respondents acknowledged using the organizational and outcome indicators, only around half (57%) of them reported using the clinical process indicators at their hospitals. In addition, when looking at the type of outcome indicators, the results showed that clinical effectiveness indicators were used significantly less than the patient safety and patient centeredness indicators. Clinical process and effectiveness indicators seem to provide a current challenge in Iranian hospitals. The lower application rate of clinical process and effectiveness indicators in Iranian hospitals can be explained by the insufficient transparency to staff (including clinicians). Moreover, lower involvement of staff in the development process and a lack of awareness of the relevance of clinical process indicators are other possible factors resulting to a lower implementation rate. Related studies also identified less utilization of clinical indicators compared to organizational and outcome indicators in hospitals. They identified difficulties in developing evidence-based clinical process and effectiveness indicators and being less transparent to patients and staff as main barriers in the use of such measures in hospitals. In addition, they found the fear of punitive measures for variation in clinical processes and a lack of awareness or familiarity of staff with quality indicators as other possible problems in the use of these indicators [13,29,30].
0.008 outcome indicators show negative results, one needs clinical process indicators to locate the causes of reduced quality levels to improve the work processes. At the moment the interest in Iran both among clinicians and hospital managers appears to be overly focused on organizational and outcome indicators. To stimulate QI activities at hospital level, it is necessary that both clinicians and hospital managers work with clinical process and effectiveness indicators alongside organizational, patient safety and patient centeredness outcome indicators.

Combining summative with formative purposes of quality indicators
Our study shows that the current quality indicators system in Iran (which has been primarily set up for summative purposes to increase the accountability), may not be used for formative purposes in practice. Our results show that despite the government's obligation to apply quality indicators, at best, only twothirds of respondents reported to use such measures in practice. This rate points to barriers in the implementation of quality measures by hospitals. It is perhaps the result of a low awareness among frontline staff and lack of involvement in the development of quality indicators. Staff's low engagement, lack of time, limited awareness and knowledge concerning quality indicators can lead to a low motivation and commitment in applying such measures in practice [14,28,29]. The results suggest that having a top-down implementation method may not be sufficient to achieve a maximum expected implementation and effective application of quality indicators. This needs to be complemented with a bottom up approach, involvement and widespread participation of hospital staff and encouragement to participate in QI activities to create a facilitating quality improvement culture [10,22,28,29].
Creating a common quality improvement culture We found that clinical and managerial staff had different views on the use, level of importance, scientific background, feasibility of data collection and cost benefit aspects of organizational, clinical process and outcome indicators. Clinical staff on average reported using quality indicators more than managerial staff (with the exception of patient centeredness indicators). The possible reasons for such differences may be different perspectives of staff on the level of importance of indicators, data and professional personnel availability for the specific indicators. Clinical staff believe that quality indicators are very important for QI in hospital, while managerial staff did not share this opinion and even around 4% of managerial staff mentioned that clinical process indicators are not important at all in the QI process in hospitals. The opinion of hospital staff can be changed if they feel more capable in monitoring and improving their own quality of services. This starts by involving them in the development phase of quality indicators, and selection of implementation strategy at hospital level. A feeling of increased autonomy, shared ownership and awareness will likely be beneficial for the actual (and appropriate) use of quality indicators [29]. However, this also requires a hospital culture characterized by learning, constructive feedback and discussion among professionals about health care performance without creating a punitive environment. Though essential, this is challenging to create.