Introduction

In 2003, the U.S. National Institute of Medicine for ‘Patient Safety: Achieving a New Standard of Care’ stated that in order to reduce the number of preventable medical accidents, information regarding patient safety, including near misses and adverse events, needs to be standardized and managed [1]. Nevertheless, due to differences in patient safety event reporting systems, differences in the definitions of accidents, measurements of their frequencies, and the tracking of patient safety events have been reported [2]. Heterogeneous data entry systems, differences in terminology and classification of patient safety, and inconsistent characteristic usage in patient safety event reporting systems make standardization difficult; furthermore, it has been reported that the analysis and interpretation of near misses and adverse events in each data set require different individual methods [3].

In order to make clear and direct efforts for patient safety, healthcare professionals rely on accurately described information regarding patient safety events, harm of event and the main causes of the event. Among these, the allocation of harm scores is a crucial step in the study of patient safety events, as it often provides an opportunity to trigger a response to the management of harm and the event on an organizational level [4]. Regarding patient safety events, the management of adverse events or sentinel events causing direct harm to patients in particular has recently become the main capability of relevant professionals [4].

Studies regarding harm of patient safety events were hardly performed until the beginning of the 1960s, and only deviations from clinical guidelines were examined in relation to preventable harm. In examining such harm, it is known to be very difficult to determine whether the harm originated from medical treatment, whether the medical treatment was unnecessary, or the harm was preventable. Consequently, from a patient-to-health care provider’s perspective, a variety of efforts such as the classification of harm from the occurrence of an event and the exact assessment are being continuously made [5].

To date, harm classification schemes have been developed by organizations such as the World Health Organization (WHO), the National Quality Forum (NQF), the National Coordinating Council for Medication Error Reporting and Prevention (NCC MERP), and the Agency for Healthcare Research and Quality (AHRQ) [4–6]. In particular, the National Coordinating Council for Medication Error Reporting and Prevention (NCC MERP) created its own index classifying medication errors by elaborating the definitions of harm levels in 2001. Moreover, in 2010, Version 1.1 of the Common Format Harm Scale was presented by the Agency for Healthcare Research and Quality (AHRQ) according to the Patient Safety and Quality Improvement Act of 2005, which requires all healthcare providers to employ a common set of data elements when reporting patient safety events for the purpose of data statistics and analysis, and to use the Patient Safety Organization’s (PSO’s) standardized method to collect and classify data. This version categorized the impact of patient safety events on patients’ functional capacities on a seven-point scale, categorized into five categories of “No Harm” and “Death,” and Version 1.2 was released in April 2012, allowing users to classify harm on a five-point scale from no harm to death [4].

Despite the widespread use of these various patient safety event reporting systems in medical institutions, factors that hinder consistent, timely and accurate reporting have been reported. First, factors that affect medical personnel who use the reporting system include the level of clarity about the purpose of the report, the usefulness of the system, and the degree to which reporting is emphasized in institutional communication [7, 8]. In addition, it is influenced by the reporter’s understanding and interpretation of the definition of harms and contributing factors, and the reporter’s perception and personal judgment, along with the organization’s safety culture [7, 9, 10]. In particular, for an accurate patient safety event reporting system, consistency is important in the definition and classification of near misses and adverse events of nurses and other medical personnel who enter data into the system [9, 11]. In many studies, it was reported that there were inconsistent results in the harm assessment in the report of patient safety events between medical staff [7, 10, 12–14]. For the consistency of risk assessment, the need for related education programs was raised [10, 11].

In Korea, the Patient Safety Act was implemented and entered in force in 2015, and in accordance with the Patient Safety Act, healthcare workers and patients who have caused or become aware of patient safety events can report them autonomously, and the medical institution implements and operates a system for reporting and learning of patient safety accidents for investigation, research and sharing of such self-reporting contents. In the current patient safety event report learning system, there are general reports and intensive reports. In the general report form, the date and time of the event, the date of discovery, and the improvement measures to prevent the recurrence of the event are entered, and in the intensive report form, the degree of harm should be assessed along with the date and time of the event. Among these, the voluntary reporting of patient safety events and a standardized assessment of the harm degree of the event and contributing factors are strongly needed to manage patient safety events and promote qualitative improvement based on reported events.

While the accurate assessment of harm by medical staff is highly important, factors such as lack of sufficient training contribute to differences in perception; therefore, in this study, a harm assessment tool comprising nine patient safety scenarios for nurses working in medical institutions was applied to evaluate the degree of differences in the perception of harm among nurses.

Materials and methods

Results

General characteristics of subjects

A total of 287 subjects were included in this study and 272 (94.8%) were female. The subjects’ working departments included 192 (66.92%) general wards, 77 (26.8%) special departments such as intensive units and emergency departments, 10 (3.5%) outpatient departments. The largest group, consisting of 136 subjects (47.4%), had worked at least 1 year and less than 5 years in their current job; 76 subjects (26.5%) had worked more than 10 years; 64 subjects (22.3%) had worked at least 5 years and less than 10 years; and 9 subjects (3.1%) had worked less than a year. Of the subjects, 194 (67.6%) reported they were satisfied with their current jobs. Currently, 283 subjects (98.6%) reported that they had received the patient safety training at work. In regard to the training methods for patient safety with duplicate answers, 217 subjects (75.6%) had trained with the “theoretical lectures”, followed by 196 (68.3%) with “certificated brochure training” and 191 (66.6%) with “department conveying training”. As for the contents of patient safety training, 261 (90.9%) reported the “understanding of patient safety”, and 257 (89.5%) reported the “incident reporting procedure”. The number of subjects who spent more than one hour and less than four hours on patient safety training was the highest at 170 (59.2%). Regarding the participating events of a patient safety such as special lectures, seminars, and campaigns at their current workplace, 150 subjects (52.3%) reported that they had participated (Table 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. General characteristics of study subjects.

https://doi.org/10.1371/journal.pone.0243583.t001

Subjects’ patient safety incident experiences

When asked about their experiences with patient safety incident at current job, 207 subjects (72.1%) reported they had experienced patient safety incidents at work. Regarding the types of patient safety incidents that they had experienced, 147 subjects (51.2%) reported with near miss and 131 subjects (45.6%) reported with no harm events as duplicate answers, while 61 subjects (21.3%) reported with adverse events and 22 subjects (7.70%) reported with sentinel events. When asked whether they had written a report regarding the patient safety incidents, 171 (59.6%) of the subjects reported that they had. In the question about the type of patient safety incidents that they had experienced, which allowed multiple answers, 152 subjects (53.0%) reported with medication errors, followed by falling, answered by 124 subjects (43.2%) (Table 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Experience of patient safety accident in study subjects.

https://doi.org/10.1371/journal.pone.0243583.t002

Consistency rate among subjects regarding the scenarios

In response to each scenario, 135 subjects responded “mild harm (D)” to the scenario “medication given via wrong route” and 74 subjects responded “moderate harm (E)”. Regarding the harm duration of the same scenario, 128 subjects responded “temporary”, but 146 responded “unknown.” In the second scenario, “body part laceration during surgery”, 151 subjects evaluated it as a “severe harm (F)” and harm duration was also rated the highest, with 153 subjects responding “permanent”. To the third scenario “contrast allergy”, 131 subjects responded “moderate harm (E)” and 115 subjects responded “severe harm (F)”, while regarding harm duration, 211 subjects answered “temporary”. The fourth scenario, “abdominal site infections” was assessed by 143 subjects as “severe harm (F)” and by 95 subjects as “sentinel event”. One hundred twenty-five subjects responded to the harm duration of this scenario with “permanent” but 97 subjects answered “unknown”. The degree of harm in the fifth scenario, “mislabeled specimen” was assessed by 110 subjects as a “moderate harm (E)” and by 103 subjects as a “sentinel event”; regarding the harm duration, 144 subjects evaluated it as “temporary”. The harm degree of the sixth scenario, “wrong site surgery” was evaluated by 203 subjects as a “sentinel event” and the harm duration was rated as “permanent” by 271 subjects. The harm degree of the seventh scenario, “chest tube drain” was rated by 121 subjects as a “mild harm (D)” and by 82 subjects as a “moderate harm (E)”; the harm duration was rated by 244 subjects as “temporary.” In the eighth scenario, “medication overdose”, 138 subjects responded “mild harm (D)”and 92 subjects responded “moderate harm (E)” and regarding the harm duration, 176 subjects answered “temporary.” With regard to the harm degree of the last scenario, “medication given at the wrong time”, 107 subjects responded “mild harm (D)” and the harm duration was rated “temporary” by 153 subjects.

Upon analysis of the interrater agreement rate among the respondents using Fleiss’ kappa, the harm evaluation yielded a value of 0.21 and the value for harm duration was 0.28. (Table 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Agreement values for harm scale and harm duration.

https://doi.org/10.1371/journal.pone.0243583.t003

Discussion

Our findings demonstrate that the overall interrater agreement for harm of patient safety incidents is 0.21 in this study, which is significantly lower than the result of the similar studies using AHRQ Common Format Ham Scale Version 1.2 [4, 18]. In the study conducted by Tamara et al. [4], the value of interrater agreement was 0.45, and other study conducted by Toni et al., the value was 0.48 [4, 18]. In the general characteristics of the study subjects, most of subjects of this study are nurses who have experience in their current workplace for more than one year and less than five years, and who have received patient safety training for more than one hour and less than four hours. While the study subjects of the research conducted by Tamara et al. [4] were quality, risk and safety managers, and thus had more likely to have an experience in assessing the harm of patient safety incidents. However, the subjects in the study by Toni et al. [18] had 5 years of experience or less which is similar to the characteristics of the subjects in our study. As the patient safety reporting system was activated in Korea in accordance with the relevant laws in 2015, it is possible that nurses in our study had less experience in harm assessment than those in previous studies but further research is needed in the future considering other factors.

Also, the reason for the low interrater agreement in this study could have been subjective judgments about harm scores because the description of harm scores was only provided with patient safety scenarios in the questionnaire. Another reason for the low interrater agreement rate in the harm assessment are that the level of understanding of the reporter’s harm scale is reported to affect the assessment [7]. When harm assessments were conducted for nine scenarios involving patient safety incidents in this study, a generally moderate agreement was shown when the harm was known, such as “body part laceration during surgery” or “wrong site surgery”. On the other hand, in cases in which the assessment of harm was unclear, such as “mislabeled specimen” or “chest tube drain” the evaluation of harm and harm duration were shown to be somewhat dispersed.

As such, the need for education among reporter is raised as a more accurate harm assessment method. In many studies including Pronovost et al., professional training and sharing of ideas in practice have been reported to help enhance the perception of medical personnel in patient safety incident scenarios [7, 10, 14, 19].

In the results of harm assessment for patient safety incidents for nurses with similar experience, the interrater agreement value of this study was significantly lower than that of other studies. Therefore, it is necessary to develop a training program in Korea to actively assist nurses in performing their patient safety duties through continuous monitoring of patient safety reports and practical training on harm assessment in reporting patient safety incidents. It is also required that the development of multidisciplinary, academic tools for the standardization of harm assessment.

Despite the fact that such research was first conducted in Korea, where much attention has been paid to patient safety along with the establishment of a patient safety reporting system, as a main limitation of this study, there is a lack of generalizability since only nurses working at two medical institutions were surveyed. Next, in terms of validity of the questionnaire, we only performed the content validity with three experts without pilot study, face validity and criterion validity.

In addition, the study involved working nurses based on the assumption that they had already received the respective primary training, so no explanation or separate training regarding harm or harm duration assessment was provided. Consequently, since the nurses were prompted to read and evaluate each scenario in the questionnaire immediately, individual differences might have affected the interpretation of the scenario and assessment, and differences in individual education and clinical experience might be reflected in their assessment.

Conclusion

In the patient safety incident reporting system, the accurate assessment of medical personnel is highly important; however, there are differences in perception due to factors such as inadequate training. Therefore, in the present study, AHRQ Common Format Ham Scale Version 1.2 was applied to identify differences in perception of harm assessment among nurses. This study analyzed the interrater agreement among respondents via Fleiss’ kappa, for which a value of 0.21 was found for harm assessment and 0.28 for harm duration. These values indicated significantly lower consistency in the harm assessment among nurses.

For the quality of the patient safety incident reporting, it is important for healthcare professionals to accurately assess the degree of patients’ harm. This study found that the assessment of the degree of harm by Korean nurses was not standardized. The reason for this variability could be due to the lack of education that takes harm assessment into account. Therefore, training in harm assessment and the development of programs to support this training are both necessary.

Supporting information

Acknowledgments

The contributions of all participants in this study are greatly appreciated. We would like to thank Editage (www.editage.co.kr) for English language editing.