https://orcid.org/0000-0002-8606-1415
Figures
Abstract
Objective
To analyse the interrater reliability of the NEUMOBACT checklist and verify whether consistent results are reproducible.
Methods
A validation study with a cross-sectional design, compliant with the GRRAS checklist, among ICU nurses attending a SIMULAZERO course with an Objective Structured Clinical Evaluation simulation format, to verify transfer from theory to clinical practice of knowledge and skills in ventilator-associated pneumonia (VAP) and catheter-related bacteraemia (CRB) prevention. A minimum sample size of 111 pairs of nurse raters was calculated. Interrater agreement was analysed using Gwet’s AC1 for each item and as a total for each of the three checklists in the NEUMOBACT instrument.
Results
A total of 95 pairs of valid NEUMOBACT checklists were completed by 190 raters with a median age of 29 [25–35] years, 93.7% were female. At the Central Venous Catheter insertion station, Gwet’s AC1 was 0.934 (95% CI [0.919–0.949]). Only 2 of the 17 items scored below 0.9. At the Endotracheal Suctioning station, Gwet’s AC1 was 0.869 (95% CI [0.851–0.886]). Of the 26 items that made up this station, 16 had an agreement percentage above 0.9, a further 9 were between 0.821 and 0.884, and item 13 had a value of 0.789. At the Patient Care station, Gwet’s AC1 was 0.911 (95% CI [0.896–0.927]). Of the 21 items, 17 showed an agreement percentage above 0.9 and 4 were between 0.810 and 0.894.
Conclusions
The interrater reliability of the NEUMOBACT checklist shows substantial agreement between pairs of raters and is therefore validated in this large sample of ICU nurses.
Relevance to clinical practice
The NEUMOBACT checklist can be useful for assessing skills before and after training in VAP and CRB prevention measures and during debriefing (post-simulation feedback), to reinforce the scientific evidence behind actions and decisions for items that have been performed incorrectly, thus consolidating training already received.
Citation: Raurell-Torredà M, Zaragoza-García I, Arrogante O, Aliberch-Raurell AM, Sánchez-Chillón FJ, Torralba-Melero M, et al. (2024) Interrater reliability and agreement of the NEUMOBACT checklist about infection-prevention performance of intensive care nurses in simulation-based scenarios. PLoS ONE 19(12): e0313175. https://doi.org/10.1371/journal.pone.0313175
Editor: Javier Fagundo-Rivera, Health Researcher, SPAIN
Received: February 21, 2024; Accepted: October 18, 2024; Published: December 31, 2024
Copyright: © 2024 Raurell-Torredà et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: This project received a Grant for clinical-simulation related research projects from the Spanish Society for Patient Simulation and Safety (Sociedad Española de Simulación Clínica y Seguridad del Paciente, SESSEP). This grant was obtained for the entire research team. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Intensive care units (ICUs) underwent structural and organisational changes as a result of the Covid-19 pandemic, and these changes lead to a two- to threefold increase in the incidence rates of ventilator-associated pneumonia (VAP) and catheter-related bacteraemia (CRB), together with increased length of lCU stays and intra-ICU mortality [1]. Measures to address these problems included a proposal by the Ministry of Health’s board of experts to “promote Zero Project training among existing, newly recruited and temporary ICU healthcare providers”. These projects include Bacteraemia Zero and Pneumonia Zero, which are training and epidemiological surveillance programmes designed to reduce the incidence of CRB and VAP, respectively, that are implemented across Spain and are supported by experts from the Spanish Society for Intensive and Critical Care Medicine and Coronary Units (SEMICYUC) and the Spanish Society for Intensive Care and Coronary Unit Nursing (SEEIUC).
The SEEIUC Simulation working group runs a training course called SIMULAZERO, which applies teaching through simulation to assess ICU nurses’ knowledge and skills in VAP and CRB prevention [2]. The SIMULAZERO course is based on an Objective Structured Clinical Evaluation (OSCE) composed of two training scenarios for technical skills (endotracheal suctioning and central venous catheter insertion, respectively) and a training scenario for non-technical skills [3], which, in this case, concerns decision-making, resource management, safe practice and reduction of adverse events related to the care of critically ill patients.
In simulation, OSCEs are considered the gold standard for competency-based assessments. In OSCEs, candidates rotate through a series of time-limited stations in a circuit designed to assess professional performance in a simulated environment. These assessments facilitate the objective evaluation of a complex range of skills, knowledge, and attitudes through direct observation [4], while providing an adequate method for engaging and integrating theory with practice [5]. These evaluations can be training or assessment based, or both [6]. Various checklists are used in the ICU to guide evidence-based clinical practice, with a compliance rate ranging from 73% to 91%, improving care outcomes [7], but no checklist was found in the literature to assess ICU nurses’ performance during an OSCE, and therefore the NEUMOBACT checklist was created, designed and face validity and content [8]. The aim of this study was to analyse the interrater reliability of the NEUMOBACT checklist and verify whether consistent results are reproducible.
Materials and methods
A descriptive study of a metric nature, reported using the GRRAS checklist (S1 Checklist) for reporting of studies of reliability and agreement [9] and was conducted among ICU nurses attending the SIMULAZERO course at the 47th National Congress of SEEIUC. Recruitment was conducted by convenience from 13 to 15 June 2022. The nurses worked at various ICUs across Spain, permitting the identification of potential heterogeneity among those ICUs. Nurses were included if they had participated in the course as raters, completed all the NEUMOBACT checklist items, and agreed to participate in the study.
Sample calculation
According to the COSMIN Study Design checklist for patient-reported outcome measurement instruments [10], a minimum of 100 pairs of assessments by at least two unbiased evaluators is required to analyze inter-observer reliability. Two independent observers should be used to assess the simulation stations under similar environmental conditions, having received the same instructions. To account for expected losses, a further 10% was added, resulting in a sample of 111 pairs of nurses to evaluate the NEUMOBACT checklist during the simulation.
Variables and information sources
The data were collected during the SIMULAZERO course, based on a multi-station OSCE-format simulation (specifically 3), whereby participants rotated around a circuit of sequential stations. Each station presented a clinical case, requiring students to perform a specific set of tasks by applying a range of knowledge, skills and attitudes, resulting in the simultaneous assessment of several competence-based components. Each participant started their test at a different station, but at the same time as each other (about 10 minutes). The scenarios were of a similar length so that participants would leave one station and start the next in synchronisation. The resources used included standard materials commonly available in intensive care units, as well as basic CPR manikins, avoiding any material limitations for future replication in other ICUs. Two simultaneous rounds were conducted, with each station requiring one technician, two assessors, and two OSCE coordinators.
Raters received specific training in their roles as raters to provide them with detailed assessment criteria and the meaning of each item. The raters were paired up and given the NEUMOBACT checklist, which had been previously developed by experts using the Delphi technique, to use for each station assessment, with only two response options for each item: correct/incorrect (Table 1). They were given 10 minutes to familiarise themselves with the checklist and were reminded of the importance of completing all the items when assessing each station. They were also instructed not to share the information with each other and for this purpose they were positioned one on each side of the simulator. Since the three NEUMOBACT checklists are independent of each other, each rater pair started the assessment at a different station and followed the circuit described above. The rater pairs were also asked to respond to sociodemographic questions on their age, gender, nursing experience, ICU nursing experience and whether they had postgraduate training.
S1 File describes the learning objective, participants’ roles and actors for each of the three stations in the circuit.
Data analysis
Qualitative variables were expressed as frequency and percentage and quantitative variables as mean and standard deviation or median and interquartile range (P25-P75), depending on the data distribution. To analyse the reliability of the instrument, interrater agreement was calculated using Gwet’s AC1 for each NEUMOBACT checklist item, and for each station as a total value. Gwet’s AC1 coefficient method was used because it is more stable and less affected by prevalence and marginal probability than Cohen’s Kappa [11, 12]. Gwet’s AC1 values range from -1 to 1, where a negative value indicates significant disagreement, zero indicates what would be expected by chance, values ≤0.3 indicate slight agreement, 0.31–0.6 moderate agreement, and ≥0.7 substantial agreement [11].
Ethical aspects
The project was approved by the Bioethics Commission of the University of Barcelona (code number: IRB00003099). The investigators undertook to comply with Organic Law 3/2018 of 5 December on personal data protection and the guarantee of digital rights. The participants were informed about the study’s purpose and invited to participate. Consent was given verbally and confirmed again verbally when participants chose to respond to the checklist. In order to enhance data protection, only the principal investigator recorded and had access to the socio-demographic and academic data of each rater.
Results
Demographic characteristics
95 pairs of valid NEUMOBACT checklists were collected. The raters (n = 190) had a median age (median [P25-P75]) of 29 [25–35] years and 93.7% were female. They had 5 [2–8] years of nursing experience and 3 [2–7] years of ICU-specific experience. A total of 54.7% reported postgraduate-level education.
Reliability
At the Central Venous Catheter (CVC) insertion station, the total value for Gwet’s AC1 was 0.934 (95% CI [0.919–0.949]). All items had an agreement percentage above 0.9 except item 10 (“Use maximum barrier measures (mask, cap, eye protection and gown, sterile drapes, sheets and gloves) for CVC insertion”) and item 17 (“Place needle-free connectors only at sites where boluses are to be administered”) (Table 2).
At the Endotracheal Suctioning (ETS) station, the total value for Gwet’s AC1 was 0.869 (95% CI [0.851–0.886]). Of the 26 items assessed at this station, 16 showed agreement percentages of >0.9, a further 9 items (specifically, items 4, 9, 10.1, 10.3, 10.5, 11.3, 11.4, 11.8 and 12) had percentages between 0.821 and 0.884, and item 13 had a value of 0.789 (Table 2).
At the Patient Care (PC) station, the total value for Gwet’s AC1 was 0.911 (95% CI [0.896–0.927]). Of the 21 items assessed at this station, 17 showed agreement percentages of >0.9, and the other 4 items (specifically, items 1e, 7, 14 and 17) had percentages between 0.810 and 0.894 (Table 2).
To improve the reliability of items with agreement percentages below 0.9, the investigators, through a focus group discussion, reviewed the checklist and proposed the following changes to make it easier for raters to understand the items. For item 10 of the CVC insertion station, the word "ALL" was added to clarify that the item should be marked as correct when all the measures are applied and as incorrect when only some of them are applied. The content of item 17 was expanded, adding "intermittent medication and/or bolus". For the ETS station, the word "ALL" was added to item 4 for the same reason as for the CVC station item. For the PC station, it was proposed to make the word “and” uppercase and bold in items 13 and 14 to highlight that both points in the item have to be performed (Table 3).
Discussion
Very good interrater reliability was achieved on all items of the NEUMOBACT checklist for use in the assessment of ICU nurses’ knowledge and skills in VAP and CRB prevention using simulation-based training. The checklist can be useful to assess skills before and after training in the Pneumonia Zero and Bacteraemia Zero project, to verify theory-to-practice transfer, which is a controversial aspect of training [13, 14] and, during debriefing (post-simulation feedback), to reinforce the scientific evidence behind actions and decisions for items that have been performed incorrectly [15–18]. Although this program has been conducted with ICU nursing professionals, it could be implemented in undergraduate studies, as various authors highlight the benefits of using simulation methodology during the pre-professional stage [19, 20].
Mogyoródi et al. [21] analysed before-and-after compliance with VAP prevention measures in clinical practice with a PowerPoint training programme and found a significant improvement in compliance three months after training, but a decrease to baseline at 12 months. Jansson et al. [22] found no impact on skills following simulation training on adherence to oral care recommendations. Gerolemou et al. [23] evaluated simulation-based training for CVC insertion and demonstrated an 85% reduction in the incidence of CRB infections in a critical care unit after the intervention.
Various VAP and CRB prevention checklists and care bundles have been published in the literature. These instruments were either created to assess compliance with pneumonia prevention measures in clinical practice [24–26] or designed to be applied during simulation of CVC-related skills [23, 27], but as far as we are aware, our NEUMOBACT checklist is the only one that contains all the necessary items for VAP and CRB prevention, created and validated by infection prevention and simulation experts [9].
Furthermore, unlike other instruments, the NEUMOBACT checklist requires no prior training or preparation for use during the simulation [28, 29]. This study, with a large sample of 190 raters, shows that if clear instructions are giving on using the checklist, (i.e., mark each item as correct/incorrect when those actions/decisions are observed during the simulation), it is feasible to achieve good interrater agreement for checklist completion.
Limitations
In the sample analysed, half of the nurses had postgraduate education. This level of education may have enhanced the nurses’ understanding of the checklist items compared to other nurses with undergraduate training only, because they are more familiar with complex ICU procedures, the theory behind those procedures, and simulation scenarios in general, as it is a widely used teaching method in postgraduate training [30]. To compensate for this possible bias, the raters were informed that the checklists used during the SIMULAZERO course were for research to assess interrater reliability, and that at no time would the checklists be analysed to assess participant performance of actions/decisions during the simulation.
It is essential to have validated rubrics in Spain to evaluate simulation-based activities [31]. Since developing an instrument is both costly and time-consuming, adapting existing instruments into another language offers several advantages. On one hand, it reduces research costs while preserving the psychometric properties of the original instrument. On the other hand, it allows for comparison of equally valid and reliable results with those from other national and international studies that have used the same instrument [32].
Conclusions
The interrater reliability of the NEUMOBACT checklist shows substantial agreement between pairs of raters and is therefore validated in this large sample of ICU nurses. The checklist requires no prior training and is validated and reliable for assessing nurses’ performance during simulation-based training in VAP and CRB preventive measures.
Supporting information
S1 Checklist. Checklist for reporting of studies of reliability and agreement (GRRAS checklist).
https://doi.org/10.1371/journal.pone.0313175.s001
(PDF)
S1 File. Description of the simulation stations used in the OSCE-format SIMULAZERO course.
https://doi.org/10.1371/journal.pone.0313175.s002
(DOCX)
Acknowledgments
The authors are grateful to the nurses who participated in the workshops at the Seville congress and to Maria Pérez Riart and Purificación Pérez Teran from the Hospital del Mar for their teaching and methodological support.
References
- 1.
Sociedad Española de Enfermería Intensiva y Unidades Coronarias [Internet]. [cited 2023 Nov 24]. Available from: https://seeiuc.org/
- 2. Raurell-Torredà M, Zaragoza-García I, Aliberch-Raurell AM, Sánchez-Chillón J, Torralba-Melero M, Arrogante O, et al. SIMULAZERO: taller de simulación para actualizar conocimientos y habilidades en la prevención de la neumonía asociada a ventilación mecánica y bacteriemia relacionada con catéter (Proyectos Zero). Enferm Intensiva. 2022;33:S45–55.
- 3.
Lioce L, editor. Healthcare Simulation Dictionary [Internet]. 2nd ed. Agency for Healthcare Research and Quality; 2020 [cited 2023 Jun 30]. Available from: https://www.ahrq.gov/patient-safety/resources/simulation/terms.html
- 4. Goh HS, Zhang H, Lee CN, Wu XV, Wang W. Value of Nursing Objective Structured Clinical Examinations: A Scoping Review. Nurse Educ. 2019;44(5):E1–6. pmid:30371544
- 5. Taylor I, Bing-Jonsson PC, Johansen E, Levy-Malmberg R, Fagerström L. The Objective Structured Clinical Examination in evolving nurse practitioner education: A study of students’ and examiners’ experiences. Nurse Educ Pract. 2019;37:115–23. pmid:31136916
- 6. Holzhausen Y, Maaz A, März M, Sehy V, Peters H. Exploring the introduction of entrustment rating scales in an existing objective structured clinical examination. BMC Med Educ. 2019;19(1):319. pmid:31438938
- 7. Erikson EJ, Edelman DA, Brewster FM, Marshall SD, Turner MC, Sarode VV, et al. The use of checklists in the intensive care unit: a scoping review. Crit Care. 2023;27(1):468. pmid:38037056
- 8. Raurell-Torredà M, Arrogante O, Aliberch-Raurell AM, Sánchez-Chillón FJ, Torralba-Melero M, Rojo-Rojo A, et al. Design and content validation of a checklist about infection-prevention performance of intensive care nurses in simulation-based scenarios. J Clin Nurs. 2024;33(8):3188–98. pmid:38348543
- 9. Kuiper JH. GRRAS checklist for reporting of studies of reliability and agreement. Available in:https://www.equator-network.org/wp-content/uploads/2012/12/GRRAS-checklist-for-reporting-of-studies-of-reliability-and-agreement.pdf
- 10. Mokkink LB, Prinsen CA, Patrick DL, Alonso J, Bouter LM, de Vet HC, et al. COSMIN Study Design checklist for Patient-reported outcome measurement instruments. [access september 5, 2024]. Available in: https://www.cosmin.nl/wp-content/uploads/COSMIN-study-designing-checklist_final.pdf#.
- 11. Gwet KL. Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters: Vol 2: Analysis of quantitative ratings. Advanced Analytics; 2021. 223–246 p.
- 12. Wongpakaran N, Wongpakaran T, Wedding D, Gwet KL. A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Med Res Methodol. 2013;13(1):61. pmid:23627889
- 13. Blot S, Conoscenti E. Why to monitor adherence to oral care protocols in intensive care units?. Aust Crit Care. 2023;36(2):169–170. pmid:34980521
- 14. Lombardo L, Ferguson C, George A, Villarosa AR, Villarosa BJ, Kong Ariana C, et al. Interventions to promote oral care regimen adherence in the critical care setting: A systematic review. Aust Crit Care. 2022;35(5):583–594. pmid:34764003
- 15. O’Grady NP. Prevention of Central Line–Associated Bloodstream Infections. Taichman DB, editor. N Engl J Med. 2023;389(12):1121–1131. pmid:37733310
- 16. Blakeman TC, Scott JB, Yoder MA, Capellari E, Strickland SL. AARC Clinical Practice Guidelines: Artificial Airway Suctioning. Respir Care. 2022;67(2):258–271. pmid:35078900
- 17. Garegnani LI, Giménez ML, Escobar Liquitay CM, Franco JVA. Oral hygiene interventions to prevent ventilator‐associated pneumonia: A network meta‐analysis. Nurs Crit Care. 2023;28(5):744–55.
- 18. Unahalekhaka A, Butpan P, Wongsaen R, Phunpae P, Preechasuth K. Contamination of antimicrobial-resistant bacteria on toothbrushes used with mechanically ventilated patients: A cross sectional study. Intensive Crit Care Nurs. 2022;68:103120. pmid:34391630
- 19. Díaz Agea JL, Ramos-Morcillo AJ, Amo Setien FJ, Ruzafa-Martínez M, Hueso-Montoro C, Leal-Costa C. Perceptions about the self-learning methodology in simulated environments in nursing students: a mixed study. Int J Environ Res Public Health. 2019 Nov 22;16(23):4646. pmid:31766613
- 20. Romero-Castillo R, Garrido-Bueno M, Fernández-León P. Nursing students’ perceptions and satisfaction with a self-learning methodology in simulated environments: A mixed-methods study. Nurse Educ Pract. 2024 Sep 12;80:104141. pmid:39293166
- 21. Mogyoródi B, Skultéti D, Mezőcsáti M, Dunai E, Magyar P, Hermann C, et al. Effect of an educational intervention on compliance with care bundle items to prevent ventilator-associated pneumonia. Intensive Crit Care Nurs. 2023;75:103342. pmid:36464606
- 22. Jansson MM, Syrjälä HP, Ohtonen PP, Meriläinen MH, Kyngäs HA, Ala‐Kokko TI. Effects of simulation education on oral care practices–a randomized controlled trial. Nurs Crit Care. 2017;22(3):161–168. pmid:28093837
- 23. Gerolemou L, Fidellaga A, Rose K, Cooper S, Venturanza M, Aqeel A, et al. Simulation-Based Training for Nurses in Sterile Techniques During Central Vein Catheterization. Am J Crit Care. 2014;23(1):40–48. pmid:24382616
- 24. Jam Gatell MR, Santé Roig M, Hernández Vian Ó, Carrillo Santín E, Turégano Duaso C, Fernández Moreno I, et al. Assessment of a training programme for the prevention of ventilator‐associated pneumonia. Nurs Crit Care. 2012;17(6):285–92. pmid:23061618
- 25. Behzadi F, Khanjari S, Haghani H. Impact of an education program on the performance of nurses in providing oral care for mechanically ventilated children. Aust Crit Care. 2019;32(4):307–313. pmid:30126677
- 26. Jansson MM, Ala-Kokko TI, Syrjälä HP, Kyngäs HA. Development and psychometric testing of ventilator bundle questionnaire and observation schedule. Am J Infect Control. 2014;42(4):381–384. pmid:24679564
- 27. Barsuk JH, Cohen ER, Mikolajczak A, Seburn S, Slade M, Wayne DB. Simulation-Based Mastery Learning Improves Central Line Maintenance Skills of ICU Nurses. J Nurs Adm. 2015;45(10):511–517. pmid:26425976
- 28. Rodríguez E, Chabrera C, Farrés M, Castillo J, Virumbrales M, Raurell M. Adaptation and validation of a Spanish version of the KidSIM Team Performance Scale. J Prof Nurs. 2021;37(5):795–803. pmid:34742507
- 29. Roldán-Merino J, Farrés-Tarafa M, Estrada-Masllorens JM, Hurtado-Pardos B, Miguel-Ruiz D, Nebot-Bergua C, et al. Reliability and validity study of the Spanish adaptation of the “Creighton Simulation Evaluation Instrument (C-SEI)”. Nurse Educ Pract. 2019;35:14–20. pmid:30640046
- 30. El Hussein MT, Harvey G, Bell N. The Influence of Nursing Simulation on Patient Outcomes and Patient Safety: A Scoping Review. Clin Simul Nurs. 2022;70:37–46.
- 31. Unver V, Basak T, Watts P, Gaioso V, Moss J, Tastan S, et al. The reliability and validity of three questionnaires: the Student Satisfaction and Self-Confidence in Learning Scale, Simulation Design Scale, and Educational Practices Questionnaire. Contemp Nurse. 2017;53(1):60–74. pmid:28084900
- 32. Escobar Bravo MÁ. Transcultural adaptation of health-related measurement instruments. Enferm Clin. 2004;14(2):102–6.