https://orcid.org/0000-0002-8246-0603
Abstract
This study is aimed at describing a protocol for a systematic review and meta-analysis to assess the effect of nontechnical skills training on the acquisition of knowledge, skills or attitudes, and changes in behavior at the workplace, of healthcare professionals working in intensive care units (ICUs), as well as the effect on outcomes at an organizational level. We will search for original studies in the PubMed/Medline, Scopus, Web of Science, Science Direct, EMBASE and PsycINFO databases. Studies with a clinical trial or quasi-experimental design will be included. Two reviewers will independently screen and assess the included studies, with any disagreements being resolved by a third reviewer. We will summarize the findings using a narrative approach and, if possible, conduct a quantitative synthesis (meta-analysis). We will conduct the protocol following the Preferred Reporting Items for Systematic Review and Meta-Analyses Protocols (PRISMA-P) guidelines. The review will summarize the current evidence on nontechnical skills training in ICUs, examining satisfaction with the training program, improvements in knowledge about nontechnical skills and the adoption of safety behaviors, as well as improvement in outcomes for the organization, such as mortality rates, length of stay and cost indicators. We expect that the systematic review could indicate effective strategies for training ICU professionals in nontechnical skills and also determine whether these strategies really improve the safety culture and professional knowledge and behaviors, as well as patient outcomes and safety.
Citation: Santiago Fernandes Pimenta ID, de Sousa Mata ÁN, Machado Bezerra IN, Carneiro Capucho H, de Souza Oliveira AC, Echevarría Pérez P, et al. (2023) Nontechnical skills training in intensive care units: Protocol for a systematic review and meta-analysis. PLoS ONE 18(1): e0280132. https://doi.org/10.1371/journal.pone.0280132
Editor: Muhammad Shahzad Aslam, Xiamen University - Malaysia Campus: Xiamen University - Malaysia, MALAYSIA
Received: October 20, 2021; Accepted: December 21, 2022; Published: January 6, 2023
Copyright: © 2023 Santiago Fernandes Pimenta et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: No datasets were generated or analysed during the current study.
Funding: This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Nontechnical skills (NTS), defined as ‘the cognitive, social and personal resource skills that complement technical skills, and contribute to safe and efficient task performance’ [1], have an important role in many high-reliability organizations, such as the aviation, nuclear and oil industries [1, 2].
Similar to these organizations, healthcare services have a complex socio-technical environment with a higher risk of errors [2]. Estimates indicate that at least 9% of hospital admissions are complicated by safety incidents, with a large number of them being preventable [3]. In intensive care units (ICUs), the prevalence of such events is higher than in other medical settings [4].
Analysis of incident reports in ICUs shows that human factors are an important underlying condition in safety incidents [5]. NTS such as teamwork, leadership, situational awareness, decision-making and task management are some of these behaviors [1, 5]. In a literature review by Reader et al. [6] at least 50% of the contributory factors in safety incidents were NTS related.
In light of this, NTS training seems to be especially valuable in ICU environments, it being necessary to identify the effect of training programs on health professionals’ performance. To evaluate the effects of training interventions in a work context, Kirkpatrick [7] propose a scale with four levels: reaction–which refers to the participant’s feedback of training experience; learning–the acquisition of knowledge, skills or attitudes by participants after training; behavior–changes in participant behavior at the workplace; and results–the measurable impact of training at an organizational level.
Some studies indicate that the implementation of NTS training in ICUs may provide better performance to the professionals and reduction of clinical complications to the patients. Mayer et al. [8] implemented TeamSTEPPS® training in two ICUs in a before-and-after study, and found significantly higher scores for observed behaviors such as team performance, leadership and mutual support after the intervention. At an organizational level, the average time for placing patients on extracorporeal membrane oxygenation (ECMO) was reduced, despite having a small effect size.
Haerkens et al. [9] implemented crew resource management training during a 3-year prospective study in an ICU with 2500–3000 admissions per year. The found an association between the implementation of the training program and a significant reduction of overall complication rate, from 66.4 at baseline to 50.9/1000 patients in the post-implementation year. In the same period, the cardiopulmonary resuscitation success rate increased from 19% to 67%, also associate with the intervention. However, none of the studies used a randomized and controlled design, which may affect the magnitude of the results.
Nevertheless, the impact of poor performance of NTS skills presents in safety incidents in healthcare services are very recognizable [6, 10] and no systematic review has investigated the effect of NTS training programs in the ICU context. Thus, this paper describes a protocol for a systematic review and meta-analysis that aims to assess the effect of NTS training on the reaction, knowledge and behavior of healthcare professionals working in ICUs, as well as its effect on outcomes at an organizational level.
Methods and analysis
Study registration and reporting
This systematic review study is registered on the PROSPERO database under number CRD42021244769. The protocol is based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols (PRISMA-P) guidelines [11]. The final report will be developed following PRISMA [12] and the Cochrane Handbook for Systematic Reviews of Interventions [13] and any changes to the protocol will be described in the Method section.
Eligibility criteria
Types of studies.
We will include randomized and nonrandomized clinical trials, quasi-experimental studies with a control group, interrupted time series studies and before-and-after studies without a control group.
Types of participants.
We will include studies with healthcare professionals working in ICUs, such as physicians, nurses, pharmacists, and respiratory therapists. We will also consider studies that include residents and interns, but only if they are integrated into the ICU staff.
Intervention.
We will consider all types of training programs that aim to improve NTS performance in a workspace context if they report data both before and after the intervention. For the purpose of this review, NTS are cognitive, social and personal skills that complement technical skills such as teamwork, situational awareness, task management and decision-making.
We will only consider interventions that focus on training a set of NTS, considering that being proficient in only a single skill is insufficient to improve safety in high-risk environments such as ICUs [1, 6].
The intervention could be delivered by a single methodology or by a combination of methods, such as e-learning modules, simulations, high-fidelity simulations, lectures or workshops. Furthermore, we will consider interventions of any duration and frequency.
Comparison.
We will consider studies with a control group that compare the intervention with no training or training in a different set of skills.
Outcomes.
The outcome measures will consider the four levels of Kirkpatrick’s scale [7]: reaction–participant’s feedback of training experience; learning–the acquisition of knowledge, skills or attitudes by participants after training; behavior–changes in participant behavior at the workplace; and results–the measurable impact of training at an organizational level.
As there are a variety of measures for each level of Kirkpatrick’s scale, the included studies may consider the following outcomes: indicators of satisfaction with the training program; mean scores in training assessments; changes in safety behaviors (e.g. teamwork, situational awareness); prevalence of adverse events or patient safety incidents; adoption of good practices or standard practices; morbidity or mortality rates; and cost indicators (e.g. return of investment rates).
Exclusion criteria.
We will exclude studies that: were implemented only with managers, administrators or leaders; provide an incomplete description of the training intervention or evaluation method; involved a large improvement strategy with multiple interventions; used coaching or mentoring intervention strategies; were pilot, validation or qualitative studies. Theses, reviews, cases reports, conference papers and abstracts were also excluded.
Search strategy
The systematic review will summarize the evidence of primary studies in the following databases: PubMed/Medline; Scopus; Web of Science; Science Direct; EMBASE, CINHAL and PsycINFO. In addition, the references of the included studies will be screened to search for other relevant studies that do not appear in the main search. No restrictions in terms of language or year of publication will be made. We will not search in gray literature.
The search strategy combined terms from the Medical Subject Headings (MeSH) and Embase Subject Headings (EMTREE), as well as using non-MeSH terms. Considering the absence of a specific framework for NTS in ICUs, the non-MeSH terms are selected based on the Anesthetists’ Non-Technical Skills (ANTS) system [14]. This framework is largely used across acute medical settings beyond anesthesia and is consistent enough to be used in ICUs [6]. Also, the NTS needed in various high-risk domains seem to be very analogous, considering that human behavior across these settings is also very similar [1].
The primary search strategy resulted in the following combination of keywords: “Non-technical skill*” OR "decision mak*" OR "Leader*" OR "Team Work*" OR "Situational Awareness" OR "Human factor*" AND "Intensive Care Unit*" AND "Training".
Study selection
After extraction of the records in all the databases, we will make an initial check for duplicates, following their proper removal. Two reviewers will independently screen the records by reading the titles and abstracts, and then read the full text to assess the eligibility criteria. Any disagreements between reviewers in any phase of study selection will be resolved by consulting a third reviewer.
To perform all phases of the study selection, we will use the Rayyan® application for systematic reviews [15].
Data extraction and management
Two reviewers will extract the data independently, using a standard electronic spreadsheet previously tested. The data will include identification of the studies (aim, year, main author, country, ICU type and study design), population characteristics (age, gender, profession), description of the intervention (educational strategy, duration and curricula), method of educational assessment (questionnaires, method of observation) and outcomes (mortality rates, cost indicators, length of stay, incidence of safety incidents, indicators of satisfaction with the training program, improvement in knowledge or safety behavior).
Dealing with missing data
In the case of interesting data missing or being unclear, the research team will try to contact the corresponding author by email, phone or correspondence. If this communication is unsuccessful, we will exclude the data from the analysis, covering this in the Discussion section.
Risk of bias and quality assessment
Two reviewers will independently assess the selected studies, using Version 2 of the Cochrane Risk of Bias (ROB 2) tool for randomized clinical trials or the Risk of Bias in Non-randomized Studies of Interventions (ROBINS-I) for nonrandomized clinical trials and quasi-experimental studies with a control group, classifying the risk of bias as low, high or unclear [13]. The EPOC Risk of Bias tool will be used to assess bias in interrupted time series studies and the National Heart, Lung, and Blood Institute quality assessment tool will be used for before-and-after studies without a control group [16].
In addition, the Grading of Recommendations Assessment, Development and Evaluation (GRADE) tool will be used to assess the quality of evidence [17]. To calculate the inter-rater reliability, we will use the kappa coefficient.
Data synthesis
We will carry out a narrative synthesis and produce summarizing tables considering the data extraction plan. Dichotomous outcomes will be summarized as risk ratios with 95% confidence interval (CI), whereas continuous outcomes will be summarized as mean differences with 95%CI. The p value of each result will also be considered [18].
If the included studies are methodologically homogeneous, we will conduct a meta-analysis. To verify the heterogeneity between studies, we will use the χ2 test with a significance of p < 0.05. In addition, the I2 statistic will be calculated to assess the consistency between studies, considering a value of 0% as no observed heterogeneity, up to 50% as a moderate level and 75% or higher as a substantial level of heterogeneity. If possible, we will perform a funnel plot analysis to indicate possible reporting biases and a linear regression approach to measure the asymmetry of the funnel plot. Review Manager software (RevMan V5.3.3) will be used for the data analysis.
Dissemination and ethics
We will publish the results of the systematic review in a peer-reviewed journal. In addition, the results will be disseminated in academic and health service spaces, such as at conferences or seminars. No ethical committee approval is required, given that the review will not use the personal data of professionals or patients.
Discussion
This protocol proposes a systematic review to summarize the current evidence on the effect of NTS training programs in ICUs from a professional perspective, as well for the patients and healthcare organizations.
Despite the technological and scientific advances in clinical practices, delivering high-quality healthcare services is still a major challenge to governments and organizations, especially with regard to patient safety [19, 20].
Inspired by high-reliability organizations, healthcare services incorporate many techniques and approaches to understand and solve safety problems. Understanding the role of human factors in patient safety is one of these approaches [21]. The literature demonstrates a close relation between poor NTS and safety incidents in ICUs [5, 6, 10]. However, reports about NTS training programs have particularly focused on surgical specialties [22–24].
Implications
The NTS training can help health professionals to perform procedures more safely and efficiently and are useful in daily clinical practice in ICUs [1, 6, 8, 9]. Skills such as gathering and interpreting information, anticipating future status, defining problems and selecting courses of action, and reviewing outcomes, are essential for every care process and should be addressed by training programs [1, 6, 14].
Safety incidents in ICUs are often related to processes such as medication preparations and administration, and the use of equipment and tubes [25, 26]. Incorporing or enhance NTS in clinical practice may help healthcare professionals to conduct procedures more safely, reducing avoidable safety incidents and providing better outcomes for the patients, such as reducing the length of stay and care-related infections, for example [8, 9].
In light of this, systematic reviews play an important role in summing up multiple initiatives for quality improvement and highlight the better practices to adopt when considering the variety of contexts [19, 27, 28]. We expect that this review could help healthcare services to implement NTS training in the ICU context, supporting the choices with evidence for more effective teaching methods that are well received by the professionals.
Limitations
We may find some limitations with this review. The inclusion of studies with before-and-after designs can compromise the quality of evidence provided by the review. This variety of study types included can also generate heterogeneity between studies, which could compromise a quantitative summary of the results (meta-analysis).
However, in quality improvement initiatives, as in Plan-Do-Study-Act cycles for example, randomized and controlled designs may not be an option due to limitations such as resource availability and ethical questions [29, 30]. In such context, nonrandomized studies assume a very important role to overcome this limitations and understand the effect of an intervention in clinical outcomes [31].
Futhermore, some methods of evaluation for each outcome could also vary, considering the large availability of instruments for measuring NTS in very different ways, such self-reported questionaries, and direct observation, for example [14].
Despite of this limitations, this research will provide an important overview of the use of NTS training programs in the ICUs context, that can colaborate with further interventions to improve care quality and safety.
Supporting information
S1 Checklist. PRISMA-P (Preferred Reporting Items for Systematic review and Meta-Analysis Protocols) 2015 checklist: Recommended items to address in a systematic review protocol.
https://doi.org/10.1371/journal.pone.0280132.s001
(DOC)
References
- 1.
Flin R, O’Connor P, Crichton M. Safety at the Sharp End: A Guide to Non-Technical Skills, 1st ed. London: CRC Press; 2017.
- 2. Reason J. The contribution of latent human failures to the breakdown of complex systems. Philos. Trans. R. Soc. Lond. B 1990;327(1241):475–84. pmid:1970893
- 3. de Vries EN, Ramrattan MA, Smorenburg SM, Gouma DJ, Boermeester MA. The incidence and nature of in-hospital adverse events: A systematic review. BMJ Qual. Saf. 2008;17(3):216–23. pmid:18519629
- 4. Panagioti M, Khan K, Keers RN, Abuzour A, Phipps D, Kontopantelis E, et al. Prevalence, severity, and nature of preventable patient harm across medical care settings: Systematic review and meta-analysis. BMJ 2019;366:l4185. pmid:31315828
- 5. Pronovost PJ, Thompson DA, Holzmueller CG, Lubomski LH, Dorman T, Dickman F, et al. Toward learning from patient safety reporting systems. J. Crit. Care 2006;21(4):305–15. pmid:17175416
- 6. Reader T, Flin R, Lauche K, Cuthbertson BH. Non-technical skills in the intensive care unit. Br. J. Anaesth. 2006;96(5):551–9. Available from: pmid:16567346
- 7.
Kirkpatrick L. Evaluating Training Programs: The Four Levels, 3th edn. San Francisco: Berrett-Koehler Publishers; 2006.
- 8. Mayer CM, Cluff L, Lin W-T, Willis TS, Stafford RE, Williams C, et al. Evaluating efforts to optimize TeamSTEPPS implementation in surgical and pediatric intensive care units. Jt. Comm. J. Qual. Patient Saf. 2011;37(8):365–AP3. pmid:21874972
- 9. Haerkens MHTM, Kox M, Lemson J, Houterman S, Van Der Hoeven JG, Pickkers P. Crew resource management in the intensive care unit: A prospective 3-year cohort study. Acta Anaesthesiol. Scand. 2015;59(10):1319–29. pmid:26079640
- 10. Siu J, Maran N, Paterson-Brown S. Observation of behavioural markers of non-technical skills in the operating room and their relationship to intra-operative incidents. Surgeon 2016;14(3):119–128. pmid:25022767
- 11. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) 2015 statement. Syst. Rev. 2015;4(1):1. pmid:25554246
- 12. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ 2021;n71. pmid:33782057
- 13.
Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al. (eds.). Cochrane Handbook for Systematic Reviews of Interventions, 2nd edn. John Wiley & Sons, Ltd; 2019. https://training.cochrane.org/handbook/current
- 14. Flin R. Identifying and training non-technical skills for teams in acute medicine. BMJ Qual. Saf. 2004;13(Suppl._1):i80–4. pmid:15465960
- 15. Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan: A web and mobile app for systematic reviews. Syst. Rev. 2016;5(1):210. pmid:27919275
- 16. Ma LL, Wang YY, Yang ZH, Huang D, Weng H, Zeng XT. Methodological quality (risk of bias) assessment tools for primary and secondary medical studies: What are they and which is better? Mil. Med. Res. 2020;7(1):1–11. pmid:32111253
- 17. Andrews JC, Schünemann HJ, Oxman AD, Pottie K, Meerpohl JJ, Coello PA, et al. GRADE guidelines: 15. Going from evidence to recommendation—determinants of a recommendation’s direction and strength. J. Clin. Epidemiol. 2013;66(7):726–35. pmid:23570745
- 18. Campbell M, McKenzie JE, Sowden A, Katikireddi SV, Brennan SE, Ellis S, et al. Synthesis without meta-analysis (SWiM) in systematic reviews: Reporting guideline. BMJ 2020;368:1–6. pmid:31948937
- 19. McDonald KM. Considering context in quality improvement interventions and implementation: Concepts, frameworks, and application. Acad. Pediatr. 2013;13(6):S45–53. pmid:24268084
- 20.
World Health Organization, World Bank Group, OECD. Delivering Quality Health Services. Washington (D.C.): World Bank Group; 2018. https://www.worldbank.org/en/topic/universalhealthcoverage/publication/delivering-quality-health-services-a-global-imperative-for-universal-health-coverage
- 21. Naik VN, Brien SE. Simulation: A means to address and improve patient safety. Can. J. Anesth. 2013;60(2):192–200. pmid:23239487
- 22. Robertson ER, Hadi M, Morgan LJ, Pickering SP, Collins G, et al. Oxford NOTECHS II: A modified theatre team non-technical skills scoring system. PLoS One 2014;9(3):e90320. pmid:24594911
- 23. Robertson E, Morgan L, New S, Pickering S, Hadi M, et al. Quality improvement in surgery combining lean improvement methods with teamwork training: A controlled before–after study. PLoS One 2015;10(9):e0138490. pmid:26381643
- 24. Komasawa N, Berg BW, Minami T. Problem-based learning for anesthesia resident operating room crisis management training. PLoS One 2018;13(11):e0207594. pmid:30452480
- 25. Merino P, Álvarez J, Martín MC, Alonso A, Gutiérrez I. Adverse events in Spanish intensive care units: the SYREC study. Int J Qual Health Care 2012;24(2):105–113. pmid:22190588
- 26. Corwin GS, Mills PD, Shanawani H, Hemphill RR. Root cause analysis of ICU adverse events in the Veterans health administration. Jt Comm J Qual Patient Saf 2017;43(11):580–590. pmid:29056178
- 27. Kilpatrick K, Paquette L, Jabbour M, Tchouaket E, Fernandez N, et al. Systematic review of the characteristics of brief team interventions to clarify roles and improve functioning in healthcare teams. PLoS One 2020;15(6):e0234416. pmid:32520943
- 28. Grimshaw J, McAuley LM, Bero LA, Grilli R, Oxman AD, Ramsay C, et al. Systematic reviews of the effectiveness of quality improvement strategies and programmes. BMJ Qual Saf 2003;12(4):298–303. pmid:12897365
- 29. Eccles M, Grimshaw J, Campbell M, Ramsay C. Research designs for studies evaluating the effectiveness of change and improvement strategies. BMJ Qual Saf 2003;12(1):47–52. pmid:12571345
- 30. Grimshaw J, Campbell M, Eccles M, Steen N. Experimental and quasi-experimental designs for evaluating guideline implementation strategies. Fam Pract 2000;17(suppl_1):S11–S16. pmid:10735262
- 31.
Penson DF, Wei JT. Clinical research methods for surgeons, 1st ed. Totowa: Humana Press; 2006.