https://orcid.org/0000-0002-0101-8208
Figures
Abstract
Fiberoptic intubation for a difficult airway requires significant experience. Traditionally only normal airways were available for high fidelity bronchoscopy simulators. It is not clear if training on difficult airways offers an advantage over training on normal airways. This study investigates the added value of difficult airway scenarios during virtual reality fiberoptic intubation training. A prospective multicentric randomized study was conducted 2019 to 2020, among 86 inexperienced anesthesia residents, fellows and staff. Two groups were compared: Group N (control, n = 43) first trained on a normal airway and Group D (n = 43) first trained on a normal, followed by three difficult airways. All were then tested by comparing their ORSIM® scores on 5 scenarios (1 normal and 4 difficult airways). The final evaluation ORSIM® score for the normal airway testing scenario was significantly higher for group N than group D: median score 76% (IQR 56.5–90) versus 58% (IQR 51.5–69, p = 0.0039), but there was no difference in ORSIM® scores for the difficult intubation testing scenarios. A single exposure to each of 3 different difficult airway scenarios did not lead to better fiberoptic intubation skills on previously unseen difficult airways, when compared to multiple exposures to a normal airway scenario. This finding may be due to the learning curve of approximately 5–10 exposures to a specific airway scenario required to reach proficiency.
Citation: Cailleau L, Geeraerts T, Minville V, Fourcade O, Fernandez T, Bazin JE, et al. (2023) Is there a benefit for anesthesiologists of adding difficult airway scenarios for learning fiberoptic intubation skills using virtual reality training? A randomized controlled study. PLoS ONE 18(1): e0281016. https://doi.org/10.1371/journal.pone.0281016
Editor: Silvia Fiorelli, Sapienza University of Rome: Universita degli Studi di Roma La Sapienza, ITALY
Received: May 7, 2022; Accepted: January 12, 2023; Published: January 27, 2023
Copyright: © 2023 Cailleau et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All ORSIM data have been uploaded to the DRYAD server (https://doi.org/10.5061/dryad.qjq2bvqjc).
Funding: The author(s) received no specific funding for this work.
Competing interests: Dr. Paul Baker retains a commercial interest in the ORSIM bronchoscopy simulator and has been involved in the design and development of this study. He is one of the owners of Airway Simulation Limited, which manufactures the ORSIM. In this study, he was excluded from data entry, scoring, or data analysis. His commercial involvement in the ORSIM was made clear to the other authors and the subjects at the commencement of the study. The authors who were involved in scoring and data analysis for this study have no commercial interest in the ORSIM.
Introduction
The management of difficult airways is associated with significant mortality in both the Intensive Care Unit (ICU) and the operating room [1, 2]. Airway management devices have evolved and guidelines in airway management have changed over the last decade [3–7]. Flexible bronchoscopy (FB) remains an accepted first choice for patients with known difficult intubation or as a rescue technique for non-anticipated difficult intubations [8]. However, learning FB remains challenging because of its infrequent use which is further compounded by increasing use of video-laryngoscopes [4, 8–11]. Simulation appears to be an efficient tool for training health care providers using otherwise rare clinical scenarios [12–14]. Many studies have indeed shown the value of simulation for bronchoscopy training but none of them have studied its use during difficult airway management [15–26].
Until the recent development of a virtual reality fiberoptic simulator with multiple difficult airway scenarios, mostly normal airways were available for high fidelity bronchoscopy simulators [15, 17, 21, 22, 27–29]. It is, however, not clear if training on difficult airway scenarios offers a significant advantage over training on normal airway scenarios [30].
We hypothesized that inexperienced anesthesia providers would improve by training on a virtual reality simulator with difficult airway scenarios when compared to training on normal airway scenarios (primary outcome: improved performance score, secondary outcome: success rates, times, collision avoidance and perceived level of difficulty).
Materials and methods
This study recruited anesthesia residents, fellows and staff physicians who had performed 10 or less fiberoptic intubations from June 2019 to March 2020. Participating centers were: Toulouse (France), Clermont-Ferrand (France), Coventry (United Kingdom) and Oxford (United Kingdom). The study protocol was not registered but communicated to all participating centers.
Consent
Each participating center obtained ethics approval through their local Research Ethics Committee. This study’s protocol has been registered with the Comité d’Ethique pour les Recherches (CER) of Toulouse University under the number: 2018–099 for the centers in Toulouse and Clermont-Ferrand (France) and by the Health and Research Authority in Wales (Ref 19/HRA/5142) for the centers in Oxford and Coventry (United Kingdom). All participants gave written informed consent.
Virtual reality high fidelity simulator
The ORSIM® virtual reality bronchoscopy simulator (Airway Simulation Limited, Auckland, New Zealand) incorporates a replica video bronchoscope, desktop sensor module, and dedicated laptop computer. The ORSIM® contains software recreating a high-fidelity FB intubation scenario for the user. It includes learning modules on airway anatomy and dexterity and uses both normal and difficult airway intubation scenarios. (https://www.orsim.co.nz/design).
ORSIM® score
Validity and reliability of the ORSIM® simulator as an assessment tool have been shown in a previous study by Baker et al. and relied on external scoring of a large number of video recording, which was not felt to be feasible for research [30]. Their results provided them the basis for establishing computer-generated metrics [30] and the development of a proprietary ORSIM® score. This ORSIM® Score (which has not been validated against expert scores) appears on the ORSIM® Session Results screen after the participant finishes a given scenario and was recorded to grade their performance.
Gopher-module
To assess pre-test dexterity of each participant, the ‘Gopher’ module, contained in the ORSIM®, was used: it consists of a challenge game to catch nine different gopher figures using the ORSIM® bronchoscope.
Data collection
Demographic data were collected including age, gender, dominant hand, years of residency training, previous FB experience, previous ORSIM® and video game experience. The ORSIM® simulator’s data were collected including type of scenario, time to successful completion, collision avoidance percentage score and minimal oxygen saturation, ORSIM® scores for each scenario and the degree of difficulty felt for each of the scenarios as judged by the study subjects.
Study design
The participants recruitment, randomization and allocation are displayed in Fig 1. After obtaining written informed consent, a study form was completed. Participants then watched a video explaining the functionalities of the ORSIM® simulator and the flexible bronchoscope. A pre-test dexterity test (Gopher module) was done before and then again after five minutes of hands-on familiarization with the bronchoscope simulator.
Afterwards participants were randomized (by random number drawing) into two groups: Group N trained on normal airway scenarios up to a maximum of 40 minutes and group D trained instead on one normal and then three difficult airway scenarios (a retropharyngeal abscess, an epiglottic cyst, and a macroglossia). Each of the participants in group D had up to 10 minutes per scenario before being prompted to move to the next scenario (i.e. up to a total of 40 minutes). Completion times, ORSIM® scores, collision avoidance, lowest oxygen saturation, perceived level of difficulty on a Visual Analogic Scale (VAS) from 0 (easy) to 100 (very difficult) were recorded.
The final step consisted of testing the participants on their fiberoptic difficult intubation skills by using one normal and four difficult scenarios (airway trauma, severe epiglottitis, false cord cyst and angioneurotic oedema). Each participant had to try to succeed once in a maximum of ten minutes for each scenario. Times, ORSIM® scores, collision avoidance, lowest oxygen saturation, perceived level of difficulty on a VAS from 0 (easy) to 100 (very difficult) were recorded. The investigator was not allowed to guide or help the participant to solve scenarios. The investigator’s role was strictly to observe and keep time, as well as manage any technical difficulty with the ORSIM® simulator.
Primary outcome
ORSIM® scores for the final test scenarios were compared between the two groups for each scenario.
Secondary outcomes
Times and ORSIM® scores in normal airway scenarios before and after training were compared between the two groups. Successful rates, times, collision avoidance and perceived level of difficulty were also compared.
Statistical analyses
Redcap® software (Research Electronic Data Capture®) was used to collect data [31, 32]. R® software (R Foundation for Statistical Computing, Vienna, Austria) was used for the statistical analyses. Power of this study had been estimated by using the data from the previous ORSIM® study by Baker et al. [30]: 150 participants were needed to show a significant difference of 25% with a power of 90%. A Chi square test and Fisher’s exact test were used to test differences between group N and D concerning categorial variables. A T-test (or Wilcoxon-Mann-Whitney for not normally distributed data) was used to test differences concerning continuous or ordinal variables. Statistical significance was taken as p<0.05. Results are expressed in median and interquartile ranges or in numbers and percentages.
Results
Recruitment and exclusions
The study began in June 2019 and was stopped in March 2020 due to the Covid-19 pandemic. This analysis includes the 86 participants out of the 150 expected enrolments (43 in group N, 43 in group D) from Toulouse (n = 25) and Clermont-Ferrand (n = 25) in France, the University Hospitals Coventry and Warwickshire NHS Trust (n = 16) and Oxford University Hospitals NHS Foundation Trust (n = 20) in the UK.
105 participations had been recruited, but nine were excluded because of missing or invalid data and a further nine because of a protocol violation (they had already performed more than 10 fiberoptic intubations). Finally, one participant had been recruited twice: only the first of those attempts was kept for analysis.
Demographics and pre-test dexterity
Table 1 shows the demographic data and Table 2 shows the pre-text dexterity. There was no difference in experience level with the fiberoptic bronchoscope or video game expertise between groups N and D and there was no difference in pre-test dexterity between the two groups.
Final ORSIM® score (primary outcome)
Table 3 shows the comparison of both groups ORSIM® scores during the final testing (primary outcome). The final test ORSIM® score for the normal airway scenario was significantly higher for group N than group D: median score 76% (IQR 56.5–90) versus 58% (IQR 51.5–69) respectively (p = 0.0039). There was no statistically significant difference between the two groups for the final test ORSIM® score on any of the difficult intubation scenarios (Table 3).
Performance evolution on normal airways
During the first try, at the beginning of training, there was no difference between groups in their performance on the normal airway scenario for ORSIM® scores, time to succeed and collision avoidance. During the final testing on the normal airway scenario (after completion of the training) the ORSIM® score for group N was significantly higher and their times and estimated level of difficulty were significantly decreased when compared to Group D (Table 4).
Comparison of perceived level of difficulty between groups during final testing
There was a significant difference of perceived level of difficulty on a VAS (0 to 100) between the two groups concerning the severe epiglottitis scenario as it appeared easier to group D than to group N: 66 (57–80) versus 79 (68–90) for the severe epiglottitis, p<0.05) (Table 5).
Discussion
This study shows a significant performance improvement for fiberoptic intubations on normal airways for the group who trained only on normal airways. In contrast, training on a variety of difficult airways did not translate to performance improvement on (other) difficult airway scenarios or on a normal airway.
Simulation appears as a new tool allowing learners to achieve a satisfactory skill level before practice. Graeser et al. [26] showed that simulator training allows for entry of the learning curve of airway management at a higher level. J.E Smith et al. [33] described a learning curve of 10 fiberoptic intubations to achieve a satisfactory competence and 30–50 fiberoptic intubations to reach expert level. Repetition of a same scenario seems to be the main determinant of learning. In our study, participants in group N had performed a median of 12 successive normal intubations before being tested (over a median of 15 minutes, Table 1), compared to only one single attempt in group D. In group D, each scenario could only be done once and participants in group D had thus performed the balance of their training on various difficult airways (over a median total of 14 minutes, Table 1). As our study shows, that training did not translate into better performance on the normal airway scenario.
In their study on validating the ORSIM® [30], Baker et al. proposed a score of 70 to characterize an expert level. Our study shows a median ORSIM® score of 76% after the 12 attempts in the final testing phase for group N on the normal airway, suggesting an satisfactory competence, or even expert level had been acquired after their repetitive training with this same normal airway scenario. On the other hand, for group D, the median ORSIM® score for each scenario remained well below 70%. It would be interesting to see if the ORSIM® score during final testing for group D would have improved significantly if Group D also would have trained until achieving ‘expert level’ (score of at least 70%) on each of their difficult training scenarios, but our experimental design did not allow to study that (and would have likely led to group D training for a much longer total time compared to group N, thus introducing a confounder).
The main finding of our study is thus that the learning may initially be limited to the actual scenario and the degree with which learning extends to other scenarios remains unclear (but clinically important). It may be necessary to introduce repetitive training sessions of each difficult scenario before the full benefits would become apparent.
Although training on a variety of difficult airways for group D did not translate to performance improvement, it did seem to lead to a reduction of perceived difficulty level some scenarios (for the severe epiglottitis scenario). Similarly, acquisition of expertise level for group N significantly reduced the perception of difficulty on the normal airway scenario. In practice, we can often observe that stress and perception of difficulty can have a negative impact on performance—and training is probably the only way to remedy this [34]. Our study shows that training on difficult airway scenarios could indeed reduce the stress during subsequent attempts on difficult airway scenarios.
Yang et al. [35] showed that psychometric skills can predict the acquisition of procedural skill performance and Louridas et al. [36], looking at laparoscopic surgery, investigated the value of psychometric testing to predict the technical performance of new residents. It would be interesting to study the relationship between dexterity and time to reach an expert level for fiberoptic intubation skills for a ‘competence by design’ approach to teach fiberoptic intubation.
Limitations of the study
Due to the unforeseen circumstances posed by COVID-19 we were forced to halt trial recruitment and decided to perform an unplanned interim analysis of the results. We put in place a plan for a single interim analysis with a symmetrical stopping rule for a p< 0.002 using the Pocock rule. When we identified a p = 0.0039 for our primary outcome in the direction of improved outcomes for the normal group we decided to stop the trial based on futility. While the number of participants in our study was smaller than planned, the magnitude of the estimated effect sizes was consistently small. Furthermore, the group trained on the difficult scenarios were as likely to show worse results as they were to show improved results. For these two reasons we postulate that the inclusion of more participants would not change the primary outcome.
Some participants in both groups had already used the ORSIM® simulator, however the numbers were small and also well distributed across both study groups, thus unlikely to create a bias. While it was originally contemplated to exclude participants with previous ORSIM® experience, it was felt that this would unnecessarily restrict the recruitment in view of the fact that the use of ORSIM® simulation was expanding rapidly.
In our study, we only assessed the level 2 of Kirkpatrick’s model of training evaluation: the testing of skills based on training [37]. It will be important to investigate actual changes in clinical practice (level 3) and ultimately changes in patient outcome (level 4) when evaluating the role of the ORSIM® simulator. The latter steps present significant logistical and ethical challenges and may involve the creation of actual patient airway through 3 D printer technology as an interim step.
Conclusion
This study was not able to demonstrate a benefit of adding difficult airway scenarios for learner’s performance during intubation training with a replica flexible bronchoscope. It only showed that repeated training on the ORSIM® simulator with the same scenario (a normal airway) led to increased intubation skills for the normal airway and that this skill was specific for the normal airway scenario. Training on normal airways did not predict better skills on difficult airways (and vice versa).
Subsequent studies should focus on the acquisition of a level of expertise in difficult airways and its reproducibility from one difficult airway scenario to another in order to validate this simulation model and better prepare the learner for clinical practice.
Acknowledgments
Professor Richard Cooper added valuable advice at the beginning of the study. Dr. Paul Baker assisted with the design and development of the study and provided a thoughtful review of the draft manuscript.
References
- 1. Cook TM, MacDougall-Davis SR. Complications and failure of airway management. Br J Anaesth. 2012; 109:i68–i85. pmid:23242753
- 2. Cook TM, Woodall N, Harper J, Benger J. Major complications of airway management in the UK: results of the Fourth National Audit Project of the Royal College of Anaesthetists and the Difficult Airway Society. Part 2: intensive care and emergency departments. Br J Anaesth. 2011; 106(5):632–42. pmid:21447489
- 3. McNarry AF, Patel A. The evolution of airway management–new concepts and conflicts with traditional practice. Br J Anaesth. 2017;119(suppl_1):i154–i166. pmid:29161401
- 4. Wanderer JP, Ehrenfeld JM, Sandberg WS, Epstein RH. The changing scope of difficult airway management. Can J Anesth. 2013; 60(10):1022–4. pmid:23897492
- 5. Langeron O, Bourgain J-L, Francon D, Amour J, Baillard C, Bouroche G, et al. Difficult intubation and extubation in adult anesthesia. Anesth Crit Care Pain Med. 2018; 37(6):639–651.
- 6. Xu Z, Ma W, Hester DL, Jiang Y. Anticipated and unanticipated difficult airway management: Curr Opin Anaesthesiol. 2018; 31(1):96–103. pmid:29176376
- 7. Sklar MC, Detsky ME. Emergent airway management of the critically ill patient: current opinion in critical care. Curr Opin Crit Care. 2019; 25(6):597–604. pmid:31490206
- 8. Collins SR, Blank RS. Fiberoptic Intubation: An Overview and Update. Respir Care. 2014; 59(6):865–78. pmid:24891196
- 9. Ahmad I, Onwochei DN, Muldoon S, Keane O, El-Boghdadly K. Airway management research: a systematic review. Anaesthesia. 2019; 74(2):225–36. pmid:30460982
- 10. Moore A, Schricker T. Awake videolaryngoscopy versus fiberoptic bronchoscopy: Curr Opin Anaesthesiol. 2019; 32(6):764–768. pmid:31356364
- 11. Kovacs G, Sowers N. Airway Management in Trauma. Emerg Med Clin North Am. 2018; 36(1):61–84. pmid:29132582
- 12. Wheeler DW, Williams CE, Merry AF. Pulling the plug on ad hoc critical incident training. Br J Anaesth. 2009; 103(2):145–7. pmid:19596758
- 13. Benhamou D, Roulleau P, Trabold F. Simulation in healthcare: a new teaching method to improve patient safety and quality of care. Réanimation. 2013; 22(S2):383–90.
- 14. L’Her E, Geeraerts T, Desclefs JP, Benhamou D, Blanie A, Cerf C, et al. Simulation-based teaching in critical care, anesthesia and emergency medicine. Anesth Crit Care Pain Med. 2020; 39(2):311–326.
- 15. Colt HG, Crawford SW, Galbraith O. Virtual Reality Bronchoscopy Simulation. Chest. 2001;120(4):1333–9.
- 16. Samuelson ST, Burnett G, Sim AJ, Hofer I, Weinberg AD, Goldberg A, et al. Simulation as a set-up for technical proficiency: can a virtual warm-up improve live fibre-optic intubation? B J Anesth. 2016; 116(3):398–404. pmid:26821699
- 17. Latif RK, Bautista A, Duan X, Neamtu A, Wu D, Wadhwa A, et al. Teaching basic fiberoptic intubation skills in a simulator: initial learning and skills decay. J Anesth. 2016; 30(1):12–9. pmid:26493397
- 18. Binstadt E, Donner S, Nelson J, Flottemesch T, Hegarty C. Simulator Training Improves Fiber-optic Intubation Proficiency among Emergency Medicine Residents. Acad Emerg Med. 2008; 15(11):1211–4. pmid:18699826
- 19. Giglioli S, Boet S, De Gaudio AR, Linden M, Schaeffer R, Bould MD, et al. Self-directed deliberate practice with virtual fiberoptic intubation improves initial skills for anesthesia residents. Minerva Anestesiol. 2012; 78(4):456–61. pmid:22310190
- 20. Chandra DB, Savoldelli GL, Joo HS, Weiss ID, Naik VN. Fiberoptic Oral Intubation: The Effect of Model Fidelity on Training for Transfer to Patient Care. Anesthesiology. 2008; 109(6):1007–13. pmid:19034097
- 21. Moorthy K, Smith S, Brown T, Bann S, Darzi A. Evaluation of Virtual Reality Bronchoscopy as a Learning and Assessment Tool. Respiration. 2003; 70(2):195–9. pmid:12740517
- 22. Martin KM, Larsen PD, Segal R, Marsland CP. Effective Nonanatomical Endoscopy Training Produces Clinical Airway Endoscopy Proficiency: Anesth Analg. 2004; 99(3):938–44. pmid:15333435
- 23. Gopal M, Skobodzinski AA, Sterbling HM, Rao SR, LaChapelle C, Suzuki K, et al. Bronchoscopy Simulation Training as a Tool in Medical School Education. Ann Thorac Surg. 2018; 106(1):280–286. pmid:29530773
- 24. Goldmann K, Steinfeldt T. Acquisition of basic fiberoptic intubation skills with a virtual reality airway simulator. J Clin Anesth. 2006; 18(3):173–8. pmid:16731318
- 25. Marsland CP, Robinson BJ, Chitty CH, Guy BJ. Acquisition and Maintenance of Endoscopic Skills: Developing an endoscopic dexterity training system for anesthesiologists. J Clin Anesth 2002; 14(8):615–9 pmid:12565124
- 26. Graeser K, Konge L, Kristensen MS, Ulrich AG, Hornbech K, Ringsted C. Airway management in a bronchoscopic simulator based setting: An observational study. Eur J Anaesthesiol. 2014; 31(3):125–30. pmid:24247408
- 27. Rowe R, Cohen RA. An Evaluation of a Virtual Reality Airway Simulator. Anesth Analg. 2002; 95(1):62–6. pmid:12088944
- 28. Casso G, Schoettker P, Savoldelli GL, Azzola A, Cassina T. Development and Initial Evaluation of a Novel, Ultraportable, Virtual Reality Bronchoscopy Simulator: The Computer Airway Simulation System. Anesth Analg. 2019; 129(5):1258–64. pmid:29481426
- 29. Schebesta K, Hüpfl M, Rössler B, Ringl H, Müller MP, Kimberger O. Degrees of Reality: Airway Anatomy of High-fidelity Human Patient Simulators and Airway Trainers. Anesthesiology. 2012; 116(6):1204–9. pmid:22481118
- 30. Baker PA, Weller JM, Baker MJ, Hounsell GL, Scott J, Gardiner PJ, et al. Evaluating the ORSIM ® simulator for assessment of anaesthetists’ skills in flexible bronchoscopy: aspects of validity and reliability. Br J Anaesth. 2016; 117(S1):i87–i91.
- 31. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—A metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009; 42(2):377–81. pmid:18929686
- 32. Harris PA, Taylor R, Minor BL, Elliott V, Fernandez M, O’Neal L, et al. The REDCap consortium: Building an international community of software platform partners. J Biomed Inform. 2019; 95:103208. pmid:31078660
- 33. Smith JE, Jackson APF, Hurdley J, Clifton PJM. Learning curves for fibreoptic nasotracheal intubation when using the endoscopic video camera. Anaesthesia. 1997; 52(2):101–6. pmid:9059089
- 34. Barré J, Michelet D, Truchot J, Jolivet E, Recanzone T, Stiti S, et al. Virtual Reality Single-Port Sleeve Gastrectomy Training Decreases Physical and Mental Workload in Novice Surgeons: An Exploratory Study. Obes Surg. 2019; 29(4):1309–1316. pmid:30689172
- 35. Yang Y, Gupta A, Gillett L, Kenig M, Athanassoglou V, Chuan A. Innate visuospatial and psychomotor ability influences early learning of flexible fibreoptic videobronchoscopy. Br J Anaesth. 2019; 123(2):e164–e166. pmid:31202560
- 36. Louridas M, Quinn LE, Grantcharov TP. Predictive value of background experiences and visual spatial ability testing on laparoscopic baseline performance among residents entering postgraduate surgical training. Surg Endosc. 2016; 30(3):1126–1133. pmid:26092028
- 37.
Kirkpatrick DL. Evaluation of training. In: Training and development handbook: A guide to human resource development. In Craig R. L. (Ed.) (2nd ed); McGraw Hill, New York., p. 301–319. 1976