https://orcid.org/0000-0003-2994-6900
Figures
Abstract
Background
Point-of-care ultrasound is one of useful diagnostic tools in emergency medicine practice and considerably depends on physician’s performance. This study was performed to evaluate performance improvements and favorable attitudes through structured cardiac ultrasound program for emergency medicine residents.
Methods
Retrospective observational study using the point-of-care ultrasound (PoCUS) database in one tertiary academic-teaching hospital emergency department has been conducted. Cardiac ultrasound education and rotation program has been implemented in emergency medicine residency program. Structured evaluation sheet for cardiac ultrasound and questionnaire toward PoCUS have been developed. An early-phase and a late-phase case were selected randomly for each participant. Two emergency medicine specialists with expertise in PoCUS evaluated saved images independently. We used a paired t-test to compare the performance score of each phase and the results of the questionnaire. Multivariable linear regression analysis was conducted to evaluate the association between the characteristics of participants and performance improvements.
Results
During the study period, a total of 1,652 bedside cardiac ultrasounds were administered. Forty-six examinations conducted by 23 emergency medicine residents were randomly selected for analysis. The performance score increased from 39.5 to 56.1 according to expert A and 45.3 to 62.9 according to expert B (p-value <0.01 for both). The average questionnaire score, which was analyzed for 17 participants, showed improvement from 18.9 to 20.7 (p-value <0.01). In multivariable linear regression analysis, younger age, higher early-phase score and higher confidence had a negative association with a greater improvement of performance, while the number of examinations had a positive association.
Citation: Kim KH, Jung JY, Park JW, Lee MS, Lee YH (2021) Operating bedside cardiac ultrasound program in emergency medicine residency: A retrospective observation study from the perspective of performance improvement. PLoS ONE 16(4): e0248710. https://doi.org/10.1371/journal.pone.0248710
Editor: Robert Ehrman, Wayne State University, UNITED STATES
Received: October 1, 2020; Accepted: March 3, 2021; Published: April 2, 2021
Copyright: © 2021 Kim et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: The authors received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Point-of-care ultrasound (PoCUS) has been used frequently and variously in emergency clinical practice because it can reduce cost [1] and can be used as an additional diagnostic test [2] that provides important clinical information in a very short time. Procedures have also been proven to benefit from PoCUS, e.g., both guidance and confirmation [3, 4]. Because ultrasound performance depends on the physician’s ability, PoCUS training should be a core component in the education of emergency physicians, especially those who are in residency programs [5]. Adding PoCUS training to an education program is considered an important milestone, [6] and many programs have implemented such training, demonstrating feasibility and participant satisfaction [7, 8]. In the United States, PoCUS has been requirement of residency training in emergency medicine [9].
Cardiac ultrasound would be a good performance indicator of PoCUS due to its well established standards for application and evaluation [10]. Furthermore, cardiac ultrasound is known to be useful for diagnosis in emergency departments [11–14]. A recent study by Davood et al. showed that emergency medicine residents can perform bedside cardiac ultrasound in the emergency department after several workshops and that this procedure yields comparable quality to traditional cardiac ultrasound performed by cardiologists [15].
Improvements in practitioners’ skills, including in acquiring proper images and interpreting the results, have not been well investigated in previous studies. The evaluation tools are usually limited to objective structured clinical exams (OSCEs) or knowledge tests [16, 17]. In several studies, the diagnostic performance of clinical competency has usually been used to evaluate practitioner performance [7, 18]. To the best of our knowledge, methods to evaluate the performance level of bedside cardiac ultrasound in a real clinical emergency setting have not been developed well.
The purpose of this study is to evaluate the improvement of performance skill and interpretation level of bedside cardiac ultrasound among emergency medicine residents following a PoCUS education-rotation program, by reviewing acquired cardiac ultrasound images. The secondary outcome was a change in attitude and confidence toward PoCUS in emergency practice. We hypothesized that structured program for education and rotation would improves performance and results in favorable attitudes.
Methods
Study design and setting
A retrospective observational study was conducted based on the ultrasound database of the Seoul National University Hospital Department of Emergency Medicine. Seoul National University Hospital is a tertiary academic-teaching hospital in the metropolitan city of Seoul, and approximately 70,000 patients visit the emergency department annually. Emergency bedside ultrasound rotation schedule with education program has been in place since 2018, as part of a residency training program. The goal of this program is to improve the quality of clinical practice. Before implementation, ultrasound was an optional procedure in the emergency department that was conducted when physicians were willing to participate, which was not frequent enough to make ultrasound a standard part of clinical practice. The sample for this study comprised 2nd- to 4th-year residents, who were required to participate in the PoCUS program since it became a formal part of the curriculum and practice protocol in the department.
Study population
Emergency medicine residents who followed emergency bedside ultrasound rotation duties during the study period, were enrolled. Participants who did not save the whole basic views of ultrasound imaging in the picture archiving communication system or document their interpretations in medical records were excluded from the analysis.
PoCUS program
The PoCUS program, consisting of an education section and a practice section, was officially designed and implemented in April 2018. All residents had to take a comprehensive ultrasound workshop for basic echocardiography, including basic view, by a cardiologist early in their 2nd years. They then automatically participated in monthly PoCUS education programs if they had been assigned to adult or pediatric emergency departments in Seoul National University Hospital. Residents on other schedules, e.g., in other hospitals or departments, did not participate. The monthly education program included a 2-hour training session and a conference. In the training session, basic and advanced knowledge of PoCUS administration, including for the lung or abdomen, was reviewed, and the participants received hands-on practice administering bedside cardiac ultrasound using an ultrasound simulation machine (US Mentor, Simbionix). Interesting cases were selected by the residents as cases supporting the clinical usability of PoCUS and presented in a conference at the end of the month.
The PoCUS practice session was a requirement of the residency program. All residents had approximately 8–16 hours of ultrasound duties in the emergency department each month. They performed examinations whenever patients presented with chest pain, difficulty breathing, syncope or palpitation. Primary physicians could also request bedside cardiac ultrasounds to residents who are in ultrasound rotation. Residents do not care for patients as primary physicians while on an ultrasound rotation duty. A portable ultrasound machine was used for all procedures (M-turbo, SonoSite) (Vivid Q, GE). Video clips and images were saved and transferred to a picture archiving communication system. Official interpretations of the results were typed in a constructed format. PoCUS rounding was conducted every other day with PoCUS faculty and residents for quality improvement and feedback. All contents were reviewed and supported by the PoCUS faculty, which consisted of 4 emergency medicine specialists and an emergent medical technician. PoCUS faculty need to undergo comprehensive ultrasound education workshops regularly and perform bedside ultrasound in official emergency practice.
Data source and acquisition
The database system was operated and supervised by the PoCUS faculty using a structured registry that contains demographic and clinical information of patients. It includes date of examination, operator’s information and official interpretation.
We retrieved data from the PoCUS database to evaluate performance improvements. Two cases were selected for each resident, one in the first week of the whole period of PoCUS rotation program (early phase) and the other in the last (late phase). Independent research coordinators selected all cases randomly in each period.
Evaluation for performance
A structured evaluation sheet was developed based on the guidelines of emergency ultrasound standard reporting [19]. The evaluation sheet was designed to evaluate handling the machine, capturing feasible views for interpretation, adjusting depth and gain properly, achieving a complete set of views and interpreting clearly. A detailed introduction to the evaluation sheet is described in Table 1.
Two independent emergency specialists were consulted to evaluate the cases. Two independent emergency specialists were consulted to evaluate the cases. They are PoCUS faculty members who have practical experience with more than 200 cardiac PoCUS and work as instructors for emergency physician and medical school student. Evaluation was based on video clips in a picture archiving communication system (PACS) and developed sheet. The specialists were blind to practitioner identity and phase. Specialists gave specific scores to 6 components of 7 basic views (parasternal long axis, parasternal short axis for 4 levels, 4-chamber and 5 chamber view) and 2 measure indexes (inferior vena cava and ejection fraction) based on the guidelines of the evaluation sheet, and the total score was calculated. One point was given for probe and orientation selection, 2 points were given for adjusting brightness properly, and 3 points were given for imaging well-identified structures of heart, providing a clinically sufficient interpretation and capturing high quality images. The maximum total score was 100 points in each case.
Survey of confidence and acceptance
A structured questionnaire that consisted of 7 questions regarding confidence in and attitudes about PoCUS use in emergency practice was developed. Each question is answered on a 5-point Likert scale of agreement. Detailed information on the questionnaire is provided in Table 2. All participants completed the questionnaire at the beginning and end of the PoCUS program. This questionnaire was used to judge quality improvements and applicability to residency programs.
Statistical analyses
A descriptive analysis was conducted by calculating the mean and standard deviation for the demographics and characteristics of the participants. Total early- and late-phase scores were compared by paired t-test for each participant, and the intraclass correlation coefficient was calculated. Sensitivity analysis for each basic core view and questionnaire analysis were conducted using the same methods.
An additional multivariable linear regression analysis was conducted to evaluate the association between the demographics and characteristics of the participants and score improvements. A stepwise method was used to develop the optimized model. Score improvement was defined as the mean difference between each phase. We selected predictors such as year of graduation (YOG), age, time elapsed between assessments, early-phase score, number of PoCUS applications, duration of program participation and answers to the questionnaires about confidence in and acceptance of bedside cardiac ultrasound. All statistical analyses were conducted in R Studio 4.3.4.
Results
Characteristics of study subjects
From April 2018 to February 2019, a total of 28 emergency medicine residents participated in the PoCUS program. 46 ultrasound data from 23 participants were analyzed in the performance evaluation, since 5 participants were excluded who did not save the whole basic ultrasound views or interpretations. In the survey analysis, 6 participants refused to fill out questionnaire. 22 residents conducted 35 pairs (35 early phase and 35 late phase) of questionnaires, since 13 residents participated POCUS each year during the study period. The demographics and characteristics of the study participants are described in Table 3. The mean period between the early and late phases, which is almost same as the gap between first and last rotation, was 20 weeks. The mean participation in the PoCUS program was 3 months, which was not the same for each individual due to rotation schedule including dispatching to other hospitals. They administered PoCUS nearly 40 times on average, and there was one exceptional practitioner who conducted 215 cardiac ultrasounds in 5 months.
Performance and survey evaluation
Table 4 and Fig 1 shows the results of the performance evaluations conducted by two emergency specialists using the structured sheet and questionnaire evaluations. A total of 46 cases from 23 participants were analyzed, and the mean score changed significantly from 39.5 to 56.1 according to expert A and 45.3 to 62.9 according to expert B. The interclass correlation coefficients were 0.85 and 0.84 for the early and late phases, respectively. The total average questionnaire score improved from 18.9 to 20.7. The most improved question was about confidence in acquiring the core basic view using adult cardiac ultrasound (Q4 in Table 2), which changed from 3.14 to 3.83.
PoCUS, point-of-care ultrasound.
Sensitivity analysis for detailed components
We conducted additional sensitivity analysis for each component of the bedside cardiac ultrasound core basic view. There was significant improvement in performance based on each expert’s evaluation of the parasternal long axis; parasternal short axis, aortic valve; parasternal short axis, mitral valve; parasternal short axis, papillary muscle; and 4-chamber view. The interclass correlation coefficient was mostly between 0.65 and 0.8, except for the inferior vena cava view (0.43 and 0.23) (Table 5).
Regression model for improvement of performance
In multivariable linear regression analysis, we found that age, early-phase score and prior high confidence had a negative association with performance improvement; the beta coefficients were -2.6 (-4.8 to -0.4) for age, -0.4 (-0.7 to -0.1) for early-phase score, and -11.0 (-18.6 to -3.4) for prior answers to questions about confidence. Otherwise, the number of cardiac ultrasound examinations had a positive association with improved performance, with a beta coefficient of 0.4 (0.2 to 0.5) (Table 6).
Discussion
We found statistically significant improvement of cardiac ultrasound performance by emergency medicine residents following structured education and rotation programs. Furthermore, attitudes about and confidence in emergency PoCUS application improved. Additional analysis showed that younger age, lower previous performance level and confidence were associated with marked improvement.
Our study used a paired t-test, which has been successfully used to evaluate various programs for resident education, to evaluate the effect of the program in each participant [20, 21]. Recent research on focused cardiac ultrasound in surgical intensive care units used similar statistical methods and standardized scoresheets [22]. Our scoresheet included similar categories to those of previous studies but had more detailed components and evaluation instructions, which were designed by the PoCUS faculty. The correlation between the two emergency specialists revealed good agreement on total score and most of the sensitivity analysis [23].
Multivariable linear regression analysis was conducted to evaluate the association between the characteristics of participants and potential improvement. Variables could be retrieved only from the clinical database since it was a retrospective observation study, and the results were reasonable and explainable. We may consider focusing on residents with low performance levels and confidence, which would be more effective.
Previous studies have been predominantly based on education programs, and evaluations have not been blinded to both raters and participants. Our evaluation was conducted based on a clinical practice database in an emergency department. We assume that all ultrasound procedures were performed independently for clinical purposes without additional consideration, which would guarantee more objective outcomes than evaluating on via the education program.
There has been little evidence of an effective curriculum and evaluation method for ultrasound training. In focused abdominal sonography for trauma (FAST), previous studies usually focused on requirements for certification or competency [24–26]. The results of this research demonstrated an effective way to increase the performance and favorable attitudes of cardiac PoCUS in residents. Anstey et al. demonstrated an effective PoCUS curriculum for internal medicine residents that produced long-term gains in knowledge and high confidence achievements [27]. Emergency medicine residents struggle in overcrowded emergency departments to manage various patients with severe needs, which results in a stressed and dissatisfied training period [28, 29]. It often leads residents to overload themselves to complete the mandatory cardiac PoCUS training sessions off duty. On the other hand, procedural training without real clinical practice can blur the effect of education programs. Furthermore, a recent study showed that doctors recognize their need for sufficient practice with certain procedures to build competency [30]. We can infer that less education and more practice would help improve both performance and attitude on cardiac PoCUS.
Our research focused on performance improvement of cardiac PoCUS with real practice, but the clinical benefit has not been well studied. We are planning an additional study to evaluate the association between the implementation of cardiac PoCUS protocol for specific population and improvement of clinical outcomes. Additionally, after operating the PoCUS program for a few more years, other components of the PoCUS program, including pediatric abdomen sonography and lung ultrasound, are thought to be analyzed with sufficient case volume.
Our study has several limitations. First, we screened all residents for the study population and included 23 participants based on compatibility for analysis rather than the number of examinations they performed. A previous study demonstrated a cutoff for improvement in ultrasound administration [31]. In our study, residents conducted cardiac ultrasound when needed to evaluate patients, but they did not log all examinations in the database, especially in urgent situations. Accordingly, the number of cases with usable data varied from 10 to over 200 for each participant but confining the study population based on the number of examinations completed may be inappropriate for study purpose of evaluating all participants in the program. Also, even assuming that the data would have been saved with well-performed ultrasound, the missing ultrasound data could lead to bias, which is one of the major limitations of the study. Second, since the yearly schedule is different for each participant, including the timing of the adult emergency department rotation, the number of program openings per educational period and the number of PoCUS rotations available would differ. The participation period varied from 1 month to 5 months. We assume that if the residents recognize the clinical importance of PoCUS, they will practice it in the course of their regular duties even if they do not need to. Next, scoring system used in this study has not been validated for various environment. Though it was developed based on the standard guidelines, further research for validation is needed. Fourth, evaluation of performance was conducted by emergency medicine specialists. In United States, certification for PoCUS competency in emergency practice has been developed for various institutes. We consider certification of assessor as appropriate with more than 200 cardiac PoCUS experienced, and qualification for PoCUS instructor in Seoul National University Hospital Department of Emergency Medicine. Detailed components to prove the practitioner’s credentials may not match perfectly due to lack of official PoCUS certificate programs in South Korea, which can be another limitation. Fifth, this study was limited to one tertiary academic hospital emergency department, and thus cannot be generalized directly to other environments. We are trying to construct a multicenter PoCUS program, which may overcome this limitation. Next, concrete description or analysis for diastolic function and valve function was not conducted. Though some description for them was in interpretation, we considered competency of participants is insufficient, which including diastolic function and valve function seems inappropriate. Last, this is a retrospective observational study, which has limitations in proving the effect of experience.
Conclusion
We found performance improvements in bedside cardiac ultrasound administration and favorable attitudes toward PoCUS in emergency medicine residents who were enrolled in a structured education and rotation program.
References
- 1. Mercaldi CJ, Lanes SF. Ultrasound guidance decreases complications and improves the cost of care among patients undergoing thoracentesis and paracentesis. Chest. 2013;143(2):532–8. pmid:23381318
- 2. Pivetta E, Goffi A, Lupia E, Tizzani M, Porrino G, Ferreri E, et al. Lung Ultrasound-Implemented Diagnosis of Acute Decompensated Heart Failure in the ED: A SIMEU Multicenter Study. Chest. 2015;148(1):202–10. pmid:25654562
- 3. Lau CS, Chamberlain RS. Ultrasound-guided central venous catheter placement increases success rates in pediatric patients: a meta-analysis. Pediatr Res. 2016;80(2):178–84. pmid:27057741
- 4. Wilson SP, Assaf S, Lahham S, Subeh M, Chiem A, Anderson C, et al. Simplified point-of-care ultrasound protocol to confirm central venous catheter placement: A prospective study. World J Emerg Med. 2017;8(1):25–8. pmid:28123616
- 5. Sanders JL, Noble VE, Raja AS, Sullivan AF, Camargo CA Jr. Access to and Use of Point-of-Care Ultrasound in the Emergency Department. West J Emerg Med. 2015;16(5):747–52. pmid:26587101
- 6. Beeson MS, Carter WA, Christopher TA, Heidt JW, Jones JH, Meyer LE, et al. The development of the emergency medicine milestones. Acad Emerg Med. 2013;20(7):724–9. pmid:23782404
- 7. Choi YJ, Jung JY, Kwon H. Effectiveness of education in point-of-care ultrasound-assisted physical examinations in an emergency department: A before-and-after study. Medicine (Baltimore). 2017;96(25):e7269.
- 8. Reaume M, Siuba M, Wagner M, Woodwyk A, Melgar TA. Prevalence and Scope of Point-of-Care Ultrasound Education in Internal Medicine, Pediatric, and Medicine-Pediatric Residency Programs in the United States. J Ultrasound Med. 2019;38(6):1433–9. pmid:30255947
- 9. Heller MB, Mandavia D, Tayal VS, Cardenas EE, Lambert MJ, Mateer J, et al. Residency training in emergency ultrasound: fulfilling the mandate. Acad Emerg Med. 2002;9(8):835–9. pmid:12153891
- 10. Labovitz AJ, Noble VE, Bierig M, Goldstein SA, Jones R, Kort S, et al. Focused cardiac ultrasound in the emergent setting: a consensus statement of the American Society of Echocardiography and American College of Emergency Physicians. J Am Soc Echocardiogr. 2010;23(12):1225–30. pmid:21111923
- 11. Nazerian P, Vanni S, Zanobetti M, Polidori G, Pepe G, Federico R, et al. Diagnostic accuracy of emergency Doppler echocardiography for identification of acute left ventricular heart failure in patients with acute dyspnea: comparison with Boston criteria and N-terminal prohormone brain natriuretic peptide. Acad Emerg Med. 2010;17(1):18–26. pmid:20078435
- 12. Wang HK, Tsai MS, Chang JH, Wang TD, Chen WJ, Huang CH. Cardiac ultrasound helps for differentiating the causes of acute dyspnea with available B-type natriuretic peptide tests. Am J Emerg Med. 2010;28(9):987–93. pmid:20825928
- 13. Anderson KL, Jenq KY, Fields JM, Panebianco NL, Dean AJ. Diagnosing heart failure among acutely dyspneic patients with cardiac, inferior vena cava, and lung ultrasonography. Am J Emerg Med. 2013;31(8):1208–14. pmid:23769272
- 14. Russell FM, Ehrman RR, Cosby K, Ansari A, Tseeng S, Christain E, et al. Diagnosing acute heart failure in patients with undifferentiated dyspnea: a lung and cardiac ultrasound (LuCUS) protocol. Acad Emerg Med. 2015;22(2):182–91. pmid:25641227
- 15. Farsi D, Hajsadeghi S, Hajighanbari MJ, Mofidi M, Hafezimoghadam P, Rezai M, et al. Focused cardiac ultrasound (FOCUS) by emergency medicine residents in patients with suspected cardiovascular diseases. J Ultrasound. 2017;20(2):133–8. pmid:28593003
- 16. Bornemann P. Assessment of a Novel Point-of-Care Ultrasound Curriculum’s Effect on Competency Measures in Family Medicine Graduate Medical Education. J Ultrasound Med. 2017;36(6):1205–11. pmid:28206672
- 17. Smith CJ, Matthias T, Beam E, Wampler K, Pounds L, Nickol D, et al. Building a bigger tent in point-of-care ultrasound education: a mixed-methods evaluation of interprofessional, near-peer teaching of internal medicine residents by sonography students. BMC Med Educ. 2018;18(1):321. pmid:30591050
- 18. Mullangi S, Sozio SM, Segal P, Menez S, Martire C, Shafi T. Point-of-care ultrasound education to improve care of dialysis patients. Semin Dial. 2018;31(2):154–62. pmid:29314256
- 19. American College of Emergency P. American College of Emergency Physicians. ACEP emergency ultrasound guidelines-2001. Ann Emerg Med. 2001;38(4):470–81. pmid:11574810
- 20. Moeller A, Webber J, Epstein I. Resident duty hour modification affects perceptions in medical education, general wellness, and ability to provide patient care. BMC Med Educ. 2016;16:175. pmid:27411835
- 21. Kornegay JG, Leone KA, Wallner C, Hansen M, Yarris LM. Development and implementation of an asynchronous emergency medicine residency curriculum using a web-based platform. Intern Emerg Med. 2016;11(8):1115–20. pmid:26951187
- 22. Townsend NT, Kendall J, Barnett C, Robinson T. An Effective Curriculum for Focused Assessment Diagnostic Echocardiography: Establishing the Learning Curve in Surgical Residents. J Surg Educ. 2016;73(2):190–6. pmid:26774938
- 23. Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 2016;15(2):155–63. pmid:27330520
- 24. Han DC, Rozycki GS, Schmidt JA, Feliciano DV. Ultrasound training during ATLS: an early start for surgical interns. J Trauma. 1996;41(2):208–13. pmid:8760525
- 25. Ma OJ, Mateer JR. Trauma ultrasound examination versus chest radiography in the detection of hemothorax. Ann Emerg Med. 1997;29(3):312–5; discussion 5–6. pmid:9055768
- 26. Mayo PH, Beaulieu Y, Doelken P, Feller-Kopman D, Harrod C, Kaplan A, et al. American College of Chest Physicians/La Societe de Reanimation de Langue Francaise statement on competence in critical care ultrasonography. Chest. 2009;135(4):1050–60. pmid:19188546
- 27. Anstey JE, Jensen TP, Afshar N. Point-of-Care Ultrasound Needs Assessment, Curriculum Design, and Curriculum Assessment in a Large Academic Internal Medicine Residency Program. South Med J. 2018;111(7):444–8. pmid:29978232
- 28. Hoonpongsimanont W, Murphy M, Kim CH, Nasir D, Compton S. Emergency medicine resident well-being: stress and satisfaction. Occup Med (Lond). 2014;64(1):45–8. pmid:24306263
- 29. Wong ML, Anderson J, Knorr T, Joseph JW, Sanchez LD. Grit, anxiety, and stress in emergency physicians. Am J Emerg Med. 2018;36(6):1036–9. pmid:29502975
- 30. Hunnalvatn KH, Ivan D, Wisborg T. Emergency medicine in the general practice internship in Finnmark county. Tidsskr Nor Laegeforen. 2017;137(23–24). pmid:29231639
- 31. Gaspari RJ, Dickman E, Blehar D. Learning curve of bedside ultrasound of the gallbladder. J Emerg Med. 2009;37(1):51–6. pmid:18439787