https://orcid.org/0000-0003-2024-0020
Figures
Abstract
Entrustable Professional Activities (EPA) are specialty specific tasks or responsibilities, combining the clinical workplace and the long-demanded competency-based medical education. The first step to transform time-based into EPA-based training is to reach consensus on core EPAs that describe sufficiently the workplace. We aimed to present a nationally validated EPA-based curriculum for postgraduate training in anaesthesiology. Using a predefined and validated list of EPAs, we applied a Delphi consensus approach, involving all German chair directors of anaesthesiology. We then conducted a subsequent qualitative analysis. Thirty-four chair directors participated in the Delphi survey (77% response) and twenty-five completed all the questions (56% overall response). Reflected by the intra-class-correlation, the consensus on the importance (ICC: 0.781, 95% CI [0.671, 0.868]) and the year of entrustment (ICC: 0.973, 95% CI [0.959, 0.984]) of each EPA reached high levels of agreement among the chair directors. The comparison of data assessed in the preceding validation and present study showed excellent and good levels of agreement (ICC entrustment: 0.955, 95% CI [0.902, 0.978]; ICC importance: 0.671, 95% CI [-0.204, 0.888]). The adaptation process, based on the qualitative analysis, resulted in a final set of 34 EPAs. We present an elaborate, fully described and nationally validated EPA-based curriculum, reflecting a broad consensus among different stakeholders of anaesthesiology. We hereby provide a further step towards competency-based postgraduate anaesthesiology training.
Citation: Ganzhorn A, Schulte-Uentrop L, Küllmei J, Zöllner C, Moll-Khosrawi P (2023) National consensus on entrustable professional activities for competency-based training in anaesthesiology. PLoS ONE 18(7): e0288197. https://doi.org/10.1371/journal.pone.0288197
Editor: Valérie Pittet, Center for Primary Care and Public Health, SWITZERLAND
Received: November 9, 2022; Accepted: June 21, 2023; Published: July 11, 2023
Copyright: © 2023 Ganzhorn et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data is provided in the paper and its supporting information.
Funding: The author(s) received no specific funding for this work
Competing interests: The authors have declared that no competing interests exist.
Introduction
Entrustable Professional Activities (EPAs) have gained great popularity over the past years [1] and have been integrated into many undergraduate and postgraduate training programs worldwide [2]. An EPA is a specialty specific task or responsibility, which should be entrusted to the trainee once a sufficient level of performance has been reached [3]. The concept of EPAs is an effective approach to combine the clinical workplace with competency-based medical education (CBME) [4].
Like in many other fields, in anaesthesiology too, the importance of CBME in postgraduate training has been emphasised: The European Society of Anaesthesiology has endorsed the transformation of anaesthesiology training into an outcome-based approach, moving from time based- towards competency-based training [5]. So far, only few studies have defined EPAs for anaesthesiology training and even fewer have constructed an elaborate training curriculum based on fully described EPAs [6–11]. Although the findings of these studies do contribute to the process of implementing EPA-based training, nevertheless, still some gaps in research emerge: Only few studies aimed to reach institutional [10] or even national consensus [9] on defined EPAs. Some only focused on the first year [6] or undergraduate training [12]. Furthermore, most of the studies included only the titles and other components of a fully described EPA were omitted. Alas, all components of an EPA are necessary to implement an EPA-based training into the clinical workplace [1]. A fully described EPA encompasses seven domains which, amongst others, include the title, specific descriptions and the stage of training at which the task should be entrusted [1].
According to the agenda of Jonker et al., the first step towards CBME in anaesthesiology training is reaching consensus on relevant EPAs [3]. Nevertheless, investigations aiming to reach national or even international consensus are scarce. In a pioneering study, Wisman-Zwarter et al. provided a list of forty-five EPAs which was validated through a national consensus procedure involving Dutch anaesthesiology program directors. Overall, the current state of research deviates from the desired state. There is a gap in research regarding the definition and validation of EPAs for anaesthesiology training in among others, German-speaking countries. No published investigation has reached consensus on EPAs for anaesthesiology training in Germany, much less in Europe [10]. Yet, reaching a broad consensus on EPAs will facilitate their implementation and acceptance [4]. Consensus should be reached on local and institutional levels (those who work directly with the EPAs), but also on more extended levels which include experts and multiple institutions (nationally or even internationally) [3, 4].
We therefore aimed to present an institutionally and nationally validated EPA-based curriculum for postgraduate training in anaesthesiology, which is in accord with the so far published European EPAs.
In a preceding study we developed and institutionally validated an anaesthesiology core curriculum based on thirty-nine EPAs, which includes the year of entrustment and the importance (ranking) of each EPA to the curriculum. Our curriculum showed a 73% accordance with the EPAs provided from the study of Wisman-Zwarter et al. in the Netherlands [8]. To extend the validation and consensus, chair directors nation-wide were targeted, as they are key figures in implementing EPA-based curricula into the workplace.
Using a national consensus procedure, which involved all German chair directors, the EPAs were validated with respect to their importance and the year of entrustment. The curriculum was adjusted accordingly, resulting in a validated core curriculum for competency-based training in anaesthesiology.
Methods
A detailed project description was sent to the local Ethic Committee of Hamburg which belongs to the General Medical Council of Hamburg (Ethik‐Kommission der Ärztekammer Hamburg, Hamburg, Germany). The project was exempted from the need of approval, as the paragraph 9 of the "Law of Healing Professions, Hamburg” (§ 9 des Hamburgischen Kammergesetzes für Heilberufe) and § 15, section 1 of the medical professional conduct (Berufsordnung für Hamburger Ärzte und Ärztinnen) and the article 6 of the “Declaration of Helsinki” did not apply. Therefore, no necessity of deliberation of the project was seen. This study was a study with- but not on humans.
Participation in the study was voluntary and anonymous. Participants declared their consent to participate by participating (this information was provided alongside the email which explained the study goal).
Study design
This study is reported in accordance with the STROBE and COREQ guidelines [13, 14].
The starting point and basis of our study was a core curriculum for postgraduate anaesthesiology training, composed of thirty-nine EPAs which was developed and validated in a preceding study [8]. We used the combination of a modified Delphi-method and qualitative analysis (expert group, template- and mapping method, direct content analysis) [15, 16], to reach consensus and to adapt the curriculum (EPAs) accordingly. The Delphi-method is an iterative technique with the goal to collect expert opinion to reach group consensus [17, 18]. Qualitative analysis has gained a broad applicability in the field of medical education and especially for developing EPAs [8, 19].
Study setting
This study was performed at the Department of Anaesthesiology, University Medical Center Hamburg-Eppendorf. The General Medical Council of each federal state sets the prerequisites for postgraduate specialty training in anaesthesiology. Specialist qualification is reached during residency (five years): The residents are assigned to several anaesthesia workplaces, where they function as the primary executing anaesthesiologist. Supervision is provided by specialists. Based on the residents´ level of training and the anaesthesiologic procedure, the amount of supervision varies from a direct form (being directly involved) to a far indirect form (availability by phone) and is not explicitly defined by law. Therefore, the level of supervision and hence teaching depends on the supervisor and on the resident, who can also actively demand for supervision. Supervision and workplace-based teaching differ in every hospital and are based on local circumstances. The numbers of specialty specific procedures, reaching from a “simple” general anaesthesia to more advanced forms (a.e. one-lung ventilation; anaesthesia for abdominal surgery; anaesthesia for neurosurgery) are recorded in a logbook which is provided by the General Medical Council. Each anaesthesiologic procedure has a minimum number which is determined by the General Medical Council and must be conducted by the resident to reach specialty qualification. When hospitals do not have departments of special surgical disciplines, anaesthesia residents rotate to other hospitals to learn and gather the specific procedures (a.e neurosurgery). After five years of residency and after reaching the minimal numbers for each procedure, the program directors can issue the specialist qualification and the resident can take the specialist examination.
Since 2018 the postgraduate education guidelines have been renewed, including competencies but not EPAs. Furthermore, minimal numbers of conducted anaesthesiologic procedures are still integral parts of specialist qualification. The program is put in practice by training site clusters (hospitals) across Germany, where according to federal regulations, each hospital has a program director, who is responsible for the local training. Amongst others, the chairs of the university hospitals (chair directors) (n = 47) have the full training authorisation.
Sample
The chair directors were eligible to participate in the study due to their broad expertise in anaesthesiology training and because of their representative and determining character. The chair directors have completed their specialty training in anaesthesiology, intensive care, pain-, and emergency medicine. All but three program directors (n = 44) of anaesthesiology across Germany were invited to participate in the Delphi survey. The chair directors can be found on the homepage of the German Society of Anaesthesiology and Intensive Care Medicine (DGAI). The chair of the Department of Anaesthesiology of Hamburg was not eligible to participate since he was involved as a researcher in the current study. Two further chair directors were not eligible, because they mainly lead the intensive care units. Thirty-four chair directors participated in the Delphi survey (77% response) and twenty-five completed all the questions (56% overall response). Data was analysed only for complete surveys.
The expert group consisted of four researchers and clinicians of different maturity, with profound knowledge in postgraduate training and medical education. Three of them are anaesthesiology specialists (n = 2 female and n = 1 male), one of them is a second-year resident (female). The mean age was thirty-five years.
Participants of the preceding study were residents, attendings and consultants of the department of anaesthesiology at the University medical Center Hamburg-Eppendorf.
Table 1. summarises the participants of the current and the preceding study.
Procedure
An overview of the study procedure is shown in Fig 1.
Before the questionnaire for the consensus procedure (Delphi survey) was drafted, the expert group developed an elaborate curriculum based on the thirty-nine EPAs, meeting the requirements of the AMEE (Association for Medical Education in Europe) guide no 99 [4].
Subsequently, the curriculum was piloted within the department for the duration of six months by n = 58 residents. The aim was to analyse the practicability of the curriculum in the daily clinical work setting, in order to detect necessary adaptations before the consensus procedure took place.
After the expert group reconfirmed the EPAs, the questionnaire for the Delphi survey was developed, discussed and piloted. In the following step, the chair directors were contacted by e-mail with the purpose of the study and were provided with background information on the concept of EPAs. The e-mail contained a Weblink via which the anonymous participation in the Delphi survey was enabled.
Outcome measures
As this study was an initial step to meet the prerequisites for a unification of postgraduate training and a transformation from time-based into competency (EPA) based training, the main goal of the study was to reach national consensus on Entrustable Professional Activities for anaesthesiology. Therefore, the first outcome measure was the validation of each EPA, to determine the relevance of each EPA for anaesthesiology training with the primary focus on the content validation. For this purpose, participants were asked to determine, if each EPA should be included in the final list of EPAs. To provide a basis for further adaptations, the importance of each EPA was also numerically assessed to calculate the content validity index (CVI). A CVI reflects the proportion of relevance and is an established parameter in validation processes [20]. The second outcome measure was the year of entrustment for each EPA. Since a broad consensus among different stakeholders of anaesthesiology is helpful to facilitate implementation, the third outcome measure was the consensus between the first two outcome measures (relevance and year of entrustment of each EPA) assessed in the preceding (institutional validation) and current study.
To assess the outcome measures, the Delphi survey was composed of the following three sections (for each EPA):
- Should this EPA be included into an anaesthesiology curriculum? (Answers: “yes”, “no”)
- Please rank each EPA concerning its importance for anaesthesiology training. (Likert- scale- ranks from 5 = “very important”, to 1 = “not important“).
- In which year of residency should this EPA be entrusted? (conduction without direct supervision) (Answers: “1st year, 2nd year, 3rd year, 4th year, 5th year, attending”)
To provide a basis for adaptations and amendments, the participants were also provided with an open-text box for additional comments and feedback for each EPA.
Data from the Delphi survey was discussed in the expert group regarding necessary amendments of the EPAs. The qualitative data analysis of the open-text comments was conducted, using the mapping method [15], content analysis and an adapted template, which was also used in the preceding study [8, 16]. A triangulation strategy was applied to assess data validity of the comments [21]. First the leader of the research group analysed the comments (preliminary analysis). A second member of the research group repeated independently the same process. Data was then analysed by an independent group of staff members (n = 3). Then the same process was repeated by the expert group and all the amendments were discussed until saturation was reached [22]. For further validation, the results of the expert group were reviewed and confirmed by an independent member of the department.
For the quantitative analysis, agreement of the study participants for each of the three questions (1. if an EPA should be part of the curriculum; 2. the importance (ranking); 3. the year of entrustment) was computed. In a next step, the year of entrustment, as well as the relevance (importance) of each EPA (assessed in the present study) was compared to data from the preceding study. Good levels of agreement (intraclass correlation over 0.6) were specified to evaluate the necessity of further changes of the EPA curriculum.
Following thresholds were set and specified (quantitative analysis) for each EPA to pass or to be included into further discussions and considerations:
- 80% agreement among the current study participants on the question, if the EPA should be part of the curriculum was specified to retain an EPA
- Good agreement (reflected by the computed intraclass correlation (ICC) between 0.6 and 0.74) among the study participants regarding the importance of each EPA
- Good agreement among the study participants regarding the year of entrustment
- Good agreement between the study participants of both studies regarding the importance of each EPA
- Good agreement between the study participants of both studies regarding the year of entrustment. If the level of agreement was given, the final year of entrustment was averaged to set the mean value as the final year of entrustment
Statistical analysis
Statistical analysis was performed with SPSS (version 28.0.1.0, IBM Corp., Armonk, New York, USA). Descriptive statistics (mean values, standard deviations and percentages) were calculated for all data. For the first consensus calculation- which determined if an EPA retained, percentages of agreement (“yes” or “no”) were computed and an 80% threshold was specified. The ranking of each EPA was calculated with the content validity index (CVI). The CVI reflects the proportion of relevance [23]. A CVI of 0.75 or higher is considered as “excellent”[24]. According to the CVI, the EPAs were split into a “high ranking”(CVI>0.75) and a “low ranking”group (CVI<0.75) [20].
The consensus (agreement) among the group of chair directors (year of entrustment and ranking) were calculated with the intraclass correlations (ICC): ICC estimates and their 95% confident intervals (95% CI) were calculated based on a mean-rating, absolute-agreement and 2-way mixed-effects model [25].
The levels of agreement with ICC estimates and their 95% CI, between data of both studies (year of entrustment and CVI of each EPA) were calculated based on a test-retest model, mean-rating, absolute-agreement and 2-way mixed-effects model [26].
Values of ICC below 0.40 reflect a poor correlation, between 0.40 and 0.59 a fair correlation, between 0.60 and 0.74 a good correlation and between 0.75 and 1.00 are interpreted as excellent correlation [27].
Results, ranking and year of entrustment of each EPA
With a mean agreement of 94%, the participants agreed on thirty-seven EPAs to be relevant for anaesthesiology training. Two (5%) EPAs did not reach the 80% consensus rate (EPA: “Indication and performance of ultrasound use and diagnostic (a.e FAST) and therapeutic consequences”; EPA: “Providing perioperative care for patients undergoing cardiothoracic surgery”).
Thirty-five EPAs achieved a CVI over 0.75 (highest CVI: 1.0; lowest CVI: 0.76) and were therefore allocated to the high-ranking group. Four EPAs were allocated to the low-ranking group (highest CVI: 0.72; lowest CVI: 0.6). The two EPAs, which did not pass the 80% consensus rate (to be included in the curriculum) were part of the low-ranking EPAs.
Table 2 displays the content validity index (CVI) of each EPA, dividing them into a high (35/39)- and low/inferior (4/39) ranking group.
The year of entrustment (year of training, in which the EPA can be conducted without direct supervision) is also displayed. For better overview, the CVI and year of entrustment assessed in the preceding study are also outlined.
Agreement on the CVI and the year of entrustment
The consensus among the chair directors (reflected by the ICC) regarding the importance (ICC: 0.781, 95% CI [0.671, 0.868]) and the year of entrustment (ICC: 0.973, 95% CI [0.959, 0.984]) reached high levels of agreement.
The comparison of data assessed in both studies regarding the year of entrustment and the relevance (CVI) showed excellent and good levels of agreement (ICC entrustment: 0.955, 95% CI [0.902, 0.978]; ICC CVI: 0.671, 95% CI [-0.204, 0.888]).
As displayed in Table 2. the chair directors ranked the EPAs of higher relevance (CVI) than in the preceding study with one exception: The EPA “Providing anaesthetic care for large laparoscopic surgery” achieved an CVI of 0.72 (low-ranking) whereas in the preceding study a CVI of 0.85 (high-ranking).
Qualitative analysis and alignment process
A total of n = 137 comments were provided by the study participants. All comments were analysed and n = 42 comments were considered for the adaptation and adjustment process. A summarised description of the alignment process is provided in the appendix (S1 Appendix).
The alignment process led to four major changes of the curriculum: Two EPAs (“Indication and performance of ultrasound use and diagnostic (a.e FAST) and therapeutic consequences”; “Providing perioperative care for patients undergoing cardiothoracic surgery”) were excluded from the core curriculum. First because they did not pass the 80% mark and secondly because the study participants declared these EPAs to be too specific for general training in anaesthesiology. The two EPAs “Communication with relatives of critically ill patients and consultation about treatment plans”; “Performing a premedication round (preoperative evaluation) including patient education” were merged to “Performing a premedication round (preoperative evaluation) including patient education and communication with relatives”, as the comments showed that there are many intersections between the competencies of those EPAs. The EPA “Providing emergency care for critically injured and ill patients during in-house transfers and diagnostic procedures” received comments that categorized its competencies into the EPA “Performing in-house transfers of critically ill patients”. Therefore, the two EPAs were merged and the competencies nested into one extended EPA (“Performing in-house transfers of critically ill and injured patients”). The EPA “Providing perioperative care for critically injured patients with increased intracranial pressure” is composed of competencies which are already covered in the EPAs “Providing anaesthetic care for intracranial surgery with (the risk of) increased intracranial pressure” and “Providing anaesthetic care and emergency management for critically injured and ill patients in the shock room”. Therefore, the EPA “Providing perioperative care for critically injured patients with increased intracranial pressure” was nested in the two above mentioned and in the EPA which describes the management of critically injured and ill patients. The EPA was extended with associated competencies in the operating room, resulting in an amended EPA: “Providing anaesthetic care and emergency management for critically injured and ill patients in the shock room and OR”.
The adaptation and adjustment process resulted in a curriculum consisting of n = 34 EPAs, which is depicted in Fig 2.
Note: The year of entrustment is indicated for each EPA; the light-blue bars display the high-ranking EPAs; the dark-blue bars display the low-ranking EPAs. Abbreviations: ASA = American Society of Anaesthesiologists; HELLP = hemolysis, elevated liver enzymes, low platelet count.
Discussion
In our prospective multi-step national consensus study, 25 experts of anaesthesiology across Germany reached a high degree of agreement on an EPA-based curriculum, which includes the year of entrustment as well as the ranking for each EPA (reflecting its importance). The curriculum was generated and pre-validated in a preceding study and initially included 39 EPAs. After the national consensus procedure (Delphi-method) and a comprehensive qualitative analysis, the curriculum was re-validated. The modifications and amendments resulted in a curriculum composed of 34 EPAs, which constitute a comprehensive range of relevant perioperative- and emergency anaesthetic care.
A strength of our study is the broad consensus on the curriculum, which has been reached from a diverse group of different stakeholders including learners as well as educators. The first validation of the preceding study included clinicians in different stages of training, who are mostly concerned by training curricula on a daily work basis (a.e attendings- functioning as educators and supervisors and trainees as the actual learners) [8]. For the validation of the present study, we have gone a step further and included the German anaesthesiology chair directors. Taking into account that the results of the EPA ranking and year of entrustment of both studies reached high levels of agreement, an extended consensus can be assumed. This broad consensus is important, because an EPA-based curriculum should create a shared mental model amongst learners and educators, resulting in greater acceptance. This would in turn facilitate its implementation and also support learner-oriented teaching [1, 10, 28]. Furthermore, the inclusion of a wide range of study participants prevented a limited and partial view of the subject (curriculum) and hereby also differentiates our study from recently published works on EPAs, which either included program directors or members of a department (trainees and supervisors) [6, 8, 10].
Interestingly, with one exception, the chair directors rated the EPAs to be more important, than they were rated in the preceding study, which resulted in higher CVIs (reflecting the importance of each EPA). Whilst remaining conscious that the levels of agreement between the CVIs of both studies were still good (ICC: 0.671), the discrepancy of the CVIs can be explained in two respects: On the one hand, different experience levels might have caused different perspectives on the EPAs, leading to higher estimates of importance (CVI), assessed within the group of greater experienced (chair directors). On the other hand, the lower estimates assessed in the preceding study might be based on the Dunning-Kruger effect [29]. A cognitive bias whereby low experienced individuals prone to overestimate their abilities. Therefore, the lower experienced anaesthesiologists from the preceding study, might have not overlooked the importance and hidden hazards of a clinical tasks, resulting in lower CVIs [29]. This theory can be supported exemplary with the CVI differences for the EPAs: “Performing a premedication round including patient education” or “Postoperative pain management”. While a trainee might not realise the relevance and importance, the expert values the outcome of good competencies while conducting the task.
Some of the discrepancies between the CVIs might be due to differing weighting and importance of the competencies at the departments. This argument can be also transferred to the EPA “Providing anaesthetic care for large laparoscopic surgery”, that received a higher CVI in the preceding study. The CVI was assessed at a department where laparoscopic procedures are of great significance. Nevertheless, all included EPAs of the new curriculum passed the more important question, if they should be integrated into the curriculum and therefore, the value of the CVI lies within the purpose of further adaptations.
One limitation of our study is that the initial curriculum was developed at a university medical center (maximum care hospital), which might have biased some perceived daily tasks. To circumvent this potential bias and to refine the curriculum accordingly, the study participants were asked to provide feedback for each EPA in an open text field. These comments led to adaptations of the curriculum, making it more transferable to non-maximum care hospitals. Furthermore, the amendments might reduce difficulties in using our results for a curriculum reform of other faculties, as we moved from an initial single center design towards a broader and more representative validation [30].
The amendments resulted in a reduction of the curriculum to thirty-four EPAs. Two EPAs were excluded from the core curriculum, as they did not reflect relevant daily tasks of the anaesthesiologic workplace (EPAs: “Indication and performance of ultrasound use and diagnostic (a.e FAST) and therapeutic consequences”; “Providing perioperative care for patients undergoing cardiothoracic surgery”). Furthermore, many comments indicated that too many EPAs of the curriculum target competencies of the care of critically ill and injured patients. Therefore, some of these EPAs were merged and their competencies nested into other EPAs. This quantitative reduction of the EPAs purses the recommendation which describes an amount of twenty to thirty EPAs per curriculum [31]. Considering the extent of our curriculum, one might argue that the number of EPAs constitute a further limitation of our study. Nevertheless, taking into account the large range of so far published EPA curricula, which include 10 to 45 EPAs [32], as well as the fact that anaesthesiology training is a comprehensive program with a lot of competencies, the number of our EPAs can be justified.
A further strength of our study is that we truly met the challenges of the concept of EPAs. We took into account and transposed the specific recommendations, that were provided by ten Cate and his co-workers regarding the development of an EPA-based curriculum [1, 4]. Considering these recommendations is an important aspect, which has been neglected in some works. As recently described, there has been a diversity in the use of the term of EPAs, diluting the concept as intended and hereby hampering the values of EPA in CBME [33].
Although the content of our curriculum itself is not generalizable to medical education, the value of the approach of our study still merits consideration. This approach could assist other medical disciplines, in graduate and undergraduate medical education, with the integration of EPAs into training.
Conclusion
Interest in EPAs is rapidly growing and their potential to facilitate the implementation of CBME into the clinical workplace has been emphasised [34, 35]. Nevertheless, few are used systematically in anaesthesiology training [36], mainly due to little availability and also because this novel approach is not well understood [37]. With our study a further step is made towards the transformation of anaesthesiology training into a more contemporary approach, by presenting a validated set of EPAs and by introducing this concept to many stakeholders of anaesthesiology. Our curriculum does not only provide specific EPA titles, but includes the importance of each EPA, the year of entrustment as well as other important components. Thereof, our study presents a broadly validated EPA-based curriculum for German speaking countries, which has also reached a 73% agreement with validated EPAs published in the Netherlands [10]. Hereby, our results provide a further base towards unifying the goal of CBME in anaesthesiology.
Acknowledgments
We would like to thank the German chair directors for participating in this study. We would like to thank all members of the Department of Anaesthesiology at the University Medical Center Hamburg-Eppendorf for their support.
References
- 1. Ten Cate O, Taylor DR. The recommended description of an entrustable professional activity: AMEE Guide No. 140. Medical Teacher. 2021;43(10):1106–14.
- 2. Ot Cate. An updated primer on entrustable professional activities (EPAs). Revista Brasileira de Educação Médica. 2020;43:712–20.
- 3. Jonker G, Hoff RG, Ten Cate OTJ. A case for competency-based anaesthesiology training with entrustable professional activities: an agenda for development and research. European Journal of Anaesthesiology (EJA). 2015;32(2):71–6. pmid:24945750
- 4. ten Cate O, Chen HC, Hoff RG, Peters H, Bok H, van der Schaaf M. Curriculum development for the workplace using entrustable professional activities (EPAs): AMEE guide no. 99. Medical teacher. 2015;37(11):983–1002. pmid:26172347
- 5. Shorten GD, De Robertis E, Goldik Z, Kietaibl S, Niemi-Murola L, Sabelnikovs O. European Section/Board of Anaesthesiology/European Society of Anaesthesiology consensus statement on competency-based education and training in anaesthesiology. European Journal of Anaesthesiology| EJA. 2020;37(6):421–34.
- 6. Marty AP, Schmelzer S, Thomasin RA, Braun J, Zalunardo MP, Spahn DR, et al. Agreement between trainees and supervisors on first-year entrustable professional activities for anaesthesia training. British Journal of Anaesthesia. 2020;125(1):98–103. pmid:32423610
- 7. Marty AP, Braun J, Schick C, Zalunardo MP, Spahn DR, Breckwoldt J. A mobile application to facilitate implementation of programmatic assessment in anaesthesia training. British Journal of Anaesthesia. 2022. pmid:35410792
- 8. Moll-Khosrawi P, Ganzhorn A, Zöllner C, Schulte-Uentrop L. Development and validation of a postgraduate anaesthesiology core curriculum based on Entrustable Professional Activities: a Delphi study. GMS journal for medical education. 2020;37(5). pmid:32984511
- 9. Woodworth GE, Marty AP, Tanaka PP, Ambardekar AP, Chen F, Duncan MJ, et al. Development and pilot testing of entrustable professional activities for US anesthesiology residency training. Anesthesia & Analgesia. 2021;132(6):1579–91.
- 10. Wisman-Zwarter N, Van Der Schaaf M, Ten Cate O, Jonker G, Van Klei WA, Hoff RG. Transforming the learning outcomes of anaesthesiology training into entrustable professional activities: a Delphi study. European Journal of Anaesthesiology (EJA). 2016;33(8):559–67. pmid:27270884
- 11. Dabbagh A, Fadaeizadeh L, Gharaei B, Ghasemi M, Kamranmanesh M, Khorasanizadeh S, et al. The Role of Entrustable Professional Activities in Competency-based Medical Education for Anesthesiology Residents: A Pilot Phase. Anesthesiology and Pain Medicine. 2022(In Press). pmid:36937177
- 12. Weissenbacher A, Bolz R, Stehr SN, Hempel G. Development and consensus of entrustable professional activities for final-year medical students in anaesthesiology. BMC anesthesiology. 2022;22(1):1–10.
- 13. Cuschieri S. The STROBE guidelines. Saudi journal of anaesthesia. 2019;13(Suppl 1):S31. pmid:30930717
- 14. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. International journal for quality in health care. 2007;19(6):349–57. pmid:17872937
- 15. Pelz C, Schmitt A, Meis M, editors. Knowledge Mapping als Methode zur Auswertung und Ergebnispräsentation von Fokusgruppen in der Markt-und Evaluationsforschung. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research; 2004: Deutschland.
- 16. Brooks J, McCluskey S, Turley E, King N. The utility of template analysis in qualitative psychology research. Qualitative Research in Psychology. 2015;12(2):202–22. pmid:27499705
- 17. Williams PL, Webb C. The Delphi technique: a methodological discussion. Journal of advanced nursing. 1994;19(1):180–6. pmid:8138622
- 18. De Villiers MR, De Villiers PJ, Kent AP. The Delphi technique in health sciences education research. Medical teacher. 2005;27(7):639–43. pmid:16332558
- 19. Basch CE. Focus group interview: an underutilized research technique for improving theory and practice in health education. Health education quarterly. 1987;14(4):411–48. pmid:3319971
- 20. Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in nursing & health. 2007;30(4):459–67. pmid:17654487
- 21. Denzin NK. Triangulation 2.0. Journal of mixed methods research. 2012;6(2):80–8.
- 22. Saunders B, Sim J, Kingstone T, Baker S, Waterfield J, Bartlam B, et al. Saturation in qualitative research: exploring its conceptualization and operationalization. Quality & Quantity. 2018;52(4):1893–907. pmid:29937585
- 23. Martuza VR. Applying norm-referenced and criterion-referenced measurement in education: Allyn & Bacon, Incorporated; 1977.
- 24. Lynn MR. Determination and quantification of content validity. Nursing research. 1986. pmid:3640358
- 25. Koch GG. Intraclass correlation coefficient. Encyclopedia of statistical sciences. 1982.
- 26. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of chiropractic medicine. 2016;15(2):155–63. pmid:27330520
- 27. Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological assessment. 1994;6(4):284.
- 28. ten Cate O, Snell L, Mann K, Vermunt J. Orienting Teaching Toward the Learning Process. Academic Medicine. 2004;79(3):219–28. pmid:14985194
- 29. Dunning D. The Dunning–Kruger effect: On being ignorant of one’s own ignorance. Advances in experimental social psychology. 44: Elsevier; 2011. p. 247–96.
- 30. Ebert TJ, Fox CA. Competency-based Education in AnesthesiologyHistory and Challenges. Anesthesiology: The Journal of the American Society of Anesthesiologists. 2014;120(1):24–31.
- 31. Ten Cate O. Nuts and bolts of entrustable professional activities. Journal of graduate medical education. 2013;5(1):157–8. pmid:24404246
- 32. Young JQ, Hasser C, Hung EK, Kusz M, O’Sullivan PS, Stewart C, et al. Developing end-of-training entrustable professional activities for psychiatry: results and methodological lessons. Academic Medicine. 2018;93(7):1048–54. pmid:29166349
- 33. Hennus MP, Nusmeier A, van Heesch GG, Riedijk MA, Schoenmaker NJ, Soeteman M, et al. Development of entrustable professional activities for paediatric intensive care fellows: A national modified Delphi study. PloS one. 2021;16(3):e0248565. pmid:33735195
- 34. Flentje M, Eismann H, Schwill S, Forstner D, Kranke P. Bedarfsanalyse zur Curriculumsentwicklung für die anästhesiologische Kreißsaalversorgung–eine deutschlandweite Umfrage. Die Anaesthesiologie. 2022;71(9):697–705.
- 35. SPÉCIALISTES UEDM. Training Requirements for the Specialty of Anaesthesiology.
- 36. Burkhart CS, Dell-Kuster S, Touchie C. Who can do this procedure? Using entrustable professional activities to determine curriculum and entrustment in anesthesiology–An international survey. Medical Teacher. 2022:1–7.
- 37. Bonal Ruiz R, Vergel J, Sotomayor Escalona R. De la formación basada en competencias a las actividades profesionales encomendables. Revista Habanera de Ciencias Médicas. 2022;21(1).