https://orcid.org/0000-0001-9838-6084
Figures
Abstract
Purpose
To identify preferred burnout interventions within a resident physician population, utilizing the Nominal Group Technique. The results will be used to design a discrete choice experiment study to inform the development of resident burnout prevention programs.
Methods
Three resident focus groups met (10–14 participants/group) to prioritize a list of 23 factors for burnout prevention programs. The Nominal Group Technique consisted of three steps: an individual, confidential ranking of the 23 factors by importance from 1 to 23, a group discussion of each attribute, including a group review of the rankings, and an opportunity to alter the original ranking across participants.
Results
The total number of residents (36) were a representative sample of specialty, year of residency, and sex. There was strong agreement about the most highly rated attributes which grouped naturally into themes of autonomy, meaning, competency and relatedness. There was also disagreement on several of the attributes that is likely due to the differences in residency specialty and subsequently rotation requirements.
Conclusion
This study identified the need to address multiple organizational factors that may lead to physician burnout. There is a clear need for complex interventions that target systemic and program level factors rather than focus on individual interventions. These results may help residency program directors understand the specific attributes of a burnout prevention program valued by residents. Aligning burnout interventions with resident preferences could improve the efficacy of burnout prevention programs by improving adoption of, and satisfaction with, these programs. Physician burnout is a work-related syndrome characterized by emotional exhaustion, depersonalization, and a sense of reduced personal accomplishment [1]. Burnout is present in epidemic proportions and was estimated to occur in over 50 percent of practicing physicians and in up to 89 percent of resident physicians pre-COVID 19. The burnout epidemic is growing; a recent national survey of US physicians reported an 8.9 percent increase in burnout between 2011 and 2014 [2]. Rates of physician burnout have also increased [3] during the COVID-19 pandemic with a new classification of “pandemic burnout” experienced by over 52 percent of healthcare workers as early as June of 2020 [4]. Physician burnout can lead to depression, suicidal ideation, and relationship problems that may progress to substance abuse, increased interpersonal conflicts, broken relationships, low quality of life, major depression, and suicide [5–7]. The estimated rate of physician suicide is 300–400 annually [8–10].
Citation: Nelson V, Dubov A, Morton K, Fraenkel L (2022) Using nominal group technique among resident physicians to identify key attributes of a burnout prevention program. PLoS ONE 17(3): e0264921. https://doi.org/10.1371/journal.pone.0264921
Editor: Sergio A. Useche, Universitat de Valencia, SPAIN
Received: March 1, 2021; Accepted: February 19, 2022; Published: March 18, 2022
Copyright: © 2022 Nelson et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data will be held within a public repository. DOI https://doi.org/10.5061/dryad.n8pk0p2vv.
Funding: The authors received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
It often takes a tragedy like the one suffered at a New York City hospital where two first-year residents died by suicide within four days, to draw attention to the seriousness of the problem [11]. A 2017 study of medical residents and causes of death found that suicidal ideation increased 370 percent over the first three months of the first year of training [12]. Burnout also negatively affects physicians’ friends and family, patient care and safety, as well as healthcare organizations and systems. In a clinical setting, evidence demonstrates that burnout results in prescription errors, and reduces the quality of medical services potentially affecting inter-professional relationships [5–7, 13]. According to recent estimates, the burnout epidemic costs the healthcare system approximately $4.6 billion a year [13].
Existing research on burnout prevention among residents focuses primarily on physician-directed interventions, such as building personal resiliency [14] and practicing mindfulness-based stress reduction [15] A recent meta-analysis of 20 controlled studies evaluating over 1500 physicians found that these interventions are associated with insignificant benefits and suggested the focus should be on the adoption of organization-directed approaches [16]. Thus, there is an urgent need to design and deliver effective interventions aimed to reduce burnout that will result in high uptake by medical residents that may extend to other physician groups. Understanding how residents value various aspects of burnout interventions is vital to both the design and evaluation of burnout prevention programs [17]. Aligning burnout interventions with residents’ preferences could improve the effectiveness of burnout prevention programs by improving adoption of, and satisfaction with, these programs.
Discrete choice experiment (DCE) research can be used to understand how stakeholders value specific components of possible programs or interventions. For example, this approach was used to model the physician preferences regarding a pay-for performance incentive program, [18] adherence promoting programs, [19] or residency choices of medical students [20]. Programs developed based on stakeholders’ preferences result in improved adoption and satisfaction with the program. Existing burnout prevention programs are rarely informed by medical residents and are consequently often underutilized. We seek to design a burnout prevention program based on the preferences of medical residents using a DCE study. The first step in designing a DCE study is to identify which attributes to include in the study design. The purpose of this focus group study is to identify the attributes to include in a future DCE study using the Nominal Group Technique (NGT).
Methods
Three resident focus groups (consisting of 10–14 participants/session) were conducted in July and August 2020, in three medical centers in California to prioritize resident preferences for burnout intervention attributes. We purposely chose programs based at diverse healthcare organizations. The first program was based at a university with 900 resident physicians across all medical specialties. Residents rotated at both the university hospital and the affiliated Veterans Affairs Medical Center. The second program was based at a large (476 beds) not-for-profit medical center with family medicine, internal medicine, and emergency medicine residency programs (over 100 residents in total). The third program was based at a community teaching hospital with 14 residents enrolled in a family medicine residency program.
All residents were emailed an invitation to participate in a focus group at a predetermined time. Anyone who presented the day of the focus group could participate. Due to the COVID-19 pandemic, two focus groups had to meet via Zoom, while one group met in person. Participants were considered eligible for inclusion in the study if they were currently part of a residency program at one of these three centers. They were purposely selected to represent the full program spectrum of age, sex, specialty, type of hospital, and year of residency.
The moderator had no reporting relationships with residents or faculty in any participating program, and a neutral note-taker was present during each focus group. Participants provided oral informed consent before discussions began. Focus groups were audio-recorded and transcribed; all identifiers were removed before analysis. After each focus group, participants were provided with information on available resources for addressing burnout. The Loma Linda University institutional review board approved the study as expedited.
Study design
NGT was selected as a strategy for identifying relevant attributes and relative ranking of importance. NGT is a qualitative exploratory research method often used in situations that require complex problem solving, decision-making, priority-setting, and consensus reaching [21]. It is defined as “a structured method for group brainstorming that encourages contributions from everyone and facilitates quick agreement on the relative importance of issues, problems, or solutions” [21–23]. Participants present their favorite ideas, and the suggestions are then discussed and prioritized by the entire group. In this way NGT combines the importance ratings of individual group members into the final weighted priorities of the group.
NGT combines quantitative and qualitative data collection in a group setting while avoiding the issues of group dynamics present in other group methods such as Delphi methods or brainstorming. Additional advantages of NGT over other group methods include reduced researcher bias due to the structured nature of the process, expedient achievement of data saturation, and its ability to simultaneously address problem identification, development of solutions, and establishing priorities for action. This method can be implemented within a narrow timeframe, and it is easily understood by participants. Responses are weighted by participants, and the process is transparent, inclusive and can be easily replicated. NGT promotes even participation and facilitates representation of the implicit views of the group [24–26].
Procedures
We conducted a literature search to identify systematic reviews of interventions to reduce burnout in groups of physicians (medical residents, interns, practicing physicians). Both individual and organization directed interventions were included. We restricted our review of existing studies to a time limit of 10 years and found eight systematic reviews [16, 17, 27–32]. Table 1 lists key characteristics of the systematic reviews, and a flow chart is included as Fig 1.
Flow Chart
Twenty-three potential important attributes of intervention programs were established from this literature review. Table 2 lists the attributes numbered from one to twenty-three.
The NGT process consisted of three steps. After a welcome from the facilitator, participants were asked to introduce themselves, including specialty and year of residency. The facilitator then described the purpose of the study, the process, and the survey list of potential attributes with a description of each attribute. These attributes were displayed in Google Sheets for all participants to view in real-time. Participants were then asked to complete the survey, ranking each potential attribute from 1–23 with one being the most important and twenty-three being the least. Ranking occurred anonymously using Google Sheets. Each participant was assigned an individual sheet displaying the list of attributes in one column and the drop-down list of ranking values (1–23) in the adjacent column. Participants ranked each attribute, assigning only one number per attribute. The aggregate ranking of each attribute was instantly updated on a separate summary sheet available to the group facilitator. The summary sheet displayed individual anonymous rankings of each group member (e.g., column B) together with the list of attributes sorted according to the rank order at the group level.
During a second step (discussion and sharing ideas), a group discussion on each of the attributes was held and included a group review of the aggregate score of the initial ranking, displayed in the summary sheet and presented to participants. This discussion reviewed if the group endorsed the highest-ranking attributes and contextualized responses and the rank-ordering. After the three group discussions, the rank order of the attributes was tallied. Finally, residents had an opportunity to reconsider their initial ranking of the attributes based on the group discussion. They were not pressured to alter their ranking nor to achieve consensus. The facilitator endeavored to ensure that all participants were given the opportunity to contribute. None of the participants opted to change their ranking of attributes following discussion.
Results
A total of 36 residents participated across the three sites with 10, 12, and 14 participants per focus group. As seen in Table 3, the residents were a representative sample of specialty, year of residency, sex, and program location. Internal Medicine and Family Medicine were the specialties represented by the greatest number of participants (n = 16 and n = 15) followed by a small number of residents from Anesthesia, Surgery, and Emergency Medicine. The groups were almost equally divided by sex and year of residency.
“Using the direct rating method,” we calculated mean importance scores and standard deviations for each program attribute. The mean importance score was calculated by dividing the total points awarded per attribute by the total number of residents participating in the focus groups. The mean scores show the group aggregate rank, whereas standard deviation show the spread, i.e., disagreement of the group’s responses around that result. Based on the mean importance score and calculated standard deviations, we made the ranking of attributes from most (lowest mean) to least (highest mean) important (Table 4). Similar to other studies using nominal group technique to identify attributes for inclusion in discrete choice experiments, we plan to include the top 10 most important attributes in our follow-up study. Consequently, only top 10 ranked attributes are highlighted in the discussion [33, 34]. The highest agreement among participants with the resulting smallest standard deviation was from attribute 6 –emphasize learning over service–with participants agreeing that it is a useful intervention (4.74 standard deviation). Next highest agreement, attribute 20 –communication skills training–with participants agreeing that it is not a useful intervention (4.87). The highest prioritized intervention was attribute 1 –flexible scheduling–with participants agreeing that this is the most wanted intervention, though with a higher standard deviation (4.91). The highest disagreement among participants with the resulting highest standard deviation was attribute 21, spiritual nurturing and care (7.51), followed by attribute 10, mindfulness and self-care training (7.40) and attribute 5, enforcing sensible patient censuses (6.71). This disagreement can be attributed to the differences in residency programs and types of hospital settings. One setting was a faith-based organization and some residents spent more time in outpatient versus inpatient settings that require the consideration of a patient census limit for trainees.
We review the top 10 priority interventions and provide participants’ quotes to help contextualize the ratings in the following sections.
Flexible schedule allowing for time off work
This attribute was identified as the essential aspect of the burnout prevention programs. In all three focus groups, residents prioritized time for personal and professional needs as the most critical aspect of burnout prevention. Residents talked about their time constraints preventing them from practicing self-care, mindfulness, or meaningful learning. Time off work and flexibility in scheduling was discussed as a baseline for any other wellness intervention. When on-call schedules do not allow residents to meet basic needs such as sleep and self-care, any additional intervention to prevent burnout is meaningless.
“Scheduling is so important. This is a change that doesn’t take much work to implement and will have a real, immediate impact. It is no stretch of the imagination to think working fewer days in a row could mean a better quality of life. More rest between shifts means a better quality of life. More time-off requests granted means more of us are happier in the long-term. This is the most immediate and tangible solution instead of stress management or meditation promoted by some programs.”
Limiting the duration of shift
Residents discussed the importance of fatigue as a precursor to burnout. They talked about how long hours, heavy workloads, and constant pressure are emotionally taxing. Participants emphasized the ability to spend time away from work to reenergize, engage in activities that can prevent burnout, and maintain a life outside of work.
“Having shorter shifts means I have time either before or after my shift to spend time with my family, work out, cook meals, and be a normal person outside of the hospital.”
“Having a shorter shift structure would let me recover sooner after night shifts. You are not completely wiped out at the end of your shift.”
Physical exercise
Residents described the protective mechanisms of physical fitness and exercise. Participants spoke about the benefits of exercise not only for physical, but also for their mental health and well-being. There was a discrepancy in ranking this intervention. University hospital residents ranked it lower than the other two programs, possibly due to their easy access to a state-of-the-art recreation and wellness center.
“Exercise is the best way for me to sweat off stress and anxiety. I keep talking to my patients about the importance of physical activity, yet my schedule is so hectic that I cannot find enough time to exercise regularly. It takes about 20 minutes to get to the gym, and I feel like our minutes are precious. I am struggling to integrate exercise into my routine.”
Protected sleep time
Despite the work hours restrictions, it remains challenging to get enough sleep while in residency. Residents linked sleep deprivation to burnout and discussed this issue as requiring system change rather than individual attention. They mentioned difficulties in noticing and addressing their sleep deprivation, exacerbated by an “I can handle it” mentality.
“Once I ended up being on call for six days in a row. Near the end, I was so tired that I stopped smiling. I was so tired of running on no sleep. During that time, we were getting late-night calls from the ER, but you are expected to see patients during the day as well. So, you are running on no sleep. At the end of the sixth day, I developed a fever and had to call in sick.”
Sensible patient census
Residents articulated the distinction between hours worked per shift and the number of admissions during the shift. Because of work hour restrictions by accreditation policies, [35] programs may only focus on hours rather than patient numbers. While their programs seemingly understand the connection between burnout and workload, residents perceived their program directors being more concerned with burnout resulting from hours worked per shift rather than the intensity of work completed during those hours. Enforcing sensible patient census to control workload was a priority for the university hospital and medical center, but less critical for the community hospital residency.
“I can have several long shifts scheduled back-to-back but uneventful and still feel fine. Sometimes my shifts are more staggered and seem more manageable, but I end up caring for a lot more patients or dealing with many end-of-life situations and feel much more run down. It is not only about how many hours I spent in the hospital, but also what happens during those hours that matters for burnout.”
Protected learning time
Meaningful learning experiences are essential during residency. Participants emphasized the fact that residency is an educational experience. They work long hours for minimal pay in exchange for the benefits of clinical education. For residents to receive the most benefit from their training, programs need to prioritize their learning by securing uninterrupted educational time. This protected time would show residents that hospitals value their personal growth and training over their service.
“It is refreshing to have an opportunity to invest in your education and improvement as a physician. At least in part, my burnout experience was connected to a feeling of not being prepared to face some of the more challenging clinical situations. One solution is taking some time away from chaos, turning off your pager, and having a meaningful learning experience. I listen better, ask more questions, and learn more when I can have no responding duties during educational conferences.”
Non-physician staff to offload EHR imposed stress
According to a recent study, the average physician spends 28 hours on clinical documentation during nights and weekends each month [21]. Residents in this study reported a high level of stress from Electronic Health Record (EHR) systems and dissatisfaction from reduced direct face time with patients. One resident commented:
“At times, I feel like EHR robs me of everything that gives me meaning in my profession–it makes me less efficient, less able to spend time with my patients. It is very disruptive to the flow of my work.”
Voice for residents to influence change
Residents identified programs’ unresponsiveness to residents’ feedback as one of the factors contributing to burnout. Residents perceived themselves as being at the bottom of the hospital hierarchy and rarely having control over their hours, schedule, and work environment. Having a sense of control and being able to influence change can moderate the intensity of workload and burnout. Residents prioritized the ability to have a forum where they can voice concerns and see them addressed.
“Giving residents a voice as much as possible is a great way to sustain our motivation and prevent apathy. Let us tell you what works and what interventions are effective (or not). We are the ones on the frontlines.”
Social connections
Participants discussed multiple ways in which residency programs and healthcare practice can lead to a sense of isolation and loneliness, resulting in burnout. They talked about many hours spent in front of their computers working on medical records, high case workload, and online learning, leading to reduced face-to-face interactions. They highlighted the connection between social support and reduced burnout.
“One of my priorities as of late is to invest in relationships. Residency can be lonely and isolated. This sense of isolation can lead to burnout. I believe we need more personal, human interaction beyond quick, professional exchanges. We need opportunities to talk with each other about our work and lives. We need to vent about bad stuff, to talk about meaningful experiences at work, to share tips about surviving this place without losing our sanity.”
Mindfulness
Some residents found mindfulness useful for combating stress and burnout. They talked about the importance of self-awareness and positive self-reinforcement to reduce their emotional response to stressors.
“I’ve been focusing on self-care and fostering my sense of connection, not only to family and friends, but to myself. I am figuring out what really inspired me, what really motivated me. This practice of mindfulness can be very cathartic or Zen.”
However, there was a high level of disagreement between the usefulness of mindfulness and stress-management classes to reduce burnout. Residents at the community hospital ranked this intervention remarkably high, while residents at the university hospital ranked it exceptionally low.
“Our program focuses a lot on this. Pretty much every retreat we go through, that is the one thing they do. And I am so glad to see the agreement that this is not helpful.”
Discussion
Although burnout during residency training has been widely acknowledged, there is less consensus regarding ways to address this problem [16, 17]. Little is known about which interventions among this variety of burnout prevention strategies residents value the most. This study provides an overview of the interventions residents value most to combat burnout, and what makes those interventions valuable. Our findings offer insight that may help focus future prevention efforts.
The study results suggest that residents prioritize interventions targeting systemic and program-level factors contributing to burnout over the individual interventions. This finding is in line with several recently published studies [36–38]. Participants connected burnout experience to their working conditions and characteristics of their programs rather than their failure to maintain a proper work-life balance or inability to manage stress. A variety of factors contribute to residents’ burnout: reduced sense of control over their schedule, their workload, time spent with patients, and their learning; reduced sense of belonging and social connectedness; reduced time for self-care and patient care; reduced sense of meaning and purpose. As a result, participants considered systems factors the most critical target for interventions, rather than individual strategies such as stress management. They prioritized flexible schedules and protected learning time to increase control over their work and learning environment and have a voice and ability to influence change. Residents wanted their programs to cultivate meaning in residency training by bringing them back to patient care (e.g., relieving the EHR burden). They talked about addressing the reasons why they are pulled away from the bedside, such as high patient census and high volume of administrative tasks. Participants referred to time as the priority for interventions to reduce burnout–limiting durations of shifts to allow for self-care and protecting sleep time. Finally, they mentioned the importance of social connectivity and practicing within a work environment that fosters a sense of belonging.
Residents prioritized several individual-level interventions such as physical exercise programs and mindfulness training. However, these interventions were on the list of the attributes with the highest disagreement among participants. Residents of one program were dissatisfied with the sole focus of their program administration on boosting physicians’ resilience as the best strategy for dealing with burnout. It is also worth mentioning the contextual differences between programs that factored into residents’ preferences. While a physical exercise program was highly prioritized by medical center and community hospital residents, it was scored lower by the university hospital residents who had easy access to an on-site, state-of-the-art wellness center that remained open in the pandemic with classes outdoors. Stress management and mindfulness training were a high priority for community hospital residents, and a low priority for medical center and university hospital residents who had this incorporated by their programs and realized it was not helpful.
The results of our study support the need for interventions to address multiple organizational factors leading to burnout as opposed to addressing a single issue such as workload or shift patterns. There is a need for complex interventions that will target several systemic and program level factors [39, 40] while including individual-level interventions when indicated. Organizational psychology suggests several factors that may support intrinsic motivation of residents: autonomy, meaning, competence, and relatedness. As our study shows, these factors tend to be reduced during residency and may result in burnout. Interventions prioritized by residents can be grouped according to these four factors: autonomy (e.g., flexible schedule, forum to influence change), meaning (e.g., sensible patient census, support staff to offload clerical burdens), competence (e.g., protected learning time, feedback from faculty), and relatedness (e.g., social support).
The literature has not clarified guidance on NGT sample size, and this study, like others before, used a small number of focus groups. We were aware that the restriction of our project timeframe to conduct focus groups may have led to limited depth of discussion of each attribute. Subsequent groups may have been able to broadened the scope and breadth. We used eight systematic literature reviews of interventions, both individual and institutional, for prevention of burnout, to develop the list of 23 attributes. In addition, we spoke with experts to refine our list. However, we cannot rule out other burnout program attributes that are important to some residents or that additional groups of residents may have added other attributes not mentioned in the current study methods. In addition, the samples for the focus groups contained a relatively high proportion of internal medicine and family medicine residents but a limited representation of several other specialties. Participants also represented three diverse healthcare centers. Some might consider our selection of participants to be less generalizable to all residency programs and medical specialties. Future studies could do this procedure on each medical speciality and compare and contrast the program attribute rankings in interesting ways. A further limitation is that we have not compared the attributes derived from NGT with other approaches (e.g., Delphi, best-worst scaling, brainstorming). Head-to-head comparisons of different techniques could help to assess and understand strengths and weaknesses between approaches.
This study identified and ranked burnout prevention preferences in a resident population, showing that there is a need to address multiple organizational factors leading to burnout. There is also a need for complex interventions that will target several systemic and program level factors. While there was variability about attributes the groups perceived to be less important, as well as variability between healthcare centers, there was strong agreement about the most highly rated attributes. The attributes group naturally into themes of autonomy, meaning, competency, and relatedness. The next steps in this process will enable the researchers to refine the methods to determine measures for each attribute. The results of this study may help residency program directors understand what specific attributes of a burnout prevention program residents will value. The results will also be used to design a DCE study that would model residents’ burnout prevention program preferences. This may advance understanding and support for evidence-based, best practices as well as increase residents participation in, and adoption of, burnout prevention programs resulting in a potential decrease in rates of resident burnout. These same attributes could also be explored with physicians across specialities to determine whether the four themes of autonomy, meaning, competency, and relatedness also apply after board certification and across specialties.
References
- 1. Maslach C JS, Leiter MP, Schaufeli WB, Schwab RL. Maslach burnout inventory. Consult Psychol Press. 1986.
- 2. Shanafelt T, Hasan O, Dyrbye L, Sinsky C, Satele D, Sloan J, et al. Changes in Burnout and Satisfaction With Work-Life Balance in Physicians and the General US Working Population Between 2011 and 2014. Mayo Clin Proc. 2015;90:1600–13. pmid:26653297
- 3. Civantos AM, Byrnes Y, Chang C, Prasad A, Chorath K, Poonia SK, et al. Mental health among otolaryngology resident and attending physicians during the COVID-19 pandemic: National study. Head Neck. 2020;42(7):1597–609. pmid:32496637
- 4. Khasne RW, Dhakulkar BS, Mahajan HC, Kulkarni AP. Burnout among Healthcare Workers during COVID-19 Pandemic in India: Results of a Questionnaire-based Survey. Indian J Crit Care Med. 2020;24(8):664–71. pmid:33024372
- 5. Dyrbye L, Shanafelt T. A narrative review on burnout experienced by medical students and residents. Med Educ. 2016;50(1):132–49. pmid:26695473
- 6. Rothenberger DA. Physician Burnout and Well-Being: A Systematic Review and Framework for Action. Dis Colon Rectum. 2017;60(6):567–76. pmid:28481850
- 7. West CP, Dyrbye LN, Shanafelt TD. Physician burnout: contributors, consequences and solutions. J Intern Med. 2018;283(6):516–29. pmid:29505159
- 8. Collier R. Physician suicide too often "brushed under the rug". CMAJ. 2017;189(39):E1240–e1. pmid:28970266
- 9. Eckleberry-Hunt J, Lick D. Physician Depression and Suicide: A Shared Responsibility. Teach Learn Med. 2015;27(3):341–5. pmid:26158336
- 10. Harvey EJ M.D. Burnout should not e a silent epidemic. Can J Surg. 2019.10,1503/cjs.000819. pmid:30689330
- 11. Slavin SJ, Chibnall JT. Finding the Why, Changing the How: Improving the Mental Health of Medical Students, Residents, and Physicians. Acad Med. 2016;91(9):1194–6. pmid:27166866
- 12. Yaghmour NA, Brigham TP, Richter T, Miller RS, Philibert I, Baldwin DC Jr., et al. Causes of Death of Residents in ACGME-Accredited Programs 2000 Through 2014: Implications for the Learning Environment. Acad Med. 2017;92(7):976–83. pmid:28514230
- 13. Han S, Shanafelt TD, Sinsky CA, Awad KM, Dyrbye LN, Fiscus LC, et al. Estimating the Attributable Cost of Physician Burnout in the United States. Ann Intern Med. 2019;170(11):784–90. pmid:31132791
- 14. Haramati A, Cotton S, Padmore JS, Wald HS, Weissinger PA. Strategies to promote resilience, empathy and well-being in the health professions: Insights from the 2015 CENTILE Conference. Med Teach. 2017;39(2):118–9. pmid:28103729
- 15. Shanafelt TD. Enhancing meaning in work: A prescription for preventing physician burnout and promoting patient-centered care. JAMA. 2009;302(12):1338–40. pmid:19773573
- 16. Panagioti M, Panagopoulou E, Bower P, Lewith G, Kontopantelis E, Chew-Graham C, et al. Controlled Interventions to Reduce Burnout in Physicians: A Systematic Review and Meta-analysis. JAMA Intern Med. 2017;177(2):195–205. pmid:27918798
- 17. Busireddy KR, Miller JA, Ellison K, Ren V, Qayyum R, Panda M. Efficacy of Interventions to Reduce Resident Physician Burnout: A Systematic Review. J Grad Med Educ. 2017;9(3):294–301. pmid:28638506
- 18. Chen T-T, Mei-Shu Lai, and Kuo-Piao Chung. "Participating physician preferences regarding a pay-for-performance incentive design: a discrete choice experiment." Int J Qual Health Care. 2016; 28(1):40–6. pmid:26660443
- 19. Mülle S, Ziemssen T, Diehm C, Duncker T, Hoffmanns P, Thate-Waschke IM, et al. "How to Implement Adherence-Promoting Programs in Clinical Practice? A Discrete Choice Experiment on Physicians’ Preferences." Patient Prefer Adherence. 2020;14.
- 20. Wang T, Wong B, Huang A, Khatri P, Ng C, Forgie M, et al. "Factors affecting residency rank-listing: a Maxdiff survey of graduating Canadian medical students." BMC Med Educ. 2011;11.1. pmid:21414226
- 21. Bromley P. Using Nominal Group Technique (NGT) to reach consensus on Graduate Attributes for nurses undertaking Postgraduate Certification in Neonatal Intensive Care in Australia. JNN. 2014;20(6):245–52.https://doi.org/10.1016/j.jnn.2014.08.002.
- 22. Nominal Group Technique [Internet]. 2020. Available from: https://asq.org/quality-resources/nominal-group-technique.
- 23. Hiligsmann M, van Durme C, Geusens P, Dellaert BG, Dirksen CD, van der Weijden T, et al. Nominal group technique to select attributes for discrete choice experiments: an example for drug treatment choice in osteoporosis. Patient Prefer Adherence. 2013;7:133–9. pmid:23412964
- 24. Fournier JP, Escourrou B, Dupouy J, Bismuth M, Birebent J, Simmons R, et al. Identifying competencies required for medication prescribing for general practice residents: a nominal group technique study. BMC Fam Pract. 2014;15:139. pmid:25084813
- 25. Justice J, Jang R. Tapping employee insights with the Nominal Group Technique. Am Pharm. 1990;NS30(10):43–5. pmid:2239690
- 26. McMillan SS, King M, Tully MP. How to use the nominal group and Delphi techniques. Int J Clin Pharm. 2016;38(3):655–62. pmid:26846316
- 27. Clough B, March S, Chan R, Casey L, Phillips R, Ireland M. Psychosocial interventions for managing occupational stress and burnout among medical doctors: a systematic review. Syst Rev. 2017;6:1–19. pmid:28077170
- 28. DeChant PF, Acs A, Rhee KB, Boulanger TS, Snowdon JL, Tutty MA, et al. Effect of Organization-Directed Workplace Interventions on Physician Burnout: A Systematic Review. Mayo Clin Proc Innov Qual Outcomes. 2019;3(4):384–408. pmid:31993558
- 29. Kalani S, Azadfallah P, Oreyzi H, Adibi P. Interventions for physician burnout: A systematic review of systematic reviews. Int J Prev Med. 2018;9(1):81–. pmid:30283613
- 30. Perski O, Grossi G, Perski A, Niemi M. A systematic review and meta-analysis of tertiary interventions in clinical burnout. Scand J Psychol. 2017;58(6):551–61. pmid:29105127
- 31. West CP, Dyrbye LN, Erwin PJ, Shanafelt TD. Interventions to prevent and reduce physician burnout: a systematic review and meta-analysis. Lancet (London, England). 2016;388(10057):2272–81. pmid:27692469
- 32. Walsh AL, Lehmann S, Zabinski J, Truskey M, Purvis T, Gould NF, et al. Interventions to Prevent and Reduce Burnout Among Undergraduate and Graduate Medical Education Trainees: a Systematic Review. Acad Psychiatry. 2019;43(4):386–95. pmid:30710229
- 33. Odu GO. "Weighting methods for multi-criteria decision making technique.". JASEM. 2019;23.8:1449–57.
- 34. Kremer IEH, Evers SMAA, Jongen PJ, van der Weijden T, van de Kolk I, Hiligsmann M. Identification and Prioritization of Important Attributes of Disease-Modifying Drugs in Decision Making among Patients with Multiple Sclerosis: A Nominal Group Technique and Best-Worst Scaling. PloS One. 2016;11(11):e. pmid:27812117
- 35. Antiel RM TS, Reed DA, James KM, Tilburt JC, Bannon MP, Fischer PR, Farley DR. ACGME duty-hour recommendations–a national survey of residency program directors. N Engl J Med. 2010; Aug 19;363(8):e12. pmid:20842785
- 36. Oetgen WJ. PRACTITIONER APPLICATION: An Organizational Intervention to Reduce Physician Burnout. J Healthc Manag. 2018;63(5):352–3. pmid:30180033
- 37. Gregory ST, Menser T, Gregory BT. An Organizational Intervention to Reduce Physician Burnout. J Healthc Manag. 2018;63(5):338–52. pmid:30180032
- 38. Collier R. Addressing physician burnout at the systems level. CMAJ. 2018;190(6):E174–E. pmid:29440342
- 39. Romani M, Ashkar K. Burnout among physicians. Libyan J Med. 2014;9:1-N.PAG. pmid:24560380
- 40. Dzau VJ, Kirch DG, Nasca TJ. To Care Is Human—Collectively Confronting the Clinician-Burnout Crisis. N Engl J Med. 2018;378(4):312–4. pmid:29365296