http://orcid.org/0000-0003-4313-743X
Figures
Abstract
Background
Despite the general agreement regarding the central role of the clinical learning environment in graduate medical education, its assessment remains challenging owing to the lack of available standardized measures. We report on the cross-cultural adaptation and psychometric assessment of the Brazilian-Portuguese version of Seelig’s Resident Questionnaire.
Methods
The present study was performed in two steps. First, a cross-cultural translation and adaptation of the Resident Questionnaire was conducted through multiple translations, synthesis of versions, back-translation, content validation, and face validation. Subsequently, a sample of 288 (72%) resident physicians enrolled in 40 residency programs at a Brazilian university hospital completed the following measures: 1) the Brazilian-Portuguese version of the Resident Questionnaire (for factor analysis and to determine internal consistency, reliability, and validity); 2) three existing, validated psychometric measures (to determine convergent and divergent validity); and 3) a self-report questionnaire.
Results
Confirmatory factor analysis results provided support for the three-dimensional model of the Resident Questionnaire in use on a sample of Brazilian resident physicians, having been previously verified for use in American samples. All three factors (emotional distress, learning environment satisfaction, and workload satisfaction) verified in the confirmatory factor analysis showed good internal consistency (α > .80), reliability (Raykov’s rho > .80), and correlations in the expected directions and magnitude with measures of depressive symptoms, duty hours, organizational conditions, and emotional exhaustion.
Conclusions
This study is the first to adapt a measure of the clinical learning environment of residency programs into Brazilian Portuguese. Our findings suggest that the adapted version of the Resident Questionnaire is valid and reliable for assessing Brazilian residency programs. This free, easy-access, and fast-application instrument may be a useful standardized measure for research and educational purposes concerning the clinical learning environments of resident physicians.
Citation: Pereira-Lima K, Silva-Rodrigues APC, Marucci FAF, Osório FdL, Crippa JA, Loureiro SR (2018) Cross-cultural adaptation and psychometric assessment of a Brazilian-Portuguese version of the Resident Questionnaire. PLoS ONE 13(9): e0203531. https://doi.org/10.1371/journal.pone.0203531
Editor: Oathokwa Nkomazana, University of Botswana Faculty of Medicine, BOTSWANA
Received: March 12, 2018; Accepted: August 22, 2018; Published: September 4, 2018
Copyright: © 2018 Pereira-Lima et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: This study received financial support from the São Paulo Research Foundation (FAPESP grant 2016/13410-0) - website: fapesp.br (K.P-L.). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
The significance of the clinical learning environment in resident physicians’ performance, well-being, and patient care has been a topic of increasing debate in recent literature [1–4]. Although the clinical learning environment is likely to play a central role in graduate medical education, its assessment remains challenging, owing to the lack of available standardized measures [5].
Among the standardized measures for evaluating the clinical learning environments of resident physicians, the Resident Questionnaire (RQ) has been demonstrated to be a valid measure for assessing residents’ perceptions of critical aspects of their programs [6]. Since its development in 1993 [7] and subsequent revision and validation in 1995 [6], the aspects measured by the RQ have been associated with mental health, education, and satisfaction of resident physicians and fellows from different programs in the United States [6, 8–11].
In the original validation study of the RQ, residents were asked to indicate whether they agreed with 33 statements using a self-report Likert scale ranging from 1 = Strongly Disagree to 5 = Strongly Agree [6]. The results identified 28 valid items categorized into the following three correlated first-order factors: emotional distress (11 items, α = .89), workload satisfaction (8 items, α = .85), and learning environment satisfaction (9 items, α = .82). The correlation analysis in the original study verified strong correlations between emotional distress and workload satisfaction (r = -.71). Less robust correlations were found between emotional distress and learning environment satisfaction (r = -.47), and between workload satisfaction and learning environment satisfaction (r = .55). These three dimensions evaluated by the RQ have been shown to be valid for the assessment of different aspects of residents’ perceptions of their programs [6].
Although other measures of the clinical environment of resident physicians have been developed in recent years, most do not present strong evidence of validity [5]. Others assess numerous factors related to the residency context/legislation of a specific country [12, 13], or are long [13, 14], which could impair their use in large surveys on resident physicians. In addition, besides its assessment of residents’ satisfaction with their workload and learning environment, the RQ has the advantage of including a measure of residents’ emotional distress. This could be useful for identifying trainees at risk of developing stress-related problems.
Herein, we set out to describe the process of the cross-cultural adaptation and psychometric assessment of a Brazilian-Portuguese version of the RQ. Our study uses a sample of resident physicians enrolled in 40 residency programs at a Brazilian university hospital. Our specific goals were as follows: a) to perform a cross-cultural translation and adaptation of the RQ to Brazilian Portuguese; b) to assess the content and face validity of this Brazilian-Portuguese version; c) to examine the factor structure of the Brazilian-Portuguese RQ; d) to assess the internal consistency and reliability of the Brazilian-Portuguese RQ; and e) to evaluate the convergent and divergent validity of the Brazilian-Portuguese RQ, with measures of both positive and negative organizational characteristics, depressive symptoms, and emotional exhaustion.
Methods
After obtaining ethical approval from the Institutional Review Board, the present study was conducted using the following two steps: 1) cross-cultural translation and adaptation of the RQ into Brazilian Portuguese and 2) psychometric assessment of the validity and reliability of the Brazilian-Portuguese RQ. All participants provided informed consent.
Step 1: Cross-cultural translation and adaptation
The RQ was translated and adapted into Brazilian Portuguese in accordance with standardized technical recommendations [15], as described below.
First, the instrument was independently translated by two bilingual psychiatry residents and two professional translators with more than five years of experience (one Brazilian and one American). These four versions were synthesized by an experienced psychiatrist and an experienced psychologist into a single Brazilian-Portuguese version. This version was then back-translated by a professional American translator.
For content validation, both forward- and back-translation were evaluated by an expert committee comprising one bilingual psychiatrist and two bilingual clinical psychologists, each with more than 10 years of experience in medical residency education. These experts were asked to evaluate whether the items in the two versions were conceptually similar to the target constructs of the original instrument. The conceptual, semantic, idiomatic, and cultural equivalence of the instructions and each of the 28 items were rated by the experts, who suggested cross-cultural adaptation of three items, as described below.
Since the term, hospital support services, is not typically used in the Brazilian health system, the experts suggested adding examples of types of hospital support services in parentheses after the term in the item, “Hospital support services sufficiently help me care for my patients.” Similarly, the experts suggested changing the term, conferences, to “reuniões clínicas” (translation: clinical meetings) in the item, “The scheduled conferences are generally a valuable learning experience,” since this term would be more appropriate for the context in which theoretical and applied subjects are taught and discussed in Brazilian residency programs. Finally, considering differences in the demands of the various specialties included in this instrument undergoing adaptation, the experts suggested adapting the item, “The average number of workups on call days is reasonable,” to “O número médio de chamados (pedidos de exames, bips, urgências, intercorrências) em dias de plantão é razoável” (translation: The average number of calls [workups, bips, urgencies, intercurrences] on call days is reasonable).
For face validation, pilot testing was conducted with a small number (n = 7) of residents from medical and surgical specialties (internal medicine, general surgery, ophthalmology, psychiatry, and radiology) to determine semantic understanding of the instructions and items of the Brazilian-Portuguese version of the RQ. Residents were asked to read and then rephrase all sentences in the instrument. Since they did not demonstrate any difficulty understanding any of the RQ items, this step was considered to be the conclusion of the translation and adaptation of the RQ into Brazilian Portuguese.
Step 2: Psychometric assessment of the validity and reliability of the Brazilian-Portuguese version of the RQ
Data collection.
A total of 400 medical residents of both sexes with varying years of residency (1st to 5th) and specialties at a Brazilian university hospital were invited to participate in the present study between August and December of 2016. The referral hospital is a public university-based institution located in a medium-sized (682,302 people) inner state city in the state of São Paulo, Brazil. Residents were eligible to participate in the study if they were Brazilian or foreign nationals who had resided in Brazil for at least two years. Residents were recruited at their workplaces and those who agreed to participate signed informed consent forms and were asked to complete the following self-report instruments: the Brazilian-Portuguese version of the RQ to obtain data for confirmatory factor analysis (CFA) and to determine the measure’s internal consistency, reliability, and validity; the Patient Health Questionnaire-9 (PHQ-9) [16]; the Burnout Syndrome Inventory (BSI) [17]; a question about weekly duty hours to determine the RQ’s convergent and divergent validity; and a self-report questionnaire on demographic variables including sex, age, residency program, and year of postgraduate study.
The PHQ-9 is a component of the Primary Care Evaluation of Mental Health Disorders inventory comprising nine self-report items designed to screen respondents for depressive symptoms [16]. The instrument has good psychometric properties, as demonstrated in validation studies with primary healthcare patients and the general population [16, 18, 19]. The Positive Organizational Conditions (BSI-POC), Negative Organizational Conditions (BSI-NOC), and Emotional Exhaustion (BSI-EE) scales of the BSI were used in the present study. The full BSI is a Brazilian instrument consisting of 35 items categorized into two parts (16 and 19 items, respectively) [17]. The first part assesses organizational factors reported in literature as triggers or modulators of occupational stress and, consequently, burnout. The second part assesses burnout syndrome through four dimensions [17]. Validation studies with participants from various professional fields, including health care, have demonstrated the psychometric adequacy of the BSI [17].
Descriptive analysis.
After verifying that data on all the variables were normally distributed, using Kolmogorov-Smirnov normality tests, we conducted descriptive analysis (mean, standard deviation, frequency, percentage) to characterize the demographic and academic profiles of the study sample.
Confirmatory factor analysis (CFA).
We conducted CFA to evaluate whether the three-dimensional model of the RQ proposed for assessing American residency programs fits the Brazilian residency context. To account for the categorical characteristics of the RQ items, we performed CFA using the Weighted Least Squares Means and Variance Adjusted estimation method in Mplus 7.11. Model fit was evaluated through the Comparative Fit Index (CFI), Tucker-Lewis Index (TLI), and the Root Mean Square Error of Approximation (RMSEA) with a 90% confidence interval. CFI and TLI values higher than .90 and RMSEA values lower than .08 represent acceptable fit, and CFI and TLI values higher than .95 and RMSEA values lower than .05 represent excellent fit [20, 21]. Items were accepted as part of a particular scale if a factor loading of at least .45 was obtained [22].
Reliability analysis.
The reliability of the Brazilian-Portuguese version of the RQ was assessed through internal consistency analysis using Cronbach’s alpha, and composite reliability using Raykov’s rho coefficient. Alpha values higher than .70 and rho values higher than .80 indicate good reliability [23, 24].
Results
Representativeness of the sample
A total of 288 Brazilian medical residents from 40 residency programs were included in the present study (response rate = 72%). Participants’ demographic and academic characteristics are presented in Table 1.
CFA, internal consistency, and reliability
The CFA of the Brazilian-Portuguese RQ showed an acceptable fit of the model proposed for American Programs in the Brazilian residency context (RMSEA 0.073, 90% CI 0.067–0.079; CFI .938, TLI .932). Pearson’s correlation analysis revealed significant correlations between emotional distress and workload satisfaction (r = -.542, p < .001), emotional distress and learning environment satisfaction (r = -.372, p < .001), and workload satisfaction and learning environment satisfaction (r = .408, p < .001). Similarly, all scales of the Brazilian-Portuguese RQ showed good internal consistency and reliability (α > .70, rho > .80).
Table 2 presents Cronbach’s alpha value and the Raykov’s composite reliability rho coefficient for each dimension of the Brazilian-Portuguese RQ, as well as the standardized loadings, R2, and error variance for each item of the instrument.
Convergent and divergent validation
Table 3 presents the correlation coefficients for the Brazilian-Portuguese RQ and related measures.
Discussion
The present study entailed a cross-cultural translation and adaptation of the RQ into Brazilian Portuguese and evaluation of the psychometric adequacy of this instrument using a sample of medical residents enrolled in 40 residency programs at a Brazilian university hospital. Our results demonstrate the adequacy of the original RQ model [6] for the evaluation of Brazilian residency programs, as well as the instrument’s good validity and reliability for use on this sample of Brazilian resident physicians.
Besides confirming the factorial structure identified in the original validation study of the RQ, the relationships between the three dimensions of the Brazilian-Portuguese version were comparable to those identified in the original study [6]. Significant correlations were found between all dimensions of the instrument. In addition, similar to the results of the original study [6], the RQ-Emotional Distress, RQ-Workload Satisfaction, and RQ-Learning Environment Satisfaction scales have shown good internal consistency and reliability in the Brazilian-Portuguese version of Seelig’s RQ.
In line with the results of previous studies using the RQ on American samples [6, 7], our results showed high correlations of measures of depressive symptoms and emotional exhaustion with the RQ–Emotional Distress scale. The results also showed moderate and weak correlations of these measures with the RQ-Workload and RQ-Learning Environment Satisfaction scales. The convergent validity analysis showed significant correlations in the expected directions and magnitude for all dimensions of the RQ with positive (BSI-POC) and negative (BSI-NOC) organizational conditions assessed by the BSI. This demonstrates the validity of the Brazilian-Portuguese RQ for assessing program-related aspects of residents’ clinical learning environment. In addition, the significant moderate correlations between the Workload Satisfaction Scale and average duty hours per week demonstrated that, although related, these are distinct constructs.
With regard to the strengths of the study, first, standardized procedures for cross-cultural translation and adaptation were used [15]. Second, the study assessed residents enrolled in 40 residency programs in medical and surgical specialties. Third, the study demonstrated content and face validity, adequate factor structure fit, good reliability, and convergent and divergent validity of the Brazilian-Portuguese RQ with measures of organizational conditions, exhaustion, depressive symptoms, and duty hours. The limitations include the assessment of residents from one university-based institution in a medium-sized inner state city in São Paulo, and the lack of test-retest reliability. In addition, although previous research has shown that self-reported duty hours match well with electronic records [25], the average number of duty hours per week may have been misreported. There is a need for further studies on residents from different types of institutions (public/private, university/community-based) and regions, and those assessing test-retest reliability and both the discriminative and predictive validity of the Brazilian-Portuguese RQ. These would enable better exploration of the psychometric adequacy of this version of the instrument.
In summary, this cross-cultural adaptation and validation study showed the Brazilian-Portuguese version of the RQ to be a valid and reliable measure for assessing different aspects of residents’ perceptions of the workload, learning environment, and emotional distress related to their programs. This free, easy-access, and fast-application instrument may be a useful standardized measure for research and educational purposes concerning the clinical learning environment of resident physicians.
Supporting information
S1 File. Brazilian-Portuguese version of the Resident Questionnaire.
https://doi.org/10.1371/journal.pone.0203531.s001
(PDF)
Acknowledgments
The authors wish to thank the medical residents who participated in this study. We would also like to thank medical residents, Juliana Moretti de Souza and Bruno Ravenna Pinheiro Kondo, and the professional translators for their independent translations of the Resident Questionnaire.
References
- 1. Nasca TJ, Weiss KB, Bagian JP. Improving clinical learning environments for tomorrow’s physicians. N Engl J Med. 2014;370(11): 991–993. pmid:24467307
- 2. Weiss KB, Bagian JP, Nasca TJ. The clinical learning environment: the foundation of graduate medical education. JAMA. 2013;309(16): 1687–1688. pmid:23613072
- 3. Jennings ML, Slavin SJ. Resident wellness matters: optimizing resident education and wellness through the learning environment. Acad Med. 2015;90(9): 1246–1250. pmid:26177527
- 4. Schumacher DJ, Slovin SR, Riebschleger MP, Englander R, Hicks PJ, Carraccio C. Perspective: beyond counting hours: the importance of supervision, professionalism, transitions of care, and workload in residency training. Acad Med. 2012;87(7): 883–888. pmid:22622207
- 5. Colbert-Getz JM, Kim S, Goode VH, Shochet RB, Wright SM. Assessing medical students’ and residents’ perceptions of the learning environment: exploring validity evidence for the interpretation of scores from existing tools. Acad Med. 2014;89(12): 1687–1693. pmid:25054415
- 6. Seelig CB, DuPre CT, Adelman HM. Development and validation of a scaled questionnaire for evaluation of residency programs. South Med J. 1995;88(7): 745–750. pmid:7597480
- 7. Seelig CB. Quantitating qualitative issues in residency training: development and testing of a scaled program evaluation questionnaire. J Gen Intern Med. 1993;8(11): 610–613. pmid:8289101
- 8. Hosseini M, Lee JG, Romano P, Hosseini S, Leung J. Educational experiences and quality of life of gastroenterology fellows in the United States. Am J Gastroenterol. 1999;94(12): 3601–3612. pmid:10606326
- 9. Kirton OC, Antonetti M, Morejon O, Dobkin E, Angelica MD, Reilly PJ, et al. Measuring service-specific performance and educational value within a general surgery residency: the power of a prospective, anonymous, Web-based rotation evaluation system in the optimization of resident satisfaction. Surgery. 2001;130(2): 289–295. pmid:11490362
- 10. Blazek BA, Zollinger TW, Look KY. Obstetrics-gynecology resident satisfaction. Am J Obstet Gynecol. 2005;193(5): 1798–1803. pmid:16260237
- 11. Gruppen LD, Stansfield RB, Zhao Z, Sen S. Institution and specialty contribute to resident satisfaction with their learning environment and workload. Acad Med. 2015;90(11): S77–S82.
- 12. Holt KD, Miller RS, Philibert I, Heard JK, Nasca TJ. Residents’ perspectives on the learning environment: data from the Accreditation Council for Graduate Medical Education resident survey. Acad Med. 2010;85(3): 512–518. pmid:20182130
- 13. Fahy BN, Todd SR, Paukert JL, Johnson ML, Bass BL. How accurate is the Accreditation Council for Graduate Medical Education (ACGME) resident survey? Comparison between ACGME and in-house GME survey. J Surg Educ. 2010;67(6): 387–392. pmid:21156296
- 14. Boor K, Van Der Vleuten C, Teunissen P, Scherpbier A, Scheele F. Development and analysis of D-RECT, an instrument measuring residents’ learning climate. Med Teach. 2011;33(10): 820–827.
- 15. Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine. 2000;25(24).
- 16. Spitzer RL, Kroenke K, Williams JB. Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study. Primary Care Evaluation of Mental Disorders. Patient Health Questionnaire. JAMA. 1999;282(18): 1737–1744. pmid:10568646
- 17.
Benevides-Pereira AMT. Análise ISB—Inventário da Síndrome de Burnout 2007. Available from: https://gepeb.wordpress.com/isb/. Cited 9 September 2015.
- 18. de Lima Osorio F, Vilela Mendes A, Crippa JA, Loureiro SR. Study of the discriminative validity of the PHQ-9 and PHQ-2 in a sample of Brazilian women in the context of primary health care. Perspect Psychiatr Care. 2009;45(3): 216–227. pmid:19566694
- 19. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001;16(9): 606–613. pmid:11556941
- 20.
Brown TA. Confirmatory factor analysis for applied research. New York: Guilford Press; 2006.
- 21. Hu LT, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Modeling. 1999;6(1): 1–55.
- 22.
Comrey AL, Lee HB. A first course in factor analysis. 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc; 1992.
- 23.
Gliem JA, Gliem RR. Calculating, interpreting, and reporting Cronbach’s alpha reliability coefficient for Likert-type scales. In: Midwest Research-to-Practice Conference in Adult, Continuing, and Community Education. Columbus, Ohio: The Ohio State University; 2003. pp. 82–88.
- 24. Raykov T. Coefficient alpha and composite reliability with interrelated nonhomogeneous items. Appl Psychol Meas. 1998;22(4): 375–385.
- 25. Todd SR, Fahy BN, Paukert JL, Mersinger D, Johnson ML, Bass BL. How accurate are self-reported resident duty hours? J Surg Educ. 2010;67(2): 103–107. pmid:20656607