Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Innovative Training with Virtual Patients in Transcultural Psychiatry: The Impact on Resident Psychiatrists’ Confidence

  • Ioannis Pantziaras ,

    ioannis.pantziaras@ki.se

    Affiliation Cultural Medicine Unit, Department of Learning, Informatics, Management and Ethics (LIME), Karolinska Institutet, Stockholm, Sweden

  • Uno Fors,

    Affiliation Department of Computer and Systems Sciences, Stockholm University, Stockholm, Sweden

  • Solvig Ekblad

    Affiliation Cultural Medicine Unit, Department of Learning, Informatics, Management and Ethics (LIME), Karolinska Institutet, Stockholm, Sweden

Abstract

Background

Virtual patients are now widely accepted as efficient and safe training tools in medical education, but very little is known about their implementation in psychiatry, especially in transcultural clinical care of traumatized refugee patients.

Objective

This study aimed at assessing the impact of training with a virtual patient on confidence in providing clinical care for traumatized refugee patients.

Methods

The authors developed an educational tool based on virtual patient methodology portraying the case of “Mrs. K”, a traumatized refugee woman with symptoms of PTSD and depression. A group (N=32) of resident psychiatrists tested the system and their confidence in different aspects of providing clinical care for this patient group was evaluated pre- and post-test by using a validated confidence questionnaire. Cronbach’s α was calculated for all clusters. Changes between pre- and post-test were compared by using the matched-pair t-test, binomial distribution for exact significance test and a calculation of effect sizes (Cohen’s d).

Results

A statistically significant improvement was exhibited in overall confidence (mean Δ: 0.34; p<0.0001; d: 0.89) as well as in four more specific domains of clinical care, with the area of identifying and evaluating trauma-related diagnoses and disability showing the most prominent improvement (mean Δ: 0.47; p<0.0001; d: 1.00).

Conclusions

This VP-system can lead to physicians’ improvement of confidence in providing transcultural clinical care for traumatized refugee patients. Further research is required to investigate improvement in actual performance and cognitive outcomes with several VPs and in a long-term effect perspective.

Introduction

Virtual Patients (VPs) have been technologically developed and researched since the 1990s as educational tools used to train different aspects of medical care, most commonly clinical reasoning, bioethics, clinical decision-making, leadership, and history taking [14]. They are broadly defined as “interactive computer simulations of real-life clinical scenarios for the purpose of medical training, education, or assessment” [5] and result in many advantages, some of them being easy accessibility, exposure to critical yet rare cases, training of skills in a safe environment, and standardization [6]. VPs have previously demonstrated an equivalent efficiency in improving the clinical performance and diagnostic ability when compared to standardized patients [7] but found to provoke lower engagement and interest levels compared to real humans [8].

While VPs have been extensively used in various areas of medicine integrating in some cases exciting technological capabilities like natural language dialogue and emotion reasoning [9], they are a rather new paradigm of education in the field of psychiatry [10]. A few studies have been published that involved the use of VPs for training assessment of suicide risk in general [11] and in bipolar patients [12], as well as diagnosing conduct disorder [13, 14], posttraumatic stress disorder in a teenager patient (PTSD) [15], schizophrenia [16], depression with panic disorder [17] as well as major depressive episode presenting with the chief complaint of fatigue and anhedonia [18]. Web-based simulation has been recently reported as a mean to teach violent risk assessment of complex psychiatric patients and the clinical management of follow-up care [19]. To the best of our knowledge, no previous study has been published concerning the formal use of VPs for transcultural training clinical management of traumatized refugee patients, with the exception of previous pilot studies published by our research team examining educational aspects of a preliminary version of our VP-system [20, 21] as well as the learners’ expectations and attitudes towards the current version [22].

We consider transcultural psychiatry to be a crucial domain of systematized knowledge, since the rapid increase of globalization (including migration and especially refugees) leading to more culturally diverse societies [23] has given rise to better understanding of the serious and often long-lasting consequences of the processes of migration and asylum seeking, reception, and exposure to traumatic life events before, during and after arrival in a receiving country. Severe mental disorders such as PTSD, depression [24, 25], cardiovascular disease [26], metabolic syndrome [27], as well as social disability as reflected by unemployment following hospitalization due to mental disorder [28], have been found to have a higher relative risk among immigrants, especially refugees, than in host populations.

We have developed a dedicated VP system that portrays refugee trauma cases for educational use, aiming to enhance the clinical, interpersonal, and cultural competence of trainees, as well as the clinical management of traumatized refugee patients. This system, including a slightly modified version developed as a part of our international cooperation with the Harvard Program for Refugee Trauma (HPRT) [20], has been evaluated with respect to user acceptance and educational potentials, as well as face and construct validity, with promising results [21, 22]. This study aimed to examine the impact of training with this VP-system on confidence in caring for traumatized refugee patients among Swedish psychiatry residents.

Method

Study Subjects

Thirty-two residents in psychiatry at the Karolinska University Hospital, Stockholm and University Hospital, Linköping, Sweden, volunteered to participate in our study in response to an e-mail invitation sent by the respective department director of studies from each hospital after communication with the research team. The director of studies for resident doctors in Sweden has as main responsibility to plan the overall study program of resident doctors, ensure the quality of it and offer advice and support during the training period to the residents as well as to their supervisors [29]. All of the residents that were active and not engaged by other clinical or educational duties at the time, accepted to participate. Twenty out of 32 participants (62.5%) were women and 12 (37.5%) were men. Their mean age was 35.6 years (female: 35.5, men: 35.9; range: 28–51) while mean time of working experience as psychiatry residents was 2.3 years (female: 2.4, men: 2.1; range: 0–5). No statistically significant differences in age and work experience were observed between male and female participants.

The Regional Ethical Committee at the Karolinska Institutet examined this study and its protocol and determined that because the project does not include any sensitive personal data and it is not considered research according to the ethics law, formal ethical approval was not required. The authors had access to information about age, sex, email address and work experience in years. This information falls under the Swedish definition of a research register bank according to the Personal Data Law. Because of that a separate application was sent and approved by the legal counsel at University Administration at the Karolinska Institutet. Data was anonymised prior to analysis.

Materials

By using Adobe Flash CS4 Professional we developed a VP system called RT-Sim (Refugee Trauma Simulation), intended to train how to manage refugee patients. The current VP case tested in this study portrays a 45-year-old female Bosnian refugee patient (“Mrs. K”) with previous exposure to severe trauma, presenting with serious symptoms of PTSD, major depression and headache (Fig. 1). The VP is presented as a series of pre-recorded video clips that are displayed according to the questions asked by the user. Generic video sequences (i.e., the patient waiting for the doctor to ask a question, coughing, crying, etc.) were also utilized in order to achieve a higher degree of realism. During the interaction with the VP, the user was able to perform a medical interview and physical examination, use screening instruments and order selected laboratory and imaging tests. The medical interview was conducted by selecting appropriate questions from a list of questions (n = 148) divided suitably into different categories (i.e., reason for contact, past medical history, family history, social situation and current problems). Upon completion of the consultation with the VP, the user was asked to provide a preliminary assessment including a brief summary of the patient’s history, social situation and current symptomatology, as well as a preliminary diagnosis and a proposed treatment plan. This was followed by a detailed, individualized and automated feedback module delivered by: (a) the VP herself giving the patient’s perspective on different aspects of the consultation, and (b) the Virtual Advisor (VA) (“the expert”), giving detailed delayed feedback regarding clinical aspects of the consultation and basic communication skills. The provided feedback was structured and formed based on pre-determined and programmed rules, depending on the learners’ actual performance during the virtual interaction and included short and relevant theoretical background. For example, if the learner did not sufficiently examine the patient’s symptoms regarding intrusive recollection of traumatic events, the VA commented:

“You didn’t obtain sufficient information about experiences of intrusive recollection of the patient´s traumatic event like for instance images, thoughts, or perceptions, recurrent distressing dreams of the event, acting or feeling as if the traumatic event were recurring. Even physiologic reactivity upon exposure to internal or external cues that symbolize or resemble an aspect of the traumatic event can be experienced.”

thumbnail
Fig 1. Screenshot of the Virtual Patient system illustrating the history-taking interview.

The user can interact with the patient by choosing relevant questions divided in searchable appropriate categories and sub-categories. For illustrative purposes only, used with written permission by the portrayed model.

https://doi.org/10.1371/journal.pone.0119754.g001

A comprehensive description of the RT-Sim system and the current VP case used, along with the participants’ acceptance, expectations and attitudes towards it have been presented in previous articles [2022].

Outcome Measures

Confidence addressing mental health problems after trauma: In order to investigate the participants’ confidence in providing medical care for traumatized refugee patients before and after the training session with the VP, we used an instrument developed by the Harvard Program for Refugee Trauma (HPRT) that examines confidence in different areas of health and illness, communication with the patient, effects of diet and exercise on health, as well as different aspects of conducting related research and teaching activities. This validated, self-administered instrument, which we refer to as the “HPRT Confidence Questionnaire,” consists of 64 six-point Likert items and takes approximately 30 minutes to complete. It has been extensively used by HPRT for evaluating the effects of training programs for local health providers in countries like Cambodia [30]. All items measure the physician’s confidence in performing certain medical procedures by asking: “How confident are you that you can…” and scoring from 1 (= not at all confident) to 6 (= extremely confident). The participants were hence asked to predict their performance on a future task (predictive self-assessment).

For this study we revised the HPRT confidence questionnaire by excluding 20 items that were considered to not be relevant to the objectives of this study, in other words, those that examined aspects of planning and conducting research and educational activities as well as those that concerned non-relevant patient populations (children and adolescents).

Procedure

All data was collected during 3 occasions (at each of the university hospitals), in quiet group rooms, where the participants had access to stationary or laptop computers and wired headphones. Upon signing the informed consent form, all participants were asked to complete a pre-test version of the HPRT confidence questionnaire. They were then given access to the RT-Sim system and were instructed to interact with the VP for up to 45 minutes and try to assess as they would do in a real-life clinical encounter. After interacting with the VP and receiving the provided feedback, the participants were asked to fill in the post-test version of the aforementioned questionnaire.

Results

All of the participants (n = 32) completed the study and returned completed pre- and post-test questionnaires. The items of the HPRT confidence questionnaire (n = 44) were classified into 4 clusters that were comprised of similar items depicting 4 different aspects of medical care for traumatized refugee patients. Cronbach’s α was calculated for each cluster as well as for the whole HPRT confidence questionnaire in order to examine the internal consistency for the questionnaire and each cluster distinctly. Cronbach’s α > 0.7 was regarded as evidence for good internal consistency [31].

Table 1 presents the four generated clusters with examples of items, number of items included, and the estimated Cronbach’s α. The calculated Cronbach’s α for the overall questionnaire was 0.97 (44 items). The four clusters exhibited high values of Cronbach’s α as well (min: 0.87; max: 0.92), providing evidence for high internal consistency [32].

thumbnail
Table 1. The four generated clusters of confidence with examples of items, number of included items (n) and Cronbach’s α.

https://doi.org/10.1371/journal.pone.0119754.t001

Table 2 demonstrates the confidence of the participants in different aspects of caring for traumatized patients, as self-reported before and after training with the VP and receiving the individualized feedback by the VP and the VA. The matched-pair t-test was used to estimate changes between pre- and post-test values of self-reported confidence. P-values ≤ 0.05 were considered as evidence of statistical significance. Effect sizes were calculated by using the equation: Cohen’s d = |M|/SD where M is the mean of differences (Δ), and SD is the standard deviation of differences. Effect size was interpreted according to the standard interpretation suggested by Cohen [21]: 0.8 = large (8/10 of a standard deviation unit); 0.5 = moderate (1/2 of a standard deviation); 0.2 = small (1/5 of a standard deviation).

thumbnail
Table 2. Changes in mean confidence scores between pre- and post-test, by cluster and gender.

https://doi.org/10.1371/journal.pone.0119754.t002

Pre-training confidence in all clusters was below 4 (“confident”), with male participants scoring higher than female (overall confidence in male participants = 4.13; female participants = 3.70), though not statistically significant (mean Δ = 0.43; CI: -0.10, 0.96; p = 0.11). Participants with working experience as residents in psychiatry over 2.5 years (n = 15) scored higher overall at baseline (mean: 4.03 ± 0.62) than those (n = 17) with less experience than 2.5 years (mean: 3.71 ± 0.81), a difference however, that was not statistically significant (mean Δ: -0.32; CI: -0.85, 0.20; p = 0.2).

There were statistically significant improvements in overall confidence after using the RT-Sim system (mean Δ: 0.34; p<0.0001; d: 0.89), as well as in all four clusters depicting more specific aspects of caring for traumatized patients. Effect size, measured by Cohen’s d, was overall large (0.89). Concerning 2 out of 4 clusters, the effect size was also large (> 0.8), whereas for the remaining two it was moderate (0.67) and small (0.44). The highest improvement was noticed in confidence in identifying and evaluating trauma-related diagnoses and disability (mean Δ: 0.47; p<0.0001; d: 1.00), in addition to confidence in treating and managing victims of torture (mean Δ: 0.46; p = 0.0006; d: 0.67). Whereas there was overall statistically significant improvement in all clusters, when analyzing by gender, it was found that confidence among male participants regarding (a) aspects of intercultural communication and (b) treatment of torture victims, was improved, but not to a statistically significant degree.

Discussion

A previous pilot study by our team examined a preliminary version of the VP system presented in this study in terms of user acceptance and educational potentials [21], whereas a recently published study [22] examined the new version of our VP system with respect to users’ expectations and attitudes toward it. In this current study we examined the actual impact of training with VPs on the resident psychiatrists’ confidence in providing medical care for traumatized refugee patients with psychiatric symptomatology. It is, to the best of our knowledge, the first published study to examine the impact of VP on confidence in providing psychiatric care for a patient group in a broader context, while one previous pilot study examined the impact of VPs on confidence within a very specific slice of a clinical encounter (completing an informed consent task) [33].

Our current results revealed a statistically significant improvement and large effect size in overall confidence, as well as statistically significant improvement in confidence in specific domains of medical care for traumatized refugee patients, as defined by the four clusters used: (a) To identify and evaluate trauma-related diagnoses and disability (large effect size), (b) to treat and manage patients with trauma-related diagnoses and disability (large effect size), (c) to treat and manage victims of torture with mental illness (moderate effect size) and (d) effectively use elements of intercultural communication in the clinical praxis (small effect size). The highest improvement in confidence was observed in the domains of identifying and evaluating trauma-related diagnoses and disability, as well as in treating and managing victims of torture. The overall questionnaire (HPRT confidence questionnaire) as well as the generated clusters demonstrated very high Cronbach’s alpha, which speaks for more than acceptable internal consistency and high reliability.

This version of VP system has been previously evaluated in terms of user’s acceptance, expectations and attitudes with very promising results. The provided delayed summative, automated and individualized feedback was previously reported as very important elements of the program and valuable sources of knowledge. An innovative feature of this system was the provided feedback by the VP herself, giving needful and appreciated insights on aspects of medical care as seen from the patient’s viewpoint. Moreover, the design of the VP system favors a dynamic interaction between the user and the VP, as the VP’s reactions and answers depend partly on the user’s choices, questions asked and accomplished trust building. This allows the user to actively train in medical history taking, physical examination, ordering laboratory tests and planning treatment for this vulnerable patient group [6]. An important limitation of the system is however the lack of the ability to formulate in written or oral form own questions during the history-taking interview, limiting the perceived realism, as demonstrated in previously published studies. [21, 22]

It must be stressed that this study demonstrated the self-reported improvement in the participants’ confidence in providing medical care for refugee-traumatized patients and not any actual change in clinical performance or competence. This should be considered as a limitation of our study, since there are currently indications that physicians have a limited ability to accurately self-assess their competence [34]. However, there are also several studies that demonstrated positive association between self-assessed competence and external observations which give support to the theory that for at least some aspects of care, self-belief in one’s ability to perform a medical task may approximate one’s competence [35]. Fernandez et al. [36] demonstrated a strong association between self-rated language and cultural competence and patient satisfaction using an established interpersonal proof of care instrument. Similarly, Robbins et al. [37] demonstrated an association between self-reported “sensitivity to emotional and psychological issues in patients,” and the actual diagnosis of these issues in practice as audited by a blinded reviewer. Further research in the field of the relationship between self-reported confidence and actual competence in clinical, interpersonal and cultural aspects of medical care is definitely required. Other limitations in this study include the possible introduction of selection bias due to the non-random inclusion of our participants.

Despite the above-mentioned limitations, we consider this study to deliver important insights about the possible positive outcomes of training with VPs in psychiatric residency programs and to bolster previous promising findings about the application of VPs in psychiatric education in general [1122, 38]. We strongly believe that with further development and fine-tuning, VPs can also act—as even previously suggested by a published study [27]- as a useful complementary tool in the process of assessing different aspects of psychiatry residents' clinical competence. The next step in our project is to examine actual cognitive outcomes pre- and post- training with our VP-system. Future studies will include a randomized control study examining actual patient care outcomes.

Author Contributions

Conceived and designed the experiments: IP UF SE. Performed the experiments: IP SE. Analyzed the data: IP UF SE. Contributed reagents/materials/analysis tools: IP UF SE. Wrote the paper: IP UF SE.

References

  1. 1. Fleetwood J, Vaught W, Feldman D, Gracely E, Kassutto Z, Novack D (2000) MedEthEx online: a computer-based learning program in medical ethics and communication skills. Teach Learn Med 12: 96–104. pmid:11228685
  2. 2. Stevens A, Hernandex J, Johnsen K, Dickerson R, Raij A, Harrison C, et al. (2006) The use of virtual patients to teach medical students history taking and communication skills. Am J Surg 191(6): 806–11. pmid:16720154
  3. 3. McGee JB, Neill J, Goldman L, Casey E (1997) Using multimedia virtual patients to enhance the clinical curriculum for medical students. Studies in health technology and informatics 52: 732–735.
  4. 4. Srinivasan M, Hwang JC, West D, Yellowlees PM (2006) Assessment of clinical skills using simulator technologies. Acad Psychiatry 30: 505–515. pmid:17139022
  5. 5. Ellaway R, Poulton T, Fors U, McGee JB, Albright S (2008) Building a virtual patient commons. Med Teach 30: 170–4. pmid:18464142
  6. 6. Saleh N (2010) The value of virtual patients in medical education. Annals of Behavioral Science and Medical Education 16: 29–31.
  7. 7. Triola M, Feldman H, Kalet AL, Zabar S, Kachur EK, Gillespie C, et al. (2006) A randomized trial of teaching clinical skills using virtual and live standardized patients. J Gen Intern Med 21(5): 424–9. pmid:16704382
  8. 8. Raij AB, Johnsen K, Dickerson RF, Lok , Cohen MS, Duerson M, et al. (2007) Comparing interpersonal interactions with a virtual human to those with a real human. Visualization and Computer Graphics. IEEE Transactions on 13(3): 443–457. pmid:17356212
  9. 9. Swartout WR, Gratch J, Hill RW, Hovy E, Marsella S, Rickel J, et al. (2006) Toward virtual humans. AI Magazine 27(2): 96.
  10. 10. Yellowlees P, Nafiz N. (2010) The psychiatrist-patient relationship of the future: anytime, anywhere? Harv Rev Psychiatry 18(2): 96–102. pmid:20235774
  11. 11. Bender E (2003) Army goes into battle against soldier suicides. Psychiatr News 38: 6–37.
  12. 12. Foster A, Chaudhary N, Murphy J, Lok B, Waller J, Buckley PF (2014) The Use of Simulation to Teach Suicide Risk Assessment to Health Profession Trainees—Rationale, Methodology, and a Proof of Concept Demonstration with a Virtual Patient. Academic psychiatry: 1–10. pmid:25416611
  13. 13. Kenny P, Parsons T, Gratch J, Leuski A, Rizzo AA (2007) Virtual patients for clinical therapist skills-training. Lecture Notes In Computer Science 4722: 197.
  14. 14. Parsons TD, Kenny P, Ntuen CA, Pataki CS, Pato MT, Rizzo AA, et al. (2008) Objective structured clinical interview training using a virtual human patient. Stud Health Technol Inform 132: 357–62. pmid:18391321
  15. 15. Kenny P, Parsons T, Gratch J, Rizzo AA (2008) Evaluation of Justina: a virtual patient with PTSD. Lecture Notes In Computer Science 5208: 394–408.
  16. 16. Lin CC, Wu WC, Liaw HT, Liu CC (2012) Effectiveness of a virtual patient program in a psychiatry clerkship. Medical education 46(11): 1111–1112.
  17. 17. Williams K, Wryobeck J, Edinger W, McGrady A, Fors U, Zary N (2011) Assessment of competencies by use of virtual patient technology. Academic Psychiatry 35(5): 328–330. pmid:22007093
  18. 18. Shah H, Rossen B, Lok B, Londino D, Lind SD, Foster A (2012) Interactive virtual-patient scenarios: an evolving tool in psychiatric education. Acad Psychiatry 36(2): 146–50. pmid:22532209
  19. 19. Gorrindo T, Goldfarb E, Chevalier L, Hoeppner BB, Birnbaum RJ, Meller B, et al. (2013) Interprofessional differences in disposition decisions: results from a standardized web-based patient assessment. Psychiatric services 64(8): 808–811. pmid:23903607
  20. 20. Ekblad S, Mollica RF, Fors U, Pantziaras I, Lavelle J (2013) Educational potential of a virtual patient system for caring for traumatized patients in primary care. BMC Medical Education 13: 110. pmid:23957962
  21. 21. Pantziaras I, Courteille O, Mollica R, Fors U, Ekblad S (2012) A pilot study of user acceptance and educational potentials of virtual patients in transcultural psychiatry. International Journal of Medical Education 3: 132–140.
  22. 22. Pantziaras I, Fors U, Ekblad S (2014) Virtual Mrs K: The learners’ expectations and attitudes towards a virtual patient system in transcultural psychiatry. J Contemp Med Edu 2(2):109–115.
  23. 23. Ekblad S, Kastrup M (2013) Current research in Transcultural Psychiatry in the Nordic countries. 2nd special issue of Transcultural Psychiatry 50(6): 841–857.
  24. 24. Sledjeski EM, Speisman B, Dierker L (2008) Does number of lifetime traumas explain the relationship between PTSD and chronic medical conditions? Answers from the National Comorbidity Survey-Replication (NCS-R). J Behav Med 31:341–349. pmid:18553129
  25. 25. Mollica RF, McInnes K, Pham T, Smith Fawzi MC, Murphy E, Lin L (1998) The dose-effect relationships between torture and psychiatric symptoms in Vietnamese ex-political detainees and a comparison group. The Journal of Nervous and Mental Disease 186(9): 543–553. pmid:9741560
  26. 26. Coughlin S (2012) Post-Traumatic stress disorder and cardiovascular disease. In: Coughlin S, editor. Post-traumatic stress disorder and chronic health conditions. Washington DC: American Public Health Association. pp. 165–186.
  27. 27. Rasmusson AM, Schnurr PP, Zukowska Z, Scioli E, Forman DE (2010) Adaptation to extreme stress: post-traumatic stress disorder, neuropeptide Y and metabolic syndrome. Exp Biol Med 235: 1150–1162. pmid:20881319
  28. 28. Hollander A-C, Bruce D, Ekberg J, Burström B, Ekblad S (2013) Hospitalisation for depressive disorder following unemployment—differentials by gender and immigrant status: a population-based cohort study in Sweden. J Epidemiol Community Health 67: 875–881. pmid:23868528
  29. 29. The Swedish Medical Association (2006) The study director as a central participant in doctors’ residence programs. Available: http://www.slf.se/upload/Lakarforbundet/studierektorn.pdf (in Swedish). Accessed 6 December 2014.
  30. 30. Henderson DC, Mollica RF, Tor S, Lavelle J, Culhane MA, Hayden D (2005) Building primary care practitioners' attitudes and confidence in mental health skills in a post-conflict society: a Cambodian example. The Journal of nervous and mental disease 193(8): 551–559. pmid:16082300
  31. 31. Santos JRA (1999) Cronbach’s alpha: A tool for assessing the reliability of scales. Journal of extension 37(2): 1–5.
  32. 32. Cohen J (1998) Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale, NJ: Erlbaum.
  33. 33. Gorrindo T, Baer L, Sanders KM, Birnbaum RJ, Fromson JA, Sutton-Skinner KM, et al. (2011) Web-based simulation in psychiatry residency training: a pilot study. Academic Psychiatry 35(4): 232–237. pmid:21804041
  34. 34. Davis DA, Mazmanian PE, Fordis M, Van Harrison RTKE, Thorpe KE, Perrier L (2006) Accuracy of physician self-assessment compared with observed measures of competence. JAMA 296(9): 1094–1102. pmid:16954489
  35. 35. Smith RC, Lyles JS, Mettler J, Stoffelmayr BE, Van Egeren LF, Marshall AA, et al. (1998) The effectiveness of intensive training for residents in interviewing: A randomized, controlled study. Ann Intern Med 128: 118–126. pmid:9441572
  36. 36. Fernandez A, Schillinger D, Grumbach K, Rosenthal A, Stewart AL, Wang F et al. (2004) Physician language ability and cultural competence: an exploratory study of communication with Spanish-speaking patients. J Gen Intern Med 19: 167–174. pmid:15009796
  37. 37. Robbins JM, Kirmayer LJ, Cathebras P, Yaffe MJ, Dworkind M (1994) Physician characteristics and the recognition of depression and anxiety in primary care. Med Care 32: 795–812. pmid:8057696
  38. 38. Fors U, Muntean V, Botezatu M, Zary N (2009) Cross-cultural use and development of virtual patients. Med Teach 31(8): 732–738. pmid:19811210