https://orcid.org/0000-0001-5574-8559
Figures
Abstract
Background
Objective structured clinical examinations (OSCEs) are extensively used in many medical schools worldwide with the stated objective to assess students’ clinical skills acquired during internships. The objective of the present study was to assess the factors associated with success in university summative OSCEs, especially the impact of previous hospital internships in corresponding disciplines and supervision during internships.
Methods
This was a cross-sectional study assessing the results in the summative OSCEs of 4th year medical students during the 2021–2022 academic year in a French medical school. The summative OSCEs included five stations for each student. Each student answered a survey at the end of summative OSCEs about previous internships, the supervision they had and perceived difficulty levels for each station. The scores in each station were assessed according to previous hospital internships in the corresponding discipline. Analysis of predictive factors of success in OSCEs, defined by a score ≥ 10/20 at each station, were performed using univariate and multivariate logistic regression.
Results
Out of the 220 students participating in the summative OSCEs, 182 (83%) answered the survey. One hundred and forty-four (79%) of these students had carried out hospital internships in at least one of the disciplines evaluated during the OSCEs. Students having completed an internship in the corresponding discipline had significantly higher OSCEs scores for interrogation, communication, therapeutic education and procedure stations compared to those who had not. Previous internship in corresponding disciplines was independently associated with success in OSCEs in interrogation (OR 9.45 [1.34–66.8] p = 0.02), clinical examination (OR 6.93 [1.88–25.57] p = 0.004, and therapeutic education (OR 3.09 [1.22–7.82] p = 0.02) stations.
Citation: Carsuzaa F, Larid G, Martin M, Coudroy R, Vallée M, Paccalin M, et al. (2024) Impact of hospital internships on success in university summative objective structured clinical examinations: Large-scale experience in a French medical school. PLoS ONE 19(6): e0302427. https://doi.org/10.1371/journal.pone.0302427
Editor: Rano Mal Piryani, Bilawal Medical College, Liaquat University of Medical and Health Sciences, PAKISTAN
Received: February 1, 2024; Accepted: April 4, 2024; Published: June 13, 2024
Copyright: © 2024 Carsuzaa et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
To date, objective structured clinical examinations (OSCEs) are extensively used in many medical schools worldwide and are being progressively developed in France [1, 2]. Up until recently, assessments of medical students in France were based only on a written exam with multiple choice questions, which do not assess students’ clinical skills or prepare them for clinical practice [3]. To address this issue, a reform of the 2nd cycle of medical studies in France in 2021 introduced OSCEs in the assessment of French medical students, with the first nationwide OSCEs planned for 2024 [4]. The results of the national summative OSCEs will be part of the results used to classify students at the end of their 6th year and consequently enable them to choose their specialty. It will increase the discriminatory capacity of current evaluation modalities in French medical schools [5] insofar as it has been shown to be an efficient tool to assess the clinical performance of medical students [6]. The stated objective of the reform is to assess students’ clinical skills acquired during hospital internships in view of enhancing their investment in the latter and of better preparing them for their future clinical practice. Indeed, from the 4th to the 6th year of medical studies in France, medical students spend 50% of their time on placements in hospital departments, generally for periods of 3 months, where they provide clinical follow-up of patients, work in teams with senior doctors and gain enhanced clinical experience in the discipline concerned. Training and supervision during these internships are not standardized, and their impact on success in university OSCEs has not been studied in France or in the international literature. Identification of factors associated with success to the OSCEs is aimed at developing pedagogical avenues to improve student preparation.
The objective of this study was to assess the factors associated with success in summative university OSCEs, and more particularly the impact of previous hospital internships in corresponding disciplines.
Methods
This was a cross-sectional study aimed at assessing the relationship between the results of the summative OSCEs of 4th year medical students (2021–2022 academic year) in the French medical school of Poitiers university and the results of a survey on OSCEs and previous hospital internships collected at the end of these OSCEs, and the scores obtained for the final faculty tests throughout the same academic year.
Organization of the summative OSCEs
The summative OSCEs took place on May, 31th 2022 for 220 4th year medical students and included a route of five stations for each, randomly assessing disciplines they studied during the academic year. The OSCE disciplines and subjects differed between the morning and the afternoon, evaluating similar skills, the objective being to avoid communication of the subjects between students, with balanced overall difficulty between the two half-days (Fig 1).
Simulated patients, trained beforehand so as to ensure evaluation homogeneity, were present at the interrogation, clinical examination, and therapeutic education stations. Each station lasted 7 minutes, during which the students read and followed the instructions. A scoring checklist validated by the educational committee was completed by two evaluators present in the room, with a final OSCE grade of 20. The two evaluators were asked to carry out their evaluation each separately. At the end of the OSCE station, they were asked to use the 2-minute transition time to reach a consensus by reviewing the differences between each evaluation grid. The consensus score was entered into the computer system. The score was normalized according to the number of items evaluated. OSCE grades were collected after each student’s OSCE.
Survey on OSCEs and previous internships
At the end of the OSCEs, students were asked to answer a survey collecting the following information: places of hospital internships carried out during the 3rd and 4th years of medical studies and whether they had made formative or summative OSCEs for each internship, whether they had performed clinical examinations supervised by a senior, and whether they had participated in consultations in an out-patient clinic. Perceived difficulty level for each station using the Likert scale (0–5) was also collected.
Ethics
This study was carried-out in accordance with French regulations. The approval of an ethics committee or the consent of the participants was not required as this study was not involving the Human person according to the French Public Health Code (Loi Jardé—n°2012–300 of March the 5th 2012, in application in November 2016—Article R1121-1: research conducted exclusively from the processing of personal data is outside the scope of the researches involving human subjects (RIPH)). The survey was validated by the data protection delegation of the University of Poitiers (n°2022.75). All the data were self-reported by the participants with their written agreement on the questionnaire. Students were free to respond and individuals who did not participate had the same learning opportunities, and their exams assessed in the same way as those who did not participate. The assessment was independent of grading and pseudonymised with the student number. Confidential information was provided as defined by the European General Data Protection Regulation.
Statistical analysis
Categorical variables were expressed as frequencies and percentages and continuous variables as means and standard deviations. Categorical variables were compared using the chi-square test and continuous variables using the Student’s t-test. Pearson correlation matrix was used to evaluate the relationship between variables. Success at a OSCEs was defined by a score ≥10/20, the minimum score required for validation of summative OSCEs. OSCEs results were analyzed according to the discipline, or the clinical competency assessed. Analysis of predictive factors of success in OSCEs was performed through univariate and multivariate logistic regression with a backward selection of included variables employing a threshold of p<0.20. Included variables were previous internship in the discipline (in 3rd or 4th medical year), activities in the internships (supervised clinical examination participation in consultations, OSCEs), perceived difficulty of the station, final faculty test results in the discipline, and gender. The significance threshold was set at p<0.05. All statistical analyses were performed using GraphPad Prism software (v. 9, GraphPad Software, Boston, USA) and MedCalc (v.20.013; MedCalc Software, Ostend, Belgium).
Results
Population characteristics and previous internships
Out of the 220 students, 182 (83%) answered the survey and were included in the study, of whom 112 (61.5%) were women. Thirty-eight (20.9%) had not carried out internships in the disciplines evaluated during their OSCEs, while 71 (39.0%) had carried out an internship in one of the disciplines evaluated, 52 (28.6%) in 2 different disciplines and 20 (11.0%) in 3 disciplines. Medical activities during hospital internships are reported in Table 1. Among the 182 students, whatever the discipline, 87 (47.8%) had previously performed ≥1 formative OSCE, 145 (79.7%) ≥1 summative OSCE, 112 (61.5%) supervised clinical examinations, while 144 (84.6%) had participated in consultations during their internships.
Performances during the OSCEs
The overall average OSCEs score was 11.3 ± 2.0 /20. Procedure and therapeutic education OSCEs were respectively the most (14.6/20 ± 3.2) and the least (8.6/20 ± 3.3) successful. The scores obtained in each station are detailed in Table 2.
Impact on OSCEs scores of prior internships in a given discipline
The scores obtained in the rheumatology (p = 0.0002), pneumology (p = 0.0294), geriatrics (p = 0.0175), endocrinology (p = 0.0004), psychiatry (p = 0.004) and cardiology (0.0011) were higher if the student had completed an internship in the discipline (Fig 2).
OSCEs: objective structured clinical examinations. **p<0.01, ***p<0.001, ****p<0.0001.
When OSCEs stations were grouped by evaluated competency, students having completed an internship in the corresponding discipline had significantly higher respective OSCEs scores compared to those who had not done so, in the interrogation (p = 0.0003), communication (p = 0.0004), therapeutic education (p<0.0001), and procedures stations (p = 0.0011) (S1 Fig).
Moreover, the levels of difficulty perceived in the rheumatology (p = 0.0120), pneumology (p<0.0001), neurology (p = 0.0023), geriatrics (p<0.0001), endocrinology (p<0.0001), psychiatry (p<0.0001) and orthopedics (p = 0.0022) units were significantly lower when the student had completed an internship in the discipline (Fig 3).
**p<0.01, ***p<0.001, ****p<0.0001.
Similarly, when OSCEs station were grouped by evaluated competency, the difficulty levels perceived by students having completed an internship in the corresponding discipline were lower compared to those who had not done so, in the interrogation (p = 0.0004), communication (p<0.0001), and therapeutic education stations (p<0.0001) (S2 Fig).
There was an inverse correlation between the difficulty level perceived by the students and the scores obtained in the interrogation (p = 0.0307), communication (p = 0.0025), therapeutic education (p<0.0001) and procedure (p = 0.0063) domains. Conversely, students having performed previous formative or summative OSCEs during internships, whatever the internship, did not have significantly different overall OSCEs scores compared to those who had not done so (11.1 ± 1.9 vs. 11.5 ± 2.1, p = 0.16 and 11.4 ± 2.1 vs. 11.1 ± 1.67, p = 0.47 respectively). The same results were observed for each clinical domain.
Performance in the OSCEs and faculty test results
There was a positive and strong correlation between overall OSCEs scores and overall faculty test results throughout the same academic year for each student (r = 0.7639, p<0.0001). Moreover, there was a positive correlation between the faculty test results in the discipline and the corresponding OSCEs score for orthopedics (communication) (r = 0.3772, p = 0.009), endocrinology (therapeutic education) (r = 0.3655, p = 0.0004), cardiology (procedures) (r = 0.3448, p = 0.0012), geriatrics (therapeutic education) (r = 0.3197, p = 0.0032), neurology (clinical examination) (r = 0.2659, p = 0.0108), rheumatology (clinical examination and interrogation) (r = 0.2486, p = 0.0009) and psychiatry (interrogation) (r = 0.2167, p = 0.0464).
Predictive factors of OSCE success
Univariate analysis of predictive factors of success in OSCEs (score ≥ 10/20) for each station and each discipline are reported in S1 and S2 Tables. In multivariate analysis, success in OSCEs was associated with previous internship in the corresponding discipline for the interrogation, clinical examination and therapeutic education stations (OR = 9.45 [1.34–66.80], p = 0.0244 OR = 6.93 [1.88–25.57], p = 0.0036 and OR = 3.09 [1.22–7.82], p = 0.0169 respectively) (Table 3). The predictive factors of success to OSCEs stations for each discipline are reported in Table 4.
Discussion
OSCEs have become an established clinical assessment tool in many countries worldwide and have been developed more recently in French medical studies. Our study is one of the first in Europe and worldwide to study the impact of hospital internships for success in summative university OSCEs. It shows that OSCEs grades are better and the level of perceived difficulties lower when students have carried out an internship in the corresponding discipline. Previous internships were an independent factor of success in OSCEs in interrogation, clinical examination, and therapeutic education stations.
Positive impact of prior internships on OSCEs scores
The relationship between better results in OSCEs and lower perceived difficulty in the station if previous internship was done in the discipline has previously been described for psychiatry with higher OSCEs scores in the event of a specialized internship [7]. Indeed, the mental status examination, phenomenology, and differential diagnosis are probably more easily apprehended by the student and better mastered if they have been exposed to these diagnoses during internships rather than by learning during lessons or in textbooks [7]. In our study, this observation was corroborated for all the clinical domains evaluated, except for OSCEs scores in clinical examination stations. The two disciplines evaluated in the latter were neurology and rheumatology, and the correlation was significant in the latter, but not in the former. Neurology is taught at an earlier stage of studies and is favored by students as an important [8] but difficult discipline [9]. In France, medical students carry out several full-time 3-month hospital internships in a given discipline, during which they receive clinical training and carry out, independently and under direct supervision, clinical examinations every day. Daily training with actual patients concretely prepares students for clinical practice, and their learning and practice of fundamental skills can help to explain its being a major success factor in completed specialized internships. This finding is supported by a study which shows that OSCEs efficiently assess the clinical performance of medical students [6].
Lack of impact of formative and summative OSCEs sessions during internships on OSCEs score
Surprisingly, OSCEs success was not associated with completion of the formative and summative OSCEs sessions organized during the various internships, even though OSCEs have been systematized in most hospital internships. This may be explained by differences and lack of homogeneity in the conduct of OSCEs during internships, which are quite new in France, which means that the relevant training of teachers as well as students has remained heterogeneous, and consequently does not yet conclusively determine success in summative university OSCEs.
Positive correlation between OSCEs scores and faculty test results
In a study assessing the predictive factors for OSCEs success among 5th-year medical students, female gender, absence of health problems during the internship, high number of internships in a medical specialty (≥6) and low number of internships in a surgical specialty (<3) were the most significantly factors associated with higher OSCEs success. In this study, success in theoretical faculty exams had no effect [10]. By contrast, in our cohort one of the factors of success in OSCEs was success in final faculty exams. On this subject, a correlation between faculty test results and OSCEs performance was clearly demonstrated in the musculoskeletal domain in a study by the Mexican Board of Rheumatology on postgraduate certification procedures [11]. However, this correlation is not perfect, especially insofar as theoretical faculty exams and OSCEs do not assess the same skills. Another recent study showed a weak correlation between OSCEs grades and faculty exams among 6th year medical students, but these latter obtained lower scores in OSCEs exams compared to other standard evaluation modalities, leading the authors to suggest that the OSCEs are not redundant with the other evaluation modalities [12].
Better results in procedural skills
The scores obtained in the clinical examination and therapeutic education stations were lower than those obtained in the procedure stations. Therapeutic education requires communication skills, which have been shown to be correlated with OSCEs performance evaluating clinical examination, problem solution prescription, and medical records [13]. On one hand, performance of some procedures (the electrocardiogram…) are part of the daily lives of medical students. On the other hand, communication with the patient, therapeutic education, and performance of specialized clinical examinations, one example being examination of the knee in rheumatology, are skills acquired by only a minority of students. This is in line with a study showing that among the different skills evaluated by OSCEs, medical students are less effective at clinical reasoning skills, while better results are found in procedural skills [14]. It is therefore appropriate to increase this form of skill acquisition in view of upgrading OSCEs results. The particularity of the musculoskeletal examination was previously described in the literature. In this domain, the addition of an interactive workshop with a focus on clinical examination of the knee in the rheumatology rotation of medicine residents led to significantly improved OSCEs performances [15].
Homogenization of internships during medical training courses and relevant activities in the different disciplines help to improve skills acquired by medical students and are likely to improve grades obtained during the OSCEs. Having completed a specialized internship improves success, and the benefit of this continuous training has been underlined in the literature and should be encouraged [16–18].
Limitations
Our study has several limitations. First, it was conducted at a single institution. However, there was a high response rate (83.1%), which strengthened the results. Moreover, a survey is associated with potential information biases and a certain number of missing data, and result should be considered with caution. In these OSCEs, we used a checklist to determine the student’s grade, but there were no global evaluation scales, which would have improved assessment according to previous internships, as previously shown [19]. In addition, for each station, the two evaluators made a consensus for the final score, and we had no individual scores to assess reliability and generalizability.
Conclusion
One of the main factors associated with OSCEs success was participation to previous hospital internships in the discipline evaluated by the OSCEs. Homogenization and increased supervised medical activities carried out during the internship will render them even more educational.
Supporting information
S1 Fig. OSCE sub-score by evaluated competency according to completion of an internship in the corresponding discipline. **p<0.01, ***p<0.001, ****p<0.0001.
https://doi.org/10.1371/journal.pone.0302427.s001
(JPG)
S2 Fig. Difficulty level (Likert scale) by evaluated competency according to completion of an internship in the corresponding discipline.
**p<0.01, ***p<0.001, ****p<0.0001.
https://doi.org/10.1371/journal.pone.0302427.s002
(JPG)
S1 Table. Predictive factors of success to OSCE stations (univariate analysis).
https://doi.org/10.1371/journal.pone.0302427.s003
(DOCX)
S2 Table. Predictive factors of success to OSCEs for each discipline (univariate analysis).
https://doi.org/10.1371/journal.pone.0302427.s004
(DOCX)
Acknowledgments
The authors acknowledge the OSCEs group of the University of Poitiers: Géraldine Allain, François Arrivé, Aurélien Binet, Vanessa Bironneau, Jérôme Danion, Romain David, Jean du Breuillac, Paul Dufay, Magali Garcia, Elisabeth Gervais, Jérémy Guenezan, Vincent Jedat, Laurent Macchi, Marie Masson-Regnault, Karine Salaun, Fabrice Schneider, Jean-Philippe Neau.
References
- 1. Agarwal A, Batra B, Sood A, Ramakantan R, Bhargava SK, Chidambaranathan N, et al. Objective structured clinical examination in radiology. Indian J Radiol Imaging. 2010;20(2):83‑8. pmid:20607015
- 2. Maker VK, Bonne S. Novel hybrid objective structured assessment of technical skills/objective structured clinical examinations in comprehensive perioperative breast care: a three-year analysis of outcomes. J Surg Educ. 2009;66(6):344‑51. pmid:20142133
- 3. Epstein RM. Assessment in medical education. N Engl J Med. 2007;356(4):387‑96. pmid:17251535
- 4.
Arrêté du 2 septembre 2020 portant modification de diverses dispositions relatives au régime des études en vue du premier et du deuxième cycle des études médicales et à l’organisation des épreuves classantes nationales—Légifrance [Internet]. [cité 25 août 2021]. Disponible sur: https://www.legifrance.gouv.fr/jorf/id/JORFTEXT000042320018
- 5. Matet A, Fournel L, Gaillard F, Amar L, Arlet JB, Baron S, et al. Impact of integrating objective structured clinical examination into academic student assessment: Large-scale experience in a French medical school. PLoS One. 2021;16(1):e0245439. pmid:33444375
- 6. Terry R, Hing W, Orr R, Milne N. Do coursework summative assessments predict clinical performance? A systematic review. BMC Med Educ. 16 févr 2017;17(1):40. pmid:28209159
- 7. Goisman RM, Levin RM, Krupat E, Pelletier SR, Alpert JE. Psychiatric OSCE performance of students with and without a previous core psychiatry clerkship. Acad Psychiatry. 2010;34(2):141‑4. pmid:20224028
- 8. Pokryszko-Dragan A, Mottershead J, Aitken G. Attitudes towards neurology among medical undergraduates. Neurol Neurochir Pol. 2018;VM/OJS/J/61584.
- 9. Pakpoor J, Handel AE, Disanto G, Davenport RJ, Giovannoni G, Ramagopalan SV. National survey of UK medical students on the perception of neurology. BMC Medical Education. 2014;14(1):225. pmid:25335431
- 10. Leclercq M, Vannier M, Benhamou Y, Liard A, Gilard V, Auquit-Auckbur I, et al. [Identification of factors associated with performance in objective structured clinical examination in Rouen medical school]. Rev Med Interne. 2022;43(5):278‑85.
- 11. Pascual Ramos V, Medrano Ramírez G, Solís Vallejo E, Bernard Medina AG, Flores Alvarado DE, Portela Hernández M, et al. Desempeño del examen clínico objetivo estructurado como instrumento de evaluación en la certificación nacional como reumatólogo. Reumatología Clínica. 2015;11(4):215‑20.
- 12. Lebdai S, Bouvard B, Martin L, Annweiler C, Lerolle N, Rineau E. Objective structured clinical examination versus traditional written examinations: a prospective observational study. BMC Med Educ. 2023;23(1):69. pmid:36707797
- 13. Pascual-Ramos V, Flores-Alvarado DE, Portela-Hernández M, Maldonado-Velázquez M del R, Amezcua-Guerra LM, López-Zepeda J, et al. Communication Skills in Candidates for Accreditation in Rheumatology Are Correlated With Candidate’s Performance in the Objective Structured Clinical Examination. Reumatología Clínica (English Edition). 2019;15(2):97‑101.
- 14. Sim JH, Aziz YFA, Mansor A, Vijayananthan A, Foong CC, Vadivelu J. Students’ performance in the different clinical skills assessed in OSCE: what does it reveal? Medical Education Online. 2015;20(1):26185. pmid:25697602
- 15. Kwiatkowski A, Shakoor N, Manadan A, Block JA, Khandelwal S. Incorporating interactive workshops into bedside teaching: completion of a multi-modal rheumatology rotation significantly increases internal medicine residents’ competency and comfort with comprehensive knee examinations. BMC Med Educ. 2022;22(1):355. pmid:35538536
- 16. Khan KZ, Ramachandran S, Gaunt K, Pushkar P. The Objective Structured Clinical Examination (OSCE): AMEE Guide No. 81. Part I: an historical and theoretical perspective. Med Teach. 2013;35(9):e1437–1446. pmid:23968323
- 17. Zayyan M. Objective structured clinical examination: the assessment of choice. Oman Med J. 2011;26(4):219‑22. pmid:22043423
- 18. Sloan DA, Donnelly MB, Schwartz RW, Strodel WE. The Objective Structured Clinical Examination. The new gold standard for evaluating postgraduate clinical performance. Ann Surg. 1995;222(6):735‑42. pmid:8526580
- 19. Hodges B, McIlroy JH. Analytic global OSCE ratings are sensitive to level of training. Med Educ. 2003;37(11):1012‑6. pmid:14629415