https://orcid.org/0000-0001-9954-1162
Figures
Abstract
Background
With the emergence of the COVID-19 pandemic and lockdown approach that was adopted all over the world, conducting assessments while maintaining integrity became a big challenge. This article aims at sharing the experience of conducting an online assessment with the academic community and to assess its effectiveness from both examiners’ and students’ perspectives.
Methods
An online assessment was carried out for the final year medical students of Hawler Medical University/Iraq during the lockdown period of the COVID-19 pandemic, June 2020. Then, an online questionnaire was sent to a sample of 61 examiners and 108 students who have been involved in evaluating the mentioned assessment process. Mann-Whitney and Kruskal-Wallis tests were used to compare the mean ranks of the overall satisfaction scores between categories of the students and examiners. Categorical data were summarized and presented as frequencies and percentages.
Results
The response rates among examiners and students were 69.4% and 88.5% respectively. The majority of the examiners were generally satisfied with the online examination process compared to only around a third of the students. However, both examiners and students agreed that online examination was not suitable for assessing the physical examination skills.
Citation: Alkhateeb NE, Ahmed BS, Al-Tawil NG, Al-Dabbagh AA (2022) Students and examiners perception on virtual medical graduation exam during the COVID-19 quarantine period: A cross-sectional study. PLoS ONE 17(8): e0272927. https://doi.org/10.1371/journal.pone.0272927
Editor: Vijayalakshmi Kakulapati, Sreenidhi Institute of Science and Technology, INDIA
Received: March 18, 2022; Accepted: July 29, 2022; Published: August 19, 2022
Copyright: © 2022 Alkhateeb et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: The authors received no specific funding for this work.
Competing interests: The authors declared that they have no competing interests.
Introduction
The COVID-19 pandemic has affected every part of societies all around the world, it has caused the largest interruption of education systems in the history of mankind involving almost 1.6 billion learners in over 200 countries [1]. The standard provision of medical education for undergraduate medical students has to be changed during this global crisis, and, it is important to develop innovative approaches of assessment that adopt the standards of medical education and consider the current social and environmental constraints caused by the COVID-19 pandemic [2].
Assessment is the measurement of the students’ learning and the way they are assessed usually influence what and how they learn. In many countries, clinical and written examinations have been suspended, cancelled, or replaced by online examinations or new methods of assessment including; Comprehensive high-stake online exams [3], Modified national Objective Structured Clinical Examination (OSCE) [4], High stakes modified OSCE [5], Web-based OSCE [6], Online oral [7], and Online assessment [8].
College of Medicine at Hawler Medical University has considered it essential to set up a modified online examination not only to maintain the students’ academic progress but also to provide enough medical graduates. Therefore, this study aims at sharing the experience of implementing a comprehensive online examination to final year students at the college of medicine during this pandemic and to assess its effectiveness and satisfaction by examiners and students.
Materials and methods
Study design and setting
This was a cross-sectional study performed at the College of Medicine, Hawler Medical University, Kurdistan Region/Iraq during June 2020.
Study participants and sampling
The study population was 72 examiners and 150 6th year students who were involved in the online assessment experience. The sample size was estimated using the Epi info 7 computer program (free program issued by WHO and CDC) where the following information was entered into the program: the size of the population (mentioned above), the confidence interval was set at 95%, the estimated prevalence was set at 50% and the absolute precision was set at 5%. Accordingly, the estimated sample size was found to be 108 students and 61 examiners. Simple random sampling was used to select the sample out of the examiners and students’ populations. The aim of the study was clarified to the participating students and examiners, and their consent was taken.
Structure of the online assessment
Over the past years (before the COVID-19 pandemic), the final year medical students’ assessment was composed of a two-part assessment; advanced clinical competence which consisted of: Objective Structured Clinical Examination (OSCE) that assesses the skills and a written component in a form of Single best answer (SBA) and Extended matching questions (EMQ) which assess the knowledge.
It’s worth mentioning that the attitude of the students is assessed as part of daily evaluation through all the courses. The college has developed a comprehensive multiform examination which include all disciplines (Pediatric, Obstetrics & Gynecology, Medicine and Surgery). This examination represents 40% of the students’ final graduation average grade of the MBChB program and the remaining 60% is collected from students’ grades in years 1–5.
During the COVID-19 pandemic lockdown, final assessment with the physical presence of students was not possible, therefore the curriculum and assessment committees of the college made a modification for the assessment process with taking the examiners’ opinions into consideration.
Planning and preparation
A committee (including the Dean, head of clinical departments, head of the Medical Education department, and the 6th year director) was formed to decide on the forms and number of stations, competencies to be assessed, assure that the blueprint was properly sampling the essential part of the curriculum and responsible on students’ and examiners’ orientation on the final exam process during COVID-19 pandemic. A grading score checklist was prepared for all sets of questions.
The 6th year director prepared 18 examination panel teams. Each team was composed of four members from major clinical specialties. A focal point was allocated for each team.
The 6th year director conducted a Zoom meeting with the 18 focal points to describe the implementation process, and discuss challenges expected and possible solutions.
To decrease the possible internet problems, the exam panel teams were asked to attend the college and to meet the students in one Zoom account (focal point’s account).
Description of the online assessment
In usual situations, clinical competencies in history taking, examination, communication, and procedural skills besides data interpretation and management competencies were assessed through OSCE stations.
In the modified online assessment, all the above-mentioned competencies were assessed except for the examination and procedural skills competencies.
The oral exam was conducted as an online Zoom meeting between the student and the examination panel. In a random manner, eighteen groups of students were created and they were distributed to the online groups (8–9 students were put into the online zoom groups). As a result, first students in each of the 18 groups would be examined with the same questions, and so on (Fig 1).
All students who assigned #1 in the 18 groups will pass through the same set of questions, the same will be applied for students #2–8. Two extra set of questions were saved to overcome any problems.
The Blueprint was regulated according to the competencies for each round of student exams, and the questions included emergency patient scenarios in different disciplines and different systems.
For the security purpose of the online exam, 10 sets of different questions according to the blueprint competencies were prepared. The 9th and 10th set were prepared as backup sets in case any connection problem appeared during the exam. In case any student experienced connection problems, they would be shifted to the end with a new set of questions i.e. 9th and 10th in order not to affect the sequence of the exam in the other examination panels.
Implementation
To overcome technical competence deficiency in some examiners, each examination panel contained one faculty member who was proficient in the Zoom application and that member was responsible for admitting the students to the Zoom meeting, controlling the waiting list and sharing the screen that show the questions to the students. Student answers were graded according to a designed checklist.
The sequence of asking questions was also the same in each examination panel starting from Surgery case followed by Medicine, Obstetrics & Gynecology, and Pediatrics.
Each exam panel examined 8 or 9 students at the same time and with the same questions for the same rounds. The question set was changed with each student to control the security of the exam (Fig 2). The time of examination for each student was 20 minutes, 5 minutes for each branch.
(A) Sequence of discipline questions asked by examination panel, (B) question sets according to assigned students numbers in each examination panel i.e. 8 set of questions used by each examination panel for the 8 students.
Stakeholder evaluation of the process
Evaluation of the online assessment process from both students and examiners perspectives was done through a questionnaire that was designed by one of the authors through literature reviews. The questionnaire was prepared and distributed to both examiners and students through a Google form. It was composed of 9 and 8 items for both examiners and students respectively that were rated on a five-point Likert scale from 1 (strongly disagree) to 5 (strongly agree). The items were related to the participants’ experience and satisfaction with the setting, implementation, and environment of the online assessment. Three open-ended questions were added to the questionnaire to indicate, the most commonly liked and disliked aspects of the online examination, and suggestions to improve it. In addition to that, one more item was included in students’ questionnaire about whether they faced any connection problem during the online assessment process. An overall score for each individual was calculated by summing up their Likert scale scores of all items [4]. Internal consistency reliability of the responses was evaluated by Cronbach alpha (α).
Data analysis
The data obtained from the Google forms was shifted to another statistical software, the Statistical Package for Social Sciences (SPSS, version 25). Categorical data were summarized and presented as frequencies and percentages. Mann-Whitney and Kruskal-Wallis tests were used to compare the mean ranks of the overall satisfaction scores between categories of the students and examiners. Thematic analysis was performed for the above-mentioned open-ended questions. P-values <0.05 were regarded as significant.
Ethical consideration
Ethical approval was obtained from the ethics committee of Hawler Medical University. (the ethical approval number: 8 on 20/4/2021) A written informed consent about providing feedback on the online assessment was obtained through Google form. Students’ identity was not disclosed for ethical reasons. The confidentiality of the participants’ information was maintained.
Results
Data from 75 students and 54 examiners were available for analysis. The response rate for the electronic questionnaire was 69.4% for students and 88.5% for examiners.
The overall score for each individual was calculated, in addition to the Mean values and standard deviations. There were no significant differences in the mean ranks of the total scores between male and female students (p = 0.392), and between male and female examiners (p = 0.095), also there were no significant differences between the different specialties (p = 0.392) (Table 1).
The mean (SD) of the examiners’ overall satisfaction score was 3.76 (0.67) out of 5 which was significantly (p<0.001) higher than that of students (3.1 (0.85)).
The student online assessment survey had good internal reliability (Cronbach alpha = 0.883). It was clear that the vast majority of the student responses were in the "Agree" category apart from their response to the statement of "the assessment process evaluated the clinical skills of the students" where only 10.7% agreed on it. According to 53.3% of students, the time allocated for answering questions was enough. Half of the students agreed that the examiners were professional, and the assessment questions were clear. In general, 37.3% of the students were satisfied with the assessment process (Table 2).
It is worth mentioning that 33 (44%) students faced problems during the assessment process which was either electricity shutdown (21.2%), internet connection (45.5%) or both (33.3%). The majority of the students’ issues were solved immediately while only 9 out of the 33 students (27.3%) waited till the end of the exam and the extra set of questions were used for them.
Table 3 shows the main themes and subthemes that emerged from the students’ perception on Pros and Cons of the online assessment. The two main areas highlighted in the Pros were the question and format of the exam, and the professional behavior of the examiners. The students mostly liked the type of the questions, being practical and covering the common clinical and emergency cases reasonably (34.6%). On the other hand, the two main themes that emerged in the Cons were related to administration issues, and question and format of the assessment. The major areas that students disliked were as follows: long waiting time and delay (22.7%), unfair distribution of marks (22.7%), connection problems (20%). Other subthemes are presented in detail in Table 3.
Around one quarter (24%) of the students had no specific suggestion for improvement, 17.3% suggested another method for assessment (other than the online), 13.3% suggested better training for the students and examiners before the exam, and 13.3% of students suggested accurate timing, and to give more time. The other suggestions are mentioned in (Table 4).
Sixty-one examiners were asked to participate in the study, but only 54 (88.5%) participated. Around half (48.1%) were males. The mentioned teaching staff belonged to the following departments: medicine, surgery, pediatrics, obstetrics and gynecology. The examiners’ online assessment survey was found to have good internal reliability (Cronbach alpha = 0.881).
It is evident in (Table 5) that the majority of the examiners were generally satisfied with the online examination process, they agreed on all the items raised by the researchers except for ’assessment of the clinical competence’ where many of them thought that the online exam could not assess the clinical competence and did not reflect real clinical practice.
Organization of the exam and teamwork were the most liked areas by the examiners (38.9% and 27.8% respectively). Four examiners (7.4%) liked the fairness of the exam, and another 7.4% liked the type of questions, being clinical and covering the curriculum. On the other hand, the most disliked areas were: the inability of the exam to reflect the clinical skills of the students” (22.2%), connection problems (18.5%), and poor organization (14.8%). Details of examiners related themes and subthemes are presented in (Table 6).
More than one third (36.4%) of the suggestions were about better organization, and to have more questions. Nine (16.4%) suggestions were about the training of the examiners before the exam. Eight (14.5%) suggestions were to make an on-campus exam with implementing safety measures where possible. The other suggestions are presented in (Table 7).
Discussion
Online assessment is an unusual method of assessment for graduating medical students and proper decision and preparation with the involvement of administration and teachers is essential. The presented study represents one of the experiences of online assessment during COVID-19 lockdown.
The COVID-19 pandemic has affected medical education in many aspects. Creative ways of assessment were important to maintain the standards of medical education during the lockdown time [9], especially the assessment of clinical competencies which is a challenging area needing innovative modification [10].
As the lockdown was imposed by the government of Iraq and Kurdistan Region, to continue learning and save the academic year, the Ministry of Higher Education and Scientific Research in Iraq and Kurdistan Region decided in March 2020 to replace in-person classes with their online equivalents. These challenges were more distressing for final year medical students waiting for their graduation assessments [11].
At Hawler Medical University, College of Medicine, the college council adapted the distribution of assessments’ marks, assigning 80% of the total course mark to coursework related activities, while the remaining 20% was allocated to final assessments to ensure a fair ranking of students. A similar approach was performed in Jordan [12].
As the face-to-face final OSCE was prohibited at the start of the pandemic, OSCE blueprint, instructions and structured marking schedules were adapted. Physical examination or procedural skills couldn’t be assessed due to absence of simulated or real patients. Therefore, it was planned to have an objective structured viva examination instead. Questions were based on emergency oriented short case vignette from all major disciplines. Technological development and accessibility to variety of digital tools have supported the education process in COVID-19 era throughout the globe. Zoom, Google meet, Microsoft team, WebEx, …etc are examples. Overall, these tools enabled sharing clinical photographs, laboratory results and imaging using the screen-sharing feature, and facilitated the assessment of clinical reasoning and data interpretation domains via examiner questioning synchronously. Problem-based questions enabled the assessment of the clinical reasoning and higher-order thinking skills [13, 14]. Hawler Medical University used Zoom as a tool for the final year assessment as the staff and students had experience with it during the early COVID-19 period.
In several countries, clinical and theory examinations have been cancelled [9], postponed, or delayed or replaced by online or new methods of assessment [15, 16].
Results indicated that the overall satisfaction of both examiners and students with the conducted assessment was high with that of the examiners’ being higher. Similar results were obtained by Tan et al. [17]. On the other hand, concerns about the number of questions used, the mark allocated for the final assessment, technical problems and poor nonverbal communication on video-conferencing were mainly disliked by a certain fraction of students (30.7%) and might have contributed in their non-satisfaction. According to Mak and his colleagues, it is hard to gain control on the full range of nonverbal communication through the online platform due to screen size, position of the student on the screen as well as impaired video quality due to internet problems [18]. This was also found in our students, as a good percentage of them used mobile screens for joining. The online assessment process has many challenges during implementation including connection problems [19]; variations in household internet access for both examiners and students was an expected problem especially in a country like Iraq. To minimize this concern, two points were considered, asking examiners in the examination panel to conduct the examination as a committee in the campus benefiting from the fact that the examiners (who are also physicians) had permission from the local authorities for free movement in the lockdown period and only students needed to be at home. Secondly, if any connection problem occurred for any of the students, an extra set of questions were present to be used at the end of the exam, in order not to affect the sequence of the exam as explained in detail in methodology.
Exam security was another challenge, as it was necessary to ensure that students have met the required learning outcome. Threats to assessment security can be alleviated by designing assessment methods that are resistant to challenges of cheating [20].
To overcome this problem, 18 committees conducted the exam at the same time and changed the question for each round as clarified in (Fig 1), and a Zoom room was created for every committee with enabling the waiting feature.
Less chance for cheating was declared by many students as one of the points that students liked about the process of assessment, however, it led to longer waiting time for their turn to enter the Zoom room.
To use technology in a low-technology context, adapting new approaches to solve local problems should be restricted to what is possible under the circumstances [21]. Poor technical capability of all examiners to use IT especially Zoom, as this exam was conducted at the beginning of COVID-19 (2019–2020 academic year) and lacking IT personnel in the college put a higher pressure on the assessment committee and the administrative team to find a solution. This problem was alleviated by choosing 18 examiners who were good at using technology specifically Zoom to be focal points for each examining committee.
In medical education, it is important to prepare medical students for real-life scenarios by designing authentic assessments [15, 22] to ensure the students’ ability to deal with emergency case scenarios. Therefore, all case scenarios that were prepared reflected real emergency case scenarios. However, around one-third of examiners and nearly half of students disagreed that this online assessment assessed clinical skills.
The strength of this study is that it reports both examiners and students’ perception on the online assessment experience. The overall satisfaction score mean of examiners were significantly higher than that of students. This agreed with Elshami et al. where satisfaction rates among students and examiners were 41.3 and 74.3% respectively [23].
Limitations: This study examines one institution experience but, it is important to share the modification of assessment process done in the pandemic time with medical educators throughout the globe especially in low resource countries. Although it was not possible to practice a mock final online assessment by students due to uncertainty at the beginning of the pandemic, good preparation, planning and prediction of the expected problems with their solutions by focal points and effective communication with the students, had a major role in the success of the process.
In conclusion, the findings from this study suggest that assessment via alternative methods during the lockdown has enlightened examiners to the value of their use in normal situations.
There is therefore, a definite need for Information communication technology training including the effective use of distance educational tools for student-oriented teaching and assessment strategies for all faculty members. This may come as an obligatory online course in alignment with preparing new university examiners and equipping them with digital tools. An important area that needs to be addressed in future researches is to explore the perception of stakeholders on online assessments utilizing focus groups and qualitative methods.
Acknowledgments
The authors appreciate all medical students and examiners who spent their time to participate in the study.
References
- 1. Sani I, Hamza Y, Chedid Y, Amalendran J, Hamza N. Understanding the consequence of COVID-19 on undergraduate medical education: Medical students’ perspective. Ann Med Surg. 2020;58: 117–119. pmid:32983429
- 2. Huber SG, Helm C. COVID-19 and schooling: evaluation, assessment and accountability in times of crises—reacting quickly to explore key issues for policy, practice and research with the school barometer. Educ Assessment, Eval Account. 2020;32: 237–270. pmid:32837626
- 3. Khalaf K, El-Kishawi M, Moufti MA, Al Kawas S. Introducing a comprehensive high-stake online exam to final-year dental students during the COVID-19 pandemic and evaluation of its effectiveness. Med Educ Online. 2020;25(1). pmid:33000704
- 4. Hytönen H, Näpänkangas R, Karaharju-Suvanto T, Eväsoja T, Kallio A, Kokkari A, et al. Modification of national OSCE due to COVID-19-Implementation and students’ feedback. Eur J Dent Educ. 2021;00: 1–10. pmid:33369812
- 5. Boursicot K, Kemp S, Ong TH, Wijaya L, Goh SH, Freeman K, et al. Conducting a high-stakes OSCE in a COVID-19 environment. MedEdPublish. 2020;9:54.
- 6. Major S, Sawan L, Vognsen J, Jabre M. COVID-19 pandemic prompts the development of a Web-OSCE using Zoom teleconferencing to resume medical students’ clinical skills training at Weill Cornell Medicine-Qatar. BMJ Simulation and Technology Enhanced Learning. BMJ Publishing Group; 2020. pp. 376–377.
- 7. Akimov A, Malin M. When old becomes new: a case study of oral examination as an online assessment tool. Assess Eval High Educ. 2020;45: 1205–1221.
- 8. Elzainy A, El Sadik A, Al Abdulmonem W. Experience of e-learning and online assessment during the COVID-19 pandemic at the College of Medicine, Qassim University. J Taibah Univ Med Sci. 2020;15: 456–462. pmid:33106752
- 9. Papapanou M, Routsi E, Tsamakis K, Fotis L, Marinos G, Lidoriki I, et al. Medical education challenges and innovations during COVID-19 pandemic. Postgrad Med J. 2022;98: 321–327. pmid:33782202
- 10. Nimavat N, Singh S, Fichadiya N, Sharma P, Patel N, Kumar M, et al. Online Medical Education in India – Different Challenges and Probable Solutions in the Age of COVID-19. Adv Med Educ Pract. 2021;12: 237–243. pmid:33692645
- 11. Choi B, Jegatheeswaran L, Minocha A, Alhilani M, Nakhoul M, Mutengesa E. The impact of the COVID-19 pandemic on final year medical students in the United Kingdom: a national survey. BMC Med Educ. 2020;20: 206. pmid:32600460
- 12. Elsalem L, N A-A, AA J, Obeidat N. Remote E-exams during Covid-19 pandemic: A cross-sectional study of students’ preferences and academic dishonesty in faculties of medical sciences. Ann Med Surg. 2021;62: 326–333. pmid:33520225
- 13. Varutharaju E, Ratnavadivel N. Enhancing higher order thinking skills through clinical simulation. Malaysian J Learn Instr. 2014;11: 75–100.
- 14. Kansal AK, Gautam J, Chintalapudi N, Jain S, Battineni G. Google Trend Analysis and Paradigm Shift of Online Education Platforms during the COVID-19 Pandemic. Infect Dis Rep. 2021;13: 418–428. pmid:34065817
- 15. O’Byrne L, Gavin B, McNicholas F. Medical students and COVID-19: the need for pandemic preparedness. J Med Ethics. 2020;46: 623–626. pmid:32493713
- 16. Birch E, Wolf M de. A novel approach to medical school examinations during the COVID-19 pandemic. Med Educ Online. 2020;25. pmid:32594865
- 17. Tan CK, Chua WL, Vu CKF, Chang JPE. High-stakes examinations during the COVID-19 pandemic: To proceed or not to proceed, that is the question. Postgrad Med J. 2021;97: 427–431. pmid:33452144
- 18. Mak V, Krishnan S, Chuang S. Students’ and Examiners’ Experiences of Their First Virtual Pharmacy Objective Structured Clinical Examination (OSCE) in Australia during the COVID-19 Pandemic. Healthcare. 2022;10:328. pmid:35206942
- 19. Guangul FM, Suhail AH, Khalit MI, Khidhir BA. Challenges of remote assessment in higher education in the context of COVID-19: a case study of Middle East College. Educ Assessment, Eval Account. 2020;32: 519–535. pmid:33101539
- 20. Gamage KAA, de Silva EK, Gunawardhana N. Online delivery and assessment during COVID-19: Safeguarding academic integrity. Educ Sci. 2020;10(11):301.
- 21. Yapa HM, Bärnighausen T. Implementation science in resource-poor countries and communities. Implement Sci. 2018;13: 1–13.
- 22. Karakus A, Şenyer N. The preparedness level of final year medical students for an adequate medical approach to emergency cases: computer-based medical education in emergency medicine. 2014;7(1):1–6.
- 23. Elshami W, Taha MH, Abuzaid M, Saravanan C, Al Kawas S, Abdalla ME. Satisfaction with online learning in the new normal: perspective of students and faculty at medical and health sciences colleges. 2021;26:1920090. pmid:33974523