Introduction

The necessities brought about by the COVID-19 pandemic required an inevitable shift towards online and distance learning, addressing the challenges posed by government directives, the need for social distancing, and continuing health professions education (HPE) [1–4]. Reviews that investigated the developments in medical education in response to the COVID-19 pandemic highlighted that in the immediate response, the majority of interventions described a pivot to online learning. However, the need for continued clinical contact remained but was often replaced in curricula with remote, distance or telehealth. In the Best Evidence in Medical Education (BEME) rapid review, BEME 63, the authors identified a significant focus on sharing experiences, rather than robust evaluation or research enquiry, and less than 50% of studies reviewed described educational outcomes. BEME Guide no.64 acknowledged that online learning will undoubtedly continue to be a feature of medical education long after the pandemic, but encouraged educators to have a deliberate and thoughtful selection of strategies and consider the differential impacts of these approaches. BEME Guide no. 71 recognised the limitations of remote learning including the loss of social interactions, lack of hands-on experiences and challenges with technology, the authors recommend its continued use in higher education due the flexibility it offered and highlighted practical advice to optimize the online environment [4]. Among the educational interventions that were adopted in response to the COVID-19 pandemic, the flipped classroom (FC) has been reported to be efficacious in responding to these extraordinary challenges in medical education [5–7]. The FC, a form of blended learning and instructional strategy, seeks to enhance student engagement and learning. It involves students completing readings autonomously outside of scheduled class time and participating in live problem-solving activities during class time. In undergraduate ophthalmology education, studies by Diel et al found high levels of satisfaction with a FC approach and reported no changes in knowledge acquisition [8], and a reduction in students’ pressure to perform, course burden and anxiety, along with increased confidence in triaging common eye complaints [9].

In our initial educational response to the pandemic at our university, we implemented a remote online flipped classroom (OFC) approach to facilitate delivery of an ophthalmology clinical attachment for medical students, and evaluated students perceptions and satisfaction with the Course Evaluation Questionnaire (CEQ) [10]. The CEQ is used globally to determine undergraduate student satisfaction and to identify areas for improvement [11, 12]. There is substantial evidence supporting its reliability and validity with undergraduate and medical students [13–19], and it has been utilised in ophthalmology interventions evaluating the FC [20–22]. However, the efficacy of the FC for ophthalmology education in a completely virtual setting is still insufficiently measured [9]. We investigated student satisfaction using the CEQ following the introduction of a remote online FC, necessitated by the COVID-19 pandemic, compared to with our usual delivery format, which provided a blend of didactic lectures and clinical skills sessions [10].

Our results contradicted existing literature on the effectiveness of a flipped-classroom approach in delivering ophthalmology content to medical students. Previous studies indicated a preference among students for the flipped classroom over the traditional lecture method, citing its benefits in developing problem-solving, creative thinking, and teamwork skills [21, 23]. We identified significant levels of dissatisfaction with problem solving, communication, staff motivation and provision of feedback [10]. As the constraints imposed by government directives and the necessity for social distancing eased, we sought to re-design the ophthalmology module to incorporate learnings from our previous findings and the evidence based recommendations from BEME reviews 63, 64, 69 and 71 [1–4]. For subsequent iterations of the ophthalmology module an educational strategy that combined online learning and in-person seminars with practical patient-centred sessions was adopted. It was anticipated that this blended learning approach would result in improved levels of student satisfaction and knowledge gain. In this study we investigated how a blend of traditional classroom-based and remote FC learning approaches combined with in person practical elements including direct patient contact would impact student satisfaction with the CEQ, and compare these results to those previously reported for the complete OFC delivery of ophthalmology content [10].

Methods

Instrument and data collection

To investigate student perceptions and satisfaction with the online flipped classroom, all students (257) were invited to complete the CEQ36 online via Survey Monkey. Each item of the questionnaire is answered using a standard 5-point Likert scale where the levels of agreement range from “strongly agree” (scoring a “1”) to “strongly disagree” (scoring a “5”). The CEQ36 measures six constructs established as important learning environment features within the context of higher education [12, 14, 16], and are presented in Table 1. In addition to the CEQ36 data, final anonymised MCQ exam scores were obtained for each student in the study.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Course Evaluation Questionnaire constructs, dimensions and indicators.

https://doi.org/10.1371/journal.pone.0305755.t001

Results

A total of 257 undergraduate medical students who received the BL delivery of the ophthalmology clinical attachment were invited to participate in this study. Of these, 59 students (23%) agreed to take part in the study and completed an online CEQ. A total of 114 students who had received OFC delivery of ophthalmology content the year prior attended online tutorials as described previously [10]. Of these, 28 agreed to participate (25%).

The demographic distribution of the participants is presented in Table 2. There was no evidence of a difference in gender or age between the OFC and BL groups for the classes as a whole (column 1 v 3), or between students in the OFC group or the BL group who participated in the online surveys (column 2 v 4).

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Demographic information of students who participated in OFC and BL deliveries.

https://doi.org/10.1371/journal.pone.0305755.t002

Student perceptions

Fig 1 graphically summarizes the responses from the students regarding the six constructs established as important learning environment features within the context of higher education: Good Teaching (GT), Generic Skills (GS), Appropriate Assessment (AA), Appropriate Workload (AW), Clear Goals and Standards (CG), Emphasis on Independence (IN) [12, 14, 16]. Overall students indicated a preference for the BL compared to the OFC approach. We observed significant differences between the responses of the OFC and BL groups regarding the learning experience, perceived value of the flipped classroom, teaching process, skill development and the evaluation system outlined in Table 3. Due to small number of respondents in some categories, Strongly Agree and Agree, and additionally, Strongly Disagree and Disagree categories were joined for analysis. Furthermore, also relating to the small number respondents, the margin of error varied substantially, ranging from 5.7% to 12.6% for estimates of Agree/ Strongly Agree in the BL group, and 16.7% to 18.5% for estimates of Agree/ Strongly Agree in the OFC group.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Global summary of CEQ comparing student perceptions within the six survey constructs.

Average values were derived from analysis of Likert scale responses to each question within a subscale. OFC = Online flipped classroom. BL = Blended learning.

https://doi.org/10.1371/journal.pone.0305755.g001

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Comparison of students’ perspectives between online flipped classroom and blended learning.

https://doi.org/10.1371/journal.pone.0305755.t003

Good Teaching scale.

We observed that the BL delivery approach resulted in significantly greater levels of student satisfaction on the GT scale compared to the OFC approach. Specifically the BL group felt that the teaching staff motivated students to do their best (Q4, p<0.001), put a lot of time into commenting on students work (Q9, p = 0.004) and made a real effort to understand difficulties that students may be having with their work (Q20, p = 0.001). However, having adjusted for multiple comparisons Q9 was no longer significant.

Furthermore, compared to the OFC group the BL students felt that faculty were extremely good at explaining course content (Q23, p = 0.05) and that they made significant efforts to make the subjects interesting (Q25, p = 0.013). Critically we observed a significant improvement of student perceptions regarding the course trying to get the best out of its students among the BL group compared to the OFC student group (Q33, p = 0.001). However, having adjusted for multiple comparisons, only improvement of student perceptions regarding the course trying to get the best out of its students among the BL group compared to the OFC student group remained statistically significantly different (Q33, p = 0.033).

Clear Goals and Standards scale.

There was no evidence of a difference in student perceptions on the goals and standards (CG) scale specifically about what was expected from them (Q18), about the standard of work required (Q1) and faculty expectations of students being made clear (Q35). Overall there was no evidence of a difference in goals and standards after Bonferroni adjustment, however, before adjustment there was some evidence the BL group were significantly more satisfied with the CG compared to the OFC. Specifically, students felt that they had a clear idea of what was going on and what was expected from them (Q8, p = 0.006) and that the aims and objectives of the course were made very clear (Q24, p = 0.027).

Generic Skills scale.

There was no evidence of a difference in student perceptions about the capacity for the OFC or BL course to improve their written communication skills (Q13). In contrast to the OFC course students those who participated in the BL course showed some evidence of a difference prior to Bonferroni adjustment, finding it helped develop their problem skills (Q2, p = 0.001), sharpened their analytical skills (Q6, p = 0.023), developed their ability to work as a team member (Q11, p<0.001), improved their confidence about tackling unfamiliar problems (Q12, p = 0.029) and developed their ability to plan work (Q28, p = 0.003). Developing their problem skills (Q2) and developing their ability to work as a team member (Q11, remained statistically significant following adjustment for multiple comparisons.

Appropriate Assessment scale.

There was no evidence of a difference between the OFC and BL group in student perceptions of the impression that staff are more interested in testing what students have memorised (Q17) or ask too many questions about facts (Q26). Additionally, there was no difference between the OFC and BL group in their perceptions of the form that feedback was given (Q29) or that just by working hard around exam times they could get through the course (Q32). We observed significantly greater levels of student satisfaction among the BL group with the impression that faculty can learn from students (Q7, p = 0.001) compared to the OFC group. Furthermore, the BL group indicated that doing well on the course required more than just a good memory (Q10, p = 0.007).

Appropriate Workload scale.

There was no evidence of a difference in student perceptions in relation to the workload (Q5), the number of topics covered in the syllabus (Q14), the amount of time given to learn (Q19), the pressure felt by students (Q27) or how the volume of work affects comprehension of topics (Q36).

Emphasis on Independence scale.

There was no evidence of a difference in student perceptions between the OFC and BL group on the IN scale regarding opportunities to choose the particular areas you want to study (Q3), that the course encouraged them to pursue their academic interests (Q15) or their opportunities to discuss how they were going to learn with lecturers (Q30). However, we observed that the BL group were significantly more satisfied with elements in the IN scale compared to the OFC group. Specifically, students in the BL group felt they had greater levels of choice regarding how they would learn (Q16, p = 0.005), the work they had to do (Q21, p = 0.031) and the ways in which they were assessed (Q34, p = 0.004).

Questions regarding the value of the flipped classroom.

Previous studies have highlighted questions within the CEQ survey, which provide insights into the perceived value of the flipped classroom [21]. The FC scale questions overlap with the GT and GS scale; specifically questions 2, 4, 5, 11, 12, 13 and 28. Student survey responses indicated a significant level of student satisfaction with the online flipped classroom approach as part of the revised BL curriculum. As mentioned above students in the BL group felt that there were more opportunities to improve their problem-solving skills (Q2, P = 0.01) and that staff did more to motivate them (Q4, p<0.001). In addition, they felt the course helped develop their ability to work as a team member compared to the OFC group (Q11, <0.001), tackle unfamiliar problems (Q12, p = 0.029) and developed their ability to plan their own work (Q28, p = 0.003). When asked to rate the statement “Overall, I am satisfied with the quality of this course” there was no evidence of a difference in the rating between the OFC and the BL group (Q 37).

Comparison of overall student performance on final multiple-choice exam.

Next we compared students’ exam scores before and after the educational intervention for all students in the OFC (n = 114) and BL groups (n = 257) and students in the OFC (n = 28) and BL groups (n = 59) who responded to the survey. Students answered 20 ophthalmology multiple-choice questions (MCQ) as part of completing the course. Each question had the same weight, and the total score was converted into a 0–100 scale. Independent samples t test was used to compare the differences between the two groups. This analysis of the final exam MCQ score showed that there were no statistical differences between the OFC and BL group (p = 0.0560). Comparison of the final exam MCQ score for survey responders between the OFC and BL found no evidence of a statistical difference in the score achieved. Overall, this indicates that the BL did not negatively influence knowledge gain.

Discussion

The imperatives of the COVID-19 pandemic mandated an inevitable transition to online and distance learning, addressing the challenges posed by government directives and social distancing requirements. In the aftermath of the COVID-19 pandemic blended learning has become an accepted approach in health professions education [27]. While student safety and wellbeing was paramount, removing medical students from the clinical context to minimise risk associated with the COVID-19 pandemic was not a feasible long-term strategy. BL involves both face-to-face and online learning components and was therefore an advantageous approach to health professions education during the pandemic as it offers the best of both approaches [27]. In this study we wanted to assess student satisfaction with a revised ophthalmology module adopting a BL format, which included online learning and in-person seminars combined with practical patient centred sessions. Our goal was to compare BL with the previous delivery format that relied solely on a remote online flipped classroom to facilitate continued delivery of the ophthalmology module. It was hypothesised that as an educational intervention, the blended learning approach would continue to facilitate delivery of content as well as maintaining or improving levels of student satisfaction and knowledge gain as determined by a CEQ and MCQ examination.

Learner satisfaction is a multidimensional construct and is related to an individual’s subjective assessment [28]. Student satisfaction hinges on the efficacy of educational courses and the individual’s enthusiasm and enjoyment in the learning process [29]. Blended learning offers learners more choice in a multimodal delivery of course content, and when compared to traditional deliver, BL has been shown to yield more favourable results in terms of knowledge outcomes when compared to traditional learning in HPE [30]. While the analysis of the final exam MCQ scores in our study showed that there were no statistical differences between the OFC and BL groups, the BL group showed higher satisfaction with the choice provided by the BL approach with how they learn, the work they completed, and the methods of assessment (Emphasis on Independence scale). The practical constraints related to the OFC approach provided the learners fewer freedoms and choice in their educational journey.

Chick et al suggested innovative technology including FC could play an essential role bridging the educational gap during the unprecedented COVID-19 pandemic [31]. The FC has been demonstrated to be accessible and user friendly [32], and was favourable among ophthalmology residents, with reported improvement in test scores [33]. The OFC approach has been utilised by many academics who found it was well received by students and resulted in similar or enhanced knowledge gain in some instances compared to the traditional delivery of teaching [5–9]. However, our previous study’s findings were in contrast to this literature, and we found significant dissatisfaction with the online flipped classroom approach [10]. Given that our initial rapid response to the challenges of delivering content during the pandemic relied significantly on a remote OFC approach we sought to determine student perceptions with this model. Overall students reported a lack of satisfaction with this model indicating a lack of staff motivation, difficulties determining the standard of work required and lack of development of critical thinking and problem solving as issues with the OFC approach for remote ophthalmology teaching [10]. We believe that a lack of faculty preparedness [34–36], digital fatigue and student uncertainty may also have contributed to student dissatisfaction with the OFC approach [37–39]. In this study compared to the OFC group the BL students felt staff excelled at explaining course content (Q23, p = 0.05) and made significant efforts to make the subjects engaging (Q25, p = 0.013). The variety in the BL approach offers more choice and enables learners to engage with material through various mediums, and this was also reflected in the BL groups’ satisfaction with the choices regarding how they would learn and be assessed. Additionally, the multimodal delivery of course content in BL, appears to have addressed some of the challenges associated with the technological challenges faced by staff in the initial response to the pandemic.

Student Satisfaction is also associated with an individual’s interaction with their peers and with faculty [28], and in a national review in the UK having a “social life and meeting people” was acknowledged as a crucial factor contributing to overall satisfaction [40]. Our resulted demonstrated BL resulted in significantly greater levels of student satisfaction on the Good Teaching scale compared to the OFC approach, specifically relating to items associated with staff motivating students to do their best and to understand student difficulties. We noted markedly higher levels of student satisfaction within the BL group with survey items relating to students feeling motivated by staff (Q4), working as part of a team (Q11), and how the course tried to get the best of students (Q33). HPE involves hands on learning and elements of teamwork and effective communication. Online learning has been associated with poor engagement and during the COVID-19 pandemic, reduced interpersonal interaction [41], and has also been associated with lower levels of preparedness and a lack of hands-on training [42]. Our revised BL curricula, including online learning with in-person seminars and practical patient centred sessions, improved students self-reported problem solving, analytical skills and ability to work as part of team.

Conclusion

The COVID-19 pandemic compelled an inevitable shift to online and distance learning to address challenges posed by government mandates and social distancing requirements. However, in post-pandemic HPE, it is crucial to assess the effectiveness and learner perceptions of online and distance learning interventions. In line with recent BEME reviews we implemented a revised curriculum which included a blend of traditional classroom-based and remote learning approaches combined with in person practical elements including direct patient contact with mitigated risk. We provided support and training of both faculty and students will help to increase digital proficiency and engagement as online elements are continuing as a central feature of medical education. These changes resulted in significant increases in student satisfaction. Our study revealed a substantial student preference for blended learning (BL) over the online flipped classroom (OFC) approach, with comparable student performances based on MCQ examinations. Importantly, this study presents a unique insight into the repercussions of introducing an educational intervention centred on blended learning amidst the pandemic. This insight focusses on student satisfaction and the enhancement of learning experiences, underlining the distinctive value of our research. These findings indicate a preference for reintegrating in-person and patient engagement activities in post-pandemic health professions education.

Supporting information

Acknowledgments

We would like to thank all the students who participated in this study and acknowledge the support provided by Cathy Fox in the Royal Victoria Eye and Ear Hospital and the Student, Academic, and Regulatory Affairs (SARA) office at RCSI, University of Medicine and Health Sciences.