Effectiveness of blended learning in pharmacy education: A systematic review and meta-analysis

Background & objective Though blended learning (BL), is widely adopted in higher education, evaluating effectiveness of BL is difficult because the components of BL can be extremely heterogeneous. Purpose of this study was to evaluate the effectiveness of BL in improving knowledge and skill in pharmacy education. Methods PubMed/MEDLINE, Scopus and the Cochrane Library were searched to identify published literature. The retrieved studies from databases were screened for its title and abstracts followed by the full-text in accordance with the pre-defined inclusion and exclusion criteria. Methodological quality was appraised by modified Ottawa scale. Random effect model used for statistical modelling. Key findings A total of 26 studies were included for systematic review. Out of which 20 studies with 4525 participants for meta-analysis which employed traditional teaching in control group. Results showed a statistically significant positive effect size on knowledge (standardized mean difference [SMD]: 1.35, 95% confidence interval [CI]: 0.91 to 1.78, p<0.00001) and skill (SMD: 0.68; 95% CI: 0.19 to 1.16; p = 0.006) using a random effect model. Subgroup analysis of cohort studies showed, studies from developed countries had a larger effect size (SMD: 1.54, 95% CI: 1.01 to 2.06), than studies from developing countries(SMD: 0.44, 95% CI: 0.23 to 0.65, studies with MCQ pattern as outcome assessment had larger effect size (SMD: 2.81, 95% CI: 1.76 to 3.85) than non-MCQs (SMD 0.53, 95% CI 0.33 to 0.74), and BL with case studies (SMD 2.72, 95% CI 1.86–3.59) showed better effect size than non-case-based studies (SMD: 0.22, CI: 0.02 to 0.41). Conclusion BL is associated with better academic performance and achievement than didactic teaching in pharmacy education.


Introduction
Evaluating the effectiveness of blended learning (BL), a thoughtful combination of both online and face-to-face instructions, is difficult because the components of BL can be extremely heterogeneous [1,2]. For instance previous systematic reviews / meta-analyses on BL have included multiple techniques such as virtual face-to-face interaction, simulations, online instruction, e-mails, computer laboratories, mapping and scaffolding tools, computer clusters, interactive presentations, handwriting capture, class room web sites, and virtual apparatuses [3]. Also, there is no standardized proportion in which BL combines online with face-to-face instructions [4].
Flipped learning 'and 'hybrid learning' are often used interchangeably with BL. In flipped learning, the learner is first exposed to online content, which will be reinforced during face-toface sessions [5]. Hybrid learning, a combination of face-to-face instruction with computer mediated instruction, is most often used in United States [6]. In all forms of BL, the learner enjoys a certain degree of autonomy in deciding the pace of learning. However, previous reported systematic reviews on BL have not taken the keyword "flipped" in their search strategy [7,8].
Increased research has been published on BL in medical education over last decades. For instance, Quian Liu et al's systematic review and meta-analysis reported that BL has consistent positive effects in comparison with no intervention for knowledge acquisition in the health professions [7]. In another systematic review, McCutcheon et al reported a deficit of evidence on implementation of BL in undergraduate nursing education [9]. Most of the published systematic review and meta-analyses in medical education were focused on medical students or nursing students or other healthcare professionals [8][9][10]. There is only one meta-analysis that evaluated the effectiveness of flipped learning in pharmacy education, with a major limitation namely, lack of prospective randomized control trials (RCT) and restrictions to the domain of flipped contexts [11]. Accordingly, we designed our objective to assess the effectiveness of BL which employed a combination of online and face-to-face instruction in blended, hybrid and flipped contexts in pharmacy education. We have considered BL as a combination of online and face-to-face instruction, excluding other computer mediated forms like virtual labs, gamifications, simulations to limit heterogeneity and included all possible synonyms of blended, hybrid, flipped learning and pharmacy education.
used traditional lecture based learning as control for two arm studies and pre-test score for single arm studies (4) reported knowledge score/ objective structured clinical examination (OSCE) score as outcome (5) were two-group controlled studies (randomised/non-randomised)/ single group pre-test-post-test studies.
We excluded studies which did not explicitly state components of BL i.e. face-to-face learning and computer assisted learning. Computer assisted learning can be any form of technologies like online learning, e-learning, video podcasts, or the application of university learner management system for posting lectures. We excluded studies which employed "virtual faceto-face" interactions (as practiced by universities with satellite campuses). Studies which did not report a quantitative outcome of knowledge (comparison of students who completed and did not complete online module, number of correct answers between the groups, comparison of pass percentage), studies which evaluated only online component of BL, and surveys were also excluded. Multi-year studies without differentiating between study term years were excluded. Reviews, short communication, conference proceedings, editorials, meeting abstracts and non-English studies were also excluded.

Data sources and literature search
A literature search employing PubMed, Scopus and Cochrane Library, was performed using a comprehensive search strategy since the inception of each database up to mid-December 2020. We employed all the MesH terms and key words for "BL" (Blended learning, blended course, blended program, hybrid learning, hybrid Course, Hybrid Program, Hybrid training, Flipped learning, Flipped Course, Flipped Program, Computer-aided learning, Computer-assisted learning, Integrated learning, Distributed learning, Distributed education Integrated instruction, Computer-aided instruction, Computer-assisted instruction) and "Pharmacy Student" which was obtained from the databases and previous studies. We employed the asterisk ( � ) as a wildcard character in keyword searches. We also searched for additional reference materials by consulting the cross references listed in the included publications, in addition to Google and Google Scholar (Details in S2 Appendix).

Study selection and data extraction
The retrieved studies from databases were screened for its title and abstracts followed by the full-text in accordance with the pre-defined inclusion and exclusion criteria (List of excluded studies provided in S3 Appendix). We compiled and collated data in a comprehensive data extraction form containing characteristics such as, author and year of publication, population, duration and subject covered, nature of BL, sample size, and outcomes. The above data extraction form was perfected by trial and error, by piloting on 3 articles. Three independent reviewers were involved in study selection and data extraction to limit the bias and any disagreements were resolved through consensus or by discussion with another member of research team.

Quality assessment
Modified Newcastle Ottawa scale (Newcastle Ottawa scale-education) was used to appraise methodological quality of included studies [12][13][14]. This tool assessed the following criteria: 1) representativeness of intervention group (1 point) 2) selection of comparison group (1point) 3) comparability of comparison group (2 point) 4) study retention (1 point) 5) blinding of assessment (1 point), totalling a maximum of 6 points. Two independent reviewers were involved to appraise the methodological quality to limit the bias and any disagreements were resolved through consensus or by discussion with another member of research team.

Data synthesis
The evidence were synthesized narratively and presented in tabular form. We employed metaanalysis whenever possible. We omitted studies from data pooling whenever data did not meet the requirements of meta-analysis, such as, participant number, mean and standard deviation [SD]. All comparisons were based on scores of consecutive years. If more than one topic was delivered by BL in same study with separate scores for each, we considered them as separate studies. RevMan 5.3 was used to conduct the meta-analysis [15]. The data were used as mean with SD and outcomes were presented as standardised mean difference (SMD) along with 95% confidence interval (CI). Studies that did not report a SD, the corresponding SD from the pvalues and standard errors were generated as per Cochrane guideline [16]. Heterogeneity was assessed by I 2 statistics and random effect model used for statistical modelling. Subgroup analysis were performed to find out potential source of heterogeneity based on factors like studies with case studies and without case studies, studies which reported outcome as a measure of multiple choice questions(MCQs) or non MCQs, and studies from developed and developing countries. Sensitivity analysis were performed to ensure the robustness of findings.

Publication bias
We employed a funnel plot for visual inspection of publication bias, which was assessed for statistical significance by Egger's and Begg's test [16].

Results
A total of 2539 records were retrieved first, of which 2448 underwent initial screening. Next, 2383 studies were omitted, yielding 65 full-text studies, of which 26 studies were included for systematic review, and 20 for meta-analysis (See Fig 1 for details

BL approaches
Two studies employed face-to-face session followed by online activities [17,34] while all other studies employed face-to-face session after watching online content. Only one study reported time spent and workload associated with BL [37].

Quality assessment of included studies
As per modified Ottawa scale requirements, we ascertained that intervention groups in all the included studies were representatives of target population. Out of 26 studies, 19 used previous year students' score as control, one used subsequent year score as control and 3 studies were randomized. Two studies used analysis of covariance(ANCOVA) for controlling covariates in final analysis [23,38] and one used linear regression [22]. In five studies there were no statistically significant differences in students demographics / pre-test (Grade Point Average) between groups by t-test [27,30,32,34,41]. However, modified Ottawa scale requires controlling for subject characteristics by statistical covariate analysis. Outcome assessment was blinded for 11 studies, as assessor cannot be influenced by group assessment (third party statistician) or assessments did not require human judgments (MCQs/ graded performance) [17, 19-20, 25, 27, 29-30, 36, 38, 40-41]. As all studies were part of curriculum in educational institution, there is no mention about drop outs. All studies obtained a score below 4 except one [19] (See S4 Appendix).

Quantitative analysis
We included 20 studies with 4525 participants for meta-analysis that employed traditional teaching in the control group and had no missing data.

Efficacy of BL versus. Traditional teaching in improving knowledge
Pooled effect of 18 studies showed that knowledge improved significantly in BL, with large effect compared to didactic teaching ((SMD 1.35, 95% CI-0.91 to 1.78, p<0.00001). In the knowledge domain, randomised controlled studies had a lower pooled effect (SMD 0.88) than cohort studies (SMD 1.41). There was significant statistical heterogeneity among studies (I 2 = 98%, p<0.00001) with individual effect sizes ranging from −0.37 to 15.54 (See Fig 2).

Efficacy of BL versus traditional teaching in improving skill
Pooled effect size (SMD 0.68, 95% CI: 0.19 to 1.16,Z = 2.74,p = 0.006) of 4 studies in improving skills, showed statistically significant moderate to large effect, compared with didactic teaching. Significant statistical heterogeneity was observed among studies (I 2 = 92%, p<0.00001) (See Fig 3).

Sensitivity analysis
A sensitivity analysis was performed in studies improving knowledge by removing two studies (Wong et al., [2,3]) which are having lesser weight (3.1% and 4.1%, respectively), and higher

Publication bias
Visual inspection of funnel plot revealed an obvious asymmetry, demonstrating possible publication bias. This was confirmed by Egger's (P = 0.00006) and Begg's (P = 0.04) test (See Fig 5).

Discussion
This systematic review and meta-analysis primarily attempted to evaluate the impact of BL approach on various outcomes in pharmacy education. We identified 26 studies relevant for systematic review, in which 18 demonstrated significant improvement in learning outcome, against controls. Two of them were single arm studies which also showed improved performance after intervention. 24 of the 26 studies included in this systematic review were controlled, among which majority (n = 19) employed examination scores of previous year(s) as the control. All studies employed first online review of contents followed by face-to-face discussion except two. Studies which employed face-to-face discussion followed by online activities also favoured BL [17,34]. The face-to-face discussion part of BL in all included studies involved either reinforcing the concepts by tutor or using learning strategies such as case studies, case discussion or group activities. In addition to the general scarcity of literature comparing BL and traditional methods, a major limitation of the previous meta-analysis by Gillette et al., was the lack of prospective RCTs [11]. Our meta-analysis included 20 of the studies included in the systematic review. Our review included 3 RCTs, all of which showed major improvements in either knowledge score or skill. We report a large pooled effect size for knowledge and a medium to large for skills. These findings were statistically significant with high heterogeneity in all analyses and are consistent with those reported by previous meta-analyses in medical education.
The majority of the studies reported knowledge score in terms of either mean examination percentage/score or OSCE, whereas 5 studies reported outcomes based on skill. Many of the studies included in this review also reports that BL has a major effect on improving teaching as well as positive student perceptions about learning. As mentioned earlier, the rich variety of components can attribute to an enhanced learning experience as well as increased engagement and learning activities such as group assessment, assessment quizzes and peer discussions. Even the studies that did not report a significant difference in acquisition of knowledge-such as those by Phillips et al., and Gloudeman et al. showed that the perceptions of both students and faculty favoured BL [35,42]. Another important finding is that BL modules which employed case studies/discussions or case-based scenarios reported better outcomes. A few studies also concluded that positive results obtained may not be attributed entirely to the suggest on that case studies need to be included in learning strategies [24,37]. There is evidence to show that case studies simulate real world situations and enhance interactive student-centred learning, particularly in the health professions. Incorporating case studies in a real-world context is extensively useful in pharmacy education, as it enhances students' complex decision-making abilities.
Out of 26 studies, only 4 originated from developing countries, possibly because of poor online connectivity, lack of resources, fear of adopting unfamiliar technology, lack of skill development program to instructors, interruption in power supply and internet connections, affordability, low bandwidth and trust deficit [17,20,26,28]. A single study that compares time budgets reported that BL techniques were completed ahead of allotted time [35]. BL approach appears to significantly improve the learning outcomes in pharmacy students and reason could be following, i. Relaxed/flexible scheduling: BL allows students to view electronic materials at their own pace and time ii. Improved interaction: BL makes classroom discussion more meaningful because of content familiarity.
iii. Variety of components: BL incorporates a rich variety of face-to-face and online components.
This study has a few limitations. First, the search was restricted to the publications in English language, which might have contributed to missing out eligible studies in non-English speaking countries. However, a comprehensive search in various databases would have covered the maximum quality publications. Second, our review also excludes conference proceeding and unpublished or grey literature. However, this may increase the credibility of our findings obtained from full length papers by avoiding the irrelevant or incomplete acquisition of the data. Third, there was high heterogeneity among the outcomes or measures of outcome, thereby restricting our choice exclusively to studies reporting quantitative outcomes. Fourth, the heterogeneous administration pattern of BL was an another challenge in this review, so we included those studies which used online teaching along with face-to-face approach, this made our result more robust and conclusive. Statistical heterogeneity was high in all analysis. However, this is in accordance with other meta-analysis in medical education [7,8,43]. Subgroup analyses did not find any source of heterogeneity. Despite the effective search strategy, one major limitation is the majority i.e. 18 of the 26 studies, were from the US, which could impact the global representativeness of the findings. Therefore, future research should address the impact of BL in diverse populations from other countries.
Publication bias was addressed by including the three major scientific databases (Pubmed, SCOPUS and Cochrane) during the literature search. This resulted in an increased number of papers which may have further increased the likelihood of selecting papers with negative results. In our review, 5 of the 26 studies reported that BL yields either equal or poorer outcomes than didactic teaching [33,35,38,40,42].