Figures
Abstract
Background
Recently, flipped classrooms (FCs) have gradually been used in Chinese higher education settings. However, few studies have focused on the effects of FCs on interdisciplinary curricula. The purpose of this study was to examine the impact of an FC on the engagement, performance, and perceptions of students and on teacher-student interaction in a pharmaceutical marketing course.
Design
A clustered randomized controlled study was conducted, with 137 junior-year pharmacy undergraduates using an FC serving as the intervention group, in contrast to students using lecture-based learning (LBL) as the control group. Flanders’ interaction analysis system (FIAS) was used to measure teacher-student interaction, and questionnaires regarding attitudes toward and satisfaction with the teaching model were administered.
Results
The students in the FC group scored significantly higher than those in the LBL group (88.21±5.95 vs. 80.05±5.59, t = -8.08, p = 0.000) on pharmaceutical marketing. The multiple linear regression results showed that the FC model had a significant impact on student performance (β = 8.16, p<0.0001). The percentages of teacher talk in the FC and LBL groups were 21% and 96%, respectively (χ2 = 2170.274, p = 0.000); however, the percentages of student talk in the FC and LBL groups were 75% and 2.6%, respectively (χ2 = 2012.483, p = 0.000). Compared with the LBL group, most students in the FC group held more positive attitudes toward the teaching model; the mean scores for the 8 attitude attributes in the FC group were significantly higher than those in the LBL group (p = 0.000). There were significant differences in the ratings of satisfaction with teacher-student interaction (p = 0.000), the students’ learning attitude (p = 0.000), the teacher’s preparatory work (p = 0.000), the teaching objective (p = 0.000), and the teaching effect (p = 0.000) between the two groups.
Conclusion
Compared with LBL methods, implementing the FC model improved student performance, increased teacher-student interaction and generated positive student attitudes toward the experience. As an effective pedagogical model, it can also stimulate pharmacy students’ learning interest and improve their self-learning abilities.
Citation: He Y, Lu J, Huang H, He S, Ma N, Sha Z, et al. (2019) The effects of flipped classrooms on undergraduate pharmaceutical marketing learning: A clustered randomized controlled study. PLoS ONE 14(4): e0214624. https://doi.org/10.1371/journal.pone.0214624
Editor: Andrew R. Dalby, University of Westminster, UNITED KINGDOM
Received: December 23, 2018; Accepted: March 15, 2019; Published: April 10, 2019
Copyright: © 2019 He et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: The study was supported by the Humanities and Social Sciences Fund of Chinese Ministry of Education (Grant No. 15YJCZH061) and the Education and Teaching Research Project of Nanjing Medical University (Grant No.YB2017051). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Concurrent with the increasing use of information technology and electronic learning (e-learning) approaches, flipped classrooms (FCs) are growing in popularity in higher education worldwide. Previous scoping reviews, which summarized the relevant research regarding the emergence of the FC approach, showed that much convincing evidence of improved course grades and learning satisfaction with the active learning or FC approach was emerging [1–8]. However, most of the studies included in the reviews were conducted in Western countries such as the United States and Australia. Only a few studies in East Asian countries have evaluated the impacts of the FC model on student satisfaction [1–8]. To date, studies that focused on FC teaching methods primarily came from Western contexts. The enormous cultural differences between Western and Asian populations could give rise to potential cultural conflicts and exert negative effects on the FC teaching model. It is necessary to expand the understanding of how the FC teaching model works in other cultural contexts, especially in China, the largest developing country worldwide.
According to statistics from China’s Department of Education, approximately 40% of Chinese pharmacy undergraduates have a willingness to engage in pharmaceutical marketing and management, becoming, for example, sales and marketing managers, medical representatives or medical product managers [9]. In China, pharmaceutical marketing is listed as one of the compulsory courses in the curriculum of pharmacy education [10]. The specialized pharmaceutical marketing course focuses exclusively on the pharmaceutical industry or health-related field, and it not only offers the same analytical and financial skills as a general marketing course but also emphasizes the knowledge and skills needed to understand the economic, financial and organizational structure unique to the pharmaceutical industry [11].
Specifically, the aims of this course are to teach pharmacy students to master the principles of marketing, economics, management theory and evaluation of marketing cases in the pharmaceutical industry. At the end of the curriculum, pharmacy students can understand and have mastery of the knowledge and skills associated with pharmaceutical marketing planning, implementation and surveying.
Traditionally, the lecture-based learning (LBL) method has been widely used to teach the undergraduate pharmacy curriculum in Chinese university classrooms. However, in classic LBL sessions, the teacher is regarded as the highly knowledgeable leader who can control the process of teaching in the classroom, while students are usually viewed as passive learners who only receive knowledge from the teacher [12,13]. As LBL is used as the only teaching method, students in lecture courses are passive recipients. This passivity leads to negative effects in which students usually perceive a lack of opportunity to take more initiative in analyzing problems and developing problem-solving skills [14, 15]. Furthermore, it almost always takes the teacher a great deal of time and effort to communicate theoretical knowledge and basic contents to the students in the classroom. On the one hand, teachers cannot satisfy the personal needs of each learner. On the other hand, students do not have enough time to receive one-on-one guidance or to obtain problem-based learning experience [16, 17]. In particular, under the LBL model, teachers usually pay little to no attention to which practical abilities students can improve. Furthermore, this passive learning process inevitably leads to student boredom and prevents students from acquiring the necessary application and communication skills [18–19].
Therefore, to transform students into “active self-explorers of knowledge” and to promote their active participation in the learning process, it is very important to develop or seek new teaching models that have the ability to improve the teaching effect during a limited period of time [4,13,18,19].
The FC is a new pedagogical method that complements traditional teaching models; it employs asynchronous video lectures and practice problems as homework and active, group-based problem-solving activities in the classroom [20,21]. In an FC, the teacher is no longer the “speaker” on the podium; instead, the teacher becomes a coordinator for the students. Some scholars have described the nature of the flipped teaching mode as follows: By blending the strengths of internet-driven instruction outside the classroom (e.g., digital videos, self-regulated learning, online discussions) and face-to-face inside the classroom (e.g., collaborative study, applied problem-solving, instructor and peer engagement), the FC model can effectively increase student engagement, improve student performance and strengthen the development of creative thinking [22–25]. For instance, as early as 1996, an education study conducted in the United States that changed the before, during and after class traditional teaching order demonstrated that the “inverted classroom” was able to effectively engage students in the learning process [20]. In the FC context, students can not only selectively watch online videos based on their knowledge levels but also freely replay core and difficult content. In the classroom, teachers and students have sufficient time to conduct face-to-face discussions. Meanwhile, to enhance knowledge integration and application, students are asked to use in-class time for group discussion. These discussions, especially those involving controversial topics, can play an important role in improving students’ understanding of the key points of the course. [26]. Under the flipped pedagogy model, students can benefit from the freedom to choose the most appropriate method to learn knowledge before class. Most importantly, this model integrates the process of knowledge internalization into class time to enhance the extensive learning cooperation between students and to increase teacher-student interaction. The “negative recipients” in traditional classroom learning can be transformed into “positive explorers” under flipped teaching through a recognition of the individual characteristics of cognitive learning [26].
Research on the use of FCs in the field of pharmacy education has grown over the past few years. [27–31]. McLaughlin et al. found that the flipped pharmacy classroom enhanced the quality of students’ experiences in a basic pharmaceutics course through a thoughtful course design, enriched dialogue, and the promotion of learner autonomy [27]. They also demonstrated that engagement with a highly interactive online preparatory tool was positively related to student learning in neurologic pharmacotherapy [28]. Pierce and Fox also revealed that the positive outcomes of an FC model for teaching a renal pharmacotherapy module resulted in improved student performance and favorable student perceptions of the instructional approach [30]. However, a study by Bossaer et al. involving a pharmacotherapy oncology module found that examination scores were poorer in the FC cohort than in the traditional LBL cohort [31].
Recently, teachers have been encouraged to initiate teaching innovation programs using FCs in Chinese higher educational contexts [32]. For instance, Zeng et al. implemented a randomized controlled study on undergraduate electrocardiogram learning. Compared with those learning in the LBL classroom, the medical students learning through the FC model reported greater self-learning abilities, increased interest and decreased boredom [33]. However, few studies have highlighted the effect of the FC model on interdisciplinary curricula, such as pharmaceutical marketing.
In this study, we define the FC as an educational technique that consists of two parts: direct web-based individual self-learning outside the classroom and interactive group learning activities inside the classroom. Specifically, the primary objective of the current study is to use mixed methods to examine the impact of FC teaching on the engagement, performance, and perceptions of junior-year pharmacy students and on teacher-student interaction with a flipped pharmaceutical marketing course. The secondary objectives include serving as a pilot for pedagogy researchers to implement and develop the FC methodology for pharmaceutical marketing and providing a theoretical foundation that will inform future innovative course design and a broader range of FC applications in the pharmacy curriculum.
Methods
Study design and participants
We conducted a clustered randomized controlled study on student performance, student satisfaction and teacher-student interaction, with the class as the unit of analysis. This study was undertaken at Nanjing Medical University (NMU) in China between September and December 2017.
The participants were recruited from the School of Pharmacy at NMU in August 2017. The inclusion criteria were (1) majoring in pharmacy, (2) being aged ≥18 years, and (3) being willing to sign a consent form to participate in the study. Students were excluded if they had transferred from other specialties or schools, were transferring to other specialties or schools for further study, or had already taken the pharmaceutical marketing course prior to this study. During the introductory session, the researcher assistant (RA), who was not involved in student outcomes or examinations, followed a standardized script to give the full details of the study to the students and asked for informed consent. The RA emphasized that participation was completely voluntary and would be independent of the students’ course grades. After distributing the informed consent forms to the students, the RA asked them to read the form and organized a discussion session regarding the study. After the introductory session, the students had at least 24 hours to decide whether to participate in the study. Moreover, they themselves submitted the informed consent form in a sealed file packet to the RA at the end of the recruitment period.
This study determined the sample size according to the rule of statistical power [34,35,18,19]. The sample size was determined using G*Power 3.1 for Windows. For the purposes of this study, the significance level was set at 0.05 (two-sided) with 80% statistical power and the effect size at 0.5 for analysis. Considering a 10% dropout rate, a minimum number of 53 pairs (106 participants) would be sufficient to evaluate the outcome. In this study, randomization was performed at the class level. Specifically, this study was further undertaken in five independent teaching classes, which were randomized into the control and the FC group. Randomization at the student level and blinding were not possible due to the nature of the teaching model. Three classes were randomly allocated to the FC group, while the other two were assigned to the LBL group. A total of 137 junior-year pharmacy students with five independent teaching classes were enrolled in this study. Eighty-one students participated in the FC group, while fifty-six students participated in the LBL group. This sample size fulfilled the requirement.
Teaching contents and methods
All participants in the two groups had the same teacher, the same teaching syllabus, and the same textbook. The teacher also set up the same teaching goals, the same teaching schedule and the same test model. This course curriculum covered ten teaching chapters belonging to the pharmaceutical marketing knowledge domain. These chapters addressed the following contents: the definition of pharmaceutical marketing; the concept of marketing; the pharmaceutical marketing mix (the 4Ps); the market environment; consumer behavior; market segmentation, targeting and positioning (STP strategy); communications (personal sales, sales promotion, public relations, publicity, advertising); planning; the product life cycle; and customer focus. In the chapter named “market segmentation, targeting and positioning (STP strategy)”, each group was taught using FC teaching or LBL. Over four consecutive weeks, the classes that were held for the chapter included four teaching units. The details of the teaching methods are as follows:
FC teaching method
The whole process of FC teaching can be divided into three phases: the knowledge delivery, knowledge internalization and knowledge consolidation phases[35–37]. The process consists of the following (Fig 1):
1. The knowledge delivery phase (out-of-class activities).
In this phase, to successfully carry out knowledge delivery before class, some out-of-class activities should be performed by the teacher and students, including preparation by the teacher and students. Three months prior to the course, the teacher received training for microlecture sessions. Based on the teaching goals and contents, the teacher decomposed his or her knowledge, compiled the PowerPoint courseware, designed and produced the microvideos, and uploaded the videos to the e-learning platform. Each video was accompanied by quizzes and exercises to help students learn by themselves and to test the effect of self-learning.
Three weeks before the classroom was flipped, the teacher introduced the aims of the teaching and the details of curriculum to the students and announced the requirements and procedures of FC learning. To facilitate group discussions, the students were divided into 8 groups. Each group was required to nominate a chair, who was the main point of contact and had overall responsibility for assembling the group, leading discussion within the group, and ensuring the timely completion of work.The chair also coordinated microlesson learning activities and communications to address problems among group members. The students were required to watch narrated microvideos. At the same time, each group was required to prepare a presentation before class time, namely, a talk in which a piece of group work was shown and explained to the teacher and students in the FC. Two weeks prior to each teaching unit, the teacher listed 5–6 presentation topics on the e-learning platform. Each group could select any one of the listed topics. In the FC, each group presented its learning results to the entire class. The students’ presentation grades accounted for 30% of their entire final course grade. Moreover, the teacher uploaded supplementary reading materials, including PowerPoint courseware, textbook sections, self-learning materials and exercise information, and sent the links via the e-learning platform and discussion forums.
The reading materials included a variety of classic marketing books, papers, journals and slides. The topics of the reading materials consisted of broad marketing research, market targeting and market segmentation, the determination of distribution, pricing and promotion strategies, the development of a communications strategy, budgeting, and how to envision long-term market development goals. For instance, the book “Marketing 4.0: Moving from Traditional to Digital”, published by Wiley, was recommended to the students [38]. During the knowledge delivery phase, group members could freely discuss the content of a small private online course (SPOC) and communicate with the members of their own groups anonymously on the website discussion boards. On the day prior to the class for each teaching unit, the chair of each group uploaded group members’ assignments and exercises to the e-learning platform, and subsequently, the teacher could evaluate the effectiveness of self-learning.
2. The knowledge internalization phase (in-class activities).
The in-class activities consisted of three stages. In the first stage, the teacher gave a concise lecture that highlighted the key points of the teaching unit based on the assignments and student feedback. The aim of this stage was to resolve the major problems and confusion shown during students’ self-learning [36]. The first stage lasted approximately 10 minutes.
In the second stage, the role of the teacher was switched from that of a leader to that of a coordinator, and in-class learning started with the student-initiated interactive learning model between the teacher and students [39,40,7,4]. The in-class activities could be carried out with different learning strategies, such as student presentations, group discussions and collaborations, and problem-solving [2,7].
First, each group made a presentation on the topic that the members had chosen. During the presentation process, the students in the other groups were encouraged by the teacher to interrupt the speaker or presenting group and to ask questions. Accordingly, the speaker or the presenting group members gave a preliminary answer to the other groups [41–42,19]. All students could also propose new issues for further discussion between groups. Each group took 8 minutes to complete this task. Second, the teacher offered a teacher-student interaction session in which the students freely raised any concerns regarding the presentations and the teacher briefly asked each group questions on key points. This session also provided an in-depth discussion about the questions that were raised by other groups during the previous presentations. Student concerns could be addressed either by students from other groups or by the teacher. After the group-based discussion, each group chose a representative to answer the questions raised, and the other group members could provide new thoughts about the answers, ask further questions, or discuss between groups [43,7,35]. This session lasted approximately 3–4 minutes for each group. In the second stage, the eight groups took a total of approximately 90 minutes to present ideas and discuss issues.
Finally, in the third stage, the teacher provided simple reviews and summaries; then, the teacher provided some exercises to all the students. Each group solved the problems through discussion within the group [44]. After the teacher presented the key analysis, the representative of each group contributed to the interpretation and supplementary conclusions. The third stage lasted approximately 20 minutes.
3. The knowledge consolidation phase.
After the classroom was flipped, the students were able to use the e-learning platform to replay the microvideo lectures. According to their individual level of knowledge of the content, they could also review the key points and marketing cases through online group discussions. Additionally, during online discussions, students could discuss difficult points with the teacher through the e-learning platform.
Lecture-based learning method
In the control group, traditional classes, known as the LBL method, were used as follows. Two weeks prior to class, the teacher distributed supplementary learning materials, including pharmaceutical marketing teaching materials, classic cases, quizzes and exercises. In the LBL class, the teacher gave a lecture lasting approximately 110 minutes, followed by 10 minutes of communications between the teacher and students. After class, the teacher uploaded the PowerPoint slides used for the lecture to the e-learning platform. Outside of class, the students were free to self-study based on the corresponding textbook chapters.
Course evaluation methods
Seven days after the teaching unit was conducted using the different teaching methods in the two groups, the students were synchronously given a test; this test aimed to assess their knowledge and level of understanding regarding the topic. In the test, the students in the two groups received the same questions but in a different order. The primary aim of the test was to assess the ability of the students to analyze typical questions on pharmaceutical marketing concerning the core knowledge that was prepared in advance by the teacher. The students’ knowledge and ability were measured by 5 short open-ended questions, 10 multiple-choice questions and 2 essay questions. The essay questions consisted of a short pharmaceutical marketing case. The students had to conduct a comprehensive analysis on this case. The short open-ended questions included the following: (1) What is the role of pharmaceutical representatives in pharmaceutical marketing activities? (2) How can a medical product be positioned in the minds of consumers, and what attributes can differentiate it from competitor products? (3) What are the two main purposes of market segmentation? (4) How are the commonly used criteria for market segmentation selected? (5) How are key opinion leaders identified through direct inquiry of physicians?
The total time for the test was fifty-five minutes. Students took three minutes and fifteen minutes to answer each multiple-choice question and essay question, respectively. Previously, we invited two educational experts to assess the two tests to ensure the integrity of the content and consistency in difficulty levels. The test scores were statistically analyzed to compare the differences between the two groups in knowledge and ability.
Primary outcomes
The outcomes of teacher-student interaction.
To evaluate the impact of the FC on the interactive activities between the teacher and students inside the classroom, we adopted Flanders’ interaction analysis system (FIAS), which was developed by Flanders (1962) to analyze teacher-student verbal behaviors in the context of classroom communication. The FIAS is a standardized observation method of interaction analysis that describes teacher-student interaction through predefined activity categories. In the FIAS system, ten categories of verbal statements were defined, with 7 categories for teacher talk, 2 categories for student talk and 1 category for silence or chaos in the FC and LBL classroom [45]. The FIAS has been employed by education researchers worldwide as an effective measuring tool for analyzing teacher-student [15, 45–48].
As shown in Table 1, we made a minor modification to the FIAS by adding “answering students’ questions” as a teacher talk category. Traditionally, Chinese college students have been regarded as passive learners, and they rarely ask questions in class [12]. To discover the impact of the FC on students’ in-class activities, we added some categories to analyze the percentage of student-initiated talk in class. Thus, “student-initiated talk: initiating a conversation between groups” (9b), “student-initiated talk: answering other group members’ questions” (9c) and “student-initiated talk: discussing problems within groups” (9d) were added as student talk categories. According to the following scheme, two trained RAs recorded the frequency of each category code of verbal behaviors in the class every 3 seconds. Further data were processed into so-called activity indices, which express teacher-student interaction as follows:
- The percentage of teacher talk: This activity index was calculated by the ratio of the frequencies of the teacher’s verbal behaviors (category codes 1–7) to the frequencies of the total verbal behaviors (category codes 1–10) inside the classroom.
- The percentage of student talk: This activity index was calculated by the ratio of the frequencies of the students’ verbal behaviors (category codes 8–9) to the frequencies of the total verbal behaviors (category codes 1–10) inside the classroom.
- The percentage of silence or chaos: This activity index was calculated by the ratio of the frequencies of silence or chaos (category code 10) to the frequencies of the total verbal behaviors (category codes 1–10) inside the classroom.
- The ratio of the teacher’s indirect influence to the teacher’s direct influence (I/D ratio): This activity index was calculated by the ratio of the frequencies of the teacher’s indirect influence (category codes 1–4) to the frequencies of the teacher’s direct influence (category codes 5–7) inside the classroom.
- The ratio of the teacher’s positive influence to the teacher’s negative influence (P/N ratio): This activity index was calculated by the ratio of the frequencies of the teacher’s positive influence (category codes 1–3) to the frequencies of the teacher’s negative influence (category codes 6–7) inside the classroom.
- The ratio of the students’ positive responses to the students’ passive responses: This activity index was calculated by the ratio of the frequencies of students’ positive responses (category codes 9a-9d) to the frequencies of students’ negative responses (category code 8) inside the classroom.
We analyzed a total of 240 teaching minutes (120 minutes in the FC group and 120 minutes in the LBL group) using the modified method FIAS, and the results were graphically and statistically processed.
Secondary outcomes
Regarding their attitudes, learning experiences and satisfaction under each teaching method, the students in both the FC and LBL groups were measured with the same self-administered questionnaire, which used a 5-point Likert scale.
A focus group interview that included teachers, students, teaching administrators and education scholars was conducted to collect the student attitude, learning experience and satisfaction attributes. In addition, some questions were adopted from previously published studies, such as those by Zeng et al. [33], Meng et al. [18] and Lin et al. [19].
The questionnaire contained 20 items and used a 5-point Likert scale, and the levels of agreement or satisfaction ranged from “strongly disagree” and “very dissatisfied” (“1”) to “strongly agree” and “very satisfied” (“5”). To collect the students’ total learning time investment outside of class, we designed a fill-in-the-blank questionnaire.
To evaluate the impact of the FC model on student learning gains in a more robust manner, we employed a linear regression model and controlled for student demographic variables, especially GPA [49]. The model for the regression is as follows:
We define the following variables for each student: Y final is the student’s score on the test; YPA is the student’s score on the prior pharmacy management course, which is the proxy for the pretest on pharmaceutical marketing; GPA is the student’s GPA; Performance is the student’s grade in the prior introductory course in pharmacy; and Intervention is a variable for whether the student was in the FC group (Intervention = 1 if the student was in the FC group; otherwise, Intervention = 0). In this study, each control variable (PA, GPA and Performance) was mean-centered by subtracting the mean of each variable from each student’s value [49].
Statistical analysis
All statistical analyses were performed using the SPSS 17.0 statistical software package (SPSS, Inc., Chicago, IL, USA). Continuous variables were reported in terms of their mean and corresponding standard deviation (SD) or standard error (SE), while categorical data were presented as percentages.
The scores of the test results and the learning time outside of class invested by the students were used for an independent-samples t-test to analyze between-group differences. Due to their distribution characteristics, the differences between the two groups were compared through appropriate nonparametric tests. When appropriate, Mann-Whitney U tests were used to compare questionnaire values between the FC and LBL groups. The relationships between variables were examined using linear regression analysis. Multiple linear regression analysis was used to test the proposed model. The independent variables were the influences on final scores in the proposed model, and they were entered simultaneously. A t-test of the beta coefficient was used to test each hypothesis. All tests were two-tailed and were assessed at the 5% level of significance.
Results
Baseline student characteristics
The demographic distribution of the participants is presented in Table 2. The baseline variable data for the FC and LBL groups were examined via chi-square and two-tailed t-tests. The results showed that there were no significant differences by gender (χ2 = 0.440, p = 0.507), age (t = -0.298, p = 0.766), and GPA (t = -1.447, p = 0.09).
In addition, there were 517 pharmacy students in the School of Pharmacy at NMU; the proportion of enrolled students accounted for 26.5% of the total number of pharmacy students at NMU. There were no significant differences between the sample and the total population in any of the variables (p>0.05). Thus, the sample could be considered representative of the population in this study.
The test scores between the two groups
As shown in Table 3, the total test scores of the students in the FC group were 88.21±5.95, which were significantly higher than those of the students in the LBL group (80.05±5.59). The differences in the scores of the short-answer questions and essay questions were also significant (p = 0.000). However, there was no significant difference between the two groups in the scores of the multiple-choice questions (p = 0.502).
As presented in Table 4, the multiple linear regression model showed an R2 value of 0.838, p<0.001. The students’ score on pharmacy management (YPA), their score in the prior introductory course in pharmacy (Performance) and whether they were in the FC group (Intervention) all had a significant influence on the students’ total final scores. Put differently, the “intervention” is significantly correlated with student performance, controlling for the influence of student demographic variables.
The estimated intercept (β0 = 80.05) means that the expected score on the final test for a student with an average YPA, average GPA, and average Performance and who did not receive the intervention is 80.05.
The coefficient on YPA (β1 = 0.64) means that, holding all other variables constant, a student’s score on the final test is expected to increase by 0.64 points for each additional point the student scored on the prior pharmacy administration course.
The coefficient on Performance (β3 = 0.29) means that, holding all other variables constant, a student’s score on the final test is expected to increase by 0.29 points for each additional point in the student’s grade on the prior introductory course in pharmacy.
The coefficient on Intervention (β4 = 8.16) means that, holding all other variables constant, the score on the final test of a student who was taught via the FC teaching method is expected to increase 8.16 points compared to a student who did not receive the FC intervention.
Comparison of the teacher-student interaction of the two groups
To better understand the teacher-student interaction results, we performed a comparative analysis of teacher-student verbal behaviors between the two groups based on the modified FIAS.
Structured observations of the verbal behaviors in the two groups were graphically recorded (Fig 2). In general, the number of observations of the teacher giving lectures (category code = 5a) in the control group was 1916, which dominated most of the verbal behaviors in the LBL class. In addition, six category codes of verbal behaviors were not recorded in the control group. Fig 2 shows in the LBL class, lecturing was the dominant activity of the teacher.
Verbal behavior category codes: 1 accepting students’ feelings in a constructive way, 2 giving encouragement or praise, 3 valuing student views, 4 posing questions (teacher), 5a giving lectures, 5b answering students’ questions, 6 giving instructions, 7 criticizing students or establishing authority, 8 teacher-initiated talk, 9a student-initiated talk, 9b initiating a conversation between groups (students), 9c answering other group members’ questions (students), 9d discussing problems within groups,10 silence or chaos.
On the other hand, we compared the number of observations of student talk. The total number of observations of students giving lectures on a topic during the presentations (category code = 9a) in the FC group was 663. The other seven groups could interrupt the lecture and initiate a conversation between groups (category code = 9b), and the number of observations was 177.
According to the questions to be initiated by the other groups, the number of observations of answering other group members’ questions (category code = 9c) was 268. The total number of observations for student-initiated talk consisting of lecturing other students during the presentations was 445 (category codes = 9b, 9c). In addition, all the students could discuss problems within groups in FC, especially for solving the problems and exercises (Category code = 9d), the number of observations for group-based discussion was 635. All these numbers of observations accounted for a large percentage of the verbal behaviors in the FC. Thus, the results show that the students played the most active role in verbal communications with the students and teacher. Table 5 displays the frequencies and activity indices of the teacher-student verbal behaviors in the two groups. The results of the chi-square test of independence show that the percentage of teacher talk in the FC and LBL groups were 21% and 96%, respectively, presenting a significant difference (p = 0.000). However, the percentages of student talk in the FC and LBL groups were 75% and 2.6%, respectively, also exhibiting a significant difference (p = 0.000). This result indicates that student talk far exceeded teacher talk in the FC group. Notably, the number of observations for other types of teacher talk, such as category codes 1, 2, 3, 4 and 7, was relatively low in the two groups. The percentages of silence or chaos in the FC and LBL groups are also shown in Table 5; they are both very low. There were detectable differences between the two groups (4% vs. 1%, χ2 = 41.880, p = 0.000).
In relation to the teacher’s verbal behavior, the I/D ratios in the FC and LBL groups were 1:4.10 and 1:23.25, respectively, showing a significant difference (p = 0.000). Then, we analyzed the students’ verbal behavior; compared with the LBL group, the FC group had a higher ratio of the students’ positive responses to the students’ passive responses (54.47:1 vs. 0.91:1, p = 0.000).
Student attitudes toward and satisfaction with the teaching model
The overall evaluation results of the questionnaires are shown in Table 6 and Table 7. As shown in Table 6, the students in the FC group showed higher mean scores on the 8 attitude attributes (p = 0.000).
Compared with the students in the LBL group, the overwhelming majority of the students in the FC group held more positive attitudes toward their teaching model.
Table 7 summarizes the results of comparing the median scores of the 8 satisfaction attributes between the FC and LBL groups with Mann-Whitney U tests. Comparing the two groups, we found that there were significant differences in the rating of satisfaction with teacher-student interaction (p = 0.000), the students’ learning attitude (p = 0.000), the teacher’s preparatory work (p = 0.000), the teaching objective (p = 0.000), and the teaching effect (p = 0.000). Furthermore, for the abovementioned five satisfaction attributes, the FC group participants had significantly higher ratings than the LBL group participants.
The time investment in learning between the two groups
As shown in Fig 3, before the class, the students in the FC group invested significantly more time in their studies than those in the LBL group (65.48 ± 11.89 vs. 33.16 ±11.56, t = -15.90, p = 0.000). However, by contrast, after class, the students in the LBL group devoted significantly more time than did those in the FC group (96.38 ± 16.96 vs. 61.94 ± 10.04, t = 14.90, p = 0.000).
Discussion
This study designs and provides a reference for flipped learning to improve teacher-student interaction and students’ learning outcomes. The primary goal of this study was to evaluate the effectiveness of the FC pedagogical model for a pharmaceutical marketing module. Specifically, this exploratory clustered randomized controlled study focused on the status of teacher-student interaction in the FC context, resulting in the following important findings.
In this study, traditional teacher-dominated lectures, during which students traditionally take notes, were replaced by an FC teaching model with highly interactive teacher-student activities. Prerecorded microvideo lectures, the main objective of which was the delivery of knowledge, were viewed independently by students prior to class. The scheduled lecture time was primarily devoted to consolidating student knowledge and developing student skills through in-class activities. Active student-centered learning, through student group presentations and group discussions, increased the opportunities for students to apply their knowledge and skill to pharmaceutical marketing case scenarios in class.
The impacts of the flipped classroom model on teacher-student interaction
Some scholars and educators have argued that frequent and meaningful teacher-student interaction in the classroom within a higher education context could play a critical role in students’ learning and effectively assist their career development [50]. Moreover, a student-centered model underscored by active learning would necessitate class participation and interaction. In recent years, there has been increasing evidence of the promotion roles played by teacher-student interaction in the Chinese cultural context [51–52]. The findings of this study, shown in Table 5, confirm that the teacher held strict control over the verbal behaviors in the LBL classroom. In contrast, some results show that in the FC, student activity prevails due to the large percentage of student talk in the FC group. As shown in Table 5, from the results of the ratios of the students’ positive responses to the students’ passive responses between the two groups, the percentage of student-initiated talk in the FC group far exceeds that in the LBL group. We conclude that the students in the FC group had sufficient opportunities to express their opinions. In the FC group, students expressed their opinions through presentations, initiated conversations between groups and actively directed discussion sections on new topics. In addition, the students spent significant time answering other group members’ questions and discussing issues within groups, which could imply that there was a relatively enthusiastic atmosphere for communication activities among the students. The findings from this educational study could challenge previous conclusions, which claimed that most Chinese college students only took notes, listened carefully and seldom attempted to impose their ideas on other students in class [53–54].
Furthermore, the I/D ratio is very high (1:23.25) in the LBL class, which is another indication of the teacher’s initiative in controlling the activities in the LBL class. Meanwhile, in the FC group, the I/D ratio could be reduced to a lower level (1:4.10). Although this finding with regard to the I/D ratio is still not ideal, we can demonstrate that there was a relatively democratic relationship between the teacher and students in the FC group. These findings can provide a possible way to change the classroom atmosphere in Chinese cultural settings.
Interestingly, the teacher never or seldom showed verbal behaviors of criticizing students or establishing authority (category code = 7) in the two groups, demonstrating that the teacher never criticized students or tried to change students’ activities during the teaching process.
The impacts of the flipped classroom teaching model on student academic performance
The findings from this study also support other educational research that reports improved academic performance associated with FCs [26–28,33]. Notably, the linear regression results provide robust evidence supporting the hypothesis that the FC model has a significant impact on student performance, controlling for the influence of other student demographic variables. In other words, even when controlling for a student’s score on the precourse, GPA and prior grades on performance, the FC intervention is still a predictive factor of the student’s score on the final test.
More specifically, based on the results of the students’ scores, we conclude that the FC model is more effective for achieving academic performance than is the LBL model. The reasons are as follows: In the test in this course, the purpose of the multiple-choice questions was to assess the knowledge points that the students acquired. Simultaneously, we established short-answer questions and essay questions to measure both the students’ knowledge and their application skills. On the one hand, the students in the FC group were better at the short-answer questions and essay questions than their counterparts in the LBL group. On the other hand, there were no significant differences in the scores of the multiple-choice questions between the two groups. This result is extremely important, as it means that the increase in scores was just an increase in performance on the final test items related to the topics of the presentations and discussions in the FC group. In the FC group, each group presented on a particular topic in class. Most of the short-answer questions and essay questions were tightly associated with the issues that the students discussed before or during the class, such as the criteria for market segmentation. As a result, the students in the FC group might have had better insights into the open-ended questions in the exam, which helped them prepare for and answer the questions on the final test. Due to the more desirable learning effects of the FC method, the FC gave students ample opportunities to acquire knowledge and apply pharmaceutical marketing skills. This result is in line with other studies [26, 55–59]. The rationale behind the FC teaching model is to promote effective teacher-student interaction and to stimulate “exploratory, group discussion-based and interactive” teaching activities in the classroom [60]. Chinese teachers should build a good environment for undergraduate students to communicate with one another and establish a positive cooperative relationship among group members to obtain a desirable learning effect.
The impacts of the flipped classroom teaching model on students’ time investment
Compared with the LBL model, the students exposed to the FC model invested more time and energy before the class. Conversely, the students exposed to the LBL method invested more time after class than those in the FC group. The possible reasons are as follows. The students in the FC group usually invested plenty of time and energy in searching for information in network databases, watching microvideos and discussing issues with peers before class. The increased investment in self-learning prior to class could be a key factor in the higher level of understanding achieved by the students in the FC group [56, 59]. Furthermore, they could also benefit from the Q&A sessions with the teacher and from class discussions with peers in the FC. Therefore, due to their familiarity with the teaching contents, the students in the FC group did not need to spend much time and effort on knowledge review after class. However, the students in the LBL group just reviewed the textbook, course materials, PowerPoint files and class notes for self-learning outside of class. Under the traditional LBL model, the lack of enough opportunities for the students to apply their knowledge and skills in the classroom may have been one of the reasons that a serious amount of time was invested in review after class.
The impacts of the flipped classroom teaching model on student attitudes and satisfaction
The findings of the questionnaire survey revealed that the students in the experimental group expressed more positive attitudes toward the FC model. The following could be a possible explanation for this finding. First, as one of the most helpful learning tools, the FC method enabled the students’ individualized learning before and after class. The process of delivering knowledge was set before class in the FC environment; therefore, the students had an opportunity to select their study patterns based on their learning habits. For example, according to their time plan, they could easily follow the online course material, such as the teaching microvideos, anytime and anywhere. Meanwhile, prerecorded lectures could be assigned to the students as homework, leaving classroom time open for interactive learning activities. Subsequent to the students’ independent acquisition of content, engaging in knowledge integration through teacher-student interaction and collaboration with other students is a hallmark of the FC. Conversely, in the LBL environment, the teacher considers the learning status of only the majority of the class, and as a result, the personalized learning needs of the students are neglected. Because different students have different study habits and abilities, student-centered teaching models, such as the FC model, can boost each student’s cognitive abilities by allowing the student to decide his or her learning schedules according to his or her own specific needs. This is a key contributing factor for students to achieve better learning outcomes [56].
Second, the proportion of cognitive learning in the FC group, such as understanding and memorizing outside class, was lower than that in the LBL group. Conversely, the vast majority of learning activities under the FC model were carried out through cooperative learning or problem-based learning; thus, the teaching activities were accomplished through teacher-student interaction and cooperation between peers [26, 57, 61]. The students preferred interactive class time more than in-person lectures. Most of the time in the FC was invested in developing high-level cognitive learning abilities that could stimulate the students’ intellectual abilities to analyze, think and solve problems. Compared with the LBL teaching method, the FC is better able to stimulate the learning interests of students, increase their self-control abilities, and more effectively guide their self-learning outside of class [26, 55,27]. Similarly, the findings of this study agree with those of previous studies [56, 59, 27]. In summary, the satisfaction of individual learning needs, the decrease in cognitive learning activities, and the improvement in one’s intellectual abilities and learning interest are possible reasons why students generally have a higher degree of satisfaction with FCs.
Strengths of this study
The current study is the first to use the FC pedagogical model for the STP strategy chapter of pharmaceutical marketing in China. The use of a previously validated FIAS methodology allows the measurement of teacher-student interaction in a reliable and standardized manner. Furthermore, this FC project discovered that the quantity of teacher-student interaction is a compelling force in improving student performance and satisfaction with the teaching model. Thus, this study can serve as reference for further innovative teaching models and to promote the application of FCs.
Limitations of this study
However, this study has some limitations. First, in this study, only one test was used to examine student performance, which is insufficient to validate the chronicity of the acquisition of knowledge and skills. Therefore, the results should be generalized with caution. Future research should employ controlled studies that objectively repeat the test to identify the possible long-term effects throughout a semester. Second, this study tested only a fraction of course content, limiting the generalizability of the results to the entire pharmaceutical marketing course system. Future studies on the FC are needed to verify the validity of the results claimed in this study by testing large-scale and more in-depth teaching contents. Third, the positive results obtained cannot be attributed solely to the FC pedagogical model. Therefore, a larger sample size and a more rigorous design should be employed to evaluate these results. Fourth, the Hawthorne effect and sample selection bias may also be potential factors that influence student learning [62], which may threaten the reliability of these results. Thus, the results should be generalized with caution. Educators who may apply the FC pedagogical model should be aware of the contextual factors and individual differences regarding experiences and achievements.
Conclusion
The current study provided an innovated concept and framework in pharmacy education in the Chinese cultural context. Compared with LBL methods, the implementation of the FC pedagogical model in this study improved student performance, increased teacher-student interaction and generated positive student attitudes toward the experience. As an effective teaching model, it can also stimulate pharmacy students’ learning interest and enhance their self-learning abilities. Further research is necessary to validate the efficacy of the FC model.
Supporting information
S5 File. Teacher-student interaction data in FC group.
https://doi.org/10.1371/journal.pone.0214624.s005
(XLS)
S6 File. Teacher-student interaction data in LBL group.
https://doi.org/10.1371/journal.pone.0214624.s006
(XLS)
S7 File. Questionnaire data on students attitudes.
https://doi.org/10.1371/journal.pone.0214624.s007
(XLS)
S8 File. Questionnaire data on students satisfaction.
https://doi.org/10.1371/journal.pone.0214624.s008
(XLS)
Acknowledgments
The authors would like to thank all students who participated in this course during the study period. Additionally, we would like to thank the reviewers and editors for their useful comments.
References
- 1. Bernard JS. The flipped classroom: fertile ground for nursing education research. International Journal of Nursing Education Scholarship, 2015, 12(1): 99–109.
- 2. Betihavas V, Bridgman H, Kornhaber R, Cross M. The evidence for ‘flipping out’: a systematic review of the flipped classroom in nursing education. Nurse education today, 2016, 38: 15–21. pmid:26804940
- 3. Joanne CSM, Lateef F. The flipped classroom: Viewpoints in Asian universities. Education in medicine journal, 2014, 6(4).
- 4. O'Flaherty J, Phillips C. The use of flipped classrooms in higher education: A scoping review. The internet and higher education, 2015, 25: 85–95.
- 5. Presti CR. The flipped learning approach in nursing education: A literature review. Journal of Nursing Education, 2016, 55(5): 252–257. pmid:27115451
- 6. Seery MK. Flipped learning in higher education chemistry: emerging trends and potential directions. Chemistry Education Research and Practice, 2015, 16(4): 758–768.
- 7. Zainuddin Z, Halili SH. Flipped classroom research and trends from different fields of study. The International Review of Research in Open and Distributed Learning, 2016, 17(3): 313–340
- 8. Zuber WJ. The flipped classroom, a review of the literature. Industrial and Commercial Training, 2016, 48(2): 97–103.
- 9. Du WQ, Xu W, Ma SP. The investigation of talents requirement of pharmaceutical enterprise and its orientation to employment. China higher medical education. 2004,2:36–39. (In Chinese).
- 10.
The Committee of Higher Pharmacy Education, Ministry of Education Of China P.R. The official national standard of higher pharmacy education. Beijing China: Science Press, 2018. (In Chinese).
- 11. Alkhateeb FM, Clauson KA, Latif DA. Availability and perceived value of masters of business administration degree programs in pharmaceutical marketing and management. American Journal of Pharmaceutical Education. 2012, 76(4): 64. pmid:22611273
- 12. Tam KYB, Heng MA, Jiang GH. What undergraduate students in China say about their professors’teaching. Teaching in Higher Education. 2009;14(2):147–159.
- 13. Li M, Zheng C, Tang X, Sang G. Exploring the nature of teacher-student interaction in small-group discussions in a Chinese university setting. Journal of Computers in Education. 2015, 2(4): 475–491.
- 14.
Hattie J. Visible learning: a synthesis of over 800 meta-analyses relating to achievement. New York: Routledge; 2008.
- 15. Schwerdt G, Wupperman AC. Is traditional teaching really all that bad? A within-student between-subject approach. Economics of Education Review. 2010;30:365–79.
- 16. Mattis KV. Flipped classroom versus traditional textbook instruction: assessing accuracy and mental effort at different levels of mathematical complexity. Technology, Knowledge and Learning. 2015;20:231–48.
- 17. Farley J, Risko EF, Kingstone A. Everyday attention and lecture retention: the effects of time, fidgeting, and mind wandering. Frontiers in psychology.2013;4:619 pmid:24065933
- 18. Meng X, Yang L, Sun H, Du X, Yang B. A Novel Student-centered Teaching Method STQD Using Multiple Teaching Methodologies to Improve Students’ Learning in Pharmacy Education. American Journal of Pharmaceutical Education, 2017: ajpe6505.
- 19. Lin Y, Zhu Y, Chen C, Wang W, Chen T, Li T, et al. Facing the challenges in ophthalmology clerkship teaching: Is flipped classroom the answer? PLoS ONE,2017,12(4): e0174829. pmid:28384167
- 20. Lage M, Platt G, Treglia M. Inverting the classroom: a gateway to creating an inclusive learning environment. The Journal of Economic Education. 2000;31:30–43.
- 21.
Baker JW. The Classroom flip: using web course management tools to become the guide by the side chambers.11th International Conference on College Teaching and Learning, vol. 1. Florida: Florida Community College at Jacksonville; 2000. p. 9–17.
- 22. Eddy SL, Hogan KA. Getting under the hood: how and for whom does increasing course structure work? CBE—Life Sciences Education. 2014;13(3):453–468. pmid:25185229
- 23. McLaughlin JE, Roth MT, Glatt DM, et al. The flipped classroom: a course redesign to foster learning and engagement in a health professions school. Academic Medicine. 2014;89(2):236–243. pmid:24270916
- 24. Persky AM, Pollack GM. Transforming a large-class lecture course to a smaller-group interactive course. American Journal of Pharmaceutical Education. 2010;74(9):Article 170.
- 25. Persky AM, Pollack GM. A modified team-based learning physiology course. American Journal of Pharmaceutical Education. 2011;75(10):Article 204.
- 26. van Vliet EA, Winnips JC, Brouwer N. Flipped-class pedagogy enhances student metacognition and collaborative-learning strategies in higher education but effect does not persist. CBE—Life Sciences Education. 2015;14:26.
- 27. McLaughlin JE, Griffin LM, Esserman DA, et al. Pharmacy student engagement, performance, and perception in a flipped satellite classroom. American Journal of Pharmaceutical Education. 2013;77(9):Article 196.
- 28. McLaughlin JE, Rhoney DH. Comparison of an interactive e-learning preparatory tool and a conventional downloadable handout used within a flipped neurologic pharmacotherapy lecture. Currents in pharmacy teaching and learning. 2015;7(1):12–9.
- 29. Persky AM, Dupuis RE. An eight-year retrospective study in “flipped” pharmacokinetics courses. American Journal of Pharmaceutical Education. 2014;78(10): Article 190.
- 30. Pierce R, Fox J. Vodcasts and active-learning exercises in a “flipped classroom” model of a renal pharmacotherapy module. American Journal of Pharmaceutical Education.2012;76(10):Article 1.
- 31. Bossaer JB, Panus P, Stewart DW, Hagemeier NE, George J. Student performance in a pharmacotherapy oncology module before and after flipping the classroom. American Journal of Pharmaceutical Education. 2016;80:31. pmid:27073284
- 32. Li D, Remedios L, Clarke D. Chinese students’ groupwork practices and experiences in China. Higher Education. 2014, 68(2): 227–241.
- 33. Rui Z, Lian-rui X, Rong-zheng Y, Jing Z, Xue-hong W, Chuan Z. Friend or foe? Flipped classroom for undergraduate electrocardiogram learning: a randomized controlled study. BMC Medical Education. 2017, 17(1): 53. pmid:28270204
- 34. Faul F, Erdfelder E, Buchner A, Lang A-G. Statistical power analyses using G* Power 3.1: Tests for correlation and regression analyses. Behavior research methods. 2009; 41(4):1149±60. pmid:19897823
- 35. Huan C. A Study on Digital Media Technology Courses Teaching Based on Flipped Classroom. American Journal of Educational Research, 2016, 4(3): 264–267.
- 36. Galway LP, Corbett KK, Takaro TK, Tairyan K., Frank E. A novel integration of online and flipped classroom instructional models in public health higher education. BMC medical education, 2014, 14(1): 181. pmid:25169853
- 37. Kong SC. An experience of a three-year study on the development of critical thinking skills in flipped secondary classrooms with pedagogical and technological support. Computers & Education, 2015, 89: 16–31.
- 38.
Kotler P, Kartajaya H, Setiawan I. Marketing 4.0: Moving from traditional to digital. John Wiley & Sons, 2016.
- 39. Sams A, Bergmann J. Flip Your Students’ Learning. Educational Leadership. 2013; 70(6): 16–20.
- 40. Fulton K. Upside down and inside out: Flip Your Classroom to Improve Student Learning. Learning &Leading with Technology. 2012; 39(8): 12–17.
- 41. DeLozier S J, Rhodes M G. Flipped classrooms: a review of key ideas and recommendations for practice. Educational Psychology Review, 2017, 29(1): 141–151.
- 42. Hung H T. Flipping the classroom for English language learners to foster active learning. Computer Assisted Language Learning, 2015, 28(1): 81–96.
- 43. Kim MK, Kim SM, Khera O, Getman J. The experience of three flipped classrooms in an urban university: an exploration of design principles. The Internet and Higher Education, 2014, 22: 37–50.
- 44. Wu JC, Chi SC, Wu CC, Kang Y. Helps from flipped classroom in learning suturing skill: The medical students’ perspective. PloS ONE, 2018, 13(10): e0204698. pmid:30278060
- 45. Flanders NA. Using interaction analysis in the inservice training of teachers. The Journal of Experimental Education. 1962,30(4):313–316.
- 46. Kožić D, Globočnik Žunac A, Bakić-Tomić L. Use of non-verbal communication channels in the classroom. Croatian Journal of Education. 2013; 15(1): 141–153.
- 47. Schempp P, Mccullick B, Pierre PS, Woorons S, You J, Clark B. Expert Golf Instructors' Student-Teacher Interaction Patterns. Research Quarterly for Exercise and Sport. 2004; 75(1):60–70. pmid:15532362
- 48. Sahlberg P, Boce E. Are teachers teaching for a knowledge society? Teachers and Teaching: theory and practice 2010, 16(1): 31–48.
- 49. Theobald R, Freeman S. Is it the intervention or the students? Using linear regression to control for student characteristics in undergraduate STEM education research. CBE—Life Sciences Education, 2014, 13(1): 41–48. pmid:24591502
- 50. Cotten SR, Wilson B. Student-faculty Interactions: Dynamics and Determinants. Higher Education. 2006;51(4):487–519.
- 51. Gu P. Emphysizing research-based teaching and promoting the reform of classroom pedagogies. China Higher Education. 2009;7: 31–38. (In Chinese).
- 52. Sun Q. The transformation of teaching strategy: From monologue to dialogue. Global Education. 2006;35(6), 42–45.
- 53. Charlesworth ZM. Learning Styles across Cultures: Suggestions for Educators. Education + Training. 2008;50(2):115–127.
- 54. Huang JK, Brown K. Cultural Factors Affecting Chinese ESL Students' Academic Learning. Education. 2009; 129:643–653.
- 55. Gross D, Pietri ES, Anderson G, Moyano-Camihort K, Graham MJ. Increased preclass preparation underlies student outcome improvement in the flipped classroom. CBE—Life Sciences Education. 2015;14:ar36. pmid:26396151
- 56. Veeramani R, Madhugiri VS, Chand P. Perception of MBBS students to “flipped class room” approach in neuroanatomy module. Anat Cell Biol. 2015;48:138–43. pmid:26140225
- 57. Munson A, Pierce R. Flipping content to improve student examination performance in a pharmacogenomics course. American Journal of Pharmaceutical Education. 2015;79:103. pmid:27168616
- 58. Sharma N, Lau CS, Doherty I, Harbutt D. How we flipped the medical classroom. Medical Teacher. 2015;37:327–30. pmid:24934251
- 59. Wakabayashi N. Flipped classroom as a strategy to enhance active learning. Kokubyo Gakkai Zasshi, The Journal of the Stomatological Society. 2015;82:1–7.
- 60.
AEI. Outline of China’s national plan for medium and long-term education reform and development. Retrieved May 15, 2013 from Commonwealth of Australia website: http://www.aei.gov.au/news/newsarchive/2010/documents/china_education_reform_pdf.pdf.
- 61. Moraros J, Islam A, Yu S, Banow R, Schindelka B. Flipping for success: evaluating the effectiveness of a novel teaching approach in a graduate level setting. BMC Medical Education. 2015;15:27. pmid:25884508
- 62. Tsai CW, Shen PD, Chiang YC, Lin C. How to solve students’ problems in a flipped classroom: a quasi-experimental approach. Universal Access in the Information Society. 2017, 16(1): 225–233.