Instructors and courses.

This research is a retrospective study of the teaching methods used by faculty at a major Research 1 university in the Northwest. This study examined 33 biology faculty and 10 different undergraduate biology courses at over a period of four academic years. At this university, large-enrollment classes are recorded using lecture capture technology installed in the classrooms and these recordings are archived. Faculty were not involved any formal professional development and were not specifically trained in the use of PORTAAL practices. Faculty had varied levels of experience using active learning, but information about experience was not collected.

There were four lower-division courses, each of which were offered multiple times across the four years, and six upper-division courses, each offered only once. Three of the lower division courses were either team-taught or taught by a single instructor depending on the offering; the other lower-division course and all upper-division courses were taught by a single instructor. When the course was taught by two instructors, each instructor taught half of the ten-week quarter and gave half of the exams. As the instructors, exams, Bloom’s level of exams, and students’ exam performance were different in each section of a team-taught course, each section was treated as independent and became a “unit” of analysis. The solo-taught courses were each their own unit. In total, we had 40 units of lower-division courses and 6 units of upper-division courses to analyze. These 46 units were taught by 33 faculty, as some instructors taught two units. Instructor and course information is described in S1 Table and Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Instructor and course information.

https://doi.org/10.1371/journal.pone.0260789.t001

Students.

There were 14,956 student data points across all courses (Table 1). Introductory courses ranged from 200 to 700 students, whereas upper-division courses ranged from 24 to 120 students. The registrar provided students’ demographic information, which included self-identified binary gender, grade point average (GPA) at the start of the term, participation in the educational opportunity program (EOP) for students identified by the university as economically or educationally disadvantaged, and whether the students self-identified as belonging to a race/ethnicity that is traditionally underrepresented in science (underrepresented minority or URM: e.g., Hispanic, Black/African-American, Pacific Islander, Native Hawaiian, Native American).

Across students in our sample, 58.9% of students (n = 8,808) self-identified as female, 9.6% of students (n = 1,436) self-identified as a race/ethnicity underrepresented in science (URM), and 16.6% of students (n = 2,490) were classified as eligible for the university’s educational opportunity program (EOP). For our analysis, we will use eligibility for the EOP as a proxy for socioeconomic status. This research was done under the approval of the University of Washington IRB protocols 38945 and STUDY00002830. Verbal consent was obtained from participants when required by the IRB protocol.

Classroom observation data.

We randomly selected three to four archived videos from each of the 46 units for a total of 150 videos (S2 and S3 Tables). To select which days to include in this study, the 10-week quarter was divided into four equal time periods excluding the first and last week of the quarter, and one day was randomly selected from each time period. For team taught courses, three videos (except in one instance when four were coded) were randomly selected from weeks two through five for the first instructor and six through nine for the second instructor. Based on a Stains and colleagues [41] study that suggested that at least four observations are necessary for reliable characterization, we coded four classroom videos from all units of analysis collected after the publication of that study.

All videos were coded by trained researchers using the classroom observation tool, PORTAAL [24]. The PORTAAL coding rubric is found in (S1 Fig). Pairs of researchers independently coded each video of the entire class session before they met to reconcile differences and come to complete consensus. Due to the fine-grained nature of the PORTAAL rubric and that every second of a class session is given at least one code, there is no single PORTAAL score for each class session. Therefore, we were not able to calculate inter-rater reliability between coders. The most common coding disagreements were Bloom’s level of the activity, time differences of a few seconds when recording the start and end of an iteration or activity, organizing the activity into a new iteration of the same activity or a new activity, or tallying the number students who responded during debrief if student voices were difficult to hear. Coders discussed these disagreements and came to complete consensus on PORTAAL values for each video. Prior to the statistical analysis, we determined an average value for each PORTAAL practice for a unit of analysis by totaling the number or duration of each PORTAAL practice across all videos for each unit of analysis and dividing by the number of videos. The average values for each PORTAAL practice were then standardized to a 50-minute period which is the length of most class sessions at this institution.

Exam data.

Most courses had three or four exams per quarter. In team-taught courses, each instructor usually gave two exams. Exams were in multiple formats, including but not limited to the following: multiple-choice, fill-in-the-blank, true/false, and constructed response questions. We collected exam questions with keys that indicated the point value of each question. We also collected the total exam score for each exam for each student. As each unit was taught by a different instructor and each instructor had exams worth a varying number of points, students’ exam scores for each exam were normalized to 100 points. The normalized exam scores for each student were totaled and averaged to create an overall exam score for each student. As this is a retrospective study, instructors could not change the composition of exams in these courses to alter student academic performance.

Small group activities.

We found that use of small group work was positively correlated with student exam performance across all units of analysis when controlling for Bloom’s level of exams. We coded a ‘small group’ activity when students were asked to work with one or more peers to discuss a prompt, whether it was a clicker question, a formal worksheet, or an impromptu question generated by the instructor. The ICAP framework suggests that class activities that provide students with the opportunity to be Interactive and Constructive are effective at improving learning [26], and our results provide empirical support of that claim. Chi and Wylie [26] categorize small group activities as Interactive as this practice provides students with the opportunity to discuss and defend their answers with peers. Similarly, Andrews and colleagues [51] hypothesized that effective learning activities require students to generate their own understanding which is the goal of most small group work. Our finding is aligned with many previous studies that demonstrate the improvement in learning that students gain from working in small groups [52–56]. Small group work may be an added benefit in classes that have more academically challenging exams as the small group work may give students the additional practice with and peer feedback on explaining and developing the logic and reasoning skills needed for increased performance on these types of exams.

While the use of small group activities was positively correlated with all students’ academic performance, the model predicted that this PORTAAL practice offered a small additional benefit to female students in courses with exams at lower Bloom’s levels. Small group work supports greater social cohesion between members of a large lecture class [57]. This cohesion can provide a safe learning environment which may contribute to female students’ willingness to engage in learning activities [58]. This additional benefit to women may contribute, even in a small way, to eliminating the documented gender gap in student academic performance [38, 59]. We were surprised not to detect a similar mediated effect at higher Bloom’s levels, but this may be due to a limited amount of variation in our sample or the effects of other PORTAAL practices on exams at higher Bloom’s levels.

Random call.

Across all units of analysis, randomly calling on students to answer a question was positively correlated with student exam performance when controlling for Bloom’s level of exams. We coded ‘random call answers’ when a student or group of students gave an answer to a question when called on by the instructor using a randomly generated list of names, seat numbers, or group names. Using random calls to solicit answers from students can be a form of accountability [24]. In classrooms where instructors regularly call on volunteers, some students may put less effort into answering questions if they are accustomed to other students volunteering an answer. When students know that their name or their groups’ name may be called on to answer, they put more effort into solving the posed question [60]. Research has also found that regular use of random call can increase all students’ frequency of voluntary participation and comfort with participating in class discussions [61]. As student volunteers also tend to be male [59, 62], this leads to a bias in student voices heard in the classroom. Using random call can therefore enhance equity and inclusion as all students have an equal chance of being called on.

However, using random call was significantly negatively correlated with student performance when exams were at lower Bloom’s levels. Research has shown that random call can induce some anxiety in students, which may decrease students’ sense of safety in the classroom and negatively impact exam performance [63, 64]. There are multiple ways to implement random call that may alleviate the anxiety [64] and instructors need to determine the procedure that best matches the characteristics of their student population. As instructors in these courses had a low incidence of random call (mean = 0.38 times per class, SD = 1.55), it is possible that students did not have enough opportunities to become comfortable sharing their answers with the whole class, and instead felt anxious about random call. However, this result could also be due to not enough variation in our sample of courses with low Bloom’s exams to detect an accurate effect of random calling on exam scores.

When pooling the data across all units of analysis, random call had a positive correlation with exam performance, which may be explained by a higher incidence of random call (mean = 1.49) and a larger variation (SD = 3.16). The more frequent use of random call across all courses may have contributed to students’ comfort level with this PORTAAL practice which may help to reduce their anxiety. This could have allowed students to use more of their working memory to learn course material rather than dealing with feelings of anxiety.

Explaining alternative answers.

Providing opportunities for students to explain alternative answers to in-class questions was correlated with increases in exam scores across the three SEM analyses. We defined and used the code ‘alternative answers’ each time the instructor or student explained why incorrect or partially incorrect answers were incorrect, or if they provided multiple correct answers to a problem. Explaining alternative answers provides students with the opportunity to compare and contrast other possible answers to a question. This often happens when explaining why the distractors in a multiple-choice option are incorrect or when discussing an open-ended question that has multiple correct answers. This practice highlights for students key distinguishing features of concepts [65] and is categorized as a type of Constructive activity in the ICAP framework [26]. Therefore, identifying incorrect answers during class may provide students with valuable test-taking skills and foster deeper conceptual understanding that can be used effectively on exams.

Opportunity for students to work alone.

Allowing students to work alone prior to answering a question had opposite associations with exam scores depending on the Bloom’s level of exams, positive for high Bloom’s exams and negative for exams at low Bloom’s levels. High Bloom’s questions are cognitively demanding as they ask students to analyze, synthesize, or evaluate course topics which requires focused attention. In a study on how the activity of working alone on an in-class question impacted their experiences, students reported that they felt it was very important to form their own opinion without the influence of others prior to answering a question [66]. Students also indicate that having the opportunity to collect and formulate their thoughts prior to peer interaction allowed them to have a more fruitful discussion as they had more to contribute [66]. These results imply that working alone allows students the time to generate and formulate their thinking on the question and therefore is a type of constructive engagement with the material. Furthermore, reinforcing this type of cognitive behavior during class time could carry over to exams and encourage students to collect and organize their thoughts prior to answering.

Courses that test at lower Bloom’s levels of recall, comprehension, and rote application often use more multiple-choice, matching, or fill-in the blank type questions [35] where students are unable to reason their way to the correct answer if they have not already memorized the required information. Research [67] indicates that students adapt their study strategies to meet but not exceed the challenge level of the exam. In other words, exams drive student learning. It is possible that in courses that test at lower Bloom’s levels, students may not benefit from reasoning through the problem on their own as the answer is usually a factual piece of information that they either know or they do not know. In this case, taking time to work alone may cause more frustration if students are unable to answer the question on their own. We acknowledge that working alone should benefit students at all Bloom’s levels of exams, and further research should investigate if there are components of this practice that could be associated with decreased performance on low Bloom’s exams.

Total student time in class.

Across all units of analysis, the total amount of time students were actively engaged in course material was positively correlated with exam performance when controlling for Bloom’s level. Student time was recorded as any time during the class in which the instructor is not actively lecturing, giving instructions, or explaining the answer to a question. Total student time includes time that students are working individually, with a group of peers, or debriefing the answer to a question to the whole class.

‘Student time’ is the broadest of the codes that PORTAAL defines. It can be interpreted as a general measure of how student-centered versus instructor-centered the classroom is. Based on the findings of numerous studies showing the positive value of transforming STEM courses from traditional passive lecture to higher student engagement, The National Science Foundation’s Vision and Change report [11] created a list of action items for faculty to undertake. Four of the eight action items included the following: engage students as active participants and not passive recipients in all undergraduate biology courses, use multiple modes of instruction in addition to traditional lecture, facilitate student learning within a cooperative context, and give students ongoing, frequent, and multiple forms of feedback on their progress. Our results provide empirical support for these suggested action items as we found that it may be beneficial for instructors to increase the amount of time students are actively working on course material and decrease the amount of time they are delivering content. This does not mean that instructors give up their role in the classroom, but rather that they put their energy into designing class activities that provide students with the opportunity to create and deepen their understanding. This increased student engagement with course material may be achieved by incorporating more of the PORTAAL practices more often in their course. It may be encouraging for instructors to realize that implementing any of the PORTAAL practices they feel comfortable using will increase total student time in learning activities which has a positive correlation with exam performance.

Performance on cognitively challenging exams.

In classes with exams at higher Bloom’s levels, two other PORTAAL practices correlated with improved exam scores: students explaining the logic underpinning answers and receiving positive feedback from the instructor. Students ‘explaining their answers’ were coded when a student or group of students provided reasoning behind their answer in front of the whole class. ‘Positive feedback’ from the instructor was coded any time the instructor used affirming language to praise the work of an individual student or the whole class.

Reasoning ability is crucial to answering cognitively challenging exam questions, and the more opportunities a student gets to practice this skill under the guidance of the instructor, the more likely the student will succeed on the exam. By having students explain the logic underlying their answers to the whole class while debriefing the question, instructors are reaffirming the necessity of generating and articulating a more sophisticated understanding of the material. In related findings, Knight et al. [68] found that when the instructor in an upper-division Biology course prompted students to explain their reasoning, more reasoning was noted in transcripts of peer discussions and during report outs. These student groups also more often arrived at the correct answer.

Providing positive feedback to students also creates a supportive climate and is observed more often in high-achieving classes [69]. Similarly, teacher confirmation of student work has been shown to increase student participation in class and greater self-reported use of study behaviors associated with cognitive learning [70]. Collectively, the additional practice of using logic to answer in-class questions and the supportive climate created by positive feedback may contribute to improved performance on cognitively challenging exams.

Practical implementation of the PORTAAL practices.

A convenient way for instructors to incorporate the PORTAAL practices the model predicted to significantly increase exam scores is by using in-class questions that students answer using a personal response system (i.e., clickers or Poll Everywhere) [71–73]. After posing the question, the instructor allows students a short period to work alone on the question and then enter their answer on their voting device. Without showing the voting results, the instructor asks the students to work in a small group with nearby peers to discuss and defend their answer. After a period of time, students re-enter their final answer. When debriefing the answer, the instructor randomly calls on a student or group of students to provide their answer and the reasoning underlying their answer. The instructor can also call on students to explain alternative answers. Activities like these can increase the amount of student time during class by allowing students to explore the problem on their own, with their peers, and to finally share their answer and reasoning with the whole class. Following this strategy could help faculty feel more comfortable and confident implementing PORTAAL practices.