Complex multiple-choice questions

The CMC question format is one in which a question is presented with a stem, a set of primary responses, and a set of secondary choices that contain various combinations of the primary responses [12]. Multiple-choice questions that do not follow this format, even if they require a student to select multiple answer choices, are not considered CMC questions for the purposes of this paper. Further, in this paper, we do not differentiate between CMC questions and type-K questions, which are a specific subtype of CMC questions with four primary responses and five of the possible sixteen responses displayed as secondary choices [12].

CMC questions are often used instead of more traditional multiple-choice questions due to the assumed benefits they provide. For example, the higher complexity of the question is assumed to enable assessment of greater complexity in thinking [13]. Given that traditional multiple-choice questions may hinder critical thinking skills in introductory science courses [14], CMC questions could provide a remedy while retaining the benefits of multiple-choice questions such as being easy to grade and implement in large-enrollment courses.

In addition, the CMC format can identify when students believe more than one response is true. A study in biology found that as many as half of students who correctly answered a traditional multiple-choice question would have also (incorrectly) selected another answer choice if given the option [15]. While that study was done with multiple-true-false questions instead of CMC questions, it suggests that traditional multiple-choice questions might not be able to capture the partial understandings students might hold. Further, the CMC format can also be useful for assessing knowledge in cases when there is more than one correct answer.

Despite the proposed benefits of this type of question, CMC questions also have disadvantages relative to traditional multiple-choice questions. First, studies in educational assessment, mathematics, and medicine have found that CMCs are harder for students than traditional multiple-choice questions [12,16–18], though the effect may depend on whether the question requires the recall of memorized information or higher-order thinking skills [19]. As a result, some have claimed that the reason for the difference in performance could be because CMCs are measuring more than just knowledge such as non-cognitive traits [17]. For example, after reformatting CMC questions as multiple-true-false questions, one study found that student performance improved [17].

Second, CMCs may inadvertently “clue” students to the correct answer [12] because not all combinations of primary responses can be present in the secondary choices if there are five or fewer answer choices (a common practice on exams). For example, in Fig 1, knowing that response 2 is incorrect eliminates answers B, C, and E. Of the remaining options, both have response 1. Therefore, a student is potentially able to get the question correct only by knowing whether responses 2 and 3 are true or false. This clueing could then help less knowledgeable students perform better on CMC questions than they would on comparable traditional multiple-choice questions [12].

Third, CMC questions might be inequitable, though the evidence is limited. One study of undergraduates in a teaching certification program found that some, but not all CMC questions in their sample showed performance gaps between male and female students [20].

Finally, there are test construction concerns about CMC questions. CMC questions have been shown to have lower reliability than traditional multiple-choice questions or multiple-true-false questions [10]. In addition, CMC questions can be more difficult to construct [10], although others disagree and believe they are easier to construct because they require fewer distractor answers than traditional questions [21].

Given the number of concerns about CMC questions, many have recommended against using the format [12,13,22]. Yet, recent work suggests that this format is still in use [23–26].

Complex multiple-choice questions in physics

There has been limited study of the CMC question format in physics. One study [27] found that students performed better on CMC questions than on equivalent versions in the multiple-true-false format with approximately one-third of students answering the multiple-true-false questions in a way that was not represented among the secondary choices in the corresponding CMC format. The study concluded that the CMC format may overestimate student knowledge.

Despite the limited direct study of CMC questions in physics, there have been many indirect studies due to their appearance on many concept inventories including the Test of Understanding of Graphs in Kinematics (items 12 and 19) [28], the Force Concept Inventory (items 12 and 22 on the original 1992 version; items 5, 18, 29, and 30 on the 1995 revision) [29], and the Physics Inventory of Quantitative Literacy (item 4) [30]. In the case of the 1995 version of the Force Concept Inventory, various studies have commented on the CMC items, noting that items 5 and 18 were able to distinguish between high- and low-proficiency students [31]; items 5, 18, and 29 are “problematic” [32]; and item 29 is “potentially malfunctioning for currently unknown reasons” [33]. Thinking toward the next generation of concept inventories in physics (e.g., Laverty and Caballero [34]), it is important to consider whether CMC-type questions should be avoided on these types of assessments.

CMC questions in physics have also appeared in studies of in-class interventions [35], of K-12 contexts [36], of teacher knowledge [37], and student learning [26]. Outside peer-reviewed studies and conceptual inventories, CMC questions have further appeared on the American Association of Physics Teachers’ F = ma exam, the first qualifier for the US Physics Team in the International Physics Olympiad, and on the physics Graduate Record Examination (GRE) which, until recently, was required for most applicants to physics graduate programs [38].

Data collection

Data for this study comes from Problem Roulette users enrolled in a second-semester calculus-based introductory physics course (referred to as Physics II herein) between the Winter 2019 and Winter 2023 semesters (13 semesters total, including summer semesters which enroll significantly fewer students than Fall and Winter offerings). Problem Roulette is a free, optional, not-for-credit, test question practice software offered to all students in introductory physics courses (and other introductory STEM courses) at the University of Michigan [39]. Students have the option to select a broad physics topic or specific exam period (e.g., Exam 1) and then are randomly served practice questions from the system. Students are given as many attempts as they need to answer the question and are told whether they are correct or not after each response. All questions are uploaded by instructors and appeared on exams during previous course iterations. Prior work has found that using this system regularly results in a significant increase in a student’s final grade in introductory physics over and above what might be expected based on their prior academic preparation (i.e., grades in their other college courses and ACT/SAT scores) [39,40]. For the time period in this study, 1685 students answered at least one Physics II question in Problem Roulette, corresponding to 41% of the students enrolled in the course and 39% of students who completed the course during the study period.

Data for this study also came from a simplified version of the university’s student data warehouse intended to provide researchers with straightforward access to student data [41]. Specifically, we had access to students’ course grades, birth sex, race/ethnicity, high school zip code, parental education status, and residency status.

This study was reviewed by the Institutional Review Board at the University of Michigan and determined to be exempt (HUM00158291). The University of Michigan owns the Problem Roulette software and the data used in this study. The data used were not collected specifically for research and, as such, are secondary data in this study. The terms of service users agree to before using Problem Roulette allow the university to perform research related to evaluating the performance of users. In addition, the Family Education Rights and Privacy Act (FERPA) allows student educational records to be used for research without consent if the goal of the research is to improve instruction. Therefore, the typical requirement to obtain active consent from participants was waived and not sought by the researchers.

To protect student privacy, potentially identifying information was stored separately from student responses and grades. Student responses and grades were assigned random numbers via Problem Roulette and the university when creating the data files. A linking file was then used to connect the two data sources and match students in each database. The connection was done via R [42] rather than manually so that no member of the research team could potentially associate a random identifier with potentially identifying information. After students were matched, the potentially identifying information in the linking file was removed from the resulting data set.

Demographic definitions

Noting that demographic categories are neither natural nor given [43], in this section, we discuss how we defined each demographic group. As the demographic data is collected by the university and not directly by us as researchers, we had no control over the options students were able to select from and acknowledge that they are not the only axes on which privilege and oppression act in physics. The five demographic categories were sex, race/ethnicity, socioeconomic status, parental education, and international status. We provide the definitions and criteria for creating groups in each of these categories below and then state why we choose to include these demographic variables.

• Sex: Sex was recorded as a binary variable (male or female) even though sex is not binary [44] and gender is actually the construct of greater interest.
• Race/ethnicity: Students were able to select any of the following five options: Asian, Black, Hispanic, Native American, and white. Any student who selected more than one option was marked as Multiracial. Due to the limited numbers of Black, Hispanic, Multiracial, and Native American students in our study population, we grouped these students into one category and we grouped Asian and white students into a second category. Given that the underrepresented minority label is considered racist and harmful [45–47], we instead refer to these groups as “B/H/M/N” and “A/w”, using the first letter of each race and ethnicity recorded by the university, following the capitalization used by Robertson et al. in a recent physics education article[48]. We acknowledge that such groupings ignore the unique situations of each group and further mask the experiences of individuals [45,49,50].
• Socioeconomic status: We defined a student as low-income with a binary variable where a student was considered low-income if the median annual household income of the student’s high school’s zip code was less than $60,000. We determined median annual household income by zip code using the 2020 American Community Survey 5-Year Estimates. We used this metric rather than the student’s self-reported estimated family income—another socioeconomic metric available to us—due to a large amount (more than 20%) of missing data and a lack of access to data about which students received Pell Grants, a common choice for denoting low-income students in education research. We also use this community-based definition of low income rather than an individually-based definition to acknowledge that educational resources students have access to are not solely based on their family but also their community. Therefore, a low-income student based on family income may have had access to different opportunities if they attend a well-resourced school compared to a low-income student based on family income who attends a less-resourced school. Students without a zip code listed (such as international students) were conservatively coded as not low-income.
• Parental education: We defined first-generation status as a binary variable, where any student with neither parent earning at least a bachelor’s degree was marked as a first-generation student. Students could select their parents’ education level as one of eleven levels. We grouped elementary school only, less than high school, high school diploma, some college, Associate’s degree, and nursing diploma into the first generation category, and we grouped Bachelor’s degree, Master’s degree, Professional Doctorate, Doctorate, and Post Doctorate into the continuing generation category.
• International status: International student status was based on whether the university considers the student to be international. The university from which the data come classifies a student as international if the student has a citizenship status of Non-Resident Alien or Alien Under Tax Treaty.

Our reasons for including these variables are as follows. We include sex and race/ethnicity as these are common demographic variables studied in physics education when examining equity issues. We include socioeconomic status and first-generation status as we assume that middle- and upper-class students and students with college-educated parents may have a greater understanding of the hidden curriculum of college, which may include how to approach and answer exam questions presented in non-standard formats such as CMC questions. Finally, we include international status under the assumption that most international students do not speak English as their first language, and the additional cognitive load of CMC questions could be especially impactful for students who are not native English speakers.

Analysis

Here, we describe our procedures for analyzing the data. The various analyses are summarized in Fig 2 and are described below.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Summary of analysis methods used in this study.

Light grey boxes represent the categories of analysis while white boxes represent the variables included at each stage of the analysis.

https://doi.org/10.1371/journal.pone.0323813.g002

Data cleaning and handling.

We first matched all responses to Physics II questions in Problem Roulette to the university’s enrollment records and removed responses from students who were not enrolled in the course and any responses that could not be matched to an enrollment record. We also removed responses that were blank and hence not marked as correct or incorrect by the system, which happens when the user is delivered a question and then exits the platform without answering it (presumably because they are done studying for the time being) as well as any question that a student chose to skip. Finally, we removed responses that were marked as “retry” so that our data set only contained the student’s first response to each question, excluding any responses provided after students received feedback and mimicking more typical testing environments in which students respond to each test item only once. We explicitly note this does not make the setting equivalent to that of a testing environment and that the data are still from a low-stakes, not-for-credit environment.

We were left with 72,611 responses to 951 questions from 1,685 students, where students and questions were crossed given that each student could theoretically respond to each question. Because questions are served randomly to students, questions have different numbers of responses. In our data set, the most encountered question had 204 responses, the least encountered question had 1 response and the median number of responses was 82. Random effects modeling was used to account for this imbalance [51,52] and is described later in this section. The cumulative distribution of responses is shown in Fig 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Cumulative distribution of responses to each question in Problem Roulette for Physics II.

The dotted line corresponds to the median number of responses, which is 82 for this data set.

https://doi.org/10.1371/journal.pone.0323813.g003

Mixed effects models.

Finally, to gain further understanding of any effects observed, we ran a series of mixed effects logistic regression models. We adopt an explanatory modeling approach in that we assume that the variables in our model relate to an underlying set of factors that cause the outcome and our goal is to minimize bias to obtain the most accurate description of the underlying theory [55]. We first ran five bivariate regression models, using one of each of the five demographic variables as a binary variable, question type (CMC or non-CMC), and an interaction term as inputs and whether the question was answered correctly (yes or no) as an output. For each of these models, we included the term the student was enrolled, the question ID, and the student as random intercepts to account for possible baseline performance differences based on the term, the individual question, and the individual student. We also included a random slope based on the student and question type because the relationship between performance on CMC and non-CMC questions might not be the same for all students. If there was a differential effect between the demographic or student groups, the interaction term would be statistically significant. We ran another two bivariate models, one with GPAO (as a continuous variable) and one with ACT score, and question type, an interaction term, and the same random intercepts and slopes as before.

To account for how some of the variables may be measuring similar effects, we ran multivariate logistic regression models including subsets of all five demographic variables, GPAO, ACT score, and all interactions between these variables and question type. These multivariate models also included the random slope and intercepts. As the demographic variables along with GPAO, ACT score, and their interactions adds 14 variables to the model and this can make the model difficult to interpret, we performed a model selection procedure to simplify our final model. We began with including only question type in the model and then adding one additional variable (e.g., sex, socioeconomic status, or GPAO) and its interaction with question type to the model, resulting in seven total models. Variables whose interaction with question type were statistically significant in the bivariate models were added first followed by variables that were statistically significant in the bivariate model, and then finally, variables that were not statistically significant.

To compare the models, we then used a likelihood ratio test [56]. In a likelihood test, the likelihood (or log-likelihood) of two models is compared with the null hypothesis that the simpler model (i.e., the one with fewer variables) is a better fit of the data. If the resulting p-value is less than 0.05, there is evidence that the more complex model is a better fit for the data. Using this approach, we are then able to select the model with the fewest terms, balancing the need to explain the underlying data but also be interpretable by researchers. Any data with missing values in any of the variables considered for the models were dropped prior to model construction.

Comparing the results across the bivariate and multivariate models, there are four possible patterns. If a variable has the same relationship with the outcome variable in both the bivariate and multivariate models and it is statistically significant, that variable likely has explanatory power. In contrast, if the variable is not statistically significant, the variable likely does not have explanatory power. If the variable is statistically significant in one of the models but not the other, there are an additional two cases. If the variable was statistically significant in the bivariate model but not the multivariate model, the variable is likely a proxy and likely correlated with another input variable. Alternatively, if the variable is statistically significant in the multivariate model but not the bivariate model (or if the variable is statistically significant in both models but the direction of the effect is different), the variable is a suppressor variable [57]. In this paper, we are especially interested in determining variables with explanatory power and less interested in understanding variables with proxy or suppressor effects. For the purposes of achieving this goal, if a variable is not present in the final selected model, we will consider it a non-significant variable since its inclusion in the model did not result in a model that was better than the model without it.

We performed all analysis in R [42] and used the lme4 package to run the nested models [58]. Following best practices of Kanim and Cid [59] and Young and Caballero [60], the percentage of responses attributed to each variable in our models are shown in Table 1. A correlogram of the data is shown in Fig 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Correlogram of the independent variables in our regression model with each other and the outcome variable, correct.

Sex, race, parental education, socioeconomic status, and international status are binary variables with female, B/H/M/N, first generation, low income, and international coded as one.

https://doi.org/10.1371/journal.pone.0323813.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Percent of responses to questions by variables in the regression models by category as well as the amount of missing data.

https://doi.org/10.1371/journal.pone.0323813.t001

Across these models, continuous variables were scaled and centered as appropriate. We centered GPAO and ACT scores based on the average GPAO and ACT among the students in the data set as a GPAO or ACT of zero is highly unlikely to occur in practice. Under this transformation, the intercept corresponds to a student with an average GPAO and average ACT score in this sample (3.49 and 32.75, respectively).

In addition, any cases with missing data were removed. For these models, there were 64,383 usable responses in 950 questions and 1,493 students, where questions and students are crossed, across 13 terms.

Descriptive statistics

Examining overall performance, we found that students performed worse on CMC questions compared to non-CMC questions. Students submitted 2,994 responses to CMC questions with an accuracy rate of 44.1% (95% confidence interval: 42.3%, 45.8%) compared to 69,617 responses to non-CMC questions with an accuracy rate of 51.9% (95% CI: 51.5%, 52.3%). That is, the rate at which students correctly answered CMC questions was 7.9 (95% CI: 6.1, 9.7) percentage points lower than that at which they correctly answered non-CMC questions. Despite this apparent increase in difficulty, we found minimal evidence that students skipped CMC questions more than non-CMC questions. There were 149 skipped CMC questions (4.74% of CMC questions served to students) and there were 2,784 skipped non-CMC questions (3.84% of non-CMC questions served to students).

To understand whether this performance difference was due to the format or content of the question (conceptual vs. “plug-and-chug”), we compared student accuracy on a subset of non-CMC questions. We found that 52.5% of responses to “plug-and-chug” non-CMC questions were correct (95% confidence interval: 51.3%, 53.7%) while 48.9% of responses to conceptual non-CMC questions were correct (95% CI: 47.4%, 50.4%).

That is, students performed 4.82 (95% CI 2.50, 7.14) percentage points better on non-CMC conceptual questions than they did on CMC conceptual questions.

The results of taking prior performance and grades earned in Physics II into account when answering CMC and non-CMC questions are shown in Fig 5. Looking at prior performance first (Fig 5A), we notice that regardless of a student’s grades in their other courses, students correctly answered non-CMC questions at a higher rate than CMC questions. However, students who typically earn “A”s in their other courses correctly answered CMC questions at a higher rate than students who typically earned less than “A”s in their other courses. Looking at the discrepancy in performance on the question types between students who earned “A”s in their other courses and those who did not, we find the gap to be about the same size as previously described—around 8 percentage points.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Percent of CMC and non-CMC questions answered correctly based on the average grade students earned in their other classes before or while enrolled in Physics II (panel A) and grade earned in Physics II (panel B).

Numbers above each data point are the number of responses from that grade group to that type of question and the error bars are 95% confidence intervals. Both groups of students do equally worse on CMC questions.

https://doi.org/10.1371/journal.pone.0323813.g005

Looking specifically at the grades students earned in Physics II, we find a similar result (Fig 5B). Students who earned “A”s in Physics II correctly responded to non-CMC questions more often than they did CMC questions and correctly answered CMC questions more often than students who didn’t earn “A”s in Physics II correctly responded to non-CMC questions. Looking at the difference in performance between the two question types based on the student’s Physics II grade, we notice that the gap is slightly smaller for students who earned “A”s compared to those who did not (6.4 vs. 8.9 points).

We then considered prior performance through the lens of standardized test scores. The result is shown in Fig 6 where the ACT score groups were chosen to correspond to roughly a quarter of students. We find that regardless of ACT score, students performed better on non-CMC questions. Notably, while performance on non-CMC questions tended to increase with ACT score, performance on CMC questions was largely unchanged.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Percent of CMC and non-CMC questions answered correctly based on student’s ACT score.

Higher ACT scores are represented in green while lower ACT scores are represented in purple. Each category corresponds to around a quarter of students in our data set. Numbers above each data point refer to the number of responses from that group to that type of question and the error bars are 95% confidence intervals. Students tend to do better on non-CMC questions compared to CMC questions.

https://doi.org/10.1371/journal.pone.0323813.g006

Next, we examined performance on the CMC and non-CMC questions from the perspective of differentiating students within a course. That is, given a student who answered a CMC or non-CMC question correctly, what was the probability that the student would earn an “A” in Physics II? If CMC questions are better at discriminating between “A” students and other students, we would expect that answering CMC questions correctly would lead to a higher probability of earning an “A”. However, we find that there is no apparent relationship between students’ rate of accuracy on CMC and non-CMC questions and their probability of earning an “A” in Physics II (Table 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Probability and 95% confidence intervals that a student who answered each type of multiple-choice question correctly would go on to earn an “A” in Physics II. We find that the probabilities are not statistically different from each other.

https://doi.org/10.1371/journal.pone.0323813.t002

We then looked at how various demographic groups performed on CMC and non-CMC questions (Fig 7). All groups performed better on the non-CMC questions, but the size of the performance gap varied between the groups. For example, the CMC-non-CMC performance gap was smallest for international students (4.6%) and largest for Black, Hispanic, Multiracial, and Native American students and female students (11% and 12% respectively). In general, we find that minoritized students have a larger performance gap between the two types of questions than their majoritized peers do, with the exception being first-generation status where the performance gap is 0.12 percentage points. The difference in the performance gap when minoritized students perform worse on CMC questions varies between 0.7% and 5.6%, with the smallest corresponding to the difference in performance gap between low-income and medium/high-income students and the largest corresponding to the difference in performance between male and female students. Practically, this result means that if male students on average had a performance penalty of percentage points on CMC questions compared to non-CMC questions, a female student on average would have a performance gap of percentage points.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Comparison of the percentage of correct responses to CMC and non-CMC questions by Physics II students split by demographic groups.

Error bars on each point represent 95% confidence intervals and the number above each point is the number of responses by that demographic group to that question type. We find all demographic groups perform worse on CMC questions than they perform on non-CMC questions.

https://doi.org/10.1371/journal.pone.0323813.g007

Finally, we compared individual student performance on the two types of questions (Fig 8). Of the students who saw at least five of each question type, most students (65.8%) answered a higher fraction of the non-CMC questions correctly compared to the CMC questions. Requiring a practically significant difference of at least ten percentage points, the equivalent of a full letter grade, we find that 44% of students performed better on the non-CMC questions, 15% performed better on the CMC questions, and 41% performed the same or within the 10 percentage point threshold (marked by the grey region in Fig 8).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Comparison of student accuracy on non-CMC questions compared to CMC questions among students who answered at least five of each question type.

The diagonal line denotes equal performance on the two types of questions. The gray region denotes where performance on the two types of questions are within ten percentage points of each other, or the equivalent of a letter grade on the standard scale where 90 to 100 equates to an A grade. The edge histograms show the relative number of students in each 10% band of accuracy. In general, students are less accurate at answering CMC than non-CMC questions, shown by there being more dots below the line of parity.

https://doi.org/10.1371/journal.pone.0323813.g008

For interested readers, additional plots with results split by demographics are included in S4 Appendix. Given limited sample sizes and that the results are not central to our argument, we do not discuss those results here.

Regression models

To assess if those differences in performance gaps were significant, we developed regression models. As the bivariate models inform our interpretation of the multivariate model, we present those results first followed by the multivariate results. The bivariate results are shown in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Odds ratios for each regression model for answering a question correctly. *** p < .001, ** p < .01, and * p < .05. Numbers in parentheses represent 95% confidence intervals. The sex-question-type interaction is the only interaction found to be statistically significant.

https://doi.org/10.1371/journal.pone.0323813.t003

First, we notice that in most models, the question type variable is statistically significant and the odds ratio is less than one. This means that the reference group in each model (Asian and white students, continuing generation students, medium and high income students, domestic students) performs worse on CMC questions than traditional multiple choice questions.

Second, we notice that because the odds ratio is less than one for the sex and parental education variables, female and first-generation students perform worse on traditional multiple-choice questions than male and continuing-generation students do. Likewise, because the odds ratios are greater than one, we find that students with higher ACT scores and GPAOs perform better on traditional multiple choice questions than students with lower ACT scores and GPAOs do. For the other variables, (race, socioeconomic status, and international status) the odds ratio is not statistically different than one.

Finally, looking at the interaction terms, we notice that only the sex-question type interaction is statistically significant. Because the odds ratio for the interaction is less than one, the performance gap on CMC questions and traditional multiple choice questions is greater for female students than it is for male students.

To determine what the odds of a female student getting a CMC question correct are to the odds of the female student getting a non-CMC question correct, we can combine the odds ratios. For the coefficients, we would add them to get the simple effect so for odds ratios, we need to multiply them (because ). For female students (), the odds ratio is the product of the question type odds ratio and the sex question type interaction odds ratio () and we find the odds of a female student answering a CMC question correctly are 0.637 the odds of a female student answering a non-CMC question correctly.

Based on the results of the bivariate models, model selection started from the null model (only question type) and then a single new variable and its interaction with question type were added one at a time, starting with sex followed by GPAO, ACT, parental education, international, race, and socioeconomic status. The log-likelihood and resulting p-value of the chi-squared test to compare each model with the previous are shown in Table 4. From the results, there is limited statistical evidence to add parental education or any of the subsequent variables to the model. Therefore, only the results of the multivariate model including question type, sex, ACT score, GPAO and their interactions with question type will be included here and are shown in Table 5. For interested readers, the odds ratios for the mixed effects model with all terms (model 7) are included in S5 Appendix.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Model selection results. The Chi-squared statistic is calculated based on the difference in log likelihoods of the current row and previous row and the degrees of freedom represents the change in the degrees of freedom between the current model and the model in the previous row. A p-value greater than 0.05 suggests the models are not statistically different. Based on the results, we selected model 3.

https://doi.org/10.1371/journal.pone.0323813.t004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Odds ratios for mixed effects model with students and questions as crossed random effects. *** p < .001, ** p < .01, * p < .05, · p < .10. Numbers in parentheses represent 95% confidence intervals. The sex-question-type interaction was found to be marginally significant and all the other interactions were not statistically significant.

https://doi.org/10.1371/journal.pone.0323813.t005

First, after controlling for sex and prior preparation, we find that question type has an odds ratio that is not statistically different from one, meaning that the odds of a male student with an average GPAO and ACT score answering a CMC question correctly are not statistically different than that of answering a non-CMC question correctly.

Second, we find that the demographic variable of sex has an odds ratio statistically different from one, with male students having higher odds of answering non-CMC questions correctly than female students do. We also, unsurprisingly, find that both measures of prior preparation have odds ratios statistically greater than one, meaning that higher grades in college courses and higher ACT scores are associated with increased accuracy on traditional (non-CMC) multiple choice questions.

Next, examining the interaction terms, we find mixed results. We find that the odds ratio of the sex-question type interaction is less than one, meaning the performance difference between male and female students is increased for CMC questions compared to non-CMC questions. Likewise, we find that the odds ratio of the interaction between ACT score and question type is less than one. In contrast, the odds ratio of the interaction between GPAO and question type is greater than one meaning that those with a higher GPAO have a smaller performance difference on CMC questions compared to non-CMC questions compared to those with lower GPAOs. However, we note that none of the interactions are statistically significant, though the sex question type interaction is marginally significant (p = 0.051).

Comparing the results of the multivariate model to the bivariate models, we find that the effects are in the same directions but whether they are statistically equivalent is not always the same. While question type and the sex question-type interaction were statistically significant in the bivariate models, they were not in the multivariate model. We examine possible reasons and explore this further in the discussion.

How does students’ performance on CMC questions compare to their performance on non-CMC questions in introductory physics?

We found that students performed 7.9 percentage points worse on CMC questions compared to non-CMC questions. When looking at the results from the regression models, we found the odds ratio of question type to be statistically different from one in some of the models but not all of them. For example, the odds of a student with an average GPAO getting a CMC question correct is 0.727 of the odds of the student getting a non-CMC question correct. When the student’s ACT score and sex were added to the model, the effect was no longer statistically significant, meaning a male student with an average ACT score and GPAO has the same odds of getting a CMC question and a non-CMC question correct. Taken together, the results suggest that there is a performance gap on CMC questions compared to non-CMC questions though whether we can measure it depends on which group or sub-group of students we examine. Such results align with previous findings [12,18] that CMC questions tend to be more difficult than non-CMC questions.

How does the type of knowledge students need to answer a question (conceptual vs. “plug-and-chug”) account for performance differences between CMC questions and non-CMC questions?

We did not find sufficient evidence that the type of knowledge students need to answer a question could account for the performance differences between CMC questions and non-CMC questions. We found that on non-CMC questions, student accuracy on “plug-and-chug” questions was 3.6 percentage points higher than it was on conceptual questions. As the 95% confidence intervals of the CMC/non-CMC performance difference and the “plug-and-chug” non-CMC/conceptual non-CMC performance differences do not overlap, these results suggest that the observed performance gap between CMC and non-CMC questions is not entirely due to CMC questions only being conceptual while the non-CMC questions were conceptual and “plug-and-chug”. In addition, that students performed worse on CMC questions than non-CMC conceptual questions further supports this conclusion.

What is the relationship between students’ prior preparation and their ability to answer CMC questions correctly?

In terms of prior preparation, we find that “A” students answer CMC questions correctly at a higher rate than non-“A” students do. However, the difference in performance between CMC and non-CMC questions is around 8 percentage points, regardless of whether the student is an “A” student in their other courses or not.

From the regression models, we find that a higher GPAO is associated with increased odds of answering a non-CMC question correctly. However, the interaction term was not statistically different from one, suggesting that there is no disproportionate difference in performance on CMC and non-CMC questions based on the student’s GPAO.

We find a similar story with ACT scores. A higher ACT score was associated with increased odds of answering a non-CMC question correctly but we do not find an interaction effect. We believe this is likely because students with higher ACT scores do better on non-CMC questions than students with lower ACT scores, but everyone performs equally badly on CMC questions (Fig 6) and therefore, any performance difference between CMC and non-CMC questions is explained by differences in non-CMC question performance, which is what the ACT score variable in the model captures.

While we cannot address the potential for clueing directly, our results suggest that if clueing is occurring, it is not helping lower-ability students perform better on CMC questions by eliminating primary answer options due to the available secondary choices as hypothesized by Albanese [12]. If clueing were helping lower-ability students, we would expect the performance gap between CMC and non-CMC questions to be smaller for non-“A” students than for “A” students or for students with lower versus higher ACT scores because clueing would not help on the non-CMC questions. In addition, clueing would only benefit the non-“A” students and students with lower ACT scores and would be of little benefit to “A” students or students with higher ACT scores who were assumed to know the material. Instead, we found the performance gap to be similar.

We also note that performance on all questions is relatively low, regardless of the student’s prior preparation. For example, students who earned “A”s in physics II only answered around 60% of non-CMC questions correctly in our study. We discuss this more in the limitations.

How well can a CMC question differentiate between students who go on to earn a high versus low grade in their physics course?

We find that CMC questions are able to differentiate between students who went on to earn “A”s in Physics II from those who did not. However, we do not find that CMC questions offer any practical differentiation benefits from non-CMC questions as the difference in performance gaps was 2 percentage points. We note that a recent paper found evidence that non-standard multiple-choice questions could be useful for differentiating high and low performing students [61] though that study did not include complex multiple choice questions. Therefore, while some non-standard multiple-choice formats may be useful for differentiating top-performing students from everyone else, our study does not support using CMC questions in this way.

When considering the probability that someone who answered a CMC question correctly would go on to earn an “A”, we find a similar result. While the probability that someone who answered a CMC question correctly earns an “A” in Physics II is higher by approximately 1.5 percentage points than the probability that someone who answered a non-CMC question correctly earns an “A” in Physics II, the result is not statistically or practically different.

How might CMC questions affect performance gaps between students of various demographic groups?

We find that all demographic groups included in our study did worse on CMC than non-CMC questions. We also find that the differences between performance on CMC and non-CMC questions varied between demographic groups with international students having the smallest performance gap and Black, Hispanic, Multiracial, and Native American students and female students having the largest performance gaps. Prior work has also found evidence that there may be differential performance between male and female students on CMC questions [20].

From the regression models, we find mixed evidence that there may be an effect of CMC questions on performance based on demographics. We found a marginally significant interaction between sex and question type in our multivariate model after controlling for ACT score and GPAO and a significant interaction in our bivariate model. In addition, the values of the odds ratio of the interaction are similar in both the bivariate and multivariate models, which suggests that the interaction may hold explanatory power. If the interaction were instead a proxy, we would expect the value of the odds ratio to be much closer to one in the multivariate model because some of the effect would be “taken” by the other variables in the model.

It is possible however that the effect is only marginally significant because we are missing variables in our model and once those variables are included, we would find a different result [62]. Indeed, in multivariate model 7 (S5 Appendix), we do find the sex-question type interaction is significant once all demographic variables and their interactions are included in the model. However, our model selection process does not provide evidence that we would be justified picking the more complex model over the model we report in the paper. While some argue that it is justifiable to keep a non-statistically significant variable in the model provided there is a strong theoretical reason for doing so [55,62], we do not have a particularly strong theoretical reason for including the additional variables and our bivariate models do not provide strong experimental reasons for including additional variables beyond parental education. Including only parental education and its interaction with question type does not change the sex-question type interaction away from marginally significant. Therefore, additional evidence is needed to fully resolve this result. Regardless, it should be noted that whatever the result, it does not mean that the student’s sex causes a difference in performance but rather, that there is something different between male and female students in our data set that relates to students’ likelihood to answer the questions correctly.

For all other demographic groups, our regression models find limited evidence that CMC questions could exacerbate existing performance gaps.

We note that the question type variable in the models is confounded by the fact that non-CMC questions are a mix of conceptual and “plug-and-chug” questions while CMC questions are only conceptual and the conclusions should be interpreted through that lens.