Setting

This retrospective cohort study was conducted in Leiden University Medical Center (LUMC), the Netherlands. The Dutch medical curriculum comprises a three-year bachelor’s program followed by a three-year master’s program. Although the bachelor courses are primarily assessed with MCQs, other formats such as extended matching, comprehensive integrated puzzle, open essay, and VSAQs are also included in the written assessments. Additionally, students participate in longitudinal training on various CanMEDS competencies [11] beyond the role of Medical Expert, such as communication skills, leadership, health promotion, and collaboration. To assess the discriminative ability of VSAQs and MCQs we analyzed summative examinations of two medical bachelor’s courses: “Regulation and Metabolism” (RM), a first-year fundamental course, and “Diseases of the Abdomen” (DA), a second-year clinical course. These courses (6 and 7 weeks, respectively) address metabolic and gastrointestinal topics. These courses were chosen because their summative assessment contained both VSAQs and MCQs within the same examination. In our prior study [5], we compared the psychometric properties of MCQs and VSAQs in formative assessments for both courses. Subsequently, the course coordinators added VSAQs to the previously MCQ-only summative exams, resulting in mixed-format examinations. This mixed format has now been used in both courses for three consecutive years (student cohorts 2020–2021; 2021–2022; 2022–2023), resulting in six summative mixed-format examinations which we selected for our analyses. This mixed-format design enabled direct within-examination comparisons between question formats under identical educational and assessment conditions. Teaching and exam preparation follows the curriculum program for all cohorts. Students receive scheduled instructional sessions, guided self-study materials, and practice assessments aligned with the exam formats. All students have equivalent access to these learning resources prior to the examination.

All assessments were administered digitally through RemindoToets (Paragin) system [12] and yielded study credits. All teachers involved in the question writing process were required to participate in a course about writing exam questions, where guidelines for writing high quality exam questions were addressed, based on existing literature [13]. Furthermore, all exam questions are reviewed by an Examination Review Board as part of a routine quality assurance system for assessments. VSAQs were marked accordingly to the standard institutional procedure. Predefined correct answers were automatically scored; all other responses were reviewed manually by the examiner. Correct but unanticipated answers (e.g., minor spelling variations) were added to the accepted answer list. Scoring was dichotomous (0/1), with a second examiner consulted when uncertainty arose.

Participants

The first population, RM participants, included all first-year bachelor medical students from the 2020–2021, 2021–2022, and 2022–2023 cohorts who completed the first sitting of the summative RM assessment (n = 902). The second population, RM & DA participants, comprised all second-year bachelor medical students from the same cohorts who completed the first sitting of both the RM and DA summative assessments in consecutive years (n = 797). Students who attempted less than 75% of all exams taken into consideration for the GPA calculation were excluded from both populations (n = 51).

Study design and data collection

We analyzed data from two summative mixed-format examinations from the RM and DA courses, administered between 2021 and 2024. For each assessment, we extracted individual questions, question formats, question scores (coded as 1 for correct and 0 for incorrect), residual item reliability-values (R_ir) for each question, and student IDs from the Remindo assessment system. The R_ir-value, representing the correlation between one test item and all other items of the test, measures item discrimination by indicating how well each question correlates with overall student performance on that test (after excluding that specific question) [14]. To assess the comparability of question formats within the examinations, we examined the distribution of each question format across the different themes. The data were accessed between 10/11/2024 and 17/12/2024 for our research. The data were anonymised using ID numbers, ensuring that the authors had no access to information that could identify individual participants.

We measured academic performance with the grade point average (GPA), a widely recognized and standardized indicator of academic success [15,16]. In our context, GPA is defined as a credit-weighted average of all numeric grades from the first and second year of study. For each student, GPA was calculated based on all courses that included a written examination. In addition to written exams, several courses also comprised other forms of examinations (e.g., oral exams, presentations, or practical assessments) which were scored by a pass or fail, thereby providing a broader representation of academic performance. Courses were weighted according to their European Credit Transfer and Accumulation System (ECTS) credits. An overview of all courses included in the GPA calculation and their corresponding ECTS weights is provided in S1 table. The RM and DA course awarded 7 and 8 ECTS, respectively. The GPA was computed by multiplying each course’s final grade by its corresponding ECTS credits and aggregating these values. All exam grades from the first and second-year courses were retrieved from the university’s administrative system. In our system (used in several European countries), grades range from 1 to 10, with 10 being the highest possible score. A grade of 5.5 or higher is considered a passing grade and is required to earn study credits. Failing grades (defined as <5.5) were included in the GPA calculation to provide a realistic representation of overall academic performance. We evaluated GPA based on two approaches: one using the most recent grade from the last sitting (including retake exams) and another using only the grade from the first sitting. For the first population (RM participants) we calculated the GPA based on first-year courses, while for the second population (RM & DA participants) the GPA included courses from both the first- and second-year. Based on these GPAs, students were categorized by quintiles into “poor performing” (first quintile), “average performing” (second through fourth quintile), or “excellent performing” (fifth quintile) students.

Data analysis

Descriptive statistics were calculated and presented for each examination, including total average scores, average VSAQ and MCQ scores, and the distribution of VSAQs and MCQs across the themes. To compare the different question formats, we calculated separate z-scores for VSAQs and MCQs based on their respective absolute scores. For the RM participants, z-scores were derived from their VSAQ and MCQ scores on the RM assessment. For the RM & DA participants, z-scores were calculated separately for VSAQs and MCQs using absolute scores from both the RM and DA assessments, ensuring that each student had one VSAQ z-score and one MCQ z-score reflecting their performance across both exams. Item discrimination was determined using the mean of the R_ir-values for each question format [13]. The R_ir-value for each question was calculated in relation to the entire exam.

Linear regression analysis was performed with GPA as linear outcome variable and the z-scores of the VSAQ and MCQ as covariates, separately for each cohort and across all cohorts. First, univariate regression analyses were conducted to assess the variance explained by each question format individually (reported as adjusted R²). Next, both formats were included in a multivariate model to account for shared variance and determine their relative predictive value. Student GPAs from the courses of the first year (RM participants) and the first two year (RM & DA participants) were used as outcome. Similarly, the same student GPAs were used to create new categorical variables: “poor performing” versus “other” (average and excellent) and “excellent performing” versus “other” (average and poor). Receiver Operating Characteristic (ROC) curves were used to evaluate the ability of MCQs and VSAQs to distinguish between poor and non-poor performing students, and also between excellent and non-excellent performing students. The Concordance-statistic (C-statistic or area under the ROC curve) was calculated as a measure of discriminatory power, reflecting how well these formats differentiate between performance levels. Significant differences between the C-statics of VSAQs and MCQs were assessed using paired bootstrapping with 95% confidence intervals. The difference was considered significant when the confidence interval (CI) of the score difference did not contain the zero. Bootstrapping was used to compare the paired C-statistics because it provides robust confidence intervals without relying on strict distributional assumptions. All statistical analyses were performed using R version 4.1.0 (R Foundation for Statistical Computing, Vienna, Austria).

Ethical approval

This study received ethical approval from the LUMC Educational Research Review Board (OEC/ERRB/20241008/1). Our ethical review board decided that informed consent was not required, as this study did not involve an intervention or experimental manipulation. Rather, VSAQs were introduced as part of an incremental improvement of the existing assessment practice, initially on a small scale to gain experience with this question format. The data analyzed were retrospective and already available from the assessment system as part of the students’ regular curriculum. We collected no additional demographic or personal data, and all data were aggregated and anonymized to ensure that individual students could not be identified.

Descriptives and item discrimination

The students included in our study for each mixed-format examination, along with the average total, VSAQ, and MCQ scores are presented in Table 1. In all mixed-format examinations, the average R_ir-values of the VSAQs were consistently higher compared to the MCQs (Table 1). The distribution of question formats across the themes was generally even, with no themes exclusively assessed by a single format (S2 Table).

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. The scores and R_ir-values of VSAQs and MCQs within the mixed-format examinations.

https://doi.org/10.1371/journal.pone.0349318.t001

Relationship between question format and academic performance

We conducted a linear regression analysis to examine the effects of VSAQ and MCQ z-scores on GPA. First, we assessed the variance explained by each question format separately using univariate models (R² values). Among all RM participants, the R² values ranged from 0.61 to 0.67 for VSAQ z-score and from 0.44 to 0.58 for MCQ z-score. In all RM & DA participants, the R² values ranged from 0.68 to 0.71 for VSAQ z-score and from 0.59 to 0.67 for MCQ z-score. The R² values represent the proportion of variance in GPA that can be explained by the VSAQ or MCQ z-score.

In the multiple regression analyses standardized beta coefficients (β) were used to compare the relative strength of associations between the VSAQ and MCQ z-scores and GPA; larger absolute β values indicate stronger relationships. The multiple regression analyses showed a significant positive association between both VSAQ and MCQ z-scores and GPA (Table 2). However, for all RM participants (n = 902), the VSAQ z-score had a stronger association with GPA when using first sitting grades (β = 0.63, t(903) = 21.28, p < .001, 95% CI [0.57, 0.69]) than the MCQ z-score (β = 0.36, t(903) = 12.17, p < .001, 95% CI [0.30, 0.42]), with the model explaining 68% of the variance in GPA (F(2, 899) = 945.79, p < .001, adjusted R² = .68). Similarly, in all RM & DA participants (n = 797), the VSAQ z-score had a stronger positive association (β = 0.53, t(794) = 21.78, p < .001, 95% CI [0.48, 0.57]) compared to the MCQ z-score (β = 0.36, t(794) = 14.84, p < .001, 95% CI [0.31, 0.41]), with the model accounting for 77% of the variance (F(2, 794) = 1303.59, p < .001, adjusted R² = .77). These results were similar when using last sitting grades and for the analyses of the separate cohorts (Table 2).

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Linear regression model parameters with grade point average as outcome based on the first and last sitting grades.

https://doi.org/10.1371/journal.pone.0349318.t002

Identification of poor and excellent performing students

We calculated the C-statistic to assess the ability of VSAQ and MCQ scores to identify poor performing (lowest GPA quintile) and excellent performing (highest GPA quintile) students. The C-statistic reflects the model’s ability to distinguish between individuals with and without the outcome of interest. Values closer to 1.0 indicate better discrimination, whereas a value of 0.5 indicates performance no better than chance. In all RM participants (n = 902), VSAQ z-scores showed higher C-statistics (i.e., greater discriminative ability) for both poor and excellent performing students compared to MCQ z-scores (Fig 1A). However, for poor performing students using first sitting grades, the difference between the C-statistics of VSAQ and MCQ was not significant. In all RM & DA participants (n = 797), the C-statistic of the VSAQ z-score remained significantly higher than that of the MCQ z-score for poor performing students using first sitting grades. For excellent performing students, there was no significant difference in discriminative ability (Fig 1B). The findings across separate cohort analyses followed the same pattern; either the C-statistic for the VSAQ z-score was significantly higher than that of the MCQ z-score, or there was no significant difference between them (Figs S1 and S2).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Receiver Operating Characteristic (ROC) curves for both poor and excellent performing students using the first sitting or last sitting grades from A) the total RM participants, and B) the total RM & DA participants.

Red = VSAQs; Blue = MCQs; CI = 95% confidence interval; TPR = true positive rate; FPR = false positive rate. NS = non-significant. C-statistics are shown above each graph, which were calculated from the area under the ROC curve.

https://doi.org/10.1371/journal.pone.0349318.g001

Strengths and limitations

This study is the first to evaluate the discriminative ability of VSAQs across multiple assessments using GPA as a measure of academic performance (evidence of relations to other variables), rather than focusing exclusively on item discrimination with individual exams (evidence of internal structure). By utilizing real-world summative assessment data from three distinct student cohorts, we minimized the biases associated with voluntary participation and ensured a robust, and representative sample size. This also allowed us to aggregate results and average out variance across examinations and populations, thereby enhancing the reliability of our findings. The inclusion of both first-year students with greater variation in academic performance, and nominal second-year students allowed for a more comprehensive analysis of performance across different populations, while mitigating selection bias. Moreover, mixed-format examinations enabled within-subjects comparisons of VSAQs and MCQs, under comparable instructional and assessment conditions. Standardizing scores further reduced variability and ensured reliable comparisons. However, this study also has limitations. While GPA is an objective and widely used measure of academic achievement, it simplifies complex learning outcomes and may overlook critical thinking and skill development [15]. Additionally, the variability in content and distribution between VSAQs and MCQs within real-world examinations could introduce bias. However, we mitigated this by aggregating data from multiple examinations and cohorts, standardizing scores, and conducting analyses across two student populations. Lastly, this study was conducted within a single medical school, which may limit the generalizability of the findings to other institutions and curricula. Although the inclusion of multiple cohorts and examinations strengthens the robustness of our findings, replication across different educational contexts is needed to determine whether the observed advantages of VSAQs are consistent across settings with varying practices and student populations.

Implications for practice and future research

Our findings indicate that VSAQs have a higher discriminative ability than MCQs, effectively distinguishing students both within individual examinations and across multiple examinations based on GPA. These findings suggest that VSAQs may represent a more robust and valid question format for evaluating academic performance and warrant further consideration within medical education assessment. Incorporation of VSAQs into assessments may enhance the ability to identify students who could benefit from early interventions and support, while also recognizing those who excel. However, given that this study was conducted within a single institution, multicenter studies and prospective cohort research are needed to confirm the generalizability of these findings and to evaluate the practical implications of implementing VSAQs across diverse educational settings before widespread adoption can be recommended.

Although the observed differences in discriminative ability between VSAQs and MCQs were modest, they were consistently observed across examinations and student cohorts. In educational practice, even relatively small improvements in discrimination may be meaningful, particularly in assessments that inform decisions regarding student progression or academic support. Improved discrimination may enhance the ability to distinguish between students with different levels of knowledge and understanding, thereby supporting more accurate assessment decisions. The potential educational benefits should therefore be considered alongside practical implementation requirements.

The practical implementation of VSAQs also warrants consideration. Adoption of VSAQs may require investments in examiner training, and adaptation of assessment procedures. However, training can be integrated into existing faculty development and item-writing workshops, allowing educators to develop VSAQs while simultaneously preparing examination content. Furthermore, previous research has shown that grading is feasible and efficient in large cohorts [5,6]. Therefore, although implementation requires investment of time and institutional support, the operational burden may be manageable and should be weighed against the potential gains in assessment validity and discrimination. Future studies could also examine grading efficiency, examiner experiences, and inter-rater reliability across different educational settings to further inform large-scale implementation of VSAQs.

Moreover, the stronger relationship between VSAQ performance and GPA suggests that VSAQs may warrant further investigation as a potential predictor of academic performance. Future prospective longitudinal studies are needed to determine whether performance on VSAQs is associated with subsequent academic outcomes and whether such assessments could have a role in selection procedures. Future studies could also expand academic performance measures beyond GPA, incorporating assessments of skill development and work-place learning. These additional metrics would provide a more holistic evaluation of the effectiveness of VSAQs and MCQs across different educational contexts. Additionally, exploring whether VSAQs are associated with long-term academic and professional success, including performance in real-world clinical settings, would provide deeper insights into their utility.