Study design

The primary goal of the study was to characterize the quality of FM MCQ’s generated via AI methods (i.e., ChatGPT-4); and compare the quality of AI-generated MCQ’s versus those generated by human experts.

Eight consenting attending physicians from the University of Toronto postgraduate Family Medicine training program were recruited to score questions [10]. These eight subject matter experts had previous question-writing experience on committees such as the Medical College of Canada, Royal College of Physicians and Surgeons of Canada, or the College of Family Physicians Canada licensing examinations. Recruitment specifically was targeted at physicians with a background in postgraduate family medicine question writing.

The eight physicians each scored 44 FM MCQs using the quality/scoring guide created in Table 1 across 17 quality domains (14 binary and 3 likert). Four approaches to medical question generation were considered:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Question rating scales used to assess multiple choice question quality and reliability.

https://doi.org/10.1371/journal.pdig.0001156.t001

1. Subject matter expert generated questions/scenarios from an existing postgraduate family medicine question bank.
2. GPT-4 generated questions using cloning from existing question banks.
3. GPT-4 generated questions using novel prompts/inputs without any reference to existing questions (see Table 2).
4. GPT-4 generated questions using novel prompts/inputs (see Table 2) with human appraisal and editing.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Steps and rationale for prompt engineering of novel ChatGPT generated multiple choice questions.

https://doi.org/10.1371/journal.pdig.0001156.t002

We utilized fourteen human-generated questions from the existing family medicine residency MCQ bank at the University of Toronto. We intended to include ten questions, however, due to human error leading to a protocol deviation, four additional questions were included, rated, and analyzed. Our a priori plan allocated an equal number of items (10/10/10/10) to each generation method to preserve balance for estimation and comparability. Owing to a protocol deviation (human error during survey set-up), 14 expert-generated items were distributed and rated. Because these items had already been scored by all raters, and removal would unnecessarily discard data and reduce precision, we retained all observations and accounted for potential imbalance with mixed-effects models. We also note this deviation as a limitation.

We created ten GPT-4 questions generated using cloning from the existing undergraduate medicine and postgraduate family medicine residency MCQ banks. Cloning involves providing GPT-4 with an existing question and asking it to create a similar question, but modifying the stem, answer, or other components of the options. We provided the question and asked GPT-4 to create similar questions based on the provided reference questions as a guide.

We created ten GPT-4 questions generated using the prompt template in Table 2.

Ten GPT-4 generated questions were created using the novel prompt template in Table 2 and were then appraised and modified by a single investigator with expertise in post-graduate medical testing (F.L.) without the use of GPT-4.

Question rating scale

Writing an effective MCQ is a challenging task, especially in the medical context [11]. Clinical vignettes, questions, answers, and distractors must be carefully crafted to assess multiple domains of psychometric performance.

Each of the 8 subject matter experts recruited, scored each of the 44 FM MCQs with respect to 17 domains of quality. 14/17 quality measures were scored using a binary (yes/no) measurement scale; whereas, 3/17 quality measures were scored using an ordinal five-point Likert scale.

In this study, we aimed to generate a comprehensive rating scale that would address diverse quality domains (as seen in Table 1). This scale was created in collaboration with experts who have served or led institutional, provincial, and national committees for medical education (MK, FL, HC). We utilized well-validated and published MCQ rating scales [12–14] and combined these with the generally accepted best practices for evaluating MCQ items.

The eight subject matter experts scoring each FM MCQ were blinded to the generation method. Raters were not asked to answer the questions; nor were they required to provide a rationale for the quality scores they provided.

Prompt engineering

There has been significant interest in the AI space on how to effectively generate the ideal prompt (the initial input, question, or task) for ChatGPT to maximize the efficiency and appropriateness of its response [15,16]. There have also been many plug-ins generated to accomplish this task [17]. We tested many different prompt formulations for creating an ideal medical MCQ. We present a concise 10-step, role and objective anchored prompting framework designed to elicit higher-order clinical reasoning, with explicit instructions for vignette structure, distractor quality, and jurisdiction-specific guidance (Table 2).

Table 2 summarizes the major components of the prompt that we believed were essential to be included: 1. Define the role of GPT-4 as an educator and specify the setting (i.e., postgraduate family medicine department). 2. Define the goal of the task (i.e., assist residents in preparing for licensing examination). 3. Define who the target audience/learner for the task is (graduating family-medicine resident). 4. Define the objective, question domain, type of question, and number of options (i.e., Create a clinical decision-making multiple-choice question with four options). 5. Provide a source or reference material for the questions for the unique jurisdiction/examination (i.e., based on the Certificant of the College of Family Physicians 105 priority topics). 6. Ensure GPT-4 provides an answer and rationale for the proposed question. 7. Ensure to specify the level of difficulty the question should be (i.e., challenging enough for a graduating family medicine resident in Canada). 8. Provide explicit instructions on how the stem should be structured and what type of information should be included, based on the type of examination (i.e., structure the question stems so they include the physician type, setting of care, patient age, past medical history, medications, pertinent symptoms, physical examination findings, relevant bloodwork with Canadian units for measurement, imaging results). 9. Provide instructions to be specific on names, doses, frequency, and duration of therapies as relevant to the question. 10. Instruct GPT-4 on the number of questions to generate or if it should generate questions based on a defined number of medical topics (i.e., provide 10 questions covering different topics versus 10 questions covering diabetes, hypertension, and dyslipidemia topics).

There were several reasons that this specific format was used for question generation. The common themes among other prompts were that the questions and answers generated were too simple and did not include specific patient clinical characteristics (i.e., no inclusion of age, past medical history, medications), did not identify key features relevant to the setting the physician was in, the distractors were of poor quality, the questions did not test higher-order thinking (defined as applying foundational knowledge to a case scenario to apply and integrate multiple key concepts in diagnosis and management) related to clinical decision making, and the vignettes were not sophisticated in the information provided. There were also some prompts which led to hallucinations where the answer rationale was not in keeping with appropriate clinical guidelines, however, this is an inherent limitation in the technology’s ability to source accurate and contemporary information (a challenge that family physicians also regularly face in their practice). Overall, this prompt engineering approach was chosen as the result of an iterative process testing each component to make the output as specific and detailed as possible.

Question content

The content of questions in this study were limited to the specialty of family medicine, specifically as it is practiced in Canada, and as it would be appropriate for a graduating family medicine resident. We did not limit GPT-4 on the types of questions it could produce within family medicine. From our human-generated question bank, we randomly chose a series of questions with the only criteria being that there were no directly similar topics within the question set. Furthermore, we opted to avoid the use of any questions or stems that would rely on images, as we did not have a robust database of expert-generated questions with images to use as control variables. Given this, topics relating to dermatology, radiology, pictures of physical examination findings, and presentations involving electrocardiograms or other visual-based inputs were not included in this study.

Hallucination quantification

A post‑hoc review of every GPT‑4–generated question was performed to identify hallucinations, specifically flagging factual errors or content that diverged from the current, high‑quality publicly available evidence. Two investigators independently reviewed every GPT-4–generated item and rationale for factual errors, outdated recommendations, or contradictions with publicly available, high-quality guidance relevant to Canadian family medicine. Discrepancies were adjudicated by a third investigator.

Statistical analysis

Counts and percentages were used to characterize ratings of the binary quality scores. Means and standard deviations were used to characterize the Likert-scale quality scores. Likert-valued responses were dichotomized before modelling/inference.

Linear mixed effect models were fitted to each of the 17 binarized quality outcomes. The linear mixed effect models included a random effect for the rater (8 raters), a random effect for the question (44 questions), and a fixed effect for the question generation method (i.e., human expert generated, GPT-4 cloned, GPT-4 novel, GPT-4 novel with human editing). Logistic generalized linear mixed models (with an identical parameterization as described above) were considered; however, design constraints (i.e., small number of raters/questions) coupled with sparse empirical response distributions resulted in non-convergence of these seemingly more appropriate models. Hence, this study utilized a more robust/stable linear mixed probability model for inference.

Model estimated probabilities of a positive quality score, conditional on question generation method (i.e., human expert generated, GPT-4 cloned, GPT-4 novel, GPT-4 novel with human editing) were obtained for each of the 17 binarized quality outcomes. We estimated differences in conditional probabilities across question generation types, with particular interest in 3 contrasts:

1. Human expert-generated questions versus GPT-4 cloned questions.
2. Human expert-generated questions versus GPT-4 novel template questions.
3. Human expert-generated questions versus GPT-4 novel templated questions with editing.

Confidence intervals about model-estimated differences in conditional probabilities were obtained. We were interested in whether upper limits of 95% confidence intervals about differences in conditional probabilities were less than a priori stated non-inferiority margins (subjectively set at 10%) [18].

As a secondary analysis, we fit a linear mixed probability model with random rater/question effects and a fixed effect for the question generation method. In this model, the question generation method was operationalized as a binary variable (i.e., human expert generated question vs. GPT-4 generated question). Conditional probabilities of a positive quality score were estimated for each group. Differences in conditional probabilities and associated 95% confidence intervals were used to assess whether AI-generated questions were non-inferior to human-expert-generated questions, using the same 10% non-inferiority margin.

Research ethics

Ethics approval for this study was granted from the University of Toronto Research Ethics Board (RIS #45124).

Discussion

Our study found that ChatGPT-4 could generate FM MCQ’s of similar quality to human experts for several quality domains (i.e., non-inferior at a 10% margin). Human‑expert–generated questions were less likely to be identified as AI‑generated compared to ChatGPT‑4–derived questions

While the current literature suggests a possible use of AI in question generation [19,20], our study adds a rigorous evaluation of the validity of the generated questions, utilizing a novel and comprehensive MCQ rating scale [21–23]. Several diverse AI question generation methods are pragmatically compared against human expert question generation, using a novel non-inferiority design (not previously implemented when evaluating AI tools in medical education settings). While this study specifically examines a prompt engineering framework specific to GPT-4, the principles generalize to other transformer-based LLMs.

Using a novel psychometric instrument for scoring family medicine MCQ’s, our findings demonstrate that ChatGPT-4 is capable of generating MCQs with quality scores comparable to those created by human experts across a range of psychometric domains. AI’s ability to assist in generating high-quality content may significantly reduce the burden on educators while maintaining high educational standards. Our experience of question generation prior to the advent of AI suggests a writing speed of 1–2 high-quality MCQ questions per hour. AI assistance thus presents a potential benefit in efficiency and cost that must be explored. Moreover, trained expert item writers are a scarce faculty resource. Using AI assistance may enhance the capacity of newly engaged item writers and make the work of creating items less daunting for non-expert faculty.

Our structured prompting complements emerging approaches that scaffold LLM outputs for education and clinical content creation. While newer models appear less prompt-sensitive, explicit structure remains useful for ensuring item difficulty calibration, clinical specificity, and distractor plausibility.

Our results also suggest that human raters were less likely to perceive human-expert generated questions as being AI generated, as compared to several of the ChatGPT-4 methods. This reinforces the notion that AI should be viewed as a supplement to, rather than a replacement for, expert input in educational content creation. Furthermore, generative-AI use in assessment raises potential concerns about bias propagation, misinformation, and over-reliance. We recommend human-in-the-loop editorial review, transparent audit trails for item provenance, periodic guideline concordance checks, and sequestered item banks with access controls to protect exam security.

Limitations and Future Work

One limitation to our study is that we randomized the question content, while keeping it inside the domain of family medicine. Future work is required to investigate the quality of AI generated medical MCQs developed to test knowledge in other medical sub-specialities. While the prompt engineering framework would be generalizable to these specialties, it is unclear without further research whether the intended end-users would find these questions appropriate, challenging, and factually accurate. Future work should also focus on an explicit iterative process to evaluate each component of the prompt for relevance.

Our work was limited to a small sample size of 44 MCQs with 8 raters. This was particularly limiting in the ability to make precise inferences regarding non-inferiority. As well, we did not provide training to the raters on addressing bias in AI-generated questions. Small sample sizes, coupled with ceiling effects on certain binary quality domains, also resulted in model convergence issues (hence why we used robust linear mixed probability models for inferences, rather than more standard logistic generalized linear mixed models). Future work in our group aims to address these sample size and design-related challenges in a larger cohort study.

Newer architecture with built-in chain-of-thought capabilities [24] (i.e., OpenAI o1) may reduce the need for explicit prompt decomposition, making an explicit 10-step prompt engineering less relevant. Our current work is a brief snapshot in the fast-moving field of LLM models and as they continue to evolve, further work iterating on this process is crucial. Our work may not extend to non-GPT models without further validation and our future work will aim to test our methodology with a diverse set of LLMs.

While our approach to prompting the LLM did not output any medical MCQs that were found to experience hallucinations on expert review, we did not employ any specific strategies to mitigate hallucinations. Future work should aim to employ retrieval-augmented generation [25], chain-of-verification [26], and reference checks as strategies to mitigate hallucinations and confirm the medical accuracy of the subject matter. Model updates since the time of our initial study have addressed many of these strategies and would strengthen future work aimed at producing high-quality medical questions.

As well, future work should evaluate knowledge-augmented generation (retrieval-augmented generation against current Canadian guidelines) to reduce factual drift and improve rationale quality, and test generalizability across specialties, jurisdictions, and larger item pools.

Given the novelty of this work, it is unclear how to a priori determine an appropriate non-inferiority margin (e.g., 10% versus 5%, 1% etc.). As AI evolves, equivalence or superiority designs could be argued as more appropriate, however this remains unclear at this time in the literature.

Finally, we did not have graduating family medicine residents complete the questions or evaluate them. This information would help further understand whether AI-generated questions would be an appropriate learning tool for trainees and will be the focus of our future work. We also did not explore the participants’ perspective on this process. The focus of future work should aim to understand the ethical implications, for trainees and faculty, that this poses to the current medical education curriculum.

Conclusions

We present a novel study using GPT-4 to create medical examination questions compared to human-generated questions. We provide a framework for the prompt engineering of novel medical exam questions, a robust rating scale to evaluate the validity of AI-generated medical MCQs, and rigorously conceptualize AI versus expert comparisons in the context of a non-inferiority study design. Our study builds on the fast-growing body of work in evaluating how AI can augment medical education and serves to guide further developments for educators. Incorporating AI in question generation presents a cost-effective alternative, reducing time and resource burdens on faculty while maintaining high-quality standards.