Conceptual framework and research question

To assess how GAI algorithms perform on doctoral-level experimental design questions, we used Bloom’s taxonomy as a framework. First developed to characterize the acquisition of knowledge, Bloom describes six ordered levels from lower cognitive functions like memorization to higher-level critical thinking skills like analysis and evaluation [26]. We focus here on a subsequent revision that introduced six verbs of increasing cognitive complexity: “remember”, “understand”, “apply”, “analyze”, “evaluate”, and “create” [27]. Bloom’s taxonomy has been widely used across disciplines from computer sciences and mathematics to music [28–30]. In the biosciences, this framework is most often applied at the undergraduate level, both to design assessments and plan learning objectives [31]. Given how assessments can shape student learning [32], courses typically span the spectrum of Bloom’s levels from “remember”–the act of retrieving knowledge from memory–to higher levels that promote critical thinking, such as “analyze”, which involves deconstructing content to achieve a new goal [33,34].

The precise links between taxonomic levels and goals within Bloom’s framework vary across disciplines [35]. In graduate biosciences, the exact assignment of levels is challenging as the competencies expected of scientists can often involve multiple levels of Bloom’s simultaneously or iteratively [36]. In BCMP 200, we ask students to answer hypothetical open-ended experimental design questions across two modalities: in-class “chalk talks” and out-of-class worksheets. Students’ use of GAI can be monitored for the in-person “chalk talks” but not for out-of-class worksheets. In this work, we sought to determine whether our worksheets can still serve as authentic assessments that promote learning despite GAI usage or if a larger redesign of our course assessment structure would be warranted instead. To assess both students and ChatGPT using Bloom’s taxonomy, we began by defining the associations between Bloom’s levels and experimental design components guided by the Blooming Biology Tool, developed for this purpose with biology exam questions (Table 1) [35]. Within this context, “remember” depends only on recalling information, apply requires predicting outcomes in new situations, and “analyze” involves interpretation to select the best conclusion [35]. “Remember” is thus characterized by verbs like identify, recall, and list, which aligns with stating the purpose of a given technique. “Apply” instead involves using information in new contexts, such as implementing positive and negative controls in new experimental designs. Finally, “analyze” refers to inferring how components come together, as would be seen in selecting the method that best addresses the question at hand. With this design, we sought to evaluate how ChatGPT and first-year doctoral students perform across the levels of Bloom’s taxonomy. By identifying weaknesses in ChatGPT’s capabilities on assignments, we hoped to define principles for designing effective out-of-class assignments to maintain meaningful student engagement with classroom content.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Defining and Contextualizing Bloom’s Levels in Experimental Design Principles.

https://doi.org/10.1371/journal.pone.0346127.t001

We hypothesized that ChatGPT would show performance comparable to that of doctoral students on the lower levels of Bloom’s taxonomy, like “remember” and “understand.” However, we predicted it would underperform at the higher taxonomic levels, such as “analyze” and “create”, whose cognitive complexity involves critical thinking. When we examined ChatGPT’s performance, the model generally underperformed and did not achieve the 80% passing threshold for BCMP 200. On text input questions, ChatGPT answered questions across the whole hierarchy of Bloom’s from remember to create with reasonable success. The notable exception came at the “apply” level, which involves rationalizing and applying experimental controls. In contrast, ChatGPT’s performance was generally poor when image inputs like scientific graphs were incorporated. Specifically, ChatGPT accurately answered “remember”-level Bloom’s questions that did not rely on the provided image but failed to contextualize information from the image when answering “apply”-level questions. Based on our results, we discuss several strategies for how best to prepare out-of-class assessments that promote student engagement and learning in the era of GAI.

Molecular biology course and student information

The study was conducted with first-year doctoral students enrolled in the BCMP 200: Principles of Molecular Biology course at Harvard Medical School. Enrollment is open to students across life sciences doctoral programs and is part of the core curriculum for the largest bioscience graduate program. The course spans a full semester (approximately 15 weeks). The course is taken by approximately 80 students and covers thematic modules focused on concepts in molecular biology, which are taught using a flipped classroom format [11]. Experimental design training is embedded into the lectures and small group discussions led by upper-year doctoral students serving as teaching fellows (TFs). At the end of each module, students are assessed by: (1) presenting a chalk talk that answers a hypothetical, open-ended experimental design question, and (2) completing a two-part out-of-class worksheet [9]. In a fill-in-the-blank table, students summarize the module’s experimental techniques by briefly describing the method and its goal. In a long answer section, we formulate a hypothetical experimental design question and provide students with hypotheses to test. They are asked to design an experiment to test the hypotheses, justify their methodology, and depict expected results. We focused here on two modules: “Module 1: DNA-Protein Interactions” and “Module 5: Transcription” to provide breadth to the content we assessed. Doctoral students are expected to maintain a minimum grade of B, so we treat 80% as a passing grade for this course.

Student responses

Student responses to experimental design questions were randomly selected from one section of the course by assigning all students a number and using a random number generator to select a subset. To avoid confounding our results with responses that utilized GAI, out-of-class worksheet answers were obtained from the Fall 2022 iteration of the course, which preceded the general release of ChatGPT. Due to practical constraints, we used a sample size of 20 for each experiment. All student responses from the cohort of 79 students were eligible for inclusion. To ensure scoring reliability, each response was independently evaluated by three graders using a standardized rubric (see “Blind grading” below). This approach prioritized grading reliability while maintaining random sampling from the full cohort. Long-answer responses were lightly edited to remove any references to depicted figures, as ChatGPT-4 Omni cannot depict predicted results.

GPT-4o generated responses

All GAI-generated responses were produced using ChatGPT with GPT-4 Omni (GPT-4o). GPT-4o, released by OpenAI in May 2024, can accept combinations of text, audio, image, and video inputs to generate text, audio, and image outputs. For multiple-choice questions, we repeated prompting 10 times per question and recorded the model’s output. For free-response questions, we used a two-step prompting approach. We initially provided the model with the question, and then prompted: “Could you rewrite that in paragraph form?” This method helped ensure stylistic similarity between student and GAI responses. Additionally, GPT-4o responses were occasionally lightly edited to remove transition statements that would not typically be found in student responses. An example of text removed is: “To explore the mechanisms by which the MAYBELESS protein disrupts expression of the deliciousfood76 gene in Drosophila melanogaster, we’ll employ distinct experimental strategies tailored to each hypothesis.” The remainder of the response was used in full.

Bloom’s level assignments

Bloom’s levels were assigned based on the Blooming Biology Tool, which was created to assist science faculty in designing undergraduate-level assessments that test higher levels of cognitive skills [35]. Levels were assigned to our questions based on the specific skills and characteristics addressed by each Bloom’s level in this tool. The “analyze” level, for instance, tests how components relate to one another and a larger process and is characterized by data interpretation and selection of the best conclusion. On our worksheets, this level corresponds to the ability to select the best experimental technique to address the question within the confines of the experimental setup. Additional examples of this mapping to Bloom’s levels are in Table 1.

Blind grading

Answers generated by students and ChatGPT were formatted into the same style and combined into a single document. Answers to a specific question were randomly assigned an identifying number and sorted according to this number. Two TFs (A.K. and C.G.) and one postdoctoral fellow (C.D.) with at least two years of experience in BCMP 200 blindly graded all free-response questions using a pre-designed rubric, and grades were then unblinded by J.J.P. The average grade from the three graders was used as the final score for each question.

Data analysis

Data visualization and analyses were conducted using GraphPad Prism 10. As normality was not assumed, comparisons between groups were performed using the non-parametric Mann-Whitney test. A p-value threshold of p < 0.05 was used for significance.

Human subjects research

The research performed in this study was determined not to be human subjects research and that consent was not required by the Institutional Review Board (IRB) of the Harvard Faculty of Medicine, protocol IRB25–0357. Student responses were accessed on May 1, 2024. As described above, a subset of student responses was selected using a random number generator and contained no identifying information.

ChatGPT performs well (but worse than students) on “remember”-level Bloom’s questions

To test ChatGPT’s performance across a range of Bloom’s levels, we revisited our out-of-class worksheets from Module 1: DNA-Protein Interactions and Module 5: Transcription and assigned a Bloom’s level to each rubric criterion (Fig 1; S1 Fig). First, we used our fill-in-the-blank experimental design question to evaluate ChatGPT’s performance on the lowest Bloom’s level of “remember” (Fig 1A). Provided with an experimental technique, students are asked to briefly describe the experimental steps and identify the method’s goal. Using DNA footprinting as an example, the respective answers would be: (1) serves to determine the region of DNA where a protein is bound, and (2) DNase-treating short pieces of protein-bound DNA that are then analyzed by gel electrophoresis. At the lowest level of Bloom’s, these questions are not meant to be challenging but instead encourage students to summarize a module’s experimental techniques. Doctoral students outperformed ChatGPT on this remember-level task with an average of 98% compared to 82% (Fig 1B).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Module 1 out-of-class worksheet components and Bloom’s taxonomy levels.

(A) Experimental approaches table in which students are provided with an approach and asked to describe the method and its goal. (B) ChatGPT performance on Bloom’s “remember” level questions compared to doctoral students. Each blank is worth 1 mark and scores have been normalized and plotted as mean ± SD. Mann-Whitney test; *p < 0.05 (n = 45) (C) Long-answer experimental design question from Module 1: DNA-Protein Interactions with (D) associated rubric used for grading.

https://doi.org/10.1371/journal.pone.0346127.g001

ChatGPT-4 underperforms on long-answer experimental design questions

We then sought to ask how ChatGPT performed on the higher levels of Bloom’s taxonomy that are built into our long-answer experimental design section (Fig 1D). In Module 1, students are provided with two hypotheses for how a fictional viral protein–MAYBELESS–may turn off expression of the deliciousfood76 gene in infected Drosophila melanogaster fruit flies to promote lethality. A transcription factor called EATSUM is known to be essential for expression of this gene. One hypothesis is that MAYBELESS binds at or near the deliciousfood76 promoter to occlude EATSUM binding. Students are asked to design a strategy to test each hypothesis including: (1) selecting and justifying a method (“analyze”), (2) concisely explaining the key steps (“understand”), (3) identifying and rationalizing positive and negative controls (“apply”), and (4) depicting and describing their experimental conditions and controls in the context of their conclusion (“apply”) (Table 1). As the version of ChatGPT used in this study cannot depict predicted results, we did not assess any depictions. We asked five postdoctoral-level bioscience educators who were not associated with our course to assess whether responses were generated by ChatGPT or students. They correctly identified 84% of ChatGPT-generated responses (true positives) and 91% of human answers (true negatives). ChatGPT achieved an average of 68% compared to 91% from a randomly selected student subset (Fig 2A). Doctoral students thus outperform GAI without prompt engineering, and ChatGPT importantly did not achieve the passing mark of 80% required to remain in good academic standing.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Testing ChatGPT across Bloom’s taxonomy levels with text-input-based long-answer worksheets.

(A) The overall score determined by experienced blind graders of iterations of ChatGPT against doctoral student performance on Modules 1 and 5. (n = 20) (B) Scores are broken down by Bloom’s taxonomy level as pre-determined in the rubric. (C) Overall score comparison between ChatGPT with additional prompt engineering to specify more detail about controls and doctoral students on Module 1. Mann-Whitney test; nd p > 0.05, *p < 0.05, **p < 0.01, ****p < 0.0001 (n = 10). (D) ChatGPT responses to a “create” Bloom’s level question were categorized and reported based on frequency of occurrence (n = 20).

https://doi.org/10.1371/journal.pone.0346127.g002

When we examined the specific Bloom’s levels associated with each rubric component, GAI performed similarly to doctoral students on the “understand” and “analyze” levels (Fig 2B). However, ChatGPT performed significantly worse on “apply”-level questions, which refers to applying positive and negative controls to new scientific contexts. Students are expected to describe the experimental condition and explain the purpose of the stated control. For the Module 1 DNA footprinting experiment, students should describe a no protein negative control to assess that DNase digestion achieved the single-base-pair resolution needed to interpret DNA footprints. ChatGPT often listed various experimental controls without classifying them as positive or negative or providing an associated rationale. One answer stated that: “Controls for this experiment would include treatments with no protein, EATSUM alone, and MAYBELESS alone to delineate specific interactions.” We hypothesized that this lack of detail could be resolved with simple prompt engineering, more similar to how GAI is used in practice [37]. After prompting ChatGPT with, “Please rewrite with more details about controls,” we repeated this blinded grading exercise with a new set of randomly selected student responses. ChatGPT frequently added a brief rationale to its controls, like “testing DNA with EATSUM alone to establish a baseline of binding” or “DNA alone to confirm that any assay signal is due to protein interaction.” With this change, ChatGPT achieved an average of 72% compared to a student average of 84%, a difference which was no longer statistically significant (p = 0.07; Fig 2C). GAI underperforms doctoral students on our long-answer experimental design questions largely due to a poor ability to answer Bloom’s “apply”-level questions pertaining to experimental controls, which can be improved in part with prompt engineering.

ChatGPT generates “create”-level hypotheses that lack precision

Despite a poor performance on apply-level questions, ChatGPT performed well on the next higher Bloom’s level of “analyze”, which involved selecting the best method to test the provided hypothesis. We therefore assessed GAI’s abilities on the highest Bloom’s level of “create”, which represents hypothesis generation according to the Blooming Biology Tool (Table 1) [35]. As generating coherent testable hypotheses can be a challenging learning objective for first-year doctoral students, we hypothesized that it would prove similarly difficult for ChatGPT. Specifically, we provided the background context described above (Fig 1C) and asked ChatGPT: “What are two possible hypotheses for how MAYBELESS disrupts the expression of deliciousfood76?” Across 20 ChatGPT responses, we strikingly found that GAI consistently generated plausible hypotheses. Some answers even aligned perfectly with our provided hypotheses, such as: “The MAYBELESS protein directly competes with the EATSUM transcription factor for binding to the 18-bp sequence in the promoter of the deliciousfood76 gene.” For evaluation, we categorized the answers into four categories: “Competitive inhibition at EATSUM binding site,” “Binds to and inhibits EATSUM,” “Chromatin remodeling,” and “Post-translational modification of EATSUM” (Fig 2D). The most frequently encountered hypothesis present in 80% of responses was “Chromatin remodeling”, which extends beyond the scope of Module 1. ChatGPT responses often listed multiple hypotheses within a broader category, such as changes in DNA methylation, histone modification, or chromatin remodeling. In contrast, we would expect students to precisely state single testable hypotheses, such as MAYBELESS sequestering EATSUM, or MAYBELESS inducing H3K9 methylation to reduce deliciousfood79 expression. Therefore, GAI can successfully answer “create”-level experimental design questions by generating coherent scientific hypotheses, but these hypotheses may lack the precision to be directly testable without additional prompt engineering.

ChatGPT performs poorly on DNA footprinting questions requiring image inference

Given GAI’s performance across Bloom’s taxonomy levels on our doctoral-level worksheets, we decided to explore alternative question designs. A key part of biomedical research is developing the ability to depict experimental results in a clear manner that is easily interpreted by other scientists. Although the version of ChatGPT we tested cannot directly output precise scientific graphs, it accepts image inputs. We therefore tested how well ChatGPT can process and interpret results as expected of first-year doctoral students. Given our earlier findings, we hypothesized that ChatGPT would perform poorly on questions at the “apply” level.

We thus designed a new question along similar lines to our Module 1 long-answer question. Students are studying how a transcription factor X regulates expression of a gene called HAPPY. We provided a poorly annotated 4-lane DNA footprint and asked students to: (1) identify the technique used to produce the gel (remember), (2) describe the key procedural steps (understand), and (3) interpret each lane as either experimental conditions or controls (apply) (Fig 3A). These Bloom’s levels match our rubric assignments from text-based inputs. Our answer key interpreted the four lanes of the footprint as an experimental titration of X from a no protein negative control in lane 1 to increasing amounts of X over lanes 2–4, consistent with an increasingly clear footprint.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Testing ChatGPT on the Bloom’s “analyze” level with an image-based DNA footprinting input.

(A) DNA footprinting analysis question provided to ChatGPT with either a DNA footprint gel that was either schematized or from primary data. (B) ChatGPT’s overall performance on this assessment with each image after blinded grading. Mann-Whitney test; nd p > 0.05 (n = 10). (C) ChatGPT’s performance across Bloom’s levels. (D) Distribution of experimental techniques identified by ChatGPT as being used to produce this gel (n = 20).

https://doi.org/10.1371/journal.pone.0346127.g003

First, we wanted to determine whether ChatGPT could successfully generate an interpretation for scientific data presented as either a primary or schematized DNA footprint (Fig 3A). We anticipated that ChatGPT would perform better on primary data, as this format would most likely be represented in its training data. Across ten answers per format, ChatGPT performed poorly, achieving only a 41% average with primary data and a 26% average with schematized data (Fig 3B). As there was no significant difference between images, we combined both samples for subsequent analysis. ChatGPT performed poorly across Bloom’s levels and respectively achieved averages of 26%, 42%, and 35% in the “remember”, “understand”, and “apply” levels. However, ChatGPT frequently stated the wrong technique used to produce the image, and DNA footprinting was only correctly identified in 45% of cases. Electrophoretic mobility shift assays, Western blot, or gel electrophoresis were respectively reported in 25%, 15%, and 10% of answers (Fig 3D). If we gave ChatGPT partial credit for naming steps relevant to DNA footprinting, such as radiolabelling DNA or running a gel, despite identifying the wrong technique, its performance on the “understand” level increased to 70%. Although still below the passing criteria, this improvement indicates that ChatGPT can consistently describe experimental steps for a given method.

On “apply”-level questions, ChatGPT was frequently unable to interpret the changing DNA footprint signal. In 33% of cases, GAI failed to identify the partial disappearance of bands in lane 2. Furthermore, lane 3 was stated to show the absence of DNA binding protein, similar to the negative control of lane 1, in 44% of answers (an incorrect interpretation). Additionally, ChatGPT failed to describe the titration of transcription factor X. Overall, GAI failed on this DNA footprinting image inference exercise.

ChatGPT performs consistently poorly on “apply”-level Bloom’s image inference across question types

GAI’s poor performance on our DNA footprinting experiment could conceivably be the result of the lack of training data on DNA footprinting in the model or the minimal information present on the provided footprint. To address these possibilities, we designed a new question using biolayer interferometry (BLI), given that it is a more recently developed technique. To avoid confounding our results with method identification, we informed ChatGPT that the graph was produced by BLI as part of a novel assay to detect antigen-specific antibodies in patient serum (Fig 4A). The question states: (1) describe what is happening in each different phase of the BLI experiment in the experimental context to produce the changes in signal (“understand” and “apply”), and (2) discuss what the three data lines represent (“apply”). Five phases are shown on the graph: baseline, loading, new baseline, association, and dissociation. Naming and describing the step is categorized as “understand”, while contextualizing the result is “apply.”

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Testing ChatGPT on the Bloom’s understand level with an image-based biolayer interferometry input.

(A) BLI data analysis question provided to ChatGPT with either a full graph including axis labels and titles, or a minimal graph missing these labels. (B) ChatGPT’s overall performance on this assessment after blinded grading. Mann-Whitney test; nd p > 0.05 (n = 10). (C) ChatGPT’s performance broken down by Bloom’s level. (D) Pie chart of number of phases identified in BLI data image. (E) Example multiple-choice questions provided to ChatGPT based on Full BLI image along with correct answer and results from GPT-4o with and without image (n = 30). (F) Overall results of ChatGPT across 14 multiple-choice questions. Mann-Whitney test; nd p > 0.05 (n = 30).

https://doi.org/10.1371/journal.pone.0346127.g004

First, we evaluated how the amount of information provided within the image affects ChatGPT’s performance by generating a full version complete with axis labels and phase demarcations, and a minimal version with the axis information removed (Fig 4A). Across ten answers per graph, we saw no difference in ChatGPT’s performance achieving an average of 50% (Fig 4B). When we broke the 20 scores down into Bloom’s levels, ChatGPT performed significantly better on the “understand” than “apply”-level questions, achieving respective averages of 40% and 25% (Fig 4C). Similar to our partial credit results for DNA footprinting, ChatGPT can discuss possible phases of a BLI experiment (“understand”) without implementing the experimental context (“apply”). However, ChatGPT only correctly identified five phases of the BLI signal in 5% of cases (Fig 4D). We found that GAI answered based on the number of phases chosen without necessarily matching its descriptions to the graph. For example, the association phase was described as the rise in the BLI signal despite the graph containing two signal increase events. Our BLI experiment reveals that ChatGPT lacks the ability to interpret scientific graphs across levels of visual detail.

To refine our assessment of ChatGPT on text- and image-based questions, we generated multiple-choice questions using the BLI graphical setup (Fig 4A). We categorized questions as requiring only background knowledge, which we deem a control (n = 8), or dependent on image interpretation (n = 6) (Fig 4E; Table 2). To directly test image interpretation, we prompted ChatGPT to answer each question 30 times where we provided the image and compared these responses against the same questions when no image was provided. In the control questions, ChatGPT achieved an average of 87% regardless of whether the image was present. On interpretation questions, ChatGPT achieved an average of 17% with the image compared to 4% without the image (Fig 4F). Notably, ChatGPT only successfully interpreted the image in a single case, resulting in this increased score. In contrast to knowledge-based questions, GAI shows a consistently poor performance on inferring meaning from scientific graphs in multiple-choice format.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Multiple-Choice Questions on BLI and ChatGPT Performance.

https://doi.org/10.1371/journal.pone.0346127.t002

Limitations

Our work has been conducted using ChatGPT with GPT-4o released May 2024, whereas the latest version, GPT-5.2, was introduced in December 2025. A limitation of our work is that the rate of advancements in GAI tools far outpaces individual studies. However, many of the reasoning errors observed in GPT-4o are observed in more recent versions of ChatGPT and appear to be long-staying properties of transformer-based large language models [38]. Alongside GPT-5.2, ChatGPT Images was also released, which allows for image generation. Although it is unclear how these developments improve upon image interpretation, the value of this study lies in identifying principles that help maintain student learning in the era of GAI. Another limitation of this work is that it is based on comparing ChatGPT’s performance against a re-assessment of past student submissions. A comparison that would allow ongoing modification of course assessments and eliciting student feedback would be particularly valuable but extend beyond the scope of this study. Finally, we have focused our analysis on our graduate-level molecular biology course grounded in experimental design. Whether and how these same lessons may apply to other disciplines or teaching goals remains to be determined.

Recommendations to design life sciences assessments that maintain learning with GAI usage

While GAI algorithms did not directly pass on our assignments, they remain powerful tools that can greatly assist students in completing out-of-class assignments. Given that differences in the accessibility or reliance on GAI between students could drive inequality in both learning and time usage, it is critical to design assessments that are capable of promoting student learning regardless of GAI usage. While objective metrics of effort like time may not directly correlate with learning, the student perception of effort may instead be more closely associated with learning gains following positive outcomes [25].

Below, we consider several lessons that educators may leverage to best ensure student learning in the era of GAI. These lessons are summarized in Table 3. First, we propose designing open-ended text-based questions that demand precision to exploit ChatGPT’s tendency to remain vague in its answers. This tendency of GAI models may be due to their deficits in performing complex reasoning, especially in novel contexts. Biosciences doctoral students are expected to propose hypotheses to explore in their theses through well-designed experiments. By designing open-ended problems, students must take a clear position to introduce the appropriate level of nuance and precision expected of their career stage. Any usage of GAI tools will require direct supplementation by student knowledge and experience to produce effective responses.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Tips for Promoting Student Learning in the Era of Generative AI.

https://doi.org/10.1371/journal.pone.0346127.t003

Second, we will prepare complementary rubrics that emphasize this need for scientific precision. Historically, we have allowed our TFs considerable liberty in assessing student worksheets to accommodate the variety of possible experimental designs that could address our open-ended questions. We have accordingly noticed that students often suggest multiple experiments where one would be sufficient. As presented in our study, a well-designed DNA footprinting experiment can determine if MAYBELESS occludes EATSUM from binding to the deliciousfood76 promoter. However, many students respond with a combination of DNA footprinting and BLI, or electrophoretic mobility shift assays and BLI. These answers likely represent attempts to offer more information to reduce the chance of being incorrect. Through revised rubrics, our TFs will now be provided with the ideal answer to the question and only grade the first experiment submitted by students. As part of this process, we will explicitly ask students to justify their chosen technique–a skill required in writing scientific proposals. Altogether, we hope to push students to learn scientific precision by coupling a single technique with critical experimental control to answer our open-ended questions.

Third, take-home assignments should focus on “apply”-level Bloom’s questions which involve implementing known procedures in new situations (Table 1), and fourth, we suggest incorporating image inference where appropriate into life sciences assessments for data interpretation. Beyond moving between Bloom’s levels given text inputs, we designed several questions that relied on both image and text inputs. When provided with a DNA footprint or BLI graph, ChatGPT’s performance in data interpretation was relatively poor. Relative to understand-level questions, ChatGPT struggled primarily with “apply”-level Bloom’s questions. As with our text-based questions, the model consistently performed poorly when asked to interpret portions of the graph within the biological context of the question. In contrast to vague answers to text inputs, ChatGPT was often confidently wrong when provided with image inputs. In our DNA footprinting experiments, ChatGPT only correctly identified the technique in 45% of cases yet continued to answer the questions with an apparent logical flow. Similarly, our BLI experiment revealed that even with five clearly demarcated phases on the graph, ChatGPT could not properly name and describe the five phases and often only identified three or four. In fact, when ChatGPT did describe five phases, it explicitly acknowledged that the regeneration phase was not present on the graph. However, we could not directly compare the performance of ChatGPT against doctoral students in these newly designed questions.

Fifth, combining multiple-choice questions with image inference forces ChatGPT to overcome the tendency to remain vague and select an answer that may be incorrect. To assess the difference in ChatGPT’s performance between the “understand” and “apply” levels more formally, we generated multiple-choice questions that were either knowledge-based controls or image inference-dependent cases. In control questions, ChatGPT achieved an average score of 87% compared to a poor performance of 17% on the inference cases. When we repeated this experiment without providing an image to ChatGPT, the model maintained an 87% average on control questions but scored only 4% on the cases. This drop was driven by successful inference on a single question. For GAI tools, these responses are called hallucinations–text that is written in a natural manner that is non-sensical in meaning or unfaithful to its training data. While these factual inaccuracies are accepted occurrences, biological scientists have instead described them as fabrications and falsifications [44]. Compounding with this phenomenon, the overall ability of ChatGPT to interpret scientific images remains poor. The consistent results on most of the image inference questions, regardless of the image being present, suggest that ChatGPT may be completely ignoring images in most situations despite GPT-4o being specifically designed to handle multi-modal inputs such as combined image and text. As we push our students for scientific precision, multiple-choice questions will force students to think through ChatGPT-generated answers or risk being misled by a fabrication. Notably, this process is fundamentally aligned with doctoral training as students must assess the information presented for accuracy similar to reading papers. Recent work has similarly found that ChatGPT’s effectiveness as a writing aid depends largely on its strategic usage by doctoral students [45]. For educators, we further note that such multiple-choice questions can equally span the spectrum of Bloom’s levels to appropriately challenge both ChatGPT and promote student thinking (Table 2).

Sixth, complementing these insights for refining out-of-class assignments, we also encourage educators to use in-class activities to assess learning objectives and build professional competencies. While it is impossible to monitor student use of GAI outside of the classroom, GAI use can be restricted inside the classroom. Opportunities such as chalk talks and TF-led small group discussions are already core components of our course, and we suggest using activities like these to assess learning objectives that can no longer be authentically assessed outside the classroom.

Finally, we encourage all educators to run their assessments through ChatGPT, or a similar GAI algorithm. While our strategies are derived from our study using graduate-level experimental design in molecular biology, whether these same strengths and weaknesses hold true for ChatGPT across disciplines remains to be determined. Our results nevertheless align with recent work showing a limited ability in clinical reasoning and practical skills when tested in a medical curriculum [46]. A limitation of the present work is that GAI tools are being continuously improved, and it is unclear whether our findings and those of existing studies would hold when tested on the latest algorithms.

Collectively, our analyses of ChatGPT’s performance on our existing and modified BCMP 200 worksheets reveal that generative AI is a powerful tool that can be leveraged for the completion of out-of-class assignments. To maintain the educational value of these assessments, we share several insights into how best to design questions that support effortful engagement with the material even with GAI usage by encouraging greater refinement of generated answers whether manually or through prompt engineering (Table 3). We encourage life sciences educators to focus on asking interpretation-based questions, rather than purely knowledge-based ones, across text and image inputs. In particular, we highlight how GAI fails to perform with the scientific precision expected of doctoral students on “apply”-level questions. In the context of selecting experimental controls necessary for data interpretation, this core conceptual threshold for doctoral students presently remains out of reach for ChatGPT. Additionally, we find a distinct utility in multiple-choice questions that rely on image interpretation to capitalize on the possibility of hallucinations in ChatGPT’s answers. Critically, this question format is highly scalable for larger classroom settings. Overall, scientific precision and data interpretation stand out as weaknesses of current GAI algorithms when applied to doctoral-level coursework. These findings will not only help re-design worksheets to ensure student learning but also help establish tips for assessment design in the era of generative AI.