Introduction

Large language models (LLM) are a type of artificial intelligence (AI) trained by deep neural networks on datasets that contain billions of parameters [1]. These models possess advanced capabilities in natural language processing, generation, and critical reasoning. One of the most well-known and easily accessible LLMs is Open AI’s ChatGPT [2]. Other LLMs, such as Meta’s LLaMA and Google’s Bidirectional encoder representations from transformers (BERT) have been made available and shown potential to perform language processing tasks [3,4].

LLMs are increasingly gaining attention for their potential applications in health care as a diagnostic or educational tool [5,6]. Researchers demonstrated that ChatGPT can accurately answer biomedical and clinical reasoning questions on the United States Medical Licensing Examination (USMLE) at a level that approached or exceeded the passing threshold [7]. Several other studies have since investigated the ability of LLMs to respond to clinical scenario questions often with moderate accuracy [8–10]. These studies often cite limitations in the training dataset and lack of genuine understanding of the clinical scenarios as reasons for LLMs’ suboptimal performances [9]. However, little attention has been paid to advanced prompting techniques. Furthermore, most investigations rely on a single LLM, typically GPT-3.5, without exploration of other models. One such study recently investigated the ability of ChatGPT to respond to clinical case scenarios. The authors found that GPT-3.5 had difficulty interpreting clinical findings and that responses lacked factual accuracy. They therefore concluded that ChatGPT has limited utility as a diagnostic tool [11].

The aim of the current study is therefore to explore the potential role for LLMs as an educational tool for medical trainees by using three LLMs and advanced prompt engineering techniques.

Methods

This study aimed to replicate and expand upon prior research investigating the diagnostic accuracy and utility of large language models in clinical decision-making. We compared the performance of three LLMs (llama-3.1-70b-versatile, llama-3.1-8b-instant, and mixtral-8x7b-32768) against previously reported GPT-3.5 performance.

Results

Correct response rate

Out of 150 cases, correct responses were provided for 119 (79%), 97 (65%), and 93 (62%) cases by llama-3.1-70b-versatile, llama-3.1-8b-instant, and mixtral-8x7b-32768, respectively. The answer provided by these LLMs aligned with the response given by the majority of Medscape users in 94 (63%), 90 (60%), and 96 (64%) cases, respectively. In all three instances, these three models outperformed responses provided by GPT-3.5 in the previous analysis (Table 1).

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Summary of LLM responses. P-values refer to Chi-Square tests performed between the LLMs selected by us and GPT-3.5 responses reported by Hadi et al. [11].

https://doi.org/10.1371/journal.pone.0325803.t001

Sensitivity analysis

Correct response rates were significantly worse when presenting llama-3.1-70b-versatile, llama-3.1-8b-instant, and mixtral-8x7b-32768 with question stems using basic prompting compared with advanced prompting techniques (p = 0.0021, p < 0.0001, p = 0.00370, respectively; Supplementary results, page 6; Table S1 in S1 Appendix). When comparing the responses generated from the three LLMs using basic prompting with previously reported GPT-3.5 results, only llama-3.1-70b-versatile outperformed GPT-3.5 (Supplementary results, page 6; Table S1in S1 Appendix). When comparing the responses generated from the three LLMs with GPT-4.0 Turbo results using advanced prompting for all models, only llama-3.1-70b-versatile outperformed GPT-4.0 Turbo (Supplementary results, page 6; Table S2 in S1 Appendix).

Upon review of the cases, 22 did not contain any images in the case description or physical exam/investigation findings. 85 contained images and provided a description or analysis of the image contents. 43 cases contained images of either physical exam findings or investigations, but did not provide a description of the image contents. We performed a sensitivity analysis excluding clinical cases with images that did not provide a description of the imaging findings to mimic the inclusion criteria initially proposed. Correct responses were provided for 87/107 (81%), 70/107 (65%), and 72/107 (67%) questions by llama-3.1-70b-versatile, llama-3.1-8b-instant, and mixtral-8x7b-32768, respectively. GPT-3.5 showed a correct response for 64/107 (60%) questions (Table 2).

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Summary of sensitivity analysis excluding cases that contained images but did not provide a description of the image content (n = 107). P-values refer to Chi-Square tests performed between the LLMs selected by us and GPT-3.5 responses reported by Hadi et al. [11].

https://doi.org/10.1371/journal.pone.0325803.t002

Discussion

In this analysis of the performance of three different LLMs when answering clinical practice scenarios, all models achieved a high rate of accuracy using advanced prompting strategies. Response rates improved when cases with clinical images were excluded from the analysis. Our findings refute previous claims that LLMs are not accurate diagnostic tools and highlight the importance of prompt engineering. Future studies should focus on education around prompt engineering to optimize LLM usability.

In recent years, interest in the use of AI technologies in healthcare applications has risen [1]. Many studies have investigated the potential of LLMs in medical education and assessed their ability to accurately respond to clinical scenario questions. However, the majority of this research has relied on basic prompting strategies that did not fully leverage LLM capabilities [10,11,20–22]. Basic prompting involves presenting LLMs with simple, straightforward question that are often generic and provide minimal context. By contrast, advanced prompting employs prompt engineering to provide LLMs with more context, multi-step instructions, and often uses output format constrains or examples to optimize their performance on various tasks. These techniques allow LLMs to better understand the nuances of challenging tasks, such as medical case scenarios [15,18]. Response rates were significantly improved when using advanced prompting techniques, demonstrating that effective prompt engineering can optimize LLM performance and lead to significantly higher accuracy when answering clinical scenarios even when lower parameter models are used. Yuan et al. have shown that advanced prompting strategies can lead to a higher rate of correct responses and more succinct and relevant outputs when presenting GPT-4.0 Turbo with management questions in the field of gastrointestinal oncology [23]. Prompt engineering is a crucial skill for health care providers wanting to interact with AI systems, but frequently overlooked in the health care literature [15]. By better utilizing and understanding prompt engineering, we can realize the full potential of LLMs in transforming health care. Future research should therefore prioritize the development of educational resources and training programs focused on prompt engineering.

One example of prompt engineering is one- or multi-shot prompting. These approaches provide the model with a single or multiple examples of the desired task or output [24]. This understanding of the underlying tasks can better equip the model to carry out more complex tasks such as clinical reasoning. Most studies assessing the capabilities of LLMs in healthcare, however, have relied on zero-shot estimates where models are expected to perform tasks solely relying on their pre-existing knowledge and the prompt instructions without providing specific examples in advance [24]. Reports assessing ChatGPTs capabilities in healthcare applications frequently cite limitations in its training dataset as reasons for suboptimal results [9,25]. Specialized LLMs, such as BioBART which was pretrained on PubMed abstracts to deliver it with specialized knowledge in the biomedical domain or Med-PaLM tailored and tested on the medical domain, have been developed to overcome this limitation with demonstrated improvement in biomedical domain performance benchmarks [26,27]. However, as we have demonstrated in our study, one-shot prompting can lead to more accurate results even when smaller parameter models without specialized training datasets or GPT-4.0 Turbo are used. The assumption that LLMs will naturally perform well on medical tasks without tailored prompting overlooks the complexity of medical knowledge and reasoning. It also leads to an underestimation of their true capabilities. Future studies should focus on multiple-prompt techniques that provide the LLM with better contextual understanding to leverage their true capabilities.

ChatGPT is one of the first and most well-known LLM due to its public availability and easy accessibility via any browser [9]. Few studies investigating LLMs in healthcare have focused on other LLMs [9]. OpenAI regularly releases model updates and recently deprecated GPT-3.5 [28]. Since GPT is proprietary to OpenAI, deprecation of models implies that users will no longer be able to access these versions. This has significant implications when these models are used in research practice as future authors will no longer be able to replicate previous studies. In research, the ability to replicate findings is fundamental to validate study results and advance knowledge. Reliance on proprietary LLMs therefore limits long-term research advancement. On the other hand, open-source models like Llama and Mixtral that were employed in this research are more transparent on their underlying architecture and training data. These models also remain accessible over time allowing future researchers to replicate and verify study results. Future studies should therefore focus on open-source models that provide long-term accessibility.

This study has several limitations. First, since GPT-3.5 has been deprecated from the online platform and the application programming interface has reference to multiple versions of GPT-3.5 and the specific model is not referenced [11]. Therefore, we were unable to replicate previous study findings using our own algorithm. Second, all LLMs evaluated in this study were trained on publicly available datasets after the cutoff date for Medscape questions, which might give them marginal knowledge advantage over the previously assessed GPT-3.5. Third, our study was designed to demonstrate that LLM performance can be improved when using advanced prompting strategies. Therefore, we did not contrast different advanced prompting strategies. Future research should focus on assessing advantages and disadvantages associated with different prompt engineering techniques. Fourth, the Medscape Clinical Challenge questions, while designed for medical professionals, may not fully represent the complexity of real-world clinical scenarios. The questions are structured for online consumption and may inadvertently simplify or standardize clinical presentations. Furthermore, the use of multiple-choice questions as a proxy for clinical reasoning skills has inherent limitations. While such questions can assess knowledge recall and basic application, they may not fully capture the nuanced, iterative process of clinical decision-making that often involves gathering additional information, considering patient-specific factors, and weighing multiple potential diagnoses simultaneously.

Using AI in medical diagnosis and education raises important ethical considerations that must be carefully approached. Though our study demonstrates the potential of LLMs in accurately answering clinical questions, it is important to recognize that these tools should act as supplement or aid and, critically, not replace, human clinical judgment. Future research should explore the use of vision language models (VLMs) to understand and interpret clinically relevant images. This could significantly enhance the diagnostic capabilities of AI systems by allowing them to analyze visual data. Additionally, future projects could explore the potential of VLMs in medical education, examining how these might be used to train medical learners to interpret clinical images.

This study demonstrates LLMs can achieve high accuracy in responding to clinical practice scenarios when advanced prompting strategies are used. Future research should incorporate prompt engineering into their study design. Additionally, there is a need to address the limitations associated with overreliance on proprietary models like ChatGPT. Version deprecation, as seen with GPT-3.5, hinders accessibility and prevents the replication of research findings. To ensure long-term research viability and reproducibility, future studies should focus on open-source models that provide transparency and sustained accessibility over time.

Supporting information