Introduction

Artificial intelligence has advanced rapidly in recent years, leading to the integration of generative large language models (LLMs) into essential healthcare services. These models are now used in applications from clinical knowledge management to telemedicine consultations. The medical field, with its complex knowledge requirements and critical decision-making contexts, provides an important opportunity to evaluate how well artificial intelligence can perform in real-world scenarios. The study in medical education demonstrates that GPT-4 can improve medical students’ diagnostic accuracy by 19% [1]. Furthermore, extensive research shows that LLMs have significant clinical knowledge, performing at expert levels across various medical specialties [2–4].

The Chinese medical licensing examination is one of the world’s largest clinical competency assessments, using various question types to evaluate foundational knowledge, clinical reasoning, and practical application skills. While researchers have investigated large language model performance on the United States medical licensing exams and similar assessments [5,6], there is a lack of comprehensive research examining how LLMs perform when evaluated across different languages and question types [7].

This research presents a new mixed-format evaluation framework for Chinese medical licensing examination, utilizing six major large language models: OpenAI’s ChatGPT-o3 and ChatGPT-4o, Google’s Gemini-2.5-pro, DeepSeek’s R1 and V3 models, and ByteDance’s Doubao-1.5-pro [8,9]. The purpose of this study is to investigate the performance of LLMs on Chinese medical licensing examination, with the goal of providing valuable insights for medical education and training.

Materials and methods

Response generation

All models were accessed through their official web interfaces using default parameters. Below are the input methods for different question types: for type A questions, each question was entered in a fresh conversation session; for type B questions, questions sharing the same stem were entered together in a single session, followed by a new conversation session for the next set of questions with a different shared stem. For type C questions, questions sharing the same set of options were entered together in one session, with a new conversation session initiated for the next group of questions with different shared options. During the experiment, research assistants used a standardized prompt: “You should act as an educational expert of Medicine. When faced with a choice question of type A, where each question has only one best answer, you need to provide the optimal response. The question is “The primary damage to DNA by ultraviolet rays is the formation of: A. Base deletion; B. Base insertion; C. Base substitution; D. Thymine dimers; E. Phosphodiester bond cleavage”. You are required to present your answer in the following structure: Response: Answer.” (Fig 1). Two medically trained assistants independently verified each model’s answer against the official key, labeling responses as correct or incorrect.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Schematic diagram of the English version of the structured prompt template.

Yellow-colored placeholder denotes the question type (A, B, C), while blue-colored placeholder denotes the question stem. Type A (Single-Best-Answer Questions): Direct, standalone multiple-choice items that test the retrieval of basic medical facts. Type B (Case-Based Analysis Questions): Full-case prompts—including chief complaint, medical history, and diagnostic findings—designed to evaluate a candidate’s clinical reasoning processes. Type C (Shared-Option Questions): Multi-part items that present a common set of five diagnoses and then pose sequential questions on pathogen identification and radiologic feature matching, thereby probing candidates’ ability to transfer and apply their knowledge in multidimensional diagnostic scenarios.

https://doi.org/10.1371/journal.pone.0346518.g001

Results

Each model’s overall accuracy (mean ± SD) was calculated across all question types and languages: ChatGPT-4o 82.8% ± 3.7%, ChatGPT-o3 86.1% ± 2.5%, Deepseek-V3 89.8% ± 1.4%, Deepseek-R1 91.0% ± 1.5%, Doubao-1.5-pro 92.0% ± 1.3%, and Gemini-2.5-pro 88.7% ± 1.7% (Fig 2). All models performed most reliably on Type A questions, maintaining accuracy above 88% on average. Type C questions, however, showed the greatest variability in performance, ranging from 72% to 95%. ChatGPT-4o scored 88.2% on Type A, 82.3% on Type B, and 71.7% on Type C in Chinese, while in English the scores were 97.4%, 77.4%, and 80.0% respectively. Deepseek-R1 stood out with particularly strong performance on Type C questions, achieving 95.0% accuracy in Chinese and 88.3% in English, which was superior to most other models in our evaluation. Notably, Doubao-1.5-pro, Deepseek-R1, and Deepseek-V3 all showed strong cross-lingual stability, with language‐induced accuracy differences under 5%. For Type C questions, the performance difference between Deepseek-R1 (the top model) and ChatGPT-4o (the lowest-performing model) was quite large—almost 25% (Fig 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Performance evaluation of generative language models on Chinese medical licensing examinations.Heatmap displays accuracy (%) of six large language models across three bilingual question formats: Type A, Type B and Type C. Columns represent models, while rows denote question-type-language combinations. Color gradient reflects model competency levels, with deep blue (>90%) indicating clinical reliability thresholds and yellow (<75%) signaling high-risk zones requiring human oversight.

https://doi.org/10.1371/journal.pone.0346518.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. The average accuracy, along with its standard error range, of different large language models (Large language models) in answering all 198 questions.

https://doi.org/10.1371/journal.pone.0346518.g003

There was a significant main effect of question type (F(2,10) = 10.22, P = 0.0038), but no significant effect of language (F(1,5) = 0.39, P = 0.56) (Fig 4). A marginal interaction between question type and language was observed (F(2,10) = 3.57, P = 0.068). The performance of Type A was significantly better than both Type B (P = 0.002) and Type C (P = 0.025). However, neither the Type B vs. Type C comparison nor the Chinese vs. English comparison reached statistical significance after correction.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. A violin plot comparing accuracy distributions between Chinese and English in educational question answering.

The central line indicates the median score, with box edges marking the 25th and 75th percentiles, defining the interquartile range (IQR) of the distributions.

https://doi.org/10.1371/journal.pone.0346518.g004

Discussion

This study systematically evaluated the performance of six LLMs on Chinese medical licensing examination questions across both Chinese and English versions and different question types. Our comprehensive analysis yielded several key findings. Question type emerged as the primary factor influencing model performance, with a significant main effect observed. All models performed most consistently on basic knowledge questions and complex integrative questions revealed substantial performance variability. Language had minimal impact on overall performance, with no significant main effect detected. This stability suggests these models have developed robust semantic representations that transcend surface linguistic differences.

We also found that complex questions have the ability to distinguish between different models, which is in line with cognitive theories. Cognitive theories suggest that higher – order thinking tasks can better highlight differences in reasoning abilities. Several studies have shown that models with specialized reasoning capabilities, such as those using chain – of – thought prompting or other reasoning – enhancement techniques, usually perform better than general – purpose models when dealing with complex medical questions [10,11]. Our research also made a similar discovery. Models like Doubao-1.5-pro and Deepseek-R1, which may adopt more advanced reasoning architectures, performed particularly well on complex questions [12,13]. Our research further expands on previous research findings. We found that as the complexity of questions increases, the performance gap between different models becomes larger. For example, on Type C questions, the difference in accuracy between Deepseek-R1 and ChatGPT-4o was nearly 25%. This indicates that complex integrative tasks may be very useful for evaluating and comparing the reasoning abilities of different LLMs in the medical field.

Our findings align with and extend previous research evaluating LLM performance on medical licensing examinations in China and internationally. Prior studies have shown that LLMs generally perform well on foundational recall questions but exhibit greater variability on complex reasoning tasks, consistent with our observation of significant question‑type effect [12,14]. Other evaluations of Chinese medical licensing examinations similarly reported that English prompts may improve performance on basic knowledge tasks but offer limited benefit for higher‑order reasoning [15]. Additionally, recent work in dental and specialty examinations demonstrated that cross-lingual performance varies substantially across models, highlighting the importance of bilingual semantic alignment [16]. Our study contributes to this literature by providing a direct cross‑lingual comparison across three question types and identifying models with particularly strong cross‑lingual stability.

Some previous studies have found that using English prompts can make models do better in basic knowledge tasks [17–19]. But our research results show that this advantage fades when it comes to more complicated clinical reasoning tasks. This is in line with other research which points out that although large English language databases can help with simple questions, they don’t offer much help for complex medical reasoning [20,21]. It’s really worth noticing that models like Doubao-1.5-pro, Deepseek-R1, and Deepseek-V3 show strong stability across different languages. This finding fits well with other studies that prove models trained with bilingual biomedical data and semantic alignment methods can have great cross-lingual performance [22–24].

This study has several limitations. Although the passing standard for this examination is 60% accuracy, the results of this study do not indicate that these LLMs can pass the real examination. This is because the analysis was based on only 198 questions rather than the complete examination set, and it was limited to text-based single-choice questions, excluding items containing images, tables, chemical formulas, and other non-text modalities that are typically included in the full examination. This study only focused on accuracy, neglecting consistency, rationale, educational value, and clinical validity. Future studies are needed to address these issues. Extending assessments to various dimensions and real-world scenarios will better establish LLMs’ utility and reliability in clinical education.

In conclusion, this study provides a comprehensive evaluation of six LLMs on the Chinese national medical licensing examination across languages and question types. A central finding is that question type, rather than language, is the primary determinant of model performance. All models showed consistent accuracy on basic knowledge questions but exhibited substantial variability on complex questions. Among the models evaluated, Doubao-1.5-pro and Deepseek-R1 demonstrated superior overall performance and maintained strong cross-lingual stability, with low language-related accuracy differences. These findings have important implications for the development and deployment of LLMs in medical education.

Acknowledgments

We would like to express our sincere gratitude to Dr. Chuanbing Wang for his kind assistance in using ChatGPT and Gemini during his stay in the United States in May 2025.