Source of questions

The test questions are extracted from the nursing (intermediate) simulation papers in the examination book designated by the Talent Exchange Service Center of the National Health and Wellness Commission of China and are in the form of objective MCQs, with a total of 400 questions divided into four units (Basic Knowledge, Relevant Professional Knowledge, Professional Knowledge, Professional Practice Ability). The basic knowledge unit examines the etiology and pathogenesis of common clinical diseases and diseases, and auxiliary examinations, of which internal medicine accounts for 35%, surgery accounts for 35%, obstetrics and gynecology accounts for 15%, and pediatrics accounts for 15%; the relevant professional knowledge examines nursing health education, hospital infection nursing, nursing management, and Chinese medicine nursing, of which nursing health education, hospital infection nursing, and nursing management each accounts for 30%, and Chinese medicine nursing accounted for 10%; professional knowledge examination of internal medicine, surgery, gynecology, pediatrics specialties of common clinical diseases, clinical manifestations, treatment points, of which internal medicine, surgical each accounted for 30%, gynecology, pediatrics accounted for 20%; professional practice ability to examine the internal medicine, surgery, obstetrics and gynecology, pediatrics comprehensive nursing content, internal medicine, surgery each accounted for 30%, gynecology, pediatrics accounted for 20%. The questions are divided into two types, A and B. Type A is a best-answer multiple-choice question; type B is a set of questions that provides a number of groups of questions, each of which shares five alternative answers, A, B, C, D, and E, listed at the front of the question, from which the answer that is most closely related to the question is chosen, and each alternative answer may be chosen once, multiple times, or not chosen.

Model version and parameter control

We selected DeepSeek-R1 and GPT-4o (via API) for comparison based on their relevance and availability at the time of study. DeepSeek-R1 represents a new domestic large language model with increasing adoption in Chinese academic and clinical settings, while GPT-4o represents the latest widely recognized international state-of-the-art model. This pairwise comparison allowed us to examine performance differences between a cutting-edge domestic model and a global benchmark within the context of the Chinese Health Professional and Technical Examination.

Both models were configured with a temperature parameter of 0 (deterministic sampling mode) to ensure answer consistency across repeated queries [25]. Generation constraints included: ①maximum output length of 10 tokens to enforce single-character responses (A-E), ②suppression of explanatory text generation, and ③mitigation of formatting errors through restricted reasoning pathways, thereby eliminating response redundancy associated with excessive model deliberation.

Standardized treatment

In order to ensure the standardization and comparability of the model response, this study designed a standardized prompt template for each question with the following structure: ①Mark the question stem with Clearly define the role of the model and the response requirements, the first line begins with [instruction], emphasizing the identity of the model as the “nurse in charge”, and requesting that the model “output only the letters of the options”;②[title], retaining the key clinical information and option content of the original question. “The first line begins with [instruction], emphasizing the model’s status as a “nurse practitioner in charge” and asking it to “output only the letters of the options the question stem is marked with [title], which retains the key clinical information and options in the original question. At the end of the question, it is stated in parentheses “(Answer Requirement: Output Option Letters Only)” to strengthen the output format restriction.; [2] ③ Remove the marking symbols in the original question and unify the Chinese and English punctuation to ensure the consistency of the input text; if the options in the original question contain alphabetic suffixes, they will be changed to consecutive capital letters; if the model outputs non-alphabetic characters, the first letter will be extracted automatically; if more than one letter is output, it will be regarded as an error in the format and marked as an invalid response.To enhance transparency and reproducibility, representative examples of the standardized prompt template and model outputs are provided in the S1 Fig. This approach was designed to evaluate the model’s factual knowledge based on its ability to select the correct answer without introducing confounding variables such as reasoning complexity or verbose explanations. By constraining the output to a single letter, we minimize the potential for variability in reasoning styles, focusing purely on whether the model “knows” the correct answer.

Assessment of indicators

Statistical analysis

Descriptive statistics were used to summarize model performance across all evaluation metrics. Categorical variables—including accuracy, consistency, and consistent accuracy—were presented as frequencies and percentages (n, %). Consistency was defined as agreement across at least 4 of 5 repeated responses for each item (80% threshold). Consistent accuracy was calculated as the proportion of items for which responses were both consistent and correct. Chi-square tests (α = 0.05) were used to compare performance differences between DeepSeek-R1 and the GPT-4o API across examination units, question types, and clinical disciplines. Because the stratified analyses involved multiple simultaneous comparisons, Holm–Bonferroni correction was applied to adjust p-values and control the family-wise error rate. Raw p-values were used for overall accuracy comparisons and graphical visualizations, whereas adjusted p-values were reported for consistency-based analyses.Invalid responses caused by format errors (e.g., outputs not conforming to single-letter requirements) were counted and excluded from comparative statistical analyses but reported descriptively. All preprocessing was conducted in Microsoft Excel 2021, and statistical tests were performed using SPSS version 29.0. Statistical significance was interpreted using conventional thresholds (***p < 0.001, **p < 0.01, *p < 0.05).

Ethics approval and consent to participate

This research utilized exclusively publicly accessible data from non-sensitive sources, encompassing neither human subjects, confidential health records, nor animal experimentation. As such, the research complies with the World Medical Association Declaration of Helsinki guidelines for non-interventional studies involving publicly available information, thereby exempt from institutional ethics review board approval requirements.

Comparison of advantages and disadvantages and analysis of causes

The superior overall accuracy of DeepSeek-R1 suggests that it may have stronger alignment with the linguistic and conceptual structure of the Chinese nursing examination, as well as potentially more effective representation of Chinese medical text. While the specific training corpus of DeepSeek-R1 is not publicly disclosed, it is plausible that domain-relevant content or Chinese-language clinical materials contributed to its performance advantages. For Chinese nursing education and clinical knowledge updating, such characteristics may allow DeepSeek-R1 to provide comparatively more reliable factual references.In contrast, although GPT-4o API demonstrated higher response consistency, its consistently correct rate was substantially lower. This pattern indicates the presence of “stable but incorrect” responses, reflecting systematic rather than random error. Prior research has shown that systematic LLM errors can reinforce misconceptions among learners and introduce biased or unsafe content into educational materials [26]. In clinical decision-support scenarios, repeated incorrect outputs may pose risks if users rely on them without verification, particularly in high-stakes domains such as drug dosage calculation or diagnostic reasoning. DeepSeek-R1, by achieving both high consistency and high consistent accuracy, appears more reliable in this regard. These findings underscore the importance of evaluating not only model stability but also the correctness of stable responses. Accordingly, healthcare professionals should apply LLM output with critical appraisal and contextual judgment to safeguard patient safety.Differences across disciplinary domains may reflect variations in model sensitivity to specific medical knowledge structures. After adjustment for multiple comparisons, DeepSeek-R1 retained significant advantages in the basic knowledge unit as well as in surgical and gynecological nursing, suggesting greater robustness in areas characterized by standardized procedures and structured clinical reasoning. Disciplines such as pediatrics showed a trend favoring DeepSeek-R1, although statistical significance was not retained after correction, indicating that additional data may be needed to confirm domain-specific differences. Other domains, including basic nursing, nursing health education, hospital infection nursing, and Chinese medicine nursing, showed no significant differences between models, which may relate to their broader conceptual knowledge bases or, in the case of Chinese medicine nursing, the presence of culturally specific theoretical content that both models may represent only partially [27]. he performance differences observed between DeepSeek-R1 and GPT-4o may also relate to differences in model architecture and semantic processing, as prior studies suggest that LLMs vary in their handling of complex logical dependencies and domain-rich language [28]. Future research could further investigate which architectural features or training strategies enhance model performance on medical text, including analyses of error patterns, linguistic sensitivity, and domain-specific reasoning pathways.

Differences in the suitability of disciplines and question types

After multiple-comparison adjustment, DeepSeek-R1 continued to show stronger performance than GPT-4o in the clinical disciplines of surgical nursing and gynecological nursing, suggesting that its training may better capture procedural knowledge, structured clinical pathways, and guideline-based decision-making. These domains typically involve clear diagnostic frameworks and standardized nursing interventions, which may align more closely with DeepSeek-R1’s model characteristics. This pattern provides practical implications for medical education: DeepSeek-R1 may serve as a more reliable auxiliary tool when teaching clinically structured content that requires procedural accuracy.In contrast, disciplines such as basic nursing, internal medicine nursing, nursing management, hospital infection nursing, and Chinese medicine nursing did not show significant differences between the two models after correction. The lack of performance gap may stem from the nature of these domains, which contain a higher proportion of generalizable principles or culturally specific knowledge (e.g., in Chinese medicine nursing), where both models may have limited or similar depth of domain representation. For areas grounded in traditional theory, existing LLMs may require more targeted training data or domain-specific adaptation to achieve meaningful improvements [29].Regarding question types, DeepSeek-R1’s advantage remained statistically significant for Type A questions (standardized, single-scenario items), indicating stronger stability in structured reasoning and factual recall. Although DeepSeek-R1 also outperformed GPT-4o in Type B questions (complex, multi-clue scenarios), the difference did not remain significant after Holm correction. Nevertheless, the directional trend still suggests that DeepSeek-R1 may hold greater potential in handling integrated clinical information, an ability that is essential in real-world nursing where dynamic and multifaceted clinical cues must be synthesized for decision-making [30]. GPT-4o’s comparatively lower performance in standardized items indicates room for optimization in addressing straightforward factual content, and the contrast between the two models reflects fundamental differences in their problem-solving strategies. These differences provide valuable insights for model refinement and for selecting appropriate LLMs for specific educational or clinical tasks.

Clinical application risks and coping strategies

The risk posed by GPT-4o API’s low consistent correctness, despite its high response agreement, warrants close attention. The phenomenon of “stable but wrong” outputs carries different implications across settings. In educational contexts, repeated exposure to incorrect answers may reinforce learner misconceptions and hinder the development of accurate clinical knowledge structures. In clinical environments, however, the potential consequences are more serious. For domains such as hospital infection prevention and control, recurrent erroneous associations could increase the risk of unsafe practices if users adopt model outputs uncritically, thereby posing threats to patient and staff safety and potentially contributing to malpractice events [31]. These findings illustrate that high consistency does not equate to reliability, and systematic errors may be more hazardous than random inaccuracies.

Given these risks, GPT-4o API should ideally be paired with a human auditing mechanism. Yet manual auditing is time-intensive, resource-demanding, and subject to variability in evaluators’ professional expertise [32]. To improve efficiency, the development of intelligent auditing tools—such as systems incorporating rule engines or knowledge-graph–based verification—may offer a scalable approach for automated error detection [33]. The proposed “Dual Model Cross-Validation” strategy represents another potential safeguard by leveraging complementary strengths of different LLMs. However, in addition to technical challenges such as managing data interchange and resolving conflicting outputs, practical considerations must be addressed. These include increased computational burden, integration into existing clinical or educational workflows, and the continued need for human oversight when models disagree. One feasible direction may be to apply dual-model validation selectively in high-stakes scenarios—such as medication safety, infection control, or ethically sensitive decision-making—while allowing single-model outputs to support lower-stakes learning tasks [34]. Future research should evaluate the cost-effectiveness, usability, and real-world safety implications of multi-model validation frameworks in authentic teaching and clinical environments. Systematic assessments using real clinical workflows will be essential to determine whether such strategies can meaningfully mitigate the risks associated with systematic LLM errors.

Research limitations and future research directions

This study has several limitations that should be acknowledged. First, only multiple-choice questions (MCQs) were used, representing a single question type that mainly assesses recognition-based factual knowledge. This narrow scope does not capture higher-order competencies such as open-ended clinical reasoning, care plan development, or ethical decision-making, which are central to nursing practice. Consequently, the ecological validity of our findings is limited: real-world nursing work often requires context-sensitive judgment, nuanced patient communication, and ethical deliberation, all of which extend beyond the ability to select a correct option.

Future studies should therefore expand the scope of evaluation by incorporating open-ended clinical problems and ethical decision-making scenarios. For open-ended items, performance could be assessed in terms of completeness, accuracy, and logical coherence, while ethical decision-making could be evaluated against established professional guidelines and expert consensus. In addition, extending the assessment to other medical specialty examinations (e.g., clinical medicine, pharmacy) would help determine whether model performance varies across domains and provide more targeted insights for medical education and clinical practice.Second, this study compared only two models (DeepSeek-R1 and GPT-4o). The absence of additional baselines such as GPT-3.5, Claude, or Gemini limits the generalizability of our conclusions. Including a wider range of large language models in future research would provide a more comprehensive benchmark and allow cross-family comparisons. Accordingly, our results should be interpreted as a comparative case study between two representative systems rather than an exhaustive evaluation of all available LLMs.Third, the data source of test items also imposes constraints. Since the official national nursing licensure examination items are not publicly released, we could not directly evaluate model performance on the true exam bank. Instead, our study relied on the officially designated practice questions, which are constructed according to the national exam syllabus and serve as the standard preparation resource for candidates. Although these items reflect the content scope and expected competency level of the official examination, they do not undergo the same psychometric validation process (e.g., calibration of item difficulty, discrimination, fairness, exposure control). As a result, the generalizability of our findings to the actual exam should be interpreted with caution. Future work would benefit from collaboration with examination authorities to access psychometrically validated items, thereby providing stronger evidence of model performance under authentic testing conditions.