2.1. Current state of text classification research

In text classification and intent recognition research, multimodal fusion, few-shot learning, and implicit intent understanding have become key focuses. For multimodal intent recognition, Li et al. [3] proposed a text-guided cross-modal attention mechanism to enhance non-textual modality features. Dong et al. [4] introduced a reinforcement learning-based multi-task framework to generate modality-relevant explanations and mitigate modality bias. Xia et al. [5] designed a method combining video feature enhancement with multimodal feature collaboration, achieving significant improvements on multimodal datasets. For context-aware intent recognition, Zhang et al. [6] combined machine reading comprehension with memory networks, utilizing self-attention and co-attention mechanisms for multi-turn dialogue intent recognition. Pan et al. [7] proposed a hybrid multi-intent recognition method for air-ground communication texts, modeling different text types separately. For small-sample and cross-lingual scenarios, Cao et al. [8] transformed cross-lingual intent recognition into masked language modeling tasks via prompt learning, significantly improving accuracy on low-resource languages. Zhang et al. [9] employed pre-trained language models and contrastive learning for gap-filling data augmentation, effectively mitigating data scarcity. For implicit intent recognition, Liu et al. [10] constructed the Chinese Implicit Intent Dataset (CIID) and pioneered prompt learning to uncover users’ true intentions. Despite progress across scenarios, existing methods still suffer from declining performance as label counts increase, hindering further development of multi-label classification techniques. Inspired by GCNet’s global context modeling in image recognition, this paper introduces a global attention mechanism to text classification tasks. By embedding global context attention modules before and after pooling operations, the model’s ability to capture long-range dependencies and global semantics is enhanced, significantly improving classification performance in multi-label text classification.

2.2. Current state of named entity recognition research

In current named entity recognition research, multiple studies have combined pre-trained language models with sequence annotation structures to enhance performance, addressing the specificity and complexity of domain-specific texts. For instance, in the field of traditional Chinese medicine, Yang et al.[11] proposed the BERT-BiLSTM-MHA-FUSION-CRF model, which enhances semantic representations through multi-head attention and multi-level fusion mechanisms, achieving significant accuracy improvements across multiple datasets. Zhu et al. [12] proposed ERPG, a prompt-guided generative approach for complex entities, effectively reducing invalid outputs and improving recognition quality. For medical texts, Song et al. [13] introduced semantic knowledge augmentation and global pointer optimization mechanisms, significantly improving nested entity recognition. Wang et al. [14] employed MCBERT combined with CNN-BiLSTM-CRF for dynamic semantic extraction, achieving an F1 score of 82.8% in electronic medical record recognition tasks. Li et al. [15] proposed a model integrating BERT with BiLSTM-CRF, incorporating part-of-speech and chunk analysis features, achieving an average F1 score of 89.45% across multiple biomedical datasets. Su et al. [16] introduced Global Pointer (GP), a novel fragment-based framework. Leveraging relative position information and a multiplicative attention mechanism, GP enhances fine-grained semantic capture and mitigates label imbalance through head-tail identification modules and a novel classification loss function. It achieves outstanding performance across multiple benchmark datasets while reducing parameters. He [17] and Qi [18] respectively enhanced local feature capture for Chinese medical and social media texts by introducing IDCNN and boundary expansion modules. Overall, current NER research is increasingly characterized by the deep integration of pre-trained models, multi-scale feature interaction, domain adaptation, and enhanced low-resource learning capabilities. Related efforts in other domains also reflect this trend toward data-efficient large-model learning under limited-data conditions. For example, Meng et al. proposed PowerMistral, a pre-trained large language model-based framework for few-shot wind power forecasting, further highlighting the broader potential of large models in data-scarce scenarios [19]. However, existing NER methods still rely heavily on large volumes of high-quality annotated data, and their performance is strongly affected by both data scale and annotation quality, while the annotation process itself remains costly and sensitive to noise. To address these limitations, this paper establishes a large-model-based medical entity recognition workflow by standardizing entity type definitions, designing precise instruction constraints, and aligning recognized entities with knowledge graph nodes. This workflow reduces dependence on manually annotated data and provides a flexible alternative for medical entity extraction in low-resource settings.

2.3. Research on intelligent question-answering mechanisms

In the research of intelligent question-answering mechanisms, the integrated application of knowledge graphs and large language models has become a key technological pathway for advancing intelligent services in specialized domains. Existing research indicates that related achievements have extensively permeated multiple vertical domains. In non-medical professional fields, system designs demonstrate deep adaptability to specific scenarios: Zhou et al. [20] combined large language models with facial recognition technology to construct a secure Q&A system capable of user identity perception, achieving intelligent access control interaction through local knowledge base retrieval and permission control; Li et al. [21] developed a specialized Q&A system for overseas COVID-19 case management using TF-IDF and Bi-LSTM+CRF models, enabling precise retrieval and visualization of pandemic-related knowledge; Li et al. [22] innovatively employed RAG frameworks and AI agent technology to address intangible cultural heritage preservation needs, establishing professional game analysis and Q&A mechanisms for data-scarce domains like Tibetan chess; Liu et al. [23] focused on geological exploration, constructing a mineral knowledge graph containing 22,000 entities based on the BERT model to achieve high-precision responses to natural language mineral queries. In the medical field, related research is particularly active, with Q&A system development emphasizing professionalism, safety, and personalized service capabilities. Zhou et al. [24] developed a medical knowledge graph Q&A system covering 14,000 diseases by combining multi-similarity matching algorithms with a Naive Bayes classifier; Guan et al. [25] proposed a joint mechanism for intent recognition and entity linking, significantly enhancing Q&A robustness under non-standard medical terminology; Qin et al. [26] integrated Neo4j knowledge graphs with large language models, employing LoRA fine-tuning to provide personalized Q&A services for diabetic patients; Sukhwal et al. [27] designed a long-form disease Q&A system leveraging LLM-KG synergy, effectively balancing medical factual accuracy with generative fluency; Hu et al. [28] employed a retrieval-based approach integrating Seq2Seq and TF-IDF models to construct a medical Q&A system; Wang et al. [29] proposed the knowledge-enhanced KEMedGPT model, focusing on medical consultation and pre-consultation scenarios; Shi et al. [30] utilized BERT and Naive Bayes methods to construct a medical knowledge graph and automated Q&A window. Although existing research has made significant progress in the accuracy and domain adaptability of professional Q&A systems, their underlying architectures still predominantly rely on graph databases like Neo4j as the core knowledge storage. However, Neo4j struggles to directly support image data storage and visualization integration, hindering the unified presentation and interaction of multimodal knowledge within Q&A interfaces. This limitation constrains the system’s ability to comprehensively express complex medical knowledge. To address this bottleneck, this paper constructs a local multimodal resource repository containing 15,183 medical images. A backend API service layer is built using the Flask framework to enable efficient interaction with Neo4j. Structured graph data is provided to the frontend via RESTful interfaces, and knowledge visualization is seamlessly integrated with Q&A functionality through ECharts. This approach provides a practical solution for integrating graph knowledge, associated images, and interactive display within the QA system.

3.1. Intent recognition based on a dual-layer attention mechanism

To accurately understand user query intent, this paper designs a dual-layer attention text classification model. By incorporating multi-scale convolutions and a global context attention mechanism(GCNet), the model effectively enhances classification performance in multi-label scenarios. The overall operational flow of the model is illustrated in Fig 2: Given an input text sequence ={text}, where L denotes the sequence length, the sequence is first segmented into tokens and converted into a numerical tensor. Subsequently, each token is mapped to a context-aware vector E through the BERT pre-training model:

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Dual-layer attention text classification model architecture.

https://doi.org/10.1371/journal.pone.0353112.g002

(1)

Here, B = 100 represents the batch size, and D = 768 denotes the dimension of the BERT hidden layer. Subsequently, the channel dimension is added, yielding the feature map .

Subsequently, multi-scale convolutional feature extraction is performed on the feature maps by applying parallel convolutional operations using convolutional kernels of various sizes to the input features. For each feature map processed with a kernel size :

(2)

Among these, ReLU is a nonlinear activation function that sets negative values to zero while preserving positive values, used to filter out irrelevant features and retain relevant ones. denotes the weight matrix of the convolutional kernel, * represents the convolution operation, is the bias vector of the convolutional kernel, N is the number of convolutional kernels, and H is the sequence length minus the kernel size, i.e., .

Subsequently, the three feature maps were processed separately in the GCNet (Global Context Attention Mechanism Layer) designed in this paper. This layer mainly consists of three steps: spatial attention pooling, channel attention fusion, and feature fusion, as shown on the right of Fig 2:

For each feature map , the attention weight matrix is first calculated through spatial attention mechanism:

(3)

Among these, is a function that converts any numerical vector into a probability distribution, where is a 1 × 1 convolution kernel. Then, the weighted context vector c is computed:

(4)

Among these,: denotes the fixed matrix’s corresponding positional dimension. Next, channel attention fusion is performed on c, which comprises two parallel branches: the channel multiplication branch:

(5)

Channel Addition Branch:

(6)

In the above branch computations, denotes the sigmoid activation function, while , , , represent the feature matrices for each stage. The matrix is obtained by fusing the original features with the attention weights:

(7)

Here, and ⊕ denote element-wise multiplication and addition, respectively. The matrix is obtained by performing global max pooling on the features at each scale:

(8)

Concatenate the pooling results from all scales into the matrix P:

(9)

The concatenated features are reshaped into a 4-channel tensor . This tensor passes through the next layer’s global context attention mechanism, calculated using Equations (3)–(7), ultimately yielding . The processed features undergo classification via a fully connected layer:

(10)

Here, n denotes the number of text categories, y represents the final probability distribution matrix across the n text categories, while and denote the feature matrix and bias of the fully connected layer.

3.2. Large language model-based named entity recognition

To accurately extract medical entities from user queries while reducing reliance on labeled data, this study employs large language models (LLMs) such as DeepSeek and Claude for named entity recognition (NER) through prompt-based extraction. This approach achieves efficient model transfer without additional training by constraining model outputs via carefully designed prompts. The workflow proceeds as follows: First, access the selected LLM service via the appropriate API. Next, define the entity types and their semantic scope based on the specific requirements of the medical question-answering task. For example, set the entity type set format as: entity_types = [‘disease’, # Disease: heart disease, cold, etc.]. This definition must align with existing entity types in the underlying medical knowledge graph to ensure effective mapping between identified entities and graph content. Next, provide explicit instruction constraints to the LLM, such as requiring: “1. Do not omit entities that appear repeatedly in the text; 2. Maintain the integrity of professional terminology.” Finally, input the text to be identified {text} and the specified output format into the model. Through these instructional controls and format constraints, relevant entity information can be extracted from user questions in a structured manner, while reducing dependence on large-scale annotated data.

Algorithm: Large Language Model Named Entity Recognition

Input: User text {text}

Output: Predicted entities in list format

1: Initialize environment and API configuration

2: Define entity type set entity_types

3: Construct the prompt variable prompt using the user input {text}

4: while retry count < 3 do:

5:   Call the selected LLM API to obtain a response

6:   Parse response text to extract entities

7:   if success then break

8: Format output as nested list: [

   [‘entity_type’, ‘entity_text’],

   [‘entity_type’, ‘entity_text’],

   ...]

3.3. Question-answering mechanism based on multimodal medical knowledge graphs

To apply the aforementioned intent recognition and entity recognition modules to practical scenarios, this section constructs a comprehensive medical intelligent question-answering mechanism. Its overall architecture, as shown in Fig 3, primarily comprises three major components: multimodal knowledge base construction, backend service logic, and frontend visual interaction.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Architecture of a question-answering mechanism for multimodal medical knowledge graphs.

https://doi.org/10.1371/journal.pone.0353112.g003

This paper constructs a multimodal knowledge base based on Neo4j, whose structure is depicted in the data storage layer at the bottom of Fig 3. This knowledge base contains entities labeled with terms such as “Disease,” along with relationships labeled as “DISEASE_ACOMPANY” (complications), while also integrating a local resource repository containing 15,183 medical images. By establishing a local HTTP server, the system achieves effective mapping between image storage paths and knowledge graph nodes, enabling multimodal information association and access.

The backend service is built using the Flask framework and provides external services via RESTful APIs. As shown in the middle section of Fig 3, its core workflow proceeds as follows. First, upon receiving a user query, the system invokes the intent recognition model described in Section 3.1 to parse the query intent. Subsequently, the entity recognition module described in Section 3.2 is used to extract key medical entities. The recognized intent and entities are then combined to generate Cypher query statements for retrieving relevant information from the Neo4j knowledge graph and local image repository. Finally, the system populates the retrieval results into a predefined response template and invokes a large language model (LLM) to summarize and refine the content, generating a final answer with a clear structure and fluent language.

The front-end visualization layer employs a combination of Layuimini and ECharts technologies. As shown in the upper part of Fig 3, the Layuimini framework handles overall layout and interactive interface construction, while ECharts specializes in rendering complex relational structures like knowledge graphs. Together, they integrate text, image, and graph data returned from the backend into a unified view, ultimately providing users with an intuitive and fluid interactive experience maintained via UUIDs.

3.4. Ethics statement

The end-to-end user evaluation in Experiment 3 involved five domain experts and twelve general participants recruited from 13 March 2026 to 17 March 2026. Ethical review exemption for this study was granted by the Medical Ethics Committee of the First People’s Hospital of Guiyang. Written informed consent was obtained from all participants prior to the evaluation. No minors were involved in the study. The general participants were anonymous, and no personally identifiable information was collected or retained.

4.1. Experimental preparation

4.1.1. Dataset.

The experiment selected four datasets. In Experiment 1: Text Classification Model Ablation Study, this paper utilized a self-built medical text classification dataset MTCD (Medical Text Classification Dataset) alongside two public datasets: Weibo and Toutiao (Weibo: https://github.com/SophonPlus/ChineseNlpCorpus. Toutiao: https://github.com/aceimnorstuvwxz/toutiao-multilevel-text-classfication-dataset/tree/master). In Experiment 2: Large Model Entity Recognition Evaluation, this study employed a public medical entity recognition dataset CMEDQANER (CMEDQANER: https://github.com/alibaba-research/ChineseBLUE) containing 11 entity labels including diseases, symptoms, and medications. When constructing the MTCD dataset, we first identified 16 distinct question-answering types based on features extracted from structured data crawled from the Xunyiwenyao website. Subsequently, we collected user-submitted question texts from open Chinese medical information websites and public medical Q&A datasets, then filtered and cleaned the data. Using the extracted textual features, we provided corresponding question-answering templates to the large language model for data augmentation, ultimately generating the MTCD dataset. Specific statistical information for the MTCD dataset is shown in Table 1, where symbols like [Disease] denote entity slots. The entire annotated dataset was divided into a test set and a training set in a 2:8 ratio, comprising a total of 27,582 samples. The Weibo dataset consists of sentiment-labeled sentences collected from Sina Weibo, with approximately 50,000 positive and negative comments each. The Toutiao dataset comprises 382,688 sentences collected from the Toutiao client, categorized under 15 news tags including automotive, agriculture, and sports. The self-constructed dataset used in this study was compiled from publicly accessible online sources, including structured disease pages from the Xunyiwenyao website, open Chinese medical information websites, and public medical question-answering datasets. Only content that was publicly accessible at the time of collection was used, and no login-protected, paywalled, or access-restricted material was collected. Data collection and subsequent analysis were conducted in accordance with the applicable terms and conditions of the original data sources. Because some raw webpage content and associated medical images may remain subject to copyright or website-use restrictions, these original materials are not redistributed in this study. Instead, we provide the processed knowledge graph data, implementation code, and related derived resources, and refer readers to the original public sources for access to the underlying raw materials.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. MTCD text classification dataset statistics.

https://doi.org/10.1371/journal.pone.0353112.t001

4.1.2. Training environment.

The model implementation was programmed in Python within a CUDA environment. We trained the model by running the program on a single NVIDIA 4090 24GB GPU. The system environment was Windows 11, utilizing the PyTorch framework version torch1.11.0 + cu113. Tables 2 and 3 list the experimental hardware and software configurations, respectively.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Experimental hardware configuration.

https://doi.org/10.1371/journal.pone.0353112.t002

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Experimental software environment.

https://doi.org/10.1371/journal.pone.0353112.t003

4.1.3. Parameter settings and evaluation metrics.

In Experiment 1: Text Classification Model Ablation Study, the pre-trained BERT-wwm model was employed to encode text data during the experimental phase. Subsequently, features were extracted from the encoded text using a convolutional layer and a dual-layer attention layer, followed by text classification. The model’s key parameters were configured as specified in Table 4, with their respective values explicitly defined.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Model parameter setting.

https://doi.org/10.1371/journal.pone.0353112.t004

In the experimental setup, each training batch comprised 100 samples, and the learning rate was set to 5e-6. For the BERT-wwm encoder, the maximum input sequence length was set to 128, while the hidden size was 768. In the TextCNN module, the number of convolutional kernels was 256. The model was trained for 100 epochs.

This study employs precision (P), recall (R), and F1-score (F1) as the basic evaluation metrics for model performance [31–33], as shown in the following equation:

(11)(12)(13)

Among these, TP (True Positive) refers to the number of samples correctly predicted as positive by the model, FP (False Positive) refers to the number of samples incorrectly predicted as positive by the model, and FN (False Negative) refers to the number of samples incorrectly predicted as negative by the model.

In Experiment 1, since the task is a multi-class text classification problem, accuracy (A) was further introduced as a supplementary evaluation metric [34]:

(14)

Where TN (True Negative) denotes the number of samples correctly predicted as negative. In addition, , Macro-F1, and Weighted-F1 were adopted to provide a more comprehensive evaluation of multi-class classification performance. Let denote the F1-score of the i-th class, C the total number of classes, the number of samples in the i-th class, and the total number of samples. Then:

(15)(16)(17)(18)

Among these metrics, reflects the overall classification performance over all samples, emphasizes balanced performance across classes, and takes class distribution into account, making it more suitable for imbalanced multi-class datasets.

In Experiment 2, which focuses on medical named entity recognition, model performance was mainly evaluated using precision, recall, and F1-score, since these metrics are the most commonly used and sufficient indicators for sequence labeling tasks.

4.2. Experiment 1: Text classification model ablation study

To evaluate the classification performance of the proposed dual-layer attention text classification model, we first replaced the TextCNN [35] layer in the baseline model with several alternative text encoding structures, including TextRNN, DPCNN [36], DPRNN [37], and FastText [38], and conducted comparative experiments on the MTCD dataset. The results are shown in Table 5. Overall, among the compared backbone structures, the baseline model using TextCNN achieved relatively strong and stable performance, supporting its use as the baseline encoder in this study. On this basis, after introducing the proposed dual-layer attention mechanism, the model achieved the best performance among the compared methods in terms of Accuracy, Recall, Micro-F1, Macro-F1, and Weighted-F1. These results indicate that the proposed attention structure can effectively improve the overall classification performance on the self-built medical text classification dataset.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Comparison of classification performance on the MTCD dataset.

https://doi.org/10.1371/journal.pone.0353112.t005

To further analyze the performance of the proposed model across different categories, Table 6 presents the fine-grained classification results for 16 labels, including the support for each category, as well as the numbers of correctly and incorrectly classified samples. For simplicity, the label categories are represented by Label ID throughout the manuscript, with the corresponding label names provided in Table 1; the same notation is also adopted in Tables 15–17. The results indicate that the proposed model achieves high classification performance in most categories, but it still encounters difficulties in distinguishing between a few semantically similar labels, such as Disease Definition, Disease Causes, and Disease Symptoms, which is reflected in relatively low Precision, Recall, and F1 values.

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. Class-wise performance of the proposed model on the MTCD dataset.

https://doi.org/10.1371/journal.pone.0353112.t006

The main confusion scenarios are summarized in Table 7. It can be observed that misclassifications primarily occur between semantically adjacent categories, such as disease symptoms, disease causes, treatment methods, and complications. Taking symptom-related questions as an example, their expressions often contain similar lexical patterns to those of cause-related or treatment-related questions, making them more prone to confusion. This indicates that the remaining classification errors primarily stem from semantic overlaps between similar medical question-answering types, rather than a lack of overall classification ability in the model.

Download:

• PNG
  larger image
• TIFF
  original image

Table 7. Major confusion cases of the proposed model on the MTCD dataset.

https://doi.org/10.1371/journal.pone.0353112.t007

To further verify the robustness of the dataset construction strategy and the generalization ability of the model, this paper also conducted supplementary experiments on MTCD-n (i.e., the self-built dataset after removing all LLM-based template augmentation) as well as the public datasets Weibo and Toutiao. The results are presented in Table 8. On MTCD-n, our model still outperforms the baseline model across all evaluation metrics, indicating that the observed performance gains do not solely rely on template-based data augmentation. This finding suggests that the model is able to capture semantically meaningful intent-related features from the original data distribution, rather than merely fitting to artificially constructed template patterns. Furthermore, on the two public datasets Weibo and Toutiao, which contain naturally occurring open-domain texts with different linguistic styles and label distributions, our model also consistently outperforms the baseline methods. These results demonstrate that the proposed method is not over-aligned with the predefined QA-like structures introduced during augmentation, but instead maintains strong transferability and robustness across both medical and general-purpose text classification tasks.

Download:

• PNG
  larger image
• TIFF
  original image

Table 8. Comparison of classification performance on MTCD-n and public datasets.

https://doi.org/10.1371/journal.pone.0353112.t008

4.3. Experiment 2: LLM-based entity recognition evaluation

To evaluate the entity recognition performance of large language models, the specific experimental methodology is as follows: by applying character-length filtering and random sampling to the original CMEDQA dataset, 300 samples ranging from 15 to 50 characters in length were obtained. This process excluded excessively short or long sentences while ensuring sufficient randomness. For the large language model-based methods, each experiment was repeated three times under the same setting, and the mean values together with standard deviations were reported to improve the reliability of the evaluation. The entity recognition task was then performed using the large language models via the method described in Section 3.2.

As shown in Table 9, the first five cases represent the CMedQANER test results of traditional deep learning models from [15] and [16], while the last four cases show the results of different large language models. Traditional deep learning models still outperform large language models in precision and overall F1. However, some large language models achieve competitive recall, and claude-sonnet-4–6 attains the highest recall (81.49% ± 0.73%). In terms of processing time, deepseek-v3.2 is the fastest among the tested large language models (243.12 ± 2.67 s). Overall, some large language models achieve relatively high recall in this task, but their substantially lower precision and F1 indicate that they are not yet competitive with the strongest traditional baselines in overall NER performance. The relatively small standard deviations also indicate that the repeated runs are reasonably stable.

Download:

• PNG
  larger image
• TIFF
  original image

Table 9. Performance comparison of different NER methods on the CMEDQA dataset.

https://doi.org/10.1371/journal.pone.0353112.t009

As shown in Table 10, a temperature sensitivity analysis was further conducted on claude-sonnet-4–6, which showed relatively strong performance in the NER task. The results indicate that the model performance remains relatively stable under different temperature settings, with only slight variations in F1 score. Among these settings, temperature = 0.8 achieves the best overall performance, with the highest F1 (55.22% ± 0.15%) and recall (83.06% ± 0.20%).

Download:

• PNG
  larger image
• TIFF
  original image

Table 10. Temperature sensitivity analysis of Claude-Sonnet-4-6 for NER.

https://doi.org/10.1371/journal.pone.0353112.t010

As shown in Table 11, we further report the entity-type-wise performance of claude-sonnet-4–6 under its best-performing setting, namely temperature = 0.8. The results show clear differences across entity types. The model performs relatively well on crowd, disease, body, and drug, while weaker results are observed for time, feature, test, and physiology. In addition, the comparison between the real and predicted counts suggests that the model tends to over-identify some entity types, especially time, feature, physiology, and symptom. These results indicate that the method is more effective for entity categories with clearer semantic boundaries, while fine-grained or ambiguous categories remain more challenging.

Download:

• PNG
  larger image
• TIFF
  original image

Table 11. Entity-type-wise NER performance of Claude-Sonnet-4-6 at temperature 0.8.

https://doi.org/10.1371/journal.pone.0353112.t011

4.4. Experiment 3: System-level validation with expert and user evaluation

The number of medical nodes and relationships crawled from the disease encyclopedia section of Xunyiwenyao.com (Seek Medical Advice Network: https://www.xywy.com/) is shown in Tables 12 and 13, comprising 9 node types and 10 relationship types. Partial knowledge graphs constructed in the Neo4j database based on these data are illustrated in Fig 4. Selected multimodal knowledge graphs visualized in the ECharts frontend interface are shown in Fig 5. Fig 6 presents example Q&A interactions using cold definition and prevention as representative cases.

Download:

• PNG
  larger image
• TIFF
  original image

Table 12. Medical entity statistics.

https://doi.org/10.1371/journal.pone.0353112.t012

Download:

• PNG
  larger image
• TIFF
  original image

Table 13. Statistical information on inter-healthcare entity connections.

https://doi.org/10.1371/journal.pone.0353112.t013

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Medical knowledge graph stored on Neo4j (Partial).

https://doi.org/10.1371/journal.pone.0353112.g004

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Multimodal medical knowledge graph based on ECharts (Partial).

https://doi.org/10.1371/journal.pone.0353112.g005

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Medical knowledge graph Q&A system Q&A Example.

https://doi.org/10.1371/journal.pone.0353112.g006

The medical text and image resources used for the multimodal knowledge graph were derived from publicly accessible third-party medical websites; details of data-source compliance and redistribution restrictions are described in the Dataset section.

Since the construction of the knowledge graph in this paper primarily relies on structured and semi-structured web page fields, rather than open semantic extraction based on free text, data noise control mainly focuses on the structured processing flow, including web page parsing result verification, null and outlier filtering, duplicate record removal, field mapping consistency check, entity naming normalization, and schema consistency check.

For image node assignment, this study achieves automatic alignment between image resources and knowledge graph nodes based on the original correspondence between entity pages and display images in the source webpages, combined with entity names and image file naming conventions. During the construction process, we eliminated samples with inconsistent names, abnormal naming, missing information, or entries that could not be uniquely matched, thereby improving the reliability of multimodal node mapping.

To ensure the usability of the knowledge graph, this paper further conducted three aspects of quality inspection after the graph construction was completed. Firstly, normalized processing and similar node screening were performed on entity names of the same type to reduce potential duplications caused by differences in naming granularity, alias expressions, or field redundancy. The inspection results indicated that most of the high-similarity nodes identified belonged to differences in concept hierarchy, subdivision of departments, or different expression granularity, rather than strictly defined duplicate entities. Secondly, focusing on the design of the graph structure, a pattern consistency check was conducted on the triples, with a focus on verifying whether the head entity type, relationship type, and tail entity type conform to predefined schema constraints. Finally, 100 triples were extracted using uniform sampling based on relationship labels, and experts in the medical field were invited to verify them from the perspectives of pattern consistency and content rationality. The results showed that most of the sampled triples were basically reasonable in terms of pattern and content, with a small number of samples (19 in total) having doubts that required further verification. These doubts mainly concentrated on diet/dish-related relationships, as well as a small number of drug and complication-related relationship types. The issues were more reflected in the instability of the semantic boundaries of relationships or insufficient medical specificity, rather than a loss of control over the overall entity type system of the graph. Overall, the constructed graph can better support the knowledge retrieval and answer generation requirements of this paper’s question-answering system, but there is still room for further optimization in some weakly structured relationships.

To more comprehensively evaluate the overall performance of the medical question-answering system in real question-answering scenarios, this study designed an end-to-end user evaluation involving both domain experts and general participants. For each intent category, three representative questions were selected for assessment, and the mean scores together with standard deviations were calculated based on the participants’ ratings. Table 14 presents the scoring criteria used in this evaluation. The criteria were developed through discussions with domain experts. System responses were assessed from four dimensions, namely Correctness, Completeness, Safety, and Match, with each dimension scored on a 2-point scale. This design provides a more intuitive reflection of answer quality in realistic QA scenarios and supports a more structured end-to-end evaluation of system performance [39].

Download:

• PNG
  larger image
• TIFF
  original image

Table 14. Scoring criteria for the end-to-end user evaluation.

https://doi.org/10.1371/journal.pone.0353112.t014

Table 15 presents the end-to-end evaluation results scored by five domain experts. Overall, the expert ratings were relatively higher in Safety and Match, whereas lower scores were mainly observed for several labels in Correctness and Completeness. In particular, Complications, Drug Applicability, Treatment Cost, and Incidence Rate showed relatively weaker performance, indicating that the system still has limitations in handling categories involving greater medical complexity, uncertainty, or individual variation. In contrast, Disease Definition, Disease Causes, Department, and Contraindications achieved relatively better results, suggesting that the system performs more reliably on well-defined and knowledge-based medical questions.

Download:

• PNG
  larger image
• TIFF
  original image

Table 15. Expert evaluation results for each label in the end-to-end user evaluation.

https://doi.org/10.1371/journal.pone.0353112.t015

Table 16 summarizes the end-to-end evaluation results scored by 12 general participants. Compared with the expert ratings, the user scores showed a similar overall trend, with relatively lower results still concentrated in several more difficult labels. Specifically, Symptoms Check, Treatment Methods, Prevention, Drug Applicability, and Treatment Cost received relatively lower scores. In contrast, Disease Definition, Disease Causes, Disease Symptoms, Contraindications, and Inspection Plan obtained relatively higher scores.

Download:

• PNG
  larger image
• TIFF
  original image

Table 16. General participant evaluation results for each label in the end-to-end user evaluation.

https://doi.org/10.1371/journal.pone.0353112.t016

Table 17 presents the combined end-to-end evaluation results by integrating the scores from domain experts and general participants, with the total scores further converted into normalized percentages for more intuitive comparison. In Table 17, the values outside parentheses represent the scores from domain experts, whereas the values inside parentheses represent the scores from general participants. The table also reports the 95% confidence interval results of the combined scores, which remain within a relatively reasonable range overall. The combined results show that the overall pattern was broadly consistent between expert and public evaluations: labels such as Contraindications, Disease Definition, Disease Causes, and Department remained among the best-performing categories, whereas Treatment Cost, Complications, Drug Applicability, and Incidence Rate consistently ranked lower. At the same time, general participants tended to assign relatively more favorable scores to common and easy-to-understand categories, while expert ratings were more sensitive to deficiencies in professional accuracy and completeness. Overall, the comparison between the two groups suggests that the proposed system is generally usable. However, further efforts are still needed to improve its performance on labels involving greater uncertainty, stronger personalization, and higher medical complexity.

Download:

• PNG
  larger image
• TIFF
  original image

Table 17. Combined end-to-end evaluation results by label.

https://doi.org/10.1371/journal.pone.0353112.t017