Background

Carbon monoxide (CO) is a colorless, odorless gas produced by the incomplete combustion of carbon-based fuels. Carbon monoxide poisoning remains the leading cause of accidental poisoning deaths worldwide. In the United States, it is estimated that CO poisoning results in approximately 50,000 to 100,000 emergency room visits and 1,500 to 2,000 deaths each year [1, 2]. Moreover, the annual hospital costs associated with CO poisoning exceed $1 billion, as it causes hypoxic damage to the brain and contributes significantly to high mortality rates [3,4]. In China, particularly in rural regions during the winter season, the predominant method of heating is the combustion of coal, which results in a considerable number of hospitalizations each year due to carbon monoxide poisoning. A notable consequence of CO poisoning is CO encephalopathy, which has been evidenced to impair cognitive function, diminish labor productivity, and impose a considerable economic burden on families and society [5]. Furthermore, individuals diagnosed with carbon monoxide toxic encephalopathy require comprehensive rehabilitation and ongoing care. In conclusion, carbon monoxide (CO) poisoning has the potential to result in long-term disability or even death. Fortunately, the occurrence of carbon monoxide poisoning can be prevented through the implementation of efficacious prevention strategies.

As the Internet has expanded, an increasing number of individuals have sought medical-related scientific information from online sources. The advent of new social media platforms has resulted in the emergence of a significant new avenue for the dissemination of health education information [6]. The advancement of artificial intelligence (AI), particularly in the domain of large language models (LLMs), has exhibited a notable acceleration in recent times. OpenAI’s Chatbot Generative Pre-Trained Transformer (ChatGPT) is a notable example, having received considerable global attention upon its release. ChatGPT functions as an intelligent conversational agent, capable of responding to a multitude of inquiries and generating text output that closely resembles that of a human being. It has been utilized extensively across a multitude of medical domains, encompassing a plethora of applications, including clinical decision support, medical inquiries, and medical documentation. The integration of AI tools, such as ChatGPT, into the healthcare sector has the potential to significantly enhance the capacity of both patients and healthcare professionals to access information and support. Firstly, the dissemination of knowledge online provides individuals with the opportunity to enhance their health literacy by acquiring information about health conditions, preventative measures, and treatment options. Furthermore, the integration of AI into the healthcare sector has the potential to enhance patient engagement and empower individuals to assume a more proactive role in their own care. Moreover, the promotion of health literacy via the Internet can facilitate the dissemination of information, thereby reducing disparities in access, particularly in communities that are underserved. In conclusion, the deployment of tools such as ChatGPT to disseminate health literacy has the potential to enhance public health education efforts and, ultimately, improve the health status and quality of life for a significant proportion of the population.

In recent years, a considerable body of research has been conducted to assess the potential applications of ChatGPT in the medical field. The majority of these studies have yielded positive findings, indicating that ChatGPT has the potential to be a valuable tool in this domain. It has been reported that ChatGPT has demonstrated the ability to successfully complete the United States Medical Licensing Examination (USMLE) [7]. Additionally, prior research has demonstrated that ChatGPT attains a consistency rate of 85.4% and an accuracy rate of 57.33% for endodontic inquiries [8]. Moreover, it is noteworthy that ChatGPT has exhibited remarkable accuracy in addressing queries related to benign prostatic hyperplasia and prostate cancer. Its accuracy rates for these topics reached 90% and 94.2%, respectively [9].

To date, no studies have been conducted to evaluate the performance of ChatGPT in answering questions related to carbon monoxide poisoning. The objective of this study was to assess the accuracy and consistency of responses generated by ChatGPT to questions pertaining to carbon monoxide (CO) poisoning.

Materials and methods

The data for this study were obtained from the "Medical Exam Assistant" (Yi Kao Bang), a widely utilized resource by medical students preparing for the Chinese Medical Licensing Examination and Intermediate Professional Titles. According to statistical data, the Medical Exam Assistant application has been downloaded a total of 3.52 million times since its initial release. The application comprises 44 single-choice questions pertaining to carbon monoxide poisoning. Consequently, a total of 44 questions pertaining to carbon monoxide poisoning were selected. Each item is a single-choice question, with four to five potential responses. The decision to utilize single-choice questions was predicated on a number of considerations, as detailed below. Primarily, the objective was to optimize the scoring methodology and operational procedures. Secondly, in the case of single-choice questions, the correct answer is unique, thereby facilitating a more objective evaluation of the accuracy of the responses generated by ChatGPT.

ChatGPT is a large language model (LLM) powered by artificial intelligence, currently available in two versions: GPT-3.5 and GPT-4. In conducting our analysis, we elected to utilize GPT-3.5 due to its status as a freely available, publicly accessible resource. An observational cohort study was conducted on December 29, 2023. The original Chinese text was entered into ChatGPT ten times for each question. Subsequently, the identical question was translated into English and entered ten times. Following the presentation of the question stem, a prompt indicating the optimal response, namely "Choose the only correct answer based on the stem," will be provided. In consideration of the variability observed in ChatGPT’s responses, the temperature parameter was set to a default value of 0.7. In essence, the temperature parameter controls the randomness of the model’s output, which has a marked effect on the probability distribution used to generate text. Lower temperature settings (e.g., 0.1 to 0.5) result in the generation of responses that are more deterministic and focused, whereas higher settings (e.g., 0.7 to 1.0 and above) lead to greater variability and creativity in the output. The responses from ChatGPT were meticulously documented in an Excel spreadsheet (Microsoft).

Results

The analysis encompassed a total of 44 single-choice questions, which were classified into four primary categories based on their content. The categories included Pathogenesis (n = 10, 23%), Examination and Clinical Manifestation (n = 8, 18%), Diagnosis and Basic Knowledge (n = 10, 23%), and Treatment (n = 16, 36%) (see Fig 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Classification questions.

https://doi.org/10.1371/journal.pone.0311937.g001

ChatGPT was successful in answering all single-choice questions and provided responses that it considered to be the most appropriate. The overall accuracy rates of the Chinese and English queries exhibited minimal discrepancy, with 61% and 57%, respectively. An analysis of the four primary categories revealed that English responses exhibited a higher degree of accuracy in the Examination and Clinical Manifestations category, with a score of 74% compared to 65% for Chinese responses. In contrast, Chinese responses demonstrated superior accuracy in the domain of diagnosis and basic knowledge, with a rate of 77% compared to 68% for English. In the domain of treatment, Chinese responses exhibited a higher degree of accuracy (65%) compared to English responses (49%). However, both exhibited a low level of accuracy in the pathogenesis category (for further details, please refer to Fig 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Accuracy in Chinese versus English.

https://doi.org/10.1371/journal.pone.0311937.g002

In order to evaluate the consistency of ChatGPT’s responses to the same questions, we employed the use of Shannon entropy (Table 1). In the English response, 24 of the 44 questions exhibited zero entropy (i.e., the responses were identical), while 15 demonstrated entropy values greater than 0.5 (i.e., the responses exhibited an unacceptable degree of ambiguity). In the Chinese response, 20 responses exhibited zero entropy, while 15 demonstrated entropy values exceeding 0.5.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Accuracy of answers and Shannon entropy when asking ChatGPT in English and Chinese.

https://doi.org/10.1371/journal.pone.0311937.t001

Discussion

The introduction of ChatGPT has prompted considerable interest globally, with the medical community displaying a notable enthusiasm for the technology [14]. The capacity of this chatbot to furnish expeditious, human-like text responses devoid of advertisements contributes to an enhancement in user engagement. Nevertheless, it is imperative to conduct a comprehensive assessment of ChatGPT’s capabilities before evaluating its potential for clinical applications.

In light of the inherent variability of large language models (LLMs), it was anticipated that discrepancies would be observed in the responses [15]. In the present study, a meticulous and comprehensive analysis was conducted to evaluate the consistency and accuracy of ChatGPT’s responses to clinical questions pertaining to carbon monoxide (CO) poisoning. With respect to consistency, approximately 50% of the items exhibited perfect consistency; however, approximately one-third of the items displayed ambiguous or conflicting responses. In terms of accuracy, the performance of ChatGPT is inadequate. With regard to responses in the Chinese language, the accuracy rate was found to be 61.1%, while in the case of responses in English, the accuracy rate was 57%. The findings indicate that ChatGPT currently lacks the requisite reliability to be regarded as a reliable clinical tool. It is imperative that the accuracy of ChatGPT with respect to medical knowledge be enhanced prior to its deployment in healthcare settings.

The results of the study conducted by Umer and Habib indicate that the acceptable accuracy threshold for medical tools should exceed 90% [16]. Nevertheless, our findings indicate that the accuracy of ChatGPT is approximately 60%, a figure that is consistent with the results of other studies in this field. For example, a study demonstrated that in endodontic inquiries, ChatGPT exhibited an accuracy rate of only 57.33% [16]. Other studies have documented accuracy rates of 79.1% for cirrhosis and 74.0% for hepatocellular carcinoma (HCC), while an accuracy rate of 55.8% has been reported for ophthalmology [14, 17]. Moreover, some studies have defined accuracy in a more comprehensive manner, emphasizing that it is also common for the responses generated by ChatGPT to be partially accurate or to comprise both accurate and incorrect information [18]. The findings suggest that, while ChatGPT displays potential in certain domains, its overall accuracy remains inconsistent and frequently fails to meet the rigorous reliability standards necessary for clinical utilization.

In terms of consistency, approximately one-third of the responses exhibited unacceptable blurring of answers. In a 2023 article entitled "Evaluation of the Accuracy of ChatGPT in Answering Clinical Questions on the Japanese Society of Hypertension Guidelines," the author utilized Shannon entropy as a metric for consistency evaluation. The final results were found to be comparable to those obtained in our study [13]. In the same year, Ana Suárez et al. conducted a study to assess the consistency and accuracy of the ChatGPT in responding to endodontic questions. The results demonstrated that the responses generated by the ChatGPT exhibited a high degree of consistency, with an average score of 85.44% [8]. However, the methodology of this article is disparate from that employed in our own study, therefore direct comparison between the two is not possible. Nevertheless, the application of ChatGPT in the field of healthcare must be conducted in accordance with the highest standards. Accordingly, the methodology for consistency assessment should also be multi-pronged. The evaluation metrics employed in this study, including entropy, can be utilized as benchmarks for the advancement of future AI models. The application of these metrics will assist developers in comprehending the diversity and consistency of model outputs, thereby facilitating the development of more efficient algorithms that diminish uncertainty and enhance the reliability of AI systems.

The user-friendly interface and rapid response times of ChatGPT render it more accessible to the general public in today’s information-driven society. However, a significant concern is the phenomenon of "hallucination," whereby the AI generates erroneous or deceptive information [18, 19]. To mitigate this risk, the research team implemented a straightforward strategy, providing a clear prompt that specified the desired outcome. "Please select the only correct option." The objective was to obtain a singular and precise response, thereby reducing the probability of erroneous answers. Furthermore, the high repetition of each question serves to increase the sample size of the article, which is conducive to both repeatability and accuracy assessment.

It appears that there is still a considerable distance to traverse before ChatGPT can be fully integrated into clinical practice. In the field of healthcare, the accuracy of chatbots is of paramount importance, as even minor errors have the potential to result in significant and irreversible consequences for patients and their families. A further crucial issue is that of reproducibility. The responses generated by ChatGPT are subject to variation depending on the phrasing of the question and may also differ over time. It is thus imperative to take into account the intrinsic constraints and potential hazards associated with the deployment of AI in healthcare. It is vital to pursue ongoing research, evaluation, and refinement to establish a robust foundation for the deployment of artificial intelligence (AI) in medical contexts [20, 21]. To guarantee the safety and reliability of AI applications, it is imperative to exercise close supervision, establish fundamental guidelines regulating their utilization, and implement continuous monitoring of AI outputs. This is of paramount importance for the maintenance of the high standards of clinical application.

Conclusion

The results of our investigation indicate that the accuracy and consistency of ChatGPT’s responses to queries pertaining to carbon monoxide poisoning are suboptimal. This underscores the necessity for meticulous and rigorous assessment when seeking medical counsel from ChatGPT. In conclusion, this study establishes the foundation for the broader implementation of AI technologies in healthcare and delineates the prospective trajectory of technological advancement.

Supporting information