1 Introduction

The rapid evolution of technology continues to transform healthcare delivery, as evidenced by advancements in engineered nanomaterials [1] and point-of-care diagnostic technologies [2]. Generative Language Models (GLMs) demonstrate advanced natural language processing capabilities, with emerging applications across medical domains including medical licensing examinations [3], patient education [4], and medical diagnosis [5], potentially supplementing traditional medical information sources. The potential shift toward AI-generated medical advice warrants careful evaluation against established clinical knowledge bases. The reliability of such information requires rigorous assessment. In healthcare contexts where erroneous information may have serious consequences [6], GLMs demand substantially greater scrutiny than applications in non-clinical domains. This is particularly relevant to specialized fields like ophthalmology, where comprehensive evaluation is needed to establish GLMs’ capacity for clinically consequential applications.

Despite being the most recent version of GLMs, ChatGPT’s capability to produce medical data for severe ophthalmic conditions has not been comprehensively assessed. This research sought to assess the accuracy of data produced by ChatGPT regarding ophthalmic disorders and examine how well its responses align with established ophthalmology standards guidelines. The results will offer valuable perspectives on the possible clinical uses of the model in ophthalmic care, while also pinpointing aspects that need enhancement. This will aid in the development of more accurate medical AI technologies.

2 Methods

3 Results

3.1 Content, publication or revision date, and structural data of EQIP instrument

The modified EQIP instrument now caps at a total of 36 points, segmented into three parts: content, publication or revision date, and structural data. The content segment can earn up to 18 points, focusing on the thoroughness and precision of the data provided. The section for the publication or revision date can earn up to 6 points, highlighting the importance of clear and identifiable information sources. The structural data segment can earn up to 12 points, evaluating the level of information organization and its practical utility. Across the five conditions examined, the modified EQIP’s median overall score was 18 (IQR 18-19), and the score distributions were uniform for all conditions. The specifics can be found in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Summarized findings evaluated with the adapted EQIP instrument.

https://doi.org/10.1371/journal.pone.0334250.t001

Regarding the content, ChatGPT showed impressive capabilities in explaining the medical facets of diseases, treatment objectives, effects on quality of life, warning indicators, and highlighting crucial points (Items 3, 4, 11, 14, and 18). Concerning the date of publication or revision, ChatGPT offered specific dates for updates or releases, along with citations from relevant, evidence-based data sources. In terms of structure, ChatGPT’s replies were succinct and straightforward, with an average sentence length of around 15 words. The conciseness was greatly valued by professionals as it improved both understanding and the effectiveness of communication. Consequently, Item 27 demonstrated its efficient use of succinct language. In terms of design logic, ChatGPT employed a "broad-narrow-broad" format, beginning with a concise overview, moving into an in-depth exploration, and wrapping up with a recap of the essential elements. The expert evaluators deemed this method appropriate, as it notably enhanced the clarity and understanding of the content. As a result, Item 33 was given a score of 1, reflecting the uniform positive feedback from the reviewers.

In the content section, Item 8 received no score due to the discrepancy between the quantitative data supplied by ChatGPT and the cited literature. In the publication or revision date section, Items 20, 21 and 22 were not assigned any points due to the lack of pertinent details from ChatGPT. In the structural data segment, Items 28 was also left unscored for the same rationale. Furthermore, Item 30 did not receive a score because of incorrect details about surgical procedures, expenses, and quantitative benefits.

Of note, AI hallucinations occurred in all responses to Item 15 (regarding costs in five conditions), representing an occurrence rate of 2.8% (5/180).

Overall, ChatGPT demonstrated the highest performance in providing medical information related to DR, with an EQIP total score of 20, ranking first among the five diseases. This was followed by glaucoma, which received a score of 19. In comparison, the model performed equally well for cataracts, refractive error, and AMD, each achieving a total score of 18. In the subdomain evaluation, in terms of content, both cataracts and DR received the highest score of 10. In the publication/revision date dimension, all conditions received identical scores of 3. Regarding structure, glaucoma achieved the highest score of 8. Detailed scoring results are presented in Fig 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Comparison of scores across different diseases.

https://doi.org/10.1371/journal.pone.0334250.g001

The agreement between the two assessors, when evaluating ChatGPT’s responses with the adapted EQIP instrument, was quantified by a Cohen’s kappa value of 0.926 (95% CI 0.875-0.977, P < 0.001).

4 Discussion

This research thoroughly evaluated the accuracy of medical data produced by ChatGPT for five prevalent ophthalmic conditions (cataracts, refractive errors, glaucoma, AMD, and DR) by employing an advanced EQIP instrument and aligning with expert guidelines. The findings showed that ChatGPT generally excelled in content quality, especially regarding the precision and comprehensiveness of details about disease explanations, treatment objectives, and the effects on quality of life. However, major flaws were found in the organized display and clarity of information origins. For instance, it did not adequately specify the originating bodies, the identities of the participants, and the financial backers [17]. Moreover, ChatGPT experienced significant hurdles when tackling intricate problems like measuring quantifying benefits and risks or the expenses associated with medical treatments [18]. In the alignment assessment, ChatGPT’s replies aligned with 84% of the professional standards’ suggestions, demonstrating a high degree of accuracy consistent with specific clinical metrics [19].

The evaluation of ChatGPT’s responses for the five ophthalmic conditions highlighted various deficiencies, mainly in these aspects: (1) Ambiguous intervention steps and applicability criteria, particularly for conservative treatments of cataracts, refractive errors, and AMD (Item 5), ChatGPT did not adequately define the boundaries of non-surgical management, neglecting the personalized treatment requirements for these conditions. For example, early-stage cataracts or dry AMD may be addressed with conservative methods, while wet AMD necessitates pharmaceutical or surgical treatments [20]. The extent of non-invasive approaches for managing refractive issues was not clearly defined, potentially causing confusion about what patients should anticipate from such treatments. For the management of glaucoma and cataracts (Item 6), ChatGPT failed to outline detailed phase-specific strategies, which might result in patients having an unclear grasp of their medical situation [21,22]. (2) Insufficient data support, when addressing the quantification of risks and benefits (Item8, 10), ChatGPT did not provide specific clinical data or metrics, resulting in responses lacking scientific evidence and making inadequate data backing. This led to responses that lacked empirical support, hindering a thorough patient understanding of potential treatment outcomes and associated risks. (3) In discussing the qualitative benefits for glaucoma and AMD (Item 7), ChatGPT neglected to address the constraints of real treatment effectiveness, especially the impact of anti-VEGF therapy in AMD, which is primarily confined to decelerating the advancement of the disease. This oversight might result in patients harboring excessively optimistic expectations about the outcomes of their treatments. (4) Regarding refractive errors (Item 9), ChatGPT did not adequately outline the possible postoperative complications and their probabilities [23,24]. In summary, the constraints of ChatGPT in producing medical content highlight difficulties in managing intricate clinical situations and personalized healthcare requirements [25,26]. It is advisable to enhance and refine answers regarding intervention sequences and the management of patient expectations by incorporating expert medical advice [27,28]. In the future, the growing reliance on data-driven insights and personalized medical advice will bolster the precision and scientific foundation of ChatGPT’s responses, thereby boosting patient confidence and the caliber of healthcare decisions [29].

An examination of ChatGPT’s medical data uncovered cases of what is commonly referred to as "AI hallucination." This phenomenon occurs when the AI generates information that appears plausible but is actually baseless or deceptive [30]. For questions about the expenses related to the five conditions (Item 15), ChatGPT gave ambiguous answers like "it varies based on the situation" or "it might be beneficial," without providing concrete cost estimates or relevant influencing factors. Despite seeming thorough, these replies were not grounded in practical data and did not offer patients useful guidance. The problem of "AI hallucinations" underscores major shortcomings in AI-produced medical advice. While AI can generate responses that are structurally sound, its answers frequently lack scientific accuracy and real-world applicability, especially in scenarios demanding clinical expertise or exact data. This highlights the necessity for robust verification processes when using AI-generated health information, to avoid patients making incorrect health choices based on inaccurate data [31].

5 Conclusion

While ChatGPT, a swiftly advancing language model, shows initial promise in producing medical content, it also faces notable constraints. Currently, ChatGPT is still in the experimental phase, and its responses can be inaccurate or lack scientific rigor in some complex situations. Nonetheless, as technology progresses and its range of uses broadens, the model is expected to assume additional functions. These could include helping medical practitioners swiftly retrieve health data, aiding in patient education and wellness guidance, and acting as a supplementary instrument for analyzing medical research [32]. Future advancements ought to concentrate on boosting the model’s learning capacities and its access to scientific information, enhancing its dependability and practicality in medical settings, and catering to the individual requirements of patients and healthcare professionals.

Supporting information

Acknowledgments

The authors thank ChatGPT-4o (OpenAI) for English language editing assistance during manuscript preparation