PONE-D-24-54284GPT-4 GENERATES ACCURATE AND READIBLE PATIENT EDUCATION MATERIALS ALIGNED WITH CURRENT ONCOLOGICAL GUIDELINES. A RANDOMIZED ASSESSMENT

Dear Dr. Cacciamani,

Thank you for submitting your manuscript "Guideline-based patient educational materials using generative artificial intelligence" to PLOS ONE. Please accept our sincere apologies for the delay in getting this report to you. The difficulty in securing appropriate reviewers with expertise in both urological education materials and artificial intelligence applications resulted in a longer review process than anticipated.

We have now received comments from one reviewer, and I have also personally reviewed the manuscript in detail - See this under editor comments below. Both reviews identify several strengths in your work, particularly the novel application of GenAI to creating patient educational materials with improved readability while maintaining comparable accuracy to traditional methods.

However, there are several methodological issues and areas for improvement that need to be addressed before your manuscript can be considered further. I have attached both the reviewer's comments and my own editorial assessment for your consideration.

Key areas requiring attention include:

Methodological details regarding the GenAI framework and prompting approach

Justification for sample sizes and assessment methodology

Statistical reporting and analysis of the translation quality findings

Enhanced discussion of limitations and regulatory considerations

Please provide a point-by-point response to all comments, either implementing the suggested changes or providing a clear rationale for why certain suggestions cannot be accommodated. 

If you require any clarification regarding the comments or need assistance with the revision process, please do not hesitate to contact our editorial office Please submit your revised manuscript by Apr 18 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org . When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols . Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols .

We look forward to receiving your revised manuscript.

Kind regards,

Isaac Amankwaa, Ph.D.

Guest Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please note that PLOS ONE has specific guidelines on code sharing for submissions in which author-generated code underpins the findings in the manuscript. In these cases, we expect all author-generated code to be made available without restrictions upon publication of the work. Please review our guidelines at https://journals.plos.org/plosone/s/materials-and-software-sharing#loc-sharing-code and ensure that your code is shared in a way that follows best practice and facilitates reproducibility and reuse.

3. Thank you for stating the following in your Competing Interests section: 

“NONE”

Please complete your Competing Interests on the online submission form to state any Competing Interests. If you have no competing interests, please state "The authors have declared that no competing interests exist.", as detailed online in our guide for authors at http://journals.plos.org/plosone/s/submit-now

 This information should be included in your cover letter; we will change the online submission form on your behalf.

4. In the online submission form you indicate that your data is not available for proprietary reasons and have provided a contact point for accessing this data. Please note that your current contact point is a co-author on this manuscript. According to our Data Policy, the contact point must not be an author on the manuscript and must be an institutional contact, ideally not an individual. Please revise your data statement to a non-author institutional point of contact, such as a data access or ethics committee, and send this to us via return email. Please also include contact information for the third party organization, and please include the full citation of where the data can be found.

5. Please include a separate caption for each figure in your manuscript.

6. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Additional Editor Comments:

• Abstract
• Line 184, page 4, under abstract – the patient summary could be slightly expanded to match the level of detail in the rest of the abstract while maintaining its improved readability. Also, ‘patient info,’ should be written in full.
• Line 161, page 4 – Young academic urologist (YAU).. revise to ensure consistency in capitalisation.
• Introduction
• Consider starting with the challenge of health literacy and readable PEMs before mentioning internet statistics
• A brief mention of previous research, specifically on AI in medical education materials, would strengthen your argument
• Add a sentence addressing the potential ethical implications or limitations of GenAI in medical information
• Consider including a line about the novelty/gap your study addresses
• Line 206: "on yearly bases" should be "on a yearly basis"
• Line 214: "different scholarity levels" - Consider using "educational" or "literacy" levels instead of "scholarity"
• Line 211: "but lacks consistent correctness when relying on the proprietary knowledge domain" - This phrase is somewhat unclear; consider rewording for clarity
• Line 209: Consider rephrasing "not meeting requirements set by the European Union (EU) for understandability of clinical summaries" for better clarity
• Methods
• The "GPT-4 framework development and custom-GPT creation" mentioned in the opening paragraph is never fully explained. Please provide some explanation to complement the figure provided to ensure reproducibility.
• Regarding your prompting methodology for GPT-4, while you mention that "prompts were based on previously published methods and modified to extract guideline information," this provides insufficient detail for reproducibility. Please consider enhancing this section by:
• Including at least one complete example prompt
• Specifying the exact modifications made to previously published methods
• Detailing any GPT-4 parameters used (temperature, token limits, etc.)
• Explaining how your prompts specifically targeted EU-recommended reading levels
• Describing any iterative refinement process used to optimize the prompt.
• Regarding the sample size of reviewers (12 for pilot, 20 for main assessment with 5 per cancer type), I note the absence of a formal justification for these numbers. While traditional power calculations may be challenging for expert review studies, I recommend:
• Providing a brief rationale for the chosen sample sizes based on: Previous similar validation studies in medical education/PEM literature
• Practical considerations regarding YAU expert availability
• Anticipated effect sizes based on pilot results
• Acknowledging this limitation and discussing how you addressed potential reliability concerns with a relatively small number of reviewers per cancer type
• There is no mention of translation validation methodology or back-translation verification.
• No explanation of how specific PEMs were selected for evaluation or whether they represent the full spectrum of urological cancers.

Results

• Line 356: Mentions "translations to Dutch, French, German, Italian and French" - French is listed twice, likely a typo
• Line 386: p-value written as "p=738." instead of "p=.738"
• Line 372-373: Only reports median values (4) for all measures in the pilot study, making it impossible to detect nuanced differences
• The study reports numerous non-significant p-values but no effect sizes or confidence intervals
• No interrater reliability metrics (e.g., Fleiss' kappa) are reported despite multiple raters
• Results show identical median values (mostly 4) for almost all comparisons, raising questions about the sensitivity of your measurement scale

Discussion

• No discussion of why the readability metrics were better for GenAI but accuracy and clarity ratings were similar
• Limited exploration of the variability in translation quality (simply stating "majority of cases" is imprecise)
• No explanation for the seemingly contradictory finding that GenAI PEMs were less favored for localized disease but more favored for metastatic disease (line 421-423)
• Limited connection to broader AI ethics literature or frameworks for medical AI implementation
• No discussion of regulatory considerations for AI-generated patient materials
• Minimal connection to patient communication theory or health literacy research
• Need to include a limitation section that:
• discusses the potential limitations in the assessment methodology (small sample of expert reviewers, potential biases)
• discusses Likert scale's ability to detect meaningful differences when most ratings clustered around 4
• reflects on whether urologists are the ideal evaluators of patient-oriented materials
• any other limitations

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Abstract, Line 158:

The sentences "A randomized assessment of the correctness of these materials was conducted by 32 urologists and urology residents." and "Thirty-two members of the Young academic urologists (YAU) groups evaluated the accuracy, completeness, and clarity of the original versus GPT-generated PEMs." seem to repeat the same information. Please use just one of them to avoid redundnacy.

This same comment applies to the same section for the sentences "Additionally, translations into other languages were evaluated for correctness and clarity by native-speaker urologists." and "The translation assessment involved two native speakers from different YAU groups for each language: Dutch, French, German, Italian, and Spanish."

Abstract, results: Please mention the number of original PEMs and AI generated PEMs assessed here, not just in the main results section.

The fact that translations of GPT-generated PEMs were rated as correct in 77.5% of cases and clear in 67.5% of cases, which indicates an error rate that should not be neglected and that AI is still not very reliable to translate such information without human supervision to avoid misinformation and chaos. Please elaborate on this in the discussion and highlight as a point of weakness in these generated PEMs, aside from highlighting their strength in more proper readability when compared to original PEM.

FKRE should stand for Flesch Kincaid Reading Ease, not Flesch Reading Ease. Please correct it in the entire manuscript.

Results, Line 368: The readability assessment is represented in figure 2, and not figure 1 as the text claims. Please correct it. Also, in line 388. The correctness assessment is represented in figure 3, and not figure 2 as the text says. The same error is i line 395 (Should be figure 4 instead of 3).

There is no Supplementary Table 3, and there is a jump from Supp. Table 2 to 4. Please rename 4 to 3 and correct in the text as well.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean? ). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy .

Reviewer #1: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/ . PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org . Please note that Supporting Information files do not need this step.

GPT-4 GENERATES ACCURATE AND READABLE PATIENT EDUCATION MATERIALS ALIGNED WITH CURRENT ONCOLOGICAL GUIDELINES. A RANDOMIZED ASSESSMENT

PONE-D-24-54284R1

Dear Authors,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager®  and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Isaac Amankwaa, Ph.D.

Guest Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments: