Developing an evaluation framework for public health environmental surveillance: Protocol for an international, multidisciplinary Delphi consensus study

Douglas G. Manuel; Carol Bennett; Emma Brown; David L. Buckeridge; Yoni Freedhoff; Sarah Funnell; Farah Ishtiaq; Matthew J. Wade; David Moher; on behalf of the PHES-EF Executive Group

doi:10.1371/journal.pone.0310342

Peer Review History

Original SubmissionSeptember 9, 2024
9 Sep 2024 Author Response https://doi.org/10.1371/journal.pone.0310342.r001
30 Dec 2024 Decision Letter - Angela Mendes Freitas, Editor PONE-D-24-33538Developing an evaluation framework for public health environmental surveillance: Protocol for an international, multidisciplinary e-Delphi studyPLOS ONE Dear Dr. Manuel, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. ============================== The reviewers have reviewed the work you have done and recognize the contribution of the research to the field. However, they have also provided constructive comments and suggestions to improve the clarity and overall quality of the manuscript, ensuring it meets the standards for publication. Please pay particular attention to the methodology section, ensuring detailed and clear information is provided regarding the selection of panelists, the statistical methods employed, and the criteria used for reaching consensus. We kindly request that you address these comments in a revised version of your manuscript. ============================== Please submit your revised manuscript by Feb 13 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org . When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript: A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols . Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols . We look forward to receiving your revised manuscript. Kind regards, Angela Mendes Freitas Academic Editor PLOS ONE Journal requirements: When submitting your revision, we need you to address these additional requirements. 1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf 2. When completing the data availability statement of the submission form, you indicated that you will make your data available on acceptance. We strongly recommend all authors decide on a data sharing plan before acceptance, as the process can be lengthy and hold up publication timelines. Please note that, though access restrictions are acceptable now, your entire data will need to be made freely accessible if your manuscript is accepted for publication. This policy applies to all data except where public deposition would breach compliance with the protocol approved by your research ethics board. If you are unable to adhere to our open data policy, please kindly revise your statement to explain your reasoning and we will seek the editor's input on an exemption. Please be assured that, once you have provided your new statement, the assessment of your exemption will not hold up the peer review process. 3. One of the noted authors is a group or consortium [PHES-EF Executive Group]. In addition to naming the author group, please list the individual authors and affiliations within this group in the acknowledgments section of your manuscript. Please also indicate clearly a lead author for this group along with a contact email address. [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. Does the manuscript provide a valid rationale for the proposed study, with clearly identified and justified research questions? The research question outlined is expected to address a valid academic problem or topic and contribute to the base of knowledge in the field. Reviewer #1: Yes Reviewer #2: Yes ******** 2. Is the protocol technically sound and planned in a manner that will lead to a meaningful outcome and allow testing the stated hypotheses? The manuscript should describe the methods in sufficient detail to prevent undisclosed flexibility in the experimental procedure or analysis pipeline, including sufficient outcome-neutral conditions (e.g. necessary controls, absence of floor or ceiling effects) to test the proposed hypotheses and a statistical power analysis where applicable. As there may be aspects of the methodology and analysis which can only be refined once the work is undertaken, authors should outline potential assumptions and explicitly describe what aspects of the proposed analyses, if any, are exploratory. Reviewer #1: Partly Reviewer #2: Yes ****** 3. Is the methodology feasible and described in sufficient detail to allow the work to be replicable? Descriptions of methods and materials in the protocol should be reported in sufficient detail for another researcher to reproduce all experiments and analyses. The protocol should describe the appropriate controls, sample size calculations, and replication needed to ensure that the data are robust and reproducible. Reviewer #1: Yes Reviewer #2: Yes ****** 4. Have the authors described where all data underlying the findings will be made available when the study is complete? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception, at the time of publication. The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: Yes Reviewer #2: Yes ****** 5. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes Reviewer #2: Yes ****** 6. Review Comments to the Author Please use the space provided to explain your answers to the questions above and, if applicable, provide comments about issues authors must address before this protocol can be accepted for publication. You may also include additional comments for the author, including concerns about research or publication ethics. You may also provide optional suggestions and comments to authors that they might find helpful in planning their study. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: Overall, this study protocol is well-prepared and addresses an important issue in public health surveillance. The authors’ structured approach, use of the Delphi method, and comprehensive dissemination plan indicate a strong foundation for a valuable contribution to public health. Nevertheless, to improve the study, the following items can be reviewed and revised: • The background and rationale could be strengthened by adding a section specifically addressing the limitations of current evaluation methods and how this framework seeks to address those gaps. • The article could include references to recent advancements or alternative applications of wastewater-based surveillance (e.g., monitoring antimicrobial resistance, drug residues). • Consider a pilot round to identify potential issues with the survey questions or consensus-building platform (Surveylet Delphi). • Additional details on post-publication use and integration of the framework in policy-making and public health systems would enhance practical applicability. • How this framework might adapt to different socio-economic and geographic contexts should be discussed. Almost, it could be say that if any other researcher were to conduct this study, they would likely use the same approach. As a result, I believe there is a lack of specific innovation or creativity in this protocol. It is recommended to consider modifications to the research methodology or interventions to enhance the protocol's appeal and impact, thereby creating more pronounced distinctions from similar studies. Reviewer #2: The methods and protocol sections are comprehensive and well-structured. This reflects a good approach to the Delphi process. I think there is an opportunity to provide additional detail to better support clarity and detail reproducibility. For instance, specifying the scoring scale used by panelists and clearly defining the criteria for achieving consensus would strengthen the methodology. The protocol emphasizes EDI, but the selection process for panelists could be better explained to outline how diversity in expertise and demographics will be ensured, and importantly how it might shape/improve the outcomes. While the quantitative analysis is detailed, further explanation of how median, range, and IQR will be used to define consensus and stability is needed. Also, elaborating on why Fisher’s exact test was chosen and its threshold for significance would provide a stronger rationale for the approach. I like to see stats choices well defined. The ethical considerations part is strong, though additional information on how informed consent will be managed during hybrid meetings will strengthen the methodology. Are there any fallback plans for managing (potential) panelist dropouts (it is a common challenge for a Delphi), particularly in subgroups? This should be noted. Grammar and Style The grammar and style of the paper are clear but could benefit from a careful edit and re-read. Several sentences, particularly in the "Study Record Retention" and "Research Data Management" sections, are lengthy and could be restructured into shorter, more readable forms. Redundancy in describing data encryption and ethical compliance could be streamlined to avoid repetition. And, yes I know this is stylistic, but use of active voice would improve the paper -- especially for professional audiences. I also suggest adding really basic introductory sentences before diving into complex details in sections (like EDI) to improve the overall flow and applicability of content. A Few Suggestions To strengthen the draft, maybe provide templates or examples for the demographic questionnaire and controlled feedback forms. These would enhance transparency. Plans for dissemination are good, particularly the integration of a website and social media platforms. Perhaps including strategies for keeping the framework dynamic and updated post-publication would be helpful? Are there strategies for participant fatigue? Such as incentives or even streamlined surveys. Recognizing the potential influence of controlled feedback on panelist opinions and detailing mitigation strategies to minimize bias could improve outcomes. Overall, the protocol has a good methodology and commitment to inclusivity, but it would benefit from more precise descriptions of statistical methods, panelist selection processes, and importantly an outline of contingency plans. Refining language for consistency, reducing redundancy, and breaking down complex sentences will enhance the clarity and accessibility. It is an applied work, so writing for such audiences is important. ****** 7. PLOS authors have the option to publish the peer review history of their article (what does this mean? ). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy . Reviewer #1: Yes: Hojatolah Gharaee Reviewer #2: No ******** [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/ . PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org . Please note that Supporting Information files do not need this step. https://doi.org/10.1371/journal.pone.0310342.r002
Revision 1
13 Feb 2025 Author Response Review Comments to the Author Reviewer #1: Overall, this study protocol is well-prepared and addresses an important issue in public health surveillance. The authors’ structured approach, use of the Delphi method, and comprehensive dissemination plan indicate a strong foundation for a valuable contribution to public health. Nevertheless, to improve the study, the following items can be reviewed and revised: 1. The background and rationale could be strengthened by adding a section specifically addressing the limitations of current evaluation methods and how this framework seeks to address those gaps. Thank you for providing these insights. We agree with you, and we have introduced a paragraph to address this suggestion. 2. The article could include references to recent advancements or alternative applications of wastewater-based surveillance (e.g., monitoring antimicrobial resistance, drug residues). We agree with your assessment and have highlighted the role WBS for antimicrobial resistance, with additional statements indicating WBS broad uses. Consider a pilot round to identify potential issues with the survey questions or consensus-building platform (Surveylet Delphi). We have incorporated your suggestion in the Procedure section. We will pilot test the survey for up to five panellists, composed of executive group members and other potential panellists matching our recruitment criteria. The pilot test will focus on assessing the clarity and relevance of background information provided for candidate items. The survey will be revised as necessary following feedback from pilot testing. 3. Additional details on post-publication use and integration of the framework in policy-making and public health systems would enhance practical applicability. We have revised the “Step 5: Dissemination of evaluation framework” section to include further details about our post-publication knowledge translation strategies. Post-publication, we will aim to keep the framework dynamic and up to date using a website, Discourse server, or other social media platforms to elicit ongoing feedback. Post-publication strategies may also include the creation of guides, toolkits, or ontologies to enhance the practical applicability of the evaluation framework and facilitate its adoption for policy and decision-making. 4. How this framework might adapt to different socio-economic and geographic contexts should be discussed. Thank you for highlighting the importance of an evaluation framework’s ability to adapt to different socio-economic and geographic contexts. Our Executive Group emphasized the importance of this perspective, and we feel that this is a strength in our methodology, however we now realize that it was under-developed in our manuscript. We have added a description of how we will report e-Delphi findings relating to equity in the “Equity, Diversity, and Inclusion (EDI)” section of the manuscript. We have also included the rating properties for candidate items in the “Defining consensus” section of the manuscript. Candidate items will be evaluated across four rating properties: (1) Relevance and practical utility; (2) scientific rigor, validity and reliability; (3) feasibility, adaptability and resource implications; and (4) equity, inclusiveness, and mitigation of bias. These rating properties were chosen to ensure that the framework addresses the needs of varied settings. From this, we will assess the level of consensus on these candidate items, specifically highlighting candidate items rated as strong or weak in terms of their equity implications. Of note, after reviewing the literature, our study is the first study that we are aware of that uses equity as a specific rating property. Where feasible, and with sufficient statistical power, we will further analyze the data by subgroup, distinguishing respondents from high-, middle-, and low-income settings, as well as different world regions. This will allow for a nuanced understanding of how these contexts influence the development of the framework and which candidate items may be most valued in these areas. 5. Almost, it could be say that if any other researcher were to conduct this study, they would likely use the same approach. As a result, I believe there is a lack of specific innovation or creativity in this protocol. It is recommended to consider modifications to the research methodology or interventions to enhance the protocol's appeal and impact, thereby creating more pronounced distinctions from similar studies. While we appreciate the perspective regarding the potential similarity of approaches among researchers, we respectfully suggest that the strength of this protocol lies in its emphasis on rigour and reproducibility. Using a multinational, multidisciplinary e-Delphi process ensures an evidence-based, comprehensive and collaborative approach. This methodology prioritizes consensus-building among diverse experts. We note that rigorous scoping reviews and consensus studies have not been performed for surveillance evaluation frameworks. The concept of international consensus for public health policy and decision-making was elevated during the Covid-19 pandemic [1–3]. Therefore, our innovation is in applying lessons learned from consensus-building during the pandemic to this space. Given, that wastewater-based surveillance is multidisciplinary and currently evolving, we feel that is would benefit from a rigorous consensus-based exercise. Works Cited: 1. Lazarus JV, Romero D, Kopka CJ, Karim SA, Abu-Raddad LJ, Almeida G, et al. A multinational Delphi consensus to end the COVID-19 public health threat. Nature. 2022;611: 332–345. doi:10.1038/s41586-022-05398-2 2. Hillmer MP, Feng P, McLaughlin JR, Murty VK, Sander B, Greenberg A, et al. Ontario’s COVID-19 Modelling Consensus Table: mobilizing scientific expertise to support pandemic response. Can J Public Health. 2021;112: 799–806. doi:10.17269/s41997-021-00559-8 3. Balestracci B, La Regina M, Di Sessa D, Mucci N, Angelone FD, D’Ecclesia A, et al. Patient safety implications of wearing a face mask for prevention in the era of COVID-19 pandemic: a systematic review and consensus recommendations. Intern Emerg Med. 2023;18: 275–296. doi:10.1007/s11739-022-03083-w Reviewer #2: The methods and protocol sections are comprehensive and well-structured. This reflects a good approach to the Delphi process. 6. I think there is an opportunity to provide additional detail to better support clarity and detail reproducibility. For instance, specifying the scoring scale used by panelists and clearly defining the criteria for achieving consensus would strengthen the methodology. Thank you. We agree and we have revised to better explain the consensus rating process. We have already incorporated the details of the fully labelled 7-point Likert scale to be used by panellists in Fig 2. However, we have now included the rating properties that will be used to score each candidate item. We have incorporated your feedback in the “Defining consensus” section of the manuscript and are including a new supporting information file as part of the revision titled “S4 Appendix. Rating properties for e-Delphi candidate item scoring.” 7. The protocol emphasizes EDI, but the selection process for panelists could be better explained to outline how diversity in expertise and demographics will be ensured, and importantly how it might shape/improve the outcomes. Thank you for your feedback. The study working group will monitor the distribution of panellists based on demographic information submitted during active recruitment. Based on the submitted information, the study working group will conduct targeted recruitment of certain cohorts (e.g., discipline subgroup, world region, etc.) if it appears that they are currently underrepresented. Targeted recruitment of specific cohorts will be conducted to ensure a diversity in expertise and demographics among panellists. This information has been incorporated in the “Recruitment of panellists” and “Equity, diversity, and inclusion (EDI)” sections of the manuscript. 8. While the quantitative analysis is detailed, further explanation of how median, range, and IQR will be used to define consensus and stability is needed. Also, elaborating on why Fisher’s exact test was chosen and its threshold for significance would provide a stronger rationale for the approach. I like to see stats choices well defined. Fisher’s exact test was chosen for its suitability in analyzing small sample sizes and categorical data, as it does not rely on the assumption of large sample sizes required by other tests, such as the chi-squared test. This makes it a good choice for comparing item ratings across subgroups, particularly given the potential variability in panellist numbers across disciplines, income levels, and regions. Regarding the threshold for significance, we will use a standard alpha level of 0.05, unless multiple comparisons necessitate an adjustment, such as using the Bonferroni correction. The median, range, and IQR will not be used to define consensus but rather to report stability and the differences between groups. They will only be used for descriptive secondary analyses of e-Delphi results. Including this rationale in the manuscript will provide greater transparency and clarity, and we appreciate your suggestion to ensure our statistical choices are well-defined and justified. 9. The ethical considerations part is strong, though additional information on how informed consent will be managed during hybrid meetings will strengthen the methodology. We have modified the “Step 4: Consensus meeting” section of the manuscript. Study executive group members will attend the consensus meeting. Therefore, an informed consent process will not be required, as executive group members are co-authors on this study. The consensus meeting will be guided by a Terms of Reference (ToR) document that executive group members agreed upon at the start of the study process. Chatham House Rule will apply to all executive group meetings, including the consensus meeting. The rule will encourage the free expression of ideas without attributing them to an individual or institution. 10. Are there any fallback plans for managing (potential) panelist dropouts (it is a common challenge for a Delphi), particularly in subgroups? This should be noted. Thank you for highlighting the importance of having a mitigation plan. Our fallback plans for managing potential panellist dropouts include recruitment and retention strategies. We aim to conduct robust recruitment approximately four weeks before the start of the e-Delphi survey. We will also continue recruiting throughout each e-Delphi survey round (approximately 4-6 weeks). We are allowing new panellists to participate in the second round to mitigate our reliance on panellist from the first round. This means panellists can participate in the first or second round only, or both, depending on their preference. However, to help retention efforts, we only offer compensation to panellists who complete all rounds of the e-Delphi survey. As noted in our response to question #7, the study working group will monitor the distribution of panellists based on demographic information submitted during active recruitment. Based on the submitted information, the study working group will conduct targeted recruitment of certain cohorts (e.g., discipline subgroup, world region, etc.) if it appears that they are currently underrepresented. Targeted recruitment will occur to help mitigate against uneven panellist dropout across cohorts, to ensure a diversity in expertise and demographics among panellists. 11. The grammar and style of the paper are clear but could benefit from a careful edit and re-read. Several sentences, particularly in the "Study Record Retention" and "Research Data Management" sections, are lengthy and could be restructured into shorter, more readable forms. Redundancy in describing data encryption and ethical compliance could be streamlined to avoid repetition. And, yes I know this is stylistic, but use of active voice would improve the paper -- especially for professional audiences. I also suggest adding really basic introductory sentences before diving into complex details in sections (like EDI) to improve the overall flow and applicability of content. Thank you for your helpful feedback. We have revised and streamlined the "Study Record Retention" and "Research Data Management" sections. We have also added two introductory sentences to the beginning of the “Equity, diversity, and inclusion (EDI)” section. We have reviewed the entire manuscript to improve the overall flow and applicability of the content. 12. To strengthen the draft, maybe provide templates or examples for the demographic questionnaire and controlled feedback forms. These would enhance transparency. Thank you for your thoughtful suggestion. We have included two new supporting information files as part of the revision titled: “S4 Appendix. Rating properties for e-Delphi candidate item scoring” and “S7 Appendix. Draft e-Delphi panellist screening questionnaire.” 13. Plans for dissemination are good, particularly the integration of a website and social media platforms. Perhaps including strategies for keeping the framework dynamic and updated post-publication would be helpful? Thank you for your valuable feedback. We have revised the “Step 5: Dissemination of evaluation framework” section to include further details about our post-publication knowledge translation strategies. Post-publication, we will aim to keep the framework dynamic and up to date using a website, Discourse server, or other social media platforms to elicit ongoing feedback. We are planning on creating a living review from the scoping studies. Those studies have entailed generating an ontology to synthesize concepts. That ontology will be finalized based on the consensus exercise and published. Post-publication strategies may also include creating guides and toolkits to enhance the practical applicability of the evaluation framework and facilitate its adoption for policy and decision-making. 14. Are there strategies for participant fatigue? Such as incentives or even streamlined surveys. Recognizing the potential influence of controlled feedback on panelist opinions and detailing mitigation strategies to minimize bias could improve outcomes. Thank you for highlighting the importance of having a mitigation plan. To help retention efforts, we will only compensate panellists who complete all rounds of the e-Delphi survey. We will administer streamlined survey pathways depending on the panellist’s expertise. For example, members of the engaged public will have the option to skip ascertainment performance measurements to allow them to focus on attributes such as communication and partnerships. Our fallback plans for managing potential panellist dropouts include recruitment and retention strategies. We aim to conduct robust recruitment approximately four weeks before the start of the e-Delphi survey. We will also continue recruiting throughout each e-Delphi survey round (approximately 4-6 weeks). We are allowing new panellists to participate in the second round to mitigate our reliance on panellist dropouts. This means panellists can participate in the first or second round only, or both, depending on their preference. As noted in our response to question #7, the study working group will monitor the distribution of panellists based on demographic information submitted during active recruitment. Based on the submitted information, the study working group will conduct targeted recruitment of certain cohorts (e.g., discipline subgroup, world region, etc.) if it appears that they are currently underrepresented. Targeted recruitment will occur to help mitigate against uneven panellist dropout across cohorts, to ensure a diversity in expertise and demographics among panellists. 15. Overall, the protocol has a good methodology and commitment to inclusivity, but it would benefit from more precise descriptions of statistical methods, panelist selection processes, and importantly an outline of contingency Attachments Attachment Submitted filename: Response to Reviewers.pdf https://doi.org/10.1371/journal.pone.0310342.r003
22 Apr 2025 Decision Letter - Angela Mendes Freitas, Editor Developing an evaluation framework for public health environmental surveillance: Protocol for an international, multidisciplinary Delphi study PONE-D-24-33538R1 Dear Dr. Manuel, We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication. An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org. If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. Kind regards, Angela Mendes Freitas Academic Editor PLOS ONE Additional Editor Comments (optional): Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. Does the manuscript provide a valid rationale for the proposed study, with clearly identified and justified research questions? The research question outlined is expected to address a valid academic problem or topic and contribute to the base of knowledge in the field. Reviewer #1: Yes Reviewer #2: Yes ******** 2. Is the protocol technically sound and planned in a manner that will lead to a meaningful outcome and allow testing the stated hypotheses? The manuscript should describe the methods in sufficient detail to prevent undisclosed flexibility in the experimental procedure or analysis pipeline, including sufficient outcome-neutral conditions (e.g. necessary controls, absence of floor or ceiling effects) to test the proposed hypotheses and a statistical power analysis where applicable. As there may be aspects of the methodology and analysis which can only be refined once the work is undertaken, authors should outline potential assumptions and explicitly describe what aspects of the proposed analyses, if any, are exploratory. Reviewer #1: Yes Reviewer #2: Yes ****** 3. Is the methodology feasible and described in sufficient detail to allow the work to be replicable? Descriptions of methods and materials in the protocol should be reported in sufficient detail for another researcher to reproduce all experiments and analyses. The protocol should describe the appropriate controls, sample size calculations, and replication needed to ensure that the data are robust and reproducible. Reviewer #1: Yes Reviewer #2: Yes ****** 4. Have the authors described where all data underlying the findings will be made available when the study is complete? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception, at the time of publication. The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: Yes Reviewer #2: Yes ****** 5. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes Reviewer #2: Yes ****** 6. Review Comments to the Author Please use the space provided to explain your answers to the questions above and, if applicable, provide comments about issues authors must address before this protocol can be accepted for publication. You may also include additional comments for the author, including concerns about research or publication ethics. You may also provide optional suggestions and comments to authors that they might find helpful in planning their study. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: I have carefully reviewed the authors' responses to my initial comments and the corresponding revisions made to the manuscript. I appreciate the thorough and thoughtful manner in which the authors have addressed all the raised concerns. Reviewer #2: The authors have addressed the reviewer comments and suggestion. I look forward to seeing how the project is implemented. It has relevance to other Delhi approaches and will be a helpful; model in other fields. ****** 7. PLOS authors have the option to publish the peer review history of their article (what does this mean? ). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy . Reviewer #1: No Reviewer #2: No ******** https://doi.org/10.1371/journal.pone.0310342.r004
Formally Accepted
Acceptance Letter - Angela Mendes Freitas, Editor PONE-D-24-33538R1 PLOS ONE Dear Dr. Manuel, I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team. At this stage, our production department will prepare your paper for publication. This includes ensuring the following: * All references, tables, and figures are properly cited * All relevant supporting information is included in the manuscript submission, * There are no issues that prevent the paper from being properly typeset You will receive further instructions from the production team, including instructions on how to review your proof when it is ready. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few days to review your paper and let you know the next and final steps. Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. If we can help with anything else, please email us at customercare@plos.org. Thank you for submitting your work to PLOS ONE and supporting open access. Kind regards, PLOS ONE Editorial Office Staff on behalf of Dr. Angela Mendes Freitas Academic Editor PLOS ONE https://doi.org/10.1371/journal.pone.0310342.r005

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .