Identification of digital clinical decision support systems for supporting diagnosis and triage of patients with shoulder disorders: A scoping review protocol

Cheyenne R. Schamerhorn; Nathaniel M. Peñas; Jared R. Fletcher; Richard Hayman; Breda H. F. Eubank

doi:10.1371/journal.pone.0327192

Peer Review History

Original SubmissionSeptember 17, 2024
Decision Letter - Reza Rabiei, Editor Dear Dr. Eubank, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please submit your revised manuscript by Apr 11 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org . When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols . Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols . We look forward to receiving your revised manuscript. Kind regards, Reza Rabiei Academic Editor PLOS ONE Journal Requirements: When submitting your revision, we need you to address these additional requirements. 1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf . Additional Editor Comments (if provided): Dear Authors, Thank you for your submission to PLOS One. We have been into the review of the manuscript and based on the reviewers' comments and our assessments, there are minor and major issues with the manuscript. Please find the concerns raised by the reviewers, including the inconsistencies remarked and clarifications indicated. We look forward to receiving your revision and a point-by-point response letter indicating how the comments or the concerns have been handled. Sincerely, [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. Does the manuscript provide a valid rationale for the proposed study, with clearly identified and justified research questions? Reviewer #1: Yes Reviewer #2: Partly ******** 2. Is the protocol technically sound and planned in a manner that will lead to a meaningful outcome and allow testing the stated hypotheses??> Reviewer #1: Partly Reviewer #2: Partly ****** 3. Is the methodology feasible and described in sufficient detail to allow the work to be replicable??> Reviewer #1: Yes Reviewer #2: Yes ****** 4. Have the authors described where all data underlying the findings will be made available when the study is complete??> The PLOS Data policy Reviewer #1: Yes Reviewer #2: Yes ****** 5. Is the manuscript presented in an intelligible fashion and written in standard English??> Reviewer #1: Yes Reviewer #2: Yes ****** Please use the space provided to explain your answers to the questions above and, if applicable, provide comments about issues authors must address before this protocol can be accepted for publication. You may also include additional comments for the author, including concerns about research or publication ethics. You may also provide optional suggestions and comments to authors that they might find helpful in planning their study. Reviewer #1: The authors describe a pressing need for better support in identifying and appropriately treating musculoskeletal conditions in primary care settings. MSK conditions are relatively common, but complex, and notoriously difficult to accurately diagnose -- especially under the resource-limited conditions of a typical primary care visit. This is an excellent use case for CDSS, but a previous review (2016) failed to identify a viable system. CDSS development has been very active in recent years; a new review has the potential to be highly valuable. The authors make a clear and compelling case for this scoping review, and I see tremendous potential impact for this work. As it is currently written, however, I see a few inconsistencies that need to be cleared up, and one important issue: It is unclear why the authors are limiting the scope to knowledge-based CDSS. They discuss potential drawbacks of AI-based CDSS on lines 112-119, but I expected that they would still include and evaluate non-knowledge-based CDSS (and incorporate those potential drawbacks into their evaluation of each CDSS). If there is justification to exclude this entire genre of CDSS at the outset, that should be more clearly explained. Especially given the complexity of the task (200+ potential diagnoses, etc.), there is an obvious potential utility to leveraging machine learning, if it can be done effectively and results checked for bias, etc. There should be clearer justification for the exclusion criteria. In particular, criteria 1 (see above) and 2, both of which seem like they have the potential to result in more useful CDSS. If there are important constraints (e.g. compatibility with legacy software?) influencing these exclusion criteria, those should be surfaced in the Introduction. The final sentences of the Discussion make it sound like the intended scope of the tool is shoulder pain specifically, but that is not reflected in the Introduction, proposed methods, or search terms (S2). MOST IMPORTANT ISSUE: The two guiding research questions on lines 145-149 are not adequately addressed in the proposed methods. The second question, effectiveness in improving patient care, is likely to be crucial in user acceptance for any new CDSS; de-emphasizing that in favor of measuring "affordability, accessibility, ease of use, compatibility, and generalizability" (line 207) feels like a serious mistake. Table 1 includes a single column for "health impact" -- this should be explained more thoroughly. What kinds of patient outcomes / health impacts will be considered? If this will arise during the course of the review, that should be made clear. The first guiding question puts emphasis on the initial patient assessment (e.g. medical history-taking), and the justification for doing so is compellingly laid out in the Introduction and Discussion. Unfortunately, data relevant to that are not included in Table 1, unless that is intended to be captured under "CDSS Design". I recommend adding a column capturing what point in the patient care process (e.g. medical history-taking, physical examination, referrals for diagnostic imaging; referrals to specialist care; and/or treatment, line 153-154) each CDSS targets. Moreover, "affordability, accessibility, ease of use, compatibility, and generalizability" (or similar wording) are mentioned several times as key qualities to look for in CDSS, but I don't see those clearly reflected in either the two guiding research questions or the data to be collected in Table 1 (and, I would argue, something like "effectiveness in improving patient outcomes" should be added to that list). My overall impression is that the authors have thought very carefully about how to conduct the article search, but the key outcomes to measure are as yet not clearly defined. I think either Table 1 should be substantially revised to better reflect the priorities laid out in the Introduction, or there should be additional guidance included for using Table 1 (i.e. a clear and detailed description for what each cell is intended to capture -- things like "authors" are obvious, but not so for the most important columns, those under Methodology and CDSS Descriptors). Small typos: Line 123 "clinicians" is missing a possessive apostrophe (clinicians' or clinician's, depending on whether you intended plural or singular). Line 204 there is a word missing in the phrase "numeric, tabular, and/or format". Line 205 "synthesize" should be "synthesis". Line 230 there is a missing or extra word in the phrase "to identify a knowledge-based CDSS exist" Line 241-243 is a sentence fragment. Reviewer #2: The authors describe a scoping review protocol to identify digital clinical decision support systems (CDSS) to support the diagnosis and triage of patients with musculoskeletal (MSK) conditions. The protocol is consistent with best practices for conducting a structured literature review, and is described at an appropriate level of detail. Despite the overall quality of the presentation, there are several instances of unclear or awkward wording that should be addressed. It appears based on the article title that the authors intend to formulate a knowledge base that will allow them to assess the state-of-the-art in clinical digital decision support tools for patients with MSK conditions, and ideally to identify a clinical decision support system suitable to their needs. Indeed, this is articulated in the Discussion section where they write, “We anticipate that this review will help to either identify a CDSS that can be used to support point-of-care providers managing patients presenting with shoulder pain or identify similar CDSS where best practices can be borrowed and adapted for our shoulder work.” However, in the last sentence of the Background they write, “This scoping review aims to identify an affordable, easy to use, easily accessible CDSS…”, and in the Discussion, “The aim of this work will be to identify a knowledge-based CDSS exist (?) for not only directing primary point-of-care MSK care providers to appropriate management pathways, but also supporting the clinical examination (i.e. medical history-taking and physical examination) process.” The goal of the scoping review which – again, based on the title – appears to be, first and foremost, to systematically organize information gleaned from multiple sources in the literature, should be clearly articulated and consistent throughout the manuscript. Identifying a specific MSK and/or describing best practices would then be secondary goals of the project. Additional minor points: The first sentence of the second paragraph of the Background section reads, “Primary point-of-care providers themselves are restricted by the human body’s limited memory and knowledge base capacity…” This reference to cognitive limitations is well-placed but somewhat awkwardly framed. In the third paragraph of the Background section the authors state that “Knowledge-based systems are logical rules…” However, this is characterization is not completely accurate because while logical rules are a key component of knowledge-based systems, they include other components such as the user interface and the algorithms that process inputs, operate on the rules, and produce outputs. The first sentence of the Exclusion criteria section begins with “The following exclusion criteria are as follows,” which should be amended for wording, e.g., “The exclusion criteria are as follows:” The first sentence of the section “Stage 5: Collating, summarizing, and reporting the results” reads: “We will provide an overview and consolidate evidence from eligible articles in a numeric, tabular, and/or format.” The reference to “and/or format” is ambiguous in this context; this should be expanded upon or rephrased. ****** what does this mean? ). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy Reviewer #1: Yes: Rose M Hartman Reviewer #2: Yes: Michael E. Bales ******** [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/ . PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org . Please note that Supporting Information files do not need this step. https://doi.org/10.1371/journal.pone.0327192.r001
Revision 1
11 Apr 2025 Author Response TOPIC: Review response letter April 11, 2025 MANUSCRIPT ID: PONE-D-24-41186 Dear Dr Reza Rabiei and Editorial Team: We thank the reviewers for their helpful review and comments and have detailed the changes made in the submitted manuscript below. As per your instructions, received by email on February 26, 2025, this is our response letter accompanying the submission of manuscript (PONE-D-24-41186) to PLOS One. Below we provide a response to each point raised by the two reviewers from PLOS One, with reference to where changes can be viewed in the submitted manuscript. Reviewer #1: The authors describe a pressing need for better support in identifying and appropriately treating musculoskeletal conditions in primary care settings. MSK conditions are relatively common, but complex, and notoriously difficult to accurately diagnose -- especially under the resource-limited conditions of a typical primary care visit. This is an excellent use case for CDSS, but a previous review (2016) failed to identify a viable system. CDSS development has been very active in recent years; a new review has the potential to be highly valuable. The authors make a clear and compelling case for this scoping review, and I see tremendous potential impact for this work. As it is currently written, however, I see a few inconsistencies that need to be cleared up, and one important issue: It is unclear why the authors are limiting the scope to knowledge-based CDSS. They discuss potential drawbacks of AI-based CDSS on lines 112-119, but I expected that they would still include and evaluate non-knowledge-based CDSS (and incorporate those potential drawbacks into their evaluation of each CDSS). If there is justification to exclude this entire genre of CDSS at the outset, that should be more clearly explained. Especially given the complexity of the task (200+ potential diagnoses, etc.), there is an obvious potential utility to leveraging machine learning, if it can be done effectively and results checked for bias, etc. There should be clearer justification for the exclusion criteria. In particular, criteria 1 (see above) and 2, both of which seem like they have the potential to result in more useful CDSS. If there are important constraints (e.g. compatibility with legacy software?) influencing these exclusion criteria, those should be surfaced in the Introduction. The final sentences of the Discussion make it sound like the intended scope of the tool is shoulder pain specifically, but that is not reflected in the Introduction, proposed methods, or search terms (S2). Author Response 1: Thank you for your review of our manuscript. Your comments were insightful. As such, we have revised the inclusion criteria based on your recommendation to include both knowledge-based and non-knowledge-based CDSS. However, since this will impact the volume of our search results, we have refined both the search terms (S2 - Appendix) and inclusion criteria to only “shoulder disorders. We have invited Dr. Richard Hayman, an expert evidence synthesis librarian, to help revise the initial search strategy and added him as a co-author on our manuscript. All co-authors have agreed to this addition (see attached email). This was an oversight on our team in the first submission – which is obviously why there was confusion in our protocol language. The addition of Dr. Richard Hayman will ensure that our process will be clear and further conform to Joanna Briggs Institute and Cochrane Reviews methodologies. We have thus revised the manuscript’s introduction, methods, and discussion section accordingly, which will address Reviewer 1’s concerns above. Reviewer #1: MOST IMPORTANT ISSUE: The two guiding research questions on lines 145-149 are not adequately addressed in the proposed methods. The second question, effectiveness in improving patient care, is likely to be crucial in user acceptance for any new CDSS; de-emphasizing that in favor of measuring "affordability, accessibility, ease of use, compatibility, and generalizability" (line 207) feels like a serious mistake. Author Response 2: We agree and have decided that both questions are confusing and do not adequately reflect the scoping review’s primary objective. We have therefore clarified Question 1 to read: “What knowledge-based or non-knowledge-based CDSSs exist for supporting the diagnosis and management of patients presenting to primary care with shoulder disorders?” (lines 16-170) and removed Question 2. Instead, we have replaced Question 2 with the following: “A secondary objective will be to consolidate the existing evidence for evaluating the: 1) effectiveness of improving patient outcomes; and 2) quality of CDSSs in terms of affordability, accessibility, ease of use, and generalizability across different demographics (e.g. sex, gender, age) and shoulder pathologies” (lines 140 – 143). Reviewer #1: Table 1 includes a single column for "health impact" -- this should be explained more thoroughly. What kinds of patient outcomes / health impacts will be considered? If this will arise during the course of the review, that should be made clear. Author Response 2: We agree that health impact was vague. Our goal is to capture effectiveness in improving health outcomes as suggested. We have revised Table 1 to reflect this change. The table column header now reads “Effectiveness in improving health outcomes (e.g. synthesis of qualitative or quantitative data/patient reported outcome measures)”. Reviewer #1: The first guiding question puts emphasis on the initial patient assessment (e.g. medical history-taking), and the justification for doing so is compellingly laid out in the Introduction and Discussion. Unfortunately, data relevant to that are not included in Table 1, unless that is intended to be captured under "CDSS Design". I recommend adding a column capturing what point in the patient care process (e.g. medical history-taking, physical examination, referrals for diagnostic imaging; referrals to specialist care; and/or treatment, line 153-154) each CDSS targets. Author Response 1: Thank you for this suggestion. Table 1 has been revised as recommended and now reads “CDSS Point in Patient Care Process (e.g. medical history-taking, physical examination, referrals for diagnostic imaging, referrals to specialist care, and/or treatment) Reviewer #1: Moreover, "affordability, accessibility, ease of use, compatibility, and generalizability" (or similar wording) are mentioned several times as key qualities to look for in CDSS, but I don't see those clearly reflected in either the two guiding research questions or the data to be collected in Table 1 (and, I would argue, something like "effectiveness in improving patient outcomes" should be added to that list). Author Response 1: Thank you for this suggestion. Table 1 has been revised as recommended under the last column “Effectiveness in improving outcomes (e.g. synthesis of qualitative or quantitative data/patient reported outcome measures)”. Reviewer #1: My overall impression is that the authors have thought very carefully about how to conduct the article search, but the key outcomes to measure are as yet not clearly defined. I think either Table 1 should be substantially revised to better reflect the priorities laid out in the Introduction, or there should be additional guidance included for using Table 1 (i.e. a clear and detailed description for what each cell is intended to capture -- things like "authors" are obvious, but not so for the most important columns, those under Methodology and CDSS Descriptors). Author Response 1: Thank you for this suggestion and as described above, the manuscript and Table 1 have been revised as you have recommended. Reviewer #1: Small typos: Line 123 "clinicians" is missing a possessive apostrophe (clinicians' or clinician's, depending on whether you intended plural or singular). Author Response 1: We have removed this sentence. Reviewer #1: Line 204 there is a word missing in the phrase "numeric, tabular, and/or format". Author Response 1: We have corrected this sentence. Reviewer #1: Line 205 "synthesize" should be "synthesis". Author Response 1: We have corrected this sentence. Reviewer #1: Line 230 there is a missing or extra word in the phrase "to identify a knowledge-based CDSS exist" Author Response 1: We have corrected this sentence. Reviewer #1: Line 241-243 is a sentence fragment. Author Response 1: We have corrected this sentence. Reviewer #2: The authors describe a scoping review protocol to identify digital clinical decision support systems (CDSS) to support the diagnosis and triage of patients with musculoskeletal (MSK) conditions. The protocol is consistent with best practices for conducting a structured literature review, and is described at an appropriate level of detail. Despite the overall quality of the presentation, there are several instances of unclear or awkward wording that should be addressed. It appears based on the article title that the authors intend to formulate a knowledge base that will allow them to assess the state-of-the-art in clinical digital decision support tools for patients with MSK conditions, and ideally to identify a clinical decision support system suitable to their needs. Indeed, this is articulated in the Discussion section where they write, “We anticipate that this review will help to either identify a CDSS that can be used to support point-of-care providers managing patients presenting with shoulder pain or identify similar CDSS where best practices can be borrowed and adapted for our shoulder work.” However, in the last sentence of the Background they write, “This scoping review aims to identify an affordable, easy to use, easily accessible CDSS…”, and in the Discussion, “The aim of this work will be to identify a knowledge-based CDSS exist (?) for not only directing primary point-of-care MSK care providers to appropriate management pathways, but also supporting the clinical examination (i.e. medical history-taking and physical examination) process.” The goal of the scoping review which – again, based on the title – appears to be, first and foremost, to systematically organize information gleaned from multiple sources in the literature, should be clearly articulated and consistent throughout the manuscript. Identifying a specific MSK and/or describing best practices would then be secondary goals of the project. Author Response 2: Thank you for your review of our manuscript. Your comments were insightful. As mentioned above, we have refined the inclusion criteria to include both knowledge-based and non-knowledge-based CDSS. However, since this will impact the volume of our search results, we have limited both the search terms (S2 - Appendix) and inclusion criteria to only “shoulder disorders. We have invited Dr. Richard Hayman, an expert evidence synthesis librarian, to help revise the initial search strategy and added him as a co-author on our manuscript. All co-authors have agreed to this addition (see attached email). This was an oversight on our team in the first submission – which is obviously why there was confusion in our protocol language. The addition of Dr. Richard Hayman will ensure that our process will be clear and further conform to Joanna Briggs Institute and Cochrane Reviews methodologies. We have thus revised the manuscript’s introduction, methods, and discussion section accordingly, which will address Reviewer 1’s concerns above. We also agree and have decided that both questions are confusing and do not adequately reflect the scoping review’s primary objective. We have therefore clarified Question 1 to read: “What knowledge-based or non-knowledge-based CDSSs exist for supporting the diagnosis and management of patients presenting to primary care with shoulder disorders?” (lines 16-170) and removed Question 2. Instead, we have replaced Question 2 with the following: “A secondary objective will be to consolidate the existing evidence for evaluating the 1) effectiveness of improving patient outcomes; and 2) quality of CDSSs in terms of affordability, accessibility, ease of use, and generalizability across different demographics (e.g. sex, gender, age) and shoulder pathologies” (lines 140 – 143). Reviewer #2: Additional minor points: The first sentence of the second paragraph of the Background section reads, “Primary point-of-care providers themselves are restricted by the human body’s limited memory and knowledge base capacity…” This reference to cognitive limitations is well-placed but somewhat awkwardly framed. Author Response 2: Thank you for your insightful comment. We have revised the statement to read: “Primary care providers are restricted by the human body’s limited memory and knowledge base, especially with over 200 possible MSK diagnoses to choose from.” (lines 90-92) Reviewer #2: In the third paragraph of the Background section the authors state that “Knowledge-based systems are logical rules…” However, this is characterization is not completely accurate because while logical rules are a key component of knowledge-based systems, they include other components such as the user interface and the algorithms that process inputs, operate on the rules, and produce outputs. Author Response 1: Thank you for your comment. We have revised the statement to read “Knowledge-based systems can employ logical rules (IF-THEN statements) drawn from literature, patient-centred protocols, clinical practice guidelines, or expert knowledge used to generate an action or output based on the data entry point.[19] The knowledge-based system will then retrieve the data to evaluate the rule and produce and action or output.[19]” (lines 110-113) Reviewer #2: The first sentence of the Exclusion criteria section begins with “The following exclusion criteria are as follows,” which should be amended for wording, e.g., “The exclusion criteria are as follows:” Author Response 1: We agree and have revised as suggested. (line 190) Reviewer #2: The first sentence of the section “Stage 5: Collating, summarizing, and reporting the results” reads: “We will provide an overview and consolidate evidence from eligible articles in a numeric, tabular, and/or format.” The reference to “and/or format” is ambiguous in this context; this should be expanded upon or rephrased. Author Response 1: We agree and have revised to say: “We will provide an overview and consolidate evidence from eligible articles in a numeric, tabular, or chart format” (lines 259-260) We again thank the editor and reviewers for their comments, and we hope these revisions are satisfactory. Respectfully Yours, Dr. Breda Eubank Attachments Attachment Submitted filename: PONE-D-24-41186_Response to Reviewers BE AND Email.pdf https://doi.org/10.1371/journal.pone.0327192.r002
Decision Letter - Mansour Alshehri, Editor Identification of digital clinical decision support systems for supporting diagnosis and triage of patients with shoulder disorders: A scoping review protocol PONE-D-24-41186R1 Dear Dr. Breda, We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication. An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org. If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. Kind regards, Mansour Abdullah Alshehri Academic Editor PLOS ONE Additional Editor Comments (optional): After reviewing the authors' response, the addition of Dr. Richard Hayman as a co-author has been appropriately justified in the response to reviewers. There are no serious methodological concerns regarding his inclusion, and his involvement is consistent with the scope and aims of the scoping review. Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. Does the manuscript provide a valid rationale for the proposed study, with clearly identified and justified research questions? Reviewer #2: Yes ******** 2. Is the protocol technically sound and planned in a manner that will lead to a meaningful outcome and allow testing the stated hypotheses??> Reviewer #2: Yes ****** 3. Is the methodology feasible and described in sufficient detail to allow the work to be replicable??> Reviewer #2: Yes ****** 4. Have the authors described where all data underlying the findings will be made available when the study is complete??> The PLOS Data policy Reviewer #2: Yes ****** 5. Is the manuscript presented in an intelligible fashion and written in standard English??> Reviewer #2: Yes ****** Please use the space provided to explain your answers to the questions above and, if applicable, provide comments about issues authors must address before this protocol can be accepted for publication. You may also include additional comments for the author, including concerns about research or publication ethics. You may also provide optional suggestions and comments to authors that they might find helpful in planning their study. Reviewer #2: The authors have made significant improvements to the manuscript and have adequately addressed this reviewer’s comments. Inclusion criterion #2 currently reads “must be designed as either a knowledge-based system or non-knowledge-based system”. The authors may consider expanding this to allow for the inclusion of systems that include elements of both knowledge-based and non-knowledge-based systems. ****** what does this mean? ). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy Reviewer #2: Yes: Michael E. Bales ******** https://doi.org/10.1371/journal.pone.0327192.r003
Formally Accepted
Acceptance Letter - Mansour Alshehri, Editor PONE-D-24-41186R1 PLOS ONE Dear Dr. Eubank, I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team. At this stage, our production department will prepare your paper for publication. This includes ensuring the following: * All references, tables, and figures are properly cited * All relevant supporting information is included in the manuscript submission, * There are no issues that prevent the paper from being properly typeset You will receive further instructions from the production team, including instructions on how to review your proof when it is ready. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few days to review your paper and let you know the next and final steps. Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. If we can help with anything else, please email us at customercare@plos.org. Thank you for submitting your work to PLOS ONE and supporting open access. Kind regards, PLOS ONE Editorial Office Staff on behalf of Dr. Mansour Abdullah Alshehri Academic Editor PLOS ONE https://doi.org/10.1371/journal.pone.0327192.r004

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .