Methods used to assess outcome consistency in clinical studies: A literature-based evaluation

Ewelina Rogozińska; Elizabeth Gargon; Rocío Olmedo-Requena; Amani Asour; Natalie A. M. Cooper; Claire L. Vale; Janneke van’t Hooft

doi:10.1371/journal.pone.0235485

Peer Review History

Original SubmissionMarch 24, 2020
26 May 2020 Decision Letter - Tim Mathes, Editor PONE-D-20-08427 Evaluation of methods in studies of outcomes consistency in clinical studies PLOS ONE Dear Dr. Rogozińska, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. You can ignore the comment on highlighting statistical significance (reviewer #2). Please submit your revised manuscript by Jul 10 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript: A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols We look forward to receiving your revised manuscript. Kind regards, Tim Mathes Academic Editor PLOS ONE Journal requirements: When submitting your revision, we need you to address these additional requirements: 1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at http://www.plosone.org/attachments/PLOSOne_formatting_sample_main_body.pdf and http://www.plosone.org/attachments/PLOSOne_formatting_sample_title_authors_affiliations.pdf [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Yes Reviewer #2: Yes ******** 2. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: Yes Reviewer #2: Yes ****** 3. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: Yes Reviewer #2: Yes ****** 4. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes Reviewer #2: Yes ****** 5. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: The authors propose to evaluate the quality of the methods used in studies on outcomes consistencies. The topic is important and has not been investigated. However a few point could be improved Major comments: 1) The author declare that their study was not a systematic review and reported having searched in the “COMET Database”. It is not clear what this COMET database is and the reference 8 does not mention it clearly either (the weblink of reference 8 is broken : http://www.comet-initiative.org/studies/search) It is time consuming for the reader to find out how this COMET database is provided (medline and scopus mainly? ). ð In the method section, the authors should explain with a few sentences how the COMET database is fuelled. 2) The title of the article does not give any idea about the method they used. We propose to add more precision about the design of their study as a scoping review see the reference of the Joanna briggs institute (https://wiki.joannabriggs.org/display/MANUAL/Chapter+11%3A+Scoping+reviews) or of the PRISMA group (http://www.prisma-statement.org/Extensions/ScopingReviews) ð “Evaluation of methods in studies of outcomes consistency in clinical studies: a scoping review” 3) At the end of the discussion, the author recommend not to use the term “systematic review” when studies are not : “However, we discourage labelling for this type of evaluation, as “systematic review” instead, proposing alternative terms such as an “evaluation of outcomes” or “outcome mapping”.” ð We also suggest to use the term of “scoping review” in order to standardized the terminology according to PRISMA and the Joanna Briggs institute. Minor comments: Table 1 : we do not understand what “total number of participants per review” does mean. Who are these participants ? Is it the number of reviewers involved in each study or the number of patients included in the studies evaluated by the review? In case of the later, how is it relevant for the current study? Reviewer #2: The abstract is not fully understandable without reading the main text. You should state concisely which type of studies are included in your review (e.g. synthesis studies evaluating outcome consistency in clinical studies). Please do not use undefined abbreviations (AMSTAR). Please provide a brief description of AMSTAR in the main text. Table 1: please move the notes into the table since they include relevant information Table 2: please explain why you used Fisher's exact test (instead of chi-squared) and to which groups of studies you applied the test; please also highlight significant results (p<0.05). Table 3: it is not understandable why denominators differ across the rows (90 or 93). Tables (in general): when you use the word "studies", please verify whether it is easy to distinguish between studies included in your review and clinical studies included in the synthesis studies you examined Please check all tables and figure captions and see if they can be more informative of the underlying content. Please discuss any practical implications of your work (such as improvements in clinical research and practice). Please identify any health area that requires more attention in future research. ****** 6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review?** For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #1: No Reviewer #2: No [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step. https://doi.org/10.1371/journal.pone.0235485.r001
Revision 1
2 Jun 2020 Author Response Reviewer #1 1. The authors propose to evaluate the quality of the methods used in studies on outcomes consistencies. The topic is important and has not been investigated. However a few point could be improved We thank the reviewer for the recognition of the importance of this topic. 2. The author declare that their study was not a systematic review and reported having searched in the “COMET Database”. It is not clear what this COMET database is and the reference 8 does not mention it clearly either (the weblink of reference 8 is broken: http://www.comet-initiative.org/studies/search) It is time consuming for the reader to find out how this COMET database is provided (medline and scopus mainly?). In the method section, the authors should explain with a few sentences how the COMET database is fuelled. We agree with the reviewer that we had not adequately described the COMET database and have now addressed this shortcoming by adding the following description in the methods: “We searched the COMET database from its inception to June 2018; the search was updated in May 2019. (8) The COMET database is an annually updated repository of the international COS literature based on the systematic searches run in MEDLINE, SCOPUS, and Cochrane Methodology Register. (9) 3. The title of the article does not give any idea about the method they used. We propose to add more precision about the design of their study as a scoping review see the reference of the Joanna Briggs Institute (https://wiki.joannabriggs.org/display/MANUAL/Chapter+11%3A+Scoping+reviews) or of the PRISMA group (http://www.prisma-statement.org/Extensions/ScopingReviews) “Evaluation of methods in studies of outcomes consistency in clinical studies: a scoping review” We thank the reviewer for the suggestion. However, our work was not set out to be a scoping review; thus we feel that labelling its design this way would be inappropriate. According to our protocol (see the PROSPERO registration) the design was described as “exploratory study” and was developed in the spirit of meta-research; discipline aiming to evaluate and improve research practices (Ioannidis et al. 2015 PLoS Biol. 13(10): e1002264). We have now added the information about the study design in the title as follows: “Methods to assess outcome consistency in clinical research: literature-baseded evaluation” 4. At the end of the discussion, the author recommend not to use the term “systematic review” when studies are not: “However, we discourage labelling for this type of evaluation, as “systematic review” instead, proposing alternative terms such as an “evaluation of outcomes” or “outcome mapping”.” We also suggest to use the term of “scoping review” in order to standardized the terminology according to PRISMA and the Joanna Briggs institute. We agree that a scoping review may be another alternative and have amended the last sentence in our final paragraph as follows: “However, we discourage labelling for this type of evaluation, as “systematic review.” Instead, we propose terms such as an “evaluation of outcomes”, “outcome mapping.” Alternatively, authors ma choose to use an established approach, such as “scoping review”, and adopt the appropriate methodological and reporting standards (34, 35).” 5. Table 1: we do not understand what “total number of participants per review” does mean. Who are these participants? Is it the number of reviewers involved in each study or the number of patients included in the studies evaluated by the review? In case of the later, how is it relevant for the current study? On reflection we agree with the reviewer and have now removed this information from Table 1. Reviewer #2 1. The abstract is not fully understandable without reading the main text. You should state concisely which type of studies are included in your review (e.g. synthesis studies evaluating outcome consistency in clinical studies). Please do not use undefined abbreviations (AMSTAR). We have now amended the abstract and checked the manuscript for consistency in the used terminology referring to included studies as “evaluations of outcome consistency in clinical studies” AMSTAR abbreviation has been explained in the abstract as follows: “The methods were assessed using a subset of A MeaSurement Tool to Assess systematic Reviews (AMSTAR) 2 items (1-9) and additional outcome-specific questions.” 2. Please provide a brief description of AMSTAR in the main text. We have now added a following description in the methods section: “In order to examine adoption of systematic review methodology, all studies (regardless of declared study design) were assessed against a tailored subset of A MeaSurement Tool to Assess systematic Reviews (AMSTAR) 2 items (23) we felt were relevant to evaluated studies (S1 Table). AMSTAR was designed as a practical critical appraisal instrument enabling a rapid and reproducible quality assessments of systematic reviews.” 3. Table 1: please move the notes into the table since they include relevant information The notes have been now incorporated into the main body of Table 1. 4. Table 3: it is not understandable why denominators differ across the rows (90 or 93). The explanation of the difference is provided at the bottom of the table and has been rewritten for clarity: “Three reports (3/93) of full core outcome sets were excluded from some assessments due to none applicability (for details see S3 Table)”. We also moved asterisks to the column heading (n/N). 5. Tables (in general): when you use the word "studies", please verify whether it is easy to distinguish between studies included in your review and clinical studies included in the synthesis studies you examined We have now amended all tables to improve their clarity according to the reviewer’s suggestion. 6. Please check all tables and figure captions and see if they can be more informative of the underlying content. We have now checked all tables and figures to improve their clarity according to this suggestion. 7. Please discuss any practical implications of your work (such as improvements in clinical research and practice). The practical implications of our work are discussed in the final paragraph of our discussion, notably that authors of these studies need to be transparent in reporting their methods and that they should not label such evaluations as systematic reviews if they do not fully adhere to stringent systematic review methodologies. Although we do not feel that our work has any direct impact on clinical practice; we hope that it will improve research practice relating to core outcome set development and reporting of such research. 8. Please identify any health area that requires more attention in future research. As our work focuses on the methods rather than a specific clinical issue and is applicable across all health areas. We have recorded and reported the health areas in which evaluations of outcome consistency were carried out, however, we have not compared methodological practices between different health areas. Attachments Attachment Submitted filename: 20Jun01 Response.docx https://doi.org/10.1371/journal.pone.0235485.r002
5 Jun 2020 Decision Letter - Tim Mathes, Editor PONE-D-20-08427R1 Methods to assess outcome consistency in clinical research: a literature-based evaluation PLOS ONE Dear Dr. Rogozińska, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please submit your revised manuscript by Jul 20 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript: A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols We look forward to receiving your revised manuscript. Kind regards, Tim Mathes Academic Editor PLOS ONE [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation. Reviewer #1: All comments have been addressed Reviewer #2: (No Response) ******** 2. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Yes Reviewer #2: Yes ****** 3. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: Yes Reviewer #2: N/A ****** 4. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: Yes Reviewer #2: Yes ****** 5. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes Reviewer #2: Yes ****** 6. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: (No Response) Reviewer #2: Thanks for addressing my previous comments. I have few additional minor suggestions: 1) replace "evaluations of outcome consistency" with "evaluation studies of outcome consistency" 2) if possibile, include such denomination among the keywords 3) check Table 1 caption (it should be outcome "consistency") 4) Table 3 caption: also in this case I would specify that methods are those of the included studies 5) Table 3: the third category ("when the distinction was made..") is a sub-category of the second one, and the way of reporting should reflect this 6) Table 3: the note is still unclear (you should fully understand the table without looking at supplementary files) 7) S3 Table caption: it should be "detailed characteristics" instead of "detail characteristics" 8) S4 Table caption is unclear: do you mean the clinical studies, or the evaluation studies that you assessed? ****** 7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review?** For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #1: No Reviewer #2: No [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step. https://doi.org/10.1371/journal.pone.0235485.r003
Revision 2
14 Jun 2020 Author Response 1 Thanks for addressing my previous comments. I have few additional minor suggestions: We thank the reviewer for his further suggestions. We have reviewed all the points and amended the manuscript accordingly. 2 replace "evaluations of outcome consistency" with "evaluation studies of outcome consistency" if possibile, include such denomination among the keywords The term "evaluations of outcome consistency" has been changed to "evaluation studies of outcome consistency" and listed the term among the keywords. 3 check Table 1 caption (it should be outcome "consistency") The caption of Table 1 has been corrected to: “Characteristics of included evaluations of outcome consistency” 4 Table 3 caption: also in this case I would specify that methods are those of the included studies The caption of Table 3 has been changed to: “Methods used to identify and assess outcomes in evaluation studies of outcome consistency” 5 Table 3: the third category ("when the distinction was made..") is a sub-category of the second one, and the way of reporting should reflect this The category and subcategories have been amended as follows: When the distinction between the type of outcome(s) was made: • The primary outcome(s) was clearly specified as a primary • The primary outcome(s) was clearly specified as a primary or used in the power calculation • The primary outcome(s) was identified using other measures (e.g. outcome mentioned in the trial title, first reported) • There were no details of how the primary outcome(s) was identified 6 Table 3: the note is still unclear (you should fully understand the table without looking at supplementary files) The note has been amended as follows: “Three reports (3/93) of full core outcome sets were excluded from some assessments (N = 90) as they focused only on providing a list of identified outcomes without reporting more details (see S3 Table)” 7 S3 Table caption: it should be "detailed characteristics" instead of "detail characteristics" The caption of S3 Table has been changed to: “Detailed characteristics of included evaluation studies of outcome consistency” 8) S4 Table caption is unclear: do you mean the clinical studies, or the evaluation studies that you assessed? The caption of S4 Table has been changed to: “Details of identification of clinical study reports in the evaluation studies of outcome consistency” Attachments Attachment Submitted filename: 20Jun14 Response to reviewers.docx https://doi.org/10.1371/journal.pone.0235485.r004
17 Jun 2020 Decision Letter - Tim Mathes, Editor Methods used to assess outcome consistency in clinical studies: a literature-based evaluation PONE-D-20-08427R2 Dear Dr. Rogozińska, We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication. An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org. If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. Kind regards, Tim Mathes Academic Editor PLOS ONE Additional Editor Comments (optional): Reviewers' comments: https://doi.org/10.1371/journal.pone.0235485.r005
Formally Accepted
22 Jun 2020 Acceptance Letter - Tim Mathes, Editor PONE-D-20-08427R2 Methods used to assess outcome consistency in clinical studies: a literature-based evaluation Dear Dr. Rogozińska: I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department. If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org. If we can help with anything else, please email us at plosone@plos.org. Thank you for submitting your work to PLOS ONE and supporting open access. Kind regards, PLOS ONE Editorial Office Staff on behalf of Dr. Tim Mathes Academic Editor PLOS ONE https://doi.org/10.1371/journal.pone.0235485.r006

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .