PONE-D-24-24225Application of video surveillance in preclinical safety studies in Canines: understanding the interobserver reliability and validity to recognize clinical behavior.PLOS ONE

Dear Dr. Kopljar,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Several of the authors had significant reservations about the experimental design. It is reasonable to withdraw this manuscript and resubmit in the future if you need additional time to resolve these issues.PLOS Data policy also requires authors make all data underlying the findings described in their manuscript fully available without restriction. If you are not able to adhere to that policy due to privacy concerns, you should consider an alternative journal.

Please submit your revised manuscript by Feb 22 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org . When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols . Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols .

We look forward to receiving your revised manuscript.

Kind regards,

Cord M. Brundage, D.V.M., Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for stating the following in the Competing Interests section:

“AlI have read the journal's policy and the authors of this manuscript have the following competing interests: all the authors were employed by company Janssen Research and Development, Janssen Pharmaceutical Companies of Johnson & Johnson.”

Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, by including the following statement: "This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests).  If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include your updated Competing Interests statement in your cover letter; we will change the online submission form on your behalf.

3. We note that your Data Availability Statement is currently as follows: [All relevant data are within the manuscript and its Supporting Information files.]

Please confirm at this time whether or not your submission contains all raw data required to replicate the results of your study. Authors must share the “minimal data set” for their submission. PLOS defines the minimal data set to consist of the data required to replicate all study findings reported in the article, as well as related metadata and methods (https://journals.plos.org/plosone/s/data-availability#loc-minimal-data-set-definition).

For example, authors should submit the following data:

- The values behind the means, standard deviations and other measures reported;

- The values used to build graphs;

- The points extracted from images for analysis.

Authors do not need to submit their entire data set if only a portion of the data was used in the reported study.

If your submission does not contain these data, please either upload them as Supporting Information files or deposit them to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of recommended repositories, please see https://journals.plos.org/plosone/s/recommended-repositories.

If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially sensitive information, data are owned by a third-party organization, etc.) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent. If data are owned by a third party, please indicate how others may request data access.

4. We note that Figure 1 in your submission contain copyrighted images. All PLOS content is published under the Creative Commons Attribution License (CC BY 4.0), which means that the manuscript, images, and Supporting Information files will be freely available online, and any third party is permitted to access, download, copy, distribute, and use these materials in any way, even commercially, with proper attribution. For more information, see our copyright guidelines: http://journals.plos.org/plosone/s/licenses-and-copyright.

We require you to either (1) present written permission from the copyright holder to publish these figures specifically under the CC BY 4.0 license, or (2) remove the figures from your submission:

a. You may seek permission from the original copyright holder of Figure 1 to publish the content specifically under the CC BY 4.0 license.

We recommend that you contact the original copyright holder with the Content Permission Form (http://journals.plos.org/plosone/s/file?id=7c09/content-permission-form.pdf) and the following text:

“I request permission for the open-access journal PLOS ONE to publish XXX under the Creative Commons Attribution License (CCAL) CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). Please be aware that this license allows unrestricted use and distribution, even commercially, by third parties. Please reply and provide explicit written permission to publish XXX under a CC BY license and complete the attached form.”

Please upload the completed Content Permission Form or other proof of granted permissions as an "Other" file with your submission.

In the figure caption of the copyrighted figure, please include the following text: “Reprinted from [ref] under a CC BY license, with permission from [name of publisher], original copyright [original copyright year].”

b. If you are unable to obtain permission from the original copyright holder to publish these figures under the CC BY 4.0 license or if the copyright holder’s requirements are incompatible with the CC BY 4.0 license, please either i) remove the figure or ii) supply a replacement figure that complies with the CC BY 4.0 license. Please check copyright information on all replacement figures and update the figure caption with source information. If applicable, please specify in the figure caption text when a figure is similar but not identical to the original image and is therefore for illustrative purposes only.

Additional Editor Comments :

Please make sure you address reviewer concerns both in your response to reviewers and in the manuscript or indicate why it would inappropriate to do so.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: No

Reviewer #3: No

Reviewer #4: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: No

Reviewer #3: Yes

Reviewer #4: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

Reviewer #3: No

Reviewer #4: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This manuscript looks at the reliability and validity of using video recordings to assess clinical observations, taking into account observer experience level and role. Overall I think the manuscript is well written and clearly describes the methods and results, while identifying the limitations of video observations, but providing thoughtful suggestions on using such measures to train machine learning technologies for automated detection of behavior. I have a few minor comments/questions below.

Materials and methods:

L127: How were the video snippets selected and by whom?

L130: What types of studies? Were the animals always in the same housing set up?

L136: How long were animals separated after feeding?

L140: How were the videos blinded and by whom?

L152-153: Why were twitches only an optional observation?

L184: Were the videos randomized in the same order for all observers or randomized per observer? How were videos randomized?

Results:

L273: Error! Reference source not found… - This error is found in numerous places throughout the manuscript.

Reviewer #2: The experimental design of the study has some important to note flaws due to the presence of only 2 experts for generating the ground truth for comparison with the observers’ ratings of the videos. Since the ground truth only includes the observations that both experts share, this is essentially sub-selecting only the easier to interpret aspects of the clinical observations and eliminating the role of less easily observed aspects of the animal behavior. This leads the conclusions of the study to be flawed as well since the authors are maintaining that non-experts can interpret as well as experts. However, the authors are only evaluating their correctness based on easier to interpret observations (only those that are shared by 2 experts).

To address these issues, I would recommend major revision of the study design by generating the ground truth based on the observations of more than 2 experts. The ground truth should be observations that show at least a majority prevalence in a sample of exerts to be reflective of the population of exerts instead of only having the study design catered to the observations that overlap between only 2 observers. This automatically eliminates from consideration of the study more difficult to observe behaviors. This leads to the author’s conclusions that the video observations are as accurate as the expert observes overall. If the video observers were assessing behaviors that were more difficult for experts to observe, then it is likely that the accuracy of the video observers would be reduced overall when they are assessing more nuanced behaviors. The current study design introduces potential error if one of the experts makes an error, which could be resolved if more than 2 experts are used to establish the ground truth. The authors should include in their study design justification a power analysis that indicates the rationale for choosing the number of experts given their ability to establish a reliable conclusion. The experts should have a greater inter-expert reliability index than non-experts if they are truly an expert in the subject matter. The study did not conduct this type of analysis for more than 2 subjects to determine what number of experts would contribute to a sufficient baseline.

For example, the authors should reference studies in a range of fields including animal behavior and ethology (Anderson et al 2014; Bohnslav et al 2021) and other fields such as medical image annotation (Athanasiou et al 2023) and radiology (Lebovitz et al 2021), for standards on the number of experts that should be used for establishing the ground truth as these studies use more experts for their study design, ranging from 3 to 23. Since studies in animal ethology have noted that 2 human expert observers only agree around 70%, it would be necessary to obtain a dataset of multiple observers (at least 3-5 experts) so that an observation can be determined to be present in most expert assessments. These standards for study design are especially important given that the authors are making conclusions regarding the relevance of their findings for the area of computational animal behavior, which would require a larger number of experts as a standard for training models than they used in this study.

The authors state that their data is not fully available due to restrictions. This does not meet the PLOS One journal requirement for data availability in the current state. The UNESCO Recommendations on Open Science state that restrictions “are only justifiable on the basis of the protection of human rights, national security, confidentiality, the right to privacy and respect for human subjects of study, legal process and public order, the protection of intellectual property rights, personal information, sacred and secret indigenous knowledge, and rare, threatened or endangered species” and that “In cases where data cannot be openly accessible, it is important to develop tools and protocols for pseudonymizing and anonymizing data, as well as systems for mediated access, so that as much data as possible can be shared as appropriate.”. The authors should address these points regarding restriction of their data, and if they are not applicable, then they should provide access to their data in a public repository such as Open Science Framework.

Overall, if these major concerns are addressed, this would lead to a scientifically rigorous approach as the study goals are worthwhile to examine. As it stands, the issue with using only 2 experts for the study leads to results that are obscured by the elimination of too much data from consideration to be reflective of whether video observers can be as reliable and as valid as experts. The author’s study purpose for evaluating the effectiveness of video analysis is important to extend the ability to acquire data from animal studies, so a more rigorous approach would be important to achieve results that support these goals.

Reviewer #3: The current manuscript presents a study assessing the validity and reliability of measures used to evaluate behavioural responses in lab dogs.

The manuscript is well written and clear. However, what authors are describing in this manuscript as original research, is a common part of the standard early steps for measuring behaviour (e.g., ethogram building, observer training and reliability assessment; see Bateson and Martin 2021), with the add on of testing whether level of experience (subjectively and self assessed) has an effect on reliability.

Indeed, the validity and reliability of behavioural measurements in safety assessment studies are quite relevant and needed. Behavioural indicators of clinical concerns in laboratory animals are often missed or unreliably scored; all of which can decrease translatability, the quality of data and compromise the welfare of animals. However, I fail to see how results from the current manuscript contribute to address this issue:

1. The study lacks an independent assessment of the validity of the measures included – video scoring performed by two experienced observers within the same department, does not validate a behavioural/clinical measure. The comparison between experts scoring of videos and the rest of observers is still an indicator of consistency and measures can be reliable/consistent but have low validity – simply put, to which degree a measure is reflective of what it is intended to measure (the phenomenon of interest; e.g., the animal’s clinical or emotional state or the adverse effects of a compound; see Nunnally 1978, for a reading on face and construct validity).

2. Definitions of selected clinical observations confound behaviours with the inferences that can be made regarding the animals’ internal emotional and clinical states, and in several cases lack a clear description of objectively measurable behaviours. I understand that these definitions follow an internal glossary, thus are practical and applicable for their research lab, but reduces the relevance of its findings outside their group.

4. Videos were selected and cut for the purpose of this study. I question whether these videos are representative of the variation (in quality and behavioural diversity) seen in real post dosing cage side observation videos; hence, reliability assessed in these videos may not be translatable to actual video scoring for assessing drug adverse effects.

5. It does not address the issue of how video recordings can be used instead of direct observation.

Minor: several references marked as: Error! Reference source not found..

Reviewer #4: 1. Rewrite lines 91-92 for better understanding.

2. Rewrite lines 107-110 for clear understanding.

3. Could you please explain why there is a huge difference in the time range of the video snippets in the lines 127-128.

4. How many dogs were observed in each video please mention in the line 140.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean? ). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy .

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

Reviewer #4: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/ . PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org . Please note that Supporting Information files do not need this step.

PONE-D-24-24225R1Application of video surveillance in preclinical safety studies in Canines: understanding the interobserver reliability and validity to recognize clinical behavior.PLOS ONE

Dear Dr. Kopljar,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Apr 29 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org . When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols . Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols .

We look forward to receiving your revised manuscript.

Kind regards,

Cord M. Brundage, D.V.M., Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments :

Please note that PlosOne requires that your article is free of all copyediting and typing errors before acceptance.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: (No Response)

Reviewer #3: (No Response)

Reviewer #4: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Partly

Reviewer #4: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors have addressed all my previous comments. I have no further comments to provide.

Reviewer #2: The authors have addressed my primary concerns of the number of experts described for establishing the necessary and optional observations for COs as well as the availability of raw data in the manuscript.

There are some remaining minor editorial changes that should be made, including:

Line 161: “When all 2 experts agreed…” should read “When all 3 experts agreed…”

Authors should make sure that they are consistent between the use of numerals and spelled out number notation ie. Three vs 3 and two vs 2 should remain consistent throughout the manuscript. This is an issue throughout the sections of the manuscript, so I will not note each case of inconsistency – the authors should thoroughly review the manuscript to maintain consistent notation. Typically, numbers are spelled out to represent numbers one through nine.

Reviewer #3: Authors have not yet satisfactorily addressed my first issue - which is non-trivial as it represents part of the core of the manuscript. In fact, this was also brought up by reviewer 1 and your response also did not address his comment. Having 3 instead of 2 experts does not “fix” the issue, their outcome is still a reliability measure rather than an evaluation of validity. Please provide a thorough explanation of the type of validity evaluated.

Reviewer #4: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean? ). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy .

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

Reviewer #4: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/ . PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org . Please note that Supporting Information files do not need this step.

Application of video surveillance in preclinical safety studies in Canines: understanding the interobserver reliability and validity to recognize clinical behavior.

PONE-D-24-24225R2

Dear Dr. Kopljar,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager®  and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Cord M. Brundage, D.V.M., Ph.D.

Academic Editor

PLOS ONE

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #2: The authors have addressed my primary concerns of the number of experts described for establishing the necessary and optional observations for COs as well as the availability of raw data in the manuscript.

The authors have addressed my editorial recommendations.

The authors have addressed Reviewer #3’s question of the types of validity that were assessed by the study aims.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean? ). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy .

Reviewer #2: No

**********