PONE-D-21-31227Comparison of ECG signal quality, QRS detection performance, and device usability based on simultaneous recordingsPLOS ONE

Dear Dr. Blaesing,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Jan 31 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Thomas Martin Deserno, Ph.D.

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf  and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for stating the following in the Acknowledgments Section of your manuscript:

“The authors acknowledge the financial support by the Federal Ministry of Education and Research of Germany in the project Montexas4.0 (FKZ 02L15A261), travel grants from the German Centre for Cardiovascular Research (DZHK, partner site Greifswald), and the support for the Article Processing Charge from the DFG (German Research September 28, 2021 Foundation, 393148499) and the Open Access Publication Fund of the University of Greifswald.”

We note that you have provided information within the Acknowledgements Section. Please note that funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form.

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows:

“DB acknowledges the financial support by the Federal Ministry of Education and Research of Germany in the project Montexas4.0 (FKZ 02L15A261 https://www.bmbf.de/bmbf/en/home/home_node.html), MV acknowledges travel grants from the German Centre for Cardiovascular Research (DZHK, partner site Greifswald https://dzhk.de/en/), all authors acknowledge the support for the Article Processing Charge from the DFG (German Research Foundation, 393148499 https://www.dfg.de/en/index.jsp) and the Open Access Publication Fund of the University of Greifswald.

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.”

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: N/A

Reviewer #2: Yes

Reviewer #3: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors present results of a study comparing signal quality of ECG recordings acquired using five commercially-available devices of which three are medical grade. Data was acquired in parallel as the devices employ different sensors technologies (e.g. conventional skin electrodes, textile sensors, chest strap) while the subjects (N=13) were performing different cognitive and physical activities. The dataset was already published within PhysioNet and preprocessing is described in a CinC 2019 contribution.

The topic is of high interest due to the increasing availability of (non-)medical grade ECG hardware - not only for interested individuals but also researchers - as it could enable ECG acquisition in scenarios that were not possible before. Evidently, the signal quality of these devices in different scenarios is a crucial point. The evaluation is based on a novel signal quality index that is proposed in this work. In conclusion, this work addresses an important topic and fits to PLOS ONE due to its multidisciplinary nature. The language of the paper is fine and the figures provided are of high-quality.

However, I have concerns regarding

i) the fact that this work introduces a novel signal quality measures without evaluating it or comparing it to other algorithms;

ii) the cited references as important up-to-date literature is not considered;

iii) the document content and structure being sometimes chaotic and not following IMRAD.

Moreover, iv) at multiple points in the work the wording is not uniform, the title requires changes and v) the level of detail is too superficial at some points.

However, I am optimistic that all issues can be addressed in an extensive revision, leading to acceptance.

i) The authors propose a new heuristic algorithm for ECG signal quality estimation. I am completely fine with that but this requires to evaluate its performance w.r.t. state-of-the-art methods in this field and/or test/synthetic signals with a pre-defined signal quality. Unfortunately, both are missing which prevents to assess the quality of the proposed method in any way.

I suggest the authors to either i) apply a state-of-the-art method to their data and compare the results to the results achieved with the proposed method or ii) apply their method to synthetic signals with manually added noise (e.g. baseline wander, muscle twitches) at pre-defined signal-to-noise levels (as done in [23]). Ideally, both experiments are conducted. This would lead the reader to appreciate the many open parameters (e.g. k=8, w=[-2,2], l_qrs=100ms, morphSQ < 10%) because at the moment the reader cannot assess how robust the proposed algorithm and results are.

Moreover, the reader cannot get "a feeling" for the proposed morphSQ values. Fig. 3 shows four ideal ECG signals and their corresponding morphSQ values but what about the special cases? What influence does baseline wander, motion artifacts etc. have on the accuracy of morphSQ?

ii) I strongly miss a "Related work" chapter in this work giving an overview of the state-of-the-art in ECG signal quality estimation. In the Introduction only two references are given which are from 2012 and 2007, respectively. I am absolutely not an expert in that field but I know some recent works that definitively have to be considered (e.g. https://doi.org/10.1109/tbme.2020.2969719 , https://doi.org/10.1109/rbme.2018.2810957) just by reading the relevant journals in that field.

iii) The document structure is chaotic at various points, especially in the first chapter:

ll. 1-14: Introduction and first mention of HR and HRV

ll. 15-23: "The aim of this study is to compare usability and data quality ..."

ll. 24-41: Fundamentals of ECG (clinical vs. Holter systems, PQRST etc)

ll. 42-50: The role of HR and HRV w.r.t. different user groups

ll. 51-60: Information on the devices used

ll. 60-68: "In this study, the ECG device comparison is based on..."

ll. 69-73: Information why a high signal quality is important in research

ll. 74-80: Short summary of ECG signal quality indices

ll. 81-85: "In this paper, a new signal quality index was developed"

As I try to underline, it is very hard for the reader to follow. I highly suggest to split the introduction into i) ECG fundamentals, ii) related work, iii) the aims of this work (or research questions), iv) and which methods will be used to reach these aims (answer the research questions).

At some other instances of the document, the authors do not follow IMRAD. For example:

- ll. 137-143: This should be raised in the introduction or discussed at the end of the work as a limitation.

- ll. 256-284: This chapter is hard to read as it mixes description of the algorithm and evaluation. I suggest to add a new subchapter "Evaluation methodology" (or similar).

- Conclusion: All the information on usability should be part of the main body of the text (method & results). The information how long a device needs to be equipped or how the authors assess the experience needed for sensor placement is not a "Conclusion" but a result.

iv) At multiple instances the wording is not uniform. Some examples are (this list is not complete):

- l. 22: "RR intervals" vs. l. 55: "RR-intervals"

- l. 54: "PPG" (not introduced)

- l. 10: "Heart Rate (HR)" vs. l. 49: "heart rate" (abbreviation not used)

- l. 129: "EDF or EDF+" (written in italics; also not introduced) vs. Table 2: "EDF" (no italics)

- l. 210: "PQT analysis" (what is that?) vs. l. 42: "PQRST"

- l. 259: "FP - false positive" vs. l. 312: "false positive (FP)"

Furthermore, the title does not contain the key information that in this work i) multiple, wearable ECG devices are compared and that ii) a novel method for signal quality estimation is proposed.

v) The description of the "Smith-Waterman-like algorithm" is too superficial. I cannot understand it from the 6 lines of text and the figure provided.

Minor issues:

- Please provide information how many annotators at what level of expertise provided beat_ref. Moreover, you report that arrhythmia periods were removed. Please report how this decision was made and how much data was removed.

- Table 1: That is a strange Euro symbol? Additionally, I think it would be good to report on the bit depth of the ECG signals.

- Table 2: Please define all abbreviations. What is EDF/SDC/ASC...?

- ll. 154-177: How long are P3 and P4?

- l. 202: I suggest to not use the wording "Statistical analysis" as no statistical testing is performed but only mean and SD values are reported.

- Abstract: "The results allow conclusions to be drawn..." -> I suggest to actually name the conclusions.

Reviewer #2: Summary

The authors compare five different ECG devices to evaluate these devices regarding QRS detection and usability. The research question is relevant. It is also important that the authors make the recorded data publicly available. However, it exists already many papers which compare different ECG devices. Before the publication, the method should be described sufficiently detailed, as some of the reasons for the selected approach are lacking. In addition, the limitations of the method should be discussed more intensively in the discussion.

Major Issues

• Introduction

o 1-4: The first reference should support this statement “[…] ongoing trend for self-quantification of personal health and fitness lead to […]” and is from 2012.

o 7-8: “A rising number of those devices are […]” here you need to add more than one reference.

o 23-41: It is more important to address wearable devices instead of the Holter ECG systems.

o 51-60: The selection of the different ECG devices belongs to the chapter “Material and Methods” and in the chapter “Introduction”.

• Material and Methods

o The authors need to address the time synchronization of the different devices. They need to explain how they can ensure that the recorded signals are the signals for the same time interval.

o Subjects: Please add the inclusion and exclusion criteria for the subjects.

o Statistical analysis: Please add the explanation why you chose the algorithm eplimited and did not choose another state-of-the-art algorithm (e.g., template matching or deep learning approach).

o 278: It is important to use the same algorithms to make the results comparable.

• Conclusion

o 401 „Fig 6. Authors' ordinal ranking of used recording devices.“: This ranking depends on the author's opinion and is not scientific. The authors need to exclude this figure.

Minor Issues

• Figures and tables

o Table 1: You should replace the euro signs with numbers.

o Figure 5: It is important to differentiate between the different use cases and to explain the results.

Reviewer #3: The authors present a study in which they compare several ECG devices for ambulatory monitoring in terms of their signal quality. The paper is very well written and the topic is of interest to the community. There are some points I would like to see addressed before publication.

- The authors introduce a novel signal quality index. I think it would make a lot of sense to either evaluate this signal quality index on existing benchmark problems first or to evaluate their data using also other signal quality indices and see whether or not there are any differences, if their new metric correlates with other metrics, etc. (as a side note, I find it somewhat unintuitive that the signal quality index has to assume low values for the signal to be of good quality. In Figure 3, the authors even write “quantify similarity”, so why not something like “1 – SQI”? Then, the authors also don’t need to talk about “Noise to Signal” in their manuscript but rather SNR...)

- The SQI cannot be applied to the Polar-device for obvious reasons. However, this makes the analysis somewhat inconsistent. Thus, I believe the authors should make a much stronger point on why to include this device (other than because it was there) or think about leaving it out.

- Along the same lines, the authors make an argument as to why they are not using a 12-lead ECG as reference (which is plausible) while, at the same time, argue that their SQI might be useful in the context of “further PQT analysis”. This seems like an ambiguity to me.

- I find the presentation of results in the large tables somewhat confusing. The authors could think about, for example, highlighting the best values, etc. Also, is there a way to perform some sort of statistical test to see whether or not the results of the different devices and test scenarios are significantly different?

- The conclusion section is rather long and also introduces a new figure, I would consider rearranging the content between discussion and conclusion.

- The paper could benefit from another round of proof reading from a native speaker ”differences in usability and practicability between all devices ARE characterized”, “As it was the goal to induce a mental state” (?). Also, it seems to me that punctuation could be improved.

- “During a first check of the recorded data from all five devices, several problems occurred that needed to be solved.” I am not sure this is a good sentence to start the paragraph as this should be a result-oriented (and not a progress-oriented) report.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PONE-D-21-31227R1ECG performance in simultaneous recordings of five wearable devices using a new morphological Noise-to-Signal index and Smith-Waterman-based RR interval comparisonsPLOS ONE

Dear Dr. Blaesing,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Sep 30 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Thomas Martin Deserno, Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Additional Editor Comments (if provided):

I suggest to consider the paper metioned by reviewer 2

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: (No Response)

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: (No Response)

Reviewer #3: N/A

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: (No Response)

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: (No Response)

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Dear authors, thank you very much for the thorough revision of your manuscript. I congratulate you to this effort and suggest to highlight this work on the PLOS ONE website.

Reviewer #3: The authors addressed all my comments (and I believe those of the other reviewers as well) sufficiently and have improved the manuscript significantly in the process. I now find the manuscript acceptable for publication.

Looking at the not super-recent literature, this paper [https://doi.org/10.3390/s18010038] introduces a “shape-based signal-to-noise ratio SNR_S” for cardiorespiratory signals that, to me, seems to have some obvious connections to the “morphological signal quality index” presented here. Thus, the authors might want to think about discussing it in the related work section.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Nicolai Spicher

Reviewer #3: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

ECG performance in simultaneous recordings of five wearable devices using a new morphological Noise-to-Signal index and Smith-Waterman-based RR interval comparisons

PONE-D-21-31227R2

Dear Dr. Blaesing,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Thomas Martin Deserno, Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments: