Predicting vasovagal reactions to needles from video data using 2D-CNN with GRU and LSTM

Judita Rudokaite; Sharon Ong; Itir Onal Ertugrul; Mart P. Janssen; Elisabeth Huis in ‘t Veld

doi:10.1371/journal.pone.0314038

Peer Review History

Original SubmissionDecember 5, 2023
19 Feb 2024 Decision Letter - Tirtharaj Dash, Editor PONE-D-23-40599Predicting vasovagal reactions to needles from video data using 2D-CNN with GRU and LSTM.PLOS ONE Dear Dr. Rudokaite, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. ============================== This paper proposes an ML-based pipeline to study vasovagal reactions to medical procedures involving needles. According to the authors, this is a first-of-a-kind paper in this field, and, in addition, the authors are working on getting a new dataset relevant to this study. The idea is to record short videos of patients and provide them with a questionnaire to note their reactions. The authors then annotate each video (VVR reaction) based on these responses. They use two well-known pre-trained networks to extract the video features and use them further on sequence models for the reaction classification task. This is an important research work in medical science using AI, and there are some interesting questions and queries from the learned reviewers. The reviewers raised some strong concerns, both in the context of medical science and deep network-based modelling, and these questions must be addressed in a rebuttal. Furthermore, I read this paper quite thoroughly. Here are my comments; I hope the authors find them helpful in improving their writeup and research. First, I enjoyed reading the manuscript; it is short and easy to follow and comprehend; the literature review is well done, and the results and downstream analysis sound reasonable. I do have some comments on that; see below. The paper is rather organised in a highly complicated way. It is very difficult to know what are various subsections are under each section. This makes reading a bit difficult for a reader. For a journal article, this needs to be done in a manner that reads like a good scientific story. The introduction seems all over the play, although connected to some extent. It can be improved with different subsections in place. I am not quite sure whether ethics approval should come directly within the methods section or should be moved to the section right after the conclusion. Please follow the journal guidelines while preparing the manuscript. It is also better to provide a dataset summary table that can reduce a lot of text or the texts can be moved to the table's caption. I would like to see a panel figure that visually describes each stage of the data collection methodology or the entire pipeline. For datasets of this kind, it is very important that the samples are distributed very rationally into train/validation/test sets. Sometimes it becomes a matter of cherry-picking to show a better performance. To have a better estimate of the performance, it is preferred, in such small datasets, to perform a k-fold cross-validation (k=5 is usually good) and provide an average estimate of the performance. This doesn't overestimate nor underestimate the model. Furthermore, since this is a highly skewed dataset, such an experiment is warranted. The authors might want to do that. Please also ensure that there is no data leakage from the training to the testing set while doing the modelling. Since this is the first study in this area, it is better to show a good modelling strategy that the research community can follow. As a paper focusing on medical domain applications, one would like to see substantial discussion on those and how previous AI or non-AI-based studies have done those. Minor and typos: RMSEprop --> RMSProp Proofreading is needed to improve the presentation. ============================== Please submit your revised manuscript by Apr 04 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript: A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols. We look forward to receiving your revised manuscript. Kind regards, Tirtharaj Dash Academic Editor PLOS ONE Journal Requirements: 1. When submitting your revision, we need you to address these additional requirements. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf 2. Thank you for stating the following financial disclosure: This research was funded by the ZonMW Veni project “FAINT” (project reference: 016.186.020) and Stichting Sanquin Bloedvoorziening (grant PPOC19-12/L2409). Please state what role the funders took in the study. If the funders had no role, please state: "The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript." If this statement is not correct you must amend it as needed. Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf. 3. Thank you for stating the following in the Competing Interests section: Authors Dr. E.M.J. Huis in ‘t Veld and J. Rudokaite are founders of AINAR B.V. The other authors declare no competing interests. We note that one or more of the authors are employed by a commercial company: AINAR B.V. a. Please provide an amended Funding Statement declaring this commercial affiliation, as well as a statement regarding the Role of Funders in your study. If the funding organization did not play a role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript and only provided financial support in the form of authors' salaries and/or research materials, please review your statements relating to the author contributions, and ensure you have specifically and accurately indicated the role(s) that these authors had in your study. You can update author roles in the Author Contributions section of the online submission form. Please also include the following statement within your amended Funding Statement. “The funder provided support in the form of salaries for authors, but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the ‘author contributions’ section.” If your commercial affiliation did play a role in your study, please state and explain this role within your updated Funding Statement. b. Please also provide an updated Competing Interests Statement declaring this commercial affiliation along with any other relevant declarations relating to employment, consultancy, patents, products in development, or marketed products, etc. Within your Competing Interests Statement, please confirm that this commercial affiliation does not alter your adherence to all PLOS ONE policies on sharing data and materials by including the following statement: "This does not alter our adherence to PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests) . If this adherence statement is not accurate and there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared. Please include both an updated Funding Statement and Competing Interests Statement in your cover letter. We will change the online submission form on your behalf." 4. In the online submission form, you indicated that the dataset collected during the current study is not publicly available due to participants’ privacy, but an anonymized preprocessed dataset can be requested when the FAINT study is completed, by contacting the data manager of the Donor Medicine Research department at Sanquin. For contact details, contact the corresponding author or dr. Elisabeth Huis in ‘t Veld. All PLOS journals now require all data underlying the findings described in their manuscript to be freely available to other researchers, either a. In a public repository, b. Within the manuscript itself, or c. Uploaded as supplementary information. This policy applies to all data except where public deposition would breach compliance with the protocol approved by your research ethics board. If your data cannot be made publicly available for ethical or legal reasons (e.g., public availability would compromise patient privacy), please explain your reasons on resubmission and your exemption request will be escalated for approval. 5. We note that Figures 2, 3 and 7 includes an image of a participant in the study. As per the PLOS ONE policy (http://journals.plos.org/plosone/s/submission-guidelines#loc-human-subjects-research) on papers that include identifying, or potentially identifying, information, the individual(s) or parent(s)/guardian(s) must be informed of the terms of the PLOS open-access (CC-BY) license and provide specific permission for publication of these details under the terms of this license. Please download the Consent Form for Publication in a PLOS Journal (http://journals.plos.org/plosone/s/file?id=8ce6/plos-consent-form-english.pdf). The signed consent form should not be submitted with the manuscript, but should be securely filed in the individual's case notes. Please amend the methods section and ethics statement of the manuscript to explicitly state that the patient/participant has provided consent for publication: “The individual in this manuscript has given written informed consent (as outlined in PLOS consent form) to publish these case details”. If you are unable to obtain consent from the subject of the photograph, you will need to remove the figure and any other textual identifying information or case descriptions for this individual. Additional Editor Comments: This paper proposes an ML-based pipeline to study vasovagal reactions to medical procedures involving needles. According to the authors, this is a first-of-a-kind paper in this field, and, in addition, the authors are working on getting a new dataset relevant to this study. The idea is to record short videos of patients and provide them with a questionnaire to note their reactions. The authors then annotate each video (VVR reaction) based on these responses. They use two well-known pre-trained networks to extract the video features and use them further on sequence models for the reaction classification task. This is an important research work in medical science using AI, and there are some interesting questions and queries from the learned reviewers. The reviewers raised some strong concerns, both in the context of medical science and deep network-based modelling, and these questions must be addressed in a rebuttal. Furthermore, I read this paper quite thoroughly. Here are my comments; I hope the authors find them helpful in improving their writeup and research. First, I enjoyed reading the manuscript; it is short and easy to follow and comprehend; the literature review is well done, and the results and downstream analysis sound reasonable. I do have some comments on that; see below. The paper is rather organised in a highly complicated way. It is very difficult to know what are various subsections are under each section. This makes reading a bit difficult for a reader. For a journal article, this needs to be done in a manner that reads like a good scientific story. The introduction seems all over the play, although connected to some extent. It can be improved with different subsections in place. I am not quite sure whether ethics approval should come directly within the methods section or should be moved to the section right after the conclusion. Please follow the journal guidelines while preparing the manuscript. It is also better to provide a dataset summary table that can reduce a lot of text or the texts can be moved to the table's caption. I would like to see a panel figure that visually describes each stage of the data collection methodology or the entire pipeline. For datasets of this kind, it is very important that the samples are distributed very rationally into train/validation/test sets. Sometimes it becomes a matter of cherry-picking to show a better performance. To have a better estimate of the performance, it is preferred, in such small datasets, to perform a k-fold cross-validation (k=5 is usually good) and provide an average estimate of the performance. This doesn't overestimate nor underestimate the model. Furthermore, since this is a highly skewed dataset, such an experiment is warranted. The authors might want to do that. Please also ensure that there is no data leakage from the training to the testing set while doing the modelling. Since this is the first study in this area, it is better to show a good modelling strategy that the research community can follow. As a paper focusing on medical domain applications, one would like to see substantial discussion on those and how previous AI or non-AI-based studies have done those. Minor and typos: * RMSEprop --> RMSProp * Proofreading is needed to improve the presentation. [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Partly Reviewer #2: Partly Reviewer #3: Partly ******** 2. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: Yes Reviewer #2: No Reviewer #3: Yes ****** 3. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: Yes Reviewer #2: No Reviewer #3: Yes ****** 4. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: No Reviewer #2: No Reviewer #3: Yes ****** 5. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: 1. How does the study align with existing theories on the interplay between psychological states, physiological responses, and the predictability of vasovagal reactions during needle-related procedures? 2.Considering the prevalence of severe vasovagal reactions, how does the study address the challenge of training machine learning models with limited data? 3.How does the proposed serious game intervention aim to utilize facial information obtained through video recordings, and what advantages does visible light imaging offer over infrared thermal imaging? 4.What informed the choice of pre-trained models such as Xception and ResNet152 for feature extraction, and how do they contribute to the classification and regression tasks in predicting VVR levels? 5.Can you elaborate on the specific physiological changes, such as increased heart rate and changes in facial pallor, associated with vasovagal reactions? Reviewer #2: In this paper, the authors have used 2D-CNN with LSTM and GRU to predict continuous VVR scores and to classify discrete (low and high) VVR values obtained during the blood donation. Although the study has practical significance, the result discussion section and the English could be further improved. This paper may be considered if the following comments are carefully addressed. 1. The text is written a bit carelessly. Proofread it. 2. The employed LSTM and GRU model parameters should be provided. 3. Tune the hyperparameters of used employed LSTM and GRU models for improved performance. 4. Although stochastic, 2D-CNN, LSTM and GRU models are employed in this paper, no statistical analysis on the obtained results are carried out. 5. It is suggested to repeat the simulations and apply non-parametric statistical tests on the obtained results to draw reliable conclusions. 6. The figure quality of figure 1,2, 5 need improvement. Reviewer #3: In the study, Rudokaite et al have investigated the prediction of vasovagal reactions (VVR) during needle-related procedures by analyzing facial information from video sequences obtained from 287 blood donors. Further, utilizing 2D-CNN with LSTM and GRU, authors achieved a notable F1 score of 0.74 for high VVR classification using a pre-trained ResNet152 model with LSTM on 25 frames, and demonstrated the feasibility of predicting VVR responses during blood donation, suggesting potential applications for preventive interventions. Below are few comments for consideration 1- The authors have provided an elaborate introduction. It could be made more concise and precise to offer sufficient context for the study. 2- Authors should also discuss the age range of the donors and if at all there was any difference. 3- Authors should elaborate on the high VVR class, and what implications do the model's performance metrics have for identifying individuals at risk of experiencing VVR symptoms? 4- A discussion on the practical implications of the study's findings is needed . How could the identified predictive features in the nose and eye regions be practically applied? What are the potential real-world applications and benefits of using this method for predicting VVR responses? 5-A brief mention of potential future work or directions for further research. Are there specific areas or aspects that could be explored in subsequent studies? ****** 6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #1: Yes: suresh a Reviewer #2: Yes: Sibarama Panigrahi Reviewer #3: No ******** [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step. https://doi.org/10.1371/journal.pone.0314038.r001
Revision 1
2 Aug 2024 Author Response Dear editor and reviewers, We thank both reviewers and the editor for their careful consideration of the manuscript, and for their knowledgeable and constructive comments. We’ve addressed all your suggestions for improvements. You can find a summary of our improvements in relationship to each comment below (marked in red). Editor comments. 1. The paper is rather organised in a highly complicated way. It is very difficult to know what are various subsections are under each section. This makes reading a bit difficult for a reader. For a journal article, this needs to be done in a manner that reads like a good scientific story. The introduction seems all over the play, although connected to some extent. It can be improved with different subsections in place. We have adjusted the names of the subsections and adjusted the introduction to improve the readability. 2. I am not quite sure whether ethics approval should come directly within the methods section or should be moved to the section right after the conclusion. Please follow the journal guidelines while preparing the manuscript. Thank you for raising this question. We verified that the ethics approval should be included in the methods section as the journal guidelines state that “methods sections of papers on research using human subjects or samples must include ethics statements” (https://journals.plos.org/plosone/s/human-subjects-research). However, in the revised manuscript we include the ethics statement after the description of the participants. 3. It is also better to provide a dataset summary table that can reduce a lot of text or the texts can be moved to the table's caption. I would like to see a panel figure that visually describes each stage of the data collection methodology or the entire pipeline. We have provided the entire pipeline from data collection stage to data analysis (see Figure 1). Fig. 1. An overview of the data collection and used methodology for data analysis. Stage 1: Data collection shows an overview of the blood donation testing procedure and stages. At each of the seven stages, donors reported their VVR levels, and a video recording was made. At each stage, the video recording lasted 1 to 2 minutes. From stage 4 until stage 6, the recording was continuous, lasting between 5 and 27 minutes. The procedure is slightly different per BCC. At two locations, donors were brought directly to the donation chair after the physician check and hence no recording at stage 3 was made. If donors had to return to the waiting area after the physician check, they would be asked for an additional self-reported VVR score. An additional recording of 1 to 2 minutes would be made. Stage 2: Data preprocessing shows an overview of the video preprocessing steps. At each step, the earliest frames were removed. i.e. the frames closer to self-report VVR measures are kept for the analysis. In addition, the number of frames was reduced from 30 FPS to 5 FPS. Stage 3: Data analysis shows an overview of model training and evaluation steps. 4. For datasets of this kind, it is very important that the samples are distributed very rationally into train/validation/test sets. Sometimes it becomes a matter of cherry-picking to show a better performance. To have a better estimate of the performance, it is preferred, in such small datasets, to perform a k-fold cross-validation (k=5 is usually good) and provide an average estimate of the performance. This doesn't overestimate nor underestimate the model. Furthermore, since this is a highly skewed dataset, such an experiment is warranted. The authors might want to do that. Please also ensure that there is no data leakage from the training to the testing set while doing the modelling. Since this is the first study in this area, it is better to show a good modelling strategy that the research community can follow. We agree with your raised concerns. We have followed your advice and re-trained our models using stratified 5-k fold validation technique to ensure that the model performance is not overly optimistic. “We used a 5-fold stratified validation technique on the training set, in which the dataset is divided into 5 subsets, and the model is trained and tested 5 times. In each iteration, one subset is used for testing, and the remaining 4 subsets are used for training. Stratified cross-validation was selected to ensure that each fold contains a representive sample of all classes, which is particularly important due to dataset imbalance to reduce the risk of biased performance. We have also separated our data “based on subject identification number to ensure that the same participants would not appear in both training and testing sets.” In conclusion, our results were consistent across all tested folds and the best performance was in line with our previously reported findings. We hope that this addresses your concerns. We included the results in the Supplementary Material. We reported the average F1 score with SD as well as the highest precision, recall, F1, PR-AUC, and MCC scores obtained. Finally, we reported the actual scores of F1 across all tested folds on the shortest video duration for all four models and compared the model performance using the non-parametric Friedman test. S1 Appendix. To ensure model robustness and reduce the risk of biased performance, we also completed an additional analysis where we used a nested 5-fold stratified validation technique, in which the dataset is divided into 5 subsets, and the model is trained and tested 5 times. In each iteration, one subset is used for testing, and the remaining 4 subsets are used for training. Stratified cross-validation was selected to ensure that each fold contains a representative sample of all classes, which is particularly important due to dataset imbalance. In addition, the data was split based on subject identification number to ensure that the same participants would not appear in both training and testing sets. A nested cross-validation was selected to optimize the hyperparameters and avoid overfitting. We used k=5 for the outer loop (model evaluation) and a k=3 for the inner loop (hyperparameter testing). In S1 Table we report the average F1 score with standard deviation obtained on the tested split for all tested folds, and in S2 Table we report the results obtained on the best performing model along with used hyperparameters. S1 Table. An average estimate of model performance across 5-fold splits using pre-trained Xception and ResNet152 with GRU and LSTM on various video lengths in classifying low and high (minority class) VVR groups. Model Number of frames Average F1 Score across Tested Splits Standard Deviation of F1 Scores across Tested Splits Pre-trained Xception model with GRU N = 150 0.66 0.02 N = 100 0.66 0.04 N = 50 0.51 0.25 N = 25 0.63 0.02 Pre-trained Xception model with LSTM N = 150 0.66 0.02 N = 100 0.65 0.01 N = 50 0.66 0.02 N = 25 0.62 0.04 Pre-trained ResNet152 model with GRU N = 150 0.68 0.01 N = 100 0.68 0.02 N = 50 0.66 0.03 N = 25 0.67 0.03 Pre-trained ResNet152 model with LSTM N = 150 0.68 0.02 N = 100 0.68 0.02 N = 50 0.67 0.02 N = 25 0.67 0.02 S2 Table. The 2D-CNN best model performance on the test split classifying low vs high VVR classes using pre-trained Xception and ResNet152 models with GRU and LSTM on various video sequences ranging from 150 to 25 frames. Model Number of frames Precision Recall F1 AUC-PR MCC Hyperparameters used Pre-trained Xception model with GRU N = 150 0.75 0.63 0.68 0.65 0.43 'batch_size': 64, 'dropout_rate': 0.1, 'epochs': 100, 'learning_rate': 0.001 N = 100 0.74 0.66 0.70 0.65 0.46 'batch_size': 64, 'dropout_rate': 0.1, 'epochs': 100, 'learning_rate': 0.001 N = 50 0.63 0.67 0.65 0.57 0.36 'batch_size': 32, 'dropout_rate': 0.1, 'epochs': 200, 'learning_rate': 0.0001 N = 25 0.71 0.61 0.66 0.61 0.40 'batch_size': 32, 'dropout_rate': 0.5, 'epochs': 200, 'learning_rate': 0.0001 Pre-trained Xception model with LSTM N = 150 0.76 0.63 0.68 0.65 0.45 'batch_size': 64, 'dropout_rate': 0.5, 'epochs': 100, 'learning_rate': 0.0001 N = 100 0.63 0.71 0.67 0.57 0.38 'batch_size': 64, 'dropout_rate': 0.1, 'epochs': 100, 'learning_rate': 0.0001 N = 50 0.68 0.71 0.69 0.60 0.43 'batch_size': 64, 'dropout_rate': 0.1, 'epochs': 200, 'learning_rate': 0.0001 N = 25 0.73 0.61 0.67 0.64 0.40 'batch_size': 32, 'dropout_rate': 0.1, 'epochs': 100, 'learning_rate': 0.0001 Pre-trained ResNet152 model with GRU N = 150 0.69 0.68 0.69 0.61 0.44 'batch_size': 64, 'dropout_rate': 0.5, 'epochs': 200, 'learning_rate': 0.001 N = 100 0.75 0.68 0.71 0.66 0.48 'batch_size': 64, 'dropout_rate': 0.1, 'epochs': 100, 'learning_rate': 0.001 N = 50 0.78 0.64 0.70 0.67 0.49 'batch_size': 32, 'dropout_rate': 0.5, 'epochs': 100, 'learning_rate': 0.001 N = 25 0.65 0.79 0.71 0.61 0.41 'batch_size': 32, 'dropout_rate': 0.5, 'epochs': 200, 'learning_rate': 0.001 Pre-trained ResNet152 model with LSTM N = 150 0.73 0.81 0.71 0.60 0.40 'batch_size': 64, 'dropout_rate': 0.5, 'epochs': 100, 'learning_rate': 0.001 N = 100 0.67 0.71 0.69 0.61 0.38 'batch_size': 64, 'dropout_rate': 0.5, 'epochs': 100, 'learning_rate': 0.001 N = 50 0.75 0.63 0.69 0.66 0.44 'batch_size': 64, 'dropout_rate': 0.5, 'epochs': 200, 'learning_rate': 0.001 N = 25 0.77 0.63 0.70 0.67 0.47 'batch_size': 64, 'dropout_rate': 0.1, 'epochs': 100, 'learning_rate': 0.001 5. As a paper focusing on medical domain applications, one would like to see substantial discussion on those and how previous AI or non-AI-based studies have done those. In our introduction, we included a section on current interventions used for preventing vasovagal reactions and what are the limitations. For example: “Applied muscle tension consists of repeated contractions of muscles in the legs and/or abdomen in order to increase blood pressure [11, 13]. Even though research shows that these techniques could work for a subset of donors [13], a meta-analysis suggested that these techniques are insufficient for the majority of donors [11] and do not reduce the rate of syncopic reactions [13]. Other preventive strategies provided by healthcare professionals range from providing extra information, social support or distractions or even administering calming medication such as low doses of benzodiazepines that allow the patients to reduce their anxiety levels (Gebhardt et al., 2018). Although benzodiazepines reduce the number of vasovagal reactions by addressing underlying fear and anxiety, some side effects might make them less favorable as a prevention strategy, especially, as blood draws or immunizations are quick procedures. All previously applied preventive techniques might be effective, however, they are costly in terms of extra time required by the staff. Hence, there is an enormous need for new prevention methods.” In addition, we included some examples of other papers that used facial video information to estimate the likelihood of experiencing pain, depression, anxiety and other psychological and physiological measures. “Video recordings of the face contain many types of useful information. For example, it contains information about facial expressions which, even when they are very subtle, enable the detection of anxiety, stress, fear, pain, and vasovagal reactions [33-38]. In addition, the face contains information of head movements, eye-gaze direction, changes in facial colors such as paleness, etc. With the recent developments in the field of deep learning and in particular automatic face analysis [39-41], it has been shown possible to predict not only mental health conditions such as depression, anxiety, or obsessive-compulsive disorder [42], but also physical symptoms such as pain [43-45]. These models can potentially be used as valuable tools for clinical diagnosis and for monitoring and altering physical responses in real time [24]. Specifically, we found a significant association between vasovagal reactions and changes in facial temperature [25] as well as facial micro-expressions [33]. Both changes in thermal fluctuations and facial action units recorded prior to blood donation showed promising results in predicting vasovagal reactions that occur during or after blood donation [24, 25, 33].” 6. Minor and typos: RMSEprop --> RMSProp. Proofreading is needed to improve the presentation. We have proofread our text and fixed the typos. Reviewer #1 Dear Reviewer, First of all, we would like to thank you for your valuable insights that allowed us to improve the manuscript. In what follows, we address each comment and demonstrate the changes that were made to the manuscript. 1. How does the study align with existing theories on the interplay between psychological states, physiological responses, and the predictability of vasovagal reactions during needle-related procedures? Previous studies show that anticipatory anxiety and stress is associated with increased physiological changes such as increased heart rate or cortisol levels that occur at the early stage of blood donation or any other needle-related procedure and can serve as early markers of stress. It is likely that these early subtle psychological and physiological changes could be captured using psychophysiological techniques such as infrared-thermal cameras and, as shown in this experiment, using a standard RGB camera and ML techniques. We included the examples of previous findings in our introduction to elaborate more on the relationship between psychological, physiological changes and vasovagal reactions: “Research shows that anticipatory fear, anxiety and stress and a history of previous vasovagal reactions are one of the most important risk factors for experiencing vasovagal reactions [16, 17, 10, 18, 19]. This is corroborated by studies from Hoogerwerf et al. (2018; 2017) [20, 21] who assessed psychological, hormonal and psychophysiological stress markers in donors throughout a blood donation and found that the objective stress markers already occur at a very early stage in anticipation of the needle insertion, at which time they peak. For example, the levels of systolic blood pressure and cortisol levels increased towards needle insertion and then decreased after the blood donation [20, 21]. In addition, higher systolic blood pressure and pulse rate were found in women and first-time donors, who are at the higher risk of experiencing VVR [14, 15].” “In one of our previous studies, we mimicked a blood donation using an experimental 'virtual' rubber arm illusion, imaging the participant with an infrared thermal imaging camera. This experiment showed that changes in facial temperature could serve as early indicators for vasovagal reactions [24, 25]. Specifically, facial temperature fluctuations in the area under the nose, chin and forehead are associated with increased risk of experiencing VVR [25]. “ 2.Considering the prevalence of severe vasovagal reactions, how does the study address the challenge of training machine learning models with limited data? This is indeed a challenge to train machine learning models with limited data of (specifically high VVR) examples. However, we tried a few approaches to improve our model performance. First of all, as we emphasized in our introduction, we used transfer learning to overcome the limited number of samples. “To train a model from scratch, a large amount of training data of donors with both high and low VVR scores is required, which is difficult to obtain. To overcome this limitation, transfer learning will be applied. Transfer learning is one of the machine learning techniques where a model, initially trained on one task, is used for a different task with some additional tuning.” Second, to increase the generalization and robustness of the trained models by exposing them to a wider range of variations in the input data, we applied various transformations to the existing video data samples in the training set. We explain this in the section “Model training, validation, and evaluation”: “Due to class imbalance and limited number of high VVR samples, we applied video data augmentation on the high VVR cases in the training data. Data augmentation is a technique used in Attachments Attachment Submitted filename: Response to Reviewers.docx* https://doi.org/10.1371/journal.pone.0314038.r002
15 Oct 2024 Decision Letter - Tirtharaj Dash, Editor PONE-D-23-40599R1Predicting vasovagal reactions to needles from video data using 2D-CNN with GRU and LSTM.PLOS ONE Dear Dr. Rudokaite, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. ============================== Dear Authors, I am happy to read your revised manuscript. I think you have addressed the concerns raised by the esteemed reviewers and me rather very well. I am satisfied with the revision and your response(s) to each of the reviewers' questions. However, after consulting with a domain-expert (new reviewer) and their comments, I am able to propose an acceptance only after you address their concerns, which I think are minor and clarification type. I believe those new comments will certainly improve the manuscript further. I look forward to receiving your revision soon. Thank you. ============================== Please submit your revised manuscript by Nov 29 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript: A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols. We look forward to receiving your revised manuscript. Kind regards, Tirtharaj Dash Academic Editor PLOS ONE Journal Requirements: Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice. Additional Editor Comments: Dear Authors, I am happy to read your revised manuscript. I think you have addressed the concerns raised by the esteemed reviewers and me rather very well. I am satisfied with the revision and your response(s) to each of the reviewers' questions. However, after consulting with a domain-expert (new reviewer) and their comments, I am able to propose an acceptance only after you address their concerns, which I think are minor and clarification type. I believe those new comments will certainly improve the manuscript further. I look forward to receiving your revision soon. Thank you. [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation. Reviewer #4: (No Response) ******** 2. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #4: Yes ****** 3. Has the statistical analysis been performed appropriately and rigorously? Reviewer #4: Yes ****** 4. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #4: Yes ****** 5. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #4: Yes ****** 6. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #4: This study investigates the prediction of vasovagal reactions (VVR) during blood donation using facial information captured in video sequences. The study demonstrates the potential to predict VVR via facial features to support possible future interventions to prevent adverse reactions. However, there are some possible limitations to consider: 1. In the introduction part - first paragraph, the authors mentioned “these reactions can result in physiological changes such as increased heart rate…”. Then authors mentioned parasympathetic activity such as drops in heart rate or blood pressure. It might be better to bring out only parasympathetic activity directly to avoid confusion. 2. How to define VVR in case group? Are the vasovagal reactions reported by patients themselves or diagnosed by doctors or trained medical staff? 3. How does this quick and short-time identification method prevent VVR? Why is it important? Authors mentioned several potential ways to prevent VVR from happening, but how can identifying VVR prevent VVR? It would be good if authors could describe more on the possible application on medical perspectives. ****** 7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #4: No ******** [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step. https://doi.org/10.1371/journal.pone.0314038.r003
Revision 2
23 Oct 2024 Author Response Dear Editor and Reviewers, We sincerely thank the new reviewer and the editor for their careful evaluation of our manuscript. We have addressed all of your suggestions for improvement, and a summary of our responses and modifications in relation to each comment is provided below. 1. In the introduction part - first paragraph, the authors mentioned “these reactions can result in physiological changes such as increased heart rate…”. Then authors mentioned parasympathetic activity such as drops in heart rate or blood pressure. It might be better to bring out only parasympathetic activity directly to avoid confusion. Thank you for pointing this out. Indeed this can be misleading, therefore, we adjusted the text to better explain how the vasovagal reactions occur: “These so-called vasovagal reactions (VVR) can manifest as nausea, dizziness, heart palpitations, hyperventilation, or even fainting with a temporary loss of consciousness. They occur when the body experiences an exaggerated response to a trigger such as blood draws or injections during the medical procedure. The VVR reactions are primarily mediated by the autonomic nervous system, which controls involuntary functions like heart rate and blood pressure. Initially, the sympathetic branch of the autonomic nervous system activates in response to the stressful or fearful situation, as part of the body's "fight or flight" mechanism, leading to an increase in heart rate [2-3], sweating, nausea, and pupil dilation [4, 5, 6]. However, in some individuals, the parasympathetic nervous system, which serves as the "rest and digest" counterpart, overcompensates for this initial response, working to slow down the heart rate and lower blood pressure in an attempt to restore balance. When this parasympathetic response becomes too strong, it can cause a rapid drop in heart rate and blood pressure [3], reducing blood flow to the brain. This decrease in blood and oxygen supply can result in dizziness, nausea, and ultimately fainting (syncope) [7].” 2. How to define VVR in case group? Are the vasovagal reactions reported by patients themselves or diagnosed by doctors or trained medical staff? The vasovagal reactions were self-reported by patients. We now specified this in the section called Procedure: “Specifically, at each stage donors were recorded using a regular video camera and had to verbally self-report their VVR score.” And under section Materials and Methods. Vasovagal reactions: “participants were asked to verbally rate 8 questions regarding experienced physiological reactions” 3. How does this quick and short-time identification method prevent VVR? Why is it important? Authors mentioned several potential ways to prevent VVR from happening, but how can identifying VVR prevent VVR? It would be good if authors could describe more on the possible application on medical perspectives. Thank you for your question. It seems this was not well explained in the text. We modified the “Applying deep learning methods for automated video analysis for a biofeedback-based serious game intervention” section in the Introduction, to better explain on how the game would use information of the risk of experiencing VVRs: “In order to address the lack of interventions that address the anticipatory risk factors for VVR in practice we developed a solution for people with needle fear that is able to not only identify covert symptoms of emotional and physical reactions at a very early stage, but also to immediately give them a tool which can help them to prevent the escalation into a vasovagal event. To achieve that, we aimed to implement the best performing model in a serious game for smartphones, which through the facial video input from the front-facing camera, controls a biofeedback mechanism which will help the player get control over their impeding VVR in an early stage. <...> However, assessing the risk of VVR is only one part of the solution. Even better would be if the patient can use this information to prevent the VVR from happening. This can be achieved through biofeedback. <....> By seeing or hearing these physiological signals in real time, patients can experiment with strategies to manage their stress or anxiety and bring their physiological state back into balance. <...> In these games, the stimuli adjust based on the player’s bodily responses, allowing the individual to practice managing their physiological reactions in a controlled and engaging environment. In the case of needle fear, our serious game solution (called AINAR, Artificial Intelligence for Needle Induced Fainting) continuously assesses the likelihood of experiencing a vasovagal reaction (VVR) through the model, which gets its input from the front-facing camera, which is then reflected in the weather, which can be sunny, rainy, or snowy. The player's task is to keep the weather nice and sunny. If it starts to rain, the player can experiment with different relaxation techniques or cognitive strategies to transition from a state of fear or stress to calmness, thus learning to control their body's automatic responses. The aim is to provide the feedback as often as possible with little delay, therefore, we aim to investigate what is the shortest length of video that is required for acceptable VVR prediction. This approach allows for real-time learning and adaptation, making biofeedback a powerful tool for overcoming anxiety and stress-related conditions.” Attachments Attachment Submitted filename: Response to Reviewers.docx https://doi.org/10.1371/journal.pone.0314038.r004
5 Nov 2024 Decision Letter - Mohammad Amin Fraiwan, Editor Predicting vasovagal reactions to needles from video data using 2D-CNN with GRU and LSTM. PONE-D-23-40599R2 Dear Dr. Rudokaite, We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication. An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org. If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. Kind regards, Mohammad Amin Fraiwan Academic Editor PLOS ONE Additional Editor Comments (optional): Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation. Reviewer #4: All comments have been addressed ******** 2. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #4: Yes ****** 3. Has the statistical analysis been performed appropriately and rigorously? Reviewer #4: Yes ****** 4. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #4: Yes ****** 5. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #4: Yes ****** 6. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #4: (No Response) ****** 7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #4: No ******** https://doi.org/10.1371/journal.pone.0314038.r005
Formally Accepted
8 Nov 2024 Acceptance Letter - Mohammad Amin Fraiwan, Editor PONE-D-23-40599R2 PLOS ONE Dear Dr. Rudokaite, I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team. At this stage, our production department will prepare your paper for publication. This includes ensuring the following: * All references, tables, and figures are properly cited * All relevant supporting information is included in the manuscript submission, * There are no issues that prevent the paper from being properly typeset If revisions are needed, the production department will contact you directly to resolve them. If no revisions are needed, you will receive an email when the publication date has been set. At this time, we do not offer pre-publication proofs to authors during production of the accepted work. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few weeks to review your paper and let you know the next and final steps. Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. If we can help with anything else, please email us at customercare@plos.org. Thank you for submitting your work to PLOS ONE and supporting open access. Kind regards, PLOS ONE Editorial Office Staff on behalf of Dr. Mohammad Amin Fraiwan Academic Editor PLOS ONE https://doi.org/10.1371/journal.pone.0314038.r006

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .