Linguistic based emotion analysis using softmax over time attention mechanism

Megha Roshan; Mukul Rawat; Karan Aryan; Elena Lyakso; A. Mary Mekala; Nersisson Ruban

doi:10.1371/journal.pone.0301336

Peer Review History

Original SubmissionJuly 28, 2023
10 Dec 2023 Decision Letter - Michal Ptaszynski, Editor PONE-D-23-22091Linguistic based Emotions Analysis Using Softmax over Time Attention MechanismPLOS ONE Dear Dr. Ruban, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please submit your revised manuscript by Jan 24 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript: A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols. We look forward to receiving your revised manuscript. Kind regards, Michal Ptaszynski, PhD Academic Editor PLOS ONE Journal Requirements: When submitting your revision, we need you to address these additional requirements. 1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf 2. Thank you for stating the following in the Acknowledgments Section of your manuscript: "The authors thank Vellore Institute of Technology, Vellore for providing ‘VIT SEED GRANT’ for carrying out the initial study related to this research work." We note that you have provided additional information within the Acknowledgements Section that is not currently declared in your Funding Statement. Please note that funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form. Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows: "The authors received no specific funding for this work" Please include your amended statements within your cover letter; we will change the online submission form on your behalf. 3. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information. [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Partly Reviewer #2: Yes Reviewer #3: Yes ******** 2. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: Yes Reviewer #2: N/A Reviewer #3: Yes ****** 3. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: No Reviewer #2: Yes Reviewer #3: Yes ****** 4. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes Reviewer #2: No Reviewer #3: Yes ****** 5. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: I read this manuscript over and over again but I failed to find what was the state-of-the-art of this research. The title says attention mechanism, but in the background it doesn't discuss why this method is proposed Multimodal classification, namely voice and textual input, also does not explain the results. Fig. 1 breaks down the two classifications, then how to vote on the classification results. I think this manuscript fails to convey the urgency of the proposed research. This can be seen because there is no emphasis on what the author did and what his contribution was in the introduction I don't see the benefit of giving colors to Figures, for example Fig. 2 to Fig. 5. To describe the distribution of a dataset, just use a regular picture or a table of numbers Reviewer #2: - The introduction should clarify: What are major research objectives? Are there any existing solutions? Which is the best? What is its main limitation? What does the author hope to achieve? - Also include the problem identification and research gaps. - The authors should highlight the manuscript's innovations and contributions. - Please make the Introduction and related work sections more productive using the following articles. Reading and using these articles and also cited in this article: Identifying emotions from facial expressions using a deep convolutional neural network-based approach. Multimedia Tools and Applications (2023): 1-22.; Facial emotion recognition and music recommendation system using CNN-based deep learning techniques. Evolving Systems (2023): 1-18.; Sentiment analysis on images using convolutional neural networks-based Inception-V3 transfer learning approach. International Journal of Information Management Data Insights 3.1 (2023): 100174.; Sentiment analysis on images using different transfer learning models. Procedia Computer Science, 218, 1640-1649. ; Image-based sentiment analysis using InceptionV3 transfer learning approach." SN Computer Science 4.3 (2023): 242. - Results may be describe in proper way. - Experimental details, simulation environment, measuring parameters etc. need to be added. - Author may compare their work with existing literature. - All figures have low quality, and please improve all of them. - Improve discussion section. It must include the strengths and weaknesses of the proposed approach. - Expand the critical results in the conclusion. Focus on the main developments in the finale. Also, write the main contributions in the conclusion. - The grammatical error should be corrected. Reviewer #3: This paper presents an innovative approach to emotion recognition, focusing on textual data analysis to complement the traditional methods that rely on speech signal features. The authors rightly identify a significant gap in current emotion detection systems, which often misinterpret emotions due to limitations in analyzing speech frequency and pitch, or can be misled by false facial expressions in video-based systems. The proposed solution, leveraging a modified Encoder-Decoder model with an attention mechanism, is a notable advancement. The authors' approach to integrate text-based emotion recognition into the feedback management process is both novel and practical. This integration could indeed enhance the accuracy of emotion detection in various applications, particularly in customer feedback and medical scenarios. The achievement of a 93.5% accuracy rate with the modified text-based classification model is impressive. This statistic suggests a significant improvement over existing models and indicates the potential of this method in real-world applications. However, the paper would benefit from a more detailed discussion of the datasets used for training and validation, as well as the specific types of emotions that were recognized and classified. Additionally, the paper might be strengthened by addressing potential limitations or biases in text-based emotion recognition, such as cultural or linguistic variations that could impact the model's effectiveness. Exploring the scalability of the model and its applicability across different domains would also be valuable. In conclusion, the research presents a promising direction in emotion recognition technology. The integration of text-based analysis using a sophisticated attention mechanism offers a more nuanced and potentially more accurate method for emotion detection. Future research could build on this work by exploring its application in diverse settings and further refining the model to account for a wider range of emotional expressions and languages. ****** 6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #1: Yes: Dewi Yanti Liliana Reviewer #2: No Reviewer #3: No ******** [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step. https://doi.org/10.1371/journal.pone.0301336.r001
Revision 1
30 Jan 2024 Author Response Linguistic based Emotions Analysis Using Softmax over Time Attention Mechanism Responses to the Reviewers' comments: Authors would like to sincerely appreciate the reviewer’s valuable comments for the improvement of our manuscript. We have revised the manuscript based on the comments received from the reviewers, and modified the manuscript to best suit the standards of the journal. We are ready for any more revisions also for the betterment of the paper. The authors state that there is no conflict of interest with the suggested reviewers. More over the authors wants to express their heart felt greetings to the reviewers for their valuable suggestion for the improvement of this work. We have incorporated all the suggestions given by the reviewers and the corrections made in the manuscript are highlighted. We have also given detailed response for the comments given by the reviewers below. Reviewer #1: 1. I read this manuscript over and over again but I failed to find what was the state-of-the-art of this research. The title says attention mechanism, but in the background it doesn't discuss why this method is proposed Response: We thank the reviewer for this observation, the state of the art of this research is that, the softmax function independently applies to each set of logits (Logits are the outputs of a neural network before the activation function is applied). This converts the logits into a probability distribution for each emotion category at each time step. This means that for every word or token in the text, you have a probability distribution over emotions. This explanation is included in the manuscript as suggested by the reviewer in Page 9 under 2.3 algorithms. More over we applied the attention mechanism to the encoder decoder model which was never tried before for text based emotion recognition system. Converting the words in the statement into their dense vector layer is the basis for the attention mechanism and incorporating it into the encoded decoded concept to get a better performance is the major contribution of this research. This is discussed in the section 3.2.2. 2. Multimodal classification, namely voice and textual input, also does not explain the results. Fig. 1 breaks down the two classifications, then how to vote on the classification results. Response: We thank the reviewer for this observation, to obtain a final emotion label, we combines the outcomes from both modalities, taking into account their respective confidence scores and output the resulted emotion which has highest confidence score. The mechanism of choosing the highest confidence level is used as the voting scheme in the proposed approach. This multimodal approach enhances the overall accuracy and reliability of our emotion classification system by harnessing complementary information from both voice and text sources. 3. I think this manuscript fails to convey the urgency of the proposed research. This can be seen because there is no emphasis on what the author did and what his contribution was in the introduction. Response: We thank the reviewer for this observation; this research highlights the pressing need for more robust emotion recognition systems and underscores the potential of transfer models with attention mechanisms to significantly improve feedback management processes and the medical applications. The major technical contribution of the authors are briefly included in the introduction section as suggested by the reviewer and it is elaborated in other sections of the manuscript. 4. I don't see the benefit of giving colors to Figures, for example Fig. 2 to Fig. 5. To describe the distribution of a dataset, just use a regular picture or a table of numbers Response: We thank the reviewer for this observation; there is no technical reason for the color images, it was given to make a good look of the article, since it was suggested by the reviewer, we have modified the images into grey scale images. Reviewer #2: - 1. The introduction should clarify: What are major research objectives? Are there any existing solutions? Which is the best? What is its main limitation? What does the author hope to achieve? - Also include the problem identification and research gaps. - The authors should highlight the manuscript's innovations and contributions. - Please make the Introduction and related work sections more productive using the following articles. Reading and using these articles and also cited in this article: Identifying emotions from facial expressions using a deep convolutional neural network-based approach. Multimedia Tools and Applications (2023): 1-22.; Facial emotion recognition and music recommendation system using CNN-based deep learning techniques. Evolving Systems (2023): 1-18.; Sentiment analysis on images using convolutional neural networks-based Inception-V3 transfer learning approach. International Journal of Information Management Data Insights 3.1 (2023): 100174.; Sentiment analysis on images using different transfer learning models. Procedia Computer Science, 218, 1640-1649. ; Image-based sentiment analysis using InceptionV3 transfer learning approach." SN Computer Science 4.3 (2023): 242. Response: We thank the reviewer for the valuable comment, as per the suggestion we have included the contribution, innovation in the article (Introduction section, section 2.3). We have highlighted the objective and included the advantage and limitations in the conclusion section of the manuscript. Thank you for suggesting related articles, we find it is very much related to the proposed research in the application context, so we have referred those articles and cited in the appropriate places in the article. 2. Results may be describe in proper way. Response: As per the suggestion, we have completely rewritten the result section and included more discussion on various plots and prediction given in the result section. 3. Experimental details, simulation environment, measuring parameters etc. need to be added. Response: We designed a versatile simulation environment having Dataset from different diversity, culture, age groups as well as gender. This simulation framework allowed us to replicate real-world scenarios, introducing controlled variations in cultural context, linguistic diversity, and emotional expressions. By simulating diverse conversational dynamics and cross-cultural variations, we assessed the adaptability of our models to different contexts. This simulation approach played a pivotal role in uncovering potential limitations, biases, and challenges that may arise due to cultural, linguistic, and contextual differences, ensuring the robustness and applicability of our models across various real-world scenarios. Biasness can be improved by fine tuning the same model with a new dataset in different simulation environment. 4. Author may compare their work with existing literature. Response: We have introduced three dataset for textual emotion classification, which is very rarely used by any other research group and moreover we have proposed an algorithm which uses dense vector of the words in the sentences to predict the emotions, which is completely a new concept and the state-of-the art approach, we could not find close researches, still we have compared the existing research work which are more close to the proposed research work. The performance comparison is included as Table-4 5. All figures have low quality, and please improve all of them. Response: As per the suggestion, all the plots which have poor resolution was improved and modified into a good resolution images. 6. Improve discussion section. It must include the strengths and weaknesses of the proposed approach. Response: As per the suggestion, the discussion section is completely rewritten and the strengths of the proposed methodology and the challenges are discussed in the revised discussion section. 7. Expand the critical results in the conclusion. Focus on the main developments in the finale. Also, write the main contributions in the conclusion. Response: As per the suggestion, the conclusion section is rewritten, the revised version includes, future directions, major focus on the application and the technical contributions of the research. 8. The grammatical error should be corrected. Response: The whole article is checked for the grammatical error and the language check is performed, and we feel that the article is much improved in the language and grammar context. Reviewer #3: This paper presents an innovative approach to emotion recognition, focusing on textual data analysis to complement the traditional methods that rely on speech signal features. The authors rightly identify a significant gap in current emotion detection systems, which often misinterpret emotions due to limitations in analyzing speech frequency and pitch, or can be misled by false facial expressions in video-based systems. The proposed solution, leveraging a modified Encoder-Decoder model with an attention mechanism, is a notable advancement. The authors' approach to integrate text-based emotion recognition into the feedback management process is both novel and practical. This integration could indeed enhance the accuracy of emotion detection in various applications, particularly in customer feedback and medical scenarios. The achievement of a 93.5% accuracy rate with the modified text-based classification model is impressive. This statistic suggests a significant improvement over existing models and indicates the potential of this method in real-world applications. However, the paper would benefit from a more detailed discussion of the datasets used for training and validation, as well as the specific types of emotions that were recognized and classified. 1. Additionally, the paper might be strengthened by addressing potential limitations or biases in text-based emotion recognition, such as cultural or linguistic variations that could impact the model's effectiveness. Exploring the scalability of the model and its applicability across different domains would also be valuable. Response: Dear Reviewer, we sincerely thank you for your appreciation for our research work, it gives us great motivation to do more research in the domain, as per your suggestion we have included about the cultural diversity and various biases in our manuscript. This inclusion is made in section 2: materials and methods. We are giving the content below for your quick reference: ‘We designed a versatile simulation environment having Dataset from different diversity, culture, age groups as well as gender. This simulation framework allowed us to replicate real-world scenarios, introducing controlled variations in cultural context, linguistic diversity, and emotional expressions. By simulating diverse conversational dynamics and cross-cultural variations, we assessed the adaptability of our models to different contexts. This simulation approach played a pivotal role in uncovering potential limitations, biases, and challenges that may arise due to cultural, linguistic, and contextual differences, ensuring the robustness and applicability of our models across various real-world scenarios. Biasness can be improved by fine tuning the same model with a new dataset in different simulation environment.’ In conclusion, the research presents a promising direction in emotion recognition technology. The integration of text-based analysis using a sophisticated attention mechanism offers a more nuanced and potentially more accurate method for emotion detection. 2. Future research could build on this work by exploring its application in diverse settings and further refining the model to account for a wider range of emotional expressions and languages. Response: Yes, there is an enormous scope for this kind of textual emotion recognition system with the fake emotion proof, as per the suggestion we have included a separate future scope section in the conclusion which gives an overview of all the possible future directions of this research work. Attachments Attachment Submitted filename: Response to the Reviewers.pdf https://doi.org/10.1371/journal.pone.0301336.r002
14 Mar 2024 Decision Letter - Michal Ptaszynski, Editor Linguistic based Emotions Analysis Using Softmax over Time Attention Mechanism PONE-D-23-22091R1 Dear Dr. Ruban, We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication. An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at http://www.editorialmanager.com/pone/ and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org. If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. Kind regards, Michal Ptaszynski, PhD Academic Editor PLOS ONE Additional Editor Comments (optional): Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation. Reviewer #1: All comments have been addressed Reviewer #2: All comments have been addressed ******** 2. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Yes Reviewer #2: Yes ****** 3. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: N/A Reviewer #2: N/A ****** 4. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: Yes Reviewer #2: Yes ****** 5. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes Reviewer #2: Yes ****** 6. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: The author has added state-of-the-art contributions to the abstract. The author has added a sentence on how to vote on emotion classification results in a multimodal system. The author has added how to calculate the softmax function for each emotion class using a probability distribution. Reviewer #2: (No Response) ****** 7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #1: Yes: Dewi Yanti Liliana Reviewer #2: No ******** https://doi.org/10.1371/journal.pone.0301336.r003
Formally Accepted
3 Apr 2024 Acceptance Letter - Michal Ptaszynski, Editor PONE-D-23-22091R1 PLOS ONE Dear Dr. Ruban, I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team. At this stage, our production department will prepare your paper for publication. This includes ensuring the following: * All references, tables, and figures are properly cited * All relevant supporting information is included in the manuscript submission, * There are no issues that prevent the paper from being properly typeset If revisions are needed, the production department will contact you directly to resolve them. If no revisions are needed, you will receive an email when the publication date has been set. At this time, we do not offer pre-publication proofs to authors during production of the accepted work. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few weeks to review your paper and let you know the next and final steps. Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. If we can help with anything else, please email us at customercare@plos.org. Thank you for submitting your work to PLOS ONE and supporting open access. Kind regards, PLOS ONE Editorial Office Staff on behalf of Dr. Michal Ptaszynski Academic Editor PLOS ONE https://doi.org/10.1371/journal.pone.0301336.r004

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .