PONE-D-21-33666The spatio-temporal features of perceived-as-genuine and deliberate expressionsPLOS ONE

Dear Dr. Namba,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

I have now received two reviews of the manuscript entitled The spatio-temporal features of perceived-as-genuine and deliberate expressions (PONE-D-21-33666) that you submitted to PLOS ONE. I was fortunate to secure reviews from two experts in the areas of facial expression analysis and emotional genuineness, both of whom provided strong insight to the strengths and weaknesses of the manuscript.  I read the original manuscript, and then again with comments provided by the reviewers. As you will see, the reviewers found the manuscript to be interesting, and the analytic techniques impressive. The topic is also relevant to the broad readership at PLOS One.  Both reviewers identified the academic merit of the manuscript, however noting that the clarity of the manuscript could be improved. Both reviewers provided guidance on improving the readability, which I encourage you to consider. There was disagreement as to the statistical & methodological rigor of the work, with Reviewer 1 taking issue with some analysis decisions. Both suggested major revisions. As such, I am rejecting the manuscript in its current form, and I invite you to resubmit in the form of major revisions.  You will find the full set of reviewer comments included below. To these I add my own two comments:  1) I would appreciate seeing a clear statement of theoretical contribution of this work. I read your 2021 Scientific Reports article with great interest. However, the 2021 work is only minimally reviewed, and readers may be left unclear as to what this work contributes to our understanding of genuine expressions. I recognize that there are differences with this work – OpenFace vs DeepLabCuts, Happy-Angry vs Surprise, ActionUnits vs landmarks, but these differences are methodological, rather than conceptual. I believe the core contribution relates to the perceptual nature of genuine expressions, but this should be more clearly contrasted from your previous works, along with those in the field. 2) The stimuli presenting Genuine-vs-Deliberate expressions are central to the work, with both perceptual and motion analyses relying on their validity. Yet there is almost no discussion as to how these stimuli were produced, and whether the scenarios would be considered ‘genuine’. This presents a serious methodological oversight which would need to be robustly addressed in your revision.

Please submit your revised manuscript by May 28 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Steven R. Livingstone

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf.

2. Please change "female” or "male" to "woman” or "man" as appropriate, when used as a noun (see for instance https://apastyle.apa.org/style-grammar-guidelines/bias-free-language/gender).

3. Thank you for stating the following financial disclosure:

“This research was supported by Early-Career Scientists (20K14256) from JSPS to S. N., Early-Career Scientists (19K20387) from JSPS to K.N., Grant-in-Aid for Scientific Research on Innovative Area (17H06344) from JSPS, and by Moonshot R&D (JPMJMS2012) from JST to K.W.”

Please state what role the funders took in the study.  If the funders had no role, please state: ""The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.""

If this statement is not correct you must amend it as needed.

Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf.

4. Thank you for stating the following in the Acknowledgments Section of your manuscript:

“This research was supported by Early-Career Scientists (20K14256) from JSPS to S. N., Early-Career Scientists (19K20387) from JSPS to K.N., Grant-in-Aid for Scientific Research on Innovative Area (17H06344) from JSPS, and by Moonshot R&D (JPMJMS2012) from JST to K.W.”

We note that you have provided funding information that is not currently declared in your Funding Statement. However, funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form.

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows:

“This research was supported by Early-Career Scientists (20K14256) from JSPS to S. N., Early-Career Scientists (19K20387) from JSPS to K.N., Grant-in-Aid for Scientific Research on Innovative Area (17H06344) from JSPS, and by Moonshot R&D (JPMJMS2012) from JST to K.W.”

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

Additional Editor Comments:

Dear Dr. Namba,

I have now received two reviews of the manuscript entitled The spatio-temporal features of perceived-as-genuine and deliberate expressions (PONE-D-21-33666) that you submitted to PLOS ONE. I was fortunate to secure reviews from two experts in the areas of facial expression analysis and emotional genuineness, both of whom provided strong insight to the strengths and weaknesses of the manuscript.

I read the original manuscript, and then again with comments provided by the reviewers. As you will see, the reviewers found the manuscript to be interesting, and the analytic techniques impressive. The topic is also relevant to the broad readership at PLOS One.

Both reviewers identified the academic merit of the manuscript, however noting that the clarity of the manuscript could be improved. Both reviewers provided guidance on improving the readability, which I encourage you to consider. There was disagreement as to the statistical & methodological rigor of the work, with Reviewer 1 taking issue with some analysis decisions. Both suggested major revisions. As such, I am rejecting the manuscript in its current form, and I invite you to resubmit in the form of major revisions.

You will find the full set of reviewer comments included below. To these I add my own two comments:

1) I would appreciate seeing a clear statement of theoretical contribution of this work. I read your 2021 Scientific Reports article with great interest. However, the 2021 work is only minimally reviewed, and readers may be left unclear as to what this work contributes to our understanding of genuine expressions. I recognize that there are differences with this work – OpenFace vs DeepLabCuts, Happy-Angry vs Surprise, ActionUnits vs landmarks, but these differences are methodological, rather than conceptual. I believe the core contribution relates to the perceptual nature of genuine expressions, but this should be more clearly contrasted from your previous works, along with those in the field.

2) The stimuli presenting Genuine-vs-Deliberate expressions are central to the work, with both perceptual and motion analyses relying on their validity. Yet there is almost no discussion as to how these stimuli were produced, and whether the scenarios would be considered ‘genuine’. This presents a serious methodological oversight which would need to be robustly addressed in your revision.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: No

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: PRAISE

This manuscript explored the interesting question of whether there are morphological and temporal differences between facial behaviors that are perceived to be genuine vs. deliberate. I liked that it examined perceptions of human facial behaviors rather than computer-generated facial behaviors and agree that this is an important validation step. I also liked the attempt to simultaneously consider morphology and temporal characteristics of the behavioral displays. I also thought the visualizations were nice and appreciated the data and materials being made open.

CONCERNS

1. The manuscript’s readability would be helped if the authors were more consistent in their use of terms. As just one example, multiple different terms are used to refer to the person producing a facial behavior (e.g., sender, encoder) or perceiving it (e.g., observer, decoder, perceiver).

2. The methodological decision to look at onset plus and minus 2 sec creates some standardization but also erodes some of the ecological validity of the stimuli. If temporal characteristics matter for perception, as the authors contend, then why cut them partway through and therefore eliminate the ability of perceivers to see their duration, offset, and other temporal features?

3. Please provide a rationale for making the genuineness vs. deliberate ratings dichotomous. Why not have them rate this quality on a continuum? Forcing a choice between two alternatives throws away valuable information about degree (DeCoster et al., 2009) and introduces (or at least exacerbates) participants’ subjective thresholds as a source of variability.

4. The first paragraph in the Statistical Analysis section (page 6) was very difficult to understand. I think the authors calculated the percentage of participants that rated each stimulus as genuine and then used that quantity to select a subset of the stimuli most consistently rated as genuine vs. deliberate. But the descriptions and some words (“prescribed” and “judgment ratios”) were not adequately clear/explained.

5. Related to concern #4 above, I did not love the methodological decision to analyze 40 stimuli rather than all 180. I think this was done in order to compare behaviors that are perceived-by-most-as-genuine to perceived-by-most-as-deliberate rather than having more ambiguous behaviors. But this selection makes the results much less generalizable. We are not learning about most facial behaviors, we are learning about the 22% least ambiguous ones. If genuineness had been rated dimensionally (as in concern #3), the authors could have regressed it on various temporal features and included all 180 stimuli. If they used a mixed effects model, they also could have explicitly modeled encoder and decoder effects.

6. The authors acknowledge that OpenFace may be incorrect but do not provide any validation evidence such as comparing it to expert FACS coding on a subset of this specific dataset. Because we don’t know how much to trust the AU intensity estimates, we don’t know how much to trust the NMF components and these are really important variables in the paper.

7. Please add some information to the Statistical Analysis section about you will be calculating 99% credible intervals at each value of t and how to interpret the figures (e.g., seeing whether delta=0 falls within that interval). Some of this information is currently in the figure captions only and it is very easy to miss.

8. In addition to plotting the 99% credible intervals at each value of t, I would be interested to see tests done on larger regions of t-space. For example, the authors could calculate the average delta score in the regions from t=0 to t=2000 in intervals of 250 or 500ms. They could also report probability of direction (pd) values for these regions in addition to credible intervals (Makowski et al., 2019).

COMMENTS

1. The claim on page 1 line 50 that perceived-as-genuine expressions have more significant impacts on observer’s behavior than perceived-as-deliberate expressions is a leap that does not follow directly from the evidence presented earlier in the paper.

2. Page 4 would be improved by providing an explicit rationale for the decision to look at anger and happiness rather than other possible emotions.

3. On page 5, please briefly describe what the four emotional scenarios were for happiness and anger.

4. Were there are quality or attention check questions to ensure that the participants were taking the task seriously? I’m not sure if this is a big issue in Japan, but in the US many crowdworkers do a poor job and this can be detected with some basic questions.

REFERENCES

DeCoster, J., Iselin, A.-M. R., & Gallucci, M. (2009). A conceptual and empirical examination of justifications for dichotomization. Psychological Methods, 14(4), 349–366. https://doi.org/10/bh86w7

Makowski, D., Ben-Shachar, M. S., Chen, S. H. A., & Lüdecke, D. (2019). Indices of effect existence and significance in the Bayesian framework. Frontiers in Psychology, 10. https://doi.org/10/ggfw2j

Reviewer #2: Thank you for inviting me to review this manuscript. The authors investigate spatio-temporal features of facial emotion expressions that help to distinguish perceived-as-genuine and perceived-as-deliberate (i.e., non-authentic) expressions. They do so by first presenting a series of dynamic facial emotion expression videos to 89 Japanese individuals and asking them to determine whether each expression is genuine or deliberate (completed for happy and angry expressions). They then reduced each expression video to its lowest dimensional features (as a function of the emotion and 'perceived as' conditions) using non-negative matrix factorisation. Both perceived-as-genuine and perceived-as-deliberate expressions showed protocol typical spatial action units, whilst deliberate expressions produced multiple facial movements in addition to these. Deliberate expressions also had faster onsets to peak expression and had a greater contribution of mouth opening movements.

The question being investigated here is novel, interesting and important. The techniques utilised to answer this question are in my opinion strong and in line with a number of recent studies applying dimensionality reduction techniques to facial emotion expressions. I commend the authors for this. I find the results very interesting and I also think they did a good job to put their findings in context as well as discuss the limitations (e.g., cultural differences that may exist and require further attention.

I have some comments on the write-up (mostly focused on the introduction) that I believe would strengthen the manuscript if rectified:

Introduction:

1) I feel the introduction could be tightened up a little bit so it really 'tells the story' for your reader to follow as to the rational for the current study. Some parts I struggled to follow the relevance of. For example, I wasn't sure I followed how the study is specifically related to the affective pragmatics theory and why this needs to be linked to an 'evolutionary account' that really had very little explanation. Perhaps you can build a story for the studies importance without these theories, especially as you don't refer back to this in the discussion, so I'm not sure exactly how relevant it is. Or alternatively, tighten up the explanation and refer back to it in the discussion.

2) I think your design is great, whereby you are able to determine what spatio-temporal features are important in the human perception or labelling of genuine vs deliberate (as opposed to determining yourselves what a genuine or deliberate expression looks like) and I agree this is in line with the work by Rachel Jack and colleagues. However, I think you could make this point and emphasise the benefit of this kind of approach clearer for the reader. You could tighten up the explanation of Jack and co.'s method.

3) My colleagues and I have investigated temporal features (mean speed) of happy, angry and sad expressions in posed and spontaneous contexts (Sowden et al., 2021; https://psycnet.apa.org/fulltext/2021-22419-001.html) which might also fit well into your introduction and/or discussion to support differences here. We found (in Experiment 1C) that speed of mouth opening and widening actions helped to differentiate between happy, angry and sad expressions only for posed and not spontaneous expressions, whilst speed of eyebrow widening movements was capable of differentiating these emotions for both posed and spontaneous expressions. This to me seems a nice finding alongside your findings, whilst your design has the nice element of looking at timecourses as opposed to our mean speed over entire expression period.

Methods

1) I think the methods section would benefit from a little more explanation of what the NMF dimensionality reduction technique is and what it is essentially doing, for readers who do not know about this technique. On this topic, the authors might be interested in (and may wish to cite) an interesting paper by Chiovetti et al. (2018; https://jov.arvojournals.org/article.aspx?articleid=2678770) who seek to reduce various facial expressions down to their lowest dimensional organisation (regarding temporal properties of expressions) whereby emotion can be determined. They also compare NMF and a Fourier-based Anechoic Demixing Algorithm in doing so.

2) Can you be slightly clearer in your methods by telling the reader that the way you arrived at 4840 data points is by taking each frame (121) in each of the 40 videos as a data point?

3) Could you clarify how exactly you made all the videos to be of the same length, this wasn't totally clear to me?

Small changes:

Page 3, line 85: I think you should refer to 'emotion categories' as opposed to mental representations to be clearer here for the reader.

Page 6, line 159: Was the ISI 'about' 300 ms or 'exactly' 300 ms? I.e., did you have a variable ISI or was this always 300ms?

Page 9, line 240: Remove the word 'mean' or 'refer to'.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Jeffrey Girard

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

The spatio-temporal features of perceived-as-genuine and deliberate expressions

PONE-D-21-33666R1

Dear Dr. Namba,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Please see the minor suggested comments of R2, which you may wish to incorporate prior to proofing.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Steven R. Livingstone

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Please see the minor suggested comments from R2 prior to your proofing.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: (No Response)

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: (No Response)

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: (No Response)

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: (No Response)

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: (No Response)

Reviewer #2: I am happy with responses to my comments and associated changes. Just three very minor comments remain:

1) When discussing the Sowden et al paper in the introduction, for clarity I would change the sentence to read: "Sowden et al demonstrated, using facial landmarks, that speed of facial movements differentiates deliberate expressions of anger, happiness and sadness."

2) Line 106: delete the words 'as to'.

3) Line 169-170: change sentence to read "If the participants responded with no questions, the main trials began."

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Jeffrey Girard

Reviewer #2: No

**********