Exploring the awe-some: Mobile eye-tracking insights into awe in a science museum

Sheila Krogh-Jespersen; Kimberly A. Quinn; William L. D. Krenzer; Christine Nguyen; Jana Greenslit; C. Aaron Price

doi:10.1371/journal.pone.0239204

Peer Review History

Original SubmissionJanuary 9, 2020
29 Apr 2020 Decision Letter - Enkelejda Kasneci, Editor PONE-D-20-00804 Exploring the Awe-some: Mobile eye-tracking insights into Awe in a science museum PLOS ONE Dear Dr. Krogh-Jespersen, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Although both reviewers acknowledge the idea behind the study, both reviewers have several and partly complementary comments regarding the hypotheses and methodology. In addition, the second reviewer has several technical questions that the authors should address accordingly in a new iteration. We would appreciate receiving your revised manuscript by Jun 13 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols Please include the following items when submitting your revised manuscript: A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out. We look forward to receiving your revised manuscript. Kind regards, Enkelejda Kasneci, Ph.D. Academic Editor PLOS ONE Journal Requirements: When submitting your revision, we need you to address these additional requirements. 1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf 2. We note that you have stated that you will provide repository information for your data at acceptance. Should your manuscript be accepted for publication, we will hold it until you provide the relevant accession numbers or DOIs necessary to access your data. If you wish to make changes to your Data Availability statement, please describe these changes in your cover letter and we will update your Data Availability statement to reflect the information you provide. 3. Please provide additional details regarding participant consent. In the ethics statement in the Methods and online submission information, please ensure that you have specified whether consent was informed 4. We note that Figures in your submission contain copyrighted images. All PLOS content is published under the Creative Commons Attribution License (CC BY 4.0), which means that the manuscript, images, and Supporting Information files will be freely available online, and any third party is permitted to access, download, copy, distribute, and use these materials in any way, even commercially, with proper attribution. For more information, see our copyright guidelines: http://journals.plos.org/plosone/s/licenses-and-copyright. We require you to either (1) present written permission from the copyright holder to publish these figures specifically under the CC BY 4.0 license, or (2) remove the figures from your submission: 1. You may seek permission from the original copyright holder of Figures to publish the content specifically under the CC BY 4.0 license. We recommend that you contact the original copyright holder with the Content Permission Form (http://journals.plos.org/plosone/s/file?id=7c09/content-permission-form.pdf) and the following text: “I request permission for the open-access journal PLOS ONE to publish XXX under the Creative Commons Attribution License (CCAL) CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). Please be aware that this license allows unrestricted use and distribution, even commercially, by third parties. Please reply and provide explicit written permission to publish XXX under a CC BY license and complete the attached form.” Please upload the completed Content Permission Form or other proof of granted permissions as an "Other" file with your submission. In the figure caption of the copyrighted figure, please include the following text: “Reprinted from [ref] under a CC BY license, with permission from [name of publisher], original copyright [original copyright year].” 2. If you are unable to obtain permission from the original copyright holder to publish these figures under the CC BY 4.0 license or if the copyright holder’s requirements are incompatible with the CC BY 4.0 license, please either i) remove the figure or ii) supply a replacement figure that complies with the CC BY 4.0 license. Please check copyright information on all replacement figures and update the figure caption with source information. If applicable, please specify in the figure caption text when a figure is similar but not identical to the original image and is therefore for illustrative purposes only. 5. We note that Figure 1 includes an image of a [patient / participant / in the study]. As per the PLOS ONE policy (http://journals.plos.org/plosone/s/submission-guidelines#loc-human-subjects-research) on papers that include identifying, or potentially identifying, information, the individual(s) or parent(s)/guardian(s) must be informed of the terms of the PLOS open-access (CC-BY) license and provide specific permission for publication of these details under the terms of this license. Please download the Consent Form for Publication in a PLOS Journal (http://journals.plos.org/plosone/s/file?id=8ce6/plos-consent-form-english.pdf). The signed consent form should not be submitted with the manuscript, but should be securely filed in the individual's case notes. Please amend the methods section and ethics statement of the manuscript to explicitly state that the patient/participant has provided consent for publication: “The individual in this manuscript has given written informed consent (as outlined in PLOS consent form) to publish these case details”. If you are unable to obtain consent from the subject of the photograph, you will need to remove the figure and any other textual identifying information or case descriptions for this individual. Additional Editor Comments (if provided): [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Yes Reviewer #2: Partly ******** 2. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: N/A Reviewer #2: N/A ****** 3. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: No Reviewer #2: No ****** 4. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes Reviewer #2: Yes ****** 5. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: This is an interesting exploratory study combining survey and mobile eye tracking methods in a science center, a real-world-setting where informal learning takes place. In this context I very much appreciated reading about "awe" as this presents a clear and specific research focus that is of potential interest to informal learning institutions. Here, fascination often comes from feelings of wonder or sensation and barriers from feelings of intimidation or insecurity that, to my mind, both tackle the concept of "awe". Consequently, I also appreciated reading the part where "awe" is defined and that the authors transferred a standardized survey (SAS Situational Awe Scale) from the lab to the field. What I am missing is a better contextualisation of this research in the area of museum and visitors studies with a huge amount of studies that deal with exhibition experiences. In this sense I also found it a bit uncommon to read about "guests" and not about "visitors" and their experiences on site. With respect to other mobile eye tracking studies, I would have liked to have clarified in what sense the authors believe that this is "the first time mobile eye-tracking is utilized in this type of research". An important argument of the text is based on the necessity of method combinations when working with mobile eye tracking following Eghbal-Azar & Widlok (2013). On a meta level the authors very much make clear that such a method combination is important also in this study. In addition, the authors demonstrate understandably the importance of working in a trans-disciplinary team. However, my main point of critique is that the presented results and their discussion do too little to empirically make a point on the importance of mixed method research designs with mobile eye tracking and their use in informal learning settings. This is to my mind, not a problem of the study per se that seems well planned and professionally conducted but a problem of data analysis that is not going into depth. On the one hand, this refers to the known problem of complex mobile tracking data analysis with the necessity of laborous manual work. On the other hand I have the impression that the researchers could have studied their data more thoroughly in order to present more detailed results. This would also enable a refined discussion how the phenomenon of "awe" can be studied through visual attention patterns and self reported emotions. To shed new light on exhibition display, as stated in the abstract, also more empirical results and more interpretative thoughts would be necessary. If the authors are able to base their arguments better in their own empirical work (also presenting more tables and figures) and discuss their results in depth (also in comparison with other studies), I am very much looking forward to see this text published. This would be another milestone in advancing the use of mobile eye tracking in informal learning institutions. Reviewer #2: 1) L81: Regarding the unobtrusiveness claim, do the authors have any data to support this claim? Some evidence towards this general direction for head-mounted eye trackers is provided by [1] with a large sample size, although that work use a different eye tracking system. Please expand on this item in the text. 2) L175: Please clarify in the text: 2.1) did the authors switched between the Tobii-provided nose pieces as necessary or a single size was used? 2.2) did the authors employ the corrective lenses if necessary or all participants reported sufficient visual acuity without glasses? 3) L210: Was each participant's data coded by a single coder or the same data was coded by multiple coders to decrease subjectivity? This too should be clear in the text. 4) Still regarding coding: Tobii provides a solution for automatic gaze mapping. Did the authors tried using it instead of manual coding? If so, a brief paragraph on their experience with the automatic gaze mapping would be interesting even if a bad one (e.g., [2]). 5) When mapping to the still images, how did the coders dealt with gaze positions? Was the gaze point translated to the exact matching position or to a "nearest target"? This is relevant since [3] suggests that, for upward gaze directions, Tobii Glasses 2 suffers from large deviations and the participants spent more time gazing upwards (L261). Please clarify in the text. 6) L258: The authors mention "interesting patterns". Personally, I didn't notice any correlation in Figs 5 and 6. I suggest the authors to clearly describe which effects they have noticed in the text and, despite the reduced sample size, conduct the appropriate tests even if they don't end up being statistically significant. 7) L364: Please expand on which exhibit features had emotional experience implications. [1] Santini, Thiago, et al. "The art of pervasive eye tracking: unconstrained eye tracking in the Austrian Gallery Belvedere." Proceedings of the 7th workshop on pervasive eye tracking and mobile eye-based interaction. 2018. [2] Herlitz, Mattias. "Analyzing the Tobii Real-world-mapping tool and improving its workflow using Random Forests." (2018). [3] Niehorster, Diederick C., et al. "The impact of slippage on the data quality of head-worn eye trackers." Behavior Research Methods (2020): 1-21. ****** 6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review?** For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #1: No Reviewer #2: No [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step. https://doi.org/10.1371/journal.pone.0239204.r001
Revision 1
18 Aug 2020 Author Response 5. Review Comments to the Author Reviewer #1: This is an interesting exploratory study combining survey and mobile eye tracking methods in a science center, a real-world-setting where informal learning takes place. In this context I very much appreciated reading about "awe" as this presents a clear and specific research focus that is of potential interest to informal learning institutions. Here, fascination often comes from feelings of wonder or sensation and barriers from feelings of intimidation or insecurity that, to my mind, both tackle the concept of "awe". Consequently, I also appreciated reading the part where "awe" is defined and that the authors transferred a standardized survey (SAS Situational Awe Scale) from the lab to the field. What I am missing is a better contextualisation of this research in the area of museum and visitors studies with a huge amount of studies that deal with exhibition experiences. In this sense I also found it a bit uncommon to read about "guests" and not about "visitors" and their experiences on site. We appreciate this feedback and have added additional literature and text to the section, “Museums and Awe” to bolster our discussion. We also include a statement in the Research Setting section addressing our use of the term “guest” in the place of “visitor”. With respect to other mobile eye tracking studies, I would have liked to have clarified in what sense the authors believe that this is "the first time mobile eye-tracking is utilized in this type of research". We appreciate the reviewer pointing out the broadness of this statement and have clarified on page 6 that we mean “to examine the psychological concept of awe within the real world setting of a science museum.“ An important argument of the text is based on the necessity of method combinations when working with mobile eye tracking following Eghbal-Azar & Widlok (2013). On a meta level the authors very much make clear that such a method combination is important also in this study. In addition, the authors demonstrate understandably the importance of working in a trans-disciplinary team. However, my main point of critique is that the presented results and their discussion do too little to empirically make a point on the importance of mixed method research designs with mobile eye tracking and their use in informal learning settings. This is to my mind, not a problem of the study per se that seems well planned and professionally conducted but a problem of data analysis that is not going into depth. On the one hand, this refers to the known problem of complex mobile tracking data analysis with the necessity of laborious manual work. On the other hand I have the impression that the researchers could have studied their data more thoroughly in order to present more detailed results. This would also enable a refined discussion how the phenomenon of "awe" can be studied through visual attention patterns and self reported emotions. To shed new light on exhibition display, as stated in the abstract, also more empirical results and more interpretative thoughts would be necessary. If the authors are able to base their arguments better in their own empirical work (also presenting more tables and figures) and discuss their results in depth (also in comparison with other studies), I am very much looking forward to see this text published. This would be another milestone in advancing the use of mobile eye tracking in informal learning institutions. We appreciate the call from the reviewer for more information about the visual patterns and more interpretation. We have included 2 tables in the revised manuscript that provide the individual patterns of attention in seconds to various features within the two exhibit spaces. We have also included exploratory correlational analyses between the eye-tracking measures and the Awe measures in supplementary materials with a note in text regarding the nature of correlations conducted with small sample sizes. We have also provided more detail regarding how mobile eye-tracking coding was conducted to shed light on this method and some of the constraints mentioned above. Reviewer #2: 1) L81: Regarding the unobtrusiveness claim, do the authors have any data to support this claim? Some evidence towards this general direction for head-mounted eye trackers is provided by [1] with a large sample size, although that work use a different eye tracking system. Please expand on this item in the text. We include additional citations regarding the use of recording devices worn on the heads of study participants as well as qualitative reports from our study team regarding participant behavior to support this claim. We did not explicitly ask participants about their comfort level with the eye-tracking Glasses beyond ensuring that they were comfortable during the calibration and set-up of the equipment. They were informed that they could remove the Glasses at any time if they experienced discomfort. We also include the recommended cites from this review to direct readers to articles addressing these issues more specifically. 2) L175: Please clarify in the text: 2.1) did the authors switched between the Tobii-provided nose pieces as necessary or a single size was used? 2.2) did the authors employ the corrective lenses if necessary or all participants reported sufficient visual acuity without glasses? We did not switch between nose pieces, and just used a single size. We excluded approaching visitors with glasses (now discussed in the manuscript) and did not use corrective lenses in the eye-tracking Glasses. 3) L210: Was each participant's data coded by a single coder or the same data was coded by multiple coders to decrease subjectivity? This too should be clear in the text. We clarify in the methods section that two coders where involved in the process: a primary coder and a reliability coder. 4) Still regarding coding: Tobii provides a solution for automatic gaze mapping. Did the authors tried using it instead of manual coding? If so, a brief paragraph on their experience with the automatic gaze mapping would be interesting even if a bad one (e.g., [2]). We have clarified in a description in the Coding section the many factors that mainly are due to the dynamic and individual nature of each guest’s exploration of the exhibit space that rendered this automatic mapping as unreliable. 5) When mapping to the still images, how did the coders dealt with gaze positions? Was the gaze point translated to the exact matching position or to a "nearest target"? This is relevant since [3] suggests that, for upward gaze directions, Tobii Glasses 2 suffers from large deviations and the participants spent more time gazing upwards (L261). Please clarify in the text. Although we are not entirely sure what is meant by “nearest target” here, we assume that means adjusting the eye-tracking data points based on an assumption of where the person was looking, which we did not do. We matched the gaze points to static images of the exhibit spaces, with examples of images for the U-505 now presented in Table 2 with more detail about individual coding and analyses. 6) L258: The authors mention "interesting patterns". Personally, I didn't notice any correlation in Figs 5 and 6. I suggest the authors to clearly describe which effects they have noticed in the text and, despite the reduced sample size, conduct the appropriate tests even if they don't end up being statistically significant. We have provided correlation tables to be presented in Supplementary Materials. Because correlations are extremely unstable at small sample sizes (e.g., Schönbrodt & Perugini, 2013), we do not wish to draw attention to the inferential statistics and away from the broader message in the main text, but are happy to provide the numbers for the reader’s interest. 7) L364: Please expand on which exhibit features had emotional experience implications. We have revised the manuscript to include the individual patterns of total attention directed toward various features within the exhibit spaces; however, we note that self-report measures were only taken at the end of the guests’ experience within an exhibit space and therefore cannot be tied to an exact moment or feature within the exhibit. We have clarified this limitation within the revised manuscript. [1] Santini, Thiago, et al. "The art of pervasive eye tracking: unconstrained eye tracking in the Austrian Gallery Belvedere." Proceedings of the 7th workshop on pervasive eye tracking and mobile eye-based interaction. 2018. [2] Herlitz, Mattias. "Analyzing the Tobii Real-world-mapping tool and improving its workflow using Random Forests." (2018).= [3] Niehorster, Diederick C., et al. "The impact of slippage on the data quality of head-worn eye trackers." Behavior Research Methods (2020): 1-21. We appreciate these suggested references and have included two in the revised manuscript. We have opted not to include the Herlitz reference as it is a master’s thesis examining the real-world mapping tool in Tobii which we did not utilize for the current study. https://doi.org/10.1371/journal.pone.0239204.r002
2 Sep 2020 Decision Letter - Enkelejda Kasneci, Editor Exploring the Awe-some: Mobile eye-tracking insights into Awe in a science museum PONE-D-20-00804R1 Dear Dr. Krogh-Jespersen, We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication. An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org. If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. Kind regards, Enkelejda Kasneci, Ph.D. Academic Editor PLOS ONE Additional Editor Comments (optional): Reviewers' comments: https://doi.org/10.1371/journal.pone.0239204.r003
Formally Accepted
9 Sep 2020 Acceptance Letter - Enkelejda Kasneci, Editor PONE-D-20-00804R1 Exploring the Awe-some: Mobile eye-tracking insights into Awe in a science museum Dear Dr. Krogh-Jespersen: I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department. If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org. If we can help with anything else, please email us at plosone@plos.org. Thank you for submitting your work to PLOS ONE and supporting open access. Kind regards, PLOS ONE Editorial Office Staff on behalf of Dr. Enkelejda Kasneci Academic Editor PLOS ONE https://doi.org/10.1371/journal.pone.0239204.r004

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .