PONE-D-23-30762A cascading model for nudging employees towards energy-efficient behaviour in tertiary buildingsPLOS ONE

Dear Dr. Casado-Mansilla,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. The crucial seems overcoming Reviewer #2 objections.

​Please submit your revised manuscript by Dec 14 2023 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Krzysztof Malarz, D.Sc., Ph.D., M.Sc.

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at 

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Note from Emily Chenette, Editor in Chief of PLOS ONE, and Iain Hrynaszkiewicz, Director of Open Research Solutions at PLOS: Did you know that depositing data in a repository is associated with up to a 25% citation advantage (https://doi.org/10.1371/journal.pone.0230416)? If you’ve not already done so, consider depositing your raw data in a repository to ensure your work is read, appreciated and cited by the largest possible audience. You’ll also earn an Accessible Data icon on your published paper if you deposit your data in any participating repository (https://plos.org/open-science/open-data/#accessible-data).

3. Please note that PLOS ONE has specific guidelines on code sharing for submissions in which author-generated code underpins the findings in the manuscript. In these cases, all author-generated code must be made available without restrictions upon publication of the work. Please review our guidelines at https://journals.plos.org/plosone/s/materials-and-software-sharing#loc-sharing-code and ensure that your code is shared in a way that follows best practice and facilitates reproducibility and reuse.

4. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match. 

When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section.

5. Thank you for stating the following financial disclosure: 

"Ministry of Economy, Industry and

Competitiveness of Spain for IoP, under Grant No.: PID2020-119682RB-I00"

Please state what role the funders took in the study.  If the funders had no role, please state: ""The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript."" 

If this statement is not correct you must amend it as needed. 

Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf.

6. Thank you for stating the following in the Acknowledgments Section of your manuscript: 

"This work is partially funded by the European Union’s Horizon 2020 Research and

Innovation Programme through the projects WHY (Grant Agreement No. 891943),

SOCIO-BEE (Grant No.: 101037648), and GREENGAGE (Grant No.: 101086530).

Furthermore, we express our gratitude to the German Federal Ministry of Education

and Research (BMBF) and the Greek General Secretariat for Research and Technology

(GSRT) in the context of the Greek-German Call for Proposals that funded the

SIT4Energy project. Besides, we acknowledge the Ministry of Economy, Industry and

Competitiveness of Spain for IoP, under Grant No.: PID2020-119682RB-I00"

We note that you have provided funding information that is not currently declared in your Funding Statement. However, funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form. 

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows: 

"Ministry of Economy, Industry and

Competitiveness of Spain for IoP, under Grant No.: PID2020-119682RB-I00"

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

7. In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

""Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

We will update your Data Availability statement to reflect the information you provide in your cover letter.

8. We note that you have stated that you will provide repository information for your data at acceptance. Should your manuscript be accepted for publication, we will hold it until you provide the relevant accession numbers or DOIs necessary to access your data. If you wish to make changes to your Data Availability statement, please describe these changes in your cover letter and we will update your Data Availability statement to reflect the information you provide.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

Reviewer #3: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: No

Reviewer #3: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

Reviewer #3: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This research designs a recommendation framework to identify individualized persuasion strategies that encourage energy-efficient behaviors in building occupants. Using a large dataset from four European countries, the framework ranks ten expert-preselected strategies via matrix factorization and random forest regression models. It demonstrates higher accuracy than non-personalized methods and offers insights into the most effective strategies based on user data.

Overall, the paper needs signficant improvements on the following aspects:

1) The literature review of this article is very terse. Many recent studies discussing energy consumption behavioral change has not been discussed in the literature review. The authors should elaborate more on this aspect by discussing the following studies: Smart Sensing and End-User Behavioral Change in Residential Buildings: An Edge Internet of Energy Perspective; Is there any room for renewable energy innovation in developing and transition economies? Data envelopment analysis of energy behaviour and resilience data; The emergence of explainability of intelligent systems: Delivering explainable and personalized recommendations for energy efficiency; Intelligent edge-based recommender system for internet of energy applications; Understanding Waste Management Behavior Among University Students in China: Environmental Knowledge, Personal Norms, and the Theory of Planned Behavior; A novel approach for detecting anomalous energy consumption based on micro-moments and deep neural networks; Reshaping consumption habits by exploiting energy-related micro-moment recommendations: A case study; REHAB-C: Recommendations for Energy HABits Change; A micro-moment system for domestic energy efficiency analysis; Using big data and federated learning for generating energy efficiency recommendations; Smart fusion of sensor data and human feedback for personalized energy-saving recommendations; IoT based smart and intelligent smart city energy optimization

2) Kindly fix the formatting problem in page 2.

3) While the sample size of 678 users across four countries is commendable, the authors should discuss the extent to which the findings are generalizable to wider populations, especially outside Europe.

4) The self-reported nature of the data can introduce biases. The authors might consider addressing this aspect and discussing potential limitations of the self-reported data.

5) The paper briefly mentions increased accuracy over non-personalized methods but lacks an in-depth comparison or analysis. This comparison would be beneficial for readers to gauge the actual improvement the proposed framework offers.

6) The conclusion highlights the potential of the framework, but there is a lack of discussion about the practical challenges or considerations in deploying such a system in real-world settings.

Reviewer #2: The authors propose a cascading/fusion approach to better predict the best nudging/persuasion strategy for individual users in the context of energy efficiency. The presented analysis is based on survey data, which was partly previously collected and partly collected in the context of the present research. The reported experiments indicate that combining the outcomes of two prediction models is synergistic and leads to rankings that are closer to the "ground-truth" ranking than other approaches (a random one and a popularity/frequency-based one).

The work is generally well prepared and easy to follow. The topic itself is relevant. The discussion of related works seems appropriate.

Overall, however, I think the contribution to the existing body of literature of this work seems too limited. The work extends the authors' own previous works, and the main addition seems to lie in the combination of two prediction models (and some additional data that is collected). The outcome of the combination is positive. For a journal publication, I would expect a much deeper analysis. I appreciate that the authors discuss limitations of their work in various respects. Still, I would have hoped that the paper delivers more than trying out some combination of predictors on a dataset (which also has some major limitations).

Unfortunately, the authors decided to not share the code and the data, which makes it impossible for others to validate the findings. The authors argue that they do not want to do this for privacy/ethical reasons, but it remains unclear why it is impossible to a) share the code for the analyses and b) share anonymized data.

Other remarks:

1) There is some difference between nudging and persuasion, according to the literature. Should be discussed.

2) It would be helpful if the distribution of top-1 ranked strategies in the original data would be shown. In general, a deeper analysis of the underlying data seems advisable.

3) The data itself, as mentioned by the authors, might be rather noisy or even biased, depending, e.g., how things were interpreted by the participants.

4) Pre-processing and feature selection: The authors perform feature selection for a given classification task, but the classification task is not explicitly described.

5) The threshold of 80% of keeping data seems a bit arbitrary.

6) "which are then fused sequentially to get aggregated results" -> it is not so clear at this stage what "fused sequentially" would mean.

7) "Specifically, it extends MF by including the latent factors of users (socio-economic features) and items" -> The paper is not self-contained here. It is also not clear at this stage what the elements of the matrix are which should be factorized.

8) Fusion: A variety of list combination strategies are possible. Why was this particular fusion strategy applied?

9) Experiments: The assumption of several analyses is that only one top-ranked strategy per user. In reality, also considering noise, there might be two or more equally suited strategies per use. The evaluation approach thus seems a bit narrow. Could be discussed.

10) Experimental scenarios: Two datasets are used, leading to different results. A deeper investigation is missing. Also, I wonder if it is meaningful to compare the used metrics on an absolute scale.

11) Figure 3 and the corresponding analysis: Why is the NDPM used here as the only measure? Why are different measures used in the other experiment? Why not reporting F1 here as well?

12) "Conducting a T-SNE to the user embeddings obtained from the model" -> Where do the user embeddings come from here?

13) "meaning that the predictions are an average of 1-1.5 units away from the true scores" -> Is this true? It is not the MAE, which can be easily interpreted in absolute terms, but here the authors use the RMSE.

14) Table 4: Now here we use different forms of the F-score. It is quite surprising that the combination of two rather weak predictors (eg 0.02 and 0.03) leads to a combined value that is multiple times higher (0.11). Unfortunately, no code and data is provided to verify this rather surprising result.

Minor:

• The writing is a bit wordy at times, starting with a quite long abstract.
• Jesse et al. -> Jesse and Jannach
• "is low enough to make the model useful in a real setting." -> How do we know?
• Starke et al. worked on nudging in the energy efficiency domain, e.g. [a]. Their works could be discussed.

[a] Alain D. Starke, Martijn C. Willemsen, Chris Snijders: With a little help from my peers: depicting social norms in a recommender interface to promote energy conservation. IUI 2020: 568-578

Reviewer #3: The manuscript is well written and is structured well. The research aims at providing data-driven personalized approaches for persuasion towards energy efficient behavior. The manuscript demonstrates robust statistical analysis and provides a good description of the methodology derived form the previous works.

A few suggestions to improve the paper-

1) Discuss what are the implications on different stakeholders- e.g. facility managers (optimized operations, lesser complaints?), occupants (requires more engagement or understanding of system/people), owners (operating costs), etc.

2) Were there any methods applied to test reporting biases?

3) How do you envisage this framework can be replicated in specialized settings - such as schools, senior living homes etc.

4) Can you comment on the micro f-score being slightly better for MVR than RF ( for fusion) in Tables 4 and 5?

5) Figure 3 What does the x and y axis represent?

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: Yes: Jeetika Malik

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PONE-D-23-30762R1A cascading model for nudging employees towards energy-efficient behaviour in tertiary buildingsPLOS ONE

Dear Dr. Casado-Mansilla,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please follow Reviewer #2 comments on revised manuscript. 

Please submit your revised manuscript by Apr 17 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Krzysztof Malarz, D.Sc., Ph.D., M.Sc.

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: (No Response)

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #2: (No Response)

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: I Don't Know

Reviewer #3: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #2: Yes

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #2: I thank the authors for the detailed answers to my questions. I am generally satisfied with the responses. For some of the remarks, however, the authors only added explanations in the authors' response, but the paper was not improved or changed (even though there apparently were questions that other readers might have as well).

I appreciate that the authors make an effort to make the code and data of their research available.

I am still not too convinced about the strength of the contribution of this paper compared to the authors previous work, but as no other reviewer questioned this, I will trust their assessment.

Here are a few remarks:

* The links to code repository should be included in the paper.

* There are issues in the GitHub repository: The "models" folder, for example, does not contain models, but some data-related things. I also miss a proper README that describes which code should be run to obtain the results reported in the paper. Given the huge increase in accuracy when combining two models, it might be helpful when a third party can validate this.

* The new discussion of the relation between persuasion and nudging needs some backing references. Also, I am not convinced by the statement "Both persuasion and digital nudging involve personalisation". Nudging does not necessarily involve personalization, and persuasion does not either.

Minor:

* This sentence seems broken: "Over the years, and as occupant behaviour has been vastly recognised as a crucial factor for Energy Efficiency"

* Sentence should be revised: "being this consumption behavior particularly"

* "behavioral" (page 2, check use of British vs American English)

* "Moreover, understanding waste management behavior". What is "waste management behavior"?

* The S1 file is mentioned, but no reference is provided where we can access it. (I could access it through the editorial system, though)

* "the Prolific platform 1." -> extra space before footnote.

* "they don’t correspond", "However, it’s important to", "or we don’t do" -> contractions like "don't" should be avoided.

* "Moreover highlights that tailored messages based" -> something is missing here.

Reviewer #3: The authors have addresses all the comments well, the paper can now be accepted in its current form for publication.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: No

Reviewer #3: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

A cascading model for nudging employees towards energy-efficient behaviour in tertiary buildings

PONE-D-23-30762R2

Dear Dr. Casado-Mansilla,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Krzysztof Malarz, D.Sc., Ph.D., M.Sc.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

When submitting the final version of the manuscript please take care on two issues mentioned by Reviewer:

- The newly added discussion of "Nudging vs. Persuasion" is set in quotes; this should be fixed.

- I suggest to add a link to the Zenodo repository to the paper.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: I Don't Know

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: No

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #2: Thank you for the revisions and for improving the GitHub repository. I only have two remarks:

* The newly added discussion of "Nudging vs. Persuasion" is set in quotes; this should be fixed. Also, the references have not been properly resolved "[?]".

* I suggest to add a link to the Zenodo repository to the paper.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: No

**********