PONE-D-22-33837

Dynamics of (mis)information flow and engaging power of narratives

PLOS ONE

Dear Dr. Brugnoli,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

==============================

ACADEMIC EDITOR:

Please address the comments provided by the reviewers.

==============================

Please submit your revised manuscript by Apr 29 2023 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Ankit Gupta

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. In your Methods section, please include additional information about your dataset and ensure that you have included a statement specifying whether the collection and analysis method complied with the terms and conditions for the source of the data.

3. In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

We will update your Data Availability statement to reflect the information you provide in your cover letter.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: No

Reviewer #2: Partly

Reviewer #3: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: I Don't Know

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: No

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: In short, the authors need to dig in a lot more to the methods to better understand the biases introduced in their data collection on both production and consumption. Conditional on that, I find the key findings in Figure 3 and 4 compelling, but really need the authors to update their figure before I can really confirm. I understand the style of moving methods into a follow-up section after the discussion, but the paper is unreadable due to too much key parts in the methods section.

1) I am still very confused about the core data collection methods, even after re-reading the introduction and methods several times. What I think the authors did was go onto Crowtangle and the API for Twitter and YouTube and search for keywords around vaccines. They then took content that was from publications (discarding any posts) and graded quality or not based on publisher level. But, Figure 1 shows percent of all content? So, maybe they collect all links? Either way, way: what are they doing to validate their keywords (recall and precision of their labelling). Presumably they get some idea of false positives when hand coding for topic and position. And, what are they doing to deal with issues of images and videos.

2) The authors are using Facebook, Instagram, Twitter, and YouTube to measure engagement with content: what exactly are they measuring? In the methods they recognize that these platforms are measuring very different types of engagement, and they are measuring them differently. For example, CrowdTangle only includes URL with greater than 100 public shares. What the author are capturing is very different from views and biased, in many ways, towards click-bait or more sensational, low quality information. The authors should address this!

3) In Figure 1 (and assorted tables): percent of all content is highly related to how they are defining the corpus. This is especially important for reliable content: are the authors including sports and entertainment sections of the papers? Questionable content is more likely constrained to politically oriented publications?

4) For Figure 2 the authors are trying to measure how engaging vaccine related content is within a publication compared to other content, and within vaccine related content by follower. I find the terms confusing, but that is obviously not critical. The discovery of content is conditional on engagement making these measures troubled. For an article to be discovered through CrowdTangle it had to be publicly shared 100+ times! For the “outside” perspective, the authors are using “followers”, presumably on the platform? How well does this corelate with consumption and engagement? I assume low quality publications get followed very differently than high-quality publications.

5) The model is predicting the labeled position (1 of 3) with 88% in the evaluation set? That is sort of shockingly good. Conditional on all my previous concerns over the corpus, I find the left side of Figure 3 engaging and meaningful.

6) Similarly, accepting a sufficient level of accuracy, Figure 4 is also has the potential be meaningful. Interesting to see the different topics by publication type.

Reviewer #2: By identifying impact of misinformation authors are contributing a good quality of research but few things needs to be considered while representation of the paper.

1. Highlighting the state of the art contribution is important.

2. Abstract is not summarizing the exact unique contribution and goal of the work is not clear from abstract as well as introduction.

3. It requires strengthening paper presentation in several parts, including an introduction (Where is the novelty of the methods, explain it clearly)

4. The proposed model should include the proposed formulation

5. more emphasis is required on how the use of Deep learning model contributes to achieve results.

6. The authors should add the conclusion part with all the descriptions and results of this article.

7. The author can improve this paper by adding more current and fit references.

Reviewer #3: The paper seems very sound in its scientific method. It is also clearly written apart from a few places where the English could be improved (eg p7 line150). My concern has to do with the conceptual framework. In its title the paper claims to engage with "the power of narratives", yet while there are references to scientific literature dealing with "false narratives", nowhere does it offer anything like a systematic theory of narrative. It is hard to know where to begin in offering feedback on how to address this shortcoming, as there are so many different theories of narrative. The authors might consult Tangherlini et al https://doi.org/10.1371/journal.pone.0233879 which offers a good example of how a scientific publication for this journal can manage to combine a full discussion of its quantitative experimental method, while at the same time scientifically laying out a (qualitative) theory narrative, and showing how it applies that theory to its quantitative experiment. This, I think, is the standard to which the journal aspires. Finally, a smaller issue has to do with the opening conceptual framework of 'the public sphere', which I do not see as informing the analysis. The primary reference is 60 years old, with the more secondary reference being to a somewhat obscure Marxist figure. There has been so much written on this framework, that I question the need for opening this can of worms at all, when it seems like the paper is more interested in (polarized) political opinion than in the normative public sphere. As I say, the core problem is one of clarifying the framework, with the biggest issue pertaining to having an actual the theory of narrative, and secondarily getting clearer on the whole business of the public sphere and how that highly normative argument informs the argument of the paper as a whole—if the authors want to hold onto the latter, at the very least they'd need to come back to it in the conclusion.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PONE-D-22-33837R1Dynamics and triggers of misinformation on vaccinesPLOS ONE

Dear Dr. Brugnoli,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

The article is considerably implemented compared to its first version. However, a further revision step is necessary in order to further improve the methodological and argumentative aspects (see comments below). 

Please submit your revised manuscript by Oct 26 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Andrea Cioffi

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: (No Response)

Reviewer #4: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #4: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #4: I Don't Know

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #4: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #4: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: 1) Why do the authors use 96 of 121 major Italian newspapers, rather than all 121? Can the authors break out this bulleted breakdown in Questionable and Reliable as shown in Table 1? I am most interested to know if all questionable come from online news or not?

2) I am not sure how I am supposed to think of transfer entropy in the pre-pandemic period. I do not know much about Italian vaccine laws, but judging by Figure 1, questionable sources start shooting up on coverage at a very similar time to the reliable sources, peak a still very low number, then peak after the law. For the rest of the time the value is near 0, to the point where inferences are basically built on noise? Meanwhile questionable sources have a valley between their peaks and then drop down before the reliable’ s second peak. I just want to understand if this time period, were all the action is, is driving the entropy results, or are they picking up on the relatively tiny movements over the rest of the timeframe? Either way, it may may sense to isolate that time period more, rather than the full pre-COVID period. I say this assuming that in the pre-COVID time period most of the people consuming anti-vax material were seeking it out with likely high personal demand for the material, making the production less interesting than the in the two times of acute demand during the new law and COVID.

3) E = interactions / (content * followers), and the authors use variations of this to see how content about vaccines and non-vaccines categorization, and/or between publications (or source types) within category do.

3a) I had to spend way too long to figure out exactly what the authors were doing, and how to interpret the plots, and I am still not sure I am reading everything correctly. In the writing the authors are leaning into differential interactions per content & followers, but they have created a pretty convoluted way to do it. I would like the authors to tie the findings in the paper directly to what they are referring to in the figures. I spent too much time searching around, especially as they are referring to specific values that are not directly noted on the charts.

3b) I assume many of the questionable content providers have very low follower counts, which could really impact the key findings?

3c) I assume higher interactions per content/followers were happening pre-pandemic because the content was more uniquely sensational, then the what was being produced during the pandemic (i.e., it is influenced both the much lower quantity of content and the click-baity nature of it). Could definitely think more about the crowding-out issues that happen during the pandemic.

I really like the work. The modelling is compelling. And the findings are pretty straight forward descriptive stats, hidden by a needlessly complex framework, that kept making me go back and forth between the methods, writing, and plots.

Reviewer #4: I find this manuscript very interesting and usefull for the scientific community involved in the study and promotion of the adequate use of vaccines. I am not an exprt in mathematical or deep learning models used in this paper, so I cannot evaluate the methodology used. To my best knowledge, the problem had a correct approach and the results provide important information regarding ways to get involved in proposing convincing counter-narratives to misinformation spread. Very important the results regarding the lack of engagement in the rhetoric of vaccine safety. We all should participate and learn from this study.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #4: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Dynamics and triggers of misinformation on vaccines

PONE-D-22-33837R2

Dear Dr. Brugnoli,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Andrea Cioffi

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

No further revisions are necessary.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: (1) Old figure 2 is still in the manuscript, although I believe it is no longer in the paper.

(2) I still think the new Figure 2 is confusing, but I will leave that up to the editor if they think it is adequate for publication.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

**********