PONE-D-23-15697Automated stance detection in complex topics and small languages: the challenging case of immigration in polarizing news mediaPLOS ONE

Dear Professor Mark Mets,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by 30 October 2023. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Pantea Keikhosrokiani

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf.

2. Thank you for stating in your Funding Statement:

“M.M., A.K., I.I., and M.S. are supported by the CUDAN ERA Chair project for Cultural Data Analytics at Tallinn University, funded through the European Union Horizon 2020 research and innovation program (Project No. 810961). The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. M.M., A.K. and I.I. received funding from Estonian media publishing company AS Ekspress Grupp (https://www.egrupp.ee/en/). The funder provided part of the data but had no role in study design and analysis, decision to publish, or preparation of the manuscript”

Please provide an amended statement that declares *all* the funding or sources of support (whether external or internal to your organization) received during this study, as detailed online in our guide for authors at http://journals.plos.org/plosone/s/submit-now.  Please also include the statement “There was no additional external funding received for this study.” in your updated Funding Statement.

Please include your amended Funding Statement within your cover letter. We will change the online submission form on your behalf.

3. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match.

When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section.

4. We notice that your supplementary figures are uploaded with the file type 'Figure'. Please amend the file type to 'Supporting Information'. Please ensure that each Supporting Information file has a legend listed in the manuscript after the references list.

5. We notice that your supplementary tables are included in the manuscript file. Please remove them and upload them with the file type 'Supporting Information'. Please ensure that each Supporting Information file has a legend listed in the manuscript after the references list.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: No

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The manuscript explores the application of large language models (LLMs) for automated stance detection in the context of immigration in polarizing news media, focusing on the Estonian language. The study involves annotating a dataset of pro and anti-immigration examples, comparing the performance of multiple LLMs as supervised learners, and investigating diachronic trends over a seven-year period in two corpora of Estonian mainstream and right-wing populist news sources.

The article covers an important and timely topic, and the research objective is clear. The use of LLMs for automated stance detection in a challenging scenario is relevant. However, there are areas where the manuscript could be improved:

1. Abstract: The abstract should provide a concise summary of the study, including the research objective, methodology, main findings, and implications. It currently lacks specific details about the dataset size, performance metrics, and significant findings.

2. Introduction: The introduction provides a good overview of the research context, but more explicit statements about the research gap and the significance of the study must be included. Please clearly articulate how the study contributes to existing literature and highlight the novelty and potential impact of using LLMs for stance detection in lower-resource languages.

3. Methodology: Provide more details about the annotation procedure, including the guidelines given to the human annotators. Explain how inter-annotator agreement was measured and addressed. Also, authors should describe the fine-tuning process of the LLMs, including the hyperparameters used. Please specify the evaluation metrics employed and discuss their appropriateness for the task.

4. Results: The authors evaluate the performance of different models and analyze the changes in stances over time. But below are some suggestions for improvement:

o Provide more details about the dataset used for training and evaluation. Include information such as the size of the dataset, data sources, and any preprocessing steps applied.

o When discussing the performance of different models, you can consider including statistical significance tests to determine if the observed differences are statistically significant.

o Perhaps you can explain the practical implications of achieving an F1 macro score of 0.66. How does it compare to existing methods or benchmarks in the field? Are there specific applications where this level of accuracy is considered acceptable?

5. Limitations:

o Please clarify the limitations attributed to human annotations. What specific challenges were encountered with human annotators? How can these limitations be addressed or minimized in future studies?

o Discuss the potential biases in the dataset and how they might impact the classifier's performance. You may consider exploring methods to mitigate biases or evaluate the impact of biases on the results.

o Please provide more details on the annotation instructions given to the human annotators. How were they trained? Did they receive any feedback or clarification during the annotation process? Clear instructions can help improve the quality of annotations.

6. Exemplary Analysis:

1. I think it is necessary to give a brief overview of the diachronic analysis methodology used for analyzing changes in stances over time. We need to understand the approach taken and the significance of the findings.

2. When discussing the exemplary analysis, more specific interpretations of the findings are needed. For example, what are the implications of the observed trends in stance changes? How do these trends align with existing literature or theories on media polarization and immigration discourse?

7. Future Research:

1. Elaborate on the suggestion of using generative models like ChatGPT for annotating data or augmenting existing annotations. How would this approach work in practice? What are the potential benefits and challenges?

2. Discuss the potential limitations or risks of relying solely on generative models for annotation. Are there any concerns related to biases or the reliability of annotations produced by these models?

3. Provide more details on the proposed evaluation of zero-shot learning and the necessity of annotated datasets. How would the accuracy of zero-shot learning be evaluated? In what scenarios would annotated datasets still be necessary?

Reviewer #2: 1. In the introduction section, mention the key contributions of this text/speech classification point by point.

2. There are a number of classifiers, why did the author choose this particular classifier for this classification problem?

3. Classification accuracy is not too high, how this model can be effective in practice?

4. There are some typos and grammatical anomalies in various sections in this article, overall language quality shall be improved.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Automated stance detection in complex topics and small languages: the challenging case of immigration in polarizing news media

PONE-D-23-15697R1

Dear Dr. Mets,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Pantea Keikhosrokiani

Academic Editor

PLOS ONE