PCLM-D-24-00031

Can Large Language Models Capture Public Opinion about Global Warming? An Empirical Assessment of Algorithmic Fidelity and Bias

PLOS Climate

Dear Dr. Peng,

Thank you for submitting your manuscript to PLOS Climate. After careful consideration, we feel that it has merit but does not fully meet PLOS Climate’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Overall, the reviewers and I are enthusiastic about the potential contribution of this paper to climate science and social science in explicating the promises of LLMs in public opinion research and analysis. That said, there are opportunities for the authors to address constructive feedback from the reviewers. Please pay close attention to Reviewer #1's recommendation to provide a more thorough discussion of the risks/limitations and benefits, including how LLMs could be complementary (and supplementary) to traditional survey research.

Along the same lines, Reviewer #2 urges the author to provide more discussion on why the models perform poorly in some cases. Reviewer #2 raises an important question about whether alignment processes between GPT-3.5 vs. GPT-4 reduce the ability of the models to replicate specific kinds of questions related to global warming. Moreover, I suggest that the authors take seriously Reviewer #2's constructive feedback on the need to explicate the differences between models that do and do not have attitudinal covariates. Reviewer #2 provide specific advice on how to do this more adequately. 

Please submit your revised manuscript by May 16 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at climate@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pclm/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

We look forward to receiving your revised manuscript.

Kind regards,

Lily Hsueh, Ph.D.

Associate Professor, Arizona State University

Academic Editor, PLOS Climate

Journal Requirements:

1. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

2. Please note that PLOS Climate has specific guidelines on code sharing for submissions in which author-generated code underpins the findings in the manuscript. In these cases, all author-generated code must be made available without restrictions upon publication of the work. Please review our guidelines at https://journals.plos.org/climate/s/materials-and-software-sharing#loc-sharing-code and ensure that your code is shared in a way that follows best practice and facilitates reproducibility and reuse.

3. Please upload all main figures as separate Figure files in .tif or .eps format only. For more information about how to convert and format your figure files please see our guidelines:

https://journals.plos.org/climate/s/figures

Additional Editor Comments (if provided):

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Does this manuscript meet PLOS Climate’s publication criteria? Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe methodologically and ethically rigorous research with conclusions that are appropriately drawn based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available (please refer to the Data Availability Statement at the start of the manuscript PDF file)?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception. The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS Climate does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: I would like to express my congratulations on your engaging and important paper concerning the suitability of utilizing large language models (LLMs) for assessing public opinion on climate change. Your study stands out for its meticulous execution and methodological robustness, contributing valuable insights to the field.

However, I would like to encourage the authors to provide a more in-depth critique of the potential risks and rewards inherent in employing LLMs for simulating public opinion. For instance, your findings reveal that GPT-4 exhibits an accuracy, from 53% to 91%, in mirroring beliefs and emotional responses concerning global warming. It is noteworthy that for certain demographic segments, this accuracy diminishes significantly. Despite the anticipation that future advancements in LLMs might enhance accuracy, the intrinsic biases present in LLMs training data and human feedback are likely to perpetuate skewed and unreliable outcomes, particularly for groups that are underrepresented in the training datasets or overlooked during model refinement through human feedback.

The concern extends to the representation of global perspectives, particularly from lower-income and developing nations, which may not be adequately captured due to limited training data and skewed human feedback. This situation echoes the limitations sometimes encountered in high-quality survey research, which, despite rigorous and replicable methodological approaches, may not fully capture public opinion. Considering that even high-quality survey-based methods encounter such difficulties, the accuracy metrics presented in your study underscore the potential for unrealistic conclusions derived from LLM-based simulations of public opinion.

I concur with your conclusion that LLMs hold promise for exploratory hypothesis generation, the design, and pre-testing of survey-and field-based research. Moreover, LLMs could potentially be useful as a supplementary data source or for validating findings in conjunction with traditional data gathering methods like surveys. Nevertheless, I harbor reservations regarding their utility in confirmatory empirical research, given the substantial risk that researchers and practitioners may increasingly gravitate towards cheaper, albeit less reliable, LLMs for simulating public opinion, potentially at the expense of more robust methods.

In conclusion, beyond the technical assessment of LLMs' capability to reflect public opinion, a more thorough exploration of the risks and benefits and the practical utility of LLMs in the realm of public opinion research on climate change is warranted. Such a discussion would greatly enhance the manuscript's contribution to the discourse on the application of LLMs in social science research.

Reviewer #2: I am reviewing the manuscript “Can Large Language Models Capture Public Opinion about Global Warming?” In this manuscript, the authors build on prior research about algorithmic fidelity and the potential for LLMs to accurately reflect political attitudes by extending the analysis to attitudes about climate change and comparing two new cutting-edge models (GPT-3.5 & GPT-4). They find that the LLMs do demonstrate a capacity for accurate reflection of public opinion about climate change, but they also highlight some important contingencies in that capacity. Specifically, the LLMs perform much better when provided with relevant attitudinal information (as opposed to relying only on demographic covariates). Also, GPT-4 performs worse than GPT-3.5 without the attitudinal covariates, and the best with them.

There are a lot of very commendable aspects to this project. The manuscript is detailed and very clearly written. Of course, there are a hundred other design and prompt decisions the authors could have made, but the ones they do make are clear, reasonable, and well-justified in the text. The authors do a very good job of walking the line between being clear about the contribution this manuscript makes (which is meaningful) without overclaiming or making a point that is too general to be supported by the studies they have conducted. The manuscript adds to both the substance (climate change opinions) and the methodological details about LLMs as silicon samples that exists in current published work. The pilot test replication of the voting data is also a very nice addition to verify their method and the models’ baseline performances. I think it is a strong manuscript that merits publication.

I have two relatively small recommendations for the authors as they finish revising. First, I understand that there is a lot that remains opaque in the training process and adjustments made to the OpenAI models. The authors' points on this in the conclusion are well-made. However, I do think the authors could do a little more to discuss why they believe the models perform poorly in some cases. For example, how should we think about the differences between 2017 and 2021, and GPT-3.5 vs. GPT-4? Are there more general principles that might serve as hypotheses for future work? Do alignment processes reduce the ability of the models to replicate specific kinds of questions related to global warming? The authors bring this up as a motivation for this study, but I think it deserves a little more follow through in the discussion of variation across models and prompts.

Second, I think the authors could do a little more to describe the differences between models that do and do not have attitudinal covariates in the prompts. On some level, it is not surprising that the models with only demographics perform relatively poorly as predictors, because – although there are small correlations for some of these items (judging from the human sample baseline on Figure 4), they are not strong predictors of the climate attitudes measured. The exception to this is of course ideology and party, which are not (as the authors note) strictly speaking demographic traits. Because of this, I would like to see Figure 4 expanded to include the attitudinal measures that are also used in the “with covariates” models. Additionally, I think if the authors spend a little more time developing that fact that the models with covariates work better precisely because the added covariates are more strongly correlated with the attitude they are predicting than the demographics were, they can work that into a broader point that the predictions work the best when provided with the most relevant background information. I occasionally encounter folks in this space who leave out the most relevant predictors because they feel like they would be too closely correlated to the outcome variable they are trying to predict. I think the authors here have an opportunity to demonstrate a really import

Can Large Language Models Capture Public Opinion about Global Warming? An Empirical Assessment of Algorithmic Fidelity and Bias

PCLM-D-24-00031R1

Dear Dr Peng,

We are pleased to inform you that your manuscript 'Can Large Language Models Capture Public Opinion about Global Warming? An Empirical Assessment of Algorithmic Fidelity and Bias' has been provisionally accepted for publication in PLOS Climate.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow-up email from a member of our team. 

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they'll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact climate@plos.org.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Climate.

Best regards,

Lily Hsueh, Ph.D.

Associate Professor, Arizona State University

Academic Editor, PLOS Climate

***********************************************************

Reviewer Comments (if any, and for reference):

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Does this manuscript meet PLOS Climate’s publication criteria? Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe methodologically and ethically rigorous research with conclusions that are appropriately drawn based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available (please refer to the Data Availability Statement at the start of the manuscript PDF file)?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception. The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS Climate does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Dear Authors,

Thank you very much for addressing all of my comments.

I am looking forward to seeing this paper published.

Reviewer #2: The authors did a nice job addressing all of my concerns, and the ethics concerns of Reviewer 1 (I thought the suggestions and revisions here were also excellent. I think the manuscript is ready for publication.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

Do you want your identity to be public for this peer review? If you choose “no”, your identity will remain anonymous but your review may still be made public.

For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

**********