PONE-D-24-59142Adaptive questionnaires and GPT-4: Tackling the cold start problem with simulated user interactionsPLOS ONE

Dear Dr. Bachmann,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Mar 09 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Carlos Carrasco-Farré

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for stating the following financial disclosure:

“Support of the Swiss National Science Foundation (SNSF) under grant ID CRSII5-205975 provided the primary funding for our research. Additionally, this work was partially supported by the Digital Society Initiative (DSI) of the University of Zurich through a grant from the DSI Excellence Program.”

Please state what role the funders took in the study.  If the funders had no role, please state: "The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript."

If this statement is not correct you must amend it as needed.

Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf.

3. Thank you for stating the following in the Acknowledgments Section of your manuscript:

“We gratefully acknowledge the support of the Swiss National Science Foundation (SNSF) under grant ID CRSII5-205975, which provided the primary funding for our research. Additionally, this work was partially supported by the Digital Society Initiative (DSI) of the University of Zurich through a grant from the DSI Excellence Program. We also extend our thanks to the team from Politools for providing the Smartvote data, and to David Camorani for his invaluable advice on scientific writing.”

We note that you have provided funding information that is currently declared in your Funding Statement. However, funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form.

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows:

“Support of the Swiss National Science Foundation (SNSF) under grant ID CRSII5-205975 provided the primary funding for our research. Additionally, this work was partially supported by the Digital Society Initiative (DSI) of the University of Zurich through a grant from the DSI Excellence Program.”

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

Additional Editor Comments:

Dear Authors,

Thank you for submitting your manuscript to PLOS ONE, which addresses the use of Large Language Models (LLMs) to mitigate the cold start problem in adaptive questionnaires. The reviewers have provided valuable feedback, which highlights both the strengths of your work and areas requiring substantial revision. After careful consideration of their comments, I am recommending a major revision for your manuscript, with confidence that addressing the concerns raised will significantly improve the quality and impact of your work. Below, I summarize the main points raised by the reviewers, grouped into major and minor concerns, to guide your revision process. However, please, provide an answer for each of the comments raised by reviewers, regarding of my highlighting.

Major Concerns to Address

1. Experimental Design and Evaluation Metrics

Reviewer 1 emphasized the need for a clearer rationale for selecting evaluation metrics such as RMSE and Candidate Recommendation Accuracy (CRA). Additionally, the inclusion of complementary metrics (e.g., groundedness or faithfulness scores) is suggested to provide a more comprehensive assessment of the system’s performance.

2. Generalizability Beyond the Political Domain

Your experiments focus exclusively on the Swiss political domain. Reviewer 1 recommends either conducting experiments in other domains (e.g., education or healthcare) or providing an in-depth discussion about potential challenges and adaptations when applying your approach to other fields.

3. Synthetic Data Diversity and Limitations

Reviewer 1 noted the limitations of GPT-4-generated data, particularly its tendency to lack diversity and perform poorly in edge cases. A more quantitative analysis of how these limitations affect model performance—especially in non-typical scenarios—is needed.

4. Decay Parameter (γ) Analysis

Reviewer 1 highlighted the insufficient analysis of the decay parameter (γ). While you mention that a sensitivity analysis will be presented in a future paper, some discussion and practical guidance for parameter tuning in real-world applications are needed in the current work.

5. Statistical Testing of Data Distributions

Reviewer 2 pointed out that while the downstream performance of GPT-generated data is presented as the gold standard, statistical tests distinguishing the distributions of synthetic and real data should be included. This would strengthen your claim that LLM-generated data approximates real-world distributions.

6. Consistency in Experimental Setup

Reviewer 2 questioned inconsistencies in user dropout points in Fig. 4A and Fig. 4B. Clarifying the rationale for these differences and aligning the setups, if possible, would enhance the paper's coherence.

Minor Concerns and Suggested Improvements

Line 276: Reviewer 2 suggests adding a plot showing decision boundaries for all questions on the same PCA plot to confirm alignment with principal dimensions.

Line 277: Provide a rationale for binarizing Likert scores or explore alternatives, such as using normalized scores directly.

Line 329: Clarify discrepancies in values in Table 3 (e.g., 0.165 vs. 0.191).

Line 334: Explain the source of missing values.

Figures: Address issues such as untranslated legends (e.g., Fig. 7) and missing data points (e.g., Fig. 7 discussion). Ensure clarity and consistency across all figures.

Party Labels (Line 210): For readers unfamiliar with Swiss politics, label parties as left/right and liberal/conservative in relevant figures and discussions.

Typographical Errors: Address minor errors, such as missing words (e.g., “accuracy” in Line 484) and incorrect terms (e.g., "LMM" instead of "LLM" in Line 512).

Editorial Decision and Next Steps

The reviewers have recognized the potential and novelty of your work while also identifying significant gaps that need to be addressed. I encourage you to revise the manuscript thoroughly, incorporating the feedback from both reviewers. A systematic response to each comment will be essential to demonstrate how their concerns have been addressed.

Once the revised manuscript is submitted, it will be re-evaluated to ensure that the revisions adequately address the reviewers' concerns. We believe that with these changes, your manuscript will make a meaningful contribution to the field and advance the understanding of LLM applications in adaptive questionnaires.

We look forward to receiving your revised manuscript. Should you have any questions about the reviewers' feedback or the revision process, please do not hesitate to contact us.

Best regards,

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The draft presents an alternative approach to mitigate the cold start problem in adaptive questionnaires using Large Language Models (LLMs). The work is methodologically interesting and indicates some potential of LLMs to generate training data. However, I suggest a major revision to address certain issues and enhance the overall quality of the research contribution. Below, I outline four specific areas for improvement:

The experimental design of the study needs further elaboration, particularly regarding the evaluation metrics. While Root Mean Squared Error (RMSE) and Candidate Recommendation Accuracy (CRA) are utilized effectively, the paper does not justify why these metrics were chosen over others, such as Groundedness Score or faithfulness (a measure to evaluate hallucination probability). This lack of discussion leaves the reader unclear on whether the selected metrics fully capture the performance of the adaptive questionnaire in all its dimensions. To improve the rigor of the evaluation, the authors should provide a clear rationale for their choice of metrics and consider including complementary measures that might give a more holistic view of the system's performance.

Another aspect of the experimental design that requires attention is the generalizability of the proposed method. The paper's experiments focus exclusively on the Swiss political domain, which raises questions about whether the findings can be applied to other areas where adaptive questionnaires are used, such as education or healthcare. The lack of broader validation limits the applicability of the study's conclusions. It would strengthen the paper by conducting additional experiments in diverse domains or providing a detailed discussion about the potential challenges and adaptations required for applying the method beyond political surveys.

The paper also highlights a limitation in the synthetic data generated by GPT-4: it tends to be overly centered and lacks the diversity seen in real-world data, potentially leading to overfitting or poor performance in edge cases. However, this limitation has not been thoroughly analyzed. The authors could provide a more quantitative investigation into how this affects the performance of their models, particularly in scenarios with outlier user profiles or other non-typical cases. Such an analysis would add depth to the discussion and demonstrate a complete understanding of the method's limitations.

Finally, the paper introduces a decay parameter, γ, for the GPTreplace dataset, which determines how quickly the synthetic data is replaced by real user interactions. While this parameter is important for the system's performance, its impact is not systematically analyzed, and the paper does not offer clear guidance on how it should be set in practice. Even authors claim that a complete sensitivity analysis of this parameter will be done in a separate paper, adding some discussion on practical tuning strategies, would greatly enhance the utility of the study for practitioners seeking to apply the method in real-world scenarios.

By addressing these points, the authors can improve the robustness, clarity, and applicability of their work, ensuring it has a broader impact in the field.

Reviewer #2: This paper addresses the cold start problem in adaptive questionnaires by using GPT-4 to generate synthetic training data, demonstrating how LLMs can effectively simulate political survey responses to improve early-stage questionnaire performance. Strengths include clear hypotheses, good experimental design with various GPT-generated datasets, two downstream tasks evaluated, and detailed discussion of results and limitations.

Main concerns:

1. Line 276: A plot where the decision boundaries for all questions shown on the same plot would be interesting. The reason is that if most boundaries align with the first or the second principal dimension, then we know that both the LR model and the choice of using PCA with 2 components are making sense.

2. Line 277: Is binarization necessary? Can you directly use the normalized Likert score as a feature? What's the rationale behind this?

3. Line 329: These numbers are 0.165 and 0.191 respectively in Table 3. Please clarify

4. Line 334: What is causing the missing values?

5. Fig 4: Fig 4 does show that the cold start problem exists and the proposed method is able to alleviate it. Once question though is why in Fig 4A users answer 10 questions before dropout while in Fig 4B users answer 40 questions before dropout. Would it be better to keep this consistent? Curious about the rationale about this.

6. Line 490: The discussion on the observed trend for break-even point for CRA could be improved. I would say that when K is very small (<20), there is no way to accurately estimate each user's characteristics, hence the benefit of the initial training data is not evident. My understanding is not necessarily correct but I think is more clear.

7. Statistical tests for distinguishing distributions: This paper includes several demonstrations of how GPT generated data is close to the real data distribution, but no statistical tests are performed. Although the downstream performance is the gold standard, at least we can know if statistical tests are useful at all in gauging the quality of LLM-generated synthetic data.

Minor issues

1. Line 194: exactly how many candidates and voters are there?

2. Line 210: for people unfamiliar with Swiss parties, can you label which parties are left/right and liberal/conservative? You might also want to label the axes in the figure.

3. Line 349: SP also lies outside the Gaussian fit

4. Line 484: missing the word “accuracy”

5. Line 512: LMM -> LLM

6. In some of the figures, the legend is Swedish. Consider replace them with English, or make sure that the translation is provided in the caption or related discussion in the main text. Otherwise it causes confusions and guessing, e.g. in Fig. 7 the Green party.

7. Fig 7: what is the standard deviation of GPT4 generated answers? The std of the candidates' answers show interesting patterns across parties. For example, for Mitte, the std is the largest for questions with the most balanced answers, as indicated by the convex shape of the silhouette of all std bars. Another example is Green, where the concave shape shows that the questions with extreme answers have the most variance. It would be interesting to see if GPT4 can also mimic this, and thus further justify the use of LLMs.

8. Fig 7 discussion: For question 32253, the black circle seems to be missing. And for question 32238, there seems to be no circle pointing it. So I’m not sure where to find it in Fig 7.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Adaptive political surveys and GPT-4: Tackling the cold start problem with simulated user interactions

PONE-D-24-59142R1

Dear Dr. Bachmann,

Thank you for submitting your revised manuscript entitled "Adaptive political surveys and GPT-4: Tackling the cold start problem with simulated user interactions" to PLOS ONE.

We have now received the expert reviewers’ evaluations of your revised submission. I am pleased to inform you that all reviewer concerns have been thoroughly addressed, and your manuscript has been accepted for publication.

Congratulations on this achievement. We appreciate the effort you and your co-authors have invested in preparing and revising your work. Your contribution will be a valuable addition to the literature on adaptive survey methodologies and the use of large language models in political science.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Carlos Carrasco-Farré

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Thank you for submitting your revised manuscript entitled "Adaptive political surveys and GPT-4: Tackling the cold start problem with simulated user interactions" to PLOS ONE.

We have now received the expert reviewers’ evaluations of your revised submission. I am pleased to inform you that all reviewer concerns have been thoroughly addressed, and your manuscript has been accepted for publication.

Congratulations on this achievement. We appreciate the effort you and your co-authors have invested in preparing and revising your work. Your contribution will be a valuable addition to the literature on adaptive survey methodologies and the use of large language models in political science.

Should you have any questions about the next steps, including production or publication timelines, please don’t hesitate to contact us.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: All my comments have been addressed by the authors in this second version of the manuscript.

Reviewer #2: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

**********