PONE-D-25-46997Multimodal Generative AI for Automated Pavement Condition Assessment: Benchmarking Model PerformancePLOS ONE

Dear Dr. Cui,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Nov 20 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Junghwan Kim

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf.

2. Please note that PLOS One has specific guidelines on code sharing for submissions in which author-generated code underpins the findings in the manuscript. In these cases, we expect all author-generated code to be made available without restrictions upon publication of the work. Please review our guidelines at https://journals.plos.org/plosone/s/materials-and-software-sharing#loc-sharing-code and ensure that your code is shared in a way that follows best practice and facilitates reproducibility and reuse.

3. Thank you for stating the following financial disclosure:

“CESU grant

Funding Opportunity No: W912HZ-19-SOI-0002”

Please state what role the funders took in the study.  If the funders had no role, please state: "The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript."

If this statement is not correct you must amend it as needed.

Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf.

4. In the online submission form, you indicated that your data will be submitted to a repository upon acceptance.  We strongly recommend all authors deposit their data before acceptance, as the process can be lengthy and hold up publication timelines. Please note that, though access restrictions are acceptable now, your entire minimal  dataset will need to be made freely accessible if your manuscript is accepted for publication. This policy applies to all data except where public deposition would breach compliance with the protocol approved by your research ethics board. If you are unable to adhere to our open data policy, please kindly revise your statement to explain your reasoning and we will seek the editor's input on an exemption.

5. If the reviewer comments include a recommendation to cite specific previously published works, please review and evaluate these publications to determine whether they are relevant and should be cited. There is no requirement to cite these works unless the editor has indicated otherwise.

Additional Editor Comments (if provided):

Please ensure to address the reviewers' comments thoroughly. Additionally, please expand the literature review of recent papers on this topic to discuss gaps in the existing studies and why those gaps are significant. There are many papers on the similar or relevant topics (i.e., LLM + Street-view images) published in geography and urban planning journals, which have not been extensively discussed in this manuscript.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

Reviewer #3: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: Yes

Reviewer #3: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This paper proposes using large language models (LLMs) to audit pavement condition. I think many technical details need to be described and considered more carefully.

Abstract

1. Page 1: The abstract starts by saying LLMs show promise in automated assessment of many things, but automated pavement condition assessment is unexplored. I don’t think this is a strong motivation (there are countless unexplored applications of LLMs). Instead, the authors should start with current challenges in pavement condition assessment and then explain how LLMs could help address these challenges.

Introduction

2. Page 2 (lines 36-39): Proper references should be cited for existing work on “structural evaluation” and “surface-level evaluation.” The authors should also specify which physical characteristics these studies focus on, with a few examples.

3. Page 3 (lines 46-56): Several claims here seem factually inaccurate. First, when you say existing deep learning models require “high-quality, structured image data,” do you mean training data? Even for LLMs, you still need high-quality image data to support detection and decision-making. Second, while deep learning models need human oversight, so do LLMs, which can hallucinate and require human review (so they can’t truly autonomously make decisions). Perhaps the authors mean that a single LLM could potentially handle not just detection but also evaluation of severity and maintenance timing, tasks that would require multiple deep learning models.

Methods

4. Page 7: First, what are the 39 prompts used? These should be spelled out either here or in an appendix, as it is currently unclear what the authors are asking the LLMs to do. Second, how are the model parameters set, such as temperature, top-k, etc.? Third, for each LLM, is it only prompted once per street view image? Typically, to enhance robustness in LLM-assisted assessment, each LLM should be prompted multiple times per image, because responses can vary due to inherent randomness.

5. Page 8: More details are needed about the manual annotation process. For example, where were the annotators recruited? Provide a detailed list of labels used for surface features and spatial distribution patterns, not just examples. The examples given here could instead go in the Introduction (page 3, lines 57–63), because terms like “spatial distribution pattern” are unclear until this point.

6. Page 11: How do you distinguish between response correctness and hallucination in practice? Both result in a mismatch between human-annotated data and LLM responses.

7. Echoing my earlier points, the authors should be explicit about what they instruct the LLMs to do, what outputs are expected, and what is contained in the human-annotated data. Without this clarity, it is difficult to understand how performance evaluation is conducted. How do we know whether the human-annotated results and LLM responses are comparable?

Reviewer #2: The manuscript titled “Multimodal Generative AI for Automated Pavement Condition Assessment: Benchmarking Model Performance” addresses an important and timely problem by exploring the application of multimodal generative AI to pavement condition assessment. The study provides a clear comparative evaluation of multiple proprietary and open-source models and presents results that highlight relative performance differences across tasks and dimensions. Here are my comments.

- While the study demonstrates that multimodal generative AI models can produce pavement maintenance recommendations, one important limitation not addressed in the manuscript is the lack of explainability of model outputs. For infrastructure management such as pavement condition assessment, where decisions directly affect safety, costs, and policy priorities, explainability is critical. Without interpretable reasoning or transparent links between input features (e.g., crack severity, traffic data) and recommendations, practitioners may be reluctant to trust or adopt such AI systems. The authors should discuss how LLMs address this.

- As one of the limitations of manual inspections, the paper noted that manual inspections are subject to observer variability (ln 40-41). This is also applicable in utilizing generative AI too. Many models generate probabilistic outputs, meaning repeated runs may not always yield identical results unless parameters are fixed. This needs to be acknowledged in this study

- For the 39 prompts used in this study, are there any domain expertise involved in their design? Given that prompt clarity and completeness strongly affect model outputs, it will benefit this paper to clarify whether experts in pavement management were consulted, and what criteria guided prompt construction.

- Response rate was used as one of the assessment dimensions. The paper does not clearly explain how response rate was measured in practice. Was it calculated manually?

Reviewer #3: The manuscript presents a timely contribution to the growing field of using MLLMs for imagery analysis. The integration of GenAI into road surface monitoring is novel, and the systematic evaluation across different proprietary and open-source models provides useful insights. The study is well motivated and relevant to both urban/transportation planning as well as AI research. But overall, the manusscript is not yet ready for publication. With clearer descriptions, stronger methodology, and better figure clarity, it could become a solid contribution to AI in transportation infrastructure. MAJOR REVISIONS REQUIRED.

Abstract:

23-25: Clear but the claim that GPT 4o provides the most favorable balance between accuracy and cost should be carefully stated

Introduction:

- HIghlights the importance of pavement condition assessment and the potential of generative AI, but it does not sufficiently discuss the study in relation to existing work.

- prior work in deep learning and computer vision for crack detection and PCI estimation mentioned but does not reference or contrast with these efforts in detail

- novelty of benchmarking multimodal generative models is not sharply presented

- The introduction would benefit from more explicitly stating how this study differs from prior research

Methods

- dataset size is small: needs justification

- 157–160: “ground truth” labels for maintenance interval based on temporal comparisons of Google Street View imagery, which may not reliably capture the actual timing or type of repairs. This introduces potential uncertainty in the benchmark labels

- 162-166: Manual annotation procedure is not described in enough detail. How many? Was inter-rater agreement measured?

- 205-208: definition of appears very subjective. any formal coding protocol or double-checking used?

- Statistical testing is ok, but effect sizes should also be reported to quantify practical differences, not jsut p values

Results:

- GPT-4o and OpenAI o1 are highlighted as top-performing, ut GEmma 3 is under-discussed even though it reached 100% response in one task, Being open-source, might be more important for reproducibility and cost.

- Hallucination analysis would benefit from systematic quantification across tasks and models

- the figures are difficult to read at this scale

Limitations:

- Actual pavement management requires more than just surface imagery?

- Fine tuning and prompt engineering are mentioned but not fully discussed in domain specific needs

- Comparison between Open source and proprietary could be expanded

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PONE-D-25-46997R1Multimodal Generative AI for Automated Pavement Condition Assessment: Benchmarking Model PerformancePLOS One

Dear Dr. Cui,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Jan 26 2026 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Junghwan Kim

Academic Editor

PLOS One

Journal Requirements:

If the reviewer comments include a recommendation to cite specific previously published works, please review and evaluate these publications to determine whether they are relevant and should be cited. There is no requirement to cite these works unless the editor has indicated otherwise.

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Additional Editor Comments:

I have received all the reviewers' reports and completed the evaluation of the revised manuscript. Reviewer #2 still has an outstanding item. Please address this. Thank you!

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: (No Response)

Reviewer #2: Many thanks to the authors for addressing my comments. I can see that the new version is well improved. All my concerns have been addressed. However, I recommend that the authors ensure that the claims added are adequately supported by appropriate references before the final version is published.

Reviewer #3: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

To ensure your figures meet our technical requirements, please review our figure guidelines: https://journals.plos.org/plosone/s/figures

You may also use PLOS’s free figure tool, NAAS, to help you prepare publication quality figures: https://journals.plos.org/plosone/s/figures#loc-tools-for-figure-preparation.

NAAS will assess whether your figures meet our technical requirements by comparing each figure against our figure specifications.

Multimodal Generative AI for Automated Pavement Condition Assessment: Benchmarking Model Performance

PONE-D-25-46997R2

Dear Dr. Cui,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. For questions related to billing, please contact billing support.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Junghwan Kim

Academic Editor

PLOS One

Additional Editor Comments (optional):

Reviewers' comments: