PCOMPBIOL-D-25-02477

Scaffold-Lab: Critical Evaluation and Ranking of Protein Backbone Generation Methods in A Unified Framework

PLOS Computational Biology

Dear Dr. Chen,

Thank you for submitting your manuscript to PLOS Computational Biology. After careful consideration, we feel that it has merit but does not fully meet PLOS Computational Biology's publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Apr 03 2026 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at ploscompbiol@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pcompbiol/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

* A letter that responds to each point raised by the editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. This file does not need to include responses to formatting updates and technical items listed in the 'Journal Requirements' section below.

* A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

* An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, competing interests statement, or data availability statement, please make these updates within the submission form at the time of resubmission. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter

We look forward to receiving your revised manuscript.

Kind regards,

Gaetano Montelione, Ph.D.

Academic Editor

PLOS Computational Biology

Nir Ben-Tal

Section Editor

PLOS Computational Biology

Journal Requirements:

If the reviewer comments include a recommendation to cite specific previously published works, please review and evaluate these publications to determine whether they are relevant and should be cited. There is no requirement to cite these works unless the editor has indicated otherwise.

1) Please provide an Author Summary. This should appear in your manuscript between the Abstract (if applicable) and the Introduction, and should be 150-200 words long. The aim should be to make your findings accessible to a wide audience that includes both scientists and non-scientists. Sample summaries can be found on our website under Submission Guidelines:

https://journals.plos.org/ploscompbiol/s/submission-guidelines#loc-parts-of-a-submission

2) Please upload all main figures as separate Figure files in .tif or .eps format. For more information about how to convert and format your figure files please see our guidelines:

https://journals.plos.org/ploscompbiol/s/figures

3) We have noticed that you have uploaded Supporting Information files, but you have not included a list of legends. Please add a full list of legends for your Supporting Information files after the references list.

4) Some material included in your submission may be copyrighted. According to PLOSu2019s copyright policy, authors who use figures or other material (e.g., graphics, clipart, maps) from another author or copyright holder must demonstrate or obtain permission to publish this material under the Creative Commons Attribution 4.0 International (CC BY 4.0) License used by PLOS journals. Please closely review the details of PLOSu2019s copyright requirements here: PLOS Licenses and Copyright. If you need to request permissions from a copyright holder, you may use PLOS's Copyright Content Permission form.

Please respond directly to this email and provide any known details concerning your material's license terms and permissions required for reuse, even if you have not yet obtained copyright permissions or are unsure of your material's copyright compatibility. Once you have responded and addressed all other outstanding technical requirements, you may resubmit your manuscript within Editorial Manager.

Potential Copyright Issues:

i) Figure 1. Please confirm whether you drew the images / clip-art within the figure panels by hand. If you did not draw the images, please provide (a) a link to the source of the images or icons and their license / terms of use; or (b) written permission from the copyright holder to publish the images or icons under our CC BY 4.0 license. Alternatively, you may replace the images with open source alternatives. See these open source resources you may use to replace images / clip-art:

- https://commons.wikimedia.org

- https://openclipart.org/.

5) Please amend your detailed Financial Disclosure statement. This is published with the article. It must therefore be completed in full sentences and contain the exact wording you wish to be published.

1) State the initials, alongside each funding source, of each author to receive each grant. For example: "This work was supported by the National Institutes of Health (####### to AM; ###### to CJ) and the National Science Foundation (###### to AM)."

2) State what role the funders took in the study. If the funders had no role in your study, please state: "The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript."

3) If any authors received a salary from any of your funders, please state which authors and which funders..

If you did not receive any funding for this study, please simply state: u201cThe authors received no specific funding for this work.u201d

6) Please ensure that the funders and grant numbers match between the Financial Disclosure field and the Funding Information tab in your submission form. Note that the funders must be provided in the same order in both places as well.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: On pg. 4 of SI it appears that some text might be missing. The authors write:

"We additionally adapted two modifications: "

after which the text moves onto a new paragraph.

Reviewer #2: In this manuscript, “Scaffold-Lab: Critical Evaluation and Ranking of Protein Backbone Generation Methods in A Unified Framework,” Authored by Zheng et al., the authors comprehensively evaluated protein backbone generation methods across various protein sizes and scaffolds and provided a systematic framework for evaluating Protein design methods. They have used extensive properties to rate and rank the available methods for protein design into multiple categories. They found that RFdiffusion performed most consistently across proteins of varying lengths. However, based on their evaluation metrics, the authors have concluded that none of the discussed methods is well-suited to all categories. Overall, these methods are biased towards generating more structured proteins and have limited capabilities to design loops and intrinsically disordered regions. They suggest users must design a framework tailored to the length and biological function of their protein sequence.

An interesting point to note is that, according to the evaluation in this paper, the authors conclude that novelty in a designed protein structure comes at the expense of its designability, questioning the emphasis on novelty in protein backbone generation methods. In general, low designability indicates that the structure is farther from the existing protein, and hence, the feasibility of in vivo expression of such proteins might be problematic. Hence, the authors have questioned the overemphasis on the novelty of designed proteins and suggest that designability of the motif scaffold is a prerequisite when users are interested in a certain biological function.

Reviewer #3 (Academic Editor)

Zheng et al describe Scaffold-Lab, a unified framework for systematic evaluation of protein  backbone generation methods. They evaluated several representative methods and provide a detailed analysis of their performance and utilities. Overall, RFdiffusion performed the best among all tested methods. Other methods also demonstrated reliable performance on short (< 300 residues) proteins

Considering the rapid advances in AI methods for de novo protein design, the authors are correct to point out there is a  need for standardized benchmarks to assess protein backbone generation methods. Scaffold-Lab provides unified framework for the evaluation of protein backbone generation methods. The results clearly demonstrate the challenge for most methods is "designability" for targets > 300 residues, and the unique strengths and weaknesses of RFDiffusion. The criteria for assessing various methods and protocols are also useful. Although the study is not as comprehensive as one might like, it does provide a significant and valuable effort in this direction.

A revised manuscript should address the following issues.

1. The authors state in the final paragraph of the Introduction: "We observed that the ability to create an ideal scaffold is a prerequisite of successful designs than simply meeting certain conditional criteria" The language here is not clear. Since this is a key point - it should be made more clearly.

2. For "structural properties", since the proteins are refolded with ESM_Fold or AlphaFold, could the authors also consider an empirical packing score like Molprobity? This has the advantage also that it is quantifiable.

3. In Methods the author state "For each unique length within each method, we generated 100 backbones with 10 sequences for each backbone to be evaluated through a refolding pipeline." This seems to relate to the Results of Fig 2 and 3, but not for the motif-scaffolding study which uses the 24-case benchmark of RFdiffusion. Perhaps this point is better made in the legends of Fig 2 and 3

4. Panel 2F. The only stars shown are ***. It appears contradictory that the distributions for Medium RFDiffusion with and without penalty look nearly the same, but have p < 0.001, while for Medium FrameDiff the distributions look different but are not statistically different. Is this all correct?

5. Both the Analysis of Structural Properties and the last paragraph of Discussion do not properly distinguish "ordered non-regular structures" from "intrinsically-disordered" regions. These are treated as synonymous, which they are not. Are the designed regions designated as "coil" non-regular ordered regions or disorder regions?

6. Ranking. It would be nice to have a histogram plot for Mean Ci vs method, ordered by decreasing Ci score. This would make it clear which methods were best scoring. One histogram plots could combine the data for short, medium, and long lengths.

7. The observed correlation between successful non-conditional designability and successful conditional motif scaffolding is very important, though perhaps obvious to some. The paper would be stronger if more additional assessments were made of different approaches to conditional design, and/or applications to specific cases of binder or active site design.  It seems that exploring strengths and weaknesses of different methods and their parameter choices in conditional design is a tremendous value of Scaffold-Lab.  Although additional studies addressing this are not required, the authors should at least discuss this point in a bit more detail.

8. Although a web server is not provided, there appears to be an extensive github site which will allow scientists to use Scaffold-Lab in their research

Minor points

page 3. Cartesian should be capitalized

Supplementary Materials should be capitalized throughout

Fig.3  Define "coll percentage" on Y axis

At several points, the English needs some refinement. For example:

page 17. "Recently studies leveraging diffusion models to generate intrinsically disordered regions of proteins have shown great potentials applicated in both protein..."

Fig 3 legend. "All data was curated by directly calculating onto protein backbones without refolding"

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Figure resubmission:

While revising your submission, we strongly recommend that you use PLOS’s NAAS tool (https://ngplosjournals.pagemajik.ai/artanalysis) to test your figure files. NAAS can convert your figure files to the TIFF file type and meet basic requirements (such as print size, resolution), or provide you with a report on issues that do not meet our requirements and that NAAS cannot fix.

After uploading your figures to PLOS’s NAAS tool - https://ngplosjournals.pagemajik.ai/artanalysis, NAAS will process the files provided and display the results in the "Uploaded Files" section of the page as the processing is complete. If the uploaded figures meet our requirements (or NAAS is able to fix the files to meet our requirements), the figure will be marked as "fixed" above. If NAAS is unable to fix the files, a red "failed" label will appear above. When NAAS has confirmed that the figure files meet our requirements, please download the file via the download option, and include these NAAS processed figure files when submitting your revised manuscript.

Reproducibility:

To enhance the reproducibility of your results, we recommend that authors of applicable studies deposit laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

Dear Prof. Dr. Chen,

We are pleased to inform you that your manuscript 'Scaffold-Lab: Critical Evaluation and Ranking of Protein Backbone Generation Methods in A Unified Framework' has been provisionally accepted for publication in PLOS Computational Biology.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests.

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us now if you or your institution is planning to press release the article. All press must be co-ordinated with PLOS.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Computational Biology.

Best regards,

Gaetano Montelione, Ph.D.

Academic Editor

PLOS Computational Biology

Nir Ben-Tal

Section Editor

PLOS Computational Biology

***********************************************************

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: The MolProbity clash score analysis is a welcome addition to the paper and clearly highlights the advantages imparted by design models that contain van der Waal loss penalizing functions.

The the active site-conditioned design benchmark nicely highlights some of the present difficulty in designing enzymes using this approach.

The addition of a Colab Notebook environment will also greatly enhance accessibility and dissemination of this work.

Reviewer #3: The authors have adequately addressed all the issues I raised.

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: None

Reviewer #3: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #3: Yes: Gaetano T. Montelione