PONE-D-24-22791Black box ‘consensus meeting’: Describing the differences between MSCA consensus meetings and Bayesian ranking recommendationsPLOS ONE

Dear Dr. Heyard,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Sep 12 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Annesha Sil, Ph.D.

Associate Editor

PLOS ONE

Journal requirements:   

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for stating the following in the Competing Interests section: 

[Rachel Heyard conceptualised and implemented the Bayesian ranking methodology used here while she was employed at the Swiss National Science Foundation. David G. Pina is employed by the European Research Executive Agency. Ivan Buljan has no competing interests to declare. Ana Marušić occasionally serves as an expert evaluator of grant proposals for the European Commission.]. 

Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, by including the following statement: "This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests).  If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared. 

Please include your updated Competing Interests statement in your cover letter; we will change the online submission form on your behalf.

3. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Additional Editor Comments (if provided):

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

Reviewer #2: Yes

Reviewer #3: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: "The title of the paper seems a bit too jargon-laced, which might impact its visibility and downplay its importance. Perhaps a title that better captures the big picture upsides of this analysis, about algorithmic vs. grant panel approaches, and impact on science funding policy would be more fruitful."

Black Box ‘Consensus Meeting’: Describing the Differences Between MSCA Consensus Meetings and Bayesian Ranking Recommendations

This paper provides a cogent contribution to questions about the roles of consensus meetings in grant review, their relationship to individual reviewer reports, lotteries, and considers an alternative more ‘algorithmic’ approach. This study is interesting and carefully crafted, and I recommend it for publication. I have a number of suggested revisions, all of which are fairly minor. Thank you to the authors for their contribution to science funding policy methodology.

(1) There are a few references missing in the framing of the paper. Some work has been done on consensus meetings and reducing discrepancies caused by gender bias (Bol, T., de Vaan, M. and van de Rijt, A. [2022]: ‘Gender-Equal Funding Rates Conceal Unequal Evaluations’, Research Policy, 51, available at <doi.org 10.1016="" 2021.104399="" j.respol.="">.), and some work has been done on the inefficiencies of consensus meetings (Erosheva, E., Grant, S., Chen, M., Lindner, M., Nakamura, R. and Lee, C. J. [2020]: ‘NIH Peer Review: Criterion Scores Completely Account for Racial Disparities in Overall Impact Scores’, Science Advances, 6, available at <doi.org 10.1126="" sciadv.aaz4868="">.) and their implications for lotteries (Lee, C. J., Grant, S. and Erosheva, E. [2020]: ‘Alternative Grant Models Might Perpetuate Black–White Funding Gaps’, The Lancet, 396, pp. 955–56.).

(2) “for some an additional expert had to be consulted to find a consensus.”

- For clarification, would the additional expert be brought in only in cases with large discrepancies in initial scores?

(3) “to reach a consensus…”

- Is the purpose of these meetings to reach a consensus? If so, this seems worth noting (with specific reference to instructions to reviewers, if available/possible). Some grant review panels aren’t aiming at consensus explicitly, but rather just giving referees chances to hear their colleagues opinions.

(4) “with the most important budget…”

- Does this mean the largest budget?

(5) It might be interesting to briefly note that there don’t seem to be large differences of acceptance/reserve/reject rates despite changes in the amount of reviews required differing by quite a bit.

(6) “In this setting, the individual evaluation report yij can be interpreted as the expert’s best guess or estimation of the proposal’s true quality θi”

- I understand that this is a controversial point and don’t expect the authors to make major revisions to their methodology on this point, but I do think its worth highlighting that this conception of true quality is controversial. For example, see Shaw, J. 2023. “Peer Review, Innovation, and Predicting the Future of Science: The Scope of Lotteries in Science Funding Policy.” Philosophy of Science, 90(5): 1297-1306 for the argument that true quality is really a retrospective judgment, rather than expert’s best guess, that provides a proposals true quality (although I am adding my interpretation to this paper).

(7) “the budget generally determines the number of proposals x that can be funded”

- I assume that the budget is fixed prior to the reviewing process, so there’s no wiggle room for budget negotiations? If so, this could be made explicit.

(8) “Each scientific panel has its own grading behaviour”

- Maybe a word or two here adding some specifics would be helpful here.

(9) “We would argue that, being such a poor predictor for the consensus report or funding ranking, they are not useful”

- It might be worth noting related studies. In the Erosheva et al. study mentioned above, they found that 43% of reviewers changed their preliminary scores as a result of panel discussions.

(10) “None of our findings can be generalised to other funding agencies or funding schemes as the analysis focused on a very specific MSCA funding scheme, on two specific call years and tree specific scientific panels”

- This might be overstating the case. Perhaps a more modest warning, like saying that generalizations should be cautious to consider the specifics of the MSCA funding scheme, would be appropriate.</doi.org></doi.org>

Reviewer #2: The manuscript highlights an important and underdeveloped area of funding practices. The authors have adequately reported the uncertainty in this field and suitably reference past evidence in this space. I have some additional comments under each of the manuscript headings, see below.

For those who may not know what the MSCCA is I think this needs to be reported in full in the title.

Abstract: There is no background in the abstract to put this research into context. Understanding the unmet need and the reason for the research/study would strengthen the abstract.

Introduction: Providing some context around describing the allocation of funding as simplistic and ignoring the underlying uncertainty needs elaboration. It is not clear what this is referring to and given the existing evidence in this space, I would suggest updating the 2008 reference used. The authors adequately provide context around alternative approaches and how this current work fits into the wider context. Perhaps, providing the three research questions to the introduction (or methods) would make it easier to see the purpose of the analysis.

Methods: the authors provide a thorough account of the approach taken. Who conducted the analysis and the software used for analysis is not reported though, which would help others to replicate in the future. Perhaps, move the research questions to the beginning of the methods section so the reader is able to see this first. Although there is a data analysis subheading there is minimal information provided, albeit a link to OSF protocol. Listing the questions outlined in the protocol would be very useful in the manuscript (at the end of the introduction or as part of the methods). Having them earlier in the manuscript would then support the information detailed in Box 1.

Why was the data from 2015-2019? As this is now 5 years ago and have processes/practices changed during this period. Explaining why the analysis was with 5 year old data needs to be reported within the manuscript to be able to place the analysis in the wider context.

The three research questions are adequately reported in the results section, along with tables and figures to further explain the results of the analysis.

The discussion relates back to the three research questions, although I'm unsure of the second paragraph and the justifications around the usefulness of individual evaluation reports. Further explanation is required, to back up the statement based on the results of the analysis. It is also challenging to understand what this paragraph is trying to say, and its relation to existing evidence. The authors provide limitations of the analysis, although the last three sentences appear out of place. It would be better placed as a final paragraph to summarise the analysis in the wider context of funding allocation, how funders can use this in future considerations (along with caveats) and what does it mean in terms of bias. As noted, points raised in the introduction around the challenges with

funding allocation (inefficient, biased, 'black box') need to be picked up in the discussion. In particular, the 'black box' terminology, mentioned in the introduction is not mentioned throughout the manuscript.

Using algorithms to attempt to remove bias and inefficiencies also has limitations, which is mentioned in the existing literature. Balancing funding allocation approaches is challenging, yet can complement each other given the right context. I feel the authors could pick up these opportunities in the manuscript more and tense out how and in what circumstances it would be acceptable to use algorithms to inform funding allocation processes and ensure these practices are transparent, fair and of high quality. How this analysis aligns and offers these opportunities more widely could be strengthened in the manuscript.

Minor corrections: There are typo errors on the figures such as 1,006 instead of 1'006 and 7,870 instead of 7'870

Typo error - [Pina et al., 2015, Graves et al. [2011], Erosheva et al. [2021]].

Reviewer #3: Technical Soundness:

The manuscript is partly technically sound. The methodology, involving a Bayesian hierarchical model to compare individual evaluation reports (IER) and consensus meeting reports (CR), is robust and well-chosen for the research questions posed. The study design includes appropriate controls and comparisons, with an adequate sample size across multiple panels and call years. However, while the Bayesian model is rigorously applied, the generalizability of the findings to other funding schemes and contexts remains uncertain. Detailed case studies and further validation of the model using additional datasets could enhance the robustness of the conclusions.

Statistical Analysis:

The statistical analysis has been performed appropriately and rigorously. The use of a Bayesian hierarchical model to account for variability and uncertainty in the data is commendable. The manuscript provides a clear explanation of the statistical methods used, and the comparative analysis between the Bayesian ranking and consensus meetings is thorough and well-supported by the data.

Data Availability:

The authors have made all data underlying the findings in their manuscript fully available. This transparency is crucial for the reproducibility of the research. The data is openly accessible through the provided links, ensuring that other researchers can verify and build upon the study's findings.

Manuscript Presentation:

The manuscript is presented in an intelligible fashion and written in standard English. The structure is logical, and the writing is clear, making the complex statistical methodologies and findings accessible to a broad audience. However, the manuscript could benefit from a more detailed discussion on the potential biases in both the Bayesian model and consensus meetings.

Additional Comments:

Generalizability: The study focuses on the MSCA funding scheme and a limited number of panels. A discussion on how the findings could be generalized to other funding mechanisms would be beneficial.

Bias Analysis: A thorough analysis of potential biases in expert evaluations and consensus discussions could provide valuable insights and strengthen the manuscript's conclusions.

Future Research: Outlining specific future research directions to test the applicability of the findings in other contexts and with different types of funding mechanisms would be a valuable addition.

While the manuscript is technically sound and well-written, addressing a few areas could enhance its clarity and robustness. Specific improvements include a more detailed discussion on the generalizability of findings, potential biases in both the Bayesian model and consensus meetings, and future research directions. Addressing these points would make the manuscript even stronger.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PONE-D-24-22791R1Assessing the potential of a Bayesian ranking as an alternative to consensus meetings for decision making in research funding: A Case Study of Marie Skłodowska-Curie ActionsPLOS ONE

Dear Dr. Heyard,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Nov 29 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Irfan Ullah, PhD

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: All comments have been addressed

Reviewer #4: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #2: Yes

Reviewer #4: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: N/A

Reviewer #4: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: Yes

Reviewer #4: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #2: Yes

Reviewer #4: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #2: (No Response)

Reviewer #4: 1 In the introduction section, the author is requested to discuss the theoretical and methodological foundations of existing research, the gaps in existing research, and the areas in which this article will advance existing theories and methods.

2 In the conclusion section, please have the author discuss the theoretical and methodological contributions of this article.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: Yes: A J Blatch-Jones

Reviewer #4: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Assessing the potential of a Bayesian ranking as an alternative to consensus meetings for decision making in research funding: A Case Study of Marie Skłodowska-Curie Actions

PONE-D-24-22791R2

Dear Dr. Heyard,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Sergio A. Useche, Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

The paper can be accepted in its current form. Thanks for the amendments and clarifications.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: I Don't Know

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #2: Thank you for addressing the feedback, I have no further comments to add to the revised version. The authors have revised the manuscript according to the feedback they have received.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: Yes: Amanda Blatch-Jones

**********