PONE-D-24-15144Enabling preprint discovery, evaluation, and analysis with Europe PMCPLOS ONE

Dear Dr. Levchenko,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. First of all I want to thank the 2 reviewers for their very useful comments that helped me to reach a decision. I choose major revisions in order to account for the opinion of reviewer 2, but I also acknowledge that most of the comments seems pretty easy to address. Therefore I'm quite confident that it will be easy for you to answer. I have a few additional "editorial" comments:- please make sure that you indicate wether your protocol was pre-registered or not. If not, please give an explanation and add some words in the limitation section.- please add a few words about limitations in the abstract and add a specific "limitations" section that summarize the main limitations of your work. This is in order to avoid any spin ; - please select the most appropriate guideline from the equator network (I agree that there is no guideline that has a perfect fit with your research but you may want to choose the closest one) ;  

Please submit your revised manuscript by Jul 18 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Florian Naudet, M.D., M.P.H., Ph.D.

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: N/A

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: In this important report, the authors share their experiences with indexing life science preprints, in an effort to promote the discoverability and acceptance of preprints. The authors comprehensively describe how they automated the ingestion of preprints to the Europe PMC database and its enrichment with metadata, including full texts available for some special collections related to COVID-19 or supported by Europe PMC funders. The manuscript includes useful search strategies that can assist researchers in their queries, including programmatic access via different APIs. The authors make a compelling case for a central preprint database that combines data from major preprint servers in a standardized manner, despite the technical difficulties associated with this endeavor.

Minor points:

Lines 71-76: It was unclear from the description and analysis code whether the total number of articles in PubMed includes preprints or not. If it does, then it is appropriate to refer to the share and percentage of preprints. If not, then the measure is preprints per article and not a percentage, nor a share. Please clarify if the number of preprints is included in the total. Incidentally, when I ran the code, the result was exactly 12 and not over 12, but this is probably due to updates in the database.

Figure 2 and lines 90-91: The numbers were higher than 30 only for the first two months, so it would be more precise to replace “few” with “two”. Using the code provided on zenodo, it seems that between 2 and 6% of the journal articles were associated with a preprint (hasPreprint == TRUE). It would be nice to reflect this in this figure, by splitting the articles into articles associated with a preprint or not. This way the conversion of preprints to journal articles becomes more apparent and complements the analysis in Fig. 6.

Line 112: From a security and privacy standpoint it is best to avoid using link shorteners, as they obscure the domain of the landing URL. Please replace with the original URL instead.

Lines 206-210: For better methods reproducibility it would be good to know which tools and external vendors are used for the conversion of PDF to JATS XML and to HTML.

Lines 230-231: From the limitations given in the discussion, it seems like this method is missing cases where the title is changed from the preprint to the accepted journal version, e.g. due to reviewer requests. Please add a sentence here whether it is possible, and if yes, how to mitigate for such cases.

Table 3: The entry for BioHackrXiv is inconsistent with the other (osf) preprints. Presumably the entry for the Versioning type should be “Single identifier for all versions”. Please, double-check and correct if necessary.

Lines 257-258: Unless there are legal reasons to completely remove entries from the database, it would be preferable to retain the original data, for the benefit of meta-research, e.g. into research misconduct. Are the reasons for removal taken into consideration? Perhaps one or two clarifying sentences would be useful here.

Reviewer #2: I enjoyed reading this manuscript that presents the Europe PMC open database of life science literature. It provides a comprehensive account of the motives and processes implemented and, when published, should become a go-to reference for anyone interested in ‘preprint metasearch’ --- I'm suggesting the authors to consider referring to this concept (https://en.wikipedia.org/wiki/Metasearch_engine) that was popular in the years 1995-2005.

Here are my comments:

* Rationale behind the naming

Line 123 reads:

“Europe PMC (https://europepmc.org/) is an open science platform and a life science literature database [35]. It is supported by a group of 35 international science funders as their repository of choice. Europe PMC is developed by the European Bioinformatics Institute (EMBL-EBI). It is part of the PubMed Central (PMC) International archive network, built in collaboration with the PMC archive in the USA. Europe”

Could the authors explain why Europe appears in the name? I understand there's a connexion to the developer (EMBL-EBI) ... but is there a particular focus on the indexing of European research?

* Preprint-Publication Linking (line 125)

Preprint-article matching is not a trivial process. The authors sketch up how their process work. Could they be a bit more specific and compare to other research addressing this information linking task (Cabanac et al., 2021; Eckmann & Bandrowki, 2023)? These 2 papers report how they perform on benchmarks, how does Europe PMC compare?

* Post-Publication Peer Review: PubPeer

Table 4 lists preprint review platforms. I believe PubPeer is missing (PubPeer hosts comments on preprints, too). See for instance https://pubpeer.com/search?q=SSRN

* Success stories

The authors thought about commenting some ‘success stories’ that Europe PMC has supported (lines 631 to 652). This is useful and should be stressed with a few more examples. Which community does Europe PMC serves, why no other option could replace the Europe PMC corpus? Why is this infrastructure critically important in some research areas?

* Non Crossref DOIs

Line 666 reads: ‘some do not’. Could the author try and quantify ‘some’. I suppose that's a minority ... and this should be clarified to understand what's missing in terms of coverage here. I saw the list of repositories on line 733 but failed to grasp the ‘magnitude of the loss’ (percentage of all preprints, say).

* Suggestions

1. In Table 1, add an estimate of the number of preprints hosted at each server

2. line 257 reads ‘and not even a notification remains.’ Indeed, this has been documented for SSRN for instance: https://retractionwatch.com/2021/05/17/preprints-are-works-in-progress-the-tale-of-a-disappearing-covid-19-paper/

3. Biological annotations (l290): How about RRIDs? https://scicrunch.org/resources

4. line 335: Which search engine technology runs Europe PMC? (e.g., Lucene, Solr, Elasticsearch?)

5. line 395: on publications following a preprint, one should perhaps acknowledge that some preprints will stay as such and no journal article will follow (because the authors decide not to).

6. line 1028: extra space to be removed: ‘Covid- 19’

7. Figure 5: add the percentage above each bar.

* References

Cabanac, G., Oikonomidi, T., & Boutron, I. (2021). Day-to-day discovery of preprint–publication links. In Scientometrics (Vol. 126, Issue 6, pp. 5285–5304). https://doi.org/10.1007/s11192-021-03900-7

Eckmann, P., & Bandrowski, A. (2023). PreprintMatch: A tool for preprint to publication detection shows global inequities in scientific publication. In PLOS ONE (Vol. 18, Issue 3, p. e0281659). https://doi.org/10.1371/journal.pone.0281659

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Vladislav Nachev

Reviewer #2: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Enabling preprint discovery, evaluation, and analysis with Europe PMC

PONE-D-24-15144R1

Dear Dr. Levchenko,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

I would like to thank the 2 reviewers for their help in assessing the manuscript and to thank you for addressing carefully all their comments. As you will see, the reviewers and I were satisfied with your edits. The manuscript is in a good shape now.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Florian Naudet, M.D., M.P.H., Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: N/A

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: To my reading all comments from all previous reviewers have been appropriately addressed.

The article describes a valuable resource for the (meta-)research community.

Reviewer #2: Thanks for addressing my comments and for providing a point-by-point response.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Vladislav Nachev

Reviewer #2: No

**********