PONE-D-20-39290

Alignment of biomedical data repositories with open, FAIR, citable and trustworthy principles

PLOS ONE

Dear Dr. Martone,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

I have a few specific comments : 

- Please state if the research was pre-registered or not explicitely in the method section ; 

- Figure 2 would be better in a supplementary material (in my opinion) ;

- On Figures 5 and 6, please provide the name of each repository as a legend (if possible) ; 

Please submit your revised manuscript by Mar 13 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Florian Naudet, M.D., M.P.H., Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for stating the following in the Financial Disclosure section:

"This work was supported by NIH grant# 3U24DK097771-08S1 from the National Institutes of Diabetes and Digestive and Kidney Diseases.  The original decision tree was developed through the  FORCE11 Scholarly Commons working group supported by an award from The Leona M. and Harry B. Helmsley Charitable Trust Biomedical Research Infrastructure Program to FORCE11.  The authors wish to thank Drs. Ko Wei Lin and Jeffrey Grethe for helpful comments.  The funders played no role in the design, data collection, analysis, decision to publish or preparation of the manuscript."

We note that one or more of the authors are employed by a commercial company: MoreBrains Cooperative Ltd, Chichester, UK

2. 1. Please provide an amended Funding Statement declaring this commercial affiliation, as well as a statement regarding the Role of Funders in your study. If the funding organization did not play a role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript and only provided financial support in the form of authors' salaries and/or research materials, please review your statements relating to the author contributions, and ensure you have specifically and accurately indicated the role(s) that these authors had in your study. You can update author roles in the Author Contributions section of the online submission form.

Please also include the following statement within your amended Funding Statement.

“The funder provided support in the form of salaries for authors [insert relevant initials], but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the ‘author contributions’ section.”

If your commercial affiliation did play a role in your study, please state and explain this role within your updated Funding Statement.

2. 2. Please also provide an updated Competing Interests Statement declaring this commercial affiliation along with any other relevant declarations relating to employment, consultancy, patents, products in development, or marketed products, etc.  

Within your Competing Interests Statement, please confirm that this commercial affiliation does not alter your adherence to all PLOS ONE policies on sharing data and materials by including the following statement: "This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests) . If this adherence statement is not accurate and  there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include both an updated Funding Statement and Competing Interests Statement in your cover letter. We will change the online submission form on your behalf.

3. Thank you for stating the following in the Competing Interests section:

"I have read the journal's policy and the authors of this manuscript have the following competing interests: Dr. Martone is on the board and has equity interest in SciCrunch Inc., a tech startup that develops tools and services in support of Research Resource Identifiers.

Dr Murphy is on the board of Dryad Data Repository."

Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, by including the following statement: "This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests). If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please respond by return email with your amended Competing Interests Statement and we will change the online submission form on your behalf.

Please know it is PLOS ONE policy for corresponding authors to declare, on behalf of all authors, all potential competing interests for the purposes of transparency. PLOS defines a competing interest as anything that interferes with, or could reasonably be perceived as interfering with, the full and objective presentation, peer review, editorial decision-making, or publication of research or non-research articles submitted to one of the journals. Competing interests can be financial or non-financial, professional, or personal. Competing interests can arise in relationship to an organization or another person. Please follow this link to our website for more details on competing interests: http://journals.plos.org/plosone/s/competing-interests

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

Reviewer #3: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: N/A

Reviewer #3: I Don't Know

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: First of all, I want to congratulate the authors on this interesting piece of research.

In times of open data, data repositories are evolving rapidly and thus, it is even more important to have scientific articles analyzing them.

A further plus of this paper is the openness and transparency which is shown by sharing the code and the data. Finally, the paper shows a well thought through study design.

However, there are some points that need to be improved.

I recommend an acceptance with minor improvements.

As the lines are not numbered, I will go through the paper section by section.

Title:

Please include the design of the research in the title and its outcome

Abstract:

Please include that you are looking for repositories focusing on diabetes and metabolic diseases.

Again, the design of the study is missing

Please include the time frame of when the research was conducted.

Introduction:

Overall, I am missing references to already established research, but particularly in the introduction section. Was this the only research initiative on NIH repositories.

Methods:

Please explain how many repositories are on the dkNet website and describe the outsourcing process. A flowchart could be useful here for readers.

Did you preregister a protocol on OSF, etc? If so, please insert the link. If not, please explain why.

Did you pilot test the questionnaire on other repositories?

Please include references for the TRUST principles and the CoreTrust Seal.

Table 2 is kind of confusing with the Q#ID for the reader. In my opinion not necessary as it is in the supplementary material S1 again listed. However, personal preference here.

Same problem in Table 3.

Is listing RRID necessary?

What is the point of the section “section”. For the AMP-T2D you put data. My understanding is that “data” would be applicable to all repositories.

Please take care that you explain abbreviations. These might not always be clear for readers who are not familiar with the subject (RRID, API, …). It would be good to have a list of abbreviations in the beginning of the manuscript.

Please state if you followed a reporting guideline, STROBE would be a possibility here. If you don’t use one, please explain why.

Results:

When you finished the research and didn’t find certain items, did you then contact the responsible or repository managers?

Keep in mind that it is highly appreciated that you created accounts in some cases.

I am sorry if I didn’t understand this point but how did you get to a 10 point scale if in some questions you had only 3 or 4 options to choose.

On page 16 you are referring to table 3 when you mean table 2.

Discussion:

I am again missing the focus on the specific disease area. You are writing in a general way on repositories but in which way do your results have external validity for other biomedical areas?

The study design is very well done, however only applied on 8 repositories; should be further discussed/ mentioned.

Reviewer #2: The authors present a framework for data repositories to support Open, FAIR, Citable and Trustworthy (OFCT) data. They describe the application of the open source PolicyModels toolkit to operationalize key aspects of OFCT principles. They use the framework to assess the current state of eight biomedical community repositories listed by the NIDDK Information Network (dkNET). The dkNET is taking an active role in interpreting and facilitating compliance with FAIR on behalf of this community. By defining a set of important functions repositories should support, the work will support two goals: (i) serve as a best practice for those developing new dk data repositories, and (ii) inform the creation of a dkNET tool to help researchers select an appropriate repository for their data.

In general, assessment of repositories and is an important area, and more attention is being paid in biomedicine to certification instruments. I like very much the online PolicyModels toolkit, its functionality and usability, and the machine-readable exports.

While I applaud the goals, in general, I have the following concerns.

1. The process, and the rationale for the development of the OFCT principles are not explained. As the authors say “To our knowledge, this is the first evaluation instrument that was designed specifically around OFCT. However, since the issuance of the FAIR data principles, several initiatives have invested in the development of tools that are designed to assess the level of data FAIRness, including those that are meant to evaluate on-line data repositories”. To my knowledge, beside FAIR and the CTS certifications, there are the several community-driven ‘principles’ to assess repositories and/or provide best pratices for repositories developers and maintainers, by the following groups: COAR (https://www.coar-repositories.org/coar-community-framework-for-good-practices-in-repositories), NIH (https://grants.nih.gov/grants/guide/notice-files/NOT-OD-21-016.html), ELIXIR (https://doi.org/10.12688/f1000research.9656.2) and a group of publishers (https://doi.org/10.5281/zenodo.4084763). To address this, I would like to see the authors explaining: (i) how OFCT was developed, in terms of community participation, and how the range of acceptable and unacceptable values in various dimensions (properties and flags) were defined; (ii) the rationale about creating these new set of principles, and how OFCT is different, complementary or more usable than the others; and (i) how the OFCT maps to these existing efforts, and to the CTS.

2. Although FAIR is only one of the elements of the OFCT principles, much of the discussion is only centered on comparing them to FAIR assessment tools, such as the FAIR Maturity Indicators, FAIR Evaluator and FAIRshake, which are solely based on FAIRness. To address this, I would like to see the authors expanding the discussion to on the rest of the elements in the OFCT principles, which can be done by addressing my point 1.

3. It seems that the application of the OFCT principles is based on manual assessment and on the interpretation of the results. The authors say: “…we deliberately refrained from assigning grades or calling out individual repositories in the work presented here. (Wilkinson et al. 2019) noted that many repositories which were evaluated early on using FAIRmetrics expressed resentment. We recognize the struggles that those who develop and host scientific data repositories undergo to keep the resource up and running, particularly in the face of uncertain funding……We therefore favor flexible approaches that allow individual communities to interpret OFCT within the norms of their community and not entirely according to the dictates of external evaluators.” And also: “The FAIR Maturity Indicators and FAIRshake toolkits differ from ours in that they are intended to employ either fully automated or semi-automated approaches for determining FAIRness. As we show here, some aspects of FAIR require interpretation, e.g., “a plurality of relevant attributes”, making it difficult to employ fully automated approaches”. I understand that fully automated approaches may not be easy, but I do not see how manual assessment would be better and work in practice, in particular be objective and scalable. To address this, I would like to see the authors explaining how they see the manual assessment (with OFCT principles), and what they call the “OFCT potential”, to help them to develop a tool to select an appropriate repository (one of their goals).

Other comments

4. The assessment was performed through inspection of documentation and interaction with the sites, without interacting with the repository’s owners. However, this is not clear. Fig 2 talks about “survey and questionnaire and interactive interview”, but then in the Scoring section authors say : “Five of the sites were reviewed independently by FM and MM…The reviewers made a good faith effort to find information on the site to provide an accurate answer for each question”. Therefore, the authors should make clear that this not a survey sent to repository’s owners or a self-assessment done by the repository’s owners. Nevertheless, the authors could contact the repository’s owners to vet the results.

5. The authors say that they have benchmarked their instrument against others for ‘similar use’, such as: the Datacite’s Repository Finders, and the Scientific Data’s Repository Questionnaire, and the FAIRsFAIR assessment tool. However, there is not information on what this benchmarked has produced since these efforts are not equivalent and the OFCT are principles and indeed not a tool.

6. The data is well stored and cited, as expected, but there are a lot of key details scattered across figures and in the GitHub repositories, and in places it is quite hard to follow.

Reviewer #3: The paper and study have were clearly set out. In particular, the discussion provided insight into the difficulty of other toolkits, limitations on a universal approach and how this study addressed these challenges.

The paper could have included more detail about how the toolkit will impact the workflows of researchers (for the better) and how the community can apply the toolkit in different contexts. It is not immediately clear how the toolkit operationalises key aspects of OFCT principles and this could be further highlighted. It should be noted, however, the toolkit and related materials have been made available for reuse.

It was an interesting paper that has built on work that has been done by other groups and provided a good framework for the community to adopt.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

A tool for assessing alignment of biomedical data repositories with open, FAIR, citation and trustworthy principles

PONE-D-20-39290R1

Dear Dr. Martone,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Florian Naudet, M.D., M.P.H., Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

I would like to thank the 3 reviewers for their comments and for pointing that the manuscript is in a better shape now. I'm happy to accept it for publication. 

As I have informed the authors, it was not possible for one of the reviewer to provide a fast answer. He informed me appropriately and I have decided to give him more time because I wanted to have his opinion on the revised manuscript. I apologize for the time taken but it was very important for me to have his opinion on this new version. I hope that the authors will understand. 

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: N/A

Reviewer #3: I Don't Know

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: I thank the authors for addressing all my comments. This is a nice piece of work and I further wish them a smooth publication process.

Reviewer #2: I thank the authors for their thorough response and revision of the manuscript, which addresses my comments in a satisfactory manner.

Reviewer #3: The revised submission has adequately addressed my comments raised previously. There is growing interest in the assessment of repositories globally and the authors have developed a framework and tool that can be adapted across other domains. The authors have provided a comprehensive overview of the current biomedical repository assessment landscape and developed an instrument that goes beyond the FAIR principles. This is an important step in the Open Science community as repositories need to adapt to best practice. The toolkit can be used for repositories to benchmark against best practice as they continue to evolve. It would have been good to see more about the exploitation or proposed uptake of the toolkit, beyond making the code openly available, although this does not detract from the merit of the paper.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Susanna-Assunta Sansone

Reviewer #3: No