PONE-D-23-20069Measuring Data Rot: An Analysis of the Continued Availability of Shared Data from a Single UniversityPLOS ONE

Dear Dr. Briney,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Given the comments by the reviewers, a major revision seems to be in order. Especially the second reviewer raises a number of issues that should be addressed. In particular, the reviewers raise several questions around the representativeness of the sample from Caltech that should be addressed.

In addition to the issues identified by the reviewers, I would like to suggest one additional methodological change. At the moment, the estimate of the 2.4% rate of disappearance per year is estimated using a linear regression. However, if each year 2.4% of the data disappears, then you would expect 1 - (1 - 0.024^t) data to have disappeared after t years, not t * 0.0024. That is, stated succinctly, it would be better to use survival analysis for the estimation of this yearly disappearance instead of a linear regression.

Please submit your revised manuscript by Nov 10 2023 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Vincent Antonio Traag, Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at 

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: N/A

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Thank you for your work, data sharing is a very timely and important topic and it is encouraging to see more research on it.

I believe this study is clear and well structured, it properly documents all the steps in the analysis and allows for full replication of the results. The discussion of the related art is comprehensive and direct comparisons are made where appropriate. The provided summary statistics and figures are appropriate for the envisaged level of analysis. Furthermore, the study concludes by highlighting best practices which should be more broadly adopted.

With respect to limitations, I see two main ones which I would encourage the authors to consider (as they partially do in the discussion):

- The data sample is from Caltech, therefore a single university with a certain disciplinary focus. While this might not be a problem per se, it would be interesting to know something more about what articles are included in the sample. For example, the disciplinary breakdown and the most represented venues. This would help the reader getting a better sense of where the results apply most (as well as ideas for future work in terms of extending the study considering complementary samples).

- I am not sure to what extent the data on related links to datasets can be considered to be fully correct and, if not, how this might effect the results. This is also something that the authors mention, yet I am left wondering about this: would it be possible to strengthen the study by addressing this question more systematically? For example, a small random sample of entries could be manually checked to understand what might be the share of missing data in the related links to datasets field.

I hope my comments can help improve and finalise this useful contribution.

Reviewer #2: The author presents a study exploring the data sharing behaviours of researchers at a single university by looking at the “supplemental data links'' included within the metadata of publications within the university’s institutional repository. Overall, the article is well written, uses methods appropriate to the research questions, and presents findings which will be of interest to the research data community. There are a few open questions and points for revision, which I outline below.

INTRODUCTION/BACKGROUND

Currently there is not a separate literature review or background section. I recommend splitting the current introduction into two sections and perhaps adding a bit more content in terms of the literature reviewed.

There are also a few areas where the referencing could be improved, or where it was unclear which reference was being used to substantiate a statement.

On page 3, line 36 - Are there references to include for the concepts of link rot and content drift?

On page 4, line 56 - It is unclear which (if any) of the preceding references are used to support the statement that “Gaining credit for sharing and support for sharing would make researchers more likely to share data but there is still a perceived risk in sharing.”

The authors provide in text citations to the R packages used in the analysis, which is great. Should these citations also appear in the reference list?

The author may also want to discuss (either in the literature review or in the discussion) how their work relates to an earlier study about data rot in the field of astronomy [1]. The combination of methods used in the astronomy study may also be interesting to the author in terms of developing future research directions to explore the motivations behind the behaviours which were observed.

METHODS

The methodology is clearly described. I wonder, though, if the author could include more detail about how the repository team “captures” the articles produced by the university. Does the repository staff perform literature searches to populate the repository? Do researchers voluntarily deposit articles? Essentially, I am wondering how complete the corpus of articles is, compared to the output of the university to understand how representative the data sharing behaviours are of researchers at the institution.

RESULTS

I find it particularly interesting that data sharing via journal supplemental material and GitHub were the least sustainable mechanisms, and that researchers link to files which essentially bypasses metadata which could be helpful in making sense of the data.

Reading the results, I had a few questions. A minor point is that on Line 153, the author mentions an article with 28 associated datasets - do they have any more insight into why this article has so many associations? I also wonder about the change in workflow which they mention in 2022; has this been changed again, so that the repository can capture information in the future?

I also wonder about the disciplinary distribution of articles/data in the corpus which was analyzed, given the tendency to use GitHub. It would be interesting and provide context to the results if the author could provide more information about disciplines here.

I also recommend that the author consider changing the ordering of the x-axes in both of the figures so that years increase rather than decrease. It is counter-intuitive and more difficult to interpret the data in the current presentation.

DISCUSSION

The author raises an interesting finding in Line 279 - that researchers are either choosing not to mint DOIs or are sharing data via a URL. The last option seems very likely to me, given existing work documenting a similar behavior for how researchers share academic articles with each other, as explored by Herbert Van de Sompel and Martin Klein. (The work from these authors could also be used in terms of content drift and link rot more broadly). It would be interesting for the author to make this comparison.

The author also draws on their findings to call for a need to educate researchers. I wonder what other recommendations can be made for other stakeholders, e.g. for curatorial staff.

As a minor point, I agree with the statement that there appears to be a solid uptake in use of the repository by Caltech researchers (Line 304). I wonder though, what evidence the author is using to back up the statement that there are no disciplinary repositories for the researchers who use CaltechDATA. It is an interesting question to think about who uses institutional repositories (as in which disciplines) and for which data. Perhaps the author has more to add to this statement.

MINOR ERRORS/TYPOS/CLARITY ISSUES

On P.3 , line 53 - “is” should be “if”

On p.11, Line 201, p.11 there appears to be a reporting error in the statement that “...of these 136 (6.5%) were URLS and 16 (0.8%) were URLs” Are both percentages for URLs?

On line 249, instead of stating that “this article is not reproducible”, it may be better to say that this methodology is not reproducible?

I suggest deleting “So” at the beginning of the sentence on Line 268.

On p.4, line 59: The statement “Many researchers, however, are actively sharing their data under data sharing requirements from journals” seems to contradict the following paragraph regarding problems with data availability statements.

Line 74 is also unclear; I wonder if the following statistic was perhaps copied over incorrectly from the Federer paper. “Federer found that 80% of data resources could be retrieved automatically, a percentage that went up to 78% for URLs and 98% of DOIs resources when testing links by hand.”

I would also encourage the author to clearly define the terms “supplemental datasets” and “supplemental data links” as early as possible, as these terms may not be intuitive to every reader.

REFERENCES

Pepe, A., Goodman, A., Muench, A., Crosas, M., & Erdmann, C. (2014). How do astronomers share data? Reliability and persistence of datasets linked in AAS publications and a qualitative study of data practices among US astronomers. PLoS ONE, 9(8). https://doi.org/10.1371/journal.pone.0104798

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PONE-D-23-20069R1Measuring Data Rot: An Analysis of the Continued Availability of Shared Data from a Single UniversityPLOS ONE

Dear Dr. Briney,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Your revisions satisfied both reviewers, and so, in principle, we could accept your publication. However, the second reviewer identified some minor errors, and I wanted to offer you the opportunity to revise those before accepting the final version, hence the decision to ask for a minor revision.

Please submit your revised manuscript by Jun 28 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Vincent Antonio Traag, Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Additional Editor Comments:

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: I Don't Know

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Thank you for addressing my remarks, I have no further comments.

Reviewer #2: Thank you for your work in revising this manuscript. The added detail, changes to the methodology, analysis, and situation in the literature has improved the manuscript. I noted only a few minor errors (typographical errors mostly), one place where an added reference would strengthen the article, and a question about the reporting of the data curation error rate.

P.3, Line 43: text should be “It can BE difficult…”

P.3, Line 50: the text in parantheses should be a new sentence

P.11, Line 215: same problem as above

P.12, Line 222: a reference is needed for this claim: “more publishers requesting that authors use data availability statements in articles instead of noting shared data as a citation or footnote.” Can the authors cite a study that shows that publishers request the use of data availability statements more often than requesting data citations?

P. 13, Line 242: Should it be clarified that the data curation error rate is calculated only based on the sample?

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Measuring Data Rot: An Analysis of the Continued Availability of Shared Data from a Single University

PONE-D-23-20069R2

Dear Dr. Briney,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Vincent Antonio Traag, Ph.D.

Academic Editor

PLOS ONE