PONE-D-21-12647

Evidence provided by high-impact cell culture studies does not support authors' claims

PLOS ONE

Dear Dr. Ôzkaya,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

It is important to assure that you reply to all concerns by the reviewers (especially Reviewers 3 and 4). Please note that the review provided by Reviewer 1 pertains only to ‘statistical aspects’ of the study and so does not comment on the ‘clinical aspects’ [like medical importance, relevance of the study, ‘clinical significance and implication(s)’ of the whole study, etc.] of the manuscript.

Please submit your revised manuscript by 6/26/2021. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Hamidreza Montazeri Aliabadi

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please consider posting the list of studies and their analysis as supporting information. If materials, methods, and protocols are well established, authors may cite articles where those protocols are described in detail, but the submission should include sufficient information to be understood independent of these references (https://journals.plos.org/plosone/s/submission-guidelines#loc-materials-and-methods).

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No

Reviewer #4: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No

Reviewer #4: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No

Reviewer #4: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: No

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Important note: This review pertains only to ‘statistical aspects’ of the study and so ‘clinical aspects’ [like medical importance, relevance of the study, ‘clinical significance and implication(s)’ of the whole study, etc.] are to be evaluated [should be assessed] separately/independently. Further please note that any ‘statistical review’ is generally done under the assumption that (such) study specific methodological [as well as execution] issues are perfectly taken care of by the investigator(s). This review is not an exception to that and so does not cover clinical aspects {however, seldom comments are made only if those issues are intimately / scientifically related & intermingle with ‘statistical aspects’ of the study}. Agreed that ‘statistical methods’ are used as just tools here, however, they are vital part of methodology [and so should be given due importance].

COMMENTS: In my opinion, this authors group [now there are only two but may include few more similar minded scientists working in same field] can very-well start developing ‘publishing guidelines’ for such articles/literature {I guess, that will be very useful (cell culture studies will likely support authors’ claims subsequently) and definitely will be a great contribution}. It seems that OECD’s ‘Guidance Document on Good In Vitro Method Practices (GIVIMP)’ is not sufficient. Is not that true? There are only nine articles quoted [included in reference, may be because only these many are available] but are good (very relevant) articles. Congratulations.

For this purpose {developing ‘publishing guidelines’ for such articles/literature }, excellent article quoted as reference-1, [Ioannidis JPA. Why Most Published Research Findings Are False. PLOS Med. 2005;2: e124. 260 doi:10.1371/journal.pmed.0020124] may be useful. The understudied aspect of this quality issue in published work highlighted is “the harmony between the hypotheses and the experimental design” is true for many areas and is appreciable. My only question is (line 52) ‘controls (positive, negative)’ {i.e. what are ‘positive controls and negative controls?}. Please clarify. Little/brief clarification of terms ‘p-hacking and cherry picking’ [like ‘P-hacking’ is the relentless analysis of data with an intent to obtain a statistically significant result, usually to support the researcher’s hypothesis and ‘Cherry-picking’ is the presentation of favourable evidence with the concealment of unfavourable evidence] could be useful for readers, I guess.

Although it is not my ‘Subject Area’, in my opinion, this one it is an excellent article. Nevertheless, I feel that overall the article is little short. Maybe adding few more case studies will do. As pointed out in ‘important note’ above “This review pertains only to ‘statistical aspects’ of the study and so ‘clinical aspects’ should be assessed separately/independently. May think of changing title (may make it more catchy).

Reviewer #2: In general, this is a well-conducted and important paper. The Authors are to be congratulated for their attempt to provide a degree of rigor that is typically lacking in preclinical cell culture studies.

In this study, the Authors examined highly-cited cell culture studies and investigated whether the claims made in each study were supported by sufficient experimental evidence.

To do this, they examined 282 claims asserted by 103 different high-impact articles. They found that many of all claims were not sufficiently supported by the evidence that was presented. They also identified troubling examples of misinterpretation of data.

They concluded there was “an alarming discordance” between the actual experimental findings and the way they were actually described in the manuscript.

To perform their analysis, the investigators searched on the key words “cytotoxicity, viability, cell death, growth inhibition, proliferation, or anti-cancer” and retrieved studies from 2017, 2018. They selected 121 publications, each of which had received at least 65 citations. Reviews and studies that did not include a claim or did not involve cell culture were excluded. This left 103 original studies for assessment. In total, the articles that were analysed were cited more than 9,000 times as of March 2021 (within two to four years of publication), so it is reasonable to assume (as the Authors have done) that despite their deficiencies, these publications have had an impact on the field.

The most useful aspects of this work include:

a) Authors provide a useful definition for each of the terms on which they searched - “cytotoxicity, viability, cell death, growth inhibition, proliferation, or anti-cancer”.

b) In Table 1 they provide a clear statement as to what constitutes sufficient and insufficient evidence for each of the terms that they defined. This is a valuable contribution as many publications do not apply the appropriate level of rigor when attempting to perform and interpret these assays.

c) The Authors also present specific proposals to improve the quality of future studies.

It would be a valuable contribution to preclinical research if the Authors’ proposals were widely adopted. This would certainly contribute to an improvement in the robustness of preclinical cell culture experiments.

Areas for improvement.

Major issues.

1) It would be helpful for the Authors to clarify their intent. Specifically, it is not clear to this Reader whether the Authors are stating that 64% of studies were supported with sufficient evidence, or 64% were not supported by sufficient evidence.

The two lines that appear to be in conflict are:

Line 33: “Our findings revealed that only 64% of all claims were sufficiently supported by evidence”

versus

Line 195: “we considered the evidence …which was asserted by 66 different studies (64%), as insufficient”

2) Two areas that were not specifically addressed, but that could further strengthen the manuscript are:

a) The Authors did not mention that subjective analyses should be performed by blinded investigators. Because it is seldom the case that these type of experiments are performed by blinded researchers, it would be valuable for the Authors to highlight its importance.

b) When considering an effect on cell proliferation or cell death, the Authors should stipulate that cell proliferation or cell death should be expressed on a logarithmic not a linear axis. Clearly cell proliferation is an exponential, not a linear function. Although many papers misrepresent their data by presenting it on a linear axis, where it looks more dramatic, this is not an appropriate way to present the data. Given the focus of this publication, this too would be a valuable addition for the Authors to comment upon.

Minor issues:

Line 162: Blc-2 should be Bcl-2

Reviewer #3: PONE-D-21-12647

The authors take on an important reproducibility issue in high-impact cell culture studies. This is certainly of high importance, and suggestions are made for improvements. Overall, the authors comment on a very important point, and one that is close to my own heart… viability vs cytotoxicity. However, they do not do this in a rigorous manner, and this is more suited for a commentary than any sort of research article or review.

1. Line 54 – is this substantiated by any references, or anecdotal?

2. The introduction and overall manuscript is lacking references

3. Lines 67+ reads more like commentary – also a bit vague. What is being discussed here? The cytotoxicity and cell viability terms?

4. What was the rationale for focusing on 2017 and 2018?

5. The inclusion/exclusion should be detailed more clearly on lines 91+

6. Since care is being taken to describe these cell health variables… viability I find a bit problematic. It is not the number of living cells, it generally represents a reaction that is often considered to have a linear relationship to the number of living cells. These enzymes, such as tetrazolium salt, etc., can be modified by exogenous factors and cannot be explicitly considered to describe living cell number.

7. Line 144 – LDH release is a cell viability assay, not a cytotoxicity assay – given the context of this paper, this is a very important point to classify these assays correctly. There are a variety of contexts described in the literature where cells leak LDH and remain otherwise alive and will respond normally via other cell viability measures. Adipocytes is one example of this, though others have been described. Unless the specific experimental context is being taken into account, it’s hard to see how these sorts of generalities are not going to adversely influence results here.

8. Table 1 – how is DNA content not included in cytotoxicity? It is the gold standard for determining cell number. You could say that it is context dependent, if there are no wash steps in a particular assay, but it cannot just be ignored as a cytotoxicity measure.

9. Given the well-described artifacts and issues with a lot of the cell viability and proliferation assays, as described by Hsieh et al., 2015, 2016, etc., this article seems to ignore many of these issues with the assays they included.

10. There doesn’t seem to be any transparency about how and why individual publications were deemed sufficient or insufficient. If highly cited publications are going to be published as insufficient, a rationale column for the decision-making process is critical to providing clarity for readers.

11. Lines 208+ - cytotoxicity is not a measure of viability, it is a different measure completely.

12. Line 206 – “we had to make assumptions on behalf of the researchers” is not very scientific. In a systematic review context, you can and should reach out to the authors for clarification. Appreciate this is not that, but if you are publishing a paper blasting the authors for having an “insufficient” study, and only rely on your interpretation, that is a problem.

13. Moreover, interpretation of who? Is this a single author performing these assumptions? Is each paper reviewed by both authors independently? A single person evaluating these is not robust – and what sort of testing was done to verify that the authors performed these assumptions in a consistent manner?

14. I feel at this point in the paper that I’m not sure of the overall goal. Are the authors just using the assay reported in the paper and checking which box on a table it fits into? This doesn’t seem like the level of discourse that is particularly worthy of a peer-reviewed publication. A commentary could fit this, but it seems more opinion than a rigorous evaluation of reproducibility – which is, of course, crucially important.

15. The first sentence of the discussion is very vague and alarmist – the only thing examined here was the cell proliferation/toxicity/viability and not any other claims these publications may have made. That is very important to make clear

Reviewer #4: The manuscript “Evidence provided by high-impact cell culture studies does not support authors’ claims” from Ozkaya and Geyik describes the use of various assays to evaluate cell state in published articles. The authors mined the literature for highly cited papers from 2017-2018 that evaluated cell viability, death, proliferation, growth, apoptosis, and cytotoxicity in cell culture systems, and considered whether or not the assay was appropriate and sufficient for the conclusions drawn. A summary of the authors’ ‘Evidence Sufficiency’ meta-analysis is presented in a single figure. They suggest that the incorrect use of assays, or over interpretation of results from specific assay classes are likely factors in the irreproducibility of preclinical studies.

Overall, the goal of the manuscript fits under the ‘Systematic Review and Meta-Analyses’ article category; however, it will require significant revisions and additional review should it be considered for publication. I cannot recommend it be published in its current form.

Please address the following points:

Minor

1. Add a citation to papers that have addressed similar issues – line 56-57.

2. Fig. 1C dehydrogenase activity is always used correctly to assess viability?

3. Fig. 1 C (discussed on lines 219-220) you say data are interpreted correctly in 73 claims, but the figure looks like it should be 48 claims. Please correct.

4. Thorough copy-editing is needed

Major

5. Nomenclature recommendations should not be buried in the Methods. You highlight this as a main conclusion of your work. There is an opportunity to be prescriptive with this work and for it to be a resource for people to turn to when selecting assays and designing experiments – I think this should be focused on much more.

6. Your definition of proliferation rate is presented as a population-level readout “how fast a group of cells divide over time” however it is then contradicted by stating that this rate only applies to surviving cells – an apparent decrease in proliferation rate of a population could occur from a slowing of the doubling time of all cells in a population or from a fractional response in which a subpopulation is killed off, and the surviving cells continue to proliferate at the same rate (or at an altered rate) resulting in a longer apparent doubling time, as you point out. Is your proposed definition of proliferation rate meant to be applied to single cell or population level data? Does it only apply to living cells at endpoint? If the endpoint is simply how fast a group of cells divide over time, then it doesn’t matter if the underlying phenotypes are different, though they should be determined with secondary assays designed specifically to measure cell death and growth rate.

7. I strongly disagree that measuring metabolic activity is sufficient for assaying cell viability. In some cases, these techniques (MTT, CellTiter-Glo, etc.) yield results that correlate linearly with live cell count, but many perturbations directly affect the metabolic activity of cells which can result in critical miss-interpretations of effects. See https://pubmed.ncbi.nlm.nih.gov/31302153/ for clear examples.

8. With respect to defining cell death, I generally agree that this needs to be measured directly, however, if the number of cells are counted at the time of drug addition, for example, then a treatment resulting in net cell loss can only result from some degree of cell death. In general, any time cells are counted, live and dead cells should be distinguished at the bare minimum. See Hafner et al. (https://pubmed.ncbi.nlm.nih.gov/27135972/) for a more nuanced and thorough discussion. Growth rate inhibition metrics, an important addition to the field, are omitted and should be discussed – they are immediately relevant and address several of the same concerns the authors highlight.

9. Apoptosis, please correct typo in “molecular switch responsible in apoptosis” to “molecular switch resulting in apoptosis” and specify whether you’re referring to cleaved caspases and/or MOMP.

10. Please clarify why you don’t consider PI sufficient to assay membrane integrity but you do consider other dyes that are also membrane impermeable to be sufficient (lines 174-177) viability assays.

11. Table 1 as is, is confusing especially for the proliferation and viability entries. It would be much more useful if it was presented by claim and then assay, followed by what each assay actually measures, and what is required for that assay to be sufficient for that claim.

12. Only nine references are provided – there is a lot of literature that has been overlooked on assay choices and pitfalls that should be considered, and incorporated into the introduction and discussion sections.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

Reviewer #4: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Attachments

Attachment

Submitted filename: renamed_ab46f.docx

PONE-D-21-12647R1

Evidence provided by high-impact cell culture studies does not support authors' claims

PLOS ONE

Dear Dr. Özkaya,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please specially address the concerns raised by reviewers 3 and 4. It might be helpful, if you can revise the title of the manuscript, so it is more specific ad does not seem dismissive towards all "authors' claims". Also, the text could also be slightly revised to clarify that the objective is not to dismiss all the findings these publications are reporting.

Please submit your revised manuscript by 9/10/2021. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Hamidreza Montazeri Aliabadi

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

Reviewer #3: (No Response)

Reviewer #4: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Partly

Reviewer #4: Partly

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: N/A

Reviewer #4: (No Response)

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: (No Response)

Reviewer #4: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: No

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: COMMENTS: As already said, that everything [like execution, methodology, statistical analyses, etc] are very good [i.e. study is excellent]. Writing/presentation is also excellent. Later, comments made [probably by other respected reviewers as well] were/are attended or answered positively/adequately, I am fully satisfied and the manuscript is improved a lot. I recommend acceptance.

Reviewer #2: Thank you for addressing the comments that I had provided.

This is a useful contribution and an important contribution.

Reviewer #3: I appreciate the reviewer edits to the manuscript, though many of my original concerns remain. This article still reads to me as a commentary - an important topic for one - but is not well enough defined for publication as a research article.

I still have major concerns with the title of this article. The authors rather blindly call out other papers yet (Evidence provided by high-impact cell culture studies does not support authors' claims) yet they have not examined anything in these papers other than the use of cytotoxicity/cell viability assays. This absolutely must be made crystal clear. I asked for this on first revision and this was not addressed. The inflammatory title, when this is explicitly not assessed, is not proper.

It also needs to be explicitly stated that the claims they are examining from these papers are in some cases only a very small part of the overall paper. The authors in the response to reviewer document repeatedly state the rationale for this article, but they have strayed far beyond that in the paper itself, as written.

I happen to disagree with the authors on the LDH assay, though appreciate there can be difference of opinion here. Though the irony of them citing the literature to defend correct interpretation is not lost on me, given the topic of their paper. I would suggest that they think about what this assay actually examines and whether that fits into the definition of a cell viability or toxicity assay. As there are clear examples where the cell type and context promote positive findings in this assay, it cannot be a cytotoxicity assay.

Some language editing is still needed here, particularly for the newly added sections highlighted in yellow.

This article should be a much shorter commentary, where it could accomplish the author's goals and not over-reach.

Again, how is DNA content listed under proliferation and not toxicity? This was not appropriately responded to in revision.

It is still unclear how some terms are being distinguished. On line 135, cytotoxicity, the definition specifically states "cell death". Yet in Table 1, cytotoxicity and cell death are separate groups on this table. I don't understand how these groups are being defined and how they are different, given the descriptions the authors are providing here.

Reviewer #4: PONE-D-21-12647_R1 review (also attached)

The revised manuscript “Evidence provided by high-impact cell culture studies does not support authors’ claims” from Ozkaya and Geyik is improved from the initial submission; however, several points still need to be addressed and clarified prior to publication. The expansion of Table 1 is a significant improvement to the previous version.

Minor points:

1. Table 1: I think it would be more intuitive to change the order slightly so the cell death claims (apoptosis, cell death, cytotoxicity) are followed by the live cell claims together (cell growth, proliferation, viability) rather than strictly listing them alphabetically.

2. Thorough copy editing is still required. Much of the text remains unchanged.

3. Referring to cell state assays as ‘cell culture techniques’ (for example on lines 41, 266 and 282) is confusing. I perceive good cell culture technique to be the proper maintenance of cells in culture (free of mycoplasma and other contaminants, log-phase growth for the assays you discuss etc). Good cell culture technique is also a critical factor in generating high quality and reproducible measures of cell viability/death/proliferation etc. It would be worth adding that all of the assays discussed can be badly confounded by poor cell culture technique. As you correctly point out, cost, simplicity, speed, and throughput likely motivate assay selection.

4. Reference to Fig 1D (line 226) should be to 1C.

5. While a valuable clinical biomarker, ki67 positivity as it relates to cell cycle and cells cycling is complicated in culture systems, and many cells in arrested states remain ki67 positive – caution in its interpretation is warranted, and like most of the endpoints discussed should be cross-validated. See https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6108547/ and https://elifesciences.org/articles/13722

6. Lines 37-38 are unclear.

7. Consider reframing line 76 from ‘we decided to define these terms ourselves’ to ‘in the current work, we propose a series of definitions and recommendations’ since this is the ultimate goal.

8. Lines 132 and 133, please be specific about the ‘morphological changes’ and ‘molecular switches’ that you are referring to.

9. Please add the citations for id#9 and id#26 in the text (lines 260 and 261) so the reader can refer to them directly without having to access your web tool.

10. Line 271: GR refers to normalized growth rate inhibition, not simply growth rate.

11. Line 272: DIP should be specified ‘drug-induced proliferation rate’

12. Lines 273-276 are unclear. Please explain.

Major points:

1. Your claim in Table 1 that ‘DH activity correlates with the number of viable cells’ is not always true; at the very least, the associated ‘Yes’ should be changed to ‘Conditional’. Moreover, you twice assert that you consider measurements of metabolic activity as sufficient evidence for viability because they’re widely used: ‘…it is the most common method used to evaluate viability…’ (line 119) and ‘…we considered measurement of metabolic activity as an indicator of viability as it is by far the most preferred viability assay…’ (lines 283-284). This is poor justification for reasons you include in the discussion and reasons brought up in the previous reviews. With proper controls/assay calibration (how does the assay output relate to live cell count in the specific context?) and cross-validation experiments to confirm that metabolic activity is not influenced by the perturbation under study, these assays of metabolic activity can be used as surrogates for viability, but these controls are critical to correct data interpretation. My concern is that these points are not sufficiently addressed, and that incorrect use of metabolic assays to evaluate cell viability and proliferation is presented as acceptable without caveats throughout the body of the paper only to be contradicted in the discussion.

2. Can you comment on how often, in the papers you included, the assays you considered are the only line of evidence for a conclusion drawn since this seems to be the most problematic case? A common scenario is one where a crude, population level measure such as MTT is used to suggest toxicity, and then followed by a microscopy or flow cytometry-based assay to confirm cell death vs change in proliferation and mechanism (apoptosis vs necrosis etc). In a case like that, would the use of a metabolic assay be considered sufficient or insufficient? The use of complimentary and confirmatory assays should be discussed and considered when scoring claims made in the papers that you evaluated.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Dr. Sanjeev Sarmukaddam

Reviewer #2: No

Reviewer #3: No

Reviewer #4: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Attachments

Attachment

Submitted filename: PLOS ONE R1 review.docx

PONE-D-21-12647R2From viability to cell death: claims with insufficient evidence in high-impact cell culture studiesPLOS ONE

Dear Dr. Ozkaya,

I would like to start by apologizing for the lengthy review process. Your manuscript, as I am sure you know, is about a controversial subject that has triggered discussions among the reviewers. As you will see below, all reviewers agree that the latest version of the manuscript has improved significantly; however, while reviewers 1 and 2 both agreed that manuscript should be accepted, two reviewers (Reviewers 3 and 4), have cited a few minor points that requires your attention. Hopefully, they are not too time-consuming to answer, and this would be the final re-submission.

Please submit your revised manuscript by 11/4/2021. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Hamidreza Montazeri Aliabadi

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

Reviewer #3: (No Response)

Reviewer #4: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: (No Response)

Reviewer #2: Yes

Reviewer #3: Partly

Reviewer #4: Partly

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: (No Response)

Reviewer #2: Yes

Reviewer #3: N/A

Reviewer #4: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: (No Response)

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: (No Response)

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: No

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: COMMENTS: As already said, methodology, statistical analyses, etc are good and the manuscript is improved a lot. I recommend acceptance.

Reviewer #2: No additional comments - my previous comments have been adequately addressed.

Congratulations on a valuable study.

Reviewer #3: The authors have done a good job of responding to most reviewer concerns, though on re-review I have a few limited concerns. The bulk of the data from this study is provided in an alternate web link, though I'm not sure how I feel about this. Is this a stable link? Is the journal providing certainty that this will stay active in perpetuity? This paper becomes completely illegible without these supporting data. I strongly suggest they be added as supplemental data to the manuscript itself to ensure that these data are retained.

I still feel there are some lingering issues, including some raised by reviewer #4. The final question about multiple assays, I think, raises some concerns with this analysis. This addresses how articles using multiple assays or measures were treated. The author answer was that they judged the best measure and judged the article on that - though insufficient details are provided on how they judged the best assay for each specific paper and condition. Also, certainly some credit should be given to authors that couple multiple assays together - these can often provide a more complete picture of cell health than any one individual assay.

I also feel like there are some interesting concerns raised by reviewer 4 in regard to controls and the author answer to it. Surely the use of an assay, without appropriate controls, is not sufficient. Yet it seems that the details of the assay and the specifics of how it was performed and controlled for are not factored into these determinations. When making this still broad statement about the credibility of these author claims, this seems an important aspect to have considered.

Reviewer #4: The article revised by Ozkaya and Geyik is improved. In particular I appreciate the addition of discussion around the caveats of the study and assays discussed. However, I find that the article has gained length rather than clarity, and that the authors are still missing an opportunity to provide clear recommendations for good cell culture technique (as it pertains to all cell-based assays) and optimal assay use to address specific biological questions accurately. While I see value in drawing attention to widespread incorrect or over-interpretation from basic assays, the potential for impact lies in being explicit about what would have been needed for a claim to be deemed sufficient. This information is only included in the Methods and is incomplete (examples are provided), the authors could consider adding a summary table of assays and the conclusions that can be drawn from them. Although I will leave it to the editor to decide whether or not this falls within the scope of the current paper.

I continue to disagree with the authors’ assessment that ‘viability measurement is the only purpose of an MTT assay that is somewhat acceptable’, when really it is evaluating changes in metabolic activity in a population of cells. The same number of cells can have vastly different MTT readouts based on treatment, and while I agree that using these assays to assert cell death is particularly problematic, the oversimplification is detrimental to the paper. An increase in signal does not always indicate an increase in cell number.

Remove references to the discussion from the Methods section.

I do not understand why cell count and DNA content are only considered sufficient evidence for proliferation in the absence of treatment. Background levels of cell death need to be assayed under all conditions, controls included.

Why is ‘Real-time cell imaging’ (which I assume to refer to time-lapse microscopy) insufficient for quantifying cell death? Watching cells over time is both a straightforward and accurate way to monitor cell fate.

DAPI is membrane permeable, how is it correctly used to test membrane integrity? It can be used for DNA content, and typically is used to complement a membrane impermeable dye to accurately evaluate the fraction of cell death. Same concern for acridine orange and Hoechst in the Viability section of the table. Unless they are used with GhostDye (or PI etc), or DNA content is accounted for, they can only yield overall cell counts.

Please clarify the difference between ‘Dye inclusion’ and ‘Membrane integrity’.

Propidium iodine should be Propidium iodide.

pH3 positive cells represent a small fraction of a population and can represent cells arrested in M-phase, their presence alone does not reflect the overall proliferative potential of a population.

Change in proliferation is not limited to decrease in rate, perturbations can also stimulate proliferation.

The discussion needs to be re-worked and copy edited. In particular lines 297-310. The discussion of IC50 values does not seem to fit – although these methods aim to improve reproducibility of large-scale drug response studies, this is somewhat separate from the issues the current paper is trying to address.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Dr. Sanjeev Sarmukaddam

Reviewer #2: No

Reviewer #3: No

Reviewer #4: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

From viability to cell death: claims with insufficient evidence in high-impact cell culture studies

PONE-D-21-12647R3

Dear Dr. Ozkaya,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Hamidreza Montazeri Aliabadi

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #3: N/A

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #3: We appreciate the extensive revisions that have been completed by the author. While we continue to disagree on some of the finer points of this topic, these are discussions that can play out in the peer-reviewed literature. They have done a commendable job in ensuring all their data is available for any further discussions that other authors would like to have. I think this will be an interesting discussion point for further considerations.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #3: No