PONE-D-22-21926Validity of ICD Codes to Identify Do-Not-Resuscitate OrdersPLOS ONE

Dear Dr. Callahan,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please submit your revised manuscript by Dec 02 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Keith M. Harris, PhD

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at 

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please provide additional details regarding participant consent. In the ethics statement in the Methods and online submission information, please ensure that you have specified (1) whether consent was informed and (2) what type you obtained (for instance, written or verbal, and if verbal, how it was documented and witnessed). If your study included minors, state whether you obtained consent from parents or guardians. If the need for consent was waived by the ethics committee, please include this information.

If you are reporting a retrospective study of medical records or archived samples, please ensure that you have discussed whether all data were fully anonymized before you accessed them and/or whether the IRB or ethics committee waived the requirement for informed consent. If patients provided informed written consent to have data from their medical records used in research, please include this information.

3. We note you have included a table to which you do not refer in the text of your manuscript. Please ensure that you refer to Table 2 in your text; if accepted, production will need this reference to link the reader to the Table.

4. Please include your tables as part of your main manuscript and remove the individual files. Please note that supplementary tables (should remain/ be uploaded) as separate "supporting information" files

Additional Editor Comments:

Reviewer 1 has pointed out several important factors to address. Authors are urged to take a more cautious approach to interpreting findings and stating conclusions. Please address each of the reviewer's comments and suggestions. If data is available, suggestions for expanding age range and ethnic diversity, etc. appear reasonable. The treatment of continuous variables and unweighted variables needs to be addressed according to PLOS' requirements for analyses to be conducted according to high standards. This is a valuable work, and we look forward to your response.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Major Points

1. My primary concern regarding the manuscript is the disconnect between the generalizability with which the authors discuss their results and the restrictive nature of their cohort. The authors use their results to discuss how discharge and claims data can be used to identify DNR orders in future studies. However, their cohort is exceedingly restrictive and may not be the most appropriate with which to identify patients admitted with heart failure. Their reported statistics of sensitivity, specificity, PPV, and NPV are only valid for this specific cohort and likely are not generalizable to other conditions or patient cohorts. The authors should first consider their patient selection along the below delineated lines and also discuss their results with much greater clarity about the limitations of their findings throughout the abstract and manuscript, not just in the limitations section of the discussion.

a. Age – the authors select patients who are >65 years of age. Use of DNR orders does increase with age meaning there would be greater prevalence of DNR orders in an older cohort. However, previous studies have indicated use of DNR orders begins to increase at around age 40. Use of an older cohort would result in higher sensitivity and specificity due to the higher prevalence. The authors should address this by either widening the cohort or doing a sensitivity analysis which includes patients down to age 40.

b. Cohort definition – the authors only include patients with a principle diagnosis of HF (ICD9 and ICD10) as the ‘HF’ cohort. However, previous work (Mehta et al. Journal of Critical Care. 2015.) demonstrates that the most severely ill patients with HF (i.e. those most likely to have a DNR order) may have a principle diagnosis of acute respiratory failure and a secondary diagnosis of HF (i.e. presented with hypoxia due to pulmonary edema) or a principle diagnosis of acute MI with a secondary diagnosis of HF. These alterative coding pathways may be used to increase reimbursement but still tend to identify patients with HF. The authors could consider a sensitivity analysis with a wider cohort definition or they should mention that their definition may have selectively excluded individuals most likely to have a DNR order.

c. Choice of heart failure – the authors restrict their analysis to HF only for unclear reasons. They suggest that studies that include all patients may include diagnoses for which DNR may not be relevant. However, DNR orders issued early in the course of admission tend to be more indicative of comorbid condition rather than the acute presenting illness. Moreover, other cohorts such as cancer may be even more relevant for DNR orders. The authors should provide greater justification for why HF was chosen, what biases such a choice may create, and limit their discussion about the utility of their findings to patients with HF.

2. My second major concern deals with the use of a single center and conclusion that can be drawn from it. I commend the authors on their sample size and efforts but I am concerned about any conclusions drawn from a single center on generalizable billing practices that might be heavily biased. The authors correctly mention several possible reasons why DNR orders may be systematically over or under estimated in billing codes from a given center. Each health center tends to bill in a unique fashion. Biases in terms of including or not including certain codes tend to happen at the hospital level, not the patient level. Therefore, the use of a single health system does not necessarily reduce the potential of coding bias to either over or under represent DNR orders in ICD codes. The authors should be far more specific in addressing the limitations of a single center in their discussion. Specifically, they should address the potential for different coding practices at different hospitals to result in significantly different results that what they have observed.

3. I am also concerned about the lack of racial diversity in this single center study. There is extensive evidence indicating that different racial groups have significantly different rates of ACPs and DNR orders when inpatient. Specifically, patients who identify as black are far less likely to have a DNR order when hospitalized. Therefore, a population with greater racial diversity may have very different performance metrics for iDNR vs eDNR. The authors should address this issue in the discussion.

4. The authors use and presentation of the CCI is confusing. The CCI is a continuous score ranging from 0 to 37 based on a weighted assessment of each individual comorbidity. The authors indicate that they are using the CCI but then present data in categories of 0, 1, 2-3, and 4+. The language used in the results seems to indicate that these are not categories of the scores but rather number of comorbidities. As each comorbidity carries different weights, simply counting the number of comorbidities is misguided. Moreover, when considering advanced directives, certain comorbidities are far more associated with a DNR order (e.g. cancer) than others. Therefore, simply looking at the CCI to compare differences between the iDNR and eDNR group can lead to misguided conclusions. I would suggest the authors clarify how the CCI is being used in the models and would present different rates of certain key comorbidities like cancer.

5. I am additionally confused by the use of cost as a potential outcome. The fact that patients with a DNR order have lower costs is not surprising or informative. Patients with a DNR order are less likely to receive high cost invasive therapies and, given associations with mortality, have shorter length of stays. Moreover, I am also confused by the primary exposure in the cost analysis as presented in Table 4. It appears that the primary exposure is a 4 level categorical variable with levels of Died, DNR, Died & DNR, and Lived & no DNR. This variable is very confusing as there are potentially overlapping categories and potentially groups not represented. For example, if a patient died but had no DNR does that mean they are level 1 or is there overlap between level 1 and level 3. Moreover, if the groups are mutually exclusive, is level 2 really Lived and DNR? The authors need to clarify why cost is an important outcome and how they are defining the exposure. One might consider looking at LOS rather than cost given that costs can vary widely between hospitals.

6. Several studies have shown significant changes to coding patterns associated with the shift from ICD-9 to ICD-10 in October 2015.While the authors address the validity of iDNR in several subgroups, they do not report differences in iDNR validity for ICD-9 vs ICD-10 codes. I would strongly suggest the authors do further subgroup analysis focusing on ICD-9 vs ICD-10 to show that secular changes in coding pratices and the shift from ICD-9 to ICD-10 has not changed validity.

Minor Points

1. In the introduction, the authors indicate that DNR is a legal document. However, in most states it is a hospital order that only applies to that hospitalization. Medical Orders for Life Sustaining Treatments (MOLST) forms have taken over in many states as the primary method of communicating goals of care between hospitalizations. In most states, MOLST forms are legal documents. The authors should just clarify this point about DNR orders.

2. The Methods Data Source section states that ‘both authors had full access to all the data’. There are 3 authors so I would suggest modifying the text to indicate whether all others, 2/3 authors, etc had access to the data.

3. In the results discussing mortality, the section around non-DNR drivers of mortality is a bit distracting. Knowing that “other pay sources” had the highest mortality is not related to the central question of iDNR vs eDNR.

4. Figure 1 is really a Table, not a figure.

5. Previous work has shown that DNR status can be a major confounder in comparative effectiveness and observational research (Bradford et al. CCM. 2014). Being able to identify DNR status in large administrative databases would enable to researchers to adjust for this key confounder if/when necessary. The authors should explore this fact in the discussion.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Attachments

Attachment

Submitted filename: callahan DNR review.docx

PONE-D-22-21926R1Validity of ICD codes to identify do-not-resuscitate ordersPLOS ONE

Dear Dr. Callahan,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Feb 18 2023 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Keith M. Harris, PhD

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Additional Editor Comments:

Thank you for resubmitting an improved manuscript. There are just a few points to address. Please take all of Reviewer 1's points carefully and address each. None of these impact on the validity of the study, but rather the validity of the language used to describe the study. Improving the accuracy and completeness will be helpful for researchers interested in DNROs. PLoS is generous with word counts, so feel free to explain these matters.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: No

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: I commend the authors for their responses to my previously submitted review. While they have improved many aspects of the manuscript, I continue to have significant reservations about the generalizability of the findings and I do not believe the authors have narrowed their conclusions and interpretation of their data. Based on their paper, iDNR may be a valid proxy for eDNR in their specific hospital for older patients with heart failure in a predominantly white population. As coding patterns vary from hospital to hospital and from one condition to another, they cannot conclude that iDNR will perform as well in another hospital or health care system. Moreover, PPV and NPV will change based on prevalence and a hospital with more non-White patients will have a very different prevalence of eDNR. The authors acknowledge this limitation in their response but I believe that the current presentation might be misleading. I would recommend two specific changes to ensure readers have a clear picture of the scope of the study. I would suggest:

1. Title - Validity of ICD codes to identify do-not-resuscitate orders. This title very much implies a generalization of the findings that is not appropriate. I would suggest a modification to something along the lines of: "Validity of ICD-codes to identify do-not-resuscitate orders among older adults with heart failure: A Single Center Study"

2. Abstract. Conclusions. "ICD codes provide a reasonable proxy for DNR orders." I would strongly argue that the authors do not have sufficient evidence to say that iDNR is a reasonable proxy for eDNR for all patients. As many readers only read the abstract, the abstract needs to be clear about the implication. I would suggest "ICD codes provide a reasonable proxy for DNR orders among hospitalized older adults with heart failure. Further research is necessary to determine if billing codes can identify DNR orders in other populations."

Minor Point

In Table 3, to more clearly state that the CCI is a score not a count of comorbidities i would change the row title to "Charlson Comorbidity Index Score"

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Validity of ICD codes to identify do-not-resuscitate orders among older adults with heart failure: A single center study

PONE-D-22-21926R2

Dear Dr. Callahan,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Keith M. Harris, PhD

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments: