Development and validation of a new prognostic index for mortality risk in multimorbid adults

Viktoria Gastens; Arnaud Chiolero; Daniela Anker; Claudio Schneider; Martin Feller; Douglas C. Bauer; Nicolas Rodondi; Cinzia Del Giovane

doi:10.1371/journal.pone.0271923

Peer Review History

Original SubmissionFebruary 3, 2022
15 Mar 2022 Decision Letter - Robert Daniel Blank, Editor PONE-D-22-03428Development and validation of a new prognostic index for mortality risk in multimorbid adultsPLOS ONE Dear Dr. Gastens, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Both reviewers request fuller contextualization of your work. Please see detailed reviewer comments. Please submit your revised manuscript by Apr 28 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript: A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols. We look forward to receiving your revised manuscript. Kind regards, Robert Daniel Blank, MD, PhD Academic Editor PLOS ONE Journal Requirements: When submitting your revision, we need you to address these additional requirements. 1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf 2. Thank you for stating the following in the Funding Section of your manuscript: "This work was supported by Swiss National Science Foundation grants (320030_188549/01 to AC; 325130_204361 to CDG). This work is part of the project “OPERAM: OPtimising thERapy to prevent Avoidable hospital admissions in the Multimorbid elderly” supported by the European Union's Horizon 2020 research and innovation programme under the grant agreement No 6342388, and by the Swiss State Secretariat for Education, Research and Innovation (SERI) under contract number 15.0137. The opinions expressed and arguments employed herein are those of the authors and do not necessarily reflect the official views of the EC and the Swiss government." We note that you have provided funding information. However, funding information should not appear in the Funding section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form. Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows: "This work was supported by Swiss National Science Foundation grants (320030_188549/01 to AC; 325130_204361 to CDG; www.snf.ch). This work is part of the project “OPERAM: OPtimising thERapy to prevent Avoidable hospital admissions in the Multimorbid elderly” supported by the European Union's Horizon 2020 research and innovation programme under the grant agreement No 6342388 (ec.europa.eu/programmes/horizon2020), and by the Swiss State Secretariat for Education, Research and Innovation (SERI) under contract number 15.0137 (www.sbfi.admin.ch/sbfi/en/home.html). The opinions expressed and arguments employed herein are those of the authors and do not necessarily reflect the official views of the EC and the Swiss government. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript." Please include your amended statements within your cover letter; we will change the online submission form on your behalf. 3. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For more information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. In your revised cover letter, please address the following prompts: a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially sensitive information, data are owned by a third-party organization, etc.) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent. b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. We will update your Data Availability statement on your behalf to reflect the information you provide. 4. Your ethics statement should only appear in the Methods section of your manuscript. If your ethics statement is written in any section besides the Methods, please move it to the Methods section and delete it from any other section. Please ensure that your ethics statement is included in your manuscript, as the ethics statement entered into the online submission form will not be published alongside your manuscript. 5. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information. 6. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice. [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Yes Reviewer #2: Yes ******** 2. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: Yes Reviewer #2: Yes ****** 3. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: Yes Reviewer #2: No ****** 4. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes Reviewer #2: Yes ****** 5. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: The authors sought to develop a risk assessment tool to predict 1-year mortality risk for elderly people with multimultimorbidity (i.e., 3+ chronic health diseases) and polypharmacy (i.e., use of 5+ drugs for 30+ days). This is a secondary analysis of 805 relevant participants from the OPERAM study, a European multicenter, cluster randomized controlled trial. Multiple imputation was performed for missing data, ranging from 0% to 10% under the assumption of missing at random. The predictor candidates were predefined and a backward stepwise selection was conducted to identify the optimal predictive model. Internal validation was carried out using bootstrapping techniques. The optimal model had the optimism-corrected C index of 0.70 and calibration slop of 0.93. A simpler model, excluding Charlson Comorbidity Index and Barthel Index with C index of 0.59 (95% CI: 0.58, 0.59) was considered of “satisfactory discriminative power”. The manuscript is overall informative and well written. The study was well designed and executed, and the analysis was conducted appropriately. There are however several issues the authors might wish to consider in the subsequent submission. 1. Please clarify why a backward stepwise selection was used to identify the optimal selection model whereas the other methods, including Bayesian model averaging (BMA) method that have been demonstrated to be statistically more robust are widely available. Wang et al (https://pubmed.ncbi.nlm.nih.gov/15505893/) and Genell et al (https://pubmed.ncbi.nlm.nih.gov/21134252/) have found BMA was superior to stepwise method in model selection. Selection of a robust statistical method can be an important discussion issue to convince the findings were rigorous and accurate. 2. It is acceptable to report a calibration slope to assess the predictive performance of a prediction model. As a calibration slop evaluates the spread of the estimated risk, the manuscript should also report the calibration intercept, known as the “calibration-in-the-large” index which quantifies the overall calibration performance or the difference between the average predicted risk and the overall event rate. 3. It is also acceptable to derive a point-based risk score by assigning points to each predictor in the optimal model. However, the authors might wish to consider a more robust and friendly alternative approach, such as a nomogram analysis. Unlike a point-based risk score, a nomogram provides more graphically intuitive and more friendly risk estimation for subjects with specific risk profiles. A nomogram also provides a practical solution for the implementation progress and makes the manuscript more appealing. 4. A simpler model without Charson-Comorbidity index and Barthel index is considered satisfactorily discriminative though its C index was only 0.59. Please provide explanation and cite valid references for this statement as it is practically a bit hard, especially for people working in daily clinical practice to consider a discrimination index of 0.59 satisfactory. Further discussion is definitely needed. 5. The authors are expected to expand the Discussion section. For instance, they might wish to discuss (i) the robustness of the methods used in this study in comparison with previous studies or other methods, (ii) the results, including but not limited to a “satisfactory discriminative power” of 0.59, and (iii) potential implication of the model. 6. Minor issues: The authors might wish to make their study rationale stronger and more appealing. For instance, they might add a sentence or two to extrapolate the reasons why several of 16 prognostic indices were “fairly accurate to predict mortality, the authors concluded that none could be recommended for a widespread use”. Reviewer #2: Important topic, clear rationale of the study. I would like to congratulate the authors with a clearly written, and concise paper. I do have a few comments: -the statistical analyses are sophisticated, and can be considered state-of-the-art in the field of health outcome predictions -I do miss an independent population to assess external validity of the model -The paper could be further strengthend by comparing performance of the model with other (simple) models that predict mortality. Is this model really more appropriate than those currently available? -as the authors do acknowledge in the discussion, assessing performance in an extended time horizon beyond 1-year would be of interest Thank you for having me as reviewer ****** 6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #1: Yes: Thach Tran Reviewer #2: Yes:** Silvan Licher [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step. https://doi.org/10.1371/journal.pone.0271923.r001
Revision 1
28 Apr 2022 Author Response April, 2022 RE: PONE-D-22-03428 – “Development and validation of a new prognostic index for mortality risk in multimorbid adults” To the Editor, We thank you for your interest and reviewing our manuscript. We are grateful for the editor’s and reviewers’ comments which helped us to clarify and improve our manuscript. We have addressed all comments raised by the editor and the reviewers in the edited version of the manuscript and in our point-by-point responses to the reviewers' comments. All references quoted in our responses are listed at the end of this letter. We have uploaded a “Revised Manuscript with Track Changes” showing revisions using tracked changes. We hope that the revised version of our manuscript will be suitable for publication in PLOS ONE. We are looking forward to hearing from you, Kind regards Viktoria Gastens, MSc Institute of Primary Health Care (BIHAM), University of Bern Population Health Laboratory (#PopHealthLab), University of Fribourg Email: viktoria.gastens@biham.unibe.ch Journal Requirements: When submitting your revision, we need you to address these additional requirements. Authors’ response: We thank the editor for their comments below allowing us to improve our manuscript. 1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf Authors’ response: We have adapted the style of the manuscript according to the PLOS ONE formatting requirements. 2. Thank you for stating the following in the Funding Section of your manuscript: "This work was supported by Swiss National Science Foundation grants (320030_188549/01 to AC; 325130_204361 to CDG). This work is part of the project “OPERAM: OPtimising thERapy to prevent Avoidable hospital admissions in the Multimorbid elderly” supported by the European Union's Horizon 2020 research and innovation programme under the grant agreement No 6342388, and by the Swiss State Secretariat for Education, Research and Innovation (SERI) under contract number 15.0137. The opinions expressed and arguments employed herein are those of the authors and do not necessarily reflect the official views of the EC and the Swiss government." We note that you have provided funding information. However, funding information should not appear in the Funding section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form. Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows: "This work was supported by Swiss National Science Foundation grants (320030_188549/01 to AC; 325130_204361 to CDG; www.snf.ch). This work is part of the project “OPERAM: OPtimising thERapy to prevent Avoidable hospital admissions in the Multimorbid elderly” supported by the European Union's Horizon 2020 research and innovation programme under the grant agreement No 6342388 (ec.europa.eu/programmes/horizon2020), and by the Swiss State Secretariat for Education, Research and Innovation (SERI) under contract number 15.0137 (www.sbfi.admin.ch/sbfi/en/home.html). The opinions expressed and arguments employed herein are those of the authors and do not necessarily reflect the official views of the EC and the Swiss government. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript." Please include your amended statements within your cover letter; we will change the online submission form on your behalf. Authors’ response: We have deleted the funding section in the manuscript. The funding information in the Funding Statement section of the online submission form is correct. 3. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For more information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. In your revised cover letter, please address the following prompts: a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially sensitive information, data are owned by a third-party organization, etc.) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent. b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. We will update your Data Availability statement on your behalf to reflect the information you provide. Authors’ response: We have updated the Data Availability statement accordingly: “This study involves human research participant data containing sensitive patient information. In the EU Horizon 2020 grant agreement for the OPERAM study, it had been specified that the data will be made available upon request if the use has been approved by an ethical committee. Therefore, restrictions to make the underlying data directly publicly available are both due to legal and ethical reasons, as health data are sensitive data. Data for this study will be made available for scientific purposes upon request for researchers whose proposed use of the data has been approved by the OPERAM publication committee. After approval and signing of a data transfer agreement ensuring adherence to privacy and data handling, data and documentation will be made available through a secure file exchange platform. Partially deidentified participant data, a data dictionary and annotated case report forms will be made available. For data access, external researchers can fill in the contact form on https://www.operam-cohort.biham.ch/ or operam-2020.eu.” 4. Your ethics statement should only appear in the Methods section of your manuscript. If your ethics statement is written in any section besides the Methods, please move it to the Methods section and delete it from any other section. Please ensure that your ethics statement is included in your manuscript, as the ethics statement entered into the online submission form will not be published alongside your manuscript. Authors’ response: We have moved the ethics statement to the Methods section of our manuscript. 5. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information. Authors’ response: We have adapted the Supporting Information files and citations accordingly. 6. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice. Authors’ response: We have reviewed our reference list. We have added references to: “Royston and Sauerbrei; Moons 2015”. This has led to a change in the order of references. Reviewer 1 The authors sought to develop a risk assessment tool to predict 1-year mortality risk for elderly people with multimultimorbidity (i.e., 3+ chronic health diseases) and polypharmacy (i.e., use of 5+ drugs for 30+ days). This is a secondary analysis of 805 relevant participants from the OPERAM study, a European multicenter, cluster randomized controlled trial. Multiple imputation was performed for missing data, ranging from 0% to 10% under the assumption of missing at random. The predictor candidates were predefined and a backward stepwise selection was conducted to identify the optimal predictive model. Internal validation was carried out using bootstrapping techniques. The optimal model had the optimism-corrected C index of 0.70 and calibration slop of 0.93. A simpler model, excluding Charlson Comorbidity Index and Barthel Index with C index of 0.59 (95% CI: 0.58, 0.59) was considered of “satisfactory discriminative power”. The manuscript is overall informative and well written. The study was well designed and executed, and the analysis was conducted appropriately. There are however several issues the authors might wish to consider in the subsequent submission. Authors’ response: We thank the reviewer for their comments below allowing us to improve our manuscript. 7. Please clarify why a backward stepwise selection was used to identify the optimal selection model whereas the other methods, including Bayesian model averaging (BMA) method that have been demonstrated to be statistically more robust are widely available. Wang et al (https://pubmed.ncbi.nlm.nih.gov/15505893/) and Genell et al (https://pubmed.ncbi.nlm.nih.gov/21134252/) have found BMA was superior to stepwise method in model selection. Selection of a robust statistical method can be an important discussion issue to convince the findings were rigorous and accurate. Authors’ response: We agree with the reviewer that stepwise variable selection can have some disadvantages such as instability of selection or biased estimation. However, according to the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) statement, backwards elimination is still recommended as an automatic selection procedure. [Royston and Sauerbrei 2008; TRIPOD 2015] We have added to our Methods section: “If using automatic selection procedures, backward elimination is recommended in TRIPOD.” 8. It is acceptable to report a calibration slope to assess the predictive performance of a prediction model. As a calibration slop evaluates the spread of the estimated risk, the manuscript should also report the calibration intercept, known as the “calibration-in-the-large” index which quantifies the overall calibration performance or the difference between the average predicted risk and the overall event rate. Authors’ response: We agree with the reviewer and have added the calibration intercept to the Results section in Table 3A and 3B. 9. It is also acceptable to derive a point-based risk score by assigning points to each predictor in the optimal model. However, the authors might wish to consider a more robust and friendly alternative approach, such as a nomogram analysis. Unlike a point-based risk score, a nomogram provides more graphically intuitive and more friendly risk estimation for subjects with specific risk profiles. A nomogram also provides a practical solution for the implementation progress and makes the manuscript more appealing. Authors’ response: We agree with the reviewer that nomogram analysis is a graphically intuitive representation. However, nomograms represent models with continuous variables. Due to the binary and categorical characteristics of our predictor variables, a nomogram analysis is not applicable. For the next part of this research project (extending to 3-year mortality risk prediction), we plan to keep variables continuous wherever possible and could therefore apply a nomogram analysis. [Gastens 2021] 10. A simpler model without Charson-Comorbidity index and Barthel index is considered satisfactorily discriminative though its C index was only 0.59. Please provide explanation and cite valid references for this statement as it is practically a bit hard, especially for people working in daily clinical practice to consider a discrimination index of 0.59 satisfactory. Further discussion is definitely needed. Authors’ response: We agree with the reviewer and revised the text as follows: “A simpler score was able to categorize 1-year mortality risk with a weaker discriminative power.” and “This simpler score showed reduced discriminative power compared to the full score.” 11. The authors are expected to expand the Discussion section. For instance, they might wish to discuss (i) the robustness of the methods used in this study in comparison with previous studies or other methods, (ii) the results, including but not limited to a “satisfactory discriminative power” of 0.59, and (iii) potential implication of the model. Authors’ response: We have expanded our Discussion section about the robustness of the methods: “We have applied a particularly robust methodological framework by following the research guidelines in this field, namely the Prognosis Research Strategy (PROGRESS) framework, and the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement. Notably, we ensured adequate sample size, used multiple imputation for missing data, and applied bootstrapping techniques for internal validation.” We further discuss the results: “In contrast to existing mortality risk indices, our index focuses specifically on multimorbid older patients and accounted for the severity of comorbidity (Charlson-Comorbidity-Index) and functional impairment (Barthel-Index).” We discuss potential implications: “Our results will be useful for both clinical and research activities by helping health care providers to tailor preventive care according to the estimated mortality risk. Eventually, our study can help preventing under- and overuse of preventive care in the growing older population.” 12. Minor issues: The authors might wish to make their study rationale stronger and more appealing. For instance, they might add a sentence or two to extrapolate the reasons why several of 16 prognostic indices were “fairly accurate to predict mortality, the authors concluded that none could be recommended for a widespread use”. Authors’ response: We agree with the reviewer and have strengthened our Introduction section: “One major limitation is that none of these prognostic indices has been tested prospectively in various samples. Key is that their transportability in other populations is unknown and clinicians cannot use these indices with confidence across different groups of patients.” Reviewer 2 Important topic, clear rationale of the study. I would like to congratulate the authors with a clearly written, and concise paper. I do have a few comments: Authors’ response: We thank the reviewer for their comments below allowing us to improve our manuscript. 13. -the statistical analyses are sophisticated, and can be considered state-of-the-art in the field of health outcome predictions Authors’ response: Thank you. 14. -I do miss an independent population to assess external validity of the model Authors’ response: We agree with the reviewer and have stated in the Discussion section: “One major limitation is the lack of external validation but we will explore opportunities to test the score in a different dataset of older multimorbid adults. We did however our best to assess internal validation by using the bootstrap method for internal validation that makes use of the entire sample.” 15. -The paper could be further strengthend by comparing performance of the model with other (simple) models that predict mortality. Is this model really more appropriate than those currently available? Authors’ response: We are not sure to understand this comment, because we have compared our model with a simplified one and because there are no other models currently available. These points have been discussed in the Introduction and Discussion section of the paper. 16. -as the authors do acknowledge in the discussion, assessing performance in an extended time horizon beyond 1-year would be of interest Authors’ response: We agree with the reviewer. As discussed in the Discussion section, the follow-up of our cohort study is still ongoing and we are currently working on extending the prognostic model to 3-year mortality risk: “Finally, prognostic information longer than 1-year mortality risk is needed. We will expand this model to 3-year mortality risk once the 3-year follow data collection is completed. [Gastens 2021]” Comments from authors During the revision process, we noticed a slight error in the calculation of the Charlson-Comorbidity-Index. We adapted the results accordingly. This did not change the findings in a substantial way or interpretation of the results. References Moons, Karel GM, et al. "Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration." Annals of internal medicine 162.1 (2015): W1-W73. Royston, Patrick, and Willi Sauerbrei. Multivariable model-building: a pragmatic approach to regression anaylsis based on fractional polynomials for modelling continuous variables. Vol. 777. John Wiley & Sons, 2008. Gastens, Viktoria, et al. "Development and validation of a life expectancy estimator for multimorbid older adults: a cohort study protocol." BMJ open 11.8 (2021): e048168. https://doi.org/10.1371/journal.pone.0271923.r002
26 May 2022 Decision Letter - Robert Daniel Blank, Editor PONE-D-22-03428R1Development and validation of a new prognostic index for mortality risk in multimorbid adultsPLOS ONE Dear Dr. Gastens, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Model performance must be thoroughly discussed. Please submit your revised manuscript by Jul 10 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript: A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols. We look forward to receiving your revised manuscript. Kind regards, Robert Daniel Blank, MD, PhD Academic Editor PLOS ONE Journal Requirements: Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice. [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation. Reviewer #1: (No Response) ******** 2. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Partly ****** 3. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: Yes ****** 4. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: Yes ****** 5. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes ****** 6. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: I would like to thank the authors for their efforts to address my concerns. There are however two issues that I appreciate their clarification. 1. Please explain and cite relevant references for your response “However, nomograms represent models with continuous variables. Due to the binary and categorical characteristics of our predictor variables, a nomogram analysis is not applicable.”. The authors might find in the current literature that the nomograms with binary or categorical predictor variables are common in many different research fields, including but not limited to oncology (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4465353/), bone (https://pubmed.ncbi.nlm.nih.gov/17370100/ ) and other research fields (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7906717/; https://pubmed.ncbi.nlm.nih.gov/31132089/ ). If the authors decide not to present the nomogram which they agree to be graphically intuitive, they might wish to add a sentence or two in the Discussion section to mention the nomogram analysis might be considered in the extended project. 2. The optimism corrected validation analysis using bootstrapping technique indicated both calibration slop and intercept significantly deviated from the null value of one (0.93; 95% CI: 0.92, 0.94) and zero (0.61; 0.56, 0.66) (Table 3A), respectively. The far less accurate calibration was noticed for the simplified model in Table 3B. These findings indicated the significant suboptimal prediction accuracy of these prediction models as the risk estimates were considered too moderate (for the calibration slop) or the model significantly underestimated the predicted risk (for the intercept) (https://pubmed.ncbi.nlm.nih.gov/31842878/ ). Regardless of their importance and significant, these results have not been discussed thoroughly in the current manuscript. The authors are expected to discuss sufficiently about the significantly suboptimal calibration of the prediction models. They might also wish to discuss the potential impact of this suboptimal prediction accuracy on the implementation of the model and future projects. Additionally, the authors might wish to cross check the assumption for calculating the 95% CI of the calibration intercept given its upper limit of 1.24 (Table 3B). 3. It is statistically very hard to interpret the simplified prediction model had “slightly weaker discrimination power” as its C index (0.59; 95% CI: 0.58, 0.59) appeared to be significantly lower than the “full” prediction model (0.70; 0.69, 0.70). Similar results were found for both calibration intercept and slop. These metrics instead the simplified prediction model had significantly weaker prediction performance, in terms of both discrimination and calibration power than the full prediction model. As a result, the authors might also wish to revise the interpretation related to the simplified prediction model accordingly and discuss its potential implication further. ****** 7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review?** For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #1: No [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step. https://doi.org/10.1371/journal.pone.0271923.r003
Revision 2
8 Jul 2022 Author Response Reviewer 1 I would like to thank the authors for their efforts to address my concerns. There are however two issues that I appreciate their clarification. 1. Please explain and cite relevant references for your response “However, nomograms represent models with continuous variables. Due to the binary and categorical characteristics of our predictor variables, a nomogram analysis is not applicable.”. The authors might find in the current literature that the nomograms with binary or categorical predictor variables are common in many different research fields, including but not limited to oncology (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4465353/), bone (https://pubmed.ncbi.nlm.nih.gov/17370100/) and other research fields (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7906717/; https://pubmed.ncbi.nlm.nih.gov/31132089/ ). If the authors decide not to present the nomogram which they agree to be graphically intuitive, they might wish to add a sentence or two in the Discussion section to mention the nomogram analysis might be considered in the extended project. Authors’ response: We agree with the reviewer and have added to the Discussion section “We will expand this model to 3-year mortality risk once the 3-year follow-up data collection is completed. [Gastens 2021] For this next research project, we might consider nomogram analysis as a graphically intuitive representation. [Balachandran 2015]”. 2. The optimism corrected validation analysis using bootstrapping technique indicated both calibration slop and intercept significantly deviated from the null value of one (0.93; 95% CI: 0.92, 0.94) and zero (0.61; 0.56, 0.66) (Table 3A), respectively. The far less accurate calibration was noticed for the simplified model in Table 3B. These findings indicated the significant suboptimal prediction accuracy of these prediction models as the risk estimates were considered too moderate (for the calibration slop) or the model significantly underestimated the predicted risk (for the intercept) (https://pubmed.ncbi.nlm.nih.gov/31842878/). Regardless of their importance and significant, these results have not been discussed thoroughly in the current manuscript. The authors are expected to discuss sufficiently about the significantly suboptimal calibration of the prediction models. They might also wish to discuss the potential impact of this suboptimal prediction accuracy on the implementation of the model and future projects. Additionally, the authors might wish to cross check the assumption for calculating the 95% CI of the calibration intercept given its upper limit of 1.24 (Table 3B). Authors’ response: We agree with the reviewer and have added to our Discussion section: “This optimism-corrected validation analysis indicates that both calibration slope and intercept deviate from the null value of one (0.93; 95% CI: 0.92, 0.94) and zero (0.61; 95% CI: 0.56, 0.66) (Table 3A), respectively. The calibration slope evaluates the spread of the estimated risks and has a target value of 1. A slope < 1 suggests that estimated risks are too extreme, a slope > 1 suggests that risk estimates are too moderate. [Van Calster 2019] The calibration intercept, which is an assessment of calibration-in-the-large, has a target value of 0; negative values suggest overestimation, whereas positive values suggest underestimation. [Van Calster 2019] Therefore, our results indicate suboptimal prediction accuracy as the risk estimates were considered too extreme (for the calibration slope) and the model underestimating the predicted risk (for the calibration intercept).”. Additionally, we checked the calculations for the 95% CI of the calibration intercept and confirm its upper limit of 1.24 (Table 3B). 3. It is statistically very hard to interpret the simplified prediction model had “slightly weaker discrimination power” as its C index (0.59; 95% CI: 0.58, 0.59) appeared to be significantly lower than the “full” prediction model (0.70; 0.69, 0.70). Similar results were found for both calibration intercept and slop. These metrics instead the simplified prediction model had significantly weaker prediction performance, in terms of both discrimination and calibration power than the full prediction model. As a result, the authors might also wish to revise the interpretation related to the simplified prediction model accordingly and discuss its potential implication further. Authors’ response: We agree with the reviewer and have adapted our Discussion section to “Another limitation is that indices such as the Barthel-Index and the Charlson-Comorbidity-Index used as predictor may reduce the ease of use of the risk score at the point-of-care. We have therefore developed a simpler score without such variables. This simpler score showed reduced discriminative power compared to the full score. This could highlight the importance of taking the severity of comorbidities and functional impairment into account to improve risk prediction in older multimorbid people.”. https://doi.org/10.1371/journal.pone.0271923.r004
11 Jul 2022 Decision Letter - Robert Daniel Blank, Editor Development and validation of a new prognostic index for mortality risk in multimorbid adults PONE-D-22-03428R2 Dear Dr. Gastens, We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication. An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org. If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. Kind regards, Robert Daniel Blank, MD, PhD Academic Editor PLOS ONE Additional Editor Comments (optional): Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation. Reviewer #1: All comments have been addressed ******** 2. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Yes ****** 3. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: Yes ****** 4. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: Yes ****** 5. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes ****** 6. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: I would like to thank the authors for their efforts to address my concerns and make the manuscript better. ****** 7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #1: No ******** https://doi.org/10.1371/journal.pone.0271923.r005
Formally Accepted
27 Jul 2022 Acceptance Letter - Robert Daniel Blank, Editor PONE-D-22-03428R2 Development and validation of a new prognostic index for mortality risk in multimorbid adults Dear Dr. Gastens: I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department. If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org. If we can help with anything else, please email us at plosone@plos.org. Thank you for submitting your work to PLOS ONE and supporting open access. Kind regards, PLOS ONE Editorial Office Staff on behalf of Professor Robert Daniel Blank Academic Editor PLOS ONE https://doi.org/10.1371/journal.pone.0271923.r006

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .