Dear Dr. Bellasi,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Feb 17 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org . When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols . Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols .

We look forward to receiving your revised manuscript.

Kind regards,

Fabio Sallustio, PhD

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1.Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please note that PLOS ONE has specific guidelines on code sharing for submissions in which author-generated code underpins the findings in the manuscript. In these cases, we expect all author-generated code to be made available without restrictions upon publication of the work. Please review our guidelines at https://journals.plos.org/plosone/s/materials-and-software-sharing#loc-sharing-code and ensure that your code is shared in a way that follows best practice and facilitates reproducibility and reuse.

3. We note that you have indicated that there are restrictions to data sharing for this study. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For more information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

Before we proceed with your manuscript, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information, data are owned by a third-party organization, etc.) and who has imposed them (e.g., a Research Ethics Committee or Institutional Review Board, etc.). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of recommended repositories, please see

https://journals.plos.org/plosone/s/recommended-repositories. You also have the option of uploading the data as Supporting Information files, but we would recommend depositing data directly to a data repository if possible.

We will update your Data Availability statement on your behalf to reflect the information you provide.

Additional Editor Comments:

Please, address the methodological issues of the model raised by reviewers, including the calibration and the external validation.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No

Reviewer #4: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously? -->?>

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No

Reviewer #4: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available??>

The PLOS Data policy

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English??>

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

**********

Reviewer #1: Hello! I just reviewed the article entitled "Usability of Machine Learning Algorithms based on Electronic Health Records for the Prediction of Acute Kidney Injury and Transition to Acute Kidney Disease: a Proof of Concept Study" by Lorenzo Ruinelli

et al. With this study, the authors proved that ML algorithms could predict AKI and/or AI to AKD transition. Overall, the study is well-written, in good English, with a goo flow and logical. Before considering publishing the article, I have some suggestions for the authors, as minor revisions:

1. Please revise the continuous variables regarding the normal distribution. I saw some high SD compared to the means and this suggest that the data is not normally distributed.

2. In table 2 and supplemental table 2, you present the estimated GFR 1st. I assume this is the calculated GFR from day 1 of hospital admission. Due to the fact that most of the patients presented already with AKI in the first day of hospital admission, I consider the use of eGFR to be inappropriate. The use of eGFR in context of acute kidney injury is unreliable. Nevertheless, you could replace the eGFR from day one with baseline eGFR - calculated with from the baseline creatinine. In this case, I would expect that patients who develop AKD to present lower baseline eGFR.

3. An important factor for transition to AKD from AKI would be the AKI cause. I consider that you should add in the limitations paragraph the absence of AKI cause in your model. Intrinsic AKI is proved to increase the risk of AKI prolongation towards AKD. Given this aspect, you indeed presented in the discussion section the clinical importance of AKI and/or AKI to AKD progression. I consider you could expand this paragraph and discuss the clinical solutions stratified by AKI causes (prerenal, intrinsic and postrenal - as many of AKI were from the urology department). Nevertheless, you should discuss the importance of the nephrotoxic medication avoidance in context of AKI.

4. Overall, a good article, logical and great flow. In the end, I would like to add that besides the diagnosis and prediction of AKI/AKD occurrence, the recognition of AKI is of great importance. As you clearly stated in your article, the AKI diagnosis rate was reduced and the nephrology reference was even lower.

5. Good luck!

Reviewer #2: Dear Authors,

I have read your work with great interest. It presents a highly intriguing application of AI in a very active research area. However, I would like to offer some observations and requests to enhance the paper’s clarity, which might be challenging to follow, and to highlight aspects that I believe could be valuable for the readers:

Incidence of AKI

The AKI incidence reported in your study appears to be lower than what is typically documented in the literature, which estimates it to be around 20% in hospitalized patients (DOI: 10.1016/S0140-6736(16)30240-9). Do you have any opinions or comments on this discrepancy?

Definition of AKI

The AKI definition used in your study is unclear. According to KDIGO guidelines, AKI diagnosis requires a creatinine increase within 48 hours or over 7 days from a baseline value. In your study, however, the baseline value could date back up to one year. Additionally, when no baseline value was available, you diagnosed AKI based on a two-day creatinine increase, but what was this increase compared against?

Definition of Renal Recovery

How was renal recovery defined in your study?

AKI Analysis

It is important to emphasize that AKI occurring during the first days of hospitalization differs from that developing later. Notably, there is a well-documented distinction between community-acquired AKI and hospital-acquired AKI, which may have different causes and outcomes (DOI: 10.1053/j.ajkd.2023.10.009, DOI: 10.2215/CJN.07920713).

For example, a recent prospective study demonstrated that the primary risk factor for community-acquired AKI is the use of nephrotoxic drugs (DOI: 10.1016/j.ejim.2024.09.016). However, this study lacks a detailed analysis of AKI causes and does not consider this factor.

Another observation is related to the models used to assess AKD risk in this study. These models are based on data collected during the early days of hospitalization, excluding AKI cases that develop later—cases that represent true hospital-acquired AKI and are generally associated with worse prognoses.

I suggest that the authors discuss these aspects in more detail (some of which are already partially addressed) and include them as study limitations.

Comparison of AKI and AKD Patients

A comparison between AKI patients who recover and those who progress to AKD would be interesting. Currently, Supplementary Table 2 lacks statistical analysis for this comparison.

SHAP Analysis Variables

The SHAP analyses include variables that are not explained, such as sCr MAX/SD/LAST. What do these variables represent, and how do they differ from sCr baseline? Additionally, how are the deltas calculated? Providing explanations for these factors would enhance the paper’s comprehensiveness.

Discrepancy Between Administrative and Biochemical Data

A recent study highlighted the discrepancy between administrative and biochemical data in AKI diagnosis and showed the existence of different patterns of AKI recognition in hospitals (DOI: 10.1093/ckj/sfae231). Including a discussion on this point could be useful, particularly for guiding future research.

Supplementary Figure 1 is unclear and could benefit from further clarification.

Reviewer #3: The study concerns a topic not treated in deep, which is of interest in a large number of patients admitted out of Intensive Care Units.

I have some key questions on the methods and results of this study.

The process of evaluation of machine learning algorithms is lacking of two steps: the first is the calibration and the second is the external validation.

The calibration test which is useful to calibrate the predictive model.

The external test set, which is necessary to prove the model performance and its ability to generalize on different geographical populations.

Consequently I suggest to review in depth the process of ML algorithms generation, because we don’t know if the model is well calibrated and if the algorithm can be widely applied.

Other aspects:

1) AKI and AKD occurrence and association with outcome. The authors report 3305 subjects who developed AKI during hospital stay. Then they report AKI in the first 24h = 1362 pts, in the 48 h 2279 and in the 72 hrs 2556. I think that it should be better to describe AKI periods: 0-24hrs, 25-48 hrs, 49-72 hrs, > 72hrs. This information can be added to supplemental figure 2.

2) A better explanation is needed for AKD transition. In this case it is not clear the definition of AKD because only 47.8% of patients with AKI in the first 24 hrs completed 7 days of observation needed for AKD assessment (see definition of AKD reported on methods sections).

3) Positive predictive value is reported, but the percentage is low. A comment is needed. Could we consider useful a prediction model with such as margin of error? In discussion this aspect is not reported or commented.

Reviewer #4: The study is very articulate and analyzed an interesting role for ML in predicting AKI or AKD occurrence in acute hospitalized patients. Nevertheless, some methodological limits (e.g. retrospective, mono-centric and selection criteria) I think this study may add interesting insights for ML in clinical practice. Below, the Authors can find my comments about the manuscript.

Minor revisions

A) Abstract

- "AKI and AKD" are used appropriately, but the definition of these conditions could be included, considering that the abstract is addressed to an audience that may not be familiar with these conditions. (e.g. According to the KDIGO guidelines?). Is it due to the word-limit?

- I suggest reformulating as follow: “…The majority of AKI episodes (77%) occurred within the first three days of hospitalization”.

- “AKI and AKD complicated 10% and 1.5% of hospitalizations”: respectively?

B) Aims

-Better “The aims of the study are as follows:” and “to minimize right censoring due to patient mortality or discharge before the completion of 7 days of observation following the onset of AKI;”.

Major revisions

Some comments for the Authors, to which I would appreciate their responses.

1) The main limitation of the Study may be the reduced representativeness of the enrolled patient population, as it excludes groups with significant renal comorbidities. However, if the aim is to develop a highly specific model for predicting AKI in a low-risk population, this exclusion may be appropriate." Indeed, by excluding patients with severe pre-existing renal disease, the dataset likely includes a higher proportion of individuals with low-risk for AKI or AKD and potentially increasing the NPV. This may limit the ML model's generalizability, as excluding patients with severe renal impairment reduces dataset representativeness for populations with higher prevalence of CKD or significant comorbidities

2) The exclusive use of sCr to define AKI and AKD, while widely accepted, may not fully include the complexity of AKI. sCr is influenced by factors such as age, sex, muscle mass, and hydration status, potentially distorting renal function assessment. Additionally, in patients with pre-existing conditions like dehydration or malnutrition, sCr may not accurately reflect renal function. For this reason, it would be interesting to have at least the reason for hospitalization and some information on comorbidities to better guide the clinical utility of the ML model.

3) The extended definition of AKI, including serum creatinine increases >26.5 μmol/L or >1.5 times baseline, may lead to overdiagnosis by identifying minor, clinically non-significant renal function fluctuations as AKI. While it improves early detection, these minor changes often may lack prognostic relevance, potentially increasing sensitivity at the expense of specificity affecting the clinical power of ML. Please, Authors comment this aspect in the text.

4) The retrospective design depends on potentially incomplete EHR data, which may affect accuracy. This could be a limitation for generalization that needs to be better considered in the discussion.

5) Right censoring due to death, discharge, or missing renal data within 7 days could skew results. The progressively smaller datasets for AKD models (T1-T4) may reduce ML model’s stability and generalizability. Additionally, time-specific models may fail to capture slower or atypical disease progressions, limiting their applicability.

In details, other comments about Discussion and Conclusion that I underline to the Authors.

- Only 35% of AKI and 52% of AKD cases were recorded, with a notably low rate of nephrology referrals. Could this suggest a significant proportion of events clinically not impactful?

- The median follow-up duration of 4.1 days, combined with right censoring due to discharge or death, may have led to an overestimation of the ML’s ability to predict long-term AKD outcomes. This requires further discussion.

- Serum creatinine (sCr) data were available in only 48% of cases. This limitation could significantly impact the ML model’s accuracy and should be addressed in the discussion.

- The AUC-ROC values (0.76–0.88), alongside missing data on patient comorbidities and medication use, are critical limitations that could affect both the ML model's performance and its clinical applicability. These aspects deserve more comprehensive consideration.

- The study is based on retrospective data from a single healthcare system. This raises concerns about the generalizability of the findings, which should be better contextualized in the discussion.

- Finally, it may be valuable to explore the relationship between hospitalization length and the occurrence of AKI in more detail. That Said, could Supplemental Figure 3 be integrated into the main text to provide greater clarity and emphasis?

**********

what does this mean? ). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

Reviewer #4: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/ . PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org . Please note that Supporting Information files do not need this step.

Dear Dr. Bellasi,

Please provide minor revision requested, in particular:-Please try to describe a hypothetical sample size in order to indicate the statistical power of the ML results and their associated NPV, particularly given the low incidence of AKI and AKD.- Please address the points from Reviewer 3, in addition to further points from Reviewer 4 and if not possible include these limitations in the discussion.

Please submit your revised manuscript by May 15 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org . When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols . Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols .

We look forward to receiving your revised manuscript.

Kind regards,

Fabio Sallustio, PhD

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

Reviewer #3: (No Response)

Reviewer #4: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions??>

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No

Reviewer #4: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously? -->?>

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No

Reviewer #4: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available??>

The PLOS Data policy

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English??>

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

**********

Reviewer #1:  The authors performed the changes. They answered all the questions and modified the manuscript accordingly

I have nothing to add. Good luck!

Reviewer #2:  Dear Authors,

I found your work interesting from the start, but I believe it has been significantly improved thanks to the revisions received and your subsequent modifications. I think this has been an excellent team effort. I have no further comments.

Reviewer #3:  The revision form of the study “usability of machine learning algorithms for the prediction of AKI and transition to AKD” of Ruinelli and co-authors still now hasn’t answered to the points previously underlined.

Mainly three aspects appear to be limiting.

The first is concerning the mathematical process of machine learning.

The external validation consists on the testing of model in a different “external” group of patients, but in this case patients of external validation are those “internal” of hospital database. Therefore there isn’t an external validation.

The external test is necessary to prove the model performance and its ability to generalize on different geographical populations. (Ref. Alfieri et al, PLoS ONE 18(7): e0287398. https://doi.org/ 10.1371/journal.pone.0287398).

The second point is the calibration test. When a curve is generated, a Brier Score can be computed, and a Bias detection can be made. This aspect is not treated in the present study.

The third point is the very low Positive Predictive Value.

All these aspects deeply limit the study.

Reviewer #4:  Some other comments

- Please verify the font size, as it appears inconsistent across the first three lines.

- Was septic state considered in the model?

- I still believe it is essential to describe a hypothetical sample size in order to indicate the statistical power of the ML results and their associated NPV, particularly given the low incidence of AKI and AKD, as mentioned by the authors.

- A possible consideration, in addition to the absence of concomitant therapies, could be the potential role of gender in the progression to AKI or AKD. Are the tools included in the model represented or describe even in supplementary?

- As pointed out by the authors, the generalizability of the results remains debatable due to the lack of an external validation system for the ML model, and the cohort bias may be a significant factor influencing the results (low prevalence of AKI and AKD).

Despite these considerations, the work provides valuable insights and is well-structured and rigorous.

**********

what does this mean? ). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy

Reviewer #1: Yes:  Lazar Chisavu

Reviewer #2: No

Reviewer #3: No

Reviewer #4: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/ . PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org

Usability of Machine Learning Algorithms based on Electronic Health Records for the Prediction of Acute Kidney Injury and Transition to Acute Kidney Disease: a Proof of Concept Study

PONE-D-24-42164R2

Dear Dr. Bellasi,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager®  and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Fabio Sallustio, PhD

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments: