PONE-D-21-11825

Rush order containment of critical drugs in ICUs

PLOS ONE

Dear Dr. Cappanera,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Aug 15 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Behzad Behdani

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for stating in your Funding Statement:

 [The work by P. Cappanera, R. Rossi and F. Visintin has been partially supported by the LINFA (Logistica INtelligente del FArmaco) project, funded by Regione Toscana under the call PAR FAS 2007-2013, Linea d'azione 1.1 - Bando FAR FAS 2014.

https://www.regione.toscana.it/-/bando-far-fas-finanziamento-di-progetti-di-ricerca-nei-settori-energia-fotonica-ict-e-robotica

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.]. 

Please provide an amended statement that declares *all* the funding or sources of support (whether external or internal to your organization) received during this study, as detailed online in our guide for authors at http://journals.plos.org/plosone/s/submit-now.  Please also include the statement “There was no additional external funding received for this study.” in your updated Funding Statement.

Please include your amended Funding Statement within your cover letter. We will change the online submission form on your behalf.

3. We note that you have stated that you will provide repository information for your data at acceptance. Should your manuscript be accepted for publication, we will hold it until you provide the relevant accession numbers or DOIs necessary to access your data. If you wish to make changes to your Data Availability statement, please describe these changes in your cover letter and we will update your Data Availability statement to reflect the information you provide.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Summary

The authors present a decision support system to manage the inventory policy of medications with low and intermittent demand at the ICU setting. They propose a simulation approach to forecast the demand and create scenarios of usage within the hospital system. They solve the inventory problem using a MILP model based on the principles of thee Lot-sizing problem. The study focuses on demand estimation on a rolling horizon, incorporating the properties and medical guidelines related to drug consumption. Thus, the tool is driven by patterns of therapies rather than patterns of drugs. The computational experiments investigate the effect of the model parameters and variants and provide a direct comparison with a standard inventory policy frequently encountered in practice.

The paper proposes a flexible quantitative methodology to solve an important challenge faced by decision makers in practice. The presented analysis highlights the benefit of this approach and its potential to positively affect the efficiency of major healthcare systems. Below you will find my observations regarding specific issues that need to be addressed to improve the paper.

Major Comments

1. The paper would be greatly improved if it included a direct quantitative and qualitative comparison of the proposed solution with what has been currently implemented in practice. My biggest concern relates to the accuracy of the forecasting tool and thus I strongly suggest a one-to-one evaluation of the inventory planning framework with the results obtained at the hospital in the same period. In other words, the authors could run a hypothetical retrospective “trial” in which they use for some period of time the proposed system in parallel to what is currently implemented in the clinic. This analysis would highlight the true benefit and applicability of the decision support tool.

2. A suggestion would be to add a section regarding the deployment and implementation of the tool in practice. What are the challenges that need to be overcome for its integration? How would the interface for the decision maker look like? This addition in combination with the one-to-one comparison will significantly improve the manuscript.

3. I would recommend to modify the abstract and emphasizing more the key findings and conclusions of the investigation. The current structure is not very coherent.

4. Why does the forecasting tool not take into consideration the patients that are scheduled to be admitted to the hospital in the planning period in addition to those currently hospitalized? Scheduled surgeries for example depending on their severity are associated with the risk of ICU admission. Thus, they could be factored as part of the forecasting tool.

5. The section regarding the demand generation process and forecast needs to be substantially refined and expanded. The authors should discuss how they learn the probabilities of the proposed flowchart and in which ways the patient’s history and current medical conditions are used to adjust the estimation. Figure 1 shows a very simplified and abstract version of the generator. Technical details are missing and thus the analysis cannot be neither evaluated nor replicated.

6. Given the data that is available to the authors, I would recommend the use of machine learning techniques to estimate the future trajectory and the demand of medication at the personalized level. These methodologies do not require probabilistic assumptions and are able to devise complex, non-linear rules that result in accurate demand estimations.

7. How are the authors evaluating the accuracy of the forecasting methodology they propose? A retrospective analysis of this approach using data from the healthcare system in Italy would provide significant evidence regarding the validity of the proposed generator.

8. To account for the uncertainty in the demand estimation, I suggest to the authors to consider uncertainty sets and optimize against the worst-case scenario. By using robust optimization, they could yield better solutions that protect the hospital system against forecasting errors. For a review of robust optimization approaches, they can resort to the work of Bertsimas et al.:

Bertsimas, D., Brown, D.B. and Caramanis, C., 2011. Theory and applications of robust optimization. SIAM review, 53(3), pp.464-501.

9. The authors should provide a Github repository where researchers can review and access the code and the data from the paper.

Minor Comments

1. The introduction is barely citing other papers that support the authors’ claims. My suggestion would be to add references to other investigations that highlight some of the information that is provided by the authors regarding the drug replenishment at the ICU setting.

2. The first paragraph of the introduction can be split into two paragraphs that capture the two different main ideas discussed in this section.

3. It would be very helpful to the reader to provide an Algorithm overview or a visualization of the entire approach as described in the Introduction.

4. The authors could consider breaking some of the longer sentences in two to help the reader understand better the content of the paper (see Examples in the Introduction and Conclusions).

5. Line 6: Typo – makes it possible continuous review policies.

Reviewer #2: I think this paper is useful and well written.

Moreover, the authors followed rigorous scientific methods.

Here are my main five comments:

1- The paper is missing a proper validation section. This is important.

2- Fig. 1 is not clear, e.g., the path from “Clinician expertise” to “Other Lab Issue” is a Possible death! …

So, I suggest that the flow chart have two exit nodes. One exit is due to death and the other exit is due to (proper) treatment.

3- Hierarchical optimization is a classical problem in Operations Research.

E.g., this is an old classical reference:

Anandalingam, G., & Friesz, T. L. (1992). Hierarchical optimization: An introduction. Annals of Operations Research, 34(1), 1-11.

So, in general, there is no adequate literature review (in the paper) about hierarchical optimization. The aim of this literature review should be to identify the state-of-the-art hierarchical optimization methods to implement one of them (in the paper). Note the implementation of hierarchical optimization in the MILP model is not the state of the art. This is a weak point in the paper.

4- In section 5.3, the authors state the following:

"The number of entities to create is determined by sampling from suitable empirical distributions based on the data collected on-site during the observation period."

I do not understand the term: "suitable empirical distributions".

Moreover, I think it is better first to fit the empirical distribution to a "suitable" theoretical distribution; then to use this theoretical distribution in the simulation model.

5- Also in section 5.3, the authors state the following:

"The algorithm is executed in shell (i.e., while the model runs, the simulation clock does not advance),"

I think the authors mean: (i.e., while the algorithm -- not the model -- runs, ...).

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Mohamed Saleh

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PONE-D-21-11825R1Rush order containment of critical drugs in ICUsPLOS ONE

Dear Dr. Cappanera,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

==============================

ACADEMIC EDITOR:I would like to thank the authors for the improvements and the revised version. As you will see bellow there are some final points by the reviewer that you need to address before re-submission.

==============================

Please submit your revised manuscript by Feb 19 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Behzad Behdani

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: I would like to thank the authors for their time and effort in conducting a major revision of the manuscript. The paper has been significantly improved as it now provides an indirect comparison of the model’s proposed solution with the actual decisions implemented in practice in June 2016. Section 7 also illustrates an online version of the tool and discusses the implications and challenges associated with it in a real-world setting. The section regarding the demand generation process and forecasting has been expanded and the methodology proposed is now clearer to the reader. Below you will find my comments to the authors. I believe that with a minor revision the manuscript would be ready for publication.

1. In Section 4.1 the authors outline in detail the data sources that they used for their study. Note that Prosafe data were recorded between 01/2011-04/2016 and miss information during the stay. Microbiology data refer to a 6 year period from 01/2010-02/2016 and not a 5 year period as indicated in the paper. The authors also used 69 digitized admissions from June 2019. How was this information subsequently consolidated? Based on the information currently available in the paper, the authors could use the information for patients present in both the Prosafe data and the Microbiology data. However, the rest could not be combined. In addition, the 69 digitized admissions do not refer to the same period (2010-2016) and thus no information from Prosafe and microbiology could be applied for those patients. What was the resulting size of the dataset? How did the authors deal with missing values? It would be great to provide a Table summarizing the mean, standard deviation, and missing percentage for the final dataset. Perhaps this information could go into the appendix, but it remains critically important for the reproducibility of the study.

2. I would suggest the authors to present the R^2 of their computational experiments in predicting length of stay as it provides a comparison with a simple baseline. Perhaps they could consider the average length of stay for each class as the baseline and then analyse the improvement they achieve using the SVR kernel regression.

3. Please add a citation for the Relieff features ranking methodology.

4. The mean of sensitivity and specificity might be misleading for the interpretation of the results. The authors can report instead in Table 6 in different columns the average sensitivity, specificity, and accuracy for a given threshold as well as the overall out-of-sample AUC. The currently presented results are not very insightful from a statistical perspective. In addition, confidence intervals are needed around this metrics.

5. I think it would greatly improve the paper if the authors could add a graphical illustration of the steps necessary to apply the tool. In other words, the timing and sequence of using the forecasting tool and the simulation in combination with the optimization. Parts of that are already described in Section 4.3. This would be a more high-level implementation and workflow of the system compared to Figure 3.

6. What are the implications in terms of cost other than just frequency from the comparisons conducted in Section 6.2? By having better planned and less rush orders what would be the approximate impact on the costing of the department?

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Rush order containment of critical drugs in ICUs

PONE-D-21-11825R2

Dear Dr. Cappanera,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Behzad Behdani

Academic Editor

PLOS ONE