Dear Dr. Kasparis,

Thank you very much for submitting your manuscript "Improving Timeliness in the Neglected Tropical Diseases Preventive Chemotherapy Donation Supply Chain through Information Sharing: A Retrospective Empirical Analysis" for consideration at PLOS Neglected Tropical Diseases. As with all papers reviewed by the journal, your manuscript was reviewed by members of the editorial board and by several independent reviewers. In light of the reviews (below this email), we would like to invite the resubmission of a significantly-revised version that takes into account the reviewers' comments.

We cannot make any decision about publication until we have seen the revised manuscript and your response to the reviewers' comments. Your revised manuscript is also likely to be sent to reviewers for further evaluation.

When you are ready to resubmit, please upload the following:

[1] A letter containing a detailed list of your responses to the review comments and a description of the changes you have made in the manuscript. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

[2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

Important additional instructions are given below your reviewer comments.

Please prepare and submit your revised manuscript within 60 days. If you anticipate any delay, please let us know the expected resubmission date by replying to this email. Please note that revised manuscripts received after the 60-day due date may require evaluation and peer review similar to newly submitted manuscripts.

Thank you again for your submission. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

Gregory Deye

Associate Editor

PLOS Neglected Tropical Diseases

Bruce Lee

Deputy Editor

PLOS Neglected Tropical Diseases

***********************

Reviewer's Responses to Questions

Key Review Criteria Required for Acceptance?

As you describe the new analyses required for acceptance, please consider the following:

Methods

-Are the objectives of the study clearly articulated with a clear testable hypothesis stated?

-Is the study design appropriate to address the stated objectives?

-Is the population clearly described and appropriate for the hypothesis being tested?

-Is the sample size sufficient to ensure adequate power to address the hypothesis being tested?

-Were correct statistical analysis used to support conclusions?

-Are there concerns about ethical or regulatory requirements being met?

Reviewer #1: see below

Reviewer #2: The study objectives are clear and there is a clear set of testable hypotheses.

However, the study design does not take into account several factors:

-Were there any time-varying factors besides the NTDDeliver and NTDPublic which may be coincident with the study time frame. Countries, manufacturers, and WHO are constantly trying new improvements. Not clear how the study design robustly controls for that?

-Presumably there is a higher likelihood of missing data in the pre-intervention sample. So without explicitly showing that missing data would be completely random (and covariance matrix etc.) it is unclear how the study design removes biases that may result from non random missing data.

-An important control variable which is missing is country characteristics e.g. country logistics factors, etc. What if there was a differences in the countries receiving the orders in the pre-and post intervention phase. The authors have controlled for WHO region, but that is too coarse a variable to account for country specific differences

-Did the authors explore the use of a RD type study design?

Reviewer #3: No issues

--------------------

Results

-Does the analysis presented match the analysis plan?

-Are the results clearly and completely presented?

-Are the figures (Tables, Images) of sufficient quality for clarity?

Reviewer #1: see below

Reviewer #2: The analysis matches what the study plan was.

The result are difficult to interpret without a clear exposition of what each of the time variables means ie. when does the clock start for each timeliness variables and then does the clock stop. The authors should consider creating a diagram with the timelines and variable definition on that.

Reviewer #3: No issues

--------------------

Conclusions

-Are the conclusions supported by the data presented?

-Are the limitations of analysis clearly described?

-Do the authors discuss how these data can be helpful to advance our understanding of the topic under study?

-Is public health relevance addressed?

Reviewer #1: see below

Reviewer #2: the conclusions are supported by the data with the caveats in the study design outlined earlier.

The work has significant relevance for public health but needs to be more robust in its analysis

Reviewer #3: -Based on the findings can the authors further clarify which stakeholder behavior changed based on the variable. E.g., Did PO timeliness reflect WHO management? Did shipping times reflect the pharmaceutical companies. Were these results possibly influenced by other stakeholders indirectly such as NGOs supporting governments efforts?

--------------------

Editorial and Data Presentation Modifications?

Use this section for editorial suggestions as well as relatively minor modifications of existing data that would enhance clarity. If the only modifications needed are minor and/or editorial, you may wish to recommend “Minor Revision” or “Accept”.

Reviewer #1: see below

Reviewer #2: (No Response)

Reviewer #3: CLARIFICATION IN THE INTRODUCTION

• “While significant progress was made towards these 2020 targets, the WHO has recently released a new NTD roadmap with 2030 targets, which pharmaceutical manufacturers have committed to continuing to support [2].” Pharmaceutical companies support the road map but not all donation programs have committed to donating pharmaceuticals through 2030.

• “Delivery delays result in waste, increased program costs, and delayed or even completely missed MDAs, leaving individuals susceptible to NTD infections [7].” More accurate to state “…..leaving communities susceptible to disease recrudescence.”

• “Figure 1. Components of the NTD supply chain from application for donation to in-country delivery to central medical stores”.

- Application: performance measure is 8 months prior to MDA (not 6)

- Manufacturing: performance measures are variable by manufacturer

- Shipment: three months prior to MDA is the UNICEF standard of 10 years ago for sea freight. This is not WHO policy.

- In-country processes: Performance measures should say varies by country

Clarify the NTD Supply Chain Forum membership. If stating the present Forum makeup, some members are missing. If wishing to indicate the makeup at the time of establishment, please make this clear.

--------------------

Summary and General Comments

Use this section to provide overall comments, discuss strengths/weaknesses of the study, novelty, significance, general execution and scholarship. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. If requesting major revision, please articulate the new experiments that are needed.

Reviewer #1: Overall, this is an interesting manuscript that attempts to utilize a pre-post design to evaluate the potential impact of the NTDeliver platform on improving supply chain management for PC diseases. The findings are of interest to the NTD funder, policy and program community and fit well within the scope of this journal. There are substantial issues with the way in which the paper is presented and written that need to be addressed. In addition, the authors need to tone down the language implying causality, as this study design can only assess associations, not causality. If these changes are made, I believe the paper will be a useful contribution to the literature.

A critical limitation of this analysis is that it only evaluates the impact on the “first mile”. It is well known that many of the delays in getting MDA drugs to communities happen between the central store and the program. As such, it is difficult to know how much of an impact this platform actually has on drug delivery. In addition, it does not seem logical to evaluate timeliness as a continuous measure – it probably makes no difference if the drugs are in the central warehouse 6 months or 5 months ahead of MDA – what matters is whether they are there and can be distributed in time for MDA. I would think a binary cutoff based on what this sufficient time is would be more practical. Finally, some of the “improvements” may be statistically significant but that does not make them programmatically relevant. I am not sure a two week improvement is meaningful for example.

The paper is written from the standpoint of a data analyst and much of the language is difficult to follow and not clear to the intended audience. The way in which the analysis is presented is not intuitive and does not align with metrics that the audience will want to understand.

The authors seem to erroneously attribute magnitude of significance with the p value in their analysis. First, it is not at all clear why a non-standard p value of .1 was chosen to represent significance, particularly given that the study design is already weak. This seems likely to have been a post-hoc decision made following a look at the results, particularly given that many analyses are right at the 0.9 level. Please revise to use a standard p value of 0.5 for significance. In addition, something is not “more significant” if its p value is smaller. The key metric is the effect size, and then whether or not it meets the threshold for significance. Please revise the text accordingly.

The results and the discussion need to be separated. There is far too much discussion included in the results section. Please just present the results cleanly and then interpret them in the discussion.

The weaknesses of this study design are a major limitation (as noted by the authors). There is no reason why an RCT or a stepped wedge study could not have been designed in coordination with WHO. One key takeaway in the conclusion is the need to plan robust evaluations from the start so that more meaningful impact assessment can be conducted.

I was confused by the language around confounders and controls – these are important cofactors or mediators of effect – not confounders.

The data could be presented graphically or at least in some much more readily digestible format. The raw output of the regression analyses are not the best way to present these data to the intended audience.

Small issue – data are pleural – datum is singular – update text accordingly.

Reviewer #2: This paper addresses and important and very interesting area for policy relevance for NTDs.

The authors utilize an interesting data set and define performance measures with direct operational relevance to NTD programs and NTD supply chains

However, there are multiple deficiencies in the study design which need to be addressed. It is important for this study to be robust in its methods for the findings to be taken seriously for further investment in the interventions analyzed.

Reviewer #3: (No Response)

--------------------

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

Figure Files:

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

Data Requirements:

Please note that, as a condition of publication, PLOS' data policy requires that you make available all data used to draw the conclusions outlined in your manuscript. Data must be deposited in an appropriate repository, included within the body of the manuscript, or uploaded as supporting information. This includes all numerical values that were used to generate graphs, histograms etc.. For an example see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5.

Reproducibility:

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

Dear Dr. Kasparis,

We are pleased to inform you that your manuscript 'Improving Timeliness in the Neglected Tropical Diseases Preventive Chemotherapy Donation Supply Chain through Information Sharing: A Retrospective Empirical Analysis' has been provisionally accepted for publication in PLOS Neglected Tropical Diseases.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests.

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us now if you or your institution is planning to press release the article. All press must be co-ordinated with PLOS.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Neglected Tropical Diseases.

Best regards,

Gregory Deye

Associate Editor

PLOS Neglected Tropical Diseases

Bruce Lee

Deputy Editor

PLOS Neglected Tropical Diseases

***********************************************************