Dear Mr. Schälte,

Thank you very much for submitting your manuscript "Missing data in amortized simulation-based neural posterior estimation" for consideration at PLOS Computational Biology. Your manuscript was reviewed by members of the editorial board and an independent reviewer, who appreciated the attention to an important topic. Based on the reviews, we are likely to accept this manuscript for publication, providing that you modify the manuscript according to the review recommendations.

Please prepare and submit your revised manuscript within 30 days. If you anticipate any delay, please let us know the expected resubmission date by replying to this email.

When you are ready to resubmit, please upload the following:

[1] A letter containing a detailed list of your responses to all review comments, and a description of the changes you have made in the manuscript. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out

[2] Two versions of the revised manuscript: one with either highlights or tracked changes denoting where the text has been changed; the other a clean version (uploaded as the manuscript file).

Important additional instructions are given below your reviewer comments.

Thank you again for your submission to our journal. We hope that our editorial process has been constructive so far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

Sincerely,

James R. Faeder

Academic Editor

PLOS Computational Biology

Pedro Mendes

Section Editor

PLOS Computational Biology

***********************

A link appears below if there are any accompanying review attachments. If you believe any reviews to be missing, please contact ploscompbiol@plos.org immediately:

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: This paper extends neural posterior estimation (NPE), in particular the BayesFlow method, to incorporate missing data. It compares 3 natural methods for doing this and, through carefully designed experiments, shows the advantage of one particular approach: augmenting the observations with binary variables indicating missingness.

My opinion is that the paper is a good contribution, acting as a proof of concept for this methodology using synthetic examples. I'd like to see a comparison with existing methods, such as multiple imputation, on real data. But I'm happy for this to be done as future work, as the authors describe in the conclusion.

Comments and suggested changes are listed below. I think these should all be easy to address.

Major comments

==============

* Algorithm 1 uses an online learning approach. Every training step generates a batch of data, which is then discarded. The authors should justify why they prefer this to an offline approach of generating the training data first and reusing it across multiple training epochs, which is discussed as an option in the Bayesflow paper and seems more efficient to me.

* Line 63: "Our presented approach is agnostic of the underlying missingness mechanism". This is not correct, as you later assume missingness indicators are sampled from π(τ), which is the definition of MCAR. The methodology could be extended to MAR and MNAR by using π(τ|x), but this is not done in the paper. Also application to MNAR would be challenging (as usual in missing data problems) because it would require correctly specifying the distribution of missingness indicators given unobserved data values.

* I liked the observation in Section 3.2 that your approach can improve on BayesFlow even outside the context of missing data as it's usually conceived. That is, it improves convergence for data of variable length. You could consider highlighting this as a secondary contribution of the paper in the abstract/introduction.

* NPE methods can be inaccurate under misspecified models (see for instance https://arxiv.org/abs/2210.06564). Adding an extra model for missingness seems to increase the potential for misspecification, especially in situations where MNAR is possible. I think addressing this issue is beyond the scope of this paper, but it would be good to add some discussion of it.

Minor comments

==============

* Line 43: "Possible reasons exist plenty" - is this a typo?

* Line 76: "imputation... holds no conceptual advantage over approaches discarding missing data". I don't think this is correct in general. For instance see the discussion on the limitations of deletion methods in Chapter 1 of "Flexible Imputation of Missing Data" by van Buuren. I suspect the authors may be referring to the case of likelihood-based inference under ignorable missingness.

* Line 121: I think the equation involving NA implicity assumes that 0*NA=0. It would be worth pointing this out. Often the convention is that 0*NA results in NA. For instance the R language does this.

* Line 128: "Simply dropping them from the data vector is no solution." This sounds like the basis of deletion methods, which are a solution to the problem and sometimes work well. Perhaps the authors mean that this solution wouldn't work **in the context of amortized inference**, as they argue in Section 1 of the supplement.

* Section 3: I couldn't find a description of what type of cINN is used, only the number of layers.

* Line 192: You say the true posterior is the distribution given only the observed data. It's worth noting that the reason this is true is that you assume ignorable missingness.

* Figure 3: The bottom row might be more easily interpretable if you used the same y-axis in all plots.

* Line 247: "This gives a mathematically inexact approximation of the loss function, as its true value is only approximated as an average over iterations, but the network parameters are updated after each iteration." I didn't fully understand this sentence. In particular stochastic gradient methods always use inexact approximations of the loss. I think the key point here is that fixing the number of observations within a batch means the loss estimate (1) remains unbiased but (2) has higher variance (see https://en.wikipedia.org/wiki/Law_of_total_variance).

* Supplement line 63: I think the last π(xobs) on this line should have a subscript "avai".

* Supplement, just after line 67: "the missingness pattern contains information about the parameters beyond the complete data". I'm not a missing data expert, but this seems like an unusual case. It's typically to assume that π(τ|x,ψ) i.e. missingness pattern can depend on the full data and parameters ψ which are separate from the model parameters θ (in Bayesian setting typically ψ,θ are assumed to be a priori independent). Assuming also a direct dependence of τ on θ is mathematically possible, but I think the authors should make a case for why this would be a useful model.

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: None

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Figure Files:

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org.

Data Requirements:

Please note that, as a condition of publication, PLOS' data policy requires that you make available all data used to draw the conclusions outlined in your manuscript. Data must be deposited in an appropriate repository, included within the body of the manuscript, or uploaded as supporting information. This includes all numerical values that were used to generate graphs, histograms etc.. For an example in PLOS Biology see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5.

Reproducibility:

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

References:

Review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript.

If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Dear Mr. Schälte,

We are pleased to inform you that your manuscript 'Missing data in amortized simulation-based neural posterior estimation' has been provisionally accepted for publication in PLOS Computational Biology.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. A member of our team will be in touch with a set of requests.

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us now if you or your institution is planning to press release the article. All press must be co-ordinated with PLOS.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Computational Biology. 

Best regards,

James R. Faeder

Academic Editor

PLOS Computational Biology

Pedro Mendes

Section Editor

PLOS Computational Biology

***********************************************************

Reviewer's Responses to Questions

Comments to the Authors:

Please note here if the review is uploaded as an attachment.

Reviewer #1: I'm happy that the authors have addressed all my comments from the previous round. I also like the presentation of the previous and revised content in the response letter.

One minor point to optionally address before publication is about "discarding missing data". In the original version I thought this referred to deletion methods, but from the response letter I think this is not exactly the meaning that was intended. (I suspect the intended meaning is things like likelihood methods using only the observed data under the assumption of ignorable missingness.) You could emphasise further in the text that there is a distinction between "discarding" and "deletion". But this is an optional suggestion as your new text does already explain this.

**********

Have the authors made all data and (if applicable) computational code underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data and code underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data and code should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data or code —e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No