Reader Comments

Post a new comment on this article

Reviewer 1: Christian P. Robert

Posted by PLOS_CompBiol on 11 Jan 2013 at 12:33 GMT

[This is a review of the original version. See Text S1 for the version history. The authors’ responses are included in line and are reflected in the published version.]

A few comments on the specific entry on ABC written by Mikael Sunnåker et al....

The entry starts with the representation of the posterior probability of an hypothesis, rather than with the posterior density of a model parameter, which makes it seems likely it could lead the novice reader astray. After all, (a) ABC was not introduced for conducting model choice and (b) interchanging hypothesis and model means that the probability of an hypothesis H as used in the entry is actually the evidence in favour of the corresponding model.

Response: We now first talk only about parameter estimation. We have also rewritten the section about model selection for better coherence of the text.

(There are a few typos and grammar mistakes, but I assume either PLoS or later contributors will correct those.)

Response: We have corrected the typos and grammatical mistakes found during the revision.

When the authors state that the "outcome of the ABC rejection algorithm is a set of parameter estimates distributed according to the desired posterior distribution", I think they are leading some of the readers astray as they forget the "approximative" aspect of this distribution.

Response: This has been changed.

Further below, I would have used the title "Insufficient summary statistics" rather than "Sufficient summary statistics", as it spells out more clearly the fundamental issue with the potential difficulty in using ABC.

Response: The title has been changed to “Summary statistics” (see also Dennis Prangle's comment below)

(And I am not sure the subsequent paragraph on "Choice and sufficiency of summary statistics" should bother with the sufficiency aspects... It seems to me much more relevant to assess the impact on predictive performances.

Response: We have toned down the issue of sufficiency. For clarity reason, we prefer to defer the discussion on predictive performance to the "pitfall and remedies" section.

Although this is most minor, I would not have made mention of the (rather artificial) "table for interpretation of the strength in values of the Bayes factor (...) originally published by Harold Jeffreys". I obviously appreciate very much that the authors advertise our warning [1] about the potential lack of validity of an ABC based Bayes factor!

Response: The section on model selection has been rewritten. In the process, the reference to Jeffreys's table has been removed.

I also like the notion of "quality control", even though it should only appear once.

Response: We have merged the two sections about quality control.

And the pseudo-example is quite fine as an introduction, while it could be supplemented with the outcome resulting from a large n, to be compared with the true posterior distribution.

Response: We have included a new figure (Fig. 3), which shows ABC with large n for full data, and summary statistics (ε = 0 and ε = 2). As suggested, it also compares the ABC results with the theoretical posterior.

The section "Pitfalls and remedies" is remarkable in that it details the necessary steps for validating a ABC implementation: the only entry I would remove is the one about "Prior distribution and parameter ranges", in that this is not a problem inherent to ABC... (Granted, the authors present this as a "general risks in statistical inference exacerbated in ABC", which makes more sense!)

Response: We would like to keep the discussion on prior distribution and parameter ranges. However, a sentence was added under “Pitfalls and remedies” to emphasize that the problem related to “Prior distribution and parameter ranges” is not specific to ABC.

It may be that the section on the non-zero tolerance should emphasize more clearly the fact that ε should not be zero. As discussed in the recent Read Paper by Fearnhead and Prangle [2] when envisioning ABC as a non-parametric method of inference.

Response: This has been changed accordingly.

At last, it is always possible to criticise the coverage of the historical part, since this is such a recent field that it is constantly evolving. But the authors correctly point out to (Don) Rubin on the one hand and to Diggle and Graton on the other. I would suggest adding in this section links to the relevant softwares like our own DIY-ABC[3]...

Response: A section listing ABC software has been added, including a new table with references to the corresponding papers (Table 3) .

No competing interests declared.

RE: Reviewer 1: Christian P. Robert

PLOS_CompBiol replied to PLOS_CompBiol on 11 Jan 2013 at 12:37 GMT

[This is a review of the first revision.]

I have nothing to add to my earlier review, I am completely happy with the current version!

No competing interests declared.