Reader Comments

Post a new comment on this article

Optimal thresholds being used are not reflected in the p-values given.

Posted by Rork_Kuick on 23 Jul 2015 at 13:33 GMT

"To assess the prognostic value of a gene, each percentile (of expression) between the lower and upper quartiles were computed and the best performing threshold was used as the final cutoff in a univariate Cox regression analysis."
I believe the p-values given are computed as if this optimization had not been performed, so I think they are overly optimistic. It's not much discussed in the paper. I am saying I think the software is performing p-hacking for the user. I believe this option is also available for the log-rank tests without the proper p-value for the procedure being computed. If people want to first pick an optimal cut-point by looking at the data, and use the optimal cut-point in a final statistical test, then they should penalize themselves for having optimized. It's possible that the software does account for this optimization, in which case I apologize, but my guess is that if it did, that would be discussed in the paper since it would be non-trivial.

No competing interests declared.

RE: Optimal thresholds being used are not reflected in the p-values given.

zsalab2 replied to Rork_Kuick on 24 Jul 2015 at 07:18 GMT

The described methodology for the optimization of a cutoff value is an optional feature of the software. Furthermore, the online interface uses the median as a cutoff as a default analysis option.

However, there is no biological reason to use the median as a cutoff in survival analysis. The optimization performs a stepwise analysis to evaluate the possible cutoffs. This is not a classical multiple testing problem, because the analyses are not independent (only 1% of the cases differ between two runs). The overall rate of false positives depends not only on the utilization of the cutoff but also on the application of diverse genes with different filtering. We have added a separate multiple testing calculator to enable the user to pick a final significance threshold according to the used project.

No competing interests declared.

RE: RE: Optimal thresholds being used are not reflected in the p-values given.

Rork_Kuick replied to zsalab2 on 05 Aug 2015 at 19:40 GMT

I can't reproduce figure 2A exactly, but get 487 patients (rather than 486) and get p=3.9e-08, HR=2.46, with "optimized" method which chose 200 low, 287 high (fig 2A cuts into 200 low, 286 high, and gets p=4.8e-08, HR=2.44), so it appears the authors used the optional feature in the paper, but did not account for the optimization in the p-value given. I am not alleging misconduct, just error.

No competing interests declared.

RE: RE: RE: Optimal thresholds being used are not reflected in the p-values given.

zsalab2 replied to Rork_Kuick on 05 Aug 2015 at 22:40 GMT

There is no error in computing the p value, because the computation for Figure 2 was performed as described in the manuscript Methods section which already includes utilization of the optimized cutoff. We believe this method has the highest biological relevance. However, it is still not widely used - for this reason, this feature is not set to “on” by default in the online analysis interface.

We have updated the database in 2015 which not only included new patients, but samples already in the database were also updated (in case updated data was available in GEO or in another publication for the same dataset). The "earlier release" filter only filters the samples, but still uses the latest clinical data. Therefore a minimal change (for example: n=467 vs. n=486, HR=2.46 vs. HR=2.44) can also be observed in case someone uses the "earlier release" option of the online software.

I would like to thank for the comment and have added and update the online interface with an additional information popup noticing the user of this issue (this will come live in the next update of the system).

No competing interests declared.

RE: RE: RE: RE: Optimal thresholds being used are not reflected in the p-values given.

Rork_Kuick replied to zsalab2 on 19 Aug 2015 at 17:53 GMT

It seems you are claiming that if I compute a p-value in a way I clearly describe, it must be correct since I did what I said.
There's a short letter by DG Altman about this issue from 1994. PMID: 9716046
There's a longer article by him and colleagues in 1994 JNCI offering formulas and charts about what the correctly computed p-value actually is when using "minimum p-value methods", which should be easy to implement. PMID:8182763. This paper talks of some of the additional dangers in using minimum p-value methods, one of which is that the effect size (the hazard ratio) is considerably overestimated, and we really don't know by how much (no "correction" is available).

No competing interests declared.