Author: Gang Zheng
Position: Mathematical Statistician
Institution: Office of Biostatistical Research, National Heart, Lung and Blood Institute
E-mail: zhengg@nhlbi.nih.gov
Submitted Date: October 03, 2006
Published Date: October 9, 2006
This comment was originally posted as a “Reader Response” on the publication date indicated above. All Reader Responses are now available as comments.

Ioannidis (2005) examined several important factors that lead to false research findings using the post prediction value (PPV), the probability of a true relationship (TR) given a significant research finding (SRF). One of the important factors he examined is that n research teams in the scientific field independently tested the same research hypothesis and at least one of the teams reported SFR. Ioannidis demonstrated that the PPV, defined as the probability of a TR given that at least one team obtains a SRF, decreases when n increases.

However, when more research teams test the same hypothesis independently with the same statistical power (1-beta) and Type I error (alpha), it is also likely that more research teams would report the SRF. Here I examine the PPV, defined as the probability of a TR given that k research teams report SRF (1< k < n). Following the notation of Ioannidis, the probability of a SRF is P(SRF) = [R/(R+1)](1-beta) + [1/(R+1)]alpha, where R/(R+1) and 1/(R+1) are the prior probabilities of a TR and a null relationship, respectively. Assume that all n research teams conduct the same research independently under the same conditions. Then, k/n is approximately equal to P(SRF) when n is large. This implies that k increases with n and is greater than 1 when P(SRF) is not too small (or when R is not too small). I derived a new table, like Tables 1-3 of Ioannidis, from which a new PPV is obtained, given by PPV=[R(1-beta)^k beta^(n-k)]/[R(1-beta)^kbeta^(n-k)+alpha^k(1-alpha)^(n-k)], where "^" is used for power. Figures of the above PPV can be plotted against R. I found that, when Type I error alpha = 0.05, power 1-beta = 0.80, and R > 0.7, the PPV increases when n increases.

One application of the above analysis is that we should encourage more research teams to conduct independent replication study to confirm the results of a previous study as the pre-study odds, R, is usually greater than 1. This would be particularly useful for replication studies of a genome-wide association study with 100K-500K SNPs.

Reference
Ioannidis JPA (2005) Why most published research findings are false. PLoS Med 2(8): e124.