Reader Comments

Post a new comment on this article

Power, Reliability and Heterogeneous Results

Posted by plosmedicine on 30 Mar 2009 at 23:46 GMT

Author: Ian Shrier
Position: physician, researcher
Institution: McGill University
Submitted Date: September 14, 2005
Published Date: September 15, 2005
This comment was originally posted as a “Reader Response” on the publication date indicated above. All Reader Responses are now available as comments.

I want to congratulate Dr. Ioannidis on his thought provoking article. I have two comments.

In Corollary 1, he suggests that small sample sizes mean smaller power and implies that larger studies with thousands of subjects are more likely to be true. I think it is important to stress that if the effect size is large (e.g. very small variance that is seen in physiological studies), then adequate power is obtained with small numbers. Further, some would argue that exposing subjects to research risks unnecessarily (e.g. when fewer subjects would yield sufficient power) is unethical. Since the analysis is based on power, we should remember that larger is not always better.

In corollary 4, Dr. Ioannidis argues that greater flexibility in designs, definitions, etc means the results are less likely to be true. I agree that replication of all aspects of the study is more likely to yield consistent results, but this does not necessarily mean true results. Since we don't know a priori which methodological details are most appropriate (e.g. dose, timing, etc), heterogenous results from different designs is an important source of information and can lead to a new and more in-depth understanding of the subject - and sometimes even paradigm shifts. I agree with the accompanying editorial to the article that we need to distinguish between the validity of the data and the validity of the authors' conclusions.

Competing interests declared: None