Author: Robert Waldmann
Position: professor
Institution: Universita di Roma "Tor Vergata"
E-mail: robert.waldmann@gmail.com
Submitted Date: March 20, 2008
Published Date: March 25, 2008
This comment was originally posted as a “Reader Response” on the publication date indicated above. All Reader Responses are now available as comments.

Kirsch et al estimate the average benefit of new-generation antidepressants (NDAs) using a simple standard meta-analysis procedure “We conducted two types of data analysis …and another using each study's drug and placebo groups' arithmetic mean (weighted for the inverse of the variance) as the meta-analytic “effect size” They first calculated an estimate of the overall average change of the Hamilton Rating Scale of Depression (HRSD) for patients who received a NDA as the precision weighted average of the changes reported for the treated subsample in each trial. Then they calculated the average change for patients who received the placebo in the same way and calculated the difference. This approach involves two methodological choices. First the use of precision weights (weights inverse to the reported variance of the estimate of the mean change in each study). Second the decision to calculate overall mean changes for treated and control groups and then take the difference rather than first calculating the difference in the HRSD then calculating a weighted average across trials of those differences. In each case, Kirsch chose a method which, under strong assumptions, gives an efficient and unbiased estimate of the true overall average benefit. In each case there are alternative approaches which are less efficient under those assumptions but which are unbiased not only when the Kirsch et al estimates are unbiased, but also for many cases in which the Kirsch et al estimates are biased. That is they are less efficient under the null but more robust. In each case the null hypothesis that the Kirsch et al estimator is unbiased has been tested and overwhelmingly rejected. The available unbiased estimate of the overall average benefit of NDA’s is equal to 2.65 HRSD units, which is considerably higher than Kirsch et al’s biased estimate.

The proof of these claims would put me over the character limit. It is available here
http://rjwaldmann.blogspo...