Reader Comments

Post a new comment on this article

Correct answers?

Posted by joshamverbeek on 31 Jan 2013 at 07:15 GMT

The authors are to be applauded to for their attempt to increase the understanding of the results of systematic reviews and meta-analyses. The correct answers to their questions illustrate that there is still a lot of ground to be gained because I believe that none of the correct answers is correct. Let’s have a look at them. The correct answer to question one should be that a relative risk smaller than 1 indicates that the intervention leads to more harm than the control condition. This is not the convention for Cochrane Reviews in which a relative risk smaller than 1 for and adverse effect indicates that the intervention is more effective. However, there is no way to know this if you have not seen how the relative risk is defined: is it intervention incidence divided by control or the other way around? Not all authors stick to the same rule. For question 2 the correct answer would be that the new and old drugs are equivalent in their adverse effects. Given the wide confidence interval this remains to be seen. The result can be due to low power of the meta-analysis or the included studies. Thus d can be the only correct answer here. The correct answer for question 3 suffers from the same problem as question one. If you don’t know if the control mean is subtracted from the intervention mean or the other way around, there is no way of telling if an MD indicates a longer or a shorter period. Not all authors stick to the same rules, so d is the correct answer here. The correct answer for question 4 is unfortunately not an answer to the question at hand. Since it is stated that there is no statistical heterogeneity, the fixed and random effects model would yield the same results. The correct answers are thus a and b.
I believe that this exercise illustrates that we need stricter guidelines and more transparency in reporting systematic reviews and meta-analyses since interpretation of the statistics is not unequivocal.

Jos Verbeek

No competing interests declared.

RE: Correct answers?

MNMavros replied to joshamverbeek on 14 Mar 2013 at 16:28 GMT

Dr Verbeek,
Thank you for your interest in our article. We are glad you agree with our conclusion that the current degree of understanding of statistical terms is suboptimal.
We would also like to thank you for taking the time to raise those issues about the "correct" answers. We understand your reasoning and we do not disagree for the most part. We did discuss those issues extensively during our internal lab meetings, and we struggled to find a "correct", yet concise way to pose the questions and the answers. The "concise" part is vital to all questionnaires, especially those not involving remuneration or other motives to the respondents - such as the one presented in this study. Otherwise, the number of participants would be significantly lower. Although we acknowledge the limitations presented by Dr Verbeek, we believe it would be safe to assume that the researchers-participants used the most "conventional" way of thinking while responding to our survey; in fact, among the 12 members of our group, there was absolute agreement regarding the "correct" answers (also taking into the conciseness of the survey). In this regard, we feel that our results remain accurate and precise.

With kindest regards,
Michael N. Mavros, MD

No competing interests declared.