We read with interest the meta-analysis by Strode et al [1] regarding the efficacy of insecticide treated bed-nets on entomological outcomes. We were however concerned by the methodological issues in this study. First, there was a unit of analysis problem in that control groups (untreated nets; UTN) were used repeatedly to compare against different arms of insecticide treated nets (ITN) as many of the studies were multi-arm. This of course is not statistically valid and in itself can lead to spurious conclusions. Another major problem was the use of the random effects model for pooling when heterogeneity was severe. In the latter situation, this statistical model defaults to the arithmetic mean (see equal weights for this model in Figure 1) but with a coverage (of the confidence interval) that is well below the nominal level (95%) [2]. Thus if we look at the low resistance data regarding mosquito mortality and trim duplicate control groups (retaining the largest ITN groups), there is a discrepancy between the three statistical models we use: the random effects [3] (RE), the inverse variance heterogeneity [4] (IVhet), and the quality effects [5-8] (QE) models. All three are implemented in MetaXL (www.epigear.com). The RE model result gives equal weight to all studies, and is least conservative in both its estimate and confidence interval (0.54 [0.27 to 0.81]). The IVhet model is most conservative and its pooled estimate, like the inverse variance proper model, favors larger studies that have less random error but with the coverage of the confidence interval remaining correct and this can be seen to overlap the null value (0.27 [-0.15 to 0.70]). Finally, the quality effects model, penalizes the largest studies for poor quality (assessments done by Strode et al [1] in supplementary material used) and moves the IVhet estimate a little in favor of smaller studies (0.42 [0.07 to 0.77]) but not as much as the RE model does. The difference between the RE model and QE model is that the RE model always transfers weights one-way [9]: from larger to smaller studies. The QE model does this based on quality assessment and indeed only defaults to the RE model when quality assessments are non-informative (random). The RE model is thus a special case of the QE model when quality is totally non-informative [5]. Our conclusion is that this study delivers little credible information in its current form and requires re-analysis as follows:
a) The unit of analysis problem must be addressed
b) Quality must be meticulously addressed and the QE model used
c) If the authors cannot re-do the quality assessment then the IVhet model should be run and results presented as these will have correct coverage
d) There is no role for the random effects model in such analyses and it should be avoided

Suhail A. R. Doi
Jan J Barendregt

References

1. Strode C, Donegan S, Garner P, Enayati AA, Hemingway J. The Impact of Pyrethroid Resistance on the Efficacy of Insecticide-Treated Bed Nets against African Anopheline Mosquitoes: Systematic Review and Meta-Analysis. PLoS Med. 2014 Mar 18;11(3):e1001619.
2. Brockwell SE, Gordon IR. A comparison of statistical methods for meta-analysis. Stat Med 2001; 20(6):825-40.
3. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials 1986; 7(3):177-88.
4. Barendregt JJ, Doi, SA. MetaXL users guide. [Web Page]. 29 April 2011; Available at http://www.epigear.com. (Accessed 7 April 2014).
5. Doi SA. Evidence Synthesis for Medical Decision Making and the Appropriate Use of Quality Scores. Clin Med Res 2014 [early online].
6. Doi SA, Barendregt JJ, Onitilo AA. Methods for the bias adjustment of meta-analyses of published observational studies. J Eval Clin Pract. 2013 Aug;19(4):653-7.
7. Doi SA, Barendregt JJ, Mozurkewich EL. Meta-analysis of heterogeneous clinical trials: an empirical example. Contemp Clin Trials. 2011 Mar;32(2):288-98. Erratum in: Contemp Clin Trials. 2013 Jan;34(1):35. Pu
8. Doi SA, Thalib L. A quality-effects model for meta-analysis. Epidemiology. 2008 Jan;19(1):94-100. Erratum in: Epidemiology. 2010 Mar;21(2):278.
9. Al Khalaf MM, Thalib L, Doi SA. Combining heterogenous studies using the random-effects model is a mistake and leads to inconclusive meta-analyses. J Clin Epidemiol. 2011 Feb;64(2):119-23.