Link to properly formatted version of this letter to the editor, including Table and Figure (recommended):
https://www.dropbox.com/s...


Dear Editor,

We are writing in response to the article 'Benefits of a Working Memory Training Program for Inattention in Daily Life: A Systematic Review and Meta-Analysis' from Spencer-Smith and Klingberg, which was published in PLOS ONE on March 20, 2015.

The main aim of the Spencer-Smith and Klingberg (S&K) meta-analysis was to evaluate whether Cogmed working memory (WM) training has benefits for inattention in daily life. The meta-analysis calculated the pooled standardized mean difference (SMD) between intervention and control groups. Eleven studies with 12 group comparisons were included in the meta-analysis, revealing a significant training effect on inattention in daily life, SMD=-0.47, 95% CI -0.65, -0.29, p < .00001. Moreover, subgroup analyses showed that this effect was significant for children, adolescents and adults, for patients with ADHD, for participants with WM impairments, for studies using control groups that were active or non-adaptive, for wait-list controlled studies, as well as for studies using specific or general measures. Six of the studies reported follow-up assessments and a meta-analysis suggested a follow-up training effect on inattention in daily life, SMD=-0.33, 95% CI -0.57 -0.09, p = .006. Based on these results the authors concluded that Cogmed WM training has significant and clinically relevant benefits for inattention in daily life.
The conclusions of this paper were immediately picked up by the media, with headlines such as 'Brain Training Helps ADHD Sufferers to Concentrate' (Jones, 2015), and are therefore likely to have a significant impact on clinical practice. This would be of great value if these conclusions were correct. However, we will demonstrate that the found effects are most likely inflated by coding errors and publication bias.

Coding errors
S&K’s figure 2 shows 12 effect sizes indexing intervention vs control group differences in post-treatment ratings of inattention in daily life. The direction of these effect sizes suggests that in all studies, intervention groups were characterized by less inattention problems than control groups. However, this is not true: In the studies of Chacko et al. (2014; presented in article of S&K as Chacko et al., 2013) and Grunewaldt et al. (2013) post-treatment ratings of inattention were reversed (e.g., a rating of 16.51 was reversed to -16.51; the correct ratings can be found in the supplementary materials of Chacko et al. and Grunewaldt et al.; see URLs in reference list below). In the statistical analyses paragraph S&K explain this as follows: 'In some cases scores included in analyses were reversed so that a negative effect always indicated decreased inattention in daily life'. However, given this explanation none of the scores should have been reversed. For instance, Chacko et al. used the parent-rated Disruptive Behavior Disorders Rating Scale (DBD; Pelham et al., 1992) as measure of inattention, 'with higher scores indicating a greater frequency of problems [quote from Chacko et al.]', and Grunewaldt et al. used the ADHD Rating Scale-IV (Du Paul et al., 1988) which is interpreted in the same way. In fact, the ADHD Rating Scale-IV was also used in another included study (see ratings of Egeland et al., 2013). However, for that study, scores were not reversed. In sum, the post-treatment ratings of inattention from Chacko et al. and Grunewaldt et al. are incorrectly coded in S&K’s meta-analyses.
To examine the impact of these coding errors on the results and conclusions of the study by S&K we have reanalyzed the data using the correct post-treatment ratings of inattention from Chacko et al. and Grunewaldt et al. (see results section below).

Publication Bias
As it is clearly undesirable for treatment studies to find null-effects or negative effects, it is important for meta-analyses of treatment studies to examine the risk of publication bias (Sterne et al., 2011; Tang & Liu, 2000). Therefore, S&K present a funnel plot. However, they do not interpret it, as '[i]n the current meta-analysis this approach is problematic because SMD is calculated using sample size and therefore SMD and the standard error are correlated'. We are not aware of any studies that have shown this to be a valid reason to abandon interpretation of a funnel plot.
Nonetheless, funnel plots should indeed be interpreted with caution, as there are multiple ways that funnel plot asymmetry may arise (Egger et al., 1997, also reproduced in the Cochrane Handbook). Also, visual inspection will always remain a subjective method for detecting funnel plot asymmetry, which is why tests for funnel plot asymmetry and sensitivity analyses have been developed. Therefore, we examined possible publication bias and the effect thereof by conducting a regression test for funnel plot asymmetry (Egger et al., 1997; using the correct mean post-treatment ratings of inattention from Chacko et al., 2014 and Grunewaldt et al., 2013) and applying the trim-and-fill method (Duval and Tweedie, 2000). This last method estimates the number of unpublished studies and provides an adjustment of the overall effect size accordingly.

Revised Results
To allow easy comparison of the original and revised results findings are presented in the same order as in the study by S&K.

Inattention in daily life after the training
The original effect size as reported by S&K was highly significant (even at the .001 level) and of medium size. After we corrected the scores from Chacko et al., 2014 and Grunewaldt et al., 2013, the effect was still significant at the .05 level, but it was smaller (small to medium; see revised forest plot in Figure 1 of Supplementary Materials). We performed a regression test (Egger et al., 1997) to test for funnel plot asymmetry (using the 11 studies with the 12 group comparison effect sizes; but with the correct scores from Chacko et al. and Grunewaldt et al.). We found marginally significant funnel plot asymmetry (at the .05 level, p = .053). This result is ambiguous, and warrants further sensitivity analyses. We therefore applied the trim-and-fill method. The original results and the revised results with and without applying the trim-and-fill method are presented below:

SMD (95% CI) p
Original of Spencer-Smith and Klingberg -0.47 (-0.65, -0.29) <0.0001
With correct coding of Chacko and Grunewaldt -0.37 (-0.63, -0.12) 0.0039
With correct coding, and trim-and-fill -0.20 (-0.46, 0.06) 0.1304

With correct coding and after applying the trim-and-fill method, the effect of Cogmed WM training on inattention in daily life was no longer significant. This indicates that this effect is not as robust as the original publication suggested, and is possibly affected by the coding errors and publication bias.

- Link to Supplementary Materials including Figure 1: https://www.dropbox.com/s...

Examination of methodological and participant characteristics (revised subgroup analyses)
We now move to subgroup analyses, for which fewer than 10 studies were available. Funnel plot asymmetry cannot be reliably established when there are fewer than 10 studies in a meta-analysis (Sterne et al., 2011). Therefore, the following subgroup analyses are interpreted without further examination of sensitivity to publication biases.
The original meta-analysis indicated significant training effects on inattention in daily life for all the examined subgroups: for studies using an active or non-adaptive control group or a wait-list control group, for studies using a specific or a general measure of inattention in daily life, for studies with children and adolescents or adults, and for studies with ADHD or WM impaired groups; all ps ≤ .005.
With the correct scores from Chacko et al. (2014) and Grunewaldt et al. (2013), the effects were no longer significant for the following subgroups: for studies using an active or non-adaptive control group, for studies using a specific measure of inattention in daily life, for children and adolescents, for patients with ADHD, and for participants with WM impairments; respective ps of .122, .325, .104, .138, .078. The results for the other subgroups (studies using wait-list control groups, studies using a general measure of inattention, studies with adults) remained significant at the .05 level (in addition see Table 1 in Supplementary Materials).

- Link to Supplementary Materials including Table 1: https://www.dropbox.com/s...

Inattention in daily life following a delay after training
Six of the studies (with 7 group comparisons) reported follow-up assessment (ranging from 2 to 4 months). A meta-analysis indicated a significant training effect on inattention at the follow-up assessment, SMD=-0.33, 95% CI -0.57 -0.09, p = .006. S&K conclude that '[t]his analysis provides initial evidence for persisting benefits of the WM training program for inattention in daily life'. We did not reanalyze these data because the analysis did not include the studies of Chacko et al. (2014) and Grunewaldt et al. (2013) and because less than 10 studies were included (i.e., funnel plot asymmetry could not be reliably established). Nonetheless, we think it is important to mention some of the limitations of this conclusion:
First, the difference between the Cogmed groups and control groups was analyzed using the mean ratings on the follow-up measures of inattention. The pre-test ratings were not taken into account, which makes it impossible to interpret which group benefits most, or if there is any benefit at all. Moreover, this is also a limitation of all previous analyses, as those analyses only included mean post-treatment ratings.
Second, half of the studies included in this analysis did not use active or non-adaptive control groups (only wait-list control or passive control), which implies that the outcome of the meta-analyses might be confounded by expectancy/placebo effects.
Finally, we fully agree with S&K’s comment that 'a limitation of this study is the small number of studies included in the analysis to examine persisting and long-term benefits of training…...The next critical step will be to adequately establish long-term benefits for inattention in daily life, which will become evident with the publication of the increasing number of trials conducting long-term follow-up.' We also agree with their final conclusion that 'It will be important for future trials to evaluate long-term benefits in order to reliably determine the persisting benefits of a WM training program.' Thus, more active or non-adaptive controlled studies are needed before any (clinically) relevant conclusions can be drawn regarding the long-term effects of Cogmed WM training.

Conclusion
After correcting for coding errors and possible publication bias the meta-analysis did no longer indicate a significant effect of Cogmed WM training on inattention in daily life. Re-examination of subgroup analyses showed that, after correcting coding errors, training effects were no longer significant for studies using an active or non-adaptive control group, for studies using a specific measure of inattention in daily life, for children and adolescents, for patients with ADHD, and for participants with WM impairments.


With kind regards,

Sebastiaan Dovis, Joost Agelink van Rentergem and Hilde M. Huizenga


Affiliations
S. Dovis: Department of Developmental Psychology, University of Amsterdam, Amsterdam, The Netherlands; Amsterdam Brain and Cognition Center Amsterdam, Amsterdam, The Netherlands; Addiction, Development, and Psychopathology (Adapt) lab, Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands; Research priority area Yield, University of Amsterdam, Amsterdam, The Netherlands.
J. Agelink van Rentergem and H.M. Huizenga: Department of Developmental Psychology, University of Amsterdam, Amsterdam, The Netherlands; Amsterdam Brain and Cognition Center Amsterdam, Amsterdam, The Netherlands; Research priority area Yield, University of Amsterdam, Amsterdam, The Netherlands.

Correspondence to Sebastiaan Dovis, Developmental Psychology, University of Amsterdam. Weesperplein 4, 1018 XA Amsterdam, The Netherlands; Tel: +3120-5256298; Email: S.Dovis@uva.nl


References
Chacko, A., Bedard, A.-C., Marks, D. J., Feirsen, N., Uderman, J. Z., Chimiklis, A., … Ramon, M. (2014). A randomized clinical trial of Cogmed Working Memory Training in school-age children with ADHD: a replication in a diverse sample using a control condition. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 55(3), 247–255. doi:10.1111/jcpp.12146

---------> Supplementary material of Chacko et al. (see Table S1): http://onlinelibrary.wile...

DuPaul, G. J., Power, T. J., Anastopoulos, A. D., & Reid, R. (1998). ADHD Rating Scale-IV: Checklists, norms, and clinical interpretation. New York: Guilford Press.

Duval, S., & Tweedie, R. (2000). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56(2), 455–463. doi:10.1111/j.0006-341x.2000.00455.x

Egeland, J., Aarlien, A. K., & Saunes, B.-K. (2013). Few effects of far transfer of working memory training in ADHD: a randomized controlled trial. PloS One, 8(10), e75660. doi:10.1371/journal.pone.0075660

Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ, 315(7109), 629–634. doi:10.1136/bmj.315.7109.629

Grunewaldt, K. H., Løhaugen, G. C. C., Austeng, D., Brubakk, A. M., & Skranes, J. (2013). Working memory training improves cognitive function in VLBW preschoolers. Pediatrics, 131(3), e747–54. doi:10.1542/peds.2012-1965

----------> Supplementary material of Grunewaldt et al. (see supplemental Table 9): http://pediatrics.aappubl...

Jones, O. (2015, April 4), Brain Training Helps ADHD Sufferers to Concentrate. Big Think, Retrieved from http://bigthink.com/ideaf....

Pelham, W. E., Gnagy, E. M., Greenslade, K. E., & Milich, R. (1992). Teacher rating of DSM-III-R symptoms for disruptive behavior disorders. Journal of Clinical Child Psychology, 8, 259–262.

Song, F., Khan, K. S., Dinnes, J., & Sutton, A. J. (2002). Asymmetric funnel plots and publication bias in meta-analyses of diagnostic accuracy. International Journal of Epidemiology, 31(1), 88-95.

Spencer-Smith, M., & Klingberg, T. (2015). Benefits of a working memory training program for inattention in daily life: a systematic review and meta-analysis. PloS One, 10(3), e0119522. doi:10.1371/journal.pone.0119522

Sterne, J. a C., Sutton, A. J., Ioannidis, J. P. a, Terrin, N., Jones, D. R., Lau, J., … Higgins, J. P. T. (2011). Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ (Clinical Research Ed.), 343(fig 1), d4002. doi:10.1136/bmj.d4002

Tang, J. L., & Liu, J. L. (2000). Misleading funnel plot for detection of bias in meta-analysis. Journal of Clinical Epidemiology, 53(5), 477-484.