Results of scoping review do not support mild traumatic brain injury being associated with a high incidence of chronic cognitive impairment: Commentary on McInnes et al. 2017

A recently published review of 45 studies concluded that approximately half of individuals who sustain a single mild traumatic brain injury (MTBI) experience long-term cognitive impairment (McInnes et al. Mild Traumatic Brain Injury (mTBI) and chronic cognitive impairment: A scoping review. PLoS ONE 2017;12:e0174847). Stratified by age, they reported that 50% of children and 58% of adults showed some form of cognitive impairment. We contend that the McInnes et al. review used a definition of “cognitive impairment” that was idiosyncratic, not applicable to individual patients or subjects, inconsistent with how cognitive impairment is defined in clinical practice and research, and resulted in a large number of false positive cases of cognitive impairment. For example, if a study reported a statistically significant difference on a single cognitive test, the authors concluded that every subject with a MTBI in that study was cognitively impaired–an approach that cannot be justified statistically or psychometrically. The authors concluded that impairment was present in various cognitive domains, such as attention, memory, and executive functioning, but they did not analyze or report the results from any of these specific cognitive domains. Moreover, their analyses and conclusions regarding many published studies contradicted the interpretations provided by the original authors of those studies. We re-reviewed all 45 studies and extracted the main conclusions from each. We conclude that a single MTBI is not associated with a high incidence of chronic cognitive impairment.

Introduction A recently published scoping review of the literature concluded that approximately half of individuals who sustain a single mild traumatic brain injury (MTBI) experience long-term cognitive impairment [1]. The authors identified 45 studies that met their inclusion criteria. Through their synthesis and analysis, they reported that "1963 participants out of 3593, or approximately 55% of our sample collapsed across all time points showed cognitive impairment" (page 10). Stratified by age, they reported that 50% of children and 58% of adults showed some form of cognitive impairment (page 11). They asserted that "a large proportion of individuals with a single mTBI will continue to demonstrate measurable impairment in various cognitive domains including executive function, learning/memory, attention, processing speed, and language function long after the initial injury" (page 13). They stated that the published literature to date represents a "gross underestimation" (pages 13 and 14) of the extent of cognitive impairment caused by a single MTBI, and "it is possible that our results represent a further underestimation of the incidence of persistent cognitive impairment following a single mTBI" (page 14).
We disagree with the findings and conclusions summarized above from the scoping review published by McInnes and colleagues [1]. Their conclusions are fundamentally different from, or run counter to, findings from numerous meta-analyses of the MTBI neuropsychological literature [2][3][4][5][6][7][8][9][10][11][12][13][14][15]. A scoping review is relatively new and still evolving approach to knowledge synthesis. Standardized methodology and reporting guidelines are not yet available [16,17], but there is consensus that the main purpose of scoping reviews is to examine the extent and nature of available research in a defined subject area [16][17][18]. A scoping review can be helpful for a subject area that has not previously been comprehensively reviewed, often to determine if there is sufficient evidence to conduct a systematic review [18,19]. This seems to be the reason why McInnes et al. selected this method of knowledge synthesis (i.e., they wrote: "the studies that assess long-term cognitive outcomes in singly-concussed individuals have not been gathered and reviewed"; page 2).
There are essential differences between scoping reviews and systematic reviews. Scoping reviews do not have rigid exclusion criteria and do not formally evaluate the quality of evidence [16,19], consistent with their goal of summarizing the breadth of literature. In contrast, systematic reviewers perform both of these tasks in order to reduce bias in trying to answer specific research questions, such as prognosis or treatment efficacy. Whereas systematic reviews often include a meta-analysis of aggregated quantitative data, scoping reviews generally provide only a descriptive narrative [16]. Scoping reviews may also include a "descriptive numerical summary" to map the time, location, and source of available research, typically reported as a frequency count of studies with certain characteristics [17][18][19][20].
McInnes et al. did not exclude studies with a high risk of bias and did not perform quality appraisals of included studies, consistent with scoping review methodology [19]. However, they went well beyond numerically summarizing the number and type studies available. They recoded and synthesized quantitative information, and from these analyses, drew conclusions about the incidence of long-term cognitive impairment following MTBI. This falls outside the purview of a scoping review and exposed McInnes et al. to the risk of flawed conclusions. In their original description of scoping review, Arksey and O'Malley explained that "unlike a systematic review the scoping study does not seek to 'synthesize' evidence or to aggregate findings from different studies. . . because the scoping study does not seek to assess quality of evidence and consequently cannot determine whether particular studies provide robust or generalizable findings" [18]. McInnes et al. used systematic review techniques to synthesize evidence without an assessment of the risk for bias or consideration of how bias might influence their results. A systematic review that omits these elements provides "critically low" confidence in their conclusions and "should not be relied on to provide an accurate and comprehensive summary of the available studies" ([21, page 6]. We have three primary concerns regarding the methodology used to synthesize and summarize data in their scoping review. First, their definition of "cognitive impairment" was idiosyncratic, not applicable to individual patients or subjects, and inconsistent with how cognitive impairment is defined in clinical practice and research. Their definition resulted in a large number of false positive cases of "cognitive impairment." In the McInnes et al. review, participants in the original studies were dichotomized into "cognitively impaired" and "cognitively unimpaired" groups. McInnes et al. [1] defined individuals as having cognitive impairment "if their outcome measure score significantly differed from those of the control groups or the normative data, or if they were below author-identified cut-off scores" (page 7). Of the 45 original studies, only 9 studies (20%) were dichotomized based on author-identified definitions of cognitive impairment [22][23][24][25][26][27][28][29][30]. The remaining studies did not define "cognitive impairment" in their text and were dichotomized by McInnes et al. based on differences on significance testing between the MTBI and control groups. Two original studies that defined cognitive impairment in an a priori manner did not report the incidence of cognitive impairment in their sample, and those studies were classified by McInnes et al. based on group comparisons [31,32]. If a study reported a statistically significant difference on a single cognitive test, the authors concluded that every subject in that study with a MTBI was cognitively impaired. This method represents a misunderstanding or misapplication of statistical significance testing. A statistically significant difference between an MTBI group and a control group means that the difference between the means of groups is not likely to be zero, thus the associated term in statistical testing is null hypothesis testing. A p-value does not provide us information regarding the practical or clinical significance of the difference, the magnitude of the difference, or whether the difference is large enough to classify people into one group or another. A statistically significant difference between groups on a test or tests cannot be used to accurately or reliably classify individual subjects as cognitively impaired. Classifying every subject in the MTBI group as cognitively impaired in these instances also does not make sense from a practical standpoint. There are likely several individuals in the MTBI groups who performed better than the mean of the control groups and/or whose scores would be interpreted as broadly normal (e.g., average or better) based on using traditional neuropsychological interpretation schemes. These methods artificially inflate the percentage of individuals classified as cognitively impaired. Second, the authors concluded that impairment was present in various cognitive domains, such as attention, memory, and executive functioning (page 13 of the Discussion), but they did not analyze or report the results from any of these cognitive domains in their review. Third, their analyses and conclusions regarding many published studies contradicted the interpretations provided by the original authors of the studies (e.g., [33][34][35]).

Review of 45 articles relating to cognitive functioning following a single MTBI
We re-reviewed the 45 articles identified in the McInnes et al. scoping review [1] to examine the sampling strategy and statistical techniques used when determining if participants who experienced an MTBI had cognitive impairment. Further, we thought it would be useful to provide a summary statement for each of these 45 studies based on the data and original authors' conclusions. We did not seek to complete our own scoping review, systematic review, or meta-analysis of these studies. Prior systematic reviews and meta-analyses have examined this topic in great detail [2][3][4][5][6][7][8][9][10][11][12][13][14][15].
From each article, we extracted the percentage of the MTBI sample with a complicated MTBI (i.e., macrostructural trauma-related intracranial abnormalities visible on computed tomography or magnetic resonance imaging), as well as the sample size, age (mean and standard deviation), and recruitment settings for the MTBI and control groups. We also extracted the number of group comparisons for cognitive outcomes (i.e., the number of test scores that were analyzed/compared between the MTBI and control groups) and the number of statistically significant group differences. We determined whether the original authors classified individual subjects as cognitively impaired or not, and whether the original authors drew conclusions about whether or not subjects were cognitively impaired. We examined whether other factors that may influence cognitive functioning were reported in the original studies (i.e., whether the original study assessed for pre-morbid or current intellectual functioning, or mental health problems). This does not necessarily mean that these variables were used in statistical models to control for their potential effect when assessing for cognitive differences between groups. Three authors (JK, AG, and DT) with experience conducting systematic reviews [11,12,36,37] completed extractions for all of the articles. Each study was reviewed by two authors. We provided a brief summary of the statistical findings and implications of each article, using quotations from the original articles whenever possible.

Results
The findings were consolidated into Table 1. Several studies summarized in the McInnes et al. scoping review [1] did not include means, SDs, or effect sizes for the statistical comparisons between groups on cognitive testing (e.g., [22,27,[38][39][40][41][42]). As such, it is not possible to draw conclusions from those studies regarding the magnitude of the difference between the MTBI group and the control group. Moreover, for most of the studies it is not known whether a subgroup within the MTBI group met criteria for cognitive impairment. As noted above, the scoping review by McInnes and colleagues [1] came to fundamentally different conclusions in comparison to numerous published meta-analyses of the MTBI neuropsychological literature [2][3][4][5][6][7][8][9][10][11][12][13][14][15]. McInnes and colleagues [1] identified some more recently published studies, since 2013, that were not included in previous systematic reviews and meta-analyses because they were published after those searches were performed [22,23,34,[43][44][45][46][47][48][49][50]. However, those more recently published studies, as a rule, did not compute the percentages of the MTBI sample that met criteria for cognitive impairment, nor did they yield results suggestive of chronic cognitive impairment (see Table 1).
McInnes et al. [1] identified 6 studies, at the 6 month post-injury time period, that they thought revealed all subjects to have cognitive impairment [25,34,39,41,55,60] (demarcated with an asterisk in Table 1), 5 studies that revealed cognitive impairment in some subjects [26,27,30,46,61] and 1 study that they did not think revealed cognitive impairment in any  Comparing cognitive performances to control participants or normative samples, the researchers categorized a minority of participants as showing signs of cognitive impairment potentially due to MTBI at 3 months post-injury. Comparisons/Differences: NR. Definition of CI: For a brief computerized test battery, "if patients performed 1 SD worse than mean for the controls in at least 2 separate. . .tests, at least 2 occasions, they were coded as having signs of cognitive impairment, possibly due to MTBI." (p. 54). For an extended neuropsychological test battery, "The performance was considered abnormal if 2 or more test results were below 1 SD or if the results of separate tests differed by 2 SD or more." (p. 55). % CI: 8% with CI per computerized testing; 30% with CI per extended neuropsychological test battery  Summary: They concluded that post-acute plasma ghrelin levels were associated with "cognitive deterioration" at 3-months following injury. Comparisons = NR, Differences = NR. Definition of CI: Authors reported "Forty patients (33.9%) had cognitive deterioration three months after MTBI." Authors did not provide an operational definition of what "cognitive deterioration" is, nor did they provide any cognitive tests scores. % CI: 33.9

Summary of Studies 6 Months Post Injury
(Continued )  Summary: Participants with MTBI and 3 or more post-concussion symptoms were significantly worse on the Stroop Color Word Interference test compared to participants with MTBI and 2 or fewer symptoms at 6 months post injury. Using the control participants as normative data, 6/9 participants with 3 or more symptoms at 6 months were <10 th percentile on the Stroop, and 1/37 participants with less than 3 symptoms at 6 months. These main effects showed significant differences, but had cognitive performances collapsed across all post-injury epochs (i.e., 1 month, 6 months, and 12 months). In turn, it was not possible to determine if there was unique impairment in cognitive functioning for any outcome at 6 months. Although criteria for CI were described for 12 months post injury, they were not described for 6 months post injury.   Comparisons/Differences = NR. Definition of CI: "Although it is subjective, many would agree that the clinical significance of a difference of one-quarter of a standard deviation between a study group and its controls would be considered minor; one-half of a standard deviation, moderate; and one standard deviation, substantial" (p.18). % CI = NR (Continued )  Levin et al., (1996) Table 1), 1 study that revealed cognitive impairment in some subjects [46,61] and 7 studies that they did not think revealed cognitive impairment in any subjects [32, 33, 48-50, 58, 66]. None of the original authors of the 20 studies concluded that all subjects with MTBIs were cognitively impaired. Some of these studies conducted numerous statistical comparisons and identified only one or two significantly different test scores [28,40,41,64]. One study that McInnes et al [1] reported all subjects had cognitive impairment actually found no statistically significant differences between the MTBI and control group at the 12-month follow-up [55], and another study cited by McInnes et al [1] as showing evidence of cognitive impairment did not actually present, analyze, or interpret any cognitive test scores [47]. Only four studies [28, 31, 32, 61] used some sort of methodology in which individual subjects were classified as having cognitive impairment. One study enrolled 69 patients with MTBIs but only 16 underwent neuropsychological testing at one year following injury [28]. In this study, the MTBI group had significantly lower scores on only 2 of 23 test scores. The definition of cognitive impairment in this subgroup tested one year following injury was having at least one score that was 1.5 SDs below the normative mean. In the MTBI group, 69% had at least one low score. However, 44% of the control group also had at least one low score.

Summary of Studies 3 Months Post Injury
Jaffe et al., 1995 [32] x ; Country: USA; % Comp MTBI: NR; MTBI N = 40, Age = 6-15, Setting: NR; Control N = 40, Age = 6-15, Setting: NR. IQ = Yes (as outcome); Psych. = No Summary: Children with mTBI showed minimal to no differences in cognitive functioning compared to controls at 3 years post injury. Comparisons/Differences = NR. Definition of CI: "Although it is subjective, many would agree that the clinical significance of a difference of one-quarter of a standard deviation between a study group and its controls would be considered minor; one-half of a standard deviation, moderate; and one standard deviation, substantial" (p.18). % CI = NR Note: � McInnes and colleagues identified these studies as showing that all subjects had cognitive impairment [1]. McInnes et al. [1] identified 8 studies, at greater than one year post-injury time period, that they thought revealed all subjects to have cognitive impairment [40-42, 65, 67-69] (demarcated with an asterisk in Table 1), 1 study that revealed cognitive impairment in some subjects [29] and 1 study that they did not think revealed cognitive impairment [32]. Several studies conducted numerous statistical tests and reported only one or two significant findings (e.g., [41,[67][68][69]). Only three studies [29, 32, 65] used a methodology in which individual subjects were classified as having cognitive impairment. None of the original authors of the 10 studies concluded that all subjects with MTBIs were cognitively impaired.

Low neuropsychological test scores may or may not reflect acquired cognitive impairment
When inferring the cause of low neuropsychological test scores, it is important to appreciate that a person might obtain a low score due to situational factors, such as a lapse of attention, temporary distraction, not fully understanding the instructions, or low enthusiasm or motivation for testing. Moreover, a substantial percentage of healthy people with no prior brain injuries will obtain one or more low test scores when administered a battery of cognitive tests [70][71][72][73][74][75][76][77][78][79][80][81]. Researchers repeatedly have shown that it is very common for children [82], adults [76,83], and older adults [84], with no known clinical conditions that might affect cognition, to obtain at least one low score when a battery of tests measuring multiple cognitive domains is administered [81]. As the number of test scores increases, the probability of a healthy person obtaining one or more low scores increases [79,85]. The probability of obtaining a low score varies based on the a priori cutoff for defining a low score. For example, some clinicians and researchers define a low score as greater than 1 standard deviation below the mean, 1.5 standard deviations below the mean, or 2 standard deviations below the mean. Obtaining at least one low test score is also common in healthy people who are administered several tests within a cognitive domain, such as working memory [83,85], learning and memory [77,78,84], speed of processing [83,85], and executive functioning [86,87]. Using one study from this review as an example, Rieger et al. [23] classified patients based on whether or not they had a single below average score (i.e., one cognitive test score < 30 th percentile). They reported that 96% of the MTBI group met this threshold, but so did 85% of their orthopedic control patients.
Demographic and personal characteristics also are associated with the probability of obtaining low cognitive test scores. African Americans and Hispanics, on average, obtain more low scores than Caucasians [88][89][90][91][92][93]. Level of education is associated with test performance; those with terminal high school diplomas obtain more low scores than those with university degrees [94]. Moreover, intelligence is correlated with neuropsychological test performance, so those with below average intelligence will obtain more low scores than those with average intelligence [78,81,[94][95][96][97][98][99]. Therefore, low neuropsychological test scores may or may not reflect cognitive impairment following MTBI in individual cases. Per Tables 3-6 in the McInnes et al. review [1], 26.7% (n = 12/45) of the studies attempted to match MTBI patients to controls based on socioeconomic status, 57.8% (n = 26/45) matched for education, and 11.1% (n = 5/ 45) matched for race. Based on our review of these studies, fewer than half assessed intellectual functioning (see Table 1; n = 21/44 studies; 47.7%). Further, many of the studies that measured intelligence used it as an outcome variable that they thought may have been affected by MTBI. Most of these studies did not match for it between groups, control for it in statistical analyses, or discuss it as a potential confound when interpreting their results [e.g., [65]]. Current mental health problems, such as depression or anxiety, can influence cognitive test scores in patients without a MTBI [100] and in patients following a MTBI [101]. Of the studies in this review, 47.7% (n = 21/44) assessed for emotional symptoms (see Table 1), with very few of these studies accounting for the effect of emotional symptoms in their analyses or their interpretation of cognitive test results.

Conclusions
Cognitive impairment can occur following a TBI of any severity. Even very mild TBIs at least temporarily impact cognition [102]. The risk of persistent or permanent cognitive impairment increases in association with the severity of the brain injury [12,14,[103][104][105][106]. There is a considerable risk for long-term cognitive deficits after a moderate or severe TBI [14,103,107], though the type and severity of residual cognitive deficits is variable. Following a MTBI, cognitive impairment, as measured by neuropsychological tests, is likely to improve and resolve in the initial days, weeks, or months [2,12]. Patients with structural abnormalities on computed tomography or magnetic resonance imaging, referred to as having a complicated MTBI, tend to perform somewhat more poorly on neuropsychological tests than patients with uncomplicated MTBIs in the first two months following injury [108][109][110][111][112]. However, sustaining a complicated MTBI may not increase the risk of long-term (i.e., >6 months) cognitive deficits [113,114].
The running header for the scoping review by McInnes and colleagues [1] asserts "A single mTBI chronically impairs cognitive function." The article has the potential to misinform scientists, clinicians, and the public. Some recently published articles have cited the McInnes scoping review as illustrating a high rate of cognitive impairment following "concussion" or MTBI [115][116][117][118]. Clinicians who review and accept the findings of McInnes et al. will be misinformed and potentially communicate an inaccurate prognosis to patients with a single MTBI. Some patients may personally misinterpret the literature as suggesting they will suffer long-term cognitive deficits following a single MTBI through their own review of this open access article.
We believe that the review by McInnes et al., especially when taken together with the aggregated literature over the past 50 years, does not support the conclusion that approximately half of individuals who sustain a single mild traumatic brain injury (MTBI) experience long-term cognitive impairment. Their scoping review included quantitative analyses based on flawed methodology. Moreover, they did not assess the quality of included studies or exclude studies with a high risk of bias. We re-reviewed the articles they identified and found that the articles themselves do not support their conclusions. Their scoping review reached conclusions that are discrepant from several prior systematic reviews involving meta-analysis [2][3][4][5][6][7][8][9][10][11][12][13][14][15], which consistently reached the conclusion that the impact of MTBI on neuropsychological performance becomes undetectable at the group-level by three months post injury. These prior systematic reviews do not provide an estimate of the incidence of chronic cognitive impairment following MTBI (i.e., risk of long-term deficits in an individual patient), but suggest that it is low [2].