To validate the three time-difference neuropsychological protocols developed by the National Institute of Health/National Institute of Neurological Disorders and Stroke (NINDS) and the Canadian Stroke Network for assessment of vascular cognitive impairment (VCI) in Mandarin-speaking subjects and to investigate the clinical application of the shortest form.
Patients aged 50 years or older who had a stroke were invited to participate in the study. Clinical diagnosis of VCI was made. The NINDS-VCI Neuropsychology Protocols, 60-, 30-, and two 5-minute protocols, were administered. The criteria validities of the cognitive protocols against the diagnoses of stroke and VCI were determined via Receiver Operating Characteristic (ROC) analysis. The optimal cut-off point for the 5-minute protocols total score was estimated for clinical use in screening.
Eighty-three patients and 53 controls were recruited during the study period. Patients with stroke performed more poorly than the control group in the three neuropsychological protocols. Forty-two patients with stroke were diagnosed with VCI. VCI was used as the standard to estimate the criteria validities. The area under the ROC curve was 0.78, 0.80, 0.75, and 0.73 for the 60-, 30-, 5-mintue protocol-A and 5-minute protocol-B, respectively.
Citation: Lin H-F, Chern C-M, Chen H-M, Yeh Y-C, Yao S-C, Huang M-F, et al. (2016) Validation of NINDS-VCI Neuropsychology Protocols for Vascular Cognitive Impairment in Taiwan. PLoS ONE 11(6): e0156404. https://doi.org/10.1371/journal.pone.0156404
Editor: Terence J. Quinn, University of Glasgow, UNITED KINGDOM
Received: January 21, 2016; Accepted: May 14, 2016; Published: June 1, 2016
Copyright: © 2016 Lin et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: The study was funded by grants from the Ministry of Science and Technology, Taiwan (NSC101-2314-B-075-037-MY2; NSC100-2314-B-037-016-MY3), and Kaohsiung Medical University Hospital, Kaohsiung Medical University, Taiwan (KMUH-103-3R55). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Cognitive impairment affects up to half of stroke survivors . Postmortem studies have indicated that up to two-thirds of patients with dementia exhibit brain vascular pathology . These studies suggested that vascular lesions play a causative role in the development of cognitive impairment. Cerebrovascular disease may predispose, precipitate or perpetuate some cognitive impairment, which has been termed “vascular cognitive impairment” (VCI) . As there are heterogeneous cerebrovascular vascular pathologies, the clinical presentation and course of VCI are varied. VCI can affect functional disability  and is a strong predictor of death . It poses a huge economic and social burden in developed countries . As vascular risk factors or diseases are modifiable or treatable, early recognition and identification of VCI is of great importance in the management of these patients.
Assessment of VCI requires valid neuropsychological protocols that are sensitive to specific cognitive deficits in the context of vascular etiologies. Current tools in use for cognitive function assessment were developed in the main based on the cognitive patterns specific to Alzheimer’s disease (AD). Evidence from previous studies has clearly indicated that there are cognitive differences between AD and VCI [4, 7]. Generally, VCI patients have greater executive dysfunction and slowing of mental speed but better memory than AD patients. There is a need to develop a cognitive instrument specific to patients with CVD, which has adequate sensitivity for the detection of cognitive impairment associated with vascular etiologies . The National Institute of Health/National Institute of Neurological Disorders and Stroke (NINDS) and the Canadian Stroke Network sponsored a VCI Harmonization Workshop and developed standard protocols taking into account areas of neuropsychological research . Three neuropsychology protocols, namely, 60- minute, 30-minute and 5-minute protocols for use in different settings for the assessment of VCI, have been proposed by the Neuropsychology Working Group. The 60-minute protocol was prepared for use in circumstances that require detailed cognitive assessment. It is constructed with four domains: executive function, language, visuospatial function and memory. Each domain is composed of several individual subtests. The 30-minute protocol is used for screening at clinics specializing in cognitive impairment. Tests were selected from within the 60-minute protocol. The 5-minute protocol was designed for potential use by primary clinics or community services, where quick screening is needed. Administration of the 5-minute protocol by telephone interview has the potential for use in the community. As there exist language differences, the neuropsychological protocols require modification in accordance with each society, and the validity of all three protocols must be established before they can be considered useful for the diagnosis of VCI in patients who speak different languages.
In this study, the set of three protocols was translated and modified into Mandarin Chinese. The purpose of this study was to validate these protocols for use in cognitive assessment among patients with stroke. Furthermore, the criteria validities of these three protocols were examined by determining the ability of these protocols to differentiate CVD patients with VCI from those without.
Subjects and Methods
This study was conducted in outpatient and inpatient units of two teaching hospitals in Taiwan. Patients with stroke were invited to participate in the study if they were aged 50 years or older and had suffered a stroke within the past 3–18 months, defined according to the NINDS Stroke Data Bank criteria , with neuroimaging evidence of symptomatic cerebral infarction and at least one focal symptom and/or sign of stroke in the acute phase. Sufficient language skills and motor function for completion of neuropsychological tests were required. An informant, who spent at least 10 hours per week with the patients, knowledgeable about the subject’s recent and past medical history was required. The following causes of cognitive impairment were criteria for exclusion: (1) comorbidity with other brain diseases, such as neurodevelopmental and neurodegenerative diseases, hypoxic/anoxic states, hypotension or hypertension encephalopathy, inflammation, infection, epilepsy, and brain tumor; (2) an acquired metabolic, endocrine, traumatic, nutritional, or toxic disorder that could be judged to be affecting the brain; (3) major psychiatric disorder, such as substance use disorder, schizophrenia, and bipolar disorder; and (4) having diagnosis of dementia before the event of stroke.
Diagnosis of VCI was made according to observation of cognitive changes after stroke made by one knowledgeable informant of each patient. The research neurologist or psychiatrist interviewed each study participant and informant simultaneously using structural study interview schedule for diagnosis of cognitive disorders. The cognitive changes might involve remembering recent events, making decisions, handling personal matters, or language expression and comprehension. If the situation informants observed worsening of the patient’s condition, research physicians confirmed whether the changes came after the stroke according to the Diagnostic and Statistical Manual-5 (DSM-5) for neurocognitive disorder.
A control sample consisting of adults of similar age to the stroke sample was also studied. The inclusion criteria for these subjects were as follows: (1) no history of stroke or transient ischemic attack; (2) aged 50 or older; (3) possession of sufficient language skills to complete neuropsychological tests; (4) provision of written consent; and (5) achievement of a score of 24 or higher on the MMSE. The exclusion criteria were similar to those for the patient group.
The protocol was approved by the Institutional Review Boards of Taipei Veterans General Hospital and Kaohsiung Medical University Hospital. Written informed consent was obtained from all subjects prior to interview. If the participants have a compromised capacity to consent, their legally authorized family will provide written consents on the behalf of participants.
All subjects provided demographic information (sex, age, number of years of education and handedness), stroke history (duration of stroke illness) and medical and personal history. The National Institutes of Health Stroke Scale (NIHSS) was used to represent stroke severity . Study participants received neuropsychological tests in the sequence of 60-minutue protocol, 5-minute protocols and MMSE. The data of 30-minute was derived from data of 60-minute one. Two clinical neuropsychologists, who were blind to the diagnosis of VCI, preformed the neuropsychological tests.
The NINDS-VCI Neuropsychology Protocols
The 60-minute protocol was composed of four domains, within which there were several individual tests.
Animal Naming : Animal naming was used to assess category fluency. The score is the number of correct animal names produced in one minute.
WAIS-III Digit Symbol-Coding : This is a brief timed coding test that is easy to administer. The score is the number of correct symbol-number pairs produced in two minutes.
Trail-making Test A part : This is a short test involving working memory and set shifting. It is scored in terms of time to completion and number of errors.
Boston Naming Test (BNT) 2nd edition, Short Form : This is the most frequently used confrontation naming test. The score is the number of items correctly named.
Rey-Osterrieth Complex Figure Copy : This test evaluates visuospatial abilities and executive function. A basic 36-item score based on location and accuracy is obtained , and an additional method allows for scoring of executive function .
Hopkins Verbal Learning Test-Revised (HVLT-R): This is a 12-item supra-span list-learning task including three learning trials, a 20–25-minute delayed recall condition and a delayed recognition task. Three scores are generated, which represent the sum of items recalled across the three learning trials, the number of items recalled upon delay, and the recognition discrimination index.
Boston Naming Test Recognition : A 20–25-minute delayed forced-choice recognition test for the BNT is administered, with an additional 15 items from the test as foils. The score is the number of pictures correctly recognized.
Digit Symbol-Coding Incidental Learning : This includes free-recall drawing of the symbols used, followed by recall of matched number-symbol pairs. The two recorded scores are the number of symbols correctly recalled and correctly matched.
Complex Figure Memory : This test includes free-recall drawing of a complex figure 20–25 minutes after viewing the initial copy.
The following sub-set of tests included in the 60-minute protocol is included in the 30-minute protocol: Animal naming, Digit Symbol-Coding, and HVLT-R. These test results were analyzed as a separate unit.
We created two versions of the 5-minute protocols. The first one, 5-minute protocol-A, consists of three subtests of the Montreal Cognitive Assessment (MoCA), and the Taiwanese version of the MoCA was validated previously by our group [19, 20]. It consists of a 20-point scale, with the total score computed as the sum of a 5-word delayed recall (5-point), a 6-item orientation (6-point), and the animal fluency test (0.5 points for each correct exemplar generated within one minute; maximum score 9 points) of the MoCA. The second test, the 5-minute protocol-B, replaced the orientation test with the WAIS-III Digit Symbol-Coding test. The rationale for the replacement is that Digit Symbol-Coding test is sensitive to brain lesions involving vascular pathology [21, 22]. It should be suitable to be include a brief screening tool. Raw scores of the WAIS-III Digit Symbol-Coding test were transformed to scaled scores  based on age. Then, 0.5 points were given for each scaled score, with a maximum score of 6 points, for the WAIS-III Digit Symbol-Coding test. The 5-minute protocol-B also yielded a maximum total score of 20. Both of these brief tasks were chosen because they assess both memory and executive function and can be used in a variety of languages and cultures.
The 30-point Mini-Mental State Examination (MMSE) is a popular general cognitive screening examination, and was used for assessment of cognitive function in this study. This test assesses orientation, registration, attention, calculation, language, and recall, and has been widely used to assess elderly people. The total score ranges from 0 to 30, with a lower score indicating poorer cognitive function.
The Geriatric Depression Scale (GDS), first created by Yesavage, et al. , has been translated in Taiwanese . We used the Short Form GDS which consists of 15 questions. The Short Form is more easily used by physically ill who may have short attention spans or feel easily fatigued.
The scores were then converted to standardized z scores, based on the scores of the control sample. For the 60-minute protocol, the z scores for the individual tests were grouped to form four domain z scores (executive, language, visuospatial function and memory), and an overall 60-minute summary z score was obtained as the average of these four z scores. One summary z score of the 30-minute protocol was computed with a single memory z score as the average of three HVLT z scores, the Animal Naming z score, and the Digit Symbol z score. Missing data will be excluded when calculating the average of each domain score. The raw total score was computed for the 5-minute protocols.
Student t tests or chi-square analyses were used for group comparisons. When the expected values in any of the cells of a 2 by 2 contingency table are below 5, we will use Fisher exact tests for comparisons. Correlations of two continuous variables were analyzed via Pearson’s correlation analysis. Statistical analyses were performed using the SPSS 14.0 statistical program, and the statistical significance level was set as p <0.05. We examined the external validity of the protocol according to how well its summary score can discriminate patients from controls using receiver operating curve (ROC) analysis. The criteria validities of the cognitive protocols against VCI were determined via ROC analysis . Areas under the receiver operating characteristic curve (AUC) were compared using the ROCCONTRAST option of the logistic regression procedure of the SAS software program. The optimal cut-off point for the 5-minute protocol total score was determined using Youden-Index for which (sensitifity + specificity) is maximal [27, 28]. Accordingly, criteria validity indices were estimated with respect to sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV).
Demographic and clinical data
Eighty-three patients and 53 controls were recruited during the study period. The mean age of the stroke patients was 66.6 ± 9.7 years, and the education level was 9.2 ± 4.8 years. Males accounted for 61.9% (n = 52) in this cohort. The mean NIHSS of the stroke patients at admission was 2.2 (SD = 2.4). The mean number of days from onset of the index stroke to cognitive assessment was 9.0 ± 5.4 months. The numbers of patients with various physical conditions were as follows: smoking 23 (27.7%), alcohol drinking 5 (6.0%), hypertension 71 (85.5%), diabetes mellitus 39 (47.0%), hyperlipidemia 42 (50.6%), myocardial infarction 7 (8.4%), atrial fibrillation 11 (13.0%), heart failure 4 (4.8%), and heart valve disease 1 (1.2%). The demographic and clinical data are shown in table 1. Differences between the patients and controls were noted in terms of education level and for the physical conditions of smoking, hypertension, diabetes mellitus, and atrial fibrillation. A brief description of the neuroimaging features of the stroke patients is provided in table 2. Three of the stroke patients (3.6%) were cases of intracerebral hemorrhage, and the remainder, ischemic stroke patients.
All measures regarding cognitive function, both individual tests and domain scores, were lower in the patients with stroke than in the controls. The protocol summary scores of the 60-minute, 30-minute and 5-minute protocols were also all lower in the patient group (table 3).
Correlations between each protocol and the MMSE were examined, and the results revealed that the Pearson correlation coefficients were 0.75 (p <0.001) for the 60-minute protocol, 0.75 (p <0.001) for the 30-minute protocol, and 0.70 (p <0.001) for both of the 5-minute protocols.
Criteria validities to differentiate stroke patients and controls
ROC analyses were used to represent the external validities of these protocols to differentiate patients with stroke from controls. The area under the ROC curve was 0.78 (95% CI = 0.70–0.87; p <0.001) for the 60-minute protocol, 0.85 (95% CI = 0.77–0.92; p <0.001) for the 30-minute protocol, 0.78 (95% CI = 0.70–0.86; p <0.001) for the 5-minute protocol-A, and 0.83 (95% CI = 0.75–0.90; p <0.001) for the 5-minute protocol-B. Comparisons of the AUCs of the three protocols revealed that the AUC of the 30-minute protocol was greater than that of the 60-minute protocol (p = 0.029) and that of the 5-minute protocol-A (p = 0.007). Other comparisons of AUCs did not show statistical significance (p = 0.83 for 60-minute vs 5-minute-A; p = 0.20 for 60-minute vs 5-minute-B; p = 0.46 for 30-minute vs 5-minute-B; and p = 0.10 for 5-minute-A vs 5-minute-B).
Criteria validities to differentiate VCI and non-VCI among patients with stroke
Forty-two patients with stroke were diagnosed with VCI, including 13 with vascular dementia, based on our research criteria. Their demographic, clinical and cognitive characteristics are shown in table 4. Compared with the patients without cognitive impairment, there were no significant statistical differences in the demographic and clinical variables except educational level, whereas all cognitive assessment results, including individual tests and domain z scores, were poorer among the patients with VCI.
For patients with stroke alone, the criteria validities of the 60-, 30-, and 5-mintue protocols to differentiate patients with VCI from stroke patients without VCI were assessed. The area under the ROC curve was 0.78 (95% CI = 0.68–0.89; p <0.001) for the 60-minute protocol, 0.80 (95% CI = 0.69–0.90; p <0.001) for the 30-minute protocol, 0.75 (95% CI = 0.63–0.86; p <0.001) for the 5-minute protocol-A and 0.73 (95% CI = 0.61–0.85; p = 0.002) for the 5-minute protocol-B. The p value of each pairwise comparison of the AUCs of all protocols did not reach statistical significance (p = 0.66 for 60-minute vs. 30-minute;, p = 0.50 for 60-minute vs. 5-minute-A; p = 0.25 for 60-minute vs. 5-minute-B; p = 0.16 for 30-minute vs. 5-minute-A;, p = 0.13 for 30-minute vs. 5-minute-B; p = 0.61 for 5-minute = A vs. 5-minute-B).
Criteria validity indices for the all protocols
The criteria validities in terms of sensitivity, specificity, PPV, and NPV for the 60-minute, 30-minute and both 5-minute protocols were examined. To differentiate patients with stroke and controls, the 60-minute protocol had a sensitivity of 78.3%, a specificity of 67.9%, a PPV of 79.3%, and a NPV of 66.7% at the optimal cut-off point of -0.16 z score. The 30-minute protocol had a sensitivity of 89.2%, a specificity of 71.7%, a PPV of 83.1%, and a NPV of 80.9% at the optimal cut-off point of -0.34 z score. As the total scores of the 5-minute protocols could be summed directly, we suggest that the optimal cut-off point for the 5-minute protocol-A is 15/16, which had a sensitivity of 73.1%, a specificity of 64.2%, a PPV of 72.1%, and a NPV of 65.4%. The 5-minute protocol-B had a sensitivity of 82.1%, a specificity of 67.9%, a PPV of 76.4%, and a NPV of 75.0% at the cut-off point of 12/13.
When differentiating stroke patients with VCI from stroke patients without VCI, the 60-minute protocol had a sensitivity of 73.8%, a specificity of 75.6%, a PPV of 75.6%, and a NPV of 73.8% at the optimal cut-off point of -0.85 z score. The 30-minute protocol had a sensitivity of 71.4%, a specificity of 68.3%, a PPV of 69.8%, and a NPV of 70.0% at the optimal cut-off point of -1.15 z score. The optimal cut-off point for the 5-minute protocol-A was 14/15, which had a sensitivity of 80.6%, a specificity of 54.8%, a PPV of 67.4%, and a NPV of 70.8%. The 5-minute protocol-B had a sensitivity of 75.0%, a specificity of 61.3%, a PPV of 69.2%, and a NPV of 67.9% at the cut-off point of 10/11.
The results of this study showed that all three Mandarin NINDS-VCI Neuropsychology Protocols had a good validity for the differentiation of patients with VCI from those without. All three protocols presented a good convergent validity with the MMSE. Using two 5-minute protocols for VCI screening purposes, there were fair to good criteria validity indices, including sensitivity, specificity, PPV, and NPV.
There have been various Chinese versions of NINDS-VCI neuropsychological batteries developed in Asian, such as in Hong Kong  and China . Including ours, the main change in these versions from original version were the replacement of the test to assess letter fluency with one to assess category fluency. Given the considerable involvement of language in the verbal fluency test, it is likely infeasible that letter verbal fluency can be directly applied in the Chinese context. As both phonemic and semantic verbal fluency deficits in vascular dementia may relate to a similar effortful retrieval requirement, and patients with small vessel disease exhibit similar impairments in both letter and category fluency tasks , the satisfactory psychometric properties of this Mandarin version suggest the feasibility of this modification. Another change was that the Trail-Making Test Part B was not adapted in this study, as a substantial proportion of the elderly in Taiwan is illiterate. The differences within the three Chinese versions were mainly in the 5-minute protocols. In our study, the 5-minute protocol-A did not include immediate (5 points) and recognition memory (5 points), unlike the other two Chinese versions. The 5-minute protocol-B replaced the orientation part (6 points) with the WAIS-III symbol digit test (6 points). Our two 5-minute protocols placed more weight on tests involving executive function and processing speed. Including our version, all Chinese versions of 5-minute protocols exhibited fair to good accuracy in differentiating stroke patients from controls. There were large effect sizes of difference across all cognitive measurements assessed, not only in differentiating stroke patients from healthy controls, but in differentiating stroke patients with VCI from those without. The effect sizes can be represented with high AUCs of 60-, 30-, and 5-minute protocols (0.78, 0.85, 0.78 in differentiating stroke patients; and 0.85, 0.88, 0.82 in differentiating VCI patients). These results of effect sizes were similar with two previous studies [29, 30], which demonstrated adequate discriminatory power. The findings supported the validity of these 3 protocols for cognitive assessment of patients with stroke, and moreover, for patients with VCI. The 60-minute protocol has the advantage of yielding an assessment of breakdown of cognitive performance across various cognitive domains. The pattern of cognitive impairment is dependent on the locations involved in the stroke, especially within the cortical area. A comprehensive assessment may cover all patterns of VCI. The use of the 60-minute protocol has considerable benefits for comprehensive measurement of cognitive status or cognitive change in clinical assessment. For the 30-minute protocol, the effect size of difference between cases and controls and the ROC AUC value appear larger than those for the 60-minute protocol, which may be explained by the fact that the content of the 30-minute protocol focuses more on executive dysfunction and slowing of mental speed associated with vascular lesions. As both protocols have distinct advantages, we suggested that which protocol is selected in use dependent on how much time available at clinical assessment.
The results of this study supported the usefulness of the 5-minute protocol in the primary clinical setting. In the primary clinical setting, it is important to identify individuals with cognitive complaints related to objective impairment. The 5-minute protocol is more simple to perform than the other two protocols, and the total score without standardization is easily obtained in the primary clinical setting. Our findings of a fair to good criteria validity indicated that the 5-minute protocol can differentiate subjects with VCI from those without, even for the target population with stroke. Among the criteria validity indices, we obtained a good sensitivity and NPV, whereas the specificity and PPV only showed fair values. A relatively large portion of stroke patients without VCI scored low of the 5-minute protocol might explain the fair specificity and PPV. That is, the false positives were high. In fact, patients with stroke may present with slowing of psychomotor activities, even without significant cognitive impairment. Sometimes, they may perform below their ability when completing a new neuropsychological test. Being allowed more time to complete tests may reduce the false positive rate. Beyond the caveats, according to the nature of free of pencil and paper tests, further studies can be conducted to examine their feasibility by phone administration.
Some study caveats have to be addressed with reference to standards for studies of diagnostic test accuracy in dementia . Diagnosis of VCI in this study was made based mainly on cognitive decline and not impairment of activities of daily living; that is, patients with VCI may include those with a diagnosis of vascular dementia. Another diagnostic issue was that neuropsychological tests, in general, are more sensitive to detect mild cognitive deficits as they are present in VCI than informant information which rather capture more obvious and severe deficits. Although the rate of VCI of these study sample was compatible to those in the literature, it is still need to be addressed for the diagnostic limitation in this study. As the study participants were not community-dwelling residents, and the validity is dependent on the prevalence of VCI in the specific setting, misclassification, including false positives, of subjects using this tool is possible. As calculating predictive values is problematic based on the prevalence from case-control data, the PPV and NPV of this case-control study should be interpreted cautiously interpreted. Generalization to other societies should be performed cautiously. Second, there is still variability even within Chinese-speaking societies, and therefore cross-culture validation in other settings and populations is necessary. Additionally, the distribution of cases in this study was mainly restricted to infarction patients and those with sufficient language skills and motor function to complete the neuropsychological tests; more information about other stroke patients is needed. Third, all study participants underwent neuropsychological tests in the following sequence: 60-minute protocol, 5-minute protocol-A, and MMSE. Data for the 30-minute protocol and the 5-minute protocol-B were derived, rather than assessed directly. It is possible that the results would exhibit minor differences if the sequence of administration was not exactly the same for each patient. Fourth, those with low motivation and a low willingness to participate in the study were excluded. As a significant proportion of patients who have suffered a stroke have depressive symptoms, we decided not to exclude patients with higher GDS scores. However, a possible confounding effect of depression cannot be ruled out. Finally, most diagnostic tests require large samples to increase the power estimates.
We found that all three Mandarin NINDS-VCI Neuropsychology Protocols had a good validity to differentiate patients with VCI from those without. The three protocols are suggested to be used in various settings and for different assessment purposes, such as comprehensive cognitive assessment or for screening of VCI.
The authors would like to thank Mr. Yi-Lung Chen for his assistance in statistical analysis.
Conceived and designed the experiments: CSC SJW JLF. Performed the experiments: HFL HMC YCY SCY MFH CSC JLF. Analyzed the data: HFL HMC YCY CSC JLF. Contributed reagents/materials/analysis tools: HMC SCY. Wrote the paper: HFL CMC SJW CSC JLF.
- 1. Madureira S, Guerreiro M, Ferro JM. Dementia and cognitive impairment three months after stroke. European journal of neurology: the official journal of the European Federation of Neurological Societies. 2001;8(6):621–7. pmid:11784347.
- 2. Strozyk D, Dickson DW, Lipton RB, Katz M, Derby CA, Lee S, et al. Contribution of vascular pathology to the clinical expression of dementia. Neurobiology of aging. 2010;31(10):1710–20. pmid:18996621; PubMed Central PMCID: PMC2888978.
- 3. Hachinski V, Iadecola C, Petersen RC, Breteler MM, Nyenhuis DL, Black SE, et al. National Institute of Neurological Disorders and Stroke-Canadian Stroke Network vascular cognitive impairment harmonization standards. Stroke. 2006;37(9):2220–41. Epub 2006/08/19. 01.STR.0000237236.88823.47 [pii] pmid:16917086.
- 4. Gorelick PB, Scuteri A, Black SE, Decarli C, Greenberg SM, Iadecola C, et al. Vascular contributions to cognitive impairment and dementia: a statement for healthcare professionals from the american heart association/american stroke association. Stroke; a journal of cerebral circulation. 2011;42(9):2672–713. pmid:21778438; PubMed Central PMCID: PMC3778669.
- 5. Tatemichi TK, Paik M, Bagiella E, Desmond DW, Pirro M, Hanzawa LK. Dementia after stroke is a predictor of long-term survival. Stroke; a journal of cerebral circulation. 1994;25(10):1915–9. pmid:8091433.
- 6. Rockwood K, Brown M, Merry H, Sketris I, Fisk J, Vascular Cognitive Impairment Investigators of the Canadian Study of H, et al. Societal costs of vascular cognitive impairment in older adults. Stroke; a journal of cerebral circulation. 2002;33(6):1605–9. pmid:12052999.
- 7. Reed BR, Mungas DM, Kramer JH, Ellis W, Vinters HV, Zarow C, et al. Profiles of neuropsychological impairment in autopsy-defined Alzheimer's disease and cerebrovascular disease. Brain: a journal of neurology. 2007;130(Pt 3):731–9. pmid:17267522.
- 8. Foulkes MA, Wolf PA, Price TR, Mohr JP, Hier DB. The Stroke Data Bank: design, methods, and baseline characteristics. Stroke; a journal of cerebral circulation. 1988;19(5):547–54. pmid:3363586.
- 9. NIH Stroke Scale Training,Part 2. Basic Instruction. Department of Health and Human Services, National Institute of Neurological Disorders and Stroke. The National Institute of Neurological Disorders and Stroke (NINDS) Version 2.0.
- 10. Barr A, Brandt J. Word-list generation deficits in dementia. Journal of clinical and experimental neuropsychology. 1996;18(6):810–22. pmid:9157106.
- 11. Wechsler D. WAIS-III Administration and Scoring Manual. San Antonio, TX: The Psychological Corporation; 1997.
- 12. Reian RM, Wolfson D. The Halstead-Reitan Neuropsychological Test Battery: Theory and Clinical Interpretation. 2nd ed. Tucson, AZ: Neuropsychology Press; 1993.
- 13. Goodglass H, Kaplan E, B. B. The Assessment of Aphasia and Related Disorders. Third ed. New York: Lippincott Williams & Wilkins; 2001.
- 14. Rey A. L’examen psychologique dans les cas d’encephalopathie traumatique. Arch de Psychologie. 1941;28:286–340.
- 15. Chiulli SJ, Haaland KY, LaRue A, Garry PJ. Impact of age on drawing the Rey-Osterrieth Figure. Clin Neuropsychol. 1995;9:219–24.
- 16. Somerville J, Tremont G, Stern RA. The Boston Qualitative Scoring System as a measure of executive functioning in Rey-Osterrieth Complex Figure performance. Journal of clinical and experimental neuropsychology. 2000;22(5):613–21. pmid:11094396.
- 17. Brandt J, Benedict RHB. Hopkins Verbal Learning Test-Revised Professional Manual. Lutz, FL: Psychological Assessment Resources, Inc.; 2001.
- 18. Kaplan E, Fein D, Morris R, Delis D. The WAIS-R as a Neuropsychological Instrument. San Antonio: Psychological Corporation; 1991.
- 19. Nasreddine ZS, Phillips NA, Bedirian V, Charbonneau S, Whitehead V, Collin I, et al. The Montreal Cognitive Assessment, MoCA: a brief screening tool for mild cognitive impairment. Journal of the American Geriatrics Society. 2005;53(4):695–9. pmid:15817019.
- 20. Tsai CF, Lee WJ, Wang SJ, Shia BC, Nasreddine Z, Fuh JL. Psychometrics of the Montreal Cognitive Assessment (MoCA) and its subscales: validation of the Taiwanese version of the MoCA and an item response theory analysis. International psychogeriatrics / IPA. 2012;24(4):651–8. pmid:22152127.
- 21. O'Sullivan M, Morris RG, Markus HS. Brief cognitive assessment for patients with cerebral small vessel disease. Journal of neurology, neurosurgery, and psychiatry. 2005;76(8):1140–5. pmid:16024894; PubMed Central PMCID: PMC1739761.
- 22. Zhou A, Jia J. A screen for cognitive assessments for patients with vascular cognitive impairment no dementia. International journal of geriatric psychiatry. 2009;24(12):1352–7. pmid:19347838.
- 23. Folstein MF, Folstein SE, McHugh PR. "Mini-mental state". A practical method for grading the cognitive state of patients for the clinician. J Psychiatr Res. 1975;12(3):189–98. Epub 1975/11/01. pmid:1202204.
- 24. Yesavage JA, Brink TL, Rose TL, Lum O, Huang V, Adey M, et al. Development and validation of a geriatric depression screening scale: a preliminary report. Journal of psychiatric research. 1982;17(1):37–49. pmid:7183759.
- 25. Liu CY, Lu CH, Yu S, Yang YY. Correlations between scores on Chinese versions of long and short forms of the Geriatric Depression Scale among elderly Chinese. Psychological reports. 1998;82(1):211–4. pmid:9520556.
- 26. Swets JA. ROC analysis applied to the evaluation of medical imaging techniques. Invest Radiol. 1979;14(2):109–21. Epub 1979/03/01. pmid:478799.
- 27. Youden WJ. Index for rating diagnostic tests. Cancer. 1950;3(1):32–5. pmid:15405679.
- 28. Schisterman EF, Perkins NJ, Liu A, Bondell H. Optimal cut-point and its corresponding Youden Index to discriminate individuals using pooled blood samples. Epidemiology. 2005;16(1):73–81. pmid:15613948.
- 29. Wong A, Xiong YY, Wang D, Lin S, Chu WW, Kwan PW, et al. The NINDS-Canadian stroke network vascular cognitive impairment neuropsychology protocols in Chinese. Journal of neurology, neurosurgery, and psychiatry. 2013;84(5):499–504. pmid:23250962.
- 30. Chen X, Wong A, Ye R, Xiao L, Wang Z, Lin Y, et al. Validation of NINDS-CSN neuropsychological battery for vascular cognitive impairment in Chinese stroke patients. BMC neurology. 2015;15:20. pmid:25886571; PubMed Central PMCID: PMC4350916.
- 31. Herbert V, Brookes RL, Markus HS, Morris RG. Verbal fluency in cerebral small vessel disease and Alzheimer's disease. Journal of the International Neuropsychological Society: JINS. 2014;20(4):413–21. pmid:24589176.
- 32. Noel-Storr AH, McCleery JM, Richard E, Ritchie CW, Flicker L, Cullum SJ, et al. Reporting standards for studies of diagnostic test accuracy in dementia: The STARDdem Initiative. Neurology. 2014;83(4):364–73. pmid:24944261; PubMed Central PMCID: PMC4115600.