Validity and reliability of two alternate versions of the Montreal Cognitive Assessment (Hong Kong version) for screening of Mild Neurocognitive Disorder

Objective Repeated testing using the Montreal Cognitive Assessment (MoCA) increases risks for practice effects which may bias measurements of cognitive change. The objective of this study is to develop two alternate versions of the MoCA (Hong Kong version; HK-MoCA) and to investigate the validity and reliability of the alternate versions in patients with DSM-5 Mild Neurocognitive Disorder (Mild NCD) and cognitively healthy controls. Methods Concurrent validity and inter-scale agreement were examined by Pearson correlation of the total scores between the original and alternate versions and the Bland-Altman Method. Criterion validity of the two alternate versions in differentiating patients with Mild NCD was tested using receiver operating characteristic curve (ROC) analysis. One-month test-retest and inter-rater reliability were examined in 20 participants. Internal consistency of the alternate versions was measured by the Cronbach’s α. Results 30 controls (age 73.4 [4.5] years, 60% female) and 30 patients (age 75.4 [5.5] years, 73% female) with Mild NCD were recruited. Both alternate versions significantly correlated with the original version (r = 0.79–0.87, p<0.001). Mean differences of 0.17 and -0.40 points were found between the total scores of the alternate with the original versions with a consistent level of agreement observed throughout the range of cognitive abilities. Both alternate versions significantly differentiated patients with Mild NCD from healthy controls (area under ROC 0.922 and 0.724, p<0.001) and showed good one-month test-retest reliability (intra-class correlation [ICC] = 0.92 and 0.82) and inter-rater reliability (ICC = 0.99 and 0.87) and high internal consistency (Cronbach α = 0.79 and 0.75). Conclusion The two alternate versions of the HK-MoCA are useful for Mild NCD screening.


Introduction
Mild Neurocognitive Disorder (NCD) is a newly added diagnostic entity recently published in the Diagnostic and Statistical Manual of Mental Disorders, 5 th Edition (DSM-5). [1] Mild NCD refers to a syndrome of cognitive decline that is of modest severity with minimal impact on daily functions. While Mild NCD and mild cognitive impairment (MCI) are conceptually similar, [2], Mild NCD encompasses individuals with a milder, possibly earlier stage of cognitive decline compared to MCI. The profile of cognitive domains affected in NCD can be heterogeneous depending on the disease etiology. For example, a predominant learning and memory impairment is expected in patients with NCD due to Alzheimer's Disease (AD), whereas NCD due to cerebrovascular disease is characterized by executive functions and attentional deficits. Furthermore, the diagnosis of Mild NCD requires a psychometric performance from -1 standard deviation (SD) relative to a matched normative group whereas MCI requires a comparatively more severe impairment from -1.5SD. Screening for mild NCD, therefore, requires cognitive instruments that are sensitive to subtle decline across a comprehensive set of cognitive domains. The Mini-mental State Examination (MMSE) has been reported to be insensitive to detect mild cognitive dysfunction in different neurological disorders and it use is currently fee charging by a commercial company. [3][4][5] A number of cognitive tests, for example, the Montreal Cognitive Assessment (MoCA), [3] the Oxford Cognitive Screen [6,7] and the Birmingham Cognitive Screen, [8,9] have been developed to assess mild cognitive dysfunctions of various etiologies and are currently available in multiple languages.
The MoCA is a 15-minute cognitive test developed for screening of MCI. It includes items to assess a broad range of cognitive domains and abilities including executive functions, visuospatial abilities, language, attention, working memory, abstraction and orientation. The MoCA has been validated in a large number of conditions including stroke, dementia, psychiatric disorders, substance abuse and is increasingly used as the cognitive outcome measure in clinical trials (see http://www.mocatest.org for an updated list of publications) and is one of the most widely used cognitive screening tests around the world. [3] The Hong Kong version of the MoCA (HK-MoCA) is administered in the Cantonese language. The HK-MoCA has been validated and investigated in a variety of clinical conditions including MCI, AD, stroke, cerebral small vessel disease and brain injury [10][11][12][13][14][15] with age-and education-adjusted normative data available for classification. [16] The HK-MoCA, along with its abbreviated version, the HK-MoCA 5-minute Protocol, [17] are officially listed as the cognitive screening tests for dementia by the Hospital Authority in Hong Kong. To-date, these tests are the most widely used cognitive screen tests in medical and social services in Hong Kong. Cantonese-speaking people constitute an important subgroup of the population in many parts of the world. Over 65 million people in Hong Kong and China speak Cantonese as a first language. Given the rapidly aging population, Cantonese-speaking patients are increasingly encountered in healthcare settings around the world.
With its widespread clinical use, a significant issue inherent in cognitive assessment is the gain of test scores due to an increased familiarity of test stimuli and procedure as a result of repeated exposures to the same test, a phenomenon known as the practice effect. [18] Individuals perform better over repeated testing because they have learned and remembered the test items from previous exposures to the same tests. For example, it has been observed that, on average, individuals with normal cognition gain one point on the MMSE when retested after one year. [19] Practice effects are problematic for longitudinal studies and for monitoring of disease progression or treatment effects because such effects can mask true cognitive changes. Unfortunately, despite the recognition of practice effects, the same version of cognitive test is often used repeatedly in clinical settings and even in large-scale clinical trials. One widely accepted method to circumvent the problem of practice effects is the use of alternate version of the same test. Alternate versions consist of sets of stimuli that are different than that of the original version so that previous exposures to one version of the test have minimal effect upon the performance in other versions. Although alternate versions of the MoCA are available for the English [20,21] and French versions, [22] there are no published alternate versions for the MoCA in Asian languages.
The objective of this study is to develop two alternate versions of the HK-MoCA and to investigate the concurrent and criterion validity, test-retest and inter-rater reliability as well as internal consistency of the alternate versions in patients with Mild NCD and cognitively normal older adults. We hypothesize that the two alternate versions have adequate psychometric properties for use as a screening instrument for Mild NCD.

Methods
Sixty participants, including 30 patients with Mild NCD and 30 cognitively normal controls, were recruited as a convenience sample from the Movement and Cognitive Disorders Clinic of the Prince of Wales Hospital and from a pool of 800 functionally independent community older adults recruited in a community study (Risk index for screening subclinical brain lesions in community-dwelling elderly persons in Hong Kong [ref. CUHK471911]). [16] Inclusion criteria for normal controls were normal cognition, as defined by performance within one SD in all cognitive domains (executive functions, learning and memory) measured using the NINDS-CSN VCI 30-minute Neuropsychology protocol (Table 1). [12] Mild NCD was diagnosed according to DSM-5 criteria [1] described in the following: 1) subjective cognitive impairment, defined as !1 positive response on the Abbreviated Memory Inventory for Chinese; [23] 2) objective cognitive impairment, defined as performance -1SD on any of the measure in the NINDS-CSN VCI 30-minute Neuropsychology protocol; [12,24] and 3) functional independency, defined as a score of !2 on Lawton's Instrumental of Daily Living Scale (IADL). [25] Common inclusion criteria for both groups were 1) aged !65 years, 2) adequate Table 1. Description of the NINDS-CSN VCI 30-minute Neuropsychology protocol.

Cognitive Domain Test Measure
Executive functions Symbol Digit Modalities Test (SDMT) total correct [26] Animal Verbal Fluency Test total correct Learning and Memory Hong Kong List Learning Test (HKLLT) Ã 10-minute delayed recall/30-minute delayed recall and recognition [27] sensorimotor and language competency to complete cognitive testing and 3) written informed consent given. Common exclusion criteria were history of stroke or active neurological or psychiatric disorders with known effects upon cognition. This study was approved by the Joint Chinese University of Hong Kong-New Territories East Cluster Clinical Research Ethics committee and written informed consent was obtained from each participant. Study procedures were performed in accordance to the Declaration of Helsinki of 1975 and its later revisions. Permissions for use of the MoCA and the development of the alternate versions as described in the study were obtained from the original author of the MoCA (co-author Dr. Ziad Nasreddine). This study is part of the "Brain Health Brings Health" Programme of the Division of Neurology at the Chinese University of Hong Kong.

Development of alternate versions of HK-MoCA
Two alternate versions of the HK-MoCA, namely HK-MoCA-A1 and HK-MoCA-A2, were developed. [28] An item was replaced in the alternate version if it was judged to be prone to practice effects by expert consensus by a group of healthcare professionals consisting of neurologists, a clinical psychologist and occupational therapists who had extensive experience in using the HK-MoCA in clinical practice. Replaced items were selected for domain-specificity and relevance to the cultural and demographic background of the intended users. A description of the adaptations made in each version is shown in Table 2. Serial subtraction: "100 minus 7" replaced by "80 minus 7".

Language
Sentence repetition: the two sentences are replaced by sentences "Mom does yoga" and "Teacher teaches writing poems."

Data collection
Study procedures were carried out at the designated research clinics at the Prince of Wales Hospital in Hong Kong. Each assessment session lasted within an hour to minimize the possibility of significant fatigue which might confound cognitive test performance. Demographic data including age, sex and years of education were collected. Participants were administrated the NINDS-CSN VCI Neuropsychology 30-minute protocol as part of the screening procedure and were classified into the cognitively healthy control group and Mild NCD group. All participants received the HK-MoCA before the NINDS-CSN VCI Neuropsychology 30-minute protocol. Tests were administered and scored by trained psychometricians according to the published protocols. [26,27,29] Each participant received the original HK-MoCA and one of the two alternate versions. Therefore, 30 subjects (regardless of group membership) received HK-MoCA-A1 whereas the remaining 30 received HK-MoCA-A2. The specific alternate version each participant was tested and the order of administration of the original and alternate versions were randomized using computer-generated codes. Administrations of the original and alternate versions spanned one month apart from each other. To assess test-retest reliability of the alternate versions, 20 participants, including 10 controls and 10 patients, were randomly selected and retested with the same alternate version after a one-month period by the same examiner. Inter-rater reliability was examined in 20 randomly selected participants (10 controls and 10 patients) by two different examiners over a one-month period.

Statistical analysis
Demographic characteristics were compared between patients with Mild NCD and healthy controls using independent sample t test for continuous data and x 2 test or Fisher's Exact test for dichotomous data. Given the potential confounding effects on cognitive test performance due to the difference in education attainment between the two groups, analysis of co-variance (ANCOVA) with adjustment years of education in addition to age was used to compare the group performance on cognitive test scores. Concurrent validity was measured by Pearson correlation between the total scores between the original (HK-MoCA-O) with the alternate versions. Inter-scale agreement between the total scores of the original and alternate versions was examined using the Bland-Altman Method, which is designed to evaluate the agreement of two continuous measurements.
[30] Using this method, the mean difference (i.e. bias) and 95% limits of agreement (LoA) of the total scores between the original and alternate versions were calculated. Smaller inter-scale difference and 95% LoA denote higher inter-scale agreement. Criterion validity of the two alternate versions in differentiating patients with Mild NCD from healthy controls was tested using receiver operating characteristic (ROC) curve analysis.
[31] The ROC analysis is a commonly used method for measuring diagnostic accuracy. The Area Under the ROC curve (AUC) represents the chance that when one sample is drawn from a truly normal population and another sample drawn from a truly abnormal population, the score of the normal sample will be higher than that of the abnormal sample (or vice versa, depending on the scoring system). It is an index of diagnostic accuracy and ranges between 0 and 1. AUCs closer to 0 or 1 represent greater accuracy and AUC of 0.5 indicates a complete lack of diagnostic value. Test-retest and inter-rater reliability of the two alternate versions were indexed by the intra-class correlation [ICC]. Internal consistency of items in the alternate versions was measured by the Cronbach's α. Statistical analysis was performed in R statistical software or IBM SPSS Statistics Macintosh version 24. Figure illustrations were created using Microsoft PowerPoint version 15.30. Sample size was estimated according to the data from our previous study on HK-MoCA validation conducted in healthy controls and patients with cerebral small vessel disease. [11] A group size of 26 will enable a large effect size [32] measured by Cohen's d of 0.8 with 80% power and α = 0.05 to detect a significant group difference on the HK-MoCA total score. Therefore, a final sample size of 60 was therefore sufficient for this study.

Results
The demographic characteristics and cognitive performance of the participants are shown in Table 3. All participants were able to communicate in the Cantonese language at a level that is adequate for the test results to be considered valid. No group difference was found in terms of age and sex. The Mild NCD group had significantly fewer years of education when compared to healthy controls.

Discussion
In this study, we developed two alternate versions for the HK-MoCA. Both versions demonstrated good concurrent and criteria validity, test-retest and inter-rater reliability as well as  This is also the first study to examine the validity of the MoCA in screening of Mild NCD. We showed that the original and alternate versions are all sensitive to Mild NCD. It is notable that the AUC observed for the HK-MoCA-A1 (0.922) was slightly higher than that of HK-Mo-CA-O (0.839), indicating that the HK-MoCA-A1 has better discriminating ability than HK-MoCA-A2. Given that the classification of Mild NCD was made partially on the basis of performance on the HK-MoCA-O, the difference in criterion validity between the two alternate versions is possibly explained by the higher correlation with the HK-MoCA-O observed for the HK-MoCA-A1 than the HK-MoCA-A2. Therefore, despite that both alternative versions are valid and reliable, preference is given to HK-MoCA-A1 in view of the better psychometric properties of this version over HK-MoCA-A2.
There are study limitations. First, the sample size was relatively small. However, this sample size was estimated a priori based on previous data of the HK-MoCA and it is shown to be sufficient for the analyses performed. Second, participants was recruited based on convenience sampling and therefore the possibility of selection bias could not be excluded. Third, objective psychometric performance for the diagnosis of Mild NCD was measured using the 30-minute protocol of the NINDS-CSN VCI Harmonization battery. While this protocol covers executive functions and memory which are impaired early in the common cognitive disorders such as Alzheimer's and vascular diseases, participants with isolated impairments in other cognitive domains such as visuospatial functions and attention may have been misclassified as normal. Fourth, in this study we did not determine sensitivity and specificity for a specific cut-off score for the alternate versions. We have previously derived age-and education-adjusted normative values for the HK-MoCA-O from a large cohort of functionally independent old adults free of significant MRI abnormalities. [16] In view of the high correlation observed between the alternate and original versions, it is reasonable to apply the same set of normative data to the alternate versions for classification of performance rather than relying on a single cut-off score. Fifth, Mild NCD was diagnosed clinically. Without information on neuroimaging and biomarkers such as those obtained in in vivo amyloid imaging or cerebrospinal fluid analysis, the etiology of the Mild NCD patients was not determined. Finally, it is common for elderly people in Hong Kong to speak one or more Chinese dialects other than Cantonese which could have influenced how the administration instructions and items of the tests were processed. Although we did not include a detailed assessment of the dialects the participants spoke, all of them were able to understand and speak Cantonese at a level well enough so that the test results could be considered valid.
In conclusion, the two alternate versions of the HK-MoCA are useful for screening of Mild NCD. The availability of the alternate versions of the HK-MoCA is expected to improve accuracy in measuring cognitive changes for monitoring disease progression and rehabilitation and treatment efficacy in clinical practice and research studies. Future investigations may focus on using digital technology to capture subtle human behaviors such as voice, response latency and multidimensional movement data to improve sensitivity and accuracy as well as to reduce administration time of cognitive screening.
Supporting information S1 Dataset. Minimal data set. The minimal data set contains all the data pertinent to the analysis and results reported in this manuscript. (SAV)