https://orcid.org/0000-0002-9094-347X
Figures
Abstract
Background
Research collaboration highlight a need for validated tests in other languages than English. Translation and culture adjustments may threaten essential features of the original instrument.
Objective
To assess the internal consistency, inter-rater and test-retest reliability, and construct validity of the Norwegian version of the Edinburgh Cognitive and Behavioural Amyotrophic Lateral Sclerosis (ALS) Screen (ECAS-N).
Methods
Performance of 71 subjects with ALS, 85 healthy controls (HC) and 6 controls with Alzheimer’s disease (AD) were assessed with the ECAS-N. Test-retest interval was four months. Internal consistency was evaluated using Cronbach’s alpha; reliability was assessed using intraclass correlation coefficient (ICC), Cohen’s kappa, and Bland Altman plot. Five hypothesis, including the Montreal Cognitive Assessment (MoCA) screen, was evaluated for construct validity.
Results
ECAS-N total score produced a Cronbach’s alpha of 0.65, had excellent inter-rater reliability (ICC = 0.99) and acceptable test-retest reliability (ICC = 0.73). Construct validity analysis suggested valid use of the ECAS-N to distinguish people with ALS-specific cognitive impairment from HC (p = 0.001) and those with AD (p = 0.002). The MoCA and ECAS-N were moderately correlated (r = 0.53).
Citation: Taule T, Eide IS, Fjær L, Myrberget M-A, Oseland MS, Renså MA, et al. (2023) Norwegian version of the Edinburgh cognitive and behavioural ALS screen: Construct validity, internal consistency, inter-rater, and test-retest reliability. PLoS ONE 18(5): e0285307. https://doi.org/10.1371/journal.pone.0285307
Editor: Fares Alahdab, Houston Methodist Academic Institute, UNITED STATES
Received: March 1, 2022; Accepted: April 19, 2023; Published: May 4, 2023
Copyright: © 2023 Taule et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data cannot be shared publicly because of legal and ethical reasons. Legally, our study involve human research participant data, which contain potentially identifying sensitive information about patients’ cognitive function. This is because ALS is a relatively rare diagnosis and some of the hospitals identified in our dataset treats only a few patients a year. Publishing our study’s minimal underlying data set would mean that we cannot guarantee participants anonymity and is therefore unacceptable. Ethically, all participants were informed that only anonymous data would be published and this is what the Data Privacy Unit at Haukeland University Hospital, Bergen, Norway took into consideration when the study was accepted. Data are available from the Data Privacy Unit at Haukeland University Hospital, Bergen, Norway (reference numbers 2016/3166) (contact via Health Bergen HF, Orthopaedic clinic, by clinic director Kjell Matre or the corresponding author of this manuscript) for researchers who meet the criteria for access to confidential data.
Funding: TT received grants from the Western Norwegian Regional Health Authority (grant number: 912158, 2017). Available from: https://helse-vest.no/en. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
1. Introduction
Amyotrophic lateral sclerosis (ALS) is a neurodegenerative multisystem disease that may entail cognitive and/or behavioral changes in addition to deleterious changes in motor function [1, 2]. The presence of cognitive and behavioral symptoms in ALS can precede motor symptoms [3] and may increase caregiver burden [4]; shorten lifespan [5, 6]; and lower adherence to treatment recommendations [7]. ALS is the third most common neurodegenerative disease, an adult onset disorder, and is characterized by rapid progression [8].
As a first step to detect declines and manage ALS challenges, early and repeated screening of possible cognitive and behavioral changes is recommended for ALS-specific healthcare [9]. During the decade, the Edinburgh Cognitive and Behavioural ALS Screen (ECAS) has proven to be an appropriate and reliable test, as it can measure the heterogeneity of cognitive and behavioral changes in ALS, and in addition takes into account substantial motor difficulties that are common in ALS [9–12]. The ECAS has been translated and adapted for clinical and research use in Asia [13, 14]; Europe [15–23]; including Norway [24].
As with all translated assessment instruments, translation of the ECAS depends on accounting for language and culture peculiarities of the target language. Failure to recognize this important factor may threaten the equivalence between the original and translated version [25]. Confidently evaluating patient results also depends on a valid translation [26]. What specific parameters are important? A cultural understanding of hypothetical concepts may differ among different demographic groups, and the meaning attached to each translated item may diverge across test subjects or administrators of different groups. In addition, test conditions and population characteristics may vary among countries. For each translated scale, trustworthiness must be established that the individual items of the translated test perform in the same way as the original test (internal consistency). In a validated test, one must also establish that test scores are not affected by who administers the test (inter-rater reliability). Additionally, for patients in a stable state, their scores must also be stable over time (test-retest reliability). Knowledge about how a test performs in a variety of situations (construct validity) will help expand the possibilities of test score interpretations [26].
As measurement properties can vary and are not a fixed feature, our study aimed to assess the reliability and construct validity of the version of the ECAS translated into Norwegian (ECAS-N). The study addressed the following research questions in patients with ALS: (a) What is the internal consistency of items in the ECAS-N? (b) What is the inter-rater reliability of the ECAS-N when administered by different testers? (c) What is the test-retest reliability of the ECAS-N for a test-retest interval of 4 months? (d) What is the construct validity of the ECAS-N when compared against the Norwegian version of the Montreal Cognitive Assessment (MoCA), and the ECAS-N results of healthy controls (HCs) and persons with Alzheimer’s disease (AD)? The ECAS-N and the MoCA are both tests of cognition, assessing similar, but not the same cognitive domains [27, 28]. While the ECAS is specifically designed for people with ALS, the MoCA is a generic test of cognition [27, 28], meaning the two tests are expected to have different sensitivity to ALS-specific cognitive problems.
For construct validity, we hypothesized that: (H1) ECAS-N and the MoCA scores for persons with ALS would be moderately positively correlated; (H2) ECAS-N scores of subjects with ALS and HCs would show a strong negative association; (H3) ALS-specific scores for subjects with ALS would be higher compared to subjects with AD; and (H4) ALS non-specific scores for subjects with AD would be higher compared to subjects with ALS; (H5) ECAS-N scores of subjects with ALS and subjects with AD would be moderately positively associated.
2. Material and methods
To ensure transparent reporting, we report our studies according to the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) [29].
2.1. Design
A validation design [30] was used to determine selected psychometric properties of the ECAS-N. This design comprise specific scientific steps to estimate the sample size, distinct statistical analyses to assess the degree to which the scores of the ECAS-N are free from measurement error (reliability) and assesses the degree to which the ECAS-N truly measure the psychological construct it purports to measure (validity) [30].
2.2. Protocol and registration
Before any study procedures were carried out with subjects, the protocol was published [31], and the study was registered at ClinicalTrials.gov (registration number NCT03579017; available at https://clinicaltrials.gov/ct2/show/NCT03579017).
2.3. Participants
Inclusion criteria for ALS subjects, HCs and controls with AD are listed in Table 1.
2.3.1. Persons with ALS.
Patients with ALS were eligible to participate if they were followed up at an ALS unit at Haukeland University Hospital (HUH), St.Olavs Hospital, Hospital of Southern Norway or Namsos Hospital within four months after being diagnosed with ALS. A member of the ALS-specific healthcare team recruited study participants between May 2017 and June 2021, and in conjunction with patients’ first visit to the hospital. Each ALS participant also chose one carer to assist them in the study.
2.3.2. Control subjects.
Testers recruited healthy volunteers from various places of employment and social organizations. Persons with AD were recruited from Haraldsplass Deaconess Hospital, a hospital with a research group focusing on AD and other types of dementia. AD patients were diagnosed according to standard guidelines used in Europe [32]. To ensure that all participants themselves were able to make an informed decision about study participation, we included only patients with mild-moderate AD. All volunteers who fulfilled the inclusion criteria (see Table 1) and agreed to participate were enrolled. We aimed to closely match the control subjects’ age and sex to those in the ALS group. Control subjects were between the ages of 35 and 85 years. We strived to have approximately equal distribution of the number of subjects for each group below and above 60 years old. Slightly more male control subjects were selected to match the sex distribution of the ALS group. The numbers of subjects for each group with lower or higher levels of education were also approximately equal.
2.4. Data collection
We used a custom-designed questionnaire to collect subjects’ age, gender and level of education. Patients were assessed with the ECAS-N using country-specific cutoff scores [24], and the Norwegian version of MoCA (version 7.1; available at www.mocatest.org).
2.4.1. The ECAS.
The ECAS is a brief, multi-domain screening test designed to detect and document ALS patients’ cognitive impairment (ECAS-cognitive screen) and behavioral changes (ECAS-behavioral screen) [27]. The cognitive screen comprises an ALS-specific subscore (verbal fluency, language and executive functions including social cognition); an ALS-nonspecific subscore (memory and visuospatial functions); and a total score. The total ECAS-cognitive screen score ranges from 0–136, the ALS-specific subscore from 0–100, and the ALS-nonspecific subscore from 0–36. For all cognitive scales lower scores indicate greater deficits. The ECAS-behavioral screen is a carer interview designed to assess and document patient performance changes in five domains of behavior and three domains of psychosis. Behavioral scores range from 0–10, and scores for psychosis changes range from 0–3. For behavioural and psychosis scale, higher scores indicate greater deficits. Systematic reviews have shown that the ECAS is an effective, valid and reliable screening test of cognitive changes in patients with ALS [11, 33].
ECAS performance can be assessed either by obtaining spoken responses or written responses from the subjects in order to accommodate physical disabilities that often accompany ALS [27]. The ECAS was translated into Norwegian language and culture by use of the scientific procedure described by Sousa et al. [34].
2.4.2. The MoCA.
The MoCA is a brief screening tool used to assess impairment in frontal lobe function and mild cognitive impairment. It assesses performance in the following domains: visuospatial/executive, naming, memory, attention, language, abstraction, delayed recall, and orientation [28]. The MoCA has been validated in patients with ALS [35]. The maximum possible score is 30, with scores below 26 indicative of cognitive impairment. Performance of subjects with 12 year of education or less is compensated by adding 1 point to the subject’s total MoCA score [28].
2.4.3. Testing procedures.
Assessment were conducted between May 2017 and October 2021. For patients with ALS and their carers, baseline performance on the ECAS-N and MoCA tests was obtained within four months of ALS diagnosis, and the retests were done within eight months after being diagnosed. The time interval between baseline and retest administration was considered long enough to minimize recall bias but short enough to reduce the possibility of study withdrawal due to symptom worsening. Previous research also provided evidence that the constructs measured by the tests are stable over our study period [36, 37].
The ECAS-N cognitive screen was administered either in the subjects’ homes, in an outpatient clinic, or during hospitalization. The ECAS-N behavioral screen was administered either through a face-to-face interview or by asking the questions over the phone. Decisions about the testing modality depended on established routines for each hospital, the wish of the test subjects, and recommendations given by local authorities regarding the Covid-19 pandemic. Eight individual testers, all of them specially trained for administering the ECAS-N, administered and scored the tests. All participants were given the same instructions, following ECAS guidelines [38]. Baseline tests and retest were conducted by the same tester.
Procedures for assessing inter-rater reliability on the ECAS-N involved two testers independently scoring the participants at the same time. Due to the extra cost and effort of this reliability assessment, only subjects from one hospital contributed data to it. Four of the testers were involved in these scorings.
2.5. Sample size estimation
For evaluation of the measurement properties relevant in our study, Terwee et al. recommend a minimum of 50 participants [39].
2.6. Statistical analyses
Data were analysed using SPSS version 26.0 [IBM 40] and Matlab version 9.0 [41]. Descriptive statistics were used to summarize group demographic variables and selected baseline characteristics for subjects with ALS and control subjects.
2.6.1. Internal consistency.
We used the Cronbach’s alpha statistic to estimate the internal consistency among the subscales and the internal consistency for the total scale of the ECAS-N. A Cronbach’s alpha coefficient of 0.70 is a commonly considered to be good internal consistency [39].
2.6.2. Inter-rater reliability and agreement.
For continuous variables, inter-rater reliability was assessed with the intraclass correlation coefficient (ICC), which reflects both the degree of correlation and agreement between measures [42]. The ICCs and their 95% confident intervals (CIs) were estimated by using a two-way random-effects analysis of variance model with the interaction term for assessing the absolute-agreement between single raters. We used this two-way random-effects model, since the ECAS is designed for routine clinical use by any therapist trained as a selected rater in this reliability assessment [43]. Single scores reflect that a single rater usually completed the ECAS assessment and it was chosen to avoid calculating a misleading high ICC. As the patients’ cognitive status was to be judged according to ECAS-N cutoff scores, absolute agreement was needed.
There is no consensus on how to interpret the magnitude of the ICC. We considered ICC values equal to 0.70 or better to be acceptable [39]. Additionally, we used the 95% CI of the estimated ICC as the basis for ICC interpretation [39, 43]. For dichotomous variables of the ECAS-N (cognitive screen), we used Cohen’s kappa statistic [44] to determine if there was agreement between the two testers regarding judgement of patients as being cognitive impaired or not. Kappa values greater than 0.60 was considered to be acceptable [25].
Inter-rater agreement between two independent testers for total ECAS-N, ALS-specific, and ALS-nonspecific scores was estimated with the Bland Altman method [45, 46]. This analysis is a straightforward way to evaluate if scores derivate from each other–in our case, two independent testers’ scoring of participants–and whether there are systematic differences. Bland Altman procedure present a plot in which systematic errors can easily be seen [45, 46]. For each participant, we calculated the absolute difference between the ECAS-N scores (separately for total, ALS-specific, and ALS-nonspecific scores) obtained by tester one and tester two (y-axis), and plotted this difference against the mean difference of the two scores (x-axis). The upper and lower limits of agreement were calculated using the mean and the SD of the differences between scores of tester one and tester two. For normally distributed data, the scatter should be evenly dispersed along the x-axis, and the mean difference should be close to zero [45, 46]. With this method, 95% of the data points should be within ±2SD of the mean difference [47]. Pre-established acceptable limits of agreement were 7 for total score, 5 for ALS-specific score, and 2 for ALS-nonspecific score.
2.6.3. Test-retest reliability and agreement.
For continuous variables, also test-retest reliability was evaluated by calculating the ICC [42]. A two-way mixed-effects analysis of variance model with the interaction for assessing the absolute agreement between single scores is often taken to be the most appropriate for test-retest studies [43, 48]. By using a two-way model we acknowledge that the sequence of testing time was an important factor and that the two assessment times were not interchangeable. Using a mixed-effects model, we considered a fixed effect of time, as it was pre-specified and identical across all study subjects. As for inter-rater reliability (see 2.6.2), single scores was used because the ECAS-N is usually administered by one rater. We decided to use absolute agreement, because we assumed that the construct of interest (cognitive function) was stable for the subjects across the two assessment times.
Again we used the 95% CI of the estimated ICC as the basis for ICC interpretation; ICC values greater than 0.70 were deemed acceptable [39]. For dichotomous variables in the ECAS-N, we used Cohens’s kappa to determine if there were meaningful differences in agreement between baseline and follow-up judgements of patients being cognitive impaired or not. Kappa values greater than 0.60 were considered to be acceptable [25].
Test-retest agreement was further assessed by the Bland-Altman method [45, 46]. For each participant, we calculated the absolute differences between the ECAS-N scores (separately for total, ALS-specific, and ALS non-specific scores) obtained at baseline and retest (y-axis), and plotted this difference against the mean difference of the two scores (x-axis). Limits of agreement were calculated using the mean and the standard deviation (SD) of the differences between baseline and retest scores. As for the inter-rater agreement assessment (see 2.6.2), we used the same pre-established acceptable limits of agreement.
2.6.4. Construct validity.
To assess construct validity of the ECAS-N, we used subjects’ baseline ECAS-N scores and their MoCA scores. Five a priori hypotheses (H1-H5) were evaluated. Pearson product-moment correlation coefficient were computed to evaluate hypothesis 1. A t-test for independent groups and chi-square test for independence was used to evaluate hypothesis 2–5. P-values less than 0.05 were considered significant.
2.7. Ethical considerations
This study was evaluated by the Regional Ethical Research Committee (reference number 2015/1221/REK vest) and approved by the Data Privacy Unit at HUH, Bergen, Norway (reference numbers 2016/3166). All the participants and their carers signed a written informed consent form in compliance with the revised Declaration of Helsinki of 1987 [49].
3. Results
A flow diagram showing participant selection, inclusion/exclusion criteria, and testing procedure summaries are presented in Fig 1.
Abbreviations: AD, Alzheimer’s disease; ALS, amyotrophic lateral sclerosis; ECAS-N, Edinburgh Cognitive and Behavioural ALS screen-translated Norwegian version; H1, hypothesis 1; H2, hypothesis 2; H3, hypothesis 3; H4, hypothesis 4; H5, hypothesis 5; HC, healthy control subjects; MoCA, Montreal Cognitive Assessment.
3.1. Subject demographics
Characteristics of 85 HCs, 6 controls with AD, and subjects with ALS are presented in Table 2.
Of a total sample of 71 subjects with ALS, 31showed evidence of behavioral changes as rated by their carer in at least one of the five behavioral domains. Of these, 21 subjects met the Strong criteria for ALS with behavioral impairment (ALSbi) [9], and for 19 of these, the impairment was based solely on identified apathy. Eight subjects who met the criteria of ALSbi also scored below the cutoffs of the ECAS-N total score. Five of these scored below the cutoffs of the ALS-specific score. Five ALS subjects exhibited psychotic symptoms, as reported by their carers. Two of the patients may have met the diagnosis of amyotrophic lateral sclerosis-frontotemporal dementia (ALS-FTD), with the presence of behavioral and cognitive impairment together with psychotic symptoms [9].
3.2. Reliability
Table 3 shows results for the reliability assessments, specifically internal consistency, inter-rater reliability, and test-retest reliability for the ECAS-N total score and subscale scores for subjects with ALS.
A descriptive visualization of reliability of the ECAS-N is provided in Fig 2.
Additional information about reliability and agreement for each specific cognitive domain of the ECAS-N is in the Supplemental files (insert link to S1 Fig and S1 Table here).
3.3. Internal consistency
For internal consistency of the ECAS-N using Cronbach’s alpha coefficient (α), we found acceptable values for the ECAS-N behavioral score (α = 0.72). Internal consistency Cronbach alpha values across the other subscales of ECAS-N ranged from α = 0.33 to α = 0.65.
3.4. Inter-rater and test-retest reliability
Continuous variable ICCs for interrater reliability ranged from 0.89 to 1.00. Kappa coefficients for categorical variables were above 0.85 for all cutoff scores. Analysis of test-retest reliability using ICCs for continuous variables showed values greater than 0.7 for the ECAS-N total score, ALS-specific score and behavioral score. The ICC was 0.61 for ALS- nonspecific scores. The Kappa coefficient for categorical variables ranged from 0.17 to 0.55.
3.5. Inter-rater and test-retest agreement (Bland-Altman)
The Bland-Altman results for inter-rater and test-retest agreement are presented in Fig 3.
Bland-Altman results for inter-rater agreement of the ECAS-N (top row), and Bland-Altman plot for ECAS-N scores at test time 1 and time 2 (bottom row). The middle solid line represents the mean difference between ECAS-N scores at time 1 and time 2. The lower and upper dashed lines represents the upper and lower 95% confidence limits, respectively.
The mean difference in ECAS-N total scores between rater 1 and rater 2 was -0.07 (95% CI: -0.71, 0.58); for the ALS-specific score, 0.26 (95% CI: -0.33, 0.85); and for the ALS-nonspecific score, -0.32 (95% CI: -0.67, 0.02). These mean differences were not significantly different from zero for the ECAS-N total score (p = 0.84, SD = 1.75); ALS-specific score (p = 0.38, SD = 1.61); and ALS-nonspecific score (p = 0.07, SD = 0.95). These outcomes indicate that, on average, the ratings of rater 1 and rater 2 were statistically indistinguishable on the ECAS-N. The 95% CIs for agreement were within the predefined acceptable limits (Fig 3). A visual inspection showed that the between rater differences plotted against the mean differences of the inter-rater values was close to zero, and few scores were beyond the upper or lower limits of agreement.
The mean difference between baseline and retest for the ECAS-N total score was
-2.24 (95% CI: -4.70, 0.21); for ALS specific score, it was -1.83 (95% CI: -3.80, 0.13); and for ALS-nonspecific score, it was -0.41 (95% CI: -1.42, 0.60). These mean differences were not significantly different from zero for the total score (p = 0.73, SD = 9.99); ALS-specific score (p = 0.07, SD = 8.00); or the ALS-nonspecific score (p = 0.42, SD = 4.11). This outcome indicates that, on average, the ECAS-N produced the same scores at baseline and retest. For all scores, the 95% agreement interval was beyond the acceptable limits defined a priori (Fig 3). Visual inspection indicated that the bias was not severe, because the test-retest differences plotted against the mean differences of the test-retest values was close to zero, and few scores were beyond the upper or lower limits of agreement.
3.6. Construct validity (hypothesis testing for H1-H5)
Of the 73 included patients, 2 were unable to complete the ECAS-N and 9 could not complete the MoCA.
In subjects with ALS, we found a moderate positive correlation (r = 0.53) between ECAS-N and MoCA scores, supporting H1. In the testing of H2, we found that ECAS-N total scores, ALS-specific score, ALS-nonspecific score and cutoff scores in subjects with ALS were significantly worse than HCs (see Table 4), supporting H2.
In the testing of H3 and H4, we found that ECAS-N ALS-specific score, ALS-nonspecific score and cutoff scores in subjects with ALS were significantly better than those of controls with AD (Table 4), supporting H3 and rejecting H4. In testing of H5, we found that ECAS-N total scores, ALS-specific score, ALS-nonspecific score and cutoff scores in subjects with ALS were significantly better than those controls with AD (Table 4), supporting H5. The association were however, higher than expected, as we initially expected that the ALS-specific score and ALS-nonspecific score would diverge in different directions. Fig 4 presents a visualization of the results of tests of H2-H5.
4. Discussion
Mortality and morbidity related to ALS is becoming more of a public health issue within the Norwegian aging population [50]. Thus, there is a great need to have a valid, trustworthy Norwegian language version of the ECAS (ECAS-N). With this multicentre study, we filled this gap and presented the first results on psychometrics of the ECAS-N. We found that the internal consistency of the ECAS-N differs from that reported for the original ECAS. ECAS-N had excellent inter-rater reliability. The test-retest reliability was acceptable for ECAS-N total score, ALS specific-score and behavioural score, however, Bland Altman plot indicated a potential bias. Evaluation of construct validity suggested that the ECAS-N and MoCA assess similar, but not the same construct, and that the ECAS-N could be used to distinguish people with ALS-specific cognitive impairment from HCs and those with AD. Discussion of several findings from our study will aid clinicians and researchers who want to routinely use the ECAS-N.
Of a total sample of 71 ALS subjects in our study -none of whom had comorbidities of importance related to cognitive function-24% scored below the cutoff on the ALS-specific ECAS-N. According to revised consensus criteria of Strong et al., these subjects are ALS patients with cognitive impairment [9]. This 24% prevalence rate is slightly below the rate of abnormal performance in ALS previously reported when using the original ECAS (29%) and the 30% rate derived from a meta-analysis of studies that used different but commonly used neuropsychological tests for diagnosis [1, 27]. These differences may be explained by individual differences in the studies, or different cutoff values used in different versions of the ECAS. Based on 69 interviews with carers, we found 30% of our sample met the Strong criteria of ALSbi [9]. This is close to the median prevalence (25%) of mild-to-moderate behavioral changes reported in a systematic review [2].
Our finding of apathy as the most frequently reported behavioral change in our sample of ALS patients is in line with what Abrahams et al. reported for the original ECAS [27], while Raaphorst et al. found the most frequently reported behavioral change was perseveration, followed by apathy [2]. Our findings indicated that 3% of the ALS subjects met the criteria of ALS-FTD, a result slightly below the prevalence reported in a systematic review (8%) and that found when using the original ECAS (6%) [2, 27].
4.1. Internal consistency
Our analysis of internal consistency of the ECAS-N produced a Cronbach’s alpha of 0.72 for the behavioral scale and 0.65 for the total ECAS scale. These results point to the importance of considering both scores instead of one or the other in isolation. The result for total score is slightly below the common acceptable alpha value (α = 0.70) [51].
Interpretation of Cronbach’s alpha is not straightforward. At worst, our finding of a Cronbach’s α = 0.65 may temper use of the ECAS-N cognitive screen at this time. Alternatively, the screen might be a measure of two different aspects of cognition (ALS-specific and ALS-nonspecific), which would mean that the internal consistency analysis of total score is not relevant [26]. The alpha of the original total ECAS scale was α = 0.75, and it was indicated that the sub-scores of the original ECAS correlate well with the total score [27]. However, information about the scales unidimensionality, a prerequisite for a clear interpretation of the internal consistency statistics [26, 51], was not provided.
We found a higher internal consistency Cronbach’s alpha for the ALS-specific score (α = 0.62) than for ALS-nonspecific score (α = 0.33), meaning that the ALS-specific score dominated the ECAS-N total score. The high levels of content validity previously reported for all items of the ECAS-N [24] support the notion that the ECAS-N validly taps into underlying constructs of cognition and behavior among respondents, allowing its use in clinical and research settings.
4.2. Inter-rater reliability
The ECAS-N exhibited excellent inter-rater reliability. This is an important outcome because it allows various testers trained in the use of ECAS to administer the test and produce comparable results. Ideally, we suggest testers obtain certification through the online certification system for ECAS training (https://ecas.psy.ed.ac.uk/training/).
4.3. Test-retest reliability
The results of our test-retest analysis of the ECAS-N is relevant only for a test-retest interval of four months, as that is the period we assessed. The observed differences revealed by Bland-Altman plots indicated a bias between the mean differences. However, the trend was not clear, and it is difficult to interpret whether the observed differences in test performance was due to a change in the patients’ illness or problems with the instrument itself. Based on available research, we assumed recall bias would probably not occur in patients with ALS [36, 37]. However, a practice effect has been demonstrated for the original ECAS in healthy control persons at a six-month test-retest interval [36]. This suggests ALS patients in our study who have no cognitive impairment may have improved on the second administration of the ECAS-N due to their experience with the same test version four months earlier. In HCs, using alternate forms of the ECAS seems to mitigate a practice effect in learning test content or adopting test-taking strategies [52]. This means that translation of alternate forms of the ECAS into Norwegian is warranted.
We relied on research suggesting cognitive changes emerge before motor symptoms and remain stable after an initial decline [53]. Our results, however, may support those who claim that cognitive deficits become more prominent over time, especially early in the course of disease [54].
4.4. Construct validity
We found a moderate positive (r = 0.53) correlation between the scores on the ECAS-N and the MoCA for persons with ALS. This is not surprising when comparing an ALS-specific test with an ALS-nonspecific test, and this has been observed for other translations of the ECAS [17, 21]. Identifying more subjects classified as impaired by the MoCA than by the ECAS-N may simply be because the MoCA is a more sensitive test for detecting mild cognitive impairment [55].
Further analysis of construct validity in the present study suggested that the ECAS-N may be used to validly distinguish persons with ALS-specific cognitive impairment from those with no cognitive impairment and those who have cognitive impairment due to AD. However, we included a small number of subjects with AD in our analysis, which limits what conclusions can be drawn.
4.5. Study strength and limitations
Our study had several strengths. Designing a multicenter study involving most of the large hospitals in Norway increases the probability that our sample was reasonably representative of Norwegian patients with ALS. Another strength is that we used experienced and highly trained testers, many of whom were officially certified for administering the ECAS. Employing the expertise of a biostatistician to guide our selection of appropriate statistical methods to use also was a strength. We have been transparent about our methodological choices, meaning others can easily evaluate the methodological quality of our study.
Readers must be aware that the sample size were not statistically estimated a-priory, but relied on a fixed analysis and recommendations in the litterature [39]. Although this is a limitation of the study, a post-hoc power analysis is meaningless. Significant p-values, indicate that we had enough power to detect differences in our hypothesis testing. However, a possible bias might be present related to a limited sample size in some of the other analysis; smaller sample size typically mean low statistical power. The baseline characteristics (Table 2), were however, well balanced between the ALS subjects included in the different analysis. In an attempt to prevent recall bias, we chose a four-month time interval between test and retest. Based on previous research, we assumed that the subjects with ALS were cognitively stable in the interim period [36]; however, we cannot be certain. Information about subjects’ respiratory support, stress level, and lack of sleep might have made it easier to interpret results of the ECAS-N. In the real-world setting, it was not reasonable to try to completely control all subjects and environmental conditions when we tested subjects. However, this factor is important for testers to be cognizant of, as the effect of the environment on ECAS-N results is not fully understood.
5. Conclusions
It is critical to rigorously evaluate the psychometric soundness of translated neurobehavioral and cognitive screening tests, since the translation process itself and adjustments for culture differences in the target language may threaten essential features of the original validated instrument. Internal consistency of the ECAS-N differs from that reported for the original ECAS. However, high levels of content validity still give reasons to argue that the two versions’ equivalency is maintained. The ECAS-N seems not to be affected by who administers the test, permitting its valid use by various clinical practitioners. Trustworthiness that the ECAS-N scores are stable between four and eight months is fragile, and results from repeated assessments must be interpreted with care. Further evidence to support the construct validity of the ECAS has been generated. It seems likely that the ECAS-N and MoCA measure similar, but not the same construct. The ECAS-N seems to distinguish people with ALS-specific cognitive impairment from people with no cognitive impairment and those who have cognitive impairment due to AD.
6. Considerations for future studies
Further research will contribute to a more robust documentation of the psychometric properties of the ECAS-N. In future studies, the authors should consider conducting a-priori power analysis to determine an appropriate sample size. Investigating the test-retest reliability at various intervals will provide more information about the ideal interval for assessing cognitive ability in patients with ALS. Gathering information about subjects’ respiratory support, stress level, and sleep could help in interpreting the results and controlling for potential confounding factors. To draw more reliable conclusions about the ECAS-N’s ability to distinguish between ALS patients and those with Alzheimer’s Disease (AD), the sample size of subjects with AD should be appropriate. To mitigate potential practice effects there is a need to translate alternate forms of the ECAS into Norwegian and assessing their psychometric properties. Within the field of ALS there is also a need to better understand how environmental factors may affect test’s performance, such there is also a need to examine the influence of different environmental factors on the ECAS-N results.
Supporting information
S1 Fig.
Bland-Altman results for inter-rater agreement of each domain of the ECAS-N (top row), and Bland-Altman plot for each domain of the ECAS-N scores at test time 1 and time 2 (bottom row). The middle solid line represents the mean difference between ECAS-N scores at time 1 and time 2. The lower and upper dashed lines represents the upper and lower 95% confidence limits, respectively.
https://doi.org/10.1371/journal.pone.0285307.s001
(TIF)
S1 Table. Internal consistency, inter-rater, and test-retest reliability of the ECAS-N cognitive screen in subjects with ALS.
https://doi.org/10.1371/journal.pone.0285307.s002
(DOCX)
Acknowledgments
We thank all the subjects and their carers, who gave their time for this study. Special thanks to Annbjørg Spilde Morland, OT-specialist, who planned and initiated the study. Her commitment and professional insight in the field of ALS made her an important discussion partner throughout the research process, which was crucial to completing this work. Thanks also to Thomas Guldvik, OT, and Synnøve Glesnes, OT-specialist, involved in planning the project and collecting the ECAS-N data and for their efforts to ensure high quality work. The Department of Occupational Therapy and the Department of Neurology at Haukeland University Hospital facilitated implementation of this project.
References
- 1. Beeldman E, Raaphorst J, Twennaar M, deVisser M, Schmand B, de Haan R. The cognitive profile of ALS: a systematic review and meta-analysis update. J Neurol Neurosurg Psychiatry. 2015;87(6):611–9. pmid:26283685
- 2. Raaphorst J, Beeldman E, De Visser M, De Haan R, Schmand B. A systematic review of behavioural changes in motor neuron disease. Amyotrophic lateral sclerosis: official publication of the World Federation of Neurology Research Group on Motor Neuron Diseases. 2012;13(6):493–501. pmid:22424127
- 3. Mioshi E, Caga J, Lillo P, Hsieh S, Ramsey E, Devenney E, et al. Neuropsychiatric changes precede classic motor symptoms in ALS and do not affect survival. Neurology. 2014;82(2):149–55. pmid:24336140
- 4. Pagnini F, Rossi G, Lunetta C, Banfi P, Castelnuovo G, Corbo M, et al. Burden, depression, and anxiety in caregivers of people with amyotrophic lateral sclerosis. Psychol Health Med. 2010;15(6):685–93. pmid:21154021
- 5. Olney R, Murphy J, Forwhew D, Garwood E, Miller B, Langmore S, et al. The effects of executive and behavioral dysfunction on the course of ALS. Neurology. 2005;65:1774–7. pmid:16344521
- 6. Elamin M, Phukan J, Bede P, Jordan N, Byrne S, Pender N, et al. Executive dysfunction is a negative prognostic indicator in patients with ALS without dementia. Neurology. 2011;76:1263–69. pmid:21464431
- 7. Khin E, Minor D, Holloway A, Pelleg A. Decisional capacity in amyotrophic lateral sclerosis. J Am Acad Psychiatry Law. 2015;43:210–7. pmid:26071511.
- 8. Renton A, Chio A, Traynor B. State of play in amyotrophic lateral sclerosis genetics. Nat Neurosci. 2014;17(1):17–23. pmid:24369373
- 9. Strong M, Abrahams S, Goldstein L, Woolley S, Mclauglin P, Snowden J, et al. Amyotrophic lateral sclerosis—frontotemporal spectrum disorder (ALS-FTSD): Revised diagnostic criteria. Amyotroph Lateral Scler and Frontotemporal Degener. 2017;18:153–74. pmid:28054827
- 10. Taule T, Søvik M, Lein R, Wehling E, Aßmus J, Rekand T. Psychometric properties of cognitive assessment in amyotrophic lateral sclerosis: A systematic review. Patient Relat Outcome Meas. 2020;11:181–94. pmid:33061727
- 11. Simon N, Goldstein L. Screening for cognitive and behavioral change in amyotrophic lateral sclerosis/motor neuron disease: a systematic review of validated screening methods. Amyotroph Lateral Scler and Frontotemporal Degener. 2018;20:1–11. pmid:30585510
- 12. Gosselt IK, Nijboer TCW, Van Es MA. An overview of screening instruments for cognition and behavior in patients with ALS: selecting the appropriate tool for clinical practice. Amyotrophic Lateral Sclerosis and Frontotemporal Degeneration. 2020;21(5–6):324–36. pmid:32157912
- 13. Ye S, Li C, Liu X, Fan D. The Edinburgh cognitive and behavioural ALS screen in a Chinese amyotrophic lateral sclerosis population. PlosOne. 2016;11(5). pmid:27195772
- 14. Watanabe Y, Ogino M, Ichikawa H, Hanajima R, Nakashima K. The Edinburgh Cognitive and Behavioural ALS Screen (ECAS) for Japanese ALS and FTD patients. Amyotrophic lateral sclerosis & frontotemporal degeneration. 2021;1–2:66–72. pmid:32757854
- 15. Bezdíček O, Marková L, Soósová N, Forgáč M. Czech version of the Edinburgh cognitive and behavioral amyotrophic lateral sclerosis screen–a pilot study. Cesk Slov Neurol N. 2018;81(2):208–12.
- 16. Kourtesis P, Christidi F, Margioti E, Demenega C, Renzos M, Evdokimidis I, et al. The Edinburgh cognitive and behavioral amyotrophic lateral sclerosis screen (ECAS): sensitivity in differentiating between ALS and Alzheimer’s disease in a Greek population. Amyotroph Lateral Scler and Frontotemporal Degener. 2019;30:1–8. pmid:31469297
- 17. Lulé D, Burkhardt C, Abdulla S, Böhm S, Kollewe K, Uttner I, et al. The Edinburgh cognitive and behavioural amyotrophic lateral sclerosis screen: a cross-sectional comparison of established screening tools in a German-Swiss population. Amyotroph Lateral Scler Frontotemporal Degener. 2015;16(1–2):16–23. pmid:25292386
- 18. Loose M, Burkhardt C, Aho-Özhan H, Keller J, Abdulla S, Bôhm S, et al. Age and education-matched cut-off scores for the revised German/Swiss-German version of ECAS. Amyotroph Lateral Scler and Frontotemporal Degener. 2016;17(5–6). pmid:27027323
- 19. Mora J, Salas T, Fernández M, Rodriguez-Castillo V, Chaverri D, Rodriguez-Santos F. Spanish adaptation of the Edinburgh cognitive and behavioral amyotrophic lateral sclerosis screen (ECAS). Amyotroph Lateral Scler and Frontotemporal Degener. 2018;19(1–2):74–9. pmid:29212378
- 20. Poletti B, Solca F, Carelli L, Faini A, Madotto F, Lafronza A, et al. Cognitive-behavioral longitudinal assessment in ALS: the Italian Edinburgh cognitive and behavioral ALS screen (ECAS). Amyotroph Lateral Scler and Frontotemporal Degener. 2018;19:387–95. pmid:29804470
- 21. Poletti B, Solca F, Carelli L, Madotto F, Lafronza A, Faini A, et al. The validation of the Italian Edinburgh cognitive and behavioural ALS screen (ECAS). Amyotroph Lateral Scler Frontotemporal Degener. 2016;17(7–8):489–98. pmid:27219526
- 22. Siciliano M, Trojano L, Trojsi F, Greco R, Santoro M, Basile G, et al. Edinburgh Cognitive and Behavioural ALS Screen (ECAS)-Italian version: regression based norms and equivalent scores. Neurol Sci. 2017;38:1059–68. pmid:28332040
- 23. Bakker LA, Schröder CD, Spreij LA, Verhaegen M, De Vocht J, Van Damme P, et al. Derivation of norms for the Dutch version of the Edinburgh cognitive and behavioral ALS screen. Amyotrophic Lateral Sclerosis and Frontotemporal Degeneration. 2019;20(1–2):19–27. pmid:30314420
- 24. Taule T, Morland A, Aßmus J, Tysnes O, Rekand T. Translation, cultural adaptation, and validation of a screening test for cognitive and behavioural changes in amyotrophic lateral sclerosis. Disabil Rehabil. 2021:1–9. pmid:34726988
- 25.
Streiner D, Norman G, Cairney J. Health measurement scales. A practical guide to their development and use. 5th ed: Oxford University Press; 2015. 399 p.
- 26. Mokkink L, Terwee C, Patrick D, Alonso J, Stratford P, Knol D, et al. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res. 2010;19(4):539–49. pmid:20169472
- 27. Abrahams S, Newton J, Niven E, Foley G, Bak T. Screening for cognition and behaviour changes in ALS. Amyotroph Lateral Scler and Frontotemporal Degener. 2014;15(1–2):9–14. pmid:23781974
- 28. Nasreddine ZS, Phillips NA, Bèdirian V, Charbonneau S, Whitehead V, Collin I, et al. The Montreal Cognitive Assessment, MoCA: a brief screening tool for mild cognitive impairment. J Am Geriatr Soc. 2005;53(4):695–9. pmid:15817019
- 29. von Elm E, Altman D, Egger M, Pocock S, Gætzsche P, Vandenbroucke J. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies. PloS Med. 2007;4(10):e296. pmid:17941714
- 30. Arafat Y. Validation study can be a separate study design. Int J Med Sci Public Health. 2015;5(11):1–2.
- 31. Taule T AS. M, Renså M, Aßmus J, Tysnes O, Rekand T. Edinburgh cognitive and behavioural amyotrophic lateral sclerosis screen (ECAS) in Norway: Protocol for validation and a prospective cohort study. Contemp Clin Trials Commun. 2019;14. pmid:30976725
- 32. Tolea M, Galvin J. Current guidelines for dementia screening: a shortcomings and recommended changes. Neurodegen Dis Manage. 2013;3(6):565–73. pmid:25152772
- 33. Taule T, Søvik M, Lein R, Wehling E, Aßmus J, Rekand T. Cognitive assessment tools used in ALS-specific health-care: A systematic review of their psychometric properties: PROSPERO: International prospective register of systematic reviews; 2017. Available from: http://www.crd.york.ac.uk/PROSPERO/display_record.php?ID=CRD42017055603.
- 34. Sousa V, Rojjanasrirat W. Translation, adaptation and validation of instruments or scales for use in cross-cultural health care research: a clear and user-friendly guideline. Journal of evaluation in clinical practice. 2011;17(2):268–74. pmid:20874835
- 35. Osborne RA, Sekhon R, Johnston W, Kalra S. Screening for frontal lobe and general cognitive impairment in patients with amyotrophic lateral sclerosis. Journal of the neurological sciences. 2014;336(1–2):191–6. pmid:24239183
- 36. Burkhardt C, Neuwirth C, Weber M. Longitudinal assessment of the Edinburgh cognitive and behavioural amyotrophic lateral sclerosis screen (ECAS): lack of practice effect in ALS patients? Amyotroph Lateral Scler Frontotemporal Degener. 2017;18(3–4):202–9. pmid:28165766
- 37. Burke T, Wynne B, O’Brien C, Elamin M, Bede P, Hardiman O, et al. Retrospective investigations of practice effects on repeated neuropsychological measures of executive functioning. The Irish Journal of Psychology. 2014;35(4):178–87.
- 38. Abrahams S, Bak T. Edinburgh Cognitive and Behavioural ALS Screen—ECAS English version 2013 Edinburgh Research Archive: The University of Edinburgh; 2013 [cited 2015 2015.04.28]. Available from: https://www.era.lib.ed.ac.uk/handle/1842/6592.
- 39. Terwee C, Bot S, de Boer M, van der Windt D, Knol D, Dekker J, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007;60(1):34–42. pmid:17161752
- 40.
SPSS. IBM SPSS Statistics for Windows, Version 26.0. Amonk, NY: IBM Corp.; 2019.
- 41.
Matlab. Matlab, Version 9.0. Natick, MA: The MathWorks Inc; 2016.
- 42. McGraw K, Wing S. Forming inferences about some intraclass correlation coefficients. Psychol Methods. 1996;1:30–46.
- 43. Koo T, Li M. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 2016;15(2):155–63. pmid:27330520
- 44. Cohen J. A coefficient of agreement for nominale scales. Educ Psychol Meas. 1960;20(1):37–46.
- 45. Bland JM, Altman DG. Measuring agreement in method comparison studies. Statistical methods in medical research. 1999;8(2):135–60. pmid:10501650
- 46. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1(8476):307–10. pmid:2868172.
- 47. Kottner J, Audigé L, Brorson S, Donner A, Gajewski BJ, Hróbjartsson A, et al. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. Journal of clinical epidemiology. 2011;64(1):96–106. pmid:21130355
- 48. Qin S, Nelson L, McLeod L, Eremenco S, Coons S. Assessing test–retest reliability of patient-reported outcome measures using intraclass correlation coefficients: recommendations for selecting and documenting the analytical formula. Quality of Life Research. 2019;28:1029–33. pmid:30547346
- 49. World Medical Association. World medical association declaration of Helsinki—Ethical principles for medical research involving human subjects. JAMA. 2013;310(20):2191–4. pmid:24141714
- 50. Nakken O, Lindstrøm J, Tysnes O, Holmøy T. Mortality trends of amyotrophic lateral sclerosis in Norway 1951–2014: an age-period-cohort study. J Neurol. 2016;263(12):2378–85. pmid:27586392
- 51. Taber K. The Use of Cronbach’s AlphaWhen Developing and Reporting Research Instruments in Science Education. Res Sci Educ. 2018;48:1273–96.
- 52. Crockford C, Newton J, Lonergan K, Madden C, Mays I, O’Sullivan M, et al. Measuring reliable change in cognition using the Edinburgh Cognitive and Behavioural ALS Screen (ECAS). Amyotrophic lateral sclerosis & frontotemporal degeneration. 2018;19(1–2):65–73. pmid:29214872
- 53. Kasper E, Zydatiss K, Schuster C, Machts J, Bittner D, Kaufmann J, et al. No Change in Executive Performance in ALS Patients: A Longitudinal Neuropsychological Study. Neurodegener Dis. 2016;16(3–4):184–91. pmid:26613522
- 54. Robinson KM, Lacey SC, Grugan P, Glosser G, Grossman M, McCluskey LF. Cognitive functioning in sporadic amyotrophic lateral sclerosis: a six month longitudinal study. J Neurol Neurosurg Psychiatry. 2006;77(5):668–70. pmid:16614030
- 55. Roalf D PJ. M, Xie S, Wolk D, Moelter S, SE A. Comparative accuracies of two common screening instruments for classification of Alzheimer’s disease, mild cognitive impairment, and healthy aging. Alzheimers Dement. 2013;9(5):529–37. pmid:23260866