Measurement of Social Cognition in Amyotrophic Lateral Sclerosis: A Population Based Study

Background: Amyotrophic lateral sclerosis (ALS) is a rapidly progressive neurodegenerative disease. Executive dysfunction is common in patients with ALS, with up to 50% of patients performing within an impaired range. There is evidence that social cognitive deficits associated with ALS are a function of deficits in executive function. The ‘Reading the Mind in the Eyes’ Test is a recognized test of social cognitive function, although the reliability of this instrument remains to be established. Methodology: Patients with ALS (n = 106), and age and IQ matched controls (n = 50) were recruited and asked to perform the Reading the Mind in the Eyes Test as part of an on-going population-based study of cognitive function. ALS patients were sub-stratified based on the presence, and/or extent of executive dysfunction. Results: Cronbach’s Alpha of .73 was observed, indicating good reliability on this measure. Split-half reliability analysis further confirms these findings (p = 0.826). The Reading the Mind in the Eyes test had excellent psychometric properties when discriminating between ALS patients who are cognitively intact, and those who have executive impairment, with an overall medium difficulty. There was a large magnitude significant difference between patients and controls (p< 0.001; η2 = .19). Post-hoc analysis revealed that controls performed significantly higher than patients with executive impairment (p< 0.001), and patients with single executive deficits (p = 0.002). Conclusion: Executive dysfunction impacts on social cognitive performance. This study contributes not only to the psychometric knowledge of this measure, but also to the usability, efficacy, and reliability of social cognitive assessment in ALS. Using population-specific normative data, we confirm the Reading the Mind in the Eyes Test is a reliable measure of social cognitive processes in ALS.


Introduction
Amyotrophic lateral sclerosis (ALS) is a rapidly progressive and fatal neurodegenerative disorder. Cognitive impairment, specifically executive dysfunction is common in patients with ALS, with up to 50% of patients performing within an impaired range [1]. Behaviour changes are also known to occur in ALS, with apathy as the most commonly reported symptom [2]. Recently, an in-depth study of the social cognitive changes in ALS found that performance in social cognitive function varies based on the measure used [3].
Social cognition can be defined as the ability to represent and attribute affective and cognitive mental states [4]. It integrates cognitive processes such as the ability to follow eye-gaze, share attention, recognize emotion, and to distinguish between self and other [5]. Initial social cognition research focused on disorders where social skills were implicated as a core feature [6,7]. However, in recent years the relationship between neurodegeneration and social cognitive processes has gained much attention [8] and assessments of neuropsychological and social cognitive performance are more routine in neurologic conditions where muscular atrophy is the dominant feature [9].
Performance on measures of social cognition in ALS has been associated with fronto-striatial dysfunction. ALS-related executive dysfunction has been reported to be the main predictor of social cognition performance when compared with demographic variables, behaviour, mood and personality [10]. Social cognitive deficits are also a well-recognised feature of Frontotemporal dementia [11] (FTD). ALS with a comorbid FTD is known to occur in 10-15% of patients [12], with a strong clinical and pathological overlap between these conditions [13]. Cognitive features, such as impaired social cognition of FTD may therefore be evident in people with ALS, whom do not meet the criteria for a comorbid dementia syndrome.
A commonly used social cognition task in neurodegenerative research is the Reading the Mind in the Eyes Test [3,10,14]. The Reading the Mind in the Eyes Test [6] (RMET) comprises photographs of eye regions of human faces where participants are required to infer their mental/emotional state given four possible choices. The RMET was chosen as a measure of affective social cognition within this cohort, as individuals can respond verbally or by pointing. Although the RMET was not specifically developed for use with an ALS cohort, it accounts for potential bulbar/spinal motor disability. The RMET has also been used with a wide variety of clinical populations with neurological conditions, including patients with ALS [15], Kennedy's disease [9], Huntington's disease [16], Lewy body dementia [17], Parkinson's Disease [18], Epilepsy [19], as well as psychiatric populations [20], children [21,22] and healthy cohorts [23][24][25]. However the reliability of the measure has been subject to criticism. Notwithstanding, recent evidence suggests that the RMET has good validity [26], and is unaffected by the testtaker's gender [27] in some but not all instances. Conflicting reports of reliability may be due to cultural differences in interpreting the stimuli.
The aim of our study was to investigate the reliability of the RMET using a control cohort, to sub-stratify ALS patients based on cognitive categorisation criteria [28] and to compare cohorts of multi-domain executive deficits (cognitively impaired), single executive deficits, and no cognitive abnormalities detectable on testing. We further investigated the difficulty and discrimination coefficients associated with this measure in an Irish population-based cohort of ALS patients and healthy controls.

Participants
A total of 108 population based ALS patients were included in this study. The research design, case ascertainment, procedure, and information on the population-based register have been previously reported in detail [1,14,29,30]. ALS patients were classified into groups specific to cognitive status, based on cognitive domain-based criteria. This methodology has previously been reported in detail [29], and is based on internationally accepted consensus criteria for cognitive impairment [30]. This method further compliments these criteria as a more stringent cut-off was used to define abnormality. Using these criteria, patients were sub-stratified based on whether they had 'no cognitive abnormalities' (n = 70), a 'single executive deficit' (n = 19), or 'multi-executive deficits/cognitively impaired' (n = 19).
Due to ALS patient stratification based on severity of executive impairment, we completed a priori power analyses to ensure our findings with F statistics would be robust with smaller groups. A standard alpha error probability of .05 was employed, with a 1-β power of .8 recruited for a 4 group design. This yielded a non-centrality parameter λ of 12.16, critical F at 2.73, and actual power of .82, which required a minimum of 19 per group.
Inclusion criteria for this research were a diagnosis of a possible, probable or definite ALS based on the El Escorial criteria [31], with supported neurophysiology for diagnostic accuracy. Exclusion criteria included the presence of co-morbid neurological and/or psychiatric condition such as stroke, psychosis, traumatic brain injury, or an active or positive history of chronic substance abuse. Clinic-and home-based assessments were used to gather clinical and neuropsychological data. In addition, 50 healthy age-and education-matched controls were recruited considering the aforementioned exclusion criteria.
The Medical Research Ethics Committee of Beaumont Hospital, Dublin, Ireland, has approved this study. Informed consent was obtained from all participants in this study.

Cognitive Measures
The Reading the Mind in the Eyes Task is a 36-item assessment where photographs of eye regions are presented, and participants are required to infer the mental/emotional from four choices e.g., terrified, upset, bored, irritated.
Participants were also assessed using a battery of standardized neuropsychological measures and cognitively categorised as noted above [29]. Variables of interest were extracted for the purpose of this study, which included the Wechsler Test of Adult Reading [32] (WTAR), a measure of premorbid function yielding a predicted Full Scale IQ (pFSIQ). Ravens Progressive Coloured Matrices [33] investigated current function through verbal/non-verbal trials.

Ethical Considerations
The Beaumont Hospital Medical Research Ethics Committee approved this study. All procedures were conducted in accordance to the principles expressed in the Declaration of Helsinki. Written informed consent was obtained from all participants in this study.

Statistical Methods
Demographic characteristics between ALS patients and controls were analysed using independent samples t-tests (age and years of education), with χ 2 used to test dichotomised variables. ANOVA were used to compare multiple dependent variables. The threshold for statistical significance was set at p<0.05 as per our power calculations. All statistical analyses were conducted using SPSS, Version 22.0.
Reliability and Validity. Classification for good internal consistency, using Cronbach's alpha, remains at the internationally accepted value of >.70. Split half reliability was also assessed using Spearman-Brown coefficient for equal length measures, to compliment analyses of internal consistency.
Item difficulty. We computed the difficulty and discrimination coefficient of each item in order to test if it yielded the necessary degree of reliability and validity [34]. This methodology has previously been employed when investigating the psychometric properties of a translated version of the RMET [35]. We infer that a reliable and valid item should be able to appropriately distinguish between those with a higher total score and those with a lower total score on the RMET. We defined the higher score and lower score groups as the upper 27% and the lower 27% of participants, according to their total score on RMET consistent with previous studies [36].

Difficulty Coefficient
The proportion of participants who answer a test item correctly relative to the test taking population measures the difficulty of an item. The higher this proportion is, the lower the item's difficulty. To calculate the difficulty of an individual test item, the number of participants who answered the item correctly (from both the upper and lower proportions) is divided by the total number of participants who took the test. It is represented as the following formula: whereby Pi = difficulty index of item; Ai = number of correct answers in upper 27% added to the number of correct answers in the lower 27%; Ni = the sum of total test takers in the upper 27% and lower 27% groups.
Discrimination Coefficient. Items of a valid and reliable test must be able to appropriately differentiate between participants who are relatively strong from those who are relatively weak; otherwise, the test is lacking sensitivity and specificity. Through the use of healthy controls, we investigated the RMET's overall discrimination coefficient. We then also applied this methodology to our ALS cohort whereby the proportion of patients who scored in the upper range reflects patients with a 'no cognitive abnormality detected' status, and the proportion of patients who scored in the lower group represents the 'multi executive deficits' apparent on testing group. Using this method we calculate the ability of this task to discriminate between ALS patients who are cognitively intact, and those who are cognitively impaired based on internationally accepted criteria.
It is calculated through the following formula: whereby Di: index of discrimination of item; Pu: the proportion of those in the upper 27% group who correctly scored on the item relative to the sum of test takers; Pl: the proportion of those in the lower 27% group who correctly scored on the item relative to the sum of test takers.
The following guidelines have been published [30]

Control Demographics and Psychometric Reliability
Control participants at the time of assessment were 61.4 years SD± 9.1, with 12.7 years of education SD±2.9. This cohort consisted of 30 males (60%). Prior to conducting reliability analyses, we investigated whether participant gender was equally distributed within our control cohort. The distribution of gender within the control cohort was found to be homogenous (p = 0.203). We investigated whether there was a significant difference in performance based on gender, with a 2-tailed hypothesis. With equal variance assumed, we report that there was no difference identified on the score of the RMET, based on gender stratification (p = 0.113).
We proceeded to conduct reliability statistics, and considering the full 36-items of the RMET, a Cronbach's Alpha of .73 was observed. The reliability of each individual test item, and how it correlates to the overall measure can be seen in Table 1. Split-half reliability analyses were conducted using Spearman-Brown Coefficient, and equal length statistics confirm the reliability of this measure (p = 0.826). Each item was analysed to investigate a difficulty coefficient (Pi) and was classified accordingly. A discrimination coefficient (Di) and classification were employed to investigate whether the individual items are accurate at discriminating between controls who performed well compared to those with a low score on the test. Overall the RMET is reported to be of medium difficulty (68.8%), with good psychometric properties for discriminating healthy controls based on performance (Di = 31.7). The breakdown of individualised items can be found in Table 2.  Individual patient groups' performance was compared to healthy controls, with groups matched for age at assessment (p = 0.622), and years of education (p = 0.940). Homogeneity of variance was satisfied (p = 0.115), and an ANOVA was conducted as per our power calculation. There was significant difference with large magnitude reported from the ANOVA (p< 0.001; η 2 = .19).
To elucidate this further, a post-hoc analysis was employed. Controls performed significantly higher than patients with executive impairment (p = 0.001), and patients with single executive deficits (p = 0.002). Controls did not significantly differ from ALS patients with no cognitive abnormalities (p = 0.157). By contrast, ALS patients without cognitive deficits did not differ from ALS patients with single executive deficits (p = 0.118), yet did perform significantly better than ALS patients with multi-domain executive impairment (p = 0.001). Interestingly, ALS patients with single executive deficits did not differ from the patients with multi-domain executive impairment (p = 0.341). Psychometrics in ALS patients. Itemised group performance can be seen in Table 3, where scoring patterns remain consistent based on executive grouping. The Reading the Mind in the Eyes test appears to have 'Excellent' psychometric properties when discriminating between patients who are cognitively intact, and those who have executive impairment (Di = 43.5), with an overall 'Medium' difficulty (54.7%). Individual item difficulty and discrimination for ALS patients are presented in Table 4.

Discussion
Our primary aim was to investigate whether the Reading the Mind in the Eyes Test (RMET) is a reliable measure. Secondary aims included addressing its efficacy and validity as a measure against a patient population, where executive dysfunction is categorised with high accuracy. These data suggest that the RMET is a reliable measure of social cognition for use with healthy and ALS cohorts (Cronbach's alpha .73; Split-half reliability p = .826). Our data further support previous work [27] that the RMET score is not affected by gender (p = 0.113).
Overall the RMET is reported to be of medium difficulty (68.8%), with good psychometric properties for discriminating between controls that perform in the upper and lower ranges (Di = 31.7). This analysis was applied to an ALS cohort, as a total group. The lower 27% of the patient group had a range of executive deficits, and the upper 27% were cognitively intact. This allowed us to infer whether this measure is useful in dichotomising executive dysfunction in ALS. These data suggest that this measure has 'Excellent' psychometric properties (Di = 43.5) for discriminating between patients with cognitive impairment and those who are cognitively intact, with an overall 'Medium' difficulty (54.7%). Consistent with previous literature, there was significant difference with large magnitude reported between patient groups and the control cohort (p< 0.001; η 2 = .19). Post-hoc analysis revealed that controls performed significantly higher than patients with executive impairment (p = 0.001), and patients with single executive deficits (p = 0.002) but not cognitively intact ALS patients (p = 0.157). Of interest were the scores of ALS patients with single executive deficits, as they did not differ from the patients with multi-domain executive impairment (p = 0.341) or cognitively intact ALS patients (p = 0.118). This pattern of performance demonstrates that ALS patients without cognitive abnormalities are similar to controls, and to those with single domain cognitive impairment (See Fig 1). Patterns of social cognitive deficits such as these are consistent with both the subtle deficits of executive function, which may present early in the disease, as well as the more severe impairment known to occur in some cases. The lack of significant difference in performance on the RMET between groups i.e., no cognitive abnormalities and single executive impairment, could be attributed to difficulties in detecting subtle cognitive impairment in this cognitive domain.
This study has limitations. Given the lower scoring profile of cognitively intact ALS patients, longitudinal follow-up would be advantageous. This point is further generalised to the other ALS cohorts to quantify the rate that social cognitive processes decline alongside the disease trajectory, with respect to executive dysfunction. Given the relevance of these findings in relation to executive function, future research could investigate the impact of social cognitive deficits on patient survival, the relationship of social cognitive deficits to behavioural presentations, and the extent to which patients differ on test performance considering genetic expansions known to implicate cognitive function i.e., C9orf72. Future studies could investigate the relationship between social cognitive performance and behavioural features associated with ALS; could determine whether social cognitive decline has a negative impact on caregiver burden in ALS; and whether there are additional negative implications on the psychological wellbeing of patients and caregivers.
In conclusion, our data confirm the Reading the Mind in the Eyes Test is a reliable measure of social cognitive processes, and illustrate how executive dysfunction mediates performance on this measure. These data provide clinically relevant normative data for use in clinical and research settings. Moreover, by stratifying by level of executive impairment here, these data provide a useful reference point in terms of severity of social cognitive deficit. Clinically these data support the routine inclusion of social cognitive measures in standardised batteries of neuropsychological tests. Furthermore, considering the RMET as a valid measure of social cognition for use with ALS patients, their overall performance on this measure should be considering in light of ALS-FTD comorbidity.