Background

Cognitive ability is important in epidemiological research, as a risk factor for health-related outcomes and as an outcome in its own right. Lower mental ability is associated with increased cardiovascular disease incidence [1] and earlier mortality [2], and cognitive decline is among the most feared aspects of ageing [3]. The global prevalence of dementia is expected to increase to 65.7 million people by 2030 [4], and it is therefore important to understand the predictors and modifiers of cognitive ability (and decline) into older age. It is essential that cognitive assessments used in epidemiological studies are valid, sensitive to underlying mental abilities and sufficiently reliable.

UK Biobank is a large general population cohort of more than 500,000 middle and older-age adults who underwent medical, sociodemographic, mental health and cognitive assessment between 2006 and 2010 [5,6]. Participants were recruited from a range of backgrounds and demographics. The cohort will be followed up at intervals, and morbidity and mortality tracked through linkage with routine medical records [5,6].

The baseline cognitive assessment in UK Biobank included brief bespoke tasks delivered in a novel computerised format. Around 20,000 participants subsequently underwent repeat assessment, including most of the cognitive battery [5]. It is important to establish that these novel cognitive tests are reliable if they are to be used to detect cognitive change on long-term follow-up.

One of the most replicated phenomena in psychology is that cognitive scores inter-correlate[7], possibly because their brain substrates have been subject to shared genetic/environmental influences (the ‘Common cause hypothesis’ [8]). This means that performance on one task (e.g. visuospatial memory) would be expected to correlate significantly with performance on another (e.g. information processing speed). This is commonly referred to as a general factor of cognitive ability (simply ‘g’), and is often investigated with principal components analysis (PCA), which identifies how much variance is shared between the tests[9]. It is important to quantify a g factor score based on the cognitive tasks administered in UK Biobank; not least because it is a common phenotype for genetic studies [10].

We aim to firstly test for and then describe aspects of a g factor in UK Biobank with PCA using baseline cognitive data. We also quantify the stability of cognitive scores in the participants using the repeat cognitive data. We intend for this to provide a useful resource and basis for future studies using cognitive data in UK Biobank.

Participants

The UK Biobank cohort recruited 502,649 participants, aged 40–69 years, between 2006 and 2010. Baseline assessments were conducted at 22 research centres located across the United Kingdom [5]. Participants completed touchscreen questionnaires to provide information on sociodemographic factors (including age, gender, ethnicity, and postcode of residence), lifestyle (including smoking status and alcohol intake) and medical history. We excluded participants who reported chronic neurological diseases which could directly affect cognitive function (see S1 Table).

Participants provided full informed consent to participate in UK Biobank. This study was covered by the generic ethical approval for UK Biobank studies from the NHS National Research Ethics Service (approval letter dated 17^th June 2011, Ref 11/NW/0382).

Materials and procedure

Analysis

Because the RT scores were significantly positively distributed we transformed the variable with a natural log transform (‘LN’). The visual memory error scores were significantly skewed and zero-value inflated and therefore transformed with an LN +1 equation, both in IBM SPSS V.22. For log RT and log visual memory scores we report exponentiated betas; the effects are multiplicative so that an effect size of 1.00 represents no change, and e.g. beta = 1.01 equates to a 1% increase in the raw scores. For reasoning and numeric memory scores we report unstandardized betas. Because of the large sample sizes, we set our threshold of statistical significance at P = 0.01, partly to offset spurious findings through multiple comparisons.

Data reduction was applied to the baseline assessment cognitive test scores, using PCA. We included four of the cognitive variables assessed in UK Biobank: log RT, verbal-numerical reasoning, numeric memory, and log visual memory errors. We did not include PM in the PCA of cognitive ability because it has a binary outcome and therefore limited variance. Strictly speaking, PCA does not produce latent ‘factors’ however this usage is common and adopted here [15]. Because the numeric memory N was much lower than other tests we ran PCA three times to compare the outputs: at first with all four tests and then with numeric memory removed, and then log visual memory errors removed, leaving just reasoning and log RT. Inter-correlations between cognitive tests were examined with Pearson bivariate correlations, except for analyses including PM (a binary variable) where we instead report point-biserial r coefficients.

For the longitudinal cognitive data, we analysed stability in cognitive scores in two ways: relative consistency (i.e. rank consistency; higher scores at baseline predict higher at follow-up), and absolute consistency (i.e. similar raw scores [16]). To examine relative consistency, we ran 2,1 (‘two-way random’) intraclass correlations (ICC), often simplified to ICC (2,1 [17]) which are designed to account for systematic error (e.g. prevalent practice effects) and also random error in cognitive scores (e.g. individual alertness[18]). In terms of absolute reliability, we ran unadjusted repeated-measures ANOVA to test for significant differences in scores. From the ANOVA statistics we also calculated the ‘smallest real difference’ (SRD) value, an index that can be used to define the difference between two scores needed to reflect ‘real’ differences, excluding systematic and random error [16]. The UK Biobank cohort will be followed up and re-tested on the described cognitive measures, and future researchers investigating cognitive decline may find it helpful to know the magnitude of change in a given individual that is likely to represent true change rather than possible measurement imprecision. The SRD is also known as the repeatability coefficient [19]. This was calculated using the formula: “standard error of measurement (SEM) multiplied by 1.96 multiplied by √2” (where 1.96 reflects the Z score associated with a 95% CI [16]). Any change in a participant’s score that is greater than the SRD is considered a ‘real’ change. Note that we did not adjust for age or interval in the SRD score calculation, because the aim here was to quantify the amount of raw change rather than to account for it. We also re-ran the SRD analyses in participants over the age of 60 at baseline, where we may expect greater cognitive decline due to older age [20].

To test for an effect of test-retest interval length, and a potential moderating/interactive role of baseline age (e.g. where participants at older ages may show greater relative change in scores), we also ran repeated-measures ANOVA’s that included baseline age and interval in years (between test sessions) in the model as well as two- and three-way interaction terms including time point, baseline age, and length of interval.

Because the PM task is a binary variable, we computed absolute agreement (in percentage); we formally tested for PM score agreement with Cohen’s kappa, which measures ‘true’ agreement beyond chance, although this can be very conservative.

Data reduction and cognitive score inter-correlations

Unadjusted correlations showed that the tests associated significantly and in the expected directions (e.g. faster RT and higher Reasoning scores; Table 2). All correlations were of small-to-medium effect size; the strongest correlation (in terms of r) was between reasoning and numeric memory scores (r = 0.391, P < 0.001), and the weakest was between memory errors and log RT (r = 0.123, P <0.001). The PM scores showed generally weak correlations; the strongest was with reasoning scores (r = 0.250, P <0.001).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Unadjusted inter-correlations between cognitive variables.

https://doi.org/10.1371/journal.pone.0154222.t002

The first PCA model included RT, verbal-numerical reasoning, numeric memory and visual memory errors, and had an eigenvalue of 1.60, accounting for 39.93% of the variance. The second model (not including numeric memory) had an eigenvalue of 1.33, accounting for 44.35% of the variance. The third model, not including numeric memory or log visual memory errors, had an eigenvalue of 1.18 and accounted for 59.00% of the variance. All individual test scores had moderate-to-high loadings on their respective first unrotated principal components (range = [–] 0.49 to 0.77; see Table 3). Inspection of eigenvalues and scree-plots suggested that a single component should be extracted from the data in all cases (see Fig 1, which shows the first PCA).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Scree plot for cognitive general factor scores.

https://doi.org/10.1371/journal.pone.0154222.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Factor loadings for principal components analysis of cognitive tests.

https://doi.org/10.1371/journal.pone.0154222.t003

Cognitive test score within-participant stability

A subsample of the main UK Biobank sample had complete two-wave data for verbal-numerical reasoning (N = 4,523), RT (N = 19,327), visual memory (N = 19,017) and PM (N = 3,879). Two-way random ICC statistics showed statistically significant correlations for verbal-numerical reasoning (ICC = 0.65, 95% CI = 0.63 to 0.67), log RT (ICC = 0.57; 95% CI = 0.56 to 0.58) and log visual memory errors (ICC = 0.16; 95% CI = 0.15 to 0.17; all P values < 0.001). In order to explore the relatively lower reliability of the visual memory task further, we obtained additional data from the pilot phase of the UK Biobank baseline wave. In this phase, N = 3,598 participants (excluding those with neurological conditions) performed the visual memory task (6-pair version) twice in immediate succession, with ICC for log visual memory errors being 0.17 (95% CI = 0.14 to 0.20; P < 0.001). In those participants who had complete data for the pilot trials and the longitudinal follow-up (N = 447; mean interval = 6.4 years, SD = 0.14, range 6.2 to 7.0), the log visual memory errors ICC between the second pilot trial and the follow-up assessment was 0.13 (95% CI = 0.03, 0.21; P = 0.016).

Cohen’s kappa for PM indicated significant agreement across assessment waves (κ = 0.36, P < 0.001); 3,799 (97.9%) of the 3,879 with data were successful at both baseline and retest sessions.

Repeated measures ANOVA’s showed significant differences across waves 1 and 2 for log RT (F = 110.06, P <0.001) and visual memory errors (F = 42.76, P <0.001; see Table 4). There was a nominal (i.e. significant at the conventional P<0.05) significant difference in verbal-numerical reasoning scores (F = 5.32, P = 0.021), although not in the context of our more conservative alpha of 0.01. Scores on verbal-numerical reasoning and visual memory errors improved, while RT scores got worse. In none of these measures were the mean changes in the sample above the estimated ‘Smallest real difference’ (Table 4). We also ran these analyses after stratifying participants into those aged under 60 (S2 Table), and 60 and above (S3 Table). This made no difference to the results.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Cognitive test scores at Time 1 and Time 2, and reliability statistics.

https://doi.org/10.1371/journal.pone.0154222.t004

We tested for interactions between time point (i.e. first or second assessment), baseline age, and interval (in days between sessions; all two- and three-way). Of these, there were no interactions significant at P <0.01.

Overview

Aspects of the cognitive assessment used in UK Biobank were novel because of the need to conduct brief computerized cognitive assessments in a very large population. This paper demonstrates two points. The first point is that the UK Biobank cognitive data significantly inter-correlates and can be used to make a ‘g-factor’ score of general cognitive ability, however the amount of variance explained by that factor score is not as high as in other cohorts [15,21]. The second is that the cognitive scores in participants with repeat testing, on average four years later, showed reasonable stability and principally showed test-retest reliability for the reasoning and RT tasks. The visual memory task did not show good stability across time. The UK Biobank cognitive data promises to be a valuable resource for future studies.

Interpretation

In terms of factor analysis with PCA, eigenvalues and scree-plots indicated an unrotated one-factor solution for the cognitive data. It is important to describe the g-factor score because it may be particularly valuable to investigations of general cognitive ability, e.g. genetic studies. The factor solution based on four measures accounted for around 40% of total variance, slightly less than in cohorts with more detailed assessments which report 40–50% [21].

Longitudinal cognitive data showed that participants on average improved very slightly on the reasoning and visual memory tasks. These improvements may reflect practice effects [22]. Aspects of the reasoning assessment may measure a degree of ‘crystallised-type’ knowledge (e.g. ‘synonyms’); this type of ability would not be expected to deteriorate with age. Note that the magnitude of cognitive change was very small for all tests. Intraclass correlations showed that reasoning and RT tasks showed good stability across time points. However, while statistically significant, the estimate was low for the visual memory task (under 0.20); this was the case when this test was repeated either immediately or after several years, and regardless of whether participants performed it twice or three times. For comparison, Johnson et al. reported a raw Pearson r correlation around 0.70 for each of several commonly used memory tests (Logical Memory total, Verbal Paired Associates and Digit Span Backwards/Letter-number sequencing tests), between the ages of 70 and 73 in the Lothian Birth Cohort 1936 [23]. The ‘Smallest real difference’ values suggested that relatively large differences in scores would be required in order to show meaningful change across time points, potentially suggesting a degree of measurement imprecision in the tests. Research based on the baseline cognitive data available in UK Biobank (i.e. collected between 2006 and 2010), should take these limitations into account.

Limitations and future research

It is beneficial to have participants assessed on several tests of each cognitive domain, allowing for general factors of (for example) processing speed or memory, because these reflect only the shared variance between tests and therefore less task-specific variance[9]. Because the assessment did not include two or more cognitive tests that ostensibly test the same domain, it is not possible to create cognitive domain scores that are intermediate between the specific test and g (e.g. ‘memory’ [24,25]). More detailed cognitive assessment will be included in future UK Biobank waves, allowing researchers to say with greater certainty that each cognitive test assesses what it intends to (in terms of cognitive domains).

It is possible that findings are affected by a degree of ‘practice effects’ in the repeat-assessment sample, where the change in cognitive test scores that may be expected across time is influenced (to some degree) by familiarity with the tests [24]. Previous studies indicate small-to-null individual differences in the impact of cognitive practice effects [22]. It is possible with these novel UK Biobank-specific tasks, however, that some aspects of the battery are more susceptible to practice effects than others.

It is also possible that a degree of floor/ceiling effects are prevalent in some of the tests (i.e. where a task’s characteristics give it an upper or lower limit where it cannot distinguish between true differences in participant abilities [26]). For example, a large number of participants scored zero or very few errors on the visual memory task, and the PM score was analysed as a binary variable; these factors may contribute to the relatively low reliability shown by these tests. There may be a degree of selection bias in terms of less impaired individuals attending baseline and then repeat assessment.

Summary

This report characterizes and describes the cognitive data in UK Biobank in important ways, demonstrating a valid g factor of mental ability, and showing general stability in cognitive scores in a sub-sample of participants with longitudinal data. The UK Biobank promises to be a valuable resource for researchers seeking determinants of cognitive ability, and related outcomes. Future research will examine potential moderators of the cognitive test scores reported here.