Participants

An opportunity sample of 100 neurologically healthy adults were recruited primarily from University campuses in Cambridge and London, local retail outlets, and via social media, of which eight participants failed to complete one or more tests and were excluded from all analyses. There were no missing data across the sample of 92 participants (mean age 40 years; range 18–70; s_age16.78), of which 30 were male, on any of the tests reported here. All were British nationals, with English as the first language, and with normal/corrected-to-normal vision and hearing. Participants self-declared that they had no history of neurological or psychiatric disorder. Extensive training in the administration and scoring of all tests was provided to three research assistants over several days by PB (an experienced neuropsychologist), and the testing sessions were closely monitored and supervised to ensure full compliance with the standardized administration and scoring procedures. All participants were recruited and tested between 2013 and 2016, in a UK University setting. The procedure was approved by the University Ethics Panel, and was conducted in accordance with the tenets of the Declaration of Helsinki. All participants had normal/corrected-to-normal vision and hearing (self reported), and spoke English as their first language.

Data collection

All participants completed the NART first and then all 10 core subtests from the WAIS-IV battery. All tests were administered following standard published instructions. Participants attended a single session of approx. 90 minutes, with breaks provided upon request.

Analysis procedure and results

The NART responses for each participant were placed in a 2-D bit matrix, to be denoted Q, in which each row (1..m) corresponded to a NART word index, and each column (1..n) to a participant number (Fig 1). Here, rows m = 50 and columns n = 92.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. 2-D bit matrix for all participants and NART word responses in which a black dot denotes a pronunciation error.

https://doi.org/10.1371/journal.pone.0205754.g001

The presence of a 1 in Q (a black dot in Fig 1) denotes an incorrect pronunciation (error), so the total number of NART errors, x_j, for each participant j from 1..n over the sequence of NART words i from 1..m, is given by Eq 1, such that x_j ∈ [0..50]. The number of NART errors per participant in our data ranged from 2 to 46 (). Corresponding WAIS-IV FSIQ results, to be denoted y, ranged from 80 to 150 (). A Kolmogorov-Smirnov test indicates that neither empirical dataset deviates significantly from a normal distribution (k = 0.98, k = 1.00, both p < .0001).

(1)

The linear Pearson product-moment correlation coefficient (PPMCC) between NART errors (x) and measured WAIS-IV FSIQ (y) is given by Eq 2. In addition to enabling a linear regression equation to be calculated (see below), the PPMCC, r, and coefficient of determination, r², are commonly used in neuropsychological literature to assess the degree of association between different test scores (e.g., see [28]), and provide one metric against which the GA-derived model described later in this article is to be evaluated. The value given by Eq 2 for our data, consistent with that reported in [20], was r₍₉₀₎ = −0.68 p < 0.000001, which is typically classified as large [35]. The coefficient of determination was r² = 0.47, a comparable number to that reported in [5,6] and many subsequent studies that correlate NART error scores against earlier iterations of WAIS IQ. It suggests that the explanatory variable (NART errors) accounts for 47% of the variance in the predicted variable (WAIS-IV FSIQ).

(2)

A linear regression equation (of the form , where denotes a predicted value of y) was produced, again in keeping with earlier approaches, with multiplicative constant a (slope) and additive constant b (y-intercept), which can then be used to predict WAIS-IV FSIQ () for any number of NART errors (x). The PPMCC, r (Eq 2), is used to calculate the line equation constants (Eq 3 for slope, a, and then Eq 4 for y-intercept, b).

(3)

(4)

Using the full set of NART words, the line equation for our data was , shown on a scatterplot of raw NART errors vs. WAIS-IV FSIQ in Fig 2 as a dotted black line with circles denoting measured values (i.e., our 92 participant test scores). The proximity of the sample points to the initial line equation is highlighted as a shaded zone (convex hull [36]).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Scatterplot of raw NART errors vs. measured WAIS-IV FSIQ (hollow circles).

Dotted black line is initial line equation; shaded zone is the convex hull.

https://doi.org/10.1371/journal.pone.0205754.g002

A correlation coefficient (or coefficient of determination) should not, on its own, be used to assess the accuracy of a linear regression model such as this, since in a comparison between two hypothetical models, greater absolute r (or greater r²) for the first model may coincide with greater predictive accuracy for the second model, since the slope of a regression line is not necessarily coupled with lower average residuals (i.e., shorter average distance of measured sample points to their corresponding predictions). An additional metric should be used that specifically assesses the accuracy with which a model predicts known values; one simple metric that can accomplish this is mean absolute error (MAE, Eq 5), which has the advantage of being in the same units as the predicted variable (here, IQ points). Using raw NART errors, MAE = 7.33 (s = 5.64), showing that, on average, the error between predicted and observed WAIS-IV FSIQ using raw NART errors for our data was 7.33 IQ points.

(5)

In addition, regression models should be validated to examine their stability to the removal of data points (i.e., the degree to which they may be affected by outliers), and their ability to make accurate predictions for samples not used in their production (i.e., the degree to which overfitting may have occurred). Alternative approaches to accomplish this include dividing data into training and testing sets, k-folds validation [37], and exhaustive leave-one-out cross-validation (LOOCV), described in [31]. The latter approach is used here, in part because it is fully reproducible (i.e., does not depend upon the randomized division of data into training and testing subsamples). In this form of validation, the predicted variable and other metrics of interest are calculated using models produced using subsamples of the original data in which one participant at a time has been left out (i.e., n subsamples of n − 1 participants, with participant k left out, such that k is iterated from 1.. n). These are sometimes referred to as jackknife samples. Thereafter, the accuracy with which each of the n models predict metrics of interest for the one left out participant not used their production is assessed. As before, MAE may be used to evaluate prediction accuracy both for the n jackknife models (which comprised n(n − 1) = 8372 individual predictions) and the n single left out sample predictions (here 92). A correlation coefficient (or coefficient of determination) can only be produced for the jackknife models since the left out samples are not associated with a single line equation.

For our data, averaging over the n jackknife models, with standard deviation shown in parenthesis, yields = −0.68 (0.01), a = −0.9750 (0.01), b = 126.3157 (0.28), and MAE = 7.33 (5.61). These values are remarkably close to where all participant data were used (reported above), indicating that outliers did not significantly affect these metrics. Next, the accuracy of the predicted variable elicited by each model using each respective single left-out participant (i.e., the participant not used in the production of that model) was assessed. This yielded MAE = 7.49 (SD = 5.78), which is fractionally greater than the MAE calculated using all data and the average MAE across the n jackknife models; however, this is to be expected given that the models are now being requested to make predictions for participants that were not used in their production. Furthermore, the differences in MAE values (between all data, 8372 jackknife subsamples, n leave-one-out samples) were not statistically significant (p > .05).

Apparatus

Statistical analyses and optimization algorithms were implemented in MATLAB (The Mathworks Inc., Natick MA). The standard regression mode, GA, validation routines and experimental data used for testing and validation are freely provided for download from the Open Science Framework (http://dx.doi.org/10.17605/OSF.IO/34BKU).

Analysis procedure

The GA described below is charged with finding the optimum subset of NART words that yields the smallest average prediction residual (MAE), working from the initial starting point of using all 50 words.

GAs search solution spaces so large that they cannot feasibly be traversed using exhaustive/analytical approaches, enabling them to address computational problems, like the present one, that have no polynomial-time exhaustive solution. The final solution returned by a GA is not necessarily the best possible answer, since they rely upon an adaptive heuristic approach that iteratively improves upon each currently held solution until a solution that is deemed acceptably good is obtained. However, if appropriately configured, GAs can produce solutions that dramatically improve upon the initial starting point. GAs, being inspired by the biological principle of natural selection by survival of the fittest, entail the representation of candidate solutions as chromosomes, the evaluation of chromosome efficacy through a fitness function, the creation of new chromosomes via mutation and/or crossover (principally from the chromosomes identified as the most fit), and a selection method by which individual chromosomes are chosen to sire subsequent generations. A termination criterion must also be decided upon to determine how long the GA will run. Alternatives include letting the GA run for a fix period of time, for a fixed number of generations, until the solution is valid (e.g., in some NP-class problems in which finding a solution that merely works is a laudable goal), or until the fitness of the solutions produced over a pre-determined period of time or number of generations ceases to improve (i.e., evolutionary stagnation).

Chromosome structure

Each chromosome, c, was a 1-D bit string (specifically, a sequence of 50 binary digits, each referred to as a gene) wherein each bit controls whether the NART word at index i should by used (c_i = 1) or not used (c_i = 0) in the calculation of each participant’s revised NART score. The number of alleles (alternatives) for each gene was therefore 2: 0 and 1. All possible solutions to the problem of finding the optimum NART subset can be represented on such a chromosome, of which there are 2⁵⁰ (one quadrillion, one hundred twenty five trillion, eight hundred ninety nine billion, nine hundred six million, eight hundred forty two thousand, six hundred and twenty four), which is the cardinality of the powerset of the set of words (w) in the original NART, |℘(w)| = 2^|w| (i.e., the size of the set of all possible subsets of w). If one were to iterate through these subsets one at a time, a tight bound algorithm of exponential time complexity ϴ(2^|w|) would be required, which is computationally impractical. To put this into perspective, if each subset took 1 sec to evaluate, it would take 36 million years to sequentially test all subsets to identify the true (guaranteed) optimum.

Settings

The GA was run for a fixed number of generations (128), which was found to be more than sufficient for fitness to reach a stable asymptote (i.e., after which no further improvement in fitness was observed), and also provided an acceptable run-time of approx. 1–2 minutes on a standard computer. The number of children produced in each generation was also set to 128. With these settings, one run produces a total of 128 × 128 = 16384 candidate solutions, with a tendency to improvement from generation-to-generation that inevitably slows as the algorithm progresses and fitter solutions become more difficult to find. A mutation rate [38] of , i.e. 10%, was used, determined experimentally to, when coupled with the use of 128 children per generation, yield fast and stable evolutionary descent.

The mutation routine entailed the negation of randomly selected genes (sometimes called bit mutation or bit flipping). Crossover (recombination) was not used, since this is not thought to be an effective approach for problems in which large changes in chromosome composition are likely to dramatically affect performance and thereby thwart evolutionary progress. A simple maximally elitist GA was used, such that only the fittest child in each generation was retained (determined as described below), which was set to be the parent of the subsequent generation. Other (less fit) chromosomes were destroyed. However, the fittest child in each generation always replaced the parent chromosome (whether fitter or not), enabling the fitness profile over time to decrease as well as increase, which is thought to prevent premature convergence (although it is acknowledged that the single-parent approach could lead to convergence to local optima, to demonstrate the general procedure, this simple approach was taken, and suitably fit solutions were indeed produced).

Fitness function

To calculate chromosome fitness (a so-called figure of merit), first a revised NART response matrix, Q′, is calculated from Q, the original response matrix, by multiplying each participant’s NART word responses with the chromosome to be evaluated (Eq 6). This had the effect of masking the responses for specific words so that they no longer contributed to the final score for any participant.

(6)

Next, the number of NART errors for the surviving NART words only, to be called the revised NART score, x′, is calculated using Eq 1 with Q′ substituting for Q. Next, a correlation coefficient is calculated using Eq 2 with x′ substituting for x, and then revised line equation constants, a′ and b’, are calculated using Eqs 3 and 4 with and s_x′ substituting for and s_x, respectively. A revised prediction, can then be calculated for each participant. Using Eq 5, with substituting for , the MAE using the adjusted NART scores can then be calculated, evaluating how accurately the current word subset approximates measured WAIS-IV FSIQ. The MAE value is returned as the fitness of the chromosome.

Over successive generations, the GA identifies the optimum subset: i.e., the optimum values in c, that when entrywise multiplied with the raw NART responses from all participants identically, minimizes MAE. As a consequence of falling MAE, the absolute correlation and coefficient of determination will also typically increase from generation-to-generation. It is worth noting that there may be multiple equally fit chromosomes, and that running the GA on multiple occasions could produce subtly different word subsets each time because several words or word combinations may each be equally suitable alternatives in c for MAE minimization (reflecting the randomness inherent in evolutionary descent, akin to nature). Selecting for the additional criterion of minimal subset cardinality is one approach that could be used to select from among equally fit alternatives (i.e., when two subsets yielding equal MAE are evaluated, particularly if the objective was to devise an abbreviated test or a new test using the fewest number of candidate questions).

Model results

Using the settings described above, repeating the GA for 100 runs, with standard deviation shown in parenthesis, yields MAE = 5.76 (0.03), a = −2.84 (0.16), b = 130.18 (1.50), r = −0.78 (0.003), and r² = 0.61 (0.004), with the average number of words at 19.65. Over the 100 runs performed, 14 unique chromosomes were produced. Of these, the GA identified an optimal subset comprising 17 of the 50 original NART words. Despite using only around one third (35%) of the words in the full NART, this subset produces more accurate WAIS-IV FSIQ predictions (MAE = 5.75 relative to 7.33), and also yields a PPMCC of r = −0.78 compared to r = −0.68 for the original approach (the corresponding coefficient of determination, r², increased from 0.46 to 0.61). The line equation constants for this chromosome were a = −3.4882 and b = 132.7113. The NART word subset identified by the GA on this run (with original NART word index shown in parenthesis) was: Capon (7), Nausea (9), Courteous (11), Naïve (14), Thyme (17), Procreate (23), Gouge (25), Superfluous (26), Simile (27), Façade (31), Drachm (33), Idyll (38), Puerperal (39), Leviathan (43), Prelate (45), Demesne (47), and Campanile (50). Raw and GA-derived participant scores (shown as ○ and + symbols, respectively) and corresponding line equations functions (shown as solid and dotted lines) are shown in Fig 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Scatterplot of NART errors vs. WAIS-IV FSIQ using original (sample points denoted ○, line equation dotted, light gray convex hull) and GA-derived (sample points denoted +, line equation dashed, dark gray convex hull) models.

https://doi.org/10.1371/journal.pone.0205754.g003

The evolutionary progression from the starting point of using all 50 NART words is shown in Fig 4, which also indicates that the number of generations that the GA was permitted to run was adequate for a stable chromosome to be formed using the mutation rate and number of children per generation described above.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Chromosome evolution over 128 generations (top row is start point) in which black dots denote NART words used in the calculation of each participant’s revised NART score.

https://doi.org/10.1371/journal.pone.0205754.g004

Validation

As with the initial regression model (above), the robustness and generalizability of the GA-derived model can be evaluated with jackknifing and LOOCV. For the GA-derived approach, averaging over the n jackknife models, with standard deviation shown in parenthesis, yields r = −0.78 (0.01), a = −2.8064 (0.14), b = 129.6049 (1.63), and MAE = 5.76 (5.45), values that are very close to the mean performance over 100 run using all participant data described above. The mean prediction was 108.69 (9.85). The chromosomes produced in each of these models are shown in Fig 5, in which a high degree of consistency implies a stable solution that is largely invariant to the removal of individual participants, and is quite consistent from run to run. For the n single left out samples, the MAE was 7.32 (6.11), with a mean prediction of 108.69 (9.79). The critical performance metrics (subset cardinality, mean prediction, r, and r², and MAE) are shown in relation to the original model and its validation in Table 1. It is apparent that the cardinality of the subset (number of retained NART words) is fractionally higher than the ‘best’ runs using all data described above (in which 17 words are retained). Indeed, due to the heuristic nature of the approach, running the all-data single-run GA multiple times also produces some results wherein 19–20 words are retained, since these alternative solutions also yield an MAE = 5.75. It likely that, with if the number of participants were increased, MAE for single left-out participants would fall.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Chromosomes for n jackknifed GA models in which participant k was left out.

https://doi.org/10.1371/journal.pone.0205754.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Critical performance metrics for each models and cross validation set (standard deviation shown in parenthesis, were available).

https://doi.org/10.1371/journal.pone.0205754.t001

In Table 1, it is apparent that the MAE for the all-data GA model and its jackknife subsamples are lower than the initial model all-data MAE and its jackknife subsamples. Comparing jackknife distributions using a t-test, this difference, although relatively small, is statistically significant [t₍₈₄₆₂₎ = 2.75,p < 0.01]. The MAE differences between the one-left-out (validation) sets in the initial and GA-derived models is not statistically significant (p = 0.84), despite that the GA-derived model uses, on average, only 19–20 words of the original 50, demonstrating that the additional words in the NART, as originally formulated, did not improve predictive accuracy for our data.