Figures
Abstract
Objective
To compare two definitions of neurocognitive impairment (NCI) in a large clinical trial of effectively-treated HIV-infected adults at baseline.
Methods
Hopkins Verbal Learning test-Revised (HVLT-R), Colour Trail (CTT) and Grooved Pegboard (GPT) tests were applied exploring five cognitive domains. Raw scores were transformed into Z-scores and NCI defined as summary NPZ-5 score one standard deviation below the mean of the normative dataset (i.e. <−1SD) or Z-scores <−1SD in at least two individual domains (categorical scale). Principal component analysis (PCA) was performed to explore the contribution of individual tests to the total variance.
Results
Mean NPZ-5 score was −0.72 (SD 0.98) and 178/548 (32%) participants had NPZ-5 scores <−1SD. When impairment was defined as <−1SD in at least two individual tests, 283 (52%) patients were impaired. Strong correlations between the two components of the HVLT-R test (learning/recall) (r = 0.73), and the CTT and (attention/executive functioning) (r = 0.66) were observed. PCA showed a clustering with three components accounting for 88% of the total variance. When patients who scored <−1SD only in two correlated tests were considered as not impaired, prevalence of NCI was 43%. When correlated test scores were averaged, 36% of participants had NPZ-3 scores <−1SD and 32% underperformed in at least two individual tests.
Conclusion
Controlling for differential contribution of individual test-scores on the overall performance and the level of correlation between components of the test battery used appear to be important when testing cognitive function. These two factors are likely to affect both summary scores and categorical scales in defining cognitive impairment.
Citation: Arenas-Pinto A, Winston A, Stöhr W, Day J, Wiggins R, Quah SP, et al. (2014) Neurocognitive Function in HIV-Infected Patients: Comparison of Two Methods to Define Impairment. PLoS ONE 9(7): e103498. https://doi.org/10.1371/journal.pone.0103498
Editor: Terence J. Quinn, University of Glasgow, United Kingdom
Received: November 1, 2013; Accepted: July 2, 2014; Published: July 31, 2014
Copyright: © 2014 Arenas-Pinto et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: PIVOT is funded by the National Institutes for Health Research Health (NIHR) Technology Assessment programme (project number 06/403/90). The views and opinions expressed herein are those of the authors and do not necessarily reflect those of the Health Technology Assessment programme, NIHR, the National Health Service or the Department of Health. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have read the journal’s policy and have the following conflicts: Dr Alejandro Arenas-Pinto has received honoraria or research grants, or been investigator, in clinical trials sponsored by Abbott, Boehringer Ingelheim, Bristol-Myers Squibb, Gilead Sciences, Pfizer, ViiV Healthcare and Janssen Cilag. Dr Alan Winston holds grants from the British Medical Research Council and the European Union (Framework 7). Has received honoraria or research grants, or been a consultant or investigator, in clinical trials sponsored by Abbott, Boehringer Ingelheim, Bristol-Myers Squibb, Gilead Sciences, GlaxoSmithKline, Janssen Cilag, Roche, Pfizer and ViiV Healthcare. Prof N Paton has received honoraria from Roche / Genentec for work on a DSMB; from AbbVie and Merck and Jansssen for speaking at meetings / attending advisory boards; and has received grant funding or donations of drugs for research studies from GSK, Gilead, Merck, Abbvie, Janssen and Pfizer. This does not alter their adherence to PLOS ONE policies on sharing data and materials. The rest of the authors have declared that no competing interests exist.
Introduction
HIV-associated neurocognitive impairment (NCI) remains a major issue in the care of chronic HIV management. Cognitive impairment is still common, even among patients successfully treated with anti-retroviral therapy (ART) [1], [2]. However controversy remains regarding the best way to, firstly, assess cognitive function and secondly, interpret cognitive function results in HIV-infected patients [3]–[5]. In research settings, a typical approach is to take the average of demographically adjusted scores on individual test domains to create a summary score as a marker of cognitive function and in some cases, to use this as markers to quantify cognitive impairment [2], [6], [7] but summary scores assume equivalent contribution of each test score on the overall summary score and do not offer differential information on the cognitive domain which may be affected. Other approaches such as the global deficit score have also been used to generate overall composite scores but in general, these do not offer differential information on the cognitive domain which may be affected [8], [9].
According to the 2007 revised research criteria for classifying HIV-associated NCI (often known as the Frascati criteria or the HIV-associated neurocognitive disorders (HAND) classification) cognitive impairment is defined as performance of at least 1 standard deviation (SD) below the mean of demographically adjusted normative datasets in at least two different cognitive domains [10].
High level of agreement between summary deficit scores and HAND classification results has been reported but the latter often identified a larger number of impaired patients as compared to summary score deficits [10], [11]. This excess in diagnosis of impairment usually correspond with mild forms of the condition which may correspond to false positive results [3], [5]. The aim of this study was to compare these two methods to assess cognitive function in a large population of effectively treated HIV-infected adult individuals. In addition, we aimed to assess acceptability and suitability of a screening test battery exploring five cognitive domains in the context of a large randomised clinical trial (RCT).
Methods
Participant selection
This analysis used baseline data from individuals recruited into a large treatment strategy trial for long-term management of chronic HIV infection – the Protease Inhibitor monotherapy Versus On-going Triple-therapy (PIVOT) trial. The trial enrolled participants receiving a stable combination anti-retroviral therapy (cART) regimen and a plasma HIV RNA <50 copies/ml at screening and for at least 24 weeks prior to study entry. 20% of the study population had a history of an AIDS-defining condition. Participants were recruited between November 2008 and July 2010 and followed-up until November 2013.
The PIVOT study is registered as EUDRACT: 2007-006448-23 and ISRCTN04857074. National Research Ethics Service (NRES) approval for the trial, including the NC assessments, was obtained from the East of England Cambridge South Ethics Committee. Written informed consent was obtained from all study participants.
Neurocognitive testing
Neurocognitive testing was performed by designated research staff at each study site after receiving appropriate training by the coordinating centre. The training procedures included a face-to-face session, a training video and practice of the tests with at least five work colleagues before being allowed to assess study participants, followed by yearly revision. Five cognitive domains were explored with three different tests: Verbal learning and memory were assessed using Hopkins Verbal Learning Test-Revised (HVLT-R) [12], fine motor skills assessed using Grooved Pegboard [13], and attention and executive function assessed using Color Trails Tests (CTT) 1 and 2 respectively [14]. Neurocognitive function was tested on all study participants at baseline and these results have been reported elsewhere, but briefly 32% and 52% of subjects showed summary deficit scores and HAND-like classification results compatible with cognitive impairment when standard normative data provided by the test manufacturers were used [11].
Test performance was considered not valid if participants decided to abandon the test before completion, in the case of investigators failing to comply with standard procedures according to the instructions or if the test was interrupted because of external factors. In addition, all scores were centrally monitored and extreme results were investigated and excluded if considered to be related to any of the situations listed above.
Statistical analysis
Only participants with complete cognitive testing results available were included in the analyses. Raw scores for each cognitive test were transformed to z-scores using the manufacturers’ normative data [12]–[14] adjusted for age (all tests) and years of education (CTT) by subtracting the mean and dividing by the standard deviation (SD) of test scores in reference populations. For the Grooved Pegboard test the z-score was obtained by taking the average of the z-scores for the dominant and non-dominant hands. Summary z-scores (NPZ-5) were then calculated by averaging z-scores of the 5 tests. For all individual test z scores and the NPZ-5, scores below zero denote below-average neurocognitive function compared to the reference population. Cognitive impairment was defined using two approaches: a) a summary NPZ-5 one SD below the mean of the normative dataset (i.e. < −1SD) (summary score) and b) test scores <−1SD in at least two individual tests (categorical scale).
Cochran’s Q test was used to ascertain heterogeneity of impairment across the individual tests. To investigate the relationship between all 5 tests, Pearson Correlation Coefficients were calculated. Principal component analyses (PCA), using correlation matrices, were performed to analyse the contribution of individual test scores, or tests scores clustering together, on the total variance of the 5 tests. First, a PCA without rotation and keeping all components was applied to get an overall picture. Principal components were orthogonal and uncorrelated. Second, rotated PCA using the varimax method was performed to reduce inter-component variance. Only those factors required to explain at least 80% of the total variance were retained in the rotated model which corresponded to 3 components based on the previous correlation analyses. Here, factors have only low or high values to facilitate interpretability of the components. Other rotation methods were also applied as was PCA using raw test scores instead of z-scores but results were very similar and therefore are not shown.
Results
Patient characteristics and acceptability of cognitive testing
Neurocognitive testing at baseline appeared to be highly acceptable. Of the 587 participants recruited into PIVOT, only one declined all tests. Overall, 548 (93%) participants produced valid results on all five tests and the proportion of missing, invalid or discontinued tests was not exceeding 2% in any of the tests except for the CTT-2 (4%). The demographic and clinical features of the 39 patients who were unable to produce complete tests did not differ from the general PIVOT population (Table 1).
Measurement of neurocognitive function
Mean NPZ-5 score was −0.72 (SD 0.98) and 178/548 (32%) of the study participants had a NPZ-5 score <−1, compatible with cognitive impairment. Mean Z-score for all tests were below the normative average and ranged from −0.20 (SD 1.46) for the CTT-2 to −1.36 (SD 1.96) for the GPT scores (Table 2); similarly the proportion of participants having Z-scores <−1SD was lowest for the CTT-2 test (22%) and highest (48%) for the GPT test (p<0.001). When cognitive impairment was defined as <−1 in at least two individual tests (categorical scale) 283 (52%) patients were impaired and of those patients 116 (21%), 84 (15%), 52 (9%), and 31 (6%) were <−1SD on 2, 3, 4, and all 5 tests, respectively (Figure 1).
Overall agreement between the two definitions of impairment was 79%, with 261 (48%) and 174 (32%) being classified as both normal and both impaired, respectively. Of the remainder, 4 patients (0.7%) with NPZ-5 score <−1SD underperformed in only one test/domain (the GPT test in all cases). The other 109 (20%) patients had Z-scores <−1SD on at least two cognitive domains but had NPZ-5 scores within the normal range.
Correlation between individual test Z-scores
27% of patients had z-scores <−1 on both HVLT-R learning and recall tests, 15% on HVLT-R learning but not recall test, and 8% on HVLT-R recall but not on learning test. 16% of patients had z-scores <−1 on both CTT-1 and CTT-2, 18% on CTT-1 but not CTT-2, and 5% in CTT-2 but not on CTT-1. Table 3 shows the correlation matrix of the 5 individual neurocognitive tests. There were strong correlations between a) HVLT-R learning and HVLT-R recall tests (r = 0.73), and b) CTT-1 and CTT-2 (r = 0.66). GPT was more correlated with the two CTTs (r = 0.33 with each) than with the two HVLT-Rs (r = 0.17 and 0.15) (Table 3).
Inter-test correlations were further analysed using PCA. The first component explained 47% of the total variance and has positive loadings of similar size on all variables, so could be interpreted as overall neurocognitive functioning. The second principal component explains 26% of the total variance and has negative loadings of similar size on both parts of the HVLT-R and positive loadings for the other tests, therefore differentiating performance in HVLT-Rs versus CTT-1, CTT-2 and GPT. The third principal component similarly differentiates performance in both CTTs versus GPT and explains 15% of the total variance. The fourth and fifth principal component only explain about 7% and 5%, and differentiate performances in CTT-1 versus CTT-2, and between the 2 parts of the HVLT-R, respectively. PCA with rotation and restricted to 3 components showed a clustering similar to the results of the correlation analyses: component 1 consisted of the two HVLT-R tests, component 2 of CTT-1 and 2, and component 3 of the GPT (Table 4).
Based on this finding, we recalculated cognitive impairment according to our categorical scale by reclassifying patients who underperformed only on two strongly correlated tests (HVLT-R both parts only (n = 36) or CTT-1 and 2 only (n = 14)) as ‘not impaired’. Prevalence of cognitive impairment would then be 43% rather than 52%. Similarly, when the two parts of the HVLT-R and CTT are averaged first and then used to calculate a NPZ-3 score with the GPT 36% of study participants showed NPZ-3<−1 whereas the proportion of participants with scores <−1 in two or three tests was 32%.
Discussion
NCI is an important morbidity in HIV-infected populations and the Frascati criteria is the most widely used and accepted method of diagnosis and classification. However there are concerns about some possible limitations of this approach, particularly in the case of asymptomatic or milder forms of the condition [3]–[5]. In this group of virologically suppressed patients on cART we performed standard neurocognitive tests and explored the effect of using different approaches to analysis and classification of impairment. These approaches yielded differing results. When using summary NPZ-5 scores, calculated by averaging Z-scores for all explored cognitive domains, the proportion of cognitively impaired patients was 32%, whereas using a categorical scale (defined by poor performance on at least two domains) it was 52%.
Summary scores have frequently been used to assess cognitive function in HIV-infected patients [15]–[17] and other medical conditions [18]. They provide single numerical results, particularly useful for monitoring change in cognition over time and have been utilised in the context of an RCT exploring interventions to treat HIV-associated NCI [19]. However, with a summary score such as NPZ-5, all individual test scores contribute equally, which may not reflect their individual relationship to overall neurocognitive dysfunction. In our study, neurocognitive testing generated data where some variables were highly correlated which may suggest that a simple average of individual tests Z-scores may not be an optimal method to identify cognitive impairment. Normalising scores by transforming them into Z-scores adds another limitation, as available manufacturers’ normative data (obtained with populations of unknown HIV status) may not be entirely appropriate for comparison with HIV-infected populations [11], [20]. Other methods also based on average scores that are used to measure cognitive function by normalising raw-scores, such as the global deficit score (GDS) and the clinical rating system (CR), may not be affected by the limitations generated by the lack of suitable normative population datasets, but these do not take into consideration the level of correlation between functional domain scores. When compared head to head, GDS and CR also generated differing results with the CR system classifying as impaired a larger proportion of patients than GDS [8].
By using 1 SD below the normative mean to define impairment, and assuming normal distribution of the data, 16% of individuals with normal cognitive function are expected to be classified as impaired for a single test. In categorical evaluation systems such as the Frascati criteria the probability of underperforming in two or more cognitive domains will depend on the number of tests performed (the higher the number of tests, the higher the probability of obtaining two or more abnormal results) and the level of correlation between them (there would be a high probability of failing two or more highly correlated tests or domains) and therefore, the false positive rate increases with multiple testing and multiple measures in a complex fashion [3], [21]. Using a more strict cut-off (e.g. <−1.5 or 2 SD) to define impairment (and perhaps limiting the number of measures included on the testing battery to explore performance on key cognitive domains, to those with low inter-correlation) may provide a more accurate approach to identify cognitive impairment. However, using a stricter cut-off may decrease the sensitivity of the testing battery, thus increasing the rate of false negatives. We chose a cut-off of −1SD for the summary score to define impairment, however, it should be noted that the standard deviation of an average is smaller than single test scores (again depending on the correlation between individual tests) and, therefore, also the proportion below −1SD in a summary score compared with a single test.
PCA is a useful tool to reduce a large set of correlated scores into clusters of independent factors which could help in better understanding their contribution in the overall variance of the different scores. PCA generates weighted average scores which could be used summarise performance on cognitive testing batteries. Although the PIVOT battery was brief and involved only one test for each of the five cognitive domains explored, PCA showed scores clustering in an expected fashion in three independent factors which explained 88% of the overall variance. If patients in whom test scores were <−1 only in two domains clustering together in a single PCA factor were reclassified as ‘not impaired’, 43% of our study population could be considered as cognitively impaired, a prevalence lower than previously reported [22], [23]. The first component from our PCA un-rotated model showed similar loadings on all tests suggesting similar contribution on the overall variance (Table 4) and therefore, may support the appropriateness of using a simple average of test scores to summarise cognitive function. However, the GPT showed a somewhat smaller contribution reflecting its lower correlation with the other tests, and it could be suspected that the cognitive domain it measures has less weight in a simple average. As an alternative approach to measure cognitive function we averaged the two parts of the HVLT-R and CTT to calculate an NPZ-3 score. Here, we observed a greater level of agreement between the two definition of NCI as 36% and 32% of study participants showed NPZ-3 <−1 and scores <−1 in two or three tests, respectively.
Our study population was large, homogenous, effectively treated and derived from a multi-centre study which confers some strength to our findings. However, our testing battery was brief and by no means comprehensive. Using a larger battery may generate different prevalence of NCI even in the same population, perhaps increasing sensitivity. However, depending of the level of correlation between tests, including a larger number of observations may increase the probability of ≥2 abnormal results [3]. Simioni et al reported a very high prevalence (84%) of NCI in effectively suppressed patients using a very large battery and adjusting their analysis for multiple comparisons but not for correlation between the 39 scores used [22]. On the other hand, a number of short (2–4 tests) batteries exploring each cognitive domain with a single test have demonstrated good sensitivity (60–87%) and specificity (83–91%) compared to a comprehensive test battery [24].
Prevalence of NCI is highly dependent not only on the definition used, but also on normative datasets utilised for analysis since a number of socio-demographic and cultural factors might impact performance on neuropsychological tests. In our original analysis we used normative data provided by the manufacturers of each test as well as ethnicity-adjusted sets for the CTT and the HVLT-R to calculate prevalence of NCI [11], [25]. Large cross-sectional studies have used additional normative datasets corrected by factors such as education, gender and age to calculate NCI have reported prevalences ranging between 52–58%, similar to the prevalence reported by us when the manufactures’ dataset was used (52%) [26], [27]. Because the main aim of this analysis was explore the discordance between the calculated prevalence of NCI with two different definitions of impairment, we decided to use the same normative datasets used for our initial analysis [11]. Exploring the PIVOT battery and clustering of test scores using PCA in a cohort of HIV-negative individuals would generate some valuable data.
In summary, our analysis showed discrepancies between the two definitions of NCI with the categorical scale being more likely to identify a larger proportion of study participants as impaired compared to the summary score. Taking into consideration correlations between individual test scores and adjusting test scores with methods such as PCA may have greater overall neuropsychological validity.
Acknowledgments
Statistical analysis performed by Dr Wolfgang Stöhr from the MRC-CTU.
We thank all the patients and staff from all the centres participating in the PIVOT trial.
PIVOT was funded by the NIHR Health Technology Assessment programme (project number 06/403/90). The views and opinions expressed herein are those of the authors and do not necessarily reflect those of the HTA programme, NIHR, NHS or the Department of Health.
Contributors
The members of the PIVOT Trial Team are: Barking Hospital: S Obeyesekera; Barts & The London: C Orkin; Birmingham Heartlands Hospital: S Taylor; Elton John Centre, Brighton: M Fisher; Central Middlesex Hospital: G Brook; Chelsea & Westminster Hospital: B Gazzard; Ealing Hospital: N O’Farrell; Avenue House/Ore Clin Eastbourne: K Aderogba; Edith Cavell Hospital: S Jebakumar; Gloucester Royal Hospital: A DeBurgh-Thomas; Homerton University Hospital: I Reeves; Harrogate District Hospital: C Lacey; James Cook Uni Hosp Middlesbrough: D Chadwick; King’s College Hospital: F Post; Leicester Royal Infirmary: A Palfreeman; Luton & Dunstable Hospital: T Balachandran; Manchester Royal Infirmary: V Lee; Royal Victoria Infirm Newcastle: E Ong; Newham University Hospital: R O’Connell; Norfolk and Norwich University Hospital: N David; North Manchester General: E Wilkins; North Middlesex University: J Ainsworth; Northwick Park Hospital: M Kapembwa; Derrifor Hospital Plymouth: Z Warwick; Royal Berkshire Hospital: F Chen; Royal Bournemouth Hospital: E Herieka; Royal Free Hospital, London: M Johnson; Royal Hallamshire Hospital: D Dockrell; Royal Liverpool University Hospital: N Beeching; Royal Victoria Hospital Belfast: S Pheng Quah; Southend University Hospital: J Day; Southmead Hospital Bristol: M Gompels; St James Hospital Leeds: J Minton; St George’s Hospital London: P Hay; St Mary’s Hospital, London: A Winston; St Mary’s Hospital, Portsmouth: V Harindra; St Thomas’ Hospital, London: J Fox; The Mortimer Market Centre, London: I Williams; University Hospital of Coventry: S Allan; Western General Edinburgh: C Leen; Queen Elizabeth Hospital Woolwich: S Kegg; Worcester Royal Hospital: M Roberts; York Teaching Hospital: C Lacey.
MRC Clinical Trials Unit: N Paton, A Arenas-Pinto, S Fleck, D Dunn, W Stöhr, K Scott, E Owen-Powell, I Kummeling, K Sanders, L Vivas; Trial Steering Committee: A Freedman (Chair), D Mercey, S Fidler, E Torok, B Cromarty, N Paton, B Gazzard, C Orkin, A Babiker; non-voting members: D Dunn, W Stöhr; Independent Data Monitoring Committee: T Peto (Chair), D Lalloo, A Phillips, R James.
Author Contributions
Conceived and designed the experiments: AAP AW DD AA NP. Performed the experiments: AAP AW JD RW SPQ JA SF. Analyzed the data: WS DD. Contributed reagents/materials/analysis tools: SF. Wrote the paper: AAP AW WS DD AA NP.
References
- 1. Robertson KR, Smurzynski M, Parsons TD, Wu K, Bosch RJ, et al. (2007) The prevalence and incidence of neurocognitive impairment in the HAART era. AIDS 21: 1915–1921.
- 2. Tozzi V, Balestra P, Bellagamba R, Corpolongo A, Salvatori MF, et al. (2007) Persistence of neuropsychologic deficits despite long-term highly active antiretroviral therapy in patients with HIV-related neurocognitive impairment: prevalence and risk factors. J Acquir Immune Defic Syndr 45: 174–182.
- 3. Gisslen M, Price RW, Nilsson S (2011) The definition of HIV-associated neurocognitive disorders: are we overestimating the real prevalence? BMC Infect Dis 11: 356.
- 4. Torti C, Foca E, Cesana BM, Lescure FX (2011) Asymptomatic neurocognitive disorders in patients infected by HIV: fact or fiction? BMC Med 9: 138.
- 5. Meyer AC, Boscardin WJ, Kwasa JK, Price RW (2013) Is It Time to Rethink How Neuropsychological Tests Are Used to Diagnose Mild Forms of HIV-Associated Neurocognitive Disorders? Impact of False-Positive Rates on Prevalence and Power. Neuroepidemiology 41: 208–216.
- 6. Winston A, Puls R, Kerr SJ, Duncombe C, Li PC, et al. (2012) Dynamics of cognitive change in HIV-infected individuals commencing three different initial antiretroviral regimens: a randomized, controlled study. HIV Med 13: 245–251.
- 7. Chang L, Tomasi D, Yakupov R, Lozar C, Arnold S, et al. (2004) Adaptation of the attention network in human immunodeficiency virus brain injury. Ann Neurol 56: 259–272.
- 8. Blackstone K, Moore DJ, Franklin DR, Clifford DB, Collier AC, et al. (2012) Defining neurocognitive impairment in HIV: deficit scores versus clinical ratings. Clin Neuropsychol 26: 894–908.
- 9. Carey CL, Woods SP, Gonzalez R, Conover E, Marcotte TD, et al. (2004) Predictive validity of global deficit scores in detecting neuropsychological impairment in HIV infection. J Clin Exp Neuropsychol 26: 307–319.
- 10. Antinori A, Arendt G, Becker JT, Brew BJ, Byrd DA, et al. (2007) Updated research nosology for HIV-associated neurocognitive disorders. Neurology 69: 1789–1799.
- 11. Winston A, Arenas-Pinto A, Stohr W, Fisher M, Orkin CM, et al. (2013) Neurocognitive Function in HIV Infected Patients on Antiretroviral Therapy. PLoS One 8: e61949.
- 12.
Brandt JB, Benedict RHB (2001) Hopkins Verbal Learning Test-Revised Professional Manual. Lutz, Florida, USA: PAR Psychological Assessment Resources, Inc.
- 13.
(2003) Grooved Pegboard User’s Manual. In: Europe LIC, editor. Loughborough, UK.: Lafayette Instrument Co. Europe.
- 14.
D’Elia L, editor (1996) Color Trails test Professional Manual. 2nd ed. Lutz, Florida. USA: PAR Psychological Assessmant Resources, Inc. 82 p.
- 15. Wright EJ, Grund B, Robertson K, Brew BJ, Roediger M, et al. (2010) Cardiovascular risk factors associated with lower baseline cognitive performance in HIV-positive persons. Neurology 75: 864–873.
- 16. Robertson KR, Su Z, Margolis DM, Krambrink A, Havlir DV, et al. (2010) Neurocognitive effects of treatment interruption in stable HIV-positive patients in an observational cohort. Neurology 74: 1260–1266.
- 17. Kanmogne GD, Kuate CT, Cysique LA, Fonsah JY, Eta S, et al. (2010) HIV-associated neurocognitive disorders in sub-Saharan Africa: a pilot study in Cameroon. BMC Neurol 10: 60.
- 18. Roalf DR, Moberg PJ, Xie SX, Wolk DA, Moelter ST, et al. (2012) Comparative accuracies of two common screening instruments for classification of Alzheimer’s disease, mild cognitive impairment, and healthy aging. Alzheimers Dement.
- 19. Nakasujja N, Miyahara S, Evans S, Lee A, Musisi S, et al. (2012) Randomized trial of minocycline in the treatment of HIV-associated cognitive impairment. Neurology.
- 20. Grant I (2008) Neurocognitive disturbances in HIV. Int Rev Psychiatry 20: 33–47.
- 21. Schretlen DJ, Testa SM, Winicki JM, Pearlson GD, Gordon B (2008) Frequency and bases of abnormal performance by healthy adults on neuropsychological testing. J Int Neuropsychol Soc 14: 436–445.
- 22. Simioni S, Cavassini M, Annoni JM, Rimbault Abraham A, Bourquin I, et al. (2010) Cognitive dysfunction in HIV patients despite long-standing suppression of viremia. AIDS 24: 1243–1250.
- 23.
Bonnet F, Amieva H, Marquant F, Bernard C, Bruyand M, et al. (2012) Cognitive disorders in HIV-infected patients: are they HIV-related? ANRS CO3 Aquitaine Cohort, Bordeaux, France, 2007–2009. AIDS.
- 24. Moore DJ, Roediger MJ, Eberly LE, Blackstone K, Hale B, et al. (2012) Identification of an abbreviated test battery for detection of HIV-associated neurocognitive impairment in an early-managed HIV-infected cohort. PLoS One 7: e47310.
- 25. Norman MA, Moore DJ, Taylor M, Franklin D Jr, Cysique L, et al. (2011) Demographically corrected norms for African Americans and Caucasians on the Hopkins Verbal Learning Test-Revised, Brief Visuospatial Memory Test-Revised, Stroop Color and Word Test, and Wisconsin Card Sorting Test 64-Card Version. J Clin Exp Neuropsychol 33: 793–804.
- 26. Bonnet F, Amieva H, Marquant F, Bernard C, Bruyand M, et al. (2013) Cognitive disorders in HIV-infected patients: are they HIV-related? AIDS 27: 391–400.
- 27. Heaton RK, Clifford DB, Franklin DR Jr, Woods SP, Ake C, et al. (2010) HIV-associated neurocognitive disorders persist in the era of potent antiretroviral therapy: CHARTER Study. Neurology 75: 2087–2096.