https://orcid.org/0000-0002-8007-0409
Figures
Abstract
Background
The Test of Visual Perceptual Skills-4th Edition (TVPS-4) is widely used for repeated measures of visual perception. This study aimed to examine the test-retest reliability, internal consistency, and practice effect of the TVPS-4 in people with schizophrenia receiving care in community psychiatric rehabilitation facilities.
Methods
A repeated assessment design was employed, involving 80 participants. Test-retest reliability was evaluated using the intraclass correlation coefficient (ICC) and Pearson’s r. Internal consistency was evaluated using Cronbach’s alpha (α). Minimal detectable change (MDC) values were calculated at 95%, 90%, and 80% confidence levels. The MDC% was determined based on the MDC with 95% certainty. Cohen’s d effect sizes and paired t-tests were utilized to assess the practice effect.
Results
The ICC and Pearson’s r of the overall scale were 0.93 and 0.95, respectively. The ICC and Pearson’s r of the seven subscales were 0.59–0.84 and 0.61–0.84, respectively. The α was 0.66–0.90 and 0.73–0.89 in the first and second assessment, respectively. The MDC95 (MDC%), MDC90, and MDC80 ranged from 4.7–16.3 (13.2–34.8%), 3.9–13.6, and 3.1–10.6, respectively. Cohen’s ds were 0.06–0.26 and paired t-test showed significant differences in scores of the overall scale and the four subscales (p < 0.05).
Citation: Lee S-C, Chiu E-C (2025) Reliability of the Test of Visual Perceptual Skills-4th Edition for people with schizophrenia. PLoS ONE 20(3): e0318148. https://doi.org/10.1371/journal.pone.0318148
Editor: Mohamed Ahmed Said,, King Faisal University, SAUDI ARABIA
Received: August 6, 2024; Accepted: January 11, 2025; Published: March 4, 2025
Copyright: © 2025 Lee, Chiu. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: This study was supported by Taipei City Government (grant no. 10901-62-047) and Taipei City Hospital (TPCH 110 66). The funder had no role in study design, collection, analysis, and writing the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Visual perception, also referred to as visual information processing, encompasses the cognitive processes involved in recognizing and interpreting visual stimuli in the environment, integrating visual information with other sensory systems, and generating appropriate responses [1, 2]. More than 60% of people with schizophrenia have visual perceptual deficits [3]. A previous study using functional magnetic resonance imaging showed impaired functioning of the visual pathway in people with schizophrenia [4]. The clinical manifestations of visual perceptual deficits in those with schizophrenia include difficulties in object discrimination, impaired perception of the entirety of objects, and challenges in distinguishing foreground from background elements [5–7]. Visual perceptual deficits affect their performance of functional activities [8]. Therefore, clinicians and researchers should use a visual perceptual measure to comprehend visual perception conditions, formulate related treatment plans, and track treatment progress effectively for people with schizophrenia.
The Test of Visual Perceptual Skills-4th Edition (TVPS-4) is a motor-free visual perceptual measure [9] and an updated version of the widely used TVPS-3, which assesses visual perception in adults [10, 11]. The TVPS-4 contains seven subtests: visual discrimination, visual memory, spatial relationships, form constancy, sequential memory, visual figure-ground, and visual closure. The TVPS-4 retains the items of the TVPS-3 and adds two items with lower item difficulty in each subtest, which is suitable for people with severe visual perception deficits. The scoring method for the TVPS-4 has been modified to discontinue a subtest when examinees obtain five out of seven items incorrectly [2]. The TVPS-4 has three following features. First, assessing visual perception requires perception, not the ability to reproduce perception through the motor process. Second, the easy-to-read design with only pictures raises the usefulness of this measure for people with cognitive impairments. Third, it has multiple subtests to assess diverse domains of visual perception, which is profitable for defining the strengths and weaknesses of visual perception comprehensively. Thus, the TVPS-4 is an adequate measure for assessing visual perception in people with schizophrenia in both clinical and research settings.
Temporal stability and internal consistency are essential psychometric properties of scores obtained from visual perceptual measures. Test-retest reliability estimates whether scores obtained from a measure yields consistent results over repeated assessments [12]. Internal consistency reflects how reliably items within a measure produce consistent results [13]. Random measurement error, an unpredictable factor affecting test results, should be taken into consideration when interpreting score changes. Minimal detectable change (MDC) refers to the minimum threshold to verify whether a score change of a person between repeated assessments is far from the random measurement error at a particular confidence level (e.g., 95%, 90%, and 80%) [14]. A score change below the MDC threshold indicates that it may be attributed to random measurement error. The practice effect is described as improvements in test results over repeated assessments owing to previous experiences in administration of the same measure [15]. To ensure meaningful interpretation of test results in both clinical and research settings, it is essential to examine test-retest reliability, internal consistency, and practice effect.
The TVPS-3 presented acceptable test-retest reliability and good internal consistency in people with stroke [16]. However, the test-retest reliability, internal consistency, and practice effect have not been investigated in previous editions nor the TVPS-4 for people with schizophrenia. Therefore, this study aimed to examine the test-retest reliability, internal consistency, and practice effect of the TVPS-4 in people with schizophrenia.
Methods
Participants
A total of eighty people with chronic schizophrenia were recruited from one psychiatric center in northern Taiwan between January 4th, 2021 and June 22nd, 2022. The inclusion criteria were as follows: (1) diagnosis of schizophrenia based on the Diagnostic and Statistical Manual of Mental Disorders, fifth edition; (2) age over 20 years; (3) stable drug compliance and adherence; and (4) willingness to participate in this study. The exclusion criteria were (1) diagnosis of brain injury and intellectual developmental disorders; (2) visual impairment; and (3) unstable psychiatric symptoms (with inconsistent scores on the Clinical Global Impressions-Severity scale [CGI-S] in two assessments). This study was approved by the research ethics committee of Taipei City Hospital (TCHIRB-10905018). Written informed consent was obtained from all participants before administering the assessments.
The eighty participants completed the two assessments and 42.5% were men, with an average age of 43.8 years. The average age of onset age for schizophrenia was 21.2 years. The details of the participants are presented in Table 1.
Procedures
One examiner administered the TVPS-4 twice with a two-week interval for each individual participant who met the recruitment criteria. The inclusion and exclusion criteria were assessed through a thorough evaluation of each participant’s medical history and current condition by the examiner. The examiner received at least 3 hours of training to familiarize herself with the measures (the test instructions, items, and scoring) and practiced administering the measures on two people with schizophrenia. The first author observed and conducted training for the examiner to ensure consistent and reliable assessments. According to the design of the TVPS-4, the subscales were conducted in the order of visual discrimination, visual memory, spatial relationships, form constancy, sequential memory, visual figure-ground, and visual closure. All participants were assessed in a quiet environment to avoid disturbance. On average, 3 to 5 participants were examined per day. The demographic data of the participants were gathered from medical charts.
Measures
The TVPS-4 assesses visual perception using seven subscales. In the visual discrimination subscale, participants are required to point one picture out of five that matches the target. In the visual memory subscale, the participants first memorize the target and then point it out on the next page. In the spatial relationships subscale, the participants point out a picture that is different from the others. In the form constancy subscale, the participants point out a picture that is similar to the target, regardless of any changes in size or position. In the sequential memory subscale, the participants memorize targets and their sequences and then point them out on the next page. In the visual figure-ground subscale, the participants point out the target that appears in different backgrounds. In the visual closure subscale, the participants point out a partial and incomplete picture of a complete target. Each subscale includes two practice items and 18 test items. Each item is rated as 0 (incorrect) or 1 (correct). The sum score (range 0-18) of the test items in each subtest represents the specific subscale function. The total score of the seven subtests (range 0-126) demonstrates general visual perception. A higher score indicates better visual perceptual ability [2].
The CGI-S is a clinician-rated scale that assesses the severity of psychiatric illness on a 7-point scale (1, no illness; 2, borderline illness; 3, mild illness; 4, moderate illness; 5, markedly ill; 6, severe illness; and 7, extremely severe illness). In this study, the CGI-S score was rated based on the examiner’s observation of the participants’ behaviors and self-reported symptoms. The score reflects the average severity level within 1 week [17, 18]. Sufficient convergent validity has been shown for the CGI-S in people with schizophrenia [19].
Data analysis
For examining test-retest reliability, intraclass correlation coefficient (ICC) and Pearson’s r were calculated. The criteria for ICC and Pearson’s r were as follows: ≤ 0.39, poor reliability; 0.40–0.59, moderate reliability; 0.60–0.79, good reliability; and ≥ 0.80, excellent reliability [20, 21]. For examining internal consistency, Cronbach’s alpha (α) was utilized. The standard of coefficient α was > 0.70 [13]. The MDC was estimated by multiplying the standard error of the measurement (SEM) by the z score and . The z score for the 95%, 90%, and 80% confidence interval were 1.96, 1.645, and 1.282, respectively. The SEM was computed by multiplying the standard deviation of the first assessment by
[14]. The MDC% was calculated by dividing the MDC95 by the maximum score and multiplied by 100. An MDC% < 30% represented an acceptable level of random measurement error [22].
We used effect size (Cohen’s d) to compute the extent of the practice effect due to repeated assessments. The Cohen’s d criteria were as follows: < 0.20, trivial effect size; 0.20–0.49, small effect size; 0.50–0.79, moderate effect size; and ≥ 0.80, large effect size [23]. A paired t-test (two-tailed, α=0.05) was applied to evaluate whether there was a significant difference between the two assessments.
Results
Table 2 displays the results of test-retest reliability and internal consistency for the TVPS-4. The ICC and Pearson’s r were 0.93 and 0.95 for the overall scale of the TVPS-4, respectively. The ICC and Pearson’s r values of the subscales were 0.59–0.84 and 0.61–0.84, respectively. The Cronbach’s α of the seven subscales in the first and second assessment were 0.66–0.90 and 0.73–0.89, respectively. The MDC95 (MDC%) for the overall scale was 16.3 (13.2%) (Table 3). The MDC90 and MDC80 were 13.6 and 10.6, respectively. The MDC95, MDC90, and MDC80 of the seven subscales were 4.7–6.3, 3.9–5.3, and 3.1–4.1, respectively. With the exception of two subscales (visual discrimination and form constancy), the MDC% of the other subscales were < 30%.
The Cohen’s d for the overall scale of the TVPS-4 was 0.19 (Table 2). The d values of the three subscales (visual discrimination, visual memory, and visual figure-ground) were > 0.20 and the d values of the other four subscales were 0.06–0.19. The paired t-test showed statistically significant differences between repeated assessments for the overall scale and four subscales (visual discrimination, visual memory, form constancy, and visual figure-ground) (p < 0.05). No significant differences were found (p = 0.128–0.489) in the remaining three subscales.
Discussion
To the best of our knowledge, this is the first study to examine the test-retest reliability, internal consistency, and practice effect of the TVPS-4 in people with schizophrenia. Our findings demonstrated excellent test-retest reliability and trivial practice effect for the overall scale. The TVPS-4 subscales revealed moderate to excellent test-retest reliability, sufficient internal consistency, and trivial to small practice effects. These results provide a solid foundation for the use of the TVPS-4 as a reliable assessment tool for visual perceptual skills in people with schizophrenia.
Four subscales of the TVPS-4 (form constancy, sequential memory, visual figure-ground, and visual closure) showed much more reliable ICC values (ICC = 0.70-0.84) in people with schizophrenia compared to those of the TVPS-3 (ICC = 0.55–0.77) in people with stroke [16]. These findings indicate that these four subscales demonstrated better test-retest reliability in assessing specific visual perceptual abilities for people with schizophrenia compared to the TVPS-3 in people with stroke, supporting its applicability in the schizophrenia population. The visual memory subscale showed relatively lower coefficient values of test-retest reliability and internal consistency, which were similar to reliability results of the TVPS-3 in people with stroke [16]. A possible reason for the lower reliability coefficient values may be the time constraint during testing. In the visual memory subscale, participants are required to memorize a target within 5 seconds and then identify the target within 20 seconds. People with schizophrenia show deficits in visual perception and require more time for visual processing [24,25]. Further studies are warranted to evaluate whether not timing or giving more time for the visual memory subscale could enhance the test-retest reliability and internal consistency.
In this study, the MDC values at the 95%, 90%, and 80% confidence levels in the overall scale and subscales of the TVPS-4 were provided. These values can assist clinicians and researchers when interpreting score changes (improvement or deterioration) in visual perception more reasonably. For example, a score change for an individual with schizophrenia exceeding the MDC95 (16.3) of the overall scale can be explained as a real change with 95% certainty. Substantial random measurement errors in two subscales (i.e., visual discrimination and form constancy, MDC% > 30%) were observed and the other subscales had MDC% of 26.1%–29.7%, which was close to 30%. The reason behind the substantial random measurement errors may be due to the long administration time which could result in fatigue and lead to a tendency of guessing answers. In this study, participants took 30–40 minutes to complete the seven subscales of the TVPS-4. The MDC% value was estimated at the 95% confidence level, which is more robust. People with schizophrenia may not achieve a real change with 95% certainty in the subscales. The MDC values at the other confidence levels (i.e., 90% and 80%) were estimated in this study. Future users could consider which confidence level they would like to assert to explain the score change for an individual with schizophrenia over two assessments.
Small effect sizes were observed, and the paired t-tests revealed significant differences between the two assessments in three subscales: visual discrimination, visual memory, and visual figure-ground. These findings suggest the presence of small practice effects in these three subscales. The results indicate that participants in this study may have easily memorized the answers or developed strategies to identify them. Two possible methods to reduce the practice effects are as follows: (1) repeating similar items more times to reach a plateau phase in the practice effect [26]; and (2) developing a computerized adaptive test, which can decrease the need to administer the same item in repeated assessments. Moreover, a computerized adaptive test can shorten the administration time which may improve the test-retest reliability of the TVPS-4. Future studies could consider developing a computerized adaptive test of visual perception to assess people with schizophrenia in a more precise manner.
This study had two limitations. First, we used a convenience sample recruited from a psychiatric center in northern Taiwan, which may restrict the generalizability of our results. Second, we did not examine the validity (e.g., construct validity with larger sample size), limiting the explanation of the visual perceptual constructs of the TVPS-4. Further validation is needed to confirm the construct validity of the TVPS-4 in people with schizophrenia.
Conclusions
The TVPS-4 demonstrates acceptable test-retest reliability, satisfactory internal consistency, and trivial to small practice effect in people with schizophrenia. The overall scale is useful for assessing general visual perception and tracking progress over the course of intervention. However, two subscales revealed substantial random measurement error and three subscales demonstrated small practice effects. These subscales should be used cautiously when explaining specific visual perception abilities in clinical and research settings.
References
- 1.
Warren M. Evaluation and treatment of visual deficits following brain injury. St Louis (MO): Mosby Elsevier; 2006.
- 2.
Martin NA. Test of Visual Perceptual Skills-4th edition. Novato (CA): Academic Therapy Publications; 2017.
- 3. Jurišić D, Ćavar I, Sesar A, Sesar I, Vukojević J, Ćurković M. New Insights into Schizophrenia: a look at the eye and related structures. Psychiatr Danub. 2020;32(1):60–9. pmid:32303031
- 4. Martínez A, Hillyard SA, Dias EC, Hagler DJ Jr, Butler PD, Guilfoyle DN, et al. Magnocellular pathway impairment in schizophrenia: evidence from functional magnetic resonance imaging. J Neurosci. 2008;28(30):7492–500. pmid:18650327
- 5. Doniger GM, Silipo G, Rabinowicz EF, Snodgrass JG, Javitt DC. Impaired sensory processing as a basis for object-recognition deficits in schizophrenia. Am J Psychiatry. 2001;158(11):1818–26. pmid:11691687
- 6. Gabrovska VS, Laws KR, Sinclair J, McKenna PJ. Visual object processing in schizophrenia: evidence for an associative agnosic deficit. Schizophr Res. 2003;59(2–3):277–86. pmid:12414085
- 7. Malaspina D, Simon N, Goetz RR, Corcoran C, Coleman E, Printz D, et al. The reliability and clinical correlates of figure-ground perception in schizophrenia. J Neuropsychiatry Clin Neurosci. 2004;16(3):277–83. pmid:15377734
- 8. Green MF, Hellemann G, Horan WP, Lee J, Wynn JK. From perception to functional outcome in schizophrenia: modeling the role of ability and motivation. Arch Gen Psychiatry. 2012;69(12):1216–24. pmid:23026889
- 9. Brown T, Peres L. An overview and critique of the Test of Visual Perception Skills - fourth edition (TVPS-4). Hong Kong J Occup Ther. 2018;31(2):59–68. pmid:30643493
- 10. Brown T, Elliot S, Bourne R, Sutton E, Wigg S, Morgan D, et al. The convergent validity of the Developmental Test of Visual Perception—Adolescent and Adult, Motor-Free Visual Perception Test—third edition and Test of Visual Perceptual Skills (non-motor) — third edition when used with adults. Br J Occup Ther. 2012;75(3):134–43.
- 11. Lee J, Chiu E, Lee S. A review of psychometric properties of commonly used visual perception assessment tools. J Taiwan Occup Ther Assoc. 2020;38:7–35.
- 12. Hobart JC, Riazi A, Lamping DL, Fitzpatrick R, Thompson AJ. Improving the evaluation of therapeutic interventions in multiple sclerosis: development of a patient-based measure of outcome. Health Technol Assess. 2004;8(9):iii, 1–48. pmid:14982653
- 13. Chiu E-C, Hsueh I-P, Hsieh C-H, Hsieh C-L. Tests of data quality, scaling assumptions, reliability, and construct validity of the SF-36 health survey in people who abuse heroin. J Formos Med Assoc. 2014;113(4):234–41. pmid:24685299
- 14. Haley SM, Fragala-Pinkham MA. Interpreting change scores of tests and measures used in physical therapy. Phys Ther. 2006;86(5):735–43. pmid:16649896
- 15. Lee S-C, Chien T-H, Chu C-P, Lee Y, Chiu E-C. Practice effect and test-retest reliability of the Wechsler Memory Scale-Fourth Edition in people with dementia. BMC Geriatr. 2023;23(1):209. pmid:37003982
- 16. Chiu E-C, Wu W-C, Chou C-X, Yu M-Y, Hung J-W. Test-retest reliability and minimal detectable change of the Test of Visual Perceptual Skills-Third Edition in patients with stroke. Arch Phys Med Rehabil. 2016;97(11):1917–23. pmid:27240434
- 17.
Guy W. ECDEU assessment manual for psychopharmacology, revised. Rockville (MD): US Department of Health, Education and Welfare, National Institute of Mental Health; 1976.
- 18. Busner J, Targum SD. The clinical global impressions scale: applying a research tool in clinical practice. Psychiatry. 2007;4(7):28–37. pmid:20526405
- 19. Turkoz I, Fu D-J, Bossie CA, Sheehan JJ, Alphs L. Relationship between the clinical global impression of severity for schizoaffective disorder scale and established mood scales for mania and depression. J Affect Disord. 2013;150(1):17–22. pmid:23517884
- 20. Chiu E-C, Lee S-C. Test-retest reliability of the Wisconsin Card Sorting Test in people with schizophrenia. Disabil Rehabil. 2021;43(7):996–1000. pmid:31361972
- 21. Esen SY, Türkmen C, Düger T. Cross-cultural adaptation, reliability and validity of the Turkish version of the Smart Tools Proneness Questionnaire (STP-Q). PLoS One. 2024;19(10):e0309299. pmid:39378201
- 22. Huang S-L, Hsieh C-L, Wu R-M, Tai C-H, Lin C-H, Lu W-S. Minimal detectable change of the timed “up & go” test and the dynamic gait index in people with Parkinson disease. Phys Ther. 2011;91(1):114–21. pmid:20947672
- 23.
Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale (NJ): Lawrence Erlbaum Associates; 1988.
- 24. Rassovsky Y, Horan WP, Lee J, Sergi MJ, Green MF. Pathways between early visual processing and functional outcome in schizophrenia. Psychol Med. 2011;41(3):487–97. pmid:20482936
- 25. Sanfratello L, Aine C, Stephen J. Neuroimaging investigations of dorsal stream processing and effects of stimulus synchrony in schizophrenia. Psychiatry Res Neuroimaging. 2018;278:56–64. pmid:29884441
- 26. Chiu E-C, Hung J-W, Yu M-Y, Chou C-X, Wu W-C, Lee Y. Practice effect and reliability of the motor-free visual perception test-4 over multiple assessments in patients with stroke. Disabil Rehabil. 2022;44(11):2456–63. pmid:33103489