http://orcid.org/0000-0002-0680-6430
Figures
Abstract
Background
To be able to identify and monitor personal needs and goals of visually impaired young adults before and during rehabilitation trajectories, the Participation and Activity for Young Adults (PAI-YA) was developed involving young adults (18–25 years) and professionals as stakeholders. The psychometric properties of this new patient-reported outcome measure were investigated in order to develop an improved version.
Methods
Young adults registered at two low vision rehabilitation centers in the Netherlands were invited to complete the 141-item PAI-YA (n = 186) in a test-retest design. To select the best items for the PAI-YA, response frequencies were assessed and a graded response model (GRM) was fitted. Item reduction was informed by response frequencies, insufficient item information, and participants’ comments. Fit indices, item and person (theta) parameters were computed, after which known-group validity, concurrent validity, test-retest reliability and feasibility were studied.
Results
Response frequencies, violation of assumptions and item information informed the elimination of 81 items, resulting in a unidimensional PAI-YA showing satisfactory fit to the GRM. Known-group validity showed significant differences for visual impairment, financial situation, sex, educational situation and employment situation. Concurrent validity with (scales of) other questionnaires showed moderate to strong expected correlations. Test-retest reliability was satisfactory for all items (kappa 0.47–0.87), as was agreement (63.1–92.0%). Four items and one response option were added to increase feasibility.
Conclusions
This study contributes to the development and assessment of psychometric properties of the PAI-YA, which resulted in an improved 64-item version. Evidence was provided for construct validity, known-group validity, concurrent validity and test-retest reliability. These results are an important step in the development of a feasible instrument to investigate and monitor rehabilitation needs of visually impaired young adults, to structure the intake procedure at low vision rehabilitation services and to evaluate the effectiveness of rehabilitation.
Citation: Elsman EBM, van Rens GHMB, van Nispen RMA (2018) Psychometric properties of a new intake questionnaire for visually impaired young adults: The Participation and Activity Inventory for Young Adults (PAI-YA). PLoS ONE 13(8): e0201701. https://doi.org/10.1371/journal.pone.0201701
Editor: Adewale L. Oyeyemi, University of Maiduguri College of Medical Sciences, NIGERIA
Received: April 13, 2017; Accepted: July 20, 2018; Published: August 7, 2018
Copyright: © 2018 Elsman et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: Funding was received from the 'VIVIS programmaraad'. The funders has no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
The prevalence of visual impairments among young adults aged 18–25 years in the Netherlands is low, affecting approximately 0.1–0.2% of the population [1]. However, having a visual impairment has a major impact on daily functioning and quality of life [2–5]. Furthermore, persons with low vision report higher prevalence of loneliness and experience more participation restrictions [6, 7]. Young adults face important life transitions, characterized by the longing for increased independence and autonomy [8]. Various developmental tasks and goals are associated with the transition to adulthood, e.g. completing school, gaining employment, living independently and selecting a partner [9–13]. Having a disability can make this transition highly challenging and might interfere with goals associated with the transition, causing psychological distress [14] and disruption in the pursuit of one’s developmental tasks [15]. For example, although young adults with a visual impairment are less often employed compared to their sighted counterparts [16], this cannot be explained by differences in dropout and graduation rates [16, 17]. Furthermore, they experience difficulties in establishing and maintaining social, intimate and romantic relationships, which may threaten psychosocial development [9–12, 18, 19]. In addition, these individuals are more likely than their sighted peers (or persons with other disabilities) to live with their family after finishing school [20–22]. Multidisciplinary rehabilitation centers (MRCs) for the visually impaired can play a role in identifying the difficulties young adults experience, and offer guidance and rehabilitation to help overcome these difficulties [23].
In the Netherlands, MRCs use the Participation and Activity Inventory (PAI, formerly known as the D-AI) to identify personal goals of adult clients [24]. This questionnaire was developed and implemented in the Netherlands and originates from the Activity Inventory created by Massof et al. [25]. The PAI consists of the nine domains of the Activity and Participation component of the International Classification of Functioning, Disability and Health (ICF) from the World Health Organization [26, 27], and was found to be feasible and to have sound psychometric properties [28, 29]. The PAI can be used to identify needs, set goals and create an individualized rehabilitation plan. Moreover, the PAI provides insight into the effectiveness of rehabilitation and can be considered a patient-reported outcome measure (PROM) [30, 31].
Currently, the PAI is used for all adults in Dutch MRCs. However, the life stage of young adults aged 18–25 years is characterized by the transition to becoming an adult, making the extensive content of the PAI less applicable for this particular population [32]. Therefore, the PAI is less often used during the intake of young adults, who often receive a semi-structured interview instead [5, 32], increasing the risk of bias and overlooking specific needs, especially in domains which are not straightforward (e.g. relationships, and recreation/leisure). Furthermore, incorrect identification of needs might influence referral to rehabilitation programs and the quality of care provided [33]. To overcome this, a preliminary version of the PAI for Young Adults (PAI-YA) has recently been developed using a concept-mapping procedure, and has been further improved in a pilot study to assess its feasibility [5, 34].
The concept-mapping procedure and pilot study contributed to the face and content validity of the PAI-YA. The aim of the present study is to assess the psychometric properties of the PAI-YA and to develop an improved version.
Methods
The PAI-YA
The preliminary version of the PAI-YA comprises 141 items, which are grouped into 17 domains which were informed by the concept-mapping procedure: reading and visual aids (RV-5 items), mobility (MO-16 items), computer skills (CS-8 items), living independent and finances (LF-8 items), household (HH-7 items), self-care (SC-7 items), leisure time (LT-11 items), holiday and going out (HG-11 items), social relationships (SR-11 items), intimate/romantic relationships (RR-3 items), peer contact (PC-6 items), communication (CO-11 items), information/regulations (IR-12 items), study (ST-8 items), applying (AP-4 items), work (WO-6 items), and acceptance/self-consciousness (AS-7 items).
Each item is scored on a 4-point Likert scale with response options: not difficult (1), slightly difficult (2), very difficult (3), and impossible (4). The response option ‘not applicable’ is treated as a missing value. After each domain a question is asked to clarify the rehabilitation needs (Do you have any questions for the rehabilitation center regarding the topic … or would you like to receive rehabilitation for this?).
Study design and participants
Young adults aged 18–25 years who were enrolled for care at two Dutch MRCs (Royal Dutch Visio, and Bartiméus) at the time of this study or in the past, were invited to participate in the present study. Participants had to have adequate knowledge and understanding of the Dutch language and sufficient cognitive ability to participate in the study. Young adults with registered extensive cognitive impairment were excluded from the selection of young adults to be invited to participate by the low vision rehabilitation centers. Young adults with low vision from any cause were eligible and there was no restriction regarding visual performance. All potential young adults were sent a letter explaining the aim, procedure and duration of the study and asking whether they would agree to participate. Young adults who did not respond were telephoned to further inform them about the study and ask for their reasons not to participate.
Young adults who accepted to participate in the study were also telephoned in order to explain the aim, procedure and duration of the study again, and a researcher assessed through conversation whether they had sufficient cognitive abilities to administer the questionnaire. Participants were asked to fill in a socio-demographic questionnaire, the PAI-YA, and a self-constructed evaluation form. They were also asked to fill in Dutch versions of comparator instruments to assess (vision-related) quality of life and participation. Young adults had the option to fill in the questionnaires through a web-based survey questionnaire, a paper-and-pencil version, a telephone interview, or a face-to-face interview (home visit), whatever option was most convenient to them. Two weeks after initial completion, young adults were asked to fill in the PAI-YA and evaluation form again, using the same mode of administration. This time interval was chosen because participants probably have remained stable, but are unlikely to remember their previous answers because of the length of the PAI-YA [35]. Participants with an excessive number of missing responses in the PAI-YA (over 50) were excluded from analysis.
The study protocol was approved by the Medical Ethical Committee of the VU University Medical Centre, Amsterdam, the Netherlands. This study was performed in accordance with the ethical standards as laid down in the Declaration of Helsinki. Written informed consent was obtained from all included participants.
Other instruments administered
To assess concurrent validity of the PAI-YA, participants were asked to fill in four other instruments during the first administration. Two generic health-related quality of life instruments were administered, the Euroqol-5 Dimensions (EQ-5D) [36, 37] and the Short Form Health Survey (SF-36) [38]. The EQ-5D consists of five dimensions of functional impairment, whereas the items of the SF-36 can be assigned to eight scales. One instrument to assess participation was administered: the Impact on Participation and Autonomy questionnaire (IPA) [39], which consists of 32 items that can be assigned to four scales. Finally, one vision-related quality of life instrument was administered, the Low Vision Quality of Life questionnaire (LVQOL) [40], of which the 18-item unidimensional version validated by Van der Aa et al. was used [41]. For each instrument, a (scale)score was calculated following standard scoring rules [38, 40, 42–44]. Except for the EQ-5D, which is a formative scale, internal consistency reliability was assessed in the study population using Cronbach’s alpha for (subscales of) all instruments.
Item selection
All analyses were performed using SPSS version 22 [45] and R using the ltm package [46]. Demographic variables were analyzed using descriptive statistics. Acceptability of the PAI-YA was determined by looking at response rates for each individual item and the evaluation forms. Item reduction occurred through a structured approach, in which emphasis was placed on creating an instrument with reliable measurement properties while maintaining content validity. In order to create a reliable and valid questionnaire, items for potential deletion were chosen using an iterative process consisting of item analysis and application of IRT.
Item analysis.
Items with missing scores >40% were eliminated immediately from further analysis, while items with missing scores 20–40% were considered for elimination. Variability of the PAI-YA scores was assessed for each item using floor effects (percentage of respondents scoring at minimum level, i.e. (1) not difficult) and ceiling effects (percentage of respondents scoring at maximum level, i.e. (4) impossible). Floor and ceiling effects of items were considered to be present if >70% of the participants chose the lowest or highest possible response option. Item-pairs were flagged when inter-item correlation was >0.7, indicating that these items were similar and one of them is potentially redundant.
Application of IRT.
IRT is a collection of modeling techniques from modern measurement theory which provides a powerful framework to build instruments which are more efficient, reliable and valid [47]. IRT represent a number of statistical models that describe the association between a person’s ability (latent trait) and the probability of a person to choose a certain response option of an item in a multi-item scale [48]. The graded response model (GRM) is one of the most commonly used polytomous models to evaluate questionnaires with ordinal response categories, and it estimates a discrimination/slope (α) parameter and extremity (β) parameters [49, 50]. Application of IRT requires three assumptions:
- One of the critical assumptions of IRT is unidimensionality, i.e. a single latent trait in the PAI-YA such as “participation” explains the covariance of items [47]. To investigate unidimensionality, principal component analysis (PCA) was conducted. The number of factors was assessed using a scree plot and acceleration factor, which uses a numerical solution for determining the coordinate where the slope of the curve changes most abruptly [51].
- Items should display local independence, i.e. item responses are independent given their relationship to the latent trait [47]. Local independence was assessed by inspection of possible excess covariation (>0.25) among items in the residual matrix resulting from PCA. Item pairs which held excess covariation were considered candidates for deletion; the least performing item was selected.
- Monotonicity means that the chance a respondent endorses a successive threshold on the response scale is larger for those with a higher latent trait score [52]. The monotonicity assumption was evaluated by assessing manifest monotonicity using Mokken scale analysis. Items with non-monotonic increasing graphs were considered candidates for deletion. Moreover, Loevinger H coefficients [53] were calculated to assess scalability of items (see also [54, 55]). The coefficient is calculated as a function of Guttman errors between item pairs. A Loevinger H coefficient of <0.30 is considered unsatisfactory, and these items were candidate for deletion.
After checking IRT assumptions, IRT analysis was conducted to assess model fit and identify items contributing little information. Basic model fit of the GRM was assessed comparing a full model [47] with a constrained model [47, 56], which is nested within the full model and has equal slope parameters across items (analogous to the Rasch model). A Likelihood Ratio test was conducted to assess whether the full model outperformed the constrained model. Subsequently, functioning of items was initially assessed by examining item information in the latent trait measured. Item information refers to the information content of an item in relation to the total test information. Therefore, information represents reliability or measurement precision [47]. Items with low information (initially contributing <0.75% of total information) across the disability continuum were considered for elimination. The Item Information Curves (IICs) and Category Response Curves (CRCs) were evaluated to support decision making. The IIC shows the range of the underlying trait over which an item is most useful to distinguish between participants. The CRC shows the relationship between the latent trait and the probability of responding to a categorical item (i.e. it displays the most likely categorical response across the latent trait). When items had similar curves, the one with least information and/or holding information over the smallest range of the latent trait was considered for elimination. Items flagged by the item analysis, evaluation of assumptions and IRT analysis were potential candidates for deletion. However, previous qualitative studies [5, 34] and comments made by young adults were also taken into consideration, as well as the researchers’ opinions.
Psychometric properties of the PAI-YA
After the item selection process was completed, the reliability and validity of the PAI-YA was assessed. First, overall fit of the IRT model was assessed using the mirt package [57] yielding the M2 statistic and several fit indices: root mean square error of approximation (RMSEA) [58], standardized root mean square residual (SRMR), comparative fit index (CFI) [59] and Tucker-Lewis index (TLI). The CFI and TLI should be around 0.95 or higher, whereas the SRMR should be around 0.08 or lower and the RMSEA around 0.06 or lower [60]. The test information curve of the PAI-YA was presented, which refers to the underlying trait range over which an instrument is most useful to distinguish between participants. Subsequently, a person-item map was plotted for the items of the PAI-YA using the WrightMap package [61] to evaluate whether item difficulty matches respondents’ ability. The person-item map shows the distribution of person parameters (thetas) of participants on the left side of the map, and the distribution of item thresholds on the right side. Next, known-group validity was assessed by analyzing differences in thetas between relevant groups to reassure the PAI-YA can differentiate between groups: visual impairment, sex, age, nationality, mode of administration, financial situation, cognitive impairment, comorbidity, education in years, educational situation, and employment situation. Thetas of relevant groups were compared using independent samples t-test, ANOVA with posthoc Tukey-tests for multiple inter-group comparisons and Spearman’s correlation. Significant differences were expected for severity of the visual impairment, financial situation, comorbidity, educational status and employment status. To correct for other variables, multiple linear regression including all variables was performed. Concurrent validity was assessed by investigating the associations between the PAI-YA and (scales of) the SF-36, EQ-5D, LVQOL and IPA using Spearman’s correlation. Negative correlations were expected between the PAI-YA and (scales of) the SF-36, EQ-5D and LVQOL. Positive correlations were expected between the PAI-YA and scales of the IPA. Moderate (i.e. 0.3–0.5) to strong (i.e. >0.5) correlations [62] were expected between the PAI-YA and scales of the SF-36 and IPA, and strong correlations between the PAI-YA and the EQ-5D and LVQOL. Last, test-retest reliability of individual items was investigated using weighted kappa and percentage agreement. Kappa values >0.40 are considered moderate, >0.60 good and >0.80 very good [63]. Agreement is considered moderate for percentages of 60–74%, percentages of 75–89% are considered good and percentages ≥90% are considered excellent [64].
Results
Participant characteristics
Of the 1085 young adults invited for participation in the study, 218 (20.1%) agreed to participate and gave their written informed consent. Main reasons for non-participation were no time (31.7%), not interested (30.6%), and not able to participate because of cognitive impairment or not being visually impaired (15.5%), as indicated by data from non-responders with whom contact by telephone could be established. Of the respondents who agreed to participate, 193 completed the first PAI-YA questionnaire (88.5%). Seven participants were excluded from the analysis due to an excessive number of missing responses. Socio-demographic characteristics of the remaining participants are presented in Table 1. The retest was completed by 151 participants. Mean time between completion of test and retest was 30.58 ± 29.20 (range 11–171, median 18) days.
Item selection
Table 2 shows the distribution of participants over the response categories for the PAI-YA items. Thirteen items had missing scores >40% and were eliminated from further analysis; 24 items had missing scores 20–40% and were considered for elimination. Infrequent endorsement of the response option “impossible” motivated collapsing the response options “very difficult” and “impossible”. Assessment of floor and ceiling effects indicated no ceiling effects, whereas floor effects were found in 37 items and these were reconsidered for inclusion. High inter-item correlations (>0.7) were found between eight item pairs, and one item of each pair was flagged for elimination.
IRT assumptions.
The acceleration factor suggested the PAI-YA consisted of one latent dimension, i.e. participation. PCA components for the one-factor solution were all positive and mostly acceptable. Inspection of item and factor content gave no reason for multidimensional solutions; a two-factor solution did not substantially add to the explained variance (29% vs. 34%). Based on these results, the 128 items comprised a unidimensional scale which is sufficient for IRT analysis. However, excess item covariation (>0.25) among 102 item pairs indicated a violation of the local independence assumption. Initially, this was not considered to be a problem, since IRT models are often robust to violations of local dependence, especially when the scale consists of >10 items [47]. Although items which formed questionable pairs were considered candidates for deletion, actual deletion was done with reluctance; the least performing item was selected. Monotonicity analysis showed that two items were not monotonically increasing and scalability analysis showed that 21 items had an H coefficient below 0.3 (Table 2). These items were considered candidates for elimination.
IRT analysis.
Five iterations of item selection and IRT analysis were conducted to identify items to be removed due to low information, weak curves or subsequent violations of the IRT assumption. A total of 68 items were deleted, 34 in the first iteration, 12 in the second, 11 in the third and 11 in the fourth iteration, resulting in a 60-item PAI-YA. Table 2 shows the distribution of responses and reasons for item removal. Item information and less violations in assumptions occurred at each iteration, although in the 60-item PAI-YA two items violated the monotonicity assumption (CS8 and LT10) and one item had a H coefficient of 0.29 (SC5). Excess covariation remained among six item pairs (RV1-RV2, RV5-CS4, MO5-MO6, CS5-CS8, SR7-RR1, ST2-ST6), violating the assumption of local independence. Despite (small) violations in assumptions, further item reduction was considered to be unfavorable for maintaining content validity. The Likelihood Ratio Test indicated that the full GRM improved fit upon the polytomous Rasch model (LRT value = 162.36, df = 59, p = 0.001). Item information for the 60 items ranged from 1.40 to 4.66, and none of the items contributed <0.75% to the total information (155.42).
Psychometric properties of the PAI-YA
The fit indices reflected an adequate overall model fit of the 60 items: RMSEA = 0.057, SRMR = 0.072, CFI = 0.967 and TLI = 0.966. However, the M2 statistic was significant (2662.32 ± 88.06, p<0.05). Fig 1 presents the test information curve of the PAI-YA, providing information about its precision across the disability continuum. The PAI-YA seemed less precise at the extremes, but covered most of the disability continuum; however, the lower extreme was not entirely covered. Table 3 presents the parameter estimates for the remaining 60 items of the PAI-YA. Note that the items have been restructured into new or existing domains, as some domains only included a limited number of items after item removal, or even none. Restructuring of items was done based on the results of the concept-mapping study used to design the PAI-YA [5]. The slope estimates ranged from 0.84 to 2.61, indicating considerable variation in item discrimination. The location parameters for the 60 items reflect a sizeable range of underlying disability (-1.40 to 3.73), but the majority of item response categories were only endorsed by respondents who had higher than average levels of disability (i.e. θ >0), indicating that the PAI-YA as a whole is most useful in discriminating among individuals at the high end of the disability continuum. This is also shown in Fig 2, which shows the item-person map of the PAI-YA; items are distributed across the largest part of the disability continuum. The distribution of patients’ scores matches the distribution of items adequately, although there are no items to discriminate between patients at the low end of the disability continuum. ANOVA with post-hoc Tukey tests showed that persons with mild vision loss had significantly lower thetas than persons with low vision or who were blind (p = 0.001 and p<0.001, respectively), indicating that they were less disabled and the PAI-YA was able to discriminate between them. Furthermore using ANOVA and independent samples t-tests, significantly lower thetas were found for financial situation (lower theta for usually money left vs. not enough money, p = 0.016), mode of administration (lower theta for online vs. paper-and-pencil, and lower theta for telephone interview vs. paper-and-pencil, p = 0.009 and p = 0.022, respectively), males (p = 0.014), participants who did not have a cognitive impairment (p = 0.035), who were currently studying (p = 0.001), and who currently had a paid job (p = 0.005). A significant positive correlation was found for theta and age (p = 0.001). To correct for other variables, multiple linear regression showed significant associations between thetas and degree of visual impairment (p = 0.001), financial situation (p = 0.028) and sex (p = 0.029). Internal consistency for the LVQOL and subscales of the SF-36 and IPA was good, with Cronbach’s alpha of >0.7 (Table 4). As expected, correlations between the PAI-YA and (scales of) the SF-36, EQ-5D and LVQOL were negative and those between the PAI-YA and scales of the IPA were positive. The PAI-YA was significantly correlated with (all scales of) the SF-36, EQ-5D, IPA and LVQOL (p<0.01). The strength of the correlations were within the expected range, except for the SR and WO scales of the IPA, which were expected to correlate moderately with the PAI-YA but showed strong correlations instead (Table 4). Test-retest reliability of the PAI-YA items was satisfactory for all items (Table 3). In total, 11 items had moderate kappa values (>0.40), 46 had good values (>0.60) and 3 items had very good values (>0.80). On inspection, 31 items had moderate agreement (60–74%), 28 had good agreement (75–89%) and 1 item had excellent agreement (≥90%).
Evaluation and optimized version of the PAI-YA
Over 90% of the young adults were neutral to positive on several aspects of the PAI-YA (Table 5). Mean self-reported administration time of the PAI YA (including questions on demographic and clinical characteristics) was 45.52 ± 27.90 (range 8–180, median 40) min. After removal of the 81 items, it was assessed whether all rehabilitation questions young adults asked could still be identified with the 60-item PAI-YA. This was possible with the exception of rehabilitation questions related to having difficulty explaining the consequences of visual impairment to others (CO11), getting into contact with peers (PC1), driving a car (MO4) and taking care of nails (SC3). Therefore, it was decided to include these items in the PAI-YA, but not to score them because they are not part of the unidimensional scale. Moreover, young adults suggested to add another response option between ‘slightly difficult’ and ‘very difficult’; the response option ‘difficult’ was added. The PAI-YA thus consists of 64 questions which can be scored on a 4-point Likert scale with response options: not difficult (1), slightly difficult (2), difficult (3) and very difficult/impossible (4).
Discussion
In this study, psychometric properties were investigated to improve the PAI-YA: a questionnaire to identify and monitor the needs of young adults with a visual impairment. Following a developmental and piloting phase, this study led to item reduction and assessed the psychometric properties of the remaining items of the PAI-YA. The PAI-YA has good psychometric properties and covers a broad range of aspects of measuring the participation of young adults aged 18–25 years with a visual impairment. Because a national sample was used to assess its properties, the PAI-YA should be applicable across the Dutch population of young adults aged 18–25 years with visual impairment from any cause. Although the PAI-YA can serve as a template for use in other countries and languages, cross-cultural validation is recommended for use outside the Netherlands.
The item reduction process followed a rigorous strategy, combining aspects from classical test theory (CTT) and IRT, while also considering content validity because it should cover broad aspects of participation. IRT has added value in the process taken to reach conclusions, as it contains detailed item-level information, and its insights are most useful when complemented by results from CTT [13] and the target group’s perspective, as was done in this study. This comprehensive strategy led to the removal of 81 items, and re-inclusion of four of these items, resulting in a PAI-YA of 64 items (but only 60 are used in the scoring of the unidimensional scale).
The IRT assumptions seemed to hold for most of the 60 items. PCA showed that the PAI-YA consisted of one factor, as confirmed by the acceleration factor. Since IRT analysis requires a unidimensional scale, these results were important for continuation of the analyses. Although Kaiser’s eigenvalue >1 and Cattell’s scree plot methods are mostly used to determine the number of factors [65], simulation studies suggest that the acceleration factor outperforms these methods in determining the number of factors [66, 67]. There were six item pairs with local dependence, and the monotonicity assumption was violated for two items. However, a compromise had to be made between the aim of this study and the overall aim of the PAI-YA, i.e. to develop an instrument with strong psychometric properties versus developing a feasible instrument which can be used to investigate a broad range of rehabilitation needs to develop a rehabilitation plan. Therefore, content representation was carefully considered and further item reduction was deemed unfavorable.
The full GRM fitted the PAI-YA data well and all fit indices were satisfactory, although M2 was significant. The good dispersion of the discrimination parameters reaffirmed that the full GRM would better fit the PAI-YA data than the constrained GRM model, which assumes that all items have equal discrimination. The extremity parameters reflected a broad range of underlying disability for the 60 items. Standard errors of the discrimination and first extremity parameters were generally small (mostly 0.1–0.2 for the location parameters and 0.1–0.3 for the discrimination parameters), suggesting that these parameter estimates were relatively precise. However, standard errors for the second extremity parameter were much higher, which is caused by the infrequent endorsement of the response option ‘very difficult/impossible’ and the low number of participants. Although there are no definite answers regarding sample size requirements, general guidelines are available. For example, Tsutakawa and Johnson recommend a sample size of approximately 500 for accurate parameter estimates [68], whereas others suggest that 200 participants or less can be adequate [69, 70]. Nevertheless, large standard errors are considered less problematic when evaluating questionnaire properties, as in this study, compared to estimating accurate person measures (thetas).
The PAI-YA seemed better targeted to young adults having thetas at the higher end of the disability continuum. As shown by the item-person map, persons with low disability were less well discriminated by items of the PAI-YA, and there might be a need for more difficult items. This is also found in studies evaluating the psychometric properties of questionnaires intended for visually impaired children [71, 72]. However, the target population of the PAI-YA does not entirely match the population of respondents included in the present study, as all these participants were currently or previously enrolled for care at an MRC and are likely to have received rehabilitation services in the past. Although IRT models should be quite robust to differences in population characteristics [73], this may have slightly biased the results, as this group might not experience difficulties in participation and activities to the same extent as young adults with a visual impairment who seek rehabilitation services.
The PAI-YA is intended to investigate the rehabilitation needs of young adults aged 18–25 years with normal cognitive functioning who seek low vision rehabilitation services. Despite the inclusion criteria of our study, seven participants did not meet the age criterion of the PAI-YA, and eight participants reported to be cognitively impaired. Removing these participants from the dataset did not alter the item parameters pattern (data not shown). For six items the extremity β2 parameter changed more than 0.5 on the disability continuum (theta). However, all items except CS5 ‘using the computer’, which shifted to the left on the disability continuum (i.e. becoming more difficult), were removed from the PAI-YA during the first iterations of the psychometric analyses. Therefore inclusion of these participants did not have a large influence on the outcomes of this study.
The results of this study support the construct validity of the PAI-YA, having a unidimensional structure. Known-group validity showed significant differences in thetas for visual impairment, mode of administration, age, financial situation, sex, educational situation and employment situation. In most cases, the differences between these groups were expected. Although women more often score worse than men on quality of life instruments, and worse scores associated with increasing age were seen in a study on quality of life in visually impaired children [38, 43, 74], the differences between males and females and the differences between age groups were not expected. Because the difference found in mode of administration was based on only two participants who filled in the questionnaire using a paper-and-pencil version, this finding is likely to be distorted. Although a difference was expected between persons with and without comorbidity, this was not found (p = 0.089). Significant associations between thetas and severity of visual impairment, financial situation and sex remained in a multiple linear regression model, after correction for all other variables.
Concurrent validity was established by relating PAI-YA scores to scores of the SF-36, EQ-5D, LVQOL and IPA, with the expected correlations found. To our knowledge, the SF-36 and EQ-5D have not been validated specifically in a visually impaired population, although Malkin et al. investigated the responsiveness of the EQ-5D in a sample of participants with a visual impairment prior to and after rehabilitation [75]. They concluded that the EQ-5D is unresponsive as outcome measure for low vision rehabilitation and has poor sensitivity for discriminating people with a visual impairment with different levels of ability. However, both the EQ-5D and SF-36 have extensively been used across a range of populations and diseases, including visually impaired populations and ophthalmic conditions (e.g. [2, 76–78]), which makes the instruments suitable for comparisons.
Despite the long time-interval between completion of test and retest for some of the participants (i.e. less than 5% of the participants completed the retest at least 3 months after completion of the test), test-retest reliability was satisfactory; moderate to very good kappa values and moderate to excellent agreement was found in all items. Excluding participants who completed the retest more than a month after the first measurement only had minimal influence on kappa values and agreement. In 32 items the kappa values improved (two from moderate to good, three from good to very good), whereas in 21 items kappa values deteriorated (one from very good to good, five from good to moderate). The largest difference was found for item SR9 ‘participating in activities’, with an improvement from 0.47 to 0.55. Agreement slightly improved in 23 items and deteriorated in 37 items, however, differences were no larger than 5%. The large number of instruments and the length of the 141-item PAI-YA used in this study might have affected the quality of the data and might have introduced acquiescence bias, i.e. the tendency to opt for the same answer regardless of the content of an item [79]. However, no indications for acquiescence bias were found, as variability in responses was observed throughout the questionnaire and the number of missing responses did not increase towards the end of the questionnaire. Furthermore, 85% of the participants completed all instruments, and participants were informed before the start of the study on the expected duration to fill in the questionnaires.
The items of the PAI-YA are formatted on a 4-point Likert scale, which makes the questionnaire easy to administer. An online survey questionnaire should be the preferred mode of administration, because it is least susceptible to errors or leaving items blank, and is least time consuming to employees of MRCs. However, if filling in an online survey questionnaire is not feasible for the young adult because of the visual impairment or other reasons, a telephone interview is the second preferred option. Although young adults were mostly neutral to positive about various aspects of the 141-item PAI-YA, including time to complete, the administration time is expected to significantly reduce with item deletion. Since the PAI-YA is planned to be used by Dutch low vision MRCs, this will make use of the PAI-YA more feasible. When the PAI-YA is used in rehabilitation practice, it will be possible to re-assess the psychometric properties of the PAI-YA because of increased response rates. This will enable further item reduction, resulting in a shorter, more precise and user-friendly instrument. Besides rehabilitation practice, the PAI-YA might also be suitable for use in research investigating participation levels of visually impaired young adults.
Future research should examine whether the model still fits the data after the addition of the response option ‘difficult’. Moreover, differential item functioning (DIF) could not be assessed because of the small sample size. In addition, responsiveness of the PAI-YA should be investigated, which may lead to some more adaptations. In future, a computerized adaptive test (CAT) to tailor the PAI-YA to the abilities of visually impaired young adults might be developed and evaluated. This application might be particularly useful to monitor the personal needs and goals of visually impaired young adults, as it ensures that young adults do not have to fill in the full PAI-YA multiple times. Furthermore, the latent disability scores of young adults, which are preferred over sum scores because of their ability to handle missing data, might in future be made immediately available to rehabilitation services providers. This could help improve patient-centered care by increasing communication between young adults and MRC professionals, enhancing MRC professionals’ understanding of the needs of young adults, and facilitating the integration of young adults’ perspectives in their rehabilitation process. Moreover, it is expected that the PAI-YA will help to structure the intake procedure at low vision MRCs and can be used to evaluate the effectiveness of rehabilitation.
In conclusion, the results of this study show that the GRM model fits the PAI-YA data well. This study provides detailed information on item parameters and has resulted in a shorter version of the PAI-YA consisting of 64 items. This study provides evidence of construct validity, known-group validity, concurrent validity and test-retest reliability of the PAI-YA. Furthermore, it is an important step in the assessment of psychometric properties of the PAI-YA, and in the process to develop a feasible instrument to investigate and monitor the rehabilitation needs of visually impaired young adults.
Supporting information
S1 Dataset. Dataset used for analyses of the PAI-YA.
https://doi.org/10.1371/journal.pone.0201701.s001
(XLSX)
Acknowledgments
We greatly thank all participating young adults for their contributions. We acknowledge dr. C.F. Peeters from the department of Epidemiology and Biostatistics of Amsterdam UMC for his guidance and advice in the statistical analyses.
References
- 1.
Limburg H. Epidemiology of visual impairment in The Netherlands and a demographic review [Epidemiologie van visuele beperkingen en een demografische verkenning]. The Hague: 2007.
- 2. Langelaan M, de Boer MR, van Nispen RM, Wouters B, Moll AC, van Rens GH. Impact of visual impairment on quality of life: a comparison with quality of life in the general population and with other chronic conditions. Ophthalmic Epidemiol. 2007;14(3):119–26. pmid:17613846.
- 3. Seland JH, Vingerling JR, Augood CA, Bentham G, Chakravarthy U, deJong PT, et al. Visual impairment and quality of life in the older European population, the EUREYE study. Acta Ophthalmol. 2011;89(7):608–13. pmid:19925518.
- 4. van Nispen RM, de Boer MR, Hoeijmakers JG, Ringens PJ, van Rens GH. Co-morbidity and visual acuity are risk factors for health-related quality of life decline: five-month follow-up EQ-5D data of visually impaired older patients. Health Qual Life Outcomes. 2009;7:18. pmid:19243624; PubMed Central PMCID: PMC2654874.
- 5. Elsman EBM, Van Rens GHMB, Van Nispen RMA. Impact of visual impairment on the lives of young adults in the Netherlands: a concept-mapping approach. Disabil Rehabil. 2016;39(26):2607–18. pmid:27794635
- 6. Lamoureux EL, Hassell JB, Keeffe JE. The determinants of participation in activities of daily living in people with impaired vision. Am J Ophthalmol. 2004;137(2):265–70. pmid:14962415.
- 7. Huurre T, Aro H. The psychosocial well-being of Finnish adolescents with visual impairments versus those with chronic conditions and those with no disabilities. J Visual Impair Blin. 2000;94(10):625–37. PubMed PMID: WOS:000089653000003.
- 8.
Arnett JJ. Emerging adulthood: the winding road from the late teens through the early twenties. Oxford, England: Oxford University Press; 2004.
- 9. Kef S. Psychosocial adjustment and the meaning of social support for visually impaired adolescents. J Visual Impair Blin. 2002;96(1):22–37. PubMed PMID: WOS:000173363100003.
- 10. Kef S, Dekovic M. The role of parental and peer support in adolescents well-being: a comparison of adolescents with and without a visual impairment. J Adolescence. 2004;27(4):453–66. PubMed PMID: WOS:000223419000006. pmid:15288754
- 11. Kef S, Bos H. Is love blind? Sexual behavior and psychological adjustment of adolescents with blindness. Sex Disabil. 2006;24(2):89–100. PubMed PMID: WOS:000239112800003.
- 12.
Sacks SZ, Wolffe KE. Teaching social skills to students with visual impairments: research to practice. New York: AFB Press; 2006.
- 13.
Wehman P. Life beyond the classroom: transition strategies for young people with disabilities. Baltimore: Brookes Publishing Co.; 2006.
- 14. Rous B, Hallam R, Harbin G, McCormick K, Jung LA. The transition process for young children with disabilities—A conceptual framework. Infant Young Child. 2007;20(2):135–48. PubMed PMID: WOS:000245308500006.
- 15. Boerner K, Wang SW, Cimarolli VR. The impact of functional loss: Nature and implications of life changes. J Loss Trauma. 2006;11(4):265–87. PubMed PMID: WOS:000237604000001.
- 16. Capella-McDonnall M. Predictors of employment for youths with visual impairments: findings from the second national longitudinal transition study. J Visual Impair Blin. 2011;105:453–66.
- 17. Nagle KM. Transition to employment and community life for youths with visual impairments: Current status and future directions. J Visual Impair Blin. 2001;95(12):725–38. PubMed PMID: WOS:000172917600002.
- 18. Kroksmark U, Nordell K. Adolescence: The age of opportunities and obstacles for students with low vision in Sweden. J Visual Impair Blin. 2001;95(4):213–25. PubMed PMID: WOS:000168073200003.
- 19. Gold D, Shaw A, Wolffe K. The Social Lives of Canadian Youths with Visual Impairments. J Visual Impair Blin. 2010;104(7):431–43. PubMed PMID: WOS:000279569000006.
- 20.
Newman L, Wagner M, Cameto R, Knokey AM. The Post-High School Outcomes of Youth With Disabilities up to 4 Years After High School: A Report From the National Longitudinal Transition Study-2 (NLTS2). National Center for Special Education Research, 2009 Contract No.: NCSER 2009–3017.
- 21.
Wagner M, Newman L, Cameto R, Garza N, Levine P. After High School: A First Look at the Postschool Experiences of Youth with Disabilities. A Report from the National Longitudinal Transition Study-2 (NLTS2). 2005.
- 22. Wolffe K, Sacks SZ. The lifestyles of blind, low vision, and sighted youths: A quantitative comparison. J Visual Impair Blin. 1997;91(3):245–57. PubMed PMID: WOS:A1997WZ18200010.
- 23. Pfeiffer JP, Pinquart M. Attainment of Developmental Tasks by Adolescents with Visual Impairments and Sighted Adolescents. J Visual Impair Blin. 2011;105(1):33–44. PubMed PMID: WOS:000286352900004.
- 24.
Bruijning JE. Implementation and application of the D-AI. Amsterdam, the Netherlands2013.
- 25. Massof RW, Ahmadian L, Grover LL, Deremeik JT, Goldstein JE, Rainey C, et al. The activity inventory: An adaptive visual function questionnaire. Optometry Vision Sci. 2007;84(8):763–74. PubMed PMID: WOS:000249000200015. pmid:17700339
- 26.
WHO. The International Classification of Functioning, Disability and Health for Children and Youyh (ICF-CY). Geneva: World Health Organization, 2007.
- 27. Bruijning J, van Nispen R, Verstraten P, van Rens G. A Dutch ICF Version of the Activity Inventory: Results from Focus Groups with Visually Impaired Persons and Experts. Ophthal Epidemiol. 2010;17(6):366–77. PubMed PMID: WOS:000284410200004. pmid:21090911
- 28. Bruijning J, van Nispen R, Verstraten P, van Rens G. A Dutch ICF version of the Activity Inventory: results from focus groups with visually impaired persons and experts. Ophthalmic Epidemiol. 2010;17(6):366–77. pmid:21090911.
- 29. Bruijning JE, van Rens G, Knol D, van Nispen R. Psychometric Analyses to Improve the Dutch ICF Activity Inventory. Optometry Vision Sci. 2013;90(8):806–19. PubMed PMID: WOS:000330345000012.
- 30. Bruijning JE, van Rens GHMB, Fick M, Knol DL, van Nispen RMA. Longitudinal observation, evaluation and interpretation of coping with mental (emotional) health in low vision rehabilitation using the Dutch ICF Activity Inventory. Health Qual Life Out. 2014;12. doi: ARTN 182 PubMed PMID: WOS:000348402000001. pmid:25539603
- 31. Bruijning JE, van Rens GHMB, Knol DL, van Nispen RMA. Evaluation of Reading, Writing, and Watching TV Using the Dutch ICF Activity Inventory. Optometry Vision Sci. 2014;91(11):1360–71. PubMed PMID: WOS:000345117300018.
- 32. Van Leeuwen LM, Rainey L, Kef S, Van Rens GHMB, Van Nispen RMA. Investigating rehabilitation needs of visually impaired young adults according to the International Classification of Functioning, Disability and Health. Acta Ophthalmol. 2015;93(7):642–50. pmid:26109392
- 33. Buchbinder R, Batterham R, Elsworth G, Dionne CE, Irvin E, Osborne RH. A validity-driven approach to the understanding of the personal and societal burden of low back pain: development of a conceptual and measurement model. Arthritis Res Ther. 2011;13(5). doi: ARTN R152 PubMed PMID: WOS:000301174600012. pmid:21933393
- 34. Elsman EBM, van Nispen RMA, van Rens G. Feasibility of the Participation and Activity Inventory for Children and Youth (PAI-CY) and Young Adults (PAI-YA) with a visual impairment: a pilot study. Health Qual Life Outcomes. 2017;15(1):98. pmid:28490332; PubMed Central PMCID: PMCPMC5426000.
- 35.
De Vet HCW, Terwee CB, Mokkink LB, Knol DL. Measurement in Medicine: a practical guide. New York: Cambridge University Press; 2011.
- 36. Herdman M, Gudex C, Lloyd A, Janssen M, Kind P, Parkin D, et al. Development and preliminary testing of the new five-level version of EQ-5D (EQ-5D-5L). Qual Life Res. 2011;20(10):1727–36. pmid:21479777; PubMed Central PMCID: PMC3220807.
- 37. Janssen MF, Pickard AS, Golicki D, Gudex C, Niewada M, Scalone L, et al. Measurement properties of the EQ-5D-5L compared to the EQ-5D-3L across eight patient groups: a multi-country study. Qual Life Res. 2013;22(7):1717–27. pmid:23184421; PubMed Central PMCID: PMC3764313.
- 38. Aaronson NK, Muller M, Cohen PD, Essink-Bot ML, Fekkes M, Sanderman R, et al. Translation, validation, and norming of the Dutch language version of the SF-36 Health Survey in community and chronic disease populations. J Clin Epidemiol. 1998;51(11):1055–68. pmid:9817123.
- 39. Cardol M, de Haan RJ, van den Bos GA, de Jong BA, de Groot IJ. The development of a handicap assessment questionnaire: the Impact on Participation and Autonomy (IPA). Clin Rehabil. 1999;13(5):411–9. pmid:10498348.
- 40. Wolffsohn JS, Cochrane AL. Design of the low vision quality-of-life questionnaire (LVQOL) and measuring the outcome of low-vision rehabilitation. Am J Ophthalmol. 2000;130(6):793–802. pmid:11124300.
- 41. van der Aa HP, van Rens GH, Comijs HC, Margrain TH, Gallindo-Garre F, Twisk JW, et al. Stepped care for depression and anxiety in visually impaired older adults: multicentre randomised controlled trial. BMJ. 2015;351:h6127. pmid:26597263; PubMed Central PMCID: PMC4655616.
- 42. Versteegh MM, Vermeulen KM, Evers SMAA, de Wit GA, Prenger R, Stolk EA. Dutch Tariff for the Five-Level Version of EQ-5D. Value Health. 2016;19(4):343–52. PubMed PMID: WOS:000378094200008. pmid:27325326
- 43.
Stolk E, Krabbe P, Busschbach J. Using the internet to collect EQ-5D norm scores: a valid alternative? 24th Scientific Plenary Meeting of the EuroQol Group. 2009:153–64.
- 44.
Van Engelen E. Detailed explanation of the measurment instrument Impact on Participation and Autonomy (IPA) [Uitgebreide toelichting van het meetinstrument Impact op Participatie en Autonomie (IPA)]: Measurement instruments in healthcare [Meetinstrumenten in de zorg]; 2013 [cited 2017 23-1-2017]. Available from: https://meetinstrumentenzorg.blob.core.windows.net/test-documents/Instrument15/15_1_N.pdf.
- 45.
Corp. I. IBM SPSS Statistics for Windows. 22.0 ed. Armonk, NY: IBM Corp.; 2013.
- 46. Rizopoulos D. ltm: An R package for latent variable modeling and item response theory analyses. Journal of Statistical Software. 2006;17(5). PubMed PMID: WOS:000242545700001.
- 47. Edelen MO, Reeve BB. Applying item response theory (IRT) modeling to questionnaire development, evaluation, and refinement. Qual Life Res. 2007;16 Suppl 1:5–18. pmid:17375372.
- 48. Reeve BB, Hays RD, Chang CH, Perfetto EM. Applying item response theory to enhance health outcomes assessment. Quality of Life Research. 2007;16:1–3. PubMed PMID: WOS:000248371600001.
- 49. Samejima F. Estimation of Latent Ability Using a Response Pattern of Graded Scores. Psychometrika. 1969;34(4p2):1–&. PubMed PMID: WOS:A1969F051500001.
- 50.
Samejima F. Graded response model. In: Van der Linden W, Hambleton RK, editors. Handbook of modern item response theory. New York: Springer; 1997. p. 85–100.
- 51. M.G.R C, Gordon M. Determining the number of factors to retain in EFA: using the SPSS R-Menu v2.0 to make more judicious estimations. Practical Assessment, Reseach and Evaluation. 2013;18(8).
- 52. Pilkonis PA, Choi SW, Reise SP, Stover AM, Riley WT, Cella D, et al. Item banks for measuring emotional distress from the Patient-Reported Outcomes Measurement Information System (PROMIS(R)): depression, anxiety, and anger. Assessment. 2011;18(3):263–83. pmid:21697139; PubMed Central PMCID: PMC3153635.
- 53. Loevinger J. The technic of homogeneous tests compared with some aspects of scale analysis and factor analysis. Psychol Bull. 1948;45(6):507–29. pmid:18893224.
- 54. Meijer RR, Baneke JJ. Analyzing psychopathology items: a case for nonparametric item response theory modeling. Psychol Methods. 2004;9(3):354–68. pmid:15355153.
- 55. Sijtsma K, Meijer RR, van der Ark LA. Mokken scale analysis as time goes by An update for scaling practitioners. Pers Indiv Differ. 2011;50(1):31–7. PubMed PMID: WOS:000284501600006.
- 56.
Bond TG, Fox CM. Applying the Rasch model—fundamental measurement in the human sciences (2ed.). Mahwah: Lawrence Erlbaum Associates Publishers; 2007.
- 57. Chalmers RP. mirt: A Multidimensional Item Response Theory Package for the R Environment. Journal of Statistical Software. 2012;48(6):1–29. PubMed PMID: WOS:000305117700001.
- 58.
Steiger JH, Lind J. Statistically based tests for the number of common factors. Psychometrika Society Meeting; Iowa City1980.
- 59. Bentler PM. Comparative fit indexes in structural models. Psychol Bull. 1990;107(2):238–46. pmid:2320703.
- 60. Hu LT, Bentler PM. Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria Versus New Alternatives. Struct Equ Modeling. 1999;6(1):1–55. PubMed PMID: WOS:000208063500001.
- 61.
Irribarra TD, Freund R. Wright Map: IRT item-person map with ConQuest integration. 2014.
- 62.
Cohen J. Statistical power analysis for the behavioural sciences. 2nd ed. Hillsdale, New Jersey: Lawrence Earlbaum Associates; 1988.
- 63.
Altman DG. Practical statistics for medical research. London: Chapman and Hall; 1991.
- 64. Singh AS, Vik FN, Chinapaw MJM, Uijtdewilligen L, Verloigne M, Fernandez-Alvira JM, et al. Test-retest reliability and construct validity of the ENERGY-child questionnaire on energy balance-related behaviours and their potential determinants: the ENERGY-project. Int J Behav Nutr Phy. 2011;8. doi: Artn 136 PubMed PMID: WOS:000302078400001. pmid:22152048
- 65. Henson RK, Roberts JK. Use of exploratory factor analysis in published research—Common errors and some comment on improved practice. Educ Psychol Meas. 2006;66(3):393–416. PubMed PMID: WOS:000237628100003.
- 66.
Raiche G, Roipel M, Blais JG. Non graphical solutions for the Cattell's scree test. The International Annual Meeting of the Psychometric Society; Montreal2006.
- 67. Ruscio J, Roche B. Determining the Number of Factors to Retain in an Exploratory Factor Analysis Using Comparison Data of Known Factorial Structure. Psychol Assessment. 2012;24(2):282–92. PubMed PMID: WOS:000304842200002. pmid:21966933
- 68. Tsutakawa RK, Johnson JC. The Effect of Uncertainty of Item Parameter-Estimation on Ability Estimates. Psychometrika. 1990;55(2):371–90. PubMed PMID: WOS:A1990DT24400011.
- 69. Orlando M, Marshall GN. Differential item functioning in a Spanish translation of the PTSD checklist: Detection and evaluation of impact. Psychol Assessment. 2002;14(1):50–9. PubMed PMID: WOS:000174303300005.
- 70. Thissen D, Steinberg L, Gerrard M. Beyond Group-Mean Differences—the Concept of Item Bias. Psychological Bulletin. 1986;99(1):118–28. PubMed PMID: WOS:A1986A368000009.
- 71. Cochrane GM, Marella M, Keeffe JE, Lamoureux EL. The Impact of Vision Impairment for Children (IVI_C): validation of a vision-specific pediatric quality-of-life questionnaire using Rasch analysis. Invest Ophthalmol Vis Sci. 2011;52(3):1632–40. pmid:21178144.
- 72. Tadic V, Cooper A, Cumberland P, Lewando-Hundt G, Rahi JS, Vision-related Quality of Life g. Measuring the Quality of Life of Visually Impaired Children: First Stage Psychometric Evaluation of the Novel VQoL_CYP Instrument. PLoS One. 2016;11(2):e0146225. pmid:26918329; PubMed Central PMCID: PMC4768881.
- 73.
Cella D, Chang CH, Heinemann AW. Item response theory (IRT): applications in quality of life measurement, analysis and interpretation. In: Mesbah M, Cole BF, Ting Lee ML, editors. Statistical Methods for Quality of Life Studies: Springer US; 2002. p. 169–85.
- 74. Chadha RK, Subramanian A. The effect of visual impairment on quality of life of children aged 3–16 years. Br J Ophthalmol. 2011;95(5):642–5. pmid:20852314.
- 75. Malkin AG, Goldstein JE, Perlmutter MS, Massof RW, Low Vision Research Network Study G. Responsiveness of the EQ-5D to the effects of low vision rehabilitation. Optom Vis Sci. 2013;90(8):799–805. pmid:23851303.
- 76. Chia EM, Wang JJ, Rochtchina E, Smith W, Cumming RR, Mitchell P. Impact of bilateral visual impairment on health-related quality of life: the Blue Mountains Eye Study. Invest Ophthalmol Vis Sci. 2004;45(1):71–6. pmid:14691156.
- 77. Tsai SY, Chi LY, Cheng CY, Hsu WM, Liu JH, Chou P. The impact of visual impairment and use of eye services on health-related quality of life among the elderly in Taiwan: the Shihpai Eye Study. Qual Life Res. 2004;13(8):1415–24. pmid:15503837.
- 78. Scott IU, Smiddy WE, Schiffman J, Feuer WJ, Pappas CJ. Quality of life of low-vision patients and the impact of low-vision services. Am J Ophthalmol. 1999;128(1):54–62. pmid:10482094.
- 79.
Holbrook A. Acquiescence Response Bias. 2008 2018/01/17. In: Encyclopedia of Survey Research Methods [Internet]. Thousand Oaks, California: Sage Publications, Inc. Available from: http://methods.sagepub.com/reference/encyclopedia-of-survey-research-methods.