Study design

This study extends our previous work reporting the development of IBs measuring PF, PM and SR [12]. In the current study, we focused specifically on the methodology and results for testing linguistic and ethnic DIF for each item bank, and the evaluation of IBs’ efficiency using CAT simulations. This study was approved by the SingHealth Centralized Institutional Review Board (Ref 2016/2337) and all participants signed written informed consent. The study was conducted according to the guidelines of the Declaration of Helsinki.

Subject recruitment and data collection

We recruited Singapore citizens or permanent residents from the community and the Singapore General Hospital from 13 September 2016 to 27 December 2016. We sampled 75% English and 25% Mandarin speaking subjects separately. Within each language sampling frame, we used purposive sampling to select subjects based on age, gender, ethnicity and presence of chronic illnesses. S1 Table provides the list of chronic illnesses based on the Singapore Burden of Disease study [17]. Subjects were classified as well, mildly unwell and unwell based on the number and severity of chronic illnesses. We excluded subjects who were unable to have a meaningful discussion with our interviewers, due to mental illnesses or cognitive impairment. In order to include subjects from a wide spectrum of health, we predefined the proportion of subjects to be recruited as 35% well, 15% mildly unwell and 50% unwell.

Subjects from the community were recruited by a survey company from a residential household sampling frame of public housing, in which more than 80% of Singaporeans reside. The primary sampling unit were plots of land with approximately equal numbers of households, stratified according to geographic location and dwelling type. Households in each primary sampling unit were selected based on fixed route rules and skip patterns based on pre-specified ethnic and age quotas. Only one respondent per household was selected to participate in the study. Three call attempts were made at different times of the day with at least 1 call on a non-work day (Saturday or Sunday) to improve the response rate. This sampling method has been used in the Singapore Health Survey [18]. The response rate of the survey was computed using the standard set by the Council of American Survey Research Organization [19], which is defined as the number of completed interviews divided by the number of eligible reporting units in sample. Assessments were completed on-site at subjects’ home. The subject was reimbursed for their time on completion of the interview.

Interviewers administered the SHAWS items, with each participant completing the questions from the either the PF, PM or SR IBs [2]. Interviewers administered the survey so that illiterate subjects could be included. The PF IB measures physical functioning and comprised 55 items, each with a 5-level response option adapted from Patient-Reported Outcomes Measurement Information System (PROMIS). The PM IB measures the positive mindset of a subject and comprised of 48 items while the SR IB measures the depth and meaningfulness of the human connections the subject has and comprised of 30 items [12, 14]. The response options were “Never”, “Seldom”, “Sometimes”, “Usually” and “Always” for items on frequency and “Not at all”, “Mildly”, “Moderately”, “Quite a lot” and “Extremely” for items on intensity [15].

Demographics including age, gender, ethnicity, education and current medical status and the participant-reported global assessment of health were also collected.

Assessment of DIF for language and ethnicity of the PF, PM and SR IBs

Each IB was analyzed using Rasch analysis with Winsteps 4.50 software (Winsteps, Beaverton, OR) using the Andrich single-rating scale model [20]. Rasch analysis estimates the relative difficulty of items (item measures) and relative abilities of respondents (person measures) and aligns them on an interval level scale, and these item calibrations are used for the CAT algorithm. Rasch analysis also provides substantial information about a scale’s psychometric properties, including assessment of DIF. We took a DIF contrast of ≥0.64 logits with a corresponding significant Rasch-Welch probability [21]. Bonferroni correction was applied for multiple DIF tests (p-value / no. items) [22].

Through DIF assessment, we identified items with DIF for ethnicity (Chinese, Malay, Indian) and/or language (English, Chinese) and considered them for removal. Item removal was indicated if 1) the DIF was particularly problematic (i.e., DIF contrast [DC] substantially greater than ±0.64 logits after Bonferroni correction), 2) item content was deemed non-essential or covered by other similar items, and/or 3) there were obvious reasons for DIF to occur, i.e., culturally specific activity). All decisions involving item deletion were reviewed and approved by the research team, comprised of members with content development and psychometric expertise (EF, EL, JT) and/or clinical expertise (JT).

Assessment of the efficiency of the item banks using CAT simulations

CATs for the PF, PM and SR IBs were developed and hosted on an online CAT testing platform (PROMinsight) using Concerto open-source software [23]. We then conducted simulations to assess the efficiency of our Winsteps threshold calibrations (JMLE; Joint Maximum Likelihood Estimation method) and associated CAT algorithm [24] in 1000 simulated respondents using R Statistical Computing Environment [25]. Individual packages were loaded in R to conduct IRT including CAT simulations (“catR”) [26]. Simulations were based on a standard normal distribution (M = 0, SD = 1) and used the Rating-Scale Model (RSM), the ML (maximum likelihood) estimator and the Maximum Fisher Information (MFI) item selection criteria. We determined the average number of items required based on three different stopping rules: SEM of 0.30 representing “high precision”, 0.387 representing “moderate precision”, and 0.521 representing “moderate-low precision”. High precision (reliability ~0.91) may be appropriate for high-stakes testing (e.g. clinical trials); moderate precision (reliability ~0.85) and moderate-low precision (reliability ~0.73) may be appropriate for lower-stakes testing such as routine clinical monitoring of patients and where testing brevity is important [27]. Model fit was assessed using the root mean square error (RMSE) and level of bias between true and estimated ability levels (low values are desirable). We also calculated the Pearson correlation coefficient between the IBs and CAT simulated person measure estimates. We hypothesized high (r ≥ 0.85) and moderate-high (0.75 ≥ r < 0.85) correlations for simulations with the high and moderate precision stopping rules, respectively. Results across the spectrum of the latent trait are summarized in Table 3. Results for specific ‘ability’ levels are provided in deciles (D1-D10 n = 100 each, where D1 and D10 includes simulees at the lowest and highest ‘ability’ levels, respectively; S2–S4 Tables).

Demographics

Table 1 summarizes the participants’ demographics of the SHAWS study. Out of 2013 subjects, 1492 (74.1%) and 521 (25.9%) completed the English and Chinese versions, respectively. The majority of recruited subjects were Chinese (50.8%), females (51.0%) and had completed at least 10 years of education (i.e. secondary school) (76.5%). The number of subjects who completed the PF, PM and SR IBs were largely equal [PF:PM:SR (n = 671, 670, 672 respectively)].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Sociodemographic characteristics of the study subjects.

https://doi.org/10.1371/journal.pone.0298141.t001

DIF assessment for the three IBs

Initial Rasch analysis of the PF IB revealed two items with DIF for ethnicity and language (Table 2). The first was item 15 “I am able to use a pair of chopsticks” (DC -2.23 Chinese-Malay and -1.94 Chinese Indian) indicating that Chinese respondents found this item easier to do than their Malay and Indian counterparts, regardless of their underlying level of physical functioning. The second was item 25 “I am able to roll onto my stomach, while lying in bed, without assistance from other people” which was significantly easier for Chinese participants compared to Malays (DC -0.97) and Indians (DC -0.99). We found that two items were harder to endorse for the English version compared to the Chinese version of the IB (DC 2.07 and 1.30, respectively), with this difference unrelated to their underlying level of physical function. In addition, item 42 “I am able to run a full marathon (42 kilometers)” displayed DIF for ethnicity, while item 61 “I am bedridden” displayed DIF for language (Table 2). Upon deletion of items 15 and 25, DIF for item 61 was no longer apparent, DIF for ethnicity for item 42 remained, and DIF for ethnicity for item 45 “I am able to carry 2 bags filled with groceries” had emerged, with Chinese people finding this item harder to do than Malays (DC 0.71). Because the DCs marginally exceeded the cut-off value of 0.64 logits for items 42 and 45, and the content was deemed to be important and not covered by other items, these two items were retained.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Differential item functioning (DIF) for ethnicity and language for the physical functioning, positive mindset, and social relationship item banks.

https://doi.org/10.1371/journal.pone.0298141.t002

Initial Rasch analysis of the PM IB revealed six items with both DIF for ethnicity and language (items 25, 26, 35–38), two items with DIF for ethnicity (items 39, 40), and two items with DIF for language (item 1 and 48; Table 2). These items also displayed substantial misfit (high noise-to-signal ratio) and were iteratively deleted to resolve the misfit and DIF. Following this, DIF emerged for language for items 5 and 46 and for ethnicity for item 24; however, these items were retained as their DC marginally exceeded the target cut-off and content was considered important by the study team.

For the SR IB, Rasch analysis revealed two items with DIF (Table 2), one each for language (item 50) and ethnicity (item 48). However, as their DC values marginally exceeded the target cut-off and item content was felt to be important, they were retained.

CAT Simulations

Overall, a good model fit (low RMSE and bias values) was achieved for all simulations (Table 3). With an 0.3 SEM stopping rule, the mean number of items administered across the spectrum of the latent trait was 8.5, 21.6 and 14.5 for the PF, PM, and SR IBs, respectively (Table 3), with the proportion of participants satisfying the stopping rule ranging between 89% (PM) to 99.6% (PF). For ‘moderate’ precision (stopping rule SEM 0.387), the mean number of items administered across the spectrum of the latent trait was 5.1 for PF, 13.0 for PM and 8.0 for SR, with 100% of participants meeting the stopping rule for all three IBs (Table 3). Finally, when the stopping rule was SEM 0.521 (‘moderate-low precision’), the mean number of items administered was very low (3.1 for PF, 5.3 for PM and 4.1 for SR) and 100% of participants met the stopping rule for all three IBs (Table 3). When looking at specific participant ‘ability’ levels, the mean number of items required was generally lowest for participants in the middle deciles (i.e., D3-D8) and highest for those at the very ‘unable’ (D1-D2) and very ‘able’ (D9-D10) ends of the ability spectrum (S2–S4 Tables).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. CAT simulation results for the physical function, positive mindset, and social relationship item banks.

https://doi.org/10.1371/journal.pone.0298141.t003

Correlations between person measures generated by the CATs compared to the full IBs were moderate to high, ranging between 0.73–0.85 for SEM 0.3, 0.75–0.84 for SEM 0.387 and 0.73–0.83 for SEM 0.521.

Strengths and limitations

Our study has several strengths. First, we had a large sample size and this is important to obtain accurate item-parameter estimates [35]. Second, we used purposive sampling to ensure applicability to age groups, gender and ethnicities, allowing a thorough investigation of ethnic and linguistic DIF. Our studies also had the following limitations. First, the IBs were not administered in Malay or Tamil which means that there may be undetected item bias for these languages. Second, we adopted an item removal approach to resolve DIF, whereas other studies have suggested managing cultural DIF using item anchoring, where parameters of items with DIF are allowed to vary, while item calibrations for the other items which displayed DIF remain constant across each country [36]. However, many of the items with DIF in our study also displayed substantial item misfit suggesting that they were not contributing to measurement and, as such, required deletion in any case. Third, the efficiency of the IBs is limited at the extremes of ability levels. We will work towards replenishing the IBs as part of our future work to better measure subjects at the very high or very low ends of the intended latent traits. Next, our data were collected pre COVID and it is possible that COVID-19 may have changed participants’ perceptions of quality of life. However, given that our IBs assess general health-related QoL and do not specifically tap into COVID-related issues, we expect the impact of COVID-19 on our IBs to be minimal. Finally, we used the standard normal distribution (M = 0, SD = 1) to run our simulations. Given that our IB calibrations were based on a relatively able participant sample with high mean person measures, simulating latent traits from a normal distribution with these parameters may have affected the accuracy of our results.