Development and psychometric properties of the Japanese Consumer Assessment of Healthcare Providers and Systems Clinician & Group Survey (CG-CAHPS)

The Consumer Assessment of Healthcare Providers and Systems Clinician & Group Survey (CG-CAHPS) is one of the most widely studied and endorsed patient experience measures for ambulatory care. This study aimed to develop a Japanese CG-CAHPS and examine its psychometric properties. We evaluated the structural validity, criterion-related validity, internal consistency reliability, and site-level reliability of the scale. Data were analyzed for 674 outpatients aged 18 years or older in 11 internal medicine clinics. The confirmatory factor analysis supported the scale’s structural validity and the same composites (Access, Provider Communication, Care Coordination, and Office Staff) as that of the original CG-CAHPS. All site-level Pearson correlation coefficients between the Japanese CG-CAHPS composites and overall provider rating exceeded the criteria. Results of item-total correlations and Cronbach’s alpha indicated adequate internal consistency reliability. We developed the Japanese CG-CAHPS and examined its validity and reliability to measure the quality of ambulatory care based on patient experience. The results of the Japanese CG-CAHPS survey will provide useful information to providers, organizations, and policy makers for achieving a patient-centered healthcare system in Japan.


45
The quality assessment of patient-centeredness from the patient's perspective is an important 46 aspect of quality of health care [1]. In recent years, better patient experience has been recognized 47 as one of the crucial goals of healthcare alongside population health and per capita cost [2]. 48 Patient experience is integrally tied to the principles and practices of patient-and family-49 centered care. Embedded within patient experience is a focus on individualized care and tailoring 50 services to meet patients' needs and engage them as partners in their care [3]. Numerous studies 51 have shown that better patient experience is consistently associated with patient health outcomes, 52 patient safety, and patient behaviors across a wide range of disease areas and settings [4][5][6][7]. 53 The Consumer Assessment of Healthcare Providers and Systems Clinician & Group Survey 54 (CG-CAHPS) is one of the most widely studied and endorsed patient experience measures for 55 ambulatory care [8]. This standardized scale was developed by the Agency for Healthcare 56 Research and Quality (AHRQ) and confirmed its validity and reliability [9,10]. Currently, in the 57 United States, CG-CAHPS results are widely used as quality measures in accountability 58 initiatives and to stimulate, guide, and monitor quality improvement efforts [11]. For example, 59 results from the CG-CAHPS have been reported on the Physician Compare website, and insurers 60 are also increasingly including patient experience data in pay-for-performance programs. 61 In contrast, in Japan, voluntary activities for the assessment of patient experience have just 62 begun in limited settings, and systematic approaches for quality assessment and improvement 63 based on patient experience measures are still unestablished. Only a few standardized scales, 64 the present study aimed to develop a Japanese version of the CG-CAHPS and to examine its 68 structural validity, criterion-related validity, and internal consistency reliability. 69 70

71
Design, setting, and participants 72 The data used in this study were collected from a multicenter cross-sectional survey in 11 internal 73 medicine clinics from February to March 2020. The 11 participating clinics voluntarily took part 74 in the survey and are in urban areas in the Tokyo Metropolis and Kanagawa Prefecture, with all 75 the clinics being privately owned and managed. In Japan, clinics are generally run by one 76 full-time physician, nurses, and medical assistants, and they provide outpatient services and 77 possibly home care. Independent surveyors distributed a self-administered questionnaire to all 78 outpatients aged 18 years or older who visited one of the participating clinics within two days of 79 the survey period using a continuous sampling method. Patients who were unable to respond to 80 the questionnaire due to severe physical or mental disorders were excluded. Of

89
. CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted August 17, 2020. . The CG-CAHPS survey uses multiple response formats: four-point Likert scales (1 = never, 2 106 = sometimes, 3 = usually, and 4 = always), and a global rating scale (0 = worst to 10 = best). To 107 make the results easier to understand, we converted all scales to normalized scores ranging from 108 0 to 100 using the following formula: 109 Normalized Score = 100 * (Respondent's selected response value -Minimum response value 110 on the scale) / (Maximum response value -Minimum response value) 111 . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted August 17, 2020. . In the Japanese version, assuming the convergence in each composite as in the original version, 112 the score for each of the four composites was computed as the mean value for all normalized 113 scores in the scale that would fall in the range of 0-100 points, with higher scores indicating 114 better performance. 115 116 Statistical analysis 117 We validated the Japanese CG-CAHPS through the following steps: 118 First, we carried out a confirmatory factor analysis to evaluate the structural validity of the 119 Japanese CG-CAHPS composites. In the factor analysis, we hypothesized the same factor 120 structure (four-factor solution) as that of the original CG-CAHPS. The appropriateness of the 121 resulting structure was determined by examining if factor loadings were 0.40 or greater [16]. 122 Model fitness was assessed using the goodness-of-fit index (GFI), comparative fit index (CFI), 123 root mean square error of approximation (RMSEA), and standardized root mean square residual 124 (SRMR). For GFI and CFI, a value of > 0.90 is considered acceptable, and a value of > 0.95 125 indicates excellent goodness of fit. Previous studies suggest that models with RMSEA < 0.07 126 and SRMR < 0.08 are representative of models with a good fit [17][18][19]. 127 Second, we used the Japanese CG-CAHPS composite scores and the overall provider rating to 128 examine criterion-related validity. Validity was assessed using Pearson correlation coefficients 129 with each Japanese CG-CAHPS composite to predict the Rating of Provider (0 = Worst to 10 = 130 Best) of the scale at the provider-level. A correlation coefficient greater than 0.30 was considered 131 meaningful [20]. Provider-level correlations are a more important criterion for measurement than 132 patient-level correlations because the former are benchmarking tools to compare one provider or 133 . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted August 17, 2020. . facility with another. To examine provider-level correlations, we used each provider's mean 134 score on CG-CAHPS composites and the Rating of Provider. 135 Internal consistency reliability was examined by item-total correlations and Cronbach's alpha. 136 For a scale to be considered sufficiently reliable, an item-total correlation of 0.30 and a 137 Cronbach's alpha value of 0.70 is recommended [21]. Finally, descriptive statistics were 138 performed on the Japanese CG-CAHPS scores, including the mean, standard deviation, and 139 observed range. To deal with missing data, in the confirmatory factor analysis, we used the full 140 information maximum likelihood estimation to enable the use of information collected from 141 participants with missing data. In the evaluation of criterion-related validity and internal 142 consistency, we conducted complete case analyses. All statistical analyses were conducted using 143 R version 3. 6 . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted August 17, 2020. .  Table 2 shows the participants' responses to each item of the Japanese CG-CAHPS. The Top 152

45-
Box score for each item, which is the percentage of participants who provided the most positive 153 responses on that item, ranged from 56.1% to 73.6%. Regarding the mean Top Box score for 154 composites, the highest score was observed for Provider Communication (70.9%), while the 155 lowest score was for Care Coordination (59.6%). The bottom box score, which is the percentage of 156 participants with the least positive responses on the item, ranged from 0.8% to 7.2%. 157 158 159 . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted August 17, 2020. .

Never
Sometimes Usually Always Data missing Not applicable a Access: Q2. In the last 6 months, when you contacted this provider's office to get an appointment for care you needed right away, how often did you get an appointment as soon as you needed? is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted August 17, 2020. . is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint  Table 3 shows the Pearson correlation coefficients between the Japanese CG-CAHPS composites 185 and the Rating of Provider as an overall provider rating at the provider-level. All correlations 186 exceeded the 0.30 criterion. Provider Communication (r = 0.85) had the highest correlation with 187 the overall rating. 188 189 Table 3. Pearson correlation coefficients between Japanese CG-CAHPS composites and overall provider rating.

Composites
Provider-level correlations is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted August 17, 2020. . Table 4 indicates the score distribution and internal consistency reliability for the Japanese 192 CG-CAHPS. All item-total correlations were above the 0.30 criteria, ranging from 0.31 to 0. 92. 193 For Access, Provider Communication, and Office Staff, the Cronbach's alpha was above 0. is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted August 17, 2020. . This scale could be used for quality improvement based on the assessment of patient experience 207 with ambulatory care and for health services research in Japan. 208 The confirmatory factor analysis supported the scale's structural validity and the same four 209 composites (Access, Provider Communication, Care Coordination, and Office Staff) as that of the 210 original CG-CAHPS. Correlation coefficients between all Japanese CG-CAHPS composites and 211 the overall provider rating for assessing criterion-related validity exceeded the meaningful value 212 at the provider-level. In internal consistency analyses, only Cronbach's alpha for Care 213 Coordination did not exceed the recommended value. Cronbach's alpha is quite sensitive to the 214 number of items in the scale; therefore, it is common to find low Cronbach's alpha for scales 215 with few items [22]. In this case, it is more appropriate to report the item-total or inter-item 216 correlation. In our study, all item-total correlations were greater than the cutoff value, which 217 indicated acceptable internal consistency of the scales. 218 The CG-CAHPS is one of the most widely studied patient experience scales for ambulatory 219 care worldwide. The CG-CAHPS has been translated into many languages in order to be used in 220 other countries so that comparisons of health service quality from the patient perspective can be 221 made. In our study, the recovery rate for the questionnaire administered was very high, 222 suggesting a low risk of selection bias. 223 However, the present study has several potential limitations. First, in this study, we evaluated 224 the structural validity, criterion-related validity, and internal consistency reliability of the 225 Japanese CG-CAHPS, other psychometric properties, including convergent and discriminant 226 validity, test-retest reliability, and interpretability, have not been assessed [23]. These 227 psychometric properties of the scale need to be evaluated in future studies. Second, our survey 228 setting was restricted to urban areas and may not have sufficiently represented the Japanese 229 . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted August 17, 2020. . national level. Therefore, the study results may have limited generalizability and a survey using 230 the Japanese CG-CAHPS in other suburban and rural areas should be conducted. 231 232

233
We developed the Japanese CG-CAHPS as a valid and reliable scale to measure the quality of 234 ambulatory care based on patient experience. The results of the Japanese CG-CAHPS survey will 235 provide useful information to providers, organizations, and policy makers for achieving a 236 patient-centered healthcare system in Japan. 237 238 . CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted August 17, 2020. . https://doi.org/10.1101/2020.08.15.20175612 doi: medRxiv preprint https://www.ahrq.gov/cahps/surveys-guidance/helpful-resources/resources/cahpsGuidelines_Translat taxonomy, terminology, and definitions of measurement properties for health-related patient-reported 293 outcomes. J Clin Epidemiol. 2010;63(7):737-45.

294
. CC-BY-NC 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted August 17, 2020. .