The present study investigated the internal consistency reliability, construct validity, and item response characteristics of a newly developed Vietnamese version of the Kessler 6 (K6) scale among hospital nurses in Hanoi, Vietnam. The K6 was translated into the Vietnamese language following a standard procedure. A survey was conducted of nurses in a large general hospital in Hanoi, Vietnam, using a questionnaire including the Vietnamese K6, other scales (DASS21, health-related QOL, self-rated health, and psychosocial work environment), and questions about demographic variables. Internal consistency reliability (Cronbach’s alpha coefficient) was calculated. A confirmatory factor analysis was conducted. Eleven hypotheses were tested (as Pearson’s correlations with the K6) to assess the scale’s construct validity. Item response theory (IRT) analysis was conducted to identify the item response characteristics. The Cronbach’s alpha coefficient was 0.864. The explanatory and confirmatory factor analyses indicated a one-factor structure. Most hypotheses tested for construct validity were supported. IRT analysis indicated that response categories were located in order according to severity. K6 provided reliable information regarding higher levels of psychological distress. The findings suggest that the Vietnamese version of the K6 is a reliable and valid instrument to measure psychological distress among hospital nurses in Vietnam.
Citation: Kawakami N, Thi Thu Tran T, Watanabe K, Imamura K, Thanh Nguyen H, Sasaki N, et al. (2020) Internal consistency reliability, construct validity, and item response characteristics of the Kessler 6 scale among hospital nurses in Vietnam. PLoS ONE 15(5): e0233119. https://doi.org/10.1371/journal.pone.0233119
Editor: Wen-Jun Tu, Chinese Academy of Medical Sciences and Peking Union Medical College, CHINA
Received: December 18, 2019; Accepted: April 28, 2020; Published: May 21, 2020
Copyright: © 2020 Kawakami et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All data files are available from the Figshare database (DOI: 10.6084/m9.figshare.12005511.v2).
Funding: NK received a grant from Japan Agency for Medical Research and Development (AMED) under Grant Number JP18jk0110014 (https://www.amed.go.jp/). The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: NK reports grants from Fujitsu Ltd, SB AtWork Corp., and TAK Ltd, personal fees from Japan Dental Association, Junpukai Health Care Center, Occupational Health Foundation, Osaka Chamber of Commerce and Industry, SB AtWork Corp., and Sekisui Chemicals outside the submitted work. This does not alter our adherence to PLOS ONE policies on sharing data and materials The other authors declare that they have no competing interests.
The Kessler 6 (K6) scale is a six-item short measure of psychological distress and is also used as a screening instrument for common mental disorders in the community [1, 2]. The items were selected from a large item pool based on item response theory (IRT) analysis . The interviewer-administered K6 showed a good screening performance for common mental disorders in the community, with the area under the curve (AUC) of receiver operating characteristic of 0.854, a sensitivity of 0.36 and a specificity of 0.96. The K6 was translated into 21 language families . Because of its brevity, the scale has been used widely in mental health epidemiology. A cross-country validation study including data from 14 countries reported a substantial concordance between interviewer-administered K6 scores and diagnoses of common mental disorders (CMD) (the median AUC, 0.83) . Country-specific validation studies have been published for the Arabic , English [6–8], Chinese , French , Japanese [11, 12], Korean , and Farsi (Persian) languages . These studies in general reported good internal consistency reliability (0.69–0.92 in Cronbach’s alpha) [5, 7, 9, 10, 12–15] and factor-based [9, 10] and other construct validity (correlations with other mental health measures) [5, 7, 8, 10, 13] across various populations (university students, community residents, employees, people living with HIV, and psychiatric patients), for both self-report and interviewer-administered versions. However, contrary to the initial assumption of uni-dimensionality [1, 2], some studies reported a multifactorial factor structure for the K6 [9, 10]. In addition, while a scoring approach based on unweighted item scores (i.e., 0-1-2-3-4 for the five response categories) has been used in most studies, the authors of the K6 argued that IRT-based weighted scoring could be a better and easy-to-use alternative . In fact, an IRT analysis combining data from 14 countries suggested the use of a weighted scoring approach (e.g., 0-0-0-1-1) . An IRT study reported that the psychometric properties of the K6 may vary depending on ethnicity in the United States . Psychometric properties of the K6 should be systematically investigated including confirmatory factor analysis and IRT analysis for specific country and ethnic populations .
There is a growing concern for mental health in low- and middle-income countries (LMICs) . One of the proposed strategies to scale up mental health service in these countries is to integrate mental health services into workplaces . In LMICs, healthcare professionals face increasing stress and burnout at work  because of a shortage of healthcare professionals [21, 22], population aging, and demands from service users (i.e., patients and families) [23–25]. This is particularly the case in South-East Asia . Increasing stress at work has been reported among nurses in Vietnam [26, 27], Thailand , and Myanmar . High levels of stress at work and poor mental health of healthcare professionals may diminish the quality of health care services in these countries . It is important to manage work-related stress and to improve mental health among nurses in Southeast Asian countries.
The present study investigated psychometric properties, i.e., internal consistency reliability, construct validity, and item response characteristics of the K6 scale translated in Vietnamese in a sample of nurses working in a large general hospital in Hanoi, Vietnam. This is the first validation study of a Vietnamese version of the K6. While the study used a sample from a specific occupational group the findings may be useful in exploring the cross-cultural validity of the K6 scale in Vietnam. This is also the first study in which IRT analysis was conducted on data from the K6 in a LMIC since a previous cross-national study used a combined data set from high-income countries and LMICs to conduct an IRT analysis .
A questionnaire survey was conducted as the baseline survey for a stress management intervention study in a large general hospital in Hanoi, Vietnam , from August 21 to September 10, 2018. Out of all registered nurses (n = 1,269) in this hospital, a total of 1,258, excluding those who were about to retire (n = 11), were invited to complete a questionnaire and return it in an envelope to collection boxes. The returned questionnaires were collected by staff at Hanoi University of Public Health and entered into a database.
The aim and procedure of the study was reviewed and approved by the Research Ethics Committee of the Graduate School of Medicine/Faculty of Medicine, The University of Tokyo (no. 11991-(1)) and the Ethical Review Board for Biomedical Research, Hanoi University of Public Health (no 346/2018/YTCC-HD3).
The K6 scale is a 6-item self-report measure of psychological distress, with a 5-point response option (4 = All of the time, 3 = Most of the time, 2 = Some of the time, 1 = A little of the time, and 0 = None of the time) . The possible range of the total score is 0 to 24, with higher scores indicating greater psychological distress. The translation of the K6 into the Vietnamese language was conducted following the standard procedure of translation and cultural adaptation of patient reported health outcomes . After obtaining permission to translate it into Vietnamese from the lead author of the research team that developed the K6 (Prof Ronald C Kessler, Harvard Medical School, United States of America), the original English version was translated into Vietnamese independently by two translators who were fluent in English and were knowledgeable about mental health symptomatology. These translations were integrated into a preliminary translation. This translation was reviewed by collaborators at the Hanoi University of Public Health, and a brief cognitive interview was conducted of a group of nurses (n = 30) to ask their opinions on draft items to know if the wordings were meaningful and relevant, followed by necessary amendments. This amended version was back-translated into English by an independent translator, and reviewed by staff of the research team at Harvard Medical School, as well as by the authors. Further careful revisions were made to make sure some of the K6 items were appropriately translated: "hopeless" in item #2; "restless or fidgety" in item #3; "depressed" in item #4. The final draft was pre-tested in a pilot study of 150 nurses in the same hospital in June 2018. A psychometric analysis showed a problem in this draft: item #5 "…that everything was an effort?" did not correlate with other items. The item was further revised based on a discussion among the collaborators. The final version of the Vietnamese version of K6 was posted on the website of the Harvard Medical School: https://www.hcp.med.harvard.edu/ncs/k6_scales.php
Other scales for testing the construct validity.
Depression, anxiety and stress. The 21-item version of the DASS scale, which measures symptoms of depression, anxiety, and stress in community settings [33, 34], is comprised of the three corresponding subscales, each with seven items. Items are scored on a 4-point scale ranging from 0 (did not apply to me at all) to 3 (applied to me very much, or most of the time). Each subscale score ranges from 0 to 21. Reliability and validity of the Vietnamese version of the DASS21 scale have previously been reported .
Health-related quality of life. The EQ-5D-5L is a widely used indicator of health-related quality of life (HR-QOL) . The EQ-5D-5L consists of five items covering mobility, self-care, usual activities, pain/discomfort and anxiety/depression, each of which is rated on a five category scale from no problems (1) to extreme problems (5). Reliability and validity of the EQ-5D-5L are well established . A Vietnamese version of DASS21 has been developed and tested, and has acceptable reliability and validity . We used the standard value set for Vietnam to calculate participants’ HR-QOL scores .
Psychosocial work environment. The Job Content Questionnaire (JCQ) was used to assess the psychosocial work environment . It includes four scales: a five-item psychological demand scale, a nine-item decision latitude scale, a four-item supervisor support scale, and a four-item coworker support scale. The items are scored on a 4-point Likert scale, ranging from 1 (strongly disagree) to 4 (strongly agree). The reliability and validity of the Vietnamese version of the JCQ has been found to be acceptable, while Cronbach’s alpha coefficients were lower for job demands and job control (0.446–0.499) than those for supervisor and coworker support (0.856–0.868) .
Self-rated health. Participants was asked to rate their health status on a 0–100 scale, in which 0 indicates the worst health status and 100 indicates the best health status. This was part of the EQ-5D-5L scale [36, 37], but not used in the calculation of the HR-QOL score.
Demographic variables were included in the questionnaire. Respondents were asked to indicate their sex (male or female), and their birth year. Personal income per month was assessed by a single-item question, “What is your current monthly income (including salary and any other sources of income)?”, with a 3-point response scale: ≤ 5 million VND (220 USD) (1), 5–10 million VND (220–432 USD) (2), and ≥ 10 million (432 USD) (3). Respondents were also asked whether they were currently married, never married, or divorced/separated, and about the nature of their labor contract: fixed-term contract for less than one year, fixed-term contract for more than one year, unspecified-term contract, permanent contract, or “other.”
Distribution of the K6 scores was examined separately for men and women. The average and standard deviation (SD), maximum and minimum values of K6 scores were calculated. The proportions of participants who had a score of 13 or higher  were calculated for men and women. Internal consistency reliability was calculated by using the Cronbach’s alpha coefficient. Then a confirmatory factor analysis was conducted to assess the fit of the data to a one-factor structure. The goodness of fit index (GFI), adjusted goodness of fit index (AGFI), and comparative fit index (CFI) greater than 0.95 and the root mean square error of approximation (RMSEA) smaller than 0.06 to 0.08 were considered to be indicators of acceptable fit  For a comparison with previous studies, an exploratory factor analysis was conducted with maximum likelihood extraction, in order to assess the number of factors using a scree plot and proportion of variance explained by the first factor.
Construct validity was assessed by using Pearson’s correlations (rs), with the following 11 hypotheses: DASS21 scores of (1) depression, (2) anxiety, and (3) stress moderately or strongly and positively (rs>0.4) with K6 score [5, 10, 13]; (4) HR-QOL and (5) self-rated health status moderately and negatively (rs = -0.4 to -0.7) correlated with K6 scores ; (6) job demands weakly or moderately positively (r = 0.2 to 0.4), and (7) job control, (8) supervisor support, and (9) coworker support weakly or moderately negatively (rs = -0.2 to -0.4) correlated with K6 scores ; socioeconomic position, such as (10) higher education and (11) personal income weakly and negatively (rs = -0.1 to -0.2) correlated with K6 scores .
A screening performance of K6 for identifying a CMD (i.e., depressive disorders and anxiety disorders) was also examined. Cases with CMD were defined as those who had 34 or greater scores of the total DASS21, that was the best cut-off score found in a previous study in Vietnam . An area under the curve (AUC) of a receiver operative curve (ROC) and the 95% CIs were calculated predicting these cases with CMD on the sore of K6. An optimum cut-off score was determined based on the best Youden index (a sum of sensitivity and specificity), and the sensitivity, specificity, and positive likelihood ratios were calculated with their standard errors .
The IRT analysis was conducted using the graded response model. Threshold parameters for response categories and slope parameter of each item were estimated. The test information curve and item information curves were presented in a graphical way.
Most of these analyses were conducted using IBM SPSS version 22. The confirmatory factor analysis and IRT were conducted using PROC CALIS and PROC IRT, respectively, from the SAS software package version 9.4.
A total of 949 (75.6% of the target population) respondents returned the questionnaires. One respondent had a missing response on one of the K6 items, and was excluded from the analysis. Participants were mostly women, married, and younger than 40 years old, with age range between 22 and 58 years (Table 1). Most were with monthly personal income of more than 5 million VND (equivalent to 220 USD) About half had completed college or higher education. About half had a permanent contract, while about one fourth were employed with a fixed-term contract for one year.
Distribution of the scale scores
The K6 scores showed a skewed distribution for both men and women (S1 Appendix). The average K6 scores of women and men did not significantly differ (Table 2). The proportion of respondents with a K6 score of 13 or greater was 2.4%, and did not significantly differ between men and women. Average item scores were higher for item #1 compared to those for the other items (Table 3). For items #4 and #5, no respondents endorsed the extreme category “all of the time”).
Internal consistency reliability
The Cronbach’s alpha coefficient for the total K6 score was 0.864. Most items substantially contributed to the Cronbach’s alpha coefficient, while item #1 did not (Table 3).
The confirmatory factor analysis assuming a one factor structure indicated a moderate fit of the one factor model: GFI = 0.957; AGFI = 0.901; RMSEA = 0.115 (90% confidence intervals, 0.098 to 0.134); CFI = 0.954. Standardized coefficients (error variance) for items 1 to 6 were 0.582 (0.662), 0.764 (0.416), 0.726 (0.472), 0.774 (0.400), 0.794 (0.369), and 0.695 (0.517), respectively. In the explanatory factor analysis, initial eigenvalues (% of variance explained) for factors 1 to 6 were 3.619 (60.3), 0,750 (12.5), 0.472 (7.9), 0.449 (7.5), 0.399 (5.2), and 0.311, respectively, yielding one single factor.
DASS21 scores for depression, anxiety, and stress correlated moderately and positively with K6 score (rs = 0.513 to 0.544) (Table 4). HR-QOL and self-rated health correlated negatively with K6 score (rs = -0.399 and -0.355, respectively), while the correlation coefficients were slightly smaller than 0.4. Job demands weakly and positively correlated with K6 score (r = 0.292). Job control, supervisor support and coworker support weakly and negatively correlated with K6 score (rs = -0.147, -0.204, and -0.160, respectively). Personal income was only marginally significantly associated with K6 (r = -0.06, p = 0.066).
Among the participants, 22 cases were identified as having CMD (i.e., 34 or greater scores of the total DASS21). The AUC predicting CMD on K6 score was 0.919 (95% CIs, 0.855–0.984). The best cut-off score for K6 was identified as 9+, with 129 (12.4%) of the participants having the scores equal to or greater than this cut-off. The sensitivity and specificity were 0.864 (SE, 0.073) and 0.876 (SE, 0.031), respectively, with the positive likelihood ratio of 6.96 (SE, 0.121).
Item-response theory analysis
For all the items, the threshold parameters almost linearly increased from mild to severe response categories (Table 5). For items #2–6, the threshold parameters for most response categories of each item were greater than 0. On the other hand, for item #1, the first two threshold parameters were less than 0. The slope parameters were similar for items #2–6 (1.512–1.770), while the parameter was smaller for item #1 (0.909). The test information curves indicated that the K6 provided more information when the trait was greater than 0 (i.e., when people had higher psychological distress than the average) (Fig 1). Item information curves indicated that most of the items provided good information where the trait was greater than 0 (Fig 2). Item #1 had markedly lower information across the latent trait continuum than the other items. However, item #1 provided better information than other items where the trait was less than -0.5.
The vertical axis indicates the latent trait of psychological distress.
The vertical axis indicates the latent trait of psychological distress.
The Vietnamese self-report version of the K6 showed high level of internal consistency reliability. The uni-dimensional structure of the K6 was supported by the confirmatory factor analyses of the items. Most hypotheses tested for construct validity were supported. Item response analysis indicated that response categories were located in order according to severity. The K6 provided information for detecting higher psychological distress as a whole, but item #1 did not. The study suggests that the Vietnamese version of the K6 is a reliable and valid instrument to measure psychological distress.
The confirmatory factor analysis indicated a good fit for the one-factor structure, with acceptable levels of most goodness of fit indicators, while RMSEA was not within an ideal range (<0.6–0.8) but close. The findings of the confirmatory factor analysis suggest that the Vietnamese K6 has a one-factor structure. The first factor had an eigenvalue of 3.6 and explained 60.3% of the total variance in the explanatory factor analysis. The one-factor structure was consistently observed across 14 countries including LMICs (Brazil, China, Colombia, India, Mexico, Nigeria, Ukraine)  with similar eigenvalues for the first factor. The one-factor structure seems applicable to most countries, including LMICs. However previous studies in France and China reported a two-factor structure consisting of items related to depression and anxiety [9, 10, 15]. The factor structure of the K6 may still vary depending on language, ethnicity, and culture.
For construct validity, all of the 11 hypotheses tested were supported by the data. DASS21 scales for depression, anxiety, and stress all correlated moderately with the K6, which is concordant with previous findings of moderate to strong correlations with other depression scales [5, 10, 13]. Consistent with previous studies , HR-QOL and self-rated health also moderately correlated with K6 score. The correlations were slightly below 0.4. Job demands, job control, supervisor support, and coworker support correlated weakly (|rs| = 0.15–0.28) in expected directions . The modest associations between job demands and job control with K6 scores may be attributable to lower reliability of these scales . Educational attainment and personal income correlated very weakly (|rs|<0.1) with the K6. These findings are in line with previous studies on socioeconomic factors associated with depression . The small correlation for educational attainment may be a result of the fact that all participants had the same occupation; the small correlation for personal income may be attributable to some participants having income from other family members. The present study suggests an adequate level of construct validity of the Vietnamese version of the K6.
A previous IRT analysis using data from 14 countries found the two most severe response categories were useful for discriminating degrees of psychological distress, and discussed the use of a weighted scoring approach (e.g., 0-0-0-1-1) . However, in the present study, the threshold parameters almost linearly increased according to the severity of response categories. The present study rather supports the 0-1-2-3-4 scoring for the Vietnamese K6. The test information curve was better when the latent trait was greater than 0, suggesting that the Vietnamese version of the K6 measure high levels of psychological distress more accurately. This version of the K6 may be useful for detecting poor mental health among hospital nurses in Vietnam, although how K6 score relates to probability of presence of diagnosable psychiatric disorder, and what is the most appropriate cutoff score, in Vietnam is currently unknown.
Internal consistency reliability measured by Cronbach’s alpha coefficient was high enough in this sample to be comparable with previous reports [5, 9, 10, 12–15]. However, item #1 did not contribute much to the internal consistency reliability. The item information curves from the IRT analysis indicated that item #1 provided poorer information than the other items. This is partly because the item #1 that originally asks “nervousness” was translated by using a Vietnamese word “stress”, that is popular wording used frequently in daily life. The item #1 can be dropped from the Vietnamese version of the K6, still keeping the current level of internal consistency reliability and the scale information. However, it might be good to have an item with such lower difficulty in the beginning of the scale, to make it easy for respondents to start responding to the scale. Potential benefits of keeping and dropping this item should be investigated further, e.g., by conducting a randomized controlled trial comparing psychometrics of the two versions.
The distribution of scores of the Vietnamese version of the K6 was skewed to the right. The pattern is similar to an exponential distribution of K6 scores found in a study in the United States . However, there was a peak at scores of 1–3 in this sample, not at the score of 0 as in the US sample . This may be because the item response function of item #1 for those response categories had lower difficulty parameters. It is hard to compare average scores of K6 in this sample (4.57) with other reports, since some reports did not indicate the average score, and available average scores varied a lot (e.g., 3.6 in a community sample in Japan ; 5.9 in a community sample in the United States ; 10.0 among older people in Korea ; and 12.9 among Palestinian social workers . The screening performance of K6 score predicting CMD defined by using DASS21 was high: the AUC was 0.919, with the sensitivity of 0.864 and the specificity of 0.876 for the best cut off score of 9+. The finding is similar to one from a previous study in Japan that the stratum-specific likelihood ratio became higher (>11) for the cut off scores of 9+ on K6 . However, the screening performance and the best cut off score of the Vietnamese version of the K6 should be assessed by comparing people who have or have not been previously diagnosed with mental disorders.
A common use of the K6 (and K10) is in large-scale population mental health surveys. Despite some reservations about varying sensitivity of K6 for different disorders  it is a strong candidate for such use in Vietnam, where a population representative national mental health survey to estimate prevalence of common mental disorders has not yet been conducted. The absence of reliable estimates of population prevalence of mental disorders is a significant problem for mental health policy-making and advocacy. The use of the K6 in such national surveys would require studies to establish the criterion validity of the K6 in Vietnam.
The present study has several limitations. First, we aimed to validate the K6 scale to screen psychological distress among nurses, a special occupational group under high stress, in Vietnam. Nurses are usually more educated than general workers, and also familiar with questions of psychological symptoms included in K6. The present findings may not be generalized to the general population or working population in Vietnam. The reliability and validity of the present scale should be further tested in these populations. Second, as we noted earlier, we did not investigate the scale’s screening performance for clinically diagnosed mental disorders. Third, the study did not examine test-retest reliability or indicators for interpretability (e.g., minimal important difference or minimally important change) which might be useful in applying the K6 in practice. Finally, 150 nurses who responded to a pilot study were also invited to the main validation study; most of them probably actually participated. Because the pilot study was anonymous, it was impossible to conduct an analysis excluding those who participated in the two surveys. Repeated surveys have been reported to result in attenuated responses to anxiety symptoms of respondents , which may affect the finding of the present study. Despite these limitations, the study clarifies the psychometric properties of the newly developed Vietnamese self-report version of the K6, including item response characteristics first examined in a LMIC.
The present study found an acceptable level of internal consistency reliability, and supported a unidimensional factor structure and other construct validity of Vietnamese self-report version of the K6. Item response analysis indicated that the K6 provided information for detecting higher psychological distress as a whole. It is suggested that the Vietnamese version of the K6 is a reliable and valid instrument to measure psychological distress at least among nurses in Vietnam.
The authors acknowledged the support and contribution of the Director Board and the Nursing Office in Bach Mai hospital to the implementation of this study. We thank all nurses who participated in and provided their information for the study and highly appreciated the contribution of a research team from Hanoi University of Public health, an important partner to make this study possible.
- 1. Kessler RC, Andrews G, Colpe LJ, Hiripi E, Mroczek DK, Normand SL, et al. Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychological Medicine. 2002;32(6):959–76. pmid:12214795.
- 2. Kessler RC, Barker PR, Colpe LJ, Epstein JF, Gfroerer JC, Hiripi E, et al. Screening for serious mental illness in the general population. Archives of General Psychiatry. 2003;60(2):184–9. pmid:12578436.
- 3. Department of Health Policy, Harvard Medical School. National Comorbidity Survey: K10 and K6 Scales: Department of Health Policy, Harvard Medical School; 2005 [cited 2019 Jan 3]. Available from: https://www.hcp.med.harvard.edu/ncs/k6_scales.php.
- 4. Kessler RC, Green JG, Gruber MJ, Sampson NA, Bromet E, Cuitan M, et al. Screening for serious mental illness in the general population with the K6 screening scale: Results from the WHO World Mental Health (WMH) survey initiative. International journal of methods in psychiatric research. 2010;19 Suppl 1:4–22. Epub 2010/06/08. pmid:20527002; PubMed Central PMCID: PMC3659799.
- 5. Easton SD, Safadi NS, Wang Y, Hasson RG 3rd. The Kessler psychological distress scale: translation and validation of an Arabic version. Health and quality of life outcomes. 2017;15(1):215. Epub 2017/10/29. pmid:29078774; PubMed Central PMCID: PMC5658946.
- 6. Furukawa TA, Kessler RC, Slade T, Andrews G. The performance of the K6 and K10 screening scales for psychological distress in the Australian National Survey of Mental Health and Well-Being. Psychological Medicine. 2003;33(2):357–62. Epub 2003/03/08. pmid:12622315.
- 7. van Heyningen T, Honikman S, Tomlinson M, Field S, Myer L. Comparison of mental health screening tools for detecting antenatal depression and anxiety disorders in South African women. Plos One. 2018;13(4):e0193697. pmid:29668725
- 8. Choi SK, Boyle E, Burchell AN, Gardner S, Collins E, Grootendorst P, et al. Validation of six short and ultra-short screening instruments for depression for people living with HIV in Ontario: results from the Ontario HIV Treatment Network Cohort Study. PLoS One. 2015;10(11). https://doi.org/10.1371/journal.pone.0142706.
- 9. Kang YK, Guo WJ, Xu H, Chen YH, Li XJ, Tan ZP, et al. The 6-item Kessler psychological distress scale to survey serious mental illness among Chinese undergraduates: Psychometric properties and prevalence estimate. Comprehensive psychiatry. 2015;63:105–12. Epub 2015/11/12. pmid:26555498.
- 10. Arnaud B, Malet L, Teissedre F, Izaute M, Moustafa F, Geneste J, et al. Validity study of Kessler's psychological distress scales conducted among patients admitted to French emergency department for alcohol consumption-related disorders. Alcoholism, clinical and experimental research. 2010;34(7):1235–45. Epub 2010/05/19. pmid:20477768.
- 11. Furukawa TA, Kawakami N, Saitoh M, Ono Y, Nakane Y, Nakamura Y, et al. The performance of the Japanese version of the K6 and K10 in the World Mental Health Survey Japan. Int J Methods Psychiatr Res. 2008;17(3):152–8. pmid:18763695.
- 12. Sakurai K, Nishi A, Kondo K, Yanagida K, Kawakami N. Screening performance of K6/K10 and other screening instruments for mood and anxiety disorders in Japan. Psychiatry and Clinical Neurosciences. 2011;65(5):434–41. Epub 2011/08/20. pmid:21851452.
- 13. Min JW, Lee SH. Validation of the K6/K10 scales of psychological distress and their optimal cutoff scores for older Koreans. International journal of aging & human development. 2015;80(3):264–82. Epub 2015/07/22. pmid:26195502.
- 14. Hajebi A, Motevalian A, Amin-Esmaeili M, Rahimi-Movaghar A, Sharifi V, Hoseini L, et al. Adaptation and validation of short scales for assessment of psychological distress in Iran: The Persian K10 and K6. International journal of methods in psychiatric research. 2018;27(3):e1726. Epub 2018/06/12. pmid:29888523.
- 15. Lee S, Tsang A, Ng KL, Ma YL, Guo W, Mak A, et al. Performance of the 6-item Kessler scale for measuring serious mental illness in Hong Kong. Comprehensive psychiatry. 2012;53(5):584–92. Epub 2011/11/20. pmid:22104556.
- 16. Kim G, DeCoster J, Bryant AN, Ford KL. Measurement equivalence of the K6 Scale: The effects of race/ethnicity and language. Assessment. 2016;23(6):758–68. Epub 2015/08/19. pmid:26282779; PubMed Central PMCID: PMC5468522.
- 17. Stolk Y, Kaplan I, Szwarc J. Clinical use of the Kessler psychological distress scales with culturally diverse groups. International journal of methods in psychiatric research. 2014;23(2):161–83. Epub 2014/04/15. pmid:24733815.
- 18. Eaton J, McCay L, Semrau M, Chatterjee S, Baingana F, Araya R, et al. Scale up of services for mental health in low-income and middle-income countries. Lancet. 2011;378(9802):1592–603. WOS:000296767500025. pmid:22008429
- 19. Thornicroft G, Deb T, Henderson C. Community mental health care worldwide: current status and further developments. World Psychiatry. 2016;15(3):276–86. WOS:000386140200019. pmid:27717265
- 20. Dugani S, Afari H, Hirschhorn LR, Ratcliffe H, Veillard J, Martin G, et al. Prevalence and factors associated with burnout among frontline primary health care providers in low- and middle-income countries: A systematic review. Gates Open Research. 2018;2:4. Epub 2018/06/11. pmid:29984356; PubMed Central PMCID: PMC6030396.
- 21. Elison NK, Verani AR, McCarthy C. National nursing and midwifery legislation in countries of South-East Asia with high HIV burdens. WHO South-East Asia journal of public health. 2015;4(102):12–9. pmid:26568920; PubMed Central PMCID: PMC4640210.
- 22. World Health Organization. The World Health Report 2006: Working Together for Health: World Health Organization; 2006.
- 23. Duvall JJ, Andrews DR. Using a structured review of the literature to identify key factors associated with the current nursing shortage. Journal of Professional Nursing. 2010;26(5):309–17. pmid:20869031
- 24. Fleming KC, Evans JM, Chutka DS. Caregiver and clinician shortages in an aging nation. Mayo Clinic proceedings. 2003;78(8):1026–40. pmid:12911051.
- 25. Toh SG, Ang E, Devi MK. Systematic review on the relationship between the nursing shortage and job satisfaction, stress and burnout levels among nurses in oncology/haematology settings. International Journal of Evidence‐based Healthcare. 2012;10(2):126–41. pmid:22672602
- 26. Nguyen HTT, Kitaoka K, Sukigara M, Thai AL. Burnout study of clinical nurses in Vietnam: Development of job burnout model based on Leiter and Maslach's theory. Asian nursing research. 2018;12(1):42–9. pmid:29463486.
- 27. Tran TTT, Nguyen NB, Luong MA, Bui THA, Phan TD, Tran VO, et al. Stress, anxiety and depression in clinical nurses in Vietnam: a cross-sectional survey and cluster analysis. International Journal of Mental Health Systems. 2019;13:3. Epub 2019/01/03. pmid:30622629; PubMed Central PMCID: PMC6317201.
- 28. Pongruengphant R, Tyson PD. When nurses cry: coping with occupational stress in Thailand. International Journal of Nursing Studies. 2000;37(6):535–9. pmid:10871663
- 29. Lwin PM, Rattanapan C, Laosee O. Nurses' intention to remain employed in hospitals: Understanding the shortage in the context of Myanmar. International journal of nursing practice. 2017;23(4). pmid:28303620.
- 30. Salmasi S, Khan TM, Hong YH, Ming LC, Wong TW. Medication errors in the Southeast Asian countries: A systematic review. PLoS One. 2015;10(9):e0136545. pmid:26340679; PubMed Central PMCID: PMC4560405.
- 31. Imamura K, Tran T, Nguyen H, Kuribayashi K, Sakuraya A, Nguyen A, et al. Effects of two types of smartphone-based stress management programmes on depressive and anxiety symptoms among hospital nurses in Vietnam: a protocol for three-arm randomised controlled trial. Bmj Open. 2019;9(4):e025138. WOS:000471157200095. pmid:30962230
- 32. Wild D, Grove A, Martin M, Eremenco S, McElroy S, Verjee-Lorenz A, et al. Principles of good practice for the translation and cultural adaptation process for patient-reported outcomes (PRO) measures: Report of the ISPOR Task Force for Translation and Cultural Adaptation. Value in Health. 2005;8(2):94–104. WOS:000227723000003. pmid:15804318
- 33. Henry JD, Crawford JR. The short-form version of the Depression Anxiety Stress Scales (DASS-21): Construct validity and normative data in a large non-clinical sample. The British journal of clinical psychology. 2005;44(Pt 2):227–39. pmid:16004657.
- 34. Lovibond PF, Lovibond SH. The structure of negative emotional states: comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories. Behaviour research and therapy. 1995;33(3):335–43. pmid:7726811.
- 35. Tran TD, Tran T, Fisher J. Validation of the depression anxiety stress scales (DASS) 21 as a screening instrument for depression and anxiety in a rural community-based cohort of northern Vietnamese women. BMC psychiatry. 2013;13:24. pmid:23311374; PubMed Central PMCID: PMC3566910.
- 36. Herdman M, Gudex C, Lloyd A, Janssen M, Kind P, Parkin D, et al. Development and preliminary testing of the new five-level version of EQ-5D (EQ-5D-5L). Qual Life Res. 2011;20(10):1727–36. pmid:21479777
- 37. Tran BX, Ohinmaa A, Nguyen LT, Nguyen TA, Nguyen TH. Determinants of health-related quality of life in adults living with HIV in Vietnam. AIDS care. 2011;23(10):1236–45. pmid:21711211.
- 38. Mai VQ, Van Minh H, Sun S, Giang KB, Sahlen KG. Valuing health—related quality of life: An EQ-5D-5L value set for Vietnam. Hanoi, Vietnam: Hanoi University of Public Health, 2018.
- 39. Karasek R, Brisson C, Kawakami N, Houtman I, Bongers P, Amick B. The Job Content Questionnaire (JCQ): An instrument for internationally comparative assessments of psychosocial job characteristics. Journal of Occupational Health Psychology. 1998;3(4):322–55. pmid:9805280.
- 40. Sasaki N, Imamura K, Thuy TTT, Watanabe K, Huong NT, Kuribayashi K, et al. Validation of the Job Content Questionnaire among hospital nurses in Vietnam. Journal of Occupational Health. 2019;62(1):e12086. WOS:000490576000001. pmid:31541525
- 41. Schreiber J, Nora A, Stage F, Barlow E, King J. Reporting structural equation modeling and confirmatory factor analysis results: A review. Journal of Educational Research. 2006;99(6):323–37. WOS:000239326800001.
- 42. Prigent A, Auraaen A, Kamendje-Tchokobou B, Durand-Zaleski I, Chevreul K. Health-related quality of life and utility scores in people with mental disorders: a comparison with the non-mentally ill general population. International Journal of Environmental Research and Public Health. 2014;11(3):2804–17. WOS:000334438200028. pmid:24608903
- 43. Theorell T, Hammarström A, Aronsson G, Träskman Bendz L, Grape T, Hogstedt C, et al. A systematic review including meta-analysis of work environment and depressive symptoms. BMC Public Health. 2015;15:738. Epub 2015/08/01. pmid:26232123; PubMed Central PMCID: PMC4522058.
- 44. Lorant V, Deliège D, Eaton W, Robert A, Philippot P, Ansseau M. Socioeconomic inequalities in depression: A meta-analysis. American Journal of Epidemiology. 2003;157(2):98–112. 12522017. pmid:12522017
- 45. Peirce JC, Cornell RG. Integrating stratum-specific likelihood ratios with the analysis of ROC curves. Medical Decision Making. 1993;13(2):141–51. pmid:8483399.
- 46. Tomitaka S, Kawasaki Y, Ide K, Akutagawa M, Yamada H, Yutaka O, et al. Pattern analysis of total item score and item response of the Kessler Screening Scale for Psychological Distress (K6) in a nationally representative sample of US adults. PeerJ. 2017;5:e2987. Epub 2017/02/09. pmid:28289560; PubMed Central PMCID: PMC5345827.
- 47. Furukawa TA, Kawakami N, Saitoh M, Ono Y, Nakane Y, Nakamura Y, et al. The performance of the Japanese version of the K6 and K10 in the World Mental Health Survey Japan. Int J Methods Psychiatr Res. 2008;17(3):152–8. WOS:000259296400003. pmid:18763695
- 48. Veldhuizen S, Cairney J, Kurdyak P, Streiner DL. The sensitivity of the K6 as a screen for any disorder in community mental health surveys: a cautionary note. Can J Psychiatry. 2007;52(4):256–9. Epub 2007/05/16. pmid:17500307.
- 49. Knowles ES, Coker MC, Scott RA, Cook DA, Neville JW. Measurement-induced improvement in anxiety: Mean shifts with repeated assessment. Journal of Personality and Social Psychology. 1996;71(2):352–63. pmid:8765486