http://orcid.org/0000-0002-7195-8086
Figures
Abstract
The psychometric properties of the ten-item Kessler Psychological Distress scale (K10) have been extensively explored in civilian populations. However, documentation of its psychometric properties in military populations is limited, and there is no universally accepted cut-off score on the K10 to distinguish clinical vs. sub-clinical levels of distress. The objective of this study was to examine the psychometric properties of the K10 in Canadian Armed Forces personnel. Data on 6700 Regular Forces personnel were obtained from the 2013 Canadian Forces Mental Health Survey. The internal consistency and factor structure of the K10 (range, 0–40) were examined using confirmatory factor analysis (CFA). Receiver Operating Characteristic (ROC) analysis was used to select optimal cut-offs for the K10, using the presence/absence of any of four past-month disorders as the outcome (posttraumatic stress disorder, major depressive episode, generalized anxiety disorder, and panic disorder). Cronbach’s alpha (0.88) indicated a high level of internal consistency of the K10. Results from CFA indicated that a single-factor 10-item construct had an acceptable overall fit: root mean square error of approximation (RMSEA) = 0.05; 90% confidence interval (CI):0.05–0.06, comparative fit index (CFI) = 0.99, Tucker-Lewis Index (TLI) = 0.99, weighted root mean square residual (WRMR) = 2.06. K10 scores were strongly associated with both the presence and recency of all four measured disorders. The area under the ROC curve was 0.92, demonstrating excellent predictive value for past-30-day disorders. A K10 score of 10 or greater was optimal for screening purposes (sensitivity = 86%; specificity = 83%), while a score of 17 or greater (sensitivity = 53%; specificity = 97%) was optimal for prevalence estimation of clinically significant psychological distress, in that it resulted in equal numbers of false positives and false negatives. Our results suggest that K10 scale has satisfactory psychometric properties for use as a measure of non-specific psychological distress in the military population.
Citation: Sampasa-Kanyinga H, Zamorski MA, Colman I (2018) The psychometric properties of the 10-item Kessler Psychological Distress Scale (K10) in Canadian military personnel. PLoS ONE 13(4): e0196562. https://doi.org/10.1371/journal.pone.0196562
Editor: Antonio Palazón-Bru, Universidad Miguel Hernandez de Elche, SPAIN
Received: April 28, 2017; Accepted: April 16, 2018; Published: April 26, 2018
Copyright: © 2018 Sampasa-Kanyinga et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The data underlying this study belong to the Federal Research Data Centre (FRDC) - Statistics Canada. Researchers may request access to data as detailed on the Statistics Canada website (http://www.statcan.gc.ca/eng/health/acces). The authors did not have special access privileges.
Funding: The authors received funding for this work from the Canadian Department of National Defence via salary support and from research contract W7714–145967 with the Canadian Institute of Military and Veteran Health Research and the University of Ottawa. This work was also partly supported by the Canada Research Chairs Program (Dr. Ian Colman). The Department of National Defence provided all funding for the collection of data for the military population by Statistics Canada. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Military personnel are known to have a significant risk of exposure to occupational trauma, which increases their risk of many mental disorders and psychological distress, a phenomenon underlined by the past fifteen years of armed conflict in southwest Asia [1]. Population mental health surveillance data are therefore necessary to guide prevention and control efforts in this at-risk population [2]. One of the key approaches for this crucial task is to administer surveys that include disorder-specific instruments, with a primary focus on conditions of particular concern in the military, notably PTSD, depression, and alcohol use disorders [3]. However, focusing only on the prevalence of specific disorders may underestimate the extent of impaired mental health, given that it is challenging to reliably assess each and every possible disorder in prevalence surveys. There is certainly evidence of a substantial prevalence in military personnel of disorders that have never been systematically assessed in Canadian military research, such as developmental disorders (such as ADHD), intermittent explosive disorder, bipolar disorders, and anxiety disorders such as specific phobias and obsessive-compulsive disorder [4, 5]. Given the practical constraints of assessing many different disorders in health surveys, using brief proxy measures, such as general psychological distress [6], is appealing.
Psychological distress is commonly measured on surveillance surveys using the 10-item Kessler Psychological Distress Scale (K10) measure, which was specifically designed for that purpose [7, 8]. Its developers relied on earlier work suggesting that while measures of psychological distress had items targeting a seemingly diverse set of cognitive, behavioural, emotional, psychophysiological processes, these items tended to load strongly on a single factor conceptualized as “generalized” or “non-specific” psychological distress, which can be contrasted with more specific types of distress related to specific disorders (e.g., distress surrounding obsessions and compulsions in obsessive compulsive disorder) [9, 10]. This important observation opened the door to the creation of a brief, unidimensional measure of generalized psychological distress for use on population health surveys [7]. In their ground-breaking work, Kessler and colleagues developed the K10 through a systematic process in which they first identified 612 candidate items from 18 different instruments, and then gradually reduced these using a variety of techniques (including factor analysis to ensure unidimensionality and item response theory to optimize their measurement of the underlying construct) to come up with the final 10 items in the K10 and its further reduced sibling, the 6-item K6 [7].
The psychometric properties of the K10 (and a further reduced six-item K6) have been extensively examined in civilian populations [11–13]. Exploratory factor analysis has suggested a unidimensional factor structure of the K6 across the general populations of at least 14 nations [14], while similar analysis with the K10 has suggested the possibility of a second factor or even up to four factors [15]. Some investigators have confirmed a single higher order factor using confirmatory factor analysis (CFA), while others found that a model with four factors loading on two first order factors (representing anxiety and depression) fit the data best. There is thus some uncertainty as to the true factor structure of the measures and the extent to which they are invariant across contexts and populations. Nevertheless, treated as a simple scale, the K10 (and K6) consistently show very good reliability as measured by Cronbach’s alpha [12, 16, 17].
The convergent validity evidence of the K10 with other measures of psychological distress (such as the General Health Questionnaire; GHQ) is well-established [8, 17]. Finally, the criterion validity of the K6 and K10 as a tool for predicting serious mental illness is very good to excellent, with the area under the receiver operating characteristic curve (AUC) ranging from 0.76 to 0.85 across 14 nations for the K6 [14] and from 0.87 to 0.88 in two nations for the K10 [7]. Strong predictive value has also generally been noted for past-year mental disorders more broadly (that is, in those with both serious and less serious mental illness) [18], though AUCs as low as 0.71 have been reported [19], and there is some evidence of heterogeneity of AUC values across those of different racial and ethnic backgrounds [19]. The K10 has also been found to correlate with other important mental health-related outcomes [8], such as functional impairment and use of health services.
The K10 is most often treated analytically as a unidimensional scale, which ranges from 0 to 40 or from 10 to 50, depending on the investigator. The remainder of the paper has adjusted results of studies using the 10 to 50 scoring to 0 to 40 scoring to facilitate comparisons. For certain applications (e.g., screening for mental disorders vs. estimation of the prevalence of significant distress) one or more cut-offs have been proposed; these cut-offs vary somewhat depending on the context and the underlying population. The most important consideration in selecting a cutoff is the intended use of the measure. For clinical screening, high sensitivity is essential, and trade-off in terms of lack of specificity can be tolerated provided that subsequent definitive diagnostic steps (e.g., a consultation with a clinician) are not resource-intensive or risky. Screening thresholds will greatly overestimate the prevalence of common mental disorders—a consequence of their low specificity. For example, at a commonly used screening cut-off of 9 or higher, 37.9% of the Canadian general population screened positive, while the prevalence of any mood or anxiety disorder in the population was 15.4% [20]. For epidemiological surveillance of the prevalence of common mental disorders, it may be preferable to have a cut-off that results in roughly an equal number of false positives and false negatives. Such a cut-off results in equal misclassification for those with and without the disease. Consequently, a less sensitive cut-off can be used to more accurately and efficiently determine the prevalence of common mental disorders in the population level or for surveillance purposes. These two types of cut-offs (“screening” vs. “prevalence estimation”) are not interchangeable, justifying the value of determining both within the same population [21]. Others have used multiple cut-offs to split populations into four bands representing low, moderate, high, and very high levels of distress at -points cut of 10, 16, 22, and 30, respectively [22, 23].
Thus, the psychometric properties of the K10 (and K6) have been extensively explored in civilian populations. The measure has also been used for both population health surveillance and/or screening of military personnel in several nations [24–26]. However, documentation of its psychometric properties in military populations is limited. The most extensive validation was undertaken by Blanc and colleagues, who explored its factor structure, internal consistency, concurrent validity, and predictive value for occupational impairments in Canadian military personnel while deployed in Afghanistan [26]. They found that a model with four first-order factors loading on a single higher order factor fit the data much better than a model with just a single higher-order factor (comparative fit indices (CFI) of 0.91 vs. 0.71, respectively) [26]. Internal consistency was very good to excellent (Cronbach’s alpha = 0.89). The K10 showed expected correlations with other commonly used measures of anxiety, depression, and post-traumatic stress. Finally, they noted that K10 score (treated unidimensionally) strongly predicted perceived occupational impairments related to mental health problems (AUC of 0.87), which was similar to the AUC of other measures of anxiety, depression, and post-traumatic stress.
Searle and colleagues explored the internal consistency and predictive value of the K10 for past-30 day and past-year mental disorders assessed using the much more detailed Composite International Diagnostic Interview (CIDI) in the general Australian military population in garrison [21]. They found that the internal consistency was excellent (Cronbach’s alpha = 0.91). Using any of nine past-30 day CIDI affective or anxiety disorders as the outcome, they identified an optimal screening cut-off of 9 or greater (sensitivity = 59%; specificity = 81% and an optimal cut-off for prevalence estimation (which they termed an “epidemiological cut-off”) of 15 or greater (sensitivity = 30%; specificity of 93%).
While these military K10 validation studies have been useful, they provide an incomplete picture of the psychometric properties of the instrument in the general military population. The findings of marginal fit for a unidimensional factor structure is concerning [26], and findings from the general population suggest potential concerns with heterogeneity in terms of its factor structure, optimal cut-offs, and predictive value for mental disorders assessed using more detailed means [27, 28].
Hence, this paper explores the psychometric properties of the K10 in serving Canadian Forces Regular Force personnel using data from the 2013 Canadian Forces Mental Health Survey. Specifically, it 1) assesses its internal structure using Confirmatory Factor Analysis, 2) reports the internal consistency of scores; 3) explores its correlation with outcomes related to mental health (self-rated mental health, disability, mental health services use, and positive mental health) for which a negative correlation is expected for self-rated mental health and positive mental health; 4) presents its association with the presence and recency of common mood, anxiety, and post-traumatic disorders assessed using the CIDI, and 5) using ROC analysis, it explores its overall predictive value and pinpoints optimal cut-offs for screening and for prevalence estimation.
Methods
Study population and data source
The Canadian Forces Mental Health Survey was a representative cross-sectional survey of serving Canadian Armed Forces personnel [29]. The target population was all Regular Force personnel as well as Reserve Force personnel who had deployed in support of the mission in Afghanistan who were in uniform in September 2012. The survey employed stratified random sampling framework (with stratification variables being military component (Regular vs. Reserve Force), military rank (in three categories), and Afghanistan mission deployment status (yes or no)) to ensure the representativeness to the Canadian Armed Forces as a whole. The sample size for the survey was determined to achieve a precision of +/-0.7% around a past-year disorder prevalence of 4.0%. Sampled individuals were contacted by Statistics Canada personnel, and data collection occurred from April—August of 2013. The data collection method was via an in-person, computer-assisted interview in the official language of the respondent’s choice (English or French). For most participants, data collection occurred in a private location in the workplace; some participants were interviewed in their homes. Participation in the survey was voluntary, and all participants provided written consent. Ethical review for the data collection occurred within the context of compliance with the relevant Statistics Canada legislation, policies, and directives, including the Statistics Act, as well as its Policy on Privacy and Confidentiality, Directive on Informing Survey Respondents, Directive on Record Linkage, and the Directive on Security of Sensitive Statistical Information. After compliance was demonstrated, the survey was approved by Canada`s Chief Statistician. The particular analysis in the present paper was approved through Statistics Canada`s Federal Research Data Centre Program (https://www.statcan.gc.ca/eng/rdc/frdc_application_process_guidelines), which is responsible for ensuring the scientific merit and compliance with ethical and other standards. This approval process involves review by the Senior Statistical Focal Point for the Department of National Defence, as well as Statistics Canada`s Departmental Audit Committee, and the Director of Subject Matter for the particular theme of the proposed analysis. Data were obtained from Statistics Canada, and the authors had no access to any potentially identifying participant information. More detailed information about the methodology of the 2013 Canadian Forces Mental Health Survey is available elsewhere [1]. The analyses of the present study are restricted to the Regular Force sample (N = 6,700; with a response rate of 80%), representative of approximately 68,000 Regular Force personnel.
Measures
Sociodemographic and military characteristics of respondents.
Sample characteristics were described using administrative data from the CAF (age, sex, element [Army, Navy, or Air Force], and rank category or via survey items developed by Statistics Canada for its household surveys (language of interview, marital status, educational attainment, and household income).
Psychological distress.
Psychological distress was measured using the 10-item K10 scale, which is described in detail in the Introduction. Responses were scored on a five-point ordinal scale reflecting how much of over the past month time respondents had experienced 10 symptoms, such as “feeling tired out for no good reason” and “sad or depressed”. The measure has five response categories ranging from 0 (none of the time) to 4 (all of the time). The items were summed to generate a total score ranging from 0 to 40, with higher scores indicating higher levels of psychological distress.
Self-rated mental health.
Self-rated mental health was assessed using a single item: “In general, would you say your mental health is…”, with response options of poor, fair, good, very good, and excellent. The item was treated as a scale variable with poor = 1 and excellent = 5, so a higher score indicated better self-rated mental health.
Disability.
Disability across six life domains was assessed using the World Health Organization Disability Assessment Schedule, Version 2 (WHODAS-2), using the “complex” scoring algorithm. WHODAS-2 scores range from 0–100, with higher scores reflecting greater disability. The WHODAS-2 has undergone validation in general population and in military samples [30].
Past-year professional mental health services use.
Past-year professional mental health services use was assessed using the “Services” module of the Composite International Diagnostic Interview (CIDI). Respondents were asked: “During the past 12 months, have you seen, or talked on the telephone to, any of the following people about problems with your emotions, mental health or use of alcohol or drugs?”, with response options of psychiatrist, psychologist, general practitioner or family doctor, nurse (including a case manager), or social worker, counselor, or psychotherapist. Those responding affirmatively to any of these were categorized as having any past-year professional mental health services use.
Positive mental health.
Positive mental health was assessed using the Mental Health Continuum, Short-form (MHC-SF), a 14-item scale assessing three dimensions of positive mental health (positive emotional, psychological, and social well-being [31]. The items assess how often, over the past month, the respondent had felt… (happy, that people are basically good, that [they] had warm and trusting relationships with others, etc.), with response categories of never, once or twice, about once a week, about two or three times a week, almost every day, and every day. The items were scored from 0 (never) to 5 (every day) and then summed (range, 0–70), with higher scores representing greater positive mental health. The MHC-SF has been validated in general populations but not, to our knowledge, in military populations. It demonstrated a Cronbach’s alpha of 0.90 and an ordinal alpha of 0.99 on the current data, indicating a high level of internal consistency.
Mental disorders.
Mental disorders were assessed with the World Health Organization World Mental Health Composite International Diagnostic Interview (WMH-CIDI), which is a comprehensive, structured lay-administered psychiatric interview for the assessment of mental disorders [32]. WMH-CIDI has shown good consistency with clinical diagnostic instruments [33, 34]. The mental disorders measured include: posttraumatic stress disorder (PTSD), major depressive episode (MDE), generalized anxiety disorder (GAD), and panic disorder (PD)”; The DSM-IV criteria were used to create an algorithm to classify people as having the mental health conditions. WMH-CIDI identified individuals who had ever met criteria for mental disorders in their lifetime (lifetime disorders) or during the past 12 months (past-year disorders). We have also identified those with a WMH-CIDI diagnosed lifetime disorder but no past-year disorder (remote disorders), those with past-year disorder but no past-month mood and anxiety disorder (recent disorders), and those with past-month mood and anxiety disorder (current disorders). For those with a past-year disorder, recency was assessed using a single item in each of the disorder-specific modules that asked how recently an episode of the key features of the disorder (e.g., a period of low mood or anhedonia in those with depression) had occurred, with options including the past month, two to six months previously, and more than six months previously. One aggregate variable was constructed indicating those with any mood or anxiety disorder (PTSD, MDE, GAD, or PD). For this indicator, “never” stands for never having had any of the four lifetime disorders; “remote” for those who had at least one lifetime disorder but no past-year disorders; “recent” for those who had at least one past-year disorder but no 30-day disorder; and “current” for individuals who had at least one 30-day disorder.
Statistical analysis
The factor structure of the K10 was examined via confirmatory factor analysis (CFA) using structural equation modeling on the K10 items (treated as ordinal variables and hence using the polychoric correlation matrix and MPlus’ WLSMV estimator). We explored two competing models based on Blanc and colleague’s CFA work on another Canadian military sample [26]. Specifically, Model 1 had all 10 items loading on a single higher-order factor representing general distress. Model 2 had the items loading on two correlated first-order factors representing depression and anxiety (Fig 1).
Note: Factor 1 = Anxiety; Factor 2 = Depression.
The version of the K10 used on the survey included three items with skip patterns. For example, one item asked how often the respondent had been “nervous”, and the next item then asked how often the respondent had been “so nervous that nothing could calm you down.” Those answering “never” to the first item would skip the second item, with a response of “never” (score = 0) being reasonably imputed for it. To address these interdependencies, we specified correlated error terms for the three pairs of items with such skip patterns in both of our models [12]. Goodness of fit was determined using absolute (i.e., chi-square and root-mean-square error of approximation (RMSEA)) and comparative (i.e. comparative fit index (CFI) and Tucker-Lewis Index (TLI)) fit indices. Model fit was considered to be acceptable if RMSEA is equal or lower than 0.08, CFI and TLI are equal or greater than 0.95, and if WRMR is equal or greater than 0.90 [35]. However, we did not rely on the chi-square test to examine model fit because current analyses used a large sample size, and chi-square is sensitive to sample size [36]. Internal consistency of the K10 items was tested using Cronbach’s alpha (treating the items as scale variables) and confirmed using an ordinal alpha that addressed potential violations of assumptions, including normality and lack of correlated errors. Given the construct-confirming nature of the CFA, these analyses were unweighted [37].
The remaining analyses were completed with SPSS version 24.0 (IBM Corp., Armonk, NY), and all were weighted to make the results representative of the underlying population. The association of the K10 score with mental health outcomes were explored using Pearson correlation coefficients (r) for disability, positive mental health, and self-rated mental health and point-biserial correlations (rpb) for past-year mental health services use. The association of the K10 score with the presence and recency of the four mood, anxiety, and post-traumatic disorders assessed by the survey was examined using the mean value of the K10 in those with current, recent, and remote disorders (as defined above). In keeping with Statistics Canada’s confidentiality restrictions for this survey, unweighted cell counts are not reported, values pertaining to a cell with an unweighted count of less than 5 are suppressed, and all proportions were calculated using weighted cell counts rounded to the nearest 20.
Receiver Operating Characteristic (ROC) analysis was used to select optimal cut-offs for the K10 [38, 39], using the presence/absence of past-month mood and anxiety disorders as the outcome. Prevalence-dependent and prevalence-independent ROC indices were calculated manually from weighted contingency table data, using proportions calculated from rounded, weighted cell counts. AUC was estimated non-parametrically using SPSS’s ROC Curve analysis command; this required rescaling of the fractional sampling weights to convert them to integers to accommodate the non-parametric approach. Confidence intervals (CI) for weighted analyses were estimated using a manual bootstrap approach with 500 bootstrap replicate weights provided by Statistics Canada. Non-overlapping 95% CIs indicate statistically significant differences [40]. In accordance with Statistics Canada’s data quality standards, estimates with high coefficient of variation (16.5–33.3%) are flagged, and those with a coefficient of variation of more than 33.3% are suppressed.
Following the approach of a previous study that examined the validity of K10 in Australian military personnel [21], we examined both a “screening” and “prevalence estimation” cut-off. The screening cut-off was explored using the Youden index [41]. This index is equivalent to maximization of the sum of sensitivity and specificity [42, 43]. We also explored the extent of misclassification at a given cut-off using contingency tables, with the optimal prevalence estimation cut-off being one in which the extent of misclassification was the same in both directions (that is, that the number of false positives and false negatives are closest together) [21]. Cases representing less than 0.1% were excluded due to missing data on K10 score.
Missing data for other variables were handled through complete case analysis.
Results
Participant characteristics
Table 1 presents demographic and military characteristics of the sample. Respondents were predominantly young and middle-aged, male, and married or in common law relationships. More than half had a post-secondary degree and a household income of $80,000 or more. Slightly more than half were in the Army and were junior non-commissioned members (NCMs).
Confirmatory factor analysis (CFA) and internal consistency
Results from the CFAs for two competing models are presented in Table 2. Model 1 had all 10 items loading on a single higher-order generalized distress factor, whereas Model 2 had the 10 items loading on 2 highly-correlated first-order factors representing anxiety and depression, respectively (Fig 1). In absolute terms, both models had adequate fit across all fit indices, as defined in the Methods section. However, the more parsimonious single factor model (Model 1) fit almost as well (RMSEA = 0.05 and 90% CI: 0.05–0.06, CFI = 0.99, TLI = 0.99, WRMR = 2.06) and thus retained for subsequent analyses. The K10 items demonstrated a Cronbach’s alpha of 0.88 and an ordinal alpha of 0.92, indicating a high level of internal consistency.
K10 and outcomes known to be related to distress and mental disorders
K10 had the expected positive correlation with outcomes known to be related to distress and mental disorders, including disability (r = 0.63; 95% CI: 0.62–0.65) and past-year mental health services use (r = 0.57; 95% CI: 0.55–0.60). In contrast, the K10 score had the expected negative correlation with self-rated mental health (r = -0.43; 95% CI: -0.40 –-0.45) and positive mental health (rpb = -0.60; 95% CI: -0.62 –-0.58).
K10 and the presence and recency of mental disorders
K10 distress was consistently related to the presence and recency of MDE, PTSD, GAD, PD, and any of the foregoing disorders (Table 3). Individuals with current (past-month) disorders had the highest levels of distress, followed by those with recent disorders. These were all followed much more distantly by those with remote disorders, though distress levels in those with remote disorders were significantly higher than those without any past disorders.
ROC analysis
Fig 2 illustrates the ROC curve using any past-month disorder as the reference. The K10 had an AUC of 0.915 (95% CI, 0.900–0.929), showing that K10 score was a strong predictor of the outcome of any past-month disorder. The ROC analysis results for prevalence-dependent and prevalence-independent indices are shown in Tables 4 and 5, respectively. Scanning the indices across the potential cut-offs, no clear K10 score thresholds are apparent with respect to most of the indices. For % correctly classified and specificity, however, the values plateau around a cut-off of 18 or so, beyond which little improvement is seen.
Note: Coordinates suppressed for cut-offs <4 or > 29 to comply with Statistics Canada’s release guidelines related to small cell sizes. Area under the ROC curve (AUC) = 0.915; 95% CI: 0.900–0.929.
For the purpose of estimation of the prevalence of any of the four past-month disorders assessed, an optimal cut off of 17 or greater (seen in 7.1% of the population) is identified in that the false positive and false negative proportions are closest to equal (3.3% each; Table 5). The Youden Index is maximized at a cut-off of 10 or greater (Table 5), which thus corresponds to an optimal screening cut-off. At that cut-off, the sensitivity and specificity are 0.869 and 0.824, respectively. At the outcome prevalence seen in our population (7.0%), the PPV at the screening cut-off is modest (0.274) but the NPV is excellent (0.988). At the screening cut-off, the positive likelihood ratio was 4.9 and the negative likelihood ratio was 0.16, reflecting very good discriminatory power at that cut-off.
Discussion
Summary of key findings
In this paper, we sought to explore the psychometric properties of a widely-used general psychological distress scale, the K10. Using CFA, we confirmed that a model with all ten items loading on a single higher-order fit the data adequately across all fit indices, using a priori values for adequacy of fit. However, a model in which the ten items loaded on two first-order factors (representing anxiety and depression) fit the data slightly better. The K10 scale had excellent internal consistency (Cronbach’s alpha = 0.88). K10 scores were positively correlated with other outcomes related to distress and mental disorders, including self-rated mental health, disability, and past-year mental health services use; it was negatively correlated with positive mental health. K10 scores were significantly higher in those with MDE, PTSD, GAD, and PD (and an aggregate outcome of any of the foregoing). K10 scores were also much higher in those with more recent disorders. In ROC analysis, the K10 was a strong predictor of the outcome of any past-month disorder (AUC = 0.914). A cut-off of 17 or greater proved optimal for the purposes of estimation of the prevalence of past-month mood, anxiety, and post-traumatic disorders. In contrast, a cut-off of 10 or higher was identified as optimal for screening purposes.
Comparison with other relevant literature
The K10 was explicitly designed to be a unidimensional measure of generalized psychological distress, and most analyses have indeed found (as we did) a single-factor structure using factor analytic techniques and favourable internal consistency [27, 28]. Our findings cohere with those of Bougie et al. (2016), in that we found that a more complex model with two first-order factors loading fit the data slightly better—though the strong correlation between the two factors (0.92) suggested that it offered little conceptual advantage over a more parsimonious single-factor model. Others have also documented the expected association of K10 scores with self-rated mental health, disability, past-year mental health services use, and positive mental health [28, 44, 45].
A number of different cut-offs have been proposed for the K10 [17, 21, 26, 28], depending on the population studied, the outcome against which it was validated (and how it was measured), and the goal of the cut-off (e.g., for clinical screening vs. estimation of the prevalence of distress in population health surveys) [21]. The most comparable work in a military population comes from the Australian Defence Force, in which a slightly lower cut-off of 15 (converted to the 0–40 scaling we used) was deemed optimal for prevalence estimation [21]. However, their cut-off showed a sensitivity (0.30) that was lower than ours (0.53). Our optimal screening cut-off of 10 is also similar to their screening cut-off of 9 (converted to the 0–40 scaling we used). Their screening cut-off again showed a lower sensitivity 0.59 than ours (0.87). It is possible that the poor final response rate for the Australian survey (24%) and the use of different outcome measures (a broader range of mood and anxiety disorders assessed using ICD-10 criteria) explain the observed discrepancies. Blanc et al. (2014) instead proposed a screening cut-off of 6 (converted to the 0–40 scaling we used, based on the maximum of sum of sensitivity and specificity) in a convenience sample of CAF personnel serving in Afghanistan [26]. Differences in the study population, the context (a combat deployment), and the outcome (the presence of occupational impairment as opposed to past-month mood and anxiety disorders) likely account for the differences in that finding from our own. However, Cornelius et al. (2013) proposed an optimal screening cut-off of 14 for K10 (converted to the 0–40 scaling), using past-month DSM-IV mental disorders including any mood, anxiety, or substance use disorders (assessed by CIDI) in a general population sample [17]. Fassaert et al. (2009) documented cut-off scores of 12.5 for Moroccan and Turkish participants and 6.5 for ethnic Dutch participants for K10 (converted to the 0–40 scaling), using one-month DSM-IV diagnosis for depressive and/or anxiety disorder [28]. These findings on differing cut-offs demonstrate the importance of exploring the psychometric properties of the K10 in a given target population.
Limitations and strengths
This study has several limitations. First, we were limited to exploring the predictive value of the K10 for the four mood, anxiety, and post-traumatic disorders assessed with the disorder-specific modules of the CIDI. A number of other mental disorders have been shown to be prevalent in military personnel, including bipolar disorder, obsessive-compulsive disorder, and phobias [4, 21]. However, our K10 cut-offs may still identify those at risk for other disorders that cause similar levels of distress as those assessed in the present survey. Second, our survey did not include other measures of generalized distress, against which convergent validity evidence could be optimally assessed. Third, imposing any single cut-off (no matter how rigorously established) for a continuous construct like psychological distress measured by the K10 will always result in loss of information and misclassification. We did indeed find a meaningful amount of misclassification (false negatives plus false positives ~ 6.6%). Finally, the predictive value of the K10 in clinical screening programs (in which barriers to disclosure may exist, especially in military personnel) [46] may differ from that demonstrated in the present analysis.
Strengths of this study include validation of the K10 against a reliable outcome (disorders assessed by the CIDI), the generous sample size, and the high response rate. Another strength relates on the identification of cutoffs for both screening and for estimation of the prevalence of mood, anxiety, and post-traumatic disorders.
Implications for research and clinical practice
Taken as a whole, these findings provide satisfactory convergent validity evidence of the K10 in the Canadian military population. Specifically, the CFA findings and internal consistency provide reassurance as to internal structure as a source of validity evidence (that is, that the measure is unidimensional as intended). The association of the K10 with outcomes known to be associated with distress and mental disorders provides further evidence of construct validity. Its negative association with positive mental health also provides convergent evidence of validity. Finally, its association with the presence and recency of mood, anxiety, and post-traumatic disorders provide further support that the K10 was designed to accurately predict the presence of mental disorders, and it did so in our population.
Those wishing to use the scale as a dependent or independent variable in future research or as a screening tool can thus do so with knowledge of its psychometric properties and the extent to which these provide evidence of its validity as a tool for the estimation of the prevalence of common mental disorders (and hence, as a potential tool for screening) in the Canadian military population. However, those using the scale in any population should be aware of the somewhat unusual objectives in its development: to efficiently and pragmatically assess the prevalence of a range of disorders constituting “serious mental illness” in a population.
Our presentation of the full range of ROC indices will allow researchers and clinicians to select a cut-off (or cut-offs) that meet their needs. Topics of future research with the K10 in military populations might include exploration of the reasons behind the observed misclassification. False positive cases may represent those who mental disorders other than those assessed on the survey or they may represent those with elevated distress in the absence of a disorder. False negative cases may represent those with either mild or sub-threshold forms (limited symptom burden, limited functional impact) of the underlying disorder. Exploration of different scoring approaches to the K10 items (such as those identified through item response theory analysis) might ultimately lead to greater predictive value and hence greater research and clinical utility [7].
Conclusion
The present study suggests that K10 scale has psychometric properties that favour its use as a measure of generalized psychological distress in Canadian military personnel, both for the purposes of screening and estimation of the prevalence of past-month mood, anxiety, and post-traumatic disorders.
Acknowledgments
The authors received funding for this work from the Canadian Department of National Defence via salary support and from research contract W7714–145967 with the Canadian Institute of Military and Veteran Health Research and the University of Ottawa. This work was also partly supported by the Canada Research Chairs Program (Dr. Colman). The Department of National Defence provided all funding for the collection of data for the military population by Statistics Canada.
References
- 1. Zamorski MA, Bennett RE, Boulos D, Garber BG, Jetly R, Sareen J. The 2013 Canadian Forces Mental Health Survey: Background and Methods. Can J Psychiatry. 2016;61(1 Suppl):10S–25S. pmid:27270738
- 2. Stoltz RF. Editorial: the mental health of our deploying generation MSMR. 2013;20(7):2–3. pmid:23927057
- 3.
Department of National Defence. The Surgeon General’s mental health strategy. Ottawa (ON): Department of National Defence;. 2013.
- 4. Kessler RC, Heeringa SG, Stein MB, Colpe LJ, Fullerton CS, Hwang I, et al. Thirty-day prevalence of DSM-IV mental disorders among nondeployed soldiers in the US Army: results from the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). JAMA Psychiatry. 2014;71(5):504–13. pmid:24590120
- 5. Hooff MV, McFarlane AC, Davies CE, Searle AK, Fairweather-Schmidt AK, Verhagen A, et al. The Australian Defence Force Mental Health Prevalence and Wellbeing Study: design and methods. Eur J Psychotraumatol. 2014;5. pmid:25206944
- 6. Wittchen H-U. Screening for serious mental illness: methodological studies of the K6 screening scale. International Journal of Methods in Psychiatric Research. 2010;19(S1):1–3.
- 7. Kessler RC, Andrews G, Colpe LJ, Hiripi E, Mroczek DK, Normand SL, et al. Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychol Med. 2002;32(6):959–76. pmid:12214795
- 8. Andrews G, Slade T. Interpreting scores on the Kessler Psychological Distress Scale (K10). Australian and New Zealand J Public Health. 2001;25(6):494–7.
- 9. Dohrenwend BP, Shrout PE, Egri G, Mendelsohn FS. Nonspecific psychological distress and other dimensions of psychopathology: Measures for use in the general population. Arch Gen Psychiatry. 1980;37(11):1229–36. pmid:7436685
- 10.
Link BG, Dohrenwend BP. Formation of hypotheses about the true relevance of demoralization in the United States. In Mental Illness in the United States: Epidemiological Estimates (ed. Dohrenwend B. P., Dohrenwend B. S., Gould M. S., Link B., Neugebauer R. and Wunsch-Hitzig R.), pp. 114±132. Praeger: New York. 1980.
- 11. Jong Won M, Sun Hae L. Validation of the K6/K10 Scales of Psychological Distress and Their Optimal Cutoff Scores for Older Koreans. The International J Aging Human Dev. 2015;80(3):264–82. pmid:26195502
- 12. Bougie E, Arim RG, Kohen DE, Findlay LC. Validation of the 10-item Kessler Psychological Distress Scale (K10) in the 2012 Aboriginal Peoples Survey. Health Rep. 2016;27(1):3–10. pmid:26788720
- 13. Oakley Browne MA, Wells JE, Scott KM, McGee MA, New Zealand Mental Health Survey Research T. The Kessler Psychological Distress Scale in Te Rau Hinengaro: the New Zealand Mental Health Survey. Aust N Z J Psychiatry. 2010;44(4):314–22. pmid:20307164
- 14. Kessler RC, Green JG, Gruber MJ, Sampson NA, Bromet E, Cuitan M, et al. Screening for Serious Mental Illness in the General Population with the K6 screening scale: Results from the WHO World Mental Health (WMH) Survey Initiative. Int J Methods Psychiatric Res. 2010;19(0 1):4–22. pmid:20527002
- 15. Brooks RT, Beard J, Steel Z. Factor structure and interpretation of the K10. Psychol Assessment. 2006;18(1):62.
- 16. Sakurai K, Nishi A, Kondo K, Yanagida K, Kawakami N. Screening performance of K6/K10 and other screening instruments for mood and anxiety disorders in Japan. Psychiatr Clin Neurosciences. 2011;65(5):434–41. pmid:21851452
- 17. Cornelius BL, Groothoff JW, van der Klink JJ, Brouwer S. The performance of the K10, K6 and GHQ-12 to screen for present state DSM-IV disorders among disability claimants. BMC Public Health. 2013;13:128. pmid:23402478
- 18. Furukawa TA, Kessler RC, Slade T, Andrews G. The performance of the K6 and K10 screening scales for psychological distress in the Australian National Survey of Mental Health and Well-Being. Psychol Med. 2003;33(2):357–62. pmid:12622315
- 19. Andersen L, Grimsrud A, Myer L, Williams D, Stein D, Seedat S. The psychometric properties of the K10 and K6 scales in screening for mood and anxiety disorders in the South African Stress and Health study. Int J Methods Psychiatr Res. 2011;20(4):215–23. pmid:22113964
- 20. Caron J, Fleury MJ, Perreault M, Crocker A, Tremblay J, Tousignant M, et al. Prevalence of psychological distress and mental disorders, and use of mental health services in the epidemiological catchment area of Montreal South-West. BMC Psychiatry. 2012;12:183. pmid:23110632
- 21. Searle AK, Van Hooff M, McFarlane AC, Davies CE, Fairweather-Schmidt AK, Hodson SE, et al. The validity of military screening for mental health problems: diagnostic accuracy of the PCL, K10 and AUDIT scales in an entire military population. Int J Methods Psychiatr Res. 2015;24(1):32–45. pmid:25511518.
- 22.
Australian Bureau of Statistics. Use of the Kessler Psychological Distress Scale in ABS health surveys. Information paper. Canberra: Australian Bureau of Statistics. 2001.
- 23. Slade T, Grove R, Burgess P. Kessler Psychological Distress Scale: normative data from the 2007 Australian National Survey of Mental Health and Wellbeing. Aust N Z J Psychiatry. 2011;45(4):308–16. pmid:21332432
- 24.
Hodson SE, McFarlane AC, Van Hooff M, Davies C. Mental Health in the Australian Defence Force– 2010 ADF Mental Health Prevalence and Wellbeing Study: Executive Report, Department of Defence: Canberra. 2011.
- 25. Reeves WC, Pratt LA, Thompson W, Dhingra SS, McKnight-Eily LR, Harrison L, et al. Mental illness surveillance among adults in the United States. MMWR Surveill Summ. 2011;60(Suppl. 3):1–29
- 26. Blanc S, Zamorski M, Ivey G, Edge HM, Hill K. How much distress is too much on deployed operations? Validation of the Kessler Psychological Distress Scale (K10) for application in military operational settings. Military Psychol. 2014;26(2):88.
- 27. Sunderland M, Mahoney A, Andrews G. Investigating the Factor Structure of the Kessler Psychological Distress Scale in Community and Clinical Samples of the Australian Population. J Psychopathol Behav Assess. 2012;34(2):253–9.
- 28. Fassaert T, De Wit MAS, Tuinebreijer WC, Wouters H, Verhoeff AP, Beekman ATF, et al. Psychometric properties of an interviewer-administered version of the Kessler Psychological Distress scale (K10) among Dutch, Moroccan and Turkish respondents. Int J Methods Psychiatric Res. 2009;18(3):159–68. pmid:19701920
- 29.
Statistics Canada. Canadian Forces Mental Health Survey (CFMHS), 2014. Retrieved from http://www23statcangcca/imdb/p2SVpl?Function=getSurvey&SDDS=5084 [Last accessed February 01, 2018].
- 30. Weeks M, Garber BG, Zamorski MA. Disability and Mental Disorders in the Canadian Armed Forces. Can J Psychiatry. 2016;61(1 Suppl):56S–63S. pmid:27270743
- 31.
Keyes CLM. Atlanta: Brief description of the mental health continuum short form (MHC-SF), 2009. https://www.aacu.org/sites/default/files/MHC-SFEnglish.pdf. [On–line, retrieved January 26, 2018].
- 32.
American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision. Washington (DC): American Psychiatric Press Inc. 2000.
- 33. Wittchen HU. Reliability and validity studies of the WHO—Composite International Diagnostic Interview (CIDI): a critical review. J Psychiatr Res. 1994;28(1):57–84. pmid:8064641
- 34. Haro JM, Arbabzadeh-Bouchez S, Brugha TS, de Girolamo G, Guyer ME, Jin R, et al. Concordance of the Composite International Diagnostic Interview Version 3.0 (CIDI 3.0) with standardized clinical assessments in the WHO World Mental Health surveys. Int J Methods Psychiatr Res. 2006;15(4):167–80. pmid:17266013.
- 35. Jackson DL, Gillaspy JA, Purc-Stephenson R. Reporting practices in confirmatory factor analysis: an overview and some recommendations. Psychol Methods. 2009;14(1):6–23. pmid:19271845
- 36. DiStefano C, Hess B. Using confirmatory factor analysis for construct validation: An empirical review. J Psychoeduc Assess. 2005;23(3):225–41.
- 37. Brummel BJ, Drasgow F. The effects of estimator choice and weighting strategies on confirmatory factor analysis with stratified samples. Appl Multivariate Res. 2010;13(2):113–28.
- 38. Streiner DL, Cairney J. What’s under the ROC? An introduction to receiver operating characteristics curves. Can J Psychiatry. 2007;52(2):121–8. pmid:17375868
- 39. Hajian-Tilaki K. Receiver Operating Characteristic (ROC) Curve Analysis for Medical Diagnostic Test Evaluation. Caspian J Internal Med. 2013;4(2):627–35.
- 40. Schenker N, Gentleman JF. On judging the significance of differences by examining the overlap between confidence intervals. Am Stat. 2001;55(3):182–6.
- 41. Zweig MH, Campbell G. Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem. 1993;39(4):561–77. pmid:8472349
- 42. Bewick V, Cheek L, Ball J. Statistics review 13: receiver operating characteristic curves. Crit Care. 2004;8(6):508–12. pmid:15566624
- 43. Youden WJ. Index for rating diagnostic tests. Cancer. 1950;3(1):32–5. pmid:15405679
- 44. Anderson TM, Sunderland M, Andrews G, Titov N, Dear BF, Sachdev PS. The 10-Item Kessler Psychological Distress Scale (K10) as a Screening Instrument in Older Individuals. The Am J Geriatric Psychiatry. 2013;21(7):596–606. https://doi.org/10.1016/j.jagp.2013.01.009.
- 45. Ustun TB, Chatterji S, Kostanjsek N, Rehm J, Kennedy C, Epping-Jordan J, et al. Developing the World Health Organization Disability Assessment Schedule 2.0. Bull World Health Organ. 2010;88(11):815–23. pmid:21076562
- 46. Zamorski MA, Rusu C, Garber BG. Prevalence and correlates of mental health problems in Canadian Forces personnel who deployed in support of the mission in Afghanistan: findings from postdeployment screenings, 2009–2012. Can J Psychiatry. 2014;59(6):319–26. pmid:25007406