https://orcid.org/0000-0003-3778-7161
Figures
Abstract
The Hospital Anxiety and Depression Scale (HADS) is commonly used to detect depressive or anxious states, but its 14-item questionnaire is time-consuming. Visual analog scales (VAS) are easy to use and quick to implement. Although the VAS has been validated to assess pain and occupational stress, VAS scores for anxiety and mood have never been evaluated in the workplace. We aimed to validate the use of visual analog scales (VAS) for anxiety and mood compared to HADS in workers. A HADS self-reported questionnaire associated with VAS assessing perceived anxiety and mood on a horizontal line of 100 mm was administered to 182 workers, with a second test (retest) proposed one week later. Sociodemographic, characteristics of work, sleep, well-being, and stress were also assessed. VAS anxiety and mood correlated with the HADS sub-scores (0.70 and 0.65, respectively). The test-retest reliability was good. Optimal VAS cut-offs were ≥ 60/100 for anxiety and ≤ 60/100 for mood, to define at-risk patients. The VAS is quick to perform, easy to use, and reliable for screening depression and anxiety in occupational medicine. We recommend validated questionnaires for at-risk patients.
Trial registration. Clinicaltrials.gov: NCT02596737. Available at: https://www.clinicaltrials.gov/ct2/show/NCT02596737.
Citation: Dutheil F, Palgen C, Brousse G, Cornet T, Mermillod M, Lakbar I, et al. (2024) Validation of visual analog scales of mood and anxiety at the workplace. PLoS ONE 19(12): e0316159. https://doi.org/10.1371/journal.pone.0316159
Editor: Alice Mannocci, Telematic University San Raffaele Rome: Universita Telematica San Raffaele Roma, ITALY
Received: August 20, 2024; Accepted: December 8, 2024; Published: December 31, 2024
Copyright: © 2024 Dutheil et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All files are available from the "Validation of VAS Anxiety and Depression" database (URL https://osf.io/6vcsg/?view_only=3f4a891812544f1999da8caaba674c28).
Funding: The study is integrally funded by WittyFit (https://www.cegid.com/fr/produits/cegid-wittyfit/), Paris, France. The funding went to FD. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: TC is director of WittyFit. This does not alter our adherence to PLOS ONE policies on sharing data and materials. Other authors have declared that no competing interests exist.
Introduction
The World Health Organization (WHO) estimates that mental illness affects one in five people globally and one in three individuals in his/her lifetime [1]. Mental disorders represent the second leading cause of sickness absence [2–4] and the leading cause of disability. Anxiety disorders and mood disorders are common when related to mental illness [5]. The estimated prevalence of anxiety is 18% and mood disorders are 9.5% [6]. Most of them appear at the beginning or during working life [5] and can negatively affect the worker, the organization and the society. Comorbid anxiety and depressive disorders have the greatest impact on long-term work disability and absenteeism, emphasizing the need to address both conditions simultaneously [7]. Psychological ill-health affects human performance and increases the risk of accidents, especially in hazardous industries [8–10]. In manufacturing, an anxious worker is more likely to experience accidents, endangering himself and coworkers, but also increasing downtime and operational costs. Anxiety and depression can lead to both absenteeism [11], and presenteeism [12]. Presenteeism accounts for a significant portion of productivity loss, surpassing direct medical costs for depression [13]. Untreated mental health issues, particularly depression, contribute to a $44 billion annual loss of productive time in the USA [14]. Workplace interventions targeting mental health are moderately effective in improving work outcomes, while programs based on exposure or that combine mental, social and physical health interventions are the most effective [15]. Companies with supportive environments observe a 30% reduction is stress-related absenteeism [16]. For every dollar spent in such programs, companies see up to a $4 return in reduced absenteeism and healthcare costs [17]. Workplace mental health programs and interventions seem to help individual employees but also reduce larger societal costs [17, 18].
Screening and assessing the severity of symptoms is therefore essential to occupational health and should especially happen during consultation by an occupational physician. Among the multiple validated tests assessing anxiety and depression [19–21], the Hospital Anxiety and Depression Scale (HADS) is the most common test. Yet, the questionnaire consists of 14 items and requires extended periods during occupational health consultation, while ooccupational health physicians have limited time to deal with a large number of workers and worksites [22]. Visual analog scales (VAS) are reproducible tools, fast, and easy to use, with good psychometric characteristics, already validated for pain [23], occupational stress [24, 25], and job satisfaction [26]. Therefore, we hypothesized that a VAS for mood and anxiety would identify at-risk workers with depressive or anxious symptoms, in comparison with the HADS.
Stress, sleep disturbance and working hours have been previously been linked with anxiety and depression scores using HADS [27, 28], and should be associated with VAS anxiety and mood.
The main objective of this study was to validate VAS mood and VAS anxiety vs. HADS in a population of active workers. The external validity was evaluated for both VAS by highlighting their relationships with sociodemographic, professional and well-being characteristics.
Materials and methods
Participants
An epidemiological, observational, descriptive and cross-sectional study was conducted between 2016-03-01 and 2016-05-12, for which self-questionnaires were sent to users of the WittyFit software [29]. The name WittyFit comes from Witty and Fitness and reflects the concept of health from the World Health Organization (WHO): to be in good physical and mental health. WittyFit is software whose objective is to promote well-being in companies, with an epidemiological and research conception. WittyFit performs a personalized evaluation by self-questionnaire. Participation in the study required written consent. This observational study received approval from the Ethics Committee of the University Hospital of Clermont-Ferrand, France (clinicaltrials.gov identifier NCT02596737) and from National Commission for Information Technology and Civil Liberties (CNIL). All data was anonymous, and the employees’ identity was not collected in the database. A human resource created a number to feed the database, which was then automatically modified by another number.
Participants were invited to participate in the study via the WittyFit software, which delivered the research invite to the entire workforce of participating companies, with the option to refuse. Any worker accepting to join the WittyFit research was considered eligible to participate. No other inclusion criteria were enforced.
Primary outcome
The primary objective of this study was to validate the two VAS (anxiety and mood) as an alternative to the HADS.
Zigmond and Snaith published HADS for the first time in 1983, in English [30–32]. This was then translated and validated in French by Lépine in 1985 and by Ravazi in 1989 [33–35]. This scale is used for screening for common psychopathological disorders. Seven items evaluate depressive symptoms (items 2, 4, 6, 8, 10, 12, 14): one for dysphoria, one for psychomotor slowdown and five for anhedonia. Seven other items measure anxiety (items 1, 3, 5, 7, 9, 11, 13). Each item is comprised of a four-point Likert scale. The scale of answers is sometimes reversed to avoid repetition bias. The questionnaire generates 2 sub-scores, corresponding to the two subscales of anxiety and depression, with two thresholds set at 7 and 11. Scores 0–7 refer to no anxiety or depressive disorders; 8–10 suspected anxiety or depressive disorders; 11–21 proven anxiety or depressive disorders [30–32, 36].
VAS assessed the perceived anxiety and mood of individuals at work, on a horizontal, non-calibrated line of 100 mm. The scales ranged from very low (0) to very high anxiety (100) for VAS anxiety; and ranging from very sad mood (0) to very good mood (100) for VAS mood.
Secondary outcomes
Secondary outcomes were defined as the measures of association between the HADS or VAS and workers’ personal and work-related characteristics. Sociodemographic data such as age, gender, education level, and marital status was collected. The characteristics of work were also collected, such as occupation, number of working hours per week, seniority within the company, the type of work schedule (fixed, shift, nightshifts) and the number of nightshifts per month. Body mass index was calculated from height and weight. Sleep quantity was evaluated by the number of sleeping hours per night [37]. Sleep quality, well-being, stress at work, and stress at home were evaluated with VAS ranging from very low (0) to very high (100) [22, 25].
Time of measurements
Participants could complete the whole questionnaire whenever they wished. All responders were also invited to complete the HADS and VAS anxiety and mood again a week later, to perform the test-retest approach. The handover time was approximately 15 minutes.
Statistical analysis
Sample size was determined according to COSMIN recommendations [38, 39] as follows. 1) “Rules-of-thumb vary from four to 10 subjects per variable, with a minimum number of 100 subjects to ensure stability of the variance-covariance matrix” and 2) “Often 0.70 is recommended as a minimum standard for reliability. We give a positive rating for reliability when the ICC or weighted Kappa is at least 0.70 in a sample size of at least 50 patients.”
The Stata software (Version 13, StataCorp, College Station, TX) was used, with a a two-sided Type I error of α = 5% to perform statistical analysis. Participants’ characteristics were expressed as mean ± standard deviation (SD) or median [interquartile range] for continuous data (assumption of normality assessed by the Shapiro-Wilk test) and as numbers and associated percentages for categorical parameters.
The test-retest reliability was performed using Pearson correlation coefficient, Lin concordance coefficient and Bland and Altman plots. The external validity was assessed using a correlation coefficient (Pearson or Spearman according to statistical distribution) between VAS and other psychological measures, such as the HADS score. Then, a ROC receiver operating characteristic (ROC) curve analysis were used to determine the optimal thresholds of VAS to predict HADS, according to clinical relevance and usual indexes reported in the literature (Youden index, Liu index and efficiency) [40, 41]. Sensitivity, specificity, positive and negative predictive values were calculated and presented with 95% confidence intervals. The agreement between the HADS anxiety and depression sub-score and corresponding VAS, according to cut-offs determined by ROC curve analysis, was evaluated using agreement rates. Finally, quantitative variables were compared between independent groups by ANOVA or the Kruskal-Wallis test, if ANOVA assumptions were not met (normality and homoscedasticity were analyzed using the Bartlett test). When appropriate, post-hoc tests were performed considering multiple comparisons (Tukey-Kramer post ANOVA and Dunn after Kruskal-Wallis). The comparisons between groups were carried out using the Chi-squared or Fischer’s exact test for categorical variables. When appropriate, a post-hoc test was used (Marascuillo procedure). External validity was assessed by comparing the associations of personal and work-related variables (sex, age, BMI, well-being VAS, sleep quality VAS, sleep duration, stress at work VAS, stress at home VAS, seniority in company, and weekly workload) between participants with and without risks of depression or anxiety, as measured by HADS and VAS. To assess the robustness of our primary analysis, a sensitivity analysis was performed on a subset of the population in which the primary outcomes (VAS anxiety and depression as well as HADS-A and HADS-D), but also the variables used to assess external validity evidence (sex, age, BMI, well-being VAS, Sleep quality VAS, Sleep duration, Stress at work VAS, stress at home VAS, seniority in company, weekly workload) were complete.
Results
Participants
Among the 1580 workers and users of WittyFit, 222 (14%) agreed to participate. The data for the primary outcomes HADS Anxiety (HADS-A), VAS anxiety, HADS Depression (HADS-D) and VAS mood was missing in 40 participants. The analysis was then conducted on 182 (82%) of the responders. Among them, 86 (47.3%) were women (with 26 missing data). The test-retest approach was calculated on 123 participants who answered twice to the HADS and the two VAS (anxiety and mood) (Fig 1). The average age was 41.4 ± 11.6 years old. Half of the workers were married (44%, 26 missing data). Most included workers were senior executives (57.7%) and had reached a master’s degree or higher (62.6%) (Table 1).
Abbreviations: *: The variables considered to define the sensitivity analysis were sex, age, BMI, well-being VAS, Sleep quality VAS, Sleep duration, Stress at work VAS, stress at home VAS, seniority in company, weekly workload. VAS: Visual Analog Scale.
Hospital anxiety and depression scale
To define the threshold on the VAS anxiety and mood compared to the HADS, we assumed for each sub score of anxiety and depression that a score ≤7 was normal and a score ≥ 8 was considered abnormally high.
The mean score of HADS-A was 6.9 ± 4.0. 108 (59.3%) participants had no anxiety symptoms (score ≤7) and 74 (40.7%) had anxiety symptoms (score ≥8) The mean score of HADS-D was 3.8 ± 3.0. 157 (86.3%) participants had no depressive symptoms (score ≤7) and 25 (13.7%) had depressive symptoms (score ≥8) (Table 2).
VAS anxiety and VAS mood: Cut-offs determination and agreement rate
Mean VAS anxiety was 50.4 ± 28.7. A best cut-off value at 63.5 (p <0.001) was calculated to assess the risk of anxiety using VAS Anxiety, with a sensibility of 71.6% (95CI: 59.9% to 81.5%) and a specificity of 86.1% (78.1% to 92.0%), an area under the curve of 0.79 (0.73 to 0.85), a positive predictive value of 78% (66.2% to 87.1%), a negative predictive value of 81.6% (73.2% to 88.2%), and an agreement of 80.2%.
Mean VAS mood was 65.4 ± 24. A best cut-off value at 58 (higher risk for values ≤ 58, p <0.001) was calculated to assess the risk of depression using VAS mood, with a sensitivity of 84% (95%CI: 63.9% to 95.5%), a specificity of 72.6% (64.9% to 79.4%), an area under the curve of 0.78 (0.70 to 0.86), a positive predictive value of 32.8% (21.6% to 45.7%) and a negative predictive value of 96.6% (91.5% to 99.1%) and agreement of 74.2%.
Therefore, we propose to set cut-offs at ≥60 for VAS anxiety and ≤60 for VAS mood. 76 (41.8%) workers had a VAS anxiety ≥60, and 67 (36.8%) workers had a VAS mood ≤ 60 (Table 3, Fig 2).
Using the VAS anxiety, data ranged from minimal to maximal values, with a reasonable standard deviation, as suggested by the variation coefficient around 0.57 for VAS anxiety. Using the VAS mood to 0.30, data ranged from 4 to 100, with a reasonable standard deviation (coefficient of variation of 0.37). No real ceiling of floor effects was observed; with 6% of participants for VAS anxiety and 7.7% for VAS mood having the lowest or highest possible score. The correlation between VAS anxiety and VAS mood was large (Spearman rho 0.51, p < 0.001). VAS anxiety and mood correlated with their respective HADS sub-scores (0.70 and 0.65, respectively, p < 0.001).
Validation of VAS anxiety and VAS mood—internal validity
Correlation between VAS anxiety/VAS mood and HADS-A/HADS-D.
Rank correlation coefficient found high correlation (0.70, p <0.001) between anxiety score from the HADS-A and the VAS anxiety, and moderate correlation (0.65, p <0.001) between depression score from the HADS-D and the VAS mood.
Test-retest reproducibility.
The analysis was performed on a subsample of 123 participants with available data for both test and retest. Lin concordance coefficient for VAS anxiety was 0.79 (95CI: 0.73 to 0.86) with difference on the retest of -1.4 ± 18.9 (95CI: -38.4 to 35.6). Lin concordance coefficient for VAS mood was 0.72 (95CI: 0.64, 0.81) with difference on the retest of -0.22 ± 18.8 (95CI: -37.1 to 36.7). The Bland and Altman plot is shown in Fig 3.
2.A. Bland-Altman plot for VAS Anxiety, 2.B. Bland-Altman plot for VAS Mood.
Validation of VAS anxiety and VAS mood—external validity
VAS anxiety.
According to the cut-off for VAS anxiety, age, BMI and seniority in the company were not correlated with anxiety symptoms but women were more affected by anxiety than men (p = 0.008). Moreover, a high degree of anxiety was correlated with a lower VAS quality of sleep (p <0.001) but not with duration of sleep (p = 0.46). Also, a high degree of anxiety was associated with higher stress at work (p <0.001) and at home (p <0.001) but not with a longer time spent at work (p = 0.7). Conversely, a lower VAS anxiety was linked to higher VAS well-being (p <0.001). Relationships were similar using the HADS-A (Table 4).
The agreement between VAS and HADS subscale is indicated for each characteristic. Abbreviations: Mean ± SD: Mean ± Standard deviation. Statistically significant results are displayed in bold.
VAS mood.
According to the cut-off for VAS mood, age, sex, BMI and seniority in the company were not correlated with anxiety symptoms. A sad mood was associated with a lower VAS quality of sleep (p <0.001) but not with duration of sleep (p = 0.2). Also, sad mood was associated with higher stress at work (p <0.001) and at home (p = 0.04) but not with longer time spent at work (p = 0.18). Conversely, good mood was linked to better VAS well-being (p <0.001). Relationships were similar using the HADS-D (Table 4).
Sensitivity analysis.
A sensitivity analysis was performed on the group of 144 patients for which complete data were available regarding the primary outcome and defining variables (sex, age, BMI, well-being VAS, Sleep quality VAS, Sleep duration, Stress at work VAS, stress at home VAS, seniority in company, weekly workload), presented in S1 Table. The distributions of HADS-A and HADS-D in this subset was similar to those observed in the entire cohort. The mean score of HADS-A was 6.9 ± 3.9. 83 (57.6%) participants had no anxiety symptoms (score ≤7) and 61 (42.4%) had anxiety symptoms (score ≥8) The mean score of HADS-D was 3.8 ± 2.96. 126 (87.5%) participants had no depressive symptoms (score ≤7) and 18 (12.5%) had depressive symptoms (score ≥8). Regarding VAS anxiety, the same optimal cutoff of 63.5 was drawn from this sample and yielded similar classification performance. The analysis led of VAS mood led to the same optimal cutoff of 42 and yielded a similar classification performance.
The analysis of test-retest reproducibility was performed on a subsample of 91 participants with available data for both test and retest and yielded similar results compared to those observed in the entire cohort. Lin concordance coefficient for VAS anxiety was 0.79 (95CI: 0.71 to 0.87) with difference on the retest of -1.37 ± 19.5 (95CI: -39.6 to 36.8). Lin concordance coefficient for VAS mood was 0.75 (95CI: 0.65, 0.84) with difference on the retest of -0.42 ± 18.7 (95CI: -36.9 to 36.1). The Bland-Altman analysis is shown in S1 Fig.
Evidence of external validity were similar in this subset compared to the entire cohort and concordant between the VAS and the HADS assessment for both anxiety and depression. The only discrepancy was observed for VAS stress at work, that was not significantly different between the groups without vs. with suspected or proven depressive disorder (HADS-D ≥ 8) (56.3 ± 22.7 vs. 61.8 ± 25.1, respectively, p = 0.33, Cohen’s d = 0.55), while the groups at-risk of depressive disorder according to the VAS Mood exhibited a significantly higher weekly workload compared to the group not at risk (62.5 ± 24.0 vs. 53.5 ± 21.8, respectively, p = 0.01, Cohen’s d = 0.33). Those results are presented in S2 Table.
Discussion
This study brings validity evidence of visual analog scales of anxiety and mood for clinical use in occupational health, i.e., the assessment of its acceptability, reproducibility, internal and external validity.
Prevalence and health outcomes
Anxiety and mood disorders are common illnesses in active or non-active populations [1, 42–50]. In our study, 41.8% and 36.8% of workers reported anxiety (VAS anxiety ≥ 60/100) and depressive (VAS mood ≤ 60/100) symptoms. The literature confirms that rates are variable, ranging from 12 to 73% depending on the profession, the country of study, and the questionnaire used [46–48, 50–53]. Those disorders increase the risk of numerous diseases [54], such as metabolic and cardiovascular diseases [55, 56]. Anxiety and depression are also major risk factors for suicide [57, 58]. Psychological disorders are not correctly detected or diagnosed [59–61], and this can significantly impact patients’ professional life such as discrimination in hiring, stigmatization in the workplace, difficulty in maintaining employment, and absenteeism [2–4, 49, 62, 63]. More than half of employers would never accept someone diagnosed with depression for a managerial position, and more than one third of workers would be anxious about such colleagues [64, 65]. Conversely, occupational factors can directly promote anxious-depressive symptomatology such as stress [47, 53, 66–68], damaging social relationships [67] and management [63], imbalanced job design [69], occupational uncertainty, or lack of value and respect in the workplace [66]. The occupational physicians hold an important role in promoting mental health through individual and collective prevention [70], early identification of mental pathologies, and maintenance during return to work [71, 72].
Acceptability
Ceiling or floor effects happen when 15% of participants have the highest or lowest possible score, indicating a lack of discrepancy for extreme scores [73]. No real ceiling of floor effects were observed; with less than 15% of participants having the lowest or highest possible score, as recommended in the literature [73]. The absence of major floor and ceiling effect demonstrate the absence of over-representation of extreme levels of mood and anxiety, and therefore the pertinence of VAS to discriminate symptoms of anxiety or depression [39, 74]. We retrieve one study mentioning ceiling effects in the group of healthy subjects regarding the use of electronic models of VAS pain, anxiety, fatigue and quality of life [75]. However, most of the available studies did not systematically stipulate floor or ceiling effects [23–26, 76–80]. Only one study evaluated the floor and ceiling effects for HADS and found a floor effect for each item but no ceiling effect. However, this study did not concern active subjects but the general population aged 65 to 80 [81]. Despite the lack of data on the time required to complete the VAS tools and the HADS, we assume that there must be a benefit in completion times, as it takes a few seconds to answer a VAS, while HADS takes up to 5–10 minutes [82–84].
Internal validity
In our study, the agreement and reproducibility of VAS anxiety and mood measured by Lin concordance coefficients were satisfying and were over 0.70 for the two VAS. The best sensitivity/specificity compromises were for a threshold of 60/100 for each VAS anxiety and VAS mood: at-risk workers being those for VAS anxiety ≥ 60/100 and those for VAS mood ≤ 60/100. The reliability, the validity and the sensibility of VAS anxiety and VAS mood were also confirmed in the literature versus other questionnaires than HADS, and within specific populations (preanesthesia, dental care for anxiety [76–78, 85], after a stroke or in geriatrics for depression and mood [86–90]. Moreover, VAS anxiety was moderately correlated with VAS mood (coefficient = 0.51; p <0.001), aligned with the literature reporting similar correlations between the HADS-A and the HADS-D (varying from 0.40 to 0.74, mean 0.56) [91–93]. The visual graphical analysis of Bland and Altman for both VAS showed no systematic error but a relatively large random error with a heterogeneous general dispersion; the 95% limits were adequate for high and low values of mean VAS anxiety and mean VAS mood test and retest. We have found similar data in the literature for VAS pain, fatigue, quality of life and anxiety, in a population with multiple sclerosis [75]. The test/retest differences were acceptable [94, 95].
External validity
This study demonstrated a very good external validity of VAS anxiety/VAS mood: the same relationships were shown between those VAS and other variables, and between HADS-A / HADS-D and those same variables. Women had more anxiety symptoms (both for VAS anxiety and HADS-A) without differences for depressive symptoms (both for VAS mood and HADS-D). The literature also frequently report higher levels of anxiety in women, with more heterogeneous data for mood [32, 48–51, 92]. Whatever the scales considered (VAS anxiety vs. HADS-A, or VAS mood vs. HADS-D), age did not influence depressive and anxiety symptoms. The influence of age on those symptoms is conflicting in the literature, depending on the population studied [28, 48, 53, 92]. Similarly, whatever the scales considered, anxiety and depressive symptoms were associated with a poor sleep quality and high levels of stress at work or at home. In line with the literature, anxiety and mood perturbations may be associated with a poor sleep [48] and work- or home-related stress [46, 96]. In our study, other factors (such as working hours and seniority in the company) were not associated with anxiety and mood. The literature did not have consensus data on the protective [50] or vulnerability [48] effects associated with seniority in the company. However, a positive correlation was found between working hours, anxiety and depression [28]. However, despite no significance, relationships were similar between VAS and HADS, and those factors.
Limitations
Compared to other French studies using a questionnaire, the response rate may seem low [29, 97–101]. However, the number of included workers made it possible to carry out the statistical analyses with the number of required subjects, determined a priori [38, 39]. Also, we had some missing data despite the volume of the survey [102]. Moreover, the proportion of participants who answered both test and retest was higher in our study than in other studies [83], demonstrating the interest of workers for questionnaires on quality of life at work. A limitation arose from participants’ characteristics with more women than men, and a higher representation of executives (57%) compared to mid-level professionals (23%). This class imbalance in terms of socio-economic levels calls for larger studies designed to assess the validity evidence of these VAS in other socio-economic classes. Considering internal validity, our study did not assess sensitivity to changes in anxiety and mood. However, test-retest reliability was acceptable. Moreover, literature previously showed a good sensitivity to changes using VAS anxiety before and after stress [76, 95]. The relatively large random error retrieved at the Bland and Altman test may traduce the non-negligible probability of a stressful event having occurred between the test and retest. Also, we suggest that the same evaluation may be repeated for people close to the at-risk threshold, which is relatively easy considering the instantaneous time required to answer. Our study may suffer from limitations in terms of external validity because we used only the HADS whereas other validated assessment tools exist to assess anxiety and depressive symptoms. However, HADS is classically considered as the gold standard with good external structure and satisfying discriminant validity [103]. Moreover, the aim of our study was not to compare the numerous scales but rather to validate the VAS anxiety and mood vs. the reference test, in a general population. We also used many sociodemographic, occupational, and clinical data for external validity. Despite a sufficient sample size for the validation of VAS anxiety and mood, insufficient data precluded further analyses on at-risk occupations for anxiety and depressive symptoms. However, all professions can be affected by the common mental disorders of anxiety and depression due to major work changes. Indeed, some studies showed that intellectual professions or higher degrees have high levels of anxiety and depression [48, 50], while others reported a high risk for lowest occupational categories (workers and technical classes) [92, 104]. Identifying at-risk workers is necessary for effective preventive strategies. Assessment of anxiety by a VAS was only described in pre-anesthesia patients [77, 78] or dental care, limiting its generalizability and its validity, as it was not compared with the HADS. Finally, considering semantics, VAS mood presented in the literature had a specific design. They used a combination of words and schematic faces indicating different mood states. Those VAS were designed for certain profiles of patients with neurologic disorders or cognitive impairment [79, 80, 86, 88]. Therefore, we were the first to introduce a unique VAS anxiety and mood model in general population.
Conclusion
Our results show that VAS anxiety and VAS mood are reliable tools for identifying at-risk workers for anxiety and depression, allowing for a quicker screening and primary prevention in the workplace. We have determined a cut-off value of 60 for each of the VAS: ≥ 60/100 for workers at risk of anxiety (Sensitivity 72%, Specificity 86%, ROC AUC 0.79) and ≤ 60/100 for depression (Sensitivity 84%, Specificity 73%, ROC AUC 0.78). The Anxiety and Mood VAS yielded satisfying test-retest reliability and their association with participants socio-demographic and work-related characteristics are similar with those observed for the HADS.
Supporting information
S1 Table. Characteristics of participants in the sensitivity analysis, defined by the subgroup in which the primary outcome and the variables used for examining external validity evidence were complete.
Those variables are sex, age, BMI, well-being VAS, Sleep quality VAS, Sleep duration, Stress at work VAS, stress at home VAS, seniority in company, weekly workload).
https://doi.org/10.1371/journal.pone.0316159.s001
(DOCX)
S2 Table. Agreement between visual analog scales vs. HADS on differences in participants’ characteristics in the sensitivity analysis cohort.
Abbreviations: Mean ± SD: Mean ± Standard deviation. Statistically significant results are displayed in bold.
https://doi.org/10.1371/journal.pone.0316159.s002
(DOCX)
S1 Fig. Agreement between the test and retest of both visual analog scales according to Bland-Altman analysis, in the sensitivity analysis cohort.
A. Bland-Altman plot for VAS Anxiety, B. Bland-Altman plot for VAS Mood.
https://doi.org/10.1371/journal.pone.0316159.s003
(DOCX)
Acknowledgments
We express our sincere gratitude to Marie-Anne Cousseau who established the public-private partnership between WittyFit and the University Hospital of Clermont-Ferrand, France.
References
- 1. Steel Z, Marnane C, Iranpour C. The global prevalence of common mental disorders: a systematic review and meta-analysis 1980–2013. Int J Epidemiol. 2014;43: 476–493. pmid:24648481
- 2. Arends I, Klink JJ, Rhenen W, Boer MR, Bultmann U. Predictors of recurrent sickness absence among workers having returned to work after sickness absence due to common mental disorders. Scand J Work Env Health. 2014;40: 195–202. pmid:24048675
- 3. Koopmans PC, Bultmann U, Roelen CA, Hoedeman R, Klink JJ, Groothoff JW. Recurrence of sickness absence due to common mental disorders. Int Arch Occup Env Health. 2011;84: 193–201. pmid:20449605
- 4. Koopmans PC, Roelen CA, Groothoff JW. Risk of future sickness absence in frequent and long-term absentees. Occup Med Lond. 2008;58: 268–274. pmid:18390880
- 5. Kessler RC, Petukhova M, Sampson NA, Zaslavsky AM, Wittchen HU. Twelve-month and lifetime prevalence and lifetime morbid risk of anxiety and mood disorders in the United States. Int J Methods Psychiatr Res. 2012;21: 169–184. pmid:22865617
- 6. Kessler RC, Chiu WT, Demler O, Merikangas KR, Walters EE. Prevalence, severity, and comorbidity of 12-month DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2005;62: 617–627. pmid:15939839
- 7. Hendriks SM, Spijker J, Licht CMM, Hardeveld F, De Graaf R, Batelaan NM, et al. Long-term work disability and absenteeism in anxiety and depressive disorders. J Affect Disord. 2015;178: 121–130. pmid:25805404
- 8. Scaife R. Identifying the impact of psychological ill-health on safety performance. 2006.
- 9.
Amati C, Scaife R. The Impact of Psychological Ill-Health on Safety. In: Bust PD, editor. 1st ed. Taylor & Francis; 2020. pp. 551–554. https://doi.org/10.1201/9781003072072-130
- 10. McCarthy JM, Trougakos JP, Cheng BH. Are anxious workers less productive workers? It depends on the quality of social exchange. J Appl Psychol. 2016;101: 279–291. pmid:26375962
- 11.
Bubonya M, Cobb-Clark DA, Wooden M. Mental health and productivity at work: does what you do matter? Melbourne: Melbourne Institute of Applied Economic and Social Research; 2016.
- 12. De Oliveira C, Saka M, Bone L, Jacobs R. The Role of Mental Health on Workplace Productivity: A Critical Review of the Literature. Appl Health Econ Health Policy. 2023;21: 167–193. pmid:36376610
- 13. Goetzel RZ, Long SR, Ozminkowski RJ, Hawkins K, Wang S, Lynch W. Health, Absence, Disability, and Presenteeism Cost Estimates of Certain Physical and Mental Health Conditions Affecting U.S. Employers: J Occup Environ Med. 2004;46: 398–412. pmid:15076658
- 14. Stewart WF. Cost of Lost Productive Work Time Among US Workers With Depression. JAMA. 2003;289: 3135. pmid:12813119
- 15. Wagner S, Koehn C, White M, Harder H, Schultz I, Williams-Whitt K, et al. Mental Health Interventions in the Workplace and Work Outcomes: A Best-Evidence Synthesis of SystematicReviews. Int J Occup Environ Med. 2016;7: 1–14. pmid:26772593
- 16. Plaisier I, De Graaf R, De Bruijn J, Smit J, Van Dyck R, Beekman A, et al. Depressive and anxiety disorders on-the-job: The importance of job characteristics for good work functioning in persons with depressive and anxiety disorders. Psychiatry Res. 2012;200: 382–388. pmid:22862911
- 17. Kelloway EK, Dimoff JK, Gilbert S. Mental Health in the Workplace. Annu Rev Organ Psychol Organ Behav. 2023;10: 363–87.
- 18. Kessler RC, Frank RG. The impact of psychiatric disorders on work loss days. Psychol Med. 1997;27: 861–873. pmid:9234464
- 19. Crocq MA. A history of anxiety: from Hippocrates to DSM. Dialogues Clin Neurosci. 2015;17: 319–325. pmid:26487812
- 20. Kupfer DJ. Anxiety and DSM-5. Dialogues Clin Neurosci. 2015;17: 245–246. pmid:26487805
- 21. Kjaergaard M, Arfwedson Wang CE, Waterloo K, Jorde R. A study of the psychometric properties of the Beck Depression Inventory-II, the Montgomery and Asberg Depression Rating Scale, and the Hospital Anxiety and Depression Scale in a sample from a healthy population. Scand J Psychol. 2014;55: 83–89.
- 22. Dutheil F, Pereira B, Moustafa F, Naughton G, Lesage F-X, Lambert C. At-risk and intervention thresholds of occupational stress using a visual analogue scale. Kou YR, editor. PLOS ONE. 2017;12: e0178948. pmid:28586383
- 23. Boonstra AM, Schiphorst Preuper HR, Reneman MF, Posthumus JB, Stewart RE. Reliability and validity of the visual analogue scale for disability in patients with chronic musculoskeletal pain. Int J Rehabil Res. 2008;31: 165–169. pmid:18467932
- 24. Lesage FX, Berjot S. Validity of occupational stress assessment using a visual analogue scale. Occup Med Lond. 2011;61: 434–436. pmid:21505089
- 25. Lesage FX, Berjot S, Deschamps F. Clinical stress assessment using a visual analogue scale. Occup Med Lond. 2012;62: 600–605. pmid:22965867
- 26. Ratinaud MC, Chamoux A, Glace B, Coudeyre E. Job satisfaction evaluation in low back pain: a literature review and tools appraisal. Ann Phys Rehabil Med. 2013;56: 465–481. pmid:23928031
- 27. Gramstad TO, Gjestad R, Haver B. Personality traits predict job stress, depression and anxiety among junior physicians. BMC Med Educ. 2013;13.
- 28. Afonso P, Fonseca M, Pires JF. Impact of working hours on sleep and mental health. Occup Med Lond. 2017;67: 377–382. pmid:28575463
- 29. Dutheil F, Duclos M, Naughton G. WittyFit-Live Your Work Differently: Study Protocol for a Workplace-Delivered Health Promotion. JMIR Res Protoc. 2017;6. pmid:28408363
- 30. Snaith RP, Baugh SJ, Clayden AD, Husain A, Sipple MA. The Clinical Anxiety Scale: an instrument derived from the Hamilton Anxiety Scale. Br J Psychiatry. 1982;141: 518–523. pmid:7150890
- 31. Snaith RP, Zigmond AS. The hospital anxiety and depression scale. Br Med J Clin Res Ed. 1986;292.
- 32. Zigmond AS, Snaith RP. The hospital anxiety and depression scale. Acta Psychiatr Scand. 1983;67: 361–370.
- 33. Lepine JP, Godchau M, Brun P. Anxiety and depression in inpatients. Lancet. 1985;2: 1425–1426. pmid:2867417
- 34. Lepine JP, Godchau M, Brun P, Lemperiere T. Evaluation of anxiety and depression among patients hospitalized on an internal medicine service. Ann Med Psychol Paris. 1985;143: 175–189.
- 35. Razavi D, Delvaux N, Farvacques C, Robaye E. Validation de la version française du HADS dans une population de patients cancéreux hospitalisés. [Validation of the French version of the Hospital Anxiety and Depression Scale (HADS) in a population of hospitalized cancer patients. Rev Psychol Appliquée. 1989;39: 295–307.
- 36. Olsson I, Mykletun A, Dahl AA. The Hospital Anxiety and Depression Rating Scale: a cross-sectional study of psychometrics and case finding abilities in general practice. BMC Psychiatry. 2005;5.
- 37. Dutheil F, Trousselard M, Perrier C. Urinary interleukin-8 is a biomarker of stress in emergency physicians, especially with advancing age–the JOBSTRESS* randomized trial. PLoS One. 2013;8. pmid:23977105
- 38. Mokkink LB, Terwee CB, Patrick DL. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res. 2010;19: 539–549. pmid:20169472
- 39. Terwee CB, Bot SD, Boer MR. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007;60: 34–42. pmid:17161752
- 40. Obuchowski NA, Bullen JA. Receiver operating characteristic (ROC) curves: review of methods with applications in diagnostic medicine. Phys Med Biol. 2018;63. pmid:29512515
- 41. Carter JV, Pan J, Rai SN, Galandiuk S. ROC-ing along: Evaluation and interpretation of receiver operating characteristic curves. Surgery. 2016;159: 1638–1645. pmid:26962006
- 42. Kessler RC, Berglund P, Demler O. The epidemiology of major depressive disorder: results from the National Comorbidity Survey Replication (NCS-R. JAMA. 2003;289: 3095–3105. pmid:12813115
- 43. Kessler RC, Berglund P, Demler O, Jin R, Merikangas KR, Walters EE. Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2005;62: 593–602. pmid:15939837
- 44. Smith K. Mental health: a world of depression. Nature. 2014;515.
- 45. Creedy DK, Sidebotham M, Gamble J, Pallant J, Fenwick J. Prevalence of burnout, depression, anxiety and stress in Australian midwives: a cross-sectional survey. BMC Pregnancy Childbirth. 2017;17.
- 46. Fawzy M, Hamed SA. Prevalence of psychological stress, depression and anxiety among medical students in Egypt. Psychiatry Res. 2017;255: 186–194. pmid:28575777
- 47. Desouky D, Allam H. Occupational stress, anxiety and depression among Egyptian teachers. J Epidemiol Glob Health. 2017;7: 191–198. pmid:28756829
- 48. Zhang ZH, Li RH, Li DF. Anxiety and depression status of coal miners and related influencing factors. Zhonghua Lao Dong Wei Sheng Zhi Ye Bing Za Zhi. 2018;36: 860–863. pmid:30646656
- 49. Torske MO, Hilt B, Glasscock D, Lundqvist P, Krokstad S. Anxiety and Depression Symptoms Among Farmers: The HUNT Study, Norway. J Agromedicine. 2016;21: 24–33. pmid:26488439
- 50. Fu A, Liu B, Jiang Y, Zhao J, Zhang G, Liu J. A Mental Health Survey of Different Ethnic and Occupational Groups in Xinjiang, China. Int J Env Res Public Health. 2017;14. pmid:28067780
- 51. Utzet M, Moncada S, Molinero E, Navarro A. Psychosocial exposures and mental health: distribution by gender and occupational class in a population-based study. J Public Health Oxf. 2016;38. pmid:28158824
- 52. Theis KA, Roblin DW, Helmick CG, Luo R. Prevalence and causes of work disability among working-age U.S. adults, 2011–2013, NHIS. Disabil Health J. 2018;11: 108–115. pmid:28476583
- 53. Tran TTT, Nguyen NB, Luong MA. Stress, anxiety and depression in clinical nurses in Vietnam: a cross-sectional survey and cluster analysis. Int J Ment Health Syst. 2019;13. pmid:30622629
- 54. Lépine JP, Briley M. The increasing burden of depression. Neuropsychiatr Treat. 2011;7: 3–7. pmid:21750622
- 55. Belialov FI. Depression, anxiety, and stress in patients with coronary heart disease. Ter Arkh. 2017;89: 104–109.
- 56. Chauvet-Gelinier JC, Bonin B. Stress, anxiety and depression in heart disease patients: A major challenge for cardiac rehabilitation. Ann Phys Rehabil Med. 2017;60: 6–12. pmid:27771272
- 57. Bentley KH, Franklin JC, Ribeiro JD, Kleiman EM, Fox KR, Nock MK. Anxiety and its disorders as risk factors for suicidal thoughts and behaviors: A meta-analytic review. Clin Psychol Rev. 2016;43: 30–46. pmid:26688478
- 58. Bolton JM, Gunnell D, Turecki G. Suicide risk assessment and intervention in people with mental illness. BMJ. 2015;351. pmid:26552947
- 59. Wang PS, Aguilar-Gaxiola S, Alonso J. Use of mental health services for anxiety, mood, and substance disorders in 17 countries in the WHO world mental health surveys. Lancet. 2007;370: 841–850. pmid:17826169
- 60. Wang PS, Angermeyer M, Borges G. Delay and failure in treatment seeking after first onset of mental disorders in the World Health Organization’s World Mental Health Survey Initiative. World Psychiatry. 2007;6: 177–185. pmid:18188443
- 61. Wang PS, Berglund P, Olfson M, Pincus HA, Wells KB, Kessler RC. Failure and delay in initial treatment contact after first onset of mental disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2005;62: 603–613. pmid:15939838
- 62. Norder G, Hoedeman R, Bruin J, Rhenen W, Roelen CA. Time to recurrence of mental health-related absence from work. Occup Med Lond. 2015;65: 574–577.
- 63. Junne F, Michaelis M, Rothermund E. The Role of Work-Related Factors in the Development of Psychological Distress and Associated Mental Disorders: Differential Views of Human Resource Managers. Occup Physicians Prim Care Physicians Psychother Ger Int J Env Res Public Health. 2018;15.
- 64. Scheid TL. Employment of individuals with mental disabilities: business response to the ADA’s challenge. Behav Sci Law. 1999;17: 73–91. pmid:10216927
- 65. Glozier N. Workplace effects of the stigmatization of depression. J Occup Env Med. 1998;40: 793–800. pmid:9777563
- 66. Harvey SB, Modini M, Joyce S. Can work make you mentally ill? A systematic meta-review of work-related risk factors for common mental health problems. Occup Env Med. 2017;74: 301–310. pmid:28108676
- 67. Rusli BN, Edimansyah BA, Naing L. Working conditions, self-perceived stress, anxiety, depression and quality of life: a structural equation modelling approach. BMC Public Health. 2008;8. pmid:18254966
- 68. Song KW, Choi WS, Jee HJ. Correlation of occupational stress with depression, anxiety, and sleep in Korean dentists: cross-sectional study. BMC Psychiatry. 2017;17.
- 69. Magnavita N, Fileni A. Association of work-related stress with depression and anxiety in radiologists. Radiol Med. 2014;119: 359–366. pmid:24297590
- 70. Cleary M, Dean S, Webster S, Walter G, Escott P, Lopez V. Primary health care in the mental health workplace: insights from the Australian experience. Issues Ment Health Nurs. 2014;35: 437–443. pmid:24857527
- 71. Arends I, Klink JJ, Rhenen W, Boer MR, Bultmann U. Prevention of recurrent sickness absence in workers with common mental disorders: results of a cluster-randomised controlled trial. Occup Env Med. 2014;71: 21–29. pmid:24158311
- 72. Nigatu YT, Liu Y, Uppal M. Interventions for enhancing return to work in individuals with a common mental illness: systematic review and meta-analysis of randomized controlled trials. Psychol Med. 2016;46: 3263–3274. pmid:27609709
- 73. Ravoux H, Pereira B, Brousse G. Work Addiction Test Questionnaire to Assess Workaholism: Validation of French Version. JMIR Ment Health. 2018;5. pmid:29439945
- 74. Wamper KE, Sierevelt IN, Poolman RW, Bhandari M, Haverkamp D. The Harris hip score: Do ceiling effects limit its usefulness in orthopedics? Acta Orthop. 2010;81: 703–707. pmid:21110703
- 75. Kos D, Raeymaekers J, Remoortel A. Electronic visual analogue scales for pain, fatigue, anxiety and quality of life in people with multiple sclerosis using smartphone and tablet: a reliability and feasibility study. Clin Rehabil. 2017;31: 1215–1225. pmid:28786335
- 76. Abend R, Dan O, Maoz K, Raz S, Bar-Haim Y. Reliability, validity and sensitivity of a computerized visual analog scale measuring state anxiety. J Behav Ther Exp Psychiatry. 2014;45: 447–453. pmid:24978117
- 77. Berghmans JM, Poley MJ, Ende J. A Visual Analog Scale to assess anxiety in children during anesthesia induction (VAS-I): Results supporting its validity in a sample of day care surgery patients. Paediatr Anaesth. 2017;27: 955–961. pmid:28707384
- 78. Facco E, Stellini E, Bacci C. Validation of visual analogue scale for anxiety (VAS-A) in preanesthesia evaluation. Minerva Anestesiol. 2013;79: 1389–1395. pmid:23860442
- 79. Barrows PD, Thomas SA. Assessment of mood in aphasia following stroke: validation of the Dynamic Visual Analogue Mood Scales (D-VAMS. Clin Rehabil. 2018;32: 94–102. pmid:28653547
- 80. Haley KL, Womack JL, Harmon TG, Williams SW. Visual analog rating of mood by people with aphasia. Top Stroke Rehabil. 2015;22: 239–245. pmid:26258449
- 81. Djukanovic I, Carlsson J, Arestedt K. Is the Hospital Anxiety and Depression Scale (HADS) a valid measure in a general population 65–80 years old? A psychometric evaluation study. Health Qual Life Outcomes. 2017;15.
- 82. Fernández-de-las-Peñas C, Rodríguez-Jiménez J, Palacios-Ceña M, de-la-Llave-Rincón AI, Fuensalida-Novo S, Florencio LL, et al. Psychometric Properties of the Hospital Anxiety and Depression Scale (HADS) in Previously Hospitalized COVID-19 Patients. Int J Environ Res Public Health. 2022;19: 9273. pmid:35954630
- 83. Reda AA. Reliability and validity of the Ethiopian version of the hospital anxiety and depression scale (HADS) in HIV infected patients. PLoS One. 2011;6. pmid:21283565
- 84. Guo C, Huang X. Hospital anxiety and depression scale exhibits good consistency but shorter assessment time than Zung self-rating anxiety/depression scale for evaluating anxiety/depression in non-small cell lung cancer. Medicine (Baltimore). 2021;100: e24428. pmid:33663054
- 85. Appukuttan D, Vinayagavel M, Tadepalli A. Utility and validity of a single-item visual analog scale for measuring dental anxiety in clinical practice. J Oral Sci. 2014;56: 151–156. pmid:24930752
- 86. House ED, Arruda JE, Andrasik F, Grazzi L. The reliability and validity of the Visual Analog Mood Scales in non-English-speaking pain patients. Pain Pr. 2012;12: 626–632. pmid:22443497
- 87. Arruda JE, Stern RA, Somerville JA. Measurement of mood states in stroke patients: validation of the visual analog mood scales. Arch Phys Med Rehabil. 1999;80: 676–680. pmid:10378494
- 88. Temple RO, Stern RA, Latham J, Ruffolo JS, Arruda JE, Tremont G. Assessment of mood state in dementia by use of the visual analog mood scales (VAMS. Am J Geriatr Psychiatry. 2004;12: 527–530. pmid:15353392
- 89. Kanamori H, Nagai K, Matsubara T. Comparison of the psychosocial quality of life in hemodialysis patients between the elderly and non-elderly using a visual analogue scale: the importance of appetite and depressive mood. Geriatr Gerontol Int. 2012;12: 65–71. pmid:21843200
- 90. Killgore WD. The visual analogue mood scale: can a single-item scale accurately classify depressive mood state? Psychol Rep. 1999;85: 1238–1243. pmid:10710979
- 91. Bjelland I, Dahl AA, Haug TT, Neckelmann D. The validity of the Hospital Anxiety and Depression Scale. An updated literature review. J Psychosom Res. 2002;52: 69–77. pmid:11832252
- 92. Bocerean C, Dupret E. A validation study of the Hospital Anxiety and Depression Scale (HADS) in a large sample of French employees. BMC Psychiatry. 2014;14.
- 93. Schonberger M, Ponsford J. The factor structure of the Hospital Anxiety and Depression Scale in individuals with traumatic brain injury. Psychiatry Res. 2010;179: 342–349. pmid:20483471
- 94. HC V, CB T, DL K, LM B. When to use agreement versus reliability measures. J Clin Epidemiol. 2006;59: 1033–1039. pmid:16980142
- 95. Guyatt G, Walter S, Norman G. Measuring change over time: assessing the usefulness of evaluative instruments. J Chronic Dis. 1987;40: 171–178. pmid:3818871
- 96. Fan LB, Blumenthal JA, Watkins LL, Sherwood A. Work and home stress: associations with anxiety and depression symptoms. Occup Med Lond. 2015;65: 110–116. pmid:25589707
- 97. Tekath M, Dutheil F, Bellini R. Comparison of the ultra-low-dose Veo algorithm with the gold standard filtered back projection for detecting pulmonary asbestos-related conditions: a clinical observational study. BMJ Open. 2014;4. pmid:24879827
- 98. Dutheil F, Delaire P, Boudet G. Cost/effectiveness comparison of the vaccine campaign and reduction of sick leave, after vaccination against influenza among the Clermont-Ferrand University Hospital staff. Med Mal Infect. 2008;38: 567–573.
- 99. Dutheil F, Kelly C, Biat I. Relation between the level of knowledge and the rate of vaccination against the flu virus among the staff of the Clermont-Ferrand University hospital. Med Mal Infect. 2008;38: 586–594.
- 100. Lanhers C, Pereira B, Garde G, Maublant C, Dutheil F, Coudeyre E. Evaluation of “I-Preventive”: a digital preventive tool for musculoskeletal disorders in computer workers-a pilot cluster randomised trial. BMJ Open. 2016;6. pmid:27660316
- 101. Lopez V, Chamoux A, Tempier M. The long-term effects of occupational exposure to vinyl chloride monomer on microcirculation: a cross-sectional study 15 years after retirement. BMJ Open. 2013;3. pmid:23794583
- 102. Edwards P, Roberts I, Sandercock P, Frost C. Follow-up by mail in clinical trials: does questionnaire length matter? Control Clin Trials. 2004;25: 31–52. pmid:14980747
- 103. Herrmann C. International experiences with the Hospital Anxiety and Depression Scale–a review of validation data and clinical results. J Psychosom Res. 1997;42: 17–41. pmid:9055211
- 104. Cohidon C, Imbernon E, Gorldberg M. Prevalence of common mental disorders and their work consequences in France, according to occupational category. Am J Ind Med. 2009;52: 141–152. pmid:19016268