The assessment of anxiety and depression through the Depression Anxiety Stress Scales

A valuable self-report questionnaire to assess internalizing symptomatology is the Depression Anxiety Stress Scales (DASS; [22]), a 42-item scale empirically developed to provide maximum differentiation between anxiety and depression in adults by reducing the importance of general symptoms and focusing on the distinctive symptoms of each construct. Although the DASS was originally designed to assess the core symptomatology of depression and anxiety, subsequent factor analysis studies pointed out a third factor including items that reflected nondiscriminating symptoms of the two syndromes and, in particular, some form of general tension; thus, the final version of the DASS resulted composed of three scales: (a) Depression, evaluating depressive symptoms such as hopelessness, dysphoria, low self-esteem, and anhedonia; (b) Anxiety, referring to subjective and physical symptoms of anxiety (e.g., breathing difficulties), and acute response to fear; (c) Stress, measuring tension, agitation, persistent arousal, and irritability [22]. Afterward, to facilitate the use of the DASS in time-constrained settings, a shorter 21-item version (i.e., DASS-21) was developed [23,24]. The DASS-21 has shown favorable psychometric and clinimetric properties; due to its short format, ease of administration, and availability in the public domain, it has become one of the most widely employed self-report instruments in research and clinical practice [25,26]. Notably, recent research has demonstrated its robust reliability and validity in specific populations, including primary and middle school teachers [27], undergraduate nursing students [28], frontline doctors [29], and caregivers of young children [30], thus substantiating its usability across various contexts. With specific regard to the factor structure of the DASS-21 in community adult samples, a bifactor model with a general factor (i.e., NA/general distress) and three domain-specific factors (i.e., Depression, Anxiety, Stress) is currently deemed the best solution [25,31], also in the Italian population [32]; in particular, such a solution has been considered consistent with the tripartite model [12] which hypothesizes the presence of a general factor comprising common symptoms of anxiety and depression as a possible reason for the high co-occurrence between these conditions [21].

The current study

The DASS-21 enables a comprehensive assessment of the symptoms of different negative affective states (i.e., anxiety, depression, and stress), providing an overview of internalizing experiences; therefore, it may represent a helpful tool to clarify whether adolescents are able or not to differentiate between different negative emotional states. Nevertheless, as previously mentioned, the literature on the factor structure of the DASS-21 in young people is still ambiguous. For instance, it seems unclear whether the use of separate scale scores assessing anxiety, depression, and stress is justified in adolescence or it would be more appropriate to calculate a total score of general distress. Furthermore, to date, no study has investigated whether the structure of the adult DASS-21 is generalizable to the Italian adolescent population; however, without evidence regarding the psychometric properties of the tool, the application of the adult scoring to youth can lead to erroneous conclusions. Based on these premises, the current study aimed to investigate the factor structure and psychometric properties of the DASS-21 in a group of adolescents (age range:14–18 years) from the general Italian population. By pursuing this objective, it is also possible to gain deeper insight into the structure of adolescents’ emotional experience as regards their ability to discriminate between anxiety, depression, and stress.

In the first place, following the recent evidence by Jovanović et al. [41], we might have expected the bifactor-ESEM model to best represent the factor structure of the DASS-21 among Italian adolescents. However, as outlined earlier, the literature is rather controversial, posing challenges in crafting a precise hypothesis. Thus, we systematically tested all models considered in previous studies (see Statistical analyses section for details); this approach was used not only to identify the optimal factorial solution, but also to ensure the comprehensiveness and comparability of our study within the existing literature. Second, we investigated Measurement Invariance (MI) across sex and age groups (14–16 years vs. 17–18 years); previous research generally found that the DASS-21 factor structure was invariant between boys and girls [18,37,38,41], while MI across age has been explored and supported only by a few studies [17,41]. Importantly, the decision to segment the age groups into 14–16 years and 17–18 years was mainly influenced by the seminal work of Jovanović et al. [41], who employed the same categorization. By adopting this approach, we aimed to facilitate a direct comparison between Jovanović et al.’s findings and our results, thereby excluding the possibility that divergent outcomes could be attributed to variations in age ranges. Then, complex latent mean differences across sex and age were addressed: Jovanović et al. [41] reported that girls scored higher than boys on anxiety and depression, while no age difference was shown; consequently, we hypothesized that similar results would emerge in the present study. Finally, we sought to examine convergent and divergent validity of the DASS-21: we investigated its associations with PA and NA and surmised that the specific factors (i.e., Depression, Anxiety, and Stress) were significant negative predictors of PA and positive predictors of NA. However, the only available study in this regard [41] found that only the Depression factor was a significant predictor of both PA and NA; therefore, this hypothesis was mainly exploratory.

Participants

The sample included 655 White adolescents (394 girls, 60.6%) aged 14 to 18 (M = 16.3, SD = 1.29). 3.6% of participants attended the Italian third class of a lower secondary school (8^th grade), while the remaining percentage were recruited from upper secondary schools. Specifically, among the latter, 1.2% attended the first class (9 ^th grade), 5.6% the second class (10 ^th grade), 34.7% the third class (11 ^th grade), and 54.8% the fourth and fifth classes (12^th grade). Participants were asked if they had ever experienced psychological difficulties for which they sought professional help. Among those who responded (n = 566), 21.7% reported current or past psychological issues, such as anxiety-related difficulties, eating disorders, and family or school problems.

Procedure

The data used in this work derive from three different but related studies, each employing an online survey with slightly different batteries of self-report questionnaires. The resulting independent datasets were merged to obtain the final sample of the present study. Consequently, not all students completed all the same self-report measures, but all completed the DASS-21. Each project was approved by the Ethics Committee for Psychological Research of the University of Padova (protocol number: 4332) and was carried out according to the recommendations of the Declaration of Helsinki.

Data were collected between 4^th October 2021 and 31^st May 2022 in lower and upper secondary schools located in Northern, Central, and Southern Italy. Following the approval of school directors, written informed consent was requested from parents of students younger than 18; for students aged 18, written consent was obtained directly. No incentives or rewards were given for participation.

Participants completed the online survey in a group setting in their schools during class time. Anonymity was ensured by using code numbers created by each student.

Measures

The Depression Anxiety Stress Scales-21 (DASS-21; Italian version by Bottesi et al. [32]) is a 21-item questionnaire measuring depression, anxiety, and stress symptoms, as well as general distress over the previous week on a four-point Likert scale (0 = Did not apply to me at all, 3 = Applied to me very much, or most of the time). The total score of the Italian version presented excellent reliability indices in both clinical (Cronbach’s α = 0.92) and nonclinical (Cronbach’s α = 0.90) adult samples.

The Positive and Negative Affect Schedule–Trait version (PANAS; Italian version by Terracciano et al. [44]) is a self-report measure that assesses two independent dimensions of PA and NA. The PA scale contains ten items referring to positive emotions (e.g., proud, determined, strong, excited, enthusiastic), while the NA scale is composed of ten items reflecting negative emotions (e.g., ashamed, hostile, irritable, upset, distressed). Participants rate the degree to which they usually experience each of the listed emotions on a five-point Likert scale, ranging from 1 = Very slightly to 5 = Very much. The Italian version of the tool showed adequate psychometric properties, with Cronbach’s α = 0.83 for PA and 0.87 for NA.

Statistical analyses

The statistical validation of the scale was conducted through the following steps, with all analyses performed in MPlus, version 7.31 [45].

The first step focused on scale dimensionality, namely the detection of an optimal measurement model for the items. To this end, a CFA approach was adopted. However, in light of previous validation studies, a semi-confirmatory approach was also implemented within the ESEM framework. Following the seminal work of Jovanović et al. [41] and the literature on the DASS-21 factor structure in adolescent samples, the following nine models were tested:

1. Model 1 –one-factor model with all items loading on a single factor [20];
2. Model 2 –two-factor model with Depression scale items set to load on the first factor, and Anxiety and Stress scales items on the second factor [46];
3. Model 3 –two-factor model comprising a PH factor (items 2, 4, 7 and 19) and a Generalized Negativity factor combining low PA and general NA items (remaining 17 items) [19];
4. Model 4 –two correlated factors (Depression and Anxiety) with a second-order NA factor [17,18];
5. Model 5 –three-factor model with three correlated factors of Depression, Anxiety, and Stress [33–35,37–39];
6. Model 6 –three-factor model consistent with a tripartite model of anxiety and depression [12] with a PH factor (items 2, 4, 7 and 19), a low PA factor (items 3, 10, 16 and 21), and a general NA factor (the remaining 13 items);
7. Model 7 –bifactor model with one general factor and three specific factors (Depression, Anxiety, and Stress [21,40]). This model allows each item to load both on the general factor and on a specific factor;
8. Model 8 –ESEM model in which all items are specified to load on all three factors [41]. In line with previous applications of ESEM, an oblique geomin rotation was used relaxing the orthogonality assumption between the latent factors [47];
9. Model 9 –bifactor-ESEM model in which all items were set to load on the general factor and cross-load on specific factors (oblique bi-geomin rotation was used) [41].

Considering the linearly ordered categorical response scale (i.e., four-point Likert scale) of the items, the robust weighted least squares (WLSMV) estimator was used. This is specifically designed for ordinal data and was shown to be robust to violations of normality; moreover, WLSMV was shown to provide less biased and more accurate estimates than robust maximum likelihood when adopted in ordinal data settings [48]. To evaluate model fit, the following indices were used: robust Satorra-Bentler scaled chi-square (SBχ²), Root Mean Square Error of Approximation (RMSEA), Standardized Root Mean Square Residual (SRMR), Comparative Fit Index (CFI), and Tucker–Lewis Index (TLI). Acceptable model fit was defined based on the recommended cut-off values: SBχ² non-statistically significant; SBχ² /df < 3; RMSEA less than 0.06, or between 0.06 and 0.08 for a reasonable fit; CFI and TLI greater than 0.95, or between 0.90 and 0.95 for a reasonable fit; SRMR less than 0.08 [49]. The SBχ² value should be non-significant to indicate a good fit, but this is rarely obtained in large samples as the chi-square value is highly sensitive to sample size.

The second step was aimed at testing the invariance of the selected model and latent mean differences across sex and age groups. In this context, MI is meant as factorial invariance. When supported by evidence, the factor score of the respondent is independent of his or her own group membership and can be used for individual comparison purposes. A multiple-group semi-CFA framework was adopted to assess MI. The theta parametrization, in which residual variances for latent response variables are allowed to be parameters in the model but scale factors are not, was used in line with Jovanović et al. [41]. Different levels of MI exist and can be tested through several increasingly restrictive nested models. Due to the adopted semi-CFA approach, the following invariance steps were performed for both sex and age factors. The procedure started by specifying a configural invariance model in which equal factorial structures were imposed across groups. Following recommendations from the MPlus literature [50,51], uniqueness of each item was fixed to 1 in the first group for identifiability purposes. In the next step, scalar invariance was assessed by specifying a stricter model in which item intercepts were fixed equal across groups. In the Mplus program, metric MI is not allowed for ESEM using estimation for categorical data [45].

The last step aimed to verify convergent and divergent validity of the DASS-21. This was evaluated using the general SEM framework by examining associations with PA and NA as measured by the PANAS.

Descriptive statistics

The mean scores obtained by participants on the DASS-21 Depression, Anxiety, and Stress scales were 8.29 (SD = 5.49), 7.11 (SD = 5.29), and 10.2 (SD = 5.06), respectively. These scores were calculated on the basis of the adult DASS-21 factor structure (32), namely: Depression: items 3, 5, 10, 13, 16, 17, 21; Anxiety: items: 2, 4, 7, 9, 15, 19, 20; Stress: items 1, 6, 8, 11, 12, 14, 18.

Descriptive statistics of empirical distribution of individuals’ responses (mean, standard deviation, skewness, and kurtosis) and inter-item Spearman’s correlations are shown in the (S1 and S2 Tables, respectively). There was a general trend of positive skewness and a slightly platykurtic shape, and the items were all significantly correlated, showing moderate to large effects.

Factorial validity

The goodness-of-fit indices of the nine models are presented in Table 1. Models 1 and 3 exhibited the worst fit according to all indices. Models 2 and 6 showed a slightly better fit compared to the previous ones, though still falling short of acceptability. Models 4 and 5 demonstrated an acceptable fit to the data, even if the SBχ² /df ratio was above the threshold of 3 for both models. Therefore, Models 1 to 6 were all excluded. Among Models 7 (bifactor CFA), 8 (ESEM), and 9 (bifactor-ESEM), all displayed optimal fit to the data with very similar indices, thus prompting a focused comparison. First, comparing Models 7 and 8, the former showed superior performance only in terms of SBχ² /df ratio, while the latter demonstrated marginally better (Δ = |0.01|) SRMR and TLI values; the other fit indices remained consistent between the two models. Then, Model 9 (bifactor-ESEM) outperformed Model 7 in residual-based fit (indicated by a lower SBχ² /df ratio) and showed a slight improvement (Δ = |0.01|) in TLI and SRMR values. However, compared with Model 8, Model 9 was less parsimonious (Model 8: 150 df; Model 9: 132 df) and introduced a marginal improvement in fit only in the SBχ² /df ratio. No other differences in fit indices between Models 8 and 9 were observed.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Fit indices of CFA and ESEM models.

https://doi.org/10.1371/journal.pone.0299229.t001

Given that Models 7, 8 and 9 demonstrated excellent and nearly identical fit indices, psychometric evidence alone seems insufficient to determine the best representation of the DASS-21 factor structure among these models. The literature underscores that the selection of the final model should be guided not only by psychometric performance but also by well-founded theoretical and interpretative reasons, especially when objective indices fail to provide a clear indication (e.g., [52]). Therefore, the following theoretical considerations come to the forefront. The introduction and application of the ESEM method for uncovering latent structure in the measurement of psychological constructs appears to be a flexible solution driven by realistic assumptions about item correlations. It is plausible, for instance, that items intended to measure a particular construct might be slightly but systematically correlated with different constructs, as in the case of the DASS-21. To be specific, from an interpretative standpoint, the DASS-21 ESEM model (Model 8) indicates that items are strongly influenced by target latent factors and minimally by non-target latent factors. In addition, the three latent factors are collectively influenced by a more general (i.e., second-order) factor, which is conceived as a broader construct encompassing Depression, Anxiety, and Stress. Thus, the ESEM model seems theoretically supported as a representation of the DASS-21 factor structure. On the contrary, in the DASS-21 context, a bifactor (Model 7) or bifactor-ESEM (Model 9) model could be particularly misleading. Indeed, CFA bifactor modeling imposes orthogonality between the general factor (i.e., general distress) and the specific factors (i.e., Depression, Anxiety, and Stress), thus assuming independence among the constructs; this assumption, however, does not seem realistic from a theoretical standpoint. Although in bifactor-ESEM modelling the orthogonality constraint is relaxed by specifying an oblique rotation among latent factors, even this solution does not seem an adequate representation of the DASS-21 structure. In fact, compared to the ESEM model (Model 8), bifactor-ESEM (Model 9) introduces an additional construct accounting for a common systematic variability among all 21 items beyond Depression, Anxiety, and Stress. Notably, the additional factor introduced in bifactor-ESEM is not a construct that includes the other three factors; instead, it assumes a source of variability that is common to all items but does not include the other three psychological constructs, thus potentially being “external” to them (e.g., common method bias).

For all the reasons outlined above, Model 8 should be preferred over Models 7 and 9 as it combines theoretical support, interpretability, and parsimony. A bifactor-ESEM model in this context could be deemed overly complex and lacking robust theoretical support, potentially resulting in an equivocal interpretation of what items are measuring. Therefore, prioritizing a clearer understanding becomes imperative, especially in the face of a non-decisive improvement in model fit.

All factor loadings of the ESEM model are reported in Table 2. The hierarchical structure of the scale (mathematically equivalent to a model with a second-level common factor) is suggested by the size and statistical significance of the respective factor loadings on the target factors. For the Depression factor, the target items displayed statistically significant loadings, with moderate to high correlations with this latent factor. A similar pattern was observed for the target items of both the Anxiety and Stress factors: their loadings were all statistically significant and demonstrated moderate to high effect sizes. Specifically, the target items of the Anxiety factor showed slightly higher factor loading values compared to those of the Depression and Stress factors. Furthermore, item loadings on the target factors were overall higher than those on non-target factors. The high omega coefficients for all the scales (ωs ranging from 0.81 to 0.93) and the total score (ω = 0.80) indicated very good to excellent score reliability estimates (Table 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Standardized factor loadings and McDonald’s omega values of the ESEM solution.

https://doi.org/10.1371/journal.pone.0299229.t002

The standardized correlation coefficients between the first-level factors of the ESEM solution were all large in magnitude and significant (Depression-Anxiety: r = 0.712, Depression-Stress: r = 0.592, Anxiety-Stress: r = 0.785; all ps < 0.001).

Finally, descriptive indices (mean, standard deviation, percentiles) of the DASS-21 based on the ESEM solution are presented in the (S3 Table).

Measurement invariance

Pertaining to sex, the configural model showed a good fit to the data, supporting the configural invariance of the DASS-21 across sex levels. Invariance with respect to the sex factor was also supported at the scalar level, showing a good fit and negligible changes in CFI and RMSEA (ΔCFI <0.001, ΔRMSEA <0.001).

Subsequently, MI across age was tested. The younger group was composed of adolescents aged 14–16 years (n = 349, 53.3%), whereas the older group included adolescents aged 17–18 years (n = 306, 46.7%), following the classification used by Jovanović et al. [41]. The configural model exhibited an adequate fit to the data. Scalar invariance was also supported, as shown by nonsensible changes in both CFI and RMSEA values between the configural and scalar models (ΔCFI <0.001, ΔRMSEA <0.001). Table 3 summarizes the abovementioned results regarding MI.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Fit Indices and difference statistics for MI models (ESEM) across sex and age.

https://doi.org/10.1371/journal.pone.0299229.t003

Latent means differences

Using the parameters of the ESEM model, the latent means across sex and age were compared. The latent means of boys and the younger adolescent group were fixed to zero, following the approach of Jovanović et al. [41]. The analyses of the latent means showed that girls reported higher levels of depression (unstandardized fitted mean(girls) = 1.011, SE = 0.120, p < 0.001) and anxiety (unstandardized fitted mean(girls) = 0.431, SE = 0.172, p < 0.05), but not higher levels of stress (unstandardized fitted mean(girls) = 0.281, SE = 0.138, p = 0.142). No significant age differences were observed in depression (unstandardized fitted mean(older) = −0.102, SE = 0.088, p = 0.181), anxiety (unstandardized fitted mean(older) = −0.145, SE = 0.081, p = 0.289), and stress (unstandardized fitted mean(older) = −0.077, SE = 0.086, p = 0.274).

Convergent and divergent validity

The fit indices of the nine structural models of the DASS-21 were evaluated in the subsample of 312 participants who completed the PANAS. The associations between the DASS-21 and PANAS were then estimated. The ESEM model of the DASS-21 provided also in this subsample a good fit to the data (SBχ² (134) = 294.710, p < 0.001, RMSEA = 0.04 (90% CI = 0.03, 0.05), CFI = 0.98, TLI = 0.97, SRMR = 0.03).

To assess both convergent and divergent validity of the DASS-21, a structural model was fitted, including direct paths from the ESEM solution to the PA and NA factors of the PANAS. Standardized path coefficients (Table 4) indicated that the Depression factor had a strong and negative association with PA but did not significantly predict NA. The Anxiety factor showed a positive association with NA and a negative one with PA, as expected. The Stress factor was not significantly associated with either PA or NA.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Standardized path coefficients for the regression of PA and NA on the DASS-21 factors using the ESEM model.

https://doi.org/10.1371/journal.pone.0299229.t004