Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Correlates of wanting to seek help for mental health and substance use concerns by sexual and gender minority young adults during the COVID-19 pandemic: A machine learning analysis

  • Anasua Kundu ,

    Roles Data curation, Formal analysis, Methodology, Project administration, Software, Validation, Visualization, Writing – original draft

    anasua.kundu@mail.utoronto.ca

    Affiliations Institute of Medical Science, University of Toronto, Toronto, Canada, Centre for Addiction and Mental Health, Toronto, Canada, Ontario Tobacco Research Unit, University of Toronto, Toronto, Canada

  • Rui Fu,

    Roles Conceptualization, Formal analysis, Funding acquisition, Methodology, Supervision, Writing – review & editing

    Affiliations Department of Otolaryngology—Head and Neck Surgery, Sunnybrook Research Institute, University of Toronto, Toronto, Canada, Dalla Lana School of Public Health, University of Toronto, Toronto, Canada

  • Daniel Grace,

    Roles Conceptualization, Funding acquisition, Writing – review & editing

    Affiliation Dalla Lana School of Public Health, University of Toronto, Toronto, Canada

  • Carmen H. Logie,

    Roles Conceptualization, Funding acquisition, Methodology, Writing – review & editing

    Affiliations Factor-Inwentash Faculty of Social Work, University of Toronto, Toronto, Canada, United Nations University Institute for Water, Environment & Health, Hamilton, Canada

  • Alex Abramovich,

    Roles Conceptualization, Funding acquisition, Writing – review & editing

    Affiliations Centre for Addiction and Mental Health, Toronto, Canada, Dalla Lana School of Public Health, University of Toronto, Toronto, Canada, Department of Psychiatry, University of Toronto, Toronto, Canada

  • Bruce Baskerville,

    Roles Conceptualization, Funding acquisition, Methodology, Writing – review & editing

    Affiliations Canadian Institutes of Health Research, Ottawa, Canada, School of Pharmacy, Faculty of Science, University of Waterloo, Kitchener, Canada

  • Christina Yager,

    Roles Conceptualization, Funding acquisition, Writing – review & editing

    Affiliation Centre for Addiction and Mental Health, Toronto, Canada

  • Robert Schwartz,

    Roles Conceptualization, Funding acquisition, Writing – review & editing

    Affiliations Centre for Addiction and Mental Health, Toronto, Canada, Ontario Tobacco Research Unit, University of Toronto, Toronto, Canada, Dalla Lana School of Public Health, University of Toronto, Toronto, Canada

  • Nicholas Mitsakakis,

    Roles Conceptualization, Funding acquisition, Methodology, Writing – review & editing

    Affiliations Dalla Lana School of Public Health, University of Toronto, Toronto, Canada, Children’s Hospital of Eastern Ontario Research Institute, Ottawa, Canada

  • Lynn Planinac,

    Roles Conceptualization, Funding acquisition, Investigation, Resources, Software, Writing – review & editing

    Affiliation Ontario Tobacco Research Unit, University of Toronto, Toronto, Canada

  • Michael Chaiton

    Roles Conceptualization, Data curation, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Writing – review & editing

    Affiliations Institute of Medical Science, University of Toronto, Toronto, Canada, Centre for Addiction and Mental Health, Toronto, Canada, Ontario Tobacco Research Unit, University of Toronto, Toronto, Canada, Dalla Lana School of Public Health, University of Toronto, Toronto, Canada

Abstract

The COVID-19 pandemic has worsened the mental health and substance use challenges among many people who are Two Spirit, lesbian, gay, bisexual, transgender, queer, questioning, and intersex (2SLGBTQI+). We aimed to identify the important correlates and their effects on the predicted likelihood of wanting to seek help among 2SLGBTQI+ young adults for mental health or substance use concerns during the pandemic. A cross-sectional survey was conducted in 2020–2021 among 2SLGBTQI+ young adults aged 16–29 living in two Canadian provinces (Ontario and Quebec). Among 1414 participants, 77% (n = 1089) wanted to seek help for their mental health or substance use concerns during the pandemic, out of these, 69.8% (n = 760) reported delay in accessing care. We built a random forest (RF) model to predict the status of wanting to seek help, which achieved moderately high performance with an area under the receiver operating characteristic curve (AUC) of 0.85. The top 10 correlates of wanting to seek help were worsening mental health, age, stigma and discrimination, and adverse childhood experiences. The interactions of adequate housing with certain sexual orientations, gender identities and mental health challenges were found to increase the likelihood of wanting to seek help. We built another RF model for predicting risk of delay in accessing care among participants who wanted to seek help (n = 1089). The model identified a similar set of top 10 correlates of delay in accessing care but lacked adequate performance (AUC 0.61). These findings can direct future research and targeted prevention measures to reduce health disparities for 2SLGBTQI+ young adults.

1. Introduction

Two Spirit, lesbian, gay, bisexual, transgender, queer, questioning, and intersex (2SLGBTQI+) young adults experience significant mental health disparities and substance use challenges [15]. In 2019, 46.8%% of lesbian, gay or bisexual (LGB) youth in the United States (US) reported having serious thoughts of attempting suicide compared to only 14.5% of their heterosexual peers. Similarly, the rate of illicit drug use was reported to be more than twice among the LGB youth (27.8% among LGB vs 12.7% among heterosexual) [4]. Recent meta-analysis found that 2SLGBTQI+ youth had 2.67 times higher odds of experiencing mental health difficulties than heterosexual and cisgender counterparts [3]. Moreover, lifetime use of substances was reported 1.48 times more likely among transgender individuals compared to the cisgender people [5].

Growing evidence has demonstrated that that the recent pandemic outbreak of Coronavirus disease 2019 (COVID-19) exacerbated the mental health and substance use problems among sexual and gender minority (SGM) people [68]. A Canadian prospective study showed that SGM participants had experienced higher and increasing rate of deterioration of mental health, suicidal thoughts, alcohol, cannabis and substance use compared to non-SGM people [6]. Moreover, multiple reports of difficulty in accessing healthcare by 2SLGBTQI+ people is concerning and needs urgent attention [7,9]. However, it is not clear which individual-level factors may have contributed to the desire to seek help for mental health and substance use concerns among 2SLGBTQI+ young adults during the pandemic. These results have crucial implications to guide policy interventions that target this specific population. We aimed to identify these factors by using machine learning analysis—a novel statistical method which has emerged as a promising means of analyzing a vast range of complex data in public health informatics [10,11]. Our recent scoping review of machine learning applications in mental health and substance use issues among SGM people found only 4 prediction modelling studies up to 2020 [12]. We recently updated the review and detected 3 additional studies [1315] that applied machine learning-based predictive modelling and none of them explored the wanting to seek help behaviours for mental health and substance use issues among this population. Hence, we addressed this research gap by this current study.

Machine learning has also been increasingly used as an important tool in the field of intersectional research of inequalities targeting socio-demographic identities and social positions (e.g., age, sex, gender, ethnicity, socio-economic status) [16,17] through providing newer techniques and methods [18,19]. Recent data suggests that 84% and 66% of Canadian homeless 2SLGBTQI+ young adults experienced severe anxiety and moderately severe depression respectively, while 57% of these young adults reported problematic substance use during the pandemic [20]. Moreover, unemployment and urbanicity have been found to increase the risk of suicidal thoughts among SGM youth who experienced mental health challenges or social stigma [13]. It is important to understand how these interactions of SGM status with different socio-economic identities influence needs of seeking help. In addition, as 2SLGBTQI+ people encounter stigma, bullying, violence and institutional discrimination throughout their lifespan [2123], further contributing to the development of mental health challenges and substance use concerns among them [3,23,24]. Hence, understanding the role of these risk factors behind wanting to seek help and accessing care could help policy makers to plan equitable distribution of healthcare.

The primary objective of our study was to identify the important correlates and their effects on the predicted risk of wanting to seek help for mental health and substance use concerns among 2SLGBTQI+ young adults during the pandemic. We also aimed to assess the prevalence and correlates of delay in accessing care among 2SLGBTQI+ young adults who wanted to seek help from health professionals as a secondary objective.

2. Materials and methods

2.1 Data source

Data for this cross-sectional analysis were taken from the 2SLGBTQI+ Tobacco Project Survey, conducted among 2SLGBTQI+ young adults aged 16–29 years living in Canada during December 2020 to March 2021. This study was approved by the Centre for Addiction and Mental Health Research Ethics Board and adhered to the principles of the Declaration of Helsinki. Individuals provided written informed consent before participating in this study. Participants were recruited from two Canadian provinces: Ontario and Quebec. We received 1511 responses and participants were compensated $10 CAD. Participants who did not complete the survey were excluded, resulting in a total of 1414 respondents.

All statistical analyses were performed using R, version 4.0.5 (R Foundation for Statistical Computing). The survey questionnaire and the R codes required to reproduce the analysis are available on the Open Science Framework (https://osf.io/tnd2u/) and the deidentified data can be accessed from the University of Toronto Dataverse [25]. The entire analytical framework of the machine learning models from data pre-processing to sensitivity analysis were presented in S1 Appendix.

2.2 Outcome measures

We assessed two binary outcomes in this analysis, including a primary outcome defined for the entire study sample and a secondary outcome defined only for a subgroup of participants. Our primary outcome was the status of wanting to seek help from health professionals for mental health or substance use concerns during the pandemic, which was denoted as March 2020 [26,27]. This variable was defined using the survey question, ‘Since March 2020, was there a time when you wanted to talk with or seek help from a health professional about stress, depression, problems with emotions or substance use?’ Next, amongst participants who answered ‘yes’ to this question, i.e., those who had ever wanted to seek help during the pandemic (n = 1089, 77.0% of all participants), we created another binary outcome variable of ‘delay in accessing care’ based on the question ‘Did you delay or not get the care you thought you needed?’

2.3 Features

For both outcomes, we considered the same set of 61 features, including 53 categorical and 8 continuous variables (see https://osf.io/tnd2u/ for full list of variables). From the survey, we extracted socio-demographic characteristics (age, sexual orientation, gender, ethnicity, education, employment, income, duration of living in Canada, and housing status); health status (self-rated general and mental health, comorbidity, disability, stress, diagnosis of different mental illness, past year suicidal thoughts and past-week depression score); frequency of smoking, vaping, other tobacco products use, alcohol drinking, cannabis and other drug use; and perceived COVID-19 impact on smoking and other substance use. Additionally, we created several composite score variables based on validated scales measures, which included overall scores of enacted and perceived stigma [28], internalized homophobia [29], identity centrality [30], community connectedness [31], average scores of outness [32] and overall scores of adverse childhood experiences (ACEs) [33].

2.4 Descriptive analysis and data pre-processing

We summarized characteristics of the participants based on the status of wanting to seek help (n = 1414) and delay in accessing care (n = 1089), respectively. For continuous variables, we used 2-sample t-test and for categorical variables, we used Fisher’s Exact Test to compare the characteristics in both descriptive analyses. Due to the presence of small cell counts (n<5), we used Fisher’s Exact Test instead of Pearson Chi-square test to robustly identify any significant difference between groups [34].

We performed standard data preprocessing procedures to create a dataset ready for the machine learning analysis. We removed or recoded low-variance variables, defined as those with percentage of unique values <5% [35,36]. Inspection was carried out for all variables to rule out collinearity (i.e., all pairwise Cramér’s V<0.6) [37]. For missing data found in our dataset (2.8% total missingness with 15 variables showing ≥5% missing data), we handled them using the Multivariate Imputation by Chained Equations (MICE) approach [38] after confirming the assumption of missing at random was plausible. Specifically, 5 MICE-imputed datasets were independently generated, where based on the performance (highest AUC) of the models, one dataset was chosen to be used in the primary analysis, with the remainder being assessed in the sensitivity analysis. Before running the final model, we created the composite score variables in the imputed data. Notably, out of 1414 participants, 11% (n = 161) declined to respond to the ACEs questionnaire, which were managed by MICE imputation before creating overall scores of ACEs. No feature selection was performed due to the relatively lower number of total variables (61 variables). We additionally used ‘missForest’, a random forest-based iterative algorithm [39] to examine the impact of missing data in the sensitivity analysis.

2.5 Random forest analysis

We performed separate random forest (RF) analysis for the primary outcome (on the entire study sample, n = 1,414) and the secondary outcome (on those who had wanted to seek help, n = 1,089), following the same analytical pipeline (S1 Appendix). We specifically chose RF due to its proven effectiveness in public health applications, particularly in predicting smoking and vaping outcomes, and provides higher level of accuracy compared to others [4042]. In each analysis, we randomly split the imputed dataset at 7:3 ratio to obtain a larger training set for model development and a smaller testing set for evaluating model performance. Using the R package ‘caret’ [43] and 10-fold cross-validation method [44], we built the RF model on the training dataset using the default setting for hyperparameters (S1 Appendix). Performance of the RF model was evaluated on the testing set using the receiver operating characteristic (ROC) curve. The area under the ROC curve (AUC) was computed to give an overall account of the classification ability of the model while high performance was declared using an AUC threshold of 0.80 [45]. Accuracy, sensitivity, and specificity of the RF models were also calculated. If the final model failed to reach a testing AUC of 0.70 [46], we implemented regularized logistic regression using the Lasso (least absolute shrinkage and selection operator) [47] to see if this would yield a better performance (see sensitivity analysis).

2.6 Interpreting the random forest findings

For each outcome, we used the respective RF model to conduct post-hoc analyses to provide interpretable policy-relevant findings. First, we identified the top ten most important contributing features (i.e., correlates) based on a relative importance score calculated for each variable. These scores corresponded to the loss of prediction accuracy of the RF model due to the exclusion of a variable [48,49]. We ranked all the variables on a scale of 0% to 100% with the most important one receiving the highest score of 100%. For our primary outcome (wanting to seek help), we additionally illustrated the one-way partial dependence plots (PDPs) for each of its top ten correlates to demonstrate their marginal effect. We further assessed the partial dependence based 2-way interaction effects [50] by identifying pairs of variables that were jointly significant in predicting the probability of wanting to seek help. This was done by examining the statistical strength of all 91 pairs of interactions formed by sociodemographic variables (such as age*gender) and/or the previously identified top 10 correlates, following the procedures conducted in a recent machine learning paper [51]. We were particularly interested in studying interactions between sociodemographic variables to demonstrate the effect of intersectionality [52]. Two-way PDPs were then used to demonstrate the top 10 strong interactions and their marginal effects.

2.7 Sensitivity analysis

Sensitivity analyses were conducted to examine the robustness of the primary findings. First, we repeated the analyses separately on the remaining 4 MICE imputed datasets. Second, we also ran completed-case analyses by only including participants with full data (n = 364, 25.7% of all participants for the primary outcome; n = 293, 26.9% of all participants for the secondary outcome), although these models were expected to have poor performance. Third, to further assess the impact of missing data, we implemented the ‘missForest’ algorithm and compared the resulted RF models with our primary models. We used ‘missForest’ because it was found to outperform other data imputation algorithm including MICE and K-nearest neighbour imputation [39]. Fourth, we derived a logistic regression model using lasso if the primary RF model failed to reach an AUC of 0.70. The lasso method achieves a sparse model by penalizing the number of predictors and only retaining one of the two correlated predictors [47].

3. Results

3.1 Sample characteristics

Among the 1414 2SLGBTQI+ participants aged 16–29 years, 77.0% (n = 1089) wanted to seek help during the pandemic, while 19.9% (n = 282) participants reported not wanting to seek any help (Table 1). Nearly all (n = 1372, 97.0%) participants lived in urban areas and had an average age of 21.9 years (standard deviation [SD] = 3.78). Participants identified themselves with a range of gender identities, 45.1% (n = 607) of participants identified as cisgender women, 16.3% (n = 220) as cisgender men, 6.9% (n = 93) as transgender women, 1.8% (n = 24) as transgender men, 29.9% (n = 402) as gender diverse including genderqueer, non-conforming, non-binary, or Two-Spirit, and 2.0% (n = 26) reported being intersex. Among the sexual orientation categories, 29.8% (n = 421) identified themselves as bisexual, 16.5% (n = 233) as lesbian, 15.9% (n = 225) as gay, 17.2% (n = 243) as queer, 1.6% (n = 22) as questioning, and 14% (n = 198) as other. Most (68.2%; n = 922) of the total participants were White and 31.8% (n = 430) were persons of colour (POC) from different ethnicities such as Asian, African American, Latin American, Middle eastern, Indigenous, and mixed heritage.

thumbnail
Table 1. Socio-demographic characteristics of participants by wanting to seek help for mental health or substance use concerns during the pandemic (n = 1414).

https://doi.org/10.1371/journal.pone.0277438.t001

Wanting to seek help was found to differ significantly by age, sexual orientation and gender identity, education, employment, and household income (p<0.01). Notably, those who wanted to seek help were slightly younger on average (mean age 21.75 ± 3.78 years). More than 80.0% of the participants with bisexual (n = 339), queer (n = 214) and others (n = 172) sexual orientation wanted to seek help. Approximately 85.0% of the transgender men (n = 79) and gender diverse (n = 344) participants as well as almost 80.0% of the cisgender women (n = 479) wanted to seek help. The rate of wanting to seek help was 80.6% (n = 741) among participants who had less than a university degree, 82.1% (n = 792) among unemployed, 83.1% (n = 458) among those who had a past year household income of less than $60,000. In addition, wanting to seek help was also significantly (p<0.01) higher among past year substance users (82%, n = 356), those with anxiety disorder (90.3%, n = 561) and depression (91.6%, n = 489), higher stress levels (86.5%, n = 753), poor or fair mental health (88.2%, n = 851) and general health (87.1%, n = 471), and who reported quitting smoking (90.7%, n = 39) or increased alcohol drinking or other substance use (86.5%, n = 428) as impacts of COVID-19.

3.2 Performance of the RF model for wanting to seek help

The RF model built for predicting the status of wanting to seek help for mental health or substance use issues showed moderately high performance, with accuracy 0.86 (95% CI 0.83, 0.89), sensitivity 0.53 (95% CI 0.42, 0.63), specificity 0.95 (95% CI 0.92, 0.97) and AUC 0.85 (Fig 1).

thumbnail
Fig 1. The receiving operating characteristic curve of the random forest on the testing set for predicting wanting to seek help.

https://doi.org/10.1371/journal.pone.0277438.g001

3.3 Top 10 correlates of wanting to seek help

The top 10 correlates of wanting to seek help identified from the RF model were (Fig 2): current self-rated mental health status (100%), average outness score (59.3%), overall enacted stigma score (51.2%), past-year suicidal thoughts (49.8%), overall internalized homophobia score (45.3%), overall connectedness score (44.2%), overall identity centrality score (43.9%), overall perceived stigma score (34.2%), age (25.8%), and overall ACEs score (25.1%). One-way partial dependence plots (PDPs) depicting the marginal effects of the top 10 correlates are presented in Fig 3. Overall, the likelihood of wanting to seek help was higher among those who rated their mental health as poor or fair, reported to have suicidal thoughts in the past year, higher enacted stigma, higher internalized homophobia, lower connectedness with the community, lower level of identity centrality, and those who rated their outness and perceived stigma much higher or much lower than average. The probability of wanting to seek help was highest among the youngest age groups, and also higher among older age groups. Participants with very high ACEs score showed the highest probability of wanting to seek help, while the probability was also quite high for those with very low ACEs score.

thumbnail
Fig 2. The top ten correlates of wanting to seek help and their scaled relative importance.

https://doi.org/10.1371/journal.pone.0277438.g002

thumbnail
Fig 3. Partial dependence of wanting to seek help on top 10 important correlates.

A) Current self-rated mental health status; B) Past year suicidal thoughts; C) Average outness score; D) Overall enacted stigma score; E) Overall internalized homophobia score; F) Overall perceived stigma score; G) Overall connectedness score; H) Overall identity centrality score; I) Age; J) Overall adverse childhood experiences (ACEs) score. Note: Numeric variables (average outness score and overall internalized homophobia score) were standardized before model training.

https://doi.org/10.1371/journal.pone.0277438.g003

3.4 Interaction effects on predicted risk of wanting to seek help

All pairwise interactions formed by socio-demographic variables and the top 10 correlates of wanting to seek help were presented in the S1 Table. The strongest interaction was found between current self-rated mental health and housing status (interaction strength 0.2). Other top 10 interactions involved intersections of housing status with sexual orientation, gender identities, past-year suicidal thoughts, average outness score and overall internalized homophobia score; interactions were found between sexual orientation and gender identities with self-rated mental health and past-year suicidal thoughts. Specifically, after stratified by mental health status, past-year suicidal thoughts, outness score and internalized homophobia score, individuals with adequate housing during the pandemic always had a higher probability of wanting to seek help than those experiencing inadequate housing (Fig 4). In addition, among young adults who self-identified themselves as bisexual, others, queer and questioning, gender diverse or cisgender, the risk of wanting to seek help was further increased by having adequate housing, poor or fair self-rated mental health or past-year suicidal thoughts (Figs 5 and 6).

thumbnail
Fig 4. Two-way partial dependence (PD) plots of strong interactions of wanting to seek help.

A) current self-rated mental health and housing status; B) past-year suicidal thoughts and housing status; C) Average outness score and housing status; D) Overall internalized homophobia score and housing status. Note: Numeric variables (average outness score and overall internalized homophobia score) were standardized before model training.

https://doi.org/10.1371/journal.pone.0277438.g004

thumbnail
Fig 5. Two-way partial dependence (PD) plots of strong interactions of wanting to seek help.

A) current sexual orientation and self-rated mental health; B) current sexual orientation and housing status; C) current sexual orientation and past-year suicidal thoughts.

https://doi.org/10.1371/journal.pone.0277438.g005

thumbnail
Fig 6. Two-way partial dependence (PD) plots of strong interactions of wanting to seek help.

A) gender identity and self-rated mental health; B) gender identity and past-year suicidal thoughts; C) gender identity and housing status.

https://doi.org/10.1371/journal.pone.0277438.g006

3.5 Descriptive statistics and predicted risk of delay in accessing care

Of the total 1089 participants who wanted to seek help during the pandemic, 69.8% (n = 760) reported delay in accessing care and 25.8% (n = 281) had no delay in accessing care (Table 2). Participants who had delay in accessing care were slightly younger on average (mean age 21.63 ± 3.73 years, p = 0.007) and differed significantly (p = 0.002) by highest level of education completed. Particularly among those who had less than a university degree, 72.1% (n = 534) reported delay in accessing care. The RF model built for predicting risk of delay in accessing care failed to achieve satisfactory performance (AUC 0.61) on the testing data (S2 Appendix). The top 10 correlates were: average outness score (100%), overall internalized homophobia score (85.7%), overall ACEs score (81.8%), age (77.4%), overall connectedness score (77.3%), overall perceived stigma score (76.2%), overall enacted stigma score (73.4%), overall identity centrality score (71.2%), diagnosis of other mental illness (27.1%), current self-rated general health (26.4%) (S1 Fig). However, due to the unsatisfactory performance of this RF model, we deemed those findings to be less reliable.

thumbnail
Table 2. Socio-demographic characteristics of participants by delay in accessing care during the pandemic (n = 1089).

https://doi.org/10.1371/journal.pone.0277438.t002

3.6 Sensitivity analysis

All alternative RF models built on the remaining 4 MICE-imputed copies of the datasets and with ‘missForest’ for wanting to seek help and delay in accessing care performed similarly with AUC ranging 0.82–0.85 and 0.56–0.61 respectively (S2 and S3 Appendix). For both outcomes, the model performance dropped considerably while the models were built on the completed cases datasets (AUC 0.75, AUC 0.52 respectively). The Lasso logistic regression model found 15 of the 61 predictors to have a non-zero coefficient when predicting the status of delay in accessing care, including 4 (average outness score, overall internalized homophobia score, self-rated general health, diagnosis of other mental illness) that were previously identified to be among the top 10 correlates by RF (S2 Appendix). However, this lasso regression model failed to show better performance on the testing set (AUC 0.57). Hence, we conclude that there is a lack of key predictors of delay in accessing care in this specific dataset.

4. Discussion

We applied machine learning to analyze factors associated with wanting to seek help for mental health and substance use concerns among 2SLGBTQI+ young adults during the pandemic. Using the resulted RF model that had moderately high performance, we identified correlates of wanting to seek help that depict vulnerable young adults in need of urgent interventions. Furthermore, we used the same RF technique to identify those who had delay in accessing care; however, this model demonstrated poor performance which impeded our ability to draw any conclusive findings on correlates and interactions.

The findings of our study support an emerging body of evidence [7,20,53] that a significant proportion of 2SLGBTQI+ young adults showed wanting to seek help during the pandemic, while most of them reported delay in accessing care (Tables 1 and 2). Moreover, we found that the rate of wanting to seek help was higher among those who decreased or quit their smoking, or increased drinking or substance use due to COVID-19 pandemic. These findings suggests that 2SLGBTQI+ young adults probably wanted services for quitting smoking or managing consequences of their substance use. The rate of wanting to seek help was significantly higher among unemployed young adults. However, it is important to recognise that most of the participants (80.6%, n = 741) in our sample did not finish a university degree and may be still studying at the undergraduate levels, which might be reflected by the higher rate of unemployment in the overall sample (Table 1).

The top 10 correlates of wanting to seek help among 2SLGBTQI+ young adults identified in our study were mainly related to mental health challenges, age, sexual and gender identity related stigma and discrimination, and ACEs (Figs 2 and 3). Consistent with previous research, we found an association between wanting to seek help and poor or fair self-rated mental health status [54,55]. Previous research reported decreased rate of seeking help among those who seriously considered or attempted suicide [54,56]. In our study, we found increased likelihood of wanting to seek help among participants who had suicidal thoughts in the past year. However, this particular finding of increased needs of seeking help among 2SLGBTQI+ young adults experiencing mental health challenges presents an opportunity for timely intervention. In addition to increased prevalence of mental health issues [6], making virtual mental health visits became more common during the pandemic [57]. Hence, 2SLGBTQI+ young adults can be encouraged to leverage these services, seek help and access urgent online supports at the time of mental health emergency.

The association of exposure to enacted stigma or discrimination, perceived stigma, homophobia, low community connectedness, low level of identity centrality with adverse mental health outcomes have been already demonstrated [22,58,59]. Our results shows that these experiences also increased risk of wanting to seek help. Previous research among 2SLGBTQI+ college students concluded that high level of outness was associated with high likelihood of affected by enacted stigma and high probability of help-seeking [60]. The same effect was demonstrated in our study for wanting to seek help. While more research are needed to explore ways to reduce stigma and discrimination against 2SLGBTQI+ people, tailored peer support programs can improve self-advocacy, community connectedness, and improve mental health as well as quality of life [61]. In addition, policy initiatives like anti-discrimination measures and LGBTQI2S+ inclusion in the socio-economic development [62] will help to reduce discrimination. Evidence suggests that high exposure to ACEs predicts poor mental health outcomes among SGM youth and adults [63,64]. Moreover, high ACEs score were found to be associated with ever-smoking and ever-vaping among the 2SLGBTQI+ young adults [61]. We found similar effect of increased likelihood of wanting to seek help among participants with high ACEs score. However, the association between wanting to seek help and low ACEs score more likely resulted from the data imputation of high percentage (11% of the total sample) of missing data for the ACEs questionnaire. While this relationship between ACEs and wanting to seek help should be further explored exclusively, the ACEs questionnaire can be used for screening mental health and substance use challenges among 2SLGBTQI+ young adults.

Our findings suggest that individuals who had adequate housing, but poor mental health or past-year suicidal thoughts were more likely to want to seek help than those with inadequate housing. However, a recent study reported that homeless 2SLGBTQI+ youth were experiencing poor mental health, suicidal thoughts, self-harm, increased substance use and other mental health issues as an impact of COVID-19 [20]. This difference may be attributable to numerous barriers (i.e., lack of affordability, lack of transportation, stigmatization) faced by homeless individuals which may discouraged them to seek healthcare [65]. Moreover, past experiences of stigmatization and discrimination from health care providers for their sexual and gender identities may further inhibit young people to seek help [9]. Consistent with this possibility, we found that participants with adequate housing and lower level of internalized homophobia or average outness had higher probability of wanting to seek help (Fig 4). Notably, in our sample, bisexual, queer and questioning, cisgender, and gender diverse participants, who experienced mental health challenges or had adequate housing, were more likely to want to seek help than others (Figs 5 and 6). This finding contradicts previous research, where trans people were similar to cisgender people or more likely to experience mental health challenges and seek help than cisgender participants [55,66]. While future research should explore the reason behind these differences, these findings should be taken into account for the purpose of targeted prevention and intervention to reduce health disparities faced by 2SLGBTQI+ young adults.

Both the RF model and the Lasso regression model built for predicting delay in accessing care had poor performance (S2 Appendix), largely attributed to our limited sample size and capture of key predictors. Notably, delay in accessing care result from a complex interplay of factors beyond the individual level (structures and processes of care), as outlined in the Donabedian model [67]. Nevertheless, a few individual-level variables (outness, internalized homophobia, self-rated general health, and other mental illness) were identified as potentially important correlates in both models, which may imply some meaningful mechanisms that underlie high risks of care delays. This aligns with research on real fears of violence, and social exclusion experienced by SGM people as well as lack of 2SLGBTQI+ friendly healthcare providers during disclosure [68]. Specifically, recent studies reported that SGM people faced significant rates of health care discrimination and harassments by healthcare providers, including refusal to provide care as well as inflicting verbal and physical abuse during the COVID-19 pandemic [7,9]. These experiences of discrimination within the healthcare system should be incorporated into future research to evaluate their impact on predicting the risk of delayed access to care. Future study with more extensive capture of variables is needed to identifying predictors of delay in accessing care by 2SLGBTQI+ young adults to enable policy planning that ameliorates health care discrimination and ensures health equity.

We demonstrated that machine learning can be used successfully to identify potential risk factors of mental health and substance use related issues among the 2SLGBTQI+ young adults, with an adequate sample size to capture of key predictors. Compared to traditional regression approaches, machine learning allowed us to investigate potentially complex relationships amongst 61 variables without imposing significant assumptions to identify statistically meaningful individual predictors as well as interactions [69,70]. This flexible data-driven technique is particularly useful in hypothesis-generating settings to support causality-testing investigations. As such, this study provides an example of deriving interpretable findings from a black-box machine learning model, which has been identified as being underperformed in the current public health literature [70,71]. Future public health researchers should be mindful of making their machine learning models interpretable and potentially use more sophisticated techniques (such as individual conditional expectation plots [72]) to enhance the real-world relevance of their interpretations. 2SLGBTQI+ people face various social and institutional barriers in accessing timely healthcare services, which are often intertwined with factors that challenge the disclosure of their identities [71]. By leveraging the data-driven strength of machine learning, future researchers could adopt our approach to uncover nuanced findings that hinder the help-seeking behaviours of this underserved population [11]. In addition, beyond supervised machine learning that we demonstrated in this study, techniques of unsupervised learning (cluster analysis [73]) can be applied to characterize unique preferences/trajectories of care that might be particularly suitable for this population to enable a patient-centred care approach.

The findings of this study should be interpreted with few key limitations. Although the RF model predicting status of wanting to seek help yielded relatively high performance, we were unable to achieve satisfactory performance for the models build for predicting delay in accessing care. This weakness of the current study was attributed to the inherent limitations of a cross-sectional survey-based study (e.g., small sample, missing responses, omission of potentially important attributes in the questionnaire). Future researchers should use more comprehensive datasets, preferably ones that link individual- and system-level factors from health services administrative databases with population-based survey data, to depict a fulsome picture of barriers to timely care receipt. Next, our primary outcome denoted the status of wanting to seek help, without confirming how many of them have actually subsequently sought (and received) professional help. This should be addressed in longitudinal studies. Furthermore, we were unable to comment on whether the participants started to experience the needs to seek help since the start of the pandemic or how the pandemic influenced changes in risk factors of wanting to seek help among 2SLGBTQI+ young adults. These unaddressed objectives can be explored in a case-control setting with data from the pre-pandemic era. Moreover, owing to the data driven nature of the machine learning techniques and cross-sectional data, the top 10 correlates identified were not ‘true predictors’, rather factors that might contribute to a larger increase in the likelihood of wanting to seek help than other variables. As such, the correlates suggested by this study should be confirmed using causality testing techniques to formally establish their statistical importance.

5. Conclusions

We conducted an explanatory machine learning analysis on Canadian 2SLGBTQI+ young adults during the COVID-19 pandemic to identify individual-level correlates of wanting to seek professional help and subsequent delay in accessing care. We identified several sexual and gender-identity related stressors influencing wanting to seek help, which should be taken into consideration by the policy makers for targeted prevention approaches and ensuring health equity. Interventions such as peer support programs, social inclusion of 2SLGBTQI+ people on socio-economic development, transportation and combined efforts for people experiencing homelessness, and 2SLGBTQI+ inclusive and affirming health care services should be considered for reducing mental health and substance use challenges as well as improving help-seeking among this population. The poor performance of the random forest models aimed to predicting delay in accessing care warrants additional research using more comprehensive datasets that include factors measured at structures and processes of care levels.

Supporting information

S1 Fig. The top ten correlates of delay in accessing care and their scaled relative importance.

https://doi.org/10.1371/journal.pone.0277438.s001

(TIFF)

S1 Table. Interaction strengths formed by socio-demographic factors and top 10 important correlates (in descending order of interaction strength) of wanting to seek help.

https://doi.org/10.1371/journal.pone.0277438.s002

(DOCX)

S1 Appendix. Analytical framework of the machine learning models.

https://doi.org/10.1371/journal.pone.0277438.s003

(DOCX)

S2 Appendix. Results of sensitivity analysis for predicting risk of delay in accessing care.

https://doi.org/10.1371/journal.pone.0277438.s004

(DOCX)

S3 Appendix. Results of sensitivity analysis for predicting risk of wanting to seek help.

https://doi.org/10.1371/journal.pone.0277438.s005

(DOCX)

References

  1. 1. The Trevor Project. The Trevor Project National Survey. 2020 [cited 2 Nov 2021]. Available: https://www.thetrevorproject.org/survey-2020/?section=Introduction.
  2. 2. Han BH, Duncan DT, Arcila-Mesa M, Palamar JJ. Co-occurring mental illness, drug use, and medical multimorbidity among lesbian, gay, and bisexual middle-aged and older adults in the United States: A nationally representative study. BMC Public Health. 2020;20: 1–9.
  3. 3. Williams AJ, Jones C, Arcelus J, Townsend E, Lazaridou A, Michail M. A systematic review and meta-analysis of victimisation and mental health prevalence among LGBTQ+ young people with experiences of self-harm and suicide. PLoS One. 2021;16. pmid:33481862
  4. 4. Centers for Disease Control and Prevention. Youth Risk Behavior Survey Data Summary & Trends Report: 2009–2019. 2019. Available: https://www.cdc.gov/healthyyouth/data/yrbs/pdf/YRBSDataSummaryTrendsReport2019-508.pdf.
  5. 5. Cotaina M, Peraire M, Boscá M, Echeverria I, Benito A, Haro G. Substance use in the transgender population: A meta‐analysis. Brain Sci. 2022;12(3): 366. pmid:35326322
  6. 6. Slemon A, Richardson C, Goodyear T, Salway T, Gadermann A, Oliffe JL, et al. Widening mental health and substance use inequities among sexual and gender minority populations: Findings from a repeated cross-sectional monitoring survey during the COVID-19 pandemic in Canada. Psychiatry Res. 2022;307: 114327. pmid:34923446
  7. 7. Dawson L, McGough M, Kirzinger A, Sparks G, Rae M, Young G, et al. The impact of the COVID-19 pandemic on LGBT+ people’s mental health. 2021. Available: https://www.kff.org/other/issue-brief/the-impact-of-the-covid-19-pandemic-on-lgbt-peoples-mental-health/.
  8. 8. Chaiton M, Musani I, Pullman M, Logie CH, Abramovich A, Grace D, et al. Access to mental health and substance use resources for 2SLGBTQ+ youth during the COVID-19 pandemic. Int J Environ Res Public Health. 2021;18: 11315. pmid:34769836
  9. 9. Mahowald L, Gruberg S, Halpin J. The state of the LGBTQ community in 2020—A national public opinion study. Center for American Progress. 2020. Available: https://www.americanprogress.org/issues/lgbtq-rights/reports/2020/10/06/491052/state-lgbtq-community-2020/. Accessed 1 Nov 2021.
  10. 10. Jordan MI, Mitchell TM. Machine learning: Trends, perspectives, and prospects. Science. 2015;349(6245): 255–260. pmid:26185243
  11. 11. Luo J, Wu M, Gopukumar D, Zhao Y. Big data application in biomedical research and health care: A literature review. Biomed Inform Insights. 2016;8: BII.S31559. pmid:26843812
  12. 12. Kundu A, Chaiton M, Billington R, Grace D, Fu R, Logie C, et al. Machine learning applications in mental health and substance use research among the LGBTQ2S+ population: Scoping review. JMIR Med Informatics. 2021;9(11): e28962. pmid:34762059
  13. 13. Kundu A, Fu R, Grace D, Logie C, Abramovich A, Baskerville B, et al. Correlates of past year suicidal thoughts among sexual and gender minority young adults: A machine learning analysis. J Psychiatr Res. 2022;152: 269–277. pmid:35759979
  14. 14. Bond MA, Stone AL, Salcido R, Schnarrs PW. How often were you traumatized? Reconceptualizing adverse childhood experiences for sexual and gender minorities. J Affect Disord. 2021;282: 407–414. pmid:33422816
  15. 15. Comulada WS, Goldbeck C, Almirol E, Gunn HJ, Ocasio MA, Isabel Fernández M, et al. Using machine learning to predict young people’s internet health and social service information seeking. Prev sci. 2021;22: 1173–1184. pmid:33974226
  16. 16. Leavitt A. Human-centered data science: Mixed methods and intersecting evidence, inference, and scalability. 2016. Available: https://cscw2016hcds.wordpress.com/papers/.
  17. 17. Morina G, Oliinyk V, Waton J, Marušić I, Georgatzis K. Auditing and achieving intersectional fairness in classification problems. arXiv. 2020. Available: https://arxiv.org/abs/1911.01468. Accessed 28 Oct 2021.
  18. 18. Schiltz F, Masci C, Agasisti T, Horn D. Using regression tree ensembles to model interaction effects: a graphical approach. Appl Econ. 2018;50: 6341–6354.
  19. 19. Oh S. Feature Interaction in Terms of Prediction Performance. Appl Sci. 2019;9: 5191.
  20. 20. Abramovich A, Pang N, Moss A, Logie CH, Chaiton M, Kidd SA, et al. Investigating the impacts of COVID-19 among LGBTQ2S youth experiencing homelessness. PLoS One. 2021;16: e0257693. pmid:34547037
  21. 21. Kosciw JG, Greytak EA, Palmer NA, Boesen MJ. The 2013 National School Climate Survey: The experiences of lesbian, gay, bisexual and transgender youth in our nation’s schools. 2014. Available: https://www.glsen.org/sites/default/files/2020-03/GLSEN-2013-National-School-Climate-Survey-Full-Report.pdf.
  22. 22. Meyer IH. Prejudice, social stress, and mental health in lesbian, gay, and bisexual populations: Conceptual issues and research evidence. Psychol Bull. 2003;129(5): 674–697. pmid:12956539
  23. 23. Duncan DT, Hatzenbuehler ML. Lesbian, gay, bisexual, and transgender hate crimes and suicidality among a population-based sample of sexual-minority adolescents in Boston. Am J Public Health. 2014;104: 272–278. pmid:24328619
  24. 24. Lee JH, Gamarel KE, Bryant KJ, Zaller ND, Operario D. Discrimination, mental Health, and substance use disorders among sexual minority populations. LGBT Heal. 2016;3: 258–265. pmid:27383512
  25. 25. Kundu A, Fu R, Grace D, Logie, Carmen H., Abramovich A, Baskerville B, et al. Wanting to seek help during the pandemic for mental health and substance use concerns among the LGBTQI2S+ young adults dataset. Borealis; 2022.
  26. 26. Public Health Ontario. Enhanced epidemiological summary-Trends of COVID-19 incidence in Ontario. 2020. Available: https://www.publichealthontario.ca/-/media/documents/ncov/epi/covid-19-epi-trends-incidence-ontario.pdf?la=en.
  27. 27. Ontario Ministry of Health. Statement- Ontario hospitals asked to take a planned approach to ramping down elective surgeries. 2020. Available: https://news.ontario.ca/en/statement/56328/ontario-hospitals-asked-to-take-a-planned-approach-to-ramping-down-elective-surgeries.
  28. 28. Logie CH, Earnshaw V. Adapting and validating a scale to measure sexual stigma among lesbian, bisexual and queer women. PLoS One. 2015;10. pmid:25679391
  29. 29. de Lira AN, de Morais NA. Validity evidences of the Internalized Homophobia Scale for Brazilian gays and lesbians. Psico-USF. 2019;24: 361–372. Available:
  30. 30. Mohr JJ, Kendra MS. The Lesbian, Gay, & Bisexual Identity Scale (LGBIS). 2012. Available: https://www.midss.org/sites/default/files/lgbis.pdf.
  31. 31. Frost DM, Meyer IH. Measuring community connectedness among diverse sexual minority populations. J Sex Res. 2012;49: 36. pmid:21512945
  32. 32. Meidlinger PC, Hope DA. Differentiating disclosure and concealment in measurement of outness for sexual minorities: The Nebraska Outness Scale. Psychol Sex Orientat Gend Divers. 2014;1: 489–497.
  33. 33. Centers for Disease Control and Prevention. Behavioral Risk Factor Surveillance System ACE Data. 2020. Available: https://www.cdc.gov/violenceprevention/aces/ace-brfss.html.
  34. 34. Fisher RA. On the interpretation of χ2 from contingency tables, and the calculation of P. J R Stat Soc. 1922;85: 87–94.
  35. 35. Kuhn M. Building predictive models in R using the caret package. J Stat Softw. 2008;28: 1–26.
  36. 36. Noel CW, Sutradhar R, Gotlib Conn L, Forner D, Chan WC, Fu R, et al. Development and validation of a machine learning algorithm predicting emergency department use and unplanned hospitalization in patients with head and neck cancer. JAMA Otolaryngol Head Neck Surg. 2022;148. pmid:35771564
  37. 37. Frey BB. Cramér’s V Coefficient. SAGE Encycl Educ Res Meas Eval. 2018.
  38. 38. van Buuren S, Groothuis-Oudshoorn K. Mice: Multivariate Imputation by Chained Equations in R. J Stat Softw. 2011;45: 1–67.
  39. 39. Stekhoven DJ, Bühlmann P. MissForest—non-parametric missing value imputation for mixed-type data. Bioinformatics. 2012;28: 112–118. pmid:22039212
  40. 40. Breiman L. Random forests. Mach Learn. 2001;45: 5–32. https://doi.org/10.1023/A:1010933404324.
  41. 41. Uddin S, Khan A, Hossain ME, Moni MA. Comparing different supervised machine learning algorithms for disease prediction. BMC Med Inform Decis Mak. 2019;19: 1–16.
  42. 42. Fu R, Kundu A, Mitsakakis N, Elton-Marshall T, Wang W, Hill S, et al. Machine learning applications in tobacco research: a scoping review. Tob Control. 2021; tobaccocontrol-2020-056438. pmid:34452986
  43. 43. Kuhn M. The caret Package. 2019. Available: https://topepo.github.io/caret/index.html.
  44. 44. Kuhn M, Johnson K. Applied Predictive Modeling. Springer. 2013. https://doi.org/10.1007/978-1-4614-6849-3
  45. 45. Rice ME, Harris GT. Comparing effect sizes in follow-up studies: ROC Area, Cohen’s d, and r. Law Hum Behav. 2005;29: 615–620. pmid:16254746
  46. 46. Obuchowski NA. Receiver operating characteristic curves and their use in radiology. Radiology. 2003;229: 3–8. pmid:14519861
  47. 47. Tibshirani R. Regression shrinkage and selection via the Lasso. J R Stat Soc. 1996;58: 267–288. Available: https://www.jstor.org/stable/2346178.
  48. 48. Breiman L, Friedman JH, Olshen RA, Stone CJ. Classification and regression trees. Classification and Regression Trees. CRC Press; 2017. https://doi.org/10.1201/9781315139470
  49. 49. Friedman JH. Greedy function approximation: A gradient boosting machine. Ann Stat. 2001;29: 1189–1232.
  50. 50. Greenwell BM, Boehmke BC, Mccarthy AJ. A simple and effective model-based variable importance measure. arXiv. 2018;abs/1805.04755.
  51. 51. Fu R, Shi J, Chaiton M, Leventhal AM, Unger JB, Barrington-Trimis JL. A machine learning approach to identify predictors of frequent vaping and vulnerable californian youth subgroups. Nicotine Tob Res. 2022;24: 1028–1036. pmid:34888698
  52. 52. Hopkins P. Social geography I: Intersectionality. Prog Hum Geogr. 2019;43: 937–947.
  53. 53. Phillips C. How covid-19 has exacerbated LGBTQ+ health inequalities. BMJ. 2021;372: m4828. pmid:33727244
  54. 54. Wu CY, Lee MB. Suicidality, self-efficacy and mental health help-seeking in lesbian, gay, bisexual and transgender adults in Taiwan: A cross-sectional study. J Clin Nurs. 2021;30. pmid:33529443
  55. 55. McNair RP, Bush R. Mental health help seeking patterns and associations among Australian same sex attracted women, trans and gender diverse people: A survey-based study. BMC Psychiatry. 2016;16: 1–16. pmid:27377408
  56. 56. Lytle MC, Silenzio VMB, Homan CM, Schneider P, Caine ED. Suicidal and help-seeking behaviors among youth in an online lesbian, gay, bisexual, transgender, queer, and questioning social network. J Homosex. 2018;65. pmid:29020574
  57. 57. Zhu JM, Myers R, McConnell KJ, Levander X, Lin SC. Trends In outpatient mental health services use before and during the COVID-19 pandemic. Health Aff. 2022;41: 573–580. pmid:35377763
  58. 58. Veale JF, Peter T, Travers R, Saewyc EM. Enacted stigma, mental health, and protective factors among transgender youth in Canada. Transgender Heal. 2017;2.1. pmid:29279875
  59. 59. Shramko M, Toomey RB, Anhalt K. Profiles of minority stressors and identity centrality among sexual minority Latinx youth. Am J Orthopsychiatry. 2018;88: 471. pmid:29355368
  60. 60. Alberts E, Rohrsetzer N. Identity and social support: LGBTQIA+ individuals and help-seeking behaviors. SUURJ Seattle Univ Undergrad Res J. 2020;4: 46–64. Available: https://scholarworks.seattleu.edu/cgi/viewcontent.cgi?article=1130&context=suurj.
  61. 61. Baskerville B. ACEs high: Cigarette and vape use among gender and sexual minority young adults. Ottawa Conference 2022 Proceedings. 2022. p. 3.
  62. 62. United Nation General Assembly. Protection against violence and discrimination based on sexual orientation and gender identity: Note by the Secretary-General. 2019. Available: www.ohchr.org/EN/Issues/SexualOrientationGender/Pages/SocioCultural.
  63. 63. Schnarrs PW, Stone AL, Salcido RE, Baldwin A, Nemeroff CB. Differences in adverse childhood experiences (ACEs) and quality of physical and mental health between transgender and cisgender sexual minorities. J Psychiatr Res. 2019;119: 1–6. pmid:31518909
  64. 64. Jonas L, Salazar De Pablo G, Shum M, Nosarti C, Abbott C, Vaquerizo-Serrano J, et al. A systematic review and meta-analysis investigating the impact of childhood adversities on the mental health of LGBT+ youth. JCPP Adv. 2022;2: e12079.
  65. 65. Ramsay N, Hossain R, Moore M, Milo M, Brown A. Health care while homeless: barriers, facilitators, and the lived experiences of homeless individuals accessing health care in a canadian regional municipality. Qual Health Res. 2019;29: 1839–1849. pmid:30810072
  66. 66. Howell J, Maguire R. Seeking help when transgender: Exploring the difference in mental and physical health seeking behaviors between transgender and cisgender individuals in Ireland. Int J Transgenderism. 2019;20: 421. pmid:32999627
  67. 67. Donabedian A. Evaluating the Quality of Medical Care. Milbank Q. 2005;83(4): 691. pmid:16279964
  68. 68. Brumbaugh-Johnson SM, Hull KE. Coming out as transgender: Navigating the social implications of a transgender identity. Journal of Homosexuality. 2018;66(8): 1148–1177. pmid:30052497
  69. 69. Ley C, Martin RK, Pareek A, Groll A, Seil R, Tischer T. Machine learning and conventional statistics: making sense of the differences. Knee Surg Sports Traumatol Arthrosc. 2022;30: 753–757. pmid:35106604
  70. 70. Molnar C. Interpretable machine learning: A guide for making black box models. 2019. Available: https://christophm.github.io/interpretable-ml-book/ice.html.
  71. 71. Morgenstern JD, Buajitti E, O’Neill M, Piggott T, Goel V, Fridman D, et al. Predicting population health with machine learning: a scoping review. BMJ Open. 2020;10: e037860. pmid:33109649
  72. 72. Greenwell BM, Boehmke BC. Variable importance plots-An introduction to the vip package. R Journal. 2020;12: 343–365. Available: https://journal.r-project.org/archive/2020/RJ-2020-013/RJ-2020-013.pdf.
  73. 73. Windgassen S, Moss-Morris R, Goldsmith K, Chalder T. The importance of cluster analysis for enhancing clinical practice: an example from irritable bowel syndrome. Journal of Mental Health. 2018;27(2): 94–96. pmid:29447026