Reviewer#1, Concern # 1: Men and women do not experience depression similarly. This needs to be core to this paper. All analyses should be by gender too. Additional tables and figures are needed.

Author response:

Exploring Gender Differences: Does it Matter from Word Cloud?

Since there are some notable differences in expression of depression between males and females. According to Cavanagh et al. (2016), depressed men reported alcohol and drugs misuse, poor impulse control, while depressed women more likely to report depressed mood, appetite, and sleep disturbance [55, 56]. There was also report which found that men have higher rate of anger [57]. Women tended to express their depression more accurately according to the criteria for depressive disorder, making it more difficult to detect same degree of depression in male patients. This section aims to explore whether gender differences influence the words produced in the word cloud from the LIME explanation.

Based on our collected data, which show a prevalence of female and male depression with a ratio of 3:1 and were distributed evenly across severity (as seen in Table 2), the word clouds of Q1 in Fig. 3 produced from depressed females and males (Fig. 3b)) and (Fig. 3c)), illustrated some shared words, such as ‘not’, ‘sad’, and ‘mood’, although in different orders. Similarly, a comparison of the word clouds produced from the normal control females and males (Fig. 3e)) and (Fig. 3f)), reveals shared words such as ‘recently’, ‘fine’, ‘normally’, and ‘working’, but with the same orders. This suggests that the model tries to classify based on the common set of words even when there are some words that only occurred on each side of the gender.

Reviewer#1, Concern # 2: Men show anger and women cry, when depressed. That means, text should be gender specific, at least in part.

Author response:

Error Analysis of Model Misclassification

From Table 3, there were 33 support cases for the depression class in Q1, indicating that 8 of 40 participants were misclassified. Among those misclassified, six were female and two were male. The sample response (English translation) from a misclassified male (Age=63, Thai HAM-D Score=15, Thai PHQ-9 Score=16) was as follows:

“Older people want to know why they were born and for what reason. But it can be concluded that, those who live must lead their own way of life and live happily as a normal human being should receive and make it the most useful for humanity in this world.”

Upon carefully consideration, it was found that there were no distinct words in the word clouds produced from Q1 in Fig. 3 that indicated a feeling of depression. Furthermore, there were no words or phrases that indicated angry emotions, which are typically associated with male depression. More interestingly, out of three participants with comorbidity of bipolar and depression, two were female and one was male. Both female participants were misclassified by the model from Q1, suggesting that the model may not be able to capture the text transcripts produced from psychiatric symptoms other than depression. Further investigation is needed to better understand the underlying factors behind the misclassification of the model.

Reviewer#1, Concern # 3: Women start using care sooner than men when depressed. That means, men have more severe depression in clinics, because they use care too late. This has implications for this study.

Author response:

We are also mindful of the gender differences in terms of communicating and describing feelings of depression. Previously reported literature has shown that there is a gender difference in self-reporting symptom of depression in males and higher rates of suicide. Potential reasons behind this include gender differences in coping style, such as relative emotional inexpressiveness, non-help seeking behavior, as opposed to the reported feminine coping styles such as help-seeking and emotional expressiveness which can lead to major differences when filling out the questionnaire [59, 60]. There might be social expectations and stigma attached for males, associated with shame when expressing emotion or depression leading to an under-diagnosis and under-treatment of mild-moderate male depression and a higher suicide reporting in males [61]. Moreover, depressive symptoms in males usually including irritability, aggression, substance abuse, and risky behaviors [62]. Unfortunately, our collected dataset consisted of only 19 male data, which was insufficient to create a reliable model. This highlights the need for further research and questionnaire development to identify and screen for nuanced variations in depression reporting among different genders.

Reviewer#1, Concern # 4: This paper does not have genera international audience. Thus, I am not sure if this is a good fit for the PLOS One.

Author response:

The methodology of this study was similar to previously reported studies in the literature, such as the work of Senn et al. (2022) [30], which used clinical transcripts from an interview with twelve questions [58] from 189 participants with the PHQ-8 as a support diagnosis. The RoBERTa model achieved an F1-score of 0.92 for classifying depression with only one question: “What are you most proud of?”. When all the questions were used, the F1-score of the RoBERTa model decreased to 0.58 ± 0.22 (mean ± S.D.) [30]. Another work by Dai et al. (2021), also used the RoBERTa model for transfer learning with 500 discharge summaries from an electronic health record (EHR) system, and the F1-score was as high as 0.83 [29]. These experiments demonstrated that transfer learning towards a RoBERTa like model can also create a robust model even with a limited set of data. This work focused on leveraging the original Thai language, which is an uncommon domestic language, to create a multi-lingual support system with the XLM-RoBERTa model. The results revealed that the accuracy of the model was up to 90.00%, with only three questions. Although the model is based on the Thai language, the context of the speech responses to the questions was found to be generalizable to all practitioners, as shown in Fig. 2. Further research and investigation into cross-language studies of this paradigm could be beneficial.

The words produced from our LIME explanation in word cloud are in line with previous published articles. Ghosh et al. (2021) used a traditional features-based method with LSTM to train a model from approximately 4.2 M tweets and achieved 87.14% accuracy. Their study noted that words such as ‘depression’, ‘stress’, and ‘sad’, as well as attributes such as ‘work-pressure’, ‘divorce’, and ‘break-up’ tended to have a positive correlation to depression [26]. This result was in alignment with the dataset collected by Wani et al. (2022), which showed that the most frequent depressed words from Facebook comments were ‘fake’, ‘suicide’, ‘die’, and ‘sadness’, while tweets were ‘stress’, ‘life’, ‘failure’, and YouTube comments were ‘trauma’, ‘worst’, and ’death’ [25]. However, the other studies did not focus on the model’s clinical implications [23, 24, 27, 29–32].

-------------------------------------------------------------------------------------------------------------------------------------------

Reviewer#2, Concern # 1: The contributions of this work are not clear. I went through the abstract and introduction, I did not get the main contribution of this work I suggest the authors spend significant efforts to enhance the main work in this research.

Author response:

We modified the abstract and enhanced the main contributions of this research to be clearer. The modified abstract is shown below. In addition, we spent significant efforts to enhance contributions of this research in the introduction section described in the author response to Reviewer#2, Concern # 2.

Abstract

Depression is a serious mental health disorder that poses a major public health concern in Thailand and have a profound impact on individuals’ physical and mental health. In addition, the lack of number to mental health services and limited number of psychiatrists in Thailand make depression particularly challenging to diagnose and treat, leaving many individuals with the condition untreated. Recent studies have explored the use of natural language processing to enable access to the classification of depression, particularly with a trend toward transfer learning from pre-trained language model. In this study, we attempted to evaluate the effectiveness of using XLM-RoBERTa, a pre-trained multilingual language model supporting the Thai language, for the classification of depression from a limited set of text transcripts from speech responses. Twelve Thai depression assessment questions were developed to collect text transcripts of speech responses to be used with XLM-RoBERTa in transfer learning. The results of transfer learning with text transcription from speech responses of 80 participants (40 with depression and 40 normal control) showed that when only one question (Q1) of “How are you these days?” was used, the recall, precision, specificity, and accuracy were 82.5%, 84.65, 85.00, and 83.75%, respectively. When utilizing the first three questions from Thai depression assessment tasks (Q1 −Q3), the values increased to 87.50%, 92.11%, 92.50%, and 90.00%, respectively. The local interpretable model explanations were analyzed to determine which words contributed the most to the model’s word cloud visualization. Our findings were consistent with previously published literature and provide similar explanation for clinical settings. It was discovered that the classification model for individuals with depression relied heavily on negative terms such as ‘not,’ ‘sad,’, ‘mood’, ‘suicide’, ‘bad’, and ‘bore’ whereas normal control participants used neutral to positive terms such as ‘recently,’ ‘fine,’, ‘normally’, ‘work’, and ‘working’. The findings of the study suggest that screening for depression can be facilitated by eliciting just three questions from patients with depression, making the process more accessible and less time-consuming while reducing the already huge burden on healthcare workers.

Reviewer#2, Concern # 2: The research findings and contribution need to be stated clearly. As well as the obtained results in this paper.

Author response:

Our four main research contributions include:

1. Fine-tuning XLM-ROBERTABASE [45] to text transcripts elicited from spoken responses to twelve questions of Thai depression assessment tasks [44].

2. Assessing the effectiveness of Thai depression assessment tasks for detecting depression from text responses in speech and outline a set of questions that yield the most accurate results in clinical settings.

3. Explaining a black-box model from fine-tuned XLM-RoBERTa using weight explained from LIME explanations [41].

4. Visualizing weights from LIME explanations [41] in a word cloud format and explaining their relevance to clinical settings.

Reviewer#2, Concern # 3: The introduction, e.g., should lead the way throughout the paper. In addition, the benefits coming from this research should be made clearer in the introduction and throughout the paper. I believe, this section needs significant efforts to make things and contributions clearer and flow of the contents.

Author response: This research has important clinical implications, as it demonstrates that a highly accurate model for classifying depression can be produced using a limited set of high-quality text transcripts from speech in clinical settings. This is particularly noteworthy for uncommon domestic languages such as Thai, where data availability is often a significant issue. Additionally, the explanations provided by LIME through the black-box model can be interpreted and understood in a clinical context.

Reviewer#2, Concern # 4: The related work section is clear, and it needs a discussion for the studied papers.

Author response:

Introduction

In recent years, several studies have developed classification models from social media posts using vanilla fine-tuning of BERT and the model can successfully achieve up to 98% accuracy [26–28, 31, 32]. Instead of using social media posts, Dai et al. (2021) provided more examples of their work. They utilized transfer learning from four variants of the BERT model to identify depression in discharge summaries, and RoBERTa yielded the highest classification F1-score of 0.830 [29]. Another example was the work by Senn et al. (2022), which used transcripts of 12 clinical interview questions and a transfer learning approach with three variants of BERT and an ensemble of extra LSTM and attention layers. The results indicated that RoBERTa tended to perform better with basic architectures individually without ensemble. However, the best result was achieved using the BERT architecture with ensemble strategy of only one question, with an F1-score of 0.93. When using all the questionnaires, the F1-score dropped to 0.62 ± 0.12 (mean ± S.D.) [30].

Discussion

The methodology of this study was similar to previously reported studies in the literature, such as the work of Senn et al. (2022) [30], which used clinical transcripts from an interview with twelve questions [58] from 189 participants with the PHQ-8 as a support diagnosis. The RoBERTa model achieved an F1-score of 0.92 for classifying depression with only one question: “What are you most proud of?”. When all the questions were used, the F1-score of the RoBERTa model decreased to 0.58 ± 0.22 (mean ± S.D.) [30]. Another work by Dai et al. (2021), also used the RoBERTa model for transfer learning with 500 discharge summaries from an electronic health record (EHR) system, and the F1-score was as high as 0.83 [29]. These experiments demonstrated that transfer learning towards a RoBERTa like model can also create a robust model even with a limited set of data. This work focused on leveraging the original Thai language, which is an uncommon domestic language, to create a multi-lingual support system with the XLM-RoBERTa model. The results revealed that the accuracy of the model was up to 90.00%, with only three questions. Although the model is based on the Thai language, the context of the speech responses to the questions was found to be generalizable to all practitioners, as shown in Fig. 2. Further research and investigation into cross-language studies of this paradigm could be beneficial.

The words produced from our LIME explanation in word cloud are in line with previous published articles. Ghosh et al. (2021) used a traditional features-based method with LSTM to train a model from approximately 4.2 M tweets and achieved 87.14% accuracy. Their study noted that words such as ‘depression’, ‘stress’, and ‘sad’, as well as attributes such as ‘work-pressure’, ‘divorce’, and ‘break-up’ tended to have a positive correlation to depression [26]. This result was in alignment with the dataset collected by Wani et al. (2022), which showed that the most frequent depressed words from Facebook comments were ‘fake’, ‘suicide’, ‘die’, and ‘sadness’, while tweets were ‘stress’, ‘life’, ‘failure’, and YouTube comments were ‘trauma’, ‘worst’, and ’death’ [25]. However, the other studies did not focus on the model’s clinical implications [23, 24, 27, 29–32].

Although Ghosh et al. (2021) attempted to explain the model, their study focused on the weights directly which have some limitations. Since weights in a model are typically calculated from many input variables and interactions between variables, it can be difficult to understand the individual contributions of each factor to the overall result [41]. In comparison, our work utilized LIME explanations, which use various methods such as sampling, data perturbation, and feature importance to determine each feature’s importance. This comprehensive method helps to better understand the importance of each model feature, as well as how changes to the data can affect the model output. Additionally, the analysis can identify any potential bias in the model, which is not possible when simply exploring the model weight directly [41].

Reviewer#2, Concern # 5: The problem scenario is not clear.

Author response:

The goal of this study is to examine the potential of using a limited-set of text transcripts from speech responses to fine-tune a T-PTLM to create a promising classification model [39]. The main issues addressed in this research are (1) the scarcity of textual transcripts from speech collected in clinical settings, (2) how to develop a highly accurate model using T-PTLM with a limited dataset, and (3) how the results can be applied and trusted in clinical settings. Figure 1 illustrates the process from data collection, data preparation, and fine-tuning of the model. The details of this process are provided in the following subsections.

Reviewer#2, Concern # 6: The mathematical notations need more focus and enhancement.

Author response:

Model Explanation and Visualization

The model will later be explained by LIME explanation [41] to determine which words contribute the most for depression screening. The idea of LIME explanation was to randomly make an absence of words from text transcripts (perturb) and to create white-box model that resembles the black-box model (fine-tuned XLM-RoBERTa model) and minimizes the following equations:

{\\mathrm{arg\\thinspmin}}_{g\\in G}\\mathcal{L}\\left(f,g,\\pi_x\\right)+\\Omega\\left(g\\right) (5)

\\pi_x\\left(z\\right)=e^{-\\frac{D\\left(x,z\\right)^2}{\\sigma^2}} (6)

where x is the instance to be explained, z is the perturbed instance, f is the black-box model to be explained, G is the hypothesis space of white-box models, \\mathcal{L}\\left(f,g,\\pi_x\\right) is the deviation between the black-box model (f) and the white-box model (g), as weight measured on a perturbed version of the text transcripts x, represented by \\pi_x\\left(z\\right), \\Omega\\left(g\\right) is a regularization term that controls the complexity of the interpretable model (g), D\\left(x,z\\right) is a cosine similarity between x and z, and \\sigma is kernel width.

The goal of LIME is to find the white-box model (g) that is closest to the black-box model (f) and at the same time has the lowest complexity, as determined by \\Omega\\left(g\\right). The result of LIME is an explanation of the predictions of the black-box model (f) for the instance x. In this work, we implement a LIME analysis using the ELI5 library [54]. With ELI5, we can obtain an interpretable description of the model’s predictions along with the individual contributions of each word (\\pi_x\\left(z\\right)).

Since binary classification models produce four different weights in LIME explanations, namely the weight of the true positive class (depression predicted as depression), the weight of the false negative class (depression predicted as normal control), the weight of the false positive class (normal control predicted as depression), and the weight of the true negative class (normal control predicted as normal control), the relationship between words and the prediction of depression or normal control will be examined by summarizing the LIME weights from the true positive and true negative classes as a word cloud. To calculate this, we summed all the weights that had a positive effect on each class and normalized them to a scale between 0 and 1.

Reviewer#2, Concern # 7: The used figures in this paper are not clear enough.

Author response:

Fig 1. A flow diagram of the article’s process including a) data collection and speech elicitation, which are then transformed into text transcriptions; b) text pre-processing and normalization, followed by data preparation for cross-validation in transfer learning to the model for each question to compute evaluation matrices; c) training of the model for LIME explanation and utilization of the weights from the explanation for creating word clouds for both the normal control group and the depression group.

Fig 2. Samples of transcription of speech response from participants with original language (Thai) and English translation.

Fig 3. Word cloud of the responses to the question “How are you these days?” from six different groups: a) depressed females and males, b) depressed females, c) depressed males, d) normal control females and male, e) normal control females, and f) normal control males.

Reviewer#2, Concern # 8: I need to see the main procedure of the proposed method.

Author response:

Fig 1. A flow diagram of the article’s process including a) data collection and speech elicitation, which are then transformed into text transcriptions; b) text pre-processing and normalization, followed by data preparation for cross-validation in transfer learning to the model for each question to compute evaluation matrices; c) training of the model for LIME explanation and utilization of the weights from the explanation for creating word clouds for both the normal control group and the depression group.

Reviewer#2, Concern # 9: Some works are worthy of mentioning in this paper: Prairie dog optimization algorithm, Gazelle Optimization Algorithm: A novel nature-inspired metaheuristic optimizer, Dwarf Mongoose Optimization Algorithm, Ebola Optimization Search Algorithm: A new nature-inspired metaheuristic algorithm, Reptile Search Algorithm (RSA): A nature-inspired meta-heuristic optimizer, The arithmetic optimization algorithm, Aquila Optimizer: A novel meta-heuristic optimization algorithm.

Author action:

Author response:

For future direction, we plan to expand our study to investigate the relationship between normal control and depression in patients with other psychiatric disorders, such as bipolar disorder and psychotic disorder. This is because those with multiple psychological comorbidities might have variations in the presentation of depression, which will have implications for detection and diagnosis. Moreover, we plan to utilize data from various modalities to strengthen the accuracy and reliability of our models. To further improve our models, we are also aiming to employ optimization techniques [63,64]. Furthermore, we are hoping to use our models for clinical studies, which will allow us to gain better insights into the development of these psychiatric illnesses. Developing an application which can detect depression will help with insufficient health care workers’ problem, and this would also be beneficial to improve health care service accessibility.

Reviewer#2, Concern # 10: The evaluation metrics should be briefly described in the experimental section. Moreover, add further details on how simulations were conducted. Similarly, system and resource characteristics could be added to Tables for clarity.

Author response:

Evaluation Matrices

Our goal is to maximize recall (sensitivity), precision, and specificity of the model, as these matrices are widely used for clinical applications. Additionally, accuracy is reported to provide a holistic view of performance. Thus, in this work, the performance of a model is evaluated based on a variety of metrics, including recall, precision, specificity, and accuracy, using the following equations [53]:

Recall = TP/(TP+FN)

Precision = TP/(TP+FP)

Specificity = TN/(TN+FP)

Accuracy = (TP+TN)/(TP+TN+TP+FN)

where TP is true positive (depression predicted as depression), TN is true negative (normal control predicted as normal control), FP is false positive (normal control predicted as depression), and FN is false negative (depression predicted as normal control).

Implementation of Fine-tuning XLM-RoBERTa Model

The XLM-RoBERTa model was implemented for fine-tuning the data using the Keras library [50] on an NVIDIA RTX A6000 with 128 GB of RAM and 1 TB of SSD. The model was trained using a triangular learning rate policy [51] with a reduction factor of 2 and a reduction on plateau of 4. Other hyperparameters included a maximum of 50 epochs of training, a batch size of 8, and a learning rate of 1e-5. Furthermore, the model for each question was trained individually to examine the potential of the questions for the performance of the classification model.

Reviewer#2, Concern # 11: The conclusion section also needs significant revisions. It should briefly describe the study's findings and some more directions for further research. The authors should describe academic implications, major findings, shortcomings, and directions for future research in the conclusion section.

Author response:

When consider the parts of text transcriptions that contributes to which one of normal control or depression When considering conducting the LIME analysis, most of word cloud built from weight of the depression class has a preponderance towards a negative word than a neutral or positive one. This was in contrasted to normal controls, where most of word cloud consisted of neutral or positive words. Some of words are in conjunction with those found on the DSM-5 criteria for depression classification, such as the 1st criteria of “Depressed mood most of the day, nearly everyday.”, which can be described by the word ‘Sad’ in 2nd rank of word cloud or word ‘mood’ in 3rd rank of word cloud from Q1 in Fig. 3a), or the 2nd criteria of “Markedly diminished interest or pleasure in all, or almost all, activities most of the day, nearly every day.”, which can be described by the word ‘negate’ in 1st rank of word cloud. Most importantly, the last criteria of “Recurrent thoughts of death, recurrent suicidal ideation without a specific plan, or a suicide attempt or a specific plan for committing suicide.” can also described be by the word ‘suicide’ in 9th rank of word cloud [46].

Reviewer#2, Concern # 12: Can you compare the proposed method with other published method to solve the same case.

Author response:

The methodology of this study was similar to previously reported studies in the literature, such as the work of Senn et al. (2022) [30], which used clinical transcripts from an interview with twelve questions [58] from 189 participants with the PHQ-8 as a support diagnosis. The RoBERTa model achieved an F1-score of 0.92 for classifying depression with only one question: “What are you most proud of?”. When all the questions were used, the F1-score of the RoBERTa model decreased to 0.58 ± 0.22 (mean ± S.D.) [30]. Another work by Dai et al. (2021), also used the RoBERTa model for transfer learning with 500 discharge summaries from an electronic health record (EHR) system, and the F1-score was as high as 0.83 [29]. These experiments demonstrated that transfer learning towards a RoBERTa like model can also create a robust model even with a limited set of data. This work focused on leveraging the original Thai language, which is an uncommon domestic language, to create a multi-lingual support system with the XLM-RoBERTa model. The results revealed that the accuracy of the model was up to 90.00%, with only three questions. Although the model is based on the Thai language, the context of the speech responses to the questions was found to be generalizable to all practitioners, as shown in Fig. 2. Further research and investigation into cross-language studies of this paradigm could be beneficial.

The words produced from our LIME explanation in word cloud are in line with previous published articles. Ghosh et al. (2021) used a traditional features-based method with LSTM to train a model from approximately 4.2 M tweets and achieved 87.14% accuracy. Their study noted that words such as ‘depression’, ‘stress’, and ‘sad’, as well as attributes such as ‘work-pressure’, ‘divorce’, and ‘break-up’ tended to have a positive correlation to depression [26]. This result was in alignment with the dataset collected by Wani et al. (2022), which showed that the most frequent depressed words from Facebook comments were ‘fake’, ‘suicide’, ‘die’, and ‘sadness’, while tweets were ‘stress’, ‘life’, ‘failure’, and YouTube comments were ‘trauma’, ‘worst’, and ’death’ [25]. However, the other studies did not focus on the model’s clinical implications [23, 24, 27, 29–32].

Although Ghosh et al. (2021) attempted to explain the model, their study focused on the weights directly which have some limitations. Since weights in a model are typically calculated from many input variables and interactions between variables, it can be difficult to understand the individual contributions of each factor to the overall result [41]. In comparison, our work utilized LIME explanations, which use various methods such as sampling, data perturbation, and feature importance to determine each feature’s importance. This comprehensive method helps to better understand the importance of each model feature, as well as how changes to the data can affect the model output. Additionally, the analysis can identify any potential bias in the model, which is not possible when simply exploring the model weight directly [41].

Attachments

Attachment

Submitted filename: Response to Reviewers.pdf