https://orcid.org/0000-0002-9314-5828
Figures
Abstract
Digital self-guided mobile health [mHealth] applications are cost-effective, accessible, and well-suited to improve mental health at scale. This randomized controlled trial [RCT] evaluated the efficacy of a recently developed mHealth programme based on cognitive-behavioral therapy [CBT] principles in improving worry and anxiety. We also examined psychological mindedness [PM] as a mediator by which app engagement is thought to improve outcomes. The Intervention group completed a 2-week “Anxiety and Worry” programme with daily CBT-informed activities, while the active waitlist-control completed a matched 2-week mHealth programme on procrastination. Participants filled out the Generalized Anxiety Disorder [GAD-7], Patient Health Questionnaire [PHQ-9], and Psychological Mindedness Scale [PMS] at baseline, post-intervention, and 2-week follow-up. App engagement was measured at post-intervention only. Contrary to prediction, the Intervention group did not perform better than the Active Control group; both groups showed significant improvements on anxiety and depressive symptoms from baseline to follow-up. From post-intervention to follow-up, only the Intervention group showed further improvements for anxiety symptoms. Higher engagement with the mHealth app predicted lower anxiety and depressive symptoms at follow-up, and this relationship was fully mediated by psychological mindedness. This study provides evidence that [a] engaging in a CBT mHealth programme can reduce anxiety and worry, and [b] Psychological mindedness is a potential pathway by which engaging with a mHealth app improves anxiety and depressive symptoms. While overall effect sizes were small, at the population level, these can make significant contributions to public mental health.
Author summary
Increasing burden of anxiety amongst young adults has made widely accessible mobile health applications a promising tool in improving anxiety symptoms at scale. We conducted a randomized controlled trial (N = 323) to examine the effectiveness of a brief, publicly available mobile health application (Intellect’s “Anxiety and Worry” programme) in reducing anxiety and worry symptoms among young adults. Participants who received the intervention showed significant reduction in anxiety and depressive symptoms, however, these effects did not significantly differ from the active control group. At post-intervention, only the Intervention group continued to experience improvements in anxiety level. We also found that higher app engagement with the mHealth app predicted better anxiety and depressive symptom outcomes, and this relationship was fully mediated by psychological mindedness. Future work would benefit from inclusion of a waitlist control group, a larger sample size, and identification of alternative mediators.
Citation: Kosasih FR, Yee VTS, Toh SHY, Sündermann O (2023) Efficacy of Intellect’s self-guided anxiety and worry mobile health programme: A randomized controlled trial with an active control and a 2-week follow-up. PLOS Digit Health 2(5): e0000095. https://doi.org/10.1371/journal.pdig.0000095
Editor: Laura M. König, University of Bayreuth: Universitat Bayreuth, GERMANY
Received: August 8, 2022; Accepted: March 24, 2023; Published: May 24, 2023
Copyright: © 2023 Kosasih et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: The study was partly funded by Intellect Pte Ltd. The study design, data management, interpretation, analysis, and reporting and the decision to publish the study are entirely independent of Intellect Pte Ltd. OS had a research collaboration with Intellect Pte Ltd at the time of the data collection and has since joined Intellect Pte Ltd as their clinical director.
Competing interests: We have read the journal’s policy and the authors of this manuscript have the following competing interests: At the time of submission OS was employed by Intellect Pte Ltd as clinical director, FRK and SHYT were employed as research associates. OS also owns shares of Intellect Pte Ltd.
Introduction
Worry is a cognitive-based construct, commonly defined as an uncontrollable chain of thoughts and images about future events that are negatively affect-laden [1,2]. Anxiety, on the other hand, is a multi-encompassing construct defined as an emotion characterized by somatic, cognitive, and behavioral responses [3,4]. Previous research has demonstrated a directional relationship between worry producing anxiety and confirmed the high correlational nature of the two constructs [2,4]. Indeed, worry has been attributed as the cognitive component of anxiety, where persistent and excessive worry is the central diagnostic criteria for generalized anxiety disorder (GAD) [2,5]. Anxiety often co-occurs with depression, a clinical disorder characterized by persistent mood disturbances, negative self-concept, and anhedonia or the inability to experience positive affect [6–8]. Although anxiety and depression have overlapping symptoms, anxiety and depression have distinct temporality (future vs. past oriented) and characteristic (hyperarousal vs. anhedonia) [2,5,6,9]. Anxiety has been associated with an increased risk for cardiovascular illness [9], mortality and suicide [10,11], as well as reduced quality of life [9]
University students are particularly vulnerable to anxiety and depression, as they are exposed to multiple stressors such as major life events (e.g. transition to university life) and societal trends (e.g. increasing financial burden) [12]. A recent meta-analysis of 19 studies involving undergraduates reported a 29% prevalence rate of anxiety and 37% prevalence rate of depression during the pandemic [13]. In addition to its high prevalence rate, anxiety and depressive symptoms are the most common problems for which university students seek help [10,14]. While effective psychological interventions exist for anxiety and depression, only few are utilizing these due to multiple barriers, such as high treatment costs, limited accessibility, lack of perceived need for help, and negative stigma [15–18]. Unfortunately, anxiety and depression usually take a chronic course, and are unlikely to resolve without appropriate interventions [19]. Therefore, there is a dire need for an effective, cost-efficient intervention on anxiety and depression.
One solution to help a large number of people is to utilize widely-accessible, secure yet inexpensive mobile health (mHealth) applications [20–22]. Emerging research has found evidence that engaging with mHealth apps can improve anxiety and depressive symptoms across both clinical [20] and non-clinical populations [23–26]. To date, several meta-analyses have highlighted promising evidence on the success of mHealth apps in decreasing anxiety and depressive symptoms using randomized controlled trials (RCT). As anxiety and depression share overlapping symptoms, some intervention components for anxiety are applicable to depression, such as cognitive restructuring and behavioral activation [27,28]. This is supported by studies of CBT-based mHealth interventions that reported significant improvements on both anxiety and depressive symptoms [29]. Although effect sizes found were small, recent meta-analyses have also highlighted promising evidence on the effectiveness of mHealth interventions on anxiety (g = 0.30–0.38) and depressive symptoms (g = 0.28–0.38) [21,30,31]. However, comparisons against Active Control often yield significantly weaker and inconsistent results [21,30,32]. Nevertheless, scaled at population level, small effects can meaningfully impact public mental health.
Despite promising evidence on the efficacy of mHealth apps on improving anxiety and depressive symptoms, especially for mild symptom severity, there is still a limited number of applications with reported effectiveness [33,34], quality [35], and evidence-based components [36]. Wang et al. [34]’s systematic review found that only 14 out of over 100 studies had reported clinically validated evidence on its effectiveness. In a review of 27 popular apps by Wasil et al. [36], most apps contained just 3 evidence-based components and incorporated limited core treatment techniques typically present in standard anxiety and depression treatment protocols, where cognitive restructuring made up just 12% and 31% of the applications, respectively. Therefore, there is a strong need for more empirical studies to gain a better perspective of the true value of mHealth apps on anxiety as well as depression, paying extra attention to the content used within the application and the type of control groups used in the studies.
While more effort has been made to examine the efficacy of publicly available mHealth apps, limited studies have investigated the underlying mechanisms that enable such impact. To our knowledge, only 5 studies evaluated such mechanisms [23,37–40], and only 3 of these studies utilized mHealth apps incorporating multiple Cognitive Behavior Therapy (CBT) components. Using MoodMission, MoodPrism, and MoodKit, Bakker and colleagues [23,37,38] demonstrated the mediating effect of coping self-efficacy in the relationship between app engagement ratings and anxiety, depressive symptoms, and mental wellbeing outcomes. Bakker and Richard [37] also found a partially mediated effect of emotional self awareness (ESA) in the aforementioned relationship. Identifying additional mechanisms that facilitated improvements in mental health outcomes can further inform mHealth apps developers to create apps that utilize multiple mediators to enhance mental health outcomes [37]. One such potential mediator is psychological mindedness (PM), defined by Denollet and Nyklíček [41] as “the intrinsic motivation to be in touch with one’s inner feelings and thoughts by monitoring and analyzing them in an adaptive way.” An important factor in the development of emotion regulation, PM has previously been linked to lower psychological distress and better coping abilities [42–49]. Previous research has also hinted at the prospect of PM as a relevant skill in psychotherapy, albeit inconsistent findings on its effect on treatment outcomes [44,50–52]. As psychotherapies encourage individuals to monitor, evaluate, and express their cognitions, emotions, and behaviors, it is likely that participating in treatments increases an individual’s PM [53–55]. Nyklíček et al. [54] demonstrated that changes in PM scores over the course of an intervention were associated with larger decreases in psychological symptoms scores, including anxiety, depressive symptoms, and sleeping problems in a clinical sample. Additionally, previous research has indicated that metacognitive skills such as PM are systematically developed during the transition from adolescence to adulthood and are considered key mechanisms required to transform behaviors during cognitive-behavior-based and mindfulness-based treatments [56,57]. Altogether, this suggests that PM might act as a mediator that intermediately affects treatment outcomes from the use of mHealth apps.
Objectives
This study aimed to evaluate the efficacy of a recently developed mHealth programme within the Intellect mHealth application called (“Anxiety and Worry”) compared to the Active Control group (“Procrastination” programme) among young adults in a randomized controlled trial (RCT) with a 2-week follow-up. We hypothesized that participants in the Intervention group would experience significant decreases in anxiety (primary outcome) and depressive symptoms (secondary outcome) compared to the Active Control group. Based on Bakker and colleagues’ [23,37,38] prior work, we also sought to examine whether PM mediates the hypothesized relationship between engagement with the mHealth programmes and improvements in anxiety and depressive symptoms.
Materials and methods
Design
This randomized controlled trial (RCT) is a 2 x 3 mixed factorial experimental design, with condition (Intervention vs. Active Control) as the between-groups factor and time of assessment (pre- intervention vs. post-intervention vs. 2-week follow-up) as the within-groups factor. The study was registered with ClinicalTrials.gov (NCT04911803; see S1 Protocol) and it was approved by the National University of Singapore (NUS)’s Institutional Review Board / NUS-IRB (Reference Code: 2021–266). The study was conducted entirely remotely between June 2021 and December 2021.
Participants
Participants were recruited through convenience sampling via the NUS’ internal Research Recruitment portal and through the NUS Department of Psychology’s Research Participation (RP) pool. Recruitment took place between June 2021 and December 2021, and potential participants were screened for eligibility via a Qualtrics survey. Inclusion criteria included (1) being able to read and understand English, (2) aged 18 years or above, (3) has not participated in similar mHealth studies. Participants were ineligible if they met any of the following exclusion criteria: (1) 18 years or below, (2) inability to read and understand English, (3) participated in other mHealth studies, (4) lack of mobile device or access to the Apple or Google Play Store, and (5) lack of access to the Internet. Participants who did not complete the baseline questionnaire were also excluded. Only participants who met the inclusion criteria were awarded either $15 reimbursement or 3 course credits for their full participation.
Materials
Intellect is a publicly available mobile health application aimed at providing mental wellbeing support using self-guided resources. Recently, these self-guided resources have been validated in previous randomized-controlled trials [58,59]. All programmes within the application were developed and have been reviewed by Intellect’s Scientific Advisory Board consisting of clinical psychologists and cognitive behavioral therapists. For the purposes of this study, a version of the Intellect Application is used, where only the “Anxiety and Worry” or “Procrastination” programmes are available, depending on the participant’s assigned condition.
“Anxiety and Worry” Programme (Intervention Condition).
This is a 14-day mHealth programme within the Intellect Application that provides psychoeducation on the negative core beliefs and/or thought patterns associated with anxiety and worry, as well as effective coping mechanisms and problem-solving techniques (see Fig 1). Each psychoeducation module is followed by short writing exercises, and a few situational-task questions (see Table 1 for a programme overview). The modules and tasks are divided into short sessions, such that participants can engage in a <5 minutes session a day. This module structure allows participants to actively practice utilizing the techniques they have learned.
The Intellect “Anxiety and Worry” Programme is a free self-guided programme within the Intellect Application.
“Procrastination” Programme (Active Control Condition).
This is a 14-day mHealth programme within the Intellect Application that provides psychoeducation on the procrastination cycle, the mental barriers that arise due to procrastination, and ways to tolerate and overcome the discomfort. The structure and duration of the Procrastination Programme (PP) is similar to that of the Anxiety and Worry programme (see Table 2 for programme overview).
Procedures
Participants signed up for the study through the University’s recruitment sites, where they were redirected to a secured Qualtrics survey link. Participants who provided digital consent to participate in the study (N = 492) were prompted to complete the baseline questionnaires, where outcome variables (PHQ-9, GAD-7, PM) were measured. Only participants who were able to download the application, have not participated in similar mHealth app studies, and completed the baseline measures (N = 323) were then automatically randomized using the randomizer function in Qualtrics to the Intervention (“Anxiety and Worry” Programme; N = 160) or the Active Control (“Procrastination” Programme; N = 163) condition. As the study is a single-blind trial, participants were broadly informed that the main purpose of the study was to evaluate the benefits of using a mHealth programme to improve wellbeing [58]. In addition, participants were not informed about the specific details of the study and their assigned condition beforehand in order to lower expectation biases [60]. Next, participants were guided to download the Intellect mHealth application and were provided a unique code to register and access a research version of the app, where only their assigned programme was available. This was done to ensure that any treatment effects observed were due to the programme, and not other features available in the application. Both programmes span over 14 days, with one session to be completed daily (i.e. <5 minutes). After completion of their allocated mHealth programmes, participants were contacted to complete the same outcome variables (PHQ-9, GAD-7, PM) and the app engagement scale (AES). The same measures (but without the AES) were administered 14 days after the completion, and participants were also debriefed about the main purposes of the study via email. A 14 day follow-up was chosen to mimic previous studies [61].
Measures
Primary Outcome Measures.
Generalized Anxiety Disorder Scale (GAD-7). Generalized Anxiety Disorder- 7 Scale (GAD-7) is a 7-item self-report instrument that measures anxiety on a continuum [62] and is widely used in research in both clinical and nonclinical populations [63]. Example items include “feeling nervous, anxious, or on edge” and “not being able to stop or control worrying.” Items are scored on a 4-point likert scale, ranging from Not At All (0) to Nearly Every Day (3). Scores of ≥ 10 indicate moderate levels of anxiety [64]. GAD-7 has an excellent internal consistency with Cronbach’s α = .84 to α = .90 in this study.
Secondary Outcome Measures.
Patient Health Questionnaire (PHQ-9). Patient Health Questionnaire (PHQ-9) is a 9-item self-report instrument that measures depressive symptomatology. Example items include “little interest or pleasure in doing things” and “feeling tired or having little energy.” Items are scored on a 4-point Likert scale, ranging from Not at all (0) to Nearly Every Day (3), where higher scores indicate more severe depressive symptoms. Scores of ≥ 10 indicate moderate severity of depressive symptoms [65]. PHQ-9 has a good internal consistency with Cronbach’s α = .85 to α = .86 in this study.
Psychological Mindedness Scale (PMS). Psychological Mindedness Scale (PMS) is a 45-item self-report instrument that measures an individual’s ability to be reflective about interpersonal relationships, psychological processes and meanings across both intellectual and emotional dimensions. Example items include “I often find myself thinking about what made me act in a certain way” and “I am sensitive to the changes in my own feelings.” Items are scored on a 4-point Likert scale ranging from Strongly Agree (4) to Strongly Disagree (1). Consistent with previous research where Cronbach’s α = .86 to α = .87 [44], the reliability of PMS in this study was acceptable with Cronbach’s α = .77 to α = .79.
App Engagement Scale (AES). App Engagement Scale (AES) is a 7-item questionnaire developed by Rickard et al. [37,66] based on the Mobile Application Rating Scale’s (MARS) items and themes, such as engagement, functionality, aesthetics, and information [67]. Example items include “overall I was satisfied with the app” and “it had about the right amount of information.” All 7 items are scored on a 5-point Likert scale ranging from Strongly Disagree (1) to Strongly Agree (5). The AES displayed good internal reliability with Cronbach’s α = .83 in this study.
Statistical Analysis
Power analysis.
Power was calculated with G*Power v.3.1 [68]. Using a small effect size from similar mHealth RCT’s [23,69], setting cronbach alpha at .05, and the power at .90, N = 88 participants per group was required. Accounting for high dropout rates observed in previous studies [23,70], we aimed to recruit a minimum of N = 200 participants.
Data cleaning.
Data cleaning included examining the data for incompleteness. For missing data, t-tests and chi-square tests were used to compare demographics and outcome measures between complete cases with and without missing data to verify whether or not data were missing completely at random (MCAR). All missing data was then addressed using the last observation carry forward imputation method [71], such that missing data from the post-intervention and follow-up assessment were replaced with data from the last completed questionnaire. The final dataset can be found in the supplementary information of this study (i.e., S1 Data).
All data were analyzed according to the intention-to-treat principle using IBM SPSS v.25. Preliminary data analyses included normality testing and checking for univariate outliers. 13 outliers (i.e. +/- 3SD from the mean) were excluded. Baseline group differences were examined with the independent samples t-tests and chi-square test with a significance level of p < .05
Main analysis.
A 2x3 Mixed Analysis of Variance (Mixed ANOVA) with time (baseline (T1), post-intervention (T2), and follow-up (T3)) as the within subject factor and condition (Anxiety, Procrastination) as the between subject factor was conducted to assess the difference in anxiety (GAD-7) and depressive (PHQ-9) symptom scores. As there were no significant demographic differences (i.e. age, gender) between conditions, no covariates were controlled for in our analysis. P levels were corrected for non-sphericity using the Greenhouse-Geisser epsilon when necessary. Significant time by condition interaction effects were further examined using Bonferroni adjusted contrast analyses to compare between group differences on the outcome measures across each time point [72]. A significant interaction effect indicates group differences in improvement over time. Next, a series of mediation analyses were performed using Hayes’ PROCESS macro plugin Model 4 with confidence intervals set at 95% using bootstrapping procedures, with 5000 samples (95% confidence interval) [73]. An effect was considered to be significant if zero was not contained in the confidence interval. Using Zhao et al. [74] and Rucker et al. [75]’s recommendation, mediation was explored even in the absence of a significant total or direct effect. Two mediated regression models were used to examine the relationship between App Engagement Ratings and outcome variables, including Anxiety symptoms (GAD-7) and Depressive symptoms (PHQ-9). The mediating variable used in both models was PM at T2 measured by PMS and the dependent variable used was the outcome variables at T3. No covariates were included in this model. All Beta (β) statistics reported in the regressions are standardized effect sizes.
The main analyses were performed on the whole ITT sample (n = 299), while the mediation analyses were performed on the ITT sample who completed AES (n = 196). Eta squared partial (η2partial ≈ .01: small; η2partial ≈ .06: medium; η2partial ≈ .14: large) was the effect size reported for all findings [76]. The significance level was set as p < .05.
Results
All results described complied with the CONSORT guidelines for reporting RCTs (see S1 Checklist).
Baseline Evaluation
No significant baseline group differences on baseline outcome measures, gender, and age, were observed between the intervention and control conditions (p>.05). There were no significant differences in application engagement between both conditions (p>.05), indicating that participants were equally engaged throughout their respective programmes. The high mean scores on application engagement seen across both conditions (Intervention: M = 27.29, SD = 3.81; Active Control: M = 27.04, SD = 3.66) may infer that participants generally found our application to be acceptable [38,58]. At baseline, the sample was mildly anxious (M = 7.25, SD = 4.95) and depressed (M = 7.51, SD = 5.33), where 92 participants (30.77%) had a GAD-7 score of 10 or over, indicating moderate levels of anxiety and 90 participants (30.10%) had a PHQ-9 score of 10 or over, indicating moderate severity of depressive symptoms. There were no significant baseline group differences observed for the number of participants that experienced moderate levels of anxiety and depressive symptoms. There were also no significant differences in application engagement between these participants in both conditions [p>.05]. Sample characteristics for the final sample of 299 participants (Mage = 22.05 years; SDage = 4.06 years, range: 18 to 54 years) are displayed in Table 4.
Dropout Analysis
In total, 323 participants were randomized to either the Intervention or Active Control condition (See Fig 2). Of these, 11 participants withdrew prior to the post-intervention assessment. Both groups also experienced attrition during the two-week intervention period (35.48% of the Intervention condition and 34.49% of the Active Control condition). Additionally, 7.74% of the Intervention condition and 12.10% of the Active Control condition were lost to follow-up. Participants who dropped out during the study period did not significantly differ in age, gender, anxiety symptoms, depressive symptoms, application engagement and psychological mindedness scores compared to participants who did not drop out (all ps>.05).
Adherence to the study was defined as completing the assigned programme (when applicable) and questionnaires. By this definition, 65.06% (203/312) adhered to the study at T2 and 55.12% (172/312) adhered to the study at every time point. There are no significant differences in adherence at each timepoint between groups (p>.05).
Test of Group Effects
The mean score and standard deviation for each outcome measure by condition and time point are shown in Table 5. Mixed ANOVA with anxiety (W(2) = .890, p < .001) and depressive symptoms (W(2) = .794, p < .001) indicated that the assumption of sphericity had been violated, therefore the Greenhouse-Geisser corrected values were reported. Significant main effects of time were found for both outcome measures, where both conditions improved in anxiety and depressive symptoms over time (Anxiety symptoms: F(1.802,535.19) = 12.306, p < .001, ηp2 = .040; Depressive symptoms: F(1.658,492.365) = 16.984, p < .001, ηp2 = .054). Results from subsequent pairwise comparisons revealed a significant mean difference from baseline (Anxiety symptoms: M = 7.553, SD = .404; Depressive symptoms: M = 7.573, SD = .436) to post-intervention (Anxiety symptoms: M = 7.140, SD = .345, p < .001, 95% C.I. = [.232, 1.152]; Depressive symptoms: M = 6.807, SD = .408, p < .001, 95% C.I. = [.354, 1.291]) and from baseline to follow-up (Anxiety symptoms: M = 6.613, SD = .357, p < .001, 95% C.I. = [.401, 1.453]; Depressive symptoms: M = 6.453, SD = .409, p < .001, 95% C.I. = [.523, 1.563]), where the Active Control condition reported lower mean on anxiety and depressive symptoms in each instance.
While there were no significant effects of time by condition for anxiety and depressive symptoms (Anxiety symptoms: F(1.802,535.194) = 2.624, p = .079, ηp2 = .009; Depressive symptoms: F(1.658,492.365) = .760, p = .445, ηp2 = .003), results from post-hoc pairwise comparisons revealed a significant decrease in anxiety symptoms from post-intervention to follow-up for the Intervention condition (p = .033, 95% C.I. = [.031, 1.023]) and a non-significant increase in anxiety symptoms for the Active Control condition (p = .585, 95% C.I. = [-.766, .229]). Although both conditions experienced a significant decrease in anxiety symptoms from baseline to follow-up (Intervention: p = .002, 95% C.I. = [.279, 1.601]; Active Control: p = .033, 95% C.I. = [.041, 1.368]), only the Active Control condition reported a significant decrease from baseline to post-intervention (p = .001, 95% C.I. = [.324, 1.622]). Altogether, this indicated that the Intervention condition experienced significant improvements at follow-up driven by continued improvements from post-intervention to follow-up, while the Active Control condition experienced significant improvement at post-intervention with no significant improvements from post-intervention to follow-up (See Fig 3). For depressive symptoms, post-hoc pairwise comparisons revealed a significant decrease from baseline to post-intervention (Intervention: p = .041, 95% C.I. = [.023, 1.510]; Active Control: p = .002, 95% C.I. = [.342, 1.3833]) and from baseline to follow-up (Intervention: p = .005, 95% C.I. = [.230, 1.703]; Active Control: p = .033, 95% C.I. = [.041, 1.368]) for both conditions. This indicated that both conditions did not significantly differ in the improvement trajectory of depressive symptoms.
Error bars represent 95% CIs.
There were no significant main effects of condition observed for both outcome measures (p>.05). The full results of the analysis are presented in Table 5.
Fig 4 shows the full results of the 2 mediation analyses performed. A significant total effect between AES and Anxiety scores was observed, β = -.184, t196 = -2.268, p = .025. However, there were no significant total effects of AES on Depressive symptoms β = -.178, t196 = -1.863, p = .064. No direct effects were significant. Although results revealed an absence of significant total effects for Depressive symptoms, mediation analyses were carried out for both outcome measures as per Zhao et al. [73]. Mediation analyses revealed significant standardized indirect effects of AES through PMS for Anxiety symptoms, β = -.047, 95% CI [-.106, -.007] (See Fig 4A), and Depressive symptoms, β = -.049, 95% CI [-.105, -.007] (See Fig 4B). Overall, these models accounted for 4.72% and 1.76% of the variance (R2) in anxiety and depressive symptoms reduction, respectively. Results indicated the presence of an indirect-only mediation of PMS at T2 for both anxiety and depressive symptoms at T3. This suggests that AES causes a decrease in anxiety and depressive symptoms indirectly by increasing PMS.
Mediated regression models using AES as the predictor and (A) Anxiety symptoms and (B) Depressive symptoms as the outcome. Whole sample (N = 196) mediated regression models using App Engagement ratings (AES) as the predictor and (A) Anxiety symptoms (GAD-7), (B) Depressive symptoms (PHQ-9) scores as the outcome, where T2 indicates post-intervention and T3 indicates 2-week follow-up. Note: bolded coefficients indicate significance, p < .05.
Discussion
This study sought to evaluate the efficacy of Intellect’s mHealth programme “Anxiety and Worry” in improving anxiety and depressive symptoms in college students. The tools implemented in the intervention are based on cognitive-behavior therapy principles, and include cognitive, behavioral, and writing activities that may increase an individual’s ability to self-reflect and introspect. Thus, we also examined PM as mediators of app engagement on anxiety and depressive symptoms.
Efficacy of the Anxiety and Worry Programme
Our first hypothesis was not fully supported. Even though the “Anxiety and Worry” programme is based on CBT principles used in anxiety interventions, participants in the “Anxiety and Worry” programme did not experience significantly greater improvements in anxiety and depressive symptoms compared to participants in the “Procrastination” programme. Rather, participants in both conditions displayed a significant decrease in levels of anxiety and depressive symptoms from baseline to follow-up. While there was no significant Time by Condition interaction on depressive symptoms, the present study showed a marginally significant Time by Condition interaction on anxiety scores, with participants in the Active Control condition reporting lower mean scores at follow-up. Further examinations demonstrated that anxiety symptoms significantly decreased after the completion of the “Anxiety and Worry” programme, i.e. from post-intervention to follow-up, whereas no such continued improvement was observed in the Active Control condition. The result from the analysis is in line with other follow-up studies that reported continued improvement of anxiety symptoms among individuals in the intervention group after the intervention ceased [77,78]. One possible explanation for this observation is Rachman’s (2017) notion that the full benefits of CBT-based intervention on anxiety will only manifest post-intervention, after participants fully understood the skills taught and practiced applying them in their day-to-day lives [79].
Altogether, the results did not demonstrate the superiority of the “Anxiety and Worry” programme to the “Procrastination” programme. However, when compared with previous studies that evaluate the efficacy of mHealth interventions against Active Control on anxiety and depressive symptoms among the general population, non-significant results were mostly reported [61,80]. SuperBetter (SB), a game-like mobile-based self-help programme, required participants aged 18 years or older to engage with the application at least 10 minutes per day for a month [61]. Roepke et al. [61] found that the intervention condition (SB with CBT and positive psychotherapy content) was more effective in reducing depressive and anxiety symptoms at post-intervention and follow-up, but not the Active Control condition (i.e. General SB application). Since this study was conducted during the COVID-19 pandemic (between June and December 2021) which may explain the increased anxiety and depressive symptoms at baseline [81–83], it is also possible that the higher baseline symptoms hindered the efficacy of the “Anxiety and Worry” intervention. For instance, McKee et al. (2007) found that individuals with a higher tendency to experience negative affectivity symptoms (i.e., anxiety and depression) were less able to engage meaningfully with the intervention and suffered from less than optimal outcomes [84]. Nevertheless, even though the “Anxiety and Worry” programme was not superior to the “Procrastination” programme in terms of lowering anxiety and depressive symptoms, it is encouraging that participants in both conditions reported a significant decline in anxiety and depressive symptoms at follow-up. While we cannot rule out the effect of third variables, it is plausible that both mHealth programmes are effective in reducing anxiety and depressive symptoms.
One explanation for the unexpectedly good outcomes in the active control group is that the procrastination mHealth programme is also based on the cognitive-behavior model and thus contains multiple techniques that may be beneficial for anxiety and depressive symptoms. For example, one module in the programme is titled “Overcoming the barrier” and contains discomfort intolerance (DI) related information and relaxation exercises targeting participants’ abilities to overcome difficult scenarios. The module aims to equip participants with the necessary skills to overcome similar scenarios in the future by teaching them to be mindful and walking them through how to challenge procrastination. Previous studies have recognized DI as central to anxiety pathology [85] In particular, discomfort avoidance, a component of DI, is primarily responsible for anxious responding [85,86]. Therefore, it is likely that this module helped participants to improve DI and thus decreased anxiety. Moreover, this module incorporates exposure tasks related to procrastination. Peris et al. [87] previously highlighted that exposure tasks in CBT treatment were often followed by significant improvement in the rate of progress in treatment, including the primary treatment outcome (i.e. procrastination) and secondary treatment outcome (i.e. anxiety). Additionally, the exposure tasks included in this module include reducing avoidance, such as encouraging participants to complete the task they previously put off. Such tasks involve principles of behavioral activation (BA), an established technique recognized as important in depression interventions [88,89]. Taken together, this suggests practical implications for future developments of mHealth programmes, in terms of including modules that address discomfort intolerance related to anxious responding and incorporating elements of exposure tasks and behavioral activation that likely help improve anxiety and depressive symptoms.
Furthermore, helping students with procrastination may have inadvertently improved anxiety and depressive symptoms more than we had anticipated. Previous studies have indicated that 50–70% of college students showed frequent procrastination behavior [90] and linked procrastination to negative well-being outcomes, including higher anxiety and depressive symptoms [91]. Indeed, a mediating effect of rumination between anxiety and procrastination as well as depression and procrastination was previously observed among students [92]. Constantin et al. [92] indicated that students experiencing anxiety and depressive symptoms might more readily procrastinate in order to manage their current distress or regulate their negative mood states. Thus, it is highly plausible that interventions targeting procrastination may improve one’s ability to manage psychological distress, especially in the student population. This is supported by previous meta-analysis that found a small effect size of procrastination intervention on anxiety and depressive symptoms among a mix of students and the general population [93]. Therefore, using PP as an Active Control may have exerted unexpected benefits on anxiety and depressive symptoms.
Nevertheless, it is encouraging that participants in the intervention condition continued to experience improvements in anxiety from post-intervention (T2) to follow-up (T3), while no such benefit was observed in the Active Control condition. This suggests a potential delayed effect of the “Anxiety and Worry” programme, where full treatment benefits were only manifested post-intervention. One explanation for the apparent delayed effect observed is the major inclusion of writing activities (>80% of activities) within the Anxiety programme. The writing activities are framed as an interactive journal (IJ) using language that elicits change strategies, thereby mimicking motivational interviewing (MI) in writing [94]. The use of MI enhances a “client-as-expert” stance, which indirectly confers a sense of autonomy and induces a belief in clients to rely on their ability to address the situation they are in. Existing MI literature has attributed this mechanism as the source of delay or the sleeper effect at post-intervention [95,96]. A relevant study by Westra et al. [97] found a steeper rate of worry decline at post-intervention for MI-CBT groups. Thus, using the same principles, it is possible that the MI-based IJ components in the Anxiety programme instills a “user-as-expert” stance, which reinforce the participants’ confidence in applying the techniques learnt to address anxiety and worry, ultimately improving their anxiety symptoms post-intervention. This suggests practical implications for future developments of anxiety related mHealth application, in terms of integrating MI-based journaling elements which could aid in long-term effective reductions of anxiety.
Psychological Mindedness as a Mediator
To our knowledge, this is the first study that investigated Psychological Mindedness as a mediator in the context of mHealth apps. Our second hypothesis is fully supported. The results of our mediation analyses demonstrated that while app engagement did not directly predict reductions in anxiety and depressive symptoms, a mediated pathway was identified via PMS. While effect sizes are small, results of the current study suggest that engaging with the “Anxiety and Worry” and “Procrastination” programmes improved one’s PM, which subsequently decreased one’s anxiety and depressive symptoms. These findings extend previous studies by Bakker’s group [23,37,38] that have observed the mediating role of coping self-efficacy (CSE) on anxiety and depressive symptoms. Previous studies have hinted upon the potential relationship between PM and CSE, particularly problem-focused or adaptive coping [54]. Pang et al. [46] have also established that the insight component of PM significantly mediates the relationship between dysfunctional coping and depressive symptoms. Hence, it is unsurprising to find that PM plays a mediating role in the relationship between App Engagement and anxiety and depressive symptoms.
The present study contributes to the growing literature of PM and CBT-based treatment. Previously, Kishon et. al. [98] established an association between PM and depressive symptoms across timepoints and demonstrated that individuals who are psychologically minded but do not have access to their feelings benefited from CBT treatment. Kishon et al. [98] also suggested the notion that techniques used in CBT-based treatment (e.g. monitoring thoughts, feelings, and behaviors) likely targets the same cognitive analytical abilities that underlie PM and that PM likely assists individuals on the process of emotion regulation during treatments on anxiety and depression. Besides that, previous studies have found PM to be associated with higher commitment and stronger involvement in therapy [37,89,90]. Thus, it is possible that an increase in PM influenced participants’ commitment and motivation to therapy, and the sustained motivation contributes to the lasting improvements in anxiety and depressive symptoms at follow-up.
However, the absence of direct effects of AES on Anxiety and Depressive symptoms might indicate the presence of an unmeasured conflicting mediator that ultimately cancels out the overall direct effect [74,99]. This is highly probable, as the current study did not measure potential negative side effects of the trial such as rumination, which is positively associated with anxiety and depressive symptoms [23,37,38]. Further research is needed to investigate other possible mediators which may have induced an undesired impact on Anxiety and Depressive symptoms to prevent unwanted effects of mHealth programmes.
Nevertheless, the present finding supported the aforementioned theory on the potential reason behind the observed delayed effect in Anxiety as well as the importance of discomfort tolerance in anxiety pathology. A study by Beitel et al. [43] found associations between high PM individuals and their ability to tolerate distressing elements of psychotherapy. Additionally, Beitel et al. [100] also found that highly psychologically minded individuals appear to expect more self-involvement in the treatment process and outcome. Hence, speculatively, it may be that individuals who engaged with the application experienced improvements in PM, which in turn allowed them to better tolerate discomfort and become intrinsically motivated to engage with the app’s “user-as-expert”-inducing features.
Limitations and Future Directions
Our study is not without limitations. First, no waitlist condition was included. Although our main objective was to assess whether the “Anxiety and Worry” programme is more effective than the active control (“Procrastination” programme) in reducing anxiety and worry, including a passive control would have yielded a clearer result pattern. Secondly, it is possible that the improvements in anxiety and depressive symptoms observed across conditions were caused by performance expectancy, or an individual’s beliefs about the benefit of using information and communications technology, rather than the use of the Intellect mHealth programmes. Previous studies have also found evidence that performance expectancy was positively associated with intention to use mHealth apps [101]. Hence, while an Active Control condition was used to limit the presence of digital placebo, as per Gruszka et al. [102]’s recommendation, simply participating in the study and engaging in mental wellbeing topics might contribute to the improvement of anxiety and depressive symptoms [103]. Third, we underestimated the relevance of CBT techniques utilized in the “Procrastination” programme for anxiety outcomes and thus did not expect that it would be as effective in reducing anxiety among young adults. For future comparison, researchers may consider utilizing active control designs that do not involve CBT-based techniques, such as attention control designs or designs based on different treatment modalities.
Fourth, the present study lacks objective measures of engagement and adherence. While no significant differences were found on the subjective AES measure, the present study did not collect the duration spent on the Intellect Application. Moreover, study adherence was heavily reliant upon completions of the self-reported outcome measures. Even though programme completion was verified with the Intellect database and participants who did not complete the assigned programme were identified, the lack of data on duration spent on the application limits the accuracy of adherence measurement. Therefore, future studies may benefit from including a more objective app engagement measure to limit its potential in acting as a third variable when examining mHealth application’s efficacy. Additionally, approximately 30% of the sample population met the cutoff (≥10) for moderate anxiety and depressive symptoms at baseline. Thus, the present results may not readily be generalized to the clinical population of individuals experiencing anxiety or depressive disorders. Lastly, although the present study provides valuable insight into potential pathways of improvements in anxiety and depressive symptoms via psychological mindedness when engaging in mHealth applications, future research should aim to identify additional mediators that might strengthen the benefit of mHealth applications.
Conclusion
The intervention and active control showed significant improvements in anxiety and depressive symptoms across time, however both groups did not differ significantly from one another. Nonetheless, our study demonstrated the continual improvement of anxiety induced by the Anxiety programme post-intervention. Altogether, this study suggests that engagement with a CBT-based mHealth programme is related to reduction in anxiety and depressive symptoms. One mechanism involved in exerting this effect is the app’s ability to improve an individual’s Psychological Mindedness. Higher engagement rating of the mHealth programme predicted improvements on anxiety and depressive symptoms. Despite small effect sizes observed, the practical implications of these cost-effective, highly accessible mHealth programmes can meaningfully impact public mental health at scale.
Acknowledgments
The authors would like to thank the students for participating in the trial, the funding listed, and the team at Intellect for adapting the application and providing support throughout the study.
References
- 1. Borkovec TD, Inz J. The nature of worry in generalized anxiety disorder: A predominance of thought activity. Behav Res Ther. 1990 Jan 1;28(2):153–8. pmid:2183759
- 2. Olatunji BO, Wolitzky-Taylor KB, Sawchuk CN, Ciesielski BG. Worry and the anxiety disorders: A meta-analytic synthesis of specificity to GAD. Appl Prev Psychol. 2010 Jun 1;14(1):1–24.
- 3.
Anxiety (Internet). https://www.apa.org. (cited 2022 Nov 21). Available from: https://www.apa.org/topics/anxiety.
- 4. Zebb BJ, Beck JG. Worry Versus Anxiety: Is There Really a Difference? Behav Modif. 1998 Jan 1;22(1):45–61.
- 5. Andrews G, Hobbs MJ, Borkovec TD, Beesdo K, Craske MG, Heimberg RG, et al. Generalized worry disorder: a review of DSM-IV generalized anxiety disorder and options for DSM-V. Depress Anxiety. 2010;27(2):134–47. pmid:20058241
- 6. Eysenck MW, Fajkowska M. Anxiety and depression: toward overlapping and distinctive features. Cogn Emot. 2018 Oct 3;32(7):1391–400. pmid:28608767
- 7. Ballenger JC. Anxiety and Depression: Optimizing Treatments. Prim Care Companion J Clin Psychiatry. 2000 Jun;2(3):71–9. pmid:15014652
- 8.
Kapalka GM. Depression. In: Nutritional and Herbal Therapies for Children and Adolescents (Internet). Elsevier; 2010 (cited 2022 Nov 23). p. 141–87. Available from: https://linkinghub.elsevier.com/retrieve/pii/B9780123749277000066.
- 9. Hohls JK, König HH, Quirke E, Hajek A. Anxiety, Depression and Quality of Life—A Systematic Review of Evidence from Longitudinal Observational Studies. Int J Environ Res Public Health. 2021 Nov 16;18(22):12022. pmid:34831779
- 10. Moreno-Peral P, Pino-Postigo A, Conejo-Cerón S, Bellón D, Rodríguez-Martín B, Martínez-Vizcaíno V, et al. Effectiveness of Physical Activity in Primary Prevention of Anxiety: Systematic Review and Meta-Analysis of Randomized Controlled Trials. Int J Environ Res Public Health. 2022 Feb 5;19(3):1813. pmid:35162835
- 11. The gap in life expectancy from preventable physical illness in psychiatric patients in Western Australia: retrospective analysis of population based registers | The BMJ (Internet). (cited 2022 Nov 28). Available from: https://www.bmj.com/content/346/bmj.f2539. pmid:23694688
- 12. Richards D, Timulak L, Rashleigh C, McLoughlin O, Colla A, Joyce C, et al. Effectiveness of an internet-delivered intervention for generalized anxiety disorder in routine care: A randomised controlled trial in a student population. Internet Interv. 2016 Nov 1;6:80–8. pmid:30135817
- 13. Wang C, Wen W, Zhang H, Ni J, Jiang J, Cheng Y, et al. Anxiety, depression, and stress prevalence among college students during the COVID-19 pandemic: A systematic review and meta-analysis. J Am Coll Health. 2021 Sep 1;0(0):1–8. pmid:34469261
- 14. Liyanage S, Saqib K, Khan AF, Thobani TR, Tang WC, Chiarot CB, et al. Prevalence of Anxiety in University Students during the COVID-19 Pandemic: A Systematic Review. Int J Environ Res Public Health. 2021 Dec 22;19(1):62. pmid:35010323
- 15. Hunt J, Eisenberg D. Mental Health Problems and Help-Seeking Behavior Among College Students. J Adolesc Health. 2010 Jan 1;46(1):3–10. pmid:20123251
- 16. Lake J, Turner MS. Urgent Need for Improved Mental Health Care and a More Collaborative Model of Care. Perm J. 2017;21:17–024.
- 17. Ohayon MM, Shapiro CM, Kennedy SH. Differentiating DSM-IV Anxiety and Depressive Disorders in the General Population: Comorbidity and Treatment Consequences. Can J Psychiatry. 2000 Mar 1;45(2):166–72. pmid:10742876
- 18. Vanheusden K, Mulder CL, van der Ende J, van Lenthe FJ, Mackenbach JP, Verhulst FC. Young adults face major barriers to seeking help from mental health services. Patient Educ Couns. 2008 Oct 1;73(1):97–104. pmid:18584997
- 19. Collins KA, Westra HA, Dozois DJA, Burns DD. Gaps in accessing treatment for anxiety and depression: Challenges for the delivery of care. Clin Psychol Rev. 2004 Sep 1;24(5):583–616. pmid:15325746
- 20. Donker T, Petrie K, Proudfoot J, Clarke J, Birch MR, Christensen H. Smartphones for Smarter Delivery of Mental Health Programs: A Systematic Review. J Med Internet Res. 2013 Nov 15;15(11):e247. pmid:24240579
- 21. Firth J, Torous J, Nicholas J, Carney R, Rosenbaum S, Sarris J. Can smartphone mental health interventions reduce symptoms of anxiety? A meta-analysis of randomized controlled trials. J Affect Disord. 2017 Aug 15;218:15–22. pmid:28456072
- 22. Hilty DM, Chan S, Hwang T, Wong A, Bauer AM. Advances in mobile mental health: opportunities and implications for the spectrum of e-mental health services. mHealth. 2017 Aug 21;3:34. pmid:28894744
- 23. Bakker D, Kazantzis N, Rickwood D, Rickard N. A randomized controlled trial of three smartphone apps for enhancing public mental health. Behav Res Ther. 2018 Oct;109:75–83. pmid:30125790
- 24. Lee M, Lee H, Kim Y, Kim J, Cho M, Jang J, et al. Mobile App-Based Health Promotion Programs: A Systematic Review of the Literature. Int J Environ Res Public Health. 2018 Dec 13;15(12):E2838. pmid:30551555
- 25. Howells A, Ivtzan I, Eiroa-Orosa FJ. Putting the ‘app’ in Happiness: A Randomised Controlled Trial of a Smartphone-Based Mindfulness Intervention to Enhance Wellbeing. J Happiness Stud. 2016 Feb 1;17(1):163–85.
- 26. Lee RA, Jung ME. Evaluation of an mHealth App (DeStressify) on University Students’ Mental Health: Pilot Trial. JMIR Ment Health. 2018 Jan 23;5(1):e8324. pmid:29362209
- 27. Rith-Najarian LR, Boustani MM, Chorpita BF. A systematic review of prevention programs targeting depression, anxiety, and stress in university students. J Affect Disord. 2019 Oct;257:568–84. pmid:31326690
- 28. Day V, McGrath PJ, Wojtowicz M. Internet-based guided self-help for university students with anxiety, depression and stress: A randomized controlled clinical trial. Behav Res Ther. 2013 Jul 1;51(7):344–51. pmid:23639300
- 29. Proudfoot J, Clarke J, Birch MR, Whitton AE, Parker G, Manicavasagar V, et al. Impact of a mobile phone and web program on symptom and functional outcomes for people with mild-to-moderate depression, anxiety and stress: a randomised controlled trial. BMC Psychiatry. 2013 Nov 18;13(1):312.
- 30. Linardon J, Cuijpers P, Carlbring P, Messer M, Fuller-Tyszkiewicz M. The efficacy of app-supported smartphone interventions for mental health problems: a meta-analysis of randomized controlled trials. World Psychiatry. 2019 Oct;18(3):325–36. pmid:31496095
- 31. Firth J, Torous J, Nicholas J, Carney R, Pratap A, Rosenbaum S, et al. The efficacy of smartphone-based mental health interventions for depressive symptoms: a meta-analysis of randomized controlled trials. World Psychiatry. 2017 Oct;16(3):287–98. pmid:28941113
- 32. Goldberg SB, Lam SU, Simonsson O, Torous J, Sun S. Mobile phone-based interventions for mental health: A systematic meta-review of 14 meta-analyses of randomized controlled trials. PLOS Digit Health. 2022 Jan 18;1(1):e0000002. pmid:35224559
- 33. Ahmed A, Ali N, Giannicchi A, Abd-alrazaq AA, Ahmed MAS, Aziz S, et al. Mobile applications for mental health self-care: A scoping review. Comput Methods Programs Biomed Update. 2021 Jan 1;1:100041.
- 34. Wang K, Varma DS, Prosperi M. A systematic review of the effectiveness of mobile apps for monitoring and management of mental health symptoms or disorders. J Psychiatr Res. 2018 Dec 1;107:73–8. pmid:30347316
- 35. Larsen ME, Huckvale K, Nicholas J, Torous J, Birrell L, Li E, et al. Using science to sell apps: Evaluation of mental health app store quality claims. Npj Digit Med. 2019 Mar 22;2(1):1–6. pmid:31304366
- 36. Wasil AR, Venturo-Conerly KE, Shingleton RM, Weisz JR. A review of popular smartphone apps for depression and anxiety: Assessing the inclusion of evidence-based content. Behav Res Ther. 2019 Dec 1;123:103498. pmid:31707224
- 37. Bakker D, Rickard N. Engagement in mobile phone app for self-monitoring of emotional wellbeing predicts changes in mental health: MoodPrism. J Affect Disord. 2018 Feb;227:432–42. pmid:29154165
- 38. Bakker D, Rickard N. Engagement with a cognitive behavioural therapy mobile phone app predicts changes in mental health and wellbeing: MoodMission. Aust Psychol. 2019 Aug 1;54(4):245–60.
- 39. Huberty J, Puzia ME, Green J, Vlisides-Henry RD, Larkey L, Irwin MR, et al. A mindfulness meditation mobile app improves depression and anxiety in adults with sleep disturbance: Analysis from a randomized controlled trial. Gen Hosp Psychiatry. 2021 Nov 1;73:30–7. pmid:34537477
- 40. Li J, Zhang C, Li X, Zhang C. Patients’ emotional bonding with MHealth apps: An attachment perspective on patients’ use of MHealth applications. Int J Inf Manag. 2020 Apr 1;51:102054.
- 41.
Denollet J, Nykliček I. Psychological mindedness: A new index to assess a major emotion-focused coping style. In: Emotional expression and health: Advances in theory, assessment and clinical applications. New York, NY, US: Brunner-Routledge; 2004. p. 185–203.
- 42. Abramowitz SI, Abramowitz CV. Psychological-Mindedness and Benefit From Insight-Oriented Group Therapy. Arch Gen Psychiatry. 1974 May 1;30(5):610–5. pmid:4824195
- 43. Beitel M, Ferrer E, Cecero JJ. Psychological mindedness and awareness of self and others. J Clin Psychol. 2005;61(6):739–50. pmid:15546143
- 44. Conte HR, Plutchik R, Jung BB, Picard S, Karasu TB, Lotterman A. Psychological Mindedness as a predictor of psychotherapy outcome: A preliminary report. Compr Psychiatry. 1990 Sep 1;31(5):426–31. pmid:2225801
- 45. Grant A, Franklin J, Langford P. The Self-Reflection and Insight Scale: A New Measure of Private Self-Consciousness. Soc Behav Personal Int J. 2002 Jan 1;30:821–35.
- 46. Pang NTP, Masiran R, Tan KA, Kassim A. Psychological mindedness as a mediator in the relationship between dysfunctional coping styles and depressive symptoms in caregivers of children with autism spectrum disorder. Perspect Psychiatr Care. 2020;56(3):649–56. pmid:32017137
- 47. Piper WE, Joyce AS, Rosie JS, Azim HF. Psychological mindedness, work, and outcome in day treatment. Int J Group Psychother. 1994 Jul;44(3):291–311. pmid:7927974
- 48.
Piper WE, Rosie JS, Joyce AS, Azim HFA. Time-limited day treatment for personality disorders: Integration of research and practice in a group program. Washington, DC, US: American Psychological Association; 1996. xv, 360 p. (Time-limited day treatment for personality disorders: Integration of research and practice in a group program).
- 49. Piper WE, McCallum M, Joyce AS, Rosie JS, Ogrodniczuk JS. Patient personality and time-limited group psychotherapy for complicated grief. Int J Group Psychother. 2001 Oct;51(4):525–52. pmid:11582899
- 50. McCallum M, Piper WE, O’Kelly J. Predicting patient benefit from a group-oriented, evening treatment program. Int J Group Psychother. 1997 Jul;47(3):291–314. pmid:9217344
- 51. McCallum M, Piper WE, Ogrodniczuk JS, Joyce AS. Relationships among psychological mindedness, alexithymia and outcome in four forms of short-term psychotherapy. Psychol Psychother Theory Res Pract. 2003;76(2):133–44. pmid:12855060
- 52. Conte HR, Ratto R, Karasu TB. The Psychological Mindedness Scale. J Psychother Pract Res. 1996;5(3):250–9.
- 53. Boylan MB. PSYCHOLOGICAL MINDEDNESS AS A PREDICTOR OF TREATMENT.: 132.
- 54. Nyklíček I, Majoor D, Schalken PAAM. Psychological mindedness and symptom reduction after psychotherapy in a heterogeneous psychiatric sample. Compr Psychiatry. 2010 Sep 1;51(5):492–6. pmid:20728006
- 55. Marciniak MA, Shanahan L, Rohde J, Schulz A, Wackerhagen C, Kobylińska D, et al. Standalone Smartphone Cognitive Behavioral Therapy–Based Ecological Momentary Interventions to Increase Mental Health: Narrative Review. JMIR MHealth UHealth. 2020 Nov 12;8(11):e19836. pmid:33180027
- 56. Grant AM. Rethinking Psychological Mindedness: Metacognition, Self-reflection, and Insight. Behav Change. 2001 Apr;18(1):8–17.
- 57. Sauter FM, Heyne D, Blöte AW, Widenfelt BM van, Westenberg PM. Assessing Therapy-Relevant Cognitive Capacities in Young People: Development and Psychometric Evaluation of the Self-Reflection and Insight Scale for Youth. Behav Cogn Psychother. 2010 May;38(3):303–17. pmid:20380777
- 58. Ong WY, Sündermann O. Efficacy of the Mental Health App “Intellect” to Improve Body Image and Self-compassion in Young Adults: A Randomized Controlled Trial With a 4-Week Follow-up. JMIR MHealth UHealth. 2022 Nov 21;10(11):e41800. pmid:36409542
- 59. Toh SHY, Tan JHY, Kosasih FR, Sündermann O. Efficacy of the Mental Health App Intellect to Reduce Stress: Randomized Controlled Trial With a 1-Month Follow-up. JMIR Form Res. 2022 Dec 14;6(12):e40723. pmid:36515984
- 60. Sil A, Kumar P, Kumar R, Das NK. Selection of Control, Randomization, Blinding, and Allocation Concealment. Indian Dermatol Online J. 2019 Aug 28;10(5):601–5. pmid:31544090
- 61. Roepke AM, Jaffee SR, Riffle OM, McGonigal J, Broome R, Maxwell B. Randomized Controlled Trial of SuperBetter, a Smartphone-Based/Internet-Based Self-Help Tool to Reduce Depressive Symptoms. Games Health J. 2015 Jun;4(3):235–46. pmid:26182069
- 62. Siddaway AP, Taylor PJ, Wood AM. Reconceptualizing Anxiety as a Continuum That Ranges From High Calmness to High Anxiety: The Joint Importance of Reducing Distress and Increasing Well-Being. J Pers Soc Psychol. 2018 Feb;114(2):e1–11. pmid:28253007
- 63. Löwe B, Decker O, Müller S, Brähler E, Schellberg D, Herzog W, et al. Validation and standardization of the Generalized Anxiety Disorder Screener (GAD-7) in the general population. Med Care. 2008 Mar;46(3):266–74. pmid:18388841
- 64. Spitzer RL, Kroenke K, Williams JBW, Löwe B. A Brief Measure for Assessing Generalized Anxiety Disorder: The GAD-7. Arch Intern Med. 2006 May 22;166(10):1092–7. pmid:16717171
- 65. Kroenke K, Spitzer RL, Williams JBW. The PHQ-9. J Gen Intern Med. 2001 Sep;16(9):606–13.
- 66. Rickard N, Arjmand HA, Bakker D, Seabrook E. Development of a Mobile Phone App to Support Self-Monitoring of Emotional Well-Being: A Mental Health Digital Innovation. JMIR Ment Health. 2016 Nov 23;3(4):e6202. pmid:27881358
- 67. Stoyanov SR, Hides L, Kavanagh DJ, Wilson H. Development and Validation of the User Version of the Mobile Application Rating Scale (uMARS). JMIR MHealth UHealth. 2016 Jun 10;4(2):e72. pmid:27287964
- 68. Faul F, Erdfelder E, Lang AG, Buchner A. G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods. 2007 May;39(2):175–91. pmid:17695343
- 69. Lecomte T, Potvin S, Corbière M, Guay S, Samson C, Cloutier B, et al. Mobile Apps for Mental Health Issues: Meta-Review of Meta-Analyses. JMIR MHealth UHealth. 2020 May 29;8(5):e17458. pmid:32348289
- 70. Hochheimer CJ, Sabo RT, Krist AH, Day T, Cyrus J, Woolf SH. Methods for Evaluating Respondent Attrition in Web-Based Surveys. J Med Internet Res. 2016 Nov 22;18(11):e301. pmid:27876687
- 71. Overall JE, Tonidandel S, Starbuck RR. Last-observation-carried-forward (LOCF) and tests for difference in mean rates of change in controlled repeated measurements designs with dropouts. Soc Sci Res. 2009 Jun 1;38(2):492–503.
- 72.
Portney LG, Watkins MP. Foundations of Clinical Research: Applications to Practice. Pearson/Prentice Hall; 2009. 892 p.
- 73. Hayes AF. Introduction to Mediation, Moderation, and Conditional Process Analysis: Third Edition: A Regression-Based Approach (Internet). (cited 2022 Jul 4). Available from: https://www.guilford.com/books/Introduction-to-Mediation-Moderation-and-Conditional-Process-Analysis/Andrew-Hayes/9781462549030.
- 74. Zhao X, Lynch JG, Chen Q. Reconsidering Baron and Kenny: Myths and Truths about Mediation Analysis. J Consum Res. 2010 Aug;37(2):197–206.
- 75. Rucker DD, Preacher KJ, Tormala ZL, Petty RE. Mediation Analysis in Social Psychology: Current Practices and New Recommendations. Soc Personal Psychol Compass. 2011;5(6):359–71.
- 76.
Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. New York: Routledge; 1988. 567 p.
- 77. Mostert J, Loxton H. Exploring the Effectiveness of the FRIENDS Program in Reducing Anxiety Symptoms Among South African Children. Behav Change. 2008 Jun 1;25(2):85–96.
- 78. Troeung L, Egan SJ, Gasson N. A waitlist-controlled trial of group cognitive behavioural therapy for depression and anxiety in Parkinson’s disease. BMC Psychiatry. 2014 Jan 27;14(1):19. pmid:24467781
- 79. Rachman S. Points of light. Behav Res Ther. 2017 Jan;88:2–6. pmid:28110673
- 80. Sun S, Lin D, Goldberg S, Shen Z, Chen P, Qiao S, et al. A mindfulness-based mobile health (mHealth) intervention among psychologically distressed university students in quarantine during the COVID-19 pandemic: A randomized controlled trial. J Couns Psychol. 20210715;69(2):157. pmid:34264696
- 81. Law M, Gasteiger N, Vedhara K, Massey A, Jia R, Ayling K, et al. Risk Factors and Changes in Depression and Anxiety over Time in New Zealand during COVID-19: A Longitudinal Cohort Study. Psych. 2022 Dec;4(4):706–16.
- 82. Hyland P, Shevlin M, McBride O, Murphy J, Karatzias T, Bentall RP, et al. Anxiety and depression in the Republic of Ireland during the COVID-19 pandemic. Acta Psychiatr Scand. 2020 Sep;142(3):249–56. pmid:32716520
- 83. Zhang Y, Bao X, Yan J, Miao H, Guo C. Anxiety and Depression in Chinese Students During the COVID-19 Pandemic: A Meta-Analysis. Front Public Health. 2021;9:697642. pmid:34485228
- 84. McKee L, Zvolensky MJ, Solomon SE, Bernstein A, Leen-Feldner E. Emotional-Vulnerability and Mindfulness: A Preliminary Test of Associations among Negative Affectivity, Anxiety Sensitivity, and Mindfulness Skills. Cogn Behav Ther. 2007 Jun 1;36(2):91–101. pmid:17530495
- 85. Schmidt NB, Richey JA, Cromer KR, Buckner JD. Discomfort Intolerance: Evaluation of a Potential Risk Factor for Anxiety Psychopathology. Behav Ther. 2007 Sep 1;38(3):247–55. pmid:17697850
- 86. Spira AP, Zvolensky MJ, Eifert GH, Feldner MT. Avoidance-oriented coping as a predictor of panic-related distress: a test using biological challenge. J Anxiety Disord. 2004 Jan 1;18(3):309–23. pmid:15125979
- 87. Peris TS, Compton SN, Kendall PC, Birmaher B, Sherill J, March J, et al. Trajectories of Change in Youth Anxiety during Cognitive Behavior Therapy. J Consult Clin Psychol. 2015 Apr;83(2):239–52. pmid:25486372
- 88. Cuijpers P, van Straten A, Warmerdam L. Behavioral activation treatments of depression: A meta-analysis. Clin Psychol Rev. 2007 Apr 1;27(3):318–26. pmid:17184887
- 89. Abbasi IS, Alghamdi NG. The Prevalence, Predictors, Causes, Treatment, and Implications of Procrastination Behaviors in General, Academic, and Work Setting. Int J Psychol Stud. 2015 Feb 16;7(1):p59.
- 90. Küchler AM, Albus P, Ebert DD, Baumeister H. Effectiveness of an internet-based intervention for procrastination in college students (StudiCare Procrastination): Study protocol of a randomized controlled trial. Internet Interv. 2019 Sep 1;17:100245. pmid:31080750
- 91. Beutel ME, Klein EM, Aufenanger S, Brähler E, Dreier M, Müller KW, et al. Procrastination, Distress and Life Satisfaction across the Age Range–A German Representative Community Study. PLOS ONE. 2016 Feb 12;11(2):e0148054. pmid:26871572
- 92.
Anxiety, Depression, and Procrastination Among Students: Rumination Plays a Larger Mediating Role than Worry | SpringerLink (Internet). (cited 2022 Nov 28). Available from: https://link.springer.com/article/10.1007/s10942-017-0271-5#Sec18.
- 93. Rozental A, Bennett S, Forsström D, Ebert DD, Shafran R, Andersson G, et al. Targeting Procrastination Using Psychological Treatments: A Systematic Review and Meta-Analysis. Front Psychol (Internet). 2018 (cited 2022 Nov 28);9. Available from: https://www.frontiersin.org/articles/10.3389/fpsyg.2018.01588. pmid:30214421
- 94. Miller WR. Interactive journaling as a clinical tool. J Ment Health Couns. 2014;36(1):31–42.
- 95. Riper H, Andersson G, Hunter SB, de Wit J, Berking M, Cuijpers P. Treatment of comorbid alcohol use disorders and depression with cognitive-behavioural therapy and motivational interviewing: a meta-analysis. Addict Abingdon Engl. 2014 Mar;109(3):394–406. pmid:24304463
- 96. Murphy DA, Chen X, Naar-King S, Parsons for the Adolescent Trials N JT. Alcohol and Marijuana Use Outcomes in the Healthy Choices Motivational Interviewing Intervention for HIV-Positive Youth. AIDS Patient Care STDs. 2012 Feb;26(2):95–100. pmid:22191456
- 97. Westra HA, Constantino MJ, Antony MM. Integrating motivational interviewing with cognitive-behavioral therapy for severe generalized anxiety disorder: An allegiance-controlled randomized clinical trial. J Consult Clin Psychol. 2016;84(9):768–82. pmid:26985729
- 98. Kishon R, Geronazzo-Alman L, Westphal M, Chen Y, Green D, Sarda A, et al. Psychological Mindedness and Alexithymia Predict Symptom Reduction in Cognitive Behavior Therapy for Major Depressive Disorder. Int J Cogn Ther. 2020 Dec 1;13(4):287–302.
- 99.
Darlington RB, Hayes AF. Regression Analysis and Linear Models: Concepts, Applications, and Implementation. Guilford Publications; 2016. 688 p.
- 100. Beitel M, Hutz A, Sheffield KM, Gunn C, Cecero JJ, Barry DT. Do psychologically-minded clients expect more from counselling? Psychol Psychother Theory Res Pract. 2009;82(4):369–83. pmid:19527565
- 101. Nunes A, Limpo T, Castro SL. Acceptance of Mobile Health Applications: Examining Key Determinants and Moderators. Front Psychol. 2019 Dec 10;10:2791. pmid:31920836
- 102. Gruszka P, Burger C, Jensen MP. Optimizing Expectations via Mobile Apps: A New Approach for Examining and Enhancing Placebo Effects. Front Psychiatry. 2019 May 31;10:365. pmid:31214057
- 103. Torous J, Firth J. The digital placebo effect: mobile mental health meets clinical psychiatry. Lancet Psychiatry. 2016 Feb 1;3(2):100–2. pmid:26851322