Background

Anxiety disorders are highly prevalent among college communities. A 2023 study indicated that 41.8% of college students and 38.6% of faculty and staff had experienced symptoms consistent with severe or extremely severe anxiety over the past week [1]. Despite these alarmingly high rates, many individuals in need of mental health services remain untreated [2]. Some of the most common barriers to receiving mental health services among college students are access, time, and stigma. Accessibility barriers specifically include lengthy weight times, the cost of services, and perceived lack of options [3,4]. Similarly, university faculty reported stigma and lack of support as their main barriers to receiving mental health care [5].

In light of these barriers, there has been increased interest in digital mental health interventions (DMHIs). DMHIs provide mental health support via technologies like web-based or mobile platforms and have the potential to expand treatment delivery options and enhance access to mental health care, particularly in rural settings [6,7]. Additionally, studies show DMHIs may be specifically effective at reducing depression, anxiety, and stress outcomes in academic settings [8,9]. A recent meta-analysis [10] assessed the efficacy of digital interventions for various anxiety disorders, examining their impact against waitlist and standard care controls for specific disorders, and found positive effects on symptom reduction for DMHI’s for generalized anxiety, mixed anxiety, and social anxiety disorders. However, it was noted that further research is necessary to investigate which intervention formats have the greatest impact on treatment outcomes, particularly regarding the type and frequency of support provided.

The use of Cognitive Bias Modification for Interpretation (CBM-I) is a mechanistic intervention that may be an effective component of treatment for anxiety [11–15], given extensive research showing that selective threat interpretations are common among individuals high in anxiety [16]. However, some studies report mixed results regarding the effectiveness of CBM-I [17,18], highlighting the need for further investigation into the efficacy of these programs and how to improve their acceptability. CBM-I training is designed to modify threat-focused cognitive biases related to interpretation processes (i.e., assigning threatening meanings to ambiguous situations). “Hoos Think Calmly” (HTC) is a mobile application based on a specific CBM-I paradigm known as the ambiguous scenarios paradigm, which presents users with a series of ambiguous or mildly threatening scenarios [19]. In the typical ambiguous scenarios format, the final word of each scenario appears as a word fragment that participants must complete. These fragments are typically designed to assign benign or neutral interpretations of the scenarios in most cases, helping to shift participants away from selectively negative or catastrophic thinking patterns. Through repeated practice and reinforcement, the goal is to help individuals develop more flexible interpretation patterns that are less rigidly negative, ultimately reducing anxiety [20]. HTC offers these CBM-I training scenarios in brief 5–10 minute training sessions that are designed to increase flexible thinking in response to stressors commonly experienced by students, faculty, and staff at a large public university. To meet the specific needs of the university community, we engaged in outreach with student groups and organizations to beta test the app, review stressor domains, and develop scenarios tailored to their needs. We also consulted with leadership from local university programs for recommendations on in-app resources to incorporate.

Although there is evidence supporting the effectiveness of various DMHIs, such as online CBM-I training (including at our team’s MindTrails website; [21–23]), their impact is hindered by high attrition rates alongside low engagement [24–26]. Better ways to overcome these common limitations of DMHIs need to be better understood. A 2021 review found that participants’ engagement with DMHIs increased when users found the content appealing, perceived the program to be a good fit, and experienced perceived changes in their mental health [27]. Thus, we need to understand what makes content appealing and a good fit to users. Research on DMHIs tends to rely on quantitative metrics to evaluate outcomes, which are very valuable but limited in their ability to contextualize data and assess individual user experiences that may impact engagement. This points to the value of also using qualitative approaches to help design, evaluate and improve DMHIs [28–30]. For instance, qualitative methods can be used during the post-trial phase of pilot studies to gain insight into users’ acceptance of and feedback on DMHIs [31]. Other studies have evaluated the acceptability of DMHIs through qualitative methods and have shown mixed feedback regarding attitudes toward and engagement with these tools. One study identified negative attitudes toward a Cognitive Behavioral Therapy (CBT) DMHI as users specifically found it to be inflexible, unrelatable, and hard to engage with [32]. Another study revealed that participants had a positive attitude toward and experienced benefits from a DMHI, but the need to further personalize and creatively engage users to promote long-term adherence was emphasized [33]. When it comes to enhancing user motivation, engagement, and enjoyment, recent research indicates that games and gamification, when combined with CBT, have produced encouraging results [34,35]. As the mobile health (mHealth) landscape continues to evolve to meet user needs, ongoing evaluation of DMHIs is essential to keep pace with the advancements in the field. This feedback can be especially valuable to guide improvements for future versions of a DMHI. Additionally, a 2022 paper emphasized the importance of distinguishing between the feasibility/acceptability of a DMHI and its effectiveness. It’s argued that a program’s efficacy cannot be accurately assessed if the intervention is not implemented properly and to its full potential [36]. Acceptability and feasibility studies help determine whether an intervention can meet users’ needs and can be implemented as intended. This is essential for evaluating user interest and satisfaction, willingness to engage with DMHIs, factors influencing participation, and reasons for continuing or dropping out—all of which are important for refining the intervention, especially in its early stages. The findings from this study will offer concrete strategies for improving DMHIs based on user feedback, which can be used to enhance the acceptability of future interventions.

Objective

The objective of this study is to collect feedback from participants in the first trial of a mobile application called “Hoos Think Calmly” (HTC) to evaluate acceptability and gain a deeper understanding of their experiences and suggestions for improvement. HTC is designed to increase flexible thinking in response to stressors commonly experienced by students, faculty, and staff at a large public university through brief ~5 minute interventions (microdoses) of CBM-I training delivered by mobile phone to university community members. Study design and analysis plans were preregistered on Open Science Framework [37].

Ethics approval and consent to participate

This study was approved by the Institutional Review Board for Social and Behavioral Sciences (IRB-SBS) at the University of Virginia. All participants provided written informed consent via an electronic consent form that outlined the study’s purpose, risks, and benefits prior to study enrollment. To ensure confidentiality, each participant was assigned a unique ID after enrolling, and all data—including questionnaire responses and audio/video recordings—were securely stored and linked only to this ID, with access restricted to the research team.

Strategies to understand and mitigate bias: Reflexivity and credibility

Our team acknowledges that we bring biases in favor of HTC, having grants that fund the program and prior tests of the parent program MindTrails that demonstrated efficacy in reducing anxiety [21–23]. It is likely that these factors may predispose us towards interpreting the current findings more positively.

To bolster methodological integrity, we employed several credibility strategies. Two of the initial coders on the research team were individuals not involved in the data collection process to help mitigate biases and allow for fresh perspectives [38]. Likewise, we established an intercoder reliability score exceeding 0.80 on a subset of interviews before proceeding with general coding to increase coding consistency. Throughout the analysis, the lead author attended a recurring journal club focused on best practices in qualitative research. During this time, the lead author also spent time examining personal judgements and beliefs to better understand how biases might influence the research process and outcomes. We encouraged peer examination, and colleagues were asked to give feedback on our research methodology and procedures. An audit trail was kept of all edits made to the codebook throughout the analysis process to help establish confirmability and trustworthiness of data [39]. This study was reported using the Standards for Reporting Qualitative Research (SRQR) guidelines [40] (S1 Text).

Intervention

In the parent randomized control trial (RCT), n = 134 participants assigned to the active treatment condition were notified to complete two microdose sessions per day over a 6-week period. Prior to beginning a microdose session, participants completed a pre-intervention survey in which they reported their current mood (and completed a very brief imaginal exercise depending on their anxiety level) and selected a specific stressor domain for targeted training. Domain options included academics/work/career development, social situations, romantic relationships, finances, physical health, mental health, family and home life, and discrimination. The content of the training scenarios was modified to match common situations encountered for the different groups (e.g., to capture fears of negative evaluation, the undergraduate group would read about giving a class presentation while the staff would read about presenting at a staff meeting). Post-intervention surveys were also administered to assess changes in mood and provide additional resources and tips for emotion regulation and application of the CBM-I learning to daily life. At the end of the day, participants were instructed to complete a nightly survey inquiring about their mood and ability to think flexibly throughout the day. During this survey, participants had the option to pre-select notification times for the next day’s microdose sessions.

A microdose session consisted of ~ 5 minutes of CBM-I training that aimed to reduce rigidly negative thinking patterns associated with anxiety. Training scenarios could consist of 10 CBM-I short scenarios (this occurred in the majority of sessions), or 1 long scenario, or 1 write-your-own scenario (except for the Discrimination domain, which did not include CBM-I) [41]. The variation was included to reduce repetitiveness and to introduce different tasks to increase flexible thinking and the personal relevance of training. A short scenario was typically three sentences in length and consisted of an emotionally ambiguous story that raised a potential threat but was then resolved when the user completed the final word in the last sentence. The emotional resolution was reinforced through a subsequent comprehension question (see Fig 1). A long scenario was the same as a short scenario except it also included a range of thoughts, feelings, or behaviors that could occur in a particular situation to help with seeing multiple different perspectives. Users were then prompted to write down their own thoughts that could be helpful in that situation. The write-your-own scenario invited participants to craft their own short CBM-I scenario based on a personally relevant anxiety-inducing situation. Users were then tasked with resolving the scenario in a positive or non-threatening way and coming up with reasons for why the chosen resolutions could be likely to occur. Fig 1 displays a screenshot of the app’s homepage, domain selection page, and a short scenario example. More details on the intervention are available via Open Science Framework [41,42].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Screenshots of the HTC app.

https://doi.org/10.1371/journal.pdig.0000601.g001

In the present study, some CBM-I scenarios were adapted from our existing pool of scenarios developed for previous MindTrails studies [21–23] and others were generated for the HTC application. Existing scenarios were adapted as needed to be relevant to university community members with different roles on campus and at different developmental stages. For example, a scenario pertaining to anxiety about handing in a paper to a teaching assistant might be kept the same (i.e., not adapted) for an undergraduate student, but modified to be focused on submitting a paper to an academic journal (graduate students) or giving an important report to a supervisor (staff members). Our goal was to preserve the core anxiety-provoking ambiguous stimulus for a scenario across groups but tailor the content to be more role-specific. This tailoring was guided in part by our qualitative interviews with university community members as well as in consultation with undergraduate students, graduate students, and faculty members on the research team who have lived experience as members of these groups. New scenarios were generated based on feedback from the qualitative interviews about topics identified as being particularly relevant for the university community. For example, we learned in feedback that many undergraduate and graduate students experience imposter syndrome and that this is a common source of stress and anxiety. We did not have any scenarios specifically pertaining to imposter syndrome in our existing pool, so we developed new scenarios focused on this topic. Scenarios were initially adapted and/or generated by a team of undergraduate research assistants who received training in how to write effective CBM-I scenarios. Subsequently, the HTC application was pilot tested, and scenarios were reviewed by other undergraduate students as well as graduate students and faculty in clinical psychology for small errors (e.g., typos) as well as for their effectiveness as CBM-I scenarios (i.e., whether the scenario vividly introduced a potentially threatening ambiguous situation and resolved that ambiguity in a positive or negative manner).

Study procedures and data collection

This study used quantitative and qualitative approaches to understand user experience and acceptability of the HTC mHealth platform. Qualitative methods often provide greater depth in understanding, while quantitative methods are viewed as more objective and generalizable [43]. Mixed-methods analysis is often used to corroborate findings, elaborate on results, or generate new ideas [44]. We employed a mixed-methods approach to corroborate findings and enhance the credibility of the results through triangulation, while also elaborating on the results to achieve a more comprehensive analysis of the data [45]. While qualitative analysis was the primary focus of the study, quantitative methods were used to complement, confirm, and expand upon the qualitative findings. Data was collected via questionnaires and semi-structured interviews. Four biweekly user experience questionnaires were administered to all participants assigned to the treatment condition throughout their study participation. The first three questionnaires were administered every two weeks during the 6-week study period, and the final questionnaire was administered at the 8-week mark as a post-study follow-up. In addition to survey questionnaires, a subset of participants who took part in the 6-week pilot RCT to test the HTC mobile application were selected to participate in a user feedback interview. The selection process involved purposive sampling based on group assignment, study progress, and time passed since enrollment. Participants who were assigned to the treatment condition and had completed at least 3 microdose sessions were randomly selected to participate in a feedback interview. The semi-structured interview consisted of a mix of open-ended questions about user perceptions of and suggestions for the application, and more targeted questions that addressed topics that would be relevant for app design and content decisions. Questions were based in part on prior qualitative interviews conducted by our team and common user experience measurement tools [46]. In prior qualitative interviews, the semi-structured interview guide was iteratively updated to prioritize questions related to different design decisions (e.g., earlier interviews focused on topics that should be covered in CBM-I scenarios, whereas later interviews focused more on developing a feasible implementation plan with respect to app notifications, session length, etc.).

The interview selection process was initiated by inviting participants with the closest study enrollment date to the time of recruitment and moving backwards until our target accrual number was met. We aimed to complete approximately 20 interviews based on norms in the literature for papers with similar goals [28,47] and also due to practical considerations. A total of 89 interview invites were sent to faculty members, staff members, graduate students, and undergraduate students, and 22 interviews were completed using a semi-structured interview guide (S2 Text). Participant characteristics are provided in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Characteristics of interviewee participants and survey respondents.

https://doi.org/10.1371/journal.pdig.0000601.t001

Patient Health Questionnaire-2 (PHQ-2) was used to assess depression symptoms. It consists of two 4-point Likert items, ranging from 0 (not at all) to 3 (nearly every day). A score of 3 or greater suggests the potential for major depressive disorder. Previous research supports the measure’s construct and criterion validity, discriminative validity, and test-retest reliability [48–50]. The Overall Anxiety Severity and Impairment Scale (OASIS) was used to assess the frequency and intensity of anxiety symptoms. It consists of five 5-point Likert items, ranging from 0 (lowest value) to 4 (highest value), with higher scores indicating more frequent and severe anxiety. The OASIS has demonstrated strong psychometric properties, including good convergent validity and test-retest reliability [51–53]. Other MindTrails studies have also found acceptable internal consistency for this measure [21,23]. Perceived Stress Scale 4 (PSS-4) was used to assess stress through four 5-point Likert items, ranging from 0 (never) to 4 (very often). A higher score indicates more stress. Previous research has found acceptable internal consistency and reliability for this measure [54,55].

Analysis

Following our pre-registration [37], interview data was analyzed using thematic analysis as outlined by Braun and Clarke [56]. All interviews were transcribed using Descript audio transcription service [57]. Each transcript was then checked against the original audio recording by a member of our research team. This was for the purpose of ensuring transcription accuracy and enhancing team familiarity with the data. During transcript review, team members noted initial themes found in the data.

After all interview transcripts had been reviewed, preliminary codes were discussed, and an initial codebook was created. We then used an iterative coding process, allowing for continual addition or deletion of codes to align with our data. To most effectively capture context and meaning, text was coded at the “level of meaning”, allowing a code to encapsulate a line, sentence, or paragraph [58]. Interviews were coded using Delve qualitative data analysis software [59]. We followed an inductive approach of open coding to allow for unexpected, data-driven themes to emerge [47]. As codes were created, they were grouped with relevant codes to highlight similarities and begin developing overarching themes. These themes were identified and agreed upon through group consensus, although they continued to be refined as codes were added and removed throughout the analysis process. Four coders were trained by conducting inter-coder reliability (ICR) tests. Inter-coder reliability was measured using Krippendorff’s alpha, a measure of reliability in content analysis [60]. We underwent six rounds of ICR testing and reached an ICR rating of 0.83. Each round of testing involved independent coding of the same transcript by each coder, followed by comparison and discussion of coding differences. Transcripts from different participant groups (faculty, staff, graduate, and undergraduate) were selected for each round of ICR testing to ensure representation of the entire data set [61]. Changes were made to the codebook throughout this process. Each codebook iteration, including the final version, can be found on Open Science Framework [62]. After an inter-coder reliability score greater than 0.80 had been established, official coding of the transcripts commenced. Due to prolonged ICR training, only two of the four trained coders were available for the official coding process. Given time constraints and the high ICR rating, deviations were made to the preregistration, with each transcript being coded by one of the two remaining coders instead of undergoing double coding. Based on norms in the literature, it is not uncommon for ICR to be calculated on a subset of data (10–25% of data units) to establish satisfactory reliability and then proceed to single coding [61,63]. In line with this approach, we double coded 27% (6/22) of our transcripts before establishing ICR and moving to single coding. However, we acknowledge that single coding may introduce limitations in analysis due to potential inconsistencies in interpretation among independent coders, as well as increased bias from relying on a single analysis. To mitigate this, weekly meetings were held to address coding challenges and refine the codebook. The lead author refrained from coding to maintain an unbiased third-party perspective and acted as a moderator to help resolve disputes or clarify coding issues. In cases of disagreement, the lead author had the final say on coding decisions.

Thematic saturation was attained after analyzing 15 transcripts and was determined when no additional codes could be added to the codebook [64]. Once coding was complete, the themes were reviewed and refined one final time by visualizing their relationships with the codes and with each other through thematic mapping. The final codebook consisted of 47 main codes (not including sub-codes) that were grouped into 10 themes. An initial thematic map was created to visualize the codes and their relationships. Of those 46 codes, 13 appeared in 50% or more interviews. A revised thematic map was created using the 13 most saturated codes, and 5 main themes were assigned (S1 Fig). The full list of codes and definitions can be found in the codebook. Survey data was analyzed using descriptive statistics, including frequency counts and percentages.

Theme 1: Effectiveness of training program

Theme 2: Feedback on training sessions

Theme 3: Barriers to using the App

Theme 4: Use patterns

Theme 5: Suggestions for improvement

Limitations

Several limitations should be considered in interpreting this interview and survey feedback. In some cases, individual qualitative interviews of participants took place a few months after their initial use of the intervention. This gap may have biased participants’ reflections regarding their experience with HTC or its effectiveness. To address this, future iterations of the intervention could schedule post-treatment interviews more promptly after the completion of the program to gather participants’ more immediate impressions of the program, as well as some interviews later to determine longer term impacts of the program.

Furthermore, while the sample size is consistent with related studies in the literature [20,30], it likely does not sufficiently represent the experience of all participants, especially considering the potential for sampling biases among the subset of users who agreed to take part in the post-intervention interviews. Specifically, we did not achieve the desired sample size for faculty, and this underrepresentation may limit our ability to draw robust conclusions. To address this, future research should prioritize targeted recruitment efforts for faculty, as they were identified as the most challenging demographic to engage. Additionally, most participants identified as White, non-Hispanic, and female, which means our results may not accurately represent the experiences of individuals from other racial, ethnic, and gender backgrounds.

Additional limitations include factors that may impact engagement, such as technical issues and study compensation. Participants reported technical issues, specifically regarding notifications, which hindered their ability to receive app reminders and likely affected their engagement. To address this limitation, future research should consider expanding notification delivery options to ensure participants receive alerts in a more accessible and effective format. Additionally, while participants received a $50 compensation for their involvement, we cannot determine how engagement might have differed without this incentive.

Moreover, we did not screen users for elevated anxiety before participation, so some participants did not meet a standard cutoff for clinical anxiety. This could skew participants’ perception of the app’s effectiveness if they do not struggle with anxiety or if they struggle with a different mental health condition that the app is not targeted to treat. In the future, screening for individuals who are high in trait, social, or generalized anxiety could provide a more appropriate sample to study the program’s effectiveness (though we also value letting users choose whether they believe the program is something they want and need, as we did in the current study). We also did not require participants to stop other interventions while using HTC; this was done to increase external validity but we do not know how the experience of using HTC varies based on whether it is used on its own or as an adjunct to other forms of care. Finally, while the coders had varying levels of familiarity with HTC and considered diverse perspectives in the interviews, there is a possibility of reflexivity within the coding process.