Participants and procedure

In the context of the STORM trial, the program of Moving Stories was implemented in two consecutive study years, namely 2017–2018 and 2018–2019. In the Netherlands the study year generally starts around September and concludes around July. Moving Stories was offered from January to May in each study year. Fifteen high schools participate within the STORM trial, of which seven were randomly assigned to the experimental condition. Moving Stories was only offered at schools in the experimental condition.

Nearly one thousand (N = 982) eight grade high-school students across seven schools in the Southeast of the province North-Brabant participated in Moving Stories. Data from the STORM trial indicated that 11–13% of these eight grade students were screen positive for depressive symptoms as measured by the Children’s Depression Inventory 2, and 5–6% of them were screen positive on suicidal thoughts or behaviors.

Inclusion criteria were that they had to be enrolled in their second year of high school (i.e. 8^th grade in the US) and were sufficiently fluent in the Dutch language to participate in classroom sessions (which were in Dutch). During their second year of high school most student in The Netherlands are 13–14 years of age. However, there are few exceptions and we did not want to exclude any students. Therefore, a wide age range of 11–15 years was used for inclusion.

Intervention: Moving Stories

Moving Stories is intended to increase recognition of depression in self and peers, decrease stigma about depression and to promote help seeking behaviours. The intervention consists of a combination of a serious game and contact-based learning, comprising an introductory face-to-face lesson led by a researcher, single player game-app and a debriefing session led by a Lived Experience Worker (LEW) with experience of depression. The app Moving Stories was made available on the Apple App Store and Google Play app on 6 October, 2017 in both English and Dutch.

During the introductory lesson a researcher would explain to students how to download the app, what was expected of them and provided log-in codes. These sessions were delivered during a regular school class in a structured manner using a ready-to-use PowerPoint presentation to ensure every student received similar instructions. The aim of the game, as described to students, is that they will enter a virtual home where they will find a girl, Lisa, who is not feeling well. Students are asked to see if they can be of any help to Lisa. The term depression is not used during this introductory session. Although the game is played individually, the students are encouraged to help each other in identifying strategies that could be of any help to Lisa. Students were instructed to download the app during this introductory classroom session. After downloading the app, participants had to enter a passcode in order to receive access to the full app to ensure only adolescents from participating schools could actively use the app. As such, only participants that were enrolled in their second year of high school could participate.

Students then go on to play the game app for 10–15 minutes in the morning, between 7AM and 11AM, for 5 days. Students only had access to the app during the five days that they were asked to play, while they could still open the app students could no longer perform any actions or start over after the five playing days of their class was over. Students play the game individually, but simultaneous with their classmates and they are expected to do this in their own time. Schools did not provide the students with time to complete the game during school hours. In the game students learn to interact with a virtual girl, Lisa, suffering from depressive symptoms. The students perform actions in the game (over 5 days) which consist of household tasks and/or offering Lisa help, and have the ability to talk to Lisa in a structured manner using preprogrammed options conversation). The aim is that they learn effective strategies when helping someone with depressive symptoms or to become more able to seek help for themselves. Students were prompted to play the game as reminders from the app would pop-up when Lisa’s house would open (7AM) and an hour before the house would close (10AM). As with any mobile app, students had the ability to turn of pop-ups from the app. Feedback is provided each day by gaining or losing points indicating a relationship score indicating the quality of the relationship with Lisa based on the action they did in the game. In addition, they receive written text messages from Lisa telling the students how Lisa felt about each action they performed. In this way, students knew why they had gained or lost points. These messages were sent after school hours, from 3PM until 7PM.

After approximately seven days, students are brought back together again for the debriefing session. This debriefing session is an in-person classroom session lead by the LEW during a regular school class which are between 40 and 50 minutes long depending on the school. The LEW was aware of the length of each sessions beforehand. The LEW talks about his or her personal experience (approximately 10 minutes) and relates these personal experiences with depression to the game, which is then related to the students’ experiences with the game. The LEWs are required to discuss the five first aid strategies from the teen Mental Health First Aid (MHFA): (1) look for warning signs, (2) ask how their friend or peer is doing, (3) listen without judgement, (4) help them connect with an adult, and (5) your friendship is important [21]. The five strategies are reviewed and wrapped up as take-home messages for the students during the debriefing sessions. For this the LEWs can choose to use the PowerPoint slides that were made available to them during their training. In the PowerPoint slides show some example questions to launch discussions about the five aid strategies, but the LEWs were free to use these slides and adapt it to their personal preference as long as the five strategies are discussed. Moreover, the structure of the debriefing session was deliberately left open so LEWs could adapt it to their personal preference and make impromptu responses to the themes brought up by the students.

In the current trial, a teacher trained as gatekeeper and a researcher were always present during both classroom sessions. However, for future implementation only a teacher would need to be present, but there is no need for an additional researcher to be present as well. A teacher could take over the role of the researcher in the first introductory classroom session. Teachers’ presence is needed for the debriefing classroom session with the LEW as they are familiar with the students in the class and to ensure the LEW can focus on debriefing rather than keeping order in the classroom. Also, the teacher, trained as gatekeeper, will be able to guide students towards school mental health professionals. Further details about the content of Moving Stories and its development are described extensively elsewhere [32].

Recruitment and training of lived experience workers

The LEW were recruited via a post on the Facebook page of the Trimbos Institute (Netherlands Institute of Mental Health and Addiction) and GGZ Oost Brabant (Mental Health Service). Additionally, we contacted several lived experience organizations. Requirements were that the LEW had some prior lived experience with depression. LEWs were informed during the recruitment process that they were required to tell their personal story to students in their second year of high school and that they would need to complete a training beforehand. Any LEW that applied was invited to participate in the training (n = 37).

The LEWs first had to complete an 8 hr training. This training for LEWs was designed by the authors, and a researcher who is also a LEW and was an experienced trainer. The training sessions were provided by these researchers as well. At the start of the training, participants were asked to play the serious game to fully understand what the students have done prior to their debriefing session.

The training itself consisted of a theoretical part and a practical part. During the theoretical part (3hrs) the LEWs learned more about the STORM program, its delivery and the goals of the Moving Stories intervention, and what their role was in the Moving Stories intervention. In the practical training (5hrs), the LEWs had the opportunity to practice telling their personal story in 10–15 minutes, and introducing the five MHFA strategies. The discussion of the five strategies from the MHFA was the most important component of the debriefing session and the LEWs were required to discuss all five strategies. Additionally, LEWs had to prepare themselves for questions that students might ask regarding their experiences, difficult topics (such as students having suicidal thoughts) and the game.

The LEW were provided with training materials (PowerPoints used during the training session) and a PowerPoint that they were free to use or adapt to their liking for the debriefing session. This PowerPoint consisted of a few example questions that the LEW could ask students as well as a slide with the five strategies from the MHFA.

The training sessions took place from September to December in both years indicating a time lapse of one to six months between training and the debriefing session. In the meantime the LEWs were updated on planning by the researchers via email and WhatsApp, but any other contact between the researchers and LEWs was minimal during the training and debriefing sessions. By design, LEWs always led the debriefing sessions in the presence of a gate keeper-trained teacher and a researcher, with the researcher taking notes.

Data collection

Data was collected during the STORM trial. The medical ethics committee (Commissie Mensgebonden Onderzoek Region Arnhem-Nijmegen) in The Netherlands approved this study (NL61599.091.17). Informed consent from adolescents and their parents was obtained for the STORM trial. During the second trial year participation in Moving Stories was allowed on voluntary basis for students not participating in the STORM trial. At all times, students were informed that a researcher would be present during the debriefing sessions who might take anonymous notes.

Data that were collected are 1) data on preparing the delivery of Moving Stories, 2) quantitative data of offering Moving Stories, see Supplemental Materials, and 3) qualitative data on offering Moving Stories.

The data on preparing the delivery of Moving Stories included number of LEWs recruited and trained (and ensuing costs), number of trainings needed for classroom sessions, number of classroom sessions, and number of trained LEWs versus LEWs that actually provided debriefing session.

The quantitative data of offering Moving Stories included app usage data of the game and costs associated with debriefing sessions. App usage data was collected, anonymously, through a secure website. Data that were recorded include number of downloads, days played per student, and number of students playing each day. The app only collects fully anonymous data; players could not be identified as researchers were fully unaware of the identity of participating students and received only aggregated data.

In total, 49 debriefing sessions were provided by 11 LEWs. Notes were taken of 18 debriefing sessions to record what the LEW and students discussed during the debriefing sessions: 8 during study year of 2017–2018 and 10 during 2018–2019 with at least one debriefing session per school (see Table 1). These notes were kept by four researchers to ensure that topics noted were not the results of reporter bias.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Overview of spread of schools and educational level used for minutes.

https://doi.org/10.1371/journal.pone.0260224.t001

Data analysis

Descriptive statistics such as means and standard deviations were computed in SPSS Version 27 [33]. With regards to the qualitative data from the notes, first we checked in how many debriefing sessions the required five aid strategies were discussed by the LEW. Additionally, we checked whether common themes arose across the different debriefing sessions. These themes were not pre-determined as there was no specific theme the LEW had to discuss with regard to their personal experiences or depressive symptoms in general. Instead, they were free to discuss several aspects of both the game and their personal experience with depressive symptoms and were encouraged to interactively tailor the session to the interests expressed by the students. Discussed topics were considered a theme if a topic occurred in at least two of the recorded notes. A researcher read and coded all notes using MAXQDA 2020 Version 2.4.0 [34]. Quotes representative for themes were then exported into Excel.

Preparing moving stories

A total of 37 LEWs were trained divided over seven training sessions consisting of 6 participants on average. Overall, a total of 56 hours were spent on training sessions. The costs for each training are divided in costs of the trainer (hourly fee and travel expenses) and costs of the LEWs (travel expenses). The LEWs did not receive any compensation for time spent in training. However, opportunity costs in the Netherlands are valued at €15 per hr, indicating €4,440 total opportunity costs. There were no costs for renting the venue where training sessions were held. However, the costs of renting a training location are €350 per training, suggesting €1,400 for rent. The total costs of the trainer were €3748 and €50 for travel expenses. The travel expenses for the training of all the 37 LEW was €874 in total during the current trial.

Offering moving stories

The introductory lessons took approximately 30 minutes per class. A teacher was present during these introductory sessions, so they were aware of what their students would be doing and also to keep order in the classroom. For future implementation, a gate-keeper trained teacher would be expected to lead the introductory lesson. Teachers would need to be instructed first in order to provide these classes. This would take one hour and would cost €30–85. To ensure that each student was able to participate, regardless of smartphone possession, ten smartphones were purchased and made available for students at €1,750 total costs.

The total amount of students that were asked to participate in Moving Stories were 1,016 in study year 2017–2018 and 846 in study year 2018–2019 across the seven school locations. A total of 339 students downloaded the game during the study year 2017–2018 and 643 students during 2018–2019, indicating a coverage rate of 33.4% and 76% respectively across all second-year students. A number of students (n = 166, 16.9%) that downloaded the app did not complete any actions for the duration of their game in either year. It was noted that the number of students actively performing actions in the app gradually declined over the five consecutive days (see Table 2), but on average more than half of the participating students played the game on each of the five days. The average amount of days played by students was three days (M = 3.14, SD = .64).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Percentage of total students that completed actions on consecutive game days.

https://doi.org/10.1371/journal.pone.0260224.t002

In total, 49 separate debriefing sessions were provided by LEWs. Most sessions were provided by a single LEW. However, some LEWs preferred to provide classroom sessions with another LEW. This made them feel more comfortable as some of them were inexperienced in providing classroom sessions. They were matched with a LEW that did have prior experience. In total, four sessions were carried out by two LEWs instead of one. Not all trained LEWs were able to provide a debriefing session: 11 of 37 trained LEWs (29.7%) were able to provide debriefing sessions. Several LEWs indicated that classroom sessions interfered with other (volunteer) jobs. Most of them never responded to invitations to provide classroom sessions; and reasons for no-show remained largely unknown.

The costs for each classroom session were divided into an hourly fee and travel expenses for the LEW. Each LEW received an hourly fee of 30 euros for providing the debriefing sessions. The total costs for classroom sessions by LEW were €4,621, which included both the hourly fee and travel expenses. An overview of the costs in euros can be found in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Overview of costs in euros.

https://doi.org/10.1371/journal.pone.0260224.t003

Discussing the five first aid strategies of the teen MHFA program were a mandatory part of the debriefing classroom session and LEWs were asked to discuss all five first aid strategies with the students. We checked in the minutes of the 18 sessions whether all five first aid strategies were indeed discussed. This was the case for all 18 sessions.

Debriefing sessions

The notes that were taken during the debriefing sessions were also used to identify themes that were brought up by the students. It was noticeable from the classroom sessions that Lisa’s (the main character in Moving Stories) behavior and responses triggered different reactions from students and captured many experiences from the LEW. Therefore, the themes that were discussed in classroom sessions were diverse, but it is worth noting that the LEW had the freedom to highlight how their experiences with depression was (dis)similar to Lisa’s. Considering the heterogeneous nature of depression, we do not view this as a problem. Additionally, students’ interests were different across sessions as well. For example, some classes were really interested in school pressure and perfectionism, whereas other classes talked more about suicidal behavior in depression.

Six themes that were identified during our review of the minutes are highlighted below. The order in which the themes are represented in this article does not reflect the order and in how many sessions these themes were actually discussed; the n represents the number of sessions in which these themes occurred. Below, the themes are ordered from the broader themes down to more specific ones. Moreover, there were several questions by students that were similar across debriefing sessions, which are also presented in theme 7. We have added quotes from students regarding several themes in italics: Quotes have been translated into English. The original Dutch quotes have been added in brackets for clarification.

Principal findings

Among the 1880 eligible students 982 (52%) downloaded the Moving Stories app and 83% of them played the game for at least one day with an average of three days out of the five days. The LEWs discussed the five MHFA strategies with students in all recorded sessions. The notes of the debriefing sessions indicated that sadness-like symptoms were readily recognized as indicative of depression, whereas dismissive or grumpy behaviors were not. The students showed great interest in the personal stories of the LEWs and in more than half of the debriefing sessions one or more students shared their own personal experiences with mental health and depressive symptoms. This may suggest that mental health literacy was enhanced and stigma reduced among students. The total cost of implementing Moving Stories across seven high schools was €11,043 (US$ 12,367), of which €4,672 (US$ 5,232) was spent on the training of the LEWs. However only 11 out of the 37 trained LEWs (30%) carried out their training in practice and led a total of 49 debriefing sessions. This suggests that selection and training of LEWs should be improved.

Strengths and limitations

A strength of the study is that at each participating school at least one debriefing session was monitored to reduce selective sampling bias and to enhance external validity. Moreover, notes of the debriefing sessions were kept by four different note takers to reduce bias.

Several limitations are important to note. The study was carried out in a small region in the Netherlands (the South-East region of Brabant), and may therefore not be generalizable to other regions and countries.

The debriefing sessions were not tape-recorded and transcribed verbatim. Tape-recording would have legal implications as participants would be identifiable. Moreover, tape-recording has several disadvantages, namely when discussing personal issues, participants are less likely to discuss these freely [52,53]. Additionally, the recording of the notes was not completed in a pre-determined standardized format. This can be problematic as it could be that the minutes are somewhat skewed to outcomes deemed important by the reporter [54]. However, the usage of multiple reporters may counter-balance reporter bias. Moreover, using an unstructured form also meant that possible unexpected topics were recorded.

Furthermore, the current study did not tap into the experiences of the students, the LEW, teachers and school administrators on how they experienced the Moving Stories program. Questions of interest could include how they rated the game and classroom sessions, and whether they would play the game again, or would recommend it to a friend or colleague. This would help to determine the acceptability of the intervention. With regards to the acceptability another study found that over 90% of students would recommend the Moving Stories game and 66% would recommend the debriefing session to a friend [20].

Conclusion

Concluding, implementation of Moving Stories during the STORM trial to prevent depressive symptoms in high-school students was feasible. However, there were few trained LEWs that were able to provide debriefing sessions suggesting that selection and training of LEWs needs to be improved. The role of the LEWs is important because they managed to grab the full attention of students and were able to launch frank discussions about depression and create a space wherein students were willing to share their personal experiences. This suggests that Moving Stories was successful in enhancing mental health literacy and reducing depression stigma.