We describe initial pilot findings from a novel school-based approach to reduce youth depression and suicidality, the Empowering a Multimodal Pathway Towards Healthy Youth (EMPATHY) program. Here we present the findings from the pilot cohort of 3,244 youth aged 11–18 (Grades 6-12). They were screened for depression, suicidality, anxiety, use of drugs, alcohol, or tobacco (DAT), quality-of-life, and self-esteem. Additionally, all students in Grades 7 and 8 (mean ages 12.3 and 13.3 respectively) also received an 8-session cognitive-behavioural therapy (CBT) based program designed to increase resiliency to depression. Following screening there were rapid interventions for the 125 students (3.9%) who were identified as being actively suicidal, as well as for another 378 students (11.7%) who were felt to be at higher-risk of self-harm based on a combination of scores from all the scales. The intervention consisted of an interview with the student and their family followed by offering a guided internet-based CBT program. Results from the 2,790 students who completed scales at both baseline and 12-week follow-up showed significant decreases in depression and suicidality. Importantly, there was a marked decrease in the number of students who were actively suicidal (from n=125 at baseline to n=30 at 12-weeks). Of the 503 students offered the CBT program 163 (32%) took part, and this group had significantly lower depression scores compared to those who didn’t take part. There were no improvements in self-esteem, quality-of-life, or the number of students using DAT. Only 60 students (2% of total screened) required external referral during the 24-weeks following study initiation. These results suggest that a multimodal school-based program may provide an effective and pragmatic approach to help reduce youth depression and suicidality. Further research is required to determine longer-term efficacy, reproducibility, and key program elements.
Citation: Silverstone PH, Bercov M, Suen VYM, Allen A, Cribben I, Goodrick J, et al. (2015) Initial Findings from a Novel School-Based Program, EMPATHY, Which May Help Reduce Depression and Suicidality in Youth. PLoS ONE 10(5): e0125527. https://doi.org/10.1371/journal.pone.0125527
Academic Editor: Nori Takei, United (Osaka U, Kanazawa U, Hamamatsu U Sch Med, Chiba U and Fukui U) Graduate School of Child Developmen, JAPAN
Received: December 10, 2014; Accepted: March 10, 2015; Published: May 14, 2015
Copyright: © 2015 Silverstone et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: Alberta Health Services Grant-002479 provided funding for this study. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Evidence from a variety of sources suggests that depressive disorders occur in approximately 10% of youth , , that the frequency of several mental disorders may be increasing in youth , and that many disorders in adulthood are first evident during youth . Accompanying the high rates of depression and other mental health disorders, suicide rates are also high in youth [4–6]. Indeed, a recent large U.S. survey reported that over 7% of High-School students had attempted suicide in the previous 12 months, of which one-third required hospital treatment , while a large European study reported that 4% of youth have attempted suicide . One way of addressing this may be to utilize school-based prevention programs , although more research is needed in order to draw definitive conclusions regarding the best approach to achieve this [10–14]. Among suggestions for the options to address youth suicidality, it has been suggested that a comprehensive multimodal approach may be most effective . Such multimodal approaches may involve a wide variety of interventions such as increasing mental health information to students , screening , and resiliency programs based on cognitive-behavioural therapy principles, although there is a relative dearth of intervention studies .
It should be noted that one problem in research in this area is that scales may not accurately predict those who go on to carry out acts of self-harm , although a recent study suggests that two questions regarding suicidal intent contained within the 9-item patient health questionnaire (PHQ-9) may be able to do this . Furthermore, in youth, the relationship between rates of self-harm and suicide rates is complex. While it is likely that less than 5% of youth who carry out self-harm will subsequently have a completed suicide , it is clear that those at greatest risk of completing suicide are individuals who have either previously carried out self-harming behaviours and/or who have depression , . Therefore, one method to reduce youth suicides may be to reduce depression rates in this group.
Many of the school-based programs which have shown benefit are based upon cognitive behavioural therapy (CBT) principles. One such approach is the Penn Resiliency Program [22–25], which has been studied for over 20 years. More recently an updated version of this program, called “OVK” (so-called because of its Dutch name “Op Volle Kracht” which is translated as On Full Power), has been created in collaboration with the original developers of the Penn Resiliency Program . The full OVK program is delivered during regular school lessons, and consists of 16 sessions. The first 8 lessons are CBT focused and the second 8 lessons are on aspects of social and educational learning. Several studies have found beneficial outcomes on depression rates with OVK in both the short-term and longer-term , , but positive results have not been found in all studies .
Several groups have suggested that the most appropriate place for interventions to reduce depression rates in youth is by intervention in schools, either for all individuals (“universal” programs) or for specific “high-risk” populations [30–34]. Universal prevention may only produce small overall changes , although a Cochrane review of over 50 studies concluded that “universal depression prevention programs may prevent the onset of depressive disorders compared with no intervention” . The other major approach to address youth depression rates is to carry out more “targeted” approaches for those students identified as “high-risk”. This is because the large number of individuals who do not need interventions mean that many students need to be screened to identify each case . Multiple different approaches have examined potential risk factors that can identify potential high-risk youth, with most utilizing existing depression scales to screen youth followed by a CBT-based program to treat depression [38–45]. However, it is likely that an additional 3–7% of youth have “sub-threshold” depression (i.e. depressive symptoms that do not meet diagnostic criteria) , , and many of these have active suicidal thoughts . Thus, one of the issues with a “high-risk” approach is that if a high score on a depression screening measure is used as the only way to determine those at risk, it is likely to miss many youth who have active suicidal thoughts. Combining both a universal and a high-risk approach may potentially have the greatest impact , , although studies to date have not found such benefits , . It is possible that other factors, such as family relationship support, may be important in the outcome of these interventions .
To help both prevent and treat depression in youth there is an increasing focus on technology approaches , , both a universal basis , and to treat youth who are depressed [55–58]. The results are generally positive , although it is unclear how they compare to in-person delivery of CBT . Overall, one of the most consistent findings from these technology-based studies is that a “guided” approach, meaning that there is a trained individual acting as a “guide” for the user at each stage of these programs, consistently improves both uptake rates and outcomes compared to individuals just using such programs on their own , , . Consistent with this are findings that simply disseminating a self-directed internet-based intervention to a school population is not effective, even if significant steps are taken to reduce barriers to use .
Findings to date are that there exists no single program that has demonstrated consistent, and longer-term, effectiveness on reducing suicidality and depression in youth, although the need for this remains clear . The present publication describes the first results from the piloting of a novel school-based approach to reduce depression and suicidality in youth. This program is called Empowering a Multimodal Pathway Towards Healthy Youth (EMPATHY). It utilizes a multimodal approach including a universal CBT program for some, screening of all students to identify a potentially higher-risk group, rapid intervention, and a guided internet-based CBT approach for those identified as depressed or at high-risk. Our primary hypothesis was that this approach would reduce both depression and suicidality, and our secondary hypotheses were that this program would also reduce anxiety and use of drugs, alcohol, and tobacco (DAT) while increasing self-esteem and quality-of-life.
Materials and Methods
The detailed TREND checklist detailing how the study addressed methodological issues is contained within the supporting information (S1 TREND Checklist). All aspects of this program were approved by the Health Research Ethics Committee of the University of Alberta on September 23, 2013, who approved all informed consent forms and all other aspects of the study. Subsequent amendments to the original protocol and consent letters were approved in January of 2014, with some additional minor changes to the informed consent letter subsequently being approved in February 2014, prior to the first student starting. The full original protocol submitted to the ethics committee, as well as changes approved in January 2014, is contained within the supporting information (S1 Protocol). All parents received written information that all school students would be offered screening as a standard part of their health classes, with information about this, including information sessions. However, specific written consent for this was not required (as it wasn’t within the school district for any other school-wide activity), although parents were able to have their children not be included in screening, and student participation was voluntary. For any individual student who was to be offered additional intervention specific written informed consent was required from both parents/guardians and the student. The first student was screened in February 2014 and the follow-up screening was completed by the end of June 2014.
This study is registered with ClinicalTrials.gov Identifier: NCT02169960. Although an application for this was completed at the time of the ethics approval, due to an unfortunate administrative oversight, the actual submission to the registration database did not occur until July 2014. This omission was corrected as soon as it was recognized. The authors confirm that all ongoing and related trials for this intervention are now registered.
In terms of study design, from a scientific viewpoint, the most rigorous findings would have come from a design in which a group of students were randomized to be screened, but not to have any interventions, compared to those who were screened and did have an intervention. Indeed, such an approach has been carried out in many studies looking at depression rates in students. This could have been done at the class level or school level. However, given that we were specifically screening for youth who were actively suicidal, having a group that was screened and found to be suicidal but then not offering an intervention would clearly have been an unethical approach. For this reason we determined that using each student as their own controls was an appropriate study design at this stage of the research process, and this was approved by the Health Research Ethics Committee.
Study location and timing
We engaged with Red Deer Public Schools in May of 2013 and a widespread multi-sectoral community partnership approach was implemented involving education services, primary health care, specialist mental health care, social services, as well as others involved with youth (such as the police services). Following extensive consultation with large numbers of interested parties, researchers, and community groups, implementation started in February 2014 with follow-up data being collected by June 2014. The present study reports on these initial findings, which is the first pilot of this comprehensive program. Longer-term follow-up studies are planned, and modifications of the program are anticipated based on the present study experiences and findings. It should be emphasized that using all schools in a region (and not selecting specific ones) is a unique approach that we are not aware has been used in previous studies.
Novel multimodal approach
Whenever possible existing screening, measurement, and intervention tools were used. To reduce costs, and improve potential future ease of uptake, only such tools that were useable at no cost were considered, and they also had to be able to be used electronically on a tablet. Additionally, all staff hired in the schools to implement the program had to have some experience working with youth, and many had an undergraduate degree. However it is important to note that they were specifically not highly trained individuals (such as psychologists or teachers), as it was felt that it would not be feasible for widespread expansion if such highly trained (and expensive) staff were required. These individuals were termed “Resiliency Coaches”, and each was attached to a specific school, becoming integrated with the staff at that school. It was made clear to all study staff and school staff that the Resiliency Coaches were not therapists or counsellors, and would not act in those roles. As noted in previous studies, obtaining “buy-in” from schools and students as well as other community members can be difficult, but is important for the success in any such school-based programs. For this reason, among others, multiple approaches were undertaken to try and increase the chances of success for the EMPATHY program, with such approaches maximizing technology whenever possible (Table 1).
The allocation of students involved in this study is shown in Fig 1, including those offered interventions after screening. All screening was carried out on dedicated electronic tablets during a 45-minute period within a standard classroom session. Students logged on using only their student IDs. Electronic data collection complied with all privacy and security requirements. The individual scales were presented to students in a randomized order, and no data was stored on the tablets as they were directly linked to the school intranet. The data was stored in a dedicated and secure database, but was immediately available to the study staff to identify those at risk.
The figure shows how many students entered the study and (on the left-hand side) how many were allocated to the CBT treatment as well as (on the right-hand side) how many completed both baseline and 12-week follow-up ratings and formed the study population.
All of the screening tools met the criteria for being free, short, and easily adapted to the “app” used on the electronic tablet. The exact questions asked, and the scores associated with each, are shown in Table 2. Note that there are several minor variations from some of the published scales, and these are described below, as we were not using these scales to diagnose students, or to indicate thresholds of abuse, but to indicate areas of concern experienced by each individual. This means that the scores from each of the components cannot always be directly compared to previous studies. It is also important to note that we did not “pilot” these changes or validate them in any way. Therefore, we recognize that this approach means that where we have changed the original scales they cannot be directly compared to scores from other studies using the original unmodified scales. Each of the scales is described below.
For depression screening, we utilized questions from the 9-item patient health questionnaire (PHQ-9) , adapted for adolescents (PHQ-A) which has been well validated in youth [63–65]. As part of this original questionnaire, one question asks about frequency of thoughts in the past month about wanting to be dead. If the answer is positive to this question, then an additional question is asked about active suicidal ideation. These two questions have previously been shown to be predictive of subsequent suicidal behaviour . To further focus on the risk of suicide, we also separated the question that asked about self-harm into two questions (Table 2). Furthermore, since it is well recognized that the most accurate predictor of actual suicide is a history of a previous suicide attempt , , we added an additional question that asked about a history of this (Table 2). In the present study, therefore, there were a maximum of 13 questions about depression, although the two additional questions were not asked of all the population and, therefore, they were not counted in the depression score (but were used to identify suicide risk—see below).
We utilized the questions from the anxiety section of the Hospital Anxiety and Depression Scale  to screen for anxiety symptoms (Table 2). It has previously been used to measure anxiety in youth in several studies [67–69].
To measure quality-of-life we utilized questions from the KIDSCREEN-10 (Table 2). This is recent measure of well-being and quality of life developed internationally for children and adolescents aged 8 to 18 years old . It is a short version of the KIDSCREEN-52 and KIDSCREEN-27 instruments and has demonstrated validity with this population . In the KIDSCREEN-10 items are measured usually on a 5-point Likert scale and summed into a score that is transformed into standard values based upon data from an international survey sample and range from -16.06 to 57.77 . Because we used a composite score in this study, only the raw data was used, namely the scores from all 11 questions (Table 2).
In the program we also measured the use of drugs, alcohol, and tobacco (DAT) (Table 2). To do this we utilized the questions contained in the CRAAFT questionnaire, which is well validated for use in youth , . However, we also used answers to all 9 questions, as this indicated use of DAT, and not simply the answers to the 6 subsequent questions which indicate risk of abuse (Table 2). Additionally, we wanted to be aware of tobacco use in this population, including smokeless tobacco use, and for this reason we asked two additional questions to those of the CRAAFT. These were “during the past 12 months did you smoke tobacco products?” and “during the past 12 months did you use smokeless tobacco products?”. For this item, therefore, there were 11 separate questions each of which was scored equally (Table 2).
Identification of the “actively suicidal” group.
After screening, the results were rapidly available. The first variable examined was suicidal risk. From the responses to the three questions, students were placed in a category of none, low, medium, or high suicide-risk (Figs 2 and 3). The study staff identified the students at greatest risk based on this algorithm (those in the high suicide risk group and medium suicide risk group). A priori we determined that we would consider those individuals who were either in the high suicide risk group or the medium suicide group to comprise those who were “actively suicidal”. These students were then seen for an interview by a trained member of staff (usually somebody familiar to them) for a 1-hour semi-structured interview, which included a more detailed assessment of suicide risk and an open discussion of issues that were relevant to the individual student. Immediately following these interviews the parents (or guardian) was contacted and informed about the findings and concerns regarding the safety of that student. At this time a safety plan was agreed upon, including potential referral to Primary Care, Emergency, or specialized Mental Health services, as well as having information provided about the subsequent plans (to offer a guided internet approach with CBT). An informed consent sheet was provided to the student and parent/guardian regarding the guided CBT being offered, and both written parental and student consent were required to take part. Unfortunately, because of time and staffing constraints, each parent/guardian and the student only had 5 working days to return signed informed consent sheets to be eligible for the additional CBT programs.
The figure shows how scores on the three questions regarding suicide risk determined if the student was in the high suicide risk group or the medium suicide risk group. Together, these students were considered the “actively suicidal" group, and were interviewed individually within 2 days for the high suicide risk group or within 5 days for the medium suicide risk group.
The figure shows how scores on the three questions regarding thoughts of suicide determined if the student was in the low suicide risk group. These students were not interviewed individually, but were offered participation in a guided internet-based cognitive behavioural therapy (CBT) program.
Each of the three suicide-risk groups was treated differently. All those who were in the high suicide risk group were interviewed within 2 working days (but in over 90%, this happened within 24 hours of completion of the screening). For those in the medium suicide risk group, the same process occurred, but they were seen with within 5 working days (Fig 2). If they were in the low suicide risk group (Fig 3), they were not interviewed but were offered the opportunity to take part in one of two guided internet-based CBT programs. The first was focused on depression and was used by over 95% of students who took part (THIS WAY UP) . The second program was offered to those High Risk students who had addiction issues (Breaking Free) , but very few students chose to utilize this. Both groups were offered as many supervised sessions as they required to complete these programs, although the actual number each student completed was not recorded in this pilot
Identification of high risk group (Top 10%).
In addition to those who had a suicide risk of any type (all of whom were offered the opportunity to take part in THIS WAY UP), we also identified the top 10% of those who had the highest overall scores on the combined EMPATHY scale score (see below). This was based on previous studies suggesting that up to 10% of students aged 10–18 may have significant symptoms of depression , . Additionally, there is strong evidence that those who have “sub-threshold” depression are at raised risk of self-harm and suicide attempts , , . It is also recognized that depression in youth is frequently associated with poor social performance, social withdrawal, increased use of drugs and alcohol, and low self-esteem . For these reasons we determined that it may be important to intervene in those who may be at high-risk, as determined by these factors, and not simply by their scores on a depression scale. For this reason we determined that is this pilot examination of the EMPATHY program we would consider students to be in the High Risk group either if they were actively suicidal or if they scored in the top 10% for combined scores for their specific grade. It was not clear from previous research how much overlap there would be between the groups who had a raised suicide risk and those who were not actively suicidal but had a large number of other symptoms. It should be recognized firstly that that we recognized that a combination of similar scores has not been previously used to determine risk, and secondly that determination of 10% of those with the highest scores belonging in the High Risk group was arbitrary, and was not backed by evidence. We have a priori planned to carry out subsequent analyses to compare outcomes using more standard measures (such as depression scale scores alone, and using a different percentage for those at risk). This will allow us to make more evidence-based decisions in the future. We were also uncertain how many students in the High Risk group who were offered the guided CBT programs would take part, as previous research has suggested that many would not .
Combined EMPATHY scale score.
Other studies have demonstrated that there is a large “invisible” group of youth who are at risk , and we planned to determine if the use of multiple indicators may help identify them. For this reason we created a summary score composed of information from all 5 of the scales utilized in screening (Table 2). In doing so, equal weight (20%) was given to each of the 5 scales being used in determining those at highest risk. This was because there is not sufficient evidence to reliably suggest the relative importance that should be given to the role of low self-esteem, the use of DAT, or the presence of anxiety in the determination of risk for subsequent suicide. While it may be argued that greatest importance should be given to scores on the depression scale, we had already identified those who were actively suicidal and felt there is not currently enough evidence to determine how much additional importance should be given to the other measures. By using a composite score (the “combined EMPATHY scale score”), which includes all 5 areas being studied, we avoided risking missing a key component, but retained the ability to subsequently determine which questions (listed in Table 2) were most useful in determining outcomes, as well as the correlations between each scale. Thus, each scale was scored with every question having equal weight. The score for each scale was then “adjusted” to a maximum of 20. For example, if the original score on the self-esteem scale (Table 2) was 15 out of a possible 30, then the “adjusted” score for the student was 10 out of 20. The “adjusted” scores for each of the scales were then totaled to provide a score out of potential maximum of 100, and this was termed the “combined EMPATHY scale score”. The combined EMPATHY scale score was therefore calculated by adding the various scores, adjusted as follows: raw depression score x 20/33 + raw anxiety score x 20/21 + raw score for DAT x 20/11 + [20 –(raw self-esteem score x 20/30)] + [20 –(raw quality-of-life score x 20/44)]. Note that higher raw scores for depression, anxiety, and DAT indicated more problems, whereas for self-esteem and quality-of-life higher raw scores indicated better outcomes, which is why the scores for the latter two were adjusted appropriately in the combined EMPATHY scale score calculation.
From the above calculations the 10% of students with the highest combined EMPATHY scale scores in each Grade were identified, with a goal that they would be interviewed within 14 days of screening, and their parent/guardian would also be contacted (in the identical manner as those who were actively suicidal). They were also offered the opportunity to take part in guided internet-based CBT programs.
All students in Grades 7 and 8 received a modified version of a CBT program designed to reduce depressive symptoms, OVK [26–29]. It is usually given as a set of 16 sessions, each during a 45-minute classroom lesson period. The first 8 sessions are focused on CBT principles and the second 8 sessions are focused on social and educational learning. However, for reasons of time availability in this pilot version of EMPATHY only the first 8 sessions, i.e. those on CBT, were given. Given this, we were aware that any findings could not be compared to previous use of this program. It was administered by the Resiliency Coaches who received a 3-day training course on this program.
The primary statistical method was a paired design, in which each student who completed both baseline ratings and follow-up ratings was their own control. As the data showed evidence of non-normality, a non-parametric test was carried out to compare the differences between the mean scores at baseline and 12-week follow-up. This involved Wilcoxon signed-rank test (paired), unless otherwise specified, which is a non-parametric statistical test for testing hypotheses on medians. Although not shown in the present publication, median values for all items are available upon request to the corresponding author.
Statistical analysis was carried out on an “intention to treat” basis utilizing R, version 3.1.0 (R Foundation for Statistical Computing, Vienna, Austria) and Stata/IC 13.1 for Windows (StataCorp, College Station, Texas, USA). Correlations were calculated using IBM SPSS Statistics 20.0 (Chicago, Illinois, USA).
The study was designed to determine potential changes in depression scale scores and in suicidality. From previous studies, it was determined that there may be a 5% decrease in both these scores. To detect a statistically significant reduction of this size would require a sample size of 65 students in a 1-sided sample size calculation completed using the G Power calculator, version 3.1, with α = 0.05. Given our sample size of 3,500 students, of which we estimated a minimum of 4% (i.e. 140 students) would be depressed with a similar number being suicidal, this study was adequately powered.
The study was carried out in 5 schools in the Red Deer Public School system—all 3 of the Middle Schools (Grades 6, 7, and 8—average age 12.3 years, range 10–14) and both High Schools (Grades 9, 10, 11, and 12—average age 15.7 years, range 13–19). In the first screen (baseline), a total of 3,244 students took part (Fig 1), of whom 1,568 were female and 1,676 were male. In the second screen, a total of 3,228 students took part, of whom 1,550 were female and 1,678 were male. The total study population was 3,682 (Fig 1) although, only 2,790 students took part in both the baseline and 12-week follow-up assessment (i.e. in addition to the 2,790 students who completed both assessments, there were 454 students who only completed baseline assessments and 438 who only completed the follow-up assessments). Unless otherwise stated, all of the results reported are on the sample of 2,790 who completed both assessments, and were their own controls. Both mean ± standard deviation are shown.
High risk groups
As previously noted, in the present piloting of the EMPATHY program we arbitrarily determined that we would include both those who were actively suicidal, as well as those who were in the top 10% of combined scores, as being in the High Risk group. Of the total 3,244 screened at baseline, 503 students were identified as being in the High Risk group, mostly because they had some suicidal symptoms (n = 378), of whom 220 (58%) also scored in the top 10% of combined scores. In the present study there were an additional 125 students who did not have significant suicidal symptoms but who scored highly on other ratings and were included in the High Risk group.
Combined EMPATHY scale scores.
The combined EMPATHY scale scores represented overall scores based on all of the 5 measurements made, and decreased from a mean of 25.96 ± 15.05 to 23.99 ± 14.81 (an 8% decrease) which was highly statistically significant (p<0.0001) using Wilcoxon signed-rank test (paired). There were statistically significant differences seen in all but Grade 8, and in 4 of the 5 schools taking part in the study (Table 3).
In terms of correlations between each of the subscales of the combined EMPATHY scale score, the depression, anxiety, self-esteem, and quality of life scores were all significantly correlated, although the DAT score was only weakly significantly correlated with the quality-of-life score but no other scales (Table 4).
In addition to examining changes in the group who consisted of the highest 10% of scores on the combined EMPATHY scale score, we also examined the group who comprised the top 25% of combined EMPATHY scale scores (n = 698) (Table 5). In this group of more symptomatic students, there was a highly significant reduction in the combined EMPATHY scale score (13.4%), depression score (28.9%), and anxiety score (17.7%). There were also small, but statistically significant, decreases in the scores for the use of DAT, self-esteem, and quality-of-life (Table 5).
Impact of guided internet-based CBT interventions.
A total of 503 students were offered guided internet-based CBT interventions. Of these, 151 students took part (30%), with over 90% completing THIS WAY UP (focused on depression). There were highly significant improvements compared to baseline with deceases in the combined EMPATHY scale score (a 14% decrease), depression scores (a 28% decrease), and anxiety scores (a 12% decrease) (Table 5). There were also small, but statistically significant decreases in scores for self-esteem and quality-of-life. Importantly, the mean change from baseline for both combined EMPATHY scale score and depression score in the 151 who took part in the CBT treatment was significantly greater than in the other members of the High Risk group who did not take part in the guided internet-based CBT treatment programs (n = 352) (Fig 4). There were no statistically significant differences between these two groups in terms of age, sex, or any baseline scores on any of the measures, and there were no statistically significant differences (p = 0.52) in those who responded and were in Middle Schools (Grades 6, 7, and 8: n = 87) and those who were in High Schools (Grades 9, 10, 11, and 12: n = 64).
There were a total of 409 individuals who were identified as being in the High Risk group who had ratings at both baseline and at 12-week follow-up. Of these 151 took part in the guided internet-based cognitive behavioural therapy (CBT) program. The figure shows the mean change in score for the EMPATHY scale and each of the 5 subscales (depression, anxiety, drugs, alcohol, and tobacco (DAT), self-esteem, and quality-of-life (QoL)) for the 151 who took part compared to changes in score in the 258 who did not. It can be seen that the group who took part had significantly better reductions in EMPATHY and depression scores. There were also smaller, but significant, reductions in self-esteem and quality of life in the group who had the CBT program. The following symbols indicate the degree of statistical significance, * p<0.05, ** p<0.001, *** p<0.0001
At baseline, the scores for the total of 3,244 students who completed the depression scales varied by Grade (Fig 5). The scores for depression were grouped according to ranges that have previously been used for indication of possible diagnosis of depressive disorder, although it is recognized that with the changes from the original PHQ-A these results are not directly comparable. The ranges used in previous studies are a score of 0–4 indicating not depressed, a score of 5–9 indicating the presence of minor symptoms, a score of 10–14 which may indicate the presence of a mild depressive disorder, a score of 15–19 which may indicate a moderate depressive disorder, and a score of 20 or more which may indicate the presence of severe depression. In the present study, it can be seen that the distribution of scores varies across the Grades. Scores compatible with mild depression are seen most frequently in Grade 9, whereas scores compatible with severe depression occurred relatively equally across all Grades from 7–12 (Fig 5).
The distribution of depression scores is shown for all 7 Grades (average ages at start of study are 11.3 for Grade 6, 12.3 for Grade 7, 13.3 for Grade 8, 14.3 for Grade 9, 15.3 for Grade 10, 16.4 for Grade 11, and 17.4 for Grade 12). Although not directly comparable to previous studies, as the depression score was not directly comparable to PHQ-A scores, and do not have diagnostic validity, we indicate the distribution of students into the ranges that in previous PHQ-A studies have been proposed to indicate the presence of a depressive disorder. These are not depressed (0–4), minor symptoms (5–9), mild depression (10–14), moderate depression (15–19), and severe depression (>20). It can be seen that the distribution of symptoms shows that scores compatible with mild depression are seen most frequently in Grade 9, whereas scores compatible with severe depression occur relatively equally across all Grades from 7–12.
Comparing baseline ratings in the 2,790 students who completed both measurements to those at the 12-week follow-up, the mean total depression score decreased significantly from 3.71 ± 3.86 to 3.17 ± 3.51 (a 15% decrease), which was highly statistically significant (p<0.0001 using Wilcoxon signed-rank test for paired samples). There were statistically significant reductions in all Grades and in all schools (Table 3).
For the 2,790 students who completed both baseline and follow-up ratings at 12-weeks, there was a marked reduction in the number of students who were actively suicidal, which we had a priori defined as those who were in either the high suicide risk group or the medium suicide risk group, from 125 at baseline to 30 at follow-up (Table 6). There were also reductions in the assessed level of suicide risk for many of these students, and out of 71 students who were in the high suicide risk group before the intervention, 41 of them transitioned to having no suicide risk at follow-up, while out of the 54 students in the medium suicide risk group at baseline, 35 transitioned to the no suicide risk group at follow-up (Table 6).
In addition to the analysis of students who completed both ratings, there was also a smaller reduction in the numbers of students who were actively suicidal in the total student group. Thus, at the baseline rating among the 3,244 students a total of 125 were actively suicidal compared to 104 students who were actively suicidal at the 12-week follow-up completed by 3,228 students (Table 7).
At baseline, the scores for the total of 3,244 students varied by Grade (Fig 6). It can be seen that more than 30% of students in Grades 9, 11, and 12 had a significant number of anxiety symptoms, but even in younger Grades more than 15% of students in every grade had frequent symptoms of anxiety. This finding is consistent with previous research regarding the frequency of anxiety symptoms in youth.
The distribution of anxiety scores is shown for all 7 Grades (average ages at start of study are 11.3 for Grade 6, 12.3 for Grade 7, 13.3 for Grade 8, 14.3 for Grade 9, 15.3 for Grade 10, 16.4 for Grade 11, and 17.4 for Grade 12). While these scores do not have diagnostic validity, we indicate the distribution of anxiety scores that in previous studies have been proposed to indicate the presence of an anxiety disorder. These are not anxious (0–7), possible anxiety disorder (8–10), and anxiety disorder (>11). It can be seen that there more than 30% of students in Grades 9, 11, and 12 how scored 11 or more, suggesting frequent anxiety symptoms. Even in the younger Grades more than 15% of students in every grade had frequent anxiety symptoms. This finding is consistent with previous research regarding the frequency of anxiety disorders in youth.
Comparing baseline ratings in the 2,790 students who completed both measurements to those at the 12-week follow-up, the mean total anxiety scores decreased significantly from 6.97 ± 4.72 to 6.22 ± 4.83 (an 11% decrease), which was highly statistically significant (p<0.0001 using Wilcoxon signed-rank test for paired samples). There were statistically significant differences in all Grades and in all schools (Table 3).
Drugs, alcohol, and tobacco (DAT) scores
At baseline, the scores for the total of 3,244 students who completed DAT scales varied widely by Grade, with only 29% of the Grade 12 students never having abused DAT compared to 95% of the Grade 6 students (Fig 7). It can also be seen that there was a consistent relationship between Grade (and age) and use of DAT, with the greatest use being in the higher Grades (i.e. older age groups).
The distribution of scores for the use of drugs, alcohol, and tobacco is shown for all 7 Grades (average ages at start of study are 11.3 for Grade 6, 12.3 for Grade 7, 13.3 for Grade 8, 14.3 for Grade 9, 15.3 for Grade 10, 16.4 for Grade 11, and 17.4 for Grade 12). While these scores do not have diagnostic validity, we indicate the distribution of these scores. It can be seen that at all scores the distribution is that the oldest youth have more use of drugs, alcohol, and tobacco, and have been involved in more significant activities that may put them at risk. It is consistent with previous studies that less than 30% of youth in Grade 12 have not used drugs, alcohol, or tobacco in the previous 12-months, compared to more than 95% of those in Grade 6.
In marked contrast to the findings for depression and anxiety, when comparing baseline ratings in the 2,790 students who completed both measurements to those at the 12-week follow-up, there no statistically differences between the mean score at baseline (mean 1.90 ± 3.59) compared to that at 12-weeks (mean 1.86 ± 3.55) (p = 0.92). There were also no statistically significant changes seen across any of the school Grades, or in any of the schools (Table 3).
Comparing baseline ratings to those at the 12-week follow-up there was a statically significant decrease in mean self-esteem scores. Comparing baseline ratings to those at the 12-week follow-up in the 2,790 students who completed both measurements there was a small (5.0%) decrease in mean total self-esteem scores from 7.32 ± 3.18 to 6.95 ± 3.25, which was statistically significant (p<0.0001 using Wilcoxon signed-rank test for paired samples). Given the improvement in depression and anxiety scores, this was unanticipated. This decrease in self-esteem was seen across most Grades and in 4 of the 5 schools (Table 3).
Similar to the findings for self-esteem, comparing baseline ratings to those at the 12-week follow-up in the 2,790 students who completed both measurements, there was a statically significant decrease in mean quality-of-life scores. Comparing baseline ratings to those at the 12-week follow-up, there was a small (3.6%) decrease in mean total self-esteem scores from 6.39 ± 3.36 to 6.16 ± 3.45, which was nonetheless statistically significant (p<0.0001 using Wilcoxon signed-rank test for paired samples). Given the improvement in depression and anxiety scores, again this finding was unanticipated. This decrease in quality-of-life score was only seen in two Grades but in 4 of the 5 schools (Table 3).
Role of OVK
In the study, two Grades, 7 and 8 (mean ages at the start of the study for students in these Grades were 12.3 and 13.3, respectively), also received a shortened version of OVK. In trying to determine whether this had a significant impact upon the depression scores, we found no differences in the mean change in depression scores between these two grades (n = 737) and those of either Grade 6 students (n = 368) or Grade 9 students (n = 460). For example, the mean decrease in the depression score for Grade 7 and 8 students who had OVK was 0.36 ± 3.1, whereas for those in Grade 9, it was 0.52 ± 3.1 (p = 0.66). These findings do not support our initial supposition that the addition of the modified OVK program would lead to further additional improvements in depression scores. However, as previously noted, given that this was only a limited version of the usual scale this finding should not be used to consider the effectiveness, or otherwise, of the OVK intervention. Further studies using the full scale are required to determine this in a similar study population.
Our primary hypothesis was that this approach would reduce both depression and suicidality, and our secondary hypotheses were that this program would also reduce anxiety and use of DAT while increasing self-esteem and quality-of-life. While our primary hypotheses was supported, many of the secondary hypotheses were not.
The results from the present study show a highly significant decrease in depression scores (approximately 15%), seen in all age groups and in all schools, and the impact appeared greater in those who had more symptoms at baseline. There were also significant decreases in anxiety scores in all Grades and in all schools. We also identified a High Risk group, who were those students who were either suicidal and/or were in the top 10% of total EMPATHY scores in each individual school. Those in the High Risk who took part in subsequent guided internet-based CBT programs did better than individuals with a similar level of severity who did not. There appeared no baseline differences between these two groups, but they were not randomized into these two groups so it is possible that this finding is due to other, currently unclear, reasons.
Possibly more importantly, there were significant reductions in the number of students who were actively suicidal. Thus, in the actively suicidal students at baseline who completed both assessments, 76 (61%) of them were in the no-risk group at the end of the 12-week follow-up period, while the number in the actively suicidal group (those in either the high risk suicide group or the medium risk suicide group) decreased from 125 to 30.
There were no significant changes in the use of DAT scores. However, this latter finding was not surprising as the questions relate to use over the previous 12-months and the follow-up was only during the previous 3-months. Longer-term follow-up will be needed to determine if there are any DAT changes. The total combined EMPATHY scale score also decreased significantly in many schools and Grades, and with the high level of correlation between all sub-scales (apart from the DAT scores), this might support the use of such an aggregate score in future studies.
In terms of the design of this study, we used a multimodal approach incorporating several components, each of which individually has been reported as having some benefits towards improving depression. Additionally, we utilized technology whenever possible and this allowed ultra-rapid identification of the high-risk group. We also carried out the program in a complete geographical region, in contrast to many previous studies which have sometimes included schools which self-selected to be involved. It is therefore clear that the EMPATHY program is somewhat different to those previously studied.
Caution in terms of the sustainability of the improvements seen in depression is required, as other studies have previously shown improvements over a 3-month period that are not sustained over longer time periods . In terms of a more comprehensive approach, the study, which is most similar to the EMPATHY program, is a program carried out in Australia containing many of the same elements . Unfortunately, this large and well-designed study found no improvements in depression over a 3-year period. This emphasizes the need for a longer-term follow-up.
If these initial positive findings are confirmed during longer-term follow-up study then, since many different and well-designed studies have failed to find similar positive outcomes, what are the key potential reasons that the present study has had such a positive effect on many measures? We believe that there are several key components that may have contributed. The first may have been the very rapid feedback from the time students completed their screen until the time that they were contacted (less than 48 hours if they were in the high suicide risk group). Feedback from individual students was that this was very helpful, as the rapid feedback made them feel that their issues were taken seriously. Supporting this possibility, a recent qualitative investigation of adolescents’ perceived mechanism of change following a universal school-based depression program found that there were improved interpersonal relationships and improved self-regulation, both stronger than originally assumed . Additionally, the fact that parent/guardian became rapidly involved, having frequently not been aware of the severity of the issue, may also have been a contributing factor to an improvement in mood, anxiety, and suicidal thinking by students , . Supporting the hypothesis that parent/guardian attitudes may have changed was the fact that during the study there were extremely frequent positive comments from parent/guardian expressing gratitude about knowing this information about their child.
A second possible factor could have been a change in the overall school environment or culture. While we did not formally study this, such changes have been reported in school environments previously , and changes in school environments can improve student mental health . The possibility that this occurred was suggested by frequent anecdotal feedback from principals, teachers, students, and parents. However, to determine if this occurred, it requires specific future study.
A third possibility that may have had an impact was that this program was carried out across a complete school district, as opposed to single schools. It is conceivable that this had the effect of heightening community awareness of, and support for, the program. It was certainly clear that children at different schools were in frequent contact about the program for a variety of reasons, including that many children had elder or younger siblings at other schools. There was also significant local media interest in the EMPATHY program. It is well established that community attitudes can be important in determining frequency of depression in youth . It is therefore conceivable that a positive change, or awareness regarding this issue, may have been one factor in the changes seen in the present study.
Fourthly, the Resiliency Coaches spent a lot of time with the students as part of the staff, including during some student recreation time, and although they were specifically trained to not act as “therapists”, they found that on many occasions students would spontaneously approach them and sometimes confide in them (including about physical and/or sexual abuse they had experienced or witnessed). The impact of this is uncertain, but this is a different role to that held by trained psychologists working at the school (as well as by the teachers, of course), and the Resiliency Coaches were surprised by how frequently their availability allowed such spontaneous interactions to occur. It is recognized that friendship skills can help adolescent depressive symptoms in combination with other approaches , so it is possible this may also have played a role. Currently, however, these four possibilities remain speculative. In future studies it will be important to examine them more specifically to determine what, if any, impact each of these has.
In terms of the potential benefit for a universal CBT intervention (OVK) this remains uncertain. Given that in the present study it was used in a different and truncated manner, and administered by a different group of individuals, than those who have previously been shown to deliver it effectively, it cannot be considered to have been adequately tested in the present study. That universal interventions may be more effective when delivered by psychologists has been suggested from other studies , means that we are uncertain what our negative findings may indicate. Determining the relative importance of this aspect of the program will be required in future studies, and most importantly it is necessary to study the effectiveness of the full 16-session program. Nonetheless, it is well recognized that it can be difficult to detect the overall effect of such universal interventions, and this may explain the large number of studies that are unable to show a benefit. The potential benefit of universal approaches such as OVK, as well as those recently examined for suicide prevention , , needs to be evaluated as part of the approach of combination programs such as EMPATHY, but are unlikely to be measures that on their own significantly reduce youth depression and suicide rates.
Unexpectedly, there were small, but statistically significant, decreases in both self-esteem and quality of life-scores (5.0% and 3.6% respectively). While the data demonstrated the expected findings that at baseline in those who had higher depression scores there was a close correlation with lower self-esteem and quality-of-life scores, the small decrease in both of these after the 12-week study period was unanticipated. Currently, we do not have a clear reason why this may have occurred, and although we believe the size of these changes means they were not clinically relevant, the fact that they appeared in several Grades means it will be important to continue to measure these factors in future studies. We also plan to carry out more detailed qualitative research to try and determine why this may have occurred.
Potential issues with the study
There were three major potential limitations of the present study, the first being that it was a short-term study, and it is uncertain if the changes identified will be sustained. It could be argued that in terms of suicidality, reducing this even for short periods may reduce the risks of completed suicide. Nonetheless, it is recognized that the links between scores for suicidality on a rating scale and actual suicide attempts, and in turn the number of completed suicides, remains uncertain. Thus, while there was a very gratifying reduction in suicidality ratings, it is not clear that this will translate into fewer episodes of self-harm or a reduction in completed suicides. What gives us some hope that this will be the case is that in about half of the students who were identified as being at high risk (either because of suicidal thoughts, or for high combined EMPATHY scale scores) they had not previously been recognized as being a concern either to their parent/guardian or to the school staff. This group may therefore have benefited significantly from recognition. Also, there is good evidence that youth who are suicidal usually show verbal, behavioural, and environmental warnings , , and so it is likely that the symptoms detected by the questions asked represent meaningful measurements of suicide risk. Our finding of a large group of high-risk children who were not previously identified is very consistent with data from a recent large study in over 12,000 adolescents, part of the Saving and Empowering Young Lives in Europe (SEYLE) project, which recruited these youth in randomly selected schools in 12 European countries . This study found that there was a large group of high-risk students with active suicidal thoughts who were “invisible”, but whose risk of suicide attempts was 6%, and we believe that this may represent a significant proportion of students identified in the current study as being high suicide risk, even if their depression scores were not very raised. Given the high rates of actual attempts at suicide in this group, it is certainly possible that reducing this risk provides a meaningful reduction in the number of youth who will attempt suicide. It remains important, however, that longer-term follow-up is carried out to determine if the improvements in mood, anxiety, and suicidality detected after 12-weeks are retained over the longer-term.
The second major issue with this study is the lack of a robust control group, and not having such a control means any conclusions about the EMPATHY program should be somewhat cautious at this time. We are, therefore, unable to determine if there would have similar improvements without any intervention. Certainly other studies in similar populations have shown that there is frequently a decrease in depression symptoms, regardless of what treatment is used, and this can be no different from controls . There were compelling ethical reasons why we did not have a control group of students (either at the same or different schools), because the most rigorous control group would have involved screening for active suicidality and then not intervening. Nonetheless, given that the improvements in depression, suicidality, and anxiety, occurred in all Grades and in all schools (which serve quite different populations), and not simply in motivated schools, we believe this unlikely although it will clearly require future additional research to determine the validity of this.
The third potential issue with the study was the use of the rating scales in a different manner to which they had designed or validated. We utilized a novel, and untested, approach to determining who was at risk. This was to combine several scales, each of which has been used independently, into a single score that was then used to determine who would receive additional intervention. Future research will need to determine the effectiveness of this approach in identifying those at highest-risk over the longer-time, and compare this to the use of an individual scale used its original format (such as PHQ-A). Such research will also clarify if the cut-off we determined (the top 10%) was appropriate, and whether any such cut-off is best applied across a Grade or across an entire school. Additionally, since many of the scales were utilized in a manner that differs from the method in which they have been previously validated (such as the additional suicide question added to the PHQ-A and the questions about tobacco added to the CRAAFT), research will need to clarify how each of the individual scales performed utilizing their original methodology. However, the high correlations between 4 of the 5 scales (all except those for the use of DAT) suggest that this approach may have some validity.
In conclusion, the EMPATHY program appears to offer the potential for meaningful improvements in depression and anxiety in large number of youth, and a potential major reduction in the number of youth who are actively suicidal. Further research is required to determine if these findings are maintained in the longer-term, and if so, then what the most important aspects of the program are. It also remains to be seen if it can be successfully implemented in other jurisdictions.
S1 TREND Checklist. This checklist details the information regarding whether or not methodological issues were addressed, and if so what page they can be seen.
Note that the page numbering refers to the original manuscript submission, and will differ from any final published format.
S1 Protocol. Original protocol with changes highlighted.
This shows the original protocol approved by the ethics review committee in September 2013, as well as the subsequent changes made which were approved in January 2014. These changes relate in large degree to a change in the measurement tools, necessitated by a desire to utilize screening tools, and cognitive behavioural treatment (CBT) tools, that would fit within the paradigms established in the study, i.e. being free, brief, and available to use with the technology available to the study and the schools. It can be seen that there varied considerably from the initial tools that were approved for use.
This study would not have been possible without the assistance of several individuals in the Red Deer community, including in Red Deer Public Schools (Elizabeth Fargey), Alberta Health Services (Michael Trew, Dwight Hunks, Noreen McCallum, Marti Rustulka, Tanya Drescher, Chris Wilkes), Red Deer Primary Care Network (Lorna Milkovich), Central Alberta Child and Family Services Authority (David Tunney), and Justice and Solicitor General Community Corrections (Kim Fay). We also had significant institutional support in Alberta including that of the Science, Policy Practice Network, a multi-sectoral partnership supported in part by the Norlien Foundation of Alberta. We also want to thank Rowella Kuijpers and Karlijn Kindt for training the study team in the correct methodology for administration of the OVK program.
Conceived and designed the experiments: PS MB VS AA KR JG SH PL RE CP. Performed the experiments: PS MB VS AA KR JG SH PL RE CP. Analyzed the data: SC IC CM. Contributed reagents/materials/analysis tools: RE. Wrote the paper: PS.
- 1. Perou R, Bitsko RH, Blumberg SJ, Pastor P, Ghandour RM, Gfroerer JC, et al. Mental health surveillance among children-United States, 2005–2011. MMWR Surveill Summ. 2013;62: Suppl 2: 1–35. pmid:23677130
- 2. Nardi B, Francesconi G, Catena-Dell'osso M, Bellantuono C. Adolescent depression: clinical features and therapeutic strategies. Eur Rev Med Pharmacol Sci. 2013;17: 1546–1551. pmid:23771545
- 3. Bor W, Dean AJ, Najman J, Hayatbakhsh R. Are child and adolescent mental health problems increasing in the 21st century? A systematic review. Aust N Z J Psychiat. 2014; 48: 606–616. pmid:24829198
- 4. Kim-Cohen J, Caspi A, Moffitt TE, Harrington H, Milne BJ, Poulton R. Prior juvenile diagnoses in adults with mental disorder: developmental follow-back of a prospective-longitudinal cohort. Arch Gen Psychiatry. 2003;60: 709–717. pmid:12860775
- 5. Hawton K, Saunders KE, O'Connor RC. Self-harm and suicide in adolescents. Lancet. 2012;379: 2373–2382. pmid:22726518
- 6. Clifford AC, Doran CM, Tsey K. A systematic review of suicide prevention interventions targeting indigenous peoples in Australia, United States, Canada and New Zealand. BMC Public Health. 2013;13: 463 pmid:23663493
- 7. Eaton DK, Kann L, Kinchen S, Shanklin S, Flint K, Hawkins J, et al. Youth risk behavior surveillance—United States, 2011. Centers for Disease Control and Prevention MMWR, Surveillance Summaries. 2012;61(no. SS-4). Available: http://www.cdc.gov/mmwr/pdf/ss/ss6104.pdf. pmid:22456193
- 8. Carli V, Hoven CW, Wasserman C, Chiesa F, Guffanti G, Sarchiapone M, et al. A newly identified group of adolescents at "invisible" risk for psychopathology and suicidal behavior: findings from the SEYLE study. World Psychiatry. 2014;13: 78–86. pmid:24497256
- 9. Strunk CM, King KA, Vidourek RA, Sorter MT. Effectiveness of the surviving the teens suicide prevention and depression awareness program: an impact evaluation utilizing a comparison group. Health Educ Behav. 2014;41: 605–613. pmid:24786795
- 10. Daniel SS, Goldston DB. Interventions for suicidal youth: a review of the literature and developmental considerations. Suicide Life Threat Behav. 2009;39: 252–268. pmid:19606918
- 11. Burns JM, Dudley M, Hazell P, Patton G. Clinical management of deliberate self-harm in young people: the need for evidence-based approaches to reduce repetition. Aust N Z J Psychiat. 2005;39: 121–128. pmid:15701059
- 12. Gould MD, Greenberg T, Velting DD, Shaffer DD. Youth suicide risk and preventive interventions: a review of the past 10 years. J Am Acad Child Adolesc Psychiat. 2003;42: 386–405.
- 13. Robinson J, Hetrick SE, Martin C. Preventing suicide in young people: Systematic review. Aust N Z J of Psychiat. 2011;45: 3–26.
- 14. Steele MM, Doey T. Suicidal behavior in children and adolescents. Part 2: treatment and prevention. Can J Psychiat. 2007;52; (Suppl 1) 5S–45S.
- 15. Weare K, Nind M. Mental health promotion and problem prevention in schools: what does the evidence say? Health Promot Int. 2011;26; (Suppl 1) i29–69. pmid:22079935
- 16. Robinson J, Pirkis J. Research priorities in suicide prevention: an examination of Australian-based research 2007–11. Aust Health Rev. 2014;38: 18–24. pmid:24262112
- 17. Saunders K, Brand F, Lascelles K, Hawton K. The sad truth about the SADPERSONS Scale: an evaluation of its clinical utility in self-harm patients. Emerg Med J. 2014;31: 796–798. pmid:23896589
- 18. Simon GE, Rutter CM, Peterson D, Oliver M, Whiteside U, Operskalski B, et al. Does response on the PHQ-9 Depression Questionnaire predict subsequent suicide attempt or suicide death? Psychiatr Serv. 2013;64: 1195–1202. pmid:24036589
- 19. Friedmann H, Kohn R. Mortality, or probability of death, from a suicidal act in the United States. Suicide Life Threat Behav. 2008;38: 287–301. pmid:18611127
- 20. Spirito A, Esposito-Smythers C. Attempted and completed suicide in adolescence. Annu Rev Clin Psychol. 2006;2: 237–266. pmid:17716070
- 21. Beghi M, Rosenbaum JF. Risk factors for fatal and nonfatal repetition of suicide attempt: a critical appraisal. Curr Opin Psychiatry. 2010;23: 349–355. pmid:20520547
- 22. Jaycox LH, Reivich KJ, Gillham J, Seligman ME. Prevention of depressive symptoms in school children. Behav Res Ther. 1994;32: 801–816. pmid:7993324
- 23. Gillham JE, Reivich KJ. Prevention of depressive symptoms in school children: A research update. Psychol Sci. 199;10: 461–462.
- 24. Gillham JE, Reivich KJ, Freres DR, Chaplin TM, Shatté AJ, Samuels B, et al. School-based prevention of depressive symptoms: A randomized controlled study of the effectiveness and specificity of the Penn Resiliency Program. J Consult Clin Psychol. 2007;75: 9–19. pmid:17295559
- 25. Gillham JE, Reivich KJ, Brunwasser SM, Freres DR, Chajon ND, Kash-Macdonald VM, et al. Evaluation of a group cognitive-behavioral depression prevention program for young adolescents: a randomized effectiveness trial. J Clin Child Adolesc Psychol. 2012;41: 621–639. pmid:22889296
- 26. Tak YR, Van Zundert RM, Kuijpers RC, Van Vlokhoven BS, Rensink HF, Engels RC. A randomized controlled trial testing the effectiveness of a universal school-based depression prevention program 'Op Volle Kracht' in the Netherlands. BMC Public Health. 2012;12: 21. pmid:22233510
- 27. Wijnhoven LA, Creemers DH, Vermulst AA, Scholte RH, Engels RC. Randomized controlled trial testing the effectiveness of a depression prevention program ('Op Volle Kracht') among adolescent girls with elevated depressive symptoms. J Abn Child Psych. 2014;42: 217–228. pmid:23893066
- 28. Kindt KCM, van Zundert RMP, Engels RC. Evaluation of a Dutch school-based depression prevention program for youths in highrisk neighborhoods: study protocol of a two-armed randomized controlled trial. BMC Public Health. 2012;12: 212–218. pmid:22433176
- 29. Kindt KCM, Kleinjan M, Janssens JMAM, Scholte RHJ. Evaluation of a School-Based Depression Prevention Program among Adolescents from Low-Income Areas: A Randomized Controlled Effectiveness Trial. Int J Environ Res Public Health. 2014;11: 5273–5293. pmid:24837666
- 30. Calear AL, Christensen H. Systematic review of school-based prevention and early intervention programs for depression. J Adolesc. 2010;33: 429–438. pmid:19647310
- 31. Bond L, Butler H, Thomas L, Carlin J, Glover S, Bowes G, et al. Social and school connectedness in early secondary school as predictors of late teenage substance use, mental health, and academic outcomes. J Adolesc Health. 2007;40: e9–18. pmid:17531766
- 32. Durlak JA, Weissberg RP, Dymnicki AB, Taylor RD, Schellinger KB. The impact of enhancing students’ social and emotional learning: a meta-analysis of school-based universal interventions. Child Dev. 2011;82: 405–432. pmid:21291449
- 33. Wiefferink CH, Peters L, Hoekstra F, Dam GT, Buijs GJ, Paulussen TG. Clustering of health-related behaviors and their determinants: possible consequences for school health interventions. Prev Sci. 2006;7: 127–149. pmid:16596470
- 34. Brière FN, Rohde P, Shaw H, Stice E. Moderators of two indicated cognitive-behavioral depression prevention approaches for adolescents in a school-based effectiveness trial. Behav Res Ther. 2014;53: 55–62. pmid:24418653
- 35. Spence SH, Shortt AL. Research Review: Can we justify the widespread dissemination of universal, school-based interventions for the prevention of depression among children and adolescents? J Child Psychol Psychiat. 2007;48: 526–542. pmid:17537069
- 36. Merry SN, Hetrick SE, Cox GR, Brudevold-Iversen T, Bir JJ. McDowell H. Psychological and educational interventions for preventing depression in children and adolescents. Cochrane Database Syst Rev. 2011;12: CD003380. pmid:22161377
- 37. van Zoonen K, Buntrock C, Ebert DD, Smit F, Reynolds CF 3rd, Beekman AT, et al. Preventing the onset of major depressive disorder: a meta-analytic review of psychological interventions. Int J Epidemiol. 2014;43: 318–329. pmid:24760873
- 38. Bond L, Butler H, Thomas L, Carlin J, Glover S, Bowes G, et al. Social and school connectedness in early secondary school as predictors of late teenage substance use, mental health, and academic outcomes. J Adolesc Health.2007;40: 357 e9–18.
- 39. Daniel SS, Goldston DB. Interventions for suicidal youth: a review of the literature and developmental considerations. Suicide Life Threat Behav. 2009;39: 252–268. pmid:19606918
- 40. Durlak JA, Weissberg RP, Dymnicki AB, Taylor RD, Schellinger KB. The impact of enhancing students’ social and emotional learning: a meta-analysis of school-based universal interventions. Child Dev. 2011;82: 405–432. pmid:21291449
- 41. Wiefferink CH, Peters L, Hoekstra F, Dam GT, Buijs GJ, Paulussen TG. Clustering of health-related behaviors and their determinants: possible consequences for school health interventions. Prev Sci. 2006;7: 127–149. pmid:16596470
- 42. Wei Y, Kutcher S. International school mental health: global approaches, global challenges, and global opportunities. Child Adolesc Psychiatr Clin N Am. 2012;21: 11–27. pmid:22137808
- 43. Chung T, Colby SM, Barnett NP, Rohsenow DJ, Spirito A, Monti PM. Screening adolescents for problem drinking: performance of brief screens against DSM-IV alcohol diagnoses. J Stud Alcohol. 2000;61: 579–587. pmid:10928728
- 44. Dever BV, Kamphaus RW, Dowdy E, Raines TC, Distefano C. Surveillance of middle and high school mental health risk by student self-report screener. West J Emerg Med. 2013;14: 384–390. pmid:23997848
- 45. O'Leary-Barrett M, Topper L, Al-Khudhairy N, Pihl RO, Castellanos-Ryan N, Mackie CJ, et al. Two-year impact of personality-targeted, teacher-delivered interventions on youth internalizing and externalizing problems: a cluster-randomized trial. J Am Acad Child Adolesc Psychiat. 2013;52: 911–920. pmid:23972693
- 46. Balázs J, Miklósi M, Keresztény A, Hoven CW, Carli V, Wasserman C, et al. Adolescent subthreshold-depression and anxiety: psychopathology, functional impairment and increased suicide risk. J Child Psychol Psychiat. 2013;54: 670–677. pmid:23330982
- 47. Bertha EA, Balázs J. Subthreshold depression in adolescence: a systematic review. Eur Child Adolesc Psychiat. 2013;22: 589–603. pmid:23579389
- 48. Teesson M, Newton NC, Slade T, Chapman C, Allsop S, Hides L, et al. The CLIMATE schools combined study: a cluster randomised controlled trial of a universal Internet-based prevention program for youth substance misuse, depression and anxiety. BMC Psychiat. 2014;14: 32.
- 49. Sheffield JK, Spence SH, Rapee RM, Kowalenko N, Wignall A, Davis A, et al. Evaluation of universal, indicated, and combined cognitive-behavioral approaches to the prevention of depression among adolescents. J Consult Clin Psychol. 2006;74: 66–79. pmid:16551144
- 50. Sawyer MG, Pfeiffer S, Spence SH, Bond L, Graetz B, Kay D, et al. School-based prevention of depression: a randomised controlled study of the beyondblue schools research initiative. J Child Psychol Psychiat. 2010;51: 199–209. pmid:19702662
- 51. Spence SH, Sawyer MG, Sheffield J, Patton G, Bond L, Graetz B, et al. Does the absence of a supportive family environment influence the outcome of a universal intervention for the prevention of depression? Int J Environ Res Public Health. 2014;11: 5113–5132. pmid:24828082
- 52. Robinson J, Hetrick S, Cox G, Bendall S, Yuen HP, Yung A, et al. Can an Internet-based intervention reduce suicidal ideation, depression and hopelessness among secondary school students: results from a pilot study. Early Intervention Psychiat. 2014; In press.
- 53. Davies EB, Morriss R, Glazebrook C. Computer-delivered and web-based interventions to improve depression, anxiety, and psychological well-being of university students: a systematic review and meta-analysis. J Med Internet Res. 2014;16: e130. pmid:24836465
- 54. Lillevoll KR, Vangberg HC, Griffiths KM, Waterloo K, Eisemann MR. Uptake and adherence of a self-directed internet-based mental health intervention with tailored e-mail reminders in senior high schools in Norway. BMC Psychiat. 2014;14: 14.
- 55. Postel MG, de Haan HA, De Jong CA. E-therapy for mental health problems: a systematic review. Telemed J E Health. 2008;14: 707–714. pmid:18817501
- 56. Calear AL, Christensen H. Review of internet-based prevention and treatment programs for anxiety and depression in children and adolescents. Med J Aust. 2010;192: (Suppl 11) S12–14.
- 57. Titov N, Andrews G, Davies M, McIntyre K, Robinson E, Solley K. Internet treatment for depression: a randomized controlled trial comparing clinician vs. technician assistance. PLOS One. 2010;5: e10939. pmid:20544030
- 58. Elison S, Ward J, Davies G, Moody M. Implementation of computer-assisted therapy for substance misuse: a qualitative study of Breaking Free Online using Roger's diffusion of innovation theory. Drugs Alc Today. 2014;14: 207–218
- 59. Anderson R, Ukoumunne OC, Sayal K, Phillips R, Taylor JA, Spears M, et al. Cost-effectiveness of classroom-based cognitive behaviour therapy in reducing symptoms of depression in adolescents: a trial-based analysis. J Child Psychol Psychiat. 2014;55: 1390–1397. pmid:24813670
- 60. Andersson G. Using the Internet to provide cognitive behavioural therapy. Behav Res Ther. 2009; 47: 175–180. pmid:19230862
- 61. Carrard I, Crépin C, Rouget P, Lam T, Golay A, Van der Linden M. Randomised controlled trial of a guided self-help treatment on the Internet for binge eating disorder. Behav Res Ther. 2011;49: 482–491. pmid:21641580
- 62. Spitzer RL, Kroenke K, Williams JB. Validation and utility of a self-report version of the PRIME-MD: the PHQ primary care study. Primary Care Evaluation of Mental Disorders. Patient Health Questionnaire. JAMA. 1999;282: 1737–1744. pmid:10568646
- 63. Ganguly S, Samanta M, Roy P, Chatterjee S, Kaplan DW, Basu B. Patient health questionnaire-9 as an effective tool for screening of depression among Indian adolescents. J Adolesc Health. 2013;52: 546–51. pmid:23299020
- 64. Richardson LP, McCauley E, McCarty CA, Grossman DC, Myaing M, Zhou C, et al. Predictors of persistence after a positive depression screen among adolescents. Pediatrics. 2012;130: e1541–1548. pmid:23166342
- 65. Richardson LP, McCauley E, Grossman DC, McCarty CA, Richards J, Russo JE, et al. Evaluation of the Patient Health Questionnaire-9 Item for detecting major depression among adolescents. Pediatrics. 2010;126: 1117–1123. pmid:21041282
- 66. Zigmond AS, Snaith RP. The hospital anxiety and depression scale. Acta Psychiatr Scand. 1983;67: 361–370. pmid:6880820
- 67. Wiklund M, Malmgren-Olsson EB, Ohman A, Bergström E, Fjellman-Wiklund A. Subjective health complaints in older adolescents are related to perceived stress, anxiety and gender—a cross-sectional school study in Northern Sweden. BMC Public Health. 2012;12: 993. pmid:23158724
- 68. van der Geest IM, van Dorp W, Hop WC, Neggers SJ, de Vries AC, Pieters R, et al. Emotional distress in 652 Dutch very long-term survivors of childhood cancer, using the hospital anxiety and depression scale (HADS). J Pediatr Hematol Oncol. 2013;35: 525–529. pmid:24060834
- 69. Wen L, Jiang G, Zhang X, Lai R, Wen X. Relationship between acne and psychological burden evaluated by ASLEC and HADS Surveys in high school and college students From central China. Cell Biochem Biophys. 2014;71: 1083–1088.
- 70. Rosenberg M. Society and the adolescent self-image. Princeton, New Jersey. Princeton University Press. 1965.
- 71. Sullivan BJ. Self-esteem and depression in adolescent diabetic girls. Diabetes Care. 1978;1: 18–22. pmid:554782
- 72. Puskar KR, Bernardo LM, Ren D, Haley TM, Tark KH, Switala J, et al. Self-esteem and optimism in rural youth: gender differences. Contemp Nurse. 2010;34: 190–198. pmid:20509803
- 73. Khajehdaluee M, Zavar A, Alidoust M, Pourandi R. The relation of self-esteem and illegal drug usage in high school students. Iran Red Crescent Med J. 2013;15: e7682. pmid:24719686
- 74. Erhart M, Ottova V, Gaspar T, Jericek H, Schnohr C, Alikasifoglu M, et al. Measuring mental health and well-being of school-children in 15 European countries using the KIDSCREEN-10 Index. Int J Public Health. 2009;54: (Suppl 2) 160–166. pmid:19652910
- 75. Ravens-Sieberer U, Erhart M, Rajmil L, Herdman M, Auquier P, Bruil J, et al. Reliability, construct and criterion validity of the KIDSCREEN-10 score: a short measure for children and adolescents’ well-being and health-related quality of life. Qual Life Res. 2010;19: 1487–1500. pmid:20668950
- 76. Bouwmans C, van der Kolk A, Oppe M, Schawo S, Stolk E, van Agthoven M, et al. Validity and responsiveness of the EQ-5D and the KIDSCREEN-10 in children with ADHD. Eur J Health Econ. 2014;15: 967–977. pmid:24233919
- 77. Knight JR, Sherritt L, Harris SK, Gates EC, Chang G. Validity of brief alcohol screening tests among adolescents: a comparison of the AUDIT, POSIT, CAGE, and CRAFFT. Alcohol Clin Exp Res. 2003;27: 67–73. pmid:12544008
- 78. Skogen JC, Bøe T, Knudsen AK, Hysing M. Psychometric properties and concurrent validity of the CRAFFT among Norwegian adolescents. Ung@hordaland, a population-based study. Addict Behav. 2013;38: 2500–2505. pmid:23770648
- 79. Costello EJ, Egger H, Angold A. 10-year research update review: the epidemiology of child and adolescent psychiatric disorders: I. methods and public health burden. J Am Acad Child Adolesc Psychiat. 2005;44: 972–986.
- 80. Goldman S. Developmental epidemiology of depressive disorders. Child Adolesc Psychiatr Clin N Am. 2012;21: 217–235. pmid:22537724
- 81. Spence SH, Sheffield JK, Donovan CL. Long-term outcome of a school-based, universal approach to prevention of depression in adolescents. J Consult Clin Psychol. 2005;73: 160–177. pmid:15709843
- 82. Shochet I, Montague R, Smith C, Dadds M. A qualitative investigation of adolescents' perceived mechanisms of change from a universal school-based depression prevention program. Int J Environ Res Public Health. 2014;11: 5541–5554. pmid:24859679
- 83. Cutuli JJ, Gillham JE, Chaplin TM, Reivich KJ, Seligman ME, Gallop RJ, et al. Preventing adolescents' externalizing and internalizing symptoms: Effects of the Penn Resiliency Program. Int J Emot Educ. 2013;5: 67–79. pmid:24634897
- 84. Sander JB, McCarty CA. Youth depression in the family context: familial risk factors and models of treatment. Clin Child Fam Psychol Rev. 2005;8: 203–219. pmid:16151618
- 85. Kutcher S, Wei Y. Mental health and the school environment: secondary schools, promotion and pathways to care. Curr Opin Psychiat. 2012;25: 311–316.
- 86. Lima-Serrano M, Lima-Rodríguez JS. Impact of school-based health promotion interventions aimed at different behavioral domains: a systematic review. Gac Sanit. 2014;28: 411–417. pmid:24923204
- 87. Everett BG. Changes in neighborhood characteristics and depression among sexual minority young adults. J Am Psychiatr Nurses Assoc. 2014;20: 42–52. pmid:24217448
- 88. Rose K, Hawes DJ, Hunt CJ. Randomized controlled trial of a friendship skills intervention on adolescent depressive symptoms. J Consult Clin Psychol. 2014;82: 510–520. pmid:24491074
- 89. Wahl MS, Adelson JL, Patak MA, Pössel P, Hautzinger M. Teachers or psychologists: who should facilitate depression prevention programs in schools? Int J Environ Res Public Health. 2014;11: 5294–5316. pmid:24837667
- 90. King KA. Practical strategies for preventing adolescent suicide. Prevention Researcher. 2006;13: 8–11.
- 91. Rudd MD, Berman AL, Joiner TE Jr, Nock MK, Silverman MM, Mandrusiak M, et al. Warning signs for suicide: theory, research and clinical applications. Suicide Life Threat Behav. 2006;36: 255–262. pmid:16805653