A recent paper reported the outcomes of a study examining a new self-guided internet-delivered treatment, the Wellbeing Course, for symptoms of anxiety or depression. This study found the intervention resulted in significant symptom reductions. It also found that automated emails increased treatment completion and clinical improvements in a subsample with elevated anxiety and depression.
To examine the clinical outcomes and the effect of automated emails at 12 months post-treatment.
Participants, who were randomly allocated to a Treatment Plus Automated Emails Group (TEG; n = 100), a standard Treatment Group (TG; n = 106) or delayed-treatment Waitlist Control Group (Control; n = 51), were followed up at 12 months post-treatment. Eighty-one percent, 78% and 87% of participants in the TEG, TG and treated Waitlist Control Group provided symptom data at 12-month follow-up, respectively. The primary outcome measures were the Patient Health Questionnaire-9 Item Scale (PHQ-9) and the Generalized Anxiety Disorder-7 Item Scale (GAD-7).
Significant improvements in symptoms of anxiety and depression were observed over time in both the TEG and TG (Fs >69, ps <.001) these were sustained from post-treatment to 12-month follow-up (ps >.05), and were associated with large effect sizes. No statistically significant differences in symptoms were found between the TEG and TG at post-treatment, 3-month or 12-month follow-up. Previously reported symptom differences between TEG and TG participants with comorbid symptoms were no longer present at 12-month follow-up (ps >.70).
The overall benefits of the Wellbeing Course were sustained at 12-month follow-up. Although automated emails facilitated Course completion and reductions in symptoms for participants with comorbid anxiety and depression from pre-post treatment, these differences were no longer observed at 12-month follow-up. The results indicate that automated emails promote more rapid treatment response for people with elevated and comorbid symptoms, but may not improve longer term outcomes.
Australian and New Zealand Clinical Trials Registry ACTRN12610001058066
Citation: Titov N, Dear BF, Johnston L, McEvoy PM, Wootton B, Terides MD, et al. (2014) Improving Adherence and Clinical Outcomes in Self-Guided Internet Treatment for Anxiety and Depression: A 12-Month Follow-Up of a Randomised Controlled Trial. PLoS ONE 9(2): e89591. doi:10.1371/journal.pone.0089591
Editor: Jerson Laks, Federal University of Rio de Janeiro, Brazil
Received: July 18, 2013; Accepted: January 19, 2014; Published: February 25, 2014
Copyright: © 2014 Titov et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: The authors gratefully acknowledge the participants for their involvement and helpful comments. This research was enabled by funding from the Australian National Health and Medical Research Council (NHMRC) Project Grant No. 630560. BFD is supported by a National Health and Medical Research Council (NHMRC) Australian Public Health Fellowship. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: NT and BFD are authors and developers of the Wellbeing Course, but derive no personal or financial benefit from it. NT, BFD, and RR are funded by the Australian Government to develop and provide a national internet and telephone-delivered treatment service, the MindSpot Clinic (www.mindspot.org), for people with anxiety and depression. This does not alter the authors' adherence to all the PLOS ONE policies on sharing data and materials. NT has acted as an associate editor for PLOSEONE. However, being a PLOSONE Associate Editor or acting as reviewers does not alter the authors' adherence to all the PLOSONE policies on sharing data and materials.
Self-guided internet-delivered psychological interventions for anxiety and depressive disorders have considerable public health potential. However, more than 90% of consumers typically withdraw after two sessions ,  and, consequently may not receive a dose of therapeutic content sufficient to lead to sustained improvement. Self-guided internet interventions that include strategies to facilitate adherence have considerable potential for optimizing outcomes. Trials of self-guided internet interventions for depression ,  and social phobia , ,  show that automated reminder emails increased completion rates and effect sizes relative to earlier trials without automated reminders. Automated emails may be particularly suitable as a medium for providing reminders given their ubiquity and low cost.
A recent paper reported an evaluation of a new self-guided internet-delivered treatment, the Wellbeing Course, for symptoms of anxiety or depression . Participants had no contact with clinicians prior to or during treatment and were randomly allocated to receive the Wellbeing Course administered with automated emails (Treatment Plus Automated Email Group: TEG, n = 100), without automated emails (Treatment Group: TG, n = 106) or to a Waitlist Control Group (Control Group; n = 51), who subsequently received treatment. At post-treatment, participants in the two Treatment Groups obtained similar and significant improvements on measures of anxiety and depression relative to the Control Group. However, those participants receiving automated emails obtained higher rates of course completion, defined as having read all five lessons of the course, than those not receiving emails (58% v. 35%). While no overall differences in symptom improvement were found between the Treatment Groups, a subgroup of participants with comorbid anxiety and depression, that is, participants with elevated symptoms of both anxiety and depression, obtained superior reductions in symptoms when they received automated emails compared to when they did not receive the emails. The pattern of these results was sustained at 3-month follow-up.
An outstanding question is whether these results are sustained in the longer term. Return or exacerbation of symptoms may occur following treatment for anxiety disorders and depression , . The primary aim of the present study was to test the hypothesis that the benefits observed at post-treatment and 3-month follow-up would be maintained at 12-month follow-up. Previous results indicate that benefits of guided internet interventions for anxiety or depression , ,  are sustained in the longer-term, but limited data exist for longer-term benefits of self-guided internet interventions . An additional aim was to examine the number of participants who deteriorated at 12-month follow-up, defined as an increase in PHQ-9 or GAD-7 scores of 5 or more points from pre-treatment. No formal hypothesis was generated with respect to deterioration, but this data was collected to provide information about the safety of the Wellbeing Course.
Ethics Statement and Trial Registration
The original study  was approved by the Human Research Ethics Committee (HREC) of the University of New South Wales (UNSW, Sydney, Australia) and ratified by the Macquarie University HREC (Sydney, Australia; HREC Reference: 5201100226). The original protocol included telephone interviews at recruitment and a fourth group who received weekly contact with a therapist. However, in order to focus on the benefits of providing automatic emails during self-guided treatment and to simulate an entirely automated treatment process, the interviews and this fourth group were omitted from the final design. The trial was registered with the Australian and New Zealand Clinical Trials registry as ACTRN12610001058066. The protocol for this trial and supporting CONSORT checklist are available as supporting information; see Checklist S1 and Protocol S1.
Design and Sample Size
The original trial randomly allocated 257 participants to one of two Treatment Groups or to a Waitlist Control Group. Participants in the Treatment Plus Automated Email Group (TEG; n = 100) received access to the Wellbeing Course with automated emails. Participants in the Treatment Group (TG; n = 106) received access to the Wellbeing Course without automated emails. The Control Group (Controls; n = 51) received access to the same treatment as the TEG following post-treatment; the data presented here are the data for the Control Group in Treatment, that is, after the Two Treatment Groups completed treatment and the Control Group received treatment.
The primary outcome measures were depression and anxiety symptom severity, as measured by the PHQ-9 and GAD-7. The GAD-7 has good convergent validity with other anxiety scales –, is sensitive to DSM-IV congruent generalised anxiety disorder, social phobia and panic disorder, with increasing scores indicating greater symptom severity . A total score of 8 on the GAD-7 has been identified as an important threshold for identifying the presence of an anxiety disorder –. A total score of 10 on the PHQ-9 has been identified as an important threshold for identifying DSM-IV congruent depression with increasing scores indicating greater symptom severity , -. Additional information collected comprised demographic details including: age, gender, educational qualifications, vocational status, treatment satisfaction and completion rates (i.e., percentage in each treatment group who read all five online lessons of the Wellbeing Course). Additional measures included changes in health status and service utilisation, which will be reported separately as part of an evaluation of the cost-effectiveness of the intervention.
TEG and TG participants completed the PHQ-9 and GAD-7 at pre-treatment, prior to each of the five lessons, at post-treatment, 3-month follow-up and 12-month follow-up. Control Group participants completed the PHQ-9 and GAD-7 at pre-treatment and post-treatment and, subsequently, commenced the same intervention as the TEG group with data collected at pre-treatment, prior to each lesson, at post-treatment, 3 month follow-up and 12 month follow-up. Their results after they received and completed the treatment phase are presented here as a partial replication in order to test the reliability of the TEG condition.
The Wellbeing Course is a five lesson transdiagnostic internet intervention based on a pragmatic model of psychotherapeutic change that assumes that symptoms of anxiety and depression are the result of unhelpful habits of thought and action, that is, maladaptive cognitions and behaviours , . This model also proposes that interventions that are structured, systematically present increasingly complex material over weeks and months, and are sufficiently engaging to achieve adherence and commitment, will result in reductions in unhelpful habits and an increase in adaptive habits. Further, such interventions are likely to result in large and sustained improvements in symptoms compared to sporadic or unstructured therapy sessions, which may only result in short-term symptom relief. More details about the Wellbeing Course can be found elsewhere .
TEG and TG participants received an email at the start of the Course providing guidelines and a recommended timetable of Course-related activities. TEG participants also received at least two additional emails per week during the Course. Emails were triggered when (1) participants read a Lesson, (2) if participants had not read a prescribed Lesson within 7 days of it becoming available, (3) at the start of each week, and (4) at times during the Course when participants were known to experience increased symptoms or to have increased difficulties practicing skills (e.g., during the weeks when graded exposure was introduced). Emails were comprised of three or four sentences, used the participant's first name, and employed a warm and supportive tone. The content informed participants about newly available or unread materials, reinforced progress and skills practice, normalised the challenges of learning new skills, and encouraged realistic expectations of recovery. All participants consented to receive the emails and no emails contained personal or detailed clinical information.
Up to five automated email reminders were sent to all participants to complete post-treatment or follow-up questionnaires. These emails were only sent where participants had not completed the questionnaires. Once participants had completed the questionnaires they were sent a brief template-based feedback email detailing their results. At the 3- and 12-month follow-up time points participants who had not completed the questionnaires after the five automated emails were telephoned once by the researchers. No clinical contact was made between the 3- and 12-month follow-up periods and no clinical content was discussed in these telephone calls. The Course material was made available for the 12-month period following the end of the 8-week treatment.
Consistent with the initial report of this trial , several separate mixed-models analyses were employed to analyse the data at post-treatment, 3-month and 12-month follow-up. These mixed-models employed an autoregressive covariance structure and maximum likelihood estimation, which provides unbiased estimates in the case of missing data; under the assumption that data is missing at random. These mixed-models specified Group (i.e., TEG and TG) and Time (i.e., pre-treatment, post-treatment, 3-month follow-up and 12-month follow-up) as fixed effects with the Intercept set at pre-treatment and Participants specific as a random effect. Primary interest was the Group x Time interactions and differences within and between the two Treatment Groups at the different time points were examined using pairwise comparisons. Consistent with the original report of this study , one set of analyses was conducted using the entire data set (i.e., the Overall Sample) and a second set of analyses was restricted to participants with elevated symptoms of both anxiety and depression (i.e., the Comorbid Sample), defined by pre-treatment scores above clinical cut-offs on both primary outcome measures (i.e., total GAD-7 ≥8 and total PHQ-9 ≥10). Effect sizes (Cohen's d) and 95% confidence intervals were calculated for both within-group and between-group effects based on the pooled standard deviation. All analyses were performed in SPSS version 19.0 (SPSS, Inc., Chicago, IL).
As described in recent dissemination studies , pre-treatment, post-treatment and follow-up PHQ-9 and GAD-7 scores were compared with clinical cut-offs to provide an index of clinically significant remission. This was defined as the proportion of participants who initially scored at or above the clinical cut-offs and then subsequently below the clinical cut-offs. Finally, the proportion of participants with significant deterioration in PHQ-9 or GAD-7 scores were reported, calculated for each group based on a completer analysis and defined as an increase in PHQ-9 or GAD-7 scores of five or more points from pre-treatment . In these analyses, missing values were handled using the Multiple Imputation  procedure where Group as well as GAD-7 and PHQ-9 scores at each time point were used to generate 20 sets of imputed data. These 20 imputed data sets where then averaged to provide pooled frequency counts, chi-square statistics and p values. All results are reported also presented for the Waitlist Control Group, after they had received treatment, as a partial replication of the results of the TEG group.
Adherence and Attrition
Twelve-month follow-up data were collected from 81%, 78% and 87% of TEG, TG and treated Waitlist Control Group participants, respectively.
Between post-treatment and follow-up participants in both treatment groups continued to log in and read the online materials. At the 12-month follow-up there was no longer a significant difference in the number of TEG (68%) and TG (59%) participants who had read the five lessons, that is, completed the Course, X2 (1) = 1.63, p = .20. Similarly, in the Comorbid Sample, no significant difference was found between the TEG (66%) and TG (59%) participants who had completed the Course at 12-month follow-up, X2 (1) = 0.67, p = .41.
Primary Outcome Measures
Note: TEG: Treatment plus automated email group. TG: Treatment group. PHQ-9: Patient Health Questionnaire-9 Item. GAD-7: Generalised Anxiety Disorder-7 Item.
The mixed-model analyses examining GAD-7 scores for the Overall Sample revealed a significant effect for Time (F3, 467 = 87.57, p<.001), but not for Group (F1, 212 = 0.11, p = .74) or the Time by Group interaction (F3, 467 = 1.27, p = .28). Pairwise comparisons revealed no significant differences between the TEG and TG groups at post-treatment, 3-month follow-up or 12-month follow-up (p range = .14 to .69). Both the TEG and TG groups improved significantly from pre-treatment to post-treatment (p<.001), but did not obtain any further significant changes from post-treatment to 3-month follow-up (p>.05) or from 3-month follow-up to 12 month follow-up (p>.05). However, the scores of both groups remained significantly lower at 3- and 12-month follow-up compared with pre-treatment (p<.001), indicating that symptom improvements were maintained.
Similarly, the mixed-model analyses examining PHQ-9 scores for the Overall Sample revealed a significant effect for Time (F3, 471 = 74.48, p<.001), but not for Group (F1, 219 = 0.01, P = .93) or the Time by Group interaction (F3, 471 = 0.79, p = .50). Pairwise comparisons revealed no significant differences between the TEG and TG groups at post-treatment, 3-month follow-up, and 12-month follow-up (p range = .35 to .81). However, both the TEG and TG groups improved significantly from pre-treatment to post-treatment (p<.001), but did not obtain any further significant changes from post-treatment to 3-month follow-up (p>.05) or from 3-month follow-up to 12-month follow-up (p>.05). The scores of both groups were sustained and remained significantly lower at 3- and 12-month follow-up compared with pre-treatment (p<.001).
The mixed-model analyses for the GAD-7 scores for the Comorbid Sample revealed a significant effect for Time (F3, 258 = 69.03, p<.001) and Group (F1, 126 = 4.21, p = .042), but not the Time by Group interaction (F3, 258 = 1.63, p = .18). Pairwise comparisons revealed that both the TEG and TG groups improved significantly from pre-treatment to post-treatment (p<.001), but did not obtain any further significant changes from post-treatment to 3-month follow-up (p>.05) or from 3-month follow-up to 12-month follow-up (p>.05). The scores of both groups were sustained and remained significantly lower at 3- and 12-month follow-up compared with pre-treatment (p>.001). Pairwise comparisons also revealed the TEG group had significantly lower scores than the TG group at post-treatment and 3-month follow-up (p range = .01 to .03), but not at 12-month follow-up (p = .75).
The mixed-model analyses of the PHQ-9 scores for the Comorbid Sample revealed a significant effect for Time (F3, 268 = 72.58, p<.001), but not for Group (F1, 138 = 2.24, p = .14) or the Time by Group interaction (F3, 268 = 2.27, p = .08). Pairwise comparisons revealed that both the TEG and TG groups improved significantly from pre-treatment to post-treatment (p<.001), but did not obtain any further significant changes from post-treatment to 3-month follow-up (p>.05) or from 3-month follow-up to 12-month follow-up (p>.05). The scores of both groups were sustained and remained significantly lower at 3- and 12-month follow-up compared with pre-treatment (p>.001). Pairwise comparisons also revealed the TEG group had significantly lower scores than the TG group at post-treatment and 3-month follow-up (p range = .02 to .04), but not at 12-month follow-up (p = .99).
Waitlist Control Group (following treatment).
The mixed-model analyses examining the scores of those in the Waitlist Control Group following treatment revealed significant effects for Time on both the GAD-7 (F3, 95.38 = 12.80, p<.001) and PHQ-9 (F3, 97.99 = 15.71, p<.001). Pairwise comparisons revealed significant improvements on both the PHQ-9 and GAD-7 from pre-treatment to post-treatment (p<.001), but no significant changes from post-treatment to 3-month follow-up (p>.05) or from 3-month follow-up to 12-month follow-up (p>.05). However, 3-month follow-up and 12-month follow-up scores on the PHQ-9 and GAD-7 remained significantly lower than pre-treatment scores (p<.001), indicating that the treatment effects were maintained.
Within- and between-group effect sizes for the outcome measures are included in Tables 1 and 2. For both the Overall and Comorbid Samples large within-group effect sizes (i.e., ≥0.80) were found for the TEG and moderate-to-large within-group (i.e., ≥0.50) effect sizes were found for the TG. Similar large within-group effect sizes were observed in the Control Group following treatment. Small and non-significant between-group effect sizes (i.e., ≤0.30) were found between the two Treatment Groups. Small-to-moderate between-group effect sizes were found between the Comorbid Samples (i.e., 0.30 to 0.50) at each of the time points, except at 12-month follow-up time point, when the effect sizes were small and non-significant (i.e., ≤0.30).
As shown in Table 3, post-treatment remission rates for the PHQ-9 and GAD-7 were sustained at the 12-month follow-up time points for the TEG (64.7% and 63.1%) and TG (68.6% and 60.8%).
As shown in Table 3, the post-treatment remission rates were also sustained at 12-month follow-up for the Comorbid Samples of the TEG (73%) and TG (74%).
Waitlist Control Group (following treatment).
The remission rates for the Control Group and the remission rates for the Comorbid Sample of the Waitlist Control Group following treatment were also sustained at 12-month follow-up.
At 12-month follow-up, up to 3.3% of TEG and 5.0% of TG participants obtained PHQ-9 or GAD-7 scores five or more points higher compared to pre-treatment. No Waitlist Control Group participants had scores five or more points higher after treatment at 12-month follow-up.
At 12-month follow-up, 3.0% of TEG and 1.8% of TG participants in the Comorbid Samples obtained PHQ-9 or GAD-7 scores five or more points higher compared to pre-treatment.
The primary finding from this study was that the self-guided Wellbeing Course resulted in clinically significant reductions in symptoms of anxiety and depression at post-treatment and these reductions were sustained at the 3- and 12-month follow-up time points. These findings are consistent with results from clinician-guided internet interventions, which demonstrate that benefits are sustained in the longer term ,  indicating the robust nature of internet interventions. The magnitude of clinical benefits from this self-guided intervention at 12-month follow-up were larger than those reported at follow-up than other self-guided internet treatments for depression or anxiety , . Benefits from the present trial also compare favourably with self-guided internet treatments for depression or anxiety where participants had contact with researchers at pre-treatment , , with trials of transdiagnostic internet interventions involving clinician guidance , ,  and trials of clinician-guided transdiagnostic interventions administered face-to-face . However, while the magnitude of gains reported here are encouraging, caution should be exercised in making comparisons across studies due to differences in samples, inclusion criteria, and methodologies.
An important secondary finding of the present study was that the benefits of automated emails, in increasing completion rates and in reducing symptoms in a subsample with elevated and comorbid symptoms, were no longer found at 12-month follow-up. In the present study, at 12-month follow-up, the TG obtained similar outcomes to the TEG on both the PHQ-9 and GAD-7 and both groups evidenced remission rates above 50% at 12-month follow-up in each group. Rates of deterioration at 12-month follow-up appeared larger in the TG, but the overall event rate was low (<5%).
The absence of benefits from automated emails appears to have been the result of continued improvements in the TG group, of whom an additional 25% completed the Course between the 3- and 12-month follow-up points; compared to 7% of the TEG group. Consequently, at 12-month follow-up, there was no difference between groups in the proportion who had read all five lessons or in the clinical outcomes of either the Overall or Comorbid Samples. The reasons for this increase in completion rates in the TG are unclear. However, it is possible that the automated assessment emails at post-treatment and 3-month follow-up prompted TG participants to return to the Course, although it is also likely that participants found the material in the Course sufficiently helpful in the first instance to justify their return.
These results raise the question of whether automated emails are helpful or necessary. Our findings suggest that automated emails facilitated more rapid symptom improvement with the Comorbid Sample who received automated emails relative to the Comorbid sample who did not receive emails. However, the results indicate that the automated emails were not absolutely necessary to achieve comparable outcomes in the longer term, as the TG group obtained similar outcomes as the TEG at the 12-month follow-up. This indicates that the decision about whether or not to include automated emails should be informed by the costs, risks and the benefits of their implementation and the value a service places on the speed of clinical improvements. Nevertheless, it is likely that more rapid reductions in symptoms would be associated with reduced health costs and lifetime burden of disease. Therefore, the inclusion of automated emails is recommended.
These results also extend debate and questions about the mechanisms of change in internet interventions . The magnitude of clinical benefits found in this study indicates that, at least for some patients, direct contact with a therapist is not necessary for clinically significant change to occur during internet treatment. This is not to say that therapeutic factors that facilitate engagement and adherence, such as the credibility of an intervention, reinforcement and unconditional positive regard, are not important. Rather, we contend that these and other factors can and should be integrated into the content, format and presentation of self-guided interventions. We suspect that as improvements are made to the content of internet-delivered psychotherapeutic interventions and in the delivery of that content, an increasing proportion of the population will be able to benefit from them. However, we caution against assuming that self-help materials can be easily developed and delivered. Just as therapists require careful training and supervised practice, internet intervention materials should also be carefully developed and improved through a process of repeated clinical evaluation and quality assurance cycles.
We also contend that these results provide support for pragmatic models of psychotherapeutic change, which emphasise the importance of systematically learning, practicing and adopting core psychological skills. These skills support participants to change unhelpful habits of thought and action, which, in turn, have broad positive impacts on emotion dysregulation. We suspect the use of case-enhanced learning strategies (e.g., detailed case studies) in the course content facilitated engagement and adherence. However, as noted by others, multiple other factors such as providing a clear deadline and expectations of feedback also facilitate adherence and clinical outcomes . It appears likely that systematic examination of other strategies to promote engagement and adherence in self-guided internet interventions, including prompts delivered by mobile technologies, will further improve outcomes in self-guided internet-delivered interventions and will also have utility across a broad range of health conditions .
These findings need to be considered in the context of the limitations of the overall study. The long-term effects of the study were not assessed alongside an active control group, although the large magnitude of the outcomes indicates there were strong treatment effects. The generalisability of these results is limited by the nature of the sample, which comprised people seeking internet-delivered treatment for anxiety and depression. However, it is important to note that a comparison of demographic and symptom profiles of participants in internet trials with those identified in epidemiological samples indicates that the results of internet trials may generalise to the wider population . Moreover, while symptom improvements were sustained, the present study relied on self-report measures rather than structured diagnostic interviews, which would have required clinical contact and may have affected outcomes. It is also important to note that the present study employed a specific transdiagnostic treatment intervention, the Wellbeing Course, which was designed and developed for use as either an entirely self-guided or clinician-guided Course. Consequently, it is possible that automated emails may facilitate greater adherence and clinical outcomes in less developed interventions or interventions initially developed to involve clinician guidance but subsequently used in a self-guided format.
The present results indicate that the benefits of the self-guided Wellbeing Course were sustained at 12-month follow-up. The inclusion of automated emails facilitated adherence, completion and more rapid symptom reduction during the Wellbeing Course in the immediate and short-term for those with comorbid symptoms. However, these emails did not improve outcomes any further in the longer term or for the overall group. We contend that automated emails are helpful, particularly for those with elevated symptoms, and should be considered when developing self-guided and clinician-guided interventions where possible. Overall, the Wellbeing Course was clinically effective and acceptable to consumers. These results are encouraging and indicate the public health potential of well-developed self-guided interventions.
Conceived and designed the experiments: NT BFD LJ PM RMR. Performed the experiments: NT BFD LJ. Analyzed the data: NT BFD LJ MDT. Contributed reagents/materials/analysis tools: NT BFD. Wrote the paper: NT BFD LJ PM MDT BW MG VF RK RMR.
- 1. Christensen H, Griffiths KM, Mackinnon AJ, Brittliffe K (2006) Online randomized controlled trial of brief and full cognitive behaviour therapy for depression. Psychol Med 36: 1737–46. doi: 10.1017/s0033291706008695
- 2. Farvolden P, Cunningham J, Selby P (2009) Using e-health programs to overcome barriers to the effective treatment of mental health and addiction problems. J Technol Hum Serv 27: 5–22. doi: 10.1080/15228830802458889
- 3. Clarke C, Eubanks D, Reid E, Kelleher C, O'Connor E, et al. (2005) Overcoming Depression on the Internet (ODIN) (2): a randomized trial of a self-help depression skills program with reminders. J Med Internet Res 7: e16. doi: 10.2196/jmir.7.2.e16
- 4. Clarke G, Reid E, Eubanks D, O'Connor E, DeBar L, et al. (2002) Overcoming Depression on the Internet (ODIN): a randomized controlled trial of an Internet depression skills intervention program. J Med Internet Res 4: e14. doi: 10.2196/jmir.4.3.e14
- 5. Titov N, Andrews G, Choi I, Schwencke G, Johnston L (2009) Internet-based cognitive behavior therapy for social phobia without clinical input is effective: a pragmatic RCT of two types of reminders. Aust N Z J Psychiatry 43: 913–9. doi: 10.1080/00048670903179228
- 6. Titov N, Andrews G, Schwencke G, Robinson E, Peters L, et al. (2010) An RCT of Internet CBT for social phobia with and without motivational enhancement strategies. Aust N Z J Psychiatry 44: 938–45. doi: 10.3109/00048674.2010.493859
- 7. Titov N, Andrews G, Choi I, Schwencke G, Mahoney J (2008) Shyness 3: An RCT of guided versus unguided Internet based CBT for social phobia. Aust N Z J Psychiatry 42: 1030–1040. doi: 10.1080/00048670802512107
- 8. Titov N, Dear BF, Johnston L, Lorian C, Zou J, et al. (2013) Improving adherence and clinical outcomes in self-guided internet treatment for anxiety and depression: randomised controlled trial. PLoS ONE 8(7): e62873. doi: 10.1371/journal.pone.0062873
- 9. Yonkers KA, Bruce SE, Dyck IR, Keller MB (2003) Chronicity, relapse, and illness-course of panic disorder, social phobia, and generalized anxiety disorder: Findings in men and women from 8 years of follow-up. Dep Anx 17: 173–9. doi: 10.1002/da.10106
- 10. Vittengl JR, Clark LA, Dunn TW, Jarrett RB (2007) Reducing relapse and recurrence in unipolar depression: a comparative meta-analysis of cognitive-behavioral therapy's effects. J Cons Clin Psych 75: 475–88. doi: 10.1037/0022-006x.75.3.475
- 11. Carlbring P, Nordgren LB, Furmark T, Andersson G (2009) Long-term outcome of Internet-delivered cognitive-behavioural therapy for social phobia: A 30-month follow-up. Behav Res Ther 47: 848–850. doi: 10.1016/j.brat.2009.06.012
- 12. Mackinnon A, Griffiths KM, Christensen H (2008) Comparative randomised trial of online cognitive-behavioural therapy and an information website for depression: 12-month outcomes. Br J Psychiatry 192: ; 130–134.
- 13. Titov N, Andrews G, Johnston J, Schwencke G, Choi I (2009) The Shyness program: Longer term benefits, cost-effectiveness, and acceptability. Aust N Z J Psychiatry 43: 36–44. doi: 10.1080/00048670802534424
- 14. Spek V, Cuijpers P, Nyklíček I, Smits N, Riper H, et al. (2008) One year follow-up results of a randomized controlled clinical trial on internet-based cognitive behavioural therapy for subthreshold depression in people over 50 years. Psych Med 38: 635–639. doi: 10.1017/s0033291707002590
- 15. Kroenke K, Spitzer RL, Williams JBW, Monahan PO, Lowe B (2007) Anxiety disorders in primary care: Prevalence, impairment, comorbidity, and detection. Ann Intern Med 146: 317–25. doi: 10.7326/0003-4819-146-5-200703060-00004
- 16. Dear B, Titov N, Sunderland M, McMillan D, Anderson T, et al. (2011) Psychometric Comparison of the GAD-7 and the PSWQ for Measure Response During Treatment of Generalised Anxiety Disorder. Cogn Behav Ther 40: 216–27. doi: 10.1080/16506073.2011.582138
- 17. Kroenke K, Spitzer RL, Williams JBW, Löwe B (2010) The Patient Health Questionnaire Somatic, Anxiety, and Depressive Symptom Scales: a systematic review. Gen Hosp Psychiatr 32: 345–359. doi: 10.1016/j.genhosppsych.2010.03.006
- 18. Kroenke K, Spitzer RL, Williams JBW (2001) The PHQ-9: Validity of a brief depression severity measure. J Gen Intern Med 16: 606–613. doi: 10.1046/j.1525-1497.2001.016009606.x
- 19. Titov N, Dear BF, McMillan D, Anderson T, Zou J, et al. (2011) Psychometric comparison of the PHQ-9 and BDI-II for measuring response during internet treatment for depression. Cognit Behav Ther 40: 126–136. doi: 10.1080/16506073.2010.550059
- 20. Gilbody S, Richards D, Brealey S, Hewitt C (2007) Screening for depression in medical settings with the Patient Health Questionnaire (PHQ): a diagnostic meta-analysis. J Gen Intern Med 22: 1596–1602. doi: 10.1007/s11606-007-0333-y
- 21. Schafer JL, Graham JW (2002) Missing data: our view of the state of the art. Psychol Meth 7: , 147–177.
- 22. Titov N, Dear B, Johnston L, Terides M (2012) Transdiagnostic internet treatment for anxiety and depression. Revista de Psicopatologia y Psicologia Clinica (RPPC) 17(3): , 237–260.
- 23. Richards DA, Suckling R (2009) Improving access to psychological therapies: Phase IV prospective cohort study. Br J Clin Psychol 48: 377–396. doi: 10.1348/014466509x405178
- 24. Meyer B, Berger T, Caspar F, Beevers CG, Andersson A, et al. (2009) Effectiveness of a novel integrative online treatment for depression (Deprexis): randomized controlled trial. J Med Internet Res 11: e15. doi: 10.2196/jmir.1151
- 25. Berger T, Hämmerli K, Gubser N, Andersson G, Caspar F (2011) Internet-based treatment of depression: A randomized controlled trial comparing guided with unguided self-help. Cogn Behav Ther 40: 2512–266. doi: 10.1080/16506073.2011.616531
- 26. Furmark T, Carlbring P, Hedman E, Sonnenstein A, Clevberer P, et al. (2009) Guided and unguided self-help for social anxiety disorder: randomised controlled trial. Br J Psychiatry 195: 440–447. doi: 10.1192/bjp.bp.108.060996
- 27. Johnston L, Titov N, Andrews G, Spence J, Dear BF (2011) A randomized controlled trial of a transdiagnostic Internet-delivered treatment for anxiety disorders. PLoS ONE 6(11): e28079. doi: 10.1371/journal.pone.0028079
- 28. Dear BF, Titov N, Schwencke G, Andrews G, Johnston L, et al.. (2011) An open trial of a brief transdiagnostic treatment for anxiety and depression. Behav Res Ther 49: ; 830–7.
- 29. Titov N, Dear BF, Schwencke G, Andrews G, Johnston L, et al. (2011) Transdiagnostic Internet treatment of anxiety and depression: A randomised controlled trial. Behav Res Ther 49: 441–452. doi: 10.1016/j.brat.2011.03.007
- 30. McEvoy PM, Nathan P, Norton PJ (2009) Efficacy of transdiagnostic treatments: A review of published outcome studies and future research directions. J Cogn Psychother 23: 20–33. doi: 10.1891/0889-8322.214.171.124
- 31. Andersson G, Carlbring P, Berger T, Almlöv J, Cuijpers P (2009) What makes internet therapy work? Cog Beh Ther 38: 55–60. doi: 10.1080/16506070902916400
- 32. Klasnja P, Pratt W (2012) Healthcare in the pocket: Mapping the space of mobile-phone health interventions. J Biomed Inform 45: 184–198. doi: 10.1016/j.jbi.2011.08.017
- 33. Titov N, Andrews G, Kemp A, Robinson E (2010) Characteristics of adults with anxiety or depression treated at an Internet clinic: Comparison with a national survey and an outpatient clinic. PLoS ONE 5(5): e10885. doi: 10.1371/journal.pone.0010885