Greater access to evidence-based psychological treatments is needed. This review aimed to evaluate whether internet-delivered psychological treatments for mood and anxiety disorders are efficacious, noninferior to established treatments, safe, and cost-effective for children, adolescents and adults.
We searched the literature for studies published until March 2013. Randomized controlled trials (RCTs) were considered for the assessment of short-term efficacy and safety and were pooled in meta-analyses. Other designs were also considered for long-term effect and cost-effectiveness. Comparisons against established treatments were evaluated for noninferiority. Two reviewers independently assessed the relevant studies for risk of bias. The quality of the evidence was graded using an international grading system.
A total of 52 relevant RCTs were identified whereof 12 were excluded due to high risk of bias. Five cost-effectiveness studies were identified and three were excluded due to high risk of bias. The included trials mainly evaluated internet-delivered cognitive behavioral therapy (I-CBT) against a waiting list in adult volunteers and 88% were conducted in Sweden or Australia. One trial involved children. For adults, the quality of evidence was graded as moderate for the short-term efficacy of I-CBT vs. waiting list for mild/moderate depression (d = 0.83; 95% CI 0.59, 1.07) and social phobia (d = 0.85; 95% CI 0.66, 1.05), and moderate for no efficacy of internet-delivered attention bias modification vs. sham treatment for social phobia (d = −0.04; 95% CI −0.24, 0.35). The quality of evidence was graded as low/very low for other disorders, interventions, children/adolescents, noninferiority, adverse events, and cost-effectiveness.
I-CBT is a viable treatment option for adults with depression and some anxiety disorders who request this treatment modality. Important questions remain before broad implementation can be supported. Future research would benefit from prioritizing adapting treatments to children/adolescents and using noninferiority designs with established forms of treatment.
Citation: Arnberg FK, Linton SJ, Hultcrantz M, Heintz E, Jonsson U (2014) Internet-Delivered Psychological Treatments for Mood and Anxiety Disorders: A Systematic Review of Their Efficacy, Safety, and Cost-Effectiveness. PLoS ONE 9(5): e98118. doi:10.1371/journal.pone.0098118
Editor: Christopher G. Davey, University of Melbourne, Australia
Received: February 23, 2014; Accepted: April 28, 2014; Published: May 20, 2014
Copyright: © 2014 Arnberg et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This work was funded by the Swedish Council on Health Technology Assessment (www.sbu.se). The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have read the journal's policy and have the following conflicts: Dr. Jonsson's co-authorship on one of the trials in the authors' review. This does not alter the authors' adherence to PLOS ONE policies on sharing data and materials.
A pressing challenge for mental health services is meeting the demand for the treatment of depression and anxiety disorders. Nearly 40% of the population is estimated to be in need of treatment at some time during their life for anxiety or depression . Each year 14–18% of the population across the age span suffer an anxiety disorder and 7–9% suffer from depression in the United States as well as in Europe , . Thus, meeting the needs of people suffering anxiety and depression with the current delivery methods is a gargantuan task –.
Only one third of depressed patients respond fully to pharmacotherapy  and patients prefer psychological to pharmacologic treatment for depression and anxiety at a 3∶1 rate . Fortunately, cognitive behavioral treatments are helpful for anxiety and depression for adults – and for children and adolescents . Other psychological therapies such as interpersonal and psychodynamic therapies have also been reported to produce significant improvements , , . However, limited access to qualified therapists restricts the utility of psychological treatments. In fact, of those with a serious problem as many as 50% in developed and 85% in undeveloped countries will simply go untreated . Of those who do receive treatment, rates of quality care are moderate to low for anxiety disorders .
The internet has offered a new avenue for providing psychological treatments, but the effectiveness of these treatments is still an issue. Most reviews to date have found support for the use of internet-delivered cognitive-behavioral therapy (I-CBT) –. For example, a meta-analytical review found that I-CBT was helpful for four distinct disorders . Similarly, Hedman et al.  reviewed randomized controlled trials (RCTs) of I-CBT and reported large effects for depression, social phobia and panic disorder. While ambitious, extant reviews nevertheless fail to address some key issues.
First, the quality of the evidence needs to be carefully considered. In previous reviews, when used at all, quality assessments were restricted to a few indices of the internal validity of the individual studies. A proper assessment of risk of bias is essential to avoid the risk of drawing false conclusions, however, and it can be justified to exclude studies of higher risk of bias from the synthesis . A recent example is a Cochrane review that found moderate clinical effect of exercise on depression when including all relevant trials regardless of risk of bias . However, when restricting the analysis to the trials with low risk of bias, the estimate indicated only a small effect of exercise that did not reach statistical significance.
Furthermore, investigators that conduct systematic reviews and meta-analyses are increasingly aware that not only individual studies but also the body of evidence needs to be systematically evaluated, because the confidence in the pooled effect estimates may be compromised not only by risk of bias in individual studies but also by several other factors (e.g., imprecision, inconsistency, indirectness, and publication bias) . The issue of quality assessment is compounded further if reviews are conducted by the trial authors themselves , . For example, the Cochrane Collaboration requires an independent assessment of eligibility and risk of bias by a second author not involved in the study/studies due to potential conflicts of interest . Also, as experts in the content area under review they may have pre-formed opinions that can influence their assessments . Given that the extant reviews were conducted by the trial authors themselves, the field would gain additional credibility from an independent evaluation.
Second, the issue of noninferiority has been largely ignored in previous reviews, but is necessary when comparing an existing evidence-based treatment (e.g., CBT) with a new one (e.g., I-CBT). In contrast to investigations of psychological therapy that involve new methods in areas where there is no known evidence-based treatment, the internet programs wisely employ known treatment techniques; only the manner of treatment delivery is altered. A greater reach and eventual cost savings could make internet therapies viable alternatives in healthcare. A critical issue, then, is whether they are noninferior to existing treatment. Noninferiority trials have gained increased attention to help in clinical decision making as the list of possible treatments grows, since a new treatment should be at least not inferior to existing evidence-based ones . The methodology for noninferiority trials differ from superiority trials  and there is a need to review the literature from this perspective. Previous reviews on internet-delivered treatments generally conclude that these treatments have effects equivalent to the established forms of treatments –, . However, the absence of a significant difference between two treatments in a clinical trial is not the same as a proof of noninferiority. Furthermore, formal indirect comparisons of treatment effect estimates between trials are only appropriate if the new and established treatments were compared against a reference that is similar both in methods and population , which, in this case seems to be a indeterminate presumption. We therefore believe that the field is ripe for an analysis that elaborates on the issue of noninferiority vs. superiority.
Noninferiority trials are difficult to design and execute well . Circumstances that strengthen inferences about superiority, because they increase similarities across treatment arms, can have the reverse effect on inferences of noninferiority. If a novel treatment is in fact inferior to established treatments, a trial with a sloppy design will be biased against finding this difference . Superiority trials mainly use intention-to-treat (ITT) samples whereas noninferiority should be demonstrated also in the per-protocol analysis because an ITT analysis tends to dilute differences. Furthermore, there should be a fairness of comparisons between the new and established treatment, such that the established treatment is implemented rigorously under conditions that do not compromise the assay sensitivity. For example, if many subjects in a trial have previously failed to respond to the control treatment, there would be a bias in favor of the new treatment . Noninferiority trials could also provide data for whether internet therapies are cost-effective, with important implications for healthcare.
Third, the previous reviews have largely ignored potential adverse events (e.g., harms, side effects, and deterioration), which may prove important for implementation of remotely delivered psychological treatments. Finally, reviews to date have focused on CBT, while trials of other treatments have begun to emerge .
The current review addresses all of the above issues. It has been conducted under the auspices of the Swedish Council on Health Technology Assessment (SBU), a government agency that has produced numerous systematic reviews evaluating the effects of various treatments (www.sbu.se/en/). The overall aim of this report is to provide a systematic review of the literature evaluating internet-delivered psychological treatment for mood and anxiety disorders with attention to methodological quality, consideration of the noninferiority perspective, and with ratings of the quality of the evidence using Grading of Recommendations Assessment, Development and Evaluation (GRADE)  by a freestanding council. Specifically, the following questions guided the review (additional questions were addressed in the governmental report):
- Is internet-delivered psychological treatment efficacious, safe and cost-effective for mood and anxiety disorders in children, adolescents and adults?
- Is internet-delivered treatment noninferior to established psychological treatments?
Protocol and registration
This systematic review was conducted at SBU. The inclusion criteria were pre-specified and a protocol was registered in advance internally at SBU (ref. no UTV2012/26), see Protocol S1.
Only published studies in English were considered for this review. The criteria for eligibility included the following characteristics.
Children, adolescents and adults with anxiety or mood disorders according to the manuals of the American Psychiatric Association  and the World Health Organization . The specific diagnoses included were major depressive disorder, dysthymia, bipolar disorder, social phobia, panic disorder, generalized anxiety disorder (GAD), posttraumatic stress disorder (PTSD), obsessive-compulsive disorder (OCD), specific phobia, and separation anxiety (in children and adolescents). Studies were excluded if the participants were selected primarily because of a specific physical illness.
Internet-delivered psychological treatments, defined as interventions based on an explicit psychological theory, not conducted at a clinic, and delivered to the patients via the internet. Any support had to be remotely delivered (e.g. email-like messages or telephone). The degree of support was categorized into pure self-help (no support), technician-assisted (e.g., non-clinical), or therapist-guided (i.e., clinical support).
Any established psychological treatments, waiting list, usual care, or attention control.
Change in symptoms of the primary disorder, adverse events, and cost per effect and per quality-adjusted life-years.
For short-term effects and risk of adverse events only RCTs were included. For long-term follow-up assessments (i.e., ≥6 months after post-assessment) RCTs and observational studies were included because of the ethical and practical dilemmas of conducting long-term RCTs. For cost-effectiveness data, economic evaluations based on individual-level data and decision models were eligible.
Electronic searches were conducted using Medical Subject Headings (MeSH) and relevant text word terms. The databases used were PubMed, Cochrane Library, CINAHL, PsycINFO, Psychology and Behavioral Sciences Collection (PBSC), TRIP database and CRD, up to March 4, 2013.
We used search terms for depression/mood and anxiety and for each disorder (e.g., panic, phobia), for a range of delivery methods (e.g., online, internet, web, computer, phone), and for therapy, psychotherapy, intervention, and terms for specific interventions (e.g., cognitive behavioral, psychodynamic, interpersonal). The detailed search strategies are found in Appendix S1.
Two reviewers independently screened the titles and abstracts identified by the search strategy. All studies of potential relevance according to the inclusion criteria were obtained in full text and two reviewers independently assessed them for inclusion. Any disagreements were resolved by discussions. Reference lists were screened for additional studies of relevance. Appendix S2 lists the efficacy and cost-effectiveness reports that were excluded after full-text reading.
Data collection process
From each included study of moderate or low risk of bias (see below), data was extracted and inserted in a table by one reviewer. A second reviewer audited the data extraction. Any disagreements were resolved by discussion.
Information was extracted from included trials on (1) participants (age, education, diagnosis, and method of diagnostic assessment); (2) treatment (including treatment paradigm, level of support, duration); (3) type of comparator (4) outcome measures of core symptoms; (5) adverse events or deterioration; (6) costs.
Risk of bias in individual studies
Two reviewers independently assessed the risk of bias with the use of checklists developed for each relevant study design . Risk of bias is the systematic tendency that any aspect of the study makes the estimated treatment effect deviate from its true value, that is, the extent to which results of an included trial can be believed. The checklist for RCTs used hereinis highly similar to the Cochrane Collaboration's tool for assessing risk of bias  and includes 31 items to consider for the randomization (methods and outcome; 3 items), treatment (blinding, compliance, therapists, confounds; 5 items) and assessment (blinding, reliability, validity, timing, analysis; 9 items) of the participants, dropout (size, balance, covariates, analysis; 5 items), reporting bias (protocol, primary/secondary outcome, adverse events, assessment, 6 items), and conflicts of interest (3 items). A rating of low, moderate or high risk of bias was given to each category of items and was combined into a global rating of the trial.
Trials that had a serious flaw were rated as high risk of bias; trials that met all or nearly all criteria were rated as low risk of bias, such as trials with a convincing comparator (e.g., an established treatment or a sham versions of attention bias modification) and no other obvious risk of bias; the remainder were rated as moderate risk of bias. Trials of moderate risk vary in their strengths and weaknesses: some trials likely provide valid results while others are only possibly valid. A high-risk trial is not valid; the results are at least as likely to reflect flaws in the study design as true differences among the trial arms. A fatal flaw may be reflected by one aspect introducing a high risk of bias or by failure to meet combinations of item criteria. The reviewers agreed on rules-of-thumb for decisions on categories and alert attention to trials that had, for example, N<30, dropout >20%, or unbalanced baseline characteristics. We included trials for the evaluation of long-term effects if they had a dropout rate of less than 30% and reported on other treatments during the follow-up period. For health-economic studies to be included they had to report both costs and effects. Any disagreements were resolved by consensus or by arbitration by a third reviewer. If necessary, study authors were solicited to provide additional information. Only studies with low or moderate risk of bias were used for further synthesis.
Planned method of analysis
We included as noninferiority designs all comparisons of internet-delivered treatment vs. established psychological therapies (e.g. I-CBT compared with individual therapist-led CBT). For these comparisons we used a predefined noninferiority margin of d = −0.2, chosen because it relates to a small effect size  and to ensure that noninferior treatments would retain an advantage over no treatment. All other comparisons were evaluated as superiority designs. Meta-analyses were carried out in RevMan 5. The calculations of the standardized mean differences were based on the groups' sample sizes, means and standard deviations at post-treatment. If the number of participants at post-treatment were not reported, the group sizes at randomization were used. Random effects models were used. All effect sizes in this report refer to between-group effects. Costs were converted to USD and the 2013 price-level .
Potential publication bias was assessed for plausibly effective interventions by inspecting funnel plots and by a trim-and-fill procedure , which yields an estimate of the effect size after taking bias into account (analyses performed in Comprehensive Meta-Analysis v2, Biostat Inc.).
Quality of evidence (GRADE)
The international grading system GRADE  was used to assess the quality of evidence for effects and safety with regard to groups of studies relevant to each treatment and support type, population, and disorder, according to the following four levels:
- High quality (⊕⊕⊕⊕) –We are very confident that the true effect lies close to that of the estimate of the effect.
- Moderate quality (⊕⊕⊕○) –We are moderately confident in the effect estimate: The true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different.
- Low quality (⊕⊕○○)–Our confidence in the effect estimate is limited: The true effect may be substantially different from the estimate of the effect.
- Very low quality (⊕○○○) – We have very little confidence in the effect estimate: The true effect is likely to be substantially different from the estimate of the effect.
In the GRADE system, evidence based on RCTs begins as high quality evidence, but may be rated down for several reasons, including study limitations, inconsistency of results, indirectness of evidence, imprecision or reporting bias. That is, for each type of treatment and support type, for each disorder and population, the quality of the evidence was assumed to be high at the outset, but subsequently rated down if there were limitations in the relevant studies. For example, there were three trials of CBT with clinical support vs. waiting list for adult participants diagnosed with panic disorder. Evidence of treatment efficacy start as being of high quality because the trials were RCTs, while study limitations (waiting list comparison [WLC]), inconsistency in the results (two trials show favorable effect, one shows no effect), and imprecision across studies (all three trials have small samples) entailed that the body of evidence finally received a low-quality rating. The quality of evidence was decided upon through discussions among the authors and input from an external group, the Quality and Priority Group at the agency. In line with agency guidelines we rated down for indirectness when only one RCT was included for a specific question, unless the included RCT was a multi-center trial.
We identified 52 relevant trials (54 reports), whereof 12 trials (13 reports) were excluded due to high risk of bias. The efficacy data thus included 39 reports with 40 RCTs of low or moderate risk of bias and 2 additional reports of long-term follow-ups of these trials that were included in the synthesis (Figure 1). Most trials recruited volunteers via advertisements, evaluated variations of therapist-guided I-CBT in self-help format carried out over 8–12 weeks and used a WLC (Table 1). The support was delivered via phone or email-like messages and took approximately 10–20 minutes per participant and week. Diagnoses were made mainly by using the MINI neuropsychiatric interview or the Structured Clinical Interview for DSM Axis-I Disorders (SCID) and the screening was performed in person or via telephone. The majority of the trials (88%) were conducted by teams from Australia or Sweden.
Mood disorders in adults
Nine trials were identified: eight had moderate risk of bias , – and one had high risk of bias (Appendix S2). The participants fulfilled criteria for a depressive episode, current or in partial remission, recurrent episodes, or dysthymia. No trials for bipolar disorder were found. Six trials included only participants with mild to moderate depression and six trials excluded participants who reported suicidal ideation.
None of the trials assessed noninferiority. Five evaluated the effect of I-CBT vs. a WLC , , , WLC and weekly symptom ratings , or WLC and access to an online discussion group . We found a large pooled effect for I-CBT as compared to a WLC (Figure 2). The quality of evidence was rated as moderate for therapist-guided I-CBT due to study limitations (WLC), see Table 2.
For the meta-analysis, the outcome chosen from each study was the Beck Depression Inventory I or II.
Three other trials were included that evaluated one intervention each: one trial with an intervention that combined components from acceptance and commitment therapy, behavioral activation, and mindfulness ; one with internet-delivered psychodynamic therapy (I-PDT) , and one with therapist-led I-CBT delivered via a chat interface . For each intervention, the quality of evidence was rated as very low. Four long-term follow-up assessments (five reports) were assessed as having a high risk of bias , , , , .
Anxiety disorders in adults
Sixteen trials were identified: three trials had low risk of bias –, 10 trials reported in 9 publications had moderate risk –, and 3 trials had high risk. One noninferiority trial with low risk of bias found that therapist-guided I-CBT was superior to live group CBT, with an effect size of d = 0.41 on the LSAS (blinded) . The 95% CI (0.03 to 0.78) was above our pre-defined noninferiority margin. We rated the quality of evidence as low because of imprecision (sample size) and indirectness (single trial).
Eight trials with moderate risk of bias evaluated the effect of therapist-guided I-CBT compared to a WLC –. The treatments conferred a large effect compared to WLC (Figure 3). The quality of evidence for therapist-guided I-CBT was rated as moderate due to study limitations (WLC). One report also evaluated whether therapist-guided I-CBT was superior to bibliotherapy . I-CBT was not found to be superior to bibliotherapy. The quality of evidence for guided I-CBT vs. bibliotherapy was rated as very low due to imprecision (small sample) and indirectness (single trial).
For the meta-analysis, the outcome chosen from each study was the Social Interaction Anxiety Scale.
Two trials (three reports) , ,  included long-term follow-ups of the treatment groups were assessed as having moderate risk of bias. Their findings suggested that participants' improvements persisted after 30 months  and 1 and 5 years , . The quality of evidence was assessed as very low due to risk of bias and imprecision. One trial found that unguided I-CBT was not superior to a WLC . The quality of evidence for unguided I-CBT for social phobia was rated as very low due to study limitations (WLC), imprecision (small sample), and indirectness (single trial).
Three trials, two of low ,  and one of moderate risk of bias , compared internet-delivered Attention Bias Modification (I-ABM) to an identical sham intervention. We found no clinically relevant pooled effect (Figure 4 includes one of three primary outcomes; plots were nearly identical for the Social Phobia Scale and the Liebowitz Social Anxiety Scale). The quality of evidence was rated as moderate for a lack of clinically meaningful effect of I-ABM (rated down due to imprecision, i.e., small sample).
Nine trials of I-CBT were identified. Five trials had moderate risk – and four had high risk of bias (e.g., due to differences among groups in baseline characteristics, sample sizes, dropout). One trial found no difference between therapist-guided I-CBT and live group CBT in participants recruited from a clinical population (d = 0.00) . Noninferiority was not established as the 95% CI (−0.41 to 0.41) included our predefined noninferiority margin of d = −0.20. One trial found no difference between I-CBT and live individual CBT . This trial was not designed as a noninferiority trial and the small sample limits the inferences to be made. The quality of evidence for the noninferiority of I-CBT vs. either individual or group CBT thus was rated as very low due to study limitations (e.g., insufficient information about treatment integrity), imprecision (small sample), and indirectness (single trial).
Three trials that compared therapist-guided I-CBT vs. a WLC with  or without ,  online information about panic found small to very large effects. No meta-analysis was undertaken because of the heterogeneity in outcome measures and effect sizes. We rated the quality of evidence as low because of study limitations (WLC, dropout) and imprecision (heterogeneous effect sizes, small samples).
Generalized anxiety disorder.
Four trials with moderate risk of bias were identified –. They evaluated therapist-guided I-CBT vs. a WLC. The pooled effect was large although heterogeneous across the trials (Figure 5). We rated the quality of evidence as low for the short-term effect because of study limitations (WLC) and imprecision (heterogeneous effect sizes, small samples).
For the meta-analysis, the outcome chosen from each study was the Penn State Worry Questionnaire.
One trial found that I-CBT with non-clinical support by a technician was more effective than a WLC . One trial included therapist-guided I-PDT . As for the I-CBT condition in this trial no effect was found for I-PDT vs. WLC. The quality of evidence for I-CBT with non-clinical support and therapist-guided I-PDT was rated as very low because of study limitations (WLC, only one technician), imprecision (small sample), and indirectness (single trial).
We identified one trial of moderate risk of bias . Four weeks of therapist-guided I-CBT did not outperform brief therapist-led exposure (one introductory session and one three-hour exposure session) according to a behavioral approach test in participants with spider phobia. The quality of evidence was rated as very low due to study limitations, imprecision (small sample), and indirectness (single trial).
Posttraumatic stress disorder.
Two relevant trials were identified; one trial with high risk of bias and one trial with moderate risk of bias that found that therapist-guided I-CBT was superior to WLC . The quality of evidence was rated as very low due to study limitations (WLC), imprecision (small sample), and indirectness (single trial).
We identified one trial of moderate risk of bias . Therapist-guided I-CBT conferred a large effect compared to supportive therapy online. The quality of evidence was rated as very low due to study limitations (no credible active comparison condition), imprecision (small sample), and indirectness (single trial).
Transdiagnostic interventions for anxiety and depression.
Six trials of moderate risk of bias were identified that included participants with mixed anxiety disorders and/or MDD –. Five trials found that therapist-guided I-CBT had moderate or large effects as compared to a WLC –, . No meta-analysis was performed due to the heterogeneity in outcome measures, diagnoses, and treatment protocols. We rated the quality of evidence as low for these interventions because of study limitations (WLC) and imprecision (small samples, heterogeneous effects and interventions). One trial included a 1- and 2-year follow-up, with results suggesting that the improvements lasted throughout the follow up . The quality of evidence was rated as very low due to study limitations (observational design, dropout) and imprecision (small sample). One trial recruited participants from an anxiety clinic and found no difference between unguided I-CBT and a WLC on the Patient Global Impression scale . The quality of evidence was rated as very low due to study limitations (WLC, dropout), imprecision (small sample), and indirectness (single trial).
Funnel plots and Duval and Tweedie's trim-and-fill procedure indicated no or trivial publication bias with respect to the pooled effect sizes for I-CBT for adults with depression, social phobia, and GAD.
Children and adolescents
We found four trials and excluded three due to high risk of bias because of various shortcomings. One trial of moderate risk of bias evaluated I-CBT for mixed anxiety disorders : 30% of the completers did no longer fulfill criteria for their primary anxiety diagnosis, compared to 10% in the WLC. The quality of evidence was rated as very low for the efficacy of internet-based psychological interventions for children and adolescents (Table 2).
Risk of adverse events
Eight trials provided information on intervention-associated risks for depression , , social phobia , , GAD , OCD , and transdiagnostic treatments , . The information provided was related to a worsening in symptoms and indicated that symptom worsening was present in 0–5% of treated participants and in 2–9% of participants in the comparison groups. The quality of evidence was rated as very low for the risk of adverse events following internet-based psychological interventions for both children and adults (⊕○○○).
Of the 139 studies screened for cost-effectiveness data, five trials met the eligibility criteria. Two had a moderate risk of bias ,  and three were excluded due to high risk of bias (e.g. incomplete information on costs; Appendix S2). One trial compared costs and effects between I-CBT and treatment as usual while on waiting list among patients with depression , and found that I-CBT had a cost per QALY of 29,384 USD compared to treatment as usual. At a willingness-to-pay for a QALY of 50,000 USD the probability was approximately 70% that I-CBT was cost-effective compared to treatment as usual. One trial compared costs and effects between I-CBT and group CBT among patients with social phobia . Compared to group CBT, I-CBT was associated with a lower cost per patient of 1,422 USD and 19% greater improvement on LSAS at the six-month follow-up. At a willingness-to-pay per additionally improved patient of 3,000 USD, the probability that I-CBT was cost-effective compared to group CBT was approximately 90%. The calculations of QALYs had not taken the time aspect of the effect on quality of life into account and are not presented.
In this review we assessed whether internet-delivered psychological treatments for mood and anxiety disorders are efficacious, noninferior to established treatments, associated with risk of adverse events, and cost-effective. We found limited to moderate evidence that for adults who seek out this treatment, therapist-guided I-CBT has a favorable short-term effect compared to waiting list for social phobia, panic disorder, generalized anxiety disorder, or mild to moderate major depression. We were not able to draw conclusions about noninferiority to proven treatments, long-term effects, adverse events, cost-effectiveness, or efficacy when given to children and adolescents.
Several reviews interpret the body of evidence such that I-CBT and established forms of CBT have comparable effects for mild to moderate depression and several anxiety disorders , , . In contrast, we found insufficient evidence to conclude whether I-CBT is noninferior to face-to-face treatment. There are important aspects that need to be attended to with regard to the methodology, and ethics, of conducting trials with a placebo/no-treatment arm when there are existing evidence-based treatments , . These issues notwithstanding, we found few trials that compared I-CBT to a face-to-face treatment. These trials were generally not adequately designed to evaluate questions of noninferiority , , with an exception of one trial, which provided tentative support for similar efficacy of therapist-guided I-CBT and group CBT for social phobia in adults .
There are a number of shortcomings with the existing trials that future studies would benefit from attending to. A common issue to these studies is that they were conducted by teams that developed the I-CBT program but had no role in developing the comparison face-to-face therapy. In addition, independent ratings of the quality of delivery of the therapy were not routinely included. Further, the face-to-face comparator was often group CBT and not individual CBT although the latter is generally the first-hand choice for anxiety and mood disorders , . The guidelines from the National Institute for Health and Care Excellence (NICE) do not support the notion of equivalence between internet-delivered and face-to-face treatment for social phobia , in part due to the aforementioned issues. More aptly designed trials are needed before we can answer clearly whether internet-delivered treatments are noninferior to face-to-face treatment. Furthermore, the lack of comparisons with established treatments provide scant data for cost-effectiveness analyses. Consequently, this review can provide no conclusions about the cost-effectiveness of I-CBT.
The diverging conclusions among extant reviews about equal efficacy between internet-delivered and face-to-face treatments highlight critical methodological aspects that set the present review apart from previous reviews , , . First, we used rigorous criteria for establishing noninferiority, whereas previous reviews seemingly have relied on subjective and indirect appraisal of the effect size differences. Second, we performed a systematic assessment of the body of evidence for each disorder  whereas previous reviews either used no formal assessment or relied only on the criteria stated by Chambless et al. , which indicate as evidence-based treatment any treatment that have been found superior to any comparison condition in two RCTs. The grading of the body of evidence that we used here entailed a reduced confidence in the results mainly due to the fact that studies were unblinded, used subjective outcome measures, were designed with waiting list or similar comparison groups, and included relatively small samples.
Third, we performed a comprehensive assessment of the risk of bias in the trials and excluded trials with high risk. Few trials for social phobia and depression were judged as having high risk of bias, which resulted in similar conclusions about short-term efficacy as the meta-analysis by Andrews et al.  Similarly, for PTSD, OCD, specific phobia, and transdiagnostic treatments only one trial (for PTSD) was excluded due to high risk. However, excluding high-risk studies resulted in fewer trials and a lower grading of the evidence for panic disorder than stated in previous reviews , . Of the four excluded publications on panic disorder two publications were from 2001, one from 2006, and one from 2008 (see Appendix S2). Given the technical progress in the field and that the reports represent studies planned and performed some years before publication, at least the 2001 publications are among the first in an emerging field and would have less resemblance of current and future practice of internet-delivered treatment packages.
We found only four relevant trials for children and adolescents, and three had high risk of bias. The three excluded trials concerned social anxiety, OCD, and diverse anxiety disorders (mainly GAD), respectively. Including them would not alter our conclusions. This turnout seems to reflect the slow progress in general among psychological interventions for children and adolescents . Although the low number of studies precludes quantitative meta-analysis, an equally important objective of a systematic review is to identify gaps in the literature. This could alert researchers and funding agencies to important research questions that are not given sufficient attention. The effect of internet-delivered interventions in general may be smaller among children , which stresses the need for more research specifically for this population.
Finally, the trials included in this review may seem few in comparison to the expanding number of publications in the literature. However, we only included studies of participants with diagnosed mood and anxiety disorders. There are many other studies on internet-treatments in which participants have not been subjected to a diagnostic interview. Several of those trials used unguided interventions, which may explain why so few trials of unguided interventions were included. Also, we did not pool studies across treatments and support types, or across disorders, and therefore each cluster of studies yielded a modest number of trials despite an impressive amount overall.
Remote delivery is one of several promising avenues for expanding the reach of psychological interventions . Indeed, a key impetus in much of the reviewed research is to improve accessibility to CBT  and attract those normally too shy to seek treatment and those without access to CBT . A central question, therefore, is whether internet-delivered treatment indeed attracts an underserved population. Among the trials of I-CBT for depression, 53–61% of participants had a history of psychological treatment , , . Among the anxiety trials 16–66% of participants had previously received psychological treatment –, , ,  and one-fourth had received CBT , . The data indicate that many depression trial participants already had access to treatment whereas this seemed to be less clear for anxiety disorders. The high level of educational attainment and employment rates among the participants raise concerns about whether the effects found in most RCTs can be generalized to those who today are underserved. Other questions also likely to be important to generalization concerns how these treatment programs can be implemented within the healthcare services and what type of changes that would be needed; for example, the training of existing therapists. Expanding the reach of psychological treatments is important . We therefore concur with the NICE guidance  and hope for further research that attend to these issues in more detail.
Several trials assessed long-term outcomes of the treatments. Yet, no clear conclusions could be drawn about long-term effects as these data were limited mainly due to the observational design, attrition, and the lack of data on participants' receipt of other treatments during the follow-up period. Only eight efficacy trials reported on deterioration, and no trial suggested that adverse events in a broader sense had been monitored. There is clearly a need for better reporting of risk of safety data , . Currently, the risk of reporting bias precludes conclusions about the risk-benefit ratio of the treatments, which is an important aspect of comparing treatments.
Correlational evidence suggests that therapist guidance is beneficial for the outcome , , . Less extensive support without adequate oversight of the patients' mental health status could also compromise patient safety. We therefore emphasize that evidence was found only for therapist-guided treatments. The lack of efficacy of I-ABM (also seen in a trial published after our final search ) compared to the effects of ABM in the laboratory , and of other remotely delivered therapies  further indicates the importance of attention to details about how interventions are delivered.
We believe that using only trials with low or moderate risk of bias is an improvement to previous reviews. We are mindful of the fact that the ratings of risk of bias were subjective, which hampers the replicability of our findings. However, it is broadly recognized as poor review practice to disregard study quality altogether , for example, because of the impact of quality on effect estimates . Instead of choosing a threshold approach, a quality-weighting approach can be used whereby low quality studies are included in the review and their influence is analyzed, thus avoiding selection bias. However, the assignment of quality weights is still fraught with subjectivity: unless rigorously implemented, it might increase the risk of over-inclusion bias and may result in inconsistency . In addition, the use of simple scoring sheets for assessing bias is not recommended . To minimize the uncertainty due to subjective judgments we performed the ratings according to best practice: The risk of conflicts of interests were minimized by the choice of independent reviewers and we used comprehensive score sheets developed for risk of bias ratings in individual trials and dual review; and we used the GRADE model for the overall assessment of the evidence . Our ratings of the strength of evidence are related not only to specific treatment packages and comparison conditions (Table 2), but also to the particular population of adults seeking out treatment themselves. The majority of the included trials were conducted in Sweden or Australia, which greatly increases external validity within these countries; however, also warranting caution before extrapolating these findings into healthcare services in other countries and cultural settings.
I-CBT for adults with mild to moderate depression and select anxiety disorders may complement existing services. More research is needed before conclusions can be drawn about the efficacy of internet-delivered treatment regarding other anxiety disorders, other treatment methods than CBT, the treatment of children, long-term effects, safety, cost-effectiveness, and noninferiority to proven forms of treatment. We believe that a shift is warranted from waiting list trials to using active comparators, particularly direct comparisons with established treatments. Nonetheless, more research is needed to understand what makes psychological treatments effective, and for whom. This field unfolds rapidly, however, and it may not be long until remaining questions can be satisfactorily answered.
Detailed search strategy.
List of reports excluded after full-text reading.
The authors would like to thank Anders Norlund, at the Swedish Council on Health Technology Assessment, for valuable contributions to the review of cost-effectiveness analyses and Hanna Olofsson, also at the agency, for diligent assistance with the literature search.
Conceived and designed the experiments: EH FA MH SL UJ. Performed the experiments: EH FA SL UJ. Analyzed the data: EH FA UJ. Contributed reagents/materials/analysis tools: EH FA SL UJ. Wrote the paper: EH FA MH SL UJ.
- 1. Wittchen HU, Jacobi F, Rehm J, Gustavsson A, Svensson M, et al. (2011) The size and burden of mental disorders and other disorders of the brain in Europe 2010. Eur Neuropsychopharmacol 21: 655–679.
- 2. Kessler RC, Chiu WT, Demler O, Walters EE (2005) Prevalence, severity, and comorbidity of 12-month DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry 62: 617–627.
- 3. Lépine JP, Briley M (2011) The increasing burden of depression. Neuropsychiatr Dis Treat 7: 3–7.
- 4. Kessler RC, Ruscio AM, Shear K, Wittchen HU (2010) Epidemiology of anxiety disorders. In: Stein MB, Steckler T, editors. Behavioral neurobiology of anxiety and its treatment. New York: Springer. pp. 21–35.
- 5. Kazdin AE, Blase SL (2011) Rebooting psychotherapy research and practice to reduce the burden of mental illness. Perspect Psychol Sci 6: 21–37.
- 6. Trivedi MH, Rush AJ, Wisniewski SR, Nierenberg AA, Warden D, et al. (2006) Evaluation of outcomes with citalopram for depression using measurement-based care in STAR*D: implications for clinical practice. Am J Psychiatry 163: 28–40.
- 7. McHugh RK, Whitton SW, Peckham AD, Welge JA, Otto MW (2013) Patient preference for psychological vs pharmacologic treatment of psychiatric disorders: a meta-analytic review. J Clin Psychiatry 74: 595–602.
- 8. Hofmann SG, Smits JA (2008) Cognitive-behavioral therapy for adult anxiety disorders: a meta-analysis of randomized placebo-controlled trials. J Clin Psychiatry 69: 621–632.
- 9. Butler AC, Chapman JE, Forman EM, Beck AT (2006) The empirical status of cognitive-behavioral therapy: a review of meta-analyses. Clin Psychol Rev 26: 17–31.
- 10. Tolin DF (2010) Is cognitive–behavioral therapy more effective than other therapies? A meta-analytic review. Clin Psychol Rev 30: 710–720.
- 11. James AC, James G, Cowdrey FA, Soler A, Choke A (2013) Cognitive behavioural therapy for anxiety disorders in children and adolescents. Cochrane Database Syst Rev 6: CD004690.
- 12. Cuijpers P, Geraedts AS, van Oppen P, Andersson G, Markowitz JC, et al. (2011) Interpersonal psychotherapy for depression: a meta-analysis. Am J Psychiatry 168: 581–592.
- 13. Shedler J (2010) The efficacy of psychodynamic psychotherapy. Am Psychol 65: 98–109.
- 14. Demyttenaere K, Bruffaerts R, Posada-Villa J, Gasquet I, Kovess V, et al. (2004) Prevalence, severity, and unmet need for treatment of mental disorders in the World Health Organization World Mental Health Surveys. JAMA 291: 2581–2590.
- 15. Stein MB (2004) Quality of care for primary care patients with anxiety disorders. Am J Psychiatry 161: 2230–2237.
- 16. Spek V, Cuijpers P, Nyklicek I, Riper H, Keyzer J, et al. (2007) Internet-based cognitive behaviour therapy for symptoms of depression and anxiety: a meta-analysis. Psychol Med 37: 319–328.
- 17. Griffiths KM, Farrer L, Christensen H (2010) The efficacy of internet interventions for depression and anxiety disorders: a review of randomised controlled trials. Med J Aust 192: S4–11.
- 18. Andersson G (2009) Using the Internet to provide cognitive behaviour therapy. Behav Res Ther 47: 175–180.
- 19. Andrews G, Cuijpers P, Craske MG, McEvoy P, Titov N (2010) Computer therapy for the anxiety and depressive disorders is effective, acceptable and practical health care: a meta-analysis. PLoS One 5: e13196.
- 20. Hedman E, Ljotsson B, Lindefors N (2012) Cognitive behavior therapy via the Internet: a systematic review of applications, clinical efficacy and cost-effectiveness. Expert Rev Pharmacoecon Outcomes Res 12: 745–764.
- 21. Guyatt GH, Oxman AD, Vist G, Kunz R, Brozek J, et al. (2011) GRADE guidelines: 4. Rating the quality of evidence–study limitations (risk of bias). J Clin Epidemiol 64: 407–415.
- 22. Cooney GM, Dwan K, Greig CA, Lawlor DA, Rimer J, et al. (2013) Exercise for depression. Cochrane Database Syst Rev 9: CD004366.
- 23. Balshem H, Helfand M, Schunemann HJ, Oxman AD, Kunz R, et al. (2011) GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol 64: 401–406.
- 24. Andersson G, Cuijpers P (2009) Internet-based and other computerized psychological treatments for adult depression: a meta-analysis. Cogn Behav Ther 38: 196–205.
- 25. Collaboration TC (2014) Cochrane editorial and publishing policy resource: Conflicts of interest and Cochrane Reviews. Oxford, UK: The Cochrane Collaboration.
- 26. Higgins J, Altman D, Sterne JAC (editors) (2011) Chapter 8: Assessing risk of bias in included studies. In: Higgins J, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 510 (updated March 2011): The Cochrane Collaboration.
- 27. Schumi J, Wittes JT (2011) Through the looking glass: understanding non-inferiority. Trials 12: 106.
- 28. Moher D, Pham B, Jones A, Cook DJ, Jadad AR, et al. (1998) Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 352: 609–613.
- 29. D'Agostino RB Sr, Massaro JM, Sullivan LM (2003) Non-inferiority trials: design concepts and issues - the encounters of academic consultants in statistics. Stat Med 22: 169–186.
- 30. International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH) (2000) Choice of control group and related issues in clinical trials: ICH harmonised tripartite guideline. Report nr E10. Geneva: ICH. Cited 23 Nov 2013. Available from: www.ich.org
- 31. Johansson R, Ekbladh S, Hebert A, Lindström M, Möller S, et al. (2012) Psychodynamic guided self-help for adult depression through the internet: a randomised controlled trial. PLoS One 7: e38021.
- 32. American Psychiatric Association (1994) Diagnostic and statistical manual of mental disorders. 4th edition. Washington, DC: Author.
- 33. World Health Organization (1992) ICD-10: International statistical classification of diseases and related health problems. 10th edition. Geneva: Author.
- 34. Swedish Council on Health Technology Assessment (SBU) (2013) Utvärdering av metoder i hälso- och sjukvården: En handbok. [Evaluation and synthesis of methods in healthcare: A handbook]. Stockholm, Sweden: SBU Swedish.
- 35. Cohen J (1988) Statistical power analysis for the behavioral sciences. 2nd edition. Hillsdale, NJ: Erlbaum. 567 p.
- 36. Shemilt I, Thomas J, Morciano M (2010) A web-based tool for adjusting costs to a specific target currency and price year. Evid Policy 6: 51–59.
- 37. Duval S, Tweedie R (2000) Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 56: 455–463.
- 38. Carlbring P, Hagglund M, Luthstrom A, Dahlin M, Kadowaki A, et al. (2013) Internet-based behavioral activation and acceptance-based treatment for depression: a randomized controlled trial. J Affect Disord 148: 331–337.
- 39. Vernmark K, Lenndin J, Bjarehed J, Carlsson M, Karlsson J, et al. (2010) Internet administered guided self-help versus individualized e-mail therapy: a randomized trial of two versions of CBT for major depression. Behav Res Ther 48: 368–376.
- 40. Johansson R, Sjoberg E, Sjogren M, Johnsson E, Carlbring P, et al. (2012) Tailored vs. standardized internet-based cognitive behavior therapy for depression and comorbid symptoms: a randomized controlled trial. PLoS One 7: e36905.
- 41. Titov N, Andrews G, Davies M, McIntyre K, Robinson E, et al. (2010) Internet treatment for depression: a randomized controlled trial comparing clinician vs. technician assistance. PLoS One 5: e10939.
- 42. Choi I, Zou J, Titov N, Dear BF, Li S, et al. (2012) Culturally attuned Internet treatment for depression amongst Chinese Australians: a randomised controlled trial. J Affect Disord 136: 459–468.
- 43. Kessler D, Lewis G, Kaur S, Wiles N, King M, et al. (2009) Therapist-delivered Internet psychotherapy for depression in primary care: a randomised controlled trial. Lancet 374: 628–634.
- 44. Berger T, Hammerli K, Gubser N, Andersson G, Caspar F (2011) Internet-based treatment of depression: a randomized controlled trial comparing guided with unguided self-help. Cogn Behav Ther 40: 251–266.
- 45. Andersson G, Hesser H, Hummerdal D, Bergman-Nordgren L, Carlbring P (2013) A 3.5-year follow-up of Internet-delivered cognitive behavior therapy for major depression. J Ment Health 22: 155–164.
- 46. Carlbring P, Apelstrand M, Sehlin H, Amir N, Rousseau A, et al. (2012) Internet-delivered attention bias modification training in individuals with social anxiety disorder: a double blind randomized controlled trial. BMC Psychiatry 12: 66.
- 47. Hedman E, Andersson G, Ljotsson B, Andersson E, Ruck C, et al. (2011) Internet-based cognitive behavior therapy vs. cognitive behavioral group therapy for social anxiety disorder: a randomized controlled non-inferiority trial. PLoS One 6: e18001.
- 48. Boettcher J, Berger T, Renneberg B (2012) Internet-based attention training for social anxiety: a randomized controlled trial. Cognit Ther Res 36: 522–536.
- 49. Andersson G, Carlbring P, Holmstrom A, Sparthan E, Furmark T, et al. (2006) Internet-based self-help with therapist feedback and in vivo group exposure for social phobia: a randomized controlled trial. J Consult Clin Psychol 74: 677–686.
- 50. Berger T, Hohl E, Caspar F (2009) Internet-based treatment for social phobia: a randomized controlled trial. J Clin Psychol 65: 1021–1035.
- 51. Titov N, Andrews G, Schwencke G, Drobny J, Einstein D (2008) Shyness 1: distance treatment of social phobia over the internet. Aust N Z J Psychiatry 42: 585–594.
- 52. Titov N, Andrews G, Schwencke G (2008) Shyness 2: treating social phobia online: replication and extension. Aust N Z J Psychiatry 42: 595–605.
- 53. Titov N, Andrews G, Choi I, Schwencke G, Mahoney A (2008) Shyness 3: randomized controlled trial of guided versus unguided internet-based CBT for social phobia. Aust N Z J Psychiatry 42: 1030–1040.
- 54. Andersson G, Carlbring P, Furmark T (2012) S. O. F. I. E. Research Group (2012) Therapist experience and knowledge acquisition in internet-delivered CBT for social anxiety disorder: a randomized controlled trial. PLoS One 7: e37411.
- 55. Carlbring P, Gunnarsdottir M, Hedensjo L, Andersson G, Ekselius L, et al. (2007) Treatment of social phobia: randomised trial of internet-delivered cognitive-behavioural therapy with telephone support. Br J Psychiatry 190: 123–128.
- 56. Furmark T, Carlbring P, Hedman E, Sonnenstein A, Clevberger P, et al. (2009) Guided and unguided self-help for social anxiety disorder: randomised controlled trial. Br J Psychiatry 195: 440–447.
- 57. Neubauer K, von Auer M, Murray E, Petermann F, Helbig-Lang S, et al. (2013) Internet-delivered attention modification training as a treatment for social phobia: a randomized controlled trial. Behav Res Ther 51: 87–97.
- 58. Carlbring P, Nordgren LB, Furmark T, Andersson G (2009) Long-term outcome of Internet-delivered cognitive-behavioural therapy for social phobia: a 30-month follow-up. Behav Res Ther 47: 848–850.
- 59. Hedman E, Furmark T, Carlbring P, Ljotsson B, Ruck C, et al. (2011) A 5-year follow-up of internet-based cognitive behavior therapy for social anxiety disorder. J Med Internet Res 13: e39.
- 60. Bergstrom J, Andersson G, Ljotsson B, Ruck C, Andreewitch S, et al. (2010) Internet-versus group-administered cognitive behaviour therapy for panic disorder in a psychiatric setting: a randomised trial. BMC Psychiatry 10: 54.
- 61. Wims E, Titov N, Andrews G, Choi I (2010) Clinician-assisted internet-based treatment is effective for panic: a randomized controlled trial. Aust N Z J Psychiatry 44: 599–607.
- 62. Carlbring P, Bohman S, Brunt S, Buhrman M, Westling BE, et al. (2006) Remote treatment of panic disorder: a randomized trial of internet-based cognitive behavior therapy supplemented with telephone calls. Am J Psychiatry 163: 2119–2125.
- 63. Carlbring P, Nilsson-Ihrfelt E, Waara J, Kollenstam C, Buhrman M, et al. (2005) Treatment of panic disorder: live therapy vs. self-help via the Internet. Behav Res Ther 43: 1321–1333.
- 64. Klein B, Richards JC, Austin DW (2006) Efficacy of internet therapy for panic disorder. J Behav Ther Exp Psychiatry 37: 213–238.
- 65. Andersson G, Paxling B, Roch-Norlund P, Ostman G, Norgren A, et al. (2012) Internet-based psychodynamic versus cognitive behavioral guided self-help for generalized anxiety disorder: a randomized controlled trial. Psychother Psychosom 81: 344–355.
- 66. Paxling B, Almlov J, Dahlin M, Carlbring P, Breitholtz E, et al. (2011) Guided internet-delivered cognitive behavior therapy for generalized anxiety disorder: a randomized controlled trial. Cogn Behav Ther 40: 159–173.
- 67. Robinson E, Titov N, Andrews G, McIntyre K, Schwencke G, et al. (2010) Internet treatment for generalized anxiety disorder: a randomized controlled trial comparing clinician vs. technician assistance. PLoS One 5: e10942.
- 68. Titov N, Andrews G, Robinson E, Schwencke G, Johnston L, et al. (2009) Clinician-assisted internet-based treatment is effective for generalized anxiety disorder: randomized controlled trial. Aust N Z J Psychiatry 43: 905–912.
- 69. Andersson G, Waara J, Jonsson U, Malmaeus F, Carlbring P, et al. (2009) Internet-based self-help versus one-session exposure in the treatment of spider phobia: a randomized controlled trial. Cogn Behav Ther 38: 114–120.
- 70. Spence J, Titov N, Dear BF, Johnston L, Solley K, et al. (2011) Randomized controlled trial of internet-delivered cognitive behavioral therapy for posttraumatic stress disorder. Depress Anxiety 28: 541–550.
- 71. Andersson E, Enander J, Andren P, Hedman E, Ljotsson B, et al. (2012) Internet-based cognitive behaviour therapy for obsessive-compulsive disorder: a randomized controlled trial. Psychol Med 42: 2193–2203.
- 72. Johnston L, Titov N, Andrews G, Spence J, Dear BF (2011) A RCT of a transdiagnostic internet-delivered treatment for three anxiety disorders: examination of support roles and disorder-specific outcomes. PLoS One 6: e28079.
- 73. Titov N, Andrews G, Johnston L, Robinson E, Spence J (2010) Transdiagnostic internet treatment for anxiety disorders: a randomized controlled trial. Behav Res Ther 48: 890–899.
- 74. Carlbring P, Maurin L, Torngren C, Linna E, Eriksson T, et al. (2011) Individually-tailored, Internet-based treatment for anxiety disorders: a randomized controlled trial. Behav Res Ther 49: 18–24.
- 75. Bell CJ, Colhoun HC, Carter FA, Frampton CM (2012) Effectiveness of computerised cognitive behaviour therapy for anxiety disorders in secondary care. Aust N Z J Psychiatry 46: 630–640.
- 76. Titov N, Dear BF, Schwencke G, Andrews G, Johnston L, et al. (2011) Transdiagnostic internet treatment for anxiety and depression: a randomised controlled trial. Behav Res Ther 49: 441–452.
- 77. Newby JM, Mackenzie A, Williams AD, McIntyre K, Watts S, et al. (2013) Internet cognitive behavioural therapy for mixed anxiety and depression: a randomized controlled trial and evidence of effectiveness in primary care. Psychol Med: 1–14.
- 78. March S, Spence SH, Donovan CL (2009) The efficacy of an internet-based cognitive-behavioral therapy intervention for child anxiety disorders. J Pediatr Psychol 34: 474–487.
- 79. Hedman E, Andersson E, Ljotsson B, Andersson G, Ruck C, et al. (2011) Cost-effectiveness of Internet-based cognitive behavior therapy vs. cognitive behavioral group therapy for social anxiety disorder: results from a randomized controlled trial. Behav Res Ther 49: 729–736.
- 80. Hollinghurst S, Peters TJ, Kaur S, Wiles N, Lewis G, et al. (2010) Cost-effectiveness of therapist-delivered online cognitive-behavioural therapy for depression: randomised controlled trial. Br J Psychiatry 197: 297–304.
- 81. Andersson G, Carlbring P, Berger T, Almlöv J, Cuijpers P (2009) What makes Internet therapy work? Cogn Behav Ther 38 Suppl 155–60.
- 82. Kimmelman J, Weijer C, Meslin EM (2009) Helsinki discords: FDA, ethics, and international drug trials. Lancet 373: 13–14.
- 83. Hey SP, Weijer C (2013) Assay sensitivity and the epistemic contexts of clinical trials. Perspect Biol Med 56: 1–17.
- 84. Rothmann MD, Wiens BL, Chan ISF (2011) Design and analysis of non-inferiority trials. Boca Raton, FL: CRC Press. 438 p.
- 85. National Institute for Health and Care Excellence (2009) Depression: the treatment and management of depression in adults (partial update of NICE clinical guideline 23). Clinical guideline 90 . Available from: http://guidance.nice.org.uk/cg90.
- 86. National Institute for Health and Care Excellence (2013) Social anxiety disorder: recognition, assessment and treatment. Clinical guideline 159 . Available: http://guidance.nice.org.uk/cg159.
- 87. Hedman E, Ljótsson B, Lindefors N (2012) Cognitive behavior therapy via the Internet: a systematic review of applications, clinical efficacy and cost–effectiveness. Expert Rev Pharmacoecon Outcomes Res 12: 745–764.
- 88. Chambless DL, Baker MJ, Baucom DH, Beutler LE, Calhoun KS, et al. (1998) Update on empirically validated therapies, II. Clinical Psychologist 51: 3–16.
- 89. Sloan DM, Gallagher MW, Feinstein BA, Lee DJ, Pruneau GM (2011) Efficacy of telehealth treatments for posttraumatic stress-related symptoms: a meta-analysis. Cogn Behav Ther 40: 111–125.
- 90. Arnberg FK, Alaie I, Parling T, Jonsson U (2013) Recent randomized controlled trials of psychological interventions in healthcare: a review of their quantity, scope, and characteristics. J Psychosom Res 75: 401–408.
- 91. Barak A, Hen L, Boniel-Nissim M, Shapira N (2008) A comprehensive review and a meta-analysis of the effectiveness of Internet-based psychotherapeutic interventions. J Technol Hum Serv 26: 109–160.
- 92. Carlbring P, Westling BE, Ljungstrand P, Ekselius L, Andersson G (2001) Treatment of panic disorder via the Internet: a randomized trial of a self-help program. Behav Ther 32: 751–764.
- 93. Peterson AL, Roache JD, Raj J, Young-McCaughan S, for the STRONG STAR Consortium The need for expanded monitoring of adverse events in behavioral health clinical trials. Contemp Clin Trials 34: 152–154.
- 94. Jonsson U, Alaie I, Parling T, Arnberg FK (2014) Reporting of harms in randomized controlled trials of psychological interventions for mental and behavioral disorders: A review of current practice. Contemp Clin Trials 38: 1–8.
- 95. Palmqvist B, Carlbring P, Andersson G (2007) Internet-delivered treatments with or without therapist input: does the therapist factor have implications for efficacy and cost? Expert Rev Pharmacoecon Outcomes Res 7: 291–297.
- 96. Johansson R, Andersson G (2012) Internet-based psychological treatments for depression. Expert Rev Neurother 12: 861–870.
- 97. Boettcher J, Leek L, Matson L, Holmes EA, Browning M, et al. (2013) Internet-based attention bias modification for social anxiety: a randomised controlled comparison of training towards negative and training towards positive cues. PLoS One 8: e71760.
- 98. Hakamata Y, Lissek S, Bar-Haim Y, Britton JC, Fox NA, et al. (2010) Attention bias modification treatment: a meta-analysis toward the establishment of novel treatment for anxiety. Biol Psychiatry 68: 982–990.
- 99. Osenbach JE, O'Brien KM, Mishkind M, Smolenski DJ (2013) Synchronous telehealth technologies in psychotherapy for depression: a meta-analysis. Depress Anxiety 30: 1058–1067.
- 100. Greenland S, O'Rourke K (2001) On the bias produced by quality scores in meta-analysis, and a hierarchical view of proposed solutions. Biostatistics 2: 463–471.