https://orcid.org/0000-0003-0480-9336
Figures
Abstract
Background
Anxiety disorders are highly prevalent and cause substantial economic burden. Blended cognitive-behavioural therapy (bCBT), which integrates Internet-based CBT and face-to-face CBT (ftfCBT), is an attractive and potentially cost-saving treatment alternative to conventional CBT for patients with anxiety disorders in specialised mental health care. However, little is known about the effectiveness of bCBT in routine care. We examined the acceptability, effectiveness and cost-effectiveness of bCBT versus ftfCBT in outpatient specialised care to patients with panic disorder, social anxiety disorder and generalised anxiety disorder.
Methods and findings
Patients with anxiety disorders were randomised to bCBT (n = 52) or ftfCBT (n = 62). Acceptability of bCBT and ftfCBT were evaluated by assessing treatment preference, adherence, satisfaction and therapeutic alliance. Costs and effects were assessed at post-treatment and one-year follow-up. Primary outcome measure was the Beck Anxiety Inventory (BAI). Secondary outcomes were depressive symptoms, general psychopathology, work and social adjustment, quality of life and mastery. Incremental cost-effectiveness ratios (ICERs) were computed from societal and healthcare perspectives by calculating the incremental costs per incremental quality-adjusted life year (QALY). No significant differences between bCBT and ftfCBT were found on acceptability or effectiveness measures at post-treatment (Cohen’s d between-group effect size on BAI = 0.15, 95% CI −0.30 to 0.60) or at one-year follow-up (d = −0.38, 95% CI −0.84 to 0.09). The modelled point estimates of societal costs (bCBT €10945, ftfCBT €10937) were higher and modelled point estimates of direct medical costs (bCBT €3748, ftfCBT €3841) were lower in bCBT. The acceptability curves showed that bCBT was expected to be a cost-effective intervention. Results should be carefully interpreted due to the small sample size.
Citation: Romijn G, Batelaan N, Koning J, van Balkom A, de Leeuw A, Benning F, et al. (2021) Acceptability, effectiveness and cost-effectiveness of blended cognitive-behavioural therapy (bCBT) versus face-to-face CBT (ftfCBT) for anxiety disorders in specialised mental health care: A 15-week randomised controlled trial with 1-year follow-up. PLoS ONE 16(11): e0259493. https://doi.org/10.1371/journal.pone.0259493
Editor: Ethan Moitra, Brown University, UNITED STATES
Received: December 12, 2019; Accepted: October 19, 2021; Published: November 12, 2021
Copyright: © 2021 Romijn et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Study data are archived at GGZ inGeest and can be made available to interested researchers upon request. Requests can be directed to the corresponding author Geke Romijn (g.a.romijn@vu.nl) or co-author Heleen Riper (h.riper@vu.nl). A non author contact where data requests can be sent is Stasja Draisma, senior researcher at GGZ inGeest (s.draisma@ggzingeest.nl). We are unable to place data in a public repository due to ethical restraints: sharing of individual participant data was not included in the informed consent approved by the Medical Ethics Committee of the Vrije Universiteit Medical Centre, Amsterdam (registration number 2015.073).
Funding: This study was funded by the Netherlands Organisation for Health Research and Development, ZonMw (project number 837002505). JK and HR obtained funding for the study. The funding body had no influence in the design, execution, analysis or interpretation of the results of this study.
Competing interests: The authors have declared that no competing interests exist.
Background
Anxiety disorders are highly prevalent, and they are associated with considerable individual suffering and a high economic burden [1–4]. The disorders can be treated effectively with cognitive-behavioural therapy (CBT) [5, 6]. Despite the demonstrated effectiveness of CBT, fewer than half of the people with anxiety disorders receive appropriate treatment [7]. Reasons for undertreatment include stigmatisation, lack of trained therapists and the costs of therapy [8].
One strategy to expand access to evidence-based therapy while lowering treatment costs could be Internet-delivered CBT (iCBT). Patients receiving iCBT usually work their way through an online modularised programme, with or without online therapist assistance [9]. iCBT has been found effective for several anxiety disorders [10–13], and there are indications for its cost-effectiveness [14]. However, most evidence thus far derives from research outside routine clinical care settings. It has not been established that the promising results from those effectiveness studies can be extrapolated to samples in routine care. For example, in a recent meta-analysis [12], a large effect size (g = 0.79) was found for anxiety symptom reduction by iCBT as compared with waitlisted controls in samples recruited from the community, while a small effect size (g = 0.28) was found in the same comparison in routine care populations. A possible explanation for the discrepancy was the greater treatment adherence in self-referred samples recruited from the community and the stricter exclusion criteria in studies with such samples.
The low uptake of iCBT in routine care complicates the investigation of effectiveness in real-world settings. Reported reasons for therapists’ reluctance to use iCBT are their concerns about the therapeutic relationship [15] and low treatment adherence, especially in patients with high symptom severity [16, 17]. Blended CBT (bCBT) combines iCBT and ftfCBT into a single standardised treatment protocol [18] and could potentially alleviate some of the aforementioned limitations associated with iCBT, while partly or fully preserving the advantages. It could help provide an attractive, and potentially cost-saving, treatment alternative for use in conventional mental health care settings. For one thing, bCBT has been found to be better received by both providers and patients than iCBT, because the face-to-face contact in the blended format makes the treatment more personal, better addresses the needs of patients with complex symptomatology, and may help improve adherence rates [15, 19–21]. A further possible advantage is that online components can be integrated into routine practice more gradually [22], making the blended format easier than iCBT to adopt for application in routine care.
Although bCBT thus seems a promising alternative to both iCBT and ftfCBT, little is known so far about the clinical and cost benefits of blended interventions for anxiety disorders. In a feasibility randomised controlled trial (RCT) comparing bCBT (n = 18) with ftfCBT (n = 18) for panic disorder, no difference was found between bCBT and ftfCBT in reducing anxiety symptoms [23].
As bCBT could possibly reduce therapist time [24] and improve self-management competencies of patients in comparison with ftfCBT, providing bCBT to patients with severe anxiety disorders in specialised mental health care might lead to equal clinical effectiveness results at lower treatment costs. We thus hypothesised that bCBT is more cost-effective than ftfCBT. We undertook a randomised controlled trial to investigate the acceptability and the clinical and cost-effectiveness of bCBT for patients with panic disorder (PD), social anxiety disorder (SAD) and generalised anxiety disorder (GAD) in outpatient specialised mental health care. The current paper describes the acceptability, the post-treatment and 12-month clinical effectiveness, and the 12-month cost-effectiveness of bCBT versus ftfCBT from both a societal and a healthcare perspective.
Methods
Study design and participants
The study design was a parallel-group randomised controlled trial. The purpose was to assess acceptability, effectiveness and cost-effectiveness of bCBT compared with ftfCBT in patients with panic disorder, social phobia or generalised anxiety disorder in routine specialised mental health care. Assessments took place at post-treatment and at one-year follow-up, respectively 15 and 52 weeks after baseline. Patients who are referred to specialized mental health care in the Netherlands are suffering from serious mental disorders [25]. Hence, participants were likely to have received psychological treatment within primary care before they were enrolled in this trial. Patients in both treatment conditions were allowed to receive other supporting therapy after the intervention.
Participant inclusion criteria were (i) age 18 or older and (ii) satisfaction of the DSM-IV criteria for panic disorder (with or without agoraphobia), social anxiety disorder or generalised anxiety disorder, as diagnosed with the Structured Clinical Interview for DSM-IV Axis I Disorders (SCID-I) [26], or the Mini-International Neuropsychiatric Interview, Plus version (MINI-Plus) [27, 28]. Exclusion criteria were (i) inadequate proficiency in Dutch, (ii) lack of e-mail address or computer with Internet access and (iii) presence of a psychotic or bipolar disorder, substance dependence or a high risk for suicide. Psychotropic medication use was allowed.
A detailed study protocol has been published elsewhere [29]. The protocol was approved by the Medical Ethics Committee of the Vrije Universiteit Medical Centre, Amsterdam (registration number 2015.073), and registered in the Netherlands Trial Register (NTR4912). The study protocol and supporting CONSORT checklist and CHEERS checklist for this trial are available as supporting information; see S1 Appendix (CONSORT Checklist), S2 Appendix (CHEERS Checlist) and S3 Appendix (Study Protocol).
Recruitment
Recruitment took place between November 2015 and July 2017 at outpatient departments of four specialised mental health care centres in the Netherlands. Mental health professionals who conducted the therapy intake session requested feasible patients’ permission to be contacted by one of the researchers. The researcher briefed interested patients about the study, sent them all relevant information on the trial, and invited them for the baseline diagnostic interview (face-to-face or by telephone). For study inclusion, the primary diagnosis was to be confirmed in that interview by a research assistant using the MINI-Plus or SCID-I. Any comorbid DSM-IV diagnoses were also assessed with the MINI-Plus or SCID-I. Written informed consent was obtained from all participants before baseline assessment and randomisation.
Randomisation
After the baseline assessment, the included participants were randomly allocated to either bCBT or ftfCBT by an independent researcher using a computer-generated block randomisation table. Randomisation was stratified across the four research sites. Due to the nature of the intervention, patients and therapists could not be blinded to treatment allocation.
Interventions
Cognitive-behavioural therapy (CBT) was provided in both treatment conditions, including evidence-based components for treatment of anxiety disorders: psychoeducation, cognitive therapy, exposure and relapse prevention [5, 6, 25]. The treatment protocols were based on the standard Dutch treatment protocols [30]. For the blended variants, face-to-face and online sessions were integrated into a single blended treatment protocol for each disorder.
Therapists taking part in the study delivered therapy to patients in both treatment conditions. All therapists had formal training and experience in delivering CBT and had received training in the delivery of the blended format.
For the three primary diagnoses, three different treatment protocols were used. In the event of comorbid anxiety disorders, the protocol of choice was based on the patient’s most prominent disorder, as established during the therapy intake session. The treatment sessions contained psychoeducation (explanation of treatment rationale and general procedures in cognitive therapy), cognitive therapy (examination of relationships between thoughts, emotions and behaviour), exposure tasks (graded exposure to feared situations) and relapse prevention (identification and adoption of strategies to prevent anxiety symptoms from reoccurring). Cognitive therapy for PD and SAD focused on reinterpreting the causes and consequences of anxiety symptoms. The protocol for GAD consisted of metacognitive therapy, which identifies underlying metabeliefs about worrying and develops more adaptive meta-beliefs, since GAD is known to respond only modestly to conventional CBT [31].
The bCBT delivery consisted of 15 weekly alternating face-to-face sessions (8) and online sessions with asynchronous therapeutic feedback (7). Online sessions were provided on a web-based treatment platform (Minddistrict, www.minddistrict.com), accessible through password-protected accounts. Online sessions contained text-based information and videos in which a therapist explained the theory, followed by exercises and homework assignments with examples from fictional patients. Feedback involved text-based messages from the therapist about the content of the online exercises performed by the patient and about treatment progress. ftfCBT entailed 15 weekly face-to-face sessions with similar content to the sessions of the bCBT protocol.
Measures
Online questionnaires were administered at baseline, at week 7 (mid-treatment), at week 15 (post-treatment) and at one-year follow-up (see S4 Appendix for an overview of measures administered at each assessment interval). All questionnaires were self-administered, except for the diagnostic interview at baseline. The Dutch versions of the questionnaires were used. Our original study protocol specified that follow-up data would be collected after 67 weeks, one year after the post-treatment assessment, but for pragmatic reasons (funder requirements in terms of final deadline), the time frame was adjusted to 52 weeks. Furthermore, to reduce burden on participants, quality of life was measured only by the EuroQol (EQ-5D-5L) [32] and not by the Short Form Health Survey (SF-36) [33] as well as both measure quality of life and anxiety severity was measured only by the Beck Anxiety Inventory (BAI) and not by the disorder-specific questionnaires as the overall sample size would be too small for robust subgroup-analyses. That means the Short Form Health Survey (SF-36) and the disorder-specific questionnaires were not administered. These changes were made prior to trial commencement (see our published study protocol [29]).
Demographic characteristics such as age, gender, education and employment were collected at baseline. Diagnoses were assessed at baseline with the Structured Clinical Interview for DSM-IV Axis I Disorders (SCID-I) [26] or the Mini-International Neuropsychiatric Interview, Plus version (MINI-Plus) [27, 28].
Acceptability.
We distinguished four aspects of acceptability: treatment preference, treatment adherence, therapeutic working alliance and treatment satisfaction. Treatment preference was assessed by asking participants to indicate their preference for bCBT or ftfCBT at baseline, prior to randomisation. Treatment adherence concerns the extent to which participants were exposed to the content of the interventions, as measured in three ways: (i) the percentage of completed prescribed sessions; (ii) the percentage of participants that finished treatment, defined as completing at least 15 sessions as described in the protocol or dropping out due to remission; and (iii) the duration of treatment in weeks.
The Revised Short Version of the Working Alliance Inventory (WAI-SR) [34, 35] was administered halfway through treatment to both patients and therapists to rate the quality of the therapeutic alliance. The WAI-SR has excellent psychometric properties [35]. To evaluate treatment satisfaction at post-treatment, we administered the Client Satisfaction Questionnaire-8 (CSQ-8) [36, 37] and, additionally for the participants randomised to bCBT, the System Usability Scale (SUS) [38, 39]. Both the CSQ-8 and the SUS scales have demonstrated reliability and validity [37, 39].
Effectiveness.
Clinical outcome variables were assessed at baseline, at post-treatment (15 weeks) and at one-year follow-up 52 weeks after baseline. The primary clinical outcome was presence and severity of anxiety symptoms, as assessed with the Beck Anxiety Inventory (BAI) [40]. It contains 21 questions and total scores range between 0 and 63, with higher scores indicating more anxiety. The BAI is a reliable and well validated self-rated measure of anxiety symptoms [41].
Secondary clinical outcome variables included depressive symptoms, general psychopathology, mastery, social and work functioning, and quality of life, likewise assessed at baseline, post-treatment and follow-up. Presence and severity of depressive symptoms were assessed using the Beck Depression Inventory-II (BDI-II) [42, 43], which has highly acceptable psychometric properties [42]. Severity of general psychopathology was evaluated by the Brief Symptom Inventory (BSI) [44, 45], whose psychometric properties are good [45]. The five-item version of the Mastery Scale [46] was administered to assess perceived control of a person’s own life; it is a psychometrically valid instrument [46]. The Work and Social Adjustment Scale (WSAS), with adequate psychometric properties [47], is a measure of impaired functioning; it assesses the impact of a person’s mental health problems on their ability to function in terms of work, home management, social leisure, private leisure and personal or family relationships.
Quality of life.
To estimate utilities the EuroQol (EQ-5D-5L) was administered [32]. We applied the Dutch tariff [48] to calculate the utilities. The EuroQol consists of five questions that gauge mobility, self-care, daily activities, pain and mood. It is the preference-based generic instrument for measuring health-related quality of life (HR-QoL) that is recommended by the Dutch guidelines for economic evaluations in healthcare and it has good psychometric properties [49]. Quality-adjusted life-years (QALYs) were calculated using the area-under-the-curve method (AUC) [50]. The health state descriptions were linked to empirical valuations of the Dutch general public, allowing utilities to be computed.
Costs.
Costs were assessed at baseline, post-treatment and one-year follow-up using the Treatment Inventory Cost in Psychiatric Patients instrument (TiC-P) [51]. Costs can be determined from several perspectives. In this study we calculated costs for both the healthcare perspective (including direct medical costs) and the societal perspective (including direct medical costs, patient costs and productivity costs). Direct medical costs consist of costs for the use of healthcare services; patient costs consist of travel costs; productivity costs include costs arising from absenteeism and presenteeism.
In the TIC-P, a maximum recall period of 15 weeks was used and cumulative costs over the one-year study period were estimated using linear interpolation. In accordance with the TiC-P manual, a specific item on the service use accountable to the bCBT intervention was added to the default TiC-P. Direct medical costs, patient costs and productivity costs were valued using Dutch indexed standard reference prices of 2018 (see S5 Appendix) [52]. The friction cost method was applied to estimate productivity losses in paid work [53].
Sample size and power
The trial was powered to investigate the joint distribution of costs and treatment effects [29]. We aimed to include 156 participants, with 78 in each condition, based on a power of 0.80 calculated by using the formula of Glick [54].
Statistical analyses
Statistical analyses were conducted using the Statistical Package for the Social Sciences version 24.0 (IBM Corporation, Somers, NY, USA) and Excel (2013). The descriptive characteristics of the bCBT and ftfCBT groups and differences between study dropouts and study completers were compared using t-test for continuous variables and chi-square test for proportions.
Acceptability.
Acceptability outcomes (treatment preference, treatment adherence, therapeutic working alliance, treatment satisfaction) were compared using t-test for continuous variables and chi-square test for proportions.
Effectiveness.
Clinical outcomes were analysed on the basis of the intention-to-treat (ITT) principle. Linear mixed model (LMM) analyses with restricted maximum likelihood (REML) were conducted to evaluate differences in symptom reduction between the bCBT and ftfCBT groups at post-treatment and one-year follow-up. The linear mixed models were adjusted for baseline scores, because using analysis of covariance the estimate of the intervention effect is not affected by baseline differences and more statistical power to detect a treatment effect is achieved [55]. The LMM approach has the ability to handle missing data, as it uses all available data to estimate parameters for missing values and can account for the correlation between repeated measures [56]. A separate model was estimated for each of the outcome measures. A Bonferroni-Holm correction was applied to adjust for repeated comparisons, yielding a significance level of p = 0.01 (.05 / 5) [57].
Effect sizes (Cohen’s d) were calculated both within and between groups from estimated means and observed pooled standard deviations. The within-group effect size was computed as d = Meandiff / SDdiff, where Meandiff is the mean difference between the values at pre-test and at post-test or follow-up and SDdiff = √(SD2pre + SD2post − 2rSDpreSDpost), with r being the correlation between the pre-test and the post-test or follow-up values. The between-group effect size was computed as d = Meandiff / SDpooled, where Meandiff is the mean difference between bCBT scores and ftfCBT scores.
Cost-effectiveness.
Yearly costs and QALYs were modelled using generalized linear models (GLM), that can manage skewness of data [58]. Missing utility values and costs at each time point were imputed using multiple imputation by predictive mean matching [59, 60]. For the estimation of costs, a log link and gamma family were used adjusting for age and baseline costs. For the estimation of QALYs a log link and gaussian family were used adjusting for age and baseline utility. The cost-effectiveness analysis was conducted by calculating the incremental costs per incremental QALY over the one-year follow-up period, resulting in the incremental cost-effectiveness ratio (ICER). The formula (C1-C2)/(E1-E2) was used, where (C1-C2) is the difference in costs between bCBT and ftf CBT, and (E1-E2) is the difference of the average effectiveness of bCBT and ftf CBT [61].
The ICER was estimated from a healthcare and societal perspective. The latter included the direct medical costs, patient costs and productivity costs, while the healthcare perspective is limited to the direct medical costs. Costs and effects are not discounted as the time-horizon of the current study did not exceeded 12 months follow-up.
Standard errors around the GLM coefficients were used to explore the uncertainty of the ICER. For this purpose, 10,000 populations were simulated using non-parametric bootstrapping. Cholesky decomposition [62] was used to retain the correlations between the parameters. The simulated results were plotted in a CE-plane [63], on which uncertainty around incremental costs and incremental effects was displayed graphically by the scatter of ICERs. From a cost-effectiveness perspective, the southeast quadrant indicates superior treatment effects and lower costs for bCBT in comparison with ftfCBT. If the ICER falls into this quadrant this indicates dominance of bCBT over ftfCBT and should lead to a positive reimbursement decision. The northwest indicates reduced treatment effects and higher costs for bCBT, thus leading to a negative reimbursement decision. ICERs in the two remaining quadrants indicate either that bCBT is less expensive but also less effective (southwest quadrant) or more effective but also more expensive (northeast quadrant). The cost-effectiveness of the latter depends on the threshold of the cost-effectiveness ratio. For the Netherlands the threshold is €20,000 to €80,000 depending on the severity of the disease. The uncertainty in the cost-effectiveness analysis was assessed using bootstrapping in Excel, with 10,000 iterations. This was expressed in a cost-effectiveness acceptability curve. The acceptability curve illustrates the probability that the cost-effectiveness ratio will be accepted for different cost limits [64].
Results
Study sample and attrition
A total of 281 participants were assessed by mental health professionals during the intake procedure; 129 eligible candidates were invited for a diagnostic interview and 114 were randomised to either bCBT (n = 52) or ftfCBT (n = 62; for details, see Fig 1). Demographic data of the included participants are presented in Table 1. The mean age of participants was 36.3 years (SD 10.6, range 19 to 69) and 60.5% were female (n = 69). Most patients had panic disorder as primary diagnosis (54.4%).
The post-treatment assessments at 15 weeks were completed by 77 (67.5%) participants (bCBT n = 34, ftfCBT n = 43) and the one-year follow-up assessments by 74 (64.9%) participants (bCBT n = 34, ftfCBT n = 40). There was no significant difference in total study dropout between the two treatment groups, χ2 (1) = 0.78, p = 0.781. We tested for significant differences in demographic variables, primary diagnosis or presence of comorbidity between those who completed all post-baseline assessments and those who did not. Participants with missing data at one or more of those assessments (n = 51) were less likely to have a comorbid diagnosis, χ2 (2) = 4.84, p = 0.028.
Acceptability
Queried prior to randomisation, patients expressed a slight preference for bCBT (54.4%) over ftfCBT (45.6%). The percentages with a bCBT preference (56.5% in the ftfCBT treatment group, 51.9% in the bCBT group) did not differ significantly between the groups, χ2 (1) = 0.23, p = 0.629.
Adherence in terms of the percentage of completed prescribed sessions was slightly but not significantly higher in the bCBT group, at 67.4% compared with 61.6% for the ftfCBT group (t = −0.515, p = 0.608). Thirty-one patients (59.6%) in the bCBT group and 32 patients (51.6%) in the ftfCBT group completed treatment (t = −0.795, p = 0.428). Treatment duration was shorter in the bCBT group, with an average of 14.4 weeks (range 0 to 56.4) compared with 16.1 weeks (range 0 to 67.7) for ftfCBT treatment (t = 0.796, p = 0.428).
The alliance assessment (WAI-SR) halfway through treatment was completed by 81 participants (71.1%) and 87 times (76.3%) by therapists. Participants in both groups reported high levels of working alliance on the WAI-SR, with a mean rating of 4.27 out of 5 (SD 0.69) in the bCBT group and 4.25 (SD 0.51) in the ftfCBT group. Therapists’ ratings were in a similar range, with scores of 4.32 (SD 0.37) in bCBT and 4.24 (SD 0.56) in ftfCBT. We found no significant difference between bCBT and ftfCBT in terms of WAI patient ratings (t = −0.111, p = 0.912) nor WAI therapist ratings (t = −0.304, p = 0.762), indicating no difference in working alliance between groups.
On average, participants in both groups reported high levels of treatment satisfaction. The mean scores on the CSQ-8 were 25.61 out of 32 (SD 4.21, range 8 to 32) for the bCBT group and 25.90 (SD 3.24) for the ftfCBT group, both lying between ‘somewhat satisfied’ (score 24) and ‘very satisfied’ (score 32). We found no significant difference between bCBT and ftfCBT in treatment satisfaction (t = 0.320, p = 0.750).
The online treatment platform was evaluated by patients randomised to the blended condition at an ‘above average’ score of 69.11 (SD 19.32) on the SUS.
Effectiveness
Mean observed scores on primary and secondary clinical outcome measures at baseline, post-treatment and one-year follow-up are displayed in Table 2, accompanied by within-group and between-group effects. No statistically significant differences emerged between the bCBT group and the ftfCBT group in terms of decreased anxiety severity, either at post-treatment (t = −0.715, p = 0.477) or at follow-up (t = 1.702, p = 0.093). Within-group effect sizes from baseline to post-treatment were d = 0.73 for bCBT and d = 0.55 for ftfCBT and from baseline to follow-up d = 0.50 for bCBT and d = 1.00 for ftfCBT.
Separate linear mixed model analyses revealed no significant effects of treatment condition at post-treatment or follow-up on secondary outcomes: depressive symptoms, general psychopathology, mastery, work and social functioning and quality of life. Within-group effect sizes at post-treatment and follow-up ranged from d = 0.13 to d = 0.98.
Cost-effectiveness
The results from the cost-effectiveness analyses are presented in Table 3. Multiple imputation (cost data: 35.1% imputed, QALY data: 32,7% imputed) followed by modelled simulations yielded average direct medical costs of €3758 for bCBT and €3841 for ftfCBT over the one-year study period. Direct medical costs were statistically significantly lower in the bCBT group (mean -83,78, 95% CI -96,96 to -70,61, p<0.001). Societal costs were €10945 for bCBT and €10937 for ftfCBT. Differences were not statistically significant (mean 26,46, 95% CI -26,46 to 42,71, p>0.1). Total costs based on available data over the treatment period and the one-year follow-up period are included in S6 Appendix. The average QALYs over the one-year study period were 0.66 for bCBT and 0.62 for ftfCBT. QALYs were statistically significantly higher in the bCBT group (mean 0.037, 95% CI 0.036 to 0.038, p<0.001). This resulted in a dominant ICER from the healthcare perspective (€-2257 per QALY) and an ICER of €219 per QALY from the societal perspective.
Uncertainty in cost and effect estimates is shown in cost-effectiveness planes (CE planes, Figs 2 and 3). The CE planes show that the greatest numbers of ICERs were situated in the southeast quadrant of the CE plane, both in the healthcare perspective (46.6%) and in the societal perspective (42.3%), indicating lower costs for bCBT as well as a superior effect in terms of quality of life. Another 37.6% were in the northeast quadrant from the healthcare perspective and 41.6% from the societal perspective respectively. From the health care perspective 6.8% and 9.0% of the estimates were in respectively in the north- and southwest quadrant. From the societal perspective these figures were 7.7% and 8.2%.
Determining the acceptability of the treatments, we calculated the proportion of ICERS that were below the threshold of 20,000 and 80,000 per QALY. The threshold is the willingness of society to pay and was varied as this is the common range for the Netherlands. The thresholds and the proportion of ICERS were subsequently plotted in the cost acceptability curve, see Fig 4. The figure shows that from a health care perspective, at a threshold of 20,000 Euro/QALY, the probability that the ratio is acceptable is more than 80%. Taking a societal perspective, the percentage that the intervention is acceptable was 67%. At a threshold of 80,000 Euro/QALY the intervention was acceptable more than 80% from both perspectives.
Discussion
Blended treatment for anxiety disorders, which integrates face-to-face therapy and Internet-based therapy, has not yet been rigorously studied. To our knowledge, RCTs investigating effectiveness and costs are lacking. This study is the first to assess the acceptability, effectiveness and cost-effectiveness of bCBT vis-à-vis ftfCBT in outpatients receiving specialised mental health care who have been diagnosed with panic disorder, social anxiety disorder or generalised anxiety disorder.
Our findings on acceptability indicate that bCBT is an acceptable treatment option for patients in specialised mental health care in terms of treatment preference, adherence, therapeutic alliance and treatment satisfaction. Over half (54.9%) of the participants would have preferred to start with bCBT above ftfCBT. Although that is the treatment preference of patients who consented to take part in the current study, and hence not a fully representative finding for all patients in specialised mental health care, it does reveal that a considerable desire for the blended treatment format exists in that population. Therapeutic alliance and treatment satisfaction were high for both bCBT and ftfCBT patients, and treatment adherence rates were comparable for both groups.
With regard to effectiveness in reducing anxiety symptoms, we found no significant differences between bCBT and ftfCBT at post-treatment (t = −0.715, p = 0.477) nor at one-year follow-up (t = 1.702, p = 0.093). Both groups exhibited moderate to large within-group effect sizes (range of d: 0.50 to 1.00). Moreover, no significant differences between the groups were found in terms of effects on depressive symptoms, general psychopathology, sense of control (mastery), work and social functioning or quality of life, with within-group effect sizes ranging from small to large (range of d: 0.13 to 0.98).
In the current study online sessions partially replaced face-to-face sessions in the blended treatment. Other studies have investigated iCBT applied as an adjunctive to ftfCBT.
Our clinical findings appear to be in line with results from those studies. For example, Nordgreen and colleagues conducted an RCT (N = 173) whereby iCBT and ftfCBT for panic disorder and social anxiety disorder were combined in a stepped-care format, with a face-to-face psychoeducation session as first step, online treatment (9 or 10 sessions) as second step and face-to-face treatment (12 sessions) as final step [65]. The stepped-care variant was compared with ftfCBT (12 sessions). No significant differences in the reductions of anxiety symptoms and depressive symptoms were found between the groups at post-treatment and one-year follow-up, and within-group effect sizes were moderate to large. Comparability with our study is limited, however, as the stepped-care format consisted of iCBT as an add-on prior to ftfCBT. Moreover, treatment attrition in the stepped-care group was high (41.2%), with the majority dropping out before starting the face-to-face treatment, meaning that they only received online iCBT. In a pilot study (N = 36) by Bruinsma and colleagues [23], a combination of 9 ftfCBT sessions supplemented with 3 iCBT sessions was compared with 12-session ftfCBT. They found no significant between-group differences at post-treatment in terms of improvement rates on panic-related symptoms and general functioning, with moderate to large within-group effect sizes.
Our cost analysis showed that societal costs were relatively larger than direct medical costs in both groups. This may be due to relatively low treatment costs and a large proportion of patients of working age. This finding is in line with literature that showed that productivity costs are commonly responsible for the majority of the total costs [3, 66]. These results highlight the substantial societal burden of anxiety disorders and the importance of making CBT for anxiety disorders more accessible. Further findings have shown that the costs for providing treatment would be compensated within two to five years by increased productivity resulting from the intervention [67].
The acceptability curves in the current study revealed that bCBT was expected to be a cost-effective intervention. While bCBT point estimates suggest slightly lower healthcare and slightly higher societal costs than ftfCBT over the one-year study period, the probabilistic results suggest a high probability of cost-effectiveness taking a threshold of €20.000 from both perspectives. In contrast, in a naturalistic study by Kenter and colleagues treatment time and costs increased for bCBT relative to ftfCBT [68]. In this study, no treatment protocol or clear guidelines on how to apply blended treatment were available and therapists turned out to have provided online sessions on top of face-to-face sessions resulting in longer treatment durations. This marked contrast to our trial possibly explains the difference in outcomes.
A strength of this study is that it is the first randomised controlled trial to explore acceptability, effectiveness and cost-effectiveness by comparing equal-intensity bCBT and ftfCBT for anxiety disorders in routine outpatient specialised mental health care. In addition, participants in our study appear to be a clinically representative sample, in view of the large proportion of patients with comorbid disorders, lower education levels and severe anxiety symptoms at baseline, in comparison with self-referred samples recruited from the community [12]. Clinical representativeness is also reflected by the high productivity costs and the low scores on measurements of work, social functioning and quality of life; disability and decreased productivity are common among patients with severe anxiety disorders [69, 70]. Although patients in both groups exhibited improvement on these scores at post-treatment and follow-up, the scores remained relatively low in comparison with those in the general healthy population [47, 71]. That further demonstrates the severity and complexity of problems in the current study sample.
Some limitations are associated with the present study. First, because the sample size was smaller than expected, only initial indications of the cost-effectiveness of bCBT in comparison with ftfCBT could be explored. Due to financial and time limits, only 114 participants rather than the intended 156 were included. However, it might be noted that such a sample size is considerable for routine specialised mental health care populations. Sample size and power challenges are common issues in trials investigating both clinical and cost-effectiveness [72]. In line with recommendations for dealing with such issues [73], the uncertainty was presented in cost-effectiveness planes. Another limitation lies in the substantial study dropout rate (35.1% at one-year follow-up), which was not considered in the power calculation. This dropout seems to reflect the reality of trials conducted in routine mental health care, as comparable rates were found in earlier clinical trials comparing iCBT with ftfCBT [65, 74–77]; it could not be prevented by our e-mail and telephone reminders. To handle missing data, a linear mixed model was used to analyse clinical effectiveness, and imputations were used for the cost-effectiveness analyses.
In addition, participants and therapists could not be blinded to treatment allocation. That was inevitable given the nature of this trial, but it may have affected results. For example, participants who know they are in the ‘experimental condition’ are more likely to provide biased effectiveness assessments than blinded participants; blinded therapists are less likely than unblinded therapists to provide additional treatment interventions [78].
Finally, we used the EQ-5D for measuring quality of life. In recent years, the usefulness of the EQ-5D to measure mental health related quality of life has been questioned [79, 80]. Other questionnaires are available that include more dimensions of quality of life relevant to populations of people with mental health problems. For example, the more recently developed Assessment of Quality of Life–Eight Dimension Scale (AQoL-8D) [81] might serve as an alternative for the EQ-5D. However, validity of this instrument has not been tested in the Dutch population, which is one of the reasons that the EQ-5D is the recommended questionnaire for economic evaluations in the Dutch context [82]. Furthermore, the EQ-5D is reasonably responsive in patients with anxiety disorders [83] and thus seems suitable in the current study. Nevertheless, other available instruments to evaluate mental health related quality of life should be considered in future research, especially when research is focusing on mental disorders such as schizophrenia and bipolar disorder [84], and validation of the AQoL-8D for the Dutch population would be desirable.
In sum, our results suggest that bCBT is an acceptable approach for patients with anxiety disorders in specialised mental health care settings. We found no indications that its clinical effectiveness differs from that of ftfCBT. Moreover, bCBT is expected to be a cost-effective alternative to ftfCBT.
Supporting information
S6 Appendix. Total costs based on observed means.
https://doi.org/10.1371/journal.pone.0259493.s006
(DOCX)
Acknowledgments
We are very grateful for the contribution of all research assistants and therapists to this study, and for the cooperation of the trial participants and the participating outpatient clinics: Altrecht, GGZ inGeest, GGZ Oost-Brabant and GGZ Noord-Holland-Noord. We also thank Robin Kok for his advice on the study protocol and Adriaan Hoogendoorn for his advice on the data analysis. Finally, we would like to thank Tim Kanters for his support on the cost-effectiveness analyses.
References
- 1. Kessler RC, Ruscio AM, Shear K, Wittchen HU. Epidemiology of anxiety disorders. Curr Top Behav Neurosci. 2010;2: 21–35. pmid:21309104
- 2. Mendlowicz M V. Quality of Life in Individuals With Anxiety Disorders. Am J Psychiatry. 2000;157: 669–682. pmid:10784456
- 3. Smit F, Cuijpers P, Oostenbrink J, Batelaan N, de Graaf R, Beekman A. Costs of nine common mental disorders: Implications for curative and preventive psychiatry. J Ment Health Policy Econ. 2006;9: 193–200. pmid:17200596
- 4. Wittchen HU, Jacobi F, Rehm J, Gustavsson A, Svensson M, Jönsson B, et al. The size and burden of mental disorders and other disorders of the brain in Europe 2010. Eur Neuropsychopharmacol. 2011;21: 655–679. pmid:21896369
- 5. NICE. Generalised anxiety disorder and panic disorder in adults. Natl Inst Heal Clin Excell. 2011. pmid:22536620
- 6. National Institute for Health and Care Excellence. Social anxiety disorder: recognition, assessment and treatment. Clinical guideline 159. Available: guidance.nice.org.uk/cg159. 2013. [doi]
- 7. Bijl R V, de Graaf R, Hiripi E, Kessler RC, Kohn R, Offord DR, et al. The prevalence of untreated mental disorders in five countries. Health Aff. 2003;22: 122–133. pmid:12757277
- 8. Andrade LH, Alonso J, Mneimneh Z, Wells JE, Al-Hamzawi A, Borges G, et al. Barriers to mental health treatment: Results from the WHO World Mental Health surveys. Psychol Med. 2014;44: 1303–1317. pmid:23931656
- 9. Riper H, Andersson G, Christensen H, Cuijpers P, Lange A, Eysenbach G. Theme issue on E-mental health: A growing field in internet research. Journal of Medical Internet Research. 2010. pmid:21169177
- 10. Olthuis J V, Watt MC, Bailey K, Hayden JA, Stewart SH. Therapist-supported Internet cognitive behavioural therapy for anxiety disorders in adults. Cochrane database Syst Rev. 2015;3: CD011565. pmid:25742186
- 11. Adelman CB, Panza KE, Bartley CA, Bontempo A, Bloch MH. A meta-analysis of computerized cognitive-behavioral therapy for the treatment of DSM-5 anxiety disorders. J Clin Psychiatry. 2014;75: e695–704. pmid:25093485
- 12. Romijn G, Batelaan N, Kok R, Koning J, van Balkom A, Titov N, et al. Internet-Delivered Cognitive Behavioral Therapy for Anxiety Disorders in Open Community Versus Clinical Service Recruitment: Meta-Analysis. J Med Internet Res. 2019;21: e11706. pmid:30994462
- 13. Emmelkamp PMG. Technological innovations in clinical assessment and psychotherapy. Psychother Psychosom. 2005;74: 336–343. pmid:16244509
- 14. Donker T, Blankers M, Hedman E, Ljótsson B, Petrie K, Christensen H. Economic evaluations of Internet interventions for mental health: a systematic review. Psychol Med. 2015; 1–20. pmid:26235445
- 15. Topooco N, Riper H, Araya R, Berking M, Brunn M, Chevreul K, et al. Attitudes towards digital treatment for depression: A European stakeholder survey. Internet Interv. 2017;8: 1–9. pmid:30135823
- 16. Christensen H, Griffiths KM, Farrer L. Adherence in Internet intervention for anxiety and depression: Systematic review. J Med Internet Res. 2009;11: 1–16. pmid:19403466
- 17. Kok RN, Beekman ATF, Cuijpers P, van Straten A. Adherence to a web-based pre-treatment for phobias in outpatient clinics. Internet Interv. 2017;9: 38–45. pmid:30135835
- 18. Riper H, Ballegooijen W Van, Kooistra L. Preventie & eMental-health. Onderzoek dat leidt, technologie die verleidt, preventie die bereikt en beklijft. Kennissynthese 2013 in opdracht van ZonMw. Amsterdam; 2013.
- 19. Titzler I, Saruhanjan K, Berking M, Riper H, Ebert DD. Barriers and facilitators for the implementation of blended psychotherapy for depression: A qualitative pilot study of therapists’ perspective. Internet Interv. 2018;12: 150–164. pmid:30135779
- 20. Schuster R, Pokorny R, Berger T, Topooco N, Laireiter AR. The Advantages and Disadvantages of Online and Blended Therapy: Survey Study Amongst Licensed Psychotherapists in Austria. J Med Internet Res. 2018;20: e11007. pmid:30563817
- 21. Erbe D, Psych D, Eichert HC, Riper H, Ebert DD. Blending face-to-face and internet-based interventions for the treatment of mental disorders in adults: Systematic review. J Med Internet Res. 2017;19. pmid:28916506
- 22. van der Vaart R, Witting M, Riper H, Kooistra L, Bohlmeijer ET, van Gemert-Pijnen LJEWC. Blending online therapy into regular face-to-face therapy for depression: Content, ratio and preconditions according to patients and therapists using a Delphi study. BMC Psychiatry. 2014;14: 1–10. pmid:25496393
- 23. Bruinsma A, Kampman M, Exterkate CC, Hendriks GJ. An exploratory study of ‘blended’ cognitive behavioural therapy (CBT) for patients with a panic disorder: results and patients’ experiences. Tijdschr Psychiatr. 2016;58: 361–370. pmid:27213635
- 24. Kaltenthaler E, Brazier J, Nigris E De, Tumur I, Ferriter M, Beverley C, et al. Computerised cognitive behaviour therapy for depression and anxiety update. 2006;10. pmid:16959169
- 25.
Dutch mental health guidelines (GGZ Richtlijnen) website. [cited 22 Dec 2018]. Available: http://www.ggzrichtlijnen.nl/angststoornissen
- 26.
First MB, Spitzer RL, Gibbon M, Williams JB. Structured Clinical Interview for DSM-IV-TR Axis I Disorders. New York State Psychiatric Institute. 2002.
- 27. Sheehan D V., Lecrubier Y, Sheehan KH, Amorim P, Janavs J, Weiller E, et al. The Mini-International Neuropsychiatric Interview (M.I.N.I.): The development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. Journal of Clinical Psychiatry. 1998. pp. 22–33. pmid:9881538
- 28. Van Vliet IM, De Beurs E. Het Mini Internationaal Neuropsychiatrisch Interview (MINI): Een kort gestructureerd diagnostisch psychiatrisch interview voor DSM-IV-en ICD-10-stoornissen. Tijdschr Psychiatr. 2007;49: 393–397. pmid:17614093
- 29. Romijn G, Riper H, Kok R, Donker T, Goorden M, van Roijen LH, et al. Cost-effectiveness of blended vs. face-to-face cognitive behavioural therapy for severe anxiety disorders: study protocol of a randomized controlled trial. BMC Psychiatry. 2015;15: 311. pmid:26651478
- 30.
Keijsers GPJ, Minnen A van, Hoogduin CAL. Protocollaire behandelingen voor volwassenen met psychische klachten. Deel 1. [Manualized treatments for adults with psychological disorders, part 1]. 2010.
- 31. Durham RC, Allan T. Psychological treatment of generalised anxiety disorder. A review of the clinical significance of results in outcome studies since 1980. Br J Psychiatry. 1993. pmid:8353693
- 32. Group EuroQol. EuroQol—a new facility for the measurement of health-related quality of life. Health Policy. 1990;16: 199–208. pmid:10109801
- 33. Gandek B, Ware JE, Aaronson NK, Apolone G, Bjorner JB, Brazier JE, et al. Cross-validation of item selection and scoring for the SF-12 Health Survey in nine countries: Results from the IQOLA Project. J Clin Epidemiol. 1998. pmid:9817135
- 34. Hatcher RL, Gillaspy JA. Development and validation of a revised short version of the Working Alliance Inventory. Psychother Res. 2006;16: 12–25.
- 35. Stinckens N, Ulburghs A, Claes L. De werkalliantie als sleutelelement in het therapiegebeuren Meting met behulp van de WAV-12, de Nederlandstalige verkorte versie van de Working Alliance Inventory. Tijdschr Klin Psychol. 2009;39: 44–60.
- 36. Larsen DL, Attkisson CC, Hargreaves WA, Nguyen TD. Assessment of client/patient satisfaction: Development of a general scale. Eval Program Plann. 1979;2: 197–207. pmid:10245370
- 37. De Brey H. A cross-national validation of the Client Satisfaction Questionnaire: The Dutch experience. Eval Program Plann. 1983;6: 395–400. pmid:10267266
- 38. Brooke J. SUS—A quick and dirty usability scale. Usability Eval Ind. 1996;189: 4–7. pmid:19172650
- 39. Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Int J Hum Comput Interact. 2008;24: 574–594.
- 40. Beck AT, Epstein N, Brown G, Steer RA. An inventory for measuring clinical anxiety: Psychometric properties. J Consult Clin Psychol. 1988;56: 893–897. pmid:3204199
- 41. Fydrich T, Dowdall D, Chambless DL. Reliability and validity of the beck anxiety inventory. J Anxiety Disord. 1992;6: 55–61.
- 42.
Van der Does W. BDI-II-NL. Handleiding. De Nederlandse versie van de Beck Depression Inventory-2nd edition. Lisse; 2002.
- 43. Beck AT, Steer RA, Carbin MG. Psychometric properties of the Beck Depression Inventory: Twenty-five years of evaluation. Clin Psychol Rev. 1988.
- 44. Derogatis LR. The Brief Symptom Inventory: An Introductory Report. Psychol Med. 1983;13: 595–605. pmid:6622612
- 45. Beurs EDE, Zitman FG. De Brief Symptom Inventory. Maandbl Geestelijke Volksgezond. 2006;61: 120–141.
- 46. Pearlin LI, Schooler C. The Structure of Coping. J Health Soc Behav. 1978;19: 2. pmid:649936
- 47. Mundt JC, Marks IM, Shear MK, Greist JH. The Work and Social Adjustment Scale: A simple measure of impairment in functioning. Br J Psychiatry. 2002;180: 461–464. pmid:11983645
- 48. Versteegh M M. Vermeulen K, Evers S M. A. A., de Wit GA, Prenger R, E A. Stolk. Dutch Tariff for the Five-Level Version of EQ-5D. Value Heal. 2016;19: 343–352. pmid:27325326
- 49.
Zorginstituut Nederland. Richtlijnen voor economische evaluatie. In: https://www.zorginstituutnederland.nl/over-ons/werkwijzen-en-procedures/adviseren-over-en-verduidelijken-van-het-basispakket-aan-zorg/beoordeling-van-geneesmiddelen/richtlijnen-voor-economische-evaluatie.
- 50. Matthews JN, Altman DG, Campbell MJ, Royston P. Analysis of serial measurements in medical research. BMJ. 1990. pmid:2106931
- 51. Bouwmans C, De Jong K, Timman R, Zijlstra-Vlasveld MC, van der Feltz-Cornelis C, Tan S, et al. Feasibility, reliability and validity of a questionnaire on healthcare consumption and productivity loss in patients with a psychiatric disorder (TiC-P). Occup Environ Med. 2013;13: 217. pmid:23768141
- 52. Hakkaart-van Roijen L, van der Linden N, Bouwmans C, Kanters T, Swan Tan S. Kostenhandleiding: Methodologie van kostenonderzoek en referentieprijzen voor economische evaluaties in de gezondheidszorg. Zorginstituut Ned. 2015.
- 53. Koopmanschap MA, Rutten FFH, van Ineveld BM, van Roijen L. The friction cost method for measuring indirect costs of disease. J Health Econ. 1995. pmid:10154656
- 54. Glick HA. Sample size and power for cost-effectiveness analysis (Part 1). Pharmacoeconomics. 2011. pmid:21309615
- 55. Vickers AJ, Altman DG, Vickers AJ, Altaian DG. Statistics Notes: Analysing Controlled Trials With Baseline And Follow Up Measurements Stable URL: https://www.jstor.org/stable/25227028 Linked references are available on JSTOR for this article: Statistics Notes Analysing controlled trials with baseli. 2001;323: 1123–1124.
- 56. Gueorguieva. Move over ANOVA? Arch Gen Psychiatry. 2011;61: 310–317.
- 57. Holm S. A simple sequentially rejective multiple test procedure. Scand J Stat. 1979.
- 58. Glick HA, Doshi JA, Sonnad SS, Polsky D. Economic Evaluation in Clinical Trials. Economic Evaluation in Clinical Trials. 2014.
- 59. Faria R, Gomes M, Epstein D, White IR. A Guide to Handling Missing Data in Cost-Effectiveness Analysis Conducted Within Randomised Controlled Trials. Pharmacoeconomics. 2014. pmid:25069632
- 60. MacNeil Vroomen J, Eekhout I, Dijkgraaf MG, van Hout H, de Rooij SE, Heymans MW, et al. Multiple imputation strategies for zero-inflated cost data in economic evaluations: which method works best? Eur J Heal Econ. 2016. pmid:26497027
- 61. Drummond MF, Stoddard GL, Torrance GW. Methods for the economic evaluation of health care programmes. Methods for the Economic Evaluation of Health Care Programmes. 2005.
- 62.
Briggs A, Sculpher M, Claxton K. Decision modelling for health economic evaluation. Oxford University Press, editor. New York; 2006. https://doi.org/10.1136/jech.2007.059576
- 63. Black WC. The CE Plane: A Graphic Representation of Cost-Effectiveness. Med Decis Mak. 1990. pmid:2115096
- 64. Fenwick E, Marshall DA, Levy AR, Nichol G. Using and interpreting cost-effectiveness acceptability curves: An example using data from a trial of management strategies for atrial fibrillation. BMC Health Serv Res. 2006;6: 1–8. pmid:16403235
- 65. Nordgreen T, Haug T, Ost L-G, Andersson G, Carlbring P, Kvale G, et al. Stepped Care Versus Direct Face-to-Face Cognitive Behavior Therapy for Social Anxiety Disorder and Panic Disorder: A Randomized Effectiveness Trial. Behav Ther. 2016;47: 166–183. pmid:26956650
- 66. Krol M, Papenburg J, Koopmanschap M, Brouwer W. Do Productivity Costs Matter? Costs of Interventions Targeted at Depressive Disorders. Pharmoeconomics. 2011;29: 601–619. pmid:21545189
- 67. Layard R, Clark DM, Knapp M, Mayraz G. Cost-benefit analysis of psychological therapy. Natl Inst Econ Rev. 2007;202: 90–98.
- 68. Kenter RMF, van de Ven PM, Cuijpers P, Koole G, Niamat S, Gerrits RS, et al. Costs and effects of Internet cognitive behavioral treatment blended with face-to-face treatment: Results from a naturalistic study. Internet Interv. 2015;2: 77–83.
- 69. Hendriks SM, Spijker J, Licht CMM, Beekman ATF, Hardeveld F, De Graaf R, et al. Disability in anxiety disorders. J Affect Disord. 2014;166: 227–233. pmid:25012435
- 70. Dewa CS, Lin E. Chronic physical illness, psychiatric disorder and disability in the workplace. Soc Sci Med. 2000;51: 41–50. pmid:10817467
- 71. EuroQol—a new facility for the measurement of health-related quality of life. Health Policy (New York). 1990;16: 199–208. pmid:10109801
- 72. Briggs A. Economic evaluation and clinical trials: size matters. Bmj. 2002;321: 1362–1363. pmid:11099268
- 73. Briggs AH. Handling Uncertainty in Cost-Effectiveness Models. Pharmacoeconomics. 2000;17: 479–500. pmid:10977389
- 74. Bergstrom J, Andersson G, Ljotsson B, Ruck C, Andreewitch S, Karlsson A, et al. Internet-versus group-administered cognitive behaviour therapy for panic disorder in a psychiatric setting: a randomised trial. BMC Psychiatry. 2010;10: 54. pmid:20598127
- 75. Botella C, Gallego MJ, Garcia-Palacios A, Guillen V, Banos RM, Quero S, et al. An Internet-based self-help treatment for fear of public speaking: a controlled trial. Cyberpsychol Behav Soc Netw. 2010;13: 407–421. pmid:20712499
- 76. Kiropoulos LA, Klein B, Austin DW, Gilson K, Pier C, Mitchell J, et al. Is internet-based CBT for panic disorder and agoraphobia as effective as face-to-face CBT? J Anxiety Disord. 2008;22: 1273–1284. pmid:18289829
- 77. Marks IM, Kenwright M, McDonough M, Whittaker M, Mataix-Cols D. Saving clinicians’ time by delegating routine aspects of therapy to a computer: a randomized controlled trial in phobia/panic disorder. Psychol Med. 2004;34: 9–17. pmid:14971623
- 78. Schulz KF, Grimes DA. Epidemiology series Blinding in randomised trials: hiding who got what. 2002;359: 696–700.
- 79. Brazier J, Roberts J, Tsuchiya A, Busschbach J. A comparison of the EQ-5D and SF-6D across seven patient groups. Health Econ. 2004;13: 873–884. pmid:15362179
- 80. Mihalopoulos C, Chen G, Iezzi A, Khan MA, Richardson J. Assessing outcomes for cost-utility analysis in depression: Comparison of five multi-attribute utility instruments with two depression-specific outcome measures. Br J Psychiatry. 2014;205: 390–397. pmid:25257063
- 81. Richardson J, Iezzi A, Khan MA, Maxwell A. Validity and reliability of the assessment of quality of life (AQoL)-8D multi-attribute utility instrument. Patient. 2014. pmid:24271592
- 82. Hakkaart-van Roijen L, van der Linden N, Bouwmans C, Kanters T, Swan Tan S. Kostenhandleiding: Methodologie van kostenonderzoek en referentieprijzen voor economische evaluaties in de gezondheidszorg. Dutch Natl Heal Care Inst. 2016; 1–73.
- 83. König HH, Born A, Günther O, Matschinger H, Heinrich S, Riedel-Heller SG, et al. Validity and responsiveness of the EQ-5D in assessing and valuing health status in patients with anxiety disorders. Health Qual Life Outcomes. 2010;8: 1–9. pmid:20053296
- 84. Brazier J. Is the EQ-5D fit for purpose in mental health? Br J Psychiatry. 2010;197: 348–349. pmid:21037210