Social anxiety disorder is one of the most persistent and common anxiety disorders. Individually delivered psychological therapies are the most effective treatment options for adults with social anxiety disorder, but they are associated with high intervention costs. Therefore, the objective of this study was to assess the relative cost effectiveness of a variety of psychological and pharmacological interventions for adults with social anxiety disorder.
A decision-analytic model was constructed to compare costs and quality adjusted life years (QALYs) of 28 interventions for social anxiety disorder from the perspective of the British National Health Service and personal social services. Efficacy data were derived from a systematic review and network meta-analysis. Other model input parameters were based on published literature and national sources, supplemented by expert opinion.
Individual cognitive therapy was the most cost-effective intervention for adults with social anxiety disorder, followed by generic individual cognitive behavioural therapy (CBT), phenelzine and book-based self-help without support. Other drugs, group-based psychological interventions and other individually delivered psychological interventions were less cost-effective. Results were influenced by limited evidence suggesting superiority of psychological interventions over drugs in retaining long-term effects. The analysis did not take into account side effects of drugs.
Various forms of individually delivered CBT appear to be the most cost-effective options for the treatment of adults with social anxiety disorder. Consideration of side effects of drugs would only strengthen this conclusion, as it would improve even further the cost effectiveness of individually delivered CBT relative to phenelzine, which was the next most cost-effective option, due to the serious side effects associated with phenelzine. Further research needs to determine more accurately the long-term comparative benefits and harms of psychological and pharmacological interventions for social anxiety disorder and establish their relative cost effectiveness with greater certainty.
Citation: Mavranezouli I, Mayo-Wilson E, Dias S, Kew K, Clark DM, Ades AE, et al. (2015) The Cost Effectiveness of Psychological and Pharmacological Interventions for Social Anxiety Disorder: A Model-Based Economic Analysis. PLoS ONE 10(10): e0140704. https://doi.org/10.1371/journal.pone.0140704
Editor: Cathy Mihalopoulos, Deakin University, AUSTRALIA
Received: June 23, 2015; Accepted: September 28, 2015; Published: October 27, 2015
Copyright: © 2015 Mavranezouli et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Data Availability: The clinical dataset that was utilised in the network meta-analysis that informed the economic analysis is provided in Mayo-Wilson et al., The Lancet Psychiatry, 1(5): 368–376 (DOI: http://dx.doi.org/10.1016/S2215-0366(14)70329-3). All other relevant data are within the paper and its Supporting Information files.
Funding: The economic analysis presented in this paper was undertaken to inform a clinical practice guideline on social anxiety disorder developed by the National Collaborating Centre for Mental Health (NCCMH), commissioned and funded by the National Institute for Health and Care Excellence (NICE) in the UK. IM, EMW, KK and SP received support from the NCCMH, which was in receipt of funding from NICE for the submitted work. SD and AEA received support from the NICE Clinical Guidelines Technical Support Unit, University of Bristol, with funding from the Centre for Clinical Practice (NICE). The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have read the journal's policy and the authors of this manuscript have the following competing interests: IM, EMW, KK and SP received support from the National Collaborating Centre for Mental Health, which was in receipt of funding from the National Institute for Health and Care Excellence (NICE), for the submitted work. SD and AEA received support from the NICE Clinical Guidelines Technical Support Unit, University of Bristol, with funding from the Centre for Clinical Practice (NICE). DMC is the developer of one of the versions of individual CBT under consideration (Cognitive Therapy following the Clark & Wells manual). The authors report no other relationships or activities that could appear to have influenced the submitted work.
Abbreviations: CBT, Cognitive behavioural therapy; GDG, Guideline Development Group; HRQoL, Health-Related Quality of Life; NICE, National Institute for Health and Care Excellence; NMA, Network meta-analysis; NMB, Net Monetary Benefit
Social anxiety disorder is one of the most persistent and common anxiety disorders, with a lifetime prevalence estimated to range between 3.9% and 13.7% in Europe . People with social anxiety disorder have difficulty forming and retaining personal and social relationships , have higher risk of leaving school early and obtaining poorer qualifications , experience impairment in their daily functioning including work/school performance and social life , and report an important reduction in their quality of life compared with people without the disorder . They also incur considerable healthcare costs, especially relating to the use of primary care services, experience high levels of productivity losses and receive higher social benefits compared with people in the general population [6–8]. It has been shown that as the number of social fears increases, so does health service utilisation . The presence of comorbid psychiatric disorders increases usage of health services and productivity losses [6,8,9].
Several studies have assessed the clinical effectiveness of psychological and pharmacological treatments for social anxiety disorder [10–13]. Recently, individually delivered psychological therapies were demonstrated to be more effective than drugs and self-help for adults with social anxiety disorder . Given the variety of available interventions for the treatment of social anxiety disorder, the high costs associated with provision of psychological interventions, and the need for efficient use of healthcare resources under conditions of restricted budgets, the objective of this study was to examine the cost-effectiveness of a wide range of psychological and pharmacological interventions for the treatment of adults with social anxiety disorder from the perspective of the British National Health Service (NHS) and Personal Social Services, using decision-analytic modelling.
This study is an update of the economic analysis that informed the development of national clinical guidance for social anxiety disorder in England and Wales, published by the National Institute for Health and Care Excellence (NICE) . The guideline was developed by the Guideline Development Group (GDG), a multi-disciplinary team consisting of clinical academics, health professionals and service user and carer representatives with expertise and experience in the field of social anxiety. The GDG contributed to the development of the economic model by providing advice on issues relating to the natural history and treatment patterns of social anxiety disorder in the UK, as well as on model inputs in areas where evidence was lacking. The analysis presented here utilised efficacy data from an updated systematic review and network meta-analysis (NMA) of interventions for adults with social anxiety disorder .
The study population comprised adults aged at least 18 years who fulfilled diagnostic criteria for social anxiety disorder and were eligible for all the treatment options included in the analysis. Full details on the study population selection criteria are provided in a related publication .
The economic analysis assessed interventions that the GDG considered as appropriate first-line treatments for adults with social anxiety disorder, for which adequate clinical evidence was available. The interventions evaluated in the economic analysis were selected among those included in a recent comprehensive systematic review and NMA . We evaluated the cost effectiveness of distinct interventions, rather than classes of treatments, as there may be differences in resource implications among interventions belonging to the same class. Details on the rationale for the selection of appropriate interventions for consideration in the economic analysis are reported in the NICE full guideline report . The following interventions were considered:
- Seven selective serotonin / serotonin-noradrenergic reuptake inhibitors (SSRIs / SNRIs): citalopram, escitalopram, fluoxetine, fluvoxamine, paroxetine, sertraline and venlafaxine (prolonged-release formulation XL)
- Two monoamine oxidase inhibitors (MAOIs): moclobemide and phenelzine
- One anticonvulsant: pregabalin
- One noradrenergic and specific serotonergic antidepressant: mirtazapine
- Self-help without therapist support: book-based (SHNS, book) and computer-based (SHNS, internet), both involving minimal contact with therapists
- Self-help with therapist support: book-based (SHWS, book) and computer-based (SHWS, internet)
- Exposure in vivo
- Mindfulness training
- Two types of group cognitive behavioural therapy: group cognitive behavioural therapy following the Heimberg model (GCBT Heimberg) ; and group cognitive behavioural therapy not following a specified model (GCBT general)
- Interpersonal psychotherapy (IPT)
- Psychodynamic psychotherapy (PDPT)
- Supportive therapy
- Four types of individually delivered cognitive behavioural therapy: individual cognitive behavioural therapy following the Hope, Heimberg and Turk model (ICBT Hope) ; cognitive therapy following the Clark and Wells model (ICBT C&W) ; cognitive therapy following the Clark and Wells model with shortened sessions (ICBT short); and individual cognitive behavioural therapy not following a specified model (ICBT general)
The economic model also included two ‘inactive’ interventions, pill placebo and wait list, in order to assess the cost effectiveness of active interventions versus non-specific medical management and a do-nothing option, respectively.
The economic analysis made simultaneous comparisons of all the above treatment options; this was enabled by employing NMA methods to synthesise available clinical evidence .
Economic model structure
A hybrid decision-analytic model consisting of a decision-tree followed by a two-state Markov model was constructed using Microsoft Office Excel 2007 to assess total costs and Quality Adjusted Life Years (QALYs) of the 28 interventions for social anxiety disorder. The model structure, shown in Fig 1, was dictated by the natural history of the disorder, its treatment patterns and associated care pathways in the UK, and the availability of relevant clinical and epidemiological data.
The model followed hypothetical cohorts of adults with social anxiety disorder initiated on each of the 28 interventions assessed. For purposes of estimation of QALYs, initial treatment lasted 12 weeks for all interventions, although in trials and clinical practice the duration of psychological interventions ranges between 9 and 16 weeks; this modelling assumption did not affect estimated resource use. Following treatment, people in each cohort either recovered, no further meeting criteria for diagnosis (thus entering a state of ‘no social anxiety disorder’) or failed to recover (remaining in the state of ‘social anxiety disorder’). People who recoved following pharmacological treatment were assumed to receive another 26 weeks of maintenace treatment with the same drug; people recovering after psychological treatment did not receive further maintenance treatment. Those not recovering were assumed to stop treatment rather than switch to an alternative intervention as the GDG suggested this is likely in a population that is usually reluctant to maintain contact with health services.
During the year post-treatment, people who did not recover remained in the ‘social anxiety disorder’ state. Those who recovered might relapse, meeting again diagnostic criteria for the disorder, and thus re-enter the ‘social anxiety disorder’ state. From that point on all people in each cohort entered the Markov model, which was run in yearly cycles. During each cycle people remained in their current health state or moved between the ‘no social anxiety disorder’ and the ‘social anxiety disorder’ states. A half-cycle correction was applied. The yearly cycles were considered appropriate and consistent with the slow, chronic course of social anxiety disorder and the low rates of clinical changes in terms of recovery and relapse characterising the disorder , and were selected for convenience in terms of applying annual discounting of costs and benefits.
The time horizon of the analysis was the 12 weeks of initial treatment plus 5 years post-treatment, comprising 1 year in the decision-tree and 4 yearly cycles in the Markov model. This time horizon was selected to enable assessment of longer term benefits and costs associated with each intervention and to allow intervention costs to be spread over a longer time period, over which benefits of interventions can be still potentially enjoyed. A 5-year time horizon following 12 weeks of initial treatment was considered appropriate and reasonable, given that long-term clinical data were available from a prospective naturalistic study that observed the long-term clinical course of anxiety disorders including social anxiety disorder over 12 years , so there was no need for extrapolation of clinical data.
Clinical data utilised in the model
Efficacy data were derived from a systematic literature review and NMA of randomised controlled trials (RCTs) of interventions for adults with social anxiety disorder ; the NMA, based on a random effects model , was conducted within a Bayesian framework using Markov Chain Monte Carlo simulation techniques implemented in WinBUGS 1.4 [23,24]. The NMA included 101 trials reporting continuous data on symptom scales of social anxiety, of which 24 also reported dichotomous recovery data (with recovery being defined as no longer meeting criteria for diagnosis). The studies reported several continuous measures of social anxiety, none of which were common to all trials, so treatment effects for each trial were calculated as a standardised mean difference (SMD). Based on published psychometric properties and data from clinically referred participants who completed several measures, continuous measures were assumed to be equally responsive and had a mean correlation of 0.65 (for details see ). The outcome measures used in the clinical analysis are provided in S1 File.
The log-odds ratio of recovery was transformed into a notional SMD for recovery using the formula .
Following inspection of the relationship between the log-odds ratio estimated from the SMD of continuous data and the transformed log-odds ratio of recovery data in studies reporting both (—see Appendix 4 in Supplementary Appendix A), we assumed a linear relationship between the pooled treatment effects of symptom scales and the transformed treatment effects on recovery that was the same across all treatments, and forced the intercept at zero. The plausibility of the assumed relationship between the log-odds ratio of the SMD of continuous data and the transformed log-odds ratio of recovery data was assessed empirically by examining the relationship in the relative effects for studies reporting both outcomes, and discussed with clinical experts who considered it reasonable.
Thus, relative treatment effects expressed as SMD versus wait list were estimated from all studies and this correspondence (between the log-odds ratio of SMD of continuous data and the transformed log-odds ratio of recovery data) and its uncertainty were used to borrow strength from studies that reported recovery, and also to transform the estimates into log-odds ratios of recovery in order to populate the economic model. These were subsequently used to estimate absolute probabilities of recovery for each intervention using the probability of recovery for wait list as baseline, which was calculated by pooling recovery data from all wait list arms in RCTs included in the NMA. We note that the method used fully incorporates the uncertainty in the estimates.
The WinBUGS model was run with an initial burn-in period of 20,000 iterations, followed by 40,000 further iterations, thinned by 4 so as to obtain 10,000 iterations for use in the economic model. Full details of the methods and results of the systematic literature review and NMA, including risk of bias assessment of the individual studies, the full dataset, the statistical analysis of clinical data and the statistical code used in the NMA are provided in a related publication .
Other clinical input parameters.
The first-year probability of relapse after recovery with a drug was estimated using pooled data from 5 placebo-controlled pharmacological RCTs on relapse prevention in adults with social anxiety disorder [26–30], identified by a systematic literature search; this probability was estimated to be higher that the pooled risk of relapse during maintenance treatment in RCT active drug arms, but lower than the pooled risk of relapse of responders to initial drug treatment who were subsequently randomised to placebo (thus not receiving maintenance treatment); for simplicity and due to lack of more suitable data the first year probability of relapse for drugs was assumed to equal the midpoint of the two pooled risks. The resulting estimate, which was validated by GDG expert opinion, was utilised across all pharmacological treatment arms of the model and pill placebo, as drug-specific relapse data were sparse and mostly unavailable.
The first-year probability of relapse after recovery with psychological interventions was calculated by dividing the first-year probability of relapse for drugs by the risk ratio of relapse of drugs versus psychological therapies. The latter was estimated from data derived from a small observational 12-month follow-up study  of responders in an RCT of phenelzine versus group CBT for social anxiety disorder  and was applied to all psychological interventions due to lack of intervention-specific relapse data.
The first-year probability of relapse after recovery in wait list was estimated using data from a naturalistic study that reported 12-year probabilities of recovery and recurrence in adults with social anxiety disorder estimated using survival analysis . Data from this study were also used to calculate the annual probabilities of recovery and relapse in years 2 to 5 post-treatment (i.e. in the Markov component of the model), which were applied to all model cohorts, regardless of initial treatment.
Utility data considered in the model
Utility scores express preferences for different states of health-related quality of life (HRQoL) and are necessary for the estimation of QALYs. Following a systematic literature search of utility data for social anxiety disorder, the economic model was populated with utility scores obtained from a Finish national health survey  that reported EQ-5D utility scores (estimated using the UK Time Trade-Off Tarrif)  for people with social anxiety disorder and people with no mental disorder over the last 12 months. The 12-month prevalence of psychiatric disorders in survey participants was assesed using the Munich version of the Composite International Diagnostic Interview (M-CIDI) . Utility scores for people with social anxiety disorder over the previous 12 months were used for the ‘social anxiety disorder’ state. Utility scores for people with no mental disorder over the last 12 months were used as a proxy for the state of ‘no social anxiety disorder’, although it is acknowledged that people recovering from social anxiety disorder may not reach the HRQoL of a person without a mental disorder, and thus their HRQoL may have been overestimated, at least over the first few months following recovery.
Another limitation of using these data was that diagnosis referred to a 12-month period prior to data collection, so some survey participants might have experienced an improvement in their condition over this period (and actually might have recovered at the point of interview), and thus the HRQoL associated with the ‘social anxiety disorder’ state may have been overestimated. These limitations were deemed acceptable as no better quality utility data were identified; moreover, since overestimation of utility scores was likely to have occurred, to some extent, in both ‘social anxiety disorder’ and ‘no social anxiety disorder’ states, the effect of overestimation was deemed to be, at least partially, cancelled out and to be affecting the results of the economic analysis only insignificantly. The economic model assumed linear changes in utility when transitioning between ‘social anxiety disorder’ and ‘no social anxiety disorder’.
Healthcare resource use and cost data
The analysis considered intervention costs and other health and personal social service costs associated with social anxiety disorder, expressed in 2015 prices.
Pharmacological intervention costs consisted of drug acquisition and general practitioner (GP) visit costs. The intervention cost of pill placebo comprised GP visit costs only. The average daily dosage for each drug was determined according to optimal clinical practice  and was consistent with dosages reported in the RCTs that were included in the NMA. Initial drug treatment lasted 12 weeks, followed, in people who recovered, by 26 weeks of maintenance treatment at the same daily dosage, according to optimal practice. Initial drug treatment included 4 GP visits; maintenance treatment for those recovering included 3 extra GP visits. These resource use estimates were based on the GDG expert opinion and are lower than descriptions of medical resource use in pharmacological trial protocols. Drug acquisition costs and the GP unit cost were taken from national sources [37,38]. Active drug intervention costs, including 7 GP visits, ranged from £339 (citalopram) to £939 (pregabalin) (Table 1).
Psychological intervention costs were calculated by combining therapists’ time (as described in the relevant source RCTs) with respective national unit costs . All therapists were assumed to be Band 7 therapists according to the NHS Agenda for Change for qualified Allied Health Professionals. An initial GP visit for referral to psychological services was also considered. No booster sessions were assumed as there is no evidence to indicate that these are necessary for psychological treatment effect sustainment, and they are not routinely offered in clinical practice. Self-help intervention costs included the cost of either a book or a computerised programme and related infrastructure or equipment required for the programme delivery (license fee or website hosting, personal computers and capital overheads). The intervention cost of wait list was zero. Estimated psychological intervention costs for self-help interventions ranged between £205-£912 per person; for group therapies £551-£1,148 per person; and for individually delivered therapies £1,588-£2,359 per person (Table 2).
Adults with social anxiety disorder incur costs to health and personal social services that are associated with their disorder. Annual health and personal social service costs associated with social anxiety disorder were taken from a published analysis  of service use data obtained from a British Psychiatric Morbidity Survey conducted in 1993–1994 . These costs included GP consultations, home visits, counselling or therapy contacts and inpatient and outpatient secondary care, and were reported separately for people with social anxiety disorder and people without psychiatric morbidity (the latter was assumed to correspond to the state of ‘no social anxiety disorder’). These costs were not applied during the period of initial treatment (over which intervention costs were incurred), to avoid doublecounting of treatment costs, which might have already been included in the estimation of health and personal social service costs associated with the disorder, but were applied immediately after completion of initial treatment, at the end of 12 weeks.
Costs were expressed in 2015 prices, inflated, where necessary, using the Hospital and Community Health Services pay and prices index . The inflation index for year 2015 was estimated using the average value of the Hospital and Community Health Services pay and prices indices of the previous 3 years. Costs and QALYs were discounted at 3.5% annually, following NICE guidance .
An overview of methods, data sources and a list of the key assumptions underpinning the economic model are provided in S2 File.
To account for the uncertainty around the input parameter point estimates, a probabilistic analysis was undertaken, in which input parameters were assigned probabilistic distributions . Subsequently, 10,000 iterations were performed, each drawing random values out of the distributions fitted onto the model input parameters. Mean costs and QALYs and the Net Monetary Benefit (NMB) for each treatment option were calculated by averaging across the 10,000 iterations. The cost-effectiveness acceptability frontier was also plotted; this shows the treatment option with the highest mean NMB over different cost effectiveness thresholds, and the probability that the option with the highest NMB is the most cost-effective among those assessed. This method of presenting uncertainty has advantages over confidence interval estimation for incremental cost-effectiveness ratios, in particular in comparisons of multiple competing interventions, and presents uncertainty in a format that is more relevant to decision making .
The distributions of the probability of recovery for each intervention at end of treatment were defined directly from values recorded in the 10,000 iterations of the NMA. The log-odds of recovery on wait list was assumed to follow a normal distribution. The log-odds ratios of recovery for each treatment relative to wait list were applied to simulated values of this normal distribution and converted onto the probability scale. This ensured that the full posterior distribution of the relative treatment effects was used to estimate the absolute probabilities of recovery for each treatment .
Other probabilities and utility scores were assigned beta distributions. The risk ratio of relapse was assigned a log-normal distribution. Annual health and personal social service costs were assigned a gamma distribution. The estimation of distribution ranges was based on respective available data reported in the published sources of evidence.
Uncertainty in intervention costs was considered by assigning probability distributions around the number of GP visits and therapist sessions in individiually delivered psychological interventions, determined by completion rates and compliance data reported in RCTs included in the NMA. The same distributions around the number of GP visits were used for all pharmacological interventions. The number of therapist sessions per person attending group psychological interventions was not assigned a probabilistic distribution because the number of group sessions remains the same, whether a participant attends the full course of treatment or a lower number of sessions. Drug acquisition costs are not subject to uncertainty and thus were also not assigned probabilistic distributions. However, if people receiving pharmacological therapy attended considerably fewer GP visits than the mode, then they were assumed to be prescribed smaller amounts of medication than optimal, and to subsequently incur lower drug acquisition costs (either at initial or during maintenance treatment). Uncertainty in intervention costs of self-help interventions was considered by applying a normal distribution around the time spent with the therapist.
Table 3 reports the values of all model input parameters and provides details on the types and range of distributions assigned to each.
Deterministic sensitivity analyses explored the following alternative scenarios:
- self-help interventions to be supported by a Band 5 therapist and group therapies to be delivered by one Band 7 and one Band 6 therapists (in the base-case analysis all therapists were in Band 7 according to the NHS Agenda for Change for qualified Allied Health Professionals, with a unit cost of £110 per hour; Band 6 and Band 5 therapist unit costs were £93 and £87 per hour, respectively) [38,39].
- use of an alternative set of utility scores, based on EQ-5D data  derived from a community-based mental health European survey . These data reflected a smaller difference in utility between social anxiety disorder and no mental disorder compared with those used in the base-case analysis (this scenario used utility values of 0.79 and 0.91 for social anxiety disorder and no social anxiety disorder, respectively).
- use of alternative time horizons of 1, 3, and 10 years following the 12 weeks of initial treatment. These scenarios were tested to explore the short-term cost effectiveness of interventions for social anxiety disorder and potential changes in their relative cost effectiveness over time and in the long term.
- the risk ratio of relapse of drugs versus psychological interventions for year 1 to equal a range of values between 1.0 and 2.0, as this may be closer to real life than the reported mean value of 3.0  that was utilised in the economic analysis.
Output of network meta-analysis used in the economic analysis
Results of the NMA that populated the economic model are provided in Table 4, which shows the probability of recovery at end of treatment for each intervention. Treatments have been ranked from the most to the least effective; the mean probability of recovery ranged from 0.62 (95% credible intervals 0.16 to 0.95) for ICBT C&W down to 0.10 (95% credible intervals 0.01 to 0.39) for wait list. Overall, the various forms of ICBT showed high probabilities of recovery (mean value in the range of 0.39–0.62) with other individually delivered psychological therapies being less effective (range of mean probabilities of recovery 0.16–0.26). The mean probabilities of recovery for drugs and self-help interventions were in the range of 0.27 to 0.40, with the exception of phenelzine, the mean probability of which reached 0.51 (second most effective option following ICBT C&W). The mean probabilities of recovery for group psychological therapies ranged from 0.19 (mindfulness) to 0.34 (GCBT general).
ICBT C&W was the most cost-effective intervention for adults with social anxiety disorder, with the highest NMB at the NICE lower cost effectiveness threshold of £20,000/QALY , despite having the highest intervention cost. The second most cost-effective option was ICBT general, while other forms of ICBT, i.e. ICBT Hope and ICBT short ranked 6th and 9th, respectively. Phenelzine ranked 3rd, while book-based self-help without / with support ranked 4th and 5th, respectively. Group-based psychological interventions were not particularly cost-effective relative to other treatments, ranking in places between 11th and 17th, except mindfulness training, which ranked 23rd. Drugs (apart from phenelzine) also tended to rank low in terms of cost effectiveness, taking up places between 8th and 22nd; following phenelzine, the order of the next most cost-effective drugs was: paroxetine, venlafaxine, fluvoxamine, sertraline, escitalopram and fluoxetine. Internet-based self-help ranked 7th (with support) and 20th (without support). Psychodynamic psychotherapy ranked 25th, just above wait list; interpersonal psychotherapy ranked 27th while supportive therapy was the least cost-effective intervention, ranking in the last (28th) place. Table 5 provides the mean total costs and QALYs per person and the percentage of adults that are well (no social anxiety disorder) at 5 years for each intervention assessed in the economic analysis; it also provides the results of incremental analysis, the mean NMB of each intervention at the NICE lower cost effectiveness threshold (£20,000/QALY), and the ranking of interventions according to their mean NMB. Detailed probabilistic results, including 95% credible intervals of costs, QALYs and NMB of each intervention, the mean ranking of each intervention and its probability of being cost-effective are provided in S1 Table.
Mean values per person 5 years after end of treatment.
The cost effectiveness plane (Fig 2) shows the incremental costs and QALYs of all interventions versus wait list (placed at the origin). The cost-effectiveness efficiency frontier (continuous line) links all interventions that are not dominated by absolute or extended dominance and are thus cost-effective at different cost effectiveness thresholds. The slope of the dotted line indicates the NICE lower cost effectiveness threshold, suggesting that IPT and supportive therapy are not cost-effective compared with wait list (since both lie on the left of the dotted line).
Wait list is placed at the origin; results are for 1,000 adults with social anxiety disorder at 5 years after treatment. The continuous line shows the cost-effectiveness efficiency frontier, while the slope of the dotted line indicates the NICE lower cost effectiveness threshold (£20,000/QALY). The data used to construct Fig 2 are provided in Table 5.
According to the cost-effectiveness acceptability frontier (Fig 3), for cost effectiveness thresholds between zero and £1,472/QALY, book-based self-help without support has the highest NMB (i.e. it is the most cost-effective treatment), but its probability of being cost-effective does not exceed 0.24. Phenelzine is the most cost-effective intervention for cost effectiveness thresholds between £1,472/QALY and £9,179/QALY, with a maximum probability of cost effectiveness just above 0.38. At even higher cost effectiveness thresholds, ICBT (C&W) has the highest NMB and a probability of cost effectiveness that increases with increasing cost effectiveness thresholds. The probability of ICBT (C&W) being cost-effective reaches 0.68 at the NICE £20,000/QALY threshold of and goes beyond 80% when the threshold exceeds £32,500/QALY.
The values used to construct Fig 3 are provided in S2 Table.
Results were robust to use of lower unit costs for therapists involved in self-help and group psychological interventions and the use of more conservative utility data. ICBT (C&W) remained the most cost-effective intervention under these scenarios, and the cost effectiveness ranking of interventions was broadly the same. Sensitivity analysis of different time horizons indicated that, in the short-term, pharmacological interventions are overall more cost-effective than psychological interventions. At a 1-year time horizon, phenelzine produced the highest NMB and was the most cost-effective among treatment options, whereas ICBT (C&W) ranked only 17th. Of the first 10 places in ranking, 8 were occupied by pharmacological interventions and the remaining 2 were occupied by book-based self-help (without / with support). However, by 3 years ICBT (C&W) became the most cost-effective intervention followed by phenelzine. The relative cost effectiveness of drugs was reduced compared with their cost effectivess at a 1-year time horizon; in constrast, the cost effectiveness of psychological interventions improved, whereas the cost effectiveness of book-based self-help remained broadly the same. By 10 years, the ranking of interventions by cost effectiveness mirrored their ranking by effectiveness (number of QALYs), with the 4 individual forms of CBT occupying the 4 first places in ranking due to their higher intervention costs having been fully offset by health and personal social service cost-savings.
As expected, use of lower risk ratios of relapse for drugs versus psychological interventions favoured drugs. When an equal risk of relapse was assumed, phenelzine became the most cost-effective intervention and, with the exception of ICBT (C&W) which ranked 2nd, the rest top-5 places in cost effectiveness ranking were occupied by drugs. However, ICBT (C&W) became the most cost-effective option when the risk ratio of relapse rose at just 1.15, and performance of psychological interventions improved substantially with increasing values of this ratio. For example, at a value of 1.5, 3 psychological interventions (ICBT, C&W; ICBT, general and SHNS, book) ranked in the top-5 places, while at a risk ratio of 2, the 5 most cost-effective options included 4 psychological interventions and phenelzine, with overall results being broadly consistent with those of the base-case analysis.
Results of deterministic one-way sensitivity analyses are provided in S3 File. The deterministic ranking of interventions according to cost effectiveness for different time horizons tested is provided in S3 Table. The deterministic ranking of interventions according to cost effectiveness at different values of the risk ratio of relapse of drugs versus psychological interventions (year 1) is shown in S4 Table.
Individual forms of CBT appear to be the most cost-effective interventions for adults with social anxiety disorder. Self-help, in particular book-based, is cost-effective compared with other options. Group-based psychological interventions and drugs (with the exception of phenelzine, which was among the most cost-effective options) do not appear to be particularly cost-effective relative to other treatments. Other individually delivered psychological interventions such as psychodynamic psychotherapy, interpersonal therapy and supportive therapy are the least cost-effective options for adults with social anxiety disorder. The emergence of individual forms of CBT as most cost-effective options despite their high intervention costs is attributed to two factors: their higher effectiveness compared with other interventions and the considerably lower risk of relapse of psychological interventions compared with drugs; consequently these interventions result in higher health benefits (QALYs), but also in a reduction in health and personal social service costs attributable to social anxiety disorder. Moreover, the cost effectiveness of individual forms of CBT appears to increase over time because their high intervention costs (which are responsible for the low performance of these interventions 1 year after treatment) are spread over a longer time period and are offset to a greater extent by health and personal social service cost savings associated with the successful management of social anxiety disorder.
Strengths and limitations
The bigest strength of our analysis is the utilisation of efficacy data derived from a systematic literature review and NMA. This methodology enabled us to consider information from both direct and indirect comparisons between interventions, and allowed simultaneous inference on all treatment options examined in trial pair-wise comparisons while preserving randomisation [20,48]. This approach for evidence synthesis is essential for populating model-based economic studies assessing more than two competing interventions. The NMA principally utilised continuous data to estimate the relative treatment effects of interventions, and then transformed the estimated SMDs into probabilities of recovery. Such a transformation is valid as long as the assumed relationship between the treatment effect based on continuous data and the treatment effect estimated using recovery data holds. This assumption could not be checked for all interventions, but available data indicated a strong relationship and therefore this transformation is unlikely to have introduced substantial bias into the analysis . These assumptions along with the limitations of the NMA model and the limitations of the RCTs considered in the NMA  may have impacted on the quality of the respective input parameters used to populate the economic model.
A number of interventions such as benzodiazepines, anticonvulsants and combination therapies that were included in the NMA  were not considered in the economic analysis. Benzodiazepines cannot be used beyond 2–4 weeks for the treatment of anxiety . Clinical evidence on anticonvulsants and combination therapies was particularly limited and overall of low quality  so that inclusion of these interventions in the analysis would not have important implications for decision making.
An important limitation of the economic analysis was the poor quality of the relapse data used, due to lack of robust evidence on the relative risk of relapse between pharmacological and psychological interventions for social anxiety disorder. Furthermore, due to lack of intervention-specific data, the economic model assumed one (common) risk of relapse applied to all pharmacological interventions, and one (common) risk of relapse across all psychological ones. Nevertheless, evidence suggests that, in constrast to pharmacological interventions, which are characterised by a relatively high relapse risk at 6 months of maintenance treatment [26–30], the effect of psychological interventions is well-maintained in the long-term, after end of treatment [49,50]. Moreover, the mean probabilities of relapse for drugs and psychological interventions estimated for the economic model (42% versus 14%, respectively) are similar to relapse rates reported in trials of these types of therapies for social anxiety disorder, very close to respective relapse rates reported for people with obsessive compulsive disorder (45% versus 12%, respectively)  and broadly consistent with respective figures reported for panic disorder (40% versus 5%, respectively) . Sensitivity analysis showed that more conservative assumptions regarding the superiority of psychological interventions over drugs in retaining long-term effects would lead to broadly the same conclusions.
Another limitation of the economic analysis was the lack of consideration of the impact of the side effects of drugs on the HRQoL and costs, due to inconsistent reporting of side effect data in the RCTs included in the NMA. In particular, use of phenelzine, which was ranked 3rd most cost-effective intervention, is associated with a potentially dangerous interaction with foods containing tyramine that may lead to episodes of high blood pressure, thus imposing dietary restrictions to people administered this drug. Omission of the impact of drug-related side effects from the economic model has likely overestimated the cost effectiveness of all pharmacological interventions. Other limitations of the model include the use of utility data that were not specific to health states of social anxiety disorder and the use of resource use data associated with social anxiety disorder collected almost 20 years ago, due to lack of better quality and more up-to-date data, respectively.
Comparison with other published studies in the field
Published economic analyses have explored the cost-effectiveness of a very limited range of interventions for social anxiety disorder and concluded that escitalopram , group CBT [54–56] and computer-based self-help [55–57] are cost-effective options. All studies were characterised by methodological limitations, such as exclusive consideration of intervention costs and omission of other relevant healthcare [54,57] or equipment (computer) [55–57] costs, an inability to accurately record relevant healthcare costs from all study participants , and inappropriate estimation of the relative effectiveness of interventions, by utilising their effect sizes versus different controls (waitlist or pill placebo), so that the effect sizes were not comparable . Therefore, these findings should be treated with caution.
Generalisability of the results and implications of the study
Our analysis has been conducted from the perspective of the British NHS and Personal Social Services. Results are likely generalisable to other settings with similar funding and structure of healthcare and personal social services and comparable care pathways for social anxiety disorder. It should be noted that conclusions on cost effectiveness ultimately rely on the cost effectiveness threshold adopted, and this depends on the policy makers’ willingness-to-pay for treatment benefits, which may vary across countries and health systems. The economic model did not incorporate productivity losses. However, more effective interventions can reasonably be expected to lead to improved functioning, and, in turn, to increased employment rates and reduced days of sick leave, thus to an increase in productivity gains. Consequently, consideration of productivity losses in our analysis would probably further favour individual forms of CBT.
Based on our findings and after considering potential drug interactions (fluvoxamine), side effects that were not considered in the model (such as possible blood pressure changes with venlafaxine and phenelzine, discontinuation symptoms associated with venlafaxine and SSRIs and in particular with paroxetine), dietary restrictions with the MAOIs, the limited evidence base for fluoxetine relative to other drugs, and concerns about the quality of phenelzine data which may have led to overestimation of the drug’s effect, the NICE clinical guideline on social anxiety disorder recommended individually delivered CBT that has been specifically designed for social anxiety disorder as first line treatment for adults with social anxiety disorder . Supported self-help was recommended as second-line psychological treatment, while SSRIs (escitalopram or sertraline) were recommended for people preferring pharmacological treatment. Additional recommendations were made for people not responding or partially responding to treatment and for those developing side effects. Further good quality research is needed on the comparative long-term outcomes of psychological and pharmacological interventions as well as on costs and utility values associated with social anxiety disorder, in order to establish the relative cost effectiveness of interventions for social anxiety disorder with greater certainty.
S1 File. Outcome measures used in the clinical analysis.
S2 File. Overview of methods, data sources and key assumptions underpinning the economic model.
S3 File. Results of deterministic one-way sensitivity analyses.
Mean values per person 5 years after end of treatment
S1 Table. Detailed results of probabilistic sensitivity analysis.
Mean values (95% credible intervals) per person, 5 years after end of treatment
Probability of cost effectiveness of the interventions with the highest net monetary benefit at varying levels of willingness-to-pay per QALY gained.
S3 Table. Ranking of interventions by cost effectiveness at different time horizons of the economic model—deterministic analysis.
We wish to thank the Guideline Development Group (GDG) for the National Institute for Health and Care Excellence (NICE) guideline Social Anxiety Disorder: Recognition, Assessment and Treatment for useful comments in the development of the economic analysis and the interpretation of the results, and in particular Andrea Malicia for his clinical expert advice in the development of the economic model and Sarah Stockton for developing the search strategy and conducting the electronic searches of the literature that informed the economic model. The members of the GDG were David Clark (chair), Stephen Pilling (facilitator), Evan Mayo-Wilson, Ifigeneia Mavranezouli, Safi Afghan, Benedict Anigbogu, Peter Armstrong, Madeleine Bennett, Sam Cartwright-Hatton, Cathy Creswell, Melanie Dix, Nick Hanlon, Kayleigh Kew, Katherine Leggett, Andrea Malizia, Jane Roberts, Kate Satrettin, Gareth Stephens, Sarah Stockton, Lusia Stopa, and Clare Taylor.
Conceived and designed the experiments: IM EMW SD KK DMC AEA SP. Performed the experiments: IM. Analyzed the data: IM. Wrote the paper: IM EMW SD KK DMC AEA SP. Contributed to data collection: IM EMW KK. Developed the statistical model and code for the network meta-analysis: SD AEA. Interpreted the findings: IM EMW SD KK DMC AEA SP.
- 1. Fehm L, Pelissolo A, Furmark T, Wittchen HU. Size and burden of social phobia in Europe. Eur Neuropsychopharmacol 2005;15: 453–462. pmid:15921898
- 2. Whisman MA, Sheldon CT, Goering P. Psychiatric disorders and dissatisfaction with social relationships: does type of relationship matter? J Abnorm Psychol 2000;109: 803–808. pmid:11196008
- 3. Van Ameringen M, Mancini C, Farvolden P. The impact of anxiety disorders on educational achievement. J Anxiety Disord 2003;17: 561–571. pmid:12941366
- 4. Aderka IM, Hofmann SG, Nickerson A, Hermesh H, Gilboa-Schechtman E, Marom S. Functional impairment in social anxiety disorder. J Anxiety Disord 2012;26: 393–400. pmid:22306132
- 5. Stein MB, Kean YM. Disability and quality of life in social phobia: epidemiologic findings. Am J Psychiatry 2000;157: 1606–1613. pmid:11007714
- 6. Patel A, Knapp M, Henderson J, Baldwin D. The economic consequences of social phobia. J Affect Disord 2002;68: 221–233. pmid:12063150
- 7. Acarturk C, Smit F, de Graaf R, van Straten A, ten Have M, Cuijpers P (2009) Economic costs of social phobia: A population-based study. J Affect Disord 2009;115: 421–429. pmid:19012968
- 8. Stuhldreher N, Leibing E, Leichsenring F, Beutel ME, Herpertz S, Hoyer J, et al. The costs of social anxiety disorder: the role of symptom severity and comorbidities. J Affect Disord 2014;165: 87–94. pmid:24882183
- 9. Acarturk C, de Graaf R, van Straten A, Have MT, Cuijpers P. Social phobia and number of social fears, and their association with comorbidity, health-related quality of life and help seeking: a population-based study. Soc Psychiatry Psychiatr Epidemiol 2008;43: 273–279. pmid:18219433
- 10. Acarturk C, Cuijpers P, van Straten A, de Graaf R. Psychological treatment of social anxiety disorder: a meta-analysis. Psychol Med 2009;39: 241–254. pmid:18507874
- 11. Ipser JC, Kariuki CM, Stein DJ. Pharmacotherapy for social anxiety disorder: a systematic review. Expert Rev Neurother 2008;8: 235–257. pmid:18271710
- 12. Canton J, Scott KM, Glue P. Optimal treatment of social phobia: systematic review and meta-analysis. Neuropsychiatr Dis Treat 2012;8: 203–215. pmid:22665997
- 13. Bandelow B, Seidler-Brandler U, Becker A, Wedekind D, Ruther E. Meta-analysis of randomized controlled comparisons of psychopharmacological and psychological treatments for anxiety disorders. World J Biol Psychiatry 2007;8: 175–187. pmid:17654408
- 14. Mayo-Wilson E, Dias S, Mavranezouli I, Kew K, Clark DM, Ades AE, et al. Psychological and pharmacological interventions for social anxiety disorder in adults: a systematic review and network meta-analysis. Lancet Psychiatry 2014;1: 368–376. pmid:26361000
- 15. NICE. Social Anxiety Disorder: Recognition, Assessment and Treatment. NICE clinical guideline 159. London: National Institute for Health and Care Excellence; 2013. Available: www.nice.org.uk/CG159
- 16. NCCMH. Social Anxiety Disorder: recognition, assessment and treatment of social anxiety disorder. Commissioned by the National Institute for Health and Care Excellence. Leicester & London: The British Psychological Society and the Royal College of Psychiatrists; 2013.
- 17. Heimberg RG, Becker RE. Cognitive-behavioural group therapy for social phobia: basic mechanisms and clinical strategies (treatment manuals for practitioners). New York: The Guildford Press; 2002.
- 18. Hope DA, Heimberg RG, Turk CL (2006) Therapist guide for managing social anxiety: a cognitive-behavioral therapy approach. New York: Oxford University Press; 2006.
- 19. Clark DM, Wells A. A cognitive model of social phobia. In: Heimberg RG, Liebowitz MR, Hope DA, Schneier FR, editors. Social Phobia: diagnosis, assessment, and treatment. New York: Guildford; 1995. pp. 69–93.
- 20. Caldwell DM, Ades AE, Higgins JP. Simultaneous comparison of multiple treatments: combining direct and indirect evidence. BMJ 2005;331: 897–900. pmid:16223826
- 21. Bruce SE, Yonkers KA, Otto MW, Eisen JL, Weisberg RB, Pagano M, et al. Influence of psychiatric comorbidity on recovery and recurrence in generalized anxiety disorder, social phobia, and panic disorder: a 12-year prospective study. Am J Psychiatry 2005;162: 1179–1187. pmid:15930067
- 22. Dias S, Sutton AJ, Ades AE, Welton NJ. Evidence synthesis for decision making 2: a generalized linear modeling framework for pairwise and network meta-analysis of randomized controlled trials. Med Decis Making 2013;33: 607–617. pmid:23104435
- 23. Lunn DJ, Thomas A, Best N, Spiegelhalter D. WinBUGS—A Bayesian modelling framework: Concepts, structure, and extensibility. Stat Comput 2000;10: 325–337.
- 24. Spiegelhalter DJ, Thomas A, Best NG, Lunn D. WinBUGS user manual: version 1.4. Cambridge: MRC Biostatistics Unit; 2003.
- 25. Chinn S. A simple method for converting an odds ratio to effect size for use in meta-analysis. Stat Med 2000;19: 3127–3131. pmid:11113947
- 26. Kumar R, Pitts C, Carpenter D. Response to paroxetine is maintained during continued treatment in patients with social anxiety disorder. Eur Neuropsychopharmacol 1999;9: S312–313.
- 27. Stein DJ, Versiani M, Hair T, Kumar R. Efficacy of paroxetine for relapse prevention in social anxiety disorder: a 24-week study. Arch Gen Psychiatry 2002;59: 1111–1118. pmid:12470127
- 28. Montgomery SA, Nil R, Durr-Pal N, Loft H, Boulenger JP. A 24-week randomized, double-blind, placebo-controlled study of escitalopram for the prevention of generalized social anxiety disorder. J Clin Psychiatry 2005;66: 1270–1278. pmid:16259540
- 29. Van Ameringen MA, Lane RM, Walker JR, Bowen RC, Chokka PR, Goldner EM, et al. Sertraline treatment of generalized social phobia: a 20-week, double-blind, placebo-controlled study. Am J Psychiatry 2001;158: 275–281. pmid:11156811
- 30. Greist GH, Liu-Dumaw M, Schweizer E, Feltner D. Efficacy of pregabalin in preventing relapse in patients with generalized social anxiety disorder: results of a double-blind, placebo-controlled 26-week study. Int Clin Psychopharmacol 2011;26: 243–251. pmid:21734588
- 31. Liebowitz MR, Heimberg RG, Schneier FR, Hope DA, Davies S, Holt CS, et al. Cognitive-behavioral group therapy versus phenelzine in social phobia: long-term outcome. Depress Anxiety 1999;10: 89–98. pmid:10604081
- 32. Heimberg RG, Liebowitz MR, Hope DA, Schneier FR, Holt CS, Welkowitz LA, et al. Cognitive behavioral group therapy vs phenelzine therapy for social phobia: 12-week outcome. Arch Gen Psychiatry 1998;55: 1133–1141. pmid:9862558
- 33. Saarni SI, Suvisaari J, Sintonen H, Pirkola S, Koskinen S, Aromaa A, et al. Impact of psychiatric disorders on health-related quality of life: general population survey. Br J Psychiatry 2007;190: 326–332. pmid:17401039
- 34. Dolan P. Modeling valuations for EuroQol health states. Med Care 1997;35: 1095–1108. pmid:9366889
- 35. Wittchen HU, Lachner G, Wunderlich U, Pfister H. Test-retest reliability of the computerized DSM-IV version of the Munich-Composite International Diagnostic Interview (M-CIDI). Soc Psychiatry Psychiatr Epidemiol 1998;33: 568–578. pmid:9803825
- 36. British Medical Association & the Royal Pharmaceutical Society of Great Britain. British National Formulary (BNF). May 2015. London: Pharmaceutical Press; 2015. Available: http://www.evidence.nhs.uk/formulary/bnf/current
- 37. NHS Business Services Authority, NHS Prescription Services. Electronic Drug Tariff for England and Wales, May 2015. Compiled on behalf of the Department of Health; 2015. Available: http://www.nhsbsa.nhs.uk/PrescriptionServices/4940.aspx
- 38. Curtis L. Unit costs of health & social care 2014. Canterbury: Personal and Social Services Research Unit, University of Kent; 2014.
- 39. Curtis L. Unit costs of health & social care 2010. Canterbury: Personal and Social Services Research Unit, University of Kent; 2010.
- 40. Kaltenthaler E, Brazier J, De Nigris E, Tumur I, Ferriter M, Beverley C, et al. Computerised cognitive behaviour therapy for depression and anxiety update: a systematic review and economic evaluation. Health Technology Assessment 2006;10(33).
- 41. Meltzer H, Gill B, Petticrew M, Hinds K. OPCS surveys of psychiatric morbidity in great britain, report 2: physical complaints, service use and treatment of adults with psychiatric disorders. London: Office for National Statistics; 1995.
- 42. NICE. Process and methods guides. Guide to the Methods of Technology Appraisal. London: National Institute for Health and Care Excellence; 2013. Available: http://www.nice.org.uk/article/pmg9/chapter/Foreword
- 43. Briggs A, Sculpher M, Claxton K. Decision Modelling for Health Economic Evaluation. New York: Oxford University Press; 2006.
- 44. Fenwick E, Claxton K, Sculpher M. Representing uncertainty: the role of cost-effectiveness acceptability curves. Health Econ 2001;10: 779–787. pmid:11747057
- 45. Dias S, Welton NJ, Sutton AJ, Ades AE. Evidence Synthesis for Decision Making 5: The Baseline Natural History Model. Med Decis Making 2013;33: 657–670. pmid:23804509
- 46. Alonso J, Angermeyer MC, Bernert S, Bruffaerts R, Brugha TS, Bryson H, et al. Disability and quality of life impact of mental disorders in Europe: results from the European Study of the Epidemiology of Mental Disorders (ESEMeD) project. Acta Psychiatr Scand Suppl 2004;420: 38–46. pmid:15128386
- 47. NICE. Social Value Judgements: principles for the development of NICE Guidance. London: National Institute for Health and Care Excellence; 2008.
- 48. Lu G, Ades AE. Combination of direct and indirect evidence in mixed treatment comparisons. Stat Med 2004;23: 3105–3124. pmid:15449338
- 49. Haug TT, Blomhoff S, Hellstrom K, Holme I, Humble M, Madsbu HP, et al. Exposure therapy and sertraline in social phobia: I-year follow-up of a randomised controlled trial. Br J Psychiatry 2003;182: 312–318. pmid:12668406
- 50. Mortberg E, Clark DM, Bejerot S. Intensive group cognitive therapy and individual cognitive therapy for social phobia: sustained improvement at 5-year follow-up. J Anxiety Disord 2011;25: 994–1000. pmid:21763101
- 51. Simpson HB, Fallon BA. Obsessive-compulsive disorder: an overview. J Psychiatr Pract 2000;6: 3–17. pmid:15990471
- 52. Clark DM, Salkovskis PM, Hackmann A, Middleton H, Anastasiades P, Gelder M. A comparison of cognitive therapy, applied relaxation and imipramine in the treatment of panic disorder. Br J Psychiatry 1994;164: 759–769. pmid:7952982
- 53. Francois C, Montgomery SA, Despiegel N, Aballea S, Roiz J, Auquier P. Analysis of health-related quality of life and costs based on a randomised clinical trial of escitalopram for relapse prevention in patients with generalised social anxiety disorder. Int J Clin Pract 2008;62: 1693–1702. pmid:18759783
- 54. Gould R, Buckminster S, Pollack M, Otto M, Yap L. Cognitive-Behavioral and Pharmacological Treatment for Social Phobia: A Meta-Analysis. Clin Psychol (New York) 1997;4: 291–306.
- 55. Hedman E, Andersson E, Ljotsson B, Andersson G, Ruck C, Lindefors N. Cost-effectiveness of Internet-based cognitive behavior therapy vs. cognitive behavioral group therapy for social anxiety disorder: results from a randomized controlled trial. Behav Res Ther 2011;49: 729–736. pmid:21851929
- 56. Hedman E, El Alaoui S, Lindefors N, Andersson E, Ruck C, Ghaderi A, et al. (2014) Clinical effectiveness and cost-effectiveness of Internet- vs. group-based cognitive behavior therapy for social anxiety disorder: 4-year follow-up of a randomized trial. Behav Res Ther 2014;59: 20–29. pmid:24949908
- 57. Titov N, Andrews G, Johnston L, Schwencke G, Choi I. Shyness programme: longer term benefits, cost-effectiveness, and acceptability. Aust N Z J Psychiatry 2009;43: 36–44. pmid:19085526