Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Is the efficacy of antidepressants in panic disorder mediated by adverse events? A mediational analysis

  • Irene Bighelli ,

    irene.bighelli@tum.de

    Affiliation Department of Psychiatry and Psychotherapy, Klinikum rechts der Isar, Technische Universität München, Munich, Germany

  • Anna Borghesani,

    Affiliation WHO Collaborating Centre for Research and Training in Mental Health and Service Evaluation, Department of Neuroscience, Biomedicine and Movement Sciences, Section of Psychiatry, University of Verona, Verona, Italy

  • Corrado Barbui

    Affiliation WHO Collaborating Centre for Research and Training in Mental Health and Service Evaluation, Department of Neuroscience, Biomedicine and Movement Sciences, Section of Psychiatry, University of Verona, Verona, Italy

Is the efficacy of antidepressants in panic disorder mediated by adverse events? A mediational analysis

  • Irene Bighelli, 
  • Anna Borghesani, 
  • Corrado Barbui
PLOS
x

Abstract

It has been hypothesised that the perception of adverse events in placebo-controlled antidepressant clinical trials may induce patients to conclude that they have been randomized to the active arm of the trial, leading to the breaking of blind. This may enhance the expectancies for improvement and the therapeutic response. The main objective of this study is to test the hypothesis that the efficacy of antidepressants in panic disorder is mediated by the perception of adverse events. The present analysis is based on a systematic review of published and unpublished randomised trials comparing antidepressants with placebo for panic disorder. The Baron and Kenny approach was applied to investigate the mediational role of adverse events in the relationship between antidepressants treatment and efficacy. Fourteen placebo-controlled antidepressants trials were included in the analysis. We found that: (a) antidepressants treatment was significantly associated with better treatment response (ß = 0.127, 95% CI 0.04 to 0.21, p = 0.003); (b) antidepressants treatment was not associated with adverse events (ß = 0.094, 95% CI -0.05 to 0.24, p = 0.221); (c) adverse events were negatively associated with treatment response (ß = 0.035, 95% CI -0.06 to -0.05, p = 0.022). Finally, after adjustment for adverse events, the relationship between antidepressants treatment and treatment response remained statistically significant (ß = 0.122, 95% CI 0.01 to 0.23, p = 0.039). These findings do not support the hypothesis that the perception of adverse events in placebo-controlled antidepressant clinical trials may lead to the breaking of blind and to an artificial inflation of the efficacy measures. Based on these results, we argue that the moderate therapeutic effect of antidepressants in individuals with panic disorder is not an artefact, therefore reflecting a genuine effect that doctors can expect to replicate under real-world conditions.

Introduction

In randomised studies allocating patients to antidepressants or placebo it is possible that differences in efficacy may, at least in part, be explained by the emergence of adverse events. In the treatment of major depression the evidence is controversial. A meta-analysis of studies comparing fluoxetine with placebo reported a strong correlation between adverse events and efficacy, and authors supposed that the rating scores for patients allocated to antidepressants might have been amplified when study participants became aware of being allocated to active treatment by experiencing adverse events [1]. A more recent study on antidepressants in depression, however, was not able to replicate this finding [2].

A similar hypothesis may be formulated for antidepressants in the treatment of panic disorder, as patients with panic disorder may be particularly prone to adverse events and these, in turn, may have an impact on efficacy measures. The hypothesis is the following. During the informed consent procedure, patients with panic disorder are told that they may receive placebo and are also informed of the side effects to be expected from the real drug. Hence, when they experience side effects, they are likely to conclude that they have been randomized to the active arm of the trial. In fact, most patients and doctors in clinical trials are successfully able to guess whether the patient has been randomized to drug or placebo ([3]; [4]; [5]; [6]). Since the placebo effect is presumed to be associated with expectancies for improvement, concluding that one has been given the real drug (after experiencing adverse events) ought to enhance the therapeutic response, whereas concluding that one has been given placebo (after not experiencing adverse events) ought to diminish it.

If drug-placebo differences are due to the breaking of blind, then they ought to be associated with the perception of adverse events. The purpose of the study reported here is to test the hypothesis that the effects of antidepressants are correlated with the perception of adverse events and to assess what remains of the drug effect when side effects are controlled statistically. We were able to test this hypothesis performing a mediational analysis.

Materials and methods

Search methods for identification of studies

The present study is based on an ongoing Cochrane systematic review and meta-analysis of published and unpublished randomised trials comparing antidepressants versus placebo for panic disorder [7]. We refer to Guaiana et al. for a detailed description of the search strategy and full methodology. Briefly, the Cochrane Collaboration Common Mental Disorder Group Trials Register (CCDANTR) was searched. This register includes relevant RCTs from the Cochrane Central Register of Controlled Trials (CENTRAL) (all years), MEDLINE (1950 to date), EMBASE (1974 to date) and PsycINFO (1967 to date). No language restriction was applied. Reference lists of relevant papers and previous systematic reviews were hand-searched. Moreover, the pharmaceutical companies marketing antidepressants, experts in this field and trial authors were contacted for additional unpublished data.

Types of studies and interventions

We included randomized double-blind comparisons of antidepressants as monotherapy versus placebo in the treatment of panic disorder. For trials that had a crossover design, only results from the first randomized period were considered.

Types of participants

Participants were in- and out-patients aged 18 years or older of both sexes, with a primary diagnosis of panic disorder with or without agoraphobia, diagnosed according to the criteria described in the Diagnostic and Statistical Manual of Mental Disorders (DSM) [8] and International Classification of Disease (ICD) [9], or according to any other clinical or standardized criteria adopted by the study authors. In case study eligibility focused on agoraphobia, rather than panic disorder, studies were to be included if operationally diagnosed according to the above-named criteria and when it could be safely assumed that at least 30% of the participants were suffering from panic disorder as defined by the above criteria. We included participants with a concurrent secondary diagnosis of another psychiatric disorder. We excluded participants with a concurrent primary diagnosis of Axis I or II disorders and participants with a serious concomitant comorbid physical disorder.

Data extraction

Two reviewers (IB and AB) independently extracted data from the included studies, and any disagreement was discussed with a third member of the review team (CB).

For the purposes of the analysis reported here, the following two outcomes were extracted: (a) efficacy, measured as number of patients responding to treatment as defined by the original investigators; (b) adverse events, measured as proportion of patients complaining with any adverse events.

Additionally, the following information was collected using an electronic spreadsheet: year of publication, type of antidepressant, sample size, inclusion of elderly participants, setting of intervention, diagnostic criteria, baseline number of panic attacks per week (as a proxy of illness severity), dose and probability of receiving placebo, rated according to Papakostas and colleagues [10]. Mean doses were converted into multiples of the defined daily dosage (DDD) for each drug by dividing the prescribed daily dosage (PDD) by the DDD (PDD/DDD). This measure is the international unit of drug use approved by the World Health Organisation for drug use studies [11,12].

Statistical analysis

Initially we investigated whether the proportion of patients experiencing adverse events significantly related to the overall efficacy of antidepressants. A meta-regression analysis was carried out with the metareg command in STATA, which performs standard random-effects meta-regression using aggregate-level data. Analyses were adjusted for the following variables: sample size, year of publication, type of antidepressant (SSRI versus any other), antidepressant dose (PDD/DDD), probability of receiving placebo (high versus low), outpatients (yes = 1, no = 0), elderly patients (yes = 1, no = 0), severity at baseline (yes = 1, no = 0) (S1 Table).

As second analytical step, the Baron and Kenny mediational model was applied to investigate whether adverse events mediated the relationship between antidepressant treatment and efficacy [13,14]. These analyses were performed at the study arm level. Therefore, in case of multi-arm trials, each arm was considered separately. We reshaped the database from a wide format, where rows identified comparisons, to a long format, where rows identified treatment arms (S2 Table). This allowed us to define the three variables of the Baron and Kenny approach: antidepressants treatment (predictor variable); responders to treatment (outcome variable); individuals with adverse events (mediator variable). According to Baron and Kenny, a mediating role of a variable exists when four conditions are met: (i) the predictor variable must be significantly related to the outcome variable; (ii) the hypothesized mediator must be significantly related to the predictor variable; (iii) the mediator must be significantly related to the outcome; and (iv) the relationship between the predictor and the outcome must be attenuated when controlling for the mediator [13]. When predictor remains significant when the mediator is controlled for, mediation is deemed to be partial. When controlling for the mediator renders the relationship between predictor and outcome non-significant, mediation is deemed complete.

A graphical representation of the model applied to antidepressant trials is presented in Fig 1.

thumbnail
Fig 1. Baron and Kenny mediational model.

The model shows both the direct and the mediated pathways by which antidepressants treatment influences efficacy. The mediated pathway investigates the potential mediational role of adverse events in the relationship between antidepressant treatment and the number of responders at endpoint.

https://doi.org/10.1371/journal.pone.0178617.g001

In model 1 we tested the relationship between antidepressant treatment and treatment response. In model 2 we tested the relationship between antidepressant treatment and adverse events. In model 3 we tested the relationship between adverse events and treatment response. In model 4 we tested the effect of antidepressant treatment and adverse effects on treatment response. Statistical significance was set at p < 0.05. Mediation was assessed by changes in statistical significance and in the magnitude of correlation coefficients (ß coefficient) between the pathways 1 and 4. To adjust for possible confounding, the four regression analyses included the following variables: sample size, year of publication, probability of receiving placebo, baseline severity, outpatients, elderly patients. A nonparametric bootstrap method of statistical accuracy was used in the four models of linear regression analyses, assuming that the observed distribution of the present sample was a good estimate of the true population distribution [15]. All calculations were performed with Stata13 (STATA Corp, College Station, TX, USA).

Results

Characteristics of included studies

Of the original 41 randomised controlled trials included in the Cochrane review, 11 were excluded because they did not report data on treatment response, and other 16 were excluded because they did not report the number of subjects experiencing adverse effects, leaving 14 studies for the present analysis [1629]. The study selection process is shown in Fig 2.

Overall, 18 comparisons and 38 treatment arms were considered. The following compounds were investigated in the included studies: imipramine (one study), paroxetine (5 studies), sertraline (3 studies), fluoxetine (one study), fluvoxamine (2 studies), citalopram (one study), escitalopram (one study) and venlafaxine (4 studies). The main characteristics of included studies are shown in Table 1. All studies were double-blind.

thumbnail
Table 1. Characteristics of included randomised controlled trials comparing antidepressants with placebo.

https://doi.org/10.1371/journal.pone.0178617.t001

Univariate and multivariable meta-regression analysis

Fig 3 shows that in univariate analysis the RR for efficacy was not associated with the frequency of antidepressant (Spearman rho -0.003, p = 0.990) and placebo (Spearman rho -0.028, p = 0.904) adverse events.

thumbnail
Fig 3. Adverse events and response.

Risk >1 favours antidepressants over placebo.

https://doi.org/10.1371/journal.pone.0178617.g003

Multivariable meta-regression analysis showed that, after adjustment for potential confounding variables, the RR for efficacy was not associated with the frequency of antidepressant (coefficient -1.752, 95% CI -4.820 to 1.316, p = 0.229) and placebo (coefficient 1.401, 95% CI -1.656 to 4.459, p = 0.327) adverse events (Table 2).

Mediational analyses

Table 3 presents the results of the Baron and Kenny approach. Model 1 tested the relationship between antidepressants treatment and efficacy measured as treatment response. In this model, antidepressants treatment was significantly associated with better treatment response (ß = 0.127, 95% CI 0.04 to 0.21, p = 0.003).

Model 2 tested the relationship between antidepressants treatment and adverse events. In this model, antidepressants treatment was not significantly associated with higher rates of subjects with adverse events (ß = 0.094, 95% CI -0.05 to 0.24, p = 0.221).

Model 3 tested the relationship between adverse events and treatment response. In this model, adverse events were inversely associated with treatment response (ß = -0.035, 95% CI -0.06 to -0.05, p = 0.022).

Finally, model 4 tested the combined effect of antidepressants treatment and adverse effects on treatment response. When adverse events were added to model 1, the relationship between antidepressants treatment and treatment response was still statistically significant (ß = 0.122, 95% CI = 0.01 to 0.23, p = 0.039), although the effect was slightly attenuated, as can be inferred from the magnitude of the coefficient.

Discussion

To our knowledge, this is the first study that formally investigated the potential role of adverse events as mediators of treatment effect in antidepressant clinical trials conducted in individuals with panic disorder. We did not find evidence that the effects of antidepressants are mediated by the perception of adverse events.

Our results do not confirm the findings of Greenberg and colleagues [1], who suggested that, in patients with depressive disorder, post-treatment efficacy ratings were inflated by the breaking of blind as a consequence of experiencing adverse events. However, the association between adverse events and improvement reported in the Greenberg et al. meta-analysis was based on four clinical trials only. Additionally, they did not carry out a mediational analysis. In our analysis the approach described by Baron and Kenny was applied, in order to disentangle the potential intermediate role of adverse events as mediators of the relationship between antidepressant treatment and efficacy in individuals with panic disorder. Our findings are consistent with a recent work carried out by Barth and colleagues, who investigated the possible mediational role of adverse events in explaining the efficacy of SSRIs in the treatment for major depression. They similarly found no evidence for a mediational role of adverse events [2].

Strengths and limitations

Strengths of this study include the following. Data were extracted from an ongoing Cochrane systematic review, which employed a comprehensive search without language restrictions, and included unpublished studies. Standard Cochrane methodology was applied throughout the steps of study selection, data handling and analysis [30]. Although we cannot rule out an even slight effect from unobserved covariates, another strength is that all analyses have been adjusted for a number of study level confounders.

However, a number of limitations should be acknowledged. First, it was impossible to investigate the breaking of blind directly, as the majority of the included clinical trials did not assess this aspect. A second limitation is that only 14 trials out of the 41 potentially relevant studies reported data on the number of participants experiencing adverse events. This is a major issue not only because the analysis might have been more powerful statistically, but also because it is difficult to speculate on whether lack of reporting happened by chance or rather it may reflect some kind of reporting bias. Another limitation is that the meta-regression analysis may suffer from low statistical power. Finally, study quality was not used as a confounding variable as the Cochrane risk of bias tool is considered a qualitative and not quantitative tool [30].

Study implications

The finding that the drug-placebo differences are not due to the breaking of blind, as there was no relationship with the perception of adverse events, is of paramount relevance clinically and methodologically. Clinically, it implies that the moderate therapeutic effect of antidepressants in individuals with panic disorder is not an artefact, therefore reflecting a genuine effect that doctors can expect to replicate under real-world conditions. Methodologically, it implies that double-blind placebo-controlled antidepressant trials are not biased by the differential emerging of adverse effects in those allocated to the active and control condition, at least in patients with panic disorders.

In terms of implications for research, we argue that strategies to check blindness in future clinical trials should always be employed. This could be achieved by using tools like the one developed by Even and colleagues, who devised a short seven-point checklist to evaluate the degree to which supposedly blind trials are protected from blindness penetration [31]. We also argue that regulatory authorities should compulsory require the routine use of such instruments in clinical trials conducted for regulatory purposes. In fact, any positive changes in terms of trial design, analysis and reporting are likely to be achieved only if they are a requirement of new legislation [32,33]. Similarly, journals editors should no longer accept for publication trials reports that do not include this information.

A different and more radical strategy to overcome this issue would be to avoid placebo-controlled antidepressant trials and require active-control trials in the evaluation of newer antidepressants. In comparative head-to-head trials it is expected that adverse events similarly occur in both treatment arms, with a substantially lower risk of breaking the blind. This seems reasonable in panic disorder, as effective drugs treatments are available in this condition to be used as reference standards [7,34,35]. Regulatory authorities may require active-control superiority clinical trials to generate evidence of superiority between competing treatments [36,33,32,37].

In summary, this analysis does not support the hypothesis that the perception of adverse events in placebo-controlled antidepressant clinical trials may induce patients to conclude that they have been randomized to the active arm of the trial, leading to the breaking of blind and to an artificial inflation of the efficacy measures.

Supporting information

S1 Table. Dataset antidepressants placebo for metaregression analysis.

https://doi.org/10.1371/journal.pone.0178617.s001

(XLS)

S2 Table. Dataset antidepressants placebo for mediational analysis.

https://doi.org/10.1371/journal.pone.0178617.s002

(XLS)

Author Contributions

  1. Conceptualization: IB AB CB.
  2. Data curation: IB AB CB.
  3. Formal analysis: CB.
  4. Methodology: IB CB.
  5. Project administration: IB.
  6. Visualization: IB CB.
  7. Writing – original draft: IB.
  8. Writing – review & editing: IB AB CB.

References

  1. 1. Greenberg RP, Bornstein RF, Zborowski MJ, Fisher S, Greenberg MD (1994) A meta-analysis of fluoxetine outcome in the treatment of depression. The Journal of nervous and mental disease 182 (10): 547–551. pmid:7931201
  2. 2. Barth M, Kriston L, Klostermann S, Barbui C, Cipriani A, Linde K (2016) Efficacy of selective serotonin reuptake inhibitors and adverse events: meta-regression and mediation analysis of placebo-controlled trials. The British journal of psychiatry: the journal of mental science 208 (2): 114–119.
  3. 3. Rabkin JG, Stewart JW, McGrath PJ, Markowitz JS, Harrison W, Quitkin FM (1987) Baseline characteristics of 10-day placebo washout responders in antidepressant trials. Psychiatry research 21 (1): 9–22. pmid:3299437
  4. 4. White K, Kando J, Park T, Waternaux C, Brown WA (1992) Side effects and the "blindability" of clinical drug trials. The American journal of psychiatry 149 (12): 1730–1731. pmid:1443253
  5. 5. Bakker A, Spinhoven P, van Balkom AJ, Matser D, van Dyck R (1999) Double-blindness procedure did not mask giving of medication in panic disorder. Journal of affective disorders 54 (1–2): 189–192. pmid:10403163
  6. 6. Piasecki MP, Antonuccio DO, Steinagel GM, Kohlenberg BS, Kapadar K (2002) Penetrating the blind in a study of an SSRI. Journal of behavior therapy and experimental psychiatry 33 (2): 67–71. pmid:12472171
  7. 7. Guaiana G, Barbui C, Chiodo D, Cipriani A, Davies SJC, Koesters M. Antidepressants versus placebo for panic disorder in adults. Cochrane Database of Systematic Reviews 2013 (7).
  8. 8. American Psychiatric Association (1994) Diagnostic and Statistical Manual of Mental Disorders (DSM-IV).
  9. 9. World Health Organization (1992) The Tenth Revision of the International Classification of Diseases and Related Health Problems (ICD-10).
  10. 10. Papakostas GI, Fava M (2009) Does the probability of receiving placebo influence clinical trial outcome? A meta-regression of double-blind, randomized clinical trials in MDD. European neuropsychopharmacology: the journal of the European College of Neuropsychopharmacology 19 (1): 34–40.
  11. 11. Nose M, Barbui C (2008) A simple approach to manage dosages in drug-epidemiology research. Epidemiologia e psichiatria sociale 17 (3): 186–187. pmid:18924556
  12. 12. World Health Organization (2016) DDD Definition and general considerations.
  13. 13. Baron RM, Kenny DA (1986) The moderator-mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. Journal of personality and social psychology 51 (6): 1173–1182. pmid:3806354
  14. 14. Kraemer HC, Wilson GT, Fairburn CG, Agras WS (2002) Mediators and moderators of treatment effects in randomized clinical trials. Archives of general psychiatry 59 (10): 877–883. pmid:12365874
  15. 15. Efron B (2011) The bootstrap and Markov-chain Monte Carlo. Journal of biopharmaceutical statistics 21 (6): 1052–1062. pmid:22023675
  16. 16. Asnis GM, Hameedi FA, Goddard AW, Potkin SG, Black D, Jameel M et al. (2001) Fluvoxamine in the treatment of panic disorder: a multi-center, double-blind, placebo-controlled study in outpatients. Psychiatry research 103 (1): 1–14.
  17. 17. Bradwejn J, Ahokas A, Stein DJ, Salinas E, Emilien G, Whitaker T et al. (2005) Venlafaxine extended-release capsules in panic disorder: flexible-dose, double-blind, placebo-controlled study. The British journal of psychiatry: the journal of mental science 187: 352–359.
  18. 18. GlaxoSmithKline (1994) A double-blind, multicentered, flexible-dose study of paroxetine, alprazolam A double-blind, multicentered, flexible-dose study of paroxetine, alprazolam and placebo in the treatment of panic disorder. GSK Clinical Study Register.
  19. 19. Liebowitz MR, Asnis G, Mangano R, Tzanis E (2009) A double-blind, placebo-controlled, parallel-group, flexible-dose study of venlafaxine extended release capsules in adult outpatients with panic disorder. The Journal of clinical psychiatry 70 (4): 550–561. pmid:19358784
  20. 20. Londborg PD, Wolkow R, Smith WT, DuBoff E, England D, Ferguson J et al. (1998) Sertraline in the treatment of panic disorder. A multi-site, double-blind, placebo-controlled, fixed-dose investigation. The British journal of psychiatry: the journal of mental science 173: 54–60.
  21. 21. Nair NP, Bakish D, Saxena B, Amin M, Schwartz G, West TE et al. (1996) Comparison of fluvoxamine, imipramine, and placebo in the treatment of outpatients with panic disorder. Anxiety 2 (4): 192–198. pmid:9160622
  22. 22. Pollack MH, Otto MW, Worthington JJ, Manfro GG, Wolkow R (1998) Sertraline in the treatment of panic disorder: a flexible-dose multicenter trial. Archives of general psychiatry 55 (11): 1010–1016. pmid:9819070
  23. 23. Pollack M, Mangano R, Entsuah R, Tzanis E, Simon NM, Zhang Y et al. (2007) A randomized controlled trial of venlafaxine ER and paroxetine in the treatment of outpatients with panic disorder. Psychopharmacology 194 (2): 233–242. pmid:17589833
  24. 24. Pollack MH, Lepola U, Koponen H, Simon NM, Worthington JJ, Emilien J et al. (2007) A double-blind study of the efficacy of venlafaxine extended-release, paroxetine, and placebo in the treatment of panic disorder. Depression and anxiety 24 (1): 1–14. pmid:16894619
  25. 25. Stahl SM, Gergel I, Li D (2003) Escitalopram in the treatment of panic disorder: a randomized, double-blind, placebo-controlled trial. The Journal of clinical psychiatry 64 (11): 1322–1327. pmid:14658946
  26. 26. Tsutsui S, Osada H, Muranaka M, Namiki M, Katsura T, Kamijima K et al. (1997) Clinical evaluation of sertraline hydrochloride, a selective serotonin reuptake inhibitor for panic disorder: a double-blind, placebo-controlled trial. Journal of Neuropsychopharmacology (19): 639–657.
  27. 27. Tsutsui S. (2000a) Clinical evaluation of paroxetine Hcl, a selective serotonin reuptake inhibitor, in the treatment of panic disorder: late phase II double-blind, parallel group study. Japanese Pharmacology and Therapeutics (28): S271–294.
  28. 28. Tsutsui S. (2000) Clinical evaluation of paroxetine Hcl, a selective serotonin reuptake inhibitor, in the treatment of panic disorder: phase III double-blind, parallel group study. Japanese Pharmacology and Therapeutics: S295–314.
  29. 29. Michelson D, Allgulander C, Dantendorfer K, Knezevic A, Maierhofer D, Micev V et al. (2001) Efficacy of usual antidepressant dosing regimens of fluoxetine in panic disorder: randomised, placebo-controlled trial. The British journal of psychiatry: the journal of mental science 179: 514–518.
  30. 30. Higgins JPT GS, editor (2011) Cochrane Handbook for Systematic Reviews of Interventions. Chichester: John Wiley & Sons.
  31. 31. Even C, Siobud-Dorocant E, Dardennes RM (2000) Critical approach to antidepressant trials. Blindness protection is necessary, feasible and measurable. The British journal of psychiatry: the journal of mental science 177: 47–51.
  32. 32. Barbui C, Bighelli I (2013) A new approach to psychiatric drug approval in Europe. PLoS medicine 10 (10): e1001530. pmid:24143137
  33. 33. Garattini S, Chalmers I (2009) Patients and the public deserve big changes in evaluation of drugs. BMJ (Clinical research ed.) 338: b1025.
  34. 34. Bighelli I, Trespidi C, Castellazzi M, Cipriani A, Furukawa TA, Girlanda F et al. (2016) Antidepressants and benzodiazepines for panic disorder in adults. The Cochrane database of systematic reviews 9: CD011567. pmid:27618521
  35. 35. Imai H, Tajika A, Chen P, Pompoli A, Guaiana G, Castellazzi M et al. (2014) Azapirones versus placebo for panic disorder in adults. The Cochrane database of systematic reviews (9): CD010828. pmid:25268297
  36. 36. Bighelli I, Barbui C (2012) What is the European Medicines Agency. Epidemiology and psychiatric sciences 21 (3): 245–247. pmid:22793154
  37. 37. (2010) Reflection paper on the need for active control in therapeutic areas where use of placebo is deemed ethical and one or more established medicines are available. EMA.