Deep brain stimulation (DBS) of the ventral capsule/ventral striatum (VC/VS) region has shown promise as a neurosurgical intervention for adults with severe treatment-refractory obsessive-compulsive disorder (OCD). Pilot studies have revealed improvement in obsessive-compulsive symptoms and secondary outcomes following DBS. We sought to establish the long-term safety and effectiveness of DBS of the VC/VS for adults with OCD.
Materials and Methods
A long term follow-up study (73–112 months) was conducted on the six patients who were enrolled in the original National Institute of Mental Health pilot study of DBS for OCD. Qualitative and quantitative data were collected.
Reduction in OCD symptoms mirrored the one-year follow-up data. The same four participants who were treatment responders after one year of treatment showed a consistent OCD response (greater than 35% reduction in Yale Brown Obsessive Compulsive Scale (YBOCS)). Another subject, classified as a non-responder, achieved a 26% reduction in YBOCS score at long term follow-up. The only patient who did not achieve a 25% or greater reduction in YBOCS was no longer receiving active DBS treatment. Secondary outcomes generally matched the one-year follow-up with the exception of depressive symptoms, which significantly increased over the follow-up period. Qualitative feedback indicated that DBS was well tolerated by the subjects.
Citation: Fayad SM, Guzick AG, Reid AM, Mason DM, Bertone A, Foote KD, et al. (2016) Six-Nine Year Follow-Up of Deep Brain Stimulation for Obsessive-Compulsive Disorder. PLoS ONE 11(12): e0167875. doi:10.1371/journal.pone.0167875
Editor: Krystof Bankiewicz, University of California San Francisco Department of Neurological Surgery, UNITED STATES
Received: January 12, 2016; Accepted: October 18, 2016; Published: December 8, 2016
Copyright: © 2016 Fayad et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All quantitative data are available at the following url: http://figshare.com/articles/OCD_DBS_six_nine_year_follow_up/1614926.
Funding: The follow-up study was unfunded; the initial pilot study was supported by the following grant: National Institutes of Health (R21MH064161); ClinicalTrials.gov Identifier: NCT00057603.
Competing interests: Dr. Sarah Fayad is on the scientific advisory board of Acadia Pharmaceuticals. Mr. Andrew Guzick, Mr. Adam Reid, Mrs. Dana Mason, Ms. Agustina Bertone, Dr. Kelly Foote and Dr. Herbert Ward have nothing to disclose. Dr. Michael Okun serves as a consultant for the National Parkinson Foundation, and has received research grants from NIH, NPF, the Michael J. Fox Foundation, the Parkinson Alliance, Smallwood Foundation, the Bachmann-Strauss Foundation, the Tourette Syndrome Association, and the UF Foundation. Dr. Okun has previously received honoraria, but in the past >60 months has received no support from industry. Dr. Okun has received royalties for publications with Demos, Manson, Amazon, Smashwords, Books4Patients, and Cambridge (movement disorders books). Dr. Okun is an associate editor for New England Journal of Medicine Journal Watch Neurology. Dr. Okun has participated in CME and educational activities on movement disorders (in the last 36) months sponsored by PeerView, Prime, Quantia, Henry Stewart, and by Vanderbilt University. The institution and not Dr. Okun receives grants from Medtronic, Abbvie, and ANS/St. Jude, and the PI has no financial interest in these grants. Dr. Okun has participated as a site PI and/or co-I for several NIH, foundation, and industry sponsored trials over the years but has not received honoraria. Dr. Wayne Goodman has participated as a site PI and/or co-I for several NIH sponsored trials, including the R21 that funded this pilot DBS trial. Related to this grant, he received travel support and a donation of DBS devices from Medtronic. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
Obsessive-compulsive disorder (OCD) is a chronic and often disabling condition that is more common than previously believed, with an estimated lifetime prevalence of 2.3%.[1–3] The World Health Organization identifies OCD in the top ten most disabling medical conditions, as the syndrome is associated with lost income and decreased quality of life. It is characterized by recurrent, unwanted, intrusive and persistent thoughts, images or impulses (obsessions) and repetitive behaviors or rituals (compulsions). Although potent serotonin reuptake inhibitors (SRIs) are often effective in the treatment of OCD, they have a slow onset of action, requiring 8–10 weeks of treatment to achieve significant reduction in symptoms, and may not be helpful in the most severe OCD cases. Psychotherapeutic interventions have also shown promise, with cognitive behavioral therapy with exposure and response prevention (CBT-ERP) emerging as the most evidence-based psychosocial treatment. During CBT-ERP, patients are challenged to confront feared stimuli without engaging in compulsive behaviors.
Despite recent advances in pharmacological and psychological treatments, some authors have suggested that as many as 20–30% of patients with OCD may remain severely disabled and could be considered candidates for deep brain stimulation (DBS).[8,9] DBS was first described as a potential intervention for debilitating, treatment resistant OCD in a study by Nuttin and colleagues. From 2003 to 2007, researchers at the University of Florida conducted a pilot study funded by the National Institute of Mental Health (NIMH) that examined DBS in six patients with treatment-resistant OCD. This study had a staggered and blinded onset, and observed that four out of six patients were responders and manifested a significant reduction in OCD symptoms with a greater than 35% reduction in Yale Brown Obsessive Compulsive Score (YBOCS) after 12 months of continuous DBS. Given the paucity of long term data on the safety and effectiveness of this procedure in this patient population, we conducted a follow-up investigation on the subjects involved in the original pilot study to assess the long term safety, effectiveness and tolerability of OCD DBS.
Only three other published studies have examined the long term effects of OCD DBS. The initial study by Nuttin and colleagues found that three out of six patients showed significant improvement in obsessive-compulsive symptoms after twenty-one months of anterior limb of the internal capsule region stimulation as evidenced by a YBOCS reduction of 35% or more. Ooms and colleagues assessed quality of life (QOL) and OCD symptom reduction in sixteen patients who had received DBS of the nucleus accumbens for OCD, and found that three to five years after surgery, their cohort experienced improvements in quality of life, OCD symptoms, anxiety, and depressive symptoms. After eight months of treatment, nine of sixteen were classified as responders, while at three to five year follow-up, two additional patients were considered responders using the same 35% reduction in YBOCS criterion. Greenberg and colleagues found in a three year follow-up that DBS of the ventral capsule/ventral striatum (VC/VS) was associated with long-term improvement in obsessive-compulsive symptoms and overall functioning; six out of eight patients demonstrated a 25% or greater reduction in obsessive-compulsive severity, while four out of eight achieved a reduction in YBOCS of at least 35%.
To date, however, no other longitudinal data has been reported that examines effectiveness and safety outcomes beyond three years post-surgery. We present data on long term safety, effectiveness and tolerability of VC/VS DBS in the six patients who were enrolled in a NIMH funded pilot study.
Materials and Methods
Subjects included two males and four females who were initially implanted between October 2003 and January 2007 during a pilot study at the University of Florida (PI: Goodman). The mean age of the six subjects during the follow-up assessment was 44.5 years (range: 33–61).
This study was approved by the University of Florida Gainesville Health Science Center Institutional Review Board. All of the interviews were conducted face-to-face. Two interviews were conducted in person at the University of Florida, and four of these assessments were conducted through video-teleconferencing from the University of Florida to remote locations. All subjects signed informed consents prior to each interview. Assessments and psychometric rating scales were administered to all subjects and information was recorded. Medication and DBS settings at follow-up were obtained through medical records. One subject did not provide consent for medical records and therefore current DBS settings are unknown, though current medications were made available through the interview. The subjects each were given psychometric rating scales, including the YBOCS, the Hamilton Rating Scale for Depression (HAM-D), the Short Form Health Survey (SF-36), and the Montreal Cognitive Assessment (MoCA).
Descriptive data on the variables of interest are provided throughout the results. An intent-to-treat analysis was conducted using multi-level modeling (MLM) due to its ability to model change in data over time with lower type 1 error than more traditional repeated measures techniques, such as repeated-measures analysis of variance.[19–21] While no power analyses are available for MLM, a general guideline is a sample of 10–30 participants with approximately 5 repeated measures.[22, 23] However, lower sample sizes are adequate when the number of repeated measures is increased,[22, 23] such as in this study in which 32 YBOCS measures were collected across time in 6 patients. The results described below test the fit of a linear growth model to the outcome variable of interest to statistically examine if an increase or decrease in severity was observed. An estimate of the average change during the follow-up period is also provided.
Obsessive-Compulsive Symptom Severity
A MLM analysis was fitted for the available monthly data of the primary outcome of the YBOCS during the 114 months of follow-up. A significant linear reduction in symptoms was observed for the group (F (1, 162.997) = 15.488, p < .001) with an average decrease of 18.5 points on the YBOCS (SD = 14.488) during the follow-up period. The course of obsessive-compulsive severity in the six subjects is displayed in Fig 1. S1, S3, S5, and S6 were all classified as “responders,” or those who experienced at least 35% reduction in their YBOCS scores at the end of follow-up period, while S2 and S4 were classified as “non-responders.” Visual inspection indicates that overall symptoms remitted, although notable fluctuations in symptom severity were observed. Categorical analyses of the data can be observed in Table 1. Echoing the results from the one-year follow-up, the same four participants met criteria for response (≥35% reduction in YBOCS symptoms and YBOCS score ≤ 16). One of the two non-responders experienced a 26% reduction in symptoms, highlighting that with the exception of the subject who was not receiving active DBS, all subjects experienced symptom reduction of at least 25% at long-term follow-up. While four of the subjects had YBOCS scores ≤ 16 at the final follow-up assessment, this did not necessarily indicate continuous symptom remission throughout time, as different numbers of subjects met response criteria at different time points.
Secondary Qualitative and Quantitative Data
A MLM analysis was fitted for the available monthly data of the secondary outcome of the HAM-D during the 114 months of follow-up. A significant linear increase in symptoms was observed for the group (F (1, 153.267) = 8.637, p = .004) with an average increase of 5.833 points (SD = 11.548) during the follow-up period.
The course of depressive symptom severity in the six subjects is displayed in Fig 2. At the end of the follow-up period, S2 and S3 finished the follow-up period classified as moderately depressed (HAM-D scores between 17 and 23) and S4 and S5 were classified as severely depressed (HAM-D≥23) per the cutoff scores recommended by Zimmerman and colleagues (2013). Similar to YBOCS scores, visual inspection of this figure suggests frequent fluctuation of depressive symptoms through the course of the follow-up period. As described above, depressive symptoms increased across the follow-up period.
HAM-D = Hamilton Rating Scale for Depression. To improve the clarity of the graph, only data collected at 12 months, 24 months, and at long-term follow-up were included in the figure.
None of the participants began treatment with HAM-D scores above the clinical depression cut-off (score ≥ 17) but four of the six subjects scored in this range at the conclusion of the follow-up period. Both of the DBS non-responders concluded the follow-up period with HAM-D scores above the clinical depression cut-off. See Table 2 for a categorical summary of depressive symptom severity through the course of follow-up.
Quality of Life/Functional Impairment.
A MLM analysis was fitted for the available monthly data assessing quality of life during the 114 months of follow-up, as measured by the SF-36. The Social Functioning subscale (F (1, 49.336) = 12.191, p = .001) significantly increased (improved) during the follow-up period, while the General Health subscale (F (1, 48.873) = 3.703, p = .060) showed improvement trending towards significance. However, the Bodily Pain subscale (F (1, 48.010) = 2.980, p = .091) showed a decrease trending towards significance. The Physical Functioning, Role-Physical, Vitality, Role-Emotional, and Mental Health subscales did not significantly increase or decrease over the follow-up period.
At the time of the follow-up interview, four subjects were married, compared with five at the beginning of the study. Two were unemployed, two reported working part-time, and two said that they had full-time jobs, compared with three who were unemployed and three who worked part-time at the beginning of the study.
Echoing findings from the original 6 and 12 month follow-ups, there was no indication of neuropsychological impairment when the MoCA was administered at the end of the follow-up period. Specifically, all subjects scored at least 27 on the assessment, exceeding the clinical screening score of 26.
DBS Lead Locations and Programming.
See Table 3 for a summary of the DBS contacts used for chronic stimulation and the programming settings. Four of six subjects had different settings at 12-month, 30-month, and long-term follow-up. One subject was no longer receiving active DBS and the other subject did not provide consent to obtain medical records for current DBS settings.
Three of the subjects (50%) were taking fewer psychotropic medications at long-term follow-up compared with the baseline visit in the pilot study. Two subjects were taking the same number, and one subject went from taking one psychotropic medication to two. See Table 4 for a summary of psychotropic medications at baseline, 30-month follow-up and long-term follow-up.
Only one serious adverse event was reported. One subject was admitted to an inpatient psychiatric facility following a DBS settings adjustment within the first year post-implantation for worsening OCD. Once the settings were appropriately readjusted the patient improved and was discharged.
Four subjects endorsed sleep disturbances, although the specific nature of the problem varied from patient to patient. One reported insomnia on the night of implanted pulse generator replacement, another reported difficulty when voltage was adjusted and too high, and another reported sleeping too much and a recurrence of OCD symptoms when the IPG was depleted. The final patient reported general insomnia unrelated to DBS. Other anecdotal adverse events include shooting tingling on the right side of body, metal taste and jaw clenching, and mild skin redness and burning on head and chest. All these adverse events were episodic, with the exception of one patient who reported slightly increased fatigue following DBS.
The five subjects whose YBOCS scores improved at least 25% subjectively reported that their symptoms had improved since starting DBS. The four responders all reported that they would elect to receive DBS again if given the choice. The subject who experienced a 26% improvement stated they would likely not choose DBS again due to discomfort of the stereotactic device during implantation and due to their desire for greater benefit in terms of obsessive-compulsive symptom relief. The patient whose DBS was turned off shortly after the beginning of the study stated they would choose DBS again, depending on the cost of the procedure.
During the interview, subjects were given an opportunity to voice any negative experiences to receiving DBS. There were a few notable complaints. One subject wished his/her other physicians knew more about DBS, and felt they may have been treated differently by medical professionals due to their interest with the procedure. Another subject reported feeling self-conscious about surgery-related scars and battery bulge, while another was surprised that there were no cosmetic issues with DBS. One subject noted that finding optimal programming settings for DBS can be problematic and expensive.
Finally, we questioned subjects about the perceived benefits and drawbacks of rechargeable IPGs versus non-rechargeable IPGs. The three patients who received rechargeable IPG’s expressed a preference for them, as frequent surgeries are necessary with non-rechargeable batteries (every 6–12 months with non-rechargeable vs. every 7–8 years with rechargeable). Patients noted the limitations of the rechargeable batteries as well, specifically that they had to be recharged for about 30 minutes per day. One patient believed that the rechargeable batteries were slightly less effective, but the lack of frequent surgery made up for any quality of life deficits.
There has been very little long term data available regarding safety and effectiveness of OCD DBS. This long term follow-up provides valuable information supporting both safety and effectiveness in this patient population over time (up to nine years and four months after implantation of DBS). There was significant reduction in OCD symptoms (p < 0.001) across this pilot group, with an average of an 18 point reduction in YBOCS scores. Four out of six subjects met criteria for response (greater than 35% reduction in YBOCS), with an additional subject having a reduction of 26% in YBOCS score. These results were very similar to previous data published by Greenberg and colleagues in which six of eight subjects achieved a 25% or greater reduction in YBOCS compared to baseline. The only subject who did not demonstrate reduction in YBOCS score had the device turned off. Across the entire follow-up period, there was only one serious adverse event reported, which occurred during the first 12 months (i.e., psychiatric hospitalization due to a DBS setting adjustment that resolved with a programming session). Other side effects were minor, usually related to setting or battery changes, and easily remediated with adjustments in settings. The safety data support OCD DBS as a reasonably safe intervention.
While there was a significant reduction in obsessive compulsive symptoms over time among the majority of patients, there were notable fluctuations in severity of both OCD and depression over time (Figs 1 and 2). These fluctuations did not correspond with battery depletion or changes. It was unclear what led to fluctuations in symptoms other than daily life stressors. It should be noted that fluctuations in obsessive-compulsive and depressive symptoms have been well-documented in naturalistic follow-up studies.[25, 26]
One notable finding in this study was that there was a significant increase in depressive symptoms (p < 0.01), with an average increase of approximately six points on the HAM-D rating scale. At baseline, only two of six subjects met criteria for a moderate depression (Table 2), with three subjects meeting criteria for mild depression and one with no depression. At the end of the long term study, two patients met criteria for moderate depression and two met criteria for severe depression, while two had no depressive symptoms based on the HAM-D. There have been subsequent DBS studies published addressing major depressive disorder. DBS was placed in the same anatomical region as our study, partially based the target selection and response to comorbid depression in OCD. These studies showed initial improvement in depression, but larger recent studies have not revealed a benefit. These findings collectively suggest that this area of the brain may not be optimal for long term treatment resistant depression (without OCD), though there have been limitations to interpretation of these data including a small sample size and a lack of knowledge on the optimal targets and settings for depression DBS. Medication adjustments as would be expected, took place over the course of the follow-up of our patients (see Table 4). However, in the two patients who were not clinically depressed at long term follow-up, one had switched from a benzodiazepine to low dose duloxetine, and the other had a significant reduction in the number medications prescribed. Of the four subjects who experienced a worsening in depression over the course of the study, only one had an overall reduction in antidepressant medication, while the other subjects’ medication regimens stayed the same, were augmented, or were increased. Thus, it did not appear that medication reduction led to the increase in depressive symptoms. Over a six to nine year follow-up, any number of stressful life circumstances may have exacerbated depressive symptoms, especially in a cohort at high risk for developing mood disorders.
In terms of disability, social functioning improved significantly, general health showed slight improvement and bodily pain worsened. None of the subjects reported chronic pain as a result of the DBS at the follow-up interview, therefore we propose the increase in self-reported bodily pain may be tied to normal aging or to elevated depressive symptomology. The remainder of subscales, including physical functioning, role-physical, vitality, role-emotional and mental health subscales did not show significant changes over the long term. There were no negative cognitive sequelae.
From our qualitative data, all subjects but one indicated they would choose to do DBS again. The subject who indicated he would not elect to do DBS again related it to discomfort with the stereotactic device in surgery, despite a 26% reduction in OCD symptoms. Subjects who received rechargeable batteries preferred these to non-rechargeable batteries, citing improved quality of life.
There are several notable limitations to this long term follow-up study. It was not controlled and subjects had medication and DBS setting adjustments throughout the course of the long-term follow-up. It is therefore difficult to attribute all changes to DBS alone in this context. In addition, the sample size was overall quite small and larger numbers of subjects will be required to fully evaluate the long term effects of DBS of the VC/VS on OCD and comorbid depression.
VC/VS DBS was safe and conferred a long term benefit in reduction of OCD symptoms. Nearly all subjects receiving the therapy would elect to receive it again. Quality of life was improved quantitatively and qualitatively, though depressive symptoms significantly increased over the follow-up period. All subjects who received rechargeable IPG’s voiced a preference over traditional IPG’s citing the frequent need for replacement surgery. Patient selection is critically important for OCD DBS and should be conducted in expert interdisciplinary centers with experienced psychiatrists managing neuropsychiatric complications which can emerge post-DBS.
- Conceptualization: SMF HEW.
- Data curation: SMF AGG DMM AB.
- Formal analysis: SMF AGG AMR.
- Funding acquisition: KDF MSO WKJ.
- Investigation: SMF HEW.
- Methodology: SMF HEW.
- Resources: DMM.
- Supervision: HEW.
- Visualization: AGG AMR.
- Writing – original draft: SMF AGG AMR DMM AB.
- Writing – review & editing: KDF MSO WKG HEW.
- 1. Ruscio AM, Stein DJ, Chiu WT, Kessler RC. The epidemiology of obsessive-compulsive disorder in the National Comorbidity Survey Replication. Mol Psychiatry 2010 Aug 26;15:53–63. doi: 10.1038/mp.2008.94. pmid:18725912
- 2. Karno M, Golding JM, Sorenson SB, Burnam A. The epidemiology of obsessive-compulsive disorder in five US communities. Arch Gen Psychiatry 1988 Dec 1;45:1094–1099. pmid:3264144
- 3. Horwath E, Weissman MM. The epidemiology and cross-national presentation of obsessive-compulsive disorder. Psychiatr Clin North Am 2000 Sep 1;23:493–507. pmid:10986723
- 4. Bobes J, Gonzalez M P, Bascaran MT, Arango C, Saiz PA, Bousono M. Quality of life and disability in patients with obsessive-compulsive disorder. Eur Psychiatry 2001 Jun;16:239–245. pmid:11418275
- 5. Bloch MH, McGuire J, Landeros-Weisenberger A, Leckman JF, Pittenger C. Meta-analysis of the dose-response relationship of SSRI in obsessive-compulsive disorder. Mol Psychiatry 2009 May 26;15:850–855. doi: 10.1038/mp.2009.50. pmid:19468281
- 6. Rosa-Alcázar AI, Sánchez-Meca J, Gómez-Conesa A, Marin-Martinez F. Psychological treatment of obsessive-compulsive disorder: a meta-analysis. Clin Psychol Rev 2008 Dec;28:1310–1325. doi: 10.1016/j.cpr.2008.07.001. pmid:18701199
- 7. Abramowitz JS. Understanding and treating obsessive-compulsive disorder: A cognitive behavioral approach. Mahwah, New Jersey: Routledge; 2005.
- 8. Fins JJ, Mayberg HS, Nuttin B, Kubu CS, Galert T, Sturm V, et al. Misuse of the FDA's humanitarian device exemption in deep brain stimulation for obsessive-compulsive disorder. Health Aff 2011 Feb;30:302–311.
- 9. Goodman WK, Ward HE, Kablinger AS, Murphy TK. Biological Approaches to treatment-resistant obsessive compulsive disorder. In: Wk Goodman, Maser J, editors. Obsessive Compulsive Disorder: Contemporary Issues in Treatment. New York: Lawrence Erlbaum Associates; 2000. p. 333–369.
- 10. Nuttin B, Cosyns P, Demeulemeester H, Gybels J, Meyerson B. Electrical stimulation in anterior limbs of internal capsules in patients with obsessive-compulsive disorder. Lancet 1999 Oct 30;354:1526.
- 11. Goodman WK, Foote KD, Greenberg BD, Ricciuti N, Bauer R, Ward H, et al. Deep brain stimulation for intractable obsessive compulsive disorder: pilot study using a blinded, staggered-onset design. Biol Psychiatry. 2010 Mar 15;67:535–542. doi: 10.1016/j.biopsych.2009.11.028. pmid:20116047
- 12. Nuttin BJ, Gabriëls LA, Cosyns PR, Cosyns PR, Meyerson BA, Andreewitch S, Sunaert SG, et al. Long-term electrical capsular stimulation in patients with obsessive-compulsive disorder. Neurosurgery 2003 Jun;52:1263–1272. pmid:12762871
- 13. Ooms P, Mantione M, Figee M, Schuurman PR, van den Munckhof P, Denys D. Deep brain stimulation for obsessive-compulsive disorders: long-term analysis of quality of life. J Neurol Neurosurg Psychiatry 2013 May 28;85:153–158. doi: 10.1136/jnnp-2012-302550. pmid:23715912
- 14. Greenberg BD, Malone DA, Friehs GM, Rezai AR, Kubu CS, Malloy PF, et al.Three-year outcomes in deep brain stimulation for highly resistant obsessive-compulsive disorder. Neuropsychopharmacology 2006 Jul 19;31:2384–2393. doi: 10.1038/sj.npp.1301165. pmid:16855529
- 15. Goodman W K, Price LH, Rasmussen SA, Mazure C, Fleischmann R, Hill CL, et al. The Yale-Brown Obsessive Compulsive Scale. I. Development, use, and reliability. Arch Gen Psychiatry 1989 Nov 1;46:1006–1011. pmid:2684084
- 16. Hamilton M. A rating scale for depression. J Neurol Neurosurg Psychiatry 1960;23:56–62. pmid:14399272
- 17. Ware JE, Sherbourne CD. The MOS 36-item short-form health survey (SF-36). I. Conceptual framework and item selection. Med Care 1992;30:473–483. pmid:1593914
- 18. Nasreddine ZS, Phillips NA, Bédirian V, Charbonneau S, Whitehead V, Collin I, et al. The Montreal Cognitive Assessment, MoCA: a brief screening tool for mild cognitive impairment. J Am Geriatr Soc 2005 Mar 30;53:695–699. doi: 10.1111/j.1532-5415.2005.53221.x. pmid:15817019
- 19. Kahn JH, Schneider WJ. It's the destination and it's the journey: using multilevel modeling to assess patterns of change in psychotherapy. J Clin Psychol 2013 Jan 24;69:543–570. doi: 10.1002/jclp.21964. pmid:23348510
- 20. Gallop R, Tasca GA. Multilevel modeling of longitudinal data for psychotherapy researchers: II. The complexities. Psychother Res 2009 Sep 22;19:438–452. doi: 10.1080/10503300902849475. pmid:20183399
- 21. Tasca GA, Gallop R. Multilevel modeling of longitudinal data for psychotherapy researchers: I. The basics. Psychother Res 2009 Sep 22;19:429–437. doi: 10.1080/10503300802641444. pmid:19235088
- 22. Maas CJ, Hox JJ. Sufficient sample sizes for multilevel modeling. Methodology 2005;1:86–92.
- 23. Snijders T, Bosker R. Multilevel modeling: An introduction to basic and advanced multilevel modeling, 1st ed. London: Sage Publishers; 1999.
- 24. Zimmerman M, Martinez JH, Young D, Chelminski I, Dalrymple K. Severity classification on the Hamilton Depression Rating Scale. J Affect Disord 2013 Jun 4;150:384–388. doi: 10.1016/j.jad.2013.04.028. pmid:23759278
- 25. Rufer M, Grothusen A, Maß R, Peter H, Hand I. Temporal stability of symptom dimensions in adult patients with obsessive-compulsive disorder. J Affect Disord 2005 Jun 1;88:99–102. doi: 10.1016/j.jad.2005.06.003. pmid:16040125
- 26. Monroe SM, Harkness KL. Recurrence in major depression: a conceptual analysis. Psychol Rev 2011 Sep 5;118:655–674. doi: 10.1037/a0025190. pmid:21895384
- 27. Malone DA, Dougherty DD, Rezai AR, Carpenter LL, Friehs GM, Eskandar EN, et al. Deep brain stimulation of the ventral capsule/ventral striatum for treatment-resistant depression. Biol Psychiatry 2009 Aug 5;65:267–275. doi: 10.1016/j.biopsych.2008.08.029. pmid:18842257
- 28. Pine DS, Cohen P, Gurley D, Brooke J, Ma Y. The risk for early-adulthood anxiety and depressive disorders in adolescents with anxiety and depressive disorders. Arch Gen Psychiatry 1998;55:56–64. pmid:9435761