https://orcid.org/0000-0002-7944-7299
Figures
Abstract
Measurement-based care has demonstrable benefits, but significant implementation barriers slow dissemination in real-world clinical settings, especially youth behavioral health care. Here, we describe use of measurement-based care in a specialty clinic offering a continuum of outpatient care for suicidal youth. We characterize strategies used to facilitate measurement-based care in this population and ways in which challenges to implementation have been addressed. We examined adherence to measurement-based care procedures relative to treatment engagement data from electronic medical records, as well as data from clinicians regarding acceptability and utility of measurement-based care. Results suggest that measurement-based care is both feasible and acceptable for use with suicidal youth. Here we provide future directions in measurement-based care in this, and other, behavioral health settings.
Citation: Victor SE, Salk RH, Porta G, Hamilton E, Bero K, Poling K, et al. (2023) Measurement-based care for suicidal youth: Outcomes and recommendations from the Services for Teens At Risk (STAR) Center. PLoS ONE 18(4): e0284073. https://doi.org/10.1371/journal.pone.0284073
Editor: Elizabeth Reynolds, Johns Hopkins School of Medicine: Johns Hopkins University School of Medicine, UNITED STATES
Received: November 23, 2021; Accepted: March 23, 2023; Published: April 6, 2023
Copyright: © 2023 Victor et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Per the University of Pittsburgh IRB, the de-identified data set may be requested via an Honest Broker system (contact: Giovanna Porta MS, portagx@upmc.edu).
Funding: This research was supported by: 1) the Pennsylvania Legislature, which provides financial support for the STAR-Center and its electronic data collection and management (no grant number, no specific awardees); 2) the University of Pittsburgh, which provides financial support for Dr. David Brent’s Endowed Chair in Suicide Studies (no grant number, awardee: DB); and 3) the National Institutes of Health, which provided funding for research effort (NIMH T32 MH018269, awardees: TG, RS, SV; NIMH P50MH115838, awardees: DB, TG). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: We have read the journal’s policy and the authors of this manuscript have the following competing interests: Dr. Brent receives research support from the National Institutes of Health, American Foundation for Suicide Prevention, the Once Upon a Time Foundation, and The Beckwith Foundation; receives royalties from Guilford Press, from the electronic self-rated version of the C-SSRS from eRT, Inc., and from performing duties as an UptoDate Psychiatry Section Editor; receives consulting fees from Healthwise; receives Honoraria from the Klingenstein Third Generation Foundation for scientific board membership and grant reviews; and is a scientific board member for the American Foundation for Suicide Prevention. With respect to intellectual property (without financial interests), funding from the National Institute of Mental Health supported the development of intellectual property for BRITE, the As Safe As Possible intervention, the Computerized Adaptive Screen for Suicidal Youth (CASSY) measure, a suicide risk machine learning algorithm, and the Screening Wizard screening tool. Dr. Goldstein has received research support from the National Institutes of Health, American Foundation for Suicide Prevention, University of Pittsburgh Clinical and Translational Science Institute, and Brain and Behavior Foundation, as well as royalties from Guilford Press. Dr. Victor has received research support from the National Institutes of Health. Ms. Poling has received royalties from Guilford Press. Dr. Salk, Ms. Porta, Mr. Hamilton, and Ms. Bero report no financial or other relationships relevant to the subject of this article. None of the above listed agencies or organizations had a role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Interests related to the above listed agencies, organizations, and commercial funders do not alter our adherence to PLOS ONE policies on sharing data and materials.
Introduction
Measurement-based care (MBC) involves “the systematic use of symptom rating scales to drive clinical decision-making” [1] and is required for Joint Commission accreditation in behavioral healthcare [2]. MBC is associated with improved patient outcomes in mental healthcare settings [3], including improved response and remission rates in the treatment of adult depression [4, 5]. Similar, although sparser, data support improved patient outcomes when MBC is used with youth [6]. MBC improves identification of stalled progress or clinical deterioration among patients, facilitating needed changes in treatment to improve patient outcomes [7], and enhances patients’ engagement with treatment [8]. Further, MBC offers important feedback to clinicians about their own practices and demonstrates the efficacy of behavioral health services to third-party payers, such as health insurance companies [9].
MBC has been used in a variety of research and clinical settings to examine treatment efficacy and outcomes for depression among adults [4, 5, 10] and youth [11–13]. Relatively less work has examined MBC in the clinical management of suicidal thoughts and behaviors [14]. This gap is concerning, given the ethical and practical necessity of regular monitoring of suicide risk in clinical practice [15]. Suicidal thoughts and behaviors vary markedly over time among high-risk individuals [16], which necessitates frequent assessment to inform decisions about imminent risk management in the case of suicidal crises [17]. Thus, MBC with suicidal youth informs critical clinical decisions, such as communication with caregivers regarding safety and the potential need for psychiatric hospitalization [18]. The most well-known example of MBC in the treatment of suicidal patients is the use of “diary cards” in dialectical behavior therapy (DBT). DBT is a well-validated treatment for suicide and self-harm across diagnostic groups [19, 20]. Diary cards help monitor patient progress and inform treatment planning; data drawn from diary cards has high utility even outside of adherent DBT provision [21].
Although the literature supporting MBC in behavioral health broadly is substantial, there remain significant gaps in its study and implementation. First, much of the support for MBC is from adult treatment settings, such as Veterans Affairs centers [22] and academic medical centers participating in nationally organized consortia [14], or from adult depression research trials [4, 5]. Even in these contexts, research on MBC for suicide-related phenomena has been rare, and typically involves very infrequent administration of suicide assessments relative to what would be typical in MBC [14]. Research that has described MBC in adolescent mental health contexts has been primarily drawn from depression treatment trials, such as TORDIA [23, 24] among others [11, 13]. In some cases, work has been published describing platforms, tools, and/or strategies that could be used to facilitate youth MBC in naturalistic treatment settings [25–28], but without empirical data to characterize its implementation in specific treatment environments. Some limited data have been presented to support MBC with youth in a school-based mental health system [29], a partial hospitalization program for mood and anxiety disorders [30], and a first-episode psychosis program [31]; however, to our knowledge, no research has focused on MBC with suicidal youth in particular.
Thus, this paper characterizes the use of MBC at the Services for Teens At Risk (STAR) Center, a well-established program providing outpatient and intensive outpatient (IOP) mental health treatment to suicidal youth. MBC is a core component of regular clinical services at STAR-Center that informs individual patients’ care and overall program development. First, we provide an overview of STAR-Center’s patient population, treatment approach, and use of MBC, as well as how challenges to MBC implementation have been addressed. Second, we summarize clinician-report data on the feasibility and utility of MBC at STAR-Center, along with evidence of successful implementation of MBC based on de-identified patient data. Finally, we describe lessons learned through STAR-Center’s use of MBC and future directions of MBC with this population, with the goal of facilitating dissemination and implementation of MBC in diverse mental health treatment settings that provide care for suicidal youth.
What is STAR-Center?
STAR-Center was founded in 1986 as a suicide prevention program for youth [32]. Supported by the Commonwealth of Pennsylvania, Western Psychiatric Hospital, and the Department of Psychiatry at the University of Pittsburgh, STAR-Center provides assessment and intervention services to suicidal youth and young adults, as well as community outreach and suicide postvention services [32]. As the STAR-Center only expanded services to young adults in 2017, and MBC data were not available for analysis during the drafting of this manuscript, the remainder of the manuscript will focus specifically on describing services provided to youth through age 18.
STAR-Center patients complete an extensive intake assessment to determine appropriateness for STAR-Center services, which includes thorough suicide risk assessment and psychiatric diagnosis. Youth appropriate for STAR-Center services are then assigned to treatment along a continuum of care, beginning either traditional outpatient psychotherapy services (one hour/week) or IOP treatment (nine hours/week) depending on clinical severity, need, and patient preference. IOP treatment includes skills group and individual psychotherapy with master’s level clinicians and medication management with a psychiatrist and psychiatric nurse. Family sessions are conducted as clinically appropriate. Patients can move between levels of care, for instance, beginning with IOP services and stepping down to traditional outpatient psychotherapy when symptoms are less acute. STAR-Center also offers a monthly continuation/maintenance group for patients who no longer require weekly support. In some cases, patients complete the IOP portion of treatment and continue ongoing treatment with community providers outside of STAR-Center. Treatment at STAR-Center uses evidence-based practices, including DBT and cognitive behavior therapy modalities, informed by each patient’s case conceptualization [33].
Beyond clinical care, STAR-Center has a strong clinical-research infrastructure, which facilitates scientific research to improve our knowledge and treatment of suicidal youth. Further, STAR-Center provides extensive training support to mental health professionals and students seeking to develop competency and expertise in suicide risk assessment and management, including masters’ level clinician trainees, predoctoral clinical psychology interns, postdoctoral fellows, and medical students, interns, and residents.
Measurement-based care at STAR-Center
As part of standard clinical care, patients, their caregivers, and clinicians complete numerous assessments over the course of treatment (see Table 1 [34–53]). Patients and caregivers complete an extensive intake assessment, including clinician-administered interviews and validated self-report and informant-report questionnaires. The clinician-administered intake evaluation includes the Suicide Circumstances Schedule [54], sections of the Columbia Suicide Severity Rating Scale [55], the Kiddie Schedule for Affective Disorders and Schizophrenia Depression Rating Scale [56], the Clinical Global Impressions Scale [57], and the Child Global Assessment of Functioning [58]. A shorter battery of self-report (patient-rated) measures is administered at regular intervals during treatment (see below for specific intervals) using a HIPAA-compliant secure web-based survey system accessed on tablets to facilitate ease of data collection and scoring. Caregiver-rated (informant-report) measures are limited to the intake assessment battery given that measures are completed at the time of the session, and in our population, caregivers infrequently attend visits after the initial intake assessment. Patients enrolled in the IOP program complete MBC assessments weekly, as well as IOP diary cards for each day they attend the program; patients seen weekly or bi-weekly in the traditional outpatient program complete assessments monthly. The high acuity and symptom severity among youth in IOP, taken with the IOP treatment intensity, render week to week changes in MBC data relevant to reflect improvement or clinical worsening. These data are regularly reviewed to determine acute treatment response and related determinations regarding appropriate level of care and risk management. These data may also be used in the process of insurance authorizations for IOP level of care to demonstrate clinical need. For youth in outpatient (weekly to biweekly) level of care, monthly MBC data enable us to note trends and treatment response while also minimizing respondent burden. Clinicians treating outpatients have the capacity to initiate interim assessment as clinically indicated. Continuation/maintenance group is held on a monthly basis, and assessments are completed at each group session (i.e., monthly). Clinicians also complete rating scales assessing depression symptoms, clinical severity, clinical improvement, suicidality and self-injury, and medication changes throughout treatment; clinician ratings are generated regardless of patient self-report completion.
The entirety of STAR-Center’s treatment team (see below) meets weekly to review MBC data. Upon completion, patient MBC data are securely transmitted to an encrypted database, where assessments are scored and compiled with previously collected data. To facilitate comparability across measures with different scoring systems, results are scaled to a 0–100 interval based on percentage of the measure’s maximum score. Results for all constructs are then displayed using line graphs to examine changes in each symptom domain over time. Data are then used to examine patient progress, deterioration, and/or stagnation, and MBC-based treatment changes are made as needed (e.g., increased/decreased level of care, increased/decreased medication dosage, trial of alternative medication(s), transition to a different therapist or program). Data are also reviewed with the patient (and caregivers, as appropriate) on a regular basis in psychotherapy and medication management appointments.
Fig 1 shows a de-identified, annotated example of how data are displayed for treatment-team meetings and patient review. At the start of IOP, this patient reported very high levels of depression and anxiety, moderate sleep problems and affective lability, and minimal difficulties with substance use. During IOP treatment, their depression decreased significantly, but their anxiety, sleep problems, and affective lability remained relatively stable. The patient was prescribed 10 mg/day of escitalopram (first vertical arrow), at which point their anxiety and affective lability decreased. The patient’s depression symptoms remained low for several weeks, leading to discharge from IOP to weekly outpatient psychotherapy and medication management (second vertical arrow). The patient continued to improve, including resolution of their sleep difficulties, at which point they were discharged from weekly individual care to monthly continuation/maintenance group treatment (third vertical arrow). The patient remained well over the following several months of continuation/maintenance group.
Scores are scaled as a percentage of the maximum score for each measure (see Table 1). Lines indicate scores on the following scales: Mood: red (Short Mood and Feelings Questionnaire); Anxiety: green (Screen for Child Anxiety Related Disorders); Sleep: black (Pittsburgh Sleep Quality Index); Mood Lability: blue (Children’s Affective Lability Scale); Substance Use: purple (Car, Relax, Alone, Forget Friends, Trouble). The black box labeled “PDW” is an example of a “flag” based on responses to the ASQ at the most recent assessment, in this case indicating endorsement of passive death wishes.
Addressing challenges in measurement-based care
Several important challenges have been identified in the implementation of MBC, which have been addressed in a variety of ways at STAR-Center. MBC assessments, like all measures used in health care settings, must be both reliable and valid for their intended purpose and developmentally appropriate [59]. To decrease patient burden, MBC assessments must also be brief and easy to complete [60]. Finally, MBC assessments must be sensitive to change over time [61].
Thus, STAR-Center uses brief versions of well-validated measures appropriate for youth MBC. Each measure administered repeatedly ranges from 5 to 20 items, for a total completion time of 10–20 minutes per follow-up assessment; the intake battery takes approximately 25 minutes. Further, each measure has been validated with youth, and exhibits sensitivity to change over treatment (see Table 1).
Measures have been digitized for tablet-based access; patients typically complete measures via web-link in the waiting room prior to their appointment. Of note, our health system quickly transitioned to provision of service via telehealth in response to local, state, and national mandates during the COVID-19 pandemic. For all telehealth visits, the treating clinician sent a web-link via email or text message (based on family preference) separately to the youth (for intake and ongoing monitoring) and caregiver (for intake only) at the time of the scheduled session. The clinician would re-send the link if the assessment battery was not completed within 5 days of the completed session. Trends in MBC assessments occurring since the onset of the COVID-19 pandemic will be discussed in a separate manuscript.
Second, any MBC strategy requires significant buy-in from clinical staff and patients. Clinician burden and lack of investment is a significant obstacle to MBC in routine clinical care [62], and lack of patient buy-in contributes to missed assessments and limited utility of completed assessments in improving patient outcomes [63]. At STAR-Center, clinical staff have been involved throughout the process of MBC-related decision-making, for instance, in the choice of measures and the format for display of data. Clinicians complete brief surveys and participate in open discussions to gauge the utility of MBC assessments; two STAR-Center clinicians participate in twice monthly meetings to incorporate staff feedback into MBC-related changes. Second, staff clinicians receive ongoing education and training related to MBC during weekly meetings and MBC-specific trainings. Finally, senior clinical staff model the use of MBC data to make treatment decisions, as well as how to review and discuss MBC data with patients, during weekly treatment-team meetings. Similar strategies have been used at STAR-Center to enhance patient engagement. Patients are educated about MBC and its role in their treatment during their orientation to services at STAR-Center and emphasized repeatedly during treatment. Clinicians set a positive tone regarding MBC, for instance, by showing investment in patients’ understanding and completion of MBC measures throughout their treatment. Finally, when reviewing data with the patient, clinicians highlight connections between changes in treatment and changes in symptoms, demonstrating the value of MBC.
Third, timely, regular feedback given to both clinicians and patients is critical to MBC [64]. At STAR-Center, MBC is integrated into all documentation, case discussions, and case conceptualizations; MBC intake data are included in all patients’ initial assessment summaries, and clinicians are encouraged to include patient MBC assessment scores in weekly notes. Digitized MBC assessments ensure results are scored and analyzed in real-time, facilitating prompt review and response to patient-reported symptoms and/or suicide risk. Clinicians can view patients’ scores on self-report measures prior to meeting with the patient using a clinician dashboard, allowing them to use MBC data to structure subsequent patient encounters. Further, clinical staff are encouraged to share MBC results with patients as clinically appropriate; once patients are accustomed to the MBC system, they often ask clinicians to view their results during meetings. Patients are regularly shown their recent and historical data using line graphs, to demonstrate symptom changes over time in an easy-to-digest format.
Finally, managing suicide risk assessment and planning is a critical component of all psychiatric care, but is especially important at STAR-Center, where patients are specifically referred due to acute suicidality. Whenever a patient completes MBC assessments, the treating clinician is required to check responses regarding suicidality immediately, while the patient is still physically present at the clinic. Patients enrolled in IOP who report suicidal planning, a suicide attempt, or non-suicidal self-injury on their daily diary cards and/or weekly MBC assessment are “flagged” for further assessment by the lead IOP clinician; regardless of treatment modality, an email alert is sent automatically to the patient’s primary clinician and the clinical supervisor that details specific high-risk response(s).
Evidence of MBC feasibility and utility at STAR-Center
Two sets of data were analyzed to examine the implementation of MBC at STAR-Center. Data were entered and examined using SPSS version 25. First, clinical staff at STAR-Center (N = 17) completed anonymous surveys to gather information about perceived benefits and costs of MBC in routine clinical practice using a measure adapted from a similar survey assessing perceptions of MBC in physical therapy [65]. Participants were given a number of statements describing possible benefits to, or problems with, MBC, and were asked to respond as to whether they definitely agreed, agreed somewhat, or disagreed. The survey was approved for distribution through the University of Pittsburgh’s Quality Improvement office, and determined not to require Institutional Review Board approval.
Clinical staff at STAR-Center were primarily cisgender women (n = 13, 76.47%) between the ages of 25 and 49 (n = 12, 70.59%). All staff held masters (n = 11, 64.71%) or doctoral (n = 6, 35.29%) terminal degrees, and fields of study included psychiatry (n = 5, 29.41%), counseling (n = 4, 23.53%), psychology (n = 3, 17.65%), social work (n = 3, 17.65%), and nursing (n = 2, 11.76%). Of the clinicians who worked (or planned to work in the future) outside of STAR-Center (n = 14), a majority (n = 9, 64.29%) rated themselves as very likely or extremely likely to use MBC in another clinical setting. Clinicians were asked about the number of years using MBC, at STAR-Center or elsewhere; data were binned to ensure anonymity given potential for reidentification. Most clinicians indicating using MBC for 1 to 4 years (n = 6, 40%) or 5 to 10 years (n = 4, 26.67%), with fewer indicating less than 1 year (n = 1, 6.67%), 11 to 30 years (n = 2, 13.33%), or a qualitative description such as “long” (n = 2, 13.33%).
Clinicians were asked about potential benefits and problems related to the use of MBC at STAR-Center (see Table 2). All surveyed definitely or somewhat agreed that MBC helped to direct patients’ care, attain better patient outcomes, and help motivate and encourage patients; clinicians were divided with respect to whether MBC decreased rates of denial for care from third-party payers. The most commonly endorsed problems with MBC were that they take too much time for patients to complete, can be difficult for clinicians to interpret, and confusing to patients. Importantly, these perceptions are inclusive of all components of MBC used at STAR-Center, such as paper diary cards, even though these measures are not integrated into the Research Registry system used to compute estimates of MBC completion rates.
Additionally, we compared a representative sample of MBC data approved for research analysis with data from electronic medical records (EMR) to determine how regularly MBC assessments were gathered during routine clinical care, relative to MBC assessments expected based on actual clinic attendance. Patients at STAR-Center have the option to permit their MBC data to be used for research purposes; those who consent have their de-identified MBC data entered into a Research Registry. The University of Pittsburgh Institutional Review Board approved the study (IRB#9912102). Written consent was obtained from participants in the STAR-Center Research Registry. EMR data were extracted and de-identified by STAR-Center staff with appropriate clearance to access EMR, to facilitate cross-comparison with MBC data without compromising patient confidentiality. MBC and EMR data were examined for a seven-month period (1/1/2018-6/31/2018). As patients may have started treatment prior to, or continued treatment after, the selected period, data presented below likely underestimate the total MBC data collected on these patients. Rates of missing data from EMR are unavailable due to the nature of the dataset being examined. Raw data were not transformed on the basis of outliers or any other method. Sample size was determined by available data over the examined period and not based on a priori power analyses.
Given variations on the measures administered to specific patients (see Table 1), as well as the addition of some measures over time, MBC assessments were counted as present if at least one self-report measure was completed within the expected MBC battery for that time window. During the selected interval, 374 patients were seen at STAR-Center, and 340 (90.91%) consented to join the Research Registry. Of those 340 patients, 326 (95.88%) had at least one MBC assessment, and 230 (67.65%) had at least 2 (M = 3.54, SD = 2.59). Based on EMR data, the average number of completed assessments per patient was 3.54 (SD = 2.59), which is slightly lower than the expected number of assessments per patient (M = 3.70, SD = 2.54). Importantly, the number of expected assessments will not match the number of patient visits attended, as MBC measures are not administered at every visit, depending upon the patient’s level of care (see above for expected assessment intervals). Overall, across all patients, 84.10% (n = 1153/1371) of expected assessments were collected. A total of 227 patients (66.67%) had at least the expected number of MBC assessments; it was possible for patients to have more than the expected number of MBC data points, if they completed MBC measures more frequently than required (for instance, completing MBC measures twice in one week of IOP treatment, even though only one administration per week is expected). Of those who were missing assessments (n = 113, 33.24%), most were missing only one (n = 64, 56.64%), with a small proportion missing two (n = 30, 26.55%) or three (n = 19, 16.81%) MBC assessments.
Rates of adherence to MBC data collection procedures varied by treatment modality. High rates of completion were found for intake appointments (93.81%, n = 197/210), IOP appointments (96.79%, n = 452/467), and monthly outpatient continuation/maintenance group (93.18%, n = 82/88). The completion rate was significantly lower for patients seen on an outpatient basis weekly (69.64%, n = 422/606).
Because data were collated on the basis of attended treatment sessions listed in the EMR system (i.e., data were not counted as missing if the patient did not receive services), missed assessments were due either to the clinician failing to administer the measures, or the patient refusing to complete them. Although this information is not available in this dataset, clinical experience in this setting suggests that patient refusal is quite rare, indicating that clinician failure to administer the measures is the most likely cause of missed assessments. This is consistent with the discrepancy in completion rates, which is likely due to the intervals at which assessments are completed in each setting. MBC assessments are expected for every intake appointment and each continuation/maintenance group, making the collection of those data more routine. Weekly MBC assessments are gathered every Monday of IOP, so missing data are unlikely unless a patient fails to attend a scheduled Monday IOP session but does attend IOP later in the week (and in this case, patients are directed to complete MBC measures when they return to IOP). For weekly outpatient treatment, however, MBC data are only gathered monthly; thus, clinicians must track when clients completed their last MBC assessment to determine whether each session requires MBC administration.
Lessons learned and future directions
The use of MBC with suicidal youth has numerous benefits. Based on our findings, clinicians find the use of MBC in this setting acceptable and beneficial, and results suggest that MBC can be implemented with reasonable fidelity even in primarily clinical (rather than research) settings. Our findings regarding staff perceptions of MBC were consistent with prior research identifying key reasons to use (or not use) progress monitoring in psychotherapy [66], as well as qualitative research with clinician stakeholders [67–69].
The STAR-Center team has implemented several strategies, in addition to those outlined above, to improve the utility and acceptability of MBC. Critically, at least one (and presently two) STAR-Center clinical staff are designated to provide additional feedback and support of the MBC system as a part of their professional duties. These staff coordinate twice monthly meetings to bridge communication between clinical staff, researchers, analysts, and programmers affiliated with STAR-Center, which facilitate identification of clinical needs not yet served by MBC and problem solving around obstacles to MBC implementation. Clinical staff liaisons play an essential role in enhancing clinician commitment to MBC, serving as advocates for the clinical staff and promoting an environment of enthusiasm and positivity about MBC. We have found that having designated leadership among clinical staff in support of the MBC program is critical to the long-term success of MBC at STAR-Center, consistent with recommendations from extant literature regarding organizational supports for MBC [70].
As evidenced by sparser MBC completion in the context of weekly outpatient care described above, ensuring clinicians gather MBC data at irregular intervals can be difficult. At STAR-Center, MBC assessments are not integrated into the EMR system, which makes routine reminders more complex; these data are instead tracked through a clinician dashboard in an external system (see Fig 2). Although the independent nature of the STAR-Center’s MBC system has some benefits, such as the ability to highly tailor its workflow, measures, and output to meet the clinic’s needs, it also has some drawbacks, such as a separate interface, the need for clinicians to initiate MBC assessments (e.g., generate and then text or email a link to youth/caregiver), and the inability to populate the EMR directly with MBC-generated data (although screenshots can be copied into clinical notes as appropriate). A future direction for STAR-Center leadership involves the design and implementation of an acceptable reminder system for clinicians to request patient-completed MBC assessments in weekly outpatient treatment and to complete clinician-rated MBC assessments at necessary intervals.
This is an example of a display for a single patient followed over time. MFQ = Short Mood and Feelings Questionnaire; SCARED = Screen for Child Anxiety Related Disorders; CALS = Children’s Affective Lability Scale; ARI = Affective Reactivity Index; CRAFFT = Car, Relax, Alone, Forget Friends, Trouble; PSQI = Pittsburgh Sleep Quality Index.
Although STAR-Center leadership is committed to the use of MBC with its patients, MBC systems are not without costs, both financial and with respect to staff time and effort. MBC systems that are more complex (e.g., allow for different measures to be administered to different patients, “flagging” high-risk responses, integration with EMR data) require significant resources, which hinders MBC implementation in many clinical environments. Policy changes that provide financial support for these programs, such as increased reimbursement rates from third-party payers for services that involve MBC, could increase the likelihood of widespread MBC adoption in mental healthcare settings. Finally, greater research is needed to examine not only the feasibility of MBC in routine clinical care for suicidal youth, but also to elucidate the ways that MBC impacts patient outcomes.
Acknowledgments
We acknowledge with gratitude Dr. David Brent’s Endowed Chair in Suicide Studies, as well as the Pennsylvania Legislature for their support of the STAR-Center and for supporting the electronic data collection and management. We are grateful to the youth and families who provided their data. We thank the STAR-Center team, including assessors, therapists, nurses, psychiatrists, data staff, students, and volunteers for their work and dedication.
References
- 1. Fortney J, Sladek R, Unützer J. Fixing behavioral health care in America: A national call for measurement-based care in the delivery of behavioral health services. 2015 [cited 2021 November 18]. In: The Kennedy Forum [Internet]. Available from: https://www.thekennedyforum.org/app/uploads/2017/06/KennedyForum-MeasurementBasedCare_2.pdf.
- 2. The Joint Commission. Complying with standard CTS.03.01.09 behavioral health care accreditation program. 2019 [cited 2021 November 18]. Available from: https://www.jointcommission.org/-/media/tjc/documents/accred-and-cert/bhc/rationale-cts030109-revised.pdf.
- 3. Fortney JC, Unützer J, Wrenn G, Pyne JM, Smith GR, Schoenbaum M, et al. A tipping point for measurement-based care. Psychiatr Serv. 2017;68: 179–188. pmid:27582237
- 4. Guo T, Xiang Y-T, Xiao L, Hu C-Q, Chiu HF, Ungvari GS, et al. Measurement-based care versus standard care for major depression: A randomized controlled trial with blind raters. Am J Psychiatr. 2015;172:1004–1013. pmid:26315978
- 5. Yeung AS, Jing Y, Brenneman SK, Chang TE, Baer L, Hebden T, et al. Clinical Outcomes in MEasurement-based Treatment (COMET): A trial of depression monitoring and feedback to primary care physicians. Depress Anxiety. 2012;29:865–873. pmid:22807244
- 6. Bickman L, Kelley SD, Breda C, de Andrade A, Reimer M. Effects of routine feedback to clinicians on mental health outcomes of youths: Results of a randomized trial. Psychiatr Serv. 2011;62:1423–1429. pmid:22193788
- 7. Scott K, Lewis CC. Using measurement-based care to enhance any treatment. Cogn Behav Pract. 2015;22:49–59. pmid:27330267
- 8. Eisen SV, Dickey B, Sederer LI. A self-report symptom and problem rating scale to increase inpatients’ involvement in treatment. Psychiatr Serv. 2000;51:349–353. pmid:10686242
- 9. Greenhalgh J, Meadows K. The effectiveness of the use of patient-based measures of health in routine practice in improving the process and outcomes of patient care: A literature review. J Eval Clin Pract. 1999;5:401–416. pmid:10579704
- 10. Morris DW, Trivedi MH. Measurement-based care for unipolar depression. Curr Psychiatr Rep. 2011;13:446–458. pmid:21935633
- 11. Berard R, Fong R, Carpenter DJ, Thomason C, Wilkinson C. An international, multicenter, placebo-controlled trial of paroxetine in adolescents with major depressive disorder. J Child Adolesc Psychopharmacol. 2006;16:59–75. pmid:16553529
- 12. Elmquist JM, Melton TK, Croarkin P, McClintock SM. A systematic overview of measurement-based care in the treatment of childhood and adolescent depression. J Psychiatr Pract. 2010;16:217–234. pmid:20644357
- 13. von Knorring A-L, Olsson GI, Thomsen PH, Lemming OM, Hultén A. A randomized, double-blind, placebo-controlled study of citalopram in adolescents with major depressive disorder. J Clin Psychopharmacol. 2006;26:311–315. pmid:16702897
- 14. Glazer K, Rootes-Murdy K, Van Wert M, Mondimore F, Zandi P. The utility of PHQ-9 and CGI-S in measurement-based care for predicting suicidal ideation and behaviors. J Affect Disord. 2020;266:766–771. pmid:29954612
- 15. Schmitz WM, Allen MH, Feldman BN, Gutin NJ, Jahn DR, Keespies PM, et al. Preventing suicide through improved training in suicide risk assessment and care: An American Association of Suicidology task force report addressing serious gaps in U.S. mental health training. Suicide Life Threat Behav. 2012;42:292–304. pmid:22494118
- 16. Kleiman EM, Turner BJ, Fedor S, Beale EE, Huffman JC, Nock MK. Examination of real-time fluctuations in suicidal ideation and its risk factors: Results from two ecological momentary assessment studies. J Abnorm Psychol. 2017;126:726–738. pmid:28481571
- 17. Schatten HT, Gaudiano BA, Primack JM, Arias SA, Armey MF, Miller IW, et al. Monitoring, assessing, and responding to suicide risk in clinical research. J Abnorm Psychol. 2020;129:64–69. pmid:31868389
- 18. Helms SW, Prinstein MJ. Risk assessment and decision making regarding imminent suicidality in pediatric settings. Clin Pract Pediatr Psychol. 2014;2:176–193.
- 19. McCauley E, Berk MS, Asarnow JR, Adrian M, Cohen J, Korslund K, et al. Efficacy of dialectical behavior therapy for adolescents at high risk for suicide: A randomized clinical trial. JAMA Psychiatr. 2018;75:777–785. pmid:29926087
- 20. Shearin EN, Linehan MM. Dialectical behavior therapy for borderline personality disorder: Theoretical and empirical foundations. Acta Psychiatr Scand. 1994;379:61–68. pmid:8010153
- 21. Klein AS, Skinner JB, Hawley KM. Targeting binge eating through components of dialectical behavior therapy: Preliminary outcomes for individually supported diary card self-monitoring versus group-based DBT. Psychotherapy. 2013;50:543–552. pmid:24295464
- 22. Landes SJ, Carlson EB, Ruzek JI, Wang D, Hugo E, DeGaetano N, et al. Provider-driven development of a measurement feedback system to enhance measurement-based care in VA mental health. Cogn Behav Pract. 2015;22:87–100.
- 23. Brent D, Emslie G, Clarke G, Wagner KD, Asarnow JR, Keller M, et al. Switching to another SSRI or to venlafaxine with or without cognitive behavioral therapy for adolescents with SSRI-resistant depression: The TORDIA randomized controlled trial. JAMA. 2008;299:901–913. pmid:18314433
- 24. Rengasamy M, Mansoor BM, Hilton R, Porta G, He J, Emslie GJ, et al. The bi-directional relationship between parent-child conflict and treatment outcome in treatment-resistant adolescent depression. J Am Acad Child Adolesc Psychiatry. 2013;52:370–377. pmid:23582868
- 25. Iorfino F, Cross SP, Davenport T, Carpenter JS, Scott E, Shiran S, et al. A digital platform designed for youth mental health services to deliver personalized and measurement-based care. Front Psychiatry. 2019;10:595. pmid:31507465
- 26. Jensen-Doss A, Douglas S, Phillips DA, Gencdur O, Zalman A, Gomez NE. Measurement-based care as a practice improvement tool: Clinical and organizational applications in youth mental health. Evid Based Pract Child Adolesc Ment Health. 2020;5:233–250. pmid:33732875
- 27. Chiauzzi E. Lessons learned in measurement-based care with youth mental health clients. Pract Innov. 2021;6:30–41.
- 28. Connors EH, Arora PG, Resnick SG, McKay M. A modified measurement-based care approach to improve mental health treatment engagement among racial and ethnic minoritized youth. Psychol Serv. 2022. pmid:35130008
- 29. Lyon AR, Pullmann MD, Whitaker K, Ludwig K, Knaster Wasse J, McCauley E. A digital feedback system to support implementation of measurement-based care by school-based mental health clinicians. J Clin Child Adolesc Psychol. 2019;48:S168–S179. pmid:28278597
- 30. Lavender J, Benningfield MM, Merritt JA, Gibson RL, Bettis AH. Measurement-based care in the adolescent partial hospital setting: Implementation, challenges, & future directions. Evid Based Pract Child Adolesc Ment Health. 2022.
- 31. Cuperfain AB, Hui K, Berkhout SG, Foussias G, Gratzer D, Kidd SA, et al. Patient, family and provider views of measurement-based care in an early-psychosis intervention programme. BJPsych Open. 2021;7:e171.
- 32. University of Pittsburgh STAR-Center. About STAR-Center. 2020 [cited 2021 November 18]. Available from: https://www.starcenter.pitt.edu/about.
- 33.
Brent DA, Poling KD, Golstein TR. Treating depressed and suicidal adolescents: A clinician’s guide. New York: The Guilford Press; 2011.
- 34. Angold A, Costello EJ, Messer SC, Pickles A. Development of a short questionnaire for use in epidemiological studies of depression in children and adolescents. Int J Methods Psychiatr Res. 1995;5:237–249.
- 35. Messer SC, Angold A, Costello EJ, Loeber R, Van Kammen W, Stouthamer-Loeber M. Development of a short questionnaire for use in epidemiological studies of depression in children and adolescents: Factor composition and structure across development. Int J Methods Psychiatr Res. 1995;5:251–262.
- 36. Sharp C, Goodyer IM, Croudace TJ. The Short Mood And Feelings Questionnaire (SMFQ): A unidimensional item response theory and categorical data factor analysis of self-report ratings from a community sample of 7- through 11-year-old children. J Abnorm Child Psychol. 2006;34:365–377. pmid:16649000
- 37. Birmaher B, Brent DA, Chiappetta L, Bridge J, Monga S, Baugher M. Psychometric properties of the Screen for Child Anxiety Related Emotional Disorders (SCARED): A replication study. J Am Acad Child Adolesc Psychiatry. 1999;38:1230–1236. pmid:10517055
- 38. Caporino NE, Sakolsky D, Brodman DM, McGuire JF, Piacentini J, Peris TS, et al. Establishing clinical cutoffs for response and remission on the Screen for Child Anxiety Related Emotional Disorders (SCARED). J Am Acad Child Adolesc Psychiatry. 2017;56:696–702. pmid:28735699
- 39. Solantaus T, Paavonen EJ, Toikka S, Punamäki R-L. Preventive interventions in families with parental depression: Children’s psychosocial symptoms and prosocial behaviour. Eur Child Adolesc Psychiatry. 2010;19:883–892. pmid:20890622
- 40. Gerson AC, Gerring JP, Freund L, Joshi PT, Capozzoli J, Brady K, et al. The Children’s Affective Lability Scale: A psychometric evaluation of reliability. Psychiatr Res. 1996;65:189–198.
- 41. Goldstein TR, Axelson DA, Birmaher B, Brent DA. Dialectical behavior therapy for adolescents with bipolar disorder: A 1-year open trial. J Am Acad Child Adolesc Psychiatry. 2007;46:820–830. pmid:17581446
- 42. Johnco C, McGuire JF, McBride NM, Murphy TK, Lewin AB, Storch EA. Suicidal ideation in youth with tic disorders. J Affect Disord. 2016;200:204–211. pmid:27136419
- 43. Buysse DJ, Reynolds CF, Monk TH, Berman SR, Kupfer DJ. The Pittsburgh sleep quality index: A new instrument for psychiatric practice and research. Psychiatr Res. 1989;28:193–213. pmid:2748771
- 44. Guo L, Deng J, He Y, Deng X, Huang J, Huang G, et al. Prevalence and correlates of sleep disturbance and depressive symptoms among Chinese adolescents: A cross-sectional survey study. BMJ Open. 2014;4:e005517. pmid:25079937
- 45. Tan E, Healey D, Gray AR, Galland BC. Sleep hygiene intervention for youth aged 10 to 18 years with problematic sleep: A before-after pilot study. BMC Pediatr. 2012;12. pmid:23216856
- 46. Knight JR, Sherritt L, Shrier LA, Harris SK, Chang G. Validity of the CRAFFT substance abuse screening test among adolescent clinic patients. Arch Pediatr Adolesc Med. 2002;156:607–614. pmid:12038895
- 47. Knight JR, Shrier LA, Bravender TD, Farrell M, Vander Bilt J, Shaffer HJ. A new brief screen for adolescent substance abuse. Arch Pediatr Adolesc Med. 1999;153:591–596. pmid:10357299
- 48. Oesterle TS, Hitschfeld MJ, Lineberry TW, Schneekloth TD. CRAFFT as a substance use screening instrument for adolescent psychiatry admissions. J Psychiatr Pract. 2015;21:259–266. pmid:26164051
- 49. Miller L, Hlastala SA, Mufson L, Liebenluft E, Yenokyan G, Riddle M. Interpersonal psychotherapy for mood and behavior dysregulation: Pilot randomized trial. Depress Anxiety. 2018;35:574–582. pmid:29719093
- 50. Stoddard J, Sharif-Askary B, Harkins EA, Frank HR, Brotman MA, Penton-Voak IS, et al. An open pilot study of training hostile interpretation bias to treat Disruptive Mood Dysregulation Disorder. J Child Adolesc Psychopharmacol. 2016;26:49–57. pmid:26745832
- 51. Stringaris A, Goodman R, Ferdinando S, Razdan V, Muhrer E, Leibenluft E, et al. The Affective Reactivity Index: A concise irritability scale for clinical and research settings. J Child Psychol Psychiatr. 2012;53:1109–1117. pmid:22574736
- 52. Ballard ED, Stanley IH, Horowitz LM, Cannon EA, Pao M, Bridge JA. Asking youth questions about suicide risk in the pediatric emergency department: Results from a qualitative analysis of patient opinions. Clin Pediatr Emerg Med. 2013;14:20–27. pmid:23908599
- 53. Horowitz LM, Bridge JA, Teach SJ, Ballard E, Klima J, Rosenstein DL, et al. Ask Suicide-Screening Questions (ASQ): A brief instrument for the pediatric emergency department. Arch Pediatr Adolesc Med. 2012;166:1170–1176. pmid:23027429
- 54. Brent D, Poling K. Suicide Circumstances Schedule (SCS). Unpublished Scale. University of Pittsburgh. 2000.
- 55. Posner K, Brown GK, Stanley B, Brent DA, Yershova KV, Oquendo MA, et al. The Columbia-Suicide Severity Rating Scale: /Initial validity and internal consistency findings from three multisite studies with adolescents and adults. Am J Psychiatr. 2011; 168:1266–1277. pmid:22193671
- 56. Kaufman J, Schweder AE. The Schedule for Affective Disorders and Schizophrenia for School-Age Children: Present and Lifetime version (K-SADS-PL). 2004.
- 57.
Guy W. ECDEU Assessment Manual for Psychopharmacology. 2nd ed. Washington, DC: US Government Printing Office; 1976. Publication ADM 76–338.
- 58. Shaffer D, Gould MS, Brasic J, Ambrosini P, Fisher P, Bird HA. Children’s Global Assessment Scale (C-GAS). Arch Gen Psychiatr. 1983;40:1228–1231.
- 59. Liu FF, Adrian MC. Is treatment working? Detecting real change in the treatment of child and adolescent depression. J Am Acad Child Adolesc Psychiatry. 2019;58:1157–1164. pmid:30825497
- 60. Goldstein LA, Connolly Gibbons MB, Thompson SM, Scott K, Heintz L, Green P, et al. Outcome assessment via handheld computer in community mental health: Consumer satisfaction and reliability. J Behav Health Serv Res. 2011;38:414–423. pmid:21107916
- 61. Trivedi MH. Tools and strategies for ongoing assessment of depression: A measurement-based approach to remission. J Clin Psychiatr. 2009;70:26–31. pmid:19922741
- 62. Mellor-Clark J, Cross S, Macdonald J, Skjulsvik T. Leading horses to water: Lessons from a decade of helping psychological therapy services use routine outcome measurement to improve practice. Adm Policy Ment Health. 2016;43:279–285. pmid:25179755
- 63. Dowrick C, Leydon GM, McBride A, Howe A, Burgess H, Clarke P, et al. Patients’ and doctors’ views on depression severity questionnaires incentivised in UK quality and outcomes framework: Qualitative study. BMJ. 2009;338:b663. pmid:19299474
- 64. Kelley SD, Bickman L. Beyond outcomes monitoring: Measurement feedback systems in child and adolescent clinical practice. Curr Op Psychiatr. 2009;22:363–368. pmid:19417669
- 65. Jette DU, Halbert J, Iverson C, Miceli E, Shah P. Use of standardized outcome measures in physical therapist practice: Perceptions and applications. Phys Ther. 2009;89:125–135. pmid:19074618
- 66. Hatfield DR, Ogles BM. Why some clinicians use outcome measures and others do not. Adm Policy Ment Health. 2007;34:281–291. pmid:17211715
- 67. Garland AF, Kruse M, Aarons GA. Clinicians and outcome measurement: What’s the use? J Behav Health Serv Res. 2003;30:393–405. pmid:14593663
- 68. Lyon AR, Knaster Wasse J, Ludwig K, Zachry M, Bruns EJ, Unützer J, et al. The Contextualized Technology Adaptation Process (CTAP): Optimizing health information technology to improve mental health systems. Adm Policy Ment Health. 2015;43:394–409.
- 69. De Beurs E, den Hollander-Gijsman ME, van Rood YR, van der Wee NJA, Giltay EJ, van Noorden MS, et al. Routine outcome monitoring in the Netherlands: Practical experiences with a web-based strategy for the assessment of treatment outcome in clinical practice. Clin Psychol Psychother. 2011;18:1–12. pmid:20238371
- 70. Lewis CC, Boyd M, Puspitasari A, Navarro E, Howard J, Kassab H, et al. Implementing measurement-based care in behavioral health: A review. JAMA Psychiatry. 2019;76:324–335. pmid:30566197