https://orcid.org/0000-0003-2185-4518
Figures
Abstract
Suicide rates have been steadily increasing in both the U.S. general population and military, with significant psychological and economic consequences. The purpose of the current study was to examine the economic costs and cost-benefit of the suicide-focused Collaborative Assessment and Management of Suicidality (CAMS) intervention versus enhanced treatment as usual (ETAU) in an active duty military sample using data from a recent randomized controlled trial of CAMS versus ETAU. The full intent-to-treat sample included 148 participants (mean age 26.8 years ± 5.9 SD years, 80% male, 53% White). Using a micro-costing approach, the cost of each condition was calculated at the individual level from a healthcare system perspective. Benefits were estimated at the individual level as cost savings in past-year healthcare expenditures based on direct care reimbursement rates. Cost-benefit was examined in the form of cost-benefit ratios and net benefit. Total costs, benefits, cost-benefit ratios, and net benefit were calculated and analyzed using general linear mixed modeling on multiply imputed datasets. Results indicated that treatment costs did not differ significantly between conditions; however, CAMS was found to produce significantly greater benefit in the form of decreased healthcare expenditures at 6-month follow-up. CAMS also demonstrated significantly greater cost-benefit ratios (i.e., benefit per dollar spent on treatment) and net-benefit (i.e., total benefit less the cost of treatment) at 12-month follow-up. The current study suggests that beyond its clinical effectiveness, CAMS may also convey potential economic advantages over usual care for the treatment of suicidal active duty service members. Our findings demonstrate cost savings in the form of reduced healthcare expenditures, which theoretically represent resources that can be reallocated toward other healthcare system needs, and thus lend support toward the overall value of CAMS.
Citation: McCutchan PK, Yates BT, Jobes DA, Kerbrat AH, Comtois KA (2022) Costs, benefits, and cost-benefit of Collaborative Assessment and Management of Suicidality versus enhanced treatment as usual. PLoS ONE 17(2): e0262592. https://doi.org/10.1371/journal.pone.0262592
Editor: Isabelle Durand-Zaleski, URCEco Ile de France Hopital de l’Hotel Dieu, FRANCE
Received: April 11, 2021; Accepted: December 30, 2021; Published: February 3, 2022
Copyright: © 2022 McCutchan et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Public sharing of data used in this study is prohibited under the protocol approved by the U.S. Army Medical Research and Development Command Office of Research Protections as well as Institutional Review Boards at Dwight D. Eisenhower Army Medical Center, The Catholic University of America, and the University of Washington. Additional restrictions apply to the availability of these data in order to protect participants’ privacy. Requests for access from interested researchers may nevertheless be considered, subject to the terms and conditions of the request and in compliance with the applicable regulations. Requests may be directed to the Office of Sponsored Programs and Research Services, The Catholic University of America, 213 McMahon Hall, 620 Michigan Ave., NE, Washington, DC 20064; Phone: 202-319-5218; Fax: 202-319-4495; Email: CUA-OSP@cua.edu.
Funding: The current study represents an unfunded secondary analysis of a randomized-controlled trial; the primary trial was supported by the Department of the Army through federal grant W81XWH-11-1-0164, awarded and administered by the Military Operational Medicine Research Program (MOMRP). The authors received no specific funding for the present work.
Competing interests: The authors have read the journal’s policy and have the following competing interests to declare: DAJ receives book royalties from the American Psychological Association Press and Guilford Press, and is the founder of CAMS-care, LLC (a professional training and consultation company; https://cams-care.com/). There are no patents, products in development or marketed products associated with this research to declare. This does not alter our adherence to PLOS ONE policies on sharing data and materials. Authors PM, BTY, AHK, and KAC have no competing interests to declare.
Introduction
Amid unprecedented increases in active duty military suicide rates in recent years, the U.S. Department of Defense (DoD) has recognized suicide prevention to be a top military priority [1,2]. Active duty service members have demonstrated continuously rising suicide rates across all branches of service. In 2019, the suicide mortality rate for the Active Component (i.e., full-time service members) across all services was 25.9 per 100,000, representing a per-year rate ratio of 1.04 from calendar years 2011 through 2019 [3]. These trends are particularly alarming in that they represent the first time in recorded history that the U.S. military suicide rate has equaled or exceeded that of comparable U.S. general population cohorts, beginning in 2008 and continuing to date. Indeed, the 2019 suicide mortality rate for the Active Component was statistically indistinguishable from the 2018 U.S. population rate after adjusting for age and gender [4]. Although the exact causes of the increase in military suicides are uncertain, the most consistently reported factor appears to be increasing co-occurrence of mental health conditions such as posttraumatic stress disorder (PTSD), depression, and substance use [5–9].
In response to this alarming increase in military suicide rates, the DoD has collaborated with other government and private agencies to make considerable investments in research geared toward determining the risk factors and correlates of suicidality and developing effective prevention and management strategies [10]. New interventions aimed specifically at managing suicidality are emerging, but additional research is needed to carefully evaluate the effectiveness of these interventions in rigorous, well-powered studies in order to inform clinical practice guidelines and facilitate integration into routine clinical care.
One promising suicide-focused approach is the Collaborative Assessment and Management of Suicidality (CAMS) [11–13], a therapeutic framework that aims to identify and address patient-articulated “suicidal drivers.” To date, CAMS has amassed a substantial evidence base demonstrating its effectiveness in reducing suicidal ideation, overall symptom distress, depression, and hopelessness, including several correlational and quasi-experimental studies [14–21] and five randomized controlled trials (RCTs) across a variety of settings [22–26]. A recent systematic review of CAMS reported it to be a promising approach to managing suicide risk and deliberate self-harm in adults [27], although it noted a high degree of heterogeneity and attrition across existing studies and emphasized the need for additional high-quality studies, particularly RCTs and meta-analyses. Further, a more recent meta-analysis reported that CAMS treatment resulted in significantly lower suicidal ideation and general distress compared to alternative interventions, concluding that it is a well-supported intervention for suicidal ideation according to Center of Disease Control and Prevention criteria [28].
In addition to its clinical effectiveness, another important factor requiring further examination is the cost of the CAMS intervention, particularly in comparison to alternatives. As with any health intervention, understanding the costs of delivering CAMS in relation to both clinical and monetary outcomes produced can be seen as an essential component of informed decision-making by policymakers, healthcare administrators, and even clinicians. The economic reality, especially in a population-based system such as the Military Health System (MHS), is that resources allocated to one intervention are then no longer available for other needed services [29,30]. Therefore, it could be helpful to examine the value of an intervention (i.e., costs relative to outcomes) compared to that of other possible alternatives in order to “provide the best to the most for the least” [31].
Preliminary analyses using archival data in one retrospective non-randomized controlled comparison study of CAMS vs. treatment as usual at two U.S. Air Force outpatient clinics found that costs of the treatment conditions did not differ [16]. However, CAMS patients utilized significantly fewer medical services (e.g., emergency room visits, medical appointments, and minutes spent in these settings) after initiating suicide-related mental health treatment, indicating relative cost savings in the form of reduced healthcare expenditures. Extrapolating the observed utilization rates to the anticipated number of suicidal patients presenting to the clinics over the next fiscal year (n = 75), the authors estimated a potential cost savings of approximately $32,500 ($434.25 per patient on average; 2004 dollars). Extrapolating across the total 80 Air Force clinics worldwide, cost savings estimates approached $2 million per year [16]. Still, the significant methodological limitations of the study must be noted, including a small and unequally distributed sample size (N = 55; n = 25 for CAMS and n = 30 for treatment as usual), lack of random assignment, and lack of fidelity measures. Consequently, it is premature to definitively conclude based on these analyses that CAMS is associated with economic advantages compared to treatment alternatives.
Although these preliminary findings were encouraging, additional examinations using more robust study designs, detailed costing methods, and systematic evaluation of treatment benefits are warranted. The purpose of the current study was to extend previous analyses by examining the costs, benefits, and cost-benefit of CAMS compared to enhanced treatment as usual (ETAU) for the treatment of suicidality in active duty Soldiers using data from a recent clinical effectiveness RCT. Results of the primary RCT indicated that CAMS treatment was associated with significant postbaseline improvements in suicidal ideation and suicide attempt behaviors across one year of follow-up. However, the ETAU condition evidenced comparable improvements, with the exception of a significantly greater reduction in probability of suicidal ideation at 3-month follow-up observed in the CAMS condition [24]. Examination of treatment costs and benefits become especially useful in such scenarios where two treatment alternatives appear similarly effective, to select the intervention which maximizes return on investment.
In the current secondary cost analysis, treatment costs (i.e., the value of resources required to implement the intervention) were determined using micro-costing and gross-costing techniques. Benefits of the intervention (i.e., the value of resources generated or saved as a result of the intervention) were assessed as potential cost savings based on healthcare expenditures across multiple categories of services. Finally, cost-benefit (i.e., the relationship between the value of resources used and the value of resources saved by an intervention) was examined using cost-benefit ratio and net benefit metrics [32].
Previous analyses found no significant differences in direct costs (i.e., resource use directly attributable to the intervention) for CAMS compared to usual care [16]. Although CAMS as delivered in the current study required more resources to support the training and consultation activities performed over and above that of the usual care condition, the costs associated with these activities were anticipated to be relatively minimal at the individual level; therefore, it was hypothesized that CAMS would be no more costly than ETAU at each follow-up timepoint (Hypothesis 1).
With regard to benefits, evidence suggests that suicidality is associated with increased health service utilization in active duty Soldiers [33–35]. Because CAMS was anticipated to lead to greater and more rapid clinical improvement in suicidality than ETAU, we predicted that health service utilization would decrease correspondingly. Indeed, usual-care patients have been found to attend significantly more medical appointments compared to CAMS patients over a one-year period [16]. Consequently, it was hypothesized that CAMS would evidence greater benefit (i.e., cost savings) in the form of lower between-group cumulative healthcare expenditures at each follow-up timepoint (Hypothesis 2) and greater reduction in within-subjects past-year healthcare expenditures at 12-month follow-up (Hypothesis 3). Because CAMS was predicted to be less costly and more beneficial than ETAU, we further anticipated that CAMS would be associated with greater cost-benefit than ETAU, as evidenced by significantly greater cost-benefit ratios (Hypothesis 4) and significantly greater net benefit (Hypothesis 5) at 12-month follow-up.
Materials and methods
Trial design
The current study was a secondary data analysis of the DoD-funded “Operation Worth Living” (OWL) study, an RCT of CAMS versus ETAU for suicidal Soldiers. The trial was conducted in the Department of Behavioral Health at an Army Medical Center on an infantry military installation in the Southern United States. Participants were randomly assigned to either CAMS or ETAU matched on histories of suicide attempts, medication class, severity of physical injury or disability, and current enrollment in outpatient behavioral health treatment (i.e., psychiatric, clinical psychology, or social work services). Participants were assessed on clinical variables and service utilization measures at baseline and 1-month, 3-month, 6-month, and 12-month timepoints after baseline. Details about the trial design, including study procedures and power analysis, are described elsewhere [24]. All study procedures were approved by The Catholic University of America, University of Washington, and Eisenhower Army Medical Center Institutional Review Boards. The current archival study was approved by the American University Institutional Review Board (IRB-2017-75).
Participants
Soldier participants in the study were 148 active duty U.S. infantry Soldiers with current suicidal ideation recruited via provider referral from the community behavioral health clinic, emergency department, and inpatient psychiatric unit. Soldiers were eligible to participate if they were over 18 years old, English-speaking, and had significant suicidal ideation (i.e., a score of ≥ 13 on the Scale for Suicidal Ideation-Current) [36]. Soldiers were excluded if they were a member of the Warriors in Transition unit (a unit providing support to soldiers being treated for chronic and/or severe injuries who cannot yet return to work), pregnant, exhibiting significant psychosis or cognitive or physical impairment, judicially ordered to treatment, or ineligible for behavioral health care at the military installation.
On-site clinicians were also consented participants in the study as they completed routine assessments. Eligible clinicians were assigned to either the CAMS or ETAU condition based on their relative self-reported allegiance to treatment models for managing suicidal patients; clinicians reporting weaker preference for a specific treatment were assigned to the CAMS condition to ensure high adherence of ETAU clinicians to their existing approach. The final sample of study therapists participating in the intent-to-treat phase of the trial (n = 4 per treatment condition) consisted of seven licensed clinical social workers and one masters-level mental health counselor. Mean years of practice experience since professional degree were 5.0 (SD = 4.7) for ETAU therapists and 13.25 (SD = 7.4) for CAMS therapists.
Treatments
CAMS is best described as a therapeutic framework that accommodates a wide range of theoretical orientations and techniques. A core tenet of CAMS therapy is an empathetic and collaborative therapeutic relationship geared toward developing a shared understanding of the patient’s suicidality. Guided by the Suicide Status Form (SSF), the fundamental goal is to engage the patient in identifying and addressing the causes (“drivers”) that compel him or her to consider suicide using appropriate driver-focused interventions, subsequently reducing suicidal ideation and behaviors as coping mechanisms are increased [11,13]. In the current study, the CAMS intervention consisted of approximately 4 to 11 weekly individual sessions (after the initial session, CAMS concludes after three consecutive sessions with resolved suicidality) following the suicide-specific CAMS framework [13].
In ETAU, clinicians provided typical care in as many sessions as appropriate to their treatment approach and theoretical orientation without constraint by the study, reflecting the real-world conditions examined within an effectiveness framework to maximize generalizability. This treatment condition was considered “enhanced” treatment as usual in that it specified a minimum number of four treatment sessions, providers had access to consultation and supervision as needed, and Soldiers were routinely engaged and evaluated at the study assessment intervals [24].
Measures
The Scale for Suicide Ideation-Current (SSI-C) [36], a 19-item interviewer-rated measure assessing suicidal ideation at its highest intensity over the past two weeks, was used to assess suicidal ideation severity (total score of 0–38) and resolution (i.e., a total score of zero). The SSI-C has demonstrated good convergent and criterion validity in previous research in psychiatric patients [36] and evidenced high internal consistency in the current study (α = .88).
Medical and behavioral health service utilization was assessed using the Treatment History Interview-Military version (THI-M), adapted from the Treatment History Interview (THI) [37] for use in the military healthcare system. The THI-M is an interviewer-administered measure that captures the history (e.g., service frequency, duration, and provider type) of outpatient psychotherapy and counseling visits, crisis medical services, medical provider visits, and medication use. The THI has been found to have high convergent validity with medical records (r = .99). Pilot studies found no significant differences between THI self-report and therapist records for number of psychotherapy hours [37]. In the current study, THI-M responses were validated against available administrative data sources (i.e., electronic health records). In the baseline assessment, the THI-M measured service utilization over the past year; in follow-up assessments, the THI-M measured service utilization since the previous assessment.
Cost assessment
The cost of each treatment condition was calculated from a healthcare system perspective, given that the MHS is the sole payor and primary decision-maker with regard to implementation of these suicide prevention services. Total treatment costs were comprised of two categories: training and implementation activity costs and treatment delivery costs.
Training and implementation activity costs.
Micro-costing was applied to determine the cost of training and implementation activities beyond standard clinical practice [31,38,39]. Costs related to these activities only applied to the CAMS condition in the current analyses as ETAU therapists did not receive any additional training or require preparatory activities for the provision of treatment. Further, although ETAU therapists were offered additional supervision and consultation as needed, key informant interviews indicated that they elected not to participate in these activities during the course of the study. This may reflect stronger allegiance to a specific treatment approach for managing suicidality. Consequently, there were no additional costs beyond those captured in the provision of clinical services, as described in a later section. For costs related to training and implementation activities, micro-costing entailed the following steps:
Itemization of activities. An inventory was developed listing all major non-clinical activities that occur for each treatment condition at the individual level, including training and other preparatory activities that must take place prior to implementing the intervention. Activities performed were identified by study protocols, intervention manuals, administrative data systems maintained by research personnel, and qualitative interviews with research personnel and study therapists as feasible and appropriate.
Estimating resource inputs and unit costs. The resources used to perform each activity were identified and then the amounts of each resource used were estimated. Resources included personnel time, travel, and materials.
Costs for personnel time were measured in hourly units based on 2018 annual salary plus fringe benefits, divided by 2080 hours (an assumed 40 hours per week). Annual base salaries for CAMS trainers and consultants were estimated using the Bureau of Labor Statistics Occupational Employment Statistics Query System based on geographical area and occupation classification. Study investigators’ salaries were included to reflect the real-world cost of CAMS training delivered directly by the developers of the intervention. Study therapists were clinical social workers assumed to be GS-12 paygrade based on key informant interviews; annual base salaries were estimated based on the 2018 General Schedule Base Pay Scale [40] and adjusted for locality. A fringe benefit rate of 29.1% was applied to annual base salaries for study investigators and therapists based on the 2018 national average for private industry workers in the South Atlantic region [41]. Annual base salaries and fringe rates for four pre-doctoral graduate student assistants who contributed to fidelity assessment were estimated based on typical grant-funding allowances reported by The Catholic University of America Office of Sponsored Programs.
Travel costs reflect resources needed for personnel to attend on-site trainings of CAMS study therapists. Two on-site trainings were conducted at the military installation as part of the OWL study; however, the current cost analyses assumed that only one on-site training would be required to better approximate real-world implementation not constrained by research conditions. Costing of on-site training assumed attendance by two CAMS study investigators and four study therapists. Because the OWL study was funded through a federal grant, travel costs were assumed to be consistent with General Services Administration allowable rates. Cost of airfare was estimated using the 2018 maximum allowable airfare rate determined by the General Services Administration City Pair Program [42]. Cost of lodging was estimated based on 2018 hotel rates negotiated by the federal government by locality [43]. Cost of meals and incidental expenses was estimated based on 2018 government allowable per diem rates by locality [44]. Traveling personnel shared one rental car at a cost estimated based on 2018 daily government rental car rates by locality (www.FedTravel.com).
The on-site training required facility space large enough to host all attendees in the same room. The size of the room was estimated to be approximately 300 square feet. Facility costs were estimated based on the current median cost for office space in the closest available locality [45], adjusted to reflect 2018 dollars. Specifically, the median annual cost per square foot was multiplied by 300 square feet and then divided by 2,080 hours (the total number of hours the room could be used annually). Hourly costs of electricity were estimated using 2018 commercial electricity costs in the region per the Department of Energy [46].
Costs for materials (for example, training or psychoeducational materials) were estimated per item based on study receipts, treatment protocols, and/or information obtained in key informant interviews.
Calculating total costs. The total cost per activity was calculated by multiplying resource inputs by per-unit costs. Costs were then summed to determine the total cost for training and non-clinical implementation activities across the CAMS condition. Because training and implementation costs were incurred at the treatment condition-level, a fixed, per-individual cost was calculated by dividing total cost of activities by the total number of individuals a trained CAMS therapist might ultimately treat with the CAMS intervention during his/her career as an MHS provider based on rates of suicidal ideation in military populations [7,33,47–50] and estimated typical caseload size and length of career for MHS behavioral health providers [51,52]. These estimates also assume a relatively high degree of adherence to CAMS across the provider’s MHS career, as supported by the absence of drift reported in the primary trial [24] and a community survey of mental health practitioners reporting generally high levels of adherence to the CAMS therapeutic approach and practice [53]. The fixed cost of non-clinical implementation activities was then added to each CAMS participant’s study treatment delivery costs to estimate the overall per-individual cost of the intervention.
Treatment delivery costs.
The cost of delivering each intervention was estimated at the individual level based on the number of treatment visits attended as documented in THI-M assessments and validated by administrative study records. Each visit was assigned an appropriate Current Procedural Terminology (CPT) [54] and costed using corresponding 2018 TRICARE/CHAMPUS Maximum Allowable Charges based on facility, non-physician provider, and appropriate locality rates. TRICARE (formerly known as CHAMPUS) is a DoD health insurance program, and its Maximum Allowable Charges rates reflect costs directly related to provision of the service (i.e., provider time used in the visit), practice expenses such as facilities and administrative staff, and malpractice insurance. This same procedure was applied to missed treatment visits (i.e., no-shows) to reflect the missed opportunity to deliver services and collect payment for dedicated provider time [55]. Out-of-session contacts that occurred as part of treatment (e.g., phone calls) were assigned a CPT code based on the purpose and duration of the communication and costed accordingly.
Benefits assessment
Benefits, in the form of cost savings, were then estimated based on healthcare expenditures for behavioral health services, medical services, crisis services, and certain medications using the THI-M. Each visit was assigned an appropriate CPT code(s) and direct care costs for these services were estimated based on the TRICARE Reimbursement Manual and CHAMPUS Maximum Allowable Charge rates. Of note, because the nature of available data did not allow for determination of exact timing of service delivery, all non-study-treatment services were captured in the assessment of benefits rather than treatment costs, including those delivered during windows with active study treatment.
Analyses
Outcome analysis used an intent-to-treat approach that included all participants who completed a baseline assessment. Multiple imputation using chained equations with predictive mean matching was used to minimize uncertainty related to the replacement of missing data for health services utilization [56]. Treatment cost data did not require imputation as data were available for all participants. Participants who did not complete at least one follow-up assessment (n = 6) were excluded from imputation and subsequent analyses. Fifteen imputed datasets were created to adequately address the observed proportion of missing data [57,58]. Planned statistical analyses were carried out on each imputed dataset, producing separate summary statistics (e.g., means or regression coefficients) with corresponding standard errors. For regression analyses, Rubin’s rules [59] were applied to derive pooled coefficients and standard errors [60]. For nonparametric tests, observed and imputed datasets were analyzed and reported separately.
Descriptive analyses were conducted to examine unadjusted costs per individual, major category of activity, and treatment condition at each follow-up timepoint. General linear mixed models (LMMs) were used to assess group differences in intervention costs at each follow-up timepoint with a random intercept and slope for participant. Wilcoxon rank-sum tests were conducted to examine differences in median costs, a useful measure of the most typical cost per individual.
Benefits were examined both between-participant (i.e., ETAU versus CAMS) and within-participant (i.e., pre- versus post-intervention). LMMs were used to assess group differences across time in total healthcare expenditures and by categories of expenditure (i.e., behavioral health, medical providers, crisis services, and medications). Linear regression was used to assess within-subjects change in pre- versus post-intervention past-year healthcare expenditures at 12-month follow-up. Cost-benefit ratios (CBRs; derived as the cumulative monetary benefit of treatment divided by the cumulative cost) and net benefit (derived as the cumulative benefit minus the cumulative cost of treatment) were calculated at the individual level. Linear regression was used to examine group differences in cost-benefit and net benefit at 12-month follow-up, as these outcomes reflect pre-intervention versus post-intervention change in past-year expenditures [61].
For all regression models of repeated-measures outcomes, treatment condition, time, and an interaction term were entered as fixed effects, with a random intercept for participants and random slope for participants by time. Nesting of participants within study therapists was examined but did not significantly improve model fit and was therefore not used in analyses. A stepwise procedure was used to identify appropriate demographic and clinical covariates as fixed effects terms, and likelihood ratio tests were used to examine random parameters. All statistical analyses were conducted using Stata v.16.0.
Results
Sample at baseline
Of the 148 individuals enrolled in the study, 73 were randomized to the CAMS condition and 75 were randomized to ETAU. The full study sample (N = 148) was predominantly male (80%), White (53%), and married (51%), with a mean age of 26.8 (SD = 5.9). Participants mostly held the rank of junior enlisted (E1-E4; 70%). The mean total SSI-C score at baseline was 20 (SD = 5.3), and half of participants reported a lifetime history of at least one suicide attempt, with 27% reporting multiple attempts. There were no statistically significant differences between the CAMS and ETAU conditions with regard to patient participants’ sociodemographic or baseline clinical characteristics.
Missing data
Study retention rates in the intent-to-treat sample were 96%, 89%, 79%, and 78% for CAMS and 90%, 83%, 79%, and 77% for ETAU at 1-, 3-, 6-, and 12-month follow-ups, respectively, with an overall missing data rate of 13% for primary outcomes. Separate mixed effect logistic regression models revealed no statistically significant differences in rates of missing data across time based on age, gender, ethnicity, marital status, education level, rank, number of lifetime suicide attempts, treatment condition, or primary study therapist.
Costs
The total cost of CAMS training and consultation activities was $7,960.32; assuming each of the four CAMS clinicians could treat 160 individuals over the course of an MHS career, this amounted to a cost of $12.44 per participant. Table 1 displays the results of micro-costing procedures for these activities [31].
Table 2 shows descriptive statistics for cumulative costs of treatment over time by treatment condition. Mean total study treatment cost at 12-month follow-up was $434.82 (SD = $209.66) for ETAU and $433.38 (SD = $248.29) for CAMS; LMM results indicated that treatment conditions did not differ significantly in treatment costs across time. Wilcoxon rank-sum tests examining group differences in median total study treatment costs were consistent with linear mixed modeling (p = .57 to .80). The average number of active study treatment sessions attended at 12 months was comparable between treatment conditions (M = 6.40 [SD = 3.53] for ETAU, M = 6.20 [SD = 3.89] for CAMS; Z = 0.69, p = .49), as was the average number of out-of-session contacts (M = 0.56 [SD = 1.27] for ETAU, M = 0.59 [SD = 1.67] for CAMS; Z = -0.37, p = .71) and average number of missed visits (M = 0.55 [SD = 1.33] for ETAU, M = 0.81 [SD = 2.20] for CAMS; Z = -0.97, p = .33).
Benefits
Due to a few extreme outliers, crisis services and total healthcare expenditure variables were separately Winsorized at the 1st and 99th percentiles to better approximate a normal distribution [62]. Table 3 presents descriptive statistics for cumulative healthcare expenditures for each treatment condition by category of service and timepoint. At baseline, mean past year total healthcare expenditures were comparable between the two treatment conditions after controlling for gender and race (p = .17). Multivariate LMMs of cumulative total healthcare expenditures across follow-up timepoints revealed a significant interaction of treatment condition x time at 6-months (β = -2,480.34, SE = 1,198.95; p = .04), indicating a decreased rate of healthcare expenditure for the CAMS condition at this timepoint.
In models of cumulative healthcare expenditures by category of services, there were no significant differences by treatment condition with respect to behavioral health services, medical provider services, or medications. However, for crisis services, a significant interaction of treatment condition x time was found at 6-month (β = -2,036.12, SE = 793.38; p = .01) and 12-month (β = -2,743.24, SE = 1,300.88; p = .04) follow-ups, indicating a decreased rate of crisis services expenditures for the CAMS condition.
In examining within-subject effects, mean changes in pre- versus post-intervention total past-year healthcare expenditures suggested a slight reduction in past-year expenditures for CAMS participants at 12-month follow-up (M = -$14.21, SD = $13,075.90), whereas past-year expenditures increased for ETAU participants (M = $3,304.72, SD = $14,756.44). Aggregated across participants by treatment condition, this represents a pre- versus post-intervention increase in total past-year healthcare expenditures of $234,637.59 for ETAU compared to a decrease of $1,009.42 for CAMS. Still, linear regression indicated that within-subject changes in pre- to post-intervention expenditures were not significantly different by treatment condition, likely due to substantial variability among observations.
Cost-benefit
Mean CBRs at 12-month follow up were -13.26 (SD = 55.10) for ETAU and 1.68 (SD = 40.53) for CAMS. Linear regression revealed that CAMS participants had significantly greater CBRs (i.e., more benefit per dollar spent) compared to ETAU condition after controlling for race and gender (β = 21.67, SE = 8.17; p < .01). Mean within-subject net benefit at 12-months was -$3,739.54 (SD = $14,755.23) for ETAU and -$419.13 (SD = $12,941.40) for CAMS, indicating that descriptively neither treatment paid for itself through reduced healthcare expenditures. However, across all datasets, one-sample Wilcoxon signed-rank tests showed that median net benefits were significantly less than zero for ETAU whereas they were not significantly different from zero for CAMS. Further, linear regression revealed that CAMS was associated with significantly greater net benefit compared to ETAU after controlling for gender and race (β = -5,167.93, SE = 2,271.79; p = .03).
Discussion
This study examined the costs, benefits, and cost-benefit of CAMS versus ETAU for the treatment of suicidality in active duty service members from a healthcare perspective. Consistent with a preliminary cost analysis [16], there were no significant differences in costs of treatment between CAMS and ETAU. As anticipated, the CAMS intervention did require training and consultation activities over and above that of usual care; however, the costs of these activities were relatively minimal when spread across the entire population of beneficiaries.
With regard to benefits at the group level, CAMS was associated with significantly reduced total cumulative healthcare expenditure compared to ETAU at 6-month follow-up, likely driven by significantly lower crisis services expenditures. However, ETAU eventually matched CAMS in total cumulative healthcare expenditures at 12-month follow-up timepoints, although crisis services expenditures remained significantly lower for CAMS. The data suggest that participants in the CAMS condition may use more crisis services than ETAU in the first month of treatment but use significantly fewer crisis services by 6- and 12-month follow-ups. The early uptick in crisis service utilization may be explained by CAMS’s suicide-specific focus which promotes accessing such services in an acute crisis, whereas the decreased utilization by six months likely reflects findings from the primary trial that CAMS participants had a lower probability of having suicidal ideation at 3-month follow-up [24]. In examining within-subject pre- versus post-intervention past-year healthcare expenditures, no significant differences were found between CAMS and ETAU participants for any category of healthcare expenditures.
CBRs were significantly greater for CAMS participants at 12-month follow-up, indicating greater benefit per dollar spent compared to ETAU. On average, CAMS participants reduced their past-year total healthcare expenditures by $0.68 for each dollar spent on treatment, whereas ETAU participants increased spending by $13.26 for each dollar spent on treatment. Neither treatment condition evidenced positive net benefit; that is, neither treatment paid for itself through reduced healthcare expenditures. Still, net benefit was significantly greater in the CAMS condition compared to ETAU by approximately $3,320 per individual, on average.
Limitations and future directions
This study had several limitations. With regard to estimating intervention costs, our micro-costing approach likely resulted in inflated estimates of CAMS costs. Micro-costing was used in the current study to maximize precision in cost estimates; however, a known limitation of micro-costing is a potential lack of generalizability beyond the specific setting or population being studied [63]. Because the current analyses were based on data from a randomized controlled trial designed to ensure internal validity, greater resources were required to provide CAMS than would be observed in real-world settings. We attempted to mitigate potential overestimation of costs related to research procedures through application of some real-world practice assumptions to enhance external validity. Still, costs of real-world CAMS delivery may be substantially lower in some settings owing to the availability of less expensive training formats (e.g., online video courses; www.CAMS-care.com) and delivery formats (e.g., group therapy) [64], as well as its suitability for use by diverse types of providers (e.g., paraprofessionals) [13]. Conversely, costs may be greater in settings where outpatient mental health services are typically delivered by doctoral-level therapists. Additional cost-inclusive evaluations of CAMS across these various implementation approaches are needed to inform wider dissemination of the framework.
A second limitation of the current study is that is solely reflects the MHS perspective. While our findings may inform decision-making that maximizes impact for the payer, it is important to consider that such a narrow scope likely undermines the true benefit of an intervention [65]. Future cost-inclusive analyses of CAMS should consider adopting a broader range of perspectives, particularly a societal perspective in which all costs and outcomes associated with an intervention are taken into consideration regardless of whom experiences them directly [66,67]. For example, it is possible that effective treatment of suicidality may also produce improvements related to productivity, employment, crime, incarceration, and consumption that further enhance societal welfare [68]. Similar improvements may also extend to others impacted by the individual’s suicidality (e.g., family members, friends, colleagues, and clinicians). Indeed, one study found that indirect costs represented 97.1% of the $58.4 billion total U.S. economic costs of reported suicide-related acts in 2013 [69]. In light of research indicating that Soldiers may be more likely to attempt suicide if there has been a past-year suicide attempt within their unit [70], it is conceivable that effective treatment of suicidality might even have a multiplicative effect within a military population. Incorporating these broader societal considerations into future cost-inclusive evaluations of suicide interventions may enhance our understanding of their true value.
Third, in the original trial, study clinicians were assigned to their respective treatment conditions based on their preferred approach for managing suicidal patients. Although this approach maximized fidelity to study treatment, it should be noted that CAMS therapists inadvertently had more than twice the years of practice experience since professional degree than ETAU therapists. This finding is consistent with research suggesting that less-experienced therapists may be more inclined to adhere to a specific treatment protocol or newly learned therapy whereas more seasoned providers may be more open to a variety of approaches [71–73]. Previous research suggests that years of practice experience is not necessarily correlated with therapist effectiveness [74,75] and statistical modeling in the current study did not suggest any therapist effects on clinical outcomes. Further, differences in practice experience did not impact treatment costs between treatment conditions as all study therapists were classified as “non-physician” under TRICARE reimbursement procedures [76]. Still, some studies indicate that greater suicide-specific practice experience is associated with enhanced suicide intervention competence and skills [77,78], and thus future CAMS investigations would be strengthened by ensuring comparable practice experience among providers to remove this potential confounding.
Finally, due to limitations in the available data, the current study was unable to determine the exact temporal sequence of healthcare services delivered to participants within each follow-up window. It is therefore possible that some non-study-related services classified as outcomes (i.e., benefit) were actually delivered concurrently with active study treatment rather than post-treatment, as the designation of outcome would imply. Although this approach allowed for the greatest use of service utilization data while avoiding conflation with study treatment costs, future cost-inclusive evaluations would benefit from greater precision in the assessment of benefits.
Conclusions
Suicide prevention represents a top force health and readiness priority for the DoD. The CAMS framework has been increasingly recognized as a promising approach for the treatment of suicidal risk. Although CAMS has amassed a robust evidence base supporting its clinical effectiveness, the current study provides evidence that CAMS may also be associated with greater benefit and cost-benefit compared to treatment as usual. Although the differences observed in our study were small, the potential impact grows appreciably once extrapolated across the population of 1.38 million active duty service members (and 9.5 million beneficiaries total) served by the MHS [79]. These economic advantages may become more pronounced as CAMS’s more cost-effective training and delivery formats become more widely adopted. Ideally, such cost savings could allow limited financial resources to be re-allocated toward meeting other healthcare system needs. Although additional evaluations are necessary to enhance the generalizability of our findings, the current study provides a valuable insight into the costs and benefits of CAMS within the context of the MHS. These findings may inform more robust cost-inclusive analyses in future trials and ultimately facilitate decision-making among the policymakers, healthcare administrators, and clinicians tasked with selecting mental health programs that will provide the biggest ‘bang for the buck’ amidst constrained resources.
References
- 1.
Department of Defense Task Force on the Prevention of Suicide by Members of the Armed Forces. The Challenge and the Promise: Strengthening the Force, Preventing Suicide and Saving Lives. Washington, DC: Department of Defense. 2010.
- 2.
Under Secretary of Defense for Personnel and Readiness. Defense Suicide Prevention Program Washington, DC: Author. 2017 Nov 6.
- 3.
Psychological Health Center of Excellence, Defense Health Agency. Department of Defense Suicide Event Report (DoDSER): Calendar Year 2019 Annual Report. 2021 Mar 15. Available from: https://health.mil/Reference-Center/Publications/2021/07/06/2019-DoDSER-Annual-Report.
- 4.
Department of Defense, Under Secretary of Defense for Personnel and Readiness. Annual suicide report for calendar year 2019. 2020 Aug 20. Available from: https://www.dspo.mil/Portals/113/Documents/CY2019%20Suicide%20Report/DoD%20Calendar%20Year%20CY%202019%20Annual%20Suicide%20Report.pdf?ver=YOA4IZVcVA9mzwtsfdO5Ew%3d%3d.
- 5. Bachynski KE, Canham-Chervak M, Black SA, Dada EO, Millikan AM, Jones BH. Mental health risk factors for suicides in the US Army, 2007–8. Inj Prev. 2012;18(6):405–12. Epub 2012/03/09. pmid:22398362.
- 6. Hyman J, Ireland R, Frost L, Cottrell L. Suicide incidence and risk factors in an active duty US military population. Am J Public Health. 2012;102 Suppl 1:S138–46. Epub 2012/03/07. pmid:22390588; PubMed Central PMCID: PMC3496445.
- 7. Nock MK, Stein MB, Heeringa SG, Ursano RJ, Colpe LJ, Fullerton CS, et al. Prevalence and correlates of suicidal behavior among soldiers: results from the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). JAMA Psychiatry. 2014;71(5):514–22. Epub 2014/03/05. pmid:24590178; PubMed Central PMCID: PMC4100464.
- 8. Pruitt L, Smolenski D, Bush N, Tucker J, Issa F, Hoyt T, et al. Suicide in the Military: Understanding Rates and Risk Factors Across the United States’ Armed Forces. Mil Med. 2019;184(Suppl 1):432–7. Epub 2018/11/14. pmid:30423136.
- 9. Ursano RJ, Kessler RC, Stein MB, Naifeh JA, Aliaga PA, Fullerton CS, et al. Risk Factors, Methods, and Timing of Suicide Attempts Among US Army Soldiers. JAMA Psychiatry. 2016;73(7):741–9. Epub 2016/05/26. pmid:27224848; PubMed Central PMCID: PMC4937827.
- 10. Conner KR, Simons K. State of innovation in suicide intervention research with military populations. Suicide Life Threat Behav. 2015;45(3):281–92. Epub 2014/10/29. pmid:25348613.
- 11.
Jobes D. Managing suicidal risk: a collaborative approach. New York: Guilford Press; 2006.
- 12. Jobes D. The Collaborative Assessment and Management of Suicidality (CAMS): an evolving evidence-based clinical approach to suicidal risk. Suicide Life Threat Behav. 2012;42(6):640–53. Epub 2012/09/14. pmid:22971238.
- 13.
Jobes D. Managing suicidal risk: a collaborative approach (2nd ed.). New York, NY: Guilford Press; 2016.
- 14. Jobes D, Jacoby A, Cimbolic P, Hustead L. Assessment and treatment of suicidal clients in a university counseling center. Journal of Counseling Psychology. 1997;44(4):368–77.
- 15. Jobes D, Kahn-Greene E, Greene J, Goeke-Morey M. Clinical improvements of suicidal outpatients: Examining suicide status form responses as predictors and moderators. Arch Suicide Res. 2009;13(2):147–59. Epub 2009/04/14. pmid:19363751.
- 16. Jobes D, Wong S, Conrad A, Drozd J, Neal-Walden T. The collaborative assessment and management of suicidality versus treatment as usual: a retrospective study with suicidal outpatients. Suicide Life Threat Behav. 2005;35(5):483–97. Epub 2005/11/05. pmid:16268766.
- 17. Ellis TE, Green KL, Allen JG, Jobes DA, Nadorff MR. Collaborative assessment and management of suicidality in an inpatient setting: results of a pilot study. Psychotherapy (Chic). 2012;49(1):72–80. Epub 2012/03/01. pmid:22369081; PubMed Central PMCID: PMC3752846.
- 18. Ellis TE, Rufino KA, Allen JG. A controlled comparison trial of the Collaborative Assessment and Management of Suicidality (CAMS) in an inpatient setting: Outcomes at discharge and six-month follow-up. Psychiatry Res. 2017;249:252–60. Epub 2017/01/28. pmid:28126581.
- 19. Ellis TE, Rufino KA, Allen JG, Fowler JC, Jobes DA. Impact of a Suicide-Specific Intervention within Inpatient Psychiatric Care: The Collaborative Assessment and Management of Suicidality. Suicide Life Threat Behav. 2015;45(5):556–66. Epub 2015/01/13. pmid:25581595.
- 20. Arkov K, Rosenbaum B, Christiansen L, Jonsson H, Munchow M. [Treatment of suicidal patients: The Collaborative Assessment and Management of Suicidality]. Ugeskr Laeger. 2008;170(3):149–53. Epub 2008/01/23. pmid:18208732.
- 21. Nielsen AC, Alberdi F, Rosenbaum B. Collaborative assessment and management of suicidality method shows effect. Dan Med Bull. 2011;58(8):A4300. Epub 2011/08/11. pmid:21827722.
- 22. Andreasson K, Krogh J, Wenneberg C, Jessen HK, Krakauer K, Gluud C, et al. Effectiveness of Dialectical Behavior Therapy versus Collaborative Assessment and Management of Suicidality Treatment for Reduction of Self-harm in Adults with Borderline Personality Traits and Disorder: A Randomized Observer-blinded Clinical Trial. Depress Anxiety. 2016;33(6):520–30. Epub 2016/02/09. pmid:26854478.
- 23. Comtois KA, Jobes DA, S SOC, Atkins DC, Janis K, C EC, et al. Collaborative assessment and management of suicidality (CAMS): feasibility trial for next-day appointment services. Depress Anxiety. 2011;28(11):963–72. Epub 2011/09/29. pmid:21948348.
- 24. Jobes D, Comtois K, Gutierrez P, Brenner L, Huh D, Chalker S, et al. A Randomized Controlled Trial of the Collaborative Assessment and Management of Suicidality versus Enhanced Care as Usual With Suicidal Soldiers. Psychiatry. 2017;80(4):339–56. Epub 2018/02/22. pmid:29466107.
- 25. Pistorello J, Jobes DA, Gallop R, Compton SN, Locey NS, Au JS, et al. A Randomized Controlled Trial of the Collaborative Assessment and Management of Suicidality (CAMS) Versus Treatment as Usual (TAU) for Suicidal College Students. Arch Suicide Res. 2020:1–25. Epub 2020/04/11. pmid:32275480.
- 26. Ryberg W, Zahl PH, Diep LM, Landro NI, Fosse R. Managing suicidality within specialized care: A randomized controlled trial. J Affect Disord. 2019;249:112–20. Epub 2019/02/17. pmid:30771641.
- 27. Hanratty D, Kilicaslan J, Wilding H, Castle D. A systematic review of efficacy of Collaborative Assessment and Management of Suicidality (CAMS) in managing suicide risk and deliberate self-harm in adult populations. Australas Psychiatry. 2019:1039856219848832. Epub 2019/06/12. pmid:31184492.
- 28. Swift JK, Trusty WT, Penix EA. The effectiveness of the Collaborative Assessment and Management of Suicidality (CAMS) compared to alternative treatment conditions: A meta-analysis. Suicide Life Threat Behav. 2021. Epub 2021/05/18. pmid:33998028.
- 29.
Drummond M, Schulpher M, Claxton K, Stoddart G, Torrance G. Methods for the Economic Evaluation of Health Care Programmes (Fourth Edition). New York, NY: Oxford University Press; 2015.
- 30.
Yates BT. Cost-benefit and cost-effectiveness analyses in evaluation research. In: Wright JD, editor. International Encyclopedia of the Social & Behavioral Sciences (2nd ed), Vol 5. Oxford: Elsevier; 2015. p. 55–62.
- 31.
Yates BT. Analyzing costs, procedures, processes, and outcomes in human services: An introduction. Thousand Oaks, CA: Sage Publications; 1996.
- 32. Yates BT. Cost-inclusive evaluation: a banquet of approaches for including costs, benefits, and cost-effectiveness and cost-benefit analyses in your next evaluation. Eval Program Plann. 2009;32(1):52–4. Epub 2008/11/04. pmid:18977532.
- 33. McKibben JB, Fullerton CS, Herberman Mash HB, Nock MK, Naifeh JA, Kessler RC, et al. Suicidal behaviors and the use of mental health services among active duty Army soldiers. Psychiatr Serv. 2014;65(3):374–80. Epub 2013/12/03. pmid:24292523.
- 34. Trofimovich L, Skopp NA, Luxton DD, Reger MA. Health care experiences prior to suicide and self-inflicted injury, active component, U.S. Armed Forces, 2001–2010. MSMR. 2012;19(2):2–6. Epub 2012/03/01. pmid:22372750.
- 35. Ryan AT, Ghahramanlou-Holloway M, Wilcox HC, Umhau JC, Deuster PA. Mental Health Care Utilization and Psychiatric Diagnoses in a Sample of Military Suicide Decedents and Living Matched Controls. J Nerv Ment Dis. 2020;208(9):646–53. Epub 2020/06/06. pmid:32502074; PubMed Central PMCID: PMC7483990.
- 36. Beck AT, Brown GK, Steer RA. Psychometric characteristics of the Scale for Suicide Ideation with psychiatric outpatients. Behav Res Ther. 1997;35(11):1039–46. Epub 1998/02/12. pmid:9431735.
- 37. Linehan MM, Heard HL. Treatment History Interview (THI). Unpublished work. 1987.
- 38.
Gold MR, Siegel JE, Russell LB, Weinstein MC. Cost-effectiveness in Health and Medicine. New York, NY: Oxford University Press; 1996.
- 39.
Yates BT. Improving Effectiveness and Reducing Costs in Mental Health. Springfield, IL: Charles C Thomas; 1980.
- 40.
U.S. Office of Personnel Management. 2018 General Pay Schedule (Base). Available from https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2018/GS.pdf. 2018.
- 41.
U.S. Bureau of Labor Statistics. Private Industry Total benefits for South Atlantic; Percent of total compensation [Time Series] 2018. Available from: http://data.bls.gov.
- 42.
U.S. General Services Administration. City Pair Program 2018. Available from: https://www.gsa.gov/travel/plan-book/transportation-airfare-rates-pov-rates-etc.
- 43.
U.S. General Services Administration. GSA Lodging 2018. Available from: https://www.gsa.gov/travel/plan-book/gsa-lodging.
- 44.
U.S. General Services Administration. Per Diem Rates 2018. Available from: https://www.gsa.gov/travel/plan-book/per-diem-rates.
- 45.
LoopNet.com [Internet]. Search commercial property listings, zip code, select property type: "Medical" and “Office.” c2019 [cited 2019 Aug 12]. Available from: http://www.loopnet.com.
- 46.
U.S. Energy Information Administration. Electric Power Monthly. Washington, D.C.: U.S. Department of Energy; 2018.
- 47. Anestis MD, Green BA. The Impact of Varying Levels of Confidentiality on Disclosure of Suicidal Thoughts in a Sample of United States National Guard Personnel. J Clin Psychol. 2015;71(10):1023–30. Epub 2015/09/09. pmid:26347943.
- 48.
Bray R, Pemberton M, Hourani L, Witt M, Rae Olmsted K, Brown J, et al. 2008 Department of Defense Survey of Health Related Behaviors Among Active Duty Military Personnel: A Component of the Defense Lifestyle Assessment Program (DLAP) Research Triangle Park, NC: RTI International. Available from: http://www.ncpgambling.org/wp-content/uploads/2019/01/2008-DoD-Study-risk-taking-section.pdf.; 2009.
- 49. Hill JV, Johnson RC, Barton RA. Suicidal and homicidal soldiers in deployment environments. Mil Med. 2006;171(3):228–32. Epub 2006/04/11. pmid:16602522.
- 50. Valuck RJ, Anderson HO, Libby AM, Brandt E, Bryan C, Allen RR, et al. Enhancing electronic health record measurement of depression severity and suicide ideation: a Distributed Ambulatory Research in Therapeutics Network (DARTNet) study. J Am Board Fam Med. 2012;25(5):582–93. Epub 2012/09/08. pmid:22956694.
- 51.
United States Government Accountability Office. Defense Health Care: Additional Information Needed about Mental Health Provider Staffing Needs (GAO-15-184). Available from: https://www.gao.gov/assets/670/668208.pdf. 2015 Jan 30.
- 52.
Hosek J, Nataraj S, Mattock M, Asch B. The Role of Special and Incentive Pays in Retaining Military Mental Health Care Providers. Santa Monica, CA: RAND Corporation. Available from: https://www.rand.org/pubs/research_reports/RR1425.html.; 2017.
- 53.
Crowley K. Collaborative assessment and management of suicidality (CAMS): Adherence to a flexible clinical framework [dissertation]. Washington (DC): The Catholic University of America; 2015.
- 54.
American Medical Association. CPT 2018: Standard Edition: American Medical Association; 2017.
- 55. Mieloszyk RJ, Rosenbaum JI, Hall CS, Raghavan UN, Bhargava P. The Financial Burden of Missed Appointments: Uncaptured Revenue Due to Outpatient No-Shows in Radiology. Current Problems in Diagnostic Radiology. 2018;47(5):285–6. Epub 2018/06/23. pmid:29929841.
- 56. Ramsey SD, Willke RJ, Glick H, Reed SD, Augustovski F, Jonsson B, et al. Cost-effectiveness analysis alongside clinical trials II-An ISPOR Good Research Practices Task Force report. Value Health. 2015;18(2):161–72. Epub 2015/03/17. pmid:25773551.
- 57. Bodner TE. What Improves with Increased Missing Data Imputations? Structural Equation Modeling: A Multidisciplinary Journal. 2008;15(4):651–75.
- 58. White IR, Royston P, Wood AM. Multiple imputation using chained equations: Issues and guidance for practice. Stat Med. 2011;30(4):377–99. Epub 2011/01/13. pmid:21225900.
- 59.
Rubin DB. Multiple Imputation for Nonresponse in Surveys. New York, NY: John Wiley & Sons, Inc.; 1987.
- 60. Donders AR, van der Heijden GJ, Stijnen T, Moons KG. Review: a gentle introduction to imputation of missing values. J Clin Epidemiol. 2006;59(10):1087–91. Epub 2006/09/19. pmid:16980149.
- 61. Hoch JS, Smith MW. A guide to economic evaluation: methods for cost-effectiveness analysis of person-level data. J Trauma Stress. 2006;19(6):787–97. Epub 2007/01/02. pmid:17195977.
- 62.
Ghosh D, Vogt A. Outliers: an evaluation of methodologies. Joint Statistical Meetings Proceedings; San Diego, CA: American Statistical Association; 2012. p. 3455–60.
- 63. Xu X, Grossetta Nardini HK, Ruger JP. Micro-costing studies in the health and medical literature: protocol for a systematic review. Systematic Reviews. 2014;3:47. Epub 2014/06/03. pmid:24887208; PubMed Central PMCID: PMC4036677.
- 64. Johnson LL, O’Connor SS, Kaminer B, Jobes DA, Gutierrez PM. Suicide-Focused Group Therapy for Veterans. Military Behavioral Health. 2014;2(4):327–36.
- 65. Polimeni J, Vichansavakul K, Iorgulescu R, Chandrasekara R. Why Perspective Matters In Health Outcomes Research Analyses. International Business & Economics Research Journal. 2013;12(11):1503–12.
- 66. Byford S, Raftery J. Perspectives in economic evaluation. BMJ. 1998;316(7143):1529–30. Epub 1998/06/06. PubMed Central PMCID: PMC1113167. pmid:9582152
- 67. Neumann PJ. Costing and perspective in published cost-effectiveness analysis. Med Care. 2009;47(7 Suppl 1):S28–32. Epub 2009/06/19. pmid:19536023.
- 68. Walker S, Griffin S, Asaria M, Tsuchiya A, Sculpher M. Striving for a Societal Perspective: A Framework for Economic Evaluations When Costs and Effects Fall on Multiple Sectors and Decision Makers. Appl Health Econ Health Policy. 2019;17(5):577–90. Epub 2019/05/18. pmid:31098947; PubMed Central PMCID: PMC6748888.
- 69. Shepard DS, Gurewich D, Lwin AK, Reed GA Jr., Silverman MM. Suicide and Suicidal Attempts in the United States: Costs and Policy Implications. Suicide Life Threat Behav. 2016;46(3):352–62. Epub 2015/10/30. pmid:26511788; PubMed Central PMCID: PMC5061092.
- 70. Ursano RJ, Kessler RC, Naifeh JA, Herberman Mash H, Fullerton CS, Bliese PD, et al. Risk of Suicide Attempt Among Soldiers in Army Units With a History of Suicide Attempts. JAMA Psychiatry. 2017;74(9):924–31. Epub 2017/07/27. pmid:28746705; PubMed Central PMCID: PMC5674988.
- 71. Bruijniks SJE, Franx G, Huibers MJH. The implementation and adherence to evidence-based protocols for psychotherapy for depression: the perspective of therapists in Dutch specialized mental healthcare. BMC Psychiatry. 2018;18(1):190. Epub 2018/06/15. pmid:29898692; PubMed Central PMCID: PMC6000963.
- 72. Monahan MF, Crowley KJ, Arnkoff DB, Glass CR, Jobes DA. Understanding Therapists’ Work With Suicidal Patients: An Examination of Qualitative Data. Omega (Westport). 2020;81(2):330–46. Epub 2018/05/11. pmid:29745779.
- 73. Wogan M, Norcross JC. Dimensions of therapeutic skills and techniques: Empirical identification, therapist correlates, and predictive utility. Psychother Theor Res Pract Train. 1985;22(1):63–74.
- 74. Goldberg SB, Rousmaniere T, Miller SD, Whipple J, Nielsen SL, Hoyt WT, et al. Do psychotherapists improve with time and experience? A longitudinal analysis of outcomes in a clinical setting. J Couns Psychol. 2016;63(1):1–11. Epub 2016/01/12. pmid:26751152.
- 75. Miller SD, Hubble MA, Chow DL, Seidel JA. The outcome of psychotherapy: yesterday, today, and tomorrow. Psychotherapy (Chic). 2013;50(1):88–97. Epub 2013/03/20. pmid:23505984.
- 76.
Defense Health Agency. TRICARE Reimbursement Manual (TRM) 6010.61-M. Available from: http://manuals.tricare.osd.mil/pages/DisplayManual.aspx?SeriesId=TR15. 2015.
- 77. Scheerder G, Reynders A, Andriessen K, Van Audenhove C. Suicide intervention skills and related factors in community and health professionals. Suicide Life Threat Behav. 2010;40(2):115–24. Epub 2010/05/15. pmid:20465346.
- 78. Jahn DR, Quinnett P, Ries R. The influence of training and experience on mental health practitioners’ comfort working with suicidal individuals. Prof Psychol Res Pr. 2016;47(2):130–8.
- 79.
Military Health System. Patients by Beneficiary Category. Available from https://www.health.mil/I-Am-A/Media/Media-Center/Patient-Population-Statistics/Patients-by-Beneficiary-Category. 2020.