Diagnostic screening vs risk screening methods

Although both the military and VA screen for common mental health conditions in primary care [9, 10], only a minority of veterans who screen positive receive adequate mental health care [11, 12]. In one study of veterans screened in a primary care system that integrated behavioral health care with primary medical care and employed a proactive referral system, less than half of veterans who screened positive for depression and/or PTSD attended mental health intake sessions or received any psychotherapy in the following 18 months [13]. Poor predictive performance of mental health screens and disconnection between perceived needs and the focus of screening may both be contributing to this lack of engagement in mental health care.

A systematic review of research on post-deployment mental health screening that uses a diagnostic model has found high variability in sensitivity of screens (.46 to .83) and low to moderate treatment initiation rates following screening [10]. This means that positive diagnostic screens are not consistently accurate in identifying those in need of care. Many with positive screens may recover on their own without mental health care. About 50% of those who screen positive for PTSD, major depression, alcohol misuse, or other mental health problems when they return from deployment to a war zone appear to recover without treatment within three to six months [3]. Thus, depending on the timing and circumstances of screening, positive diagnostic screening for current mental health problems may not be a good indicator of risk for persisting mental health problems.

Another contributor to veterans disinclination to accept mental health services may be that information given to them about their current status in of a mental illness diagnosis does not seem relevant or actionable. Among many systemic and individual factors that can be barriers to care [14–20], veterans’ reluctance to engage in mental health care may be the greatest. Qualitative and quantitative research has found that many veterans expect their problems to resolve without treatment, have a strong desire to solve problems without help, and hold beliefs that discourage engagement in care [21, 22]. Specifically, many have general attitudes of mistrust of others and negative attitudes about mental illness diagnostic labels, about standard models of mental health care offered by VA and about those who develop mental health problems and those who seek care [12, 16, 22, 23]. Information about current symptoms of a mental illness diagnosis may not be sufficient to overcome these barriers and persuade some veterans that there is a pressing need to engage in mental health care.

Screening veterans for risk of future mental health problems may address predictive accuracy and relevance problems associated with diagnostic screening. Information about risk for later mental health problems may seem more relevant and actionable to veterans than current diagnostic information. Those at risk could be told that veterans with risk screen scores like theirs did not recover on their own. Research has shown that risk information about outcomes of concern to patients can play a critical role in determining health-related behaviors. For example, in a large RCT of primary care patients, participants who completed a screen for cancer risk and received generic risk information or information about their personal colorectal cancer risk subsequently completed colorectal cancer medical tests at three times the rate of those who did not complete a risk screen and received no risk information [24]. Our success in developing a multi-risk mental health screening tool for hospital patients that performed well to predict later mental health problems [25] encouraged us to pursue a similar approach for veterans who were primary care patients.

Theory and research on risk factors for posttraumatic and postdeployment mental health problems in veterans

This study was based on theoretical frameworks for the impact of traumatic stress [26, 27] and deployment stress [28] and research on risk factors for PTSD and depression following traumatic stress [29]. The etiologies of post-traumatic and post-deployment mental health problems are complicated because of overlap in exposure to trauma and deployment and because a large number of factors affect people’s responses to both types of experiences. These factors include characteristics of individuals (such as genetic or biological tendencies, developmental level and experiences, past trauma exposure, life stress, and gender), characteristics of stressors (such as severity, intentional or accidental nature, and duration) and time-of-trauma factors (such as social support), and posttraumatic life experiences, responses, and resources (such as social support, availability and use of medical and psychological treatment, and posttraumatic life stress) [26, 27, 29]. These factors may influence responses to potentially traumatic stressors by affecting perceptions of stressful events, expectations about the impact of events, the capacity to cope with strong emotions related to events, or some combination of these.

Military personnel and veterans are subject to the same risk factors as those exposed to other types of traumatic stressors, and also to risks associated with serving in a war zone or deployment to other assignments inside or outside the country. Most research on veterans has focused on combat-related experiences, but other aspects of deployment may also influence health and mental health outcomes [28, 30]. These include perceived threat, aftermath of battle, difficult living and working environment, sense of preparedness, exposure to toxic agents, concerns about life and family disruptions, sexual or general harassment, social support during and after deployment, and life stress following deployment. Moral injury has also been identified as deleterious to mental health in veterans [31–35]. Moral injury has been defined as psychological damage caused by transgressions of deeply held moral or ethical beliefs [33]. More recently, moral injury has been defined as disturbances in virtues, character, and identity that follow a moral failure event, and it has been posited that the pathways of influence of moral injury and traumatic stress are independent [36]. Because traumatic stress and moral injury events often co-occur, the experience of both may be especially damaging. To date, moral injury research has focused on veterans who were treatment-seeking, so little is known about the incidence of moral injury in other populations or its value as a prospective predictor of later mental health problems.

The quality of relationships with partners, family members, friends, and co-workers during and after military deployment can also influence mental health. Military personnel deployed in recent years often served longer in war zones and have been deployed for multiple tours more often than prior military cohorts [11]. These extended periods of time apart and experiences of war zone stress and traumatic stress make the transition from deployments to the homefront stressful. Veterans of Iraq and Afghanistan have reported high levels of stress in marital, family, and work relationships during and after military service [37, 38], but there has been little examination of the predictive value of reports of stress in relationships related to deployment or transition to civilian life and later mental health.

Development and cross-validation of a veterans mental health risk screen in current project

In the current project, we sought to create a screen for risk of later mental health problems in veterans seeking primary care in a VA clinic and to perform cross-validation by examining the predictive performance of selected risk factor items in a new sample [39]. We focused on risk factors that are thought to play causal roles in the development of mental health problems in order to identify factors that are clinically relevant and modifiable. In the Screen Development Study, we conducted secondary analyses of data from prior research to identify which risk factors are most strongly related to PTSD and/or depression and to select small subsets of items to measure these risk factors. In the Cross-Validation Study, we collected data in a new sample of VA primary care patients (VA Primary Care II) on the selected risk factor items and on PTSD and/or depression six months later, and we calculated the performance of the risk screen results to prospectively predict these mental health problems [40, 41].

Gulf War veterans participants, procedures and measures

Data from Gulf War veterans were collected as part of a large national study that had the primary aim to develop a measure of deployment risk and resilience [44]. Potential participants were identified through records held by the Defense Manpower Data Center (DMDC), a central repository for Department of Defense personnel data and the VA Gulf War Health Registry. These veterans were deployed to the 1990–1991 Gulf War conflict (Gulf War I), and the data used were collected approximately ten years after deployment. Complete data were obtained from 317 (64%) of 495 who were sent questionnaires by mail. A detailed description of the study methods is provided in King (2006). Risk and resilience factors were assessed using the Deployment Risk and Resilience Inventory (DRRI) [44]. The DRRI is an collection of 14 scales for assessing predeployment, deployment, and postdeployment psychosocial factors that put veterans at risk for poor post-deployment health and adjustment. Posttraumatic stress symptoms were assessed with the military PTSD Checklist (PCL-M) [45].

Iraq and Afghanistan (IA) military personnel participants, procedures and measures

Data from military personnel recently returned from deployment to Iraq or Afghanistan were collected as part of a study of mental and physical health status and health services use [42]. Potential participants were identified through the DMDC. A sample of 2,000 / (OEF/OIF) personnel who had returned from deployment to Iraq (Operation Iraqi Freedom) or Afghanistan (Operation Enduring Freedom) between 3 and 12 months prior was stratified by service component (50% Active, 25% National Guard, and 25% other Reserve) and gender (>50% women within each service). Of 1,833 eligible individuals, 1,043 (57%) received the initial survey, and 596 (57%) returned completed initial (T1) surveys in the time frame allowed. Survey respondents came from every state except Wyoming and from the District of Columbia, Puerto Rico, and the US Virgin Islands. Follow-up data (T2) were collected from 512 veterans (86% of the 596 who completed the T1 survey) between six and nine months following receipt of the T1 survey [46]. Risk and resilience factors were assessed at T1 with the DRRI [44] (described above). Symptoms at T1 and T2 were assessed with the PCL-M (described above) and the Behavior and Symptom Identification Scale (BASIS-24) [47]. The BASIS-24 uses 24 items to assess six symptom domains: depression/functioning, interpersonal relationships, psychotic symptoms, alcohol/drug use, emotional lability, and self-harm.

Traumatic injury survivors participants, procedures and measures

Data from traumatic injury survivors were collected as part of research on psychological responses in patients hospitalized after severe, sudden injury and family members of severely injured patients [25]. Most risk factors, including early symptoms, were assessed upon enrollment between one and fourteen days after injury in 147 participants (54% patients and 46% family members). Social support and social constraints were assessed between eight and twenty-one days after injury in 64 participants. PTSD symptoms assessed two months post-injury were available for 129 participants. There were no significant differences between patients and family members on any risk factor or any outcome. PTSD was assessed with the Screen for Posttraumatic Stress Symptoms (SPTSS) [48, 49]. In psychiatric inpatients, scores on the SPTSS showed strong internal consistency (α = 0.91) and good concurrent validity when correlated with other measures of PTSD [48]. In a study of recent combat veterans with a 15% rate of PTSD, scores on the SPTSS had a sensitivity of 0.89 and specificity of 0.89 to predict PTSD diagnosis from the SCID PTSD module [49].

VA residential veterans participants, procedures and measures

Data were collected from 240 Vietnam War veterans participating in a U.S. Veterans Health Administration residential treatment program for chronic and severe PTSD related to combat or other trauma exposure during military service. Data were collected at the time of admission as part of research on the psychometric properties of a measure of dissociation [50]. Data included in this study were scores on the PCL-M (described above) and the Dissociative Symptoms Scale (DSS). The DSS assesses dissociative symptoms, which are distortions in perceptions, attention, and memory that are strongly related to exposure to traumatic stress and to PTSD symptoms [50]. Scores on the DSS in this sample showed strong reliability and validity in Carlson et al. [50].

VA homeless domiciliary residents participants, procedures and measures

Data were collected from 115 veterans participating in a U.S. Veterans Health Administration residential program for homeless veterans. Data were collected as part of research on military and civilian trauma exposure and posttraumatic symptoms [51]. Data included in this study were scores on the PCL-M and the DSS (both described above). Findings of strong reliability and validity for scores on the DSS in this sample are reported in Carlson et al. [50].

VA primary care (I) participants, procedures, and measures

Data were collected from 131 VA primary care patients at a VA medical center. In order to limit the potential for errors in reports of past experiences due to the passage of time, inclusion was limited to veterans who were within five years of military discharge. Appointments were identified in the electronic medical records for primary care, laboratory services, audiology, dental, optometry, orthopedics, gastro-intestinal, and hand and upper extremity. While in the waiting area, veterans who did not opt out were invited to participate and provided informed consent. Data reported here included demographics, employment, and military service.

Stress in relationships related to deployment experiences or transition to civilian life was assessed in the 118 veterans who had been deployed during military service using a measure of stress in relationships called the Deployment Transitions Stress Scale (DTSS) that was developed to quantify stress in marital/partner and family relationships related to deployment and return from deployment and about changes in relationships with friends and in the work place since deployment. The DTSS was based on the Transitioning Families Questionnaire, an unpublished clinical interview authored by Erika Curran that was used to collect relationship information from VA residential PTSD treatment program patients and their spouses. The DTSS includes items focused on stress in the relationship with a spouse or partner, family members, friends, and work. Example items include “Compared to before deployment, how many disagreements do you and your partner have now? (fewer, about the same, somewhat more, a lot more); “Since returning, how hard is it to talk or answer questions about your war experiences with your children or other family members?” (not at all hard, a little bit hard, somewhat hard, very hard); “How much have you felt alienated from your friends since returning from deployment?” (not at all, a little bit, somewhat, very much); “How hard has it been for you to get along with coworkers in a new workplace since returning from deployment?” (not at all, a little bit, somewhat, very hard). Data on Relationship Stress: Spouse/Partner were available for the 32 veterans who were still in relationships with the same spouse/partner as at the time of deployment. Data on Relationship Stress: Friends were available for N = 104 of 118 who had deployed during military service. Cronbach’s alpha value for the total DTSS was 0.93 and values for the subscales were 0.76 (spouse/partner), 0.64 (family); 0.64 (friends), and 0.82 (work).

The SPTSS (described above) was used to assess DSM-IV PTSD symptoms.

Data analysis

We used common methods for shortening scales [52–54] to select subsets of items with the goal of creating very brief risk factor measures that could predict mental health outcomes. Creating short forms of hypothetical constructs that produce scores with strong psychometric properties typically involves shortening measures of 30 or more items to about 12 items [53]. However, a screen made up of a collection of typical short forms would be too long to use for screening in health care settings. Further, since some of the most predictive risk factors (e.g., PTSD symptoms) are multifactorial [55], combining short forms that assessed all of the underlying factors of these multi-factor risks would likewise produce a multi-risk screen that was too long for practical use. For these reasons, we did not seek to create brief measures of the entire content domains of risks or produce scores with psychometric properties strong enough to serve as independent measures of the constructs. Instead, we sought to identify a set of risks that could be assessed with relatively few items and demonstrate good predictive validity of future mental health status.

Some risk factor variables we studied are hypothetical constructs (e.g., social support), whereas others are composite measures made up of items that represent discrete, possibly uncorrelated experiences (e.g., combat experiences) [56]. To select items for brief measures of risk factors that are hypothetical constructs, we examined item-total correlations and favored items that were most highly correlated with total scores [52]. We also conducted factor analyses to examine whether risk measures included only one or multiple factors, and eliminated lower-loading items by factor. To shorten measures of risk factors that are composite variables, we relied primarily on removing items that related less strongly to total scores. For all risk factors (constructs and sets of experiences), in order to maximize criterion-related validity with respect to the original full scales, we conducted forward regressions with items as predictors of the score for the full risk factor measure and retained enough items to account for at least 90% of the variance in the full measure score. We used data from the IA Military sample to select items for brief measures of some deployment-related variables rather than data from the Gulf War sample because the former was a much larger sample and the data were collected prospectively and over a comparable time frame between risks and PTSD symptoms to be predicted.

Participants and procedures

Patients were recruited for the cross-validation study using the same methods and procedures as described above in the VA Primary Care I sample. At the time of enrollment, participants completed risk factor measures. Six months following enrollment, participants completed mental health measures and reported on major life stressors since enrollment.

Measures

The risk factors identified in the Screen Development Study (shown in Table 3) were assessed with the items selected in the Screen Development Study. To make them applicable to all veterans, simplify wording, and combine very similar constructs, some items were modified. For example, items originally focused on deployment were reworded to “During military service,” and items originally specifying spouses or partners were modified to apply to those who did not have partners or spouses (e.g., refer to “your relationships” instead of “partner or spouse”). For all items, a simple, uniform set of response options was used of “not at all”, “a little bit”, “some”, “a lot” with risk factor scores coded to have higher scores reflect less favorable experiences or higher levels of symptoms.

Moral injury was also assessed with four novel items developed in consultation with four experts on moral injury. All members of the expert panel had doctoral level training in psychiatry or psychology and lived experience of military service and had published and presented on the concept of moral injury. After discussion about the construct of moral injury, four items were developed to assess the construct: “been bothered about things that happened during military service that were just not right”; “felt bad about yourself because of things you saw or did during military service”; “felt guilty about things you did or didn’t do while in the military”; and “been bothered by killing people during military service”. When administered, items were preceded by “In the past month, how much have you…” and response options were “not at all”, “a little bit”, “some”, “a lot”.

The SPTSS (described above) was used to assess PTSD symptoms six months after enrollment. SPTSS scores of 20 or higher were used to categorize participants as having elevated PTSD symptoms. This was based on strong performance of scores on the SPTSS to predict PTSD diagnosis by the Structured Clinical Interview for DSM-IV (SCID) in a sample of veterans (N = 317; SE = .89; SP = .89) [49] and by performance of scores of 20 or more on the SPTSS to predict PTSD diagnosis on the Clinician-Administered PTSD Scale (CAPS) in a sample of adults assessed two months after traumatic injury of themselves or a close family member (N = 40, SE = .90, SP = .80) [40].

The Patient Health Questionnaire—9 (PHQ-9) was used to assess depression symptoms over the past two weeks. The PHQ-9 has 9 items with 4 response options that range from “not at all” to “nearly every day”. The PHQ-9 has strong internal consistency (α = .89), test-retest reliability (r = .84), and correlations with other measures of depression severity and excellent SE (.95) and SP (.84) to a depression diagnosis [59, 60]. PHQ-9 scores of 10 or more were used to categorize participants as having elevated depression based on Veterans Health Administration/ Department of Defense guidelines for assessment of depression in primary care [61].

To identify veterans who experienced new traumatic stressors between the time of enrollment and the follow-up assessment six months later, the follow-up assessment included the question “Since you completed the first part of this study (about 6 months ago), have you had any major life events or new major stressors?” and descriptions of the events/stressors were collected for those who answered “yes”. These descriptions were used to identify veterans who had been exposed to potentially traumatic stressors.

Data analysis

Potentially traumatic stressors and multiple major stressors experienced in the six months following enrollment were expected to increase mental health symptoms and affect accuracy of prediction for veterans expected to have no disorder at six months. Therefore, data were excluded from analyses for four participants whose screen results predicted no disorder at six months and who reported major stressors between enrollment and follow-up that were considered potentially traumatic. All analyses reported here were conducted on the 228 participants who completed the follow-up and reported no new major life events or stressors that were considered likely to be traumatic stressors.

The sum of standardized scores on the SPTSS and the PHQ-9 was used as an index of mental health that reflected symptoms of both PTSD and depression. To determine the subset of risk factors that most strongly predicted elevation in combined PTSD and depression symptoms six months later, we conducted a forward regression to predict the index of mental health. The statistical software analyzed the variance in the outcome (index of mental health six months after screening) associated with all risk factors listed in Table 3 and added risk factors to the model in a stepwise fashion. At each step, the risk factor was added that provides the best improvement to the model and contributes significant additional variance with set at p < .05. We conducted a Receiver Operating Characteristic (ROC) analysis to identify a cut score on the subset of risk factors that maximized SE to predict mental health status without letting SP fall below 0.70. We examined the performance of the cut score to predict elevated mental health symptoms six months later.