Introduction

Every year in the U.S., 16 to 18 million adults are admitted to hospitals after receiving emergency medical care [1–3] and are at risk for mental health problems. In injured patients, rates of depression, anxiety disorders, and/or posttraumatic stress disorder (PTSD) up to 34% have been reported [4–9]. We have recently reported comparable rates of mental health problems in patients hospitalized with acute illnesses [10]. Research in the U.S. has shown that most who experience traumatic stress never receive mental health treatment [11], and treatment delay after onset (or first diagnosis) for those who do receive care was 5 to 8 years for those with depression and 12 to 15 years for those with PTSD [11, 12]. Screening hospital patients admitted after emergency care to identify at-risk patients could reduce treatment delays and enable preventive care that could reduce or prevent mental health problems.

To address mental health risks in patients after trauma, the American College of Surgeons Committee on Trauma has set a standard for Level 1 and 2 trauma centers to screen patients for mental health risk [13]. Measures of PTSD, depression, and anxiety symptoms have been developed and validated to identify patients who have PTSD, major depression, or anxiety disorders at the time of the assessment, but these measures were not designed to predict future mental health problems and have not shown great promise as screening tools for mental health risk. Specifically, when given to hospitalized injury patients, the 20-item Posttraumatic Stress Checklist has shown mixed results to predict later PTSD symptoms, with sensitivities to predict PTSD one month later ranging from poor at .38 [14] to low at .67 [15], poor SE to predict PTSD 4 months later of .43 [14], and low to good SE to predict PTSD 6 months later of .69 [16] and .77 [15]. When given to injury patients during hospitalization, the 4-item Primary Care PTSD Screen (PC-PTSD) and the 8-item Patient Health Questionnaire both had poor sensitivity (SE = .47) to predict high PTSD symptoms six months later [17]. These findings are consistent with past research which has found that symptoms assessed in the days following exposure to traumatic stress do not accurately predict who will develop later mental health problems [18]. Therefore, our approach has been to identify which patients are at high risk for a mental health condition. The distinction between screens for presence and screens for high risk is common in medicine. For example, a mammogram is used to screen for presence of breast cancer, whereas risk assessment tools such as the Gail model are used to screen for high risk of breast cancer [19].

Screening tools have been created for injury patients, but performance of only a few of these have been studied in new samples beyond the sample used to develop the screen [20]. Replication of findings is essential in the development of predictive screens. Initial studies of screens can yield positive results but fail to replicate because the items were strongly related to outcomes in the development study sample but not in a new sample from the population the screen was intended for.

Given that patients hospitalized with acute illness have mental health risk comparable to that of injury patients [10], we included these patients in our study to develop the Hospital Mental Health Risk Screen (HMHRS). Similarly, given ethnoracial disparities in risk factors and in mental health outcomes after hospitalization [21, 22], in our screen development research we included sufficient numbers of patients who identified as Asian American/Pacific Islander (AAPI), Black, Latinx, Multirace, or White to inform selection of risks and items and to examine performance within the ethnoracial subgroups studied. In contrast, other screen development samples have not included patients with acute illness nor sufficient numbers of patients from diverse ethnoracial identities to inform item selection or to examine screen performance within ethnic/racial subgroups of patients [20]. Lastly, we included patients whose preferred language was Spanish or Chinese.

In the study to develop the HMHRS, data on a wide range of posttraumatic mental health risk factors were assessed in 1,320 adults admitted after emergency care for injury or acute illness in hospitals in California, Ohio, and Maryland that had Level 1 trauma centers [23]. Data from these patients who were diverse in gender, age, socioeconomic status, and ethnoracial identity was analyzed to select the most predictive risk factors, the most predictive items to assess those risk factors, and the fewest items to accurately predict later mental health. We used the total symptom burden to be predicted rather than diagnostic status because of the advantages of dimensional measurement of psychopathology and the significant limitations of traditional categorical diagnostic taxonomies [24, 25]. Specifically, PTSD, depression, and anxiety disorders are highly comorbid with about 50% of those with PTSD also meeting diagnostic criteria for depression and 20% also meeting diagnostic criteria for anxiety disorder [26]. When the goal of screening is to identify those at risk for mental health problems, a focus on diagnosis may not serve patients who have symptoms from different diagnostic categories that collectively warrant clinical attention.

In the overall sample and within ethnoracial subgroups, scores on 10 items accurately identified 75% of patients who had high levels of depression, anxiety, and PTSD symptoms two months later (sensitivity = .75) [27]. Sensitivity (SE) in ethnoracial identity groups ranged from .73 in patients who identified as White to .86 in Spanish speaking patients who identified as Latinx or Hispanic. The current study replicates the predictive performance in a new sample, which is essential to confirm that the HMHRS can reliably perform across different samples of patients.

Methods

Results

We enrolled 631 patients who self-identified as one or more of ethnoracial groups (Table 1). Patients ranged in age from 18 to 87 years (mean = 50.03, SD = 16.14), and 90.4% of those who reported an identity of Latinx reported only that identity. The mean number of days between admission and study enrollment was 3.5 (SD = 3.1) with 85% of patients enrolled within 5 days of admission.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Characteristics of 631 patients.

https://doi.org/10.1371/journal.pone.0311256.t001

The overall follow-up rate was 72.4% (457 of 631). Analyses including follow-up data were conducted on N = 452 as there was missing data for some mental health measures at follow-up for five patients. There was no difference in retention of patients who scored as “at-risk” on the HMHRS (10 or higher; 70.3% retained) and those who scored as “low-risk” (74.0%). Similarly, there was no significant difference in HMHRS scores between those who did and did not complete the follow-up. There were differences among the five ethnoracial groups in retention, Χ² (4) = 33.68, p < .001. The retention rates by ethnoracial subgroups were: 87% for AAPI, 59% for Black, 75% for Latinx, 82% for Multirace/AI/AN, and 81% for White.

HMHRS scores ranged from 0 to 41. Mean score was 10.7 (SD = 8.83), median score was 9. The percent of each ethnoracial subgroup scoring 10 or more on the HMHRS was 48% for AAPI, 52% for Black, 41% for Latinx, 58% for Multirace, and 42% for White. However, there were no significant differences between any of the ethnoracial groups in mean score.

Table 2 shows the values in the overall sample, in injury patients, in acute illness patients, and by ethnoracial identity for Area Under the Curve (AUC), sensitivity (SE), specificity (SP), positive predictive value (PPV), and negative predictive value (NPV) and 95% confidence intervals for SE, SP, PPV, and NPV [39]. SE is the proportion of those who later had elevated Mental Health Symptoms that the screen classified as having high risk. SP is the proportion who did not later have elevated Mental Health Symptoms that the HMHRS classified as low risk. PPV is the proportion of all positive scores that were true positive scores. NPV is the proportion of all negative scores that were true negative scores.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Performance of an HMHRS score of ten or more to predict high mental health symptoms at follow-up by ethnoracial subgroup.

https://doi.org/10.1371/journal.pone.0311256.t002

Discussion

In this replication study, performance of the HMHRS to identify patients who reported elevated levels of mental health symptoms at follow-up was strong, with 79% of these patients correctly identified. HMHRS scores also correctly identified 72% of the patients who did not report high levels of mental health symptoms at follow-up. Notably, sensitivity and specificity were also very good to excellent within all the ethnoracial subgroups we studied. PPVs for the ethnoracial subgroups indicate that the risk for those who screen positive is at a meaningful level, and high scores provide valuable information. NPVs for the ethnoracial subgroups were quite high, indicating that a low score on the screen is a strong indicator that the patient is at low risk for later mental health problems.

Performance overall and for most of the subgroups was superior to performance in the development study, most likely due to higher retention of at-risk patients in this study. In the prior study, 53.4% of patients scoring as “at-risk” on the screen were retained, which was significantly lower than the retention rate of 68.4% for “low-risk” patients [23]. In this study, 70.3% of at-risk patients were retained, and there was no difference in retention between low-risk and high-risk patients. Similarly, HMHRS scores were significantly higher in those lost to follow-up than in those retained in the development study, but not in this study. Screen performance is better with better retention of at-risk patients because those with higher risk scores are easier to identify as they tend to have higher symptoms at follow-up.

Reasons for higher retention and better retention of at-risk patients in the replication study are important to note. In both studies, we used follow-up methods aimed at maximizing retention [31]. These included collecting follow-up data via each patient’s preferred modality and repeated attempts to reach each patient for follow-up. In the replication study, we increased the number of attempts to reach each patient and had research assistants who were fluent in Spanish or Mandarin make follow-up calls to patients in their preferred language. We also used methods during recruitment to increase the chances of reaching patients for follow-up, such as giving patients a study timeline document with the follow-up date noted on it and having patients store the study phone number as a contact on their cell phone if they wished to be contacted via text.

Unfortunately, retention for Black patients in this study was lower than for other ethnoracial groups. Challenges to retention at one site included interruptions in staffing due to the pandemic and staff attrition, being unable to reach patients due to phones no longer in service, and a high proportion of patients with violent injuries (48% of injured patients). Patients with more severe injuries may have more difficulty completing follow-up measures.

Retention was higher for Black patients in this study than in the development study (59% vs 50%). This improvement may have resulted from changing one of our study sites to a historically Black college/ university (HBCU). At the new HBCU site, the staff had established connections and liaisons with community members which likely enhanced their ability to recruit and retain these patients. Lower retention of Black patients is common in medical research. One major reason for this is low levels of trust in medical researchers, which is largely attributed to past mistreatment of Black patients in medical studies, including being subjected to dangerous and unethical research procedures without their consent [40]. Other reasons for Black patients’ reluctance to participate in medical research include competing demands for time, concern about stigma, and concern about discrimination by health insurance companies [41]. Research conducted in hospitals associated with HBCUs may fare better in recruiting and retaining Black patients as research participants.

There are no prior studies to compare our findings to as no screens have been developed for or tested in patients with acute illness. In addition, no prior studies have included patients that reflect the full range of ethnoracial identity diversity of the U.S. and none have reported on their performance within ethnoracial groups. Sensitivity of the HMHRS compares favorably to that of the Injured Trauma Survivors Screen, a brief screening interview conducted and scored by study staff. In a sample of 261 U.S. injury patients screened during hospitalization and assessed for mental health one to three months later, SE = .80 and SP = .66 for depression, and SE = .73 and SP = .79 for PTSD [15]. The study was limited because the sample included few patients who were not White or Black and none who were non-English speaking. In addition, the ITSS differs from the HMHRS because training is required to administer and score the ITSS, whereas the HMHRS is a self-report measure that requires no training to score.

There were a number of strengths of this study. It is the first mental health risk screen that was designed to be used with patients admitted to hospitals after emergency care for acute illness, with patients in the five largest U.S. ethnoracial subgroups, and with patients who preferred to complete a screen in Spanish or Chinese. This was also the first such screen to have its performance tested and replicated in acute illness patients, patients in the five largest U.S. ethnoracial subgroups, and patients who preferred the Spanish language. It was also a strength of the study that we allowed patients to select their ethnoracial identity(ies) without a forced choice option for race. This approach is increasingly recommended [42] and yields results that better reflect patients’ identity. This is especially true for Latinx patients as in this study 90.4% selected only a Latinx identity. It was a strength to include and test performance of the screen in patients who identify with multiple ethnoracial subgroups. Few studies include enough of these patients to inform results or test performance. Notably, in the development study, patients who reported multiple ethnoracial identities scored higher on all risks and mental health outcomes [10, 21], so these patients are among those with the highest mental health care needs. Lastly, the study recruitment methods resulted in inclusion of patients with a wide range of education and economic circumstances, which allows generalization of findings to more of the U.S. population.

In selecting the cut point score for the HMHRS, we maximized sensitivity because the “cost” of low specificity and false positive screens is low. Patients whose score is 10 or more can be told that patients who scored like they did often had later mental health symptoms of depression, anxiety, or PTSD. A graphic is available from the authors that shows scores associated with low, medium, or high risk of later mental health problems. Given the PPV and NPV findings, when patients are given screen results, the probabilistic nature of the prediction should be emphasized. It is also important to note that while the HMHRS includes the most predictive risk items we studied, these are not the only relevant risk factors, and they may not even be the most important variables influencing mental health outcomes. Some known and major risk factors, such as social support [29, 43–47], were not included because their impact cannot be accurately assessed very soon after traumatic stress.

Preventive care programs have been found to be effective for hospital injury patients [48–52] and should be offered to patients whose scores indicate they are at risk. Because the unmet need is great, preventive programs that use self-help or guided-self-help models [53] should be considered. Hospitals that do not have capacity to provide professionally-guided preventive mental health care may give patients who screen positive for risk a self-help guide for recovery [54] that also direct patients to additional self-help and professional care if needed.

Availability of an accurate and reliable predictive screen could foster research on preventive mental health care. In past research on preventive mental health care for trauma survivors, those at risk could not be identified. This meant that a large proportion of patients assigned to a control group that received no intervention recovered naturally, which may have obscured effects of interventions for those at risk. Research that included only those at risk would make it easier to detect effects of preventive care.

Limitations of this study include that we did not collect information or report on the number of patients who were recruited, but declined to participate in the study and that we were unable to collect follow-up data from all enrolled patients. No significant difference in risk scores of those retained and those lost to follow-up indicate that those retained were similar in risk to those enrolled, but the sample that performance was calculated for may have differed in unknown ways from the population of patients who would be screened. Another limitation is that the majority of Latinx patients were of Mexican heritage, and their risk screen scores may differ from those of Latinx patients who immigrated from or whose heritage is from other countries. The sample of AAPI patients was not large enough to examine performance in subgroups of AAPI patients. Studies of subgroups of Latinx and AAPI patients are needed.

Conclusions

The performance of the HHMRS to accurately predict later mental health problems was replicated in the overall sample and in the five ethnoracial subgroups studied. The HMHRS in English (S1 File) and Spanish (S2 File) are provided in Supporting Information files, and a graphic score explainer is available from the first author (EBC). The ease of use of the HMHRS and the meaningful information it provides about mental health risk make it a valuable addition to diagnosis, treatment, and prevention efforts. Routine screening of all patients admitted after emergency care could improve health equity by ensuring that the well-being of all patients is addressed.

Availability of an accurate and reliable predictive screen could foster research on preventive mental health care and may be valuable to identify at-risk trauma survivors in other contexts. Performance of the screen could be studied in other countries and in other populations of recent trauma survivors, such as survivors of disaster or mass violence.

Supporting information

Acknowledgments

We thank the patients who shared their experiences with us. We thank Rossana Castillo-Cabanas, Megan Martau, Felicia Yen, Teivon Johnson, Gabriella Amato, and Kristen Harris, who recruited patients and collected follow-up data. This work would not have been possible without their efforts.