Research transparency

This project was inappropriate for preregistration because components of our analysis plan evolved as we improved understanding of these analytic methods that remain somewhat novel to our research group. However, we value the principles of research transparency that are fundamental to the robustness and replicability of science and took several steps to follow emerging open science guidelines [29]. Thus, we still aimed to maximize our transparency via several complementary methods.

First, some decisions were already and irreversibly made such as the available participant pool from which the current sample is drawn and the collected predictor variables. Second, we fully describe all measures used as predictor variables, exclusionary criteria for our analysis sample, and analysis procedures within this paper. We note when we made decisions that involved researcher degrees of freedom and outline why we selected that option. Critically, our most important analytic decision–to use the Bayesian correlated t-test–was selected at project outset during a master’s thesis proposal and never changed. Many researcher degrees of freedom were restricted via cross-validation procedures, which can robustly guide analytic decisions. Cross-validation prioritizes out-of-sample prediction such that any internal analytic decisions that improve model fit in the held-in sample are only valuable to the extent that they improve prediction in the held-out sample. Finally, we completed a transparency checklist [30], which appears in S1 Text.

Participants

Participants for the current study were adult men (n = 322) and women (n = 72) who were recruited from three prisons in the Midwest region of the United States as part of an ongoing, long-term research project with incarcerated individuals. Participants were included in the parent research project if they met the following criteria: between the ages of 18 and 55, no history of a psychotic disorder or bipolar disorder, a fourth grade reading level or above, and a score of at least 70 on the Wechsler Adult Intelligence Scale-Revised [31]. The first and last authors had access to identifiable data, which were stored separately.

Our sample for the current study included all individuals from the parent research project who provided SCID-5 data, the key data source for mental illness variables at the center of this project’s research questions. This criterion resulted in a final sample for the present study of 322 individuals incarcerated at a men’s facility and 72 individuals incarcerated at a women’s facility (total sample: 394 incarcerated adults). In accordance with DSM-5 updates, we shifted from using the SCID-IV to the SCID-5 in 2017; thus, our sample was recruited between the years of 2017–2020. Individuals with a history of psychosis and bipolar disorder were excluded from the parent study due to concerns for safety and competency to consent to research.

Procedure

All procedures were approved by the [location masked] Institutional Review Board. Informed consent was obtained from all participants before data collection, and each participant was told that their participation in research was completely voluntary and had no impact on their incarceration status. Participants were compensated $5/hour for participation, which represents a higher wage than individuals typically receive for other employment during their incarceration but not so high as to add undue influence to participate in our study.

Participants completed two interview sessions to assess psychiatric symptoms and psychopathy. In addition, they filled out a packet of self-report questionnaires designed to address a variety of research questions associated with the ongoing parent research project; measures relevant to the present study are described below.

Predictor variables

A full list of predictor variables is available in Table 1. Features are divided into four sets: crime, demographics, substance, and mental illness. For analyses, the crime and demographic feature sets were paired together.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Feature set variables.

https://doi.org/10.1371/journal.pone.0297448.t001

Outcome measure

Our outcome measure (recidivism) is defined retrospectively as history of a previous adult incarceration before the incarceration sentence at the time of the assessments in line with other research that has defined recidivism similarly [28,35]. History of a previous incarceration was found through an online public DOC database. Thus, individuals with a history of previous adult incarceration were classified as positive for our outcome (cases), and individuals without a history of previous adult incarceration (i.e., for whom the current incarceration at the time of assessment was their first) were classified as negative for our outcome (controls). In the current sample, 31.7% of individuals were classified as positive for recidivism.

Analysis strategy

We used the tidymodels framework in R for our analyses. Tidymodels is a collection of multiple packages used for machine learning within the context of the tidyverse [36], another collection of packages used for data science.

Model comparisons.

To determine the degree to which the mental illness feature set and substance use feature set predicted recidivism, we conducted three model comparisons [38]. Each model comparison compared two model configurations that differed with respect to the feature sets included. Each augmented model included one or more additional sets of features not included in the corresponding compact model, which allowed us to evaluate the unique predictive value of the differing (i.e., focal) feature set(s). Specifically, we compared models with and without mental illness + substance use feature sets, models with and without the mental illness feature set, and models with and without the substance use feature set (Table 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Model comparisons.

https://doi.org/10.1371/journal.pone.0297448.t002

Our 10x10-fold cross validation procedure yielded 100 estimates of performance for each compact and augmented model. We compared these performance estimates from compact and augmented models using a Bayesian correlated t-test [38]. The Bayesian correlated t-test calculates the posterior probability (the likelihood given our data) that the augmented model performs better than the compact model for each comparison. In contrast to a frequentist approach (null hypothesis significance testing), Bayesian approaches provide the posterior probabilities of alternative hypotheses to allow us to estimate the certainty of our results. The Bayesian correlated t-test is specifically designed to adjust for non-independence in the data due to overlapping datasets used during resampling. We display the likelihood that the augmented models performed better than the compact model by plotting the posterior probability distributions for the parameter estimates of interest (i.e., differences in model performance).

Shapley additive explanations for relative feature importance

We computed Shapley Values [39]to provide a consistent, objective explanation of feature importance (i.e., contribution to predictions) across our augmented and compact models. Within each model, Shapley values show relative feature importance of the included features. We calculated Shapley values from the 100 held-out folds (10 repeats of 10-fold cross-validation) using the DALEX package in R. We averaged the 10 Shapley values for each participant for each feature across the 10 repeats to increase stability. To calculate global importance for categories of features (e.g., cannabis use disorder diagnosis and symptom counts), we added Shapley values across all features in a category separately for each participant, then averaged the absolute value of the Shapley values of all features in the category across all participants.

Sample characteristics

Demographic characteristics for our sample, grouped by recidivism status, are reported in Table 3. Gender identity data were not available for this sample; consequently, gender is assumed by the gender of the facility. One hundred and twenty-five (31.72%) participants had a previous adult incarceration, which is consistent with the three-year prison recidivism rate [40]. Thus, the baseline performance for our models was 68.3%. Compared to the state’s prison population, Black individuals (27.9%) are underrepresented in our sample [41].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Demographic characteristics by recidivism status.

https://doi.org/10.1371/journal.pone.0297448.t003

Type of offense by recidivism status is reported in Table 4. Overall, the most common charge for current incarceration was a sex crime (29.19%), followed by drug (18.78%) and murder (16.24%) offenses. The overrepresentation of individuals with sex charges is unsurprising given the presence of a sex offender treatment program at one of the prisons used

for data collection.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Type of offense by recidivism status.

https://doi.org/10.1371/journal.pone.0297448.t004

Lastly, Table 5 reports the presence of mental illness diagnoses as well as symptom count distributions by recidivism status. For the purpose of reporting sample characteristics, diagnoses were grouped into depressive, anxiety, trauma, personality, and substance use disorders. Obsessive compulsive disorder was included in the anxiety category. The rate of SUDs was particularly high in our sample at 86.29% (82.53% for no recidivism and 94.4% for recidivism classes), even when compared to other incarcerated samples.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Mental health disorders and symptoms by recidivism status.

https://doi.org/10.1371/journal.pone.0297448.t005

Model comparisons

For each model comparison, the posterior probability that the augmented model performed better than the compact model, obtained from the Bayesian correlated t-test, is provided. Greater posterior probabilities (higher percentages) indicate increased likelihood that the focal feature set(s) improve model performance and thus allow us to test the unique contribution that the focal feature set(s) made to recidivism prediction. Posterior probabilities for the main model comparisons are displayed in Fig 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Posterior probabilities for main model comparisons.

The vertical dotted line represents equal performance (i.e., no difference in accuracy) between the associated augmented and compact model. A) There is a 38.76% chance that adding both the mental illness and substance use feature sets improves model performance above and beyond the crime and demographic feature sets. B) There is a 60.32% chance that adding the mental illness feature set would improve model performance above and beyond the crime, demographic, and substance use feature sets. C) There is a 53.11% chance that adding the substance use feature set would improve model performance above and beyond the crime, demographic, and mental illness feature sets.

https://doi.org/10.1371/journal.pone.0297448.g001

As demonstrated in Fig 1A, results showed that there was a 38.76% chance that adding the mental illness and substance use feature sets in the augmented model increased model accuracy above and beyond the crime and demographic feature sets. Fig 1B shows that there was only a 60.32% chance that adding the mental illness feature set would increase model accuracy above and beyond the crime, demographic, and substance use feature sets. Fig 1C shows a similar result: there was a 53.11% chance that adding the substance use feature set would increase model accuracy above and beyond the crime, demographic, and mental illness feature sets. These results indicate that all mental illness and substance use variables as measured in our study did not add incremental predictive value.

Feature importance

Across the augmented and compact models, four demographic and crime feature categories (sex crime charge, age, education level, and being charged under the age of 18) emerge consistently as the most important feature categories relative to other features in the model. Three of these feature categories (sex crime charge, age, and education level) are noticeably more important than others (i.e., contribute relatively more to predictions). The global importance (mean absolute values of Shapley values) for feature categories for each model configuration (Compact Models 1–3, Augmented Model) appears in S1 Fig.

The combination of age emerging as a globally important feature and our retrospective recidivism definition encouraged us to look more closely at age in our sample. Means and standard deviations of age for individuals overall and by recidivism status appear in Table 3. Additionally, density plots of age by recidivism status appear in S2 Fig.

Exploratory analyses

Due to the lack of incremental predictive utility from mental illness and substance use variables observed in our model comparisons, we conducted exploratory analyses to investigate if these feature sets had predictive utility when compared to a null model. A null model, also called a no-information model (e.g., an intercept-only linear model), is created by regressing the outcome (recidivism) on no predictors, such that only the mean (proportion) of the outcome variable will influence predictions. These follow-up analyses would clarify whether the mental illness and substance use feature sets did not have any predictive utility or did not add incremental predictive utility over and above the crime and demographics feature sets. We again used Bayesian correlated t-tests to conduct three additional model comparisons: a model with crime and demographics features vs. a null model; a model with mental illness features vs. a null model; and a model with substance use features vs. a null model.

Posterior probabilities from these comparisons are displayed in Fig 2. There was a 99.60% likelihood that including the crime and demographic feature sets increases model accuracy from a null model, indicating strong signal and clear predictive utility for recidivism (Fig 2A). There was a 65.70% likelihood that the substance use feature set increased model accuracy from a null model (Fig 2B). This result indicates that the substance feature set does not reliably increase predictive utility for recidivism. There was a 28.05% likelihood that including the mental illness feature set increased model accuracy from the null model (Fig 2C). These results indicate that the mental illness feature set did not increase predictive utility for recidivism when compared to a mean-only null model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Posterior probabilities for null model comparisons.

The vertical dotted line represents equal performance (i.e., no difference in accuracy between the feature set models vs. the null model). A) There is a 99.60% likelihood that a model with crime and demographic variables increases recidivism prediction from a null model. B) There is a 65.70% likelihood that a model with substance use variables increases recidivism prediction from a null model. C) There is a 28.05% likelihood that a model with mental illness variables increases recidivism prediction from a null model.

https://doi.org/10.1371/journal.pone.0297448.g002

Recidivism

Our outcome measure was retrospective recidivism, not prospective recidivism—meaning that our models were predicting whether an individual had a prior incarceration in the past, rather than in the future. As a consequence, older individuals were more likely than younger participants to be coded positive for recidivism (Table 3 and S2 Fig). Age emerged as a relatively important feature across all model configurations, among several other demographic and crime features (S1 Fig). This finding supports age as an important feature relative to other features in the model. However, because age was included in all models, model comparisons cannot show its unique predictive value. Rather, the model comparisons were considering whether mental illness variables offered unique predictive value above and beyond demographic variables such as age. Age has also emerged as an important feature in research predicting prospective recidivism [8,45].

In addition, while prospective studies have the advantage of prediction of future events, they, too, are limited in their scope. Typical prospective recidivism studies will use a specified time frame to recidivism (e.g., 1–3 years) as the outcome measure. This strategy does not capture those who recidivate after the specified time frame. Although our retrospective study was less likely to categorize younger individuals as recidivists, and therefore there are likely some individuals who were coded as negative for recidivism who will go on to recidivate, it was able to capture a longer time period of recidivism. The current study follows prior research that has utilized retrospective data as a criminal justice outcome [28,35]. Greenberg & Rosenheck (2014) found that a substance use diagnosis was a strong risk factor for a previous incarceration. Using machine learning models in an incarcerated population with schizophrenia, Kirchebner et al. (2020) found that illegal opioid use and anxiety upon discharge were relevant predictors of criminal recidivism; therefore, a signal from mental illness and substance use have been found using a retrospective outcome.

There are opponents to the use of recidivism as a measure of “success” after release from incarceration [46]. Relying on recidivism may lead to incomplete conclusions due to the multitude of ways that personal factors (e.g., race, ethnicity, gender, etc.) intertwine with social factors (e.g., employment opportunities, neighborhood context, etc.) and lead to return to prison or jail. So, it is possible that the effect of mental illness on recidivism is far outweighed by factors like employment and policing, for example, both of which are associated with recidivism [47,48]. Employment rates after release from prison are dismal. Only 55% of individuals reported any earnings in the first year post release, and only 20% of those people earned more than $15,000 [49]. Formerly incarcerated individuals face stigma and background checks that deem them ineligible for many types of employment. Unemployment and an inability to provide for oneself or one’s family, as well as an inability to pay necessary legal fees such as parole fees, may then impact mental illness which can in turn add barriers to finding and maintaining a job with a steady income [50]. In addition, disproportionate policing in communities of color, including higher rates of stop-and-frisk for Black and Hispanic/Latino men in particular, may outweigh the risk for recidivism more than any personal characteristics, such as mental illness [51,52]. The variables in the demographic feature set in the current study only took into account personal attributes (see Table 1), but not the context in which the individual was immersed after release.

In sum, recidivism as an outcome variable is inherently limited. Although it is a useful indicator of one type of success after release from incarceration, it may be too simple to measure without larger social context. We included personal identity factors such as age, race, ethnicity, and gender in our models, as well as both personal and parental education levels which have been used as a proxy for socio-economic status [53]. However, we were not able to account for social factors such as employment opportunities and neighborhood context that may impact recidivism directly or indirectly–including by impacting mental illness.

Substance use

It was surprising that the substance use feature set did not increase predictive utility, as the relationship between substance use and recidivism is widely accepted in the literature [17,18]. However, an overwhelming majority of our sample met criteria for a history of SUD (86%). The lack of variability therefore may have hidden the impact of substance use. Previous studies that have found a clear relationship between substance use disorders and recidivism have had more balance with respect to SUD diagnoses among their participants. For example, 52% of subjects in Wilson and Wood (2014), 59% in Sadeh and McNeil (2015), and 68% in Walters & Crawford (2013). Each of these studies utilized DOC files in order to obtain SUD diagnoses, which likely captured only the most severe cases of SUD. A strength of the current project was the use of the SCID-5, which provided a more comprehensive assessment of mental illness including identifying individuals with mild or moderate SUD diagnoses. The contrast between the lack of predictive utility in our study compared to positive findings in other research that has relied on DOC files for diagnosis, then, may suggest that only severe SUD at the time of arrest can predict recidivism.

Additionally, our SCID-5 data did not provide us with a temporally specific estimate of substance use or other mental illness disorders. For example, a participant who had been in remission from an opioid use disorder for 10 years prior to their incarceration would be coded the same (“yes”) as someone who had an opioid use disorder at the time of arrest. One third of people in prison report that they were under the influence of drugs or alcohol at the time of their offense [54]. It is possible that with more temporal specificity, we would have replicated the additive value of substance use features in the prediction of recidivism.

Mental illness beyond substance use

The experience of mental illness symptomatology across diagnoses is dynamic. Static diagnostic measurement may not capture the precision needed to investigate complex relationships, like the one between recidivism and mental illness. Some researchers have utilized ecological momentary assessment (EMA), a methodological approach that involves repeated sampling of symptoms throughout the day, to provide a more in-depth picture of the fluctuations in symptom severity [55,56]. EMA has shown to be more effective than self-report measures for demonstrating clinically meaningful change [56]. In this study, we were interested in whether a more comprehensive measurement of history of mental illness would increase predictive accuracy of recidivism. However, research on the relationship between mental illness and recidivism could benefit from a more temporally precise estimate of mental illness symptomatology during the period of release between prison sentences. A more temporally precise estimate of mental illness could also include a record of mental health treatment utilization. Treatment can significantly alter the course and outcome of both mental health disorders and pre- and post-release adjustment.

Lastly, although we were able to investigate the relationship between mental illness and recidivism in a general population sample without history of psychosis or bipolar disorder, it is possible that we would have found more predictive signal from the mental illness data if individuals with these severe mental illnesses were included. Prior research demonstrating a relationship between mental illness and criminal justice outcomes such as recidivism for a new crime or reincarceration after a parole violation has typically utilized samples with more severe forms of mental illness including psychotic disorders [12,15,16].

Mental illness and the legal system

Although the current study does not support the idea that mental illness, excluding psychosis and bipolar disorder, increases risk for recidivism, it is imperative to understand why mental health treatment during incarceration is necessary and how it could have an indirect impact on recidivism. First, the rates of mental illness in our sample were high (Table 5), highlighting there is critical need for treatment in this population. Also, there are critical human rights, safety, and rehabilitation concerns for incarcerated people with mental illness. Withholding adequate medical and mental healthcare from incarcerated individuals was ruled cruel and unusual punishment by the U.S. Supreme Court [57,58]. Ruiz v. Estelle (1980) mandated that prisons implement a systemic program for screening and evaluating mental health, required that treatment consist of more than transfer to segregated housing, and necessitated a sufficient number of trained mental health professionals on staff to treat and identify mental illness. Although research shows that many institutions offer mental healthcare on a superficial basis [59], there remains a legal imperative to provide mental health care to incarcerated individuals who need it. Engagement in treatment with a compassionate clinician is one way that incarcerated individuals can begin to rebuild a sense of personal dignity, which can lead to desistance from crime once released [60].

Mental illness can also disrupt aspects of the correctional environment. For example, individuals with mental illness receive more disciplinary infractions than those without mental illness [61–64]. Individuals with major depression as well as psychosis are more likely to receive all types of infractions, including those that involve aggression [62]. Those with co-occurring mental illness and substance use at the time of admission report more variety and number of infractions when compared to those without co-occurring psychiatric disorders [63]. Infractions can lead to time in solitary confinement, and placement in solitary confinement increases the odds of both general and violent recidivism [65].

Mental illnesses such as depression and anxiety may also interfere with the ability to engage in rehabilitation programs that are aimed at reducing recidivism such as anger management, substance use treatment, and vocational training. There is evidence that treatment of PTSD with cognitive behavioral therapy (CBT) is associated with a decrease in anger [66] and that using CBT for the treatment of depression in those with SUD decreases not only depressive symptoms but also substance use symptoms [67]. Therefore, treating mental illness could increase positive outcomes for rehabilitation programs. Given the evidence that engagement in vocational training during incarceration reduces recidivism and the evidence that unemployment contributes to recidivism, treating mental illness could also be a vital step to improving employment outcomes [49,68]