Integrating human services and criminal justice data with claims data to predict risk of opioid overdose among Medicaid beneficiaries: A machine-learning approach

Health system data incompletely capture the social risk factors for drug overdose. This study aimed to improve the accuracy of a machine-learning algorithm to predict opioid overdose risk by integrating human services and criminal justice data with health claims data to capture the social determinants of overdose risk. This prognostic study included Medicaid beneficiaries (n = 237,259) in Allegheny County, Pennsylvania enrolled between 2015 and 2018, randomly divided into training, testing, and validation samples. We measured 290 potential predictors (239 derived from Medicaid claims data) in 30-day periods, beginning with the first observed Medicaid enrollment date during the study period. Using a gradient boosting machine, we predicted a composite outcome (i.e., fatal or nonfatal opioid overdose constructed using medical examiner and claims data) in the subsequent month. We compared prediction performance between a Medicaid claims only model to one integrating human services and criminal justice data with Medicaid claims (i.e., integrated model) using several metrics (e.g., C-statistic, number needed to evaluate [NNE] to identify one overdose). Beneficiaries were stratified into risk-score decile subgroups. The samples (training = 79,087, testing = 79,086, validation = 79,086) had similar characteristics (age = 38±18 years, female = 56%, white = 48%, having at least one overdose = 1.7% during study period). Using the validation sample, the integrated model slightly improved on the Medicaid claims only model (C-statistic = 0.885; 95%CI = 0.877–0.892 vs. C-statistic = 0.871; 95%CI = 0.863–0.878), with small corresponding improvements in the NNE and positive predictive value. Nine of the top 30 most important predictors in the integrated model were human services and criminal justice variables. Using the integrated model, approximately 70% of individuals with overdoses were members of the top risk decile (overdose rates in the subsequent month = 47/10,000 beneficiaries). Few individuals in the bottom 9 deciles had overdose episodes (0-12/10,000). Machine-learning algorithms integrating claims and social service and criminal justice data modestly improved opioid overdose prediction among Medicaid beneficiaries for a large U.S. county heavily affected by the opioid crisis.


Introduction
In the United States (U.S.), opioid overdose deaths quintupled from 1999 to 2017, with 47,600 opioid overdose deaths in 2017 [1]. The total annual cost for opioid overdose, abuse and dependence is greater than $78 billion, and includes health care costs along with lost productivity and criminal justice system costs [2].
To mitigate the opioid crisis, stakeholders (e.g., payers, health systems, and policy makers) have developed policies and programs, such as increased naloxone distribution and improved access to medications for opioid use disorder (OUD); however, these interventions do not always target those most at risk. Current methods to identify individuals at high risk for overdose use simple criteria (e.g., high opioid dose measured by morphine milligram equivalent) and have significant limitations [3,4]. For example, recent studies suggest that the opioid highrisk measures used by the Centers for Medicare & Medicaid Services (CMS) miss more than 90% of beneficiaries with an overdose or OUD diagnosis [5,6].
Social determinants of health that fundamentally shape individuals' health, risk behaviors, access to health resources and social support systems are associated with risk of OUD and opioid overdose [7]. For example, individuals involved with the criminal justice system or who are homeless may be at particularly high risk of OUD or opioid overdose [8]. However, social determinants of health data are managed in different agencies and often not integrated in ways that facilitate their use with health care data. In 2017, the President's Commission on Combating Drug Addiction and the Opioid Crisis recommended strengthening data integration across different agencies and systems and using advanced data analytics to improve identifying individuals at high risk of overdose [9,10]. Linking Medicaid claims with public human services data can account for important social determinants (e.g., incarceration and social services use) of opioid overdose [8]. Death certificate data can capture some overdoses not receiving medical attention. Furthermore, recent studies indicate the shortcomings of opioid risk prediction tools in current use and recommend the development of more advanced models to better identify individuals who are at risk (or no risk) of an opioid overdose [5,[11][12][13][14]. Machine-learning techniques may improve opioid-overdose risk prediction because of its capabilities handling a large number of variables and complex interactions [6,[15][16][17], as we have recently demonstrated among Medicare beneficiaries.
In this analysis, we used integrated human services and Medicaid claims data from the Allegheny County Data Warehouse in Pennsylvania to develop and validate a machine-learning algorithm to improve prediction of opioid overdose among Medicaid beneficiaries. Pennsylvania ranks second among US states in drug overdose mortality, with an opioid-related overdose death rate of 35 per 100,000 population in Allegheny County in 2018 [1,18]. The county maintains a unique data warehouse, including human services data from criminal justice records from courts and the county jail, medical examiner's autopsy data, Medicaid claims data, and others social service data for County residents. In addition to creating the machine learning algorithm, we also stratified beneficiaries into subgroups that had similar risks of developing an opioid overdose to assist a range of interventions by health, human services, or criminal justice systems.

Design and data sources
This prognostic study used a retrospective cohort design. It was approved by the University of Pittsburgh Institutional Review Board. This study followed the Standards for Reporting of Diagnostic Accuracy (STARD) and the Transparent Reporting of a Multivariable Prediction Model for Individual Prognostic or Diagnosis (TRIPOD) reporting guidelines (S1 and S2 Appendices) [19,20].
We obtained data from the Allegheny County Department of Human Services (ACDHS) Data Warehouse in Pennsylvania, U.S. The ACDHS Data Warehouse is a central electronic repository of social, human services, and health data on clients and the services financed and/ or managed directly by ACDHS and those delivered by a variety of other government entities [21]. As of 2018, the ACDHS Data Warehouse contained population-level data from more than 1.2 million clients residing in Allegheny County from over 30 sources. We limited analyses to Medicaid beneficiaries because of their disproportionately high risk of drug overdose [22][23][24]. Among Medicaid beneficiaries in Allegheny County eligible for this analysis, we merged claims data with encounter data to capture the use of physical and behavioral health services financed by Medicaid, records of county-funded behavioral health services, records on dates of incarceration in Allegheny County Jail, criminal courts' offense records from the Magisterial District Court or Court of Common Pleas, and several other publicly-funded human services encounter data including receipt of aging, child welfare system, homelessness and housing supports, independent living, intellectual disability services, and other public benefits services [21,25]. We also linked to autopsy reports from the Allegheny County Medical Examiner's Office to identify persons who died of opioid overdose in Allegheny County [21,25].

Study sample
We identified Medicaid beneficiaries who were Allegheny County residents any time from 2015 to 2018. An index date was defined as the first observed date of Medicaid enrollment during our study period. We excluded beneficiaries who: (1) were under 12 years of age because of this group's very low opioid-related mortality; (2) had invalid dates (e.g., multiple death dates, an index date before the date of birth or after the date of death); and (3) had a fatal opioid overdose in the first 30 days after the index date (because of a lack of predictor information); (see S1 Fig). Beneficiaries remained in the cohort once eligible, until censored because of death or Medicaid disenrollment.

Outcome variables: Opioid overdose
The primary outcome was a composite end point of any occurrence of either fatal opioid overdoses recorded in Medical Examiner data, or opioid overdose events captured in Medicaid claims, both of which were measured in 30-day periods from the index date of Medicaid enrollment [6]. We chose 30-day periods to better measure the immediate risk of individuals recently released from jail. We used the International Classification of Diseases codes (ICD-9/ ICD-10; S1 Table) to identify overdose events from prescription or other opioids including heroin from inpatient or emergency department (ED) settings, which were overwhelmingly non-fatal, as <1% had a fatal overdose documented within 7 days. Hereafter, we use the term non-fatal opioid overdose to refer to these overdose events. Non-fatal overdose was defined as an opioid overdose code as the primary diagnosis, or other drug overdose or substance use disorders as the primary diagnosis (S2 Table) with opioid overdose as non-primary diagnosis, as defined previously. 14 Fatal opioid overdose was measured using medical examiner's autopsy data and defined as an accidental overdose death in which an opioid (e.g., oxycodone, fentanyl, heroin) was the primary cause or a contributing factor. In secondary analyses, we examined non-fatal opioid overdoses (measured in claims) and fatal opioid overdoses separately.

Candidate predictors
We compiled 290 candidate predictors identified from prior literature  and our previous work (S3 Table) [6] including socio-demographics, health status, prescription opioid use patterns, and human services and criminal justice records to account for some of the social determinants of health. All of the candidate predictors were measured in 30-day periods. Patient sociodemographic characteristics included age, sex, race, type of Medicaid eligibility, and duration of continuous enrollment in Medicaid each month. Health status factors (e.g., number of ED visits, mental health disorders, history of opioid overdose) were derived from the literature and used in our prior work [6]. We included candidate predictors related to prescription opioid use (e.g., average morphine milligram equivalent [MME]) and other relevant medication use (e.g., benzodiazepines, gabapentinoids) when prescription claims data were available.
From ACDHS, we included monthly indicators of receipt of over a dozen publicly-funded human services programs, including receipt of aging, child welfare system, homeless and housing supports, independent living, intellectual disability services, and other public benefits services. For criminal justice, we included 18 indicators. Using records from Allegheny County Jail, we included an indicator for any jail release in the 30-day period, along with a continuous variable for the number of jail releases in each 30-day period. Using information from the Magisterial District Court and Court of Common Pleas we constructed variables for the number and type of criminal offenses in 30-day periods, by 8 types of offenses with 2 different levels (e.g., drug misdemeanor, drug felony, property misdemeanor, property felony). We measured and updated candidate predictors in 30-day periods after the index date to account for changes in predictors over time for predicting the risk of opioid overdose in the subsequent 30-day period (S2 Fig). This time-updating approach for predicting opioid-overdose risk in the subsequent 30 days mimics active surveillance a health system might conduct [6].

Gradient boosting machine approach and prediction performance evaluation
Our analysis comprised two steps: (1) creating risk prediction scores for opioid overdose for each 30-day period for each individual, and then (2) risk stratifying individuals into subgroups with similar overdose risks. First, we randomly and equally divided the cohort into training (developing algorithms), testing (refining algorithms), and validation (evaluating algorithm's prediction performance) samples. We developed and tested prediction algorithms for opioid overdose using gradient boosting machine (GBM; Appendix Methods in S3 Appendix). Studies consistently show that GBM yields the best prediction results with an ability to handle complex interactions, which were likely given the complicated nature of opioid overdose with multifaceted factors involved [6,49]. Beneficiaries may have multiple 30-day periods until occurrence of a censored event including disenrollment or death. Sensitivity analyses were conducted using iterative patient-level random subsets (i.e., using one 30-day period with predictors measured to predict each patient's risk in the subsequent month) from the validation data to ensure the findings' robustness.
To assess whether the integrated data (e.g., health and human services and criminal justice system data) improved discrimination performance (i.e., extent to which predicted high-risk patients exhibit higher opioid-overdose rates compared to those predicted as low risk) compared to Medicaid claims data alone, we compared the C-statistics (0.7 to 0.8: good; >0.8: very good) and precision-recall curves between two GBM models [50]. Given that opioid-overdose events are rare outcomes and C-statistics do not incorporate outcome prevalence information, we report metrics including sensitivity, specificity, positive predictive value (PPV), negative predictive value, positive likelihood ratio (PLR), negative likelihood ratio (NLR), number needed to evaluate to identify one overdose (NNE), and estimated rate of alerts (S3 Fig) [51,52].
No single prediction probability threshold suits every purpose, so we present these metrics at multiple levels of sensitivity and specificity (e.g., arbitrarily choosing 90% sensitivity as an anchor). We also classified the validation sample's beneficiaries into subgroups using the decile of their predicted overdose risk score, with the highest decile split into three strata based on the top 1 st , 2 nd to 5 th , and 6 th to 10 th percentiles to allow closer examination of patients at highest risk of developing opioid overdose. We evaluated calibration plots (extent to which predicted opioid-overdose risk agreed with observed risks) by risk subgroup. Furthermore, we examined age, sex, and race differences by risk subgroup based on the prior literature indicating demographic-specific differences in substance use [53][54][55].
We performed several additional analyses to ensure the algorithm's practical utility. First, we report the top 30 important predictors of individuals possessing specific risk or protective factors. Rather than p values or coefficients, the GBM reports the importance of predictor variables included in a model. Importance is a measure of each variable's cumulative contribution toward reducing the squared error, or heterogeneity within the subset, after the data set is sequentially split based on that variable. Thus, it reflects a variable's impact on prediction. Absolute importance is then scaled to give relative importance, with a maximum importance of 100. Second, we conducted sensitivity analyses using 3-month measurement periods instead of 30-day periods because some prescription drug monitoring programs and health plans update data and evaluation risks quarterly [31,32,56].

Statistical analysis
We compared beneficiaries' characteristics by training, testing, and validation samples with two-tailed Student's t-test, chi-square test, and analysis of variance, or corresponding nonparametric test, as appropriate. Analyses were performed using SAS 9.4 (SAS Institute Inc, Cary, NC) and Salford Predictive Modeler software suite v8.2 (Minitab LLC, State College, Pennsylvania, USA).

Beneficiary characteristics
Our analysis followed beneficiaries for an average of 34

Prediction performance using gradient boosting machine (GBM)
The four prediction performance measures for GBM models with Medicaid claims only versus with integrated data are summarized in  Fig 1A). Based on the area under the curve, the integrated model also had slightly improved precision-recall performance (Fig 1B).
S4 Table presents the prediction performance measures by sensitivity and specificity level (90%-100%). When at the threshold that balances sensitivity and specificity (based on Youden  (Fig 1C and 1D). When, specificity was set at 90% (i.e., to attempt identifying 90% of actual non-overdoses), the integrated model had a 65.2% sensitivity, 0.42% PPV, 99.9% NPV, NNE = 238, and 10 positive alerts/100 beneficiaries). Sensitivity analyses using randomly and iteratively selected patient-level data overall yielded similar results (see S4A-S4D Fig for  an example). As shown in S5 and S6 Figs, similar to the main findings, the integrated model resulted in slight improvement from the models with Medicaid claims only for the two separate secondary outcomes (i.e., fatal vs. nonfatal overdose). Compared to the lower risk-groups, individuals in the top decile had at least a 35-fold higher overdose rate (e.g., observed overdose rate in the subsequent month: decile 1 = 0.47%, decile 2 = 0.12%). The overdose rates were minimal (0 to 3 per 10,000) for the 3rd through 10th decile subgroups. As shown in Table 2, those in the higher risk-groups (e.g., top 1 st percentile) were generally more likely to be aged 25-34 and 35-44 years, male, and White.

Discussion
By combining Medicaid claims data with human services and criminal justice records in Allegheny County, Pennsylvania, a county heavily impacted by the opioid crisis, we developed  machine-learning models with strong performance for predicting beneficiaries' risk of opioid overdose. Our findings highlight the potential utility of machine-learning approaches when predicting opioid overdoses. Although the positive predictive value of the model was low, as expected given the low incidence of overdose in a 30-day period [51], the integrated model successfully segmented the sample into different risk groups based on the predicted risk scores. Approximately 90% of the population had a minimal risk of overdose, and the top decile group captured almost 70% of the beneficiaries who had an opioid overdose. The ability to identify these risk groups is important for payers and policy makers whose interventions are presently based on measures of risk that are less accurate [5]. The integrated models slightly improved on the models with Medicaid claims data only, highlighting the potential value of breaking down data silos to incorporate information on social determinants of health in addressing the opioid crisis. We identified seven prior published opioid prediction models, each focusing on predicting opioid overdose over different time windows (e.g., 6 months), with only one linking to statewide corrections and hospital databases [6,43,46,48,[57][58][59]. Most of the studies had key limitations, including relying on single sources, such as administrative claims data, electronic health records or prescription drug monitoring programs; not measuring predictors over time but at baseline; only capturing the first overdose episode; inability to identify complex, non-linear or non-intuitive relationships (interactions) between the predictors and outcomes; and having suboptimal prediction performance. Our study overcomes these limitations by using machinelearning approaches capable of incorporating non-linear or complex interactions, linking to human services use and criminal justice records data, and updating predictor measures monthly to reflect dynamic condition changes in prediction. We also used a population-based sample (including Medicaid beneficiaries with and without a filled opioid prescription) to predict beneficiaries' overdose risk in the subsequent 30-day period instead of using a lengthy time period.
Our models integrating human services and criminal justice data resulted in slight improvements in prediction performance compared to using Medicaid claims data alone, indicating the role of key social determinants of health in opioid overdose prediction [5,8,60]. Although the improvement in performance was not large and varied by the selection of probability thresholds, it was consistent across all measures of prediction performance. Creating state-or county-level data warehouses that link individual-level records across multiple public service systems is a promising and immediate strategy to guide public health interventions for those at high risk of overdose [10,59,61]. For example, Chapter 55 of the Acts of 2015 in Massachusetts allows comprehensive linkage across different sources of datasets at the individual level to gain deeper understanding of circumstances influencing fatal and non-fatal overdoses [62,63]. Our risk prediction scores and risk stratification approach can be used in healthcare settings (e.g., ED), and also by the state, county, or other public health stakeholders and agencies (e.g., community behavioral health organizations, criminal justice system) to more efficiently identify patients at high risk of overdose to timely target interventions.
Regardless of the different probability thresholds used to identify high-risk individuals in our study, the low incidence rates of overdose over 30-day periods resulted in low PPV (<2%). In order to maximize the clinical utility and minimize false positives of any clinical tool for predicting rare outcomes, identifying subgroups with different risk profiles can be valuable to provide guidance on how to target interventions and allocate limited resources more efficiently. First, over 90% of individuals with minimal or no overdose risk can be excluded using our risk stratification strategy. For the remaining individuals, those in the highest risk subgroup (e.g., top 1 st percentile of predicted risk scores) may benefit from interventions offering close monitoring by case managers or other specialists, although these programs can be costly. Individuals in the moderate risk subgroups (e.g., top 2 nd -5 th percentile of risk scores) may benefit from lower cost or low-risk harm prevention approaches such as naloxone kits distribution [61]. Given the high morbidity and mortality from overdose and the interventions currently being deployed to many individuals who are at much less overdose risk using less powerful prediction criteria [5,6], the false positive rate may be acceptable. Stakeholders or agencies who will implement the prediction algorithm can choose the thresholds for identifying individuals at high risk based on their interventions and resource capacities.
Our study has limitations. First, although we used a validated algorithm using ICD codes to identify opioid-overdose events in medical claims (PPV = 81%-84%) [64] and medical examiner's autopsy records to identify fatal overdose, we could not capture nonfatal overdoses not receiving medical attention, or fatal overdoses that occurred outside of the county or state. Second, some potential predictors such as socio-behavioral information (e.g., family history) and laboratory results are not captured in our data but could improve the model. At the time of the study, we only had prescription claims for those receiving behavioral health services from the county. Although our algorithm was able to incorporate prescription information when available, its performance may be further improved when we have complete prescription claims data in the future. Third, our study describes the development and validation of the prediction model but does not address challenges to implementation. For example, technical issues related to updating risk scores in a real-time manner need to be considered. The demographic differences in risk subgroups that we observed indicate that health services may not be provided in an equitable manner based on race or socioeconomic status. These ethical issues and potential biases point to the importance of performing comprehensive bias assessments and identifying potential approaches to improve algorithm fairness prior to model implementation. Finally, prediction algorithms derived from the Medicaid population in a large county in Pennsylvania may not generalize to other states or populations with different demographic profiles or program benefits. However, the demographic characteristics of opioid-related overdose in Allegheny County were generally similar to other U.S. counties heavily impacted by overdose during the study period [65,66].

Conclusions
In conclusion, integrating human services and criminal justice data with Medicaid claims using machine learning showed small but potentially informative improvements in risk prediction for opioid overdose among Medicaid beneficiaries. These findings demonstrate the potential utility of machine-learning approaches for opioid-overdose risk prediction, and highlight the value of breaking down data silos allowing state, county or other public health stakeholders and agencies to provide more timely public health interventions.   Table. Summary of predictor candidates (n = 290) measured in 30-day windows for predicting subsequent opioid overdoses a . a: Details for the operational definitions for each variable and corresponding diagnosis and procedure codes and National Drug Codes can be provided per request to the corresponding author. b: We used an "as-prescribed" approach that assumes patients taking all prescribed opioids on the schedule recommended by their clinicians. ( Table. Prediction performance measures for predicting opioid overdose (fatal/nonfatal) varying sensitivity and specificity using gradient boosting machine: With integrated data vs. Medicaid claims data only models. a : Scores were calculated by predicted probability multiplied by 100. Score threshold refers to the score used to classify or predict individuals with opioid overdose (i.e., � the threshold) vs. non-overdose (i.e., <threshold). b : Optimized threshold was calculated by the Youden Index to achieve balanced sensitivity and specificity.