Main outcome.

The main problem with child maltreatment is that it is not directly quantifiable in the administrative registers. Hence, to implement a prediction model for child maltreatment, we have to proxy it by observable information. This is a standard problem in predictive risk modeling [16].

To proxy the occurrence of child maltreatment, we use a binary variable reflecting child removal within the mandatory four-month investigation period by CPS.

This choice of proxy is motivated by various factors. First, higher risks of child removal and placement are typically associated with extreme cases of maltreatment, which also indicate an increased likelihood of other adverse child outcomes. Table 3 corroborates the claim, presenting the prevalence of eight adverse outcomes among three groups of children in Denmark as of January 1, 2017: (i) not referred in 2017, (ii) referred in 2017 but not removed, and (iii) referred in 2017 and removed. The table reveals that referred children were more likely to experience adverse outcomes, and those removed were even more likely. For instance, only 0.4% of non-referred children were charged with an offense or a crime in 2017, compared to 6.1% of referred children and 15.3% of those who experienced removal. Thus, the proportion of children charged with an offense or a crime in 2017 was almost 40 times greater among those removed than among the non-referred population of children.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Indicators of maltreatment in three distinct groups.

https://doi.org/10.1371/journal.pone.0305974.t003

Second, child removals are infrequent but not extremely rare in Denmark. Table 2 illustrates that within the four months after a referral was received (Panel B), removal and subsequent placement occurred in 3.6% of the cases. This percentage increases to 6.9% when considering the 12 months following receipt of the referral. In contrast, child deaths, often used as an indicator of child maltreatment in the literature [8, 17], are exceedingly rare events in Denmark. For instance, only five children died due to homicide in 2019 [18].

Although other adverse events, as listed in Table 3, occur more frequently, they may not always have a direct link to maltreatment and may not be well-defined for children of all ages. Additionally, certain variables are not reported daily in the administrative registers. For example, school absence statistics are reported monthly, and somatic illnesses are reported yearly, posing challenges in determining whether a specific event occurred before or after a referral was received.

When using child removal as the outcome variable, it is important to consider the non-uniform age distribution of removals in Denmark. Most removals occur for teenagers or shortly after birth, which can introduce potential bias favoring these age groups in risk predictions. However, child maltreatment can occur across various age groups to a similar extent. To address this, we present analysis results with “age-neutralized” predictions in S1 Appendix. These results show that conclusions from the main analysis remain unchanged.

Alternative child maltreatment outcomes.

In addition to using child removals as a proxy variable for child maltreatment, we also collect data on twenty other adverse child outcomes linked to child maltreatment to validate the model trained to predict child removals. The validation outcomes are selected to cover different aspects of child maltreatment to provide a nuanced view of the model’s performance. The complete description of twenty outcomes is presented in S1 Appendix. The outcomes related to crime, physical and mental health, dental quality, and school absence are all based on Danish administrative registers covering the full population of children over the period considered in this study. The outcomes related to well-being and personality traits are derived as proposed by Andersen et al. [19] using survey data from Danish public schools. Hence, measurement errors may influence these outcomes more than the register-based information, as they are not directly observable but rather estimated based on survey data.

The choice of adverse outcomes is also based on prior evidence from the literature that has established a relationship between child maltreatment and the development of mental disorders, fractures, criminal behavior, poor dental quality, and school absenteeism [20–24].

Information sets.

We create two information sets. The first one (henceforth referred to as the “limited information set”) is based on the information available to caseworkers for case assessment. It includes details such as the municipality receiving the referral, date of receipt, notifier (e.g., school, health institution, family member, or police), type of concern registered, and whether the referral is considered severe (e.g., cases of violent or sexual abuse, parental drug abuse, abusive parenting, or inadequate care). This set also contains detailed data from CPS on past referrals, removals, and preventive services for the referred child’s municipality. Additionally, it provides information about parents and siblings, like their ages, number of siblings, living arrangements, household size, and household mobility history. For the full list of variables in the limited information set, please refer to S1 Appendix.

The second information set (henceforth referred to as the “full information set”) encompasses all the data included in the limited information set along with a broader range of additional information available in the administrative registers (see S1 Appendix for details). In addition to the limited information set, the full information set comprises data about the child’s criminal background, health, and school performance, as well as the parent’s education, labor market status, income, criminal background, and type of residence. It also contains a complete history of the child’s past interactions with all Danish CPS. We use the full information set to examine the importance of the information constraint faced by caseworkers, as not all information in this set is currently accessible to CPS caseworkers. Therefore, the potential use of PRMs by Danish CPS would be constrained to be based solely on the limited information set.

Since all variables in both information sets are derived from Danish administrative registers, the dataset has a low degree of missingness. In cases where data is missing, we impute the missing values using the median of the available data points in the training sample (as defined in the Methods section).

If a PRM were to be utilized by Danish CPS, the country’s legislation would mandate the exclusion of ethnicity and sex information of the child and parents from the model’s information set. Hence, we have excluded this type of information from both information sets, even though it is accessible to caseworkers. In unreported analyses, we find that including information on ethnicity and sex in the models does not significantly alter our results and conclusions.

Predictive risk models.

To predict child removal we estimate four different machine learning models with varying degrees of complexity.

To train and test these models, we randomly split the full sample of referrals into two samples: a training sample (70%) and a test sample (30%). The former comprises 120,395 referrals (representing 63,303 different children), while the latter contains 52,649 referrals (representing 27,341 different children). The level of randomization is the mother’s social security number. This allows us to ensure that referrals concerning children from the same mother are all either in one group or the other and hence to reduce arbitrary overfitting [13]. All four models are calibrated using the same sample splits and data so that the predictions made by the models are directly comparable.

The output of the models is the predicted probability that a child removal occurs within four months of the referral receipt. We translate these probabilities into risk scores as follows. First, we rank all predicted probabilities in the test set and divide them into ten equal groups, or deciles. Each decile represents 10% of the distribution, with the lowest decile containing the lowest predicted probabilities and the highest decile containing the highest. We then assign a risk score from 1 to 10 based on the decile in which each referral’s predicted probability falls. Referrals in the first decile receive a score of 1, indicating the lowest risk, while those in the tenth decile receive a score of 10, indicating the highest risk. This method ensures that the risk scores are evenly distributed and directly tied to the empirical probability distribution.

The first model we consider is the logistic regression model estimated using maximum likelihood. The model is widely used across various disciplines and is specifically designed for binary dependent variables. In our application, the model specifies the log odds of removal to be a linear function of the variables in the model. The model offers the advantage of easy interpretation as the estimated coefficients indicate the change in the log odds of the outcome per unit change in explanatory variables. The main drawback of the logistic regression model is that it assigns a non-zero weight to all included variables, which makes it more prone to overfit the data.

The second model is the logistic regression model with the Least Absolute Shrinkage and Selection Operator (LASSO) [25]. This model is a generalization of the standard logistic regression model as it additionally performs both variable selection and regularization to enhance the accuracy of its predictions. The amount of regularization in the logistic regression model is determined by the parameter λ. This parameter is selected using five-fold cross-validation, and we follow the recommendation of Krstajic et al. [26] and use the value of λ that maximizes the cross-validated AUC-ROC minus one standard error. To avoid overfitting problems, the five folds are chosen such that children from the same mother always belong to the same fold.

The third model is a tree-based random forest model developed by Breiman [27]. The random forest model uses bagging as an ensemble learning method, which involves training different decision trees on different random subsets of data in parallel. In addition to bagging, random forest models also perform a random selection of explanatory variables for each decision tree. The fact that the random forest model is based on decision trees, makes it suitable to detect nonlinearities in the relationship between the outcome and the explanatory variables. We implement the random forest algorithm with 1,000 trees, using the ranger package [28]. We use a Bayesian optimization approach to find the optimal values for the number of variables the algorithm can use at each node (mtry) and for the minimal node size (min.node.size). More precisely, we search for the optimal hyperparameters that yield the highest AUC-ROC based on five-fold cross-validation on the training sample. The Bayesian optimization procedure is implemented using the mlrMBO package [29].

The fourth predictive model is the extreme gradient boosting (XGBoost) model proposed by Chen & Guestrin [30]. XGBoost is a tree-based method that uses boosting as an ensemble learning method. Boosting entails training different individual models sequentially in a way that allows each new model to learn from the mistakes made by the previous models. XGBoost is a computationally efficient boosting model that can handle nonlinear features in data, such as interaction effects, in a data-driven way and works in the presence of missing data in the predictors. It also employs several techniques to prevent overfitting, including regularization, early stopping, and pruning. The model is estimated using the xgboost package. To optimize the algorithm’s performance, we specify seven hyperparameters using Bayesian optimization. Using the terminology of the xgboost package, the hyperparameters we consider are the following: (i) max.depth; (ii) eta; (iii) gamma; (iv) subsample; (v) colsample_bytree; (vi) colsample_bynode; and (vii) min_child_weight. The remaining parameters are set to their default values. These hyperparameters are chosen to maximize the cross-validated area under the receiver operating characteristic curve across the five folds of data.

Predictive accuracy.

The output of the models we consider is the predicted probability of the outcome variable. In this case, model performance is often assessed using the area under the receiver operating characteristic curve (AUC-ROC). The receiver operating characteristic (ROC) curve plots the true positive rate of a predictive model against its false positive rate for each decision threshold τ ∈ [0, 1]. Formally, the AUC-ROC gives the probability of a model assigning a higher predicted probability to a randomly chosen true positive (a child who is removed) than to a randomly chosen true negative. Hence, the AUC-ROC may be interpreted as the model’s ability to distinguish between referrals leading to child removal and those that do not. Note, that a model making predictions at random would lead to an AUC-ROC of 50%. The AUC-ROC is the main performance criterion in this paper.

We also consider the area under the precision-recall curve (AUC-PR) as a supplementary performance metric. In a context of highly imbalanced data, where the number of positives is considerably smaller than the number of negatives, it is particularly relevant to consider the AUC-PR criterion, which reflects a model’s ability to predict the positive class in the data [31].

The precision of a binary classifier is determined by the ratio of true positives to the total number of predicted positives, while recall corresponds to the true positive rate. By varying the classification threshold τ between zero and one for a specific prediction model, the precision-recall curve can be obtained, from which the AUC-PR can be calculated as the areal under this curve. An ideal model would achieve an AUC-PR value of one, while random guessing would yield a score equal to the proportion of positives in the data. Higher values of AUC-PR are associated with better model performance for a given dataset, although AUC-PRs should generally not be compared across different datasets.

Identifying top predictors of the PRM.

Next, we identify the most predictors in the XGBoost model using Shapley additive explanation (SHAP) values [32, 33]. SHAP values provide a model-agnostic approach to explain the contribution of each variable to the predictions of interest. They measure how much each variable influences the model’s predictions and can account for interactions and correlations among the variables. This enhances the interpretability and transparency of the model, which is essential for the practical usage of such models in child welfare settings.

In Fig 2 we present the ten variables identified as being the most influential for predicting child removals in both the limited and the full model. In both cases, the age of the child is found to have the highest impact on the predictions, with a strongly heterogeneous effect on the predicted removal probabilities. The other identified variables by the SHAP methodology can broadly be categorized as variables related to prior or current involvement with CPS. This is also the case in the full model, where a much wider range of variables is included, suggesting that very detailed individual and family background variables are not a prerequisite for developing useful PRMs in a child welfare setting.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. SHAP values for predicting child removals.

This figure illustrates the SHAP values for the ten most influential variables based on the XGBoost model. The variables are ordered by descending variable importance as measured by the average absolute SHAP value. The figure is based on the referrals in the test set.

https://doi.org/10.1371/journal.pone.0305974.g002

Internal validation.

In Fig 3, we display the observed four-month intervention rates among the individuals in our test sample within each decile of the predicted risk distribution generated by the XGBoost model. Reassuringly, we observe a monotonically increasing relationship between the risk score and the observed child removal rates. The relationship is convex, indicating that the XGBoost model effectively discriminates between severe and less severe cases. For instance, the removal rate is close to zero for referrals with a risk score of five or below but increases sharply to around 18% for referrals receiving a risk score of ten. We also observe a robust relationship between the risk score and the observed rates for preventive services, usually initiated before removal. However, we notice a higher likelihood of removal than preventive service initiation for referrals receiving a risk score of ten. This is indicative of the model’s ability to rank children based on their underlying risk of maltreatment consistently. Despite these promising results, it is essential to stress that we cannot assert the model directly predicts child maltreatment, as this phenomenon remains inherently unobservable to us as researchers.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Observed intervention rates.

The figure illustrates the observed rates of child removals and preventive services as a function of the risk scores generated by the XGBoost model, during the four-month period following the receipt of a referral. The figure is only based on test sample data. The vertical error bars correspond to the 95% confidence intervals.

https://doi.org/10.1371/journal.pone.0305974.g003

Reducing assessment errors by caseworkers.

The effectiveness of PRMs in reducing errors made by CPS is a critical consideration. These errors can take the form of false positives, where CPS erroneously intervenes in a family, or false negatives, where CPS fail to intervene when it would have been the right decision to make. Despite its significance, there is currently no clear answer to this question in extant literature.

Occurrences of both false positives and false negatives are not straightforward to identify in the observed data. Nevertheless, we attempt to provide a crude estimate of the false negative rate by CPS by identifying referrals for which no intervention was implemented in the four months after the referral, but where an intervention was implemented in the following eight months. False positive cases, on the other hand, are more difficult to detect. For instance, a successful removal would often imply that the child who suffered from maltreatment can move back in with their parents after some time. Situations like this make it challenging to classify cases as false positives.

In Table 5, we use our definition of false negatives to investigate the extent to which our model would have flagged them as high-risk cases. To carry out this analysis, we focus on the 52,649 referrals in the test sample.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Can predictive models help CPS identify cases of maltreatment?.

https://doi.org/10.1371/journal.pone.0305974.t005

First, our estimation of CPS’s error rate suggests that there is room for improvement in caseworkers’ decision-making process. Indeed, we find that, in the four months following the end of the mandatory investigation period, a preventive service was implemented in 7.5% of the cases and a removal in 2% of the cases. Similar removal rates are found for 9 to 12 months after the receipt of a referral. This suggests that these results are not merely due to delays in the implementation of CPS decisions made during the mandatory investigation period, but rather to CPS’ decision not to intervene. Overall, 15.2% of the referrals identified as requiring no form of intervention were later reclassified as problematic.

Second, focusing on the cases that were reclassified after the end of the four months, we report their average decile risk score based on the XGBoost model at the time when the referral was received. Irrespective of the model specification, we find that a very significant share of these cases belongs to the very top of the risk distribution. Strikingly, more than 60% of the referrals for which a removal subsequently occurred receive a score of either 9 or 10, when the overall share of referrals with these risk scores by construction is 20%. Note that the models are trained to predict removal within the four months following receipt of a referral and not beyond. Consequently, there is no mechanical reason to expect that the models will necessarily predict removal accurately beyond these four months.

In S1 Appendix, we present descriptive statistics of the children for whom CPS made an error. This subgroup exhibits a particularly high prevalence of various negative child outcomes. Furthermore, when considering children with a risk score above 8, the observed rates are similar to those who experience removal within four months, indicating that the model correctly identifies the most vulnerable children at the top of the risk distribution.

Risk scores for removed children in past referrals.

In this section, we examine whether the model can offer an early indication of children’s maltreatment risk, which is crucial for their well-being. We achieve this by analyzing the risk assessments provided by the model in the previous referrals leading up to the initiation of a CPS intervention. To do this, we focus on the children in the test sample who had at least five referrals from April 2016 to March 2017. We divide them into three exclusive subsets based on their status between April 2017 and December 2017: (i) children who experienced removal; (ii) children with preventive services initiated but no removal; and (iii) children without any intervention. For the last group, we randomly assign a placebo intervention date between April 2017 and December 2017.

Fig 5 presents the results of this analysis, showing the average decile risk scores for the five most recent referrals leading up to the event, along with the average duration in days between the event and the referral. Early referrals for children who are eventually removed have a significantly higher average risk score compared to the other two groups. The average risk score for removals in the earliest referral is approximately 8.5, 7 for preventive services and 6.6 for referrals without CPS intervention. As we move to the most recent referral, these risk scores increase to around 9.5, 8.4, and 8.1, respectively. Hence, the risk scores in earlier referrals are already higher for children where CPS eventually intervenes, especially for child removals, where the average risk scores are considerably elevated several months before the removal occurs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Risk score in past referrals.

This figure depicts the average risk scores generated by the XGBoost model using full information for the most recent five referrals leading up to the initiation of a CPS intervention. The numbers displayed in the graph indicate the average number of days between the intervention date and the five most recent referrals.

https://doi.org/10.1371/journal.pone.0305974.g005

Identifying cases of maltreatment is a challenging task, particularly in the context of limited resources. Hence, our results are promising, indicating that the use of PRMs may assist CPS in identifying cases of child maltreatment and accelerating the implementation of appropriate measures to support at-risk children.