What did the researchers do and find?

• We undertook a systematic review and meta-analysis to summarise the predictive properties of machine learning algorithms to predict suicide and self-harm.
• The overall quality of the research in this area was poor, with most studies at either high or unclear risk of bias.
• We found that the predictive properties of these machine learning algorithms were poor and no better than traditional risk assessment scales.

What do these findings mean?

• Machine learning algorithms incorrectly classify more than half the people who subsequently present to hospital for self-harm or die by suicide as low risk.
• A classification of high risk poorly forecasts who will engage in suicide or self-harm.
• There is insufficient evidence to warrant changing recommendations in current clinical practice guidelines about risk assessment.
• The findings are limited by the exclusion of studies where we could not extract the information required to undertake a meta-analysis and by the included studies assessing the outcomes over different time periods.

Introduction

Numerous studies have sought to identify patients at high risk of suicide or self-harm so that treatment can be provided specifically to them [1,2]. The risk assessment scales that have been developed stratify patients into high or low risk categories, with treatment pathways based on the classification [3]. The main clinical group that has been the focus of risk stratification is patients treated for self-harm (self-poisoning or self-injury) in the general hospital setting. Patients classified as high risk are typically prioritised for more intensive aftercare interventions than patients classified as low risk. Immediate interventions are classically psychiatric inpatient admission, close nurse observation or more urgent, frequent or intense community-based treatment (supervision). A high-risk classification, however, is not necessary to allocate effective, longer-term therapy-based interventions for suicidal behaviours like cognitive behavioural therapy in unselected self-harm populations [4], dialectical behaviour therapy in selected populations [5] or for suicide prevention in various clinical populations [6].

There is clear evidence that the traditional risk assessment scales used to predict suicide or self-harm have modest sensitivity and low positive predictive values [7–9]. In keeping with these findings, clinical guidelines do not recommend using risk stratification to allocate treatment in hospital-treated self-harm populations, and the US Preventive Services Task Force does not recommend screening for suicide risk in primary care [10–12], although conversely, the US Joint Commission recommends screening for suicide ideation for all patients over 12 years of age in all behavioural health services [13].

Efforts to improve risk prediction have recently focussed on using machine learning to develop algorithms that can predict suicide and self-harm. Machine learning is a branch of artificial intelligence in which prediction algorithms are developed by automatically and iteratively testing for complex associations between many factors in a dataset. Many studies emphasise the improved accuracy of their algorithms [14], suggesting that the poor accuracy of the traditional instruments has been overcome. But an important limitation of some of these studies, is a reliance on a case-control data to develop and evaluate algorithms. The use of the case-control design in diagnostic accuracy studies has previously been criticised as this design overestimates accuracy [15]. This overestimate occurs because the prevalence of the outcome is determined by the study design, and it is common in case-control studies to use a sample comprising half cases and half controls (meaning the apparent prevalence is 50%). The positive and negative predictive values of any risk score, however, are closely related to the prevalence of the outcome [16]. Suicide and self-harm are rare events, even in populations where the prevelence of these behaviours is high [14]. Thus, the high positive predictive values reported in some studies may be an artefact of the case-control design. This criticism is less likely to apply to cohort studies, although the retrospective nature of many cohort studies, where exposure data are collected when the outcome is already known, may be another potential source of bias.

To test the predictive accuracy of risk prediction algorithms developed using machine learning techniques, we undertook a systematic review and meta-analysis, paying particular attention to study design issues and their implications for prevalence. Our goal was to estimate a range of accuracy statistics of algorithm performance, namely, the area under the curve, sensitivity, specificity, likelihood ratios and positive and negative predictive values. We focused on studies that predict either hospital-treated self-harm or suicide mortality as these are clinically relevant outcomes used by clinicians to differentially allocate treatment for high-risk patients and which usually rely on the same institutional data sources to identify outcomes for all participants.

Methods

We report our study using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA 2020) statement [17] (S1 PRISMA Checklist). The study was registered (PROSPERO: CRD42024523074). Screening, full text review, data extraction and quality assessment were undertaken using Covidence.

Results

Our search identified 7,319 studies, together with 15 additional studies which were identified from citation searching and other sources (Fig 1). After removing duplicates, we screened the titles and abstracts of 2,853 studies. 2,613 of these were excluded (including three for which we could not obtain a full text version of the article) leaving 240 studies that were assessed for eligibility using full text screening. 187 of these were excluded: 98 because they did not examine suicide or hospital-treated self-harm, 48 because of insufficient data, 21 because they were not primary research, 15 because of the wrong study design, and five for other reasons. This left 53 studies [32–84]. These studies analysed 35 million records and 249,000 occurrences of suicide and self-harm.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. PRISMA flow diagram.

https://doi.org/10.1371/journal.pmed.1004581.g001

The study characteristics are summarised in Table 1. 30 studies were conducted in the United States, five studies in Denmark, five studies in the United Kingdom, three studies in Canada, two studies in South Korea, two studies in Sweden, and one study each in China, France, Iran, the Netherlands, Spain and Turkey. All studies were published between 2015 and 2025 with 44 of these from 2020 onwards.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Characteristics of included studies.

https://doi.org/10.1371/journal.pmed.1004581.t001

Thirty-six studies used a retrospective cohort design and 17 used a case-control design. Thirteen studies predicted suicide, 30 predicted hospital-treated self-harm, 7 predicted suicide/self-harm and three studies developed algorithms separately for suicide and self-harm. The time frame over which these predictions were made were 30 days (3 studies), 90 days (12 studies), 180 days (3 studies), 1 year (11 studies) and >1 year (15 studies). In nine studies, the prediction window was not reported. There was considerable variation in the study populations. Twenty-four studies developed their algorithms in general population or general patient populations. Twenty-two studies developed algorithms in patients treated for psychiatric problems. Six studies developed algorithms in patients presenting to hospital for self-harm or with a history of self-harm. One study used data from another population (patients with multiple sclerosis). The data were predominately drawn from electronic health records, insurer claims data and registry data. The studies used a variety of machine learning methods, including random forests (10 studies), gradient boosted trees (8 studies), classification and regression trees (5 studies), LASSO models (5 studies) naive Bayes classifiers (3 studies) and ensemble learning (3 studies).

The findings of nine studies were reported using TRIPOD or STARD guidelines. Of the 31 items in the TRIPOD checklist, three items were judged to be not relevant for most studies and were removed from the quality assessment. Of the remaining 28 items, the mean number of checklist items that were adhered to across the studies was 20. Checklist items with low adherence were explaining how the sample size had been arrived at (11 studies), describing how missing data was handled (19 studies), reporting unadjusted associations between candidate predictors and the outcome (10 studies) and providing details about how the risk groups were created (11 studies) (S1 Table).

For patient selection, 18 studies were judged as having low risk of bias, 24 were at high risk of bias with the remaining 11 studies at unclear risk of bias (Fig 2 and S2 Table). For choice of index test, 6 studies were at low risk of bias, three were at high risk of bias and 44 studies were at unclear risk of bias. For the reference standard, 17 studies were at low risk of bias, 5 studies were at high risk of bias and 31 studies were at unclear risk of bias. For flow and timing of patients, 19 studies were at low risk of bias, 4 were at high risk of bias and 30 studies were at unclear risk of bias. Overall, three studies were judged to be at low risk of bias, 26 studies at high risk of bias, and 24 studies at unclear risk of bias.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Risk of bias assessments.

https://doi.org/10.1371/journal.pmed.1004581.g002

The pooled AUROCs ranged from 0.69 to 0.93 with the lowest value for the prediction of suicide from cohort studies and the highest for predicting self-harm from case-control studies (Table 2). The pooled sensitivities ranged from 45% to 82% and the specificities from 91% to 95%. The LR+ values ranged from 6.5 to 9.9 and the LR− from 0.2 to 0.6. Using baseline 1-year prevalence values of 0.7% for suicide, 2.1% for self-harm and 1.6% for suicide/self-harm, the positive predictive values were 6% for suicide (in cohort studies), 16% and 17% for self-harm (in case-control and cohort studies, respectively) and 9% for suicide/self-harm (in cohort studies). The corresponding negative predictive values were 99% for suicide, 90% and 96% for self-harm and 97% for suicide/self-harm.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Pooled diagnostic accuracy statistics of machine learning instruments to predict suicide, self-harm and suicide/self-harm.

https://doi.org/10.1371/journal.pmed.1004581.t002

Table 3 shows the positive predictive values for a hypothetical algorithm with LR+ values ranging from 1 to 50 alongside indicative 1-year probabilities of suicidal behaviours in different populations. In a low prevalence population, for example, predicting suicide in the general population (baseline prevalence 0.01% per year [30]), in the LR+ values we observed (LR+ = 6–10), the positive predictive values ranged from 0.06% to 0.10%. In a medium prevalence population, for example, predicting suicide after discharge from hospital for the treatment of self-harm (baseline prevalence 2% per year [29]), positive predictive values ranged from 11% to 17%. In a high prevalence population, for example, predicting self-harm after discharge from hospital for self-harm (baseline prevalence 16% per year [29]), the positive predictive values ranged from 53% to 66%. Positive predictive values improved at higher LR+ values than we observed in our pooled analysis. At LR+ = 20, the positive predictive value was 29% in a medium prevalence population (79% in a high prevalence population), and at LR+ = 50, it was 51% (91% in a high prevalence population).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Positive predictive values by 1-year base prevalences and positive likelihood ratios.

https://doi.org/10.1371/journal.pmed.1004581.t003

The sROCs are contained in S1 Fig. The figure shows that the estimates are generally clustered close together on the two dimensions (sensitivity and the false positive rate). I² estimates ranged from 1.6% to 11.4%.

Discussion

In this systematic review and meta-analysis of algorithms developed using machine learning tools to predict suicidal behaviour, we found these algorithms had good accuracy when assessed using a global measure, the area under the curve, but poor accuracy when assessed against more clinically relevant individual measures. We found that the algorithms had modest sensitivity and high specificity. This combination of sensitivity and specificity meant that while the algorithms are good at identifying people who will not re-present for self-harm or die by suicide, they are generally poor at identifying those who will. The modest sensitivity observed in the cohort studies indicates that more than half of those who repeat self-harm or die by suicide are misclassified as low risk.

The sensitivity and specificity values we observed translate into LR+ values that are just under the clinically meaningful minimal threshold of LR+ ≥ 10 [85]. However, the low baseline prevalence of suicidal behaviour, taken either from the cohort studies included in our review or from externally derived prevalence estimates [26–30], meant that the positive predict values of these algorithms were also very low. To illustrate, the in-sample positive predictive values were 6% for suicide, 16%—17% for self-harm and 9% for suicide/self-harm. When an LR+ value of 10 was applied to low, medium and high prevalence populations, the positive predictive values were 0.10%, 17% and 66%. The only theoretical scenarios where the positive predictive values were high enough to be clinically useful would be when the LR+ was ≥50 and the base prevalence was ≥6.5% per year (equivalent to an event rate of 6,500 per 100,000 person years) or when LR+ was ≥20 and the base prevalence was ≥16% per year (16,000 per 100,000 person years). These high positive predictive values are unlikely to be realised in real-world clinical settings for two reasons. First, predictions in high prevalence populations, such as in patients who have been discharged from a psychiatric facility or received treatment in hospital for self-harm, will be most clinically useful over a shorter window than the 1-year prevalence estimates used here (for example, 24 or 48 h after discharge through to 30 days). Prevalence will therefore be much lower than the values we used, and consequently, the positive predictive values will also be lower. To illustrate, while the 1-year baseline probability of self-harm after discharge from a psychiatric facility is 6.5%, the 4-week estimate is only 2.1% [28]. At this value, the positive predictive value is only 30% for an LR+ value of 20. Second, it is difficult to develop an algorithm with high LR+ values as it requires identifying a threshold with both high sensitivity and high specificity. In practice, there is a trade-off between sensitivity and specificity. Finding a threshold that increases the value of one of these measures will result in decreasing the value of the other. Most ways of getting a high LR+ value require very high specificity values (≥97%) which means sensitivity is likely to be corresponding low, leading to most cases being misclassified as low risk.

There appear to be two reasons for the recent focus on the development of new algorithms to predict suicide and self-harm. One reason is to screen for risk of suicide and self-harm [34,36,41,44,45,48,50,56,71,72,76,79,81,82,84]. In this model, the algorithm flags high-risk patients in the electronic medical record and these patients then undergo a further risk assessment. Most studies have focussed on this two-stage process being applied to psychiatric inpatients and outpatients, but it has also been suggested that this be applied to general practice patients [76]. If these algorithms are to be used for automatically screening medical records then they should meet the criteria set down for a viable, effective and appropriate screening programme [86]. Yet against the 12 consolidated screening principles [87] the algorithms for suicide and self-harm appear to meet one criterion: the epidemiology of the disease or condition is adequately understood; and another partially: that there is an agreed-upon course of action for screening participants with a positive test result. The algorithms do not appear to meet the other criteria, namely: the natural history of the disease or condition is clearly understood; the target population for screening is clearly defined; the screening test has sufficient performance characteristics; the screening test results are clearly interpretable; there is adequate infrastructure to allow for timely access to all components of the screening programme; the screening programme is coordinated with the broader healthcare system; the screening programme is acceptable and ethical; the overall benefits of the screening programme outweigh the harms; the full costs of the screening programme have been assessed in an economic evaluation; and the screening programme has clear goals and it is evaluated against these goals. On this basis, none of the algorithms we studied appear to be suitable as a screening tool for suicide or self-harm in unselected clinical populations.

The second reason machine learning algorithms have been developed is to prioritise the highest risk individuals for expensive or high-intensity interventions (for example, psychiatric hospitalisation or intensive case management by psychiatric services after discharge) [59,64,66,78,88]. One illustrative study found that among those with mental health speciality visits, those in the top 5% of risk accounted for 43% of suicide attempts and 48% of suicides over a 90-day prediction window [71]. The problem with this approach is that it results in algorithms with modest sensitivity and poor positive predictive values [3,89]. As the threshold that defines a positive test result is raised, the number of cases of suicide or self-harm detected by the algorithm (the true positives) decreases and the number of undetected cases increases (the false negatives). At a very high threshold (for example, the top 5% of risk continuum), it is likely that the number of undetected cases outnumber the detected cases (i.e., sensitivity will be <50%). Regarding the specificity, increasing the threshold will benefit the specificity of the algorithms because the number of non-cases that fall below the threshold (the true negatives) will increase. This is the pattern of results we see in our meta-analysis. The pooled sensitivities were generally below 50% and the specificities above 90%, and when combined with the low case prevalence, meant the positive predictive values were very low (because of the large proportion of false positives). The implication of using a high threshold to allocate treatment is that most cases of suicide and self-harm will be misclassified as low risk, and most people who test positive will receive an intervention they may not need. In other words, these intensive and expensive services will largely be delivered to the wrong people.

One argument in favour of using risk prediction algorithms is that it may be a cost-effective way of allocating expensive interventions. This strategy could be appealing to third-party payers and public health providers in an environment where healthcare resources are scarce. Some research has examined the circumstances under which a suicide risk prediction test might be cost-effective [88]. In a simulation study, the authors found an active contact and follow-up intervention could be cost-effective when people were allocated to this intervention using a test with sensitivity of 17% or greater when specificity was 95%. Similarly, the same study showed cognitive behavioural therapy could be cost-effective when the test used for allocation had sensitivity of 36% or greater when specificity was 95%. However, there are important caveats to these findings. First, the low sensitivity implies that a high threshold was being used to allocate treatment in these simulations, but a high threshold means that only a small number of people are allocated to these interventions. Second, and as discussed above, when sensitivity is less than 50%, there are more undetected cases at that threshold than detected cases. Many of these people could have benefited from the intervention, but would not have received it as they scored below the threshold on the risk assessment.

The diagnostic accuracy of machine learning algorithms for suicide and self-harm is similar to that of traditional risk assessment instruments [7,9]. The poor accuracy of these traditional instruments was one of the factors that led to clinical guidelines in several countries recommending that risk stratification not be undertaken in order to allocate aftercare services and that alternatives, such a needs-based psychosocial assessment be offered to patients instead in order to foster and focus aftercare interventions [10,12]. The National Institute of Clinical and Health Care Excellence guidelines [12] recommend that after an episode of self-harm, a mental health professional should carry out a psychosocial assessment to develop a therapeutic relationship with the patient and a shared understanding of why they have self-harmed, undertake needs assessment, ensure the patient is offered the care they need, and give family and carers information about the patient’s condition and diagnosis. Ideally, mental health clinicians should develop a therapeutic alliance with the patient that is organised around four components: predisposing factors (their history of self-harm, mental health and other relevant events), modifiable factors (things that are changeable, such as relationship issues, substance use, mood and mental health and access to means), future factors (anticipated events such as anniversaries, discharge from hospital or criminal proceedings) and protective factors (problem-solving skills, social and family support, engagement with services, insight and hope) [90]. Given that machine learning algorithms, including those that use dynamic risk formulation [68,70,77], appear to be no better at predicting suicide or self-harm than traditional risk assessment instruments, we see no compelling new evidence to warrant a change to these guidelines.

More generally, there are a number of effective aftercare interventions suitable for people presenting to hospital for self-harm that can be applied without first undertaking risk stratification to determine the allocation of treatment. Examples of interventions that have been shown to be effective for reducing the repetition of self-harm include psychological and psychosocial interventions (e.g., cognitive behavioural therapy or interventions with an interpersonal focus [4,91], brief contact interventions [92], multilevel interventions for the reduction of suicide and suicide attempts in clinical populations [6], and safety planning interventions [93]. All these interventions have financial and non-financial costs associated with them, and decisions about whether to deploy them in a hospital setting should be made with due consideration of whether the intrusiveness, burdensomeness and ethicality are proportionate to the benefits. Finally, we are concerned that the focus on risk assessment can be falsely reassuring and a distraction from the delivery of basic clinical services like ensuring all patients who present to the emergency department for self-harm are seen in a timely manner, are properly assessed and receive appropriate follow-up care [94]. In the UK and Australian context, this is apparent in the concerns of patients and service users about impersonal tick boxes rather than holistic assessments [95] and clinicians or health services being preoccupied with potential blame rather than delivering high-quality care [96,97].

Instead of predicting suicide and self-harm, there may be other ways in which artificial intelligence could be used to contribute to better outcomes for suicidal patients. Future research could consider how machine learning methods could be used to augment existing collaborative psychosocial assessments. Specifically, can machine learning methods be used to identify modifiable risk factors for suicide and self-harm for individual patients? This may be a more tractable problem as the prevalence of many risk factors is likely to be higher than the prevalence of suicide or self-harm. If such modelling can be done, then there are interesting follow-on questions about the acceptability of this approach for patients and clinical staff, and whether such an approach is superior to gathering information directly from patients and caregivers. Another interesting question for future research is to consider how artificial intelligence could be used to inform clinical decision support tools. This is distinct from using a risk classification to allocate treatment; rather, it is a question about whether artificial intelligence, when combined with information about an individual patient, can make suggestions for treatment pathways. Some work on this has been undertaken in other areas of medicine (e.g., in oncology to optimise drug dosage for individual patients [98]), and it is an open question as to whether this approach can be applied to the treatment of individuals with psychiatric symptoms and disorders.

Our study has a number of strengths. We used a broad set of search terms to capture studies that have used machine learning to predict suicidal behaviour. We searched eight databases that comprised a wide range of disciplines (e.g., medicine, psychology, health sciences, engineering and computer sciences). We focussed only on suicide and hospital-treated self-harm as the outcomes, not self-reported behaviour or scores on an instrument. The included studies use a variety of different machine learning methods. We were able to assess the quality of the literature, and we showed that at least half the studies on this topic are at high risk of bias and a substantial number are at unclear risk of bias. We were able to examine a range of diagnostic accuracy statistics, and we were able to recalibrate case-control studies to estimate positive and negative predictive values using the prevalence from cohort studies. Finally, we were able to estimate the positive predictive values for different outcomes (suicide or self-harm) in different populations (the general population, psychiatric patients, patients treated for self-harm).

Against this, our study had limitations. We had to exclude 48 studies because they did not present sufficient information for data extraction. The period over which follow-up outcome data were gathered varied between studies, from 30 days to 2 years for the majority of studies. Most of the included studies were judged to be at high or unclear risk of bias. We were unable to estimate pooled values for two groups of studies: case-control studies of suicide and case-control studies of suicide/self-harm. We were unable to assess publication bias as tools have not been developed to assess publication bias of diagnostic and accuracy studies. We were unable to assess the potential biases in individual algorithms. Finally, a number of studies used data collected from the same health system or data-linkage system. We were unable to adjust for this in our analyses.

In conclusion, our systematic review and meta-analysis has shown that algorithms developed using machine learning tools to predict suicide and self-harm suffer from the same problems as the traditional risk scales used to predict suicidal behaviour. The algorithms have modest sensitivity and low positive predictive values, resulting in most cases of suicide or self-harm occurring amongst those classified as low risk, and a large proportion of false positives in those classified as high risk.

Supporting information