Introduction

Current estimates suggest that approximately 700,000 individuals die by suicide each year, making suicide a priority in mental health research and prevention [1]. There are a variety of prevention approaches that have been widely used and recommended, with media reporting about suicide being one of the focal areas in most suicide prevention programmes and strategies [2, 3]. Research suggests that media can influence suicidal thoughts and feelings and potentially, suicidal behaviours in vulnerable individuals, in both a harmful and beneficial direction. For example, a recent-meta-analysis shows evidence for increases of suicides after news about celebrity deaths by suicide within two months after the death, making the reporting of celebrity suicide a core characteristic considered to carry some risk of harmful effects [4]. Specifically, after the reporting of celebrity suicides, suicide rates tend to increase by 13% in one to two subsequent months [4]. Imitation of suicidal behavior such as after celebrity suicide news is commonly referred to as the Werther effect [5]. Studies suggest that other types of media content may have a protective effect. With regard to positive media effects, the strongest evidence exists for stories of hope and recovery from psychosocial crises and suicidal thoughts [6–9]. A recent meta-analysis of randomized controlled trials suggests that media stories that feature individuals who managed to cope with their adversity and suicidal thoughts lead to a small-sized reduction in suicidal thoughts [8]. This beneficial media effect has been coined the Papageno effect and has received increasing attention in prevention and research over the last decade [6].

In order to mitigate the risks associated with suicide-related reporting in broadcast and online media, media recommendations for the reporting on suicide have been developed by national and international public health organizations [3]. These recommendations list content characteristics that have been deemed as potentially harmful and protective. On the one hand, recommendations include suggestions such as not focusing on suicide death; but rather on suicidal ideation in prevention-reporting. Further, they discourage extensive reporting on celebrity death from suicide, and recommend not to perpetuate common public misconceptions and myths about suicide. On the other hand, media recommendations also encourage reporting characteristics that might be helpful to vulnerable audiences. Specifically, they encourage the reporting of alternatives to suicidal behaviour, reporting of healing stories (i.e., stories of individuals who managed to recover from suicidal thoughts), and the reporting of positive outcome of crisis. Although not all of the specific recommendations have solid empirical evidence (many of them are based on expert consensus), current media recommendations are considered to be the most available and practical tool to help prevent suicide from suicide-related media reporting [3].

In order to prevent harm from media reporting on suicide, screening for proactive identification of media reporting has been identified as a priority in suicide prevention [2, 3]. In case of reporting that might do harm, media screening can speed up the process of identifying media reports and potentially distributing media reaching out to media professionals. Such screening, however, has been hampered by the considerable resources needed for content analytic work. Machine learning approaches to automatically detect specific media reporting characteristics might be helpful in this regard. Further, automatic detection of reporting characteristics across a large body of media items would also help with building the evidence base for specific media recommendations, as this would increase the feasibility of testing of how specific media reporting characteristics are associated with suicide and other important outcomes, such as help-seeking.

With regard to broadcast media content, machine learning approaches to categorize content are largely missing. This is in spite of the fact that particularly news media reporting has frequently been found to be associated with suicide [5], and although global efforts to reduce imitation suicide using media recommendations for safe portrayals have largely focused on news media [3]. In contrast, some research has been done on social media content. Specifically, there are now three machine learning studies that categorized Twitter posts into frequent content types, including celebrity suicide posts, posts signaling suicidal ident, awareness campaigns, prevention information, condolences and flippant remarks [10, 11]. These studies have typically used word frequency statistics as predefined features for model training. However, the meaning of words also depends on their context, and more novel deep learning models [see e.g., 11, 12] are needed to take context better into account.

For the present research, we build on previous studies by using a comprehensive human annotation scheme that systematically identifies selected putatively harmful and protective characteristics in broadcast reports about suicide. We selected a number of reporting characteristics from media recommendations for suicide, starting with a difficulty level that we assumed to be more simple (e.g., identifying a report on a celebrity suicide), to increasingly difficult classification tasks (e.g., identifying stories of healing) and ultimately, highly complex tasks such as the identification of the main focus of a news item. Using this selection of content characteristics with different difficulty levels for classification, we want to assess the feasibility of current machine learning approaches for the detection of suicide-related media reporting characteristics.

Materials and methods

Machine learning

We used Python 3.6 for our analysis, including the packages ktrain wrapper [18] for the deep learning library Tensor Flow Keras [19], and the sklearn library [20] for TF-IDF & SVM (see below). For links to the model, the code and data, see the data and code availability statement.

Text preprocessing.

We apply standard preprocessing strategies. see e.g., [21]. We converted the PDF files into text files using the Linux package pdftotext (https://pdftotext.com/). The original transcripts contained metadata about the radio/TV channel, the time of broadcast, a unique identifier (ID), and the transcribed broadcast. We first separated these metadata from the text using Python, and focused on analysis of the broadcasted text only for all following analyses. Input for BERT (one of the models we trained) is limited to a sequence length of 512 tokens, with remaining tokens being cut off, resulting in information loss. We compensated for this loss by training the word-frequency based models (see below) on the full sequences and comparing their performances to BERT. The majority of the 2519 transcripts in the final sample (83.2%) were shorter than 512 tokens, with a mean length of 316, the 95^th percentile at 944, and the 99^th percentile at 1411 tokens.

Machine learning models and model training.

The primary objective of machine learning is to make correct predictions on previously unseen data. To achieve this goal, the dataset is split into three distinct subsets: the training set, the validation set, and the test set. The training set is used for fitting the parameters of the model, the validation set to tune its hyperparameters and for evaluation of the model throughout development. The test set, finally, consists of data that the model has no access to during training, and is used to evaluate the model’s ability to generalize to new data. We divided the dataset of 2519 transcripts into a random training (64%), validation (16%), and test set (20%). Sklearn’s train_test_split was used for this purpose.

We used different text representations and models to classify transcripts:

Majority classifier.

This is a naive classifier that always predicts the most frequent class.

TF-IDF & SVM.

Term-frequency-inverse document frequency (TF-IDF) represents text by assigning a weight to each word in a document (transcript) based on how often it appears in this document and how uncommon it is across all documents in a corpus. We used a small modification to the standard formula for TF-IDF by adding 1 in both the numerator and denominator. This change guarantees that every word appears at least once and prevents division by zero [22]:

This value increases proportionally as a word becomes more frequent in a document, and is balanced by the number of documents in the corpus that include the word. This gives more weight to uncommon words, as they are more useful in distinguishing one document from another than words that appear in every document. Using the TF-IDF representation as features, we then trained a linear Support Vector Machine (SVM) classifier. To identify the best combination of parameters, we ran a grid search across different variations of the TF-IDF representation and the SVM classifier. For TF-IDF, these included (1) inclusion of unigrams and/or bigrams, the reduction of text to the top n features by term frequency (10.000, 25.000, 50.000). Inclusion of both uni- and bigrams, and fewer top features (10.000 or 25.000) were better in most classification tasks (see S3 Table for details). Regarding SVM hyperparameters, we tested (1) the applied penalty L₂,(2) different regularization strength C ∈ (0, 1], (3) automatic adjusting of class weights inversely proportional to their frequency (balanced vs. unbalanced class weights), and (4) the type of kernel (linear, radial) and (5) decision function shapes (one-vs-one and one-vs-rest). Across classification tasks, balanced class weights, a linear kernel, and a one-vs-one decision function shape produced optimal results. The optimal values for regularization strength varied (range: [0.46; 0.91]) between classification tasks (see S3 Table).

BERT-base.

We used the pre-trained BERT (Bidirectional Encoder Representations from Transformers) base-uncased-model [23], and fine-tuned a separate model for each classification task (i.e., characteristic). BERT is a deep learning language model of the transformer architecture by Google AI. It has 12 transformer layers, 12 self-attention heads, and a hidden size of 768. Its pre-training included a masked language modeling task: a randomly chosen 15% of words in a sentence are masked, and the model learns to predict them from the sequences of words before and after the mask. Additionally, BERT was pre-trained to predict if a sentence follows a given sentence.

We fine-tuned the BERT-base-uncased model by adding a dense output layer that reduces the dimensions of the last layer to the number of labels per classification task, training all parameters simultaneously [24]. The different layers of BERT can capture different levels of semantic and syntactic information, with lower layers probably containing more general information. Therefore, we fine-tuned the layers with different decaying learning rates, following the approach of Howard and Ruder (2018) [25]. To find the maximal learning rate that is associated with a still-falling loss, that is, residuals (prior to the loss diverging), we ran a hyperparameter search. S1 Fig shows that learning rates up to 10e-5 are still associated with falling losses. We report results for BERT with a LR = 1e-5 and text length of 512, for the epoch with best results on the validation set out of maximally 12 epochs (this was usually around epoch 4–7).

Evaluation metrics.

We indicate precision, recall, and their harmonic mean (the F₁ score) for each classification task, all of which range from 0 to 1, with 1 indicating high performance. Precision is calculated by dividing the predictions for a specific category (for example, all transcripts labeled as mentioning a characteristic, such as suicide death) by all predictions (for example, those mentioning plus those not mentioning suicide death). It thus indicates how often the model’s label for this specific class is correct.

Recall reflects how many out of all true cases (e.g., transcripts mentioning a suicide death according to human labels) are detected by the model (e.g., labeled as mentioning a suicide death), and corresponds to the sensitivity of a medical test.

The F₁-score summarizes precision and recall as the harmonic mean.

Precision, recall, and F₁-score are first calculated for each class of a variable (intraclass scores) and can then be averaged across classes to get mean performance scores per classification task (macro-averages).

We report macro-averages for all scores and all models in the validation and test sets. For intraclass performance scores, we provide scores for the best two models, TF-IDF and BERT. We do not report accuracy scores, because our dataset contains mostly characteristics with large imbalances in the sample size between different classes. In such cases, accuracy can be very high even if a classifier always predicts the majority class, and is therefore a biased estimate.

Results

Percentage and number of transcripts per category

Fig 1 shows the proportion of transcripts per manually coded category for all characteristics, on average across the entire dataset. S4 Table reports the absolute frequency and proportion of transcripts per category separately for the training, validation, and test datasets. Since most texts only contain a few characteristics, the positive instance (the characteristic is present in the media item) is usually a minority class for all binary classification tasks (Fig 1A). Positive instances made up 6 to 14% of all transcripts for binary tasks (class size median = 51.0, min = 28, max = 284 in the test set), with suicide death mentions being the only exception (56.4%, n = 909 training set, n = 284 test set). For the type of narrative (Fig 1B), 57.9% (n = 884 training set, n = 277 test set) of transcripts described suicide predominantly as a problem, whereas 21.2% (n = 324 training set, n = 101 test set) focused on solutions, and 19.4% (n = 296 training set, n = 93 test set) emphasized both aspects equally.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Number of transcripts per category in each classification task in the training set.

(A) Binary characteristics. (B) Problem vs. solution narrative. (C) Main focus of a transcript. Proportions were similar across the training (n = 1811), validation (n = 453), and test set (n = 566). S4 Table reports precise sample sizes for the validation and test set. The maximum of y-scales of panel A, B and C differs.

https://doi.org/10.1371/journal.pone.0300917.g001

Regarding the main focus of transcripts (Fig 1C), items focusing on suicide deaths were clearly the most frequent (25%). Advocacy (11%), murder-suicide (9.6%), and policy (11.9%) were also among the more frequent focus areas, while healing stories (3%), or suicidal thoughts (0.5%), were rarely the major focus. Given these low percentages, considering the absolute number of examples that models were trained on is crucial before interpreting performance scores. The median number and range of broadcast items for a particular main focus was 89 (min 8, max 381) in the training set, and 27.5 (min 2, max 121) in the test set. The most rarely mentioned main focus areas definitely have too few transcripts in the test set to allow a conclusion about their performance scores. These include assisted suicide (n = 10), suicide clusters (n = 7), healing stories (n = 14), and suicidal ideation (n = 2). Another group of main foci (attempted suicide, mass murder, prevention, research) had below 30 items in the test set, and should be interpreted only with great caution. Only performance for the most frequent main foci, including advocacy (n = 55), suicide death (121), legal issues (n = 38), murder suicide (n = 46), and policy (n = 57) will be reported in the text below. Still, caution is warranted in their interpretation.

Model comparison

Fig 2 illustrates that both Tf-idf with SVM and BERT clearly outperformed the naive majority classifier in all classification tasks. In the following, we therefore focus on BERT and Tf-idf with SVM performances. Fig 2 shows that their performances do not differ substantially for all classification tasks and metrics. In general (but not always), BERT reached higher precision, whereas Tf-idf with SVM reached higher recall for level 2 and 3 classification tasks. As F₁-scores are a balance of these two, they are overall rather similar for BERT and Tf-idf with SVM.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Macro-averages of performance scores (precision, recall, F₁) in the test set.

https://doi.org/10.1371/journal.pone.0300917.g002

Intra-class performance and correlation with class sample size

We report intraclass performance scores for the two better models (BERT and Tf-idf & SVM) in the test set in S4 Table, and illustrate recall and precision for all tasks in Fig 3. Which of the two models was better varies across classification tasks, with mostly minor differences between the two models.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Intra-class precision and recall for Tf-idf & SVM and BERT in each classification task in the test set.

(A) Binary characteristics. (B) Problem vs. solution narrative. (C) Main focus of a transcript.

https://doi.org/10.1371/journal.pone.0300917.g003

For most binary classification tasks (Fig 3A), both BERT and Tf-idf with SVM achieved very high performance scores (above .90) when predicting that a transcript did not contain a characteristic (precision for negative class) or detecting texts without the characteristic (recall for negative class). Predicting the less frequent presence of a characteristic (the positive class) was usually more difficult for both models (often between .50 and .60). Transcripts without the characteristic were the majority class by a large margin, except for suicide deaths, where texts mentioning a suicide were more frequent (see Fig 1A). Suicide death was also one of the two classification tasks where performance was similar for both classes (negative class: >.75, positive class: >.82). The only other such binary tasks is celebrity suicide, where models can achieve high performance by learning only the names of celebrities who died by suicide from April 1 2019 to March 30 2020, and thus do not require as many training examples.

The pattern of high performance for the negative majority class, and low performance for the rare positive class, clearly indicates that the number of texts the models could learn from had a large influence on performance scores. Indeed, Fig 4 shows strong correlations between sample size per class and precision as well as recall for both models. For class sizes above 250, performance increases steeply and linearly; for lower class sizes, performance and class size are not clearly related. The scatter plots thus reveal that sample size can explain many of the differences in classification performances between classes, in particular for class sizes above 250. Performance and sample size for the negative category is lower for suicide death (the lonely red dot outside the red at n = 703) than all other binary tasks. For positive categories, performance scores for suicide death (blue dot at n = 909) is higher than for almost all positive categories in binary tasks. Only the positive class celebrity suicide (blue dots at performance around .85-.95) can be classified even better despite lower sample size (n = 209 in training set), likely thanks to celebrity names as highly indicative keywords.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Scatter-plot of intraclass performance scores for each model with class size in the training dataset.

For binary tasks, the positive (blue) and negative (red) classes have different colors. For main focus, all but three categories discussed in the main text are shown in the same color. All values displayed in this figure can be found in S4 Table.

https://doi.org/10.1371/journal.pone.0300917.g004

Fig 4 also reveals that sample size determines performance of multi-class tasks: Problem focus of broadcast items (cyan dot) has very similar sample sizes and performance scores as transcripts mentioning suicide deaths, although this task had four rather than two classes. Solution-focus performance (yellow dot) is much lower, as is the sample size of this class. Regarding class sizes below 250, it is interesting to note outliers with very high performance despite very low sample size: The upper left corner of some of the four panels in Fig 4 features some of the violet dots, indicating the main focus categories Assisted suicide, Murder suicide, Attempted suicide. All of these may be explained by highly indicative keywords, like “attempt”, “euthanasia” or “murder”, and”homicide”. Finally, outliers in the left bottom corner of panels, include the main focus categories Suicidal ideation and Other. With a class size of n = 2 in the test set for the first, and n = 19 for the second, these scores cannot be interpreted. The same applies for the category “Neither [problem or solution focus]” (dark green dot) with a precision of almost 1, as it had only n = 7 in the test set.

Pearson’s correlation coefficients between class size and performance scores are shown in Table 1. Correlations are reported across all categories, and for the subset of categories with a minimal number of 50 transcripts in order to exclude unreliable outlier performance scores which are based on very few labeled examples. This threshold of 50 in the training set corresponds to 15 in the test set. Applying it excludes the categories problem vs. solution focus = neither, and main focus = assisted suicide, cluster, healing story and suicidal ideation. Removing these unreliable performance scores results in correlations around .7–0.8 for both models and metrics.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Correlation of class sample size in the training set with classification performance.

https://doi.org/10.1371/journal.pone.0300917.t001

Exceptions from the tight sample-size performance correlation may indicate which aspects of a characteristic or model additionally influence performance. A first set of exceptions are tasks which can likely be solved with a few indicative keywords, including celebrity suicides (attempted, assisted and murder suicide). Possibly, the high precision (>.80) of both models for the positive outcome of a suicidal crisis, despite the small number of positive cases to train on (n = 118), could also be explained by indicative keywords accurately identifying this characteristic.

Further, an interesting pattern across positive classes in binary tasks (except suicide death) is the generally lower recall of BERT (around .40), but often higher precision. For example, Tf-idf with SVM had a recall advantage over BERT was .22 for monocausality, .17 for alternatives to suicide, and even .40 for enhancing myths (.40). This suggests that the recall metric of BERT was especially sensitive to small class sizes, while precision did not suffer as much. BERT’s precision scores were most clearly better (by .15-.20) than Tf-idfs with SVM for moncausality, suicidal ideation and enhancing myths. Possibly, these are tasks where connotation of words, their context, and syntax matters more than for other tasks.

Finally, classifying the main focus of the item was the task with the most classes and the most severe class imbalances. Many categories had too few example transcripts to allow interpretation of their performance scores. Of more frequent categories, transcripts focusing on murder suicides were classified quite well (BERT F₁ = .8, Tf-idf with SVM F₁ = .83), with Tf-idf with SVM being better for precision, possibly because a few keywords (e.g., “homicide”) were sufficient to label these correctly. Reporting focusing on suicide death was classified similarly well (BERT F₁ = .76, Tf-idf with SVM F₁ = .67), with BERT being better at detecting many cases. This result demonstrates that at least well-defined main foci of a text, with sufficient training examples, can be classified reliably. For broader main focus topics like Policy, performance was very similar, but only with BERT (F₁ = .76, Tf-idf with SVM F₁ = .62). For such more complex topics, the deep learning model might provide an advantage. Finally, a main focus on Advocacy was also classified moderately well (BERT F₁ = .72, Tf-idf with SVM F₁ = 0.67), with similar precision and recall. In general, class imbalances as seen here are a common issue in suicide prevention, as texts featuring each characteristic are much rarer than texts without the charateristic. We tested the effect of data augmentation techniques for some of the classification tasks.³³ The augmentation techniques did not consistently improve or otherwise change the performance scores on any of the classification tasks, suggesting that artificially augmenting rare data categories did not improve the performance of our models.³³

Discussion

The current study evaluated automatic labeling methods to categorize specific suicide-related content in broadcast media (i.e. radio and TV), as well as online media with machine learning. Based on a comprehensive annotation scheme for suicide-related content, we selected some content characteristics, aiming to include various classification types and difficulty levels that are relevant based on media guidelines for the reporting of suicide [3]. We then tested the ability of different machine learning models to accurately classify these characteristics, which is relevant for both research in the area of media effects as well as for surveillance efforts of suicide-related broadcast media content.

Overall, our results demonstrate that machine learning models can in principle achieve very satisfactory results for classifying various types of characteristics in suicide-related broadcast media content, including characteristics we estimated to be simpler and more difficult, as well as multi-class characteristics. This adds to earlier work that demonstrated the efficiency of machine learning in classifying social media content [11] and relevance for prevention [26]. In the current study, both the word frequency based representation Tf-idf with a linear SVM classifier as well as the deep learning model BERT were clearly better than the naive majority classifier across all tasks. However, neither BERT nor Tf-idf with SVM were clearly better than the other model. Both models also achieved similar performance in the test and validation set for almost all tasks, suggesting they generalized well to new data not seen during training.

For certain characteristics, our classifiers achieved high classification performances. The primary factor explaining these high performances, as well as the difference in performance across classification tasks, seemed to be the number of labeled examples for models to train on. Instead, if a classification required recognizing the narrative or the emotional connotation and meaning of a broadcast item, or the mere detection vs. the extent to which a characteristic was described, seemed less important. Given the importance of sample size, sound conclusions are possible only for categories that include a sufficient number of broadcast items. The first of these are all negative classes in binary tasks (a characteristic was absent), which were predicted and detected with scores above .90. This suggests that about 1250 examples may be enough to achieve highly reliable machine learning classification for suicide-related media content. Positive classes (a characteristic is present) were usually the minority class, with below 250 training samples, and their performance scores thus low (F₁ = .4-.6) Nevertheless, the high performances for predicting the absence of these same binary characteristics suggest that machine learning models can in principle achieve very satisfactory results. The only required prerequisite seems to be more training examples for broadcast items that mention a characteristic, that is, the positive class in each task.

A second characteristic that could be classified with high precision and recall was suicide deaths reports, where sufficient examples for the negative and positive class, allowed for scores above .80. This is likely much better than what keyword searches for suicide reports can achieve, which have the problem of mixing heterogeneous types of content containing the same keywords, together. Niederkrotenthaler et al., 2020 noted accordingly that most of the research about associations of suicide reporting and subsequent suicides was based on keyword searches only, mixing together entirely different types of narratives and contents and ignoring their potentially different meaning and effects [4].

A third characteristic classified with high precision and recall (>.90) was the problem focus of a broadcast item. This is particularly encouraging, since problem focus was one out of four classes, and requires detecting what a text focuses on, rather than just whether a characteristic is present. This suggests that many characteristics of similar complexity can be recognized with machine learning tools, as long as sufficiently large training samples can be put together. Good classification performance for suicide death as one out of 14 possible main foci of broadcast items (BERT F₁ = .76) further underlines this encouraging finding. It demonstrates that even 14 class classification problems can be solved reasonably well with as little as 400 training examples. A fifth case, solution focus, is a similar case: with only around 300 training samples, precision was around .70 in this four-class classification problem. In sum, the good performance for several complex characteristics, that require predicting which of many possible areas is the focus of a text, suggest that training models on larger samples is a very promising avenue for future research.

Regarding the featuring of a “healing story”, which have been at the forefront of the discourse of positive media potentials, ie the Papageno effect [6–9], the sample size in the current sample (there were n = 190 total healing story transcripts included) was comparable to a previous study conducted with Tweets [11]. For tweets, precision and recall using tf-idf with SVM were 0.44 (precision) and 0.64 (recall) in a 6-category classification task. For BERT, precision was 0.76 and recall was again 0.76 [11]. In the current study, sample size really determined performance (category yes with precision and recall of ~0.5 vs. no 0.97). The Twitter study suggests that having more balanced classes can improve performance for BERT for such small sample sizes.

For other classification tasks, e.g. suicidal ideation, comparisons with other samples are much more difficult, particularly because of the nature of the text analyzed. Several studies have investigated suicidal online communication and have applied machine learning particularly in order to differentiate suicidal from non-suicidal users [29–31]. Many of these models performed better as compared to the present model, which is likely due to their considerably larger sample sizes positive for suicidal ideation as compared to the present analysis.

A few points are interesting to mention regarding a comparison of Tf-idf with SVM and BERT results. Overall, we observed no clear advantage of BERT over Tf-idf with SVM. In contrast, BERT clearly outperformed Tf-idf with SVM when predicting similar suicide-related characteristics in tweets, that is, very short texts [11]. Possibly, the deep learning model’s capacity to predict the meaning of each word from the specific context it occurs in may be less crucial when working with longer content types, like the current broadcast media items. Here, we generally observed lower recall scores for BERT than Tf-idf with SVM for the positive class in all binary tasks where there were few training examples for the positive class (all except suicide death). This shows that the frequency of keywords is a useful feature to detect more instances when training sample sizes are small, and that especially the recall (the detection rate) of the deep learning model is sensitive to small class size (as here for positive classes).

Yet, at the same time, BERT often still achieved higher precision than Tf-idf with SVM for these rare positive classes. Compared to Tf-idf with SVM, BERT’s precision was most clearly better (by .15-.20) for moncausality, suicidal ideation and enhancing myths. This may hint that connotation of words, their context, and syntax matters more for these than other tasks, possibly because such descriptions are more subjective and nuanced, and simple word frequencies cannot capture this as well. Finally, BERT seemed to generalize better in the multi-class task of detecting the main focus of a text. Possibly, the ability of deep learning models to rely less on specific features of datasets helped BERT to generalize better for complex and multi-class classification tasks like detecting the main focus of a text. Taken together, the potential advantages of deep learning over word-frequency based models could not fully play out given the size of positive classes in our training data set. Still, they were recognizable in generalization abilities when training on few samples for multiple classes, and in higher precision for rarer characteristics that may require detecting nuance and changes in the meaning of words in particular contexts.

Independent from sample size, some types of classifications were indeed easier than others as we had assumed. Celebrity suicide could be classified very well (F₁>.90) despite low positive case examples. The main focus categories assisted suicide, attempted suicide or murder-suicide are further such examples, very reasonably high performance (F₁>.80) was possible with very low training sample sizes. However, at least in some of these cases, machine learning does not provide a substantial advantage over keyword searches, since they are easily identified with a few highly indicative keywords (names of celebrities + suicide, euthanasia, suicide bombing, attempt + suicide etc).

From a suicide prevention standpoint, the current machine learning approach developed in this work is particularly relevant when it comes to the assessment and categorization of large quantities of media items that would normally be retrieved with keyword searches only. As noted previously, searches based solely on keywords have the considerable disadvantage of mixing together entirely different narratives such as media items about celebrity suicide and suicide prevention items or items about hope and recovery [4]. The evaluation metrics of the present work clearly show that the present approach is superior to that type of keyword searching. In particular, the current findings appear helpful to make differentiations regarding characteristics that have been sufficiently represented in the current media items. This includes the determination A) if the item is about suicide reporting or not; B) includes information on celebrity suicide vs. not; C) the absence of any of the binary characteristics that were assessed (but not necessarily their presence), D) if the focus is placed on the problem or prevention of suicide; and E) if the main focus is about suicide death.

Although these categorizations do not capture all of the characteristics listed in media guidelines, they are of high relevance for suicide research and prevention as they allow for a more fine-grained categorization of texts as compared to simple keyword searches.

Conclusions

The current work makes a relevant contribution to suicide prevention research in that it investigates and confirms, for the first time, that machine learning can be successfully applied to assess prevention-relevant characteristics in broadcast media items. This included not only relative straight-forward but also more difficult characteristics of suicide-related media content such as the solution or problem focus of a media item. Recent studies about media content characteristics and their association with suicides have been limited in terms of number of media items included due to the huge amount of resources needed to assess media items. Similar limitations apply to the screening and surveillance of suicide-related media content that would allow a more pro-active approach to prevention such as the early reaching out to journalists and media professionals in instances of harmful reporting. Although this work does not replace human coding, it provides a strong basis for the extension to other media characteristics of interest, and subsequently, for the automatic assessment of large numbers of media items and their possible associations with behavioral outcomes of interest. Taken together, the current results highlight the relevance of machine learning approaches for future media studies related to suicide and prevention.

Supporting information