Problem

Existing multilingual hate speech detection solutions predominantly rely on general-purpose embeddings, often trained in English. In contrast, low-resource languages (in the hate speech detection task) like Spanish remain under-treated. This study addresses the gap by introducing domain-specific multilingual word embeddings for hate speech. These low-dimensional embeddings are computationally cheap to use and perform competitively with general-purpose ones and transformer-based models.

Research questions

1. RQ1: How effective are existing multilingual hate speech detection solutions, considering they are based on general-purpose embeddings and pre-trained models predominantly trained in English?
2. RQ2: What is the comparative effectiveness of low-dimensional hate-specific embeddings in cross-lingual hate speech detection, as opposed to other general-purpose representations and large pre-trained models, in English, Spanish, and Italian?
3. RQ3: How do hate-specific multilingual embeddings contribute to capturing non-traditional translations of words?

Contributions

The following are our main contributions:

1. We introduce the first domain-specific multilingual word embeddings for hate speech classification. The main characteristic of these embeddings is their simplicity, making them easily replicable for other languages.
2. We present a comprehensive quantitative evaluation of different approaches for cross-lingual hate speech detection, demonstrating the effectiveness of our proposal.
3. We propose a strategy for qualitatively evaluating these hate speech multilingual embeddings by exploring words’ contexts. The relationships we found support the hypothesis that the context of words in a hateful scenario is different from that in a general scenario.

Reproducibility

All the code, experiments, and resources will be publicly available in the form of a repository.

Organization of the paper

We first describe the relevant related work in Section 2. In Section 3. we describe our approach to creating hate speech word embeddings. In Section 4. we present a quantitative evaluation of monolingual and cross-lingual hate speech detection. In Section 5. we describe our qualitative evaluation and results. Section 6 presents a summary of findings and conclusions.

Warning

Because of the topic of our research, certain examples contain potentially offensive terms.

2.1 Monolingual hate speech detection

As for other NLP tasks, English has been the most addressed language in hate speech detection. In the related literature, several methods are leveraged for in-domain English evaluation. Some of these approaches, mainly in the early years of task development, use traditional machine-learning models [3, 10, 11] such as Support Vector Machine (SVM). Logistic Regression (LR). and Random Forest (RF). These algorithms are commonly applied using existing software tools such as WEKA(https://www.cs.waikato.ac.nz/ml/weka/), scikit-learn (http://scikit-learn.org/stable/index.html), and combined engineered features [12, 13].

Deep-learning methods have gained popularity in recent years for addressing the task, in addition to the conventional machine-learning algorithms. Convolutional Neural Networks (CNNs) [14, 15] and Recurrent Neural Networks (RNNs) [1, 2, 16, 17] are popular architectural choices for detecting hate speech. Models such as BERT [18]. which are based on Transformers, have shown success in various NLP tasks [19–22]. BETO (Spanish BERT) [23]. Italian BERT [24]. Chinese BERT [25]. and various others have been developed to facilitate progress in different languages.

In addition, there have been critical analyses of English-based systems and datasets to provide a better understanding of the problem and the possible biases existing in datasets and models [26–28].

2.2 Cross-lingual hate speech detection

As demonstrated for other tasks [29, 30]. a multilingual approach to the hate speech detection problem could help improve the state of the art for under-represented languages.

For languages with little to no labeled resources, we need an approach in which no information about the target language is used during the training process. We refer to this constrained scenario as cross-lingual.

Translating training and testing data into a common language is one of the strategies employed for this task [31]. Meta-information from network dynamics and text messages has been used as features in the literature [26]. This type of feature is considered language-independent as it is not directly related to the language in which the text is written. LASER [32] is a recently proposed model for producing multilingual embeddings for sentences. These embeddings have been combined with traditional machine learning models [6, 7] for cross-lingual hate speech detection. Another approach involves fine-tuning pre-trained multilingual models like BERT [33] (https://github.com/huggingface/transformers) [5, 6, 34] or XLM (https://github.com/facebookresearch/XLM) [35] on the training data.

2.3 Specific embeddings for the hate speech detection

There is limited research on specialized word representations and specific pre-trained models for this task, as seen in the works cited [36–38]. Similar to our work, some papers [37, 38] describe the construction of task-specific word embeddings, using techniques such as Word2Vec or GloVe to construct low-dimensional word embeddings. These works also utilize unlabeled data extracted from social networks considering specific hateful queries. Other works describe the construction of specific task word embeddings using different techniques. For example, Badjatiya et al. [1] use an LSTM model for building word embeddings from a dataset of labeled hateful tweets. There have also been efforts to adapt existing pre-trained models, as in the case of HateBERT [39]. where the authors retrained BERT using English-banned comments from Reddit.

Despite these efforts in the monolingual, mainly English scenario, current cross-lingual techniques often use general-purpose features or general pre-trained models. To our knowledge, no works are creating specialized multilingual representations (word embeddings) for this problem. Considering the particularities of the hate speech phenomenon, domain-specific representations will contribute to improving cross-lingual classification, and we focus our work on this. In contrast to established methodologies, our proposed approach distinguishes itself by addressing the challenge of multilingual hate speech classification through the construction of specific representations.

2.4 Projection-based multilingual word embeddings

One approach for creating multilingual embeddings is by using the so-called projection technique [40–42]. This requires resources that are relatively easy to obtain for most tasks. The idea is to linearly project two vector spaces into a common space by optimizing the relationship between dictionary-paired vectors obtained from bilingual dictionaries. The bilingual dictionaries can be induced from the data (unsupervised methods) [29, 43, 44] or provided beforehand (supervised methods) [9, 41, 45, 46].

We select this type of approach for the creation of our multilingual hate speech embeddings.

3.1 Domain-specific monolingual embeddings

We describe the creation of monolingual word embeddings from social media. First, using a set of seed terms (or queries), we retrieve a set of social media text messages. In particular, we use the online social network Twitter, which is a microblog (i.e., short-text-based) platform. Our seeds are based on terms contained in public lexicons of hateful words [11, 48]. These seeds guarantee that the retrieved messages contain hateful terms, causing, as an overall effect, a higher probability for hateful messages to appear in the resulting corpora. We focus on English, Spanish, and Italian. Specifically, we collected 30 million tweets in English and 10 million tweets in Spanish and Italian, each.

Using this corpus, we train Word2Vec [49] 100-dimensional word embeddings for each language individually. By applying the Word2Vec technique, we obtain (mathematical) close vectors for semantically similar words. Since our corpus is biased towards hateful content—thanks to the weak supervision provided by the seed terms—we consider our resulting embeddings to be domain (hate) specific. Notice that the previous result proceeds for any other algorithm for creating the monolingual embeddings.

3.2 Alignment of monolingual spaces

In this section, we describe how different monolingual word embeddings are aligned into a single embedding space. As an alignment algorithm, we adopt a technique based on canonical correlation analysis (CCA) proposed by Faruqui and Dyer [9]. In this process, a pair of monolingual word vectors are projected into a common space by learning two projection matrices V and W that maximize the correlation between the dictionary-paired projected vectors.

More specifically, assume that and are two word-embedding matrices corresponding to two different languages, where every embedding is a column in each matrix. We note that n₁ and n₂ might be different as the sets of embeddings might be created from vocabularies of different sizes.

4.1 Datasets for evaluation

We used three labeled Twitter hate speech datasets for our experiments. A summary of these datasets is presented in Table 1. Each dataset is detailed next:

1. English dataset: This dataset consists of the English dataset by Arango et al. [26]. created in 2019, and the one created for SemEval in 2019 by Basile et al. [50]. Both datasets contain hate speech against immigrants and women. They originated in the United States; therefore, hate targets, as well as specific terms, are framed within that particular cultural context.
2. Italian dataset: This dataset is composed of the dataset by Sanguinetti et al. [52]. which is part of the “Hate Speech Italian Monitoring Program”. The hate targets are women and immigrants.
3. Spanish dataset: This dataset consists of the dataset by Pereira et al. [51]. which contains hate speech related to racism, sexism, and xenophobia. Additionally, we used the Spanish portion of the SemEval 2019 dataset by Basile et al. [50]. which includes hate speech against immigrants and women. The tweets originated in Spain.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Description of the datasets used in our evaluation.

For each dataset, we show the number of tweets per class.

https://doi.org/10.1371/journal.pone.0306521.t001

Datasets within the same language were merged, and their labels were binarized, following a commonly used strategy in this area for creating larger collections [6, 53].

4.2 Bilingual dictionary

We use a bilingual dictionary consisting of word-aligned pairs from the Hurtlex lexicon [48]. Hurtlex is a multilingual lexicon that we use to match hateful terms between different languages. It has been successfully used for cross-domain hate speech detection [54, 55]. The dictionaries comprise more than ten thousand words typically used in the hate speech domain and their translations to a second language. The particularity of Hurtlex is that it includes terms with different colloquial equivalents that are not usually included in generic dictionaries, as well as words that could typically appear in hateful content. An example comparing the diversity of Hurtlex with the Muse dictionary can be found in our code repository. According to Shakurova et al. [56]. better results are obtained when the bilingual lexicon is from the specific domain of the task.

4.3 Quantitative evaluation

Overall, we study cross-language classification in a transfer learning scenario, when the classifier is trained on labeled data for one language and then is used to classify in another.

The Algorithm 1 describes the experimental process. We used the three datasets described in Section 4.1 in languages English (EN), Spanish (SP), and Italian (IT). For each of the possible training testing combinations of the three datasets (SETUPS), we compare different multilingual embeddings (EMBEDDINGS) including our proposal. For performing the comparison we tested several models (MODELS) and reported the best performance by setup and by word embedding (RESULTS). In Tables 2 and 3 we show monolingual and cross-lingual results. Next, we detail every section of the process.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Experimental results for monolingual hate speech detection.

The cells show the F1 score of the best model for each combination of setup and input representation (embedding). The bold numbers represent the best score per setup.

https://doi.org/10.1371/journal.pone.0306521.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Experimental results for cross-lingual hate speech detection.

The cells show the F1 score of the best model for each combination of setup and input representation (embedding). The bold numbers represent the best score per setup.

https://doi.org/10.1371/journal.pone.0306521.t003

SETUPS. We test the nine possible source and target language combinations (e.g. SP → IT).

Each dataset described in Section 4.1 is split into three sets: training (80%), testing (10%), and validation (10%) for adjusting the hyper-parameters of the models. Even though our main interest is the cross-lingual scenario, we also include monolingual experiments to have reference performance of models. Intuitively, the closer the cross-lingual results are to the monolingual ones, the better they are at transferring knowledge from one language to another.

Multilingual embeddings can be used in monolingual scenarios, although their aligned characteristic are not useful in this case.

Algorithm 1 Experimental process for comparing our hateful embeddings with general purpose word embeddings.

1: SETUPS = [ES → ES, EN → EN, IT → IT, EN → ES, EN → IT, ES → EN, ES → IT, IT → EN, IT → ES];

2: EMBEDDINGS = [LASER, MUSE, mBERT, HATE_EMB];

3: MODELS = [LR, XGB, SVM, RF, DT, NB, CNN, FNN, LSTM, MHATTN, LSTMCNN, LSTMATTN];

4: ALL_RESULTS = {};

5: for setup in SETUPS do

6:  train, val, test = get_partitions(setup)

7:  for emb in EMBEDDINGS do

8:   best_result = 0

9:   for model in MODELS do

10:   best_model = h_tunning (emb, model, train, val)

11:    result = test (emb, best_model, train, test)

12:    if result > best_result then

13:     best_result = result

14:    end if

15:   end for

16:   ALL_RESULTS[setup][emb] = best_result

17:  end for

18:  if is_monolingual (setup) then

19:   BERT = get_BERT (setup)

20:   best_BERT = fine_tunning (BERT, train, val)

21:   result = test (best_BERT, train, test)

22:   ALL_RESULTS[setup][fine_tuned_BERT] = result

23:  end if

24: end for

25: Return ALL_RESULTS

EMBEDDINGS. We consider five types of multilingual embeddings: MUSE [29]. a set of general embeddings aligned for multilingual contexts; BERT [18] and XLM [57]. general purpose pre-trained models for NLP that can be used to produce embeddings for sentences (sequences of words); LASER [32]. a recent model for producing multilingual sentence embeddings and HATE_EMB our proposed embeddings described in Section 3.

In the cross-lingual setups, we use multilingual BERT (mBERT for short). In addition, we use BERT pre-trained with monolingual data Italian [24]. Spanish [58] and English [18]. those for performing fine-tuning in the specific task of hate-speech detection. We evaluate the usefulness of these three representations by using them to generate input features for several classification models. At the same time, they perform as baselines for comparing the hate embeddings.

MODELS. We evaluate several traditional machine learning models including Logistic Regression (LR), XGBoost (XGB), Support Vector Machines (SVM), Random Forest (RF), Decision Trees (DT), and Naive Bayes (NB) classifiers. We also incorporate deep learning models, including Convolutional Neural Networks (CNN), Feedforward Neural Networks (FNN), Long Short-Term Memory Networks (LSTM), and Multi Head Attention (MHATTN). In addition, we combine LSTM and CNN layers (LSTMCNN), as well as LSTM with Attention [59] (LSTMATTN). We tuned these models to find the best possible values for the different hyperparameter combinations (e.g. batch size, learning rate, dimensions, number of layers, etc.). The complete grid search and the best resulting parameters will be described in a section of our code repository. In the case of monolingual setups, we perform fine-tuning for the pre-trained BERT models according to the setup. case, the BERT model serves as input representation and classification model (BERT Italian [24]. BERT Spanish [58] and BERT English [18]).

4.4 Results

Next, we detail our results for the monolingual and cross-lingual classification task, as described in Section 4.3.

5.1 Cross-lingual relations in vectors spaces

BLI inspires our first qualitative evaluation, extracting the most related terms across different languages given a seed term. This exercise aims to identify equivalent hateful meanings rather than direct translations in both vector spaces and within the labeled datasets.

In Table 4. we present a sample of the relationships between terms comparing the hate embeddings (HateEmb) with the general-purpose multilingual embedding MUSE.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Terms extracted from the datasets and the 1-nearest neighbors in vector spaces of different languages.

https://doi.org/10.1371/journal.pone.0306521.t004

We manually selected these terms to represent groups that have been targets of hate, ensuring they were not in the bilingual dictionary used for embedding alignment. This allowed us to display a new relationship.

For each selected source term. we show the nearest neighbors (NN) terms in a language different from the source language. In most cases, the nearest neighbors in the MUSE space are terms whose standard meanings are the same in both languages. For example, in Table 4. for the Italian word “migranti”, we found that the nearest terms in the MUSE space are “migrants” in English and “migrantes” in Spanish. These are literal translations for the three languages. On the other hand, the nearest neighbors in the hate-specific embedding space are “illegals” (English) and “palestinos” (Spanish).

We can observe that these are neither direct nor neutral translations of the original word “migranti”. However, these words are likely to appear in similar contexts for hateful text. Within this scenario, the term “illegals” is used commonly to refer to a person who migrates to the U.S. illegally. Similarly, the word “palestinos” (that means a person of Palestine origin) is associated in hate-related contexts to a person who is an immigrant from an Arabian country.

We argue that evaluating the hate embeddings considering literal translations is not suitable since in the hateful content, words like “migranti” could acquire different meanings. The correct equivalence that should have been found is unknown, due to the complexity of the hate speech problem. Moreover, expecting the same relationships across different languages (e.g. “migrants”—“terrorist” = “migrantes”—“terroristas”) is not correct either. The targets of hate in different languages are different depending on the socio-cultural scenario. In most of the cases, we were able to observe non-trivial translations when exploring our hate embeddings. However, in a few cases, we could observe that the equivalences are the same as in MUSE (e.g. “negros” in Spanish, as “blacks” in English and “neri” in Italian). More experimentation is needed to derive a more robust conclusion, but our qualitative results provide positive evidence that our domain-specific embeddings capture non-trivial meanings and translations.

5.2 Cross-lingual relations in labeled datasets

In this section, we introduce a method for qualitatively exploring the ability of our embeddings to capture equivalences between hateful concepts in different languages over a labeled dataset. In the previous section, we use similarity measures (nearest neighbors) over the general embedding space, considering all the vocabulary used to construct those embeddings (unlabeled general data). In this section, we focus exclusively on texts from the positive class of hate speech labeled datasets in different languages. That is, we focus on multilingual data that we know that contains hateful information. We use the hate embeddings plus association rules to devise a similarity measure among terms in different languages. This experiment would serve as an intrinsic qualitative evaluation as we can assess how good the translations obtained for simple hateful terms. We next explain in more detail the method we devised to obtain the equivalences.

5.2.3 Context-based similarity for hateful words.

For each dataset of every language, we first selected some frequent words (seed terms) appearing in the hateful-labeled texts. Then, for each seed, we selected the words that are more (context-) similar to all the words appearing in hateful labeled texts in a different language. Table 5 shows examples of terms and the top two most similar words using our hate-specific embeddings and Table 6 shows the results for a similar experiment but using MUSE embeddings. As a comparison, the tables also show a similar experiment for the non-hate texts. More examples can be found in our repository.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. For some terms, we found the top most similar ones in different languages using the HateEmb embeddings for hateful and non-hateful classes.

We can observe different relations depending on the nature of the expressions. The numbers represent the similarity achieved in each case (%).

https://doi.org/10.1371/journal.pone.0306521.t005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Results for an experiment similar to the one presented in Table 5.

But considering the general-purpose MUSE multilingual embeddings instead of our hate-specific embeddings. We can observe different relations depending on the nature of the expressions. The numbers represent the similarity achieved in each case (%).

https://doi.org/10.1371/journal.pone.0306521.t006

Even though the labeled datasets are relatively small and from specific types of hate, we still find interesting cross-lingual relations. As expected, these relations are different depending on the nature of the text (hateful versus non-hateful).

For example, using hate-embeddings, for the Italian word “terroristi”, which is a neutral translation of the English “terrorists”, we found the words “muslims” and “fascistas” (see Table 5). In particular, “fascistas” is an adjective related to fascism and is used pejoratively. Another example that illustrates the results of using hate-embeddings is that the most similar word to “girls” in English was “perra” for Spanish. This last term is a very demeaning way to refer to women in that language.

On the other hand, for Muse embeddings (see Table 6) the closest term to “girl” is “gusta” (like) which is not semantically related. In addition, the most similar term to “gitano”, which means gypsy in Spanish, was “invaders”. However, using Muse embeddings the closest word is “hopefully”, which is not meaningfully related.

The relationships that we have found using our hate embeddings can be interpreted as a cross-cultural similarity in how concepts are related to each other within hateful contexts. Furthermore, these relationships, although only qualitative, are very difficult to find when we repeat this experiment on general-purpose multilingual embeddings.

5.3 Limitations

CCA, the method used for aligning the word embeddings into a multilingual space, relies on two resources: unlabeled data for constructing monolingual embeddings and bilingual dictionaries for the alignment process. The first resource is relatively easy to obtain, but bilingual dictionaries may be unavailable for certain languages.

The dictionary used in this paper, Hurtlex, contains equivalences relative to the hate speech phenomenon in 50 languages. However, the technique’s effectiveness may still depend on the specific characteristics of these dictionaries, such as the number of equivalences and quality of them. However, CCA’s applicability to low-resource languages can be improved using some additional strategies. For example, refining the model after the creation of multilingual embeddings. The initial dictionary can be augmented by inferring additional bilingual equivalences from these vectors. This expanded dictionary enables another iteration of the method, allowing the process to be repeated multiple times to achieve improved embeddings.

Another limitation of projecting multilingual embeddings, as well as using pre-trained language models, is the risk of introducing biases. The training data for creating the monolingual embeddings may contain stereotypes or discriminatory content that can reinforce cultural prejudices and impact the performance of hate speech detectors. In addition, the projection techniques, relying on statistical correlations between language embeddings, may introduce a skewed representation of languages based on the bilingual dictionaries used.