Lexicon-based classifier

Lexicon-based classification is the most widely used approach to text classification in the social sciences. It is an unsupervised technique that is based on the filtering or labeling of texts with the help of a lexicon, or a list of relevant key terms (e.g., [8, 10]). Typically, the lexicon that is used is created by an expert in the area of interest, who has knowledge of the terminology used in context in this area.

When we use lexicon-based classification to categorize Twitter data from UN agencies, the content of tweets is tokenized (or split up into separate words) and then inspected. If one or more of the tokenized words in the tweet matches with key terms contained within the lexicon, then the tweet is classified as dealing with climate change. If no words match, then the tweet is classified as not dealing with climate change. The number of tokenized words which need to match for positive classification is a design parameter.

There are many possible ways to classify tweets with respect to the topic of climate change. However, there is no standard lexicon for this type of classification. For this article, in order to benchmark the ML methods with a reasonable classifier, we select the lexicon from [3]. It consists of the following key terms: “climate”, “climatechange”, “globalwarming”, “climaterealists” and “agw”(an abbreviation for “anthropogenic global warming”). The lexicon has a decent performance and constitutes a good baseline for our empirical performance assessment.

Supervised approach to text classification

An efficient alternative to lexicon-based classifiers is the use of supervised ML methods. This class of methods requires a manually labeled dataset (see Eq (1) and Fig 3 in the S1 Appendix). By fitting a set of N examples given in this labeled dataset, the methods obtain a model that approximates the predictor function. To fit the model, we choose a mapping that minimizes the empirical average loss. This means that, in fitting the model, the computer attempts to minimize the difference between the predicted value and the actual observation. This can be formally expressed as: (1) where is some chosen loss function that reflects the closeness of the model prediction for features x⁽ⁱ⁾ to the true label y⁽ⁱ⁾ for observation i.

The model , being a function of the features x, is also a function of a set of its own parameters w = [w₀, …, w_K−1]. The fitting of the model is therefore carried out by searching the parameter vector w that minimizes the average loss in Eq (1). This process is referred to as the training of the model. By monitoring loss we can track the in-sample performance of the model, or more specifically how closely corresponds with the data to which it is being fitted. Because the model learns the “average” mapping from features into labels, we also expect the model to have a decent out-of-sample performance (i.e., performance on new and as-yet-unseen examples). Unfortunately, the in-sample performance of a model might not be indicative of its out-of-sample performance, as we shall see below.

If the model is too simple (i.e., it has too few parameters w to capture the behavior of the data), it is unable to fit the data well and has poor in-sample performance. This is known as underfitting and is an indicator that the model is poor in general. To avoid underfitting, more complex models have to be considered. The true predictor f(⋅) can be a complicated relation between features and labels. To approximate it well, the model may sometimes need to be complicated and include many parameters K. For example, a linear model of this kind could be given by the following expression: , where φ_k(x) is some function of features x for k = 1, …, K − 1. More complicated models (e.g., polynomials) might involve even larger numbers of parameters.

The more complex the model , the greater the possibility to decrease the empirical loss in Eq (1) and improve its in-sample performance on the given dataset . However, when the number of parameters becomes too high (relative to the number of examples N in the dataset), the training process might start confusing the noise in the data with the behavior of the feature variables. The model becomes tied to particular examples in the dataset and does not generalize to unseen data. That is, the in-sample performance of the model on the dataset becomes highly misleading and does not reflect its out-of-sample performance, which is known as overfitting. Thus, a general rule for picking the model is the following: the model should be complex enough to overcome underfitting, but simple enough to not reach overfitting.

To detect overfitting, the entire dataset can be split into two parts: a training set and a validation set . This makes it possible to validate the out-of-sample performance of a model on the validation set. Then several models can be fitted to the training set, while also validating their out-of-sample performance using the validation set. This validation can be used as a basis for selecting a model that is complex enough to generalize to new examples and performs the best in terms of average loss among all other considered models. Following this, the best model can be retrained using the entire dataset (minus the test set), whilst also ensuring that it generalizes well when applied to as-yet-unseen data.

This approach makes it possible to validate the models and select the best one in terms of the out-of-sample loss. However, it might still not be possible to assess the actual performance of the model on as yet unseen data. This is because the process of model selection contaminates the dataset, injecting a dependency between the model and the data based on which it was selected. Hence, the validation loss is not representative of the actual performance on yet unseen data. To tackle this problem, it is necessary to make yet another partition of the dataset . That is, prior to cutting subsets and from the dataset , another subset of data, called a test set is withheld from the validation set. This test set is not touched during the training and validation, and is used purely for assessing the classifier’s performance on unseen data.

Traditional machine-learning classifiers

By traditional ML classifiers we mean a collection of ML algorithms which have previously been developed for classification tasks. Those are built on different principles and have different computational complexities. Their performance depends on the task at hand. In this section, we will review the most popular ML classifiers and specify the variants of the classifiers that will be used for our subsequent performance comparisons. In our analysis, we use the SciKit-Learn implementations [27] of the traditional ML algorithms, whereas for the deep neural classifiers, we use the TensorFlow implementations set up via the Keras API [28]. For each of the methods, we test the three available vectorizers, as well as optimize the number of features to maximize the classifier’s performance.

Logistic regression.

Logistic regression (LR) is a classification algorithm that was invented in [29] and has been widely applied in text classification [30, 31]. The classifier models a binary random variable representing the label y given the data x. The model, parameterized by weights w, is given by (2) where σ(⋅) is the sigmoid function representing the conditional probability of a positive class label given the observed features x₁, …, x_M.

An example of a LR classifier in action is shown in Fig 1 on exemplary synthetic data. The two classes are separated by a decision boundary, which is determined by the equal-probability line on the sigmoid-based probability map (see the black solid line on Fig 1). For all data points with probability above the decision threshold we predict , while for the rest we predict .

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Illustration of a logistic regression classifier.

The sigmoid-based surfaces shows the probability of the positive class given the observed data.

https://doi.org/10.1371/journal.pone.0290762.g001

The advantages of the LR classifier are its simplicity, efficient training, interpretability and the absence of assumptions about the class distribution. The downsides are overfitting for a large number of features (compared to the number of observations) and sensitivity to multicollinearity among predictors.

A classifier is characterized by a set of hyperparameters which refer to parameters related to the model’s architecture and whose values are set before the learning process begins. This is in contrast to the model’s own parameters which are learned during the training. The SciKit-Learn implementation of the LR classifier has several hyperparameters, including regularization strength and maximum number of iterations. For our numerical performance assessment, these are tuned to get the best classification performance.

Support vector machines.

Support vector machines (SVMs) were first proposed in [32] for linearly separable problems and later generalized to non-linear problems in [33]. SVMs have been frequently used for the classification of political texts [12, 13, 18]. They are sometimes seen to be the classification method to be used for short texts in social sciences [8]. Unlike the LR method which provides a probability as its output, the output of an SVM gives the direct prediction of the label.

SVM classification works by fitting a hyperplane that separates the classes and maximizes the separation margin [32]. Support vectors refer to those samples that are the closest to the separating hyperplane. These are the only samples that impact the SVM training, as they are most likely to cause misclassification. The margin is given by the length of the projections of the supporting vectors onto the vector of parameters w which is perpendicular to the separating hyperplane. The average loss is given by Eq (1) with a hinge loss function.

The above intuition only works for cases where classes are linearly separable (see Fig 2a). In the general case, when classes may overlap in such a way that it is not possible to fit a separating hyperplane between their data points, the SVM principle can be extended to soft margins. This allows some observations to reside on the other side of the hyperplane [34]. Furthermore, a non-linear transformation ϕ(⋅) can also be applied to the data, expanding the feature space to a higher dimension in which there may be a linearly separating hyperplane. This transformation can be done, e.g., by adding higher-order polynomial terms. However, a problem with this approach is that it might explode the computational complexity of the algorithm and lead to overfitting. In order to overcome this problem, the “kernel trick” can be invoked. Namely, by means of Lagrangian duality, the maximization of the soft margin is reformulated in terms of scalar products between data points. There are kernel functions k(⋅) where k(x⁽ⁱ⁾, x^(j)) = ϕ(x⁽ⁱ⁾)ϕ(x^(j))—e.g., polynomial, Bessel, radial basis functions (RBFs). This means that it is possible to skip computing the data transformation ϕ(x). Instead, only computing the kernel function will give the dot products in the transformed feature space for the optimization. The obtained separating hyperplane translates back to the original feature space as a non-linear decision boundary (see Fig 2b that illustrates an SVM with an RBF kernel on synthetic data).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Illustration of a support vector machine classifier.

A hyperplane is fitted to separate the two classes and maximize the margin. Linear kernel (a) is used for the case of linearly separable classes, while radial basis function kernel (b) is used for the case with class overlap.

https://doi.org/10.1371/journal.pone.0290762.g002

The advantages of the SVM classifier are its operation efficiency with high-dimensional data, ability to model non-linear decision boundaries, and efficient handling of small datasets. The downsides are its sensitivity to the choice of the kernel function, high computational complexity for very large datasets, and the lack of probabilistic interpretation (unlike in LR).

Because it is known that text classification is usually linearly-separable [12], we use an SVM implementation with a linear kernel for our empirical analysis. The regularization hyperparameter, defining the “softness” of the margin, is tuned to maximize the performance of the SVM.

Random forest.

Random forest (RF) classifiers [35, 36] are widely used for text classification (see, e.g., [37, 38]). The algorithm uses the so-called decision tree approach [39, 40]. The latter relies on the intuition that, instead of seeking a complicated mapping function , one can partition the feature space into disjoint regions and fit a simpler model in each of those. This is done during training by recursively splitting the space of possible values for each feature at a certain threshold. For example, at a given iteration, the entire input space is split into a pair of half-spaces {x : x_j ≥ t_j} and {x : x_j < t_j}, where t_j is a threshold for a certain feature j. The split is done in a greedy way to minimize some loss function (e.g., misclassification error, Gini impurity, or entropy), without looking ahead at future splits. This process is repeated on each of the half-spaces recurrently until a stopping criterion is fulfilled (e.g., there is no information gain from further splits). In this way, a binary “decision tree” is constructed, which sets the thresholds for the classification of new samples.

Once the training is done, a new observation (x, y) is passed through the entire decision tree. As it passes through the tree, the observation experiences a sequence of learned conditional statements (greater or lower than a threshold) for each feature x_j where j = 1, …, M. At the bottom of the tree (in a leaf node, where there is no further split), the prediction is obtained as the majority class of the data points satisfying all the conditions along the path. Illustrations of predictions with decision trees are shown as one block in Fig 3. Each circle depicts a conditional statement over a feature with respect to a threshold. The thick lines depict the paths of observations as they travel through the decision tree, satisfying all the conditions on their way. The numbers below the trees indicate the predicted classes.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Illustration of a random forest classifier.

The classifier consists of a collection of decision trees, each making its own prediction regarding the class of an unseen data point. The predictions are then used in a majority voting for producing the final prediction.

https://doi.org/10.1371/journal.pone.0290762.g003

An issue with decision trees is that they typically overfit training data. A widely-used way to improve generalizability is to average the predictions of a number of decision trees by means of bagging [41]. With bagging, several bootstrap training sets are randomly sampled (with replacements). A decision-tree model is then fitted to each of these training sets. The predictions from the models are combined, e.g., using majority voting over decision trees in the forest. This improves the model’s generalizability by reducing the variance of the classifier, without increasing its bias. The RF approach is an extension of bagging that also randomly selects subsets of features used in each data sample. Fig 3 illustrates an RF classifier in action. A total of B decision trees are formed by considering various subsets of available features. Each decision tree makes its own prediction on its own subset of the training data. Then majority vote defines the final prediction label of the random forest classifier.

The RF classifier has the advantages of good interpretability, possibility to assess the importance of input features, and efficient avoidance of overfitting with many trees. The main disadvantage of this method is its computational complexity and slow speed with a large number of trees, which limits its applicability for real-time operation.

For our empirical analysis, we adjusted the maximum depth of the decision trees and the number of estimators in order to optimize the RF classifier’s performance.

k nearest neighbors.

The k nearest neighbors (KNN) algorithm [42] is a simple classifier which has been widely used by researchers for text categorization (see, e.g., [43, 44]). In contrast to other ML algorithms, KNN does not have a training phase. Instead the training set is simply stored and the actual computation is deferred to the prediction phase. In a nutshell, given an unlabeled observation x, the algorithm calculates the distances d(x, x⁽ⁱ⁾) to training observations x⁽ⁱ⁾, for i = 1, …, N, in the feature space and finds k nearest neighbors to x. The classes of these neighbors are used to weigh the available classes for the given observation. That is, majority voting is performed on the observations within the set of k nearest neighbors. Fig 4 illustrates a KNN classifier in action on synthetic data.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Illustration of a k nearest neighbors classifier.

For an unseen data point, the distances towards its k nearest neighbors are computed, and the majority class gets assigned.

https://doi.org/10.1371/journal.pone.0290762.g004

In our analysis, we use KNN with uniform weights. Because of the majority voting procedure, we set the hyperparameter k to an odd number. We sweep k and another hyperparameter, leaf size, to optimize the KNN performance.

Naïve Bayes.

Naïve Bayes (NB) is a probabilistic classifier whose basic idea is to use the joint probabilities of words and categories to estimate the conditional category probabilities given a data point [45]. It has been successfully used for text classification [46, 47]. It is called “naïve” because it makes an assumption that features are conditionally independent given the label. This assumption simplifies the computations carried out by the NB classifier, when compared to a non-naïve Bayes classifier, because the NB classifier does not use word combinations as predictors.

Given an observation x, such as a short text, the NB classifier assigns the most probable category according to (3) due to Bayes’ theorem. The term p(x) is not dependent on class c and hence skipped. Since it is difficult to estimate p(x|c), the naïve Bayes assumption is made. Namely, we factorize this conditional distribution as p(x|c) = p(x₁|c)p(x₂|c) ⋯ p(x_M|c), which simplifies the computations. Moreover, p(c) is estimated as the fraction of the tweets in class y in the training set. Meanwhile, p(x_j|c) is estimated as the relative count of feature x_j in class c to all features, with applied Laplace smoothing.

The advantages of the NB classifier are its simplicity, good scalability, quick training and insensitivity to irrelevant data. The disadvantages of the method are the assumption of independent predictor features and zero-probability problem for features present in the test set, but not occurring in the training set.

There are several versions of the NB classifier (e.g., Gaussian, Bernoulli, multinomial). The operation of a Gaussian NB classifier on exemplary data is demonstrated in Fig 5. Studies have reported that a multinomial mixture model shows improved performance scores on several data collections, when compared to other commonly used versions of the NB approach [46]. Because of this, for our empirical analysis, we use a multinomial NB classifier with the smoothing hyperparameter tuned to achieve the best performance.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Illustration of a Gaussian naïve Bayes classifier.

The classifier fits two Gaussian distributions (depicted with contour plots) to the categorized labels in the training set. The decision boundary is determined by the point where the probability densities for the two categories take the same value.

https://doi.org/10.1371/journal.pone.0290762.g005

Deep learning

Deep learning (DL) is a prominent supervised ML framework for data-driven predictive modeling. Despite its promising classification capacities, DL continues to be a much less used approach to text classification in social science, when compared to its use in such fields as computer science. A DL classifier can be thought of as a black box: data go in, decisions come out (see Fig 6a). Inside the black box there are a large number of parameters that are learned during the training. In this way, the DL model approximates a predictor function that describes the observed data. Because of the way its operations resemble those of the human brain, the broader class of methods to which DL belongs is often referred to as artificial intelligence (AI).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Illustration of a deep-learning model.

The classifier provides a complex non-linear function (a) that maps an input feature vector into a predicted label. The function is learned from the training set by adjusting the parameters of a set of internal units arranged in, e.g., a layered fully-connected structure (b). (a) Deep-learning model as a black box. (b) Fully-connected deep neural network.

https://doi.org/10.1371/journal.pone.0290762.g006

A central element of DL is the concept of an artificial neural network (NN) which describes the inner-workings of the DL black box. It consists of a layered structure of units conducting mathematical operations on their inputs and passing along the results (see Fig 6b). The usefulness of an NN lies in its ability to infer a function from observations. In this way, the DL model can approximate the predictor function that describes the observed data which is essential for both classification and regression tasks. The idea of an NN was first proposed by Hebb [48] as an abstraction of a real biological network of neurons in the mammalian brain.

A biological NN comprises numerous layers, each consisting of a set of neurons (see Fig 4 in the S1 Appendix). Each neuron consists of dendrites that receive input signal pulses, a cell body that performs a transformation of the incoming combination of pulses, and an axon. The axon carries the output pulse through its trunk (covered with isolating myelin sheaths) to its ramified terminal, which has a set of synapses that connect to the dendrites of the upstream neurons in the network. The functioning of biological NNs has been studied and formalized independently in [49, 50].

Each neuron in the network conducts a non-linear operation on the superposition of the signals received by its dendrites. These signals are weighted by the strength of the connection of its synapses of the upstream neurons. The process carried out by each neuron can therefore be modeled as a simple unit computing the weighted sum of its inputs x₁, …, x_M with weights w_j,1, …, w_j,M, adding a bias term b_j and applying a non-linear transformation a(⋅), called an activation function on the weighted sum (see Fig 4a in the S1 Appendix). This unit, first proposed in [51], is referred to as an artificial neuron and constitutes the basic building block of every NN.

Information travels through the NN from the input layer towards the output layer, passing through all neurons on its way. For binary classification tasks, the output layer consists of a single neuron that outputs the probability of the observation belonging to one of the classes. The layers in between are called hidden layers, as they are not directly visible from the outside of the black box. An NN is referred to as a deep NN if it contains two hidden layers or more. Note that NNs could equally well be adapted for regression and multi-class classification tasks. For the latter, the last layer would consist of the same number of neurons as there are classes, each having a softmax activation function and outputting the probability of belonging to that class.

It has been proven that sufficiently deep NNs are able to approximate any arbitrarily complex function f(⋅) [52]. Hence, deep NNs are particularly useful in applications where there are large amounts of data and significant computational resources available, and where manual analysis is tedious or the performance of traditional ML algorithms is unsatisfactory. In the context of text classification, NNs were first applied in [53, 54]. Only recently have they started to be used for political science research [55, 56]. There are various neural architectures available for this task. We review the most relevant ones below.

Data

For the analysis, we have collected a dataset [65] with tweets from eight Twitter accounts: @UNOCHA, @UNDP, @FAO, @WHO, @UNICEF, @UNDP, @UNDRR, and @Refugees. As elaborated in the introduction, these accounts correspond to international organizations in the UN system that deal with climate change in different policy areas [25, 26].

The tweets were downloaded and parsed via the Twitter Academic Research API. In total, the dataset contains 222,191 tweets posted by the eight UN organizations from their official accounts. This number represents the total number of tweets by these eight selected UN organizations between the beginning of their tweeting history and the end of 2019. The data comprises all tweets posted by the official accounts of the selected UN organizations, whether or not these tweets are concerned with climate change. Each tweet is considered as an observation in our dataset.

From the entire dataset, 5,750 tweets were randomly selected and labeled manually as either “climate change-related” or “not climate change-related”. Three research assistants, with training in climate governance, each labeled 2,000 tweets, working independently from each other. After this manual coding, we obtained a dataset containing 5,750 observations, see Table 1. We “lost” 250 observations due to overlaps between the three datasets produced by the research assistants, with 125 overlaps between each pair. This overlap between the datasets was used to track the agreement between the three research assistants’ coding.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Summary of the collected dataset.

https://doi.org/10.1371/journal.pone.0290762.t001

The inter-coder reliability was high: on average, the coders agreed on 96% of coding decisions, which corresponds to Cohen’s κ = 0.95 [66], viz., very high inter-coder agreement [67]. Two senior researchers trained the research assistants and supervised the coding. In this manual coding, tweets about climate change mitigation and adaptation were both classified as climate change-related, i.e., assigned label “1”.

Performance assessment

For model training and performance assessment, we split the dataset into a training set (85%) and a test set (15%). To avoid data leakage, we made sure that the test set is never used until the final step of the performance evaluation. The latter was carried out by comparing the labels outputted by a classifier to the ground truth (labels marked by our research assistants).

Additionally, a part of the training set (ca. 15%) was taken as a validation set for hyperparameter tuning.

There are many known optimization algorithms used for hyperparameter tuning, e.g., Bayesian optimization [68], random search [69] and metaheuristics, such as genetic algorithm [70], simulated annealing [71], differential evolution [72] and swarm intelligence [73]. For tuning the hyperparameters of all the methods considered herein, we have chosen to use Bayesian optimization, i.e., learning the probability distribution of the model loss conditioned on a given hyperparameter from the historical data gathered from its previous iterations. In particular, we used the tree-structured Parzen estimator [74] for our evaluations.

For the traditional ML methods, in addition to model-specific parameters, we chose the best feature vectorization approach (count vectorizer vs. TF-IDF). For DL models, the Keras tokenizer is used for the conversion of texts to sequences. We also treated the maximum number of considered features as a hyperparameter and tuned it for all models. All NNs have a common architecture. They contain the following layers: an embedding layer, a dropout layer, a layer with a certain type of units (FCNN, CNN or LSTM) and a ReLU activation function, a flattening layer (where applicable), and a fully connected layer. The actual weights and biases of the neural network are optimized with the help of the Adam optimizer [59]. The weights and biases of the NNs are initialized using the Xavier rule [75] to avoid problems with vanishing gradients.

During the training of NNs, we tracked the loss values on the training and validation sets (binary cross-entropy is used as a loss function). We picked the model that achieves the lowest validation loss. We have also used the technique of early stopping to catch the minimum of the validation loss before potential overfitting kicks in. Also, note that in all DL-based methods we utilize self-learned embeddings. An investigation of the improvements from pre-trained embeddings (see, e.g., [76, 77]) has been left for future research. Further details on the NN architectures and optimized hyperparameters can be found in our repository. Finally, it is worth noting that we run all our experiments in the Google Colab environment [78], using graphic processing units (GPUs) to accelerate the training of the DL classifiers.

Performance metrics

The choice of performance indicators is central to our analysis. For this purpose, it is necessary to consider that our dataset is imbalanced, as only ca. 9% of the tweets in the dataset are about climate change. Prediction accuracy, defined as the share of correct predictions out of all predictions made, is typically used as a performance metric for classification problems. However, for an imbalanced dataset, accuracy becomes a misleading metric. For instance, for our dataset, a simple guess that a tweet is not about climate change would yield 91% accuracy. However, this is not useful for the classification task at hand. Therefore, when choosing evaluation criteria, one has to take into account the dataset imbalance (see [79] for a discussion).

Thus, we relied on alternative performance indicators based on the so-called confusion matrix [79]. For each tweet in the test set, based on the observed features x, the classifier under consideration makes a categorization . It classifies the tweet as either “positive” (about climate change, ) or “negative” (not about climate change, ). When we compared the predicted label (or a classifier’s decision) with the corresponding manually-coded label y, each observation falls into one of four sets: true negatives (with N_TN entries), true positives (with N_TP entries), false positives (with N_FP entries), false negatives (with N_FN entries)—see Fig 7.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Confusion matrix containing counts of four sets of test observations.

The categorization of each observation is based on the comparison of the predicted label with the true one.

https://doi.org/10.1371/journal.pone.0290762.g007

Based on the confusion matrix, precision refers to the fraction of correctly classified examples among all the examples classified as being in a category. Meanwhile, recall refers to the fraction of correctly classified examples amongst all the examples that belong to a category in the dataset. These two metrics can have values between 0 and 1, and are computed as (7) The higher the value of each of these two metrics, the better the performance of the classifier. However, each of these metrics are in a trade-off against the other. Changing the classifier’s cutoff threshold (which determines which of the categories an observation falls into) can improve precision—at the expense of recall. And vice versa, improvements in recall can result in losses in precision.

Given this trade-off, the preferred metric in this article is the so-called F1 score as a compromise metric [79, 80] which is computed as the harmonic mean of the two aforementioned metrics, i.e., F₁ = 2PR/(P + R). Various alternative metrics have been used in the literature for assessing the performance of classifiers on imbalanced datasets, e.g., ROC AUC [81], PR AUC [82] and MCC [83]. However, all of these come with their respective advantages and disadvantages. For the purpose of imbalanced classification, there is no single performance metric that can capture all aspects and tell us exactly how well a particular classifier performs. It therefore might be more informative to separately report several metrics, including precision and recall [84]. Nonetheless, in this article, we focus on the F1 score as the primary performance metric as it has been widely used in the existing literature to assess the performance of classifiers applied to imbalanced datasets.

Performance results

Table 2 presents the performance of the ML algorithms under consideration in terms of all of the metrics discussed above given the original collected dataset. The results suggest that the best-performing approaches are the LR and RF classifiers. This finding is remarkable since these classifiers beat all the sophisticated neural classifiers at much lower computational complexity.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Classification performance of considered classifiers on the original collected dataset.

https://doi.org/10.1371/journal.pone.0290762.t002

At the same time, we note that the performance scores are generally quite low (the best method, LR, having F₁ ≈ 65% only). This is likely due to the characteristics of the dataset: imbalance and a rather small size. Also notable is the comparatively high performance of the lexicon-based classifier, which reaches F₁ ≈ 62%. Another interesting result is the poor performance of the KNN algorithm (with as little as F₁ ≈ 29%). Finally, it is noteworthy that the advanced DL methods (FCNN, CNN and LSTM) require significantly longer training (see Table 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Training and execution times (in seconds) of considered classifiers on the original collected dataset.

https://doi.org/10.1371/journal.pone.0290762.t003

The poor classification performance of all the classifiers discussed above can likely be attributed to class imbalance in the dataset. To investigate the influence of this imbalance, we prepare a balanced dataset. To do this, we supplement our original dataset with additional tweets taken from Kaggle’s “Twitter Climate Change Sentiment Dataset” [85]. This dataset contains 43, 943 tweets about climate change, manually labeled for sentiment. We are not interested in the sentiment labels, but instead take all of these tweets as ones about climate change (or labeled with “1”). We pick enough samples from the Kaggle dataset to balance out our own dataset and run the same algorithms on the new balanced dataset. We ensure that the same train/test split (i.e., 85%/15%) is maintained and the training and test sets in the original dataset remain parts of the new training and test sets, without mixing between the two. In this way, we make sure that the models see only new data during the performance evaluation stage.

Table 4 shows how the classification algorithms perform on the new, balanced dataset. It is notable how dramatically the overall performance scores have improved when the classification methods are applied to the balanced dataset. Almost all methods now exhibit very high F1 scores, larger than 96%. The only outliers are the KNN algorithm (F₁ ≈ 90%) and the lexicon-based classifier (F₁ ≈ 86% only). We note that the poor performance of the KNN classifier, compared to other ML algorithms, is in line with the conclusions of [86]. Meanwhile, the relative performance of the lexicon classifier was higher for the imbalanced dataset. This is likely because the proposed keywords better fit the collected dataset of UN organizations, rather than the Kaggle dataset containing all kinds of tweets.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Classification performance of considered classifiers on the artificially balanced dataset.

https://doi.org/10.1371/journal.pone.0290762.t004

In the new setting, the ranking of the best-performing methods has changed. The number one is now CNN with F₁ ≈ 97%. This indicates that CNN might be an appropriate choice for classification with balanced datasets when the best possible performance needs to be squeezed out at a cost of higher complexity. In Table 4, CNN is closely followed by the RF and LR classifiers. It is however unclear whether the differences in the performance of these classifiers is statistically significant or not. To investigate this, we conduct pair-wise McNemar’s tests [87] on every possible pair of classifiers under consideration. The corresponding methodology is described in the S1 Appendix, where Fig 8 in the S1 Appendix presents the results of the statistical tests. The results show that the performance of almost all the considered methods does not significantly differ from one another, with the exception of KNN and lexicon classifiers. That is, for scenarios where the dataset is balanced, it matters less which classification algorithm is chosen (with the exception of the KNN and lexicon classifiers).

We also sought to investigate the role of the imbalance further. To do this, we sweep the degree of balance in the dataset, adding new data from the Kaggle dataset to the training and test sets, keeping the same class proportions in both. We then compute the F1 scores for all the considered methods (see Fig 8). The dataset imbalance is captured by the balance ratio, which is defined as the ratio of positive to negative labels in the dataset. As mentioned before, for our original collected dataset, this ratio is only 0.09, hence it is heavily imbalanced. For the new completely balanced dataset, the balance ratio becomes 1.0. For each balance-ratio point in this experiment, we keep the balance between the classes the same for the training, validation and test sets.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. F1 score vs. balance ratio for the classifiers under consideration.

Solid lines show the performance of traditional ML methods, dashed lines show that of DL-based approaches, while the dash-dotted line illustrates the performance of the unsupervised lexicon-based classifier.

https://doi.org/10.1371/journal.pone.0290762.g008

Fig 8 shows that the performance of all classification algorithms improves with a higher balance ratio. The difference in performance diminishes as the balance ratio increases. This underpins the above observation that for nearly balanced datasets, the classification algorithms tend to perform similarly, with the exception of less capable KNN and lexicon classifiers. At the same time, the same algorithms—namely LR, RF and, to some extent, SVM and LSTM—perform best for almost all degrees of balance.

Taken together, we conclude that, for short text classification, it appears to make more sense to turn to traditional ML algorithms instead of computationally-demanding deep neural architectures. The benefits of the latter are marginal, while their training requires significantly more time. In contrast, DL methods might be more useful when datasets are balanced and of a large scale, and when extensive computational resources are available.