2.1 Notation

In this paper we use the following notation. By x we indicate a document drawn from a domain of documents, while by y we indicate a class drawn from a set of classes (also known as a codeframe) . Given and , a pair (x, y) denotes a document with its true class label. Symbol σ denotes a sample, i.e., a non-empty set of (labelled or unlabelled) documents drawn from . By p_σ(y) we indicate the true prevalence of class y in sample σ, by we indicate an estimate of this prevalence, and by we indicate the estimate of this prevalence obtained by means of quantification method M (consistently with most mathematical literature, we use the caret symbol (^) to indicate estimation). Since 0 ≤ p_σ(y) ≤ 1 and for all , and since , the p_σ(y)’s and the ’s form two probability distributions across the same codeframe.

By we denote an evaluation measure for quantification; these measures are typically divergences, i.e., functions that measure the amount of discrepancy between two probability distributions. By L we denote a set of labelled documents, that we typically use as a training set, while by U we denote a set of unlabelled documents, that we typically use for testing purposes. We take a hard classifier to be a function , and a soft classifier to be a function , where s(x) is a vector of posterior probabilities (each indicated as Pr(y|x)), such that ; Pr(y|x) indicates the probability of membership in y of item x as estimated by the soft classifier s. By δ_σ(y) we denote the set of documents in sample σ that have been assigned to class y by a hard classifier.

2.2 Why do we need quantification?

Quantification may be seen as the task of training, via supervised learning, a predictor that estimates an unknown true distribution p_σ, where p_σ is defined on a sample σ and across the classes in a codeframe , by means of a predicted distribution . In other words, in quantification one needs to generate estimates of the true (and unknown) class prevalence values , where . In this paper we consider a ternary sentiment quantification task (an example of single-label multiclass quantification) in which the codeframe is , and where these three class labels will be indicated, for brevity, by the symbols {⊕, ⊙, ⊖}, respectively. All the 11 datasets discussed in Section 3.5 use this codeframe.

The reason why true quantification methods (i.e., different from the trivial “classify and count” mentioned in Section 1) are needed is the fact that many applicative scenarios suffer from distribution shift, the phenomenon according to which the distribution p_L(y) in the training set L may substantially differ from the distribution p_σ(y) in the sample σ of unlabelled documents that one needs to label [23, 24]. The presence of distribution shift means that the well-known IID assumption, on which most learning algorithms for training classifiers are based, does not hold; in turn, this means that “classify and count” will perform less than optimally on samples of unlabelled items that exhibit distribution shift with respect to this training set, and that the higher the amount of shift, the worse we can expect “classify and count” to perform.

2.3 The APP and the NPP

There are two main experimental protocols that have been used in the literature for evaluating quantification; we will here call them the artificial-prevalence protocol (APP) and the natural-prevalence protocol (NPP).

The APP consists of taking a standard dataset (by which we here mean any dataset that has originally been assembled for testing classification systems; any such dataset can be used for testing quantification systems too), split into a training set L of labelled items and a set U of unlabelled items, and conducting repeated experiments in which either the training set prevalence values of the classes, or the test set prevalence values of the classes, are artificially varied by means of subsampling (i.e., by removing random elements of specific classes until the desired class prevalence values are obtained). In other words, subsampling is used either to generate s training samples , or to generate t test samples , or both, where the class prevalence values of the generated samples are predetermined and set in such a way as to generate a wide array of distribution drift values. This is meant to test the robustness of a quantifier (i.e., of an estimator of class prevalence values) in scenarios characterized by class prevalence values very different from the ones the quantifier has been trained on. For instance, in the binary quantification experiments carried out in [15], given codeframe , repeated experiments are conducted in which examples of either y₁ or y₂ are removed at random from the test set in order to generate predetermined prevalence values for y₁ and y₂ in the samples thus obtained. In this way, the different samples are characterised by a different prevalence of y₁ (e.g., ) and, as a result, by a different prevalence of y₂. This can be repeated, thus generating multiple random samples for each chosen pair of class prevalence values. Analogously, random removal of examples of either y₁ or y₂ can be performed on the training set, thus bringing about training samples with different values of and .

This protocol has been criticised (see [25]) because it may generate samples exhibiting class prevalence values very different from the ones of the set (L or U) from which the sample σ was extracted, i.e., class prevalence values that might be hardly plausible in practice. As a result, one may resort to the NPP, which consists instead of doing away with sample extraction and directly using, as the samples for conducting the experiments, the test set U (or portions of it obtained by subdividing it into bins) and the training set L that have been sampled IID from the data distribution. In other words, no perturbation of the original class prevalence values is performed for extracting samples. An example experimentation that uses the NPP is the one reported in [25], where the authors test binary quantifiers on 52 × 99 = 5,148 samples. This results from the fact that, in using the RCV1-v2 test collection, they consider the 99 RCV1-v2 classes and bin the 791,607 test documents in 52 bins (each corresponding to a week’s worth of data, since the RCV1-v2 data span one year) of 15,212 documents each on average, and use the resulting bins as the samples. However, it is not always easy to find test collections with such a large amount of classes and annotated data, and this limits the applicability of the NPP. (It should also be mentioned that, as Card and Smith [26] noted, the vast majority of the 5,148 RCV1-v2 test samples used in [25] exhibit very little distribution shift, which makes the testbed used in [25] unchallenging for quantification methods).

The experimentation conducted by [GS2016] on tweet sentiment quantification is also an example of the NPP, since it relies on 11 datasets of tweets annotated by sentiment from which no extraction of samples at prespecified values of class prevalence was performed. For each dataset, the authors use the training set L as the sample σ_L on which to train the quantifiers, and the test set U as the sample σ_U on which to test them. However, what the authors of [GS2016] overlooked is that, while in classification an experiment involving 11 different datasets probably counts as large and robust, this does not hold in quantification if only one test sample per dataset is used. The reason is that, since the objects of quantification are sets (i.e., samples) of documents in the same way that the objects of classification are individual documents, testing a tweet sentiment quantifier on just 11 samples should be considered, from an experimental point of view, a drastically insufficient experimentation, akin to testing a tweet sentiment classifier on 11 tweets only.

As a result, we should conclude that the authors of [GS2016] (unintentionally) carried out a weak evaluation, and that the results of that experimentation are thus unreliable. We thus re-evaluate the same quantification methods that [GS2016] tested (plus some other more recent ones) on the same datasets, this time following the robust and by now consolidated APP; in our case, this turns out to involve 5,775 as many experiments as run in the original study, even without considering the experiments on quantification methods that had not been considered in [GS2016]).

It might be argued that the APP is unrealistic because it generates samples whose class prevalence values are too far away from the values seen in the set from where they have been extracted, and that such scenarios are thus unlikely to occur in real applicative settings. However, in the absence of any prior knowledge about how the class prevalence values are allowed or expected to change in future data, the APP turns out to be not only the fairest protocol, since it relies on no assumptions that could penalize or benefit any particular method, but also the most interesting for quantification, since quantification is especially useful in cases of distribution shift.

Yet another way of saying this comes from the observation that, should we adopt the NPP instead of the APP, a method that trivially returns, as the class prevalence estimates for every test sample, the class prevalence values from the training set (this trivial method is commonly known in the quantification literature as the maximum likelihood prevalence estimator – MLPE), would probably perform well, and might even beat all genuinely engineered quantification methods. The reason why it would probably perform well is that the expectations of the class prevalence values of samples drawn IID from the test set coincide with the class prevalence values of the test set, and these, again by virtue of the IID assumption, are likely to be close to those of the training set. In other words, the reason why MLPE typically performs well when evaluated according to the NPP, does not lie in the (inexistent) qualities of MLPE as a quantification method, but in the fact that the NPP is a weak evaluation protocol.

3.1 Evaluation measures

As the measures of quantification error we use Absolute Error (AE) and Relative Absolute Error (RAE), defined as (1) (2) where p is the true distribution, is the estimated distribution, and is the set of classes of interest ( in our case). (The sample σ on which we quantify is left implicit in order not to overload the notation).

Note that RAE is undefined when at least one of the classes is such that its prevalence in the sample σ is 0. To solve this problem, in computing RAE we smooth all p(y)’s and ’s by means of additive smoothing, i.e., we compute (3) where denotes the smoothed version of p(y) and the denominator is just a normalising factor (same for the ’s); following [6], we use the quantity ϵ = 1/(2|σ|) as the smoothing factor. We then use the smoothed versions of p(y) and in place of their original non-smoothed versions in Eq 2; as a result, RAE is now always defined.

The reason why we use AE and RAE is that from a theoretical standpoint they are, as it has been recently argued [27], the most satisfactory evaluation measures for quantification. This means that we do not consider other measures used in [GS2016], such as KLD, NAE, NRAE, and NKLD, since [27] shows them to be inadequate for evaluating quantification.

3.2 Quantification methods used in [GS2016]

We now briefly describe the quantification methods used in [GS2016], that we also use in this paper.

The simplest quantification method (and the one that acts as a lower-bound baseline for all quantification methods) is the above-mentioned Classify and Count (CC), which, given a hard classifier h, consists of computing (4) where indicates the number of documents classified as y_i by h and whose true label is y_j. CC is an example of an aggregative quantification method, i.e., a method that requires the (hard or soft) classification of all the unlabelled items as an intermediate step. All the methods discussed in this section are aggregative.

The Adjusted Classify and Count (ACC) quantification method (see [6, 28]) derives from the observation that, by the law of total probability, it holds that (5) where δ(y_i) denotes (see Section 2.1) the set of documents that have been assigned to class y_i by the hard classifier h. Eq 5 can be more conveniently rewritten as (6) Note that the leftmost factor of Eq 6 is known (it is the fraction of documents that the classifier has assigned to class y_i, i.e., ), and that (which represents the disposition of the classifier to assign y_i when y_j is the true label), while unknown, can be estimated by k-fold cross-validation on L. Note also that p_σ(y_j) is unknown (it is the goal of quantification to estimate it), and that there are instances of Eq 5, one for each . We are then in the presence of a system of linear equations in unknowns (the p_σ(y_j)’s); ACC thus consists of estimating these latter (i.e., computing ) by solving, by means of the known techniques, this system of linear equations.

CC and ACC use the predictions generated by the hard classifier h, as evident by the fact that both Eqs 4 and 6 depend on . Since most classifiers can be configured to return “soft predictions” in the form of posterior probabilities Pr(y|x) (from which hard predictions are obtained by choosing the y for which Pr(y|x) is maximised), and since posterior probabilities contain richer information than hard predictions, it makes sense to try and generate probabilistic versions of the CC and ACC methods [29] by replacing “hard” counts with their expected values, i.e., with . If a classifier natively outputs classification scores that are not probabilities, the former can be converted into the latter by means of “probability calibration”; see e.g., [30].

One can thus define Probabilistic Classify and Count (PCC) as (7) and Probabilistic Adjusted Classify and Count (PACC), which consists of estimating p_σ(y_j) (i.e., computing ) by solving the system of linear equations in unknowns (8) The fact that PCC is a probabilistic version of CC is evident from the structural similarity between Eqs 4 and 7, which only differ for the fact that the hard classifier h of Eq 4 is replaced by a soft classifier s in Eq 7; the same goes for ACC and PACC, as evident from the structural similarity of Eqs 6 and 8.

A further method that [GS2016] uses is the one proposed in [31] (which we here call SLD, from the names of its proposers, and which was called EMQ in [GS2016]), which consists of training a probabilistic classifier and then using the EM algorithm (i) to update (in an iterative, mutually recursive way) the posterior probabilities that the classifier returns, and (ii) to re-estimate the class prevalence values of the test set, until mutual consistency, defined as the situation in which (9) is achieved for all .

Quantification methods SVM(KLD), SVM(NKLD), SVM(Q), belong instead to the “structured output learning” camp. Each of them is the result of instantiating the SVM_perf structured output learner [32] to optimise a different loss function. SVM(KLD) [25] minimises the Kullback-Leibler Divergence (KLD); SVM(NKLD) [33] minimises a version of KLD normalised by means of the logistic function; SVM(Q) [34] minimises Q, the harmonic mean of a classification-oriented loss (recall) and a quantification-oriented loss (RAE). Each of these learners generates a “quantification-oriented” classifier, and the quantification method consists of performing CC by using this classifier. These three learners inherently generate binary quantifiers (since SVM_perf is an algorithm for learning binary predictors only), but we adapt them to work on single-label multiclass quantification. This adaptation consists of training one binary quantifier for each class in by applying a one-vs-all strategy. Once applied to a sample, these three binary quantifiers produce a vector of three estimated prevalence values, one for each class in ; we then L1-normalize this vector so as to make the three class prevalence estimates sum up to one (this is also the strategy followed in [GS2016]).

3.3 Additional quantification methods

From the “structured output learning” camp we also consider SVM(AE) and SVM(RAE), i.e., variants of the above-mentioned methods that minimise (instead of KLD, NKLD, or Q) the AE and RAE measures, since these latter are, for reasons discussed in Section 3.1, the evaluation measures used in this paper for evaluating the quantification accuracy of our systems. We consider SVM(AE) only when using AE as the evaluation measure, and we consider SVM(RAE) only when using RAE as the evaluation measure; this obeys the principle that a sensible user, after deciding the evaluation measure to use for their experiments, would instantiate SVM_perf with that measure, and not with others. (Quantification is a task in which deciding the right evaluation measure to use for one’s application is of critical importance; in fact, [27] argues that some applications demand measures such as AE, while the requirements of other applications are best mirrored in measures such as RAE.) These methods have never been used before in the literature, but are obvious variants of the last three methods we have described.

We also include two methods based on the notion of quantification ensemble [18, 35]. Each such ensemble consists of n base quantifiers, trained from randomly drawn samples of q documents each, where these samples are characterised by different class prevalence values. At testing time, class prevalence values are estimated as the average of the estimates returned by the base members of the ensemble. We include two ensemble-based methods recently proposed by Pérez-Gállego et al. [35]; in both methods, a selection of members for inclusion in the final ensemble is performed before computing the final estimate. The first method we consider is E(PACC)_Ptr, a method based on an ensemble of PACC-based quantifiers to which a dynamic selection policy is applied. This policy consists of selecting the n/2 base quantifiers that have been trained on the n/2 samples characterised by the prevalence values most similar to the one being tested upon (where similarity was previously estimated using all members in the ensemble). We further consider E(PACC)_AE, a method which performs a static selection of the n/2 members that deliver the smallest absolute error on the training samples. In our experiments we use n=50 and q=1,000.

We also report results for HDy [36], a probabilistic binary quantification method that views quantification as the problem of minimising the divergence (measured in terms of the Hellinger Distance) between two cumulative distributions of posterior probabilities returned by the classifier, one coming from the unlabelled examples and the other coming from a validation set. HDy looks for the mixture parameter α that best fits the validation distribution (consisting of a mixture of a “positive” and a “negative” distribution) to the unlabelled distribution, and returns α as the estimated prevalence of the positive class. We adapt the model to the single-label multiclass scenario by using the one-vs-all strategy as described above for the methods based on SVM_perf.

ACC and PACC define two simple linear adjustments to be applied to the aggregated scores returned by general-purpose classifiers. We also use a more recently proposed adjustment method based on deep learning, called QuaNet [37]. QuaNet models a neural non-linear adjustment by taking as input (i) all the class prevalence values as estimated by CC, ACC, PCC, PACC, and SLD; (ii) the posterior probabilities Pr(y|x) for each document x and for each class , and (iii) embedded representations of the documents. As the method for generating the document embeddings we simply perform principal component analysis and retain the 100 most informative components. (Note that, since the datasets we use are available not in raw form but in vector form, we cannot resort to common methods for generating document embeddings, e.g., methods that use recurrent, convolutional, or transformer architectures that directly process the raw text.) QuaNet relies on a recurrent neural network (a bidirectional LSTM) to produce “sample embeddings” (i.e., dense, multi-dimensional representations of the test samples as observed from the input data), which are then concatenated with the class prevalence estimates obtained by CC, ACC, PCC, PACC, and SLD, and then used to generate the final prevalence estimates by transforming this vector through a set of feed-forward layers (of size 1,024 and 512), followed by ReLU activations and dropout (with drop probability set to 0.5).

3.4 Underlying classifiers

Consistently with [GS2016], as the classifier underlying CC, ACC, PCC, PACC, and SLD, we use one trained by means of L2-regularised logistic regression (LR); we also do the same for E(PACC)_Ptr, E(PACC)_AE, HDy, and QuaNet. The reasons of this choice are the same as described in [GS2016], i.e., the fact that logistic regression is known to deliver very good classification accuracy across a variety of application domains, and the fact that a classifier trained by means of LR returns posterior probabilities that tend to be fairly well-calibrated, a fact which is of fundamental importance for methods such as PCC, PACC, SLD, HDy, and QuaNet. By using the same learner used in [GS2016] we also allow a more direct comparison of results.

As specified above, the classifier underlying SVM(KLD), SVM(NKLD), SVM(Q), SVM(AE), SVM(RAE), is one trained by means of SVM_perf.

3.5 Datasets

The datasets on which we run our experiments are the same 11 datasets on which the experiments of [GS2016] were carried out, and whose characteristics are described succinctly in Table 1. As already noted at the end of Section 1, [GS2016] makes these datasets available already in vector form; we refer to [GS2016] for a fuller description of these datasets.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Datasets used in this work and their main characteristics.

Columns L_Tr, L_Va, U contain the numbers of tweets in the training set, held-out validation set, and test set, respectively. Column “Shift” contains the values of distribution shift between L ≡ L_Tr ⋃ L_Va and U, measured in terms of absolute error, columns p_L(⊕), p_L(⊙), and p_L(⊖) contain the class prevalence values of our three classes of interest in the training set L, while columns p_U(⊕), p_U(⊙), and p_U(⊖) contain the class prevalence values for the unlabelled set U.

https://doi.org/10.1371/journal.pone.0263449.t001

Note that [GS2016] had generated these vectors by using state-of-the-art, tweet-specific preprocessing, which included, e.g., URL normalisation, detection of exclamation and/or question marks, emoticon recognition, and computation of “the number of all-caps tokens, (…), the number of hashtags, the number of negated contexts, the number of sequences of exclamation and/or question marks, and the number of elongated words” [GS2016, §4.1]; in other words, every effort was made in [GS2016] to squeeze every little bit of information from these tweets, in a tweet-specific way, in order to enhance accuracy as much as possible.

In the experiments described in this paper we perform feature selection by discarding all features that occur in fewer than 5 training documents.

According to the principles of the APP, as described in Section 2.3, for each of the 11 datasets we here extract multiple samples from the test set, according to the following protocol. For each different triple (p(⊕), p(⊙), p(⊖)) of class prevalence values in the finite set P = {0.00, 0.05, …, 0.95, 1.00} and such that the three values sum up to 1, we extract m random samples of q documents each such that the extracted samples exhibit the class prevalence values described by the triple. In these experiments we use m = 25 and q = 100. For each label y ∈ {⊕, ⊙, ⊖} and for each sample, the extraction is carried out by means of sampling without replacement. (Here it is possible to always use sampling without replacement because each test set contains at least q = 100 documents for each label y ∈ {⊕, ⊙, ⊖}. If a certain test set contained fewer than q = 100 documents for some label y ∈ {⊕, ⊙, ⊖}, for that label and that test set it would be necessary to use sampling with replacement.)

It is easy to verify that there exist |P|(|P| + 1)/2 = 231 different triples with values in P. (This follows from the fact that, when p_σ(⊕) = 0.00, there exist 21 different pairs (p_σ(⊙), p_σ(⊖)) with values in P; when p_σ(⊕) = 0.05, there exist 20 different such pairs; …; and when p_σ(⊕) = 1.00, there exists just 1 such pair. The total number of combinations is thus .) Our experimentation of a given quantification method M on a given dataset thus consists of training M on the training tweets L_Tr, using the validation tweets L_Va for optimising the hyperparameters, retraining M on the entire labelled set L ≡ L_Tr ⋃ L_Va using the optimal hyperparameter values, and testing the trained system on each of the 25 × 231 = 5,775 samples extracted from the test set U. This is sharply different from [GS2016], where the experimentation of a quantification method M on a given dataset consists of testing the trained system on one sample only, i.e., on the entire set U.

3.6 Parameter optimisation

Parameter optimisation is an important factor, that could bias, if not carried out properly, a comparative experimentation of different quantification methods. As we have argued elsewhere [38], when the quantification method is of the aggregative type, for this experimentation to be unbiased, not only it is important to optimise the hyperparameters of the classifier that underlies the quantification method, but it is also important that this optimisation is carried out using a quantification-oriented loss, and not a classification-oriented loss.

In order to optimise a quantification-oriented loss it is necessary to test each hyperparameter setting on multiple samples extracted from the held-out validation set, in the style of the evaluation described in Section 3.5. In order to do this, for each combination of class prevalence values we extract, from the held-out validation set of each dataset, m samples of q documents each, again using class prevalence values in P = {0.00, 0.05, …, 0.95, 1.00}. Here we use m = 5 and q = 100; we use a value of m five times smaller than in the evaluation phase (see Section 3.5) in order to keep the computational cost of the parameter optimisation phase within acceptable bounds.

For each label y ∈ {⊕, ⊙, ⊖} and for each sample, the extraction is carried out by sampling without replacement if the test set contains at least p_y⋅q examples, and by sampling with replacement otherwise. (Unlike when extracting samples in the evaluation phase—see Section 3.5, it is here sometimes necessary to use sampling with replacement because, in some dataset, the validation set does not contain at least 100 documents per class).

In the experiments that we report in this paper, the hyperparameter that we optimise is the C hyperparameter (that determines the trade-off between the margin and the training error) of both LR and SVM_perf; for this we carry out a grid search in the range C ∈ {10ⁱ}, with i ∈ [−4, −3, …, + 4, + 5]. We optimise this parameter by using, as a loss function, either the AE measure (the corresponding results are reported in Table 2) or the RAE measure (Table 3). We evaluate the former batch of experiments only in terms of AE and the latter batch only in terms of RAE, following the principle that, once a user knew the measure to be used in the evaluation, they would carry out the parameter optimisation phase in terms of exactly that measure.

Hereafter, with the notation M^D we will indicate quantification method M with the parameters of the learner optimised using measure D.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Values of AE obtained in our experiments; each value is the average across 5,775 values, each obtained on a different sample.

https://doi.org/10.1371/journal.pone.0263449.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Values of RAE obtained in our experiments; each value is the average across 5,775 values, each obtained on a different sample.

https://doi.org/10.1371/journal.pone.0263449.t003

4.1 Difference between systems and their statistical significance

Concerning the differences between rank positions in the experimentations of this paper and of [GS2016] reported in Tables 4 and 5, we want to remark that they are just meant to provide an additional, quick reading of how differently the methods perform in the two experimentations, and should not be considered a substitute of the original numerical results from which they are obtained, as available from Tables 2 and 3.

While those differences are only qualitative in nature, we also want to investigate differences between systems from a quantitative way. We thus study, separately in our batch of experiments and in the experiments of [GS2016], the extent to which the differences in performance amongst methods (as quantified by differences in error scores, and not as differences of rankings) are indeed significant (in a statistical sense) depending on the evaluation protocol. The results of the pairwise comparisons (in terms of a two-sided Wilcoxon signed-rank test on related paired samples) are reported in Tables 6 and 7, for AE and RAE, respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Pairwise comparisons, according to the Wilcoxon test, for the experiments run in this work (left) and the experiments from [GS2016] (right) when adopting AE as the evaluation measure. The symbol ‘>’ (resp. ‘<’) indicates that the method in the row is better than (resp., is worse than) the method in the column, with a confidence level of 99%, while symbol ‘≈’ indicates instead that the difference between the two is not significant. Symbols ‘≫’ and ‘≪’ are used in place of ‘>’ and ‘<’ when the differences in performance are found to be significant at a higher confidence level of 99.9%.

https://doi.org/10.1371/journal.pone.0263449.t006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Pairwise comparisons, according to the Wilcoxon test, for the experiments run in this work (left) and the experiments from [GS2016] (right) when adopting RAE as the evaluation measure. The symbol ‘>’ (resp. ‘<’) indicates that the method in the row is better than (resp. is worse than) the method in the column, with a confidence level of 99%, while symbol ‘≈’ indicates instead that the difference between the two is not significant. Symbols ‘≫’ and ‘≪’ are used in place of ‘>’ and ‘<’ when the differences in performance are found to be significant at a higher confidence level of 99.9%.

https://doi.org/10.1371/journal.pone.0263449.t007

Something that jumps to the eye is that the results derived from our experimentation tend to be much more conclusive (in the sense of statistical significance) when it comes to judging the superiority of one method over another. Indeed, all differences resulting from our experiments, as reported in Table 6, turn out to be statistically significant at a very high level of confidence, while no fewer than 75% of the comparisons obtainable from the results in [GS2016] are inconclusive; in Table 7), instead, only 2 differences out of 56 turn out to be not significant in our experiments (namely, the comparisons between PACC and ACC), while this happens in 34 cases out of 56 for the experiments of [GS2016]. After all, it is not surprising that a test of statistical significance deems more significant the differences found for a set of experiments based on 63,525 samples than for a set of experiments based on 11 samples.