Multi-label Classification and classical approaches to handling MVs

In single-label classification problems, a set of class labels is predetermined, and each object must be associated with one and only one label [23]. Formally, let X denote the input/feature space, and y denote the class value, where y ∈ L, which is the output space (a set of disjoint class labels). In this case, each sample is strictly associated with a single class label [24, 25]

However, there are increasingly more contexts in which data may belong to more than one class label. This classification condition is referred to as Multi-Label classification. Initially, MLC primarily focused on tasks such as text categorization, protein function classification, music categorization, semantic scene classification, and medical diagnosis [23, 24, 26]. Recently, new applications have emerged in Computer Vision, Natural Language Processing, and Data Mining, including Video Annotation, Legal Text Mining, and User Profiling [27]. According to [25, 28], similar to SLC, MLC is represented by X and y, where each sample x ∈ X is assigned a subset of the output space (a set of non-disjoint class labels). Table 1 illustrates a toy example depicting the difference between SLC and MLC, adapted from [29]. Considering that the data in Table 1 comprises 5 instances (x₁, x₂, x₃, x₄, x₅) and 3 labels (y₁, y₂, y₃).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Comparison of SLC and MLC using a toy example with 5 instances and 3 labels.

https://doi.org/10.1371/journal.pone.0297147.t001

Table 1a illustrates the SLC scenario, where five data instances (x₁ to x₅) are each strictly associated with a single label (y₁ to y₃). For instance, x₁ is associated with y₁, x₂ is associated with y₂, and so on. On the other hand, MLC allows data instances to be associated with multiple labels simultaneously. Table 1b demonstrates the MLC scenario, where the same five data instances (x₁ to x₅) can have multiple labels assigned to them. For example, x₁ is associated with both y₁ and y₂, x₂ is associated with both y₂ and y₃, and so forth. This distinction highlights how SLC restricts each data instance to a single label, while MLC permits instances to belong to multiple labels simultaneously, making it more suitable for scenarios where objects or data points can be associated with different classes.

Although the difference is subtle in theory, MLC tends to be more challenging in practice. Gonçalves et al. [23] and Sá et al. [25] enumerated the following reasons for this:

• The possible classes of a given instance (output space) in MLC grow exponentially from the increasing number of labels. Therefore, when considering that a problem has L distinct labels, the size of the output space in MLC is 2^L (combination of labels) while it is only L in SLC;
• An MLC algorithm must consider whether there exists or not a correlation between labels. This kind of correlation is an essential step to ensure the effectiveness of several MLC processes [24, 30, 31];
• MLC systems performance evaluation uses different metrics than those traditionally used in SLC [32]. In SLC, the rating of a new instance can be either correct or wrong. On the other hand, in MLC, the result can be partially correct. It occurs when the classifier predicts some correct labels but includes some incorrect predictions or even omits a label that should be predicted. This problem requires cautious attention since some metrics follow contrasting aspects to define what is a good MLC prediction [25, 33];
• Unlike SLC problems, which traditionally involve the analysis of relational (structured) data, MLC applications typically address big data tasks, which involve semi-structured or unstructured data [24, 34].

All these challenges have amplified the complexity associated with handling MVs. Nevertheless, finding studies that relate MLC and MV is not straightforward, as demonstrated in [4, 8, 17].

In this context, we emphasize a limited number of studies that specifically address the issue of missing labels [35, 36], which means focusing on predicting an unknown label. Wang et al. [35] present a multi-label feature selection that considers feature interaction. For that, the authors use the definitions of multi-label neighborhood information entropy and multi-label neighborhood mutual information to mitigate the negative impact of missing labels. Cheng, Song & Qian [36] focus on addressing missing labels by leveraging label correlations and implementing a two-level kernel extreme learning machine autoencoder. The authors verified the proposed method on both missing and complete label datasets. Since these studies primarily focus on missing labels rather than missing values (predictive features), to the best of our knowledge, there is no work addressing missing values in the predictive feature space in an ML scenario. Thus, this constitutes one of the contributions of the present study.

Bio-inspired computation for the handling of MVs

Tran, Zang, and Andreae [37] proposed a data imputation method by adopting an approach based on genetic programming called GPMI. An MI strategy was applied in this method, and an estimation of missing values was performed using regression techniques. The GPMI was compared with seven imputation methods through an experiment carried out in eight datasets and applying seven different missing values ratios (5, 10, 20, 30, 40, and 50) with the aid of MCAR as a missing data mechanism. The classifier’s accuracy was the performance measure adopted. The results suggest that the planned method performed better than all methods. According to the authors, genetic programming was primarily responsible for these results because the algorithm initially used random samples to fill the gaps before being submitted to genetic processes. The results confirmed that strategies based on evolutionary algorithms are feasible alternatives for missing values treatment.

Shahzad, Rehman, and Ahmed, in their study, “Missing Data Imputation using Genetic Algorithm for Supervised Learning” [38], employed GA to search for plausible values for missing data imputation. An exciting strategy adopted in this study is using information gain to observe how solutions are found as the process grows. In an experiment with five datasets that originally contained missing values, the proposed method was compared with other imputation approaches: the average, lowest value, highest value, zero, and MI. They used the following performance measures: predictive accuracy, precision, recall, F-measure, and the area under the Receiver Operating Characteristic (ROC) curve, with the following classifiers: NB-tree, PART, JRIP, Naive Bayes, KNN, and J48. The authors noted that the GA-based method showed promising results and worked well in datasets with a high percentage of missing values.

In [39], an algorithm called MOGAImp was proposed for multiple imputation datasets based on genetic algorithms. One of the exciting strategies of this work is to apply a multi-objective approach, which until then had not been adopted in the literature for the performance analysis of imputation techniques. This approach involves simultaneously employing two or more evaluation measures. It can be explained by the fact that there are distinctions between various performance measures because, while one increases, the other declines. In the case of MOGAImp, two conflicting measures were used: the classifier accuracy and the predictive accuracy of the imputation method, calculated using Normalized Root-Mean-Square error (NRMSE) and the Pareto front.

Another critical factor in the study conducted by [39] concerns population initialization, which employs a pool of candidate solutions based on each attribute. The solution pool involves grouping all possible dataset values for the attribute that has a missing value (by lexicographically comparing two strings in cases of categorical variables). The method was experimentally compared with other well-known techniques in the literature, employing benchmarking through several databases with missing values. The results demonstrated that the method achieved competitive performance and, according to the authors, demonstrated potential for real-world applications. However, high computational power is required for handling the MVs individually with MOGAImp and through the solution pool. Additionally, this strategy is an excellent alternative to a mixture of genetic materials. Therefore, it has been adopted in EvoImp as a baseline for mutation operations.

In [15], the authors created a scheme based on genetic algorithms, which served as a baseline for developing and analyzing the method employed in this study. The strategy, nominated as MultImp, predicts multiple imputations of datasets in a multi-label classification model. In this study, the authors conducted experiments using four databases that were initially completed. Subsequently, 5% of the missing values were added through the MCAR mechanism. Binary relevance (BR) was employed as the multi-label classifier, with C4.5 as a parameter. In the test scenario, MultImp was compared with two other imputation methods (K-Nearest Neighbors Imputation—KNNI and Most Common—MC) and evaluated lexicographically using the following measures: Exact Match (EM), Accuracy, and Hamming Loss (HL). The preliminary results of this study proved to be promising, particularly in the case of EM, where the performance achieved by the method was better in all the datasets used and justified adopting the lexicographical approach.

For a comprehensive summary of the works discussed in this section, we have provided a detailed table in our supplementary material, available on the project’s GitHub repository (https://github.com/jacobjr/EvoImp).

Individual encoding and population initialization

The individual encoding of EvoImp took place in the following form: the variables in the datasets represent individual genes. Genes initially marked with “?” represent the missing values (Fig 1(a)). Each individual is represented by a completed (“accomplished”) instance of the databases (Fig 1(b)). The phenotype consists of imputed values, while the genotype represents these values in binary form, as illustrated in Fig 1(c).

The initial population comprised five simple imputation methods for the generation of each individual (Fig 1(d)). All imputation methods are well-known and established in the literature [7]: K-means Clustering Imputation, KNNI, WKNNI, Concept Most Common (CMC), and MC. The parameters for the KNNI, WKNNI, and KMI methods followed the guidelines set by the authors. This kind of population initialization was adopted in EvoImp to reduce the search space and, hence, the computational costs.

The methods employed are as follows [7]:

• KNNI: Whenever there is a missing value, the K-nearest neighbors closest to the instance containing the MV are determined. The most common value among the K-nearest neighbors was used to impute nominal attributes. For numerical attributes, imputation is performed by calculating the average of the neighboring values;
• WKNNI: This technique involves determining the distances between K-nearest neighbors and a weighting distribution regarding the distances between each neighbor. After this, the KNNI process was repeated;
• KMI: This technique divides a database into clusters based on their features. Once this has been done, the K-nearest neighbors technique is applied when deciding which value should be imputed;
• MC: In this method, the most common value is adopted for imputation in nominal attributes and the average of all corresponding attributes in the case of numerical attributes;
• CMC: This method does the same thing as MC but only employs the referenced attribute class with MV.

In contrast to MOGAImp [39], which employs random initialization of the initial population, the proposed method optimizes simple imputations through evolutionary processes to perform multiple imputations. This approach reduces the search space and introduces a novel method. This reduction in search space is particularly beneficial in scenarios where computational cost is critical in objective function calculations, such as multi-label classification.

It is also noteworthy that the presented work has two innovative contents: 1) using simple imputation methods as a priori solution, reducing the search space; 2) treating missing values in the multi-label scenario. To our best knowledge, there is no similar study in the literature.

Genetic operators

The individual selection involves a tournament in which two (or more) members of the previous population are selected, and the better one is chosen based on fitness value, as illustrated in Fig 1(e). This procedure was followed until a limited number of individuals from the current generation were obtained. The best individual is always selected through elitism [40].

In the literature, numerous methods for parameter tuning and control have been proposed and analyzed. [41] describes some of these methods and discusses various trends and challenges in the field. Specifically, [42] conducted experiments to find appropriate settings for these parameters when applying evolutionary algorithms to a multi-objective problem class. They concluded that determining the value of the scaling factor can be difficult and is highly dependent on the specific problem. Considering these findings, initial tests were conducted to define the parameters used in our study. In line with the work of [42], the initial percentage of Crossover was delimited to [0.8, 1.0], following the standard proposal for non-separable problems like the one tackled in our research. EvoImp employs a crossover for 80% of the individuals using an n-point crossover operator [43], as shown in Fig 1(f). It is also consonant with the work of [44].

The mutation process is performed on 20% of the individuals chosen randomly, except for the best one. For each individual to be mutated, the imputed value is exchanged for a candidate value. The mutation is applied only to genes that contain missing values. To accomplish this, each attribute in the dataset has a set of solutions, as shown in Table 2. This set is formed by considering all possible response options for that attribute in the evaluated dataset.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Candidate solutions for each attribute used for the mutation process.

https://doi.org/10.1371/journal.pone.0297147.t002

Table 2a displays a toy dataset containing five records and four attributes: “Year”, “Gender”, “Age”, and “Have Credit”. Some values in the dataset are missing and are represented by “?”. Table 2b lists the possible values for each attribute. For example, the “Year” attribute can have values 1998, 2005, or 2010; and the “Gender” attribute can have values M or F. The same reasoning is applied to the other attributes.

Lobato et al. [39] adopted this technique to initiate the first MOGAImp population. The mutation operator was not implemented in MultImp. The lack of it caused a premature convergence, limiting the method’s robustness. That operator is one of the main differences between MultImp and EvoImp. In other words, the proposed method implements a strategy to avoid local minimum.

The algorithm’s search and optimization process occurs over predetermined generations. The population goes into a growth phase, starting with the number of MI methods adopted in the population initialization and increasing by its cross-over. This strategy aims to provide population diversity. In the second phase, the population is gradually reduced, achieving the same initial population size, allowing the analysis to choose the best solution qualitatively.

Fitness function

As mentioned earlier, the method was evaluated on an MLC scenario. For this, EvoImp performs a classification process on each individual. The goal is to analyze the performance of the classifier and, consequently, the data imputation efficiency.

Three performance measures were adopted to evaluate the classifier, as with MultImp: Exact Match, Accuracy, and Hamming Loss. The notation used by [15, 45] were adopted to describe these measures: (i) n: number of instances in the test set; (ii) q: number of labels; (iii) Y_i: set of original labels, for instance, i; and (iv) Z_i: set of predictive labels, for instance, i.

• Exact Match calculates, using a binary system, whether all the instance labels are predicted correctly. This measure, as expressed in Eq 1, is assumed to be trivial because it ignores partial predictions: (1)
• Accuracy is also a measure that counts the correctly predicted labels of an instance. In this case, partial predictions are taken into account. Eq 2 expresses the mathematical model of this measure: (2)
• Hamming loss is a measure that, in contrast to accuracy, evaluates the classifier’s performance by finding the average of incorrect predictions. Eq 3 describes this measure: (3)

These measures were used in lexicographical order; in other words, this approach prioritizes all the problem’s objectives and then tries to satisfy them, keeping a list of priorities [46]. Thus, the fitness (f) for the problem solution can be expressed as Eq 4: (4) where n is the number of objectives defined; f_n is an optimization goal. Given two fitness evaluations f₁ and f₂ and a precision threshold t, the lexicographic relation between them (noted as ≺_l and ⪯_l) can be defined [47]: (5) (6) (7)

As can be observed, the Eq 5 shows f₁ ≺_l f₂, which means that f₁ is lexicographically less than f₂. This relationship is established when there exists an index k in in the range [0, n₀) ∩ , such that , indicating that the k-th component of f₁ is less than the k-th component of f₂. Additionally, the difference between and is greater than or equal to t. This ensures that the k-th components differ significantly by at least t. Finally, the absolute differences between corresponding components and should be less than t for all i less then k. In essence, this relation means that f₁ is superior to f₂ in terms of some objectives. The Eq 6 determines equality in lexicographical order (f₁ = _lf₂). This occurs when the absolute differences between corresponding components and are all less than t for all i in the range . In other words, f₁ and f₂ are considered equal regarding their performance across objectives. Finally, the Eq 7 presents f₁ ⪯_l f₂, which means that f₁ is either less than or equal to f₂ in lexicographical order. It combines the ≺_l and ⪯_l relations, indicating that f₁ is either better than or equal to f₂ in terms of the defined objectives.

These equations are used to rank and compare solutions or fitness evaluations in optimization problems, considering the objectives, prioritization, and performance. The lexicographical order approach allows for precise, multi-objective optimization when there are multiple criteria or objectives to be considered. Once the threshold t has been introduced, this formulation differs from the pure mathematical lexicographic relation. It permits the decision maker to choose the precision to compare two fitness functions. This relation allows the ranking of solutions of EvoImp as follows:

1. The EM behavior is evaluated;
2. If two or more individuals match their respective scores, the ACC evaluation is checked;
3. If the tie remains, the HL evaluation is used.

This approach allows different performance measures to be added to a single evaluation [45]. It is similar to the classical lexicographical approach, but once evolutionary algorithms are adopted, local optima can be avoided [47].

The EvoImp algorithm

As shown in Algorithm 1, EvoImp begins the execution by creating and evaluating individuals for the initial population. The datasets are initially imputed using simple imputation methods: KNNI, CMC, MC, KMI, and WKNNI (lines 1–5). Afterward, the population is evaluated and ranked based on each individual’s performance (line 6). The algorithm applies the genetic operators if the stopping criterion is not attained (e.g., the number of generations).

Algorithm 1: EvoImp

Input: datasets with MV and parameters (see Table 4)

Output: complete datasets

1 foreach Simple Imputation Method do

2  Generate a new Individual: individual;

3  Evaluate the individual;

4  Add individual to the Current Population: currentPop ← individual;

5 end

6 Order currentPop using Lexicographical order;

7 while Stop criterion not reached do

8  Add to the Current population the Best Individual: currentPop ← bestIndividual;

9  while currentPop < Number individuals of the new generation do

10   Select Parents;

11   Appy Crossover;

12   Evaluate the new Individual;

13   Add the Individual to Current population: currentPop ← individual;

14  end

15  while Number of mutated individuals < 20% of Individuals of the new generation do

16   Randomly choose an individual from the Current population;

17   Apply Mutation;

18   Evaluate the Individual;

19   Add the Individual to Current population: currentPop ← individual;

20  end

21  Order currentPop using Lexicographical order;

22 end

23 return bestIndividual;

The elitist individual is always passed on to the next generation (line 8). The selection is performed using the tournament selection operator (line 10). Two individuals are randomly drawn in this process. These two parents exchange genetic material using a crossover operator. These steps are repeated until the population is complete. Afterward, the mutation follows the established rate (lines 15–20). The new population is arranged, and the iterative process continues until the stopping criterion is reached. The return of the algorithm is the individual that achieves the best performance (line 23).

In summary, EvoImp adopts the configuration for the parameterization of MultImp [15], except for the mutation operator, as pointed out earlier. Besides, we also corrected bugs and optimized the code, bearing in mind maintainability and reuse. Moreover, we implemented the lexicographic strategy and expanded the computational tests, expanding the technical-scientific contribution of the present work.

Datasets

The experiments were designed using six multi-label datasets from the UCI Machine Learning repository (https://archive.ics.uci.edu/). The quantity datasets agree with the literature review conducted by [17], which mapped 48 papers related to experiments in the context of data imputation. Chiu’s work [17] shows that most papers (77%) use up to six datasets in experiments. Another interesting finding of Chiu et al. [17] is that the UCI Machine Learning Repository is the most used. Regarding the characteristics of the datasets, most use small-scale datasets, which contain fewer than 15 attributes and 800 instances. Table 3 shows the datasets used and their features.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Datasets used in experiments.

https://doi.org/10.1371/journal.pone.0297147.t003

Regarding multi-label datasets, the works of [35, 48] must be mentioned. These studies, as well as EvoImp, used datasets obtained at the UCI repository and formatted using the Mulan library (http://mulan.sourceforge.net/). The datasets used in these papers have similar characteristics (cardinality, density, and the number of instances) to those chosen in this paper. This observation highlights the experimental setup consonance with the state of the art and the EvoImp potential applicability in real-world problems.

Experimental setup

In the experiments, the missing values were artificially added to each dataset with the following rates: 5%, 10%, 15%, 20%, 25%, and 30%. This “amputation” process was carried out using the MCAR mechanism, as described in Santos (2019) [49]. The complete experimental configuration consisted of 36 datasetss with missing data, and these datasetss underwent a comparative evaluation. This evaluation involved five simple imputation methods: KNNI, CMC, MC, KMI, and WKNNI.

The following classification methods were used for the multi-label learning tasks: Binary Relevance (BR), Hierarchy of Multi-label classifiER (HOMER), Multi-Label K-Nearest Neighbors (ML-KNN), Classifier Chains (CC), and Ensembles of Classifier Chains (ECC) [21, 50]. K-fold cross-validation was used for the classification model’s evaluation (learning and testing). Table 4 summarizes the overall parameters which were used in the experiments.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Parameter settings used in the experiments.

https://doi.org/10.1371/journal.pone.0297147.t004

Regarding the simple imputation methods, the parameters recommended by [7] were used. The mutation rate (MR) chosen is higher than the typical usage rates because the starting point is not random. Therefore, considering that the initial population is obtained by other methods, parameterization experiments demonstrated that a higher MR yields better results, providing fast convergence. The entire experimental setup and the obtained results are available as supplementary material on the project’s GitHub (https://github.com/jacobjr/EvoImp).

Implementation

The GA was programmed in the Java language, version 8.1, based on the works of [15, 44]. Other components used third-party implementations as follows:

• For the multi-label classifiers, Mulan’s library (https://mulan.sourceforge.net/) was used [51]. This library also contains some classifiers implemented in Weka (https://www.cs.waikato.ac.nz/ml/weka/index.html) [52].
• The simple imputation methods used for forming the first population of EvoImp and in the comparative analyses are implemented in KEEL-software (http://www.keel.es/) [53].

It is noteworthy that GA used in the EvoImp was fully implemented by the authors despite KEEL providing a framework for evolutionary computation. This design decision aimed to give us more control over the experiments. The computational complexity is another crucial aspect to consider in implementing this proposed method. It plays a vital role in determining the feasibility and efficiency of applying bio-inspired techniques to solve optimization problems. Addressing this concern and reducing computational complexity enhances the algorithm’s applicability and scalability. As a result, it makes it more suitable for handling larger datasets and complex optimization landscapes, particularly in multi-label classification tasks. More detailed information about EvoImp’s computational complexity can be found in the supplementary materials on the project’s GitHub repository.

Binary relevance

In the learning performed with the BR classifier, the results showed that the EvoImp was numerically superior (Table 5). In the EM evaluation, EvoImp outperformed its competitors in 35 of the 36 datasets evaluated (97.22%). The proposed method demonstrated superior performance compared to others in 18 scenario datasetss (50%) regarding the Accuracy evaluation measure. Finally, considering the HL, EvoImp outperformed the baseline methods in 16 datasets (44.44%).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Experimental results for the binary relevance classifier.

https://doi.org/10.1371/journal.pone.0297147.t005

It is essential to highlight the priorities adopted in the EvoImp lexicographic order, prioritizing the evaluation with EM, as mentioned in the Subsection “Fitness Function”, which explains the performance decrease for the ACC and HL metrics considering the binary relevance classifier.

Hierarchy Of Multi-label Classifier (HOMER)

The results for the HOMER classifier are given in presented in Table 6. Analyzing the results, it is possible to observe that EvoImp is also superior to the others in 35 of the 36 datasets used in the experiments (97.22%) regarding the EM metric. These results corroborate the ones obtained from the Binary Relevance classifier.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Experimental results for HOMER classifier.

https://doi.org/10.1371/journal.pone.0297147.t006

Continuing analyzing Table 6 results, regarding the ACC evaluation measure, EvoImp outperformed the baseline methods in 23 datasets (63.88%). The HL results show that EvoImp had the slightest error in classification in 19 out of 36 datasets (52.78%). In summary, EvoImp outperformed the methods for all performance measures for HOMER classifier, in consonance with the results for BR classifier as well.

Multi-Label k-Nearest Neighbors

The results obtained with the ML-KNN classifier is shown in Table 7. As can be seen, EvoImp showed similar performance to the previous scenarios considering the BR and the HOMER classifiers. For instance, considering the primary analyzed metric (EM), EvoImp outperformed the baseline methods at 97.22%. Considering the ACC and HL, the EvoImp presented superior performance for 20 (55.55%) and 22 (61.11%) datasets, respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Experimental results for the ML-KNN Classifier.

https://doi.org/10.1371/journal.pone.0297147.t007

Classifier Chains

The results for the Classifier Chains are presented in Table 8. Again, EvoImp outperformed the baseline methods for all evaluation measures considered: EM with superiority in 32 out of 36 datasets (88.88%), ACC with 30 (83.33%), and HL with 22 (61.11%).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. Experimental results for the Classifier Chains.

https://doi.org/10.1371/journal.pone.0297147.t008

Ensembles of Classifier Chains

The last scenario analyzed was considering the Ensemble Classifier Chains method. The results are shown in Table 9. The results obtained with the ECC (Table 9) also show a significant advantage of EvoImp over competitors in the analyses performed. However, EvoImp had the lowest performance, with numerical superiority in 29 (80.55%) datasets in the evaluation with EM, 16 (44.44%) for ACC, and 17 (47.22%) for HL.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 9. Experimental Results for the Ensemble of Classifier Chains.

https://doi.org/10.1371/journal.pone.0297147.t009

In summary, the EvoImp performance for the ECC presents the same pattern described in the previous scenarios, demonstrating the EvoImp robustness.