Introduction

Researchers at Cornell University in the United States were the first to propose kinship verification via facial images [1], which aims to determine whether a given pair of facial images are kin-related. Here, kin-related refers to the blood relationship between parents and children. Kinship verification can be applied in social media analysis, social relationship classification, enhancing the performance of face recognition, and finding missing children [2].

Since facial image-based kinship verification shares a closely related objective with traditional face verification, researchers, in early research efforts, frequently adapted methodological frameworks from face verification to address kinship verification challenges, primarily due to the lack of specialized algorithms for familial facial analysis. This cross-disciplinary adaptation gave rise to two dominant paradigms: 1) feature extraction-based approaches [1,3], which aim to characterize kin-related facial images through handcrafted descriptors, and 2) similarity learning methods [4,5], which seek to establish a latent transformation space to capture pairwise relationships between relatives. While these pioneering methods have achieved notable success on benchmark datasets such as KinFaceW [4], their performance remains constrained by the intrinsic nonlinear manifold structure of facial data.

Fortunately, deep learning can effectively extract discriminative information relevant to labeled data from complex nonlinear datasets, as evidenced in diverse fields including face recognition [6], human-object interaction detection [7], and biomedical analysis [8–10]. Researchers have also introduced deep learning into the field of kinship verification. For instance, [11] fused different deep features. [12] proposed one deep scattering wavelet convolutional neural network. However, due to privacy concerns, the issue of small sample size in kinship verification is particularly severe. And relying solely on limited kin data of specific relationship types (e.g., father-son, father-daughter, mother-son, or mother-daughter in the KinFaceW dataset) for learning limits deep learning-based methods in two ways: 1) insufficient data volume for specified relationships causes inadequate gradient information, hindering efficient parameter updates; 2) inability to use extra discriminative information beyond the specified kin relationships, resulting in information isolation.

Recently, several studies [13–15] have adopted a generative approach to produce kin data of specific types, aiming to augment the sources of discriminative information. While these methods offer potential benefits, they incur computational overhead and pose a risk of concept drift to the algorithm, as they lack a thorough discussion on the relationship between the generated kin data and the original data. Moreover, these methods overlook information beyond specified kin data, failing to break information isolation. Therefore, developing a computationally efficient algorithm that simultaneously addresses data scarcity and information isolation constitutes a pivotal research direction in this field.

To address this, we propose a novel correlation calculation-based multi-task learning (CCMTL) method for kinship verification. The core concept of our CCMTL method lies in effectively leveraging both the given kin pairs of specific type and the correlations among various kinship types to resolve the information isolation issue in kinship verification, as depicted in Fig 1. Our approach is motivated by prior psychological research [16], which has revealed that facial feature sets derived from all kin relationships are usually correlated, attributed to the high heritability of several key facial traits. Complementary empirical evidence in [17] further confirms the existence of such correlations among different kinship types. By emphasizing these correlations during the learning process and transforming them into auxiliary discriminative information, our method effectively resolves information isolation in kinship verification.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Discriminative information sources (indicated by blue and red rounded rectangles) of the proposed method.

https://doi.org/10.1371/journal.pone.0329574.g001

To this end, our contributions to kinship verification are fourfold.

First, we utilize the correlations among various kinship types as auxiliary discriminative information to boost kinship verification performance. Contrary to approaches that rely solely on kin data, incorporating such inter-kinship-type correlations in kinship verification can fundamentally address the information isolation issue. To the best of our knowledge, this is the first work that integrates the inter-kinship-type correlations within the kinship verification pipeline.

Second, instead of learning distance metrics from given and generated kin data of a specified type, we propose to learn a set of metrics by leveraging not only the specified kin data but also the inherent correlations among various kinship types. In contrast, our approach eliminates the necessity for extensive parameter tuning within deep networks, thus resulting in significantly lower computational costs.

Third, we implemented two baseline approaches to evaluate the effectiveness of our proposed correlation exploitation strategy for kinship verification. The results demonstrate that models trained on data enriched with both refined kin data and the inter-kinship-type correlations outperform those trained solely on raw kin data.

Finally, to effectively integrate correlation exploitation and metric learning, we develop an algorithm within the framework of multi-task learning. Experimental results show that our method performs favorably in kinship verification.

The remainder of this paper is organized as follows. We briefly review some of the related work in the Related work section and detail our proposed method in the Proposed methods section. Experimental results are given in the Experimental results and analysis section. We conclude this paper in the Conclusions section.

Related work

This section briefly reviews the work related to facial image-based kinship verification, which was first proposed by Fang et al. [1]. Since then, a series of effective methods have been applied to solve this task, which can be roughly classified into feature representation-based and model-based learning methods.

Feature representation-based methods attempt to extract handcrafted or learning-based features that preserve kin-related properties from facial images. Fang et al. [1] firstly proposed utilizing local/holistic texture features to characterize facial images for facial image-based kinship verification task. Later, various features were developed to describe kin-related characteristics. [3] proposed to tackle kinship verification based on mid-level features, i.e., binary and relative attributes. [18] and [19] utilized texture color features from different color spaces and static wavelet transform features, respectively, to characterize facial images of kin relationships. Although these features have shown strong capability in capturing variations caused by uncontrollable factors such as illumination and rotation, they are still limited by their inherent nonlinear expressive power, which results in their inability to effectively characterize the similarity between parents and children. Thanks to the nonlinear representation capability of deep learning, researchers have leveraged deep neural networks( DNNs) to seek deep discriminative features. For instance, [11] fused different deep features. [12] proposed one deep scattering wavelet convolutional neural network. [20] and [21] combined shallow and handcrafted features with deep features to exploit their complementary strengths in characterizing facial images. [22] and [23] introduced the attention mechanism into deep networks. [24] utilized specific attention mechanisms to refine high-frequency and low-frequency features for more accurate local details and overall structure. In contrast, [25] utilized deep relational networks to mine multi-scale discriminative information from local regions. Furthermore, [26] proposed fusing deep face and general features through their tensor design. [27] compared the results of using different shallow features and deep features. While deep features demonstrate enhanced representational power compared to shallow counterparts, their inability to model age-progression effects in parent-child facial similarity significantly limits their capacity to capture such fine-grained similarity patterns. To tackle large divergence between the images of parent and child caused by significant age gaps, [28] presented a feature learning method, which consists of a two-stream generation-specific feature learning module and a generation-shared discrepancy reducing module. [29] and [30] respectively proposed an age-invariant adversarial feature learning module and Curvelet Transform features to address the large age gap between kin’s faces. Recently, [31] utilized deep networks to generate new facial images with different ages, ethnicities, and imaging environments to address the large divergence between the images of parent and child.

Model-based learning methods establish verification strategies to seek a feature transformation space through transfer learning [32–35] or metric learning [4]. And metric learning-based ones are frequently used in the facial image-based kinship verification task. Its primary objective is to learn a similarity metric such that the similarity between samples with kin relationship is greater than that between samples without kin relationship. Existing metric learning-based kinship verification methods can be further divided into single metric learning and multi-metric learning. NRML [4] encompasses the most representative single metric learning method and seeks only one transformation matrix, which pulls kin samples to be as close as possible and pushes non-kin samples in a neighborhood to be as far as possible. [5,36–40] learn multiple transformation matrices. Unlike the above multi-metric learning methods, [2] proposed to learn multiple local distance metrics, and one distance metric for one facial feature point. To improve nonlinear distance modeling in metric learning, researchers have integrated DNNs into metric learning frameworks. For instance, [41] and [42] proposed two deep metric learning methods that leverage hierarchical feature transformations for improved similarity measurement. And [43] developed a model called the adaptively weighted k-tuple metric network (AWk-TMN) for kinship verification, where a cross-pair metric learning loss based on k-tuple loss was introduced to capture features from multiple negative pairs. Beyond DNNs, kernel methods provide a robust alternative for nonlinear feature modeling, motivating researchers to evaluate their efficacy in kinship verification. Notably, [44] and [45] have validated the effectiveness of kernel-based metric learning methods in achieving nonlinear feature transformations that enhance discriminative power for kinship verification.

Unlike the above methods that focus solely on feature learning or metric learning, an end-to-end deep learning framework has been introduced in [46], which has jointly optimized feature learning, metric learning and meta-learning. Relatively speaking, this method, which draws inspiration from the ensemble learning concept, is more prone to achieving improved validation performance by integrating diverse models. In summary, numerous studies have been proposed for kinship verification, and a comprehensive summary of these studies can be found in a recent review article( e.g., [47]). Upon re-examining the aforementioned existing methods, we broadly divided them into two categories: those that use existing data and those that leverage additional data.

For the methods that use existing data, they follow the processing steps of face verification and seek discriminative features or similarity metrics from given specific type of kin data, inevitably falling into the information isolation issue. In general, these methods can make decisions, but the amount of discriminative information they can leverage is quite limited due to the scarcity of given specific type of kin data.

Relatively speaking, the methods that leverage additional data are more likely to utilize more discriminative information because they employed additional data. An earlier attempt was proposed in [13], wherein a generative adversarial network was utilized to generate adversarial kinship face pairs. The projection space was learned using both original and generated kinship face pairs. Recently, [14] proposed to extract feature descriptors from the generated fused image of each identity, where the fused image was a weighted linear combination of features obtained from multiple samples of the corresponding identity. And [15] presented a face age transformation model to generate facial images of various age groups to tackle age differences in the kinship images. Compared with methods that solely utilize existing data, these methods have the potential to use more discriminative information, thereby often achieving superior performance. However, such methods employ a generative approach to produce additional data, which not only increases the computational burden but also exposes the algorithm to the risk of concept drift due to the lack of discussion on the relationship between the generated additional data and the original data.

In this paper, we aim to tackle kinship verification task by leveraging additional data through a non-generative approach. Existing research has demonstrated the existence of correlations among various kinship types [17]. Furthermore, earlier psychological studies have indicated that feature sets extracted from all types of kin relationships are usually correlated, due to the high heritability of several key facial features [16]. Motivated by these findings, we propose a novel correlation calculation-based multi-task learning (CCMTL) method, which effectively exploits the correlations among various kinship types and converts them into auxiliary discriminative information, thereby mitigating the information isolation issue in kinship verification. A review of existing representative kinship verification methods is summarized in Table 1. Here, the column “Deep networks?" indicates whether the corresponding method incorporates deep neural networks, and the column “Additional data?" refers to whether the corresponding method explores discriminative information from data except given specific type of kin data (“*" denotes that the additional data originate from a non-generative approach).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Review of existing representative kinship verification methods.

https://doi.org/10.1371/journal.pone.0329574.t001

Proposed methods

In this section, we present our proposed kinship verification method which can exploit the correlations among various kinship types, i.e., father-son (FS), father-daughter (FD), mother-son (MS) and mother-daughter (MD). Let be the training set of specific kinship type , where p_qi and c_qi are two d-dimensional feature vectors, of the i-th parent and child image pair, respectively, indicates whether the i-th pair is parent-child relationship. We aim to learn a function to check whether two previously never seen images (p,c) have a valid kin relationship.

Kinship verification under isolated information environment

For kinship verification under isolated information environment, a separate classifier would be trained on given specific type of kin data . Let W_q denote the aimed transformation matrix, most existing metric learning methods take the following cost function form:

(1)

where is the empirical loss function, is a regulariser and λ is a trade-off parameter.

For , one can deploy a loss function that can be used to handle the binary classification problem. Hinge loss function is an important one used in kinship verification algorithm. Thus, we define , where is a similarity measure function and adopts the following form [53]:

(2)

To control the complexity of Eq (1) , we apply the Frobenius norm of matrix W_q, i.e., . The above algorithm for handling kinship verification under isolated information environment is hereinafter labeled as ‘Isolated_Learning’.

From isolated environment to correlated information utilization

Note that the ‘Isolated_Learning’ model introduced in Eq (1) is a similarity model for specific kinship type without utilizing additional data except given type of kin data. In this work, we are interested in the empirical evidence presented in [17], which confirms the existence of correlations among different kinship types. These correlations can be interpreted as additional data, while to better model the similarity between a mother and a daughter for example, one should consider it within the context of a broader range of kinship data, i.e., instead of leading the isolated view [e.g., seeking similarity metric on facial image pairs with mother-daughter relationship] to the model design, leading the connected view [e.g., seeking similarity metric not only on facial image pairs with mother-daughter relationship but also on facial image pairs with other correlated kin relationships]. Compared with methods that introduce additional data through generative approaches, the proposed method offers two key advantages. On the one hand, it can fundamentally alleviate the information isolation issue, by establishing connections between kin data of the specific type and other kin data with various kinship types. On the other hand, it incurs lower computational costs and avoids the risk of concept drift, by introducing additional data concerning the given type of kin data into the similarity learning process in a non-generative manner.

Let us denote all kinship types as , given specific kinship type as and the correlated kinship types concerning q as , respectively. In this work, each of U is also formulated as a bilinear function , i.e., as in Eq (2), where W_u denotes the dependence structure between parent and child with kin relationship u. Under the viewpoint of correlated information utilization, one should learn the dependence structure W_q in Eq (2) for kin relationship q through exploiting the correlation between W_q and . This suggests a multi-task learning procedure: learning these transformation matrices simultaneously. Consequently, we deploy a multi-task learning framework to jointly learn these transformation matrices and the relationship structures among them, which allows us to leverage the correlations between the given specific kinship type and other correlated kinship types in a non-generative manner. Specifically, we first exploit the correlations between the specific kinship type and other kinship types by analyzing their spatial positional relationships in the feature space. Subsequently, we learn a similarity measure by treating the specific kinship type and its correlated counterparts as distinct tasks within a multi-task learning framework. We detail the procedure in the following subsections.

Exploiting the correlations.

The primary problem is how to seek the correlated kinship types U concerning specific kinship type q. We observe that the Support Vector Data Description (SVDD) model is a one-class SVM whose boundary is solely determined by the support vector data points, capable of capturing the spatial structure of target class data, including its average characteristics and degree of dispersion. In light of this, we propose to transform the objective of seeking U into an investigation of the relationship between the distribution of support vectors and the centroid locations for a specific type of kinship data, in comparison to other types of kinship data.

Particularly, our algorithm has three steps.

Step 1: Conducting SVDD.

For kin data with relationship type q, we first extract the positive pairs of facial images. Then, we represent those pairs using the absolute values of the differences between the feature vectors of the parent and child. Finally, we deploy SVDD and obtain the centroid c_q.

Step 2: Acquiring correlations.

For kin data with relationship type q, we calculate the correlation between it and kin data with other types of relationships using . Thus we obtain a correlation vector .

Step 3: Obtaining U.

After conducting SVDD and calculating correlations, r encodes the correlations between specific kinship type q and other types of kin relationships. We can obtain the correlated types of kin relationships U concerning q via r.

Specifically, we first sort r in ascending order. Then, we collect the top K values as and, at the same time, record the types of kin relationships corresponding to these values as U.

Learning a similarity measure.

Now, we can utilize the correlation between given type of kin relationship q and each of U. From the perspective of multi-task learning, q and each of U are treated as separate tasks but are learned jointly based on some formal definition of how they are related. In this work, motivated by the studies of human genetics [16,54], we model W_q and in two parts:

(3)

where and are unique to the corresponding kin relationship, while all kin relationships share a common dependence structure W₀. Now, we plug the above into Eq (2) and obtain the following similarity measure functions:

(4)

These dependence structures are learned using the following regularized optimization problem:

(5)

where weight represents the degree of influence of kin relationship u, which is related to given type of kin relationship q, on the model.

Here, we believe that kin data that are spatially closer to given type of kin data are more relevant and could have a greater impact on the model. Therefore, we use the reciprocal of the previously calculated correlation vector r to represent the weights in Eq (5).

Based on this, we assign an extremely small value to given type of kin data itself, take its reciprocal as the weight for the first term of Eq (5), and then normalize it with . Now, we can combine and rewrite the first three terms of Eq (5) to obtain the final optimization objective:

(6)

Performing metric learning.

To estimate the parameters in Eq (6), i.e., the transformation matrices W₀, and the weights , we adopt an alternating iteration strategy to solve it.

First, we update W₀ by fixing other parameters. In this situation, Eq (6) is equivalent to the following expression:

(7)

To obtain W₀, we employ the stochastic sub-gradient descent method to solve Eq (7). And the gradient of objective function with respect to W₀ can be calculated as:

(8)

Now, one can update W₀ by using

(9)

where α is the learning rate.

Then, we update by fixing others. Similar to W₀, we can update by using:

(10)

where can be computed as:

(11)

Finally, we update with other new parameters. For this, we construct a Lagrange function as follows:

(12)

where θ is a Lagrange multiplier. We calculate the derivative of L in Eq (12) with respect to and obtain:

(13)

Now, we can update by using

(14)

We repeat the above steps until the algorithm meets a specific convergence criterion. The detailed process of our proposed CCMTL algorithm is summarized in Algorithm 1.

Algorithm 1. CCMTL.

Input:

  Training set: be the given type of

  kin relationship ;

           be the other types of

           kin relationships ();

  Parameters: trade-off parameters λ and η, the number of

  correlated kin types K, learning rate α, Lagrange multiplier

  θ, iteration number Nu, and convergence error τ

Output:

  The correlated types of kin relationships U concerning given

  type q;

  The transformation matrices W₀ and ;

  The correlation weights ;

1: Step1: Exploiting correlated information:

2: Step1.1: Conducting SVDD:

3:    For kin relationship q and other types of kin

  relationships , repeat:

4:       Extract the positive pairs of facial images;

5:       Represent pairs using abs. values of diff.

  between parent and child feature vectors;

6:       Deploy SVDD and obtain centroids, i.e., c_q and

  ;

7: Step1.2: Acquiring correlations:

8:    For each other type of kinship , repeat:

9:       Compute the correlation using ;

10:    Obtain a correlation vector ;

11: Step1.3: Obtaining the correlated kinship types U:

12:    Sort the correlation vector r in ascending order;

13:    Collect the top K values as ;

14:    Record the types of kin relationships corresponding to

  these values as U;

15: Step2: Learning a similarity:

16: Step2.1: Initilization:

17:    Set , ;

18:    Randomly Generate an extremely small number s ;

19:    Normalize 1/s and together and set them as

  the weights in Eq (6);

20: Step2.2: Local optimization:

21:    For , repeat

22:       Randomly select a pair of samples from

  ;

23:       Compute the gradient in Eq (8);

24:       Update W₀ by using Eq (9);

25:       for do

26:         Compute the gradient in Eq (11);

27:         Update by using Eq (10);

28:       end

29:       for do

30:         Compute the gradient in Eq (13);

31:         Update by using Eq (14);

32:       end

33:       Calculate F_r according to Eq (6);

34:       if r > 2 and , go to Step3.

35: Step3: Output:

36:    Output U, W₀, and .

Having obtained our CCMTL model, we apply it to each pair to measure the similarity using Eq (4). We then train an SVM classifier to predict whether each test pair has a kin relationship or not.

Experimental results and analysis

In this section, experimental evaluations for the proposed method are conducted on the largest publicly available kinship face image dataset, namely, KinFaceW [4]. We detail the experimental settings and results in the following subsections.

Experimental results

Baseline results.

To assess the efficacy of our proposed correlated information utilization strategy for kinship verification, we implemented two baseline methods. Specifically, we used Isolated_Learning as one baseline, employing Eq (1) and Eq (2). Additionally, we applied Isolated_Learning to the training data that had undergone SVDD filtering, and designated this approach as ‘Isolated_Learning+SVDD’.

Tables 2 and 3 compare the results of our method with two above baselines on the KinFaceW-I and KinFaceW-II datasets. The observations are as follows:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Comparisons (%) of our proposed method with two baselines on the KinFaceW-I dataset.

https://doi.org/10.1371/journal.pone.0329574.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Comparisons (%) of our proposed method with two baselines on the KinFaceW-II dataset.

https://doi.org/10.1371/journal.pone.0329574.t003

1) Isolated_Learning exhibits the lowest performance on both subsets. In contrast, Isolated_Learning+SVDD achieves average improvements of and across four kin relationships in the KinFaceW-I dataset, and improvements of and on the four relationships in the KinFaceW-II dataset, respectively. These improvements demonstrate the effectiveness of incorporating the SVDD model. Furthermore, the proposed CCMTL method further enhances performance, illustrating the efficacy of mining and leveraging correlations among diverse kinship types. To further illustrate this, we visualize the learned correlations on the KinFaceW dataset in S1 Fig. We can observe that the correlation between FD and MS is the highest, while that between FS and MD is the lowest, and such information is effectively captured and utilized by our model.

2) LBP and SIFT can yield promising performance. This demonstrates the effectiveness of these two types of features, which is consistent with some previous kinship verification results [4,13]. When integrated with SIFT, our proposed method, along with two baseline approaches, demonstrates superior average performance compared to when combined with LBP. This is primarily because SIFT exhibits superior robustness to scale changes, rotations, and illumination variations, making it more suitable for characterizing the variable similarities between parent-child pairs.

To better emphasize the effectiveness of our proposed correlated information utilization strategy and illustrate the differences between two feature representation methods, the receiver operating characteristic(ROC) curves of different methods are presented in S2 Fig. As evidenced by these figures, our proposed CCMTL model, which incorporates the correlations among various kinship types, significantly outperforms the two baseline approaches in terms of performance improvement. Furthermore, the SIFT feature yields the best performance, as indicated by the ROC curves.

Comparison with existing multi-task learning methods.

We conducted empirical comparisons with two competing methods in the realm of subspace regularized multi-task learning: No-group MTL [57], which assumes a low-dimensional subspace shared by all interrelated task parameters, and GO-MTL [58], a recently proposed method assuming each task parameter vector can be expressed as a linear combination of a finite set of basis tasks. Additionally, for both of these methods, we developed variant algorithms that utilized data filtered through SVDD, termed ‘No-group MTL [57] +SVDD’ and ‘GO-MTL [58] +SVDD’, respectively.

Table 4 summarizes the results. The observations are as follows:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Mean verification accuracy (%) of different methods with SIFT on the KinFaceW dataset.

https://doi.org/10.1371/journal.pone.0329574.t004

1) The SVDD versions of two competing multi-task learning methods outperform Isolated_Learning+SVDD, once again indicating that mining and utilizing the correlations among various kinship types can effectively improve generalization performance in kinship verification.

2) Both competing methods achieve higher performance when learning on data filtered by SVDD, demonstrating the effectiveness of the SVDD model.

3) By introducing an overlapping group structure, GO-MTL [58]+SVDD improves upon No-group MTL [57]+SVDD by and on KinFaceW-I and KinFaceW-II, respectively. This suggests that there is a certain structure between the kinship type to be learned and other correlated kinship types. Furthermore, the proposed CCMTL method achieves additional improvements of and over GO-MTL [58]+SVDD on two datasets. The reason is that, compared to the overlapping group structure assumed by GO-MTL [58]+SVDD, the proposed CCMTL method shares discriminative information across tasks via a common transformation matrix in a flexible manner, potentially leading to higher performance.

Comparison with existing metric learning methods.

Given that the proposed method necessitates learning transformation matrices, which aligns with the learning objectives of existing metric learning methods such as NRML [4], as well as its modified versions, NRCML [49] and PDFL+NRML [59], we further compared our method with these three metric learning approaches on the KinFaceW dataset. The performance of all comparison methods was sourced from the corresponding literature.

Tables 5 and 6 present the results on KinFaceW-I and KinFaceW-II, respectively. We can observe that PDFL+NRML [59] achieves the highest average performance among all compared methods. This superiority is attributed to its utilization of an intermediate-level discriminative feature learned from the LFW dataset, which can capture the parent-child similarity characteristics more accurately than the low-level features utilized by other methods. Compared to PDFL+NRML [59], the proposed CCMTL method, leveraging LBP and SIFT features, achieves average improvements of and across the four relationships in KinFaceW-I, and and on KinFaceW-II, respectively. These results further validate the effectiveness of our proposed method.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Comparisons (%) of our proposed method with three metric learning approaches on the KinFaceW-I dataset.

https://doi.org/10.1371/journal.pone.0329574.t005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Comparisons (%) of our proposed method with three metric learning approaches on the KinFaceW-II dataset.

https://doi.org/10.1371/journal.pone.0329574.t006

Furthermore, given the inevitable presence of noise in images captured in natural settings, it is crucial to assess the robustness of learning models against such noise. To this end, we artificially introduced - label noise into the training data by randomly altering the labels of the training samples. It should be noted that the labels of the test samples remained unchanged during this process. We conducted ten repeated experiments for both our proposed method and the comparison methods, and compared their average performances. Fig 2 presents the experimental results. One can see that the performance of all methods gradually deteriorates with increasing noise levels. With the continuous addition of noise, the performance of all compared metric learning methods decreases significantly, whereas the proposed method still achieves better performance in such challenging scenarios, indicating its noise robustness.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Mean verification accuracy of different methods with varying degree of label noise.

(a) On the KinFaceW-I dataset. (b) On the KinFaceW-II dataset.

https://doi.org/10.1371/journal.pone.0329574.g002

Comparison with state-of-the-art methods.

We also conducted a comparative analysis of our proposed method against several state-of-the-art learning approaches: feature-based Fusion [18], SILD+WCCN/LR [20], FDFN [24] and CLT [30], deep learning-based ASML [13] and KML [42], metric learning-based TXQDA [60], and hybrid-based [46]. More specifically, Fusion [18] utilizes texture color features from different color spaces, SILD+WCCN/LR [20] combines deep and shallow features, FDFN [24] introduces a deep learning framework that decouples features into high and low-frequency components and subsequently fuses them using attention mechanisms, CLT [30] leverages Curvelet Transform features and a CNN-based feature selection process, ASML [13] introduces generative adversarial concepts into kinship verification, KML [42] employs a pair of deep neural networks to learn a deep transformation projection space, TXQDA [60] extends cross-quadratic discriminant analysis (XQDA) by leveraging tensors to preserve kin data, and [46] jointly optimizes feature learning, metric learning and meta-learning. The performances of all comparison methods are sourced from the corresponding publications.

Table 7 presents the mean verification accuracy of our proposed CCMTL method and four other learning methods on the KinFaceW dataset. Observations are summarized as follows:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Comparisons (%) of our proposed method with state-of-the-art methods on the KinFaceW dataset.

https://doi.org/10.1371/journal.pone.0329574.t007

1) ASML [13] is closest to the proposed CCMTL method because both learn feature transformation matrices based on traditional manual features. We can observe that the proposed CCMTL method can achieve comparable performance to ASML, demonstrating its effectiveness. However, it is noteworthy that the projection space obtained by ASML [13] is learned from both original and GAN-generated kin data. In contrast, the proposed CCMTL method learns from existing kin data only, which not only incurs a lower computational cost but also mitigates the risk of concept drift.

2) The other comparison methods outperform CCMTL, primarily due to their utilization of multi-view, multi-scale or deep network techniques. More specifically, Fusion [18] utilizes texture color features from different color spaces and TXQDA [60] leverages two local texture descriptors at multiple scales. These two methods have achieved the two highest performance levels among all methods that utilize shallow features. This demonstrates the importance of leveraging multi-view and multi-scale discriminative information in kinship verification, as they enable the capture of fine-grained similarities between parent-child pairs that single-view or single-scale features cannot. SILD+WCCN/LR [20] and FDFN [24] leverage features extracted through deep neural networks. All two methods have achieved remarkably high performance. This emphasizes the significance of deep networks’ nonlinear modeling in kinship verification, since face images lie in nonlinear manifolds. Note that the proposed method focuses on leveraging correlations among diverse kinship types in the multi-task learning framework, thus only adopting widely-used shallow features. Additionally, it can be combined with multi-view, multi-scale and deep features, potentially improving its performance.

3) Notably, [46] achieves the highest performance, attributable to several factors. To start, it concatenates multi-scale features extracted from different convolutional blocks and constructs a Family ID Module to learn more discriminative representations. Moreover, it incorporates a deep nonlinear metric learning process to better capture the underlying relationships within kin data. Additionally, a meta re-weighting network is constructed to ensure algorithm robustness. In contrast to other approaches that focus solely on feature learning, metric learning, or deep learning, the hybrid [46] is more likely to achieve superior performance. This is because it formulates a comprehensive solution by integrating the strengths of these individual learning paradigms.

Parameter analysis.

Since the proposed CCMTL method involves the selection of the number of correlated kinship types K, we also investigated the algorithm’s performance across various K values.

Fig 3 illustrates the average performance of our proposed method versus different values of K on the KinFaceW dataset. Here, K = 0 corresponds to the ‘Isolated_Learning’ scenario. We can observe that the proposed method exhibits the lowest performance under this setting. As K increases, the performance of the proposed method improves due to the model’s capacity to leverage correlation information from diverse related kinship types, thereby enhancing its discriminative capabilities. However, when K attains its maximum value, the performance of the proposed method declines slightly, which is attributed to the interference from low-related kinship types during the model’s learning process. Therefore, we recommend setting K = 2 in practical applications.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Mean verification accuracy of our proposed method versus different values of parameter K.

(a) On the KinFaceW-I dataset. (b) On the KinFaceW-II dataset.

https://doi.org/10.1371/journal.pone.0329574.g003

Conclusions

Facial image-based kinship verification has a wide range of applications, such as social media analysis, social relationships classification, and missing children identification. However, current methods face significant limitations in effectively characterizing the facial similarity between parents and children. This paper introduces a novel correlation calculation-based multi-task learning (CCMTL) method for kinship verification. The proposed CCMTL learns a set of metrics from both the specified kinship data and the correlations among diverse related kinship types within a multi-task learning framework. On the one hand, by incorporating inter-kinship-type correlations, the proposed method can fundamentally resolve the information isolation issue that existing approaches face. On the other hand, by utilizing these correlations as additional discriminative information in a non-generative manner, the proposed method eliminates the necessity for extensive parameter tuning within deep networks, thereby reducing computational costs.

Experimental results have demonstrated the effectiveness of the proposed method, and they also advocate the incorporation of inter-kinship-type correlations into kinship verification to enable more efficient training and deployment, particularly when computational resources are limited. In future work, one can construct a unified framework that integrates correlation utilization, feature learning, and metric learning to further improve the performance of kinship verification.

Supporting information

Acknowledgments

The authors are grateful to the anonymous reviewers for their constructive feedback, which strengthened the methodological rigor and analytical depth of our study, and to Dr. Wei Chen for her expert proofreading and linguistic refinement, ensuring the manuscript adheres to the highest standards of scholarly communication.