https://orcid.org/0000-0001-6829-9705
Figures
Abstract
In the field of data security, biometric security is a significant emerging concern. The multimodal biometrics system with enhanced accuracy and detection rate for smart environments is still a significant challenge. The fusion of an electrocardiogram (ECG) signal with a fingerprint is an effective multimodal recognition system. In this work, unimodal and multimodal biometric systems using Convolutional Neural Network (CNN) are conducted and compared with traditional methods using different levels of fusion of fingerprint and ECG signal. This study is concerned with the evaluation of the effectiveness of proposed parallel and sequential multimodal biometric systems with various feature extraction and classification methods. Additionally, the performance of unimodal biometrics of ECG and fingerprint utilizing deep learning and traditional classification technique is examined. The suggested biometric systems were evaluated utilizing ECG (MIT-BIH) and fingerprint (FVC2004) databases. Additional tests are conducted to examine the suggested models with:1) virtual dataset without augmentation (ODB) and 2) virtual dataset with augmentation (VDB). The findings show that the optimum performance of the parallel multimodal achieved 0.96 Area Under the ROC Curve (AUC) and sequential multimodal achieved 0.99 AUC, in comparison to unimodal biometrics which achieved 0.87 and 0.99 AUCs, for the fingerprint and ECG biometrics, respectively. The overall performance of the proposed multimodal biometrics outperformed unimodal biometrics using CNN. Moreover, the performance of the suggested CNN model for ECG signal and sequential multimodal system based on neural network outperformed other systems. Lastly, the performance of the proposed systems is compared with previously existing works.
Citation: A. El_Rahman S, Alluhaidan AS (2024) Enhanced multimodal biometric recognition systems based on deep learning and traditional methods in smart environments. PLoS ONE 19(2): e0291084. https://doi.org/10.1371/journal.pone.0291084
Editor: Bilal Alatas, Firat Universitesi, TURKEY
Received: December 17, 2022; Accepted: August 22, 2023; Published: February 15, 2024
Copyright: © 2024 A. El_Rahman, Alluhaidan. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper.
Funding: The authors extend their appreciation to the Deputyship for Research & Innovation, Ministry of Education in Saudi Arabia for funding this research work through the project number RI-44-0349. funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Multimodal biometric recognition systems are the future of smart environments’ security [1]. The systems that use biometrics have different features over the traditional systems, which use pin-based authentication [1, 2]. Multimodal biometrics are executed at various fusion levels and fulfill greater identification performance than unimodal biometrics [3, 4]. Also, it provides anti-spoofing measurement, where the intruder will find it difficult to spoof multiple biometric modalities Electrocardiogram (ECG) information along with other biometrics will minimize credentials forgery and intrusion. Utilizing ECG as one of biometrics provides an advantage to the system where the heartbeat is inherited to a subject that is secured, confidential, and difficult to be forged. These days, fraud identification in smart environments is considered one of the most criminal security issues. To prohibit these issues, many methods are utilized like the techniques of biometrics identification, because of the necessity of extremely reliable security methods in confidential systems [5]. Biometric identification systems are highly used in several public security systems [6, 7] such as surveillance, recognition, law enforcement to attain superior recognition performance [8, 9]. In smart environments’ context, multimodal biometrics system has been employed to verify individuals and make verification simpler (Rajasekar et al., 2022). They can identify subjects depend on behavioral features, physical features [10], or physiological signals, such as ECG and EEG (Electroencephalogram) [6, 11]. Real-time recognition systems use the patterns and templates which are extracted from individuals and matched with enrolled ones [7] to decide for acceptance or rejection [8, 12, 13].
Problem statement
There are some issues in real-time unimodal systems, like spoof attacks, sensed data artifacts, interclass variations, and interclass similarities [14, 15]. Multiple biometric sources are used now and needed for reliable identification. With advances in artificial intelligence more accurate results and better performance are achieved. Therefore, a multimodal biometric system is suggested to reduce some of these issues by the fusion of ECG and fingerprint modalities. ECG provides the benefit of liveness recognition to enhance biometric system robustness [14]. Studies of the combination of ECG with a fingerprint as multimodal biometric systems are few [15, 16] conducted different traditional classifiers on multimodal biometric systems based on different fusion levels of ECG and fingerprint. This study presents a study of the performance of fusing ECG and fingerprint biometric modalities with different classification techniques and different fusion levels.
Contribution
To summarize, previous studies were more focused on one modality or one biometric for authenticity. Our approach increases the accuracy of classification by combining parallel and sequential multimodal biometric systems with various feature extraction and classification methods. The proposed multimodal systems of ECG with fingerprint modalities comparing the deep learning approaches with traditional methods become more proper and efficient to be applicable. The key contribution of this study can be briefly described as:
- Developing enhanced unimodal and multimodal biometric systems for smart environments based on CNN and traditional methods using different fusion levels of fingerprint and ECG signal.
- In comparison to other computational intelligence techniques, the proposed models can be used for recognition with satisfactory results and achieve improved rating optimization. A multimodal recognition system that combines an ECG signal with a fingerprint is effective.
- The efficiency of the proposed methodology is proved and the performance of the multimodal systems outperformed unimodal systems employing different classifiers, fusion levels, and rules.
The paper is organized as follows: Section 2 presents a literature review. Section 3 covers the details of the unimodal and multimodal biometrics methodology. Section 4 shows the results of the experiments. Section 5 has the conclusion.
Related works
Various unimodal and multimodal biometric systems have been developed and presented by the researchers with the explanation of the success metrics of these systems.
Fingerprint recognition system
Worldwide, fingerprint biometric is massively used by forensic experts and laboratories in criminal investigations, presence monitoring systems, and law enforcement systems [10, 17–19]. A fingerprint consists firstly of dark lines series that present the peaking and apparent part of the friction ridge skin, and secondly the white spaces that present the valleys between the ridges [20]. The ridge of a finger image is a curved line that is either a continuous line or ceases at a ridge ending. When 2-ridges intersect at a certain point, it is called a bifurcation. The bifurcation and ridge ending is considered as the minutiae points that are utilized to extract the unique features of fingerprints [10]. In fingerprint biometric systems, a feature extraction process is critical similar to the minutiae template matching [21].
Some researchers considered different methods such as neural networks, fuzzy logic, and deep learning to develop advanced robust fingerprint recognition techniques [18, 22, 23]. In [24] a fingerprint recognition system is suggested which utilizes wavelet transformation with simple minutiae equivalent to increase the reliability of fingerprint analysis and identification. Ali et al. [2] presented fingerprint identification and authentication method with minutiae equivalent approach with Euclidean distance. The tests are performed using two fingerprint databases. The research by Kahraman et al. [7]suggested fingerprint identification with image processing techniques. Multilayer perceptron method for features extraction and Regression Neural Network is applied for identification. Dale et al. [25] presented a system with the DCT features of a fingerprint while in Gupta & Walia [26] the suggested system with Gabor filter. Tico et al. [27] developed fingerprint recognition methods with diverse techniques such as Euclidian distance. Dale et al. [25], Borra et al. [8], and Mote [28] studied a classification technique of fingerprint with wave-atom transform for removing noise along with an adaptive genetic neural network for classification. Depending on the minutiae points matching or on the similarities in the fingerprint structure, there are many fingerprint systems are developed as listed in Table 1.
ECG recognition system
ECG can be employed solely for subjects’ identification or combined in a multimodal biometric recognition system [10]. It is considered a rising modality for the human identification system. The difference between the developed unimodal and multimodal systems is the extracting method of features and the features set characteristics [29]. ECG is used as a measurement of the heartbeat electrical activities [8]. It is considered one of the best biometrics since it represents the evidence of the individual’s aliveness [10]. The ECG is a physiological low-frequency signal [30] and includes repolarization and depolarization of the muscle fibers constructing the heart. The repolarization is corresponding to T-wave and the depolarization is corresponding to the QRS wave, P-Wave, and the other regions are considered as a baseline [31]. There are many ECG systems are developed as shown in Table 2. The motivation of using ECG is:
- The heartbeat signal of a person holds a unique signature and it is stable for a long time.
- The heartbeat signal offers direct solutions to the aliveness detection as it cannot be captured from deceased body parts, fake finger, or a high-resolution video. It is also difficult to steal and replicate a heartbeat signal. So the imposters will face a greater challenge to collect an illicit copy of the heartbeat signal of the actual user.
Fusion level in identification
A multibiometric fusion method for identification using two modalities has been explored in research. For example, the iris and the fingerprint are processed to generate different score levels then fusion is applied on each score. Each score is split into three zones of interest. The fusion is applied using two approaches: classification and fuzzy logic. The proposed fusion methods outperform single modality approaches [34]. For a robust human identification system, the score level fusion of face and iris attributes are combined and re-classified to improve the individual unimodal systems performance. The result provide a proof on how accurate is the multimodal biometric system [35]. The paper by [36] suggests a methodology for combining the identification results of face and ECG data. The accuracy was 98.8%. “By using a fusion approach the identification accuracy improved to 99.8%” [36].
Multimodal biometrics system
A single biometric modality method measures only one behavioral or physiological feature. So, it leads to a lower reliable system and can be impacted by the actual environmental conditions [6]. The multimodal biometric system reduces some of the issues using the acquired information from different sources [37]. The significance of biometric identification in smart environments is highlighted in (Rajasekar et al., 2022). The multimodal biometrics fusion can be implemented at various levels. The fusion levels are matching score, feature extraction, raw data, or decision level [7, 38–40]. The main frameworks of classification techniques combination are both, the first one; the same input modality presentation is utilized to all classifiers. In the second one, the different input modality presentation is considered for each classifier [41]. The fusion models are two categories that are sequential and parallel, where, parallel fusion provides higher accuracy [42]. In general, ECG is used as an auxiliary biometric in a multimodal biometric fusion that providing some advantages [43]. Where the fusion of many biometric modalities can improve the system performance and support anti-spoofing [6, 38]. Deep learning techniques are utilized in biometrics recognition to improve the system performance of the traditional classifiers. Several applications use traditional and deep learning techniques for biometric modalities. Gupta et al. [44] developed a new cancelable multimodal system that combines biometrics using the projection-based algorithm. The experiments are conducted by three chimeric multimodal datasets and the findings fulfill high performance. Hossain et al. [45] presented a multi-stage verification system utilizing multibiometrics and verifiers of multibiometrics. The researchers studied the symmetric rejection effectiveness for their proposed verification systems. Kant & Chaudhary [46] presented a multimodal biometric scheme that fusing the fingerprint, finger knuckles print, and palm print using score level. Arora et al. [47] implemented a CNN deep learning technique for combining the extracted features of the face and iris traits. Yudistira and Kurita [24] studied a multimodal CNN for multimodal correlations capturing over arbitrary time-stamps. The action recognition using a deep CNN was performed using spatial and temporal streams. The researchers developed a correlation network with a Shannon fusion approach by averaging the two streams to learn a pretrained CNN. This model was established to complement the existing approaches of network fusion [48]. Alay & Al-Baity [21] developed a multimodal identification system using three CNN deep learning models for human recognition with the modalities of face, iris, and finger vein. The researchers used CNNs for the features extraction and a softmax classifier is used for image classification. The system was developed by combining the three CNN models of the three modalities. Each CNN mode was built using the VGG-16 model, the optimization was applied using the loss function (categorical cross-entropy) and Adam algorithm. To prevent overfitting, image augmentation and dropout techniques were applied. Score and feature level fusion techniques were used for the CNN models fusion. There are different multimodal biometric systems established the traditional and the deep Learning techniques and using various fusion levels and modalities are developed. Most findings show that multimodal biometric systems outperformed unimodal biometric systems as indicated in Table 3.
Methodology
For setting experiment, data was prepared from multiple sources then augmented as detailed below. Models are explained in this section. We used Thinkpad Intel Core i3 with RAM 4GB and 256 GB hard disk and MATLAB for constructing CNN and performing the analysis.
Dataset information
In many multimodal biometric systems, the databases of multiple biometrics are combined to generate virtual individuals instead of a gathering of the real biometrics from each individual [52]. In this study, a virtual database of multimodal biometrics is constructed to be conducted in the performance evaluation. It is gathered from two public databases of fingerprint (FVC2004) and ECG [54]. FVC 2004 (FVC2004) fingerprint database is utilized that contains four various datasets that contain 80 fingers each. The MIT-BIH [54] dataset is an ECG database that contains 47 subjects. The virtual multimodal dataset is generated using ECG and the fingerprint datasets for the performance evaluation. One individual from the ECG dataset is randomly allocated to an individual from the fingerprint dataset. The multi-modal dataset is gathered from 47 subjects, everyone has been assigned to two samples of ECG and fingerprint [16]. Extensive experimental testing is conducted for the evaluation of the proposed models with the original virtual dataset without augmentation (ODB) and the virtual dataset with augmentation (VDB).
A multimodal biometric system is a robust solution for several biometric recognition systems flaws. Integrating multiple biometrics is observed to be a general issue in diverse real biometric recognition systems. In this section, the performance of the suggested unimodal and multimodal recognition systems is assessed. The tests are performed on ECG and fingerprint datasets. The two databases used in the evaluation are MIT-BIH by Wang et al. [12] for ECG and FVC2004 (FVC2004) for fingerprint grounded on NN, FL, LDA classifiers. Extended experiments are implemented to assess the proposed system using 47 subjects’ data derived from a virtual multimodal database. Moreover, a ten-fold cross-validation technique is used for testing the recognition implementation. The following subsections describe average results. Denoting, in the first proposed parallel multimodal system, a score level fusion is used while in the second sequential multimodal system of ECG and fingerprint, a decision level fusion is applied. In this study, unimodal and multimodal biometric systems with CNN using different fusion levels of fingerprint and ECG signal are evaluated. More experiments are being conducted to evaluate the proposed models with ODB and VDB datasets.
Data augmentation.
Data augmentation is utilized to develop a more robust model for overfitting. It is an approach to create an artificial dataset sample using the original ones [55]. It is utilized for increasing the size of training data and enhancing the performance [56, 57]. The augmentation implementation of the proposed dataset is based on the methods proposed in [56, 57]. Fingerprint database is augmented to construct a database by translations (two times), reflections (two times), and color manipulation (two times). The constructed dataset is six times greater than the original dataset. ECG dataset is augmented to construct a database by 15% scaling (two times) and 30% scaling (two times). The constructed dataset is four times greater than the original dataset.
Biometric models based on traditional methods
ECG biometric model.
Fig 1 shows the model of ECG biometric. It is evaluated using MIT-BIH database [54] and the experimental testing is conducted on 47 subjects. The noise and artifacts in ECG signal are attenuated to be corrected by digital filters before QRS Complex feature detection. The QRS detection processing is achieved utilizing the Pan and Tompkins algorithm [58] which performs a competent process for extracting features of the QRS complex. The extracted features are entered into Linear Discriminant Analysis (LDA), Fuzzy Logic (FL), or Feature Map-Neural Network (SOM-NN) classifiers [17] to apply the features matching. Further details explaining the proposed technique are found in [16, 17].
Fingerprint biometric model.
The fingerprint recognition model is indicated in Fig 2. The model consists of different phases, first phase is fingerprint image preprocessing, following phases are extraction of minutiae, detection of core point, then extracting features, and ultimately Identification. For a reliable recognition system, preprocessing of fingerprint images is the necessary phase before extracting the features and matching them. Preprocessing phase increases the clarity of ridge structure [12]. It is performed by Histogram Equalization (HE) and Fast Fourier Transform (FFT), then the preprocessed image is now available for subsequent processes. A very critical factor here is to have the right estimation of local ridge orientation during the extraction of minutia points and matching. The orientation map of the image is formed with a 3×3 Sobel filter [7]. Computationally, the Sobel filter is convolved with the image to calculate the derivatives using 2 kernels (3×3) (one for vertical and the second for horizontal changes). To calculate the oriented gradients Eq (1) and Eq (2) are utilized and Eq (3) is utilized to calculate all pixels directions.
(1)
(2)
(3)
where: A is the fingerprint image, Gh is horizontal and Gv is vertical derivatives, while θ is directions of all pixels.
Fig 3 indicates the preprocessing phases for the demonstrated sample image. The features are extracted after core point detection. The minutia point’s histograms are computed in a certain interesting region, which is considered as a vector of features. Then, a similar approach is applied to bifurcations and ridge endings. Fig 4 shows a sample for bifurcation and ridge ending combined. The feature vector length is 1460 which includes every feature extracted from images after they were filtered. This feature vector is entered into SOM-NN, LDA, or FL classifiers [16, 17] for the identification of fingerprint images. The model is evaluated with the FVC2004 database (FVC2004). More details about the proposed model are found in [16].
Fingerprint and ECG multimodal systems.
In this study, two multimodal biometric systems are presented using fingerprint and ECG biometrics with different fusion levels. The fusion Levels are decision level and score level. The first system is a sequential multimodal system using the decision level fusion of fingerprint and ECG for human identification. The extraction of features is fulfilled, and then the biometric templates are constructed from these features. The extracted features are then fed into the matching stage. At this point, the NN, FL, and LDA classifiers are employed. Finally, the final decision is taken according to the decision level fusion. The second system is a parallel multimodal system using the score level fusion of fingerprint and ECG for human identification. When the extraction of features is fulfilled, the extracted features entered into the matching stage. The same classifiers used in the first system were used again at the matching stage. In the end, the final decision is taken according to the score level fusion.
Sequential and parallel multimodal system
The block diagram of a sequential multimodal biometric system is shown in Fig 5A. The sequential multimodal structure depends on decision level fusion starts with ECG signal identification to guarantee that the accepted score is generated from an alive subject. Whereas ECG identification is better for rejecting impostors and fingerprint identification is better for accepting genuine subjects. Rejection is the final decision of the system if the subject is rejected according to its ECG. If the subject is accepted, it is fed to the multimodal biometric system to combine them at the decision level to produce the final decision. After the genuine acceptance and the impostors’ rejection by the ECG system, the other subjects are recognized by the decision level fusion of fingerprint and ECG. The combination of decision outputs is formed by applying (OR) logic rule to fulfill the greatest system performance. The block diagram of a parallel multimodal biometric system is demonstrated in Fig 5B. The parallel multimodal structure depends on score level fusion for fingerprint and ECG. The fingerprint and ECG of the subject are obtained in the verification stage, unlike the sequential multimodal system which does not require authenticating by both modalities. As features extraction is completed, the extracted features of both modalities are moved to classifiers for finding matching scores. Finally, a fusion rule is applied to the matching scores, where, ECG signal scores of accepted users are fused with their scores of the fingerprint to produce the final decision upon score level fusion. If the fused score < 90% (a predefined threshold), the user is rejected as an impostor, otherwise the user is accepted as genuine. In this study, Fuzzy Logic (FL), Linear Discriminant Analysis (LDA), or Neural Network (NN) is used for the fusion. One of these classification methods is executed to get the scores of each biometric. Subsequently, the fusion rule is applied to fingerprint scores and ECG scores to obtain the final score. One of three fusion rules that are Product, Sum (Summation), and (MAX) Maximum rules is applied. The applicable threshold is 90%, yet, BE et al. [51]set a low threshold (75%) in their proposed model.
Biometric models based on CNN.
Recently, Deep learning is used in different domains especially CNN. The structure of CNN includes hidden layers and different parameters [57]. There are several CNN models like Caffe-Net, Alex-Net, and VGG-Net for large-scale image classification [21]. Where VGG-Net is utilized for feature extraction of fingerprint and ECG in the proposed CNN model because its architecture is much deeper than other models [57].
Simonyan and Zisserman [59] proposed the VGG-Net to deepen the network and enhance classifier performance by integrating multiple convolutional layers. This may be accomplished by using several stacked kernel filters in place of a large sized filter. Every few layers, the max pooling layer is utilized to minimize the spatial dimensions. The first network structure to use a block-based architecture is the VGG. All hidden layers now include ReLU nonlinearity. VGG has more weight parameters than AlexNet, but it converges faster because of taking fewer epochs due to the regularization implication by its depth and the small size of convolution filter [60]. More details in [59].
In this study, the VGG-Net architecture is indicated in Fig 6. It involves five convolutional (Conv) layers which are followed by a pooling layer for each and 3 FC (fully connected) layers and Fig 6 shows all the parameters that are used in this work. The activation function is Rectified Linear Unit (ReLU) in all hidden layers.
The features that obtained from both ECG and fingerprint modalities are further processed based on a fusion module as indicated in Fig 8. Where, the fingerprint feature vector is concatenated into an ECG feature vector to generate a single features vector. The concatenated feature is entered an additional fully connected layer to determine the final class.
ECG biometric system based on CNN.
ECG-CNN recognition system using a CNN model unlike ECG recognition system using the traditional methods, where it used ECG signals without noise correction of segmentation methods. The main phases of ECG-CNN include the transformation of ECG signals (1-D) into ECG images (2-D) by plotting each in a 224×224 grayscale image as in [57]. Consequently, 2-D CNN can be applied to ECG images. Then, the features template is formed by the ECG-CNN model before the classification phase.
Multimodal systems based on CNN
Sequential multimodal system based on CNN.
This system is a sequential multimodal biometric system with CNN and the decision level fusion of fingerprint (FP-CNN) and ECG signal (ECG-CNN) for subject identification. The extraction of modalities features is achieved by utilizing CNN and these features formed the biometric templates. Then, the classification techniques are applied to enhance the system performance and the decision level fusion is applied for making the decision. This system according to the decision level fusion begins with ECG signal identification to maintain accepted score generated from an alive subject. Whereas, ECG identification is suitable for rejecting impostors and fingerprint identification is better for accepting genuine individuals. Fig 5 explains the sequential multimodal biometric system. Rejection is the final decision of the overall system if the subject is rejected due to its ECG. Yet, If the subject is accepted, it is fed to the multimodal system to combine them at the level of decision to produce the final decision. After the genuine acceptance and the impostors’ rejection by the ECG model, the other individuals are identified by the decision fusion of fingerprint and ECG. The combination of decision outputs is formed by OR Rule to fulfill the best system performance.
Parallel multimodal system based on CNN.
This system is a parallel multimodal biometric system utilizing CNN and the feature fusion of fingerprint and ECG signal for subject identification. The extraction of modalities features is achieved by utilizing CNN and the resulting feature vectors are combined using the fusion level method. The parallel multimodal system with CNN for fingerprint and ECG is shown in Fig 7. The fingerprint and ECG of the subject are presented in the verification stage, unlike the sequential multimodal that does not require authenticating by two modalities. The extracting feature of specific modalities is achieved by the same CNN utilized in the sequential multimodal. The extracted feature vectors of both modalities are applied to the feature level fusion before the classification as indicated in Fig 8.
Results and discussion
Performance measurements
There are various metrics to measure the system performance for evaluation, which are described below [16]:
- ■ ROC Curve: is a graphical representation which describes the performance with classification accuracy [12].
- ■ Area Under the ROC Curve (AUC): computes total 2-dimensional area below the ROC curve, as AUC indicates better performance when it is increased near to 1 [12].
- ■ Specificity (True Negative Rate): computes negatives proportion correctly identified as in Eq (4) [17]:
(4) where FP is False Positive that represent number of imposter acceptance, and TN is True Negative that represent imposter rejection number [17, 56, 57].
- ■ Sensitivity (True Positive Rate): computes positives proportion correctly identified as in Eq (5) [16]:
(5) where FN is False Negative which represents legitimate rejection number, and TP is True Positive that represent legitimate acceptances number [17, 56, 57].
- ■ Efficiency: produces the ratio of times where test produces correct result to overall number of tests as in Eq (6) [17]:
(6)
Simulation results
Fingerprint biometric recognition system.
The fingerprint recognition system is examined using the FVC2004 database (FVC2004) with a ten-fold cross-validation algorithm. Findings (AUC %) for fingerprint recognition system are (CNN = 86.3, NN = 86.6, FL = 83, LDA = 83.2). The performance results of proposed system are reported in Table 4 and while Fig 9 shows the ROC curve for fingerprint recognition system. The presented system implies a comparable performance of other good systems. Generally, the suggested system signifies an acceptable performance with more robust results, reliable and effective computational time and computational cost compared to the works cited in Table 1.
ECG biometric recognition system.
The implementation of the ECG recognition system is examined using 47 subjects MIT-BIH database [54], utilizing ten-fold cross-validation algorithm, to explain, ECG signals are divided into ten equal folds, nine folds are used for training while the residual is used for testing. The metrics (AUC %) for the ECG recognition system are (CNN = 99.863, NN = 95.1, FL = 93.688, LDA = 90.214). The performance results of proposed system are reported in Table 5 and while Fig 10 shows the ROC curve for the ECG recognition system. The suggested system signifies a comparable performance of other good systems. Generally, the suggested system shows better performance with more robust results compared to the works cited in Table 2.
ECG and fingerprint multimodal biometric systems
Proposed sequential multimodal system.
The virtual multi-modal dataset is used in the assessment of the suggested parallel multi-modal system according to the decision level fusion. The metrics (AUC %) for sequential multimodal biometric recognition system are (CNN = 98.1, NN = 98.5, FL = 94.1, LDA = 90.8). The performance results of proposed system are reported in Table 6 as well as ROC curves for multimodal biometric recognition system based on 3 different fusion rules at threshold 90% are shown in Fig 11.
Proposed parallel multimodal system.
The virtual multi-modal dataset is used in the evaluation of the proposed parallel multi-modal approach according to the score level fusion. The metrics for parallel multimodal biometric recognition system are stated in Table 7 as well as ROC curves for multimodal biometric recognition system based on 3 different fusion rules for each classifier at threshold 90% are shown in Fig 12.
Simulation results based on data augmentation.
The dataset augmentation impact on the proposed multimodal system is indicated in Table 8. The dataset augmentation impact in terms of the error reduction on the suggested multimodal system is 0.01845 without augmentation and 0.003 with augmentation. The average error of the proposed system without augmentation is greater than the average error with augmentation. So, the accuracy of the proposed system with augmentation has been improved.
Comparison between the suggested approach and other systems.
According to this study, the performance of ECG and fingerprint fusion biometric modalities with different classification techniques and different fusion levels is evaluated. By comparing with existing systems, the proposed unimodal (ECG recognition system & fingerprint recognition systems) and multimodal systems have achieved acceptable recognition results. Overall, this work has a good contribution and has presented sufficient experimental results.
The comparison between the unimodal and multimodal based on traditional and deep learning models is indicated in Fig 13. Most of the research either focused on certain features and artificial intelligence algorithms for identification or did not incorporate fusion levels to produce accurate results. Measurement of accuracy depending on the features used are reported. Because there is no complete database to test the approach, the data was combined as described in section “Dataset Information” and it is widely approach for this type of research. The approaches proposed by [42, 51, 57, 56] are authentication techniques that are combination of (ECG) and fingerprint multimodal. Some of the proposed systems suggested to test model on a real database. [21, 38, 47] have used face and other biometric traits in biometric identification systems and obtained better results than using two or one biometric traits. [52, 53] used ECG, Face, and Fingerprint for authentication, these multibiometric system fusing the ECG signal with the face and fingerprint biometrics. Yudistira & Kurita [24] Suggest a correlation network with a Shannon fusion for learning a pre-trained CNN it also emphasizes the importance of multimodal correlation as it enhanced the accuracy of the recognition results. As most of research relies on classical algorithms and few were dependent on deep learning, we propose the novelty in employing advanced deep learning into multi and single modalities with different fusion levels. For effectiveness of the approach, we utilize the known metrics in classification as each scholar presented different metrics depending on their method. Thus, even if different metrics used, the usability and reliability of the system can be implied. Generally, the suggested multimodal system implies an acceptable performance, signifies a similar performance of other good systems, and is more robust than previous works covered in Tables 3 and 9. So, the suggested multimodal systems can be adopted for recognition with adequate performance.
Conclusion
A strong multiple biometric modalities can greatly enhance the security of a system. They provide anti-spoofing measurement, where the intruder will find it difficult to spoof multiple biometric modalities, ECG information combined with other biometrics will minimize credentials forgery and intrusion. A superior recognition performance with deep learning can be obtained in a multimodal biometric system at different fusion levels over unimodal technique. Utilizing ECG as one of the biometrics provides an advantage to the system where the heartbeat is inherited to a subject that is secured, confidential, and hard to be forged. In this study, unimodal and multimodal biometric systems using CNN are presented using different levels of fusion of fingerprint and ECG signal. The performances of CNN and different traditional classification techniques with unimodal and multimodal biometric systems using ECG and fingerprint have been studied. A powerful data combination that is not easily guessed can increase the security of the system and reduce marginal error. The tests are performed on ECG and fingerprint records to assess the performance of the suggested multimodal biometric system. FVC2004 database is utilized for the fingerprint, MIT-BIH database is utilized for ECG, and other experiments are being implemented to assess the suggested multimodal system with 47 subjects using a virtual multimodal dataset. Further tests are being conducted to evaluate the proposed models with ODB and VDB datasets.
The suggested CNN model for ECG signal and sequential multimodal system based on neural network outperformed other systems while optimum performance of the parallel multimodal achieved 0.96 Area Under the ROC Curve (AUC) and sequential multimodal achieved 0.99 AUC, in comparison to unimodal biometrics which achieved 0.87 and 0.99 AUCs, for the fingerprint and ECG biometrics. The performance of the proposed multimodal biometrics outperformed unimodal biometrics using CNN. Generally, the results imply that the suggested unimodal and multimodal biometric systems based on ECG and fingerprint using deep learning have satisfactory identification results compared to other previous works.
References
- 1. Rajasekar V, Predić B, Saracevic M, Elhoseny M, Karabasevic D, Stanujkic D, et al. Enhanced multimodal biometric recognition approach for smart cities based on an optimized fuzzy genetic algorithm. Sci Rep. 2022 Jan 12;12(1):622. pmid:35022474
- 2. Ali M, Mahale V, Yannawar P, Gaikwad A. Overview of Fingerprint Recognition System. In 2016.
- 3. Dhiman G, Chahar V. Spotted Hyena Optimizer: A Novel Bio-inspired based Metaheuristic Technique for Engineering Applications. Advances in Engineering Software. 2017 Jun 1;114.
- 4. Alkeem EA, Yeun CY, Yun J, Yoo PD, Chae M, Rahman A, et al. Robust Deep Identification using ECG and Multimodal Biometrics for Industrial Internet of Things. Ad Hoc Netw [Internet]. 2021 Oct 1 [cited 2022 May 24];121(C). Available from: https://doi.org/10.1016/j.adhoc.2021.102581
- 5. Oulhiq R, Ibntahir S, Sebgui M, Guennoun Z. A fingerprint recognition framework using Artificial Neural Network. In: 2015 10th International Conference on Intelligent Systems: Theories and Applications (SITA). 2015. p. 1–6.
- 6. S J, Maheswari N, Venkatesh R. Neural network based automatic fingerprint recognition system for overlapped latent images. Journal of Intelligent and Fuzzy Systems. 2015 Aug 10;28:2889–99.
- 7. Kahraman N, Cam Taskiran Z, Taskiran M. Novel Feature Extraction Methodology with Evaluation in Artificial Neural Networks Based Fingerprint Recognition System. Tehnicki Vjesnik. 2018 May 27;25.
- 8. Borra SR, Jagadeeswar Reddy G, Sreenivasa Reddy E. Classification of fingerprint images with the aid of morphological operation and AGNN classifier. Applied Computing and Informatics. 2018 Jul 1;14(2):166–76.
- 9.
Pal A, Singh YN. ECG Biometric Recognition. In: Ghosh D, Giri D, Mohapatra RN, Savas E, Sakurai K, Singh LP, editors. Mathematics and Computing. Singapore: Springer; 2018. p. 61–73. (Communications in Computer and Information Science).
- 10. Noureddine B, Naït-ali A, Fournier R, bereksi reguig F. ECG Based Human Authentication using Wavelets and Random Forests. International Journal on Cryptography and Information Security. 2012 Jun 1;2.
- 11. Srivastva R, Singh YN, Singh A. Statistical independence of ECG for biometric authentication. Pattern Recognition. 2022 Jul 1;127:108640.
- 12. Wang J, Li J, Cao L. An Improved Fast Thinning Algorithm for Fingerprint Image and Its Application. Journal of Computational Information Systems. 2011 Jul 1;7.
- 13. Dhiman G, Kaur A. STOA: A bio-inspired based optimization algorithm for industrial engineering problems. Eng Appl Artif Intell. 2019 Jun 1;82(C):148–74.
- 14. Acharya UR, Oh SL, Hagiwara Y, Tan JH, Adam M, Gertych A, et al. A deep convolutional neural network model to classify heartbeats. Comput Biol Med. 2017 Oct 1;89:389–96. pmid:28869899
- 15. Biel L, Pettersson O, Philipson L, Wide P. ECG analysis: a new approach in human identification. IEEE Transactions on Instrumentation and Measurement. 2001 Jun;50(3):808–12.
- 16. El_Rahman SA. Multimodal biometric systems based on different fusion levels of ECG and fingerprint using different classifiers. Soft Comput. 2020 Aug 1;24(16):12599–632.
- 17. El_Rahman SA. Biometric human recognition system based on ECG. Multimed Tools Appl. 2019 Jul 1;78(13):17555–72.
- 18. Singh YN. Individual Identification Using Linear Projection of Heartbeat Features. Applied Computational Intelligence and Soft Computing. 2014 Aug 10;2014:e602813.
- 19. Chatterjee I. Patenting Machine-learning: Review and Discussions. International Journal of Modern Research. 2021 Aug 20;1(1):15–21.
- 20. Safdarian N, Jafarnia Dabanloo N, Attarodi G. A New Pattern Recognition Method for Detection and Localization of Myocardial Infarction Using T-Wave Integral and Total Integral as Extracted Features from One Cycle of ECG Signal. Journal of Biomedical Science and Engineering. 2014 Jan 1;07:818–24.
- 21. Alay N, Al-Baity H. Deep Learning Approach for Multimodal Biometric Recognition System Based on Fusion of Iris, Face, and Finger Vein Traits. Sensors. 2020 Oct 6;20. pmid:32992524
- 22. Kumar R, Dhiman G. A Comparative Study of Fuzzy Optimization through Fuzzy Number. International Journal of Modern Research. 2021 Aug 20;1(1):1–14.
- 23. Vaishnav PK, Sharma S, Sharma P. Analytical Review Analysis for Screening COVID-19 Disease. International Journal of Modern Research. 2021 Aug 20;1(1):22–9.
- 24. Yudistira N, Kurita T. Correlation Net: Spatiotemporal multimodal deep learning for action recognition. Signal Processing: Image Communication. 2020 Mar;82:115731.
- 25. Dale MP, Joshi MA, Sahu MK. DCT feature based fingerprint recognition. In: 2007 International Conference on Intelligent and Advanced Systems. 2007. p. 611–5.
- 26. Gupta K, Walia G. Novel approach for multimodal feature fusion to generate cancelable biometric. The Visual Computer. 2020 Jun 13.
- 27. Tico M, Immonen E, Ramo P, Kuosmanen P, Saarinen J. Fingerprint recognition using wavelet features. In: ISCAS 2001 The 2001 IEEE International Symposium on Circuits and Systems (Cat No01CH37196). 2001. p. 21–4 vol. 2.
- 28. Mote MP. Multimodal Biometric system using Gabor Filter. 2012.
- 29. Sim T, Zhang S, Janakiraman R, Kumar S. Continuous verification using multimodal biometrics. IEEE Trans Pattern Anal Mach Intell. 2007 Apr;29(4):687–700. pmid:17299225
- 30. Song Q, Du DQ, Yang L, Meng GJ, Mao XF. Drop Fingerprint Recognition Method Based on Cluster Analysis and BP Neural Network. Applied Mechanics and Materials. 2014;543–547:2099–102.
- 31. Nawal M, Purohit GN. ECG Based Human Authentication: A Review. International Journal of Emerging Engineering Research and Technology.: 8.
- 32. Singh YN, Gupta P. ECG to Individual Identification. In: 2008 IEEE Second International Conference on Biometrics: Theory, Applications and Systems. 2008. p. 1–8.
- 33. Liu B, Liu J, Wang G, Huang K, Li F, Zheng Y, et al. A novel electrocardiogram parameterization algorithm and its application in myocardial infarction detection. Comput Biol Med. 2015 Jun;61:178–84. pmid:25201457
- 34. Aizi K, Ouslim M. Score level fusion in multi-biometric identification based on zones of interest. Journal of King Saud University—Computer and Information Sciences. 2022 Jan 1;34(1):1498–509.
- 35. Hamd MH, Rasool RA. Score Level Fusion Technique for Human Identification. IOP Conf Ser: Mater Sci Eng. 2020 Nov;990(1):012021.
- 36. Truong T, Graf J, Yanushkevich S. Hybrid Score- and Rank-level Fusion for Person Identification using Face and ECG Data. In: 2019 Eighth International Conference on Emerging Security Technologies (EST) [Internet]. 2019 [cited 2023 Feb 11]. p. 1–6. Available from: http://arxiv.org/abs/2008.03353
- 37. Hong L, Jain A, Pankanti S. Can Multibiometrics Improve Performance? Proc AutoID. 2000 Jan 8;2.
- 38. Jain A, Nandakumar K, Ross A. Score normalization in multimodal biometric systems. Pattern Recognition. 2005 Dec 1;38(12):2270–85.
- 39. Dhiman G. ESA: a hybrid bio-inspired metaheuristic optimization approach for engineering problems. Eng with Comput. 2021 Jan 1;37(1):323–53.
- 40. Dhiman G, Oliva D, Kaur A, Singh KK, Vimal S, Sharma A, et al. BEPO: A novel binary emperor penguin optimizer for automatic feature selection. Knowledge-Based Systems. 2021 Jan 9;211:106560.
- 41. Kittler J, Hatef M, Duin RPW, Matas J. On combining classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence. 1998 Mar;20(3):226–39.
- 42. Pouryayevali S. ECG Biometrics: New Algorithm and Multimodal Biometric System [Internet] [Thesis]. 2015 [cited 2021 May 11]. Available from: https://tspace.library.utoronto.ca/handle/1807/70509
- 43. Del Espiritu J, Rolluqui G, Gustilo RC. Neural network based partial fingerprint recognition as support for forensics. In: 2015 International Conference on Humanoid, Nanotechnology, Information Technology,Communication and Control, Environment and Management (HNICEM). 2015. p. 1–5.
- 44. Gupta K, Walia GS, Sharma K. Quality based adaptive score fusion approach for multimodal biometric system. Appl Intell. 2020 Apr 1;50(4):1086–99.
- 45. Hossain MS, Balagani KS, Phoha VV. Effectiveness of symmetric rejection for a secure and user convenient multistage biometric system. Pattern Analysis and Applications. 2021 Feb;24(1):49–60.
- 46.
Kant C, Chaudhary S. A Multimodal Biometric System Based on Finger Knuckle Print, Fingerprint, and Palmprint Traits. In: Sharma MK, Dhaka VS, Perumal T, Dey N, Tavares JMRS, editors. Innovations in Computational Intelligence and Computer Vision. Singapore: Springer; 2021. p. 182–92. (Advances in Intelligent Systems and Computing).
- 47.
Arora S, Bhatia MPS, Kukreja H. A Multimodal Biometric System for Secure User Identification Based on Deep Learning. In: Yang XS, Sherratt RS, Dey N, Joshi A, editors. Proceedings of Fifth International Congress on Information and Communication Technology. Singapore: Springer; 2021. p. 95–103. (Advances in Intelligent Systems and Computing).
- 48. Dehghani M, Montazeri Z, Dehghani A, Ramirez-Mendoza RA, Samet H, Guerrero J, et al. MLO: Multi Leader Optimizer. International Journal of Intelligent Engineering and Systems. 2020 Aug 30;13:364–73.
- 49. Zokaee S, Faez K. Human Identification Based on ECG and Palmprint. International Journal of Electrical and Computer Engineering (IJECE). 2012 Feb 21;2.
- 50. Fatemian SZ. A Wavelet-based Approach to Electrocardiogram (ECG) and Phonocardiogram (PCG) Subject Recognition [Internet] [Thesis]. 2010 [cited 2021 May 11]. Available from: https://tspace.library.utoronto.ca/handle/1807/18293
- 51. BE M, Appaji A, Thriveni J, K R V, Patnaik L. Multimodal Biometric Authentication using ECG and Fingerprint. International Journal of Computer Applications. 2015 Feb 18;111:33–9.
- 52. Raju AS, Udayashankara V. Database Evaluation of ECG Fingerprint and Face Multimodal Biometric System. International Conference on Signal, Image Processing Communication and Automation (ICSIPCA). 2017;207–15.
- 53. Singh YN, Singh SK, Gupta P. Fusion of electrocardiogram with unobtrusive biometrics: An efficient individual authentication system. Pattern Recognition Letters. 2012 Oct 15;33(14):1932–41.
- 54. Moody GB, Mark RG. The impact of the MIT-BIH Arrhythmia Database. IEEE Engineering in Medicine and Biology Magazine. 2001 May;20(3):45–50. pmid:11446209
- 55. Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Commun ACM. 2017 May 24;60(6):84–90.
- 56. Hammad M, Wang K. Parallel score fusion of ECG and fingerprint for human authentication based on convolution neural network. Computers & Security. 2019 Mar 1;81:107–22.
- 57. Hammad M, Liu Y, Wang K. Multimodal Biometric Authentication Systems Using Convolution Neural Network Based on Different Level Fusion of ECG and Fingerprint. IEEE Access. 2019;7:26527–42.
- 58. Pan J, Tompkins WJ. A Real-Time QRS Detection Algorithm. IEEE Transactions on Biomedical Engineering. 1985 Mar;BME-32(3):230–6. pmid:3997178
- 59. Simonyan K, Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition [Internet]. arXiv; 2015 [cited 2023 Sep 7]. Available from: http://arxiv.org/abs/1409.1556
- 60. Patel K, Li K, Tao K, Wang Q, Bansal A, Rastogi A, et al. A comparative study on polyp classification using convolutional neural networks. PLOS ONE. 2020 Jul 30;15(7):e0236452. pmid:32730279