Case study of 3D fingerprints applications

Human fingers are 3D objects. More information will be provided if three dimensional (3D) fingerprints are available compared with two dimensional (2D) fingerprints. Thus, this paper firstly collected 3D finger point cloud data by Structured-light Illumination method. Additional features from 3D fingerprint images are then studied and extracted. The applications of these features are finally discussed. A series of experiments are conducted to demonstrate the helpfulness of 3D information to fingerprint recognition. Results show that a quick alignment can be easily implemented under the guidance of 3D finger shape feature even though this feature does not work for fingerprint recognition directly. The newly defined distinctive 3D shape ridge feature can be used for personal authentication with Equal Error Rate (EER) of ~8.3%. Also, it is helpful to remove false core point. Furthermore, a promising of EER ~1.3% is realized by combining this feature with 2D features for fingerprint recognition which indicates the prospect of 3D fingerprint recognition.


Introduction
As one of the most widely used biometrics, fingerprints have been investigated for more than a century [1]. Meanwhile, with the rapid development of fingerprint acquisition devices and the advent of advanced fingerprint recognition algorithms, effective Automated Fingerprint Recognition Systems (AFRSs) are available in the market. However, they are almost based on 2D fingerprint features, even though the fact is that human fingers are 3D objects. Distortions and deformations will be introduced, while 3D information will be loss when 2D fingerprint images are used, which degrades the performance of AFRSs.
3D fingerprints come into researchers' view in recent years with the expansion of acquisition technology [2][3][4][5][6][7][8][9][10][11][12][13][14]. Most of the work focused on the acquisition and preprocessing of 3D fingerprints, or at the most expanded two dimensional fingerprint features into 3D space but no recognition results were given [2][3][4][5][6][7][8][9][10]. Until 2015, researchers from Biometric Research Centre, the Hong Kong Polytechnic University [13,14] began to investigate the utility of 3D fingerprint features and reported some experimental results of user authentication using the acquired biometric information. In [13], new low level of fingerprint features were proposed and used as additional features to fingerprint recognition. In [14], the authors reported the result of 3D minutiae matching, as well as the matching result of 3D curvature features. In a1111111111 a1111111111 a1111111111 a1111111111 a1111111111 both of those studies, low Equal Error Rate (EER) (from 13% to 35%) was obtained when features from 3D fingerprint images were used for recognition. The EER was not high either for 3D minutiae matching (around 10% when unknown users were added). There may be two reasons why those features are not so effective. One is that both of the mentioned 3D fingerprint images were reconstructed by 3D reconstruction techniques. The accuracy of 3D finger shape is degraded by reconstruction algorithms. The other one is that the distinctiveness of features extracted from 3D finger shape is lower than traditional three levels of features on 2D fingerprint images. However, the EER was greatly improved if those features were combined with 2D traditional features extracted from 2D fingerprint images for matching. Thus, there is no doubt that higher accuracy will be achieved if 3D fingerprint images are used for recognition. This paper is motivated by analyzing the advantages and disadvantages of current 3D fingerprint recognition techniques. The contributions of this paper include two parts: (i). More accurate 3D finger shape was obtained by using Structured-light Illumination (SLI) method. The SLI method is widely used as a 3D imaging method for its high accuracy, speed and stability. (ii). Features on 3D fingerprint images were studied thoroughly and detailedly. By analyzing their distinctiveness, 3D fingerprint features can be used for different applications. It can be found that: (1) an EER of~15% can be achieved if the whole 3D finger point cloud data are taken as 3D shape features for fingerprint recognition. However, it is very effective to align two fingerprints in a short time. (2) The proposed distinctive 3D shape ridge feature are suitable for assisting fingerprint recognition and also helpful to remove false core point. Fusion strategy is employed to combine 2D and 3D fingerprint matching results to figure out the effectiveness of improving recognition accuracy by including 3D fingerprint features. An EER of~8.3% can be obtained when only distinctive 3D shape ridge feature is used for fingerprint recognition, while EER of~1.3% is achieved by combing this 3D feature with 2D fingerprint features.
The organization of this paper is as follows. In Section 2, the acquisition and preprocessing of 3D finger point cloud data is introduced. 3D fingerprint features are investigated in Section 3. The corresponding feature extraction and matching algorithms are also given in Section 3. The applications and experimental results are shown in Section 4. Section 5 finally concludes this work and suggests the future research.

3D finger point cloud data acquisition and preprocessing
Currently, there are mainly two ways to generate 3D fingerprints. One is 3D reconstruction technique and the other one is Structured-light scanning [6,15,16]. Generally speaking, the 3D reconstruction technique has the advantage of low cost. However, it has the disadvantage of low accuracy, because it is difficult to find and match corresponding point pairs in two or more images [14,17]. Structured-light imaging has high accuracy as well as a moderate cost [6,15,16]. Considering the requirements of accuracy for biometric authentication, we choose to use structured-light scanning to acquire the finger depth information.
As we all know, structured-light imaging is widely used in 3D imaging for its high accuracy, speed and stability. The structure diagram of the collection device is shown in Fig 1. A projector casts 13 structured-light stripes onto finger surface, the structured-light is modulated by the surface shape, and the modulated stripes are captured by a CCD camera at a constant distance from the projector. The distance from the measured surface to the reference plane can be calculated according to the modulated stripe images and the geometric correlation between the measured surface, projector and CCD camera. The structure diagram of the collection device is shown in Fig 1. Fig 2 illustrates the principle of the adopted structure light imaging technique. In Fig 2, the relative height of point D at spatial position (x, y) on the 3D object surface can be calculated by following Eq (1) [16]. Where the height of the reference surface is defined as 0. P 0 is the wavelength of the projected light on the reference surface, Q 0 is the projecting angle, Q n is the angle between the reference surface and the line which passes through the current point and the CCD center, and ϕ CD is the phase difference between points C and D. Since the phase of point D is equal to the phase of point A on the reference surface, ϕ CD can be calculated as Eq (2).
The depth information of the 3D object surface can be retrieved by using Eq (1) and the phase shifting and unwrapping technique in [16]. Interested readers can refer to [15,16] for more details. With this processing, the relative height of each point h(x, y) could be calculated. The range data of the finger surface can then be obtained. Fig 3 shows an example of finger point cloud data we obtained. In the captured data, the size of the 3D image is 640×480 with about 380 dpi resolution, i.e., there are totally 307,200 cloud points to represent the 3D fingerprint information.
Since it is inevitable to introduce noises (e.g. false estimated depth near the edge of Region of Interest (ROI), missing or misestimated depth in ROI) when data was collected, it is necessary to pre-process the original point cloud data. In this paper, the point cloud data was firstly projected into 2D plane, i.e. a 3D point (x, y, z) corresponds to a point (row, column and intensity) on a figure, as shown in Fig 4(a). Then, the morphological operators (e.g. interpolation of inner missed points, edge corrosion) provided by MATLAB toolbox were adopted to remove edge points and fill in the inner points. As shown in Fig 4(b), inner missed points and edge points are removed when it was compared with Fig 4(a). The area with the largest size in the figure was chosen as the ROI of the finger. Then, Gaussian smoothing algorithm with window size of 5 × 5 pixels was used to process the missed or misestimated depth points in ROI. Simple MAX-MIN rule was finally employed to normalize the depth value into [0, 1], the preprocessed 3D point cloud data of fingerprint is shown in Fig 4(c). The normalized depth value (W.norm) can be calculated: W.norm = (W-min (W)) / (max (W) − min (W)) when the MAX-MIN rule [18] is used. Where W is the set of depth value.

Investigation of 3D fingerprint features
After preprocessing, stable and unique features are expected to be extracted for further application. Through observation (Fig 5), we found that 3D depth information reflects the overall structure of human finger. However, there are many invalid points in the whole 3D finger shape due to the structure of human finger. As shown in Fig 5(b), the horizontal sectional profiles are almost parabola-like curves in different fractions (the real data are labelled in green dashed line and the fitted data are labelled in red solid line in Fig 5(b)). The vertical profiles are also fitted by parabolic equation (red labelled in Fig 5(c)), it can be seen that most of lines can be well fitted by parabola, however, the closer the line near the highest point, the less like the real line to parabola. Thus, we extracted this line in the paper, and defined it as the distinctive 3D shape ridge feature. The detailed method to extract the distinctive 3D shape ridge feature is as follows: i). Fit each horizontal sectional profile by a parabolic equation; ii). Find the maximum value of each fitted line; iii). Connect all of maximum values to form a vertical line, namely the distinctive 3D shape ridge feature. Fig 5(d) shows the distinctive 3D shape ridge feature extracted using the proposed method. The method to extract the distinctive 3D shape ridge feature can be formulated as Eq (3). Where x is the variable of row-coordinate of the image, and y is the variable of column-coordinate of the image. N row × M column is the size of the image. a x , b x , and c x represent the coefficients of the fitting parabola functions.
In addition, we introduced the matching method used in this paper here since it is very simple and classical. Intuitively, the iterative closest point (ICP) algorithm is suitable for solving such matching problem. ICP method [19] is widely used in many 2D image and 3D object recognition systems for matching. In this paper, we slightly modified the ICP method to measure the distances between two sets of points. The flowchart of the algorithm is given in Fig 6.

3D fingerprint applications and the experimental database
As described in detail previously [20,21], a minutia feature in a 2D fingerprint image can be extended to 3D space. (e.g. in 2D: location and orientation, in 3D:, where x, y and z is the spatial coordinates. and is the orientation of the ridge). Thus, fingerprint recognition with higher security can be achieved by matching features in 3D space (e.g. 3D minutia matching [14,22]). This paper tried to discover the additional 3D fingerprint features and their applications to fingerprint recognition. Experiments were implemented in our own established 3D fingerprint database. The database contains 440 samples from 22 volunteers, including 12 males and 10 females between 20 to 40 years old. The 3D fingerprint samples were collected in two separate sessions, and in each session, each finger of a person was taken as one subject and was collected once. i.e. the database contains 440 samples from 220 fingers, 2 pictures for each finger (part of the database can be downloaded from S1 Database.zip). The average time interval between the two sessions was about two weeks. It is noted that the data were collected under the guidance of the collector to ensure the frontal view of each finger was taken. Three cases were summarized and studied in the paper, as shown in the following subsections.

Case 1: Coarse alignment by 3D finger point cloud data
Coarse alignment is a crucial step in fingerprint recognition, especially for fingerprint matching. In 2D fingerprint images (Fig 7(a)), finger skin is forced to plane. Level-1 fingerprint features (e.g. orientation map, core point) are usually extracted at first for coarse alignment in fingerprint recognition [23][24][25][26][27].
Obviously, it takes some time to extract fingerprint features, and the accuracy is also affected by the results of feature extraction. While in 3D fingerprint images, as shown in Fig  7(b), two images can be aligned quickly by aligning their corresponding finger shapes. This is because the center part of the human finger is higher than the side part in 3D space from the frontal view of the finger. By making full use of this character, 3D fingerprint images can be quickly coarse aligned before matching in fingerprint recognition. Fig 8 shows an example of coarse alignment result according to their corresponding 3D finger point cloud data in our database. We also tried to match this feature directly to see its discriminative performance. This feature was matched by 3D ICP method introduced in Section 3 since it is 3D point cloud data. The mean distance between matched pairs (Mdist) and the percentage of matched points (Pm = matched pair number/overall pairs to be matched) are taken as the match score. The EERs were obtained from 220 genuine scores and 48,180 imposter scores (generated from 220 fingers, 2 pictures of each finger). Fig 9 shows the Receiver Operating Characteristic Curves (ROCs) of different match score indexes evaluated on the established database. It can be seen from the result that the EERs are all very high (best one: 15%), which shows the identifiability of this feature is not very high. However, it is no doubt that this feature can be used for coarse alignment.
Case 2: Core point detection under the guidance of distinctive 3D shape ridge feature It is interesting to find that core points in fingerprint images almost locate at the center part of the finger with highest curvature by observing human fingers. Thus, the position of the core point will be more likely located near the distinctive 3D shape ridge feature we defined in this paper. By making full use of the distinctive 3D shape ridge feature, false core points which are far away from the distinctive 3D shape ridge feature can be easily removed. Fig 10(a) shows an example of core point detection results by Poincare Index introduced in [1]. It can be seen that many false core points are detected by using this method. We then used the extracted distinctive 3D shape ridge feature (Fig 10(b)) to guide the location of the true core point. The final result is given in Fig 10(c), which shows the helpfulness of the usage of distinctive 3D shape ridge feature to true core point detection. The statistical result of true core point detection  from original core point set detected by Poincare Index in our established database with 220 samples (each finger selects one picture) is shown in Fig 11. From the result, it can be seen that most of the false core points are removed from original core point set even though not only the true core point is detected each time, which fully demonstrates the effectiveness of using the distinctive 3D shape ridge feature to true core point detection.
Case 3: User authentication using distinctive 3D shape ridge feature As mentioned in Section 3, the defined distinctive 3D shape ridge feature is in fact a representative vertical section profile of 3D finger point cloud data. The dimension of this feature is reduced from 3 to 2. Thus, this feature can be matched by 2D ICP. In the study, Mdist and Pm mentioned in subsection 4.2 are taken as the match score, too. Fig 12 shows the ROCs of different match score indexes evaluated on the established database. From the results, we can see that the index of mean distance between matched pairs is better than the percentage of matched points. Best EER of around 8.3% can be obtained when the distinctive 3D shape ridge feature was used for matching by simple ICP algorithm, which demonstrates this feature is helpful to distinguish different fingers even though it is not as accurate as other higher level fingerprint features.
Since both 2D fingerprint images and 3D finger point cloud data of the same finger are provided simultaneously by 3D fingerprints, we tried to study whether improved performance can be achieved by combining 2D and 3D fingerprint features or not. Here, for 2D fingerprint features, the Distal Interphalangeal Crease based (DIP-based) feature was selected due to its effectiveness on our database compared with minutiae and Scale Invariant Feature Transformation (SIFT) features [28]. The angular distance was used as the match scores (MS 2D ). DIP is defined as the only permanent flexion crease which is located between medial and distal segments of finger except thumb (between proximal and distal segments), and the DIP-based feature was formed by coding the region using competitive coding scheme introduced in [29]. The detailed DIP feature extraction and matching algorithms can be obtained in [28] which was attached as supporting information. Meanwhile, the mean distance between matched pairs be matching the defined distinctive 3D shape ridge feature was taken as the match score (MS 3D ). A simple adaptive weighted sum rule is used to combine the 2D and 3D matching scores. The combined score can be expressed as: The weight w is adaptively tuned to provide the best verification results at step length of 0.01. Fig 13 shows the ROCs achieved by matching DIP-based feature and the distinctive 3D shape ridge feature separately, as well as their combination. It is notable that the DIP-based feature clearly outperforms curve-skeleton in terms of accuracy. However, the best result is achieved when combining these two features where an EER of 1.3% is achieved. This experiment fully demonstrates that higher accuracy can be achieved if adding the distinctive 3D Case study of 3D fingerprints applications shape ridge feature. Thus, we concluded that this feature, at least, can be taken as an additional feature to current fingerprint feature set for user authentication.
Comparisons with the art-of-the-state 3D fingerprint matching performance Due to the lack of 3D fingerprint database, few studies reported the matching performance using 3D fingerprint images. Currently, the art-of-the-state 3D fingerprint recognition results were given in [13] and [14]. It is noted that different features, as well as different feature extraction and matching methods were adopted in their papers. What's more, their results were obtained based on their own established database. In this paper, we summarized and compared the matching results using different features extracted from our own established 3D fingerprint database. The fusion results of these additional features with DIP based feature extracted from 2D fingerprint images were also given. Table 1 shows the EERs obtained by matching features mentioned in [13], [14] and this paper.
From the results, it can be seen that the distinctiveness of all these features are poor. They are not suitable to be used alone to identify one's identity. However, the accuracy is greatly improved when they are combined with 2D fingerprint features. Thus, there is no doubt that  improved recognition accuracy can be achieved when 2D fingerprint features are combined with these additional features extracted from 3D fingerprint images. From Table 1, it also can be seen that the result of matching distinctive 3D shape ridge feature outperforms other features used in [13,14], which further demonstrates the usefulness of the proposed distinctive 3D shape ridge feature.

Conclusions
This paper had investigated the possible applications of 3D fingerprints. Thanks to the availability of 3D fingerprint images, more features can be extracted. The 3D finger shape features and the newly defined distinctive 3D shape ridge feature were then studied and extracted, the applications of these features were finally discussed as well. Results showed that coarse alignment could be easily achieved when 3D finger point cloud data were available. EER of~8.3% can be obtained when the distinctive 3D shape ridge feature was used for personal authentication. Also, it was discovered to be helpful to remove false core point. Furthermore, promising EER was realized by combining this feature with 2D features for fingerprint recognition which indicated the prospect of 3D fingerprint recognition. In the paper, just simple feature extraction and matching methods were used. We believe that higher accuracy can be achieved if better feature extraction and matching methods are adopted. Thus, discovering more effective 3D feature extraction and matching methods (e.g. the 3D Gabor features, SSLBP features, improved SVM classifiers, sparse representation scheme, deep learning scheme etc. introduced in [30][31][32][33][34][35][36][37][38][39][40]) are our further work. Meanwhile, our future work includes exploring the relationship between different levels of fingerprint features and proposing more powerful fusion strategy. Information security is of great importance to current electronic age. 3D fingerprints provides more complicated features and increases dimensions of 2D fingerprint features which will be more suitable to embedded into the schemes of preventing attacks to current devices (e.g. 3D printer, multi-cloud-server) [41][42][43][44][45][46]. These will be our future direction to extend 3D fingerprints applications.
Supporting information S1 Database. Part of the database used in the paper. (ZIP)