https://orcid.org/0000-0002-2849-0342
Figures
Abstract
Distinguishing different time series, which is determinant or stochastic, is an important task in signal processing. In this work, a correlation measure constructs Correlation Fuzzy Entropy (CFE) to discriminate Chaos and stochastic series. It can be employed to distinguish chaotic signals from ARIMA series with different noises. With specific embedding dimensions, we implemented the CFE features by analyzing two available online signature databases MCYT-100 and SVC2004. The accurate rates of the CFE-based models exceed 99.3%.
Citation: Wu J, Wan Q, Zhang Z, Xu J, Cheng W, Chen D, et al. (2024) Correlation Fuzzy measure of multivariate time series for signature recognition. PLoS ONE 19(10): e0309262. https://doi.org/10.1371/journal.pone.0309262
Editor: Kshatresh Dutta Dubey, Shiv Nadar University, INDIA
Received: June 4, 2024; Accepted: August 8, 2024; Published: October 7, 2024
Copyright: © 2024 Wu et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: Jun Wu was supported by the Natural Science Foundation of Hubei Province (Grant No. 2022CFB959), the Educational Commission of Hubei Province of China (Grant No. Q20221802), the Hubei Key Laboratory of Applied Mathematics (Grant No. HBAM202105) and the Doctoral Fund of Hubei University of Automotive Technology (Grant No. BK202114).Specify the role(s) played. The funder is the first and the corresponding author, who provided main ideas and took most experiments.
Competing interests: The authors have declared that no competing interests exist.
1 Introduction
The tasks of time series classification mainly include two aspects that assessment the complexity and distinguish the noise. Over recent years, Time Series Classification (TSC) have made significant improvement over the previous state of the art. A new set of TSC algorithms have been developed which involves building predictive models for a discrete target variable from ordered, real valued attributes. Although a great success of algorithms has been achieved in time series analysis, fewer studies focus on the multivariate TSC (MTSC) problems [1].
Amounts of advanced methods for TSC usually consist of the following steps: data acquisition, pre-processing, feature extraction, measures-based machine learning and the final decision. Algorithms for MTSC can be categorized in similar with TSC on whether they are based on: distance measures, shapelet, histograms over a dictionary, interval summarizing, or deep learning/neural networks [2, 3].
It seems a common choice of setting Dynamic Time Warping (DTW) as an initial measure for MTSC [4]. Shokoohi-Yekta et al. [5] proposed an adaptive solution where the decision in about between independent and dependent dynamic time warping. They present a modified idea of selecting which distance to use is based on a threshold found from the full factorial experiment design. Xia et al. [6] discuss the DTW with signature curve constraint method that is used to select discriminative features among candidates for reducing the influences of the fluctuations based on training data. However, special method still should be designed as the DTW is valid but not accurate for the real time series.
While the distance measures have an up boundary of accurate rate in MTSC, more work should be done on feature analysis and classifier design. The Random Convolutional Kernel Transform [7] uses a simplistic convolution kernel to return the maximum value and proportion of positive values (PPV) as a novel feature, resulting feature maps in conjunction with a linear classifier (ridge regression or logistic regression). The time series tree uses a large number of random tree structure, comparing selected attributes at structured nodes and performing no pruning. Middlehurst et al contribute the Canonical Interval Forest [8] which is an ensemble of time series tree [9] classifiers built using the Canonical Time-Series Characteristics, Catch22 [10] features and simple summary statistics extracted from phase dependent intervals. Then, it is extended for MTSC by a random dimension choosing strategy. However, the tree introduces a novel tie breaking measure in the form of entrance gain.
Most features in time dependent methods are local properties which computed with neighborhood data. For this reason, enough length of time series should be valid in training classifiers. But in practice, the online data we acquired, that should be charge which status is, often are short series. It is a challenge work that methods for short MTSC need to be find.
2 Related works
In recent years, complex networks show its benefit on probing the determinism of time series. For the concept of networks derives from the phase space perform well for short time series. The resulting network is still faulted by the noise and difficult to choose appropriate embedding dimensions and time lags. The resulting network inherits several properties associated with temporal evolution of time series in the network structure, and even simple measures from network science, such as mean degree, can be used to characterize complex system [11].
Bandt-Pompe symbolization method [12] transforms series into ordinal patterns for investigating system dynamics. A great success of ordinal network has extended the applicability of ordinal network [13–15] and developed the method [16, 17]. On one hand, most studies focus on the topological features. On the other hand, the information in transition probabilities between nodes remains attention until the global node entropy (GNE) is proposed [18].
To solve the problems of short and noisy recordings in physiological signals, Pincus presented a biased statistic approximate entropy (ApEn) as a measure of complexity. This measure is applicable to noisy while lacks relative consistency and requires series from medium-sized datasets [19]. To solve problems of the bias caused by self-matching, Richman and Moorman [20] developed another statistic, sample entropy (SampEn), by investigating the mechanism responsible for the bias. SampEn displays relative consistency and less dependence on data length. However, SampEn has the same function of similarity definition of vectors with ApEn. The Heaviside function exists inherent flaws when small parameters are involved. For this reason, W. Chen [21] investigates the concept of Zadeh’s fuzzy sets and derives fuzzy entropy (FE). Zhang ZL et al [22] stress that the fuzzy ordinal method could distinguish chaos from noise effectively. By improving the fuzzy entropy (FE), they design most recently a new related family of statistics, Fuzzy Permutation Entropy (FPE), for detecting determinism in time series [23].
In this work, we try to improve the method with correlation measure method for MTSC. As the Correlation distance is sensitive to the change of segment series, and tolerate local difference in neighbor position, a Correlation Fuzzy Entropy (CFE) is calculated to distinguish multi-dimensions chaos, random walking, discrete chaos from ARIMA and White Gaussian Noise (WGN). It should be emphasized that the CFE recognize determinant signal with low sensitivity to noise. Finally, we implement CFE on 2 online signature datasets for some natural application in the last part. CFE features conduce excellent accurate rate in cross validation with classifiers like Trees and KNN in experiment.
3 Methodology
A. A measure method using discrete Correlation distance
It is generally known that local characters of series, such as stationary points, curvature, histogram and power, are sensitive with noise signal. For this reason, a robust measure can be considered as a distance between vectors should be calculated on subsequence of series. For example, DTW and ordinal distance perform well in univariate time series. The DTW distance mainly adopt the nearest neighbors for two time series with different length, it takes suitable node in pair into account. This makes two series have better uniformity. The ordinal distance use ranks of the series to reconstruct the series, which can decrease the influence of noise signal. To measure similarity multi-dimensions time series, we also should consider these two aspects.
As the general type of weighted dimension is
(1)
The correlation distance is
(2)
As ‖⋅‖ is a norm of φ and in a Banach space, it satisfies
(4)
As we also have |φ(x)| = 1, this deduces
(5)
Considering F1 and F2 have same distribution, the norm-equivalence theorem ensures
(6)
Where M(t) is decided by modulus of the finite segment in time series.
Let X, Y are two continuous random variances, Cov(X,Y) is their covariance, the correlation distance is calculated as follows
(7)
where σi is the Standard Deviation.
We can use L2 norm to control the x-dimension measure under Correlation distance. We define a discrete Correlation distance of MTS with this note.
Let T(n, x) is a discrete MTS, the discrete Correlation distance can be calculated as
(8)
where Tm,τ means the m-th segment with length of τ. ‖⋅‖2,x denotes modulus of multi-dimension.
Although the Correlation coefficient is a quick and excellent measure of similarity between different time series recommended by Theorem 1, we posit that it is still existing some disadvantage on dealing with outlier values. To eliminate this problem, we chose Fuzzy Entropy regarding the global similarity to improve the correlation distance. Moreover, it overcomes effects on normal value caused by outliers in noise.
B. Correlation Fuzzy Entropy of multivariate time series
Fuzzy entropy (FE) is a measure of global level of random for topological data. After giving distance between nodes, a Correlation Fuzzy entropy (CFE) can be computes as follow steps in correspondence with Fig 1.
Step 1 Calculate the similarity degree of each pair of segments Ti,τ and Tj,τ using a fuzzy function
, as
(9)
Where m = 2, r = 0.15. There is a general assumption here that the variance of a sequence is 1, but the actual sequence does not satisfy this property. Therefore, in actual calculations, we usually use the following formula, as
(10)
Step 2 for a series with length of N, a global quantity is defined, that is
(11)
Step 3 given ϕτ and ϕτ+1, the correlation fuzzy entropy is calculated from
(12)
4 Numerical tests
In this section, we firstly distinguish different types of multi-dimension signals, including WGN, Chaos, Discrete Chaos and random walk (RW) series, based on our FPE method. And secondly, we show the robust CFE for discriminating stochastic signals from chaos with several levels of additive noise. The test carries on a work station with MATLAB 2021b.
A. Distinguish Chaos and stochastic time series
Here, WGN is a fast standard normal distribution created by pseudo-random generating algorithm [24]. The Rössler system provides continuous chaotic time series with a time lag of Δt = 1.
Besides, as the step size varying according to the standard normal distribution, RW is a Gaussian random walk. In this experiment, The Chirikov standard map,
(14)
is solved with forth/fifth Runge–Kutta algorithm. The p and x denote momentum and coordinate respectively.
We estimate CFE with N = 400 time series and embedding dimension τ∈{11, 12, …,20}. Calculation of CFE values repeat on 30 independent sequences for each τ. We also demonstrate the mean and standard deviation as a function of τ which is shown in Fig 2.
Mean of (a) FE [25], (b) FPE [22], (c) CFE values for WGN, RW, continuous chaos (Rössler system), and discrete chaos (Chirikov map).
Fig 2(A) reveals that FE is a valid measure for recognizing some signals. FE values of Discrete Chaos signal is obviously same with WGN signals as they have similar curves along to the segment lengths. FE of Chaos and RW seem to have common level, which is different from the first two signals. Besides, FE of all signals rise to the same level when τ increase to 19.
A complex situation actions in FPE (see Fig 2B), FPE values of Chaos and DisChaos across at some points, τ = 11, 17. Although FPE is not as stable as FE for signals, we stress that orbits of Chaos and Dischaos oscillates drastically as τ increases. It means that, on one hand, Chaos and Dischaos are similar when the continuous signal transferred to an ordinal style. On the other hand, FPE of WGN and RW have less standard deviation, which can be contributed to distinguish deterministic signals from stochastic ones.
Mean values of CFE are shown in Fig 2C. Curves increase separately as τ increases. CFE can recognize all 4 signals when τ is larger than 11. We can find when τ have a large value over 18, FE and FPE perform weakly on distinguishing different signals. Because longer segments in different distribution may have similar correlation degree, distance methods of FE and FPE are difficult to find the difference.
For details in 30 experiments, we chose mean and standard deviation of FE, FPE and CFE to demonstrate the results (see Fig 3). Error bars demonstrate the mean and Std. of entropy values. The curves in different figures show that FE is still robust for two groups signals on overall embedding dimensions. It is worse for FPE as the ranges overlap each other severely. The CFE of Chaos and RW are not distinguishable as shown in Fig 3. Hence, CFE recognize DisChaos from other signals. Furthermore, CFE classifies WGN from Chaos and RW signals. According to the analysis of Fig 3, we stress that fluctuation degree of FPE over all embedding dimensions maybe contribute to classify Chaos and other signals. Std of FE, FPE and CFE have been calculated for the over view on τ ∈{11, 12, …, 20} by 10 times.
Error bars of (a) FE, (b) FPE, (c) CFE values for WGN, RW, continuous chaos (Rössler system), and discrete chaos (Chirikov map).
Fig 4 demonstrates that stdFe still could discriminate DisChaos and WGN from Chaos and RW signals. Especially, stdfpe of Chaos and Dischaos signal exhibit gap to WGN and RW signals. This is corresponding to the above guess. For CFE, the mixture of RW and Chaos is more visible. In summary, FE performs well to recognize the two groups signals in phase of values and phase of Std overall embedding dimensions. FPE verifies itself in the phase of global spread, which can be used to classify deterministic signals from stochastic ones. StdCFE has excellent ability in recognizing WGN and Dischaos while RW and Chaos signals are mixture. We can classify these two signals using mean of CFE in previous part. We stress that CFE can be adapt to discriminate random signals from deterministic ones.
Standard deviation of (a) FE, (b) FPE, (c) CFE.
B. Distinguish between Chaos and stochastic time series with noise
For methods of discriminating deterministic signals from stochastic ones, it is common that they are sensitive to noise. As we have mentioned above, most real-world data contain observational noise which often has inverse effects on recognizing signals. It is more difficult to overcome this problem for short series as same as in multi-dimensions. Here, we will show that CFE is robust to noise in a continuously Chaos series. Then, the CFE method can be further applied in real time signal recognition tasks.
When we test signals with noise, the range of τ is assigned between 10 and 20. Similar with Sec. 3 A, CFE is used to distinguish 6 groups of 30 independent series of N = 400. ARMA and the Rössler system which are mixed with different levels of noise, for which distributed with meanμ = 0 and variance σ2. The 2d ARMA series have parameters that constant = [1 1], coefficients = [0.2–0.1; -0.4 0.3] and σ = [0.2 0.3; 0.3 0.7]. The sequences of Chaos are generated with SNR = 10, 3, 1, and 0.5 dB, respectively.
Fig 5 demonstrates the influence caused by different noise. The mean and standard deviation of CFE is shown in Fig 5(A). Moreover, Fig 5(B) exhibits StdCFE of different signals. As shown in Fig 5(A), as the strength of noise increases, the CFE values of signals is closer to the stochastic signal respectively. We can find that the change of CFE is negatively correlated with the variance of external noise. It is obvious that there is visible interval between signals with noise and ARIMA even for SNR = 1dB. The same situation actions for standard deviation of CFE of all signals in Fig 5(B).
(a) Error bars of CFE values and (b) Standard deviation of CFE overall segments for ARIMA, continuous chaos (Rössler system) and signal with different noise.
The stdCFE overall τ lie at corresponding locations. As the strength of noise increases, the oscillation of stdCFE values decreases. It can be attributed to that the stochastic segments have same amplitude characters. This result verifies that the CFE method is robust to discriminate determinant signals and stochastic ones.
5 Application
In this section, CFE is employed to analyze 2 publicly available online signature databases MCYT-100 and SVC2004 Task2 and. SVC2004 database includes 1600 signatures from 40 users. There are 800 genuine signatures and 800 skilled forgeries collected with the WACOM graphic tablet. The data information contains the x-coordinate and y-coordinate, pressure, azimuth angle, inclination angle, timestamp and pen up/down status [26]. MCYT-100 database includes 5000 Western signatures from 100 users, of which has 25 genuine signatures and 25 skilled forgeries for each user. The information that x-coordinate, y-coordinate, pressure, azimuth, and altitude of data are collected at 100 Hz [27]. Some samples are displayed in Fig 6.
(a) (b) Genuine signature and forged signature in MCYT-100, (c) (d) Genuine signature and forged signature in SVC2004.
In a realistic scenario, we have to distinguish genuine signatures from the forgeries. But, we often have not much access to forgery signatures. For this reason, we should train the model with negative samples which include 2 parts. One of them is a group of some forgeries and the other one is the part of genuine signatures from other uses. We take samples of two databases as shown in Table 1. For each user, we randomly chose equal number of other real signatures as NRU. The signatures of other users provide different determinant and stochastic information in the model.
Generally, training signatures set 2 modes that using single samples or pairs samples. Here, we only discuss CFE on information of signature locus. It means that samples are two dimensions time series in single or in pairs. Then, CFE values are calculated as the inputs at τ = {13, 14, 15, 16, 17, 18 19, 20}, corresponding to the result of Section 3. The Classification Toolbox of Matlab R2021b is used in this experiment.
A. Single samples in classification
Data in single samples mode includes 25 positive samples and 50 negative samples for each user in MCYT100 dataset. 20 positive samples and 40 negative samples are selected from SVC2004 for users. We first compare CFE values of real and forged signatures in Fig 7. For each user, a box-and-whisker plot includes mean and spread range for different segment.
(a) signature samples in MCYT (b) signature samples in SVC2004.
The boxes of real signature and forgeries in Fig 7 distribute with obvious gap. This is evidence that CFE values are available in distinguishing signatures as features. In this experiment, KNN, Linear Discriminant, Quadratic Discriminant, Logistic Regression, Naïve Bayes, SVM and Fine Trees models test the CFE features by 5-folds cross validation. The average Error Rate of 10 replication are shown in Table 2.
Fine Trees classifier performs well on MCYT dataset. It has the least FAR 5.8% and EER 5.74%. SVM takes the best EER on SVC. We can find that the KNN model and Fine Trees model could distinguish signatures with lower EER both on MCYT and SVC. Naïve Bayes model performs well on MCYT dataset with error of 8.13%. Thus, the CFE features is valid in solving complex task that try to classify multi-dimension time series.
B. Pairs samples in classification
When chose single samples in our test, there are only 75 samples in total. The good performance of classifier may just take advantage of the small set. In order to eliminate this factor, we chose second method that pairing samples to acquire more data. For MCYT100, 25 positive signatures are combained in pairs for 300 positive samples. Then, 25 random signatures and all forgeries are selected for 1250 negative samples. For SVC2004, there are 190 positive samples and 800 negative samples.
Fig 8 displays positive and negative observations of one result in the replication experiments. In our experiments, when the number of samples becomes lager, the pairs-feature improved most classifiers. KNN classifiers and Fine trees classifier still perform the best outcomes for pairs-samples as shown in Table 3. We can find that SVM and Naïve Bayes model have same level on MCYT with 1.97/13.7 and 11.36/9.15 respectively. Besides, the FAR of KNN is under 10% while the EER is 15.36 on SVC. It is due to that KNN models have a worse result on recognizing positive signatures. Specially, FAR and EER of the Fine trees model have dropped below 1%.
(a) ERR 0.3% Fine trees on 24th user in MCYT (b) ERR 1.5% SVM on 32nd user in SVC2004.
C. Discussion
From above two tests, CFE feature exhibits its effectiveness in KNN, Naïve Bayes SVM and Fine Trees on both single samples and pairs-samples. This means that CFE values over segments represent information included in time series. Signatures of each person have common structure and slight random change when users write them. It is the deterministic writing habit that forms the difference between genuine signatures and forgeries. At the same time, signatures include system noise captured by the devices. CFE conduces to deal with those signals with noise in Section 3, which make it effective in our application.
6 Conclusion
In this paper, a novel measure is presented to analyze multi-time series based on the correlation theory. It improves the Fuzzy Entropy and derives the Correlation Fuzzy entropy. CFE is utilized to distinguish multi-series such as WGN, continuous, or discrete chaotic time series over multiple embedding dimensions. Besides, we have illustrated the behavior of CFE on discriminating ARIMA signals from chaotic series with different levels of noise.
The method exhibits its effectiveness and stableness on distinguishing stochastic signals from determinant signals. Finally, using CFE features, we can verify user’s signatures. It is proved that the CFE is a valuable quantity for analyzing multi time series in engineering.
References
- 1. Goswami A. D.; Bhavekar G. S.; Chafle P. V. Electrocardiogram Signal Classification Using VGGNet: A Neural Network Based Classification Model. International Journal of Information Technology 2023, 15 (1), 119–128.
- 2. Pahuja S.; Veer K.; others. Recent Approaches on Classification and Feature Extraction of EEG Signal: A Review. Robotica 2022, 40 (1), 77–101.
- 3. Bagnall A.; Flynn M.; Large J.; Lines J.; Middlehurst M. On the Usage and Performance of the Hierarchical Vote Collective of Transformation-Based Ensembles Version 1.0 (HIVE-COTE v1.0). AALTD 2020, Ghent, Belgium, September 18, 2020, Revised Selected Papers; Springer-Verlag: Ghent, Belgium, 2020; pp 3–18. https://doi.org/10.1007/978-3-030-65742-0_1.
- 4. He L.; Tan H.; Huang Z.-C. Online Handwritten Signature Verification Based on Association of Curvature and Torsion Feature with Hausdorff Distance. Multimedia Tools and Applications 2019, 78, 19253–19278.
- 5. Shokoohi-Yekta M.; Hu B.; Jin H.; Wang J.; Keogh E. Generalizing DTW to the Multi-Dimensional Case Requires an Adaptive Approach. Data mining and knowledge discovery 2017, 31, 1–31. pmid:29104448
- 6. Xia X.; Song X.; Luan F.; Zheng J.; Chen Z.; Ma X. Discriminative Feature Selection for On-Line Signature Verification. Pattern Recognition 2018, 74, 422–433.
- 7. Dempster A.; Petitjean F.; Webb G. I. ROCKET: Exceptionally Fast and Accurate Time Series Classification Using Random Convolutional Kernels. Data Mining and Knowledge Discovery 2020, 34 (5), 1454–1495.
- 8. Middlehurst M.; Large J.; Bagnall A. The Canonical Interval Forest (CIF) Classifier for Time Series Classification. IEEE 2020, 188–195.
- 9. Deng H.; Runger G.; Tuv E.; Vladimir M. A Time Series Forest for Classification and Feature Extraction. Information Sciences 2013, 239, 142–153.
- 10. Lubba C. H.; Sethi S. S.; Knaute P.; Schultz S. R.; Fulcher B. D.; Jones N. S. Catch22: CAnonical Time-Series CHaracteristics: Selected through Highly Comparative Time-Series Analysis. Data Mining and Knowledge Discovery 2019, 33 (6), 1821–1852.
- 11. Kulp C. W.; Chobot J. M.; Freitas H. R.; Sprechini G. D. Using Ordinal Partition Transition Networks to Analyze ECG Data. Chaos: An Interdisciplinary Journal of Nonlinear Science 2016, 26 (7). pmid:27475074
- 12. Bandt C.; Pompe B. Permutation Entropy: A Natural Complexity Measure for Time Series. Physical review letters 2002, 88 (17), 174102. pmid:12005759
- 13. McCullough M.; Small M.; Stemler T.; Iu H. H.-C. Time Lagged Ordinal Partition Networks for Capturing Dynamics of Continuous Dynamical Systems. Chaos: An Interdisciplinary Journal of Nonlinear Science 2015, 25 (5). pmid:26026313
- 14. McCullough M.; Sakellariou K.; Stemler T.; Small M. Regenerating Time Series from Ordinal Networks. Chaos: An Interdisciplinary Journal of Nonlinear Science 2017, 27 (3). pmid:28364757
- 15. Zhang J.; Zhou J.; Tang M.; Guo H.; Small M.; Zou Y. Constructing Ordinal Partition Transition Networks from Multivariate Time Series. Scientific reports 2017, 7 (1), 7795. pmid:28798326
- 16. Yang G.; Xu D.; Zhang H. Catastrophe Pre-Warning of Multi-Modular Floating Platforms with Ordinal Partition Networks. International Journal of Computational Methods 2020, 17 (10), 2050010.
- 17. Pessa A. A.; Ribeiro H. V. Mapping Images into Ordinal Networks. Physical Review E 2020, 102 (5), 052312. pmid:33327134
- 18. McCullough M.; Small M.; Iu H.; Stemler T. Multiscale Ordinal Network Analysis of Human Cardiac Dynamics. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 2017, 375 (2096), 20160292. pmid:28507237
- 19. Bruhn J.; Röpcke H.; Hoeft A. Approximate Entropy as an Electroencephalographic Measure of Anesthetic Drug Effect during Desflurane Anesthesia. The Journal of the American Society of Anesthesiologists 2000, 92 (3), 715–726. pmid:10719951
- 20. Richman J.; Moorman J. ‘Physiological Time Series Analysis Using Approximate and Sample Entropy’. American Journal of Physiology: Heart and Circulatory cPhysiology 2000, 278, H2039–2049.
- 21. Chen W.; Wang Z.; Xie H.; Yu W. Characterization of Surface EMG Signal Based on Fuzzy Entropy. IEEE Transactions on neural systems and rehabilitation engineering 2007, 15 (2), 266–272. pmid:17601197
- 22. Zhang Z.; Zhang M.; Chen Y.; Xiang Z.; Xu J.; Zhou X. Distinguishing Chaotic from Stochastic Dynamics via the Complexity of Ordinal Patterns. AIP Advances 2021, 11 (4).
- 23. Zhang Z.; Wu J.; Chen Y.; Wang J.; Xu J. Distinguish between Stochastic and Chaotic Signals by a Local Structure-Based Entropy. Entropy 2022, 24 (12), 1752. pmid:36554157
- 24. Marsaglia G.; Tsang W. W. A Fast, Easily Implemented Method for Sampling from Decreasing or Symmetric Unimodal Density Functions. SIAM Journal on scientific and statistical computing 1984, 5 (2), 349–359.
- 25. Chen W.; Zhuang J.; Yu W.; Wang Z. Measuring Complexity Using Fuzzyen, Apen, and Sampen. Medical engineering & physics 2009, 31 (1), 61–68. pmid:18538625
- 26. Yeung D.-Y.; Chang H.; Xiong Y.; George S.; Kashi R.; Matsumoto T.; et al. SVC2004: First International Signature Verification Competition. ICBA 2004, 16–22.
- 27. Ortega-Garcia J.; Fierrez-Aguilar J.; Simon D.; Gonzalez J.; Faundez-Zanuy M.; Espinosa V.; et al. MCYT Baseline Corpus: A Bimodal Biometric Database. IEE Proceedings-Vision, Image and Signal Processing 2003, 150 (6), 395–401.
- 28. Cpałka K.; Zalasiński M.; Rutkowski L. New Method for the On-Line Signature Verification Based on Horizontal Partitioning. Pattern Recognition 2014, 47 (8), 2652–2661.
- 29. Cpałka K.; Zalasiński M. On-Line Signature Verification Using Vertical Signature Partitioning. Expert Systems with Applications 2014, 41 (9), 4170–4180.
- 30. Sharma A.; Sundaram S. An Enhanced Contextual DTW Based System for Online Signature Verification Using Vector Quantization. Pattern Recognition Letters 2016, 84, 22–28.