2.1 Traditional machine learning methods

The most typical traditional machine learning method for handwriting comparison is the KNN algorithm. The algorithm first transforms the image that needs to be compared into a matrix, finds the K neighbors in training set closest to the matrix, and determines the type of the new data based on their main classification. The main classification can be based on different criteria, such as “most numerous” “Euclidean distance” or “weighted distance.”

In the KNN algorithm [9] for handwriting style identification, signature images are usually used as input. The position, angle, spacing, and signature size can cause significant errors(shown in Fig 1). Since the proportion of signature in the image is relatively small (the information used for discrimination is sparse when the image is converted to a matrix), the gray-scale value, font color, and stroke thickness from the original scanned image have an enormous impact on the discrimination results. Therefore, the KNN algorithm requires complex pre-processing of the image: resizing, filtering, and color grading to get a good result. In particular, the KNN algorithm is less efficient when the number of images compared in the training data is large. There are other traditional machine learning algorithms such as SVM [10], LBP [11], and GMM [12] besides KNN.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Typical noise effects of signature images.

For comparison, (a) is a standard signature image, the signature in (b) has rotation, the characters in (c) have a larger interval, the background in (d) has noise due to scanning or photography, the size of the characters in (e) is not the same as the normal size, and of course, some signatures that in many of these situations at the same time. The Chinese characters shown in the signature images read “Mao Huaqing,” which is the name of one of the authors.

https://doi.org/10.1371/journal.pone.0353503.g001

Critical analysis of traditional methods:

• Strengths: KNN and SVM are computationally efficient, interpretable, and work well with limited data.
• Limitations: They require extensive preprocessing (resizing, filtering, color grading); are particularly sensitive to character rotation, variable spacing, and background noise; have limited feature learning capability; and show poor scalability to large datasets.
• Why Insufficient: These factors are common in real-world Chinese handwriting, making traditional methods unreliable for character-level comparison.

2.2 Deep learning methods

Relative to traditional machine learning algorithms, deep neural networks can learn features autonomously and have achieved excellent results in face recognition, object detection, and identification applications. Some neural network models have also been used for handwriting recognition comparison [13,14]. A commonly used unsupervised model for image feature extraction is auto-encoder, which automatically acquires an image’s feature representation to minimize the reconstruction error between encoder and decoder. The deep neural network model also has two deficiencies in the handwriting recognition and comparison task. The first is that each person writes a different character. When the model automatically acquires the feature representation of the picture, it acquires the handwriting information. It learns additional information about the text itself, which is not helpful for handwriting comparison but makes it more difficult for the model to distinguish handwriting. Also, a general deep neural network can only handle a single input of an image, which is less efficient in scenarios where two or more images are compared.

For the first issue, there are two solutions for feature disentangle, and one is to use the concept of adversarial training [15], add additional text classifiers as the discriminator, and add a training goal for the encoder, which is to make the discriminator can not distinguish the character, the model structure is shown in Fig 2a.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Two feature disentangle methods [15,16] applied in voice conversion, the adversarial training method is more generalized, but the complex architecture is difficult to train; another method is easy to implement, but it can not process the voice never present in the training set, so it has less generalization ability.

https://doi.org/10.1371/journal.pone.0353503.g002

The solution is relatively complex, which requires iterative training of the discriminator and encoder and its convergence is difficult. Another solution is adding character information to the model [16]. The information can represent by one-hot encoding. Since training data already provide additional character information, encoder and decoder are trained not to involve character context information (Shown in Fig 2b. However, this solution cannot handle new characters, and for those characters that are not in the training data, the UNK(Unknow character) is not suitable for this situation. As the characters to be compared in the signature comparison task are known in advance, it is most convenient to use this solution to filter the character context information and focus on the handwriting feature extraction.

The second problem can be solved using a siamese network that can process two inputs simultaneously. Siamese has two neural networks (CNN/LSTM/RNN) with the same structure and shared weights, which can acquire the feature vectors of both inputs simultaneously and then construct a contrastive loss function employing a distance metric to determine the similarity and train the network. Siamese network has a wide range of applications, initially mainly used for signature authentication identification. However, later it has many applications in both CV and NLP fields, such as face verification [17], handwriting recognition [18], visual tracking algorithm [19], semantic similarity analysis of the sentence [20], matching of question and answer [21]. In the Kaggle kernel competition called Quora Insincere Questions Classification, the siamese network determines whether two questions are the same question. Since its origin as one of the earliest siamese applications, handwriting authentication has been plentiful research. Many related word-based signature datasets have been published, e.g., CEDAR [22,23], MCYT-75/100 [24], BHSig-260 [25,26], and GPDS [27,28]. Those signatures in English are split by word, but a signature usually contains more than one Chinese character [29]. There are also some public Chinese signature datasets, such as SigComp11 [30], CSD [31], GTR [32] etc. The signatures in these datasets are based on multiple characters. Therefore, training from the character level requires re-collecting data to mitigate the adverse effects of noise.

Critical analysis of deep learning methods:

• Strengths: CNNs can automatically learn features, show better robustness to variations, and are scalable.
• Limitations: They are data-hungry, have black-box nature, and struggle with feature disentanglement.
• Why Insufficient: Standard CNNs typically process entire signatures or multiple characters together, learning both handwriting style AND character structure simultaneously, which leads to confusion when comparing characters with similar structures but different handwriting styles.

2.3 Attention mechanism

The attention mechanism was inspired by brain signal processing mechanisms specific to the human visual system(HVS). The human vision quickly scans the global image to obtain the target area to be focused, known as the attention zone, and then devotes more additional resources to this area to obtain more details about the target and block out other unimportant information. Initially used for machine translation [3], attention mechanism has been widely used in recent years in various areas of NLP, such as speech recognition [33], recommendation system [34], text comprehension [35], document classification [36].

The attention mechanism also has many applications in computer vision, where its primary purpose is to allow deep neural networks to learn the regions of each image that need attention. There are two different types of attention mechanisms in computer vision: soft attention and hard attention. In particular, soft attention mechanisms are divided into: regional attention networks, channel attention networks, and hybrid domain attention networks. Soft attention is deterministic attention, which can be generated directly by the neural network after training. The critical point is that soft attention is differentiable, so the gradient can be calculated by the neural network and the forward propagation and backward feedback can be used to learn the attention weights [37]. The idea of hard attention [38], unlike soft attention, is that every point in an image has the potential to be attended to, and hard attention is a stochastic prediction process with more emphasis on dynamic changes. The crucial thing is that hard attention is non-differentiable and the training process is often trained by reinforcement learning. Also, there are many attention mechanisms related to research, such as residual attention network [39], interaction-aware spatiotemporal pyramid attention networks [40], recursive attention mechanisms [41], and cross-attention mechanism [42,43].

2.4 Comparison with recent studies

Recent advances in handwriting verification have demonstrated the effectiveness of deep learning approaches. Chao-Qun et al. proposed an offline handwriting verification method based on Siamese networks with multi-channel fusion, achieving state-of-the-art performance on Chinese handwriting datasets [44]. Their dual-channel architecture shares similarities with our CLASN, but our approach incorporates spatial attention mechanisms and character one-hot encoding for better feature disentanglement.

Liu et al. developed a multi-scale SE attention model for handwritten Chinese character recognition, achieving 98.6% accuracy on writing grade examination datasets [45]. While their recognition task differs from our comparison task, their attention mechanism design validates the effectiveness of attention-based approaches for Chinese character analysis.

Kulkarni & Wagh proposed a Transfer Learning-enhanced Siamese Neural Network for signature verification, achieving 90.10% accuracy with EER of 10.08% [46]. Our CLASNF model achieves comparable performance (90.78% ACC, estimated EER 10.6%) on the more challenging character-level comparison task.

Afzali et al. developed a Customized Siamese CNN for writer verification using combined loss functions, demonstrating the effectiveness of Siamese architectures across different languages (English and Arabic) [47].

Huang et al. proposed MS-SigNet with co-tuplet loss for offline signature verification and introduced HanSig, a large-scale Chinese signature dataset [48]. While their work focuses on signature-level verification, our CCSD dataset is specifically designed for character-level Chinese handwriting comparison.

3.1 Overview

In this section, we will introduce our proposed methods for handwriting comparison tasks in detail. First, we present the overview of the model in Section Overview. Second, we give the detail of the proposed siamese networks in Section Siamese networks architecture. Finally, Section Attention explains the attention mechanism in the model.

We will start by defining some notations and describe the handwriting comparison task. Our model accepts two Chinese character images as input. The task is to determine whether those two character images come from the same person. Unlike traditional handwriting verification tasks, our task is base on character instead of multiple words or phrases. The handwriting comparison task can be model as finding the similarity of two images of the character. The Siamese architecture is chosen because handwriting comparison is fundamentally a similarity learning problem. By using two identical networks with shared weights, we ensure that: (1) The same feature extraction process is applied to both inputs, (2) The learned embeddings are directly comparable, (3) The model can generalize to unseen user pairs. An overview of our proposed method is shown in Fig 3. Firstly, we generate the list of data pairs. The number of positive data pairs is approximately equal to the number of negative data pairs, which can avoid the unbalanced data problem. Then we divide the training, validation, and test datasets in the list. Secondly, after we feed the paired sample data into the model, we get two images’ output embedding. Thirdly, we concatenate the one-hot encoding of character to the embedding and calculate the Euclidean distance between data pairs’ results. Finally, the model determines whether the same user writes two characters according to the margin value r, which is the hyper-parameter relate to the error rate.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. The overview of our proposed method.

In the examinations, candidates need to write down the same content character by character in the squares of the test paper, and the handwriting comparison is used to identify the cases of surrogate exam-takers. The figure shows handwriting samples from three different candidates; each sample contains the same three Chinese characters arranged from left to right. The first character (leftmost) is jin (meaning today or now), the second is tian (meaning day or sky), and the third (rightmost) is wei (meaning dimension or to maintain).

https://doi.org/10.1371/journal.pone.0353503.g003

3.2 Siamese networks architecture

The Siamese architecture is chosen because handwriting comparison is fundamentally a similarity learning problem. By using two identical networks with shared weights, we ensure that: (1) The same feature extraction process is applied to both inputs, (2) The learned embeddings are directly comparable, (3) The model can generalize to unseen user pairs.

The main idea of the siamese network used in this paper is to map the input images to the target space by a series of convolutional layers with spatial attention mechanism and compare the similarity in the target space using simple distances metrics (Euclidean distances, cosine distance, or Manhattan distance) [49]. In the training phase, the goal is to minimize the value of the loss function for a pair of character images from the same user and maximize the value of the loss function for many samples from different users. The siamese network architecture is shown in Fig 4, which is a 2-channel convolutional neural network. After each convolutional layer, we use the batch normalization layer to prevent vanishing gradient and speed up the training process; and we use the dropout layer to prevent overfitting. Although two CNNs are used to extract character features in the architecture, they are equivalent to one network because they share weights.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. An illustration of the architecture of CLASN explicitly showing the forward propagation of two Chinese character images.

The one-hot encoding of the character is concatenated to the output embedding before we calculate the contrastive loss. Those two channels share weights, and the input of each channel is dimensional, and the parameter number of 5 convolutional and 2 fully connected layers is given by 80/2,336/18,496/73,856/295,168/16,385,000/10,010. We will show the specific details of Conv1 and Conv2’s attention operation in Section Attention.

https://doi.org/10.1371/journal.pone.0353503.g004

As depicted in Fig 4, the network contains seven layers with weights; the first five are convolutional, and the other two are fully connected. The spatial attention mechanism of the convolutional layer will be detailed in Section Attention. The output of the last fully connected layer is concatenated to the one-hot encoding of the input character. We apply the ReLU non-linearity to the output of every convolutional and fully connected layer, except the last layer. After every convolutional layer besides the first one, We employ the max pooling, batch normalization, and dropout layers throughout our network, with pooling kernel size = 2, stride = 2, dropout rate = 0.2. To prevent the loss of handwriting information near the edges of the image, we perform a reflection pad operation before each convolutional layer, with padding size = 1. All five convolutional layers have the same with a stride of 1 pixels. The convolutional kernels in these convolutional layers are 8, 32, 64, 128, 256, respectively. The two fully connected layers have 16384 and 10 neurons, and the outputs of them are 1000 and 10 each. Since the character table in our dataset has 60 characters, the one-hot encoding of character is 60 dimensions which can be expressed as . Character one-hot encoding is inspired by feature disentanglement research [18]. By explicitly providing character identity, we enable the network to: (1) Filter out character structure information, (2) Focus exclusively on handwriting style features, (3) Compare characters with different structures more effectively.

Assume that there are a total of N users involved in the dataset, and each person has a character table containing 60 characters. We can donate the input data as:

(1)

where is the grayscale handwriting image with and is corresponding writer.To simplify the notation, we donate the character image as:

(2)

is shot notation for , it represents the distance between paired image i and j and it can be calculated as follows:

(3)

where G represents the siamese network with parameters , the output embedding of network is concatenated with the one-hot encoding of character . The distance calculation method can choose L₂ norm distance.

With the distance, we can use pair-based metric learning loss to train the model. Experiments show that the softmax loss function in the handwriting recognition MNIST test set (handwriting digital image set) exhibit little discrimination boundary, so we use contrastive loss [50] function to process the paired data as follows:

(4)

where r is margin, when the input data is a negative sample pair(), if the distance between two samples is greater than r, the model loss value will be 0, implying that the model is not updated, and mean the two input character is written by the same user (Positive data pair). Vice versa means that they were written by different users(Negative data pair). []₊ is hinge function, which can be expressed as:

(5)

3.3 Attention

Following the previous work [51], we adopt the convolutional block attention module(CBAM) as our attention mechanism. A channel and spatial-based cascading attention mechanism are used in CBAM to enhance the feature map’s valuable features and suppress the useless features so that the model can focus on the critical regions of different input data. The channel attention module focuses on the meaningful feature map; The spatial attention module focuses on the informative part. Spatial attention is chosen over channel attention because: (1) Character images are grayscale (single channel), making channel attention less informative, (2) Different strokes within a character have varying discriminative power, (3) Attention maps provide interpretability by highlighting important regions. Since our character comparison task processes a grayscale image of characters, which contains only one channel, we ignore the channel attention module and directly perform spatial attention operations on the results of the first two convolutional layers in siamese networks. The specific steps are shown in Fig 5.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. The spatial attention operation of Conv1.

After the attention operation, the output turns out to have the same shape size as Conv1.

https://doi.org/10.1371/journal.pone.0353503.g005

Firstly, we apply the global average pooling and global max-pooling in the channels’ dimension, and we concatenate two pooling feature maps () from the shape input F. To keep the output shape the same as the input shape, the kernel size we can use is or . Here we choose the former option, which can reduce the number of parameters and computational complexity. Then we use the convolution kernel to generate a spatial attention map . Finally, we multiply the input with the obtained new feature maps to get the spatially adjusted feature maps, which can be calculated as:

(6)

4.1 Data augmentation and dataset

We had 800 users write all the same 60 characters onto the paper within squares, then scanned and cut by squares, and some of the results are shown in Fig 6a. After getting the raw data, we still need to perform data augmentation. Data augmentation is a typical operation in the CV field, and it is the simplest and most direct way to improve model performance. Standard data augmentation methods include geometric transformations such as cropping, flipping, rotating, scaling, warping, pixel scrambling, adding noise, illumination adjustment, contrast adjustment, sample summation or interpolation, and segmentation patches. Generally speaking, the purpose of data augmentation is to extend the sample distribution space of the original dataset, not to spoil it; for example, flipping the handwriting-recognized image and then training it will reduce the effect. In this paper, we use two approaches to extend the original dataset.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. The Chinese character image in CCSD.

(a) is the Chinese character signature image written by different users; (b) is examples of characters image from different fonts, each font has 60 characters. The five Chinese characters displayed in all samples are arranged from left to right as follows: the first is yi (meaning to serve or labor), the second is mi (meaning secret or dense), the third is juan (meaning volume or scroll), the fourth is xu (meaning to allow or permit), and the fifth is lu (meaning dew). Those 10 examples of Chinese fonts by serial number are FZYaoti, SimSun, Microsoft YaHei, Malgun Gothic, FZShuTi, STLiti, STCaiyun, STXinwei, STHupo, and LiSu.

https://doi.org/10.1371/journal.pone.0353503.g006

The first method is shown in Fig 6b, which uses the different fonts to generate 60 characters according to the character table. We regard the same font’s characters as written by the same user and the characters of different fonts as written by different users. The second method is to use positive and negative sample pairs instead of random sampling. Instead of dividing the training, validation, and testing datasets directly by character images, the datasets are divided by users, and then image pairs are sampled as positive sample pairs () for identical users, and image pairs are sampled as negative samples () for different users. Since the sample pairs can be different permutations, the number of training samples can be greatly increased. When the number of positive and negative sample pairs is approximately equal(), the problem of data imbalance can be avoided. Besides, we can use the same raw data to generate positive and negative sample pairs of different magnitudes to perform comparative tests on the model’s scalability. The paired data generation pseudocode is shown in Algorithm 1.

Algorithm 1. Paired data generation algorithm

Input:

 Input1: userList includs all users in the dataset

 Input2: negUserNum is the number of negative random user samples

 Input3: negCharNum is the number of negative random character samples

Output:

 Output1: posDataList

 Output2: negDataList

 1:  for do

 2:   userCharList = getCharList()

 3:   //Iterate over each character for each user

 4:   for i = 0; i < len(userCharList); i++ do   //Positive paired data generation

 5:    for j=0; j<len(userCharList); j++ do

 6:     Add (userCharList[i],userCharList[j]) to posDataList

 7:    end for

 8:    //Negative paired data generation

 9:    for k=0; k < negUserNum; k++ do

10:      = randomSampleUser(userList)

11:     for n = 0; n < negCharNum; n++ do

12:       = randomSampleChar()

13:      Add to negDataList

14:     end for

15:    end for

16:   end for

17:  end for

18: return posDataList,negDataList

There are approximately 800 individuals and different kinds of the character of 50 fonts in our Chinese character signature dataset, and We divide the CCSD into training, validation and test sets by user and font. The statistics of the CCSD division are shown in Table 1.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Statistics of original data. Train Set, Validation Set, Test Set denote the number of users or fonts in those datasets, respectively. Total Image is the total Chinese character image number belong to the user or font. The symbol “*” indicates that the invalid or blank images are excluded.

https://doi.org/10.1371/journal.pone.0353503.t001

4.2 Evaluation metrics and experimental setup

Since the handwriting comparison is a special case of handwriting verification, we choose the false acceptance rate (FAR) and the false rejection rate (FRR) instead of the character error rate (CER) and character accuracy rate (CAR) [52,53]. To measure the model’s quality, we follow the previous works [31,54,55] and use FAR, FRR, and ACC (Accuracy) to evaluate our method and compare it with other existing methods. FAR is the measure of the likelihood that the model will incorrectly accept an impostor as normal data. The FAR of the model typically is stated as the fraction of the number of false acceptances divided by the number of all impostor attempts. FRR is the measure of the likelihood that the model will incorrectly reject normal data. The FRR of the model typically is stated as the fraction of the number of false rejections divided by the number of identification attempts. ACC is the most common evaluation metric in face or fingerprint recognition tasks. The ACC of the model is stated as the fraction of the number of samples classified correctly divided by all samples.

To train the proposed CLASN model, we employ the Adam optimizer with contrastive loss (margin r = 2.0). Given the importance of distinguishing different users’ handwriting patterns, we adopt an asymmetric sampling strategy where negative samples outnumber positive samples by a 1.5:1 ratio, which enhances the model’s discriminative capability for forgery detection. Following standard practices, input images are normalized and resized before feeding into the network, and dropout regularization is applied to prevent overfitting. We train the model for 2000 epochs and select the checkpoint with the best validation accuracy as the final model for testing.

While ROC curves and EER (Equal Error Rate) are commonly used in biometric verification, we note that FAR and FRR inherently capture the same trade-off information. Specifically, FAR corresponds to False Positive Rate (FPR) and (1-FRR) corresponds to True Positive Rate (TPR) in ROC analysis. Based on our reported results in Table 2, the approximate EER values can be estimated as the midpoint between FAR and FRR: CLASN achieves approximately 11.4% EER (midpoint of 12.72% FAR and 10.17% FRR), and CLASNF achieves approximately 10.6% EER (midpoint of 11.93% FAR and 9.36% FRR). These values are competitive with recent signature verification studies [48].

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Comparison between the models mentioned on CCSD. FAR, FRR, ACC denote False Acceptance Rate, False Rejection Rate, and Accuracy, respectively. The symbol “+” indicates that the higher the value is the better the model performs. The symbol “-” is the opposite.

https://doi.org/10.1371/journal.pone.0353503.t002

4.3 Result

The experimental results of our model against other models on CCSD are shown in Table 2. The results show that our proposed model achieves the best performance in the main evaluation metrics we mentioned in Section Evaluation Metrics and Experimental Setup. Specifically, the ablation studies demonstrate that each proposed component contributes significantly to the final performance.

Beginning with the baseline model (without attention mechanism and character encoding), it achieves 18.42% FAR, 14.23% FRR, and 83.67% ACC. When incorporating the Spatial Attention module alone, the model obtains a reduction of 3.24% FAR, 1.78% FRR, and an improvement of 2.52% ACC. Similarly, adding Character Encoding individually yields a reduction of 3.75% FAR, 2.41% FRR, and an improvement of 3.09% ACC. Notably, when both components are integrated (CLASN), the model achieves a reduction of 5.70% FAR, 4.87% FRR, and an improvement of 4.77% ACC over the baseline, indicating that the spatial attention and character encoding modules have complementary effects.

The proposed CLASN model enhance by the Fonts dataset achieves a reduction of 44.06% FAR, 42.6% FRR, and an improvement of 16.12% ACC over the baseline SVM. Additionally, compared with traditional deep learning models such as CSN and SCSN, our CLASN model also shows substantial improvements, and the complete CLASNF model further widens this gap. Furthermore, our model outperforms the traditional deep learning model by a large margin. For example, our proposed CLASN model enhance by the Fonts dataset gets a reduction of 38.88% FAR, 29.13% FRR, and an improvement of 11.15% ACC over the CNN-based siamese model. It is worth noting that even without the Fonts dataset, our proposed CLASN model (12.72% FAR, 9.36% FRR, 88.44% ACC) still outperforms all baseline methods including SVM, CSN, and SCSN, validating the effectiveness of the proposed attention and encoding mechanisms.

Moreover, the CLASN model is remarkably improved by additional Fonts dataset. The CLASN model with Fonts data set receives a reduction of 6.21% FAR, 22.54% FRR, and an improvement of 2.65% ACC on the CCSD test set contrasted with the CLASN without Fonts dataset. This confirms that the Fonts dataset provides valuable supplementary information for cross-domain text recognition, particularly in reducing false rejections.

4.4 Analysis and discussion

Usually, the handwriting identification task does not limit the characters to be discriminated, and the characters set is large, so it is not easy to use the one-hot encoding to filter the character meaning during the training process. While in the handwriting comparison task, there is a fixed set of characters, which can provide the model with additional character one-hot encoding during the training process, thus making the model focus on the strokes feature extraction of signature image and not suitable for the other datasets. The downstream task in this paper is to prevent surrogate exam-takers, where a test is divided into multiple exams, each of which requires the candidate to write the same paragraph of text, and then determine whether the same person writes the character on different examination papers. We use a single character to discriminate against the user. We can stack the multiple character result to improve the discriminative accuracy in the downstream task. In previous work, traditional machine learning methods require much work for image prepossessing, which is fast but not highly scalable for verifying signatures; deep learning networks can not only automatically extract features but also have better robustness to panning and blocking of signature images, and the additional one-hot encoding can prompt the model to remove the information of the character from the feature extraction process. Fig 7 has two meanings that show the strokes feature information is essential in the evaluation process. First, comparing the images in column 1 with those in columns 2 and 3, we can see that the dimensional reduction of character images are randomly distributed, and the siamese network can classify the original images; second, comparing the images in columns 2 and 3 we can observe that the CNN construct in the siamese network extracts some additional character information without the one-hot encoding of characters, for example, in the second row and second column of the plot, the different user wrote the characters still have less spatially distant due to the same glyphs except user 4; while with the extra one-hot encoding, the siamese network will focus on extracting the handwriting information, thus making the character cluster interval of each user very obvious, so the characters were written by the different user have more significant spatially distant.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. The figure shows the distribution of t-SNE [56] dimensional reduction for character encoding in different scenarios.

The first row is the result of clustering by the character with 60 different characters; the second row is the result of clustering by the user with 10 users. The first column is the result of dimensional reduction by the original image information; the second column results from clustering without using one-hot encoding; the third column results from clustering after concatenating the one-hot encoding of characters.

https://doi.org/10.1371/journal.pone.0353503.g007

The specific similarity distance results of positive and negative paired data in Fig 8 show that the same users all have low similarity values for their handwritten characters. In contrast, different users, regardless of whether the characters are the same, have high similarity values. The classification result in the first column of the third row is 0.58, which is slightly larger than the threshold value of 0.5. The characters of the same user are incorrectly discriminated against as characters of different users. This error is that there are fewer character strokes and insufficient handwriting information that can be provided to the model for discriminating.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. The number in the upper left corner of each image represents the similarity of the sample pairs.

The ten sample pairs in the first and second columns are from two different users; the five sample pairs in the third column show the similarity of the same characters from different users, and the five sample pairs in the fourth column show the similarity of different characters from different users. The Chinese characters in each pair are described from left to right. In the first and second columns (from top to bottom): (1) yi (to serve or labor) and mi (secret or dense); (2) li (to stand or establish) and juan (volume or scroll); (3) ben (root or book) and ge (measure word or individual); (4) ren (person or people) and juan (volume or scroll); (5) xu (to allow or permit) and shi (real or true). In the third column, the identical characters written by different users (from top to bottom) are yi, mi, li, juan, and ben, each paired with itself. In the fourth column, the different characters from different users (from top to bottom) are: (1) ge (individual) and xi (breath or rest); (2) juan (volume) and xian (present or to appear); (3) shi (real) and tiao (strip or article); (4) li (reason or principle) and wan (complete or finish); (5) ge (standard or lattice) and gui (rule or regulation).

https://doi.org/10.1371/journal.pone.0353503.g008

Training data is crucial for the deep learning models. The character-based signature dataset we collected can be used to obtain balanced training data by randomly sampling positive and negative sample pairs and avoid the long-tail problem. In this paper, we also add dozens of font character images to extend the training data and enhance the model generalization ability.

For the attention mechanism of the CLASN model, we use the spatial attention module in CBAM. Experiments show that channel attention does not significantly improve the effect for grayscale images like signature images. Using only spatial attention can make the model discriminate better for Chinese characters with simple strokes on the one hand and improve its computational efficiency on the other. The visualization of the attention matrix is usually used in NLP, which can visualize the attention weight between tokens; for example, in reading comprehension tasks, the visualization of the attention matrix can represent the article-question attention in different colors. The research in this paper belongs to the CV domain. It uses the spatial attention mechanism, so we observe the model’s attention region to the handwriting through the feature map. The specific results are shown in Fig 9. We extract the feature map of the spatial attention layer after the second convolutional layer and then convert it into a heat map and put it together with the original character picture. We can notice that the spatial attention layer mainly focuses on the handwriting strokes information in the character picture. The location of the brighter color in the heat map indicates that the region has a higher weight. Since the original character picture’s background is not processed, the model also has different weights for different background colors.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 9. Comparison of the original image with the result of the overlay of the spatial attention heat map, the graph contains the comparison result of 20 characters, the value from small to large corresponds to the color from dark to light.

For the original grayscale images (left half of each pair), the characters are described column by column from left to right, and row by row from top to bottom. Column 1: yi (to serve or labor), mi (secret or dense), li (to stand or establish), juan (volume or scroll), and ben (root or book). Column 2: ge (measure word or individual), ren (person or people), xu (to allow or permit), shi (real or true), and lu (dew). Column 3: li (reason or principle), wu (mistake or error), ge (grid or pattern), di (ground or earth), and xi (breath or rest). Column 4: zun (to comply with or observe), xian (to appear or present), guan (to manage or tube), tiao (strip or article), and du (alone or independent).

https://doi.org/10.1371/journal.pone.0353503.g009