2.1 Applications of phoneme recognition

Phoneme-level pronunciation accuracy assessment methods are widely applied in the field of pronunciation evaluation. Leung et al. proposed an end-to-end phoneme recognition system based on CNN-RNN-CTC, combining convolutional neural networks (CNN), recurrent neural networks (RNN), and connectionist temporal classification (CTC) technology [12], for detecting and diagnosing mispronunciations in second language learners. Building on this, Zhang et al. proposed an end-to-end automatic pronunciation error detection system based on an improved hybrid CTC/Attention architecture by combining the advantages of CTC (Connectionist Temporal Classification) and attention mechanisms [13]. This system effectively avoids the complex module segmentation and forced alignment requirements commonly found in traditional speech recognition systems. Meanwhile, Yan et al. focused on pronunciation error detection for second language (L2) English learners and proposed an end-to-end approach based on a novel anti-phone modeling technique [14]. Traditional ASR methods often struggle to accurately recognize approximate or distorted pronunciations between native language (L1) and target language (L2) phonemes. The system proposed in this paper enhances the system’s comprehensive recognition ability for both categorical and non-categorical mispronunciations by expanding the original L2 phoneme set to include the corresponding anti-phone set.

2.2 English phoneme recognition

Speech recognition-based English phoneme recognition technology has developed rapidly. With the rise of deep learning techniques, convolutional neural networks (CNN) have been introduced into phoneme recognition. Abdel-Hamid et al. proposed a CNN-based acoustic model [15], which significantly enhanced the model’s adaptability to complex acoustic environments. CNNs excel in phoneme recognition tasks by automatically extracting local features to capture important patterns in audio signals. In addition, Tóth et al. used a hierarchical convolutional deep Maxout network [16], which further enhanced the model’s ability to recognize phonemes. Building upon convolutional networks, Passricha et al. introduced Convolutional Support Vector Machines (CSVM) [17], attempting to combine the benefits of convolutional feature extraction and support vector machines to improve phoneme recognition performance. CSVM integrates the feature extraction capabilities of convolutional layers with the classification power of SVM, providing a new approach to phoneme recognition. Ravanelli et al. proposed SincNet [18], an interpretable convolutional filter model that better captures frequency information in audio signals. SincNet improves frequency feature modeling by using the Sinc function as a filter in the convolutional layers. In the integration of convolutional and recurrent networks, Zhao et al. proposed a Recurrent Convolutional Neural Network (RCNN) [19], which leverages the advantages of both convolutional and recurrent layers in processing speech signals, thereby improving phoneme recognition accuracy.

2.3 Chinese phoneme recognition

Mandarin phoneme recognition and confusion analysis provide targeted teaching guidance for Chinese language teachers. Ding et al. collected learners’ self-reported pronunciation difficulties and coping strategies through methods such as questionnaires, reading tasks, and recall interviews [20]. The study found that learners generally face challenges in consonants, vowels, and tones during pronunciation, but there is a discrepancy between self-reports and actual performance. Jiang et al. explored the correlation between the pronunciation of Mandarin consonants and the distance in speech signals, revealing the causes of consonant pronunciation confusion from the perspective of signal processing [21]. The experimental results show that confusable consonant pairs (such as /l/ and /n/) have the shortest signal distance at the signal level, which aligns with the fact that they are prone to confusion during pronunciation. Arkin et al. conducted an in-depth analysis of the phonemes in the pronunciation of 50 Uyghur native speakers learning Mandarin Chinese using automatic speech recognition technology [22]. The study found that Uyghur students often misidentify the target phoneme “a” as “e” or “i” when speaking Mandarin, and misidentify “i” as “e” or “ie”.

3.1 Deep Zipformer block

The Zipformer encoder models features across different speech frames, which results in a relatively large number of parameters while maintaining performance. To address this issue, this paper optimizes the Zipformer model by reducing the number of Encoder Blocks to decrease the overall model size. Additionally, a strategy is introduced that reuses the attention weights calculated by MHAW (Multi-Head Attention Weight) to save computational resources. The Zipformer Block’s two-layer SCF (Self-Attention, Convolution, and Feed-forward) module group is expanded to three layers, aiming to improve the overall performance of the model by increasing its depth.

The improved Zipformer Block is shown in Fig 2. The speech features are input into the MHAW module to compute the attention weights, which are then passed into the Self Attention module and the NLA (Non-Linear Attention) module within the SCF module group. Meanwhile, the speech features are also fed into the Feed Forward module, processed by the NLA module, and then passed through three consecutive SCF module groups. Finally, the output is normalized by the BiasNorm module.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Structure of the Improved Zipformer Block Module.

https://doi.org/10.1371/journal.pone.0324048.g002

The MHAW module maps the input embedding to Query, Key and relative position vectors through linear projection. It calculates the attention scores using matrix multiplication, models the sequence position relationships by incorporating relative position embeddings, and normalizes the scores into attention weights via softmax, which are then used for downstream sequence modeling tasks. The process can be described as follows:

(1)

Where Q, K are the query and key obtained by linear projection of the input tensor, respectively. is the relative position encoding with shape (batch_size, 2*seq_len-1, embed_dim). is a linear transformation that maps the original relative position embedding to the required dimension of each attention head, providing independent position information for each head computation. Masking is used to mask invalid positions, preventing the model from focusing on information that fills the region or future time steps.

The weights computed through the MHAW module are fed into the Non Linear Attrntion module and the SCF module groups, respectively. In this case, the Non Linear Attention module aggregates the embedding vectors on the time axis using the attention weight values computed by MHAW. The module splits the input features into three parts by linear projection: s, x and y. s is input to the tanh activation function after Balancer processing, and x is normalised and multiplied with the processed s at the element level. Subsequently, the features are weighted by pre-computed attention weights to complete the aggregation of sequence information. The aggregated features are recovered to the input dimension by linear projection and normalised to generate the final output features. The specific formula is as follows:

(2)

Where, whiten is a data normalisation technique used to reduce the correlation between the input features. ScaledLinear is a scaling based linear transformation layer. tanh is a nonlinear activation function. Balancer is used to limit the activation values to prevent training instability.

In this paper, groups of three serially connected SCF modules are cascaded, and a Bypass module is embedded between each group of SCF modules with additional inputs of MHAW-calculated Attention Weights and Non Linear Attention output values.The Bypass module is specifically described as:

(3)

where x is the input value of the previous module, y is the output value of the previous module, and c is the learning channel scalar weight.

The SCF module group includes Self Attention, Convolution and Feed Forword respectively. Fig 3 shows Self Attention.The module requires input features X and attention weights W. Firstly, the features X are mapped to X_linear through a linear layer, after which the attention weights W and X_linear are computed by matrix multiplication. The resulting results are processed through Whiten in order to adjust the covariance of the features during backpropagation to be closer to the unit matrix for better feature decorrelation and improved generalisation of the model. Convolution uses a standard one-dimensional deep convolution module with symmetric padding at the end of the sequence in order to keep the output size constant. Feed Forword uses a feed-forward neural network layer to achieve deep processing and decorrelation of input features.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Structure of Self Attention module using input weights in this paper.

https://doi.org/10.1371/journal.pone.0324048.g003

3.2 GELU-based pred network module

RNN-T models perform well in tasks such as speech recognition, but the complexity and number of parameters of their model structure for processing labels is large, leading to difficulties when applied on resource-constrained devices. To address the challenge of reducing the model size in RNN-T architectures without significantly degrading the recognition performance, Ghodsi et al. proposed the stateless Pred Network [23], which simplifies the model structure by removing the recurrent layer in the Pred Network while maintaining the performance of the model. However, the limitation of neuron deactivation occurs when the inputs to the ReLu activation function in this model are negative, i.e., the outputs are forced to zero, which restricts its ability to handle complex data patterns. In order to solve this problem, this paper improves the stateless Pred Network by introducing the GELU (Gaussian Error Linear Unit) activation function, which makes the activation function more sensitive to changes in the inputs, and exhibits better performance and generalisation ability when dealing with complex high-dimensional data.

Fig 4 shows the structure of the improved stateless Pred network, transform text label sequences into a continuous feature space through embedding layers, the Balancer is used to control the distribution of activation values to avoid the negative impact of extreme values on the model training, and then a one-dimensional convolutional layer is used to capture local information in the sequence, and the GELU activation function is used to introduce the nonlinearities to help the model extract more complex features. The GELU activation function is used to introduce nonlinearities and help the model extract more complex features. The activated feature vector is passed through the Balancer layer again to further adjust the range of values in the output, and the final output is a vector with the shape of (N, U, decoder_dim). This structure ensures that the output vector captures the features of the input text while remaining numerically stable.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Improved Pred Network module structure.

https://doi.org/10.1371/journal.pone.0324048.g004

The GELU activation function provides a smoother activation curve and an adaptive gating mechanism, which makes the activation function more sensitive to changes in the input and reduces the instability of the gradient during the training process. The GELU activation function formula is as follows:

(4)

Where is the error function.

The image of the GELU activation function is shown in Fig 5, which exhibits a smooth S-shaped curve that gradually tends to zero in the negative region and gradually increases linearly in the positive region, which is more natural and continuous compared to ReLU’s segmented linearity and hard thresholding (where the negative values go directly to zero). This smoothness allows the model to be more delicate with inputs close to zero, avoids the complete neglect of negative values in ReLU, and is more stable during gradient propagation, thus aiding the optimisation process and the learning of complex features.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Improved Pred Network module structure.

https://doi.org/10.1371/journal.pone.0324048.g005

It has been proved by previous experiments that the GELU activation function can show better performance and generalisation ability when dealing with complex high-dimensional data, and it has been widely used in natural language processing models such as BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer), GPT (Generative Pre-trained), XLNeT (Generalised Autoregressive Pretraining for Language Understanding), Transformer and other natural language processing models.

3.3 Mixed pruned RNN-T/CTC loss

The loss function is a key measure of the gap between the model’s predicted results and the true results, and is a crucial part of optimising the model’s performance.The Zipformer model introduces the Pruned RNN-T method proposed by Kuang et al. in the module of calculating the loss function [24], but the pruning strategy used in this method discards some information that is crucial in the alignment process, which reduces the model’s accuracy and generalisation ability. To solve these problems, this paper introduces the CTC loss function on top of the Pruned RNN-T loss function, and proposes the hybrid Pruned RNN-T/CTC loss function method, which combines the Pruned RNN-T loss function and the CTC loss function in a weighted manner and balances the effects between them by adjusting the weights, aiming to improve the performance of the model by combining multiple different loss functions through the combination to improve the performance of the model.

The overall structure of the hybrid Pruned RNN-T/CTC Loss is shown in Fig 6, and the overall structure is divided into the two loss functions of Pruned RNN-T and the CTC loss function.The Pruned RNN-T loss values include the Simple Loss and the Pruned Loss, and the Simple Loss refers to the loss value of the model before the pruning operation, based on its complete structure and all parameters, whereas Pruned Loss is the loss value recalculated after the model has been pruned, i.e., after some of the network parameters and/or nodes have been discarded to reduce model complexity and computation. In calculating the CTC loss, the encoder output is first mapped to a space more suitable for CTC through a linear layer, followed by a Dropout layer to reduce the overfitting and enhance the model generalisation by randomly dropping some of the outputs. Afterwards, the LogSoftmax layer converts these vectors into log-probability form, ensuring that each dimension represents the category log-probability and sums to 0. Finally, these log-probabilities and labels are subjected to CTC loss calculation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Improved Pred Network module structure.

https://doi.org/10.1371/journal.pone.0324048.g006

The Hybrid Pruned RNN-T/CTC Loss is derived by weighted summation of the above three loss values and is given by the following formula:

(5)

Where L_total is the total loss value, L_pruned, L_simple and L_ctc are the Pruned loss value, the Simple loss value and the CTC loss value, respectively, W_pruned and W_simple are the scaling factors for the Pruned loss value and the Simple loss value, respectively.

The calculation formulas for the two scaling factors are as follows:

(6)

(7)

Where is the current training batch index, and is the step threshold.

As training progresses, the scaling factors of the loss function gradually transition from their initial values to the predetermined target values, enabling dynamic adjustment of the loss. Specifically, and are adjusted according to a linear progression during the training process, thus assigning appropriate weights to the smooth loss and pruning loss at different stages of training. Experimental results show that when the weight coefficient of the CTC loss is set to 0.085, the model achieves optimal overall performance. This approach effectively balances the influence of different loss components, significantly improving the model’s recognition accuracy and stability.

4.1 Experimental data

The experiments in this study primarily use the Mandarin speech datasets AISHELL1 and ST-CMDS. The AISHELL1 dataset was originally recorded by 400 speakers from different regions of China using a variety of devices, including high-fidelity microphones, Android phones, and iOS phones, with an audio sampling rate of 16 kHz. The ST-CMDS dataset is an open-source Mandarin Chinese speech dataset primarily used for tasks such as speech recognition, speech synthesis, and speaker recognition. The dataset contains a large number of monophonic audio files with a 16 kHz sampling rate, covering various daily conversation scenarios, such as online chatting and voice command for smart devices.

To adapt the dataset for phoneme recognition tasks, this study performed a phoneme label conversion on the original Chinese character text labels, aiming to construct the AISHELL1-PHONEME and ST-CMDS-PHONEME Chinese phoneme datasets. The core step in this process was to replace the original Chinese character labels with phoneme labels, utilizing the mapping relationship between Chinese characters and phonemes. Due to the phenomenon of polyphonic characters in Chinese, where the pronunciation of a character may differ in different word groups or contexts, a simple character-by-character replacement approach cannot meet the need for accurate labeling. Therefore, this study employed a phrase-level phoneme mapping table to replace the labels, thereby avoiding the phonetic ambiguity that arises from character-by-character substitution of polyphonic characters. This method allows the dataset to more accurately reflect the phonetic characteristics of phonemes in different contexts, adapting to more fine-grained, phoneme-level speech recognition tasks. The dataset divisions for AISHELL1-PHONEME and ST-CMDS-PHONEME are shown in Tables 1 and 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. The division of the AISHELL1-PHONEME dataset.

https://doi.org/10.1371/journal.pone.0324048.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. The division of the ST-CMDS-PHONEME dataset.

https://doi.org/10.1371/journal.pone.0324048.t002

To enhance the robustness of the model, this paper mixes the MUSAN dataset into the AISHELL1-PHONEME dataset to simulate noise interference in real-world environments, thereby improving the model’s adaptability under various noise conditions. The MUSAN dataset is a comprehensive corpus that includes various types of audio data, such as music, speech, and noise. These audio data come from multiple sources and cover a wide range of noise scenarios, including background noise, traffic noise, wind noise, animal sounds, as well as various types of music and speech materials, greatly enriching the dataset’s diversity.

By combining the MUSAN dataset with the AISHELL1-PHONEME dataset, various noise interference situations closer to real-world environments can be simulated, including noise from different frequency bands, dynamic noise, and sudden bursts of noise. Introducing this noise during the training process can effectively improve the model’s robustness, enhancing its ability to adapt to different environmental noises, allowing it to better handle variable and complex speech inputs in practical applications. Table 3 shows the classification of the MUSAN dataset.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. The categorization of the MUSAN dataset.

https://doi.org/10.1371/journal.pone.0324048.t003

4.2 Evaluation metrics

The experiments in this paper use three metrics to evaluate the model’s performance and efficiency: Word Error Rate (WER), Real Time Factor (RTF), and the number of parameters (Params).

4.2.1 WER.

Word Error Rate (WER) is a key metric for measuring the degree of difference between the output phonemes of a speech recognition model and the reference phonemes. It is used to evaluate the accuracy of a speech recognition model in transcribing speech. Specifically, WER quantifies the accuracy of the model under various conditions by calculating the difference between the model’s output and the true labels (reference phonemes). It considers three basic types of errors: substitution (Sub), insertion (Ins), and deletion (Del), providing a comprehensive reflection of the potential biases that may occur during the recognition process.

The calculation method of WER normalizes the total occurrences of substitution, insertion, and deletion errors by the total number of reference phonemes, yielding a ratio. This ratio indicates the extent of errors made by the model when processing a segment of speech. The lower the WER, the higher the model’s recognition accuracy, as it better matches the true phoneme labels. The calculation formula for WER is typically as follows:

(8)

Where Sub represents the number of substitution errors, which refers to the times the model’s output phonemes do not match the reference phonemes; Ins represents the number of insertion errors, which refers to the number of extra or inserted phonemes in the model’s output; Del represents the number of deletion errors, which refers to the number of phonemes missing or deleted in the model’s output. Num represents the total number of reference phonemes, which is the number of phonemes in the true labels.

By calculating the total of these error types and normalizing it with the total number of reference phonemes, WER provides a quantitative assessment of the accuracy of the speech recognition model. It is important to note that WER not only considers the model’s phoneme recognition accuracy but also reflects its performance in handling complex cases such as homophones, polyphonic characters, and noise interference. Therefore, WER is widely used as a standard to evaluate the performance of speech recognition systems.

In practical applications, WER also has its limitations, especially in cases with long texts or complex contexts, where a single error type may not fully capture the model’s performance. As a result, in addition to WER, other metrics such as Real-Time Factor (RTF) and Parameters (Params) are often used in combination to comprehensively evaluate the model’s recognition performance and operational efficiency.

4.2.2 RTF.

Real-Time Factor (RTF) is one of the key metrics used to evaluate the real-time performance and processing speed of speech recognition models, particularly in applications that require fast response times. The real-time factor represents the ratio between the time taken by the model to process a segment of audio and the actual duration of that audio. In the domain of phoneme recognition, RTF is used to measure how much audio the model can process within a given unit of time. It is typically employed to assess the response speed and processing capability of a speech recognition model when handling real-time audio data.

Specifically, the real-time factor quantifies the processing efficiency of a model by calculating the ratio between the time taken by the model to process audio data and the actual duration of the input audio. When the real-time factor is less than 1, it indicates that the model can process audio in real-time (i.e., within the same duration as the input), meaning the model processes more than 1 second of audio per second, which typically implies high processing speed and low latency. Conversely, when the real-time factor exceeds 1, it suggests that the model’s processing speed cannot meet real-time requirements, as the time taken exceeds the duration of the audio, resulting in increased latency and making the model unsuitable for real-time applications. Therefore, the smaller the real-time factor, the faster the processing speed and the shorter the response time, making it more suitable for practical applications that require rapid responses, such as voice assistants, telephone speech recognition, and intelligent customer service. The formula for calculating the real-time factor is as follows:

(9)

Where refers to the time taken by the model to process the input audio, typically measured in seconds; and refers to the duration of the input audio data, also typically measured in seconds.

A low Real-Time Factor (RTF) is crucial for the practical application of speech recognition systems, especially in edge devices or resource-constrained environments. To optimize the real-time factor, it is typically necessary to improve the computational efficiency of algorithms, leverage hardware acceleration, and reduce the complexity of the model to enhance the system’s response speed. In the design of speech recognition systems, the real-time factor needs to be balanced with other metrics (such as accuracy, model size, etc.) to ensure that the system can process data efficiently while maintaining good accuracy and reliability.

4.3 Model validation

4.3.1 Comparison of mainstream models.

This set of experiments adopts the state-of-the-art encoder-decoder architecture to thoroughly evaluate the performance of three mainstream acoustic models—Zipformer, Conformer, and Transformer—on the Mandarin phoneme recognition task. The experiments are based on the AISHELL1-PHONEME and ST-CMDS-PHONEME datasets, using the Fbank technique to extract speech features. Eight different phoneme recognition models are trained, and their performance is evaluated using word error rate (WER), real-time factor (RTF), and model parameter count from both the development and test sets. In the experiments, Zipformer-RNN-T(Pruned) demonstrates excellent performance on both the AISHELL1-PHONEME and ST-CMDS-PHONEME datasets, exhibiting strong robustness and high generalization ability.

As shown in Table 4, on the AISHELL1-PHONEME dataset, Zipformer-RNN-T(Pruned) achieves a Dev WER of 2.28% and a Test WER of 2.12%, significantly outperforming other models. In particular, compared to Conformer-CTC (Test WER 2.88%), Transformer-CTC (Test WER 5.78%), and Zipformer-CTC (Test WER 2.80%), Zipformer-RNN-T(Pruned) significantly reduces the error rate on the test set. Additionally, Zipformer-RNN-T(Pruned) has an inference speed (RTF) of 0.002, which is significantly faster than Conformer-RNN-T (RTF of 0.012) and Transformer-RNN-T(Pruned) (RTF of 0.003), demonstrating a clear advantage in inference efficiency.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Comparative experimental results of mainstream models on the AISHELL1-PHONEME dataset.

https://doi.org/10.1371/journal.pone.0324048.t004

As shown in Table 5, on the ST-CMDS-PHONEME dataset, Zipformer-RNN-T(Pruned) achieves a Dev WER of 4.28% and a Test WER of 4.51%, outperforming all other comparison models. Especially when compared to Transformer-CTC (Test WER 15.77%), Conformer-CTC (Test WER 6.47%), and Zipformer-CTC (Test WER 6.43%), Zipformer-RNN-T(Pruned) performs excellently in terms of accuracy. Although its Dev WER (4.81%) and Test WER (4.97%) are not significantly different from those of Conformer-RNN-T(Pruned), Zipformer-RNN-T(Pruned) still holds an advantage in inference speed (RTF of 0.002) and model size (61.1M parameters), demonstrating a balanced trade-off between efficiency and performance.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Comparative Experimental Results of Mainstream Models on ST-CMDS-PHONEME Dataset.

https://doi.org/10.1371/journal.pone.0324048.t005

These results indicate that Zipformer-RNN-T(Pruned) achieves optimal performance on both the AISHELL1-PHONEME and ST-CMDS-PHONEME datasets, with the fewest parameters, lowest error rate, and fastest inference speed, showcasing its superior robustness and generalization ability in phoneme recognition tasks. Compared to all other models, Zipformer-RNN-T(Pruned) leads across all evaluation metrics, proving its advantage as the optimal model.

In the phoneme recognition task, misclassifications are typically categorized into three types: substitution, insertion, and deletion. As shown in Figs 7 and 8, on both the AISHELL1-PHONEME and ST-CMDS-PHONEME datasets, the number of substitution errors is significantly higher than that of insertion and deletion errors. In the specific misclassification statistics, Zipformer-RNN-T(Pruned) achieves the lowest error count across all error types. On the AISHELL1-PHONEME dataset, the numbers of substitution, insertion, and deletion errors are 7165, 380, and 356, respectively; on the ST-CMDS-PHONEME dataset, they are 7890, 741, and 820. Compared to other models, Zipformer-RNN-T(Pruned) performs exceptionally well in reducing misclassification counts, demonstrating strong modeling capability and stability both in handling the dominant substitution errors and the relatively less frequent insertion and deletion errors.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Word error rates and the number of three types of misrecognized phonemes on the development set for different models(AISHELL1-PHONEME).

https://doi.org/10.1371/journal.pone.0324048.g007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Word error rates and the number of three types of misrecognized phonemes on the development set for different models(ST-CMDS-PHONEME).

https://doi.org/10.1371/journal.pone.0324048.g008

According to the phoneme dictionary provided by the AISHELL1 dataset, Mandarin pronunciation is subdivided into 66 distinct phonemes. As shown in Figs 9 and 10, the proposed model Zipformer-RNN-T(Pruned) demonstrates good phoneme recognition accuracy on both the AISHELL1-PHONEME and ST-CMDS-PHONEME datasets, covering all phoneme categories. The accuracy for the majority of phonemes is higher than that of other comparative models.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. The phoneme accuracy percentage on the development set for different models (AISHELL1-PHONEME).

https://doi.org/10.1371/journal.pone.0324048.g009

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. The phoneme accuracy percentage on the development set for different models (ST-CMDS-PHONEME).

https://doi.org/10.1371/journal.pone.0324048.g010

4.3.2 Noise simulation experiment.

To enhance the model’s generalization ability and robustness, this study conducts a data augmentation experiment. The MUSAN dataset is used as the background noise source and mixed with the original Train, Dev, and Test sets of the AISHELL1-PHONEME and ST-CMDS-PHONEME datasets, generating augmented datasets: Train+, Dev+, and Test+. The experiment adopts a dual-track parallel training strategy, where phoneme recognition models are trained based on both the Train+ and Train datasets, and validated and tested on the Dev+ and Test+ datasets. As shown in Tables 6 and 7, the models trained on Train+ outperform those trained on Train in both Word Error Rate (WER) and Real-Time Factor (RTF), indicating the effectiveness of the data augmentation strategy in improving model performance. This strategy not only strengthens the model’s ability to accurately recognize speech content, but also significantly enhances its robustness when faced with noise interference. Additionally, the proposed Zipformer-RNN-T(Pruned) model demonstrates exceptional performance on both training sets in the AISHELL1-PHONEME and ST-CMDS-PHONEME datasets, with significantly lower WER and RTF compared to other comparative models. This highlights the model’s strong robustness and ability to maintain stable performance in complex and dynamic acoustic environments.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Comparison of Model Performance Under Different Data Augmentation Strategies on the AISHELL1-PHONEME Dataset.

https://doi.org/10.1371/journal.pone.0324048.t006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Comparison of Model Performance Under Different Data Augmentation Strategies on the ST-CMDS-PHONEME Dataset.

https://doi.org/10.1371/journal.pone.0324048.t007

4.4 Ablation experiment

In phoneme recognition tasks, the misalignment between the input speech features and the output phoneme sequence is a common issue. The CTC (Connectionist Temporal Classification) loss function addresses this problem by introducing a blank symbol and allowing multiple possible alignment paths, which enables automatic alignment between the input and output sequences. This improves the model’s robustness and generalization ability. The weight configuration of the CTC loss function is an essential part of model performance tuning, and setting the appropriate weight can further enhance the accuracy of phoneme recognition.

In the experiment of adjusting the CTC weight coefficient, the model was first trained on the Train+ dataset to ensure that it could sufficiently learn the mapping between speech features and phoneme labels in the standard training set. Then, the mixed MUSAN dataset was applied to the Dev+ and Test+ datasets for testing, to evaluate the model’s performance when noise interference was introduced. Fig 11 presents the experimental results showing the difference in model performance on the Dev+ and Test+ datasets with different CTC weighting factor settings.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. The influence trend of different CTC weight coefficients on Word Error Rate (WER).

https://doi.org/10.1371/journal.pone.0324048.g011

When the CTC weight coefficient was set to 0.085, the model achieved the lowest Word Error Rate (WER) on both the Dev+ and Test+ datasets, with rates of 1.95% and 2.11%, respectively. This result indicates that the setting of the CTC weight coefficient has a significant impact on the performance of the phoneme recognition task. In particular, in complex noisy environments, an appropriate weight coefficient can significantly improve the model’s recognition accuracy and stability. Under this weight coefficient, the model was able to effectively capture phoneme information in noisy and misaligned speech data, thereby optimizing the accuracy of speech recognition.

When the CTC weight coefficient deviated from 0.085, the experimental results showed a gradual increase in WER, demonstrating a certain degree of performance degradation. This trend suggests that both excessively high and low CTC weight coefficients can negatively impact model training. For example, if the CTC weight coefficient is too high, the model may overly rely on the blank symbol, thus ignoring phoneme label information and leading to a decline in recognition accuracy. Conversely, if the coefficient is too low, the model may not be flexible enough in alignment, which negatively affects its ability to capture phonemes and leads to more errors. This gradually increasing WER trend further validates the role of the CTC weight coefficient in regulating the model’s training and inference process. An appropriate weight coefficient helps the model better balance the selection of alignment paths, avoiding over-reliance on blank symbols or other unreasonable alignment methods, thus improving the accuracy of phoneme recognition.

To systematically assess the impact of each component on the overall performance of the model, this paper designs a series of ablation experiments based on the Zipformer-RNN-T(Pruned) architecture. The aim is to quantify the influence of different modules (such as the Block3 module, GELU activation function, and CTC loss function) on recognition accuracy (Word Error Rate, WER), real-time factor (RTF), and model complexity (parameter count, Params) by comparing the model’s performance under different configurations.

First, the Zipformer-RNN-T(Pruned)-L model is constructed, using an Encoder-Block layer configuration of 2,2,4,5,4,2, but without the inclusion of the Block3 module, GELU activation function, or CTC loss function. As shown in Table 8, the model with this configuration has a parameter count of 65.1M, and the word error rates on the Dev+ and Test+ datasets are 2.04% and 2.23%, respectively, indicating that the model’s performance still requires further optimization. The preliminary results of this experiment suggest that, although the model can complete basic recognition tasks without too many additional modules, its recognition accuracy and efficiency are suboptimal, highlighting the need for the support of specific modules.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. Ablation experiment.

https://doi.org/10.1371/journal.pone.0324048.t008

Next, the improved Zipformer-RNN-T(Pruned) model is constructed, with the Encoder-Block layer configuration adjusted to 2,2,3,5,3,2, reducing the model’s parameter count to 61.1M. The model then progressively incorporates the Block3 module, GELU activation function, and CTC loss function. The experimental results show that when the model fully includes the Block3 module, GELU activation function, and CTC loss function, the word error rates on the Dev+ and Test+ datasets decrease to the lowest levels of 1.95% and 2.11%, while the real-time factor (RTF) remains stable at 0.002. This indicates that the collaboration of the Block3 module, GELU activation function, and CTC loss function significantly improves the model’s recognition accuracy while maintaining high real-time performance and relatively low computational complexity, validating the importance of these components in phoneme recognition tasks.

Subsequently, when the Block3 module is removed, the word error rate on the Dev+ dataset increases to 1.99%, and there is also a slight increase in the Test+ dataset’s word error rate. Similarly, when the GELU activation function or CTC loss function is removed, a slight increase in the word error rate is observed, further confirming the indispensability of these components for the model’s performance. The removal of the GELU activation function leads to a decline in the model’s nonlinear representation capability, affecting its generalization ability. On the other hand, removing the CTC loss function reduces the model’s performance in alignment tasks, leading to an increase in the word error rate.

These results show that the introduction of each component plays a crucial role in the model’s performance. The Block3 module helps the model better handle complex speech features, the GELU activation function provides stronger nonlinearity, and the CTC loss function effectively solves the alignment problem between the input and output sequences.

4.5 Inference experiments

In machine learning, inference is the process of predicting or classifying test data using a model that has already been trained.In this experiment, five search algorithms are used to inference the Zipformer-RNN-T(Pruned) model.

As shown in Table 9, the analysis in terms of word error rate shows that fast-beam-search has the highest word error rate, which is 50.3% and 46.74% in Dev+ and Test+ respectively, which is due to the fact that the algorithm expands multiple candidate paths at each step, which leads to the accumulation of errors and thus increases the word error rate. In comparison, fast-beam-search-nbesth and fast-beam-search-nbest-oracle perform significantly better, but both have a real-time factor of 0.016 and 0.007, respectively, which is higher than fast-beam-search’s 0.006. The WER of modified beam search and greedy search are the same, at 1.95% and 2.11% on Dev+and Test+, respectively. However, the real-time factor of greedy search is only 0.002, which is much lower than the 0.008 of modified beam search. The greedy search has the lowest real-time factor while ensuring a low WER.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 9. Search algorithm performance comparison.

https://doi.org/10.1371/journal.pone.0324048.t009

4.6 Phonological analysis of Mandarin

The Zipformer-RNN-T(Pruned) model based on the Zipformer-RNN-T(Pruned) model shown in Fig 12 suffers from the following typical confusions in predicting Mandarin phonemes:Firstly, in terms of vowel confusion, the bilabial bursts /B/ and /P/ are often confused with each other due to subtle differences in amplitude and spectrum in the speech signal; Secondly, for front and back nasals, the difference between /IN/ and /ING/ is mainly in the presence or absence and duration of the nasal rhyme-final; Finally, in the case of the distinction between flat and warped consonants, such as /Z/ and /ZH/, /S/ and /SH/, /C/ and /CH/, the key to the distinction lies in the subtle differences in the articulatory position of the tip of the tongue. However, these subtle pronunciation differences are affected by a variety of factors such as individual differences, variation in speech rate, pronunciation habits, and dialectal environments, making them difficult to clearly define in actual recognition.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 12. Confusion matrix for Mandarin phoneme recognition based on Zipformer-RNN-T(Pruned) model.

https://doi.org/10.1371/journal.pone.0324048.g012

Fig 13 shows the distribution of accuracy in predicting Mandarin phonemes based on the Zipformer-RNN-T(Pruned) model. Of the 66 phonemes, 64 (97%) of the phonemes had an accuracy of greater than 95%, with only two phonemes lying between 90% and 95%, /OO/ and /UA/, with the highest accuracy being /UENG/, due to the extremely low frequency of this factor in the dataset, which contained only two samples. The model proposed in this paper performs with high accuracy on most of the phonemes, demonstrating the effectiveness and potential of the model in the task of Mandarin phoneme recognition.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 13. Accuracy of Mandarin phonemes based on Zipformer-RNN-T(Pruned) model.

https://doi.org/10.1371/journal.pone.0324048.g013