3.1. Multimodal data integration

Multimodal data fusion is a process that amalgamates data from diverse sources and formats for all-encompassing analysis [18]. This paper focuses on the integration of speech and text data. The initial step involves utilizing Long Short-Term Memory (LSTM) to scrutinize the audio features of spoken English and to recognize the corresponding oral content. Subsequently, these recognized results are depicted as probability distributions, paving the way for modal fusion. Upon decoding the speech, the textual information is transformed into coherent text data, a process illustrated in Fig 1.:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Speech recognition process.

https://doi.org/10.1371/journal.pone.0329381.g001

The collected oral English audio sequence is represented as , where is the length of the sequence. Mel frequency cepstrum coefficients are used for feature extraction to obtain feature sequence . The expression of the input gate, forget gate, and output gate for each time step is processed with as input. The formulas are [19,20]:

(1)

(2)

(3)

Table 1 presents the definitions of the variables in Formulas (1) to (3):

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Definitions of the variables in Formulas (1) to (3).

https://doi.org/10.1371/journal.pone.0329381.t001

Using the LSTM gating mechanism, the mechanism and hidden state at the current time can be calculated through the formulas [21]:

(4)

(5)

(6)

Table 2 presents the definitions of the variables in Formulas (4) to (6):

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Definitions of the variables in Formulas (4) to (6).

https://doi.org/10.1371/journal.pone.0329381.t002

Audio files undergo preprocessing, after which their features are extracted. These features are then fed into the LSTM network, where speech processing is executed based on specific gating mechanisms. The use of LSTM facilitates the transfer of feature information to the fully connected layer, endowing the classification output with the requisite nonlinear transformation capability. Subsequently, the probability distribution of characters that might occur at each time point is predicted by transforming the results of the fully connected layer via Softmax. This process enables the joint temporal classification loss function to decode the probability distribution of characters and generate a series of text sequences.

The English oral translation system is enhanced by integrating visual information, specifically lip movements, as an additional input modality alongside speech and text data. Utilizing lip-reading technology, both speech and text inputs can be processed. This allows for the use of visual information to decode alterations in mouth shape, thereby providing supplementary clues particularly beneficial in instances where the speech signal is unclear or disrupted by noise interference.

BERT, a pre-trained language model, possesses a robust capability for processing and understanding natural language texts [22,23]. In this study, BERT is employed to integrate the derived text sequence with the original speech features, thereby creating a unified modal representation. To more effectively harness the cross-modal information and the long-term relationships within each modality, a joint feature representation is adopted for the vectors. This process encompasses embedding data from three modalities and amalgamating multiple vectors from a singular modality to produce a comprehensive multimodal vector, as depicted in Fig 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Multimodal fusion.

https://doi.org/10.1371/journal.pone.0329381.g002

In Fig 2, it is postulated that corresponding features are extracted from N modal data and subsequently transformed via a feature extraction network. The single-mode features of text, sound, and visual information modes are mapped onto a linear layer, facilitating the high-dimensional conversion of these three single-mode features; this entails converting the output vector dimension to a higher one. Ultimately, the feature vectors from both modalities are projected into the global feature space [24,25]. In Fig 2, N linear layers, corresponding to the three modal vectors, are employed to transform N feature vectors.

In order to ensure a distinct embedding for each modality, the BERT vector embedding simulates the process by embedding all the information from each individual modality, symbolized by. During the fusion of feature vectors, all information pertaining to the modality is accepted, and the features of that modality are pooled to the maximum extent. This process is illustrated in Fig 3:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Maximum pooling operation.

https://doi.org/10.1371/journal.pone.0329381.g003

According to Fig 3, after performing the maximum pooling operation, the most significant feature in this modality is taken as the initial value for the fused feature:

(7)

The sequence representation of input features fused with feature vectors is:

(8)

In order to process multimodal data efficiently, the positions of each modality are encoded separately using a modal embedding, and corresponding models are subsequently established. During the process of positional embedding, a modal encoding method is employed to ensure that the input feature sequence is distinguishable among different modalities. For N modal data, each modality is recognized by the modal code of N positions, and the coding dimension of each modal position is . The position modal encoding sequence of the fused vector is represented as:

(9)

The feature vectors of each modality contain semantic and modal information. On this basis, a unified modal representation is defined by the formula:

(10)

3.2. Construction of context-awareness model and optimization of attention mechanism

The Transformer model boasts flexibility and scalability in the field of Natural Language Processing (NLP) [26,27]. The resulting unified modal expression is input into the Transformer network. On this basis, the self-attention mechanism in Transformer is utilized to perform context-awareness and understanding of oral-transcribed text. By establishing a contextual model to determine each part of the text sequence, omissions and non-standard grammar can be recognized.

The context-awareness model is predicated on the architecture of the Transformer encoder, illustrated in Fig 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Context-awareness model.

https://doi.org/10.1371/journal.pone.0329381.g004

In Fig 4, the Add&Norm function incorporates residual connections and Layer-Normalization operations. The role of residual connections is to mitigate the loss of gradient during the backpropagation process. On the other hand, Layer-Normalization serves as a normalized method capable of discerning feature differences within a singular sample, thereby enhancing the stability of the model. In the context of feedforward neural networks, fully connected layers and ReLU activation functions are employed.

The Transformer encoder’s primary role is to convert the input into a hidden layer, a mathematical representation encompassing natural language sequences [28]. The decoder subsequently maps this hidden layer back to the natural language sequence, thus allowing the semantic model to address complex problems effectively [29,30]. In accordance with Bayes’ rule, given the context encoding sequence and the initial target vectors for each target variable, the distribution is broken down into the product of the conditional distributions for each target vector:

(11)

The limitations of self-attention mechanisms in creating effective models for high-dimensional input vectors necessitate the establishment of multi-head attention mechanisms [31,32]. are replaced with matrix coefficients , and they are mapped to a semantic space 1, as shown in Formula (12):

(12)

Here, represent input features. On this basis, the Attention operation is performed and expressed as [33]:

(13)

represents a single vector sequence, and its length is the same as . By using matrix coefficients to transform input features into semantic space 2, the Attention operation is repeatedly performed to obtain . Similarly, can also be obtained.

Finally, by integrating the formula, a complete multi-head attention representation is obtained:

(14)

In multimodal scenarios, each modality’s data is subjected to numerous Attention operations, potentially impacting the model’s learning efficiency and effectiveness. This paper employs a linear transformation method to compute multiple head attention points and aggregates them, thereby enhancing the efficiency and capacity of attention representation. This approach is depicted in Fig 5.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Optimization of attention mechanism.

https://doi.org/10.1371/journal.pone.0329381.g005

The linear transformation incorporates an additional linear layer into the traditional attention computation, thereby enhancing attention efficiency. During the calculation of queries, keywords, and values, this paper employs a weighted matrix for linear transformation alongside the original word embedding vectors. This approach augments the model’s capacity to capture input features, rendering the attention mechanism more versatile in weight distribution and more effective in identifying long-distance correlations within sequences. Through strategic adjustments to the weight matrix in linear transformation, an adaptive modification of attention allocation strategies under varying input conditions can be accomplished. Consequently, the representation ability and generalization effect of the model are significantly improved.

Each modality’s input data undergoes a linear transformation to generate various attention heads, which are subsequently weighted together for an integrated expression. This approach to multi-attribute recognition, based on sparse representation, effectively addresses the prevalent issue of redundancy in traditional multi-attribute recognition algorithms. It also minimizes unnecessary calculations, thereby enhancing the efficiency of multimodal data processing.

The output vector is ultimately derived by merging and transforming the results of each attention head through an additional linear transformation matrix. This “multi-head” architecture not only bolsters the capture of intricate patterns but also augments computational efficiency via parallel processing. Building on this, the paper employs hierarchical normalization and residual connectivity techniques to ensure robust learning and performance enhancement of self-interested factors. Consequently, the model can concentrate more effectively on pertinent information without being disturbed by inter-modal noise.

Firstly, by obtaining , the dot product of and below the i-th head is calculated and represented as . The calculation method is shown in Formula (15) [34,35]:

(15)

The parameter matrix is multiplied with the calculation result so that each dot product contains the attention information of all heads, and it is projected to space . is used to represent the parameter matrix. The linear transformation is calculated in Formula (16):

(16)

In Formula (16), represents a parameter in a matrix. Its spatial information is normalized using Softmax and then multiplied by a numerical matrix to obtain the head attention. All the head attentions are concentrated and multiplied by matrix to obtain mixed multi-attention. Formula (17) shows the calculation process:

(17)

In Formula (17), represents the value after spatial normalization; represents the value matrix; represents the head attention. Mixed head attention can correlate information from multiple heads of attention, thereby improving the translation quality of the model.

3.3 Synchronous enhancement algorithm

Dynamic Time Warping (DTW) serves as a metric for gauging the similarity between two time series, allowing them to be matched [36]. The DTW algorithm is adept at identifying the optimal alignment path between two sequences, even in instances where their timing isn’t strictly synchronized. It essentially seeks a pathway between two sequences that minimizes the cumulative distance of the entire route [37]. In a cross-modal translation context, there often exists a pronounced temporal mismatch between speech and text. This discrepancy arises as speakers’ paces can cause the content of a speech to either extend or contract in duration. Conventional timing window alignment techniques prove versatile in addressing such inherent timing inconsistencies. However, the DTW algorithm stands out by enabling precise positioning, owing to its ability to search for the best possible path based on temporal shifts. This method formulates a two-dimensional cost matrix where each entry denotes the distance between corresponding positions of the two sequences. Leveraging dynamic programming principles, it then discerns the most abbreviated cumulative route from its inception to its ultimate destination. Employing the DTW algorithm for data alignment ensures meticulous mapping and transformation of information across different modalities during conversion. Moreover, the robust adaptability of the DTW algorithm renders it well-suited for managing both linear temporal relationships and intricate temporal structures inherent in multimodal data within cross-modal translation tasks.

This article addresses the issue of processing time series with varying lengths by utilizing the Dynamic Time Warping (DTW) algorithm for time series alignment. This is achieved through the development of a context-aware mechanism and the optimization of an attention mechanism, which facilitate real-time synchronization between speech and text. Consequently, the fluency of translation is enhanced. The process is illustrated in Fig 6.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Sequence synchronous enhancement.

https://doi.org/10.1371/journal.pone.0329381.g006

Assuming that the oral audio feature sequence is and the text feature sequence is , Euclidean distance is used to calculate the distance matrix between and :

(18)

Cumulative distance matrix is calculated:

(19)

Starting from the bottom right corner of matrix , the optimal path is found through backtracking:

(20)

The principle of backtracking is to move along the direction of minimum value:

(21)

The conventional Dynamic Time Warping (DTW) method exhibits reduced computational efficiency when applied to data with extended time series. This article uses Locality Sensitive Hashing (LSH) to preprocess speech sequences and maps them to a low dimensional space to improve matching efficiency. Firstly, using the LSH algorithm, its dimensionality is reduced, and is defined as a set of hash functions. In the low dimensional space, the audio sequence data is expressed as:

(22)

On this basis, feature extraction is performed on the data processed by LSH to capture temporal dependencies in the sequence. The oral audio feature sequence is matched with the text feature sequence . The aligned sequence is:

(23)

The Dynamic Time Warping (DTW) algorithm’s matching results are seamlessly integrated with the module. We apply a context-aware model, which is combined with an optimized linear variation-based attention mechanism, to influence the final translation decision process collectively. The translation is executed via an aligned sequence, facilitating the automatic generation of the target language text.

4.1 Experimental data

The translation model in this article is shown in Fig 7.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Translation model in this article.

https://doi.org/10.1371/journal.pone.0329381.g007

This study utilizes LibriSpeech as the experimental dataset, which provides both text and audio formats. It comprises approximately 1000 hours of 16kHz read English speeches, meticulously segmented and normalized. These speeches are divided into audio files of roughly 10 seconds each, accompanied by text annotations. Prior to experimental analysis, both audio and text data from the original dataset undergo preprocessing. The audio files are edited to remove noisy sections, while the text data is stripped of special characters and punctuation, retaining only standard text. Text differentiation is achieved through the use of lowercase letters, and it is further segmented into sub-word units to create a vocabulary.

To simulate a diverse spoken language environment and ensure model adaptability to dialects and intricate structures, additional processing is performed on the experimental dataset. Extra oral English data is sourced from the TED talk dataset, with the supplementary corpus featuring distinctly marked Indian English, Australian English, Irish English, Scottish English, and Southern English. Conversational data from the Common Voice database is also included. To reflect real-world usage scenarios, the Switchboard corpus provides additional voice data from everyday phone conversations. Texts from the Penn Treebank database and COCA Complex Sentences that contain nested clauses, split sentences, and inverted sentences undergo speech conversion, representing 12% of the corpus. Moreover, the CELEX corpus is employed to embed specialized terms for high-frequency applications into various texts, such as news reports, academic speeches, and classroom discussions, thereby generating new speech samples. The refined dataset comprises 1200 hours of speech samples, with the dataset feature breakdown detailed in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Summary Table of Key Dataset Characteristics.

https://doi.org/10.1371/journal.pone.0329381.t003

The combination of these datasets provides a large and diverse data source, which is beneficial for the training of the English oral translation model. LibriSpeech dataset is the basic part, and other datasets make up the diversity of dialects, conversational style, complex grammar and special vocabulary. The use of this large-scale dataset has made the translation model more accurate, fluent and robust in different scenarios of oral translation.

To mitigate potential translation bias and safeguard speech data privacy, this study employs the Word Embedding Association Test to ascertain the presence of gender, race, and other biased vocabulary in training samples. By detecting such hidden biases, techniques like semantic substitution and data class balancing are implemented to diminish model output bias. Furthermore, the voice anonymity algorithm, predicated on Voice Masking, prevents the unauthorized disclosure of users’ voice identities. This is achieved by ensuring non-identification of voice data while maintaining voice recognition. Moreover, differential confidentiality approaches are utilized to secure data.

The oral data collected is automatically segmented into approximately 10-second intervals using the Librosa library in Python. These segmented audio clips are then annotated with text using the original audio paired with subtitles. For consistency with the LibriSpeech data, all audio files are converted to the WAV format. This processed oral data is subsequently integrated with the LibriSpeech dataset, resulting in a comprehensive corpus encompassing both formal reading and oral expressions. The entire corpus is categorized into 10 subsets based on specific spoken language scenarios. The composition of the final experimental data is detailed in Table 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Basic information of experimental data.

https://doi.org/10.1371/journal.pone.0329381.t004

To assess the efficacy of the model presented in this paper, we compare it with existing cross-modal translation research using experimental data from Table 4. Specifically, we evaluate its performance against two baseline models: the bilingual attention-based NMT model from reference 14 and the context-aware NMT model from reference 16. The bilingual attention-based NMT model employs an encoder-decoder architecture that dynamically adjusts focus during cross-modal translation by computing attention weights between the source and target languages. In contrast, the context-aware NMT model integrates multimodal information to form context representations, leveraging context windows and guiding generation during decoding through context vectors.

In our experiments, we utilize a rigorous experimental setup to guarantee the reliability and validity of the model. For training, we meticulously curated a comprehensive dataset encompassing LibriSpeech, TED talk dataset, Common Voice database, Switchboard corpus, Penn Treebank database, COCA Complex Sentences, and CELEX corpus. Each dataset underwent preprocessing to eliminate noise and extraneous data, and the audio data was partitioned into roughly 10-second segments, accompanied by text annotations. Model parameters were initialized using the Xavier initialization method to preserve the variance of the activations and gradients. Training was executed using an Adam optimizer with a learning rate of 0.001. We conducted 50 training sessions with a batch size of 32, shuffling the training data at each session to maintain diversity. The model’s learning was guided by a combination of cross-entropy loss and attentional loss. Post-initial training, we fine-tuned the model via domain adaptation, employing domain-specific subsets, data enhancement techniques, and iterative improvements involving multiple training and evaluation cycles. Experiments were performed on a high-performance computing cluster featuring NVIDIA® 3090 GPUs, multi-core CPUs, and 512GB of RAM. The software environment comprised the PyTorch deep learning framework and various Python libraries such as NumPy, SciPy, pandas, Librosa, NLTK, and SpaCy for data and text processing tasks. The parameter settings of the model in this study are detailed in Table 5.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Model parameter settings.

https://doi.org/10.1371/journal.pone.0329381.t005

In the experiment, we tuned the main hyperparameters to achieve the best model performance. The learning rate and batch size are selected by grid and random search within a specific range of values. Specifically, the learning rate is searched between 0.0001 and 0.01, and finally 0.001 is chosen as a balance between convergence speed and model stability. Similarly, the batch size is searched between 16 and 64, and 32 is chosen as a balance between memory consumption and training efficiency. Meanwhile, we observed the model performance on the validation set to select the hyperparameters that produced the lowest validation loss and the highest translation accuracy.According to the settings of the model parameters, the model is trained and tested in each sub-set. We use the same parameter settings for bilingual attention NMT model and context-aware NMT model. As shown in the experimental results, the model converges after 50 training epochs and the BLEU score increases gradually before it converges to a stable value. Moreover, the proposed model has a lower error rate and syntax error rate, which indicates that the proposed model has a good translation fluency. The synchronization speed and translation consistency of the proposed model are also better than those of the baseline model.

In this study, we strictly adhered to ethical guidelines for data use. All experimental data were sourced from publicly available academic databases, such as LibriSpeech and the TED Talk Dataset, and their use complied with the licensing agreements of the data providers. Additionally, to protect user privacy, we employed speech anonymization algorithms to ensure that speech data could not be identified as belonging to specific individuals. Furthermore, we conducted a Word Embedding Association Test to identify and address any potential biased vocabulary in the training data, thereby reducing bias in the model’s outputs. All these measures were implemented to ensure the ethical integrity of the research and the legitimacy of the data.

4.2. Evaluation indicators

1. (1) Translation accuracy

The BLEU score is employed to evaluate the accuracy of translation, using the similarity between the generated translation results and the reference translation as the primary evaluation criterion. A high BLEU score suggests a substantial agreement between the model’s interpretation effect and the reference translation [38,39]. The calculation formula is as follows:

(24)

Among them, there is:

(25)

In Formulas (24) and (25), is the length penalty factor; and represent the weight and precision of the n-gram, respectively; is the length of the candidate translation; is the closest translation length to in the reference translation.

1. (2) Translation fluency

The fluency of translation is assessed using two metrics: perplexity and grammar error rate. Perplexity, in particular, serves as an indicator of the fluency and naturalness of the text [40]. On the other hand, the grammar error rate quantifies the ratio of grammatical errors to the total word count. The formula for computing perplexity is as follows:

(26)

represents the total number of words in the sentence, and represents the logarithmic probability of the sentence.

1. (3) Synchronization speed

The synchronization speed is quantified using the metric “tokens per second,” which denotes the number of tokens processed each second. A higher value for this indicator corresponds to a more rapid synchronization speed in oral translation.

1. (4) Translation consistency

Translation consistency is quantified using TER, which assesses the editing distance between the translation output of each model and the corresponding reference translation. This metric indicates the frequency of required edits in relation to the total number of words in the reference translation.

1. (5) METEOR

Compared to BLEU, METEOR focuses more on the semantic similarity between the translation and the reference translation, rather than strict lexical matching. It can better reflect the quality of the translation, especially in terms of grammar changes, vocabulary usage, and other aspects. Its calculation formula is as follows:

(27)

4.3. Experimental results

1. (1) Comparison of translation accuracy

The variation of BLEU model under the number of training rounds in this article is shown in Fig 8.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. BLEU training changes.

https://doi.org/10.1371/journal.pone.0329381.g008

As illustrated in Fig 8, the BLEU score experiences a rapid ascent from 12.4 to 41.2 during the initial phases of training, thereby indicating the model’s swift adaptation to fundamental translation patterns. This early progress can largely be attributed to the model’s capacity for identifying prevalent language patterns from its original random state and for learning relatively straightforward syntactic and lexical alignment relationships. However, a deceleration in the growth of the BLEU score is evident from the 16th epoch onwards, with the rate of improvement gradually diminishing. This suggests that the model incrementally masters basic translation rules and starts addressing more intricate syntactic structures and semantic relationships. Overall, the model exhibits high training efficiency, achieving performance convergence within a confined number of training iterations.

The BLEU scores of the model in this article are compared with the bilingual attention NMT model and the context-aware NMT model in each test subset, and its ability to handle complex language structures and long-term dependencies is explored. The results are shown in Fig 9.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Comparison results of BLEU.

https://doi.org/10.1371/journal.pone.0329381.g009

As illustrated in Fig 9, the model presented in this article outperforms both the bilingual attention Neural Machine Translation (NMT) model and the context-aware NMT model across all test subsets in terms of BLEU score. Specifically, the average BLEU score for the model in this article is approximately 30.7, compared to 28.1 and 24.2 for the other two models, respectively. This represents an improvement of about 9.3% and 26.9% over these models, respectively. The bilingual attention NMT model struggles with effective integration of multimodal information, leading to issues such as incomplete oral translation. Similarly, the context-aware NMT model tends to overlook details during long sequence translations. In contrast, the model in this article harmoniously merges speech and text data, thereby leveraging a broad spectrum of information resources and enhancing translation comprehension for each test set, ultimately resulting in more accurate translations.

1. (2) Comparison of translation fluency

In the analysis of translation fluency, a comparison of the perplexity results and grammar error rate results of various models is made, as shown in Fig 10 and Table 5.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Comparison results of perplexity.

https://doi.org/10.1371/journal.pone.0329381.g010

The perplexity comparison illustrated in Fig 10 demonstrates that the model articulated in this article consistently outperforms its counterparts across all subsets, exhibiting notably lower perplexity results. A comparative analysis of the average perplexity reveals that the model discussed in this paper yields an average perplexity of approximately 11.9, whereas the translation models founded on bilingual attention NMT and context-aware NMT yield average perplexities of 13.1 and 14.0, respectively. This indicates that the model in this article is superior by margins of 9.2% and 15%, respectively. The attention mechanism has been optimized through linear variation within the article, thereby focusing attention across various text segments and assigning appropriate weights during translation to enhance text coherence. The relatively limited capacity of both the bilingual attention and context-aware NMT models to extrapolate distant information could potentially compromise the coherence and fluency of the translation.

In Table 6, the model presented in this article boasts an average grammar error rate of 2.5%, compared to the averages of 3.5% and 3.9% for the other two models. These comparative results indicate that the grammar accuracy of the translations produced by our model is superior. Our model consistently generates translations that are both natural and linguistically appropriate while maintaining fluency.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Comparison results of grammar error rate.

https://doi.org/10.1371/journal.pone.0329381.t006

1. (3) Comparison of synchronization speed

In evaluating synchronization speed, we utilize the metric of “tokens per second” to quantify the number of tokens processed per second by each model within the test set. The conclusive results are depicted in Fig 11.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. Comparison results of tokens per second.

https://doi.org/10.1371/journal.pone.0329381.g011

In Fig 11, a pronounced disparity is observed in the processing speed of tokens per second among the models evaluated in the test set. The model presented in this study processes an average of 265.7 tokens per second. In contrast, the bilingual attention and the context-aware NMT model process an average of 225.4 and 227.5 tokens per second, respectively. Notably, during oral translation tasks, the synchronization speed of the proposed model surpasses that of the other two models by 17.9% and 16.8%, respectively. While the bilingual attention and the context-aware NMT models typically operate in a serial mode characterized by limited parallelism—with each time step’s output being contingent on the preceding step’s state—the model in this study employs the DTW method to minimize the cumulative distance between two sequences. This strategy effectively reduces conversion delays and expedites the synchronization process, making its translation synchronization speed optimal across each test subset.

1. (4) Translation consistency

In the analysis of translation consistency, TER is used to compare and verify the translation effects of various models. The lower the TER, the higher the consistency of the translation. The final comparison results are shown in Fig 12.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 12. Comparison results of TER.

https://doi.org/10.1371/journal.pone.0329381.g012

In Fig 12, the average TER (Translation Edit Rate) of the model presented in this paper across each test subset is 15.6%. In comparison, the average TER values for the bilingual attention NMT (Neural Machine Translation) model and the context-aware NMT model are 18.9% and 20.2%, respectively. This suggests that the translations produced by the model in this study align more closely with the reference translations, indicating superior translation quality. The model in question employs a multi-head attention mechanism and captures the semantic nuances of spoken language across modalities using a context-awareness mechanism. This approach facilitates a deeper understanding of the meaning within the broader context. On the other hand, the bilingual attention NMT model encounters challenges in effectively incorporating contextual information, especially in longer sentences. Moreover, the attention mechanism of the context-aware NMT model has a limited capacity for expression, leading to suboptimal consistency performance.

1. (5) Comparison of METEOR

The METEOR results of various models in the same dataset are compared, as shown in Table 7:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. METEOR comparison results.

https://doi.org/10.1371/journal.pone.0329381.t007

As indicated by the METEOR comparison results in Table 7, the model discussed in this paper achieves a mean METEOR score of 55.8 across various test subsets. This surpasses the mean METEOR scores of 53.7 and 51.4 achieved by the bilingual attention NMT and context-aware NMT models respectively. In a detailed comparison against these other two model types, the model in this paper demonstrates an increase in the mean METEOR score by 3.9% and 8.6% respectively. These results suggest that the model examined in this paper performs more effectively in acknowledging context awareness and incorporating differing modal information. As such, it enhances the comprehension and translation of multimodal information, particularly demonstrating increased robustness and accuracy when faced with diverse and complex inputs.

1. (6) User feedback

This study employs a block-based self-attention mechanism, optimized for addressing computational efficiency issues in long sequence translation. The input speech signal is segmented into fixed 1.5-second windows, with the linear transformation method calculating the local attention for each window. To reduce algorithmic complexity, sparse attention is applied at a global scale. The model’s performance is validated in real-world scenarios by deploying it within a real-time English oral translation system used in business conferences to translate speeches from multiple parties simultaneously. In order to evaluate the model’s performance, data is collected on the inference time pre- and post-optimization as well as user assessments of translation accuracy and fluency. A group of 10 users who participated in the conference were asked to rate the model on a scale from 1–10, with higher scores indicating superior translation performance. The results are presented in Table 8.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. User feedback results.

https://doi.org/10.1371/journal.pone.0329381.t008

The user feedback results presented in Table 8 demonstrate that the optimization strategy, which is based on the block self-attention mechanism, significantly diminishes the time complexity of the model and substantially enhances the processing efficiency for extended speech inputs. Considering the average time per sentence, the model’s inference time has decreased by approximately 38.5% compared to pre-optimization, thereby improving real-time performance. In terms of user ratings, the average value indicates a substantial enhancement in both translation accuracy and fluency following optimization.

Based on the aforementioned considerations, we conducted interference testing on the model to ascertain its robustness in real-world settings. We introduced office environment noise, quantified by the signal-to-noise ratio (SNR), to the input speech. Concurrently, other human voices were integrated into the speech sample to assess the model’s capability in isolating the primary speech signal and subsequently translating it. Metrics such as the BLEU score, METEOR, and user subjective evaluation were employed to gauge the model’s performance. The findings are presented in Table 9.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 9. Interference test results.

https://doi.org/10.1371/journal.pone.0329381.t009

The comparative results before and after interference testing indicate a slight decrease in the BLEU score, METEOR, and user subjective evaluation of the model, following the addition of environmental noise and speech interference. However, the translation performance remains relatively robust, suggesting that the model’s translation performance is resilient.

1. (7) Comparison of advanced methods

In order to thoroughly assess the performance of the proposed model, this study employs a selection of cutting-edge and recent models for comparative experiments.

FairseqS2T is a speech-to-text translation model, leveraging the capabilities of the Transformer architecture, and demonstrates robust context capture capabilities.

ESPnet-ST is a sequence-to-sequence learning-based speech translation model, adept at translating various language pairs.

Utilizing this foundation, we conducted longitudinal comparisons among a variety of models. We employed newly collected translation data, which was gathered every three months, as well as the accumulated speech data, to train the respective models. The performance of these models was evaluated based on their BLEU, TER, and METEOR scores, using the same test dataset for each. The outcomes of these evaluations are presented in Table 10.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 10. Comparison results of advanced methods.

https://doi.org/10.1371/journal.pone.0329381.t010

In Table 10, from a horizontal comparison standpoint, the model in this article demonstrates BLEU, TER, and METEOR scores of 30.7, 15.6%, and 55.7 respectively when using the first set of data. In contrast, the Fairseq S2T yields BLEU, TER, and METEOR scores of 29.8, 16.3%, and 54.1 respectively, while ESPnet ST records 29.6%, 16.6%, and 53.7% respectively. From a longitudinal view, as more temporal data is introduced, the model in this paper shows marked improvements in BLEU, TER, and METEOR evaluations, effectively capturing a greater diversity of speech and translation details and enhancing translation accuracy and fluency. Both horizontal and vertical comparative experiments indicate that the model in this paper surpasses leading-edge models in terms of both BLEU and METEOR metrics, thereby showcasing exemplary translation performance.

1. (8) Experimental analysis of statistical tests

In order to strengthen the significance of the results, we conducted an experimental analysis of statistical tests, the results of which are shown in Table 11.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 11. Experimental results of statistical tests.

https://doi.org/10.1371/journal.pone.0329381.t011

Table 11 presents the BLEU scores, 95% confidence intervals, and p-values for the model discussed in this paper, juxtaposed against the bilingual attention NMT model and the context-aware NMT model across five test subsets. The findings indicate that the model’s BLEU scores consistently surpass those of the other two models in all test subsets. Moreover, with p-values all below 0.05, the enhancement in BLEU scores is statistically significant. Additionally, the 95% confidence intervals for the BLEU scores of the models discussed in this paper do not encompass zero, further bolstering the assertion that the observed improvements are not attributable to chance. These results suggest that the model introduced in this paper is superior in capturing the semantic nuances of spoken language and generating precise translations. The incorporation of p-values and confidence intervals lends greater validity to the experimental results, offering a more robust evaluation of the model’s performance.