Literature review

With the advancement of deep learning, multimodal information integration methods have made significant strides. Liu et al. (2024) proposed a multimodal Transformer architecture that employs a cross-modal attention mechanism to integrate speech, text, and visual data. This approach markedly enhanced the accuracy of emotion recognition tasks [9]. In parallel, the development of multimodal datasets has played a critical role in supporting research progress. The Kuhn-Munkres algorithm, originally designed for solving optimal matching problems in bipartite graphs, operates through a process of node labeling and path extension. It has been widely applied in domains such as task allocation and resource scheduling [10]. For instance, in transportation systems, the algorithm is used to optimize vehicle-to-order assignments to improve operational efficiency [11]. Despite these advancements, two key challenges remain in current multimodal fusion methods. First, most models rely on a unified embedding space across modalities, which limits their adaptability to tasks involving significant modality differences. Second, deep learning models often rely on implicit learning to assign modality weights during training. This lack of structural constraints reduces the interpretability of the final fusion results.

Despite its effectiveness in traditional matching tasks, the direct application of the Kuhn-Munkres algorithm to multimodal information integration presents several challenges. Most existing approaches assume fixed matching weights between modalities. However, in real-world scenarios, multimodal data are inherently heterogeneous, and the distribution of modality feature weights often shifts dynamically based on context [12,13]. The algorithm’s efficiency also declines significantly when handling large-scale, high-dimensional data, especially in complex tasks where additional constraints must be introduced. These factors lead to increased computational costs and reduced scalability [14]. To overcome these limitations, researchers have proposed various enhancements, such as heuristic search-based acceleration techniques [15]. However, these methods largely focus on fixed-weight conditions and offer limited support for scenarios requiring dynamic multimodal weighting. Current attempts to apply the Kuhn-Munkres algorithm in multimodal contexts often simplify modality data structures into standard bipartite graphs. Yet, multimodal data frequently involve asymmetric feature dimensions, temporal misalignments, and context-dependent relationships—factors that traditional algorithms struggle to model effectively. Moreover, the Kuhn-Munkres algorithm lacks mechanisms for dynamically assigning modality importance, which often results in unstable or semantically inconsistent matching outcomes in multimodal applications.

In summary, existing research still falls short in areas such as deep fusion architecture design, modality-specific weighting, and algorithm interpretability. In particular, there remains a critical gap between modeling dynamic modality importance and achieving structured matching. To address this, the present study proposes an extended Kuhn-Munkres-based framework tailored for multimodal tasks. This new approach retains the algorithm’s efficient matching capabilities while integrating a dynamic modality weighting mechanism and cross-modal constraints. It aims to resolve key challenges in ensuring consistency and adaptive integration across heterogeneous modalities.

Theoretical framework and methodological innovation

Based on an in-depth analysis of the inherent challenges in multimodal interaction tasks—namely strong heterogeneity, temporal misalignment, and complex inter-modal correlations—this study proposes an interpretable and structured theoretical framework for multimodal information integration. This framework extends the classic Kuhn-Munkres algorithm through the incorporation of two core mechanisms:

First, a dynamic weighted matching mechanism is introduced. The traditional Kuhn-Munkres algorithm relies on static edge weights, which limits its ability to accommodate variations in semantic importance across modalities in different contexts. To address this limitation, the study proposes a modality variance-based feature importance scoring method. This approach enables the dynamic adjustment of modality weights, allowing the matching process to adapt in real time to the relative contributions of speech, vision, and text within a given task. This mechanism enhances contextual adaptability and improves the algorithm’s generalizability under heterogeneous modality conditions.

Second, a modality correlation constraint mechanism is incorporated to mitigate semantic mismatches that may occur during the matching process. A cross-modal correlation matrix is constructed and introduced as a structural constraint within the algorithm. By integrating this matrix with the edge weights in the matching process, the semantic consistency of the results is significantly improved, and irrational pairings between weakly related modalities are effectively avoided. This approach embeds structural relationship modeling into the traditional graph matching workflow, marking a substantial extension of the Kuhn-Munkres algorithm for multimodal applications.

In summary, the theoretical innovation of this study lies in its integration of feature importance modeling and structural inter-modal constraints into a graph-based matching framework. The proposed method offers both interpretability and practical utility, serving not only as a theoretical foundation for algorithmic implementation but also as a modeling reference for task matching in complex multimodal scenarios.

Data sources and preprocessing

This study employs the CMU-MOSI and IEMOCAP datasets to ensure both scientific rigor and broad applicability in constructing and validating the multimodal information integration model. The CMU-MOSI dataset contains data from three modalities—speech, vision, and text—and is primarily used for multimodal sentiment analysis. It includes annotations of sentiment intensity in video-based comments, making it well-suited for investigating inter-modal coordination mechanisms and evaluating improvements in weighted matching methods. The IEMOCAP dataset also includes speech, vision, and text modalities, but its data originate from emotional expressions in human-to-human conversations, offering a more dynamic and realistic interaction context. This makes it an ideal dataset for testing the effectiveness of the improved Kuhn-Munkres algorithm in multimodal data matching scenarios.

The complementary characteristics of these two datasets allow the study to assess algorithm performance in both sentiment intensity analysis and dynamic interaction tasks, thereby enhancing the generalizability of the results. In both datasets, speech data are stored in audio format, visual data are represented as sequences of facial expression frames, and textual data consist of manually transcribed and annotated content [16,17]. These well-annotated and validated data types provide robust support for modeling modality feature importance and assigning dynamic weights during the integration process.

To ensure comprehensive coverage of different modal expressions, this study selected 500 annotated video clips from the CMU-MOSI dataset as training and testing samples. Additionally, 600 complete dialogue turns were selected from the IEMOCAP dataset, encompassing samples from various genders and emotional labels. During the experiments, each task was independently repeated five times using different random seeds to initialize model weights and data sequences, thereby minimizing sensitivity to initial conditions. All results are reported as mean values accompanied by standard deviations to reflect variability and ensure the reproducibility and robustness of the experimental findings.

To guarantee the consistency and effectiveness of multimodal data integration, several preprocessing steps were applied: noise reduction, feature extraction, and temporal alignment across modalities. For speech signals, noise removal was performed using Short-Time Fourier Transform (STFT) and spectral subtraction. These techniques enhance the signal-to-noise ratio by eliminating environmental noise and low-frequency interference [18,19]. Specifically, let x(t) represent the original audio signal and X(f) its corresponding frequency spectrum. By estimating the noise spectrum N(f), the denoised spectrum Xf can be obtained as shown in Equation (1):

(1)

Visual data denoising is carried out using median filtering, which effectively reduces random noise in frame sequences caused by lighting variations or sensor inconsistencies [20]. To enhance the robustness of facial expression feature extraction, optical flow techniques are further applied to detect changes in key regions across consecutive frames.

For the speech modality, feature extraction is performed using Mel-Frequency Cepstral Coefficients (MFCC), which simulate human auditory perception by mapping spectral energy onto the Mel scale. The Mel frequency m, corresponding to a given frequency f, is calculated as shown in Equation (2). By extracting MFCC feature vectors, the short-term spectral characteristics of speech signals can be accurately captured.

(2)

Visual features are extracted using keypoint detection techniques implemented with the Dlib library, which identifies critical facial landmarks. Based on these landmarks, features related to the Facial Action Coding System (FACS) are derived to represent facial expressions [21]. For the text modality, feature extraction is conducted using word embedding techniques. Specifically, the Bidirectional Encoder Representations from Transformers (BERT) model is employed to convert text into high-dimensional vector representations, allowing for the capture of rich contextual semantic information [22].

Temporal alignment of multimodal data is a key step in achieving information integration between modalities. In this study, the Dynamic Time Warping (DTW) algorithm is used for temporal alignment. DTW minimizes the distance between time series of different modalities, addressing the issue of inconsistent step sizes across modalities. Let the two modality time series be A= {a₁, a₂, …, a_n} and B= {b₁, b₂, …, b_m}, the DTW distance D(i, j) is given by Equation (3):

(3)

∥a_i − b_j∥ is the Euclidean distance between the features of the two modalities, and the final alignment path is derived using dynamic programming.

The denoising, feature extraction, and temporal alignment steps described above ensure high-quality data input for the subsequent development and validation of the multimodal information matching algorithm.

Improvement of the Kuhn-Munkres algorithm

The Kuhn-Munkres algorithm is a classical approach for solving maximum-weight bipartite graph matching problems. Its main objective is to identify an optimal pairing between vertices in a bipartite graph such that the total matching weight is maximized, as illustrated in Fig 1.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Structure of the Kuhn-Munkres Algorithm.

https://doi.org/10.1371/journal.pone.0326662.g001

Consider a bipartite graph G= (U, V, E), where U and V are two disjoint sets of vertices, and E represents the set of edges connecting them. Each edge e(u,v) has an associated weight w(u,v). The goal of the Kuhn-Munkres algorithm is to traverse the graph and identify an optimal matching that maximizes the total weight of selected edges. The algorithm operates by constructing an auxiliary graph and iteratively updating vertex labels through a process of path extension and adjustment. Let L(u) and R(v) represent the labels of vertices u ∈ U and v ∈ V, respectively. These labels satisfy a constraint that ensures the matching condition holds.

During each iteration, the labels are updated to guide the matching process toward optimality. The Kuhn-Munkres algorithm has a time complexity of O(∣U ∣²⋅∣V ∣), making it efficient for solving standard bipartite graph matching problems [23,24]. However, in the context of multimodal interaction design and information matching, the traditional algorithm lacks the capability to handle weighted allocation across multiple modalities [25,26].

To address this limitation, this study proposes an enhanced version of the Kuhn-Munkres algorithm by introducing a multimodal weighting mechanism. In this framework, multimodal features are treated as attributes of the vertices, and the edge weights are dynamically adjusted based on the relative importance of each modality. Let the multimodal feature set be {M₁, M₂, …, M_k}. The revised edge weight calculation is defined in Equation (4):

(4)

α_i is the weight of modality M_i, and fi(u,v) is the similarity function between modality features. By reasonably allocating the weight α_i, the model can more precisely reflect the integration needs of multimodal information.

Before each matching round begins, the system extracts a set of features from the current input batch and calculates the variance of each modality’s features. This variance serves as the basis for determining the importance score of each modality. Through normalization, a modality weight vector α = {α₁, α₂,..., α_k} is generated for the current iteration. Subsequently, for each candidate matching pair (u,v) in the bipartite graph, the similarity scores f_i(u,v) are computed for all modalities. These scores are then combined using the corresponding weights to produce the final weighted edge score.

To further improve the algorithm’s adaptability across different scenarios, this study introduces a dynamic weight adjustment mechanism. The value of each α_i is adaptively updated based on the calculated importance of each modality. This importance is quantified by evaluating both the discriminability and confidence of the modality features [27,28]. The detailed formulation is presented in Equation (5):

(5)

Var(f_i) is the variance of modality M_i’s feature, representing its contribution to the matching result. The dynamic adjustment mechanism allows for real-time weight adjustment for each modality based on the distribution of multimodal data, thus improving the matching accuracy of the algorithm.

After each graph matching round, the system analyzes the discriminability of each modality’s feature pairs along the matched paths. Based on this analysis, it recalculates the variance of each modality and updates the weight vector α. To balance convergence speed and computational cost, the weight update is performed at fixed intervals (e.g., once every 10 rounds). At each update, variances are recomputed using a sliding data window and then normalized. This “window-based adaptive” update strategy ensures that the weight adjustment mechanism remains responsive to changes in the data, maintaining optimal matching performance throughout task progression.

Multimodal data often exhibit intrinsic correlations—for example, speech and facial expressions frequently co-express emotional states [29,30]. To effectively leverage this inter-modal correlation during the matching process, this study introduces a cross-modal correlation matrix C as a structural constraint. The final matching results are required to satisfy the condition defined in Equation (6):

(6)

denotes the cross-correlation between modalities M_i and M_j, and ε is a predefined correlation threshold. By introducing this constraint, the algorithm filters out semantically inconsistent matches between modalities, thereby enhancing the robustness of the matching process. Specifically, during the construction of the edge weight matrix, the algorithm incorporates the cross-modal correlation matrix C. For any candidate match (u,v), it first retrieves the correlation score C(i,j) for the associated modality pair M_i and M_j. The edge weight is retained only if the product exceeds the threshold . Otherwise, the candidate edge is either assigned a zero weight or excluded entirely, preventing ineffective matches between weakly correlated modalities. The pseudocode for the improved algorithm is provided in S1 Appendix.

Experimental design

To comprehensively assess the practical performance of the improved Kuhn-Munkres algorithm in multimodal interaction design, this study conducted two experimental evaluations: Information Matching Efficiency Testing and a User Interaction Satisfaction Survey.

To ensure a robust performance comparison, five state-of-the-art multimodal data matching algorithms were selected as baseline methods:

Traditional Kuhn-Munkres Algorithm: Serves as the foundational baseline to evaluate the improvements achieved by the proposed method.

Deep Canonical Correlation Analysis (DeepCCA): A deep learning-based method for modeling cross-modal correlations, used to assess the effectiveness of semantic correlation modeling.

Multimodal Tensor Fusion (MMTF): Utilizes tensor decomposition for multimodal feature fusion, included to compare fusion efficiency.

Graph-Matching Network (GMN): A graph neural network-based approach for multimodal graph matching, used to benchmark structural modeling capabilities.

Cross-Attention Transformer (CAT): A method leveraging cross-modal attention mechanisms, selected to evaluate the effectiveness of dynamic weight adjustment in collaborative modality optimization.

To comprehensively evaluate each algorithm, five performance metrics were employed, encompassing matching accuracy, computational efficiency, and user experience:

Matching Accuracy: Evaluates the precision of multimodal information matching. The calculation is provided in Equation (7).

(7)

Matching Time: Represents the duration required for the algorithm to complete a full matching task, measured in seconds. This metric reflects the algorithm’s computational efficiency.

Computational Complexity: Evaluates the theoretical time complexity of the algorithm using Big-O notation, providing insight into its scalability and performance in large-scale scenarios.

Information Integration Efficiency: Combines both matching accuracy and execution time to produce a comprehensive score that reflects the algorithm’s overall efficiency in integrating multimodal information.

User Satisfaction Score: Reflects user experience based on responses to a standardized interaction satisfaction survey, with ratings on a scale from 1 to 10.

The hyperparameter configurations used in this study are detailed in Table 1.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Hyperparameter settings.

https://doi.org/10.1371/journal.pone.0326662.t001

Based on the configuration in Table 1, an initial weight value of 0.5 was selected to ensure balanced influence among all modalities at the beginning of the matching process. Subsequent experiments demonstrated that the dynamic adjustment mechanism rapidly optimized the weight distribution, making it unnecessary to skew the initial values toward any specific modality. The correlation threshold was established through analysis of the cross-correlation matrix. Statistical evaluation of the correlation distribution among modality pairs indicated that a threshold of 0.3 effectively filtered out low-correlation matches while retaining meaningful associations. Experiments with varying step sizes revealed that a value of 0.05 offered the optimal trade-off between convergence speed and matching accuracy. The algorithm typically converged within 200 iterations; thus, 200 was set as the maximum iteration count. Sensitivity analyses were conducted across a range of 50–500 iterations to validate this choice. When comparing Euclidean distance and cosine distance for similarity measurement, Euclidean distance demonstrated superior performance in handling numerical features such as MFCC and facial keypoints. Therefore, it was adopted as the default metric for this task.

Hyperparameter tuning results

The hyperparameter tuning process is illustrated in Figs 2 and 3.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Initial weight values and correlation thresholds.

https://doi.org/10.1371/journal.pone.0326662.g002

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Adjustment step size.

https://doi.org/10.1371/journal.pone.0326662.g003

In Fig 3, when αᵢ = 0.5, the matching accuracy and information integration efficiency were optimal. Additionally, the user satisfaction score was also at its peak, and the computational complexity remained moderate. Therefore, αᵢ = 0.5 was selected as the default initial value. When ∊ = 0.3, all metrics performed at their best, with the highest matching accuracy and user satisfaction scores, while maintaining moderate computational complexity. Thus, ∊ = 0.3 was chosen as the final threshold. For Δα = 0.05, matching accuracy reached its highest value, the convergence iteration count was low, and both information integration efficiency and user satisfaction were optimal. Therefore, Δα = 0.05 was selected as the optimal adjustment step size. Considering the balance between accuracy and runtime, T_max = 200 was chosen as the default value. Based on these experimental results, Euclidean distance was selected as the default distance metric.

Matching efficiency and accuracy

To ensure the robustness of the results, all performance tests employed 5-fold cross-validation, with random partitioning of the training and validation sets in each round. The accuracy, runtime, and information integration efficiency of each method were independently recorded for each fold. The final results were reported as averages with standard deviations. The error bars in Fig 4 represent the standard deviation from the 5-fold cross-validation. To assess the performance of the improved Kuhn-Munkres algorithm in multimodal information matching tasks, this study compared its matching accuracy and information integration efficiency against five benchmark methods. These include the traditional Kuhn-Munkres algorithm, DeepCCA, MMTF, GMN, and CAT. These benchmark methods represent state-of-the-art techniques in the multimodal matching field, covering a range of paradigms from traditional algorithms to deep learning-based approaches. The experimental data were derived from the CMU-MOSI and IEMOCAP datasets, with the test tasks covering speech-visual, speech-text, and full-modal matching. The performance of each method was evaluated in terms of matching accuracy and information integration efficiency, and the results from multiple rounds of experiments are shown in Fig 4.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Comparison of matching accuracy and information integration efficiency across different methods.

https://doi.org/10.1371/journal.pone.0326662.g004

In Fig 4, the improved Kuhn-Munkres algorithm achieved a matching accuracy of 87.7%, marking a 28.2% improvement over the traditional Kuhn-Munkres algorithm. When compared to other advanced algorithms, the improved method outperformed DeepCCA, MMTF, GMN, and CAT in matching accuracy, exceeding them by 15.3%, 12.9%, 9.6%, and 5.8%, respectively. This demonstrates a significant advantage in multimodal matching tasks. Regarding information integration efficiency, the improved algorithm showed an 18.7% improvement, outperforming DeepCCA (8.5%), MMTF (10.2%), GMN (12.7%), and CAT (15.3%). These results suggest that the combination of dynamic weight adjustment and cross-modal correlation constraints greatly enhanced the algorithm’s performance in complex multimodal tasks. In multimodal combination tasks, such as speech-visual or full-modal matching, dynamic weight adjustment for the speech modality further improved matching accuracy. Additionally, strengthening the robustness of the visual modality contributed to better integration efficiency. Notably, in full-modal tasks, the improved algorithm’s integration efficiency was 21.3% higher than that of the traditional Kuhn-Munkres algorithm.

To ensure that the improvement in matching accuracy achieved by the modified algorithm is statistically significant, a 5-fold cross-validation scheme was employed. Statistical tests were performed on the matching results of each algorithm in each fold, as shown in Table 2:

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Statistical test results for multimodal matching accuracy (based on 5-fold cross-validation, n = 5).

https://doi.org/10.1371/journal.pone.0326662.t002

As shown in Table 2, the improved algorithm achieved an average matching accuracy of 87.7% with a standard deviation of ±0.53, significantly outperforming all baseline methods. Compared to the CAT method, the difference was statistically significant (t = 6.14, p = 0.002), with a large effect size (Cohen’s d = 1.54). When compared to the traditional Kuhn-Munkres algorithm, the improvement in matching accuracy had an effect size of 3.58. This indicates that the improvement is both statistically significant and of substantial practical importance.

Algorithm computational complexity analysis

To assess the computational efficiency of the improved algorithm, this study compared its theoretical complexity and actual runtime with five benchmark methods: traditional Kuhn-Munkres, DeepCCA, MMTF, GMN, and CAT. The traditional Kuhn-Munkres algorithm has a theoretical complexity of O(n³). In contrast, the proposed improved algorithm reduces certain operations to O(n²logn) by incorporating dynamic weight adjustment and cross-modal correlation matrix constraints. This optimization leads to a reduction in computational costs. The algorithm’s runtime was tested using multimodal matching tasks across five datasets of varying sizes to evaluate its efficiency at different scales, as shown in Fig 5.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Comparison of computational complexity and actual running time across different methods.

https://doi.org/10.1371/journal.pone.0326662.g005

In Fig 5, the theoretical computational complexity of the improved algorithm was optimized to O(n²logn), similar to GMN. However, its combined optimization of dynamic weight adjustment and inter-modality correlation constraints resulted in better actual runtime performance. For sample sizes of 100, 500, and 1000, the running times of the improved algorithm were 0.58 seconds, 2.89 seconds, and 5.73 seconds, respectively. Compared to the traditional Kuhn-Munkres algorithm, the improved algorithm reduced the average runtime by 15.4%, showing a significant improvement in computational efficiency, particularly for larger datasets.

To further explore how dataset size affects runtime, this study plotted the relationship between dataset size and runtime, as shown in Fig 6.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Curve of data size vs. running time.

https://doi.org/10.1371/journal.pone.0326662.g006

The results in Fig 6 show that, as the dataset size increases, the improved Kuhn-Munkres algorithm consistently outperforms other methods in terms of runtime. Particularly with larger datasets, the algorithm’s runtime growth is significantly slower, demonstrating excellent scalability. Compared to the traditional algorithm, the average runtime is reduced by 15.4%, fully validating the practical improvements of the optimization.

To further explain the underlying mechanisms of the observed experimental phenomena, this study conducts a theoretical complexity breakdown of the overall structure and the computational overhead of each module in the improved algorithm. First, during the multimodal graph construction, the algorithm must compute the modality feature similarity for all node pairs and perform weighted fusion. Since the number of modalities is fixed, the overall computational load is primarily determined by the number of node pairs, leading to a typical quadratic cost. Second, the dynamic weight mechanism updates the importance scores for each modality at fixed intervals, requiring minimal computational time, and thus has a negligible overall impact. Third, the introduction of the inter-modality correlation matrix as a constraint for potential matching paths further improves the matching accuracy but is computationally a lightweight operation. Finally, the core matching step has been structurally optimized from the original Kuhn-Munkres algorithm, reducing its time complexity from cubic to approximately quadratic-logarithmic, significantly lowering time consumption under large data conditions. Overall, the time complexity of the proposed improved algorithm is better than that of traditional methods. Both theoretical and experimental evaluations demonstrate its superior matching performance and enhanced scalability. This structural optimization not only applies to the current task scenario but also provides a universal computational foundation for multimodal large-scale model matching.

User interaction satisfaction survey results

To assess improvements in user interaction, this study designed multimodal interaction tasks and gathered user satisfaction scores through a questionnaire. The scoring range for the questionnaire was from 1 to 10, and 30 volunteers participated in the experiment. Each volunteer completed 10 sets of multimodal interaction tasks, including speech, visual, and text modality matching, followed by providing feedback. The improved algorithm showed a 19.5% increase in user satisfaction scores compared to the traditional Kuhn-Munkres algorithm and other benchmark methods. The detailed statistical data are shown in Fig 7. Similar to Fig 4, all performance tests in Fig 7 used 5-fold cross-validation to ensure the robustness of the results. In each round, the training and validation sets were randomly divided. The error bars in Fig 7 represent the standard deviation from the 5-fold cross-validation.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. User satisfaction scores for different algorithms.

https://doi.org/10.1371/journal.pone.0326662.g007

In Fig 7, the improved algorithm’s advantages in feedback speed and matching accuracy were the main reasons for the increased user satisfaction. Its average satisfaction score was 8.9, with a standard deviation of only 0.38, indicating high consistency in the ratings.

To comprehensively evaluate the performance of the improved algorithm in user interaction experience, this study compared the satisfaction scores of six methods in multimodal tasks and conducted statistical significance analysis using an independent sample t-test. The results are shown in Table 3:

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. User satisfaction score statistical analysis (n = 30).

https://doi.org/10.1371/journal.pone.0326662.t003

As shown in Table 3, the improved Kuhn-Munkres algorithm achieved the highest satisfaction score (8.9 ± 0.38). The difference between the improved algorithm and all benchmark methods is statistically significant (p < 0.01), with effect sizes exceeding 1.0, indicating substantial practical differences. In particular, compared to the traditional Kuhn-Munkres algorithm, the effect size was as high as 3.77, demonstrating that the algorithm’s improvement has a strong impact on user experience.

Ablation experiment

This study designed four models for ablation experiments:

• Model A (Full Model): Complete improved algorithm, including dynamic weight adjustment and correlation constraints.
• Model B (No Dynamic Weight Adjustment): Fixed weights with correlation constraints.
• Model C (No Correlation Constraints): Dynamic weight adjustment with no correlation matrix.
• Model D (No Improvements): Fixed weights with no correlation constraints (degraded to traditional Kuhn-Munkres).

The results of the ablation experiment are compared in Table 4, with data sourced from 5-fold cross-validation averages:

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Ablation experiment results comparison.

https://doi.org/10.1371/journal.pone.0326662.t004

As shown in Table 4, Model A achieved the best performance across all metrics, with a matching accuracy of 87.7%, information integration efficiency of 84.9%, a satisfaction score of 8.9, and stable runtime. When the dynamic weight adjustment mechanism was removed (Model B), the accuracy dropped to 81.3%, and the satisfaction score decreased by 0.8. Removing the correlation constraints (Model C) also led to a performance drop, although to a lesser extent than Model B, indicating that dynamic weight adjustment had a larger impact on matching accuracy. Model D performed the worst, with significantly lower results in all metrics compared to the improved model (p < 0.01), validating the necessity of both modules. Overall, the ablation experiment results demonstrate that the dynamic weight mechanism and the inter-modality correlation constraint play significant and complementary roles in enhancing the algorithm’s performance. Their combination enables a comprehensive optimization of accuracy, efficiency, and user experience.