Methods based on static knowledge

Initially proposed by Hinton et al. in 2015 [11], Knowledge Distillation (KD) is generally considered an efficient model compression technique. Its core idea is to use a larger teacher model to guide the training of a smaller student model. In subsequent research, scholars have proposed various forms of teacher knowledge, enabling students to mimic different aspects of the teacher, mainly divided into softened logits and intermediate features. For example, earlier KD methods encourage students to mimic the logits of the teacher softened by temperature [9,11,15,20,21]. However, while Knowledge Distillation (KD) ensures the stability of knowledge transfer by leveraging supervision from a pre-trained teacher model, the static nature of such knowledge inherently constrains the generalization capability of the student model. The current state-of-the-art methods are based on intermediate feature knowledge [22–29]. For instance, FitNet [28] directly transfers the teacher’s features to the student, while other approaches transfer the correlations between samples captured by the teacher to the student [23]. Additionally, NRKD (A new similarity-based relational) [30] transfers knowledge of neighborhood relationships by selecting the K-nearest neighbors for each sample based on similarity metrics. ReviewKD (via knowledge review) [29], first proposed matching feature information through cross-stage connection paths. However, most intermediate feature-based knowledge distillation methods require consistent feature resolutions between the teacher and the student. This requirement implies that the teacher and student must either share the same model structure or rely on complex feature alignment modules [31]. Therefore, this study utilizes the softened logits to jointly learn static and dynamic knowledge rather than solely relying on the teacher model.

Methods based on dynamic knowledge

Deep Mutual Learning (DML) [15] proposed an online knowledge distillation method that does not require a pre-trained teacher model, using the simultaneous training of multiple student models to enable them to learn from each other. To reduce training costs and complexity, ONE [16] uses a single multi-branch model and employs a gating module to aggregate branch predictions, generating a dynamic teacher for knowledge transfer. However, multi-branch architectures suffer from homogenization issues due to insufficient diversity in the acquired knowledge across branches. Subsequent research has further enhanced the stability and diversity of online distillation. For example, OKDDip (Online knowledge distillation with diverse peers) [32] utilizes a self-attention mechanism to derive a separate target for auxiliary peers, increasing diversity during training and employing second-level distillation to transfer the knowledge from aggregated predictions to the target student. PCL (Peer collaborative learning) [33] uses the temporal average model of peers as an ensemble teacher and transfers knowledge among students through two different ensemble strategies, making the optimization more stable. However, the student model is susceptible to noise in the early stages of training, which may lead to the aggregation of incorrect information in the dynamic knowledge. In the later stages, dynamic knowledge can further optimize the model, so the loss weights are adjusted to control the impact of dynamic knowledge at different stages.

Multi-branch architectures and feature fusion

Compared to training multiple independent models in DML [15], ONE [16] uses a multi-branch model architecture. This architecture reduces the parameters and avoids the asynchronous updates of independent models, simplifying the distillation process. However, in multi-branch architectures, the knowledge acquired by each branch during the training process suffers from limited diversity, resulting in a homogenization problem that ultimately constrains the model’s performance. This phenomenon occurs because parallel branches tend to converge toward similar feature representations under shared optimization objectives. Additionally, feature fusion can effectively utilize the feature information from each branch, further enhancing the representational capability of the model. For example, FFL (Feature fusion for online mutual knowledge distillation) [34] applies multiple pointwise convolutions to features fused along the channel dimension, which applies to any model architecture and improves the performance of both the fused classifier and each student model. AFF (Attentional feature fusion) [35] focuses on feature information from different dimensions and assigns different weights to this information using an attention mechanism. However, most methods remain constrained by the absence of efficient fusion mechanisms, leading to suboptimal utilization of inter-branch information. Compared to traditional feature fusion methods, Concat results in many parameters due to linear combinations, while average fusion ignores differences in feature contributions. Our proposed heterogeneous branches differ from the target branch regarding model depth and feature representation. The progressive feature fusion module, designed with a spatial attention mechanism and a hierarchical progressive structure, decomposes the global fusion problem into multiple local fusion problems, thereby effectively improving feature utilization efficiency and feature quality.

Notations

The student model adopts a multi-branch architecture where each branch shares parameters in the low-level layers but has independent parameters in the high-level layers, functioning as multiple parallel-trained students. Branch-1 is designated as the target branch, forming the baseline model together with the low-level layers. We denote the total number of branches as B, the branch index as , and the features generated by the branch as F_b.

Given a dataset with K classes, where , and a training sample x with its corresponding one-hot label , where and , when x is input into the student model, the output probability of the b-th branch for class k is calculated as:

(1)

where ] is the logit of the b-th branch. For multi-class classification tasks, training is performed by minimizing the cross-entropy loss between the predicted vector and the ground-truth labels:

(2)

where N represents the number of samples, y_i,k is the ground-truth label (one-hot encoded) of sample i, and p_i,k is the predicted probability by the model that sample i belongs to class k.

Knowledge distillation

The core of traditional knowledge distillation (KD) methods is to use the softened teacher outputs as the supervisory signal for training the student model. The formulas for the softened teacher and student output probabilities are as follows:

(3)

where and represent the softened outputs of the teacher and student, with being the temperature parameter. Then, the Kullback-Leibler divergence between and is minimized, and the distillation loss is calculated as follows:

(4)

the total loss for knowledge distillation is calculated as follows:

(5)

As the temperature increases, the gradient of the distillation loss decreases. Therefore, L_kl is multiplied by to ensure that the gradient contributions of the cross-entropy loss and the distillation loss remain roughly consistent despite changes in temperature [11].

Heterogeneous branch structure

The student model is trained using a multi-branch structure. However, due to the consistent construction among branches, each student’s capabilities are identical, leading to the homogenization problem during training, which hinders the diversification of dynamic knowledge. Therefore, this study designs heterogeneous branches by adding additional auxiliary blocks to the target branch structure to construct students with different capabilities. As shown in the Heterogeneous Branch section of Fig 2, Branch-1 is set as the target branch. In contrast, other branches gradually add auxiliary blocks with channels, where C is the number of channels in the high-level layers. Thus, the channel count of the auxiliary blocks increases in linear multiples, aiming to capture differentiated feature maps and providing a foundation for the diversity of collaborative learning. Let denote the feature map input to the auxiliary block, where C represents the number of channels in the feature map. The specific formulation is as follows:

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. An overview of two-stage optimization based on heterogeneous branch fusion for knowledge distillation (THFKD).

(1) It consists of pre-trained teacher and multi-branch student models. The teacher model guides each branch to form the L_tg loss. (2) The student model constructs heterogeneous branches by adding auxiliary blocks and introduces PFF for progressive fusion. The Collaborative Learning Target comprises Fusion Logit and Ensemble Logit, with losses denoted as L_fl and L_el, respectively. (3) The logits of the four branches are concatenated along the branch dimension, and the loss for each branch is computed concerning the teacher logit, collaborative learning target, and labels.

https://doi.org/10.1371/journal.pone.0326711.g002

(6)

Specifically, the auxiliary blocks are encapsulated as a basic unit that primarily includes two 1 1 convolutional layers and one 3 3 convolutional layer. Where Conv1 is employed to increase the number of channels (C − C_A), Conv2 extracts higher-level feature information through the expanded channels, and Conv3 uniformly restores the channel count to C. The output is then combined with the original feature map via a residual connection to mitigate overfitting. Auxiliary blocks with a gradually increasing number of channels are added to the branches, creating differences in receptive field expansion and feature dimensions. This design forces different branches to learn complementary feature representations: shallow branches learn the fundamental features of samples, while deep branches capture subtle differences.

For a more straightforward presentation, we take ResNet32 architecture, for example, to elaborate on how to add the auxiliary blocks. The ResNet model is structured into three stages: the first two stages are low-level layers shared by each branch, and after the high-level layers in the third stage, auxiliary blocks are added to construct heterogeneous branches. In ResNet32, the number of channels in the third stage is 64, so the auxiliary blocks added to the b-th branch have channels.

Progressive feature fusion module

Efficient fusion methods can enhance the utilization of features. AFF [35] suggests that the initial quality of feature fusion profoundly impacts the final fusion weights when attention mechanisms are introduced. Therefore, feature fusion modules can recursively integrate received features with another fusion module [8]. The heterogeneous branches designed in this work increase the network depth and capture feature information with different receptive fields by multiplying the number of channels, providing a rich feature basis for the fusion of global and local information. As shown in Fig 2, the Feature Fusion section includes three modules: PFF-1, PFF-2, and PFF-3. THFKD employs a progressive fusion approach to gradually integrate features, effectively improving fusion quality compared to simultaneously fusing features from B branches.

The Progressive Feature Fusion (PFF) module is shown in Fig 3. First, the features F_b from the b-th branch are initially fused with the fused features from the previous level. The initial fused features are then processed through two paths: one path includes two 1 1 convolutional layers to capture local information. In contrast, the other path employs an additional global pooling layer to capture global information. A Sigmoid function activates the features from both paths to obtain weight scores. These weight scores are then used to multiply the features F_b and , with parameter g adjusting the importance of each branch, g is a scalar parameter with a value range of 0 to 1, when g approaches 1, the current branch features dominates the fusion output, when g approaches 0, the fused features from the previous level receive higher weight. In our method, g is set to 0.5, ensuring equal importance of both current branch features and prior fused features, ultimately resulting in the fused features for this level.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Progressive feature fusion module.

The incoming features are processed through two distinct paths, and an attention mechanism (sigmoid) generates weight scores (score) to capture important information.

https://doi.org/10.1371/journal.pone.0326711.g003

The feature map of the first branch is used as the initial input fused feature, i.e., . The expression for the features outputted by subsequent progressive fusion is:

(7)

(8)

where is the fusion function from the Progressive feature fusion module, which is iteratively utilized throughout the training process to progressively amalgamate features. For instance, with the number of branches in the model set to four, the ultimate fused feature is denoted as , where the fusion process is articulated as .

Collaborative learning target

This study designs a collaborative learning target to transfer diversified dynamic knowledge effectively. The collaborative learning targets consist of two parts: (1) Fusion Logits, which are obtained from the fused features ; and (2) Ensemble Logits, which are derived from the integration of branch logits. The Fusion Logits and Ensemble Logits are then softened using Eq (3) to obtain and , respectively. The two types of distillation losses are calculated as follows:

(9)

(10)

where L_fl and L_el represent the KL divergence between each branch and the Fusion Logits and Ensemble Logits, respectively. Therefore, the total loss for the collaborative learning targets is:

(11)

Teacher guidance

The pre-trained teacher can provide stable and comprehensive guidance, using softened logits to transfer knowledge to the student. Similar to Eq (9), the distillation losses for all branches with respect to the teacher are represented as:

(12)

Two-stage optimization and training

This study introduces a ramp-up weight [36] and designs a two-stage optimization strategy to control the weight of the distillation loss at different stages:

(13)

where E determines the duration of the weight change period, and i is the i-th epoch with the rate of weight increase being smoothed through the empirical application of a factor of -5. During the training process, the initial parameter a is utilized to regulate the epoch at which the ramp-up weight commences, with the setting range of a being between 0 and 300. Finally, the overall loss function for THFKD is represented as:

(14)

where is the cross-entropy loss between the b-th branch and the ground-truth label y. During the early stages of training, the parameters of the student model are not yet stabilized and are prone to being affected by noise. Therefore, the initial value of w(i) is set to a relatively small magnitude, and the weight is gradually increased over a predetermined period to guide the student model’s optimization smoothly. In Eq (14), the loss term L_tg dominates, enabling the pre-trained teacher model to provide more comprehensive guidance in the initial phases. When i = E, the period ends, and the weight is set to 1. At this point, the student has gained a more comprehensive understanding, and diversifying the collaborative learning target becomes more important in the subsequent training. During training, the teacher model’s parameters remain unchanged. Taking ResNet32 on CIFAR-100 as an example, the shape of the concatenated logits of all branches is 100 4 (where 4 is the number of branches B), and the loss for each branch is computed along the branch dimension. The performance of the model is primarily evaluated by classification accuracy(%), which represents the proportion of correctly classified samples to the total number of samples across multiple datasets.

Experimental setups

Datasets and model architectures. The THFKD method was evaluated on two widely used datasets: CIFAR-100 [37] and Tiny-ImageNet [38]. The CIFAR-100 dataset consists of 100 classes with a total of 60,000 RGB images of size 32 32, with the training set containing 50k images and the validation set containing 10k images. The Tiny-ImageNet dataset contains 200 classes with 20,000 RGB images of size 64 64. It includes 100,000 training images and both 10,000 testing and 10,000 validation images. All images were processed and normalized using channel-wise mean and standard deviation, and standard data augmentation techniques were applied. The experiments used various model architectures, including ResNet [1], WideResNet [39], MobileNet [40], and ShuffleNet [2].

Settings. Optimization is performed using stochastic gradient descent with Nesterov momentum, with an initial learning rate set to 0.05 and momentum to 0.9 [29]. Training is conducted for 300 epochs with a batch size of 128. The learning rate is divided by 10 at epochs 150 and 225, weight decay is set to 5E-4, and the temperature is set to 3. The number of branches B for the THFKD experiments is 4 [33]. All experiments are conducted using the PyTorch framework on an NVIDIA 4090 GPU.

Comparison with KD methods

Results on CIFAR-100. The experiments on CIFAR-100 demonstrate that THFKD consistently outperforms cross-entropy-trained students by at least 3% across various architectures, as shown in Table 1. Compared to SOTA methods, THFKD achieves superior performance with identical teacher-student architectures. For instance, it surpasses ReviewKD by 1.52% with ResNet110-ResNet32 and by 0.66% with ResNet56-ResNet20. Different architectures impact knowledge distillation performance. High-capacity model pairs, such as ResNet32x4-ResNet8x4 and WRN40-2-WRN16-2, are harder to improve due to overfitting with limited data. Using heterogeneous architectures, THFKD and ReviewKD achieved comparable performance on ResNet50-MobileNetV2 and ResNet32x4-ShuffleNetV2, limited by the incompatibility of ShuffleNet with the multi-branch architecture.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Accuracy (%) of KD methods on CIFAR-100.

https://doi.org/10.1371/journal.pone.0326711.t001

Result on Tiny-ImageNet. To further evaluate the generalization capability of the THFKD method, we conducted experiments on the large-scale Tiny-ImageNet dataset. As shown in Table 2, THFKD using ResNet34 as the teacher model and ResNet18 as the student model outperforms Vanilla KD by 1.98%. Compared to previous methods, THFKD surpasses the SOTA method NRKD by 0.71%, achieving the best performance. This demonstrates that the proposed method is also effective on large-scale datasets, and the diverse knowledge provided by the THFKD architecture can better enhance the student’s generalization ability.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Accuracy (%) of KD methods on Tiny-ImageNet. ResNet34-ResNet18.

https://doi.org/10.1371/journal.pone.0326711.t002

Result on CIFAR100-LT. A long-tailed dataset has a highly imbalanced sample distribution, where the number of samples in a few categories is significantly larger than that in most categories. As shown in Table 3, although THFKD is not explicitly designed for long-tail classification, it has demonstrated strong classification performance on the long-tailed dataset CIFAR100-LT. The heterogeneous branches designed in this paper allow each branch to specialize in different categories and data characteristics, thereby enhancing the model’s ability to capture information from tail classes. Building on this architecture, our progressive feature fusion module enables effective integration of differentiated features, while the knowledge distillation mechanism facilitates knowledge sharing and complementarity specifically for tail categories. This indicates that the student model is capable of learning comprehensively across all categories, avoiding bias toward the more frequent classes in the early stages of training, and further optimizing the model with diversified knowledge to enhance its generalization ability, thereby improving the classification performance of the model on the tail categories.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Accuracy (%) on CIFAR100-LT with imbalance factor of 100 and 50. ResNet110-ResNet32.

https://doi.org/10.1371/journal.pone.0326711.t003

Ablation experiments

Effect of different loss.

Different combinations of loss impact classification accuracy, as shown in Eq (11):  +  L_el. As shown in Fig 4, training with the collaborative learning target (L_cl) loss yields a 1.17% improvement in classification accuracy compared to training with the teacher-guided (L_tg) loss. This indicates that the collaborative learning target provides diverse dynamic knowledge during training, enhancing the student’s generalization ability. However, training with only static or dynamic knowledge performs less than joint training. Using and losses leads to an improvement of approximately 0.4%. Ultimately, training with the proposed THFKD (L_cl  +  L_tg) approach through the two-stage optimization strategy provides appropriate knowledge to the student model at different stages, achieving the best performance.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Effect of different loss on accuracy.

https://doi.org/10.1371/journal.pone.0326711.g004

Effect of loss in different stages.

We analyze the impact of loss at different stages by visualizing the changes in accuracy and loss curves. As shown in Fig 5(a), during the early phase (the first 150 epochs), THFKD demonstrates a mean advantage in classification accuracy over the individual use of L_cl and L_tg for training, with an improvement ranging approximately from 2% to 5%. Furthermore, it can be observed that between epochs 90 and 110, the L_cl  +  L_tg curve exhibits more minor fluctuations, with an amplitude of approximately 3%, whereas the fluctuations for L_cl and L_tg alone are around 12% and 10%, respectively. In Fig 5(b), although the loss value of L_tg  +  L_cl is approximately four times that of L_cl, the loss value continues to decrease as training progresses, indicating that the teacher’s guidance effectively directs the optimization trajectory of the student. Additionally, it can be observed that the L_tg loss accounts for a larger proportion of the total loss compared to the L_cl loss, suggesting that the teacher’s guidance dominates the early gradient updates of the student model. After the learning rate drops at epoch 150, entering the later training stage, the red accuracy curve (L_cl) in Fig 5(a) is slightly higher than the cyan one (L_tg), suggesting that diversified dynamic knowledge further optimizes the student model. However, a reversal occurs in the final stages, which can be attributed to the lack of guidance from the teacher in the early phases. This absence leads to the accumulation of erroneous information within the collaborative learning objectives, and the impact of this phenomenon becomes further amplified as the training progresses into the later stages.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Effect of loss in different stages.

(a) Accuracy curves with different losses. (b) Loss value change curves.

https://doi.org/10.1371/journal.pone.0326711.g005

Effect of two-stage optimization.

By controlling the period parameter E and a the starting parameter, we evaluated the impact of the weight increase period and the two-stage optimization strategy on the model performance. As shown in Table 4, when the period length is controlled during the early stage of training (before the learning rate decreases at epoch 150), the impact on model accuracy is minimal. However, when the period extends into the later stages of training, the change in weight loss is detrimental to the convergence of the student model, leading to a significant drop in accuracy. When E is 0 or 300 (i.e., a single-stage optimization strategy), the loss weight gradually increases or remains constant throughout the training process, resulting in the classification accuracy of THFKD being lower than that obtained with the two-stage optimization strategy.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Impact of the period parameter E. (Accuracy%)

https://doi.org/10.1371/journal.pone.0326711.t004

In the training phase, the starting parameter a controls at which epoch THFKD begins to use ramp-up weights. As shown in Fig 6, when a = 0, the accuracy curve of the model steadily rises, indicating that THFKD completes the early learning phase within 80 epochs, laying the foundation for subsequent training, thereby maintaining the highest accuracy. However, when a is set to 30, 60, or 90, the accuracy sharply increases at epoch=a (when the ramp-up weight is first applied) but then drops immediately, suggesting that changing the loss weight mid-training disrupts the optimization of the model. Therefore, by using ramp-up weight at the beginning of training and setting the period length to 120, THFKD achieves the best two-stage optimization strategy.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Impact of the parameter a.

https://doi.org/10.1371/journal.pone.0326711.g006

Impact of temperature parameter .

The temperature parameter has a notable impact on the performance of the THFKD method. As shown in Table 5, the highest accuracy (75.41%) is achieved when , making it the optimal setting. A lower temperature () may result in insufficient soft target information, limiting the learning capability of the student model. On the other hand, as the temperature increases to and , the performance gradually declines, indicating that excessively high temperatures may overly smooth the probability distribution, leading to the loss of critical information. Therefore, represents a balanced temperature that maintains sufficient information while avoiding excessive smoothing, providing effective guidance for the student model.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Impact of temperature parameter . (Accuracy%)

https://doi.org/10.1371/journal.pone.0326711.t005

Effect of branch number.

We set the total number of branches B to range from 2 to 6 to analyze the impact of branch quantity on performance. As shown in Fig 7, THFKD exhibits a performance improvement of about 2% (from 73% to 75%) when the number of branches is increased from 1 to 4. However, further increasing the number of branches does not effectively improve performance. This is because the multi-branch architecture is independently constructed at higher layers. When the number of branches is increased, the complexity of the model increases, and the knowledge transfer becomes more chaotic. Thus, continuously increasing the number of branches does not sustain performance improvement. We observed comparable improvements in accuracy when increasing the number of branches from 3 to 4 and from 4 to 5. Although adding branches during testing does not incur additional costs, using 5 branches during training increases computational overhead, including higher GPU memory usage and longer training times, as the number of channels in the auxiliary blocks grows exponentially. Throughout the experiments, we balanced the performance gains against the computational costs and found that 4 branches were sufficient to achieve substantial performance improvements. Consequently, we selected 4 branches as the optimal configuration for our experimental setup.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Effect of branch number.

https://doi.org/10.1371/journal.pone.0326711.g007

Homogenization problem.

We compared the online knowledge distillation methods based on multi-branch models, including CL [48], ONE [16], and OKDDip [32] (all configured with four branches during training), with AFID [49], which employs dual independent models. As shown in Table 6, the first row presents the baseline classification accuracies of the high-parameter models ResNet8x4 and WRN16-2, which are 72.50% and 73.26%, respectively, both higher than the 71.14% accuracy of ResNet32. However, when trained using CL, ONE, and OKDDip, the classification accuracies of these high-parameter models (ResNet8x4 and WRN16-2) decrease compared to ResNet32 (72.56% > 61.81% and 72.43%; 73.50% > 61.66% and 72.86%; 74.32% > 72.75% and 73.96%). Notably, in the AFID, the independent model training approach does not exhibit such accuracy degradation (74.05% < 74.56% and 74.94%). In contrast, THFKD shows a 1.09% improvement over OKDDip when using ResNet8x4 and a 1.37% improvement over AFID when using WRN16-2 while avoiding the phenomenon where high-parameter models (ResNet8x4 and WRN16-2) underperform ResNet32 in classification accuracy. This demonstrates that THFKD adapts to various model architectures and effectively mitigates the problem of homogenization.

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. Accuracy (%) of OKD on CIFAR-100.

https://doi.org/10.1371/journal.pone.0326711.t006

Diversity analysis.

By visualizing the Euclidean distance between the logits of each pair of branches and the ensemble prediction accuracy of the branches, we analyze the improvement in diversity. As shown in Fig 8, the Fig 8(a) compares the Euclidean distances of the four methods. It can be seen that after several rounds of optimization, the diversity rapidly increases, with THFKD maintaining stability, while OKDDip, ONE, and CL exhibit a downward trend. Our method is clearly superior to the other three methods throughout the training process. The Fig 8(b) shows the ensemble prediction accuracy of the four methods. Benefiting from the increased diversity, THFKD consistently outperforms the other methods in terms of ensemble accuracy, indicating that THFKD effectively enhances the diversity of dynamic knowledge, improving the model’s generalization ability.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. Diversity comparison with ResNet8x4 as the student model.

(a): Euclidean distance. (b): Ensemble prediction.

https://doi.org/10.1371/journal.pone.0326711.g008

Ensemble predictions.

Ensembling the predictions of multiple branches can lead to stronger performance. As shown in Table 7, we retain three branches for the ensemble output (THFKD-E). Benefiting from the increased diversity, THFKD-E achieves a more effective ensemble. When the number of parameters in the ensemble model is less than or equal to that of the teacher model, the ensemble’s predictions achieve higher accuracy than those made by the teacher model.

Download:

• PNG
  larger image
• TIFF
  original image

Table 7. Ensemble prediction Accuracy (%) and parameters (MB) of THFKD with three branches on CIFAR-100.

https://doi.org/10.1371/journal.pone.0326711.t007

Discussion

Branches with different abilities.

THFKD’s performance in the 300th epoch is shown under three model architectures. As shown in Table 8, the performance of the four branches gradually improves as the number of parameters increases (with the channel count in the auxiliary blocks growing multiplicatively), demonstrating differences in abilities. This provides a foundation for diversifying collaborative learning targets, allowing THFKD to meet varying precision requirements and enabling more flexible deployment.

Download:

• PNG
  larger image
• TIFF
  original image

Table 8. Accuracy (%) of branches with different abilities on CIFAR-100.

https://doi.org/10.1371/journal.pone.0326711.t008

To analyze the impact of heterogeneous branches in terms of width and depth, we conducted experiments on the ResNet32 model by varying the number of auxiliary blocks (NA) and their channel size (CA). Specifically, NA was set to 0, 1, 2, where NA = 0 indicates that each branch has the same structure without auxiliary blocks, and NA = 1 corresponds to the original experimental setting of THFKD. CA was set to either 1 or 2 the number of channels in the high-level layers (CA = , ).

As shown in Table 9, although the absolute differences in accuracy are relatively small (mostly within 1%), some consistent trends can still be observed. Firstly, when NA=0, all branches perform poorly (ranging from 74.3% to 74.6%), which may be due to the lack of auxiliary structures limiting feature diversity. Introducing auxiliary blocks (NA=2) generally improves performance (e.g., Branch-3 reaches up to 75.64%), but the trend is not strictly monotonic—overly complex branches such as Branch-4 show a slight performance drop, possibly due to increased optimization difficulty. This suggests that simply stacking more modules does not guarantee better results, and moderate structural diversity may be more beneficial.

Download:

• PNG
  larger image
• TIFF
  original image

Table 9. Accuracy (%) of heterogeneous branches at different depths and widths.

https://doi.org/10.1371/journal.pone.0326711.t009

Impact of the number of channels (CA): Doubling the number of channels improves performance in some branches (e.g., Branch-2 and Branch-3 reach 75.77% and 75.90%, respectively), indicating that selective capacity expansion can enhance feature learning ability. However, for CA=C 1, C 2, the accuracy fluctuations of Branches 2–4 are relatively small (with the maximum and minimum accuracy differences being 0.45% and 0.16%, respectively), which suggests that uniform channel expansion (i.e., using identical auxiliary blocks) leads to similar capabilities among branches, thereby reducing the diversity of dynamic knowledge. The best performance is observed under the THFKD configuration (NA=1, CA=C1), with Branch-1 achieving an accuracy of 75.03%. Although the performance improvement is modest, the accuracy variation among Branches 2–4 is relatively large (the maximum difference of 1.41% is observed between Branch 3 and Branch 4), indicating that well-designed heterogeneous branches in both depth and width help promote diversity in feature representation.

Feature visualization.

This study uses Grad-CAM [50] to visualize the feature regions attended to by the model, as shown in Fig 9, including the heatmaps of each branch and PFF-3, along with the ground truth labels and predicted classes. Columns 1 and 2 show the cases where each branch and PFF-3 classify correctly, demonstrating high certainty for simple images. In Column 2, the larger-capacity branches focus more on local features. In Column 3, although each branch and PFF-3 focus on the correct regions, they predict three different categories. This demonstrates that the student can learn more comprehensively by softening logits to transfer similar information between categories and combining both static and dynamic knowledge. Only Branch-4 can correctly predict the class in the fourth column, demonstrating that THFKD can further optimize the student model in the later stages. We also observe that the feature heatmap of PFF-3, compared to those of Branches 1-4, captures more accurate feature information, indicating that PFF effectively fuses the differentiated feature representations.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 9. Fusion module and the Grad-CAM for each branch are visualized in the Tiny-ImageNet dataset.

https://doi.org/10.1371/journal.pone.0326711.g009