Fault diagnosis methods based on image representations and CNN

CNNs have achieved remarkable success in image processing and pattern recognition because of their strong local feature extraction capability, and have been widely applied to rotating machinery fault diagnosis tasks [15,16]. By transforming vibration signals into time-frequency maps or image-based representations such as GAF, CNNs can automatically learn discriminative feature representations, thereby reducing the dependence of traditional methods on manually designed features [17,18].

Previous studies have shown that CNN-based models can achieve high diagnostic accuracy in bearing and gear fault identification. For example, Zhao et al. [19] proposed a CNN model based on multi-information fusion, which effectively enhanced fault feature representation. Ruan et al. [20] improved the ability of the model to capture fault frequency characteristics under different conditions by optimizing the design of convolution kernel parameters. In addition, lightweight or deep CNN architectures, such as ResNet and MobileNet, have also been introduced into fault diagnosis tasks to reduce computational complexity while maintaining diagnostic accuracy [21,22].

However, CNN essentially rely on local convolution operations, and their receptive fields are limited by the kernel size and network depth. Therefore, long-range dependencies are difficult to model effectively. Moreover, traditional CNN architectures usually adopt fixed hierarchical designs, which limits their adaptability to multi-scale dynamic features in complex industrial data. These problems are particularly prominent in the analysis of nonstationary and strongly coupled vibration signals, thereby restricting further improvements in model performance.

Transformer-based fault diagnosis methods

Transformer was first proposed by Vaswani et al. [23] and has achieved breakthrough progress in natural language processing. Its core component is the self-attention mechanism, which can directly model dependencies between any positions in a sequence. Therefore, Transformer has significant advantages in global feature modeling. In recent years, Transformer has gradually been introduced into time-series analysis and fault diagnosis, showing promising application potential [24,25].

In rotating machinery fault diagnosis, Transformer is usually combined with CNN or other feature extraction methods to integrate local feature learning with global dependency modeling. For example, the Anomaly Transformer proposed by Xu et al. [26] improved anomaly detection capability through an association discrepancy mechanism. In some studies, extracted features have been fed into Transformer modules to further model complex patterns [27]. In addition, improved models incorporating attention mechanisms have also been used to enhance the focus of models on key features [28].

Although Transformer has advantages in global modeling, several limitations still remain in existing studies. On the one hand, most methods mainly focus on dependencies along the temporal or spatial dimension, while importance differences among feature channels are not explicitly modeled. This may introduce redundant information and degrade diagnostic performance. On the other hand, most studies use a single feature input or a simple fusion strategy, and the complementary information among multi-source features is not sufficiently exploited. Consequently, the representation capability of these models for complex fault patterns is limited.

Overall architecture

To address the limited feature representation capability of one-dimensional vibration signals and the insufficient utilization of multi-source information by deep models in rotating machinery fault diagnosis, a fault diagnosis framework integrating GAF with a dual-branch ConvNeXt and a CTransformer is proposed. The overall architecture is shown in Fig 1. It should be noted that the proposed framework is not a simple cascade of GAF, ConvNeXt, and Transformer modules. Instead, it is specifically designed around the feature representation requirements of vibration-based fault diagnosis.In this framework, GASF and GADF are used to provide two complementary types of information, namely global structural information and local dynamic information. The dual-branch ConvNeXt is employed to independently model these two heterogeneous representations, thereby reducing premature mixing between different feature spaces. The CTransformer further enhances the fused features from both the channel dimension and the global dependency dimension. In this way, the proposed method forms a continuous optimization process from signal representation and decoupled feature extraction to channel-aware global modeling, which constitutes the main methodological innovation of this study.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Overall framework of the proposed GAF-ConvNeXt-CTransformer model.

https://doi.org/10.1371/journal.pone.0354912.g001

First, in the signal preprocessing stage, the raw one-dimensional vibration signals are segmented using a fixed window to construct the sample set. Meanwhile, normalization is performed to map the data into the range of [−1, 1], thereby eliminating scale differences and improving the stability of model training. This process provides a unified data scale for subsequent image-based modeling while preserving the original signal information as much as possible.

Then, the GAF method is used to transform one-dimensional time series into two-dimensional image representations, namely GASF and GADF. Specifically, GASF mainly reflects the overall trend and global correlation of the time series, whereas GADF focuses on local variation relationships and dynamic difference features. By constructing dual-view image representations, the raw signals can be represented from different structural spaces, thereby enhancing the representation capability of the data.

In the feature extraction stage, a dual-branch ConvNeXt network is constructed to model GASF and GADF images in parallel. The key idea of this design is to decouple two types of image representations with different structural characteristics, enabling the model to learn global structural information and local variation features separately. Compared with a single-branch structure, the dual-branch architecture can effectively exploit the complementary relationship between the two representations, thus improving the completeness and expressiveness of the extracted features. In addition, ConvNeXt enhances feature learning capability while maintaining computational efficiency by introducing large convolution kernels and normalization strategies. On this basis, the feature fusion module integrates the features extracted from the two branches. This process not only preserves key information from different branches but also enables collaborative representation of multi-source features, providing richer inputs for subsequent global modeling.

Finally, in the classification stage, a CTransformer module is introduced to further model the fused features. Based on the conventional Transformer structure, this module incorporates a channel attention mechanism. By explicitly modeling the importance distribution of feature channels, fault-related key features are adaptively enhanced, and the interference of redundant information is suppressed. Meanwhile, the self-attention mechanism is used to capture global dependencies among features, thereby enabling efficient representation and accurate classification of complex fault patterns.

Gramian Angular Field

GAF [29] is an image-based representation method that maps a one-dimensional time series into a two-dimensional image. Its core idea is to encode the amplitude information of a sequence through polar coordinate transformation, thereby characterizing the structural features of the time series in a two-dimensional space. This method can preserve temporal dependencies while converting the raw sequence into an image representation suitable for visual feature modeling, thus providing richer structural information for subsequent feature extraction.

Given a time series , X is first normalized as follows:

(1)

Then, the normalized sequence is mapped into the polar coordinate system:

(2)

where denotes the polar angle of the i-th sampling point, and denotes its radius.

On this basis, the two-dimensional representation of the time series can be obtained by constructing trigonometric relationships between different time points. Specifically, GAF mainly includes two forms: GASF and GADF. GASF describes the variation relationship of the time series through the cosine of the angle summation, which is defined as

(3)

where represent the polar angles of two time points.

GADF describes the variation relationship of the time series through the sine of the angular difference, which is defined as

(4)

By simultaneously constructing GASF and GADF images, the original vibration signal can be modeled from two perspectives: global structure and local dynamics. GASF is more effective in capturing global structural correlations, whereas GADF is more sensitive to local dynamic variations. Fig 2 presents the GASF and GADF representations of the same vibration signal. It can be observed that the two representations exhibit distinct structural characteristics, indicating that they can provide complementary information for subsequent fault feature extraction.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Illustration of GASF and GADF representations: (a) GASF image; (b) GADF image.

https://doi.org/10.1371/journal.pone.0354912.g002

Dual-branch ConvNeXt feature extraction module

ConvNeXt integrates the advantages of traditional CNN and Transformer-inspired design principles, thereby improving feature representation while maintaining computational efficiency. A ConvNeXt block is mainly composed of depthwise convolution, layer normalization, pointwise convolution, and residual connection. Its basic computation can be formulated as

(5)

where denotes pointwise convolution, denotes depthwise convolution, and denotes layer normalization.

Specifically, ConvNeXt enlarges the effective receptive field by introducing a large convolution kernel, such as 7 × 7, thereby enhancing the ability to capture global spatial information. In addition, Layer Normalization and GELU activation are employed to improve training stability and batch consistency. Therefore, ConvNeXt can effectively extract spatial texture features and structural patterns from GAF images.

After GAF encoding, GASF and GADF characterize the structural properties of time series from different perspectives. GASF mainly reflects the overall trend and global correlation of the sequence, whereas GADF is more sensitive to local variations and dynamic differences. If early fusion is directly performed at the input stage, or if the two types of features are simply concatenated along the channel dimension, information from different representation spaces may be mixed prematurely during low-level feature learning, thereby weakening their distinct representational capabilities. To address this issue, a dual-branch ConvNeXt structure is adopted to decouple the modeling of GASF and GADF. The two branches are used to learn global structural features and local dynamic features separately, and feature fusion is then performed in the high-level semantic space. This design helps mitigate feature confusion caused by simple fusion and enables the complementary fault information between the two image representations to be more fully exploited.

Let the GASF and GADF images be denoted as X_GASF and X_GADF, respectively. The dual-branch feature extraction process can be expressed as

(6)

(7)

where denotes the ConvNeXt feature extraction network.

After dual-branch feature extraction, feature fusion is required to form a unified representation. In this paper, a weighted fusion strategy is used to integrate the two types of features, which is formulated as

(8)

where α is the fusion weight. This strategy preserves the structural simplicity of the model while achieving effective integration of multi-source features. The fused representation contains both global trend information and local dynamic characteristics, thereby constructing a more complete feature space.

The dual-branch ConvNeXt feature extraction module not only improves the utilization of multi-source information but also provides discriminative high-dimensional features for the subsequent CTransformer. This design lays a solid foundation for the accurate identification of complex fault patterns.

Channel-aware transformer

After the dual-branch features of GASF and GADF are fused, the fault-sensitive information contained in different deep feature channels is not evenly distributed. Some feature channels may show strong responses to local impacts, periodic modulation, structural texture variations, or changes in global correlations, whereas other channels may contain redundant information or noise-related responses. Therefore, a CTransformer is introduced after feature fusion.

It should be noted that the “channels” in this study do not refer to the original vibration sensor channels. Instead, they are deep feature channels formed after GASF and GADF images are extracted and fused by the dual-branch ConvNeXt. Each feature channel can be interpreted as a latent fault-related semantic response learned by the model. The channel-aware mechanism adaptively enhances fault-sensitive channels and suppresses redundant responses according to their contributions to fault discrimination, thereby improving the utilization efficiency of the fused features.

Let the fused feature be denoted as . First, global average pooling and global max pooling are performed along the spatial dimension to extract channel-level statistical information:

(9)

(10)

Then, the two types of statistical features are fed into a shared multilayer perceptron for nonlinear mapping, and a channel weight vector is generated through a sigmoid function:

(11)

where denotes the Sigmoid function, and represents the channel attention weight vector.

Finally, the channel weights are applied to the original fused feature to achieve channel recalibration:

(12)

where denotes element-wise multiplication.

After channel recalibration is completed, F′ is reshaped into a sequence and fed into the Transformer encoder for global dependency modeling. Specifically, the feature dimension is used as the sequence representation, and linear projections are adopted to generate the query, key, and value matrices:

(13)

where are learnable parameters.

Based on the self-attention mechanism, the correlations among features are calculated as:

(14)

Then, a multi-head attention mechanism is used to capture dependencies among features in different subspaces, thereby enhancing the ability to model complex fault patterns. In addition, residual connections and layer normalization are employed in the Transformer encoder to ensure stable information propagation. The calculation process can be expressed as:

(15)

(16)

Finally, the global feature representation is fed into a fully connected layer, and the fault category is predicted through the Softmax function:

(17)

where W and b denote the learnable weight matrix and bias term, respectively.

In summary, CTransformer enables efficient modeling of fused features through a two-stage mechanism that combines channel recalibration and global dependency modeling. The channel attention mechanism selects critical features from the channel dimension and improves feature utilization, whereas the self-attention mechanism captures global dependencies and enhances the modeling capability for complex fault patterns.

Datasets and experimental settings

Paderborn university bearing dataset.

The Paderborn University (PU) bearing dataset from Germany [30] is a publicly available benchmark dataset for bearing fault diagnosis. This dataset consists of data collected from six healthy bearings, twelve artificially damaged bearings, and fourteen bearings with real damage generated through accelerated lifetime tests. Therefore, different bearing health states, including normal states, artificial damage states, and real damage states, are covered. The original vibration signals were collected under different operating conditions, including rotational speed, load torque, and radial load. Thus, the dataset contains vibration signals under different bearing damage states and reflects, to some extent, the influence of speed and load variations on vibration signal characteristics.

The detailed information on the selected bearing condition states is listed in Table 1. A total of 1200 samples were selected from the PU dataset, including 12 bearing condition states with 100 samples for each state. Each sample contained 1600 sampling points. Therefore, the window length and stride were both set to 1600, corresponding to an overlap ratio of 0%. The samples were randomly divided into training and testing sets at a ratio of 4:1.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Bearing condition state categories in the PU dataset.

https://doi.org/10.1371/journal.pone.0354912.t001

To ensure sufficient experimental data and cover different bearing states, including normal, artificial damage, and real damage states, 12 bearing health or damage states were selected from the PU dataset. Specifically, K001-K004 represent different healthy bearing states; KA01, KA03, KA04, and KA15 represent outer-race damage states; and KI01, KI03, KI04, and KI14 represent inner-race damage states. Therefore, the 12 labels used in the PU dataset more accurately represent different bearing health states or damage states, rather than coarse-grained fault types. This setting was adopted to evaluate the ability of the proposed method to identify fine-grained differences among bearing states.

After the training and testing sets were determined, normalization and GASF/GADF image transformation were performed separately within each set.

University of Connecticut gear dataset.

The University of Connecticut (UC) gear dataset [31] was collected from a two-stage gearbox with replaceable gears at a sampling frequency of 20 kHz. The detailed definitions of the gear fault categories are listed in Table 2. This dataset contains nine gear condition categories, including healthy condition, missing tooth, root crack, spalling, and five degrees of chipping damage. Each category contains 104 samples, resulting in a total of 936 samples. Each sample consists of 3600 sampling points. Therefore, the window length and stride were both set to 3600, corresponding to an overlap ratio of 0%. The samples were randomly divided into training and testing sets at a ratio of 10:3.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Gear fault categories in the UC dataset.

https://doi.org/10.1371/journal.pone.0354912.t002

Accuracy, macro-averaged Precision, macro-averaged Recall, and macro-averaged F1-score were used as evaluation metrics, as defined in Eqs. (18)-(21). In these equations, TP, TN, FP, and FN denote true positives, true negatives, false positives, and false negatives, respectively. For the multiclass classification task, Precision, Recall, and F1-score were calculated using macro averaging. Specifically, the corresponding metric was first calculated for each class and then averaged across all classes, so that the diagnostic performance for different fault categories could be evaluated fairly.

(18)(19)(20)(21)

To improve the reproducibility of the proposed method, the main parameter settings for data processing, GAF image transformation, ConvNeXt branches, the CTransformer module, and model training are summarized in Table 3. The same data processing procedure and training parameters were used in all comparative and ablation experiments to ensure fairness and comparability. Accuracy, macro-averaged Precision, macro-averaged Recall, and macro-averaged F1-score were used as evaluation metrics.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Main parameter settings of the proposed method.

https://doi.org/10.1371/journal.pone.0354912.t003

Comparative experiments

To further verify the effectiveness and superiority of the proposed method, several deep learning models were selected for comparison, including CNN, ResNet, ConvNeXt, Transformer, and ViTransformer. These models represent classical convolutional neural networks, residual networks, modern convolutional architectures, and self-attention-based methods, respectively, and can comprehensively reflect the performance of different modeling mechanisms. To ensure the fairness of the comparison, all models were trained under the same data partition, training epochs, and optimization strategy, and their performance was evaluated using the same metrics. The basic settings of the comparison models are as follows.

1. CNN adopts a classical convolutional neural network structure composed of multiple convolutional and pooling layers. The convolution kernel size was set to 3 × 3, and the model was used to extract local features.
2. ResNet introduces residual connections into deep convolutional networks, which alleviates the gradient vanishing problem during training and enhances feature representation capability.
3. ConvNeXt adopts a modern convolutional network architecture, in which large convolution kernels and layer normalization are introduced to improve feature perception and training stability.
4. Transformer adopts the standard Transformer encoder structure and uses the multi-head self-attention mechanism to model global dependencies among features.
5. Vision Transformer divides the input image into multiple patches and models their relationships through the self-attention mechanism, thereby learning global image features.

As shown in Table 4, the proposed method achieved the best results in terms of Accuracy, Precision, Recall, and F1-score over five repeated experiments on the PU dataset. Compared with CNN, ResNet, ConvNeXt, Transformer, and ViTransformer, the proposed method exhibited higher average classification performance. Specifically, the Accuracy reached 97.08%, which was 2.32% higher than that of the second-best ViTransformer. The Precision, Recall, and F1-score reached 97.16%, 96.94%, and 97.05%, with improvements of 2.08%, 3.02%, and 2.55%, respectively. These results indicate that the collaborative use of GASF/GADF dual-view representation, dual-branch ConvNeXt feature extraction, and the CTransformer effectively improves the recognition performance for PU bearing health states.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Evaluation metrics for various comparative networks (PU).

https://doi.org/10.1371/journal.pone.0354912.t004

In addition, the proposed method showed small standard deviations across the five repeated experiments. The standard deviations of Accuracy, Precision, Recall, and F1-score were 0.36%, 0.32%, 0.39%, and 0.34%, respectively, indicating good stability under different random splits. The 95% confidence intervals of Accuracy, Precision, Recall, and F1-score were further calculated as [96.63%, 97.53%], [96.76%, 97.56%], [96.46%, 97.42%], and [96.63%, 97.47%], respectively. These narrow confidence intervals further demonstrate the stable diagnostic performance of the proposed method on the PU dataset.

As shown in Table 5, the proposed method achieved the best results in terms of Accuracy, Precision, Recall, and F1-score over five repeated experiments on the UC gear dataset. Specifically, the Accuracy reached 98.15%, while the Precision, Recall, and F1-score reached 98.24%, 98.03%, and 98.13%, respectively. Compared with the second-best ViTransformer, the proposed method improved Accuracy, Precision, Recall, and F1-score by 2.34%, 2.06%, 3.00%, and 2.53%, respectively. These results indicate that the proposed method can extract more discriminative features from UC gear data and achieves better classification performance in gear fault diagnosis.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Evaluation metrics for various comparative networks (UC).

https://doi.org/10.1371/journal.pone.0354912.t005

In addition, the proposed method showed small standard deviations across the five repeated experiments. The standard deviations of Accuracy, Precision, Recall, and F1-score were 0.34%, 0.31%, 0.37%, and 0.33%, respectively, indicating good stability under different random splits. The 95% confidence intervals of Accuracy, Precision, Recall, and F1-score were further calculated as [97.73%, 98.57%], [97.86%, 98.62%], [97.57%, 98.49%], and [97.72%, 98.54%], respectively. These narrow confidence intervals further demonstrate the stable diagnostic performance of the proposed method on the UC dataset.

Ablation study

To verify the effectiveness of GASF, GADF, dual-branch ConvNeXt, and CTransformer, ablation experiments were conducted under the same experimental settings. Model components were gradually removed or replaced to systematically analyze the contribution of each module to the overall performance. To avoid computational redundancy caused by repeated experiments while ensuring the representativeness of the analysis, the PU dataset, which contains more complex operating conditions and richer fault categories, was selected for the ablation study. Accuracy, precision, recall, and F1-score were used as the evaluation metrics.

In the first two experiments, GASF and GADF were used separately as single inputs, and a single-branch ConvNeXt was adopted for feature extraction. In the third experiment, both GASF and GADF were introduced, and a dual-branch ConvNeXt was used for parallel feature modeling. In the fourth experiment, the CTransformer was further incorporated into the dual-branch ConvNeXt, forming the complete model. By progressively adding model components, the contribution of each module to the diagnostic performance could be systematically evaluated.

The ablation results are shown in Table 6. When only a single GAF representation was used, the Accuracy values of the models using GASF and GADF were 93.28% and 94.16%, respectively. This indicates that both GAF representations can provide effective fault-related features, but a single representation is still insufficient to fully characterize complex vibration signals. When GASF and GADF were jointly introduced and modeled using the dual-branch ConvNeXt, the Accuracy increased to 96.42%, and the F1-score reached 96.47%. This demonstrates that the dual-branch structure can effectively integrate the complementary information of the two GAF representations and enhance the feature representation capability of the model. After CTransformer was further introduced, the complete model achieved an Accuracy of 97.08%, with Precision, Recall, and F1-score values of 97.34%, 97.08%, and 97.21%, respectively. These results indicate that CTransformer can strengthen critical features and model global dependencies, thereby further improving fault recognition performance. Overall, each module contributes positively to the performance improvement of the proposed model.

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. Ablation experiment results on the PU dataset.

https://doi.org/10.1371/journal.pone.0354912.t006

To further verify the effectiveness of the dual-branch ConvNeXt structure and determine whether the performance improvement mainly results from an increased number of parameters, comparative experiments with different fusion structures were designed. Three structures were compared:

1. Early fusion: GASF and GADF were directly used as a two-channel image input to a single ConvNeXt network at the input stage, and feature extraction and classification were performed by the same network.
2. Channel concatenation: Features were extracted from GASF and GADF separately and then concatenated along the channel dimension before being fed into the subsequent classification module.
3. Proposed dual-branch: Two ConvNeXt branches were used to perform decoupled modeling of GASF and GADF, respectively, and feature fusion was performed in the high-level feature space.

As shown in Table 7, different fusion strategies had a clear influence on diagnostic performance. Early fusion directly fused the GASF and GADF images at the input level and achieved an Accuracy of 94.17%. Channel concatenation extracted features from GASF and GADF separately and then concatenated them along the channel dimension, improving the Accuracy to 95.42%. This indicates that separate feature extraction helps preserve the distinct information contained in the two image representations.

Download:

• PNG
  larger image
• TIFF
  original image

Table 7. Comparison of different fusion strategies on the PU dataset.

https://doi.org/10.1371/journal.pone.0354912.t007

In contrast, the proposed dual-branch high-level fusion method achieved the best performance, with Accuracy, Precision, Recall, and F1-score values of 97.08%, 97.34%, 97.08%, and 97.21%, respectively. In addition, channel concatenation had more parameters than the proposed method, with 2.59M parameters compared with 2.53M, but achieved lower diagnostic performance. This result indicates that the performance improvement was not simply caused by an increased number of parameters, but was mainly attributed to the dual-view representation and high-level feature fusion strategy.

To analyze the feature attention mechanism of the CTransformer, the attention weights generated by the channel recalibration module were extracted, and the average channel weights for different fault categories were calculated. After normalization, the results were visualized in Fig 7. In this figure, the horizontal axis represents the deep feature channel index, the vertical axis represents the bearing health state, and the color indicates the normalized attention weight. It should be noted that these channels are not the original sensor channels, but deep feature channels obtained after GASF/GADF images were extracted and fused by the dual-branch ConvNeXt.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Normalized channel attention heatmap for different fault classes.

https://doi.org/10.1371/journal.pone.0354912.g007

As shown in Fig 7, the distributions of high-weight channels vary across different fault categories, indicating that the model can adaptively focus on different fault-sensitive features according to the fault state. Some channels show strong responses across multiple categories, which may reflect common fault-related features, whereas other channels are more prominent only in specific categories and may be associated with local impacts, dynamic differences, or texture mutations. These results suggest that the channel attention mechanism helps improve both the discriminative capability and interpretability of the proposed model.