Surface defects in railway systems

An intact and reliable rail network is essential for the safe operation of rail vehicles. Surface defects on the rails pose a serious risk to operational safety. In daily use, rails are subject to continuous wear, which in the long term leads to rolling contact fatigue (RCF). This can initially cause microscopic cracks to form, which, if undetected, can later develop into characteristic defects. Early detection of this damage is therefore crucial for safe operation. In this study, we examine seven different defect classes that were collected under real-world conditions in 2024 [18].

1. Cracks: Cyclic loading causes microcracks to form on or below the surface, which spread over time and can lead to other types of defects.
2. Flakings: Sidewall deformation occurs on the tread, typically caused by high lateral forces in tight corners.
3. Grooves: Longitudinal grooves along the rail, usually caused by abrasive wear or faulty grinding work.
4. Joints: Irregularities at rail joints or weld seams, often caused by poor connections or thermal stresses.
5. Shellings: Material chips off the surface when subsurface cracks propagate upward, typically caused by rolling contact fatigue.
6. Spallings: Larger surface breakouts form when multiple shelling cracks merge, usually as a result of advanced fatigue damage.
7. Squats: are localized depressions with a typical crack pattern that are often caused by dynamic wheel forces, such as imbalances.

Not every type of defect has the same impact on the condition of the rail infrastructure. While grooves or flaking are considered largely tolerable in the literature, depending on their location and severity, cracks or squats are considered an acute safety risk and require immediate repair. Shelling and spalling usually indicate advanced damage caused by RCF and require treatment in the medium term. Joints are considered potential weak points and should also be checked regularly. Therefore, a precise classification of defect types across all classes is crucial for condition-based maintenance of rail networks. Existing conventional testing methods, such as visual or ultrasonic inspection, are time-consuming and costly, and can therefore only be used selectively. Automated, image-based fault classification can be used more widely and can therefore help to detect critical faults at an early stage and initiate targeted measures.

Deep learning for railway surface defect detection

Due to deep learning’s ability to classify images and their context at a high level, deep learning-based approaches have also been used for the assessment of railway quality for several years. A distinction can be made between segmentation or object recognition approaches and those that classify images. While the former primarily detects the position of the anomaly to be recognized, the latter approaches classify the image shown into predefined classes. Segmentation and object detection approaches, therefore, provide information about the type of damage as well as its (pixel-precise) position, which is a more difficult task and therefore tends to result in lower performance. One of the most effective approaches in this first group is the work of Haroon et al. [11], who perform object detection based on 3,500 images. Taking five classes (e.g., defective fasteners and fishplates) into account, the authors achieve a mean average precision of 95%. Zhang et al. [13] go one step further and use eight different classes and 400 samples. The authors, who also perform object detection, achieve a mean average precision (@0.5) of 85.6% [13]. The approach with the most diverse dataset from the group of studies performing object recognition or segmentation is chosen by Zhao et al. [12] in 2024, which includes 15 different defect types based on 5,557 samples. Based on this, they achieve a mean average precision (@0.5) of 88.1%.

Nevertheless, there is a conceptual problem: It is questionable whether the additional information provided by object detection or segmentation regarding the (pixel-precise) localization of the error is significant enough for the actual real-world problem that the resulting trade-off in terms of model performance is worth this additional information. This is not necessarily the case: as long as it is not yet possible to automatically correct the detected errors, knowing the exact location of the error is less important than detecting the error as accurately as possible. Therefore, the focus should be on high-performance and robust classification of rail images. To achieve this, two aspects are important: first, rails can exhibit a variety of error patterns. The developed model must therefore cover as many relevant error types as possible in order to avoid the risk of misclassifications due to error patterns not provided for in the architecture. Second, in such safety-critical applications, it is particularly important that the developed model achieves the reported performance robustly and methodically at the highest level, which is only possible through stratified k-fold cross-validation. Table 1 therefore provides an overview of the literature relevant to this paper, its coverage of the range of defect types, and whether it employs a cross-validated approach.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Comparison of relevant classification literature on rail surface defect detection regarding coverage of five or more defect types (broad coverage) and the use of cross-validation.

https://doi.org/10.1371/journal.pone.0340789.t001

The first group of papers covers only a limited range of defect types, typically addressing fewer than five classes. Ackigoz/Korkmaz [19] develop a residual convolutional network to distinguish healthy samples from joints, squats, and so-called “SSquats”, using 1,838 images and no cross-validation. They report an accuracy of 99.83%. Alvarenga et al. [20] also classify joints and squats, but distinguish them from welding points and achieve a non-cross-validated accuracy of 98.00% based on 2,848 samples. Dang et al. [14] use non-optical sensors and ultrasonic waves to differentiate between intact and damaged rails (with cracks or bumps). The authors achieve an accuracy of 98.00% (900 images, non-cross-validated). Li et al. [24] also use so-called B-Scan ultrasonic images and achieve an accuracy of 85.00% for a problem with more classes (four) but fewer samples (280). Lu et al. [22] again use B-Scan ultrasonic images, but instead of training a CNN, they train a Vision Transformer architecture. This allows global features to be extracted efficiently, resulting in an accuracy of 98.92% for a four-class problem, but also not cross-validated (283 samples). Aydin et al. [15] use 1,598 images to assess fasteners (healthy, damaged, missing) and are able to achieve an accuracy of 99.70%. This work is the only one in the group of studies that does not aim for broad error coverage and that performs cross-validation. Three studies cover a wider range of defect types and therefore use five or more classes. Hu et al. [16] use ultrasonic images to classify eight different degrees of rail defects. The authors use different transfer learning approaches and vision transformers, with a hierarchical classification model showing the highest performance with 88.56% accuracy. Jatoi et al. [17] use the same dataset as the present study (seven classes). As in the work of Hu et al. [16], various vision transformers and transfer learning backbones are tested, but no cross-validation is performed. Using a train-test split, DenseNet121 achieves the highest performance with an accuracy of 92.80%. Finally, Wu et al. [25] use 920 samples in six classes, e.g., different screw cracks, and achieve an accuracy of 97.3%, which is also not cross-validated. Overall, the analysis shows that most studies lack either broad defect coverage or cross-validation and are therefore only suitable to a limited extent for the methodologically and empirically robust development of a model relevant to such a safety-critical application. Aydin et al. [15] do use such validation, but only cover three classes. Models that cover five or more classes are few and do not use cross-validation. Although both broad defect-type coverage and robust evaluation are essential from a practical point of view, existing studies do not meet these requirements simultaneously, leaving a relevant research gap that the present work addresses. Compared with the approaches summarized in Table 1, our method differs in three aspects. First, we address seven different defect classes in a peer-reviewed dataset, thereby offering broader coverage than existing approaches, which typically consider fewer than five types of defects. Second, this study adopts a novel dual-branch architecture that combines CNN-based local feature extraction with a Transformer branch capturing global context, whereas previous studies have mainly relied on single-branch approaches with CNNs. Third, the proposed architecture, DualSightNet, is rigorously validated using stratified 5-fold cross-validation, while most existing works report single train–test splits. Together, these distinctions demonstrate that our approach simultaneously achieves broad defect coverage, hybrid feature extraction, and robust evaluation, criteria that have not been jointly met by any previous study.

Model architecture

DualSightNet.

The proposed architecture, DualSightNet, integrates convolutional and transformer-based components, leveraging ResNet50 to extract fine-grained local features and Swin Transformer V2 to get more global relationships. This combination is particularly beneficial for the detection of rail defects, as such defects can vary in size, shape, and intensity. The integration of local and global perspectives contributes to the precise classification of these defects. The network is initialized with pretrained ImageNet weights, which promote faster convergence and improved generalization [26]. ResNet50 has been shown to be a robust and scalable backbone [27], with a structural design that helps reduce overfitting [28], thereby enhancing its generalization capabilities [29]. The Swin Transformer V2 enhances DualSightNet by enabling the modeling of global dependencies between local defect features and the broader rail structure. This global contextual understanding is particularly valuable in railway surface defect classification, where small or subtle defects often resemble non-defective components such as the rail bed or surface texture. By integrating global self-attention with local feature extraction from ResNet50, the model is able to distinguish visually similar patterns more accurately and classify defects with higher robustness and precision. The structure of DualSightNet is shown in Fig 1. Both ResNet50 and Swin Transformer V2 independently generate deep feature maps from the input image. ResNet50 produces a feature map () of size (B, 2048, 7, 7), whereas Swin Transformer V2 outputs a vector () of size (B, 1024), which is already in a reduced one-dimensional form. The feature map for ResNet50 is then reduced using adaptive average pooling (AAP), resulting in a feature map size of (B, 2048, 1, 1) and then transformed into a one-dimensional vector of size = (B, 2048) by flattening. Eq 1 illustrates this process.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Visualization of DualSightNet.

The two architectures, ResNet50 and Swin Transformer V2, each extract key features from the input images of the dataset [18]. From the resulting feature maps, one-dimensional vectors are generated and subsequently fused with the gating mechanism. The fused vector is then processed by the Efficient Channel Attention (ECA) module, which highlights the most relevant features. This enhanced vector is passed through two fully connected layers to produce the final classification output.

https://doi.org/10.1371/journal.pone.0340789.g001

(1)

Afterward, both one-dimensional vectors are individually passed through batch normalization to stabilize and normalize the intermediate representations [30]. The batch-normalized vectors ( and ) are then scaled to the same number of channels (1024). This is followed by fusion using a gating mechanism [31], where g is a learnable weighting vector:

(2)

In the implementation used, g is a channel-wise vector with the same dimensionality as the fused feature representation (1024). It is computed independently for each input sample. The output O is then normalized using batch normalization, followed by an ECA attention module, which highlights the most informative channels within the fused feature vector. ECA provides an efficient attention mechanism with minimal computational overhead. The enhanced feature vector, which retains the same dimension, is then passed through two fully connected layers (FC) separated by a dropout layer to mitigate overfitting. The number of neurons in the FC layer is determined by hyperparameter tuning, followed by a ReLU activation and dropout. The second FC layer contains 7 neurons corresponding to the number of classes and produces raw logits without an activation function, as the softmax operation is implicitly applied by the cross-entropy loss during training. This architecture leverages the complementary strengths of the fused features, resulting in a more robust and reliable classifier for complex image classification tasks.

Efficient channel attention.

Efficient Channel Attention (ECA), shown in Fig 2, is a lightweight attention module designed to selectively emphasize the most relevant channels in a feature map. Compared to the earlier Squeeze-and-Excitation (SE) module, ECA introduces several improvements. Unlike SE, which relies on complex non-linear transformations and dimensionality reduction, ECA avoids such operations by applying a simple 1D convolution without reducing channel dimensions, making it both efficient and parameter-light [35]. The process begins with Global Average Pooling (GAP), which produces a 1D vector. This vector is then passed through a 1D convolutional layer with an odd kernel size k. The kernel size is not fixed but adaptively determined based on the number of channels C as follows [35]:

(3)

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Schematic representation of efficient channel attention (ECA) [35].

https://doi.org/10.1371/journal.pone.0340789.g002

This method maintains both effectiveness and efficiency. While the SE block relies on computationally expensive fully connected layers, ECA achieves similar performance with significantly lower overhead by using a lightweight 1D convolution with an adaptively determined kernel size [35]. γ represents the scale factor and b the offset. Finally, the result is rounded to the nearest odd number [35]. To emphasize the most important feature channels, the resulting vector is multiplied element-wise with the original feature map. [35]. As a lightweight module, ECA effectively highlights the most relevant features. This is especially beneficial after feature fusion, where global representations from the Swin Transformer V2 are combined with local features from ResNet50. In this context, ECA enables the model to automatically identify and enhance the most informative channels within the fused representation.

Process of training

The entire training process is shown in Fig 3. For training and testing the architecture, an NVIDIA L40S GPU with 48 GB memory and PyTorch 2.5.0 were used. Python version 3.11.7 and CUDA 12.4.1 were also employed. Before model training, a stratified 5-fold cross-validation was applied using the scikit-learn library [36] (version 1.5.2) to divide the dataset into five equally sized subsets. This method was chosen to ensure that each fold maintains the original class distribution, unlike standard k-fold cross-validation, which may produce biased results due to class imbalance. In each iteration, four folds (80%) are used for training, and one fold (20%) is held out for testing. This process is repeated five times, rotating the test fold each time to ensure robust performance evaluation. Within the training set of each fold, 10% is further reserved for validation and hyperparameter tuning. This proportion yielded better results in preliminary tests than validation splits of 15% and 20%. Note that the validation data is only used for hyperparameter tuning and not for training or testing. Using a higher percentage, therefore, reduces the training dataset, which could explain the worse results. No data augmentation techniques or other image enhancement methods were applied before the training process. The original images from the dataset were used. All images are resized to 224 × 224 pixels with three RGB channels and normalized using the ImageNet mean and standard deviation. To address the imbalance between the individual defect classes, the class weights are integrated into the loss function () in order to also take rarer classes into account. To reduce overfitting and improve generalization, the training data is augmented using RandomRotation(degrees=10), RandomResizedCrop(scale=(0.9, 1.0)) and RandomAffine(degrees=0, translate=(0.1, 0.1). These transformations account for variability in image scale and position, making the model more robust to input variations. All transformations are applied on-the-fly during training. These techniques are only applied to the training data. The validation and test data are not augmented. Each model is trained for a maximum of 100 epochs with a batch size of 16. The training consists of two phases: transfer learning and fine-tuning. A batch size of 16 was used during transfer learning to maintain stable optimization and ensure comparability between runs, since only the classification head was updated at this stage. This size offered a practical balance because smaller batches can produce noisier gradients, while larger ones increase memory consumption. During fine-tuning, when all layers were unfrozen, batch sizes of 8, 16, and 32 were tested to examine their influence on convergence and generalization. During transfer learning, the pretrained feature extraction layers (based on ImageNet weights [26]) are frozen, while the fully connected layers remain trainable. In the fine-tuning phase, the entire model is updated. Hyperparameter optimization is performed separately for both training phases using Optuna (version 4.2.1) with a Tree-Structured Parzen Estimator (TPE) approach, testing the following parameters: dropout, learning rate, weight decay, number of units in the first fully connected layer, batch size, and optimizer. Some parameters are only tested in transfer learning or fine-tuning. An overview of these parameters with the corresponding search interval can be found in Table 2; no other parameters are tested. For each fold, 20 trials are conducted, and the configuration that yields the lowest validation loss is selected. Each trial is allowed to run up to 100 epochs. To avoid overfitting and reduce computational overhead, early stopping (patience=10, min_delta=0.001) is implemented, terminating training if the validation loss does not decrease over ten consecutive epochs. After identifying the optimal hyperparameters, a final transfer learning model is trained using the same configuration as during optimization. The model with the lowest validation loss is saved and used as the base for the subsequent fine-tuning phase. In fine-tuning, the hyperparameters (see Table 2) are re-optimized, and additional feature extraction layers are unfrozen to enable further domain adaptation. The hyperparameter tuning now again includes dropout, learning rate, weight decay, and the number of units in the first fully connected layer, and additionally incorporates the batch size. Again, the best-performing model based on validation loss is stored. After training, the final model is evaluated on the 20% test set using the following performance indicators such as Accuracy (Eq 4), Balanced accuracy (Eq 5), True positive rate (Eq 7) (sensitivity or recall), True negative rate (Eq 8) (Specificity), Positive predictive value (Eq 9) (precision), Negative predictive value (Eq 10), Cohen’s Kappa (Eq 6), and F1-Score (Eq 11). These metrics are computed for each fold, stored, and finally averaged across all five folds to provide a comprehensive overview of model performance. The training time for DualSightNet was 14 hours and 39 minutes, whereas the baseline models required less time for comparison, with 4 hours and 25 minutes for ResNet50 and 12 hours and 48 minutes for Swin Transformer V2.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Training and Evaluation approach: The dataset is split into a training set and a test set.

Data augmentation is applied to the training set, followed by model training using transfer learning and fine-tuning. Finally, the resulting model is evaluated on the test set.

https://doi.org/10.1371/journal.pone.0340789.g003

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Overview of the hyperparameters used for hyperparameter tuning.

TL = Transfer learning; FT = Fine-Tuning.

https://doi.org/10.1371/journal.pone.0340789.t002

Dataset

This study makes use of the Railway Track Surface Faults Dataset by Arain et al. [18], which offers a comprehensive and diverse set of images designed for the detection and classification of different defects on tracks and is available under DOI:10.17632/8hxtgyyxrw.2. The dataset was chosen because its seven defect classes capture most defect types encountered in real railway environments, and the images were recorded under authentic operational conditions. The resulting data include natural noise, shadows, and variations in the ballast bed, creating realistic variability that supports robust model generalization. These properties make the dataset well-suited for training and evaluating reliable railway defect-classification models. Data collection took place at Kotri Junction, Pakistan, one of the busiest sections of the national railway network. Two EKEN H9R cameras were mounted on a railway inspection vehicle, each operating at 120 frames per second with a field of view of approximately 14 inches. To ensure sufficient image detail, the inspection vehicle maintained a controlled speed of 20 km/h throughout recording. Based on the available images, the data was collected during the day. Although there are no signs of adverse weather conditions such as rain, precise weather conditions cannot be determined. After data collection, the recorded video material was pre-processed to remove irrelevant sections, followed by frame extraction and manual annotation. No further information about subsequent data augmentation or image enhancement is mentioned. The dataset contains a total of 5,153 images with a resolution of 1280 × 720 px, divided into seven defect categories as can be seen in Fig 4: Cracks, Flakings, Grooves, Joints, Shellings, Spallings, and Squats. The respective number of individual classes can be found in Table 3. The defects occurred naturally, with some stemming from actual operational incidents, thus providing authentic examples for research.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Overview of example images showing defects present in the dataset [18].

https://doi.org/10.1371/journal.pone.0340789.g004

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Overview of the seven defect classes with the number of images per class used in this study.

https://doi.org/10.1371/journal.pone.0340789.t003

Evaluation metrics

We employ multiple performance metrics to evaluate the effectiveness of the proposed model. These include accuracy, kappa, true positive rate, true negative rate, balanced accuracy, positive predictive value, negative predictive value, F1-score. The following metrics refer to the multi-class case, taking into account the class weighting, n_i represents the number of samples per class and S denotes the total number of samples. The following provides a brief explanation of each metric:

1. Accuracy
   Accuracy [37] reflects the overall proportion of correct predictions made by the model. It is defined as:(4)
   where TP, TN, FP and FN denote true positives, true negatives, false positives and false negatives, respectively.
2. Balanced Accuracy
   Balanced accuracy is particularly useful when dealing with imbalanced class distributions. In multiclass settings, as addressed in this study, it is computed as the average of the TPRs across all classes [38]. N denotes the number of classes.(5)
3. Kappa
   Cohen’s kappa [39] () measures the agreement between predicted and actual classifications, adjusted for the agreement that might occur by chance. It ranges from –1 (complete disagreement) to 1 (perfect agreement):(6)
   where P_o is the observed agreement and P_e the expected agreement by chance.
4. True Positive Rate (TPR)
   Also known as recall or sensitivity, TPR indicates the model’s ability to correctly identify positive cases [40]. TPR is calculated by [41]:(7)
5. True Negative Rate (TNR)
   TNR, also known as specificity, is analogous to the TPR, but for the negative class. It measures how effectively actual negative instances are correctly classified as negative [42]:(8)
6. Positive Predictive Value (PPV)
   PPV [43], also known as precision, quantifies the proportion of positive predictions that are actually correct:(9)
7. Negative Predictive Value (NPV)
   In contrast to the PPV, the NPV indicates the proportion of true negative classifications among all instances that were predicted as negative [44]:(10)
8. F1-Score
   The F1-score [37] represents the harmonic mean of precision (PPV) and recall (TPR), balancing both metrics in a single value. It is particularly useful for evaluating models on imbalanced datasets. F1 is calculated as follows for the multiclass case [45]:(11)

Interpretation of the results

The effectiveness of DualSightNet with ECA is illustrated in Fig 9. This figure shows the absolute number of misclassifications for each architecture: ResNet50, Swin Transformer V2, DualSightNet without ECA, and DualSightNet with ECA. A clear reduction can be seen here, starting with ResNet50, which has the most errors with 229, followed by Swin Transformer V2 with 210. A more substantial improvement appears with DualSightNet without ECA, which has 147 errors. This number is further reduced by DualSightNet with ECA, which achieves the lowest overall error count. Compared to the other architectures, this results in 102 (ResNet50), 83 (Swin Transformer V2), and 20 fewer errors. In other words, the proposed DualSightNet manages to reduce errors by 44.54% (ResNet50), 39.52% (Swin Transformer V2), and 13.61% (DualSightNet without ECA). This result is of great significance in quality control and in the detection of rail defects. Certain defects on the rail can be safety-relevant depending on their severity. If these defects are not detected or are incorrectly classified, the train traveling on that rail can ultimately be endangered, putting human lives at risk. The proposed architecture reduces unnecessary follow-up investigations by reducing classification errors. Thanks to more accurate classification, maintenance can be carried out earlier and more reliably. Safety-critical areas can also be prioritized more precisely. As a result, personnel and equipment can be deployed more efficiently, reducing costs and making more effective use of human resources. In summary, DualSightNet helps improve the quality of defect detection and, consequently, safety.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 9. Representation of the absolute errors of the respective architectures.

DualSightNet with ECA achieves the best value.

https://doi.org/10.1371/journal.pone.0340789.g009

Although some classes are severely underrepresented in the dataset, such as cracks (40 images), joints (11), or grooves (8), and other classes such as flakings (2,829) and squats (1,844) are heavily overrepresented, the results of the confusion matrix (Fig 6) show that the model can still reliably detect these underrepresented defects. This indicates that the weighted loss function and the architecture can effectively learn and recognize minority classes. However, there remains a certain risk arising from the limited number of real training examples. Even if the balanced accuracy is high, it is based on the few images available for the underrepresented classes. This means that variations in the images or new forms of these defects during real operation may not have been fully covered by the training data. As a result, model performance for these rare defects may fluctuate more and have lower generalization ability under real-world operating conditions than for classes with many training examples, such as spalling or squats, especially for new, previously unknown variants of the defects.

When comparing DualSightNet with relevant studies, two aspects stand out in particular. A significant proportion of studies do not evaluate their models using robust methods such as cross-validation [14,16,17,19,20,22,24,25]. The use of train-test splits can obscure the variability of model performance, as, for example, a particularly advantageous split may be used for validation, falsely suggesting high overall model performance. Our approach uses stratified 5-fold cross-validation and provides a more reliable robustness estimation, better meeting the requirements of safety-critical railway inspection. Only in [15] is cross-validation used, whereby the focus here is on three fastener states (healthy, damaged, missing). Furthermore, the fact that many approaches focus on fewer than five defect classes limits their use in real-world inspection conditions, as it means that many defects are overlooked. In our approach, we use a dataset that contains seven different types of errors recorded under realistic environmental conditions. This achieves broader defect coverage, which is important for inspections. Jatoi et al. [17] also used the same dataset, enabling us to compare our study with it most closely. However, in a direct comparison, they did not use hyperparameter tuning or cross-validation. They also employed data augmentation techniques to artificially expand the dataset. Their best model, DenseNet121, achieved a test accuracy of 92.80%. In contrast, we achieved an average accuracy of 96.53% and a balanced accuracy of 97.55%, both of which are cross-validated.

Implications of the study

The findings of this work indicate that deep learning-based image classification can serve as a valuable component in the automation of railway surface defect detection. The strong performance of the proposed architecture underlines its practical relevance and makes it a suitable candidate for integration into a larger defect-classification system. A key question for future implementation is the optimal location for deploying such a system. One possibility is to integrate it directly into passenger or freight trains, enabling continuous monitoring during regular operation. However, it is essential to first assess whether such an installation is technically feasible. Trains are often closed systems, and retrofitting them, provided that space is available, requires thorough evaluation to ensure that the deep learning system does not interfere with existing onboard technologies. Moreover, sufficient computational resources must be available to allow for fast and reliable classification. These are critical factors, especially if the system is to be integrated into regular passenger or freight trains, as no external monitoring vehicles may be available. In such cases, the system must function without altering existing operational conditions such as train speed. Due to these constraints, this option appears to be less suitable for practical deployment. A more practical solution is the use of a mobile platform, such as an inspection vehicle. Since such vehicles are not intended for passenger or freight transport, safety-critical retrofitting requirements are likely to be less restrictive. Sufficient computing resources are still required for reliable defect detection. One major advantage of inspection vehicles is their ability to adjust speed. If, for example, real-time detection at 150 km/h is not feasible due to limited processing capacity, these vehicles can reduce their speed to ensure optimal inspection performance. Another possible solution is the implementation of the system on drones. These can fly along railway lines and assess the condition of track sections from above. However, this method has certain limitations, including a restricted range, limited battery life, and potential issues with weather conditions that may hinder flight operations. The hybrid approach used in this study achieves an average inference time of 3.00 ms per image while operating at 21.20 GFLOPs. Although the hybrid architecture has higher computational costs and longer inference time compared to simpler architectures, such as ResNet50 (1.33 ms/img, 4.11 GFLOPs) and Swin Transformer V2 (2.48 ms/img, 17.08 GFLOPs), it achieves increased balanced accuracy in a safety-critical area. In practical applications, the extent to which longer inference times or higher computing costs are critical should be examined. Switching to simpler models may result in a decrease in recognition accuracy, which, in the worst case, could endanger human lives.

Limitations

This study introduced DualSightNet, a novel and high-performing hybrid deep learning architecture. Although the results highlight the effectiveness and robustness of the proposed approach, certain limitations of the work should be acknowledged. The dataset employed in this study includes seven predefined defect classes, which means the strong performance of the proposed architecture is currently limited to this specific set of defects. In practice, however, additional and potentially unforeseen defects may appear on railway tracks, and the detection of such defects cannot be evaluated with this dataset. Consequently, the model’s ability to generalize to a wider variety of defects remains uncertain. Furthermore, since the dataset does not include a wide range of weather conditions, no conclusions can be drawn about the robustness of DualSightNet under such varying environmental factors.

Future work

Building on the outlined limitations, several directions for future research emerge. To enhance the external validity and generalizability of DualSightNet, future studies should aim to evaluate the model on additional datasets that contain not only different imaging conditions but also new defect classes not present in the current dataset. Such evaluations would allow for a more comprehensive understanding of the model’s robustness across varying scenarios. Furthermore, future research could address the current limitations related to the narrow range of environmental conditions, such as weather variations or different times of day, by leveraging generative approaches. Generative Adversarial Networks or diffusion models offer promising tools for this purpose, as they can be used to augment existing datasets with realistic synthetic variations that reflect diverse real-world conditions. Additionally, generative models can be employed to tackle dataset imbalance by producing representative samples for underrepresented defect types or rare scenarios, ultimately contributing to more balanced and effective model training.