Classification model structure

The classification model is based on a combination of the pre-trained VGG19 model, enriched with additional layers designed specifically for the image classification task. The CNN architecture devised by the Visual Geometry Group (VGG) is widely recognised for its depth, characterised by many convolution layers. It has played a key role in the evolution of object recognition models, outperforming many benchmarks on various tasks and datasets. It is pretrained with ImageNet datasets and can still outperform unseen datasets, making it one of the most used image recognition architectures. Multiple variants exist for such a network, including VGG-16 and VGG-19, differing only in the total number of layers. VGG19 is an advanced version of the VGG architecture, incorporating 19 convolution layers [51]. It consists of several convolutional blocks, each comprising multiple convolutional layers followed by pooling layers. The model utilises small filters (3×3) with a stride pattern of 1 and padding of 1 to preserve extracted feature sizes. Using this pattern for the convolutional layers means that the convolutional filters move one pixel at a time across the input data, and one layer of zero pixels is added around the input to maintain its size during the convolution process. After the convolutional blocks, the network connects to fully connected layers for classification. While the VGG19 architecture is relatively established and widely used in image classification tasks, it was chosen for this project due to its depth, simplicity, and ability to generalise across unseen datasets. VGG19 remains a competitive architecture for tasks involving image recognition, and its extensive use allows for leveraging pretrained weights on ImageNet, speeding up training and improving accuracy. In this work, the classification model comprises both the VGG19 model and additional layers. The VGG19 layers adhere to the standard architecture, featuring blocks of convolutional layers followed by pooling layers. These layers progressively capture features across different scales and complexities. Additional layers are then introduced after the VGG19 layers to tailor the model for the image classification task. These supplementary layers include:

• a flattened layer to transform the outputs into a one-dimensional vector;
• dense (fully connected) layers for final classification, including dropout layers for regularisation;
• batch normalisation layers for normalising activations and stabilising learning;
• the final dense layer, with neurons corresponding to the number of classes (letters) in the classification task.

In our implementation, the input image size is (64,64,3), and the feature extraction part strictly follows the VGG19 configuration. It consists of five blocks of convolutional layers with small 3x3 kernels and a stride of 1. After flattening the output of the final block, we added two dense (fully connected) layers with 512 units each. Both are activated using ReLU and followed by a Dropout layer with a rate of 0.5 to reduce overfitting. Each dense block also includes Batch Normalisation to stabilise learning and improve convergence. The final output layer contains 26 neurons (one per ASL letter), activated using Softmax. The network was trained using the Adam optimiser with a 10⁻⁴ learning rate and categorical cross-entropy loss. Fig 3 shows the full architecture of the proposed model in a horizontal layout for clarity and reproducibility.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Architecture of the proposed CNN model based on a modified VGG19 structure.

The network consists of five convolutional blocks (Conv1–Conv5), each followed by ReLU activations and max-pooling operations. After the feature extraction phase, the feature maps are flattened and passed through two fully connected (Dense) layers of 512 units each, followed by ReLU activations, Dropout (rate = 0.5), and Batch Normalisation. The final output layer uses a Softmax activation with 26 neurons to classify ASL alphabet signs.

https://doi.org/10.1371/journal.pone.0336481.g003

Model limitations and motivation for improvement

The results obtained show a test accuracy of 95%; however, when evaluated on out-of-distribution images, not present in the original dataset, the model exhibits a clear divergence in performance (https://www.kaggle.com/code/damionjoyner/sign-language-classification-cnn-vgg19). It fails even to recognise simple hand signs, revealing significant limitations in its generalisation ability. A more in-depth analysis of the model behaviour uncovers several fundamental weaknesses that severely undermine its capacity for real-world deployment. The training dataset is imbalanced in class distribution and limited in subject diversity, as it includes samples from only five individuals, all with similar skin tones. This dual constraint introduces a strong learning bias: the model tends to overfit to frequent classes and person-specific features, rather than learning gesture-invariant representations. As a result, its performance drops notably for rare classes and when applied to new users not represented in the training set. Furthermore, the small size of the dataset, combined with its imbalance, significantly increases the risk of overfitting. Although the overall accuracy appears high, this metric can be misleading. The model may simply memorise dataset-specific patterns rather than learning transferable features. This concern is reinforced by the inconsistencies observed during additional evaluations, particularly on underrepresented classes. Another critical limitation lies in the way multimodal information is handled. Although the model uses both RGB and depth data, the fusion is performed through simple concatenation, which does not fully leverage the complementary strengths of these modalities. This naive fusion strategy may cause the model to overlook essential modality-specific cues or to resolve ambiguities inadequately, shortcomings that more advanced fusion mechanisms could potentially overcome. These interconnected limitations expose the structural weaknesses of the current approach and motivate the need for methodological improvements. Specifically, they justify the development of a more robust architecture that employs advanced multimodal fusion techniques, applies targeted data augmentation to mitigate class imbalance, and incorporates regularisation strategies to promote generalisation. These enhancements form the basis of the improvements proposed in the following section.

An enhanced interpretable solution for SLR

To overcome the limitations of existing ASL datasets and improve the robustness and interpretability of gesture recognition models, we propose a three-step approach: (1) data augmentation and diversification, (2) a mask-based preprocessing strategy, and (3) an explainable classification framework. These three steps are described in the following subsections.

Data augmentation and diversification.

The accuracy of ASL recognition models is often hindered by the limited variability in existing datasets, particularly with respect to hand shape, skin tone, background, and lighting conditions. To address this, we significantly changed and diversified our dataset. Instead of relying on a standard collection of 100,000 images covering 24 letters, we compiled a new dataset of 325,000 cropped RGB images representing all 26 letters of the ASL alphabet as illustrated in Fig 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Comparison of label distributions between the initial and the newly collected ASL letter datasets.

The left bar chart shows the original dataset with imbalanced counts across 24 letters, while the right chart illustrates the new dataset, which includes 12,500 per letter and covers all 26 letters of the ASL alphabet with uniform distribution.

https://doi.org/10.1371/journal.pone.0336481.g004

These images were gathered from four publicly available sources to ensure heterogeneity and cover a wide range of environments and hand configurations:

• Kaggle - ASL Alphabet https://www.kaggle.com/datasets/grassknoted/asl-alphabet
• Kaggle - ASL Alphabet Dataset https://www.kaggle.com/datasets/debashishsau/aslamerican-sign-language-aplhabet-dataset
• Kaggle - ASL Alphabet Test https://www.kaggle.com/datasets/danrasband/asl-alphabet-test
• Kaggle - Synthetic ASL Alphabet https://www.kaggle.com/datasets/lexset/synthetic-asl-alphabet

The dataset was constructed by merging multiple publicly available ASL alphabet datasets. After removing duplicates and balancing the classes, we obtained approximately 325,000 images (12,500 per class). Since the images were mixed and reorganized into a unified dataset, we report only the final distribution. The processed dataset and split files are made publicly available at (https://www.kaggle.com/datasets/fatimaelqoraychy/dataset). Although ‘j’ and ‘z’ are dynamic ASL signs requiring motion to be fully expressed, some of the datasets we employed include static image approximations for these letters. In this study, we used these static representations to cover the complete 26-letter ASL alphabet. While these approximations provide useful proxies, they cannot fully reproduce the motion aspect of the signs. In this study, we did not employ geometric or photometric data augmentation (e.g., rotations, shifts, zooms, flips, or brightness variations). The only preprocessing applied was pixel-value rescaling (1/255), performed online during training using the Keras ImageDataGenerator. This ensured consistent normalization across the dataset without altering the original distribution of hand images.

Model training and evaluation.

Following the segmentation process, two distinct gesture recognition models were trained. The first model was trained on the original RGB images, while the second model utilised the binary hand masks generated by the U-Net segmentation model. The dataset collection and processing workflow began with acquiring a total of 325,000 labelled images. Each image underwent two distinct preprocessing stages: first, it was used as an RGB image in its original form, and second, it was passed through the segmentation model to generate the corresponding hand masks. All images were resized to a fixed resolution of 64×64 pixels, which is the required input dimension for the VGG19-based architecture used in our models as presented in Fig 5. To enable reliable model evaluation, the dataset was split into 76.5% for training, 13.5% for validation, and 10% for testing, using random shuffling and a fixed random seed labeled SEED equal to 99 to ensure reproducibility. This guarantees consistent data partitioning across different runs. To monitor and mitigate overfitting during training, we recorded both training and validation losses at each epoch across all 25 training epochs. Early stopping was applied by monitoring the validation loss. The training was halted using an early stopping criterion once no improvement was detected over a set number of consecutive epochs. The test set was used to evaluate the final model after training was complete. A summary of the training hyperparameters is presented in Table 2. All experiments were conducted using the Kaggle cloud-based environment, which provides NVIDIA Tesla P100 GPUs. Each model was trained for approximately six hours.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Overview of the gesture classification process using RGB images and segmented hand masks.

Both input types are preprocessed and passed through a CNN for gesture prediction.

https://doi.org/10.1371/journal.pone.0336481.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Hyperparameters used in training the classification models.

https://doi.org/10.1371/journal.pone.0336481.t002

Real-time prediction interface

We developed a real-time prediction interface using OpenCV and TensorFlow to evaluate model performance in real-world settings. The interface captures live images of hand gestures through a webcam and performs on-the-fly predictions using the pre-trained model. It simultaneously displays the predicted ASL letter and overlays visual explanations generated through integrated XAI methods.

The interface operates by continuously capturing frames, resizing them to 64×64 pixels to match the model’s input dimensions, and normalising the pixel values. The processed frame is then passed through the model to obtain prediction probabilities, and the class with the highest score is selected as the predicted letter. The predicted class index is converted back into a human-readable label using a loaded label binarized object.

Performance evaluation. To quantify the real-time usability of the system, we measured both frame rate (frames per second, FPS) and per-frame latency (milliseconds). On our test hardware (Intel CPU), the interface achieved an average of 10 FPS with a mean latency of approximately 100 ms per frame.

Explainable AI for SLR model enhancement

We apply XAI techniques to our SLR models to enhance transparency and interpretability. In ASL recognition, it is crucial to ensure that a model focuses on semantically meaningful regions of the input, such as hand shapes and orientations, rather than background artefacts. This is particularly important for model validation and for fostering confidence among users and system developers. Explainability also serves as a practical tool for model debugging and for detecting potential biases or spurious correlations learned during training [44]. For instance, if a model consistently misclassifies a gesture due to attention on irrelevant background elements, explainability methods can help identify and correct such failures. Furthermore, it provides traceability for misclassifications by highlighting decision-relevant input regions. However, due to ASL gestures’ complexity and spatial variability, no single XAI technique is sufficient. We therefore employ three complementary and widely recognised XAI methods: Grad-CAM, LIME, and integrated gradients.

Gradient-weighted class activation mapping.

Gradient-weighted Class Activation Mapping (Grad-CAM) [23] is a widely used post-hoc explainability technique that provides visual explanations for CNNs by highlighting image regions most relevant to a model’s decision. Given an input image and a target class c, Grad-CAM produces a heatmap that localises the spatial regions that have the strongest influence on the output score y. This is particularly useful in visual tasks like SLR, where fine-grained hand features drive classification. To compute Grad-CAM, we first calculate the gradient of the output score y concerning the feature maps A of a chosen convolutional layer:

These gradients are globally averaged to obtain importance weights for each feature map:

Where Z is the total number of pixels in the feature map.

The class-discriminative localization map is then computed as:

The ReLU operation retains only features with a positive influence on the class score. The resulting heatmap is then upsampled and overlaid on the input image using a JET colourmap, where red and yellow indicate high importance and blue indicates low relevance.

Local interpretable model-agnostic explanations.

Local Interpretable Model-agnostic Explanations (LIME) [16] is a widely used technique designed to interpret the predictions of any black-box model by approximating it locally with an interpretable model. Unlike Grad-CAM, which is specific to CNNs, LIME is model-agnostic and can be applied to any classifier. The core idea is to understand a model’s decision for a specific instance x by generating a set of perturbed samples around x and observing how the model’s predictions vary. For each perturbed sample , the model predicts an output , and a locality-aware weight is assigned based on the proximity to the original instance x. A simple, interpretable model g, such as a linear regression or decision tree, is trained on this locally weighted dataset to approximate the complex model f in the vicinity of x. The optimisation objective is:

where measures the fidelity of g to f locally, defines the locality, and is a complexity penalty for g. In image classification, LIME segments the image into super pixels and perturbs it by randomly turning these super pixels on or off. It then identifies which super pixels most strongly influence the prediction by observing the model’s output across these perturbations.

Integrated gradients.

Integrated Gradients [53] is a model-specific explainability method designed to attribute a deep neural network’s prediction to its input features. It addresses the limitations of standard gradients, which can be noisy or vanish due to saturation. The method works by computing the average gradients as the input transitions from a baseline (typically a zero or black image) to the actual input. For an input x and baseline , the attribution for the i-th feature is defined as:

This integral is approximated numerically via summation over m steps:

The integrated gradients method satisfies desirable axioms such as sensitivity and implementation invariance, making it theoretically sound for attribution tasks.

Evaluation metrics for saliency maps

To quantitatively assess the compactness and dispersion of the explanations, We computed two complementary metrics on the saliency maps produced by Grad-CAM, LIME, and Integrated Gradients.

.

Energy concentration.

Let denote the normalized saliency map, with and . After sorting all pixel intensities in descending order , the energy concentration at fraction p (e.g., p = 0.05 for top-5%) is defined as:

(1)

A higher E(p) indicates that a small fraction of pixels captures most of the attribution energy, i.e., the explanation is more compact and localized.

Entropy.

Considering the normalized heatmap as a discrete probability distribution, The entropy is defined as:

(2)

where ε is a small constant to prevent numerical instability. Lower entropy values correspond to more focused saliency maps, while higher values reflect more diffuse explanations.

A multi-level interpretability strategy for SLR

The decision to integrate multiple XAI techniques, rather than rely on a single method, is motivated by the complementary nature of the insights each one provides. Grad-CAM, LIME, and integrated gradients represent distinct families of interpretability methods: Grad-CAM leverages convolutional activations to visualise class-specific saliency maps, LIME builds local surrogate models through feature perturbation, and integrated gradients quantify the cumulative contribution of each input pixel via a path-integrated gradient from a baseline. Each method brings specific strengths and limitations. Grad-CAM produces intuitive heatmaps but may lack fine-grained spatial resolution. LIME offers detailed, instance-level attributions but can be unstable across similar inputs. The integrated gradients method provides pixel-level precision but is sensitive to baseline selection and may miss high-level spatial structure. By combining all three, we obtain a multi-level perspective of the model’s internal logic—from broad spatial focus to localised feature attribution and cumulative influence. This combined framework is particularly crucial in SLR, where classification hinges on nuanced finger position, contour, and orientation differences. Relying on a single method may result in incomplete or misleading interpretations. Therefore, the integration of Grad-CAM, LIME, and integrated gradient methods is not redundant—it is essential for achieving robust, trustworthy interpretability, enabling the diagnosis of both successful and failed predictions with higher fidelity.

For quantitative evaluation, we report precision, recall, and F1-score. These metrics are particularly appropriate for our context due to class imbalance and the presence of visually ambiguous gestures. Precision measures the exactness of the predictions, recall assesses completeness, and F1-score offers a harmonic balance of both. Unlike overall accuracy, these metrics provide a more nuanced assessment of model behaviour, especially on minority or error-prone classes. To guide both interpretability and performance analysis, we selected representative subsets of ASL letters:

• Simple Signs: ‘a’ and ‘l’
• Challenging Pairs: ‘i’ vs. ‘j’, and ‘m’ vs. ‘n’

The first pair (‘a’ and ‘l’) serves as a baseline to verify that the model correctly handles straightforward gestures. The second and third sets represent more ambiguous cases with high visual similarity. These allow us to test the model’s discriminative ability and examine how explanation methods respond in both correct and incorrect classifications under challenging conditions. All scripts and code used to generate the results in this study are available on Kaggle at https://www.kaggle.com/code/fatimaelqoraychy/sign-language-classification-cnn-vgg19-1. The notebook is publicly accessible and can be used to reproduce the experiments reported in this manuscript.

Model for gesture recognition

As shown in Figs 6 and 7, both models converge early during training. The RGB model reached optimal performance at epoch 13, while the mask-based model continued improving until epoch 24. This difference reflects the RGB model’s faster learning due to richer visual features, whereas the mask model, relying solely on shape information, required more epochs to generalise effectively. In both cases, the use of early stopping ensured stable convergence and prevented overfitting. Both models demonstrated high performance in recognising ASL gestures during the test, with strong average precision, recall, and F1-scores. While Table 4 provides an overview of global performance, a more detailed breakdown is presented in Table 3, which reports the exact values of each metric for every letter in the alphabet.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Training and validation accuracy and loss curves for the RGB model.

https://doi.org/10.1371/journal.pone.0336481.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Training and validation accuracy and loss curves for the Mask-based model.

https://doi.org/10.1371/journal.pone.0336481.g007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Classification of ASL letters in the testing phase and comparison of classification performance (Precision, Recall, F1-score) between the RGB and Mask models.

https://doi.org/10.1371/journal.pone.0336481.t003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Performance comparison between the RGB and Mask models for ASL gesture recognition.

https://doi.org/10.1371/journal.pone.0336481.t004

To evaluate model performance in a real-world scenario, we implemented a real-time prediction interface capable of recognising ASL letters as presented in Sect . During this phase, we observed that the RGB model consistently predicted the correct letter, even for complex signs. However, in the case of ‘i’ and ‘j’, despite their similarity, the model provided the correct prediction. To better understand the source of these errors, we applied explainability methods to analyse which image regions the model relied on during prediction.

Explainability and model validation

To better understand our system’s prediction mechanisms, we analyze visual explanations generated by Grad-CAM, LIME, and integrated gradient methods across the two classification models. While the RGB model achieves consistently high accuracy across all letters, the mask-based model occasionally struggles, particularly confusing visually similar signs such as ‘n’ and ‘m’. Interestingly, it correctly classifies ‘m’ and successfully distinguishes other subtle variations such as ‘i’ and ‘j’. To systematically investigate this behaviour, we selected representative signs for qualitative analysis: simple and clearly distinguishable gestures (‘a’, ‘l’), a visually similar but correctly classified pair (‘i’, ‘j’), and a confused pair (‘m’, ‘n’). This strategy allows us to assess what features the model learns, how it fails, and how explanations differ between RGB and mask-based models.

Simple Signs ‘a’ and ‘l’: For the simple letters ‘a’ and ‘l’, all three methods show strong alignment between the model’s focus and the relevant gesture areas, as illustrated in Fig 8. Grad-CAM consistently highlights the hand’s central region, capturing the gesture’s overall shape, particularly the closed fist for ‘a’ and the raised thumb and index for ‘l’. This suggests the model is using spatially relevant information to make correct predictions. LIME provides contour-based insights, outlining the hand boundaries precisely. The highlighted regions correspond well with the hand’s silhouette, indicating that the model is sensitive to the global shape and the number of extended fingers. The integrated gradients method shows high activation in areas with clear structural edges, such as knuckles, fingertips, and wrists. These results confirm that pixel-level contributions are coherent with human intuition. This consistency across methods confirms the model’s robustness in recognising simple, well-separated gestures. These signs serve as effective baselines for validating that the explainability tools reflect correct model behaviour.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Explainability results for the simple ASL letters ‘a’ and ‘l’ using Grad-CAM, LIME, and integrated gradients methods.

Each column corresponds to a different explanation technique, while each row shows original inputs and their corresponding binary masks.

https://doi.org/10.1371/journal.pone.0336481.g008

Challenging Signs ‘i’ vs. ‘j’: The distinction between i’ and j’ lies primarily in the little finger’s slight curvature or motion-related aspect. Despite their visual similarity, both the RGB and mask-based models correctly classify these signs (Fig 9). Grad-CAM tends to produce similar heatmaps for both letters, often centred on the palm and lower fingers, suggesting reliance on coarse spatial features. LIME adds nuance while ‘i has tightly bound contours around the little, ‘j’ frequently shows extended contours, sometimes trailing into the curved stroke direction, hinting that LIME can capture implied motion. The integrated gradients method exposes localised pixel importance, with ‘j’ often showing extended activation beyond the little, consistent with its curved nature. The combined analysis suggests that while Grad-CAM alone may miss subtle motion cues, LIME and integrated gradients provide additional evidence that the model captures fine variations in finger trajectory.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Explainability analysis for visually similar ASL signs: ‘i’ vs. ‘j’ and ‘m’ vs. ‘n’ using Grad-CAM, LIME, and integrated gradients methods.

Each column corresponds to a different explanation technique, while each row shows original inputs and their corresponding binary masks.

https://doi.org/10.1371/journal.pone.0336481.g009

Challenging Signs ‘m’ vs. ‘n’: The pair ‘m’ and ‘n’ presents a fine-grained classification challenge, both involve fingers folded over the thumb, differing primarily in the number of visible knuckles—three for ‘m’ and two for ‘n’. While the RGB model correctly classifies both signs, the segmentation-based model systematically misclassified ‘n’ as ‘m’, as shown in Fig 9. Grad-CAM shows that in the mask-based model, both ‘m’ and ‘n’ result in nearly identical heatmaps, with broad activation spread over the entire hand region. This suggests that the model fails to localise the discriminative area, the number of folded fingers, which is critical for distinguishing the two letters. In contrast, the RGB model shows slightly more concentrated activation toward the finger area for ‘m’, reflecting finer spatial awareness. LIME provides a more informative contrast. For the RGB model, LIME outlines distinct superpixels around three folded fingers for ‘m’ and two for ‘n’, helping the model separate the two signs. However, in the mask-based model, the superpixel boundaries for ‘n’ are either missing, merged, or appear very similar to those for ‘m’. This visual similarity may explain why the model maps to the same internal representation. The integrated gradients method confirms this limitation. In the RGB model, the attribution maps for ‘m’ and ‘n’ highlight different finger segments with clear separation, three highlighted areas for ‘m’ , two for ‘n’. However, in the mask-based model, both maps show overlapping attribution across the top of the hand, with weak contrast between the two and three finger zones. This blurs the structural boundary between the two classes. These results suggest that the mask model fails to attend to the correct discriminative region (i.e., finger count), likely due to the absence of texture, shading, and depth cues. Without this fine-grained information, the visual difference between ‘m’ and ‘n’ collapses, and the model overgeneralizes, classifying both as ‘m’. In contrast, the RGB model benefits from additional visual signals that help preserve subtle structural differences.

To quantitatively assess the compactness and dispersion of the explanations, we computed two complementary metrics on the saliency maps produced by Grad-CAM, LIME, and Integrated Gradients. Energy concentration was defined as the proportion of attribution energy contained in the top k% most salient pixels (). Higher values indicate that the explanation is concentrated on fewer, more relevant regions. Entropy was calculated over the normalized heatmap to measure dispersion. Lower entropy values correspond to more focused explanations, while higher values indicate more diffuse attention. These metrics were computed for six representative ASL letters (a, l, i, j, m, n) and across both models (RGB- and mask-based), then averaged to provide a global comparison. Table 5 summarizes the averaged results across all letters. The results reveal distinct behaviors across the three explanation techniques. LIME produces the most compact explanations, with the highest concentration values (e.g., 21.6% at top-10%) and the lowest entropy (7.9), indicating highly localized attribution maps. Grad-CAM, by contrast, exhibits more diffuse saliency maps, reflected in the highest entropy (9.3) and only moderate energy concentration (13.5% at top-10%). Integrated Gradients yields the lowest energy concentration overall (12.4% at top-10%), with entropy values (8.3) falling between LIME and Grad-CAM, suggesting smoother and more spread attributions. Overall, these results confirm that the three XAI methods provide complementary perspectives: LIME highlights precise local dependencies, Grad-CAM emphasizes broader spatial regions, and Integrated Gradients distributes importance across both local and contextual cues. This quantitative analysis strengthens the qualitative observations and underlines the benefit of using multiple explanation techniques for robust model auditing in sign language recognition.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Average energy concentration (computed at top-k% pixels) and entropy.

Values are averaged across six representative letters (a, l, i, j, m, n) and across both input modalities (RGB and mask). Higher energy concentration indicates more compact explanations; lower entropy indicates less diffuse maps.

https://doi.org/10.1371/journal.pone.0336481.t005

Implications for big data

Recent advances in sign language recognition highlight the increasing availability of large-scale datasets covering multiple sign languages, signing styles, and recording conditions. Such variability reflects a Big Data challenge, where models must generalize across highly heterogeneous samples. Our work is relevant in this context because it integrates explainability into SLR, providing tools to validate and interpret models when trained on large and diverse datasets. In particular, the integration of multiple XAI methods (Grad-CAM, LIME, and Integrated Gradients) facilitates model auditing at scale by enabling transparent validation of predictions, even in massive datasets where manual inspection of each sample is infeasible. Moreover, as highlighted in recent studies on the AI-powered evolution of big data [54], explainability is increasingly recognized as a key requirement for deploying trustworthy models in real-world, data-intensive environments. Therefore, we view our contribution as a step toward scalable and interpretable SLR systems capable of operating under Big Data conditions.

Discussion

The experimental results presented in this study demonstrate high predictive performance for both RGB and mask-based models and expose deeper insights into the model behaviors through the lens of explainability. While the overall classification accuracy appears strong for both models (RGB: 99%, Mask: 98%), a deeper examination reveals key differences in generalisation and robustness. The RGB-based model consistently achieves better performance across ambiguous sign pairs, such as ‘m’ vs. ‘n’, which differ primarily by the number of extended fingers. This advantage stems from its access to rich visual cues, such as texture, shading, and skin tone, that enable the capture of subtle features. In contrast, despite eliminating background noise, the segmentation-based model lacks these visual depths and hence struggles with fine-grained discrimination. Moreover, the mask-based model demonstrates better robustness under noisy or cluttered environments, particularly in real-time settings where background conditions are variable. Its reliance on binary shape information makes it less sensitive to colour variations, lighting inconsistencies, or camera artefacts. This trade-off between robustness and discriminative power highlights the need for hybrid strategies that can balance visual richness with noise resilience.

We did not include a quantitative comparison with the baseline VGG19 model of Joyner, as this model failed to provide valid predictions on our datasets. This limitation was also reported by the original author on the project website, where it is noted that the model struggles with out-of-distribution samples. Our proposed models, by contrast, achieved consistent and interpretable results under the same conditions. One of this work’s most salient contributions is the comparative use of three XAI methods: Grad-CAM, LIME, and integrated gradients. Each explainability method brings a unique perspective: Grad-CAM identifies broader spatial attention, LIME reveals local sensitivity to super pixel-level features, and integrated gradients provide cumulative pixel-level attribution. While Grad-CAM offers intuitive heatmaps, it often lacks the resolution to distinguish fine-grained structural differences. LIME, though more precise, can vary between samples. The integrated gradients method excels at identifying localised saliency but struggles to convey global context. Their combined use triangulates the model’s behaviour, helping mitigate the limitations of any one method. Results show that for simple and clearly separated gestures such as ‘a’ and ‘l’, all three methods align in highlighting the correct hand regions across both models. This consistency supports the model’s ability to learn relevant features and provides confidence in its correct predictions. However, for visually similar gestures, particularly ‘m’ vs. ‘n’ divergence emerges. While the RGB model correctly classifies both signs with precise attribution maps (e.g., three vs. two finger contours), the mask-based model frequently misclassified ‘n’ as ‘m’. Explainability analysis reveals the root cause: Grad-CAM fails to focus on the relevant discriminative region; LIME outlines are inconsistent or overlapping; and integrated gradients show blurred attribution across the fingers. These results suggest that the mask-based model lacks the resolution and richness to differentiate subtle anatomical differences due to its reliance solely on shape. In contrast, both models perform well for ‘i’ vs. ‘j’. This is likely because the primary distinction, little finger curvature or implied motion, is still partially preserved in the binary mask. This highlights that not all structural differences are equally affected by the loss of RGB information; the ability to recognise dynamic or curvature-based features may remain intact in segmented representations, whereas subtle finger count distinctions do not. The RGB model outperforms the mask-based model not only in quantitative metrics but also in interpretability fidelity. The presence of colour, texture, and shading allows the RGB model to extract richer features and localise attention more effectively. These visual cues are especially crucial in gestures with fine-grained variations. Conversely, the binary mask strips away such information, which hinders the model’s ability to resolve ambiguous patterns like those in ‘m’ and ‘n’. This analysis underscores that segmentation-based input alone is insufficient for high-resolution gesture classification. To address this, several strategies are recommended:

• Feature Enhancement: Preprocessing techniques such as edge enhancement or skeletal overlays could improve the visibility of finger contours.
• Hybrid Input Models: Combining RGB and mask inputs could provide complementary benefits, shape consistency from masks, and detail richness from RGB.
• Attention Mechanisms: Using attention-based networks may help the model focus more selectively on discriminative regions.
• Explainability Guided Retraining: Insights from Grad-CAM or integrated gradients could guide targeted data augmentation or hard example mining during training.

In contrast to recent approaches such as SignExplainer [45], which primarily rely on LIME and SHAP to interpret feature contributions within ensemble classifiers or [48] introduces self-supervised learning with vision transformers and interprets decisions using SHAP, our work offers a multimodal, multi-technique explainability framework tailored to deep learning architectures. More recently, FedXAI-ISL [55] has proposed a federated and explainable framework for Indian Sign Language, and an explainable real-time SLR pipeline [56] has been introduced to address deployment at scale under big data conditions. While these contributions highlight the growing relevance of explainability in sign language recognition, most rely on a single explainability paradigm or are limited to specific learning setups.

By integrating Grad-CAM, LIME, and Integrated Gradients across RGB and segmentation-based models, we provide spatial, local, and structural insights beyond feature attribution and support deeper model auditing. Trust and transparency are critical in the context of assistive technology. An accurate yet opaque model is insufficient for user acceptance. The explainability framework presented here allows developers and users alike to understand model limitations, validate predictions, and adapt the system more effectively to real-world scenarios. A final implication is to integrate explainability into the model design pipeline rather than treating it as a post-hoc tool. Future models could incorporate attention mechanisms or self-supervised learning schemes explicitly guided by XAI feedback. For instance, models could be penalised during training if their attention diverges from annotated hand regions, encouraging alignment between learned representations and interpretable features. A limitation of our approach is the treatment of dynamic ASL signs ‘j’ and ‘z’. Since our models were trained on static images, we relied on dataset-provided approximations of these letters. While this enables classification of all 26 alphabet signs, the absence of true temporal modeling may affect recognition performance for such dynamic gestures. Future work will address this gap by incorporating video-based datasets and spatio-temporal architectures (e.g., 3D CNNs, temporal convolutional networks, or recurrent models).