3.2.1 Data collection and development.

Recently, there has been a surge in proposing novel research approaches utilizing low-resource languages, such as Hindi, Arabic, and Urdu, for various purposes, i.e., sentiment analysis, opinion missing, and other NLP-related tasks. Urdu is widely recognized as a low-resource language, as several computational resources are unavailable or inaccessible [46]. In line with that, we are interested in exploring sarcasm identification in the low-resourced Urdu language in the online user-generated data that have been given less concentration in the literature. However, we were confronted with a data availability problem for sarcasm detection. Therefore, we developed a research data set comprising user tweets from social media platforms publicly available for research purposes. We target the social media platform that people mostly use to express their opinions using their native languages. Also, there is a scarcity of Urdu sarcastic datasets. We curated and released a comparatively large dataset named Urdu Sarcastic Tweets (UST) Dataset, comprising user-generated comments from . Additionally, we argue that the proposed research data set dataset for sarcasm detection in Urdu is the first publicly available Urdu-sarcastic-tweets-dataset https://www.kaggle.com/datasets/shumailakhan/urdu-sarcastic-tweets-dataset of Urdu sarcastic tweets.

We have collected data by scrapping for ’tweets’ and ’replies’ written in Urdu. We utilized a Python-based online library, Tweepy, to crawl Urdu tweets from . As part of configuring the data retrieving process, we selected the Urdu language. To determine the search query, we iterated through a list of trending hashtags representing various aspects of sports and political tweets. The sources for data collection include BBC Urdu Urdu News, as well as images shared by the Pakistani and Indian public, which were collected from . The dataset contains tweet information, including tweet ID, text, gender, emoji/emoticons, date and time, and location. A Python script extracts emojis from tweet text, gathering their descriptions and sentiment scores [47]. The tweets are encoded in UTF-8 (Unicode Transformation Format version 8) and saved in a text file. The research dataset is balanced, comprising several instances under sarcastic and non-sarcastic categories.

3.2.2 Grounded theory approach.

In this step of the proposed methodology, a coding guideline document is developed by manually analyzing the test sample of collected tweets from X in the dataset. Grounded theory is a qualitative research methodology aiming to develop theories based on the frequently occurring concepts in data collected systematically from the participants. Also, to develop an automated approach capable of classifying social media text written in a low-resourced Urdu language into sarcastic and non-sarcastic instances, we need to build a grounded theory that can provide a systematic and comprehensive understanding of how individuals can differentiate between sarcastic and non-sarcastic tweets. The coding guidelines document outlines the process for conducting grounded theory research and the coding procedures for analyzing data related to Urdu sarcasm detection. The final output is then used to label the dataset in the content analysis phase, contributing to minimizing the disagreement between the different annotators when labelling the dataset for the ML experiments, as shown in Fig 1. The developed coding guideline includes definitions of each fundamental concept with examples and instructions for annotating user tweets within the dataset. Also, the coding process is conducted iteratively. The first two authors of the paper discuss the coding guidelines, refine them, and resolve disagreements through discussions and reconciliation sessions, if any, enhancing and stabilizing them.

3.2.3 Analyzing content manually.

After developing the coding guideline, the third step in the proposed approach is to annotate the user tweets in the dataset using a content analysis approach. For this purpose, the first three authors of the paper manually annotated the user tweets in the dataset to make the data pars able for the ML classifiers. The proposed approach annotation process is inspired by Riloff et al. [48] and the content analysis approach [49]. We aim to answer RQ1 by focusing on Urdu’s linguistic nuances and cultural references, highlighting how sarcasm manifests in specific linguistic contexts, deepening our understanding of language-specific cues, and enriching sentiment analysis. In the exploration to discern language-specific cues within Urdu that distinguish between sarcastic and non-sarcastic comments, RQ2 aligns to identify linguistic features inherent to Urdu. Investigating these nuanced aspects within Urdu allows for a deeper grasp of how sarcasm manifests within language, including cultural references, idiomatic expressions, or linguistic subtleties. However, identifying the most practical features or linguistics patterns for resource-less languages is also challenging due to language diversity, data sparsity, noise, limited annotated data, cultural nuances and context, the complexity of sarcasm, and the lack of computational tools. In addition to addressing RQ3, which involves assessing the effectiveness of ML models in sarcasm detection within Urdu, essential for identifying the most suitable models for this language, our study tackles the intricacies inherent in sarcasm detection within low-resource languages.

During the content analysis phase, several crucial steps were performed: First, we created a sample dataset from the collected dataset. Secondly, an annotation form was created using Microsoft Excel to annotate the human annotators’ user-generated comments. These annotators utilized the developing code guidelines and identified features for the annotation process. Thirdly, the selected human coders had the annotation form and coding guidelines to annotate each comment. We used inter-annotator agreement to measure the level of agreement between three human annotators who independently annotated the same data set. It is often used in NLP and ML tasks where data needs to be labelled with predefined categories or annotations. The inter-annotator agreement can be measured using various metrics such as Cohen’s kappa, Fleiss’ kappa, and Krippendorff’s alpha [50].

3.2.4 Automated classification.

The final phase of the proposed research approach aims to identify how well we automatically classify sarcasm in Urdu. For this purpose, we executed the subsequent steps. Initially, we conducted the pre-processing steps, including stop words removal, punctuation marks, unnecessary characters, hashtags, URLs, digits, all non-Urdu words, and special characters using iNLTK https://inltk.readthedocs.io/en/latest/and NLP Toolkit https://www.nltk.org/. After the pre-processing, we employed feature engineering techniques to convert textual data into numerical format to understand ML models. The pseudocode for the methodology process is depicted in Algorithm 1. To accomplish this task, we implemented TF-IDF and CountVectorizer techniques on the textual data. Finally, to evaluate the performance of ML techniques in automatically classifying sarcasm in text, we used standard evaluating matrices like receiver operating characteristic curve (ROC), confusion matrix, precision, recall, and a harmonic mean known as the F1-Score.

Algorithm 1: Data Pre-processing and Processing Algorithm

Step 1: Collect Data

data = CollectDataFromXPlatform()

Step 2: Identify Data Sources

sources = IdentifyDataSources()

Step 3: Ensure Data Availability

dataAvailabilityIssues = RecognizeDataAvailabilityIssues()

Step 4: Manually Annotate Dataset

codingGuidelines = DevelopCodingGuidelines()

annotatedDataset = AnnotateDatasetManually(data, codingGuidelines)

Step 5: Manual Analysis of Dataset Content

analysisResults = AnalyzeContentManually(annotatedDataset, codingGuidelines)

Step 6: Pre-process Text Data

nlpLibrary = "SpaCy"

preprocessingSteps = ["Tokenization", "Stopword Removal", "Lowercasing"]

for each tweet in annotatedDataset:

       # Apply specified pre-processing steps using NLP library

       preprocessedTweet = PreprocessText(tweet, nlpLibrary, preprocessingSteps)

       preprocessedDataset.append(preprocessedTweet)

Step 7: Extract Textual Features

featureExtractionTechniques = ["Countvectorization"]

for each tweet in preprocessedDataset:

       # Extract specified textual features using chosen techniques

       extractedFeatures = ExtractFeatures(tweet, featureExtractionTechniques)

       featureExtractedDataset.append(extractedFeatures)

Step 8: Handle Emojis

emojiLibrary = "Emoji"

transAPI = "Google Translate"

for each tweet in featureExtractedDataset:

       # Detect and translate emojis for better understanding

       translatedTweet = HandleEmojis(tweet, emojiLibrary, transAPI)

       datasetWithTranslatedEmojis.append(translatedTweet)

Step 9: Handle Slangs

urbanDictionary = "UrbanAPI"

for each tweet in datasetWithTranslatedEmojis:

       # Replace slangs and abbreviated words with their original forms

       originalSlangsTweet = HandleSlangs(tweet, urbanDictionary)

       datasetWithOriginalSlangs.append(originalSlangsTweet)

Step 10: Augment Data

augmentationTechniques = ["Synonym Replacement", "Back Translation"]

augmentedDataset = []

for each technique in augmentationTechniques:

       for each tweet in datasetWithOriginalSlangs:

              # Apply specified data augmentation technique

              augmentedTweet = ApplyAugmentation(tweet, technique)

              augmentedDataset.append(augmentedTweet)

Step 11: Save Pre-processed Data to File

SaveDatasetToFile(augmentedDataset, "UTF-8", "preprocessed_data.txt")

Sarcastic sentences.

The code of "sarcastic" is assigned to user tweets written in Urdu that portray an opposite meaning to what they are saying, having the intention to insult or mock someone. Sarcasm often conflicts between explicit expressions and the intended meaning, relying on subtle cues, such as tone or context, to convey the underlying irony. For example, "بہت ہی عمدہ کام کیا آپ نے!!!!!!!", and “واہ! کیا خوبصورتی ہے آپ کی!", both apparently praising the subject. However, the exaggerated usage of exclamation marks in the first sentence and the seemingly enthusiastic tone in the second sentence indicate sarcasm. These linguistic elements contrast the intended meaning and the literal interpretation. Sarcasm in Urdu exploits these linguistic inconsistencies to convey contradictory yet nuanced sentiments, adding depth and layers to the communicated message. Understanding these nuances is essential to incorporate sarcastic expressions within Urdu.

Non-sarcastic sentences.

The code of non-sarcastic is assigned to user tweets that demonstrate straightforward meaning, i.e., are easy to interpret and understand by reading them and show the same meaning the user wants to portray. In contrast to sarcastic expressions, non-sarcastic sentences do not require a deeper interpretation of linguistic cues. They present clear, unambiguous information, making them easily understandable. Understanding the distinction between non-sarcastic and sarcastic expressions in Urdu is essential for effective communication and accurate interpretation of intended meanings within conversations and written text. For example, “ہمیشہ اچھے کام کرو", "میں نے کتاب پڑھی". These sentences convey straightforward and literal meanings without any ironic or sarcastic intent. They express typical statements or factual information in Urdu. Detecting sarcasm in Urdu often relies on recognizing discrepancies between the explicit words used and the speaker’s underlying tone, context, or intent.

Annotation criteria.

The following criteria are identified when annotating the user tweets into sarcastic and non-sarcastic.

• Sarcasm means using language to convey the opposite of the literal meaning, often mocking, criticizing, or expressing irritation. For example,

" ہاں، حکومت نے تو ہمیشہ کی طرح چمکتے ہوئے منصوبے شروع کیے ہیں، جو ہمیشہ کی طرح مکمل ہوتے ہیں"۔

• Tone and Context: Detecting Urdu sarcasm requires comprehending contextual intricacies, lexical variations, and intricate linguistic nuances. i.e.

اس شخص کو دیکھو، ہر بار وعدے کرتا ہے کہ دنیا بھر کو بہتر بنائے گا۔""

"ہاں بلکہ ہمیں تو اب تک کچھ نظر نہیں آیا اس کی خدمات سے!"

• Linguistic Devices: The annotators are also guided to annotate tweets as sarcastic when a negative situation follows a positive sentiment or when contempt emojis follow positive tweets and vice versa, as these are considered indicators of sarcasm. Identify linguistic cues such as exaggerated punctuation (e.g., excessive exclamation/question marks), unexpected word usage, or contradictory expressions that signify sarcasm. Another rule is that Tweets with admiration or exclamation words (like Cheers, Yay) followed by unpleasant emoticons are also considered sarcastic. For example,
  1. ○ Exaggerated Punctuation:
     "واہ! آپکی حکومت نے کام کرنے کا طریقہ بلکل بے مثال ہے!!!"
  2. ○ Contradictory Expressions:
     "میرے پیارے نظریے دوست، یہ ٹریفک واقعی مزیدار ہوتا جا رہا ہے۔ #TrafficFun"
  3. ○ Admiration Words Followed by Unpleasant Emoticons:
     "واہ کیا خوبصورت مناظر ہے! "

These examples describe how linguistic devices like exaggerated punctuation, contradictory expressions, and the combination of admiration words with unpleasant emoticons can be used to convey sarcasm in Urdu tweets.

Annotation instruction.

All the individuals involved in the annotation process are experienced professors and postgraduate computer science students with diverse data annotation experience. The coders were instructed to disregard digits, symbols, and code-mixed Roman and Arabic texts and solely concentrate on pure Urdu. Furthermore, we specified the required storage format, specifically the UTF encoding. The dataset was labeled by assigning a ’sarcastic’ label for sarcastic tweets and a ’non-sarcastic’ label for non-sarcastic tweets, respectively. The annotated outputs from the collected tweets are used as the "ground truth" to evaluate how effectively the various machine learning algorithms perform in automatically identifying sarcastic tweets in low-resourced language [48].

Subjectivity and interpretation.

Annotators’ diverse perspectives lead to varying interpretations of text, resulting in differing annotations for sentiment, tone, or sarcasm. To mitigate this, we have developed a comprehensive coding guideline for annotators. This guideline includes definitions and examples, aiding annotators in navigating the annotation process consistently.

Cultural nuances.

The presence of intricate cultural subtleties and regional variations within Urdu makes accurate text labeling challenging for annotators unfamiliar with these nuances. We set some basic examples for better understanding.

Sarcasm and irony.

The identification of sarcasm or irony in Urdu proves intricate due to indirect expressions, diverse cultural backgrounds, and varying degrees of subtlety in conveying sarcasm. The coding guideline offers specific examples and guidelines on identifying indirect expressions, ensuring a more nuanced approach to sarcasm detection.

Linguistic complexity.

The Urdu script’s intricate calligraphy, diacritics, and script variations pose challenges for annotators and annotation tools alike. Our coding guideline includes visual aids and explanations to help annotators navigate through linguistic complexities, ensuring consistency in annotation.

Annotation consistency.

Ensuring uniform annotations across multiple annotators becomes daunting due to divergent interpretations and understandings of Urdu language subtleties. The coding guideline acts as a reference point, fostering consistency by providing a shared understanding of annotation principles.

Limited labeled data.

The scarcity of pre-labeled datasets in Urdu restricts the creation of robust training sets essential for machine learning models. To overcome this challenge, we actively collaborate with language experts and continuously update our coding guidelines to include diverse examples from contemporary sources.

Resource intensiveness.

Manual annotation necessitates a substantial allocation of human resources, time, and expertise. The coding guideline streamlines the annotation process, maximizing efficiency and ensuring that resources are utilized effectively.

5.1.1 Text pre-processing.

Cleaning input data (textual data) is considered significant for the ML experiments. For this purpose, we performed a series of steps as discussed below.

1. Data cleaning. For cleaning Urdu, various functions can be applied to enhance the quality and consistency of the data. i.e. to remove special characters and eliminate HTML symbols, URLs, and digits, the re (regular expression) module, NLTK, and spaCy are used, which is part of the Python standard library.
   In Urdu, the conversion of lowercase does not result in any visible changes, as Urdu does not have distinct uppercase and lowercase forms for most of its characters. However, this operation is still performed for consistency, normalization, and alignment with common practices in text processing, where case consistency is often desirable. The "lower()" function is a built-in method in Python that converts all characters in a string to lowercase. This operation is language-agnostic and works for Urdu as well.
2. Stop words. Stop words are frequently occurring words in a language with no semantic value. We used the list of stop words created by Humayoun et al. [57]. Examples of frequently used stop words are listed in Table 4.
3. Tokenization. Tokenization is breaking down text data into individual words or tokens, which can be utilized as features in ML algorithms. This technique aids in reducing the dimensionality of the text data. Tokenization helps to improve the classification models’ accuracy by capturing the essential linguistic features of the text data. To prepare the dataset for ML processing, the iNLTK python library performed various NLP tasks such as vector embedding, tokenization, and sentence similarity. The library provides the functionalities a developer would require to build NLP applications. Since iNLTK supports Urdu, it was a suitable choice for processing the Urdu sarcastic tweets dataset.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. List of Urdu stop words.

https://doi.org/10.1371/journal.pone.0307186.t004

5.1.2 Feature extraction.

In the proposed methodology, we extract the most frequent textual features that perform well on short text-like tweets [58]. The most common textual features including Bag of Words (BoW), Term frequency- Inverse Document Frequency (TF-IDF), and N-grams with ML classifiers [51].

The BoW approach involves creating a dictionary including all words within the corpus. Each document is represented by a vector, where each element corresponds to the frequency of a specific word in the document. Similarly, the TF-IDF technique operates on the premise that words occurring frequently in the text. It considering both the frequency of a word in a document (term frequency) and its inverse document frequency (which reduces the weight of common words). The TF-IDF value increases with the number of times a word appears in the document but decreases with the word’s frequency in the entire corpus.

N-grams are continuous sequence of n-words from a given text. Essentially, they help capture the context by considering word combinations like uni-grams (n = 1): "Cherries" "are", "red". Where bi-grams (n = 2): "Cherries are", "are red", and tri-grams (n = 3): "Cherries are red". By incorporating N-grams as features enhance model accuracy by providing more context compared to single words in detecting sarcasm in Urdu [14].

For the number of n continuous values, n-gram can be calculated as (2)

Where x represents occurrences of words in a sentence

N = the number of continuous words.

1. Count vectorization. A technique used in natural language processing (NLP) for converting a collection of text documents into numerical vectors. Each document is represented as a sparse vector of word counts. Each vector element corresponds to a particular word in the corpus, and the value represents the frequency of that word in the document. For example: "Cherries are red,” can be represented as [1,0,1]
2. Emoji handling. In the context of sarcasm detection, emojis play a significant role by providing nuanced information that is often essential for understanding the true sentiment. A Python Online emoji library https://github.com/github/gemoji, Google Trans API, and emoji 0.6.0 were used to detect and translate emojis into text. By translating the emojis into text, the model can better understand the sentiment of the text and make more accurate predictions. For example: Λ might be translated to "sad.” This process not only enhances the fluency and clarity of the text but also ensures that ML algorithms can effectively process and analyze the nuanced sentiments conveyed through emojis.
3. Slangs/short abbreviated words. Slangs and abbreviated words are informal and non-standard words or phrases commonly used in social media and text messages and can be challenging for NLP models. We utilize Urban Dictionary https://www.urbandictionary.com/, replacing short/abbreviated words with their root/original form, enhancing the proposed system’s efficiency and accuracy. For example, the slang "IMO" might be replaced with "in my opinion".
4. Data augmentation. To increase the diversity of the dataset to get accurate performance, various data augmentation techniques were applied using NLTK and the transformer https://huggingface.co/ library:
   • Back translation, wherein the original Urdu was translated into English and then back into Urdu, ensuring the augmented data’s uniqueness, introducing variations while maintaining the original meaning.
   • Synonym replacement was utilized to randomly replace words with their synonyms.
   • Random deletion selectively eliminated words based on a predefined threshold,
   • Random swaps involved shuffling the order of words.

5.1.3 Assessment & training.

We used a stratified fivefold cross-validation approach to the textual data to train and validate the supervised ML algorithms. Four folds of the cross-validation approach were employed to train the ML algorithms, and one fold of cross-validation was used to validate the algorithm. The benefit of the cross-validation approach for training and validating an ML classifier is to check how well a model works if limited data is available. Currently, the stratified K-fold cross-validation approach is most commonly and frequently used to train and validate ML classifiers. Each fold has roughly the same proportion of labels representing each class. To assess the effectiveness of the classifiers, we compute and summarize the average results obtained from the tenfold cross-validation runs.

The classifier accuracy can be measured by evaluating different performance parameters. The most common parameters for text classifications are accuracy, F-measure, precisions, recall, and AUC.

Accuracy (ACC). ACC shows the percentage ratio of the predicted instance. It measures overall correctly classified instances. Mathematically, accuracy can be described as:

Where;

TP: True Positive, and is predicted positive.

TN: True Negative and is predicted negative.

FP: False Positive predicted positive but false actually.

FN: False Negative predictions are false but true.

Precision (Pre). The computation ratio of true positive over the positive result.

Recall (Rec). The recall is the percentage of actual positives that are expected to be positive. It represents the ratio of true positives to all true.

F-measure (F1). The F1-measure represents the harmonic mean of precision and recalls when false positives and false negatives are equal. The standard F-measure values precision and recall equally.

5.1.4 Experimental results.

This section unveils groundbreaking findings that underscore the efficacy of the proposed machine-learning approach for sarcasm detection in Urdu. Through rigorous evaluation across diverse classifiers and feature engineering techniques, the study establishes Linear Regression and Random Forest as superior performers, achieving remarkable accuracy of 0.89 and 0.88, respectively. This breakthrough not only validates the robustness of the methodology but also paves the way for future advancements in sentiment analysis and natural language processing for low-resource languages like Urdu. To comprehensively explore sarcasm detection in Urdu tweets, we employed two different experiments, i.e., a featured-based N-gram approach and a pipeline architecture using textual features. Each setup serves a unique purpose and contributes complementary insights into the intricacies of sarcasm detection.

The need for a comprehensive exploration of sarcasm detection methodologies determines the choice of two different experiments. The feature-based approach allows a deep dive into the impact of specific linguistic structures (N-grams) on classifier performance. On the other hand, the pipeline architecture provides a broader assessment of the overall effectiveness of diverse machine-learning algorithms. Together, these setups offer a well-rounded understanding of sarcasm detection in Urdu tweets—from the intricacies of language structures to the holistic evaluation of classifiers in a unified framework.

1. Featured-based approach with N-grams. The proposed featured-based approach incorporates n-grams, a contiguous sequence of ’n’ items from a given text sample, for effective feature extraction in Urdu, as shown in Table 5. Specifically, we applied SVM, Multinomial NB, Bernoulli NB, XGBoost, DT, RF, and K-NN, tailoring each to different n-gram types. Surprisingly, the performance of the proposed approach when using uni-gram (n = 1) features on the proposed dataset, we found that most of these classifiers have effective results in performance metrics. Interestingly, SVM-Countvectorization-unigram and Bernoulli NB-Countvectorization-unigram perform best, achieving high accuracies of 0.81 and 0.80, respectively. These results show that, for the uni-gram feature extraction method, SVM and Bernoulli NB are the most effective classifiers for sarcasm detection in Urdu.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Results of featured-based classifiers.

https://doi.org/10.1371/journal.pone.0307186.t005

SVM-Countervectorizer-bigram performed well with an accuracy of 0.81. Preferably, the K-NN-Countervectorize-bigram achieved the highest precision at 0.96, though its overall accuracy remains lower at 0.51. MNB, BNB, and XGBoost remain consistent at an accuracy of 0.78 or 0.79. In comparison, the DT and RF got moderate accuracy. Furthermore, for tri-gram, the SVM classifier has the highest accuracy of 0.79 among all the classifiers. While K-NN stands out with the highest precision of 0.95, its accuracy is 0.50, indicating many false positives. RF has a good recall of 1.00, representing that the positive samples were identified correctly. However, its precision and F1-score are relatively lower. Meanwhile, multinomial NB, Bernoulli NB, and XGBoost remain slightly consistent with an accuracy of 0.78 or 0.77.

Uni-gram accuracy is better than bi-gram and tri-gram because a single word is most prominent in the text. Fig 2 depicts the performance of the suggested model’s n-gram feature. This graph depicted only the accuracy of using uni-grams, bi-grams, and tri-grams with various ML classifiers. The proposed n-gram feature evaluation results suggest that the value at n = 1 is superior to that at n = 2 and n = 3. We can determine the true meaning of the sentence if it is completed (which means both the subject and predicate are present in the sentence); otherwise, the sentence is regarded as ambiguous. So, the entire sentence is considered to analyze the optimum results for sarcasm detection. It is challenging to examine the effects of a single word when considering the n-gram because the word’s impact on the sentence depends on whether it is positive or negative.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Comparison of n-gram models.

https://doi.org/10.1371/journal.pone.0307186.g002

(B) Pipeline architecture using textual features. In this phase, we adopted a pipeline approach to encapsulate the whole process of sarcasm classification. Multiple ML classifiers, including SVM, RF, LR, Bernoulli NB, Multinomial NB, and DT, are integrated into a unified workflow. The pipeline approach enables a comprehensive comparison of different classifiers, considering their overall performance in the sarcasm detection task. The results obtained from various machine learning algorithms in the sarcasm classification experiments are presented in Table 6, with bold values indicating the highest scores for each evaluation metric. LR and RF algorithms demonstrated relatively high accuracy, achieving 0.89 and 0.88, respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Results analysis of ML textual classifiers.

https://doi.org/10.1371/journal.pone.0307186.t006

The ROC curves for LR and RF are shown in Fig 3. The ROC curve is a plot of the true positive rate (TPR) versus the false positive rate (FPR) of a classification model at different classification thresholds. The TPR is the proportion of positive cases that the model correctly classifies, and the FPR is the proportion of negative cases that are incorrectly classified as positive by the model. An ideal ROC curve would be a diagonal line from the bottom left corner of the graph to the top right corner. This would indicate that the model can distinguish between positive and negative cases perfectly.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. ROC curve showing the performance of ML classifiers.

https://doi.org/10.1371/journal.pone.0307186.g003

Both Multinomial NB and Bernoulli NB also performed well, reaching an accuracy of 85%. Although SVM showed relatively high performance when utilizing feature n-grams, its results with textual features were unsatisfactory, exhibiting lower accuracy compared to other algorithms. The figures suggest that SVM requires more generalization. The training and validation curves for RF and LR are shown in Fig 4. The learning curve shows that the training accuracy and validation accuracy both increase as the number of training samples increases. This is because the model can learn more about the features that are important for classifying sarcasm as the number of training samples increases. The training accuracy is always higher than the validation accuracy.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Learning curves to train the machine learning classifier.

https://doi.org/10.1371/journal.pone.0307186.g004

For Urdu classification, LR and RF stand out as the best-performing algorithms. When classifying user-generated comments into sarcastic and non-sarcastic categories, LR consistently outperforms other machine learning classifiers. However, the choice of algorithm depends on the application’s specific needs, such as whether it is more important to minimize false positives or false negatives. It is also worth noting that the performance of these algorithms may be enhanced further by fine-tuning the hyperparameters with approaches like grid search or random search. In addition, with the performance metrics, learning curves were generated to provide additional insights into the training and validation performance of the proposed model.

The confusion matrix of the RF and LR classifier is shown in Fig 5, which is a way to visualize the performance of a classification model. In this case, the model is trying to classify comments as either sarcastic or not sarcastic. The rows of the matrix represent the true labels of the comments, and the columns represent the predicted labels. In the context of sarcasm detection, the most important metrics are likely to be precision and recall. Precision is the proportion of comments that were labelled as sarcastic that was sarcastic. Recall is the proportion of sarcastic comments that were correctly classified. In this case, the precision is 79%, and the recall is 82%. are relatively high, which suggests that the model is doing a good job of detecting sarcasm.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Confusion matrix generated for the ML classifier 4a for logistic regression and 4b for random forest.

https://doi.org/10.1371/journal.pone.0307186.g005

5.1.5 Comparative analysis with state-of-art approach.

The proposed approach is compared with the previously published work Tanz-Indicator [16]. The Tanz-Indicator framework utilizes a rule-based to assess linguistic characteristics in Urdu sarcastic expressions. They employ Naïve Bayes and Support Vector Machine (SVM) classifiers to predict sarcasm in the Perso-Arabic script. The proposed framework adopts a feature-based Machine Learning (ML) approach, and secondly, we integrate ML techniques using textual features on the UST dataset. The paper presents a comprehensive analysis of performance metrics (Precision, Recall, F1-score) for different models applied to distinct datasets utilizing various sarcasm classification techniques, detailed in Table 7 and Fig 6.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Performance comparison of classifiers on different datasets.

https://doi.org/10.1371/journal.pone.0307186.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Tanz-Indicator model compared with the proposed model comparison.

https://doi.org/10.1371/journal.pone.0307186.t007

Fig 6 illustrates the learning curves for the RF, Multi NB, Bernoulli NB, LG, and DT classifiers. These curves demonstrate the evolution of performance metrics across training iterations, shedding light on how well the model generalizes and adapts to different classifiers. The performance of the models varies depending on the dataset used. The Tanz-Indicator dataset comprises 3000 tweets, divided into 70:30 for training and testing. In contrast, the UST dataset encompasses 20000 tweets, split into 80:20 for training and testing purposes. For the evaluation process, we employed a robust k-fold cross-validation approach to ensure the reliability of our results.

In the context of the UST Dataset, the findings indicate that Multi Naïve Bayes outperforms SVM in precision, recall, and F1-score. Conversely, the Tanz-Indicator dataset demonstrates competitive performance between SVM and Naïve Bayes, displaying slight differences in precision and recall.

This study underscores the significance of considering the dataset when selecting the appropriate algorithm. The Multi Naïve Bayes model consistently exhibits superior performance across both datasets compared to SVM. This can be attributed to Naïve Bayes’ adaptability, demonstrating strong performance in smaller datasets due to its ability to handle limited instances and scaling effectively to larger datasets owing to computational efficiency and enhanced learning from increased data volumes.

The comparative analysis with the Tanz-Indicator framework is pivotal in contextualizing the advancements made by our proposed approach. Tanz-Indicator, as a rule-based system, represents a foundational effort in Urdu sarcasm detection. However, the proposed ML methodology, particularly the SVM and Multi Naïve Bayes models, markedly outperforms Tanz-Indicator across all key metrics—precision, recall, and F1-score. This performance disparity underscores a fundamental shift in approach efficacy. While rule-based systems like Tanz-Indicator rely on predefined linguistic patterns, they may struggle with the fluidity and contextual nuances inherent in sarcastic Urdu expressions. In contrast, our ML models, trained on the extensive UST dataset, can capture these nuances more effectively. The higher F1-scores, 0.84 for SVM and 0.82 for Multi Naïve Bayes compared to Tanz-Indicator’s 0.323 and 0.324 reflect a balanced improvement in both precision and recall. This balance is crucial in sarcasm detection, where misclassifications can significantly impact sentiment analysis and user experience. The comparative analysis also highlights the scalability and adaptability of the proposed approach. By leveraging a larger, more diverse dataset UST with Tanz-Indicator’s dataset, our models demonstrate robustness in handling the variability of user-generated content. This scalability is vital for real-world applications where the volume and diversity of data are ever-increasing. The juxtaposition with Tanz-Indicator not only quantifies the performance gains of our ML-based approach but also qualitatively illustrates its superiority in capturing the linguistic intricacies of Urdu sarcasm. This comparison underscores the potential of data-driven, machine-learning methodologies in advancing NLP tasks for low-resource languages like Urdu, paving the way for more nuanced, context-aware sentiment analysis tools.

5.1.6 Statistical significance.

We conducted a paired t-test with a significance level (alpha) of 0.05 to assess the significance of the UST dataset. In this analysis, the one-tailed and two-tailed p-values are notably small (less than 0.05), indicating compelling evidence supporting the significance of the proposed ML model concerning the UST dataset. The results of the significant test are shown in Table 8.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. Significance of our UST dataset in terms of T-Pair test.

https://doi.org/10.1371/journal.pone.0307186.t008