Urdu language

Altaf et al. [15] perform SA at the aspect level following the standards of SemEval-2015 in the Urdu language. Four types of information are identified during the annotation process including aspect, aspect category, aspect polarity, and aspect category polarity.TF-IDF and n-grams are used for feature extraction and K Nearest Neighbor (KNN), Random Forest (RF), and Naïve Bayes (NB) are used for feature classification.

Sadia et al. [16] elaborated a lexicon and rule-based approach for SA in the Urdu language focusing on three main approaches. The first method is supervised, the second is lexicon rule-based, and the third is a hybrid method. Lexicon-based methods do not yield high accuracy as they heavily depend on the size and quality of training data and because many lexical objects have a positive script in one domain while a negative script in another domain. Sentiment-annotated lexicons are produced that add subjectivity-related details to a word’s syntactic and morphological characteristics. Using the shallow parsing technique, Syed et al. [17] perform ABSA for the Urdu language. They identify and extract the SentiUnits which contain the aspects having some sentiment information. For this, a lexicon-based approach is employed. The dataset comprises instances of product and movie reviews. They achieved an accuracy score of 0.72 and 0.78 respectively.

Mukhtar et al. [18] use a rule-based approach and unlike other approaches that mostly focus on nouns, adjectives, and negations, they also focus on verbs and other context associating words. The dataset was collected from online Urdu blogs related to the 14 different genres. Due to the addition of more rules and the expansion of the sentiment lexicon with more words, their approach achieves higher precision and accuracy scores. To perform ABSA in the Urdu language, Rana et al. [19] adopt a rule-based methodology and manually create various syntactic rules for identifying opinion words and their targets using this technique. To identify opinion words in a sentence and related targeted words/phrases, these rules make use of opinion lexicons. Polarities are assigned to each aspect after extracting the associated aspects. The identifier performs more effectively with the rules than without them. Ahmad et al. [20] propose an approach for ABSA in the Urdu language following SemEval-2014 guidelines. Hand annotations are performed for the aspect terms, aspect categories, and aspect polarity.

Hassan et al. [21] use Bag of Words (BOW) model and SEGMODEL to perform SA in the Urdu language on a dataset comprising reviews of 401 electronic devices and reviews of 443 automotive parts. The BOW model is used in their first experiment, and SEGMODEL is used in the second independent experiment. The identification of sub-opinions is the primary goal of this study. Most studies focus on one opinion while this study handles multiple opinions, also known as sub-opinions. Ahmed et al. [22] propose a new Meta-Learning Ensemble (MLE) method for the Urdu language SA by incorporating both ML and DL algorithms. DL models need a large dataset to perform well while meta-learning needs only a few tasks to give the expected output. The proposed ensemble technique combines the predictions of both the inter- and intra-committee classifiers on two separate levels, resulting in reduced training complexity and overfitting. The dataset comprises 28,921 Urdu language reviews with positive, negative, or neutral labels. N-gram along with other word embedding including Continuous Bag of Words (CBOW) and skip-gram are used. The proposed MLE method outperforms the baseline achieving an accuracy of 86.42%.

Altaf et al. [23] perform SA at aspect level in Urdu language. Sports tweets are collected from X (formerly known as twitter) and fine-grained SA is performed by extracting ATEs ATPs, ACTs and, ACPs using ML and deep learning approaches. Additionally, implicit ATEs are also identified from the data. For the task of ATPs, ACTs, and ACPs classical ML outperformed DL models while for ATEs DL approach performed best as compared to traditional ML classifiers.

Shabbir et al. [24] purpose a framework for Urdu SA using DL techniques which comprises of data collection, preprocessing, and sentiment classification modules. A publicly available movie dataset i.e., IMBD is translated in Urdu and three deep learning models i.e., 1-Dimensional Convolutional Neural Network (1D-CNN), Multilingual-MiniLM-L12-H384 transformer, and Long Short-Term Memory (LSTM) are used for SA. The experimental results illustrate that Multilingual-MiniLM-L12-H384 transformer based model is most suitable architecture for SA that acheived the highest accuracy of 89%.

Ashraf et al. [25] use deep learning approach for Urdu Text Sentiment Analysis (USA-BERT) using Bidirectional Encoder Representations. An Urdu dataset for SA-23 (UDSA-23) is introduced for the classification of sentiments. Their proposed approach comprises four modules i.e., preprocessing of data using BERT tokenizer, creation of BERT embeddings for each review, and fine tuning of deep learning BERT classifier. In the last module, Pareto principle is employed on two state-of-the-art datasets (UDSA-23) and (UCSA-21) to evaluate the performance of USA-BERT. The experimental results show that USA-BERT outperformed existing techniques by significantly improving the accuracy by 26 and 25% respectively.

Altaf et al. [26] perform SA on Urdu language sentence level using news dataset. A dataset is developed that comprises idioms, proverbs, and news domain. Different types of linguistic features are extracted from the data which includes part-of-speech tag-based features, numeric features, and boolean features. Then, machine learning approach is used for the classification of sentiments and several machine learning classifiers are used. J48 performed best with the accuracy of 90% and F-measure of 88%.

Roman Urdu

Zahid et al. [27] introduce the task of Aspect Based Opinion Mining (ABOM) for Roman Urdu which involves extracting an aspect of the sentence after classification and giving that aspect a polarity. Different Supervised ML approaches are used, including SVM, Decision Tree (DT), LR, RF, and KNN. The reviews are classified into the relevant category using the Support Vector Classifier (SVC). Rana et al. [28] propose an unsupervised approach to SA for Roman Urdu. First, the proposed model normalizes the text to account for different word spelling variations. They use Roman Urdu and English opinion lexicons after normalizing the text to identify users’ opinions from the text correctly. Negation terms and stemming are used to assign polarities to each extracted opinion. Additionally, the proposed model assigns a score to each sentence based on the polarities of extracted opinions and categorizes each sentence as positive, negative, or neutral. They validate their approach using two publicly available datasets for Roman Urdu and compare it with an existing model.

Mahmood et al. [29] perform SA for Roman Urdu using rule-based, N-grams, and Recurrent Convolutional Neural Network (RCNN). They perform binary and tertiary classification and yield an accuracy score of 65% and 57% for binary and tertiary classification problems, respectively. Ali et al. [30] perform ABSA in Roman Urdu/Hindi and English language on Uber reviews collected from Facebook. The reviews are converted into English using APIs and multiple aspects are extracted using Parts of Speech (POS) features like nouns, adverbs, and adjectives to get prominent aspects, and polarity is assigned to every aspect. Lexicon-based, ML and DL approaches are used to get better results. Khan et al. [31] use the CNN-LSTM-based approach for the Roman Urdu. They also use the SVM classifier and Word2Vec CBOW word embedding models that prove to be more beneficial for Roman Urdu achieving 90% accuracy. They also use TF-IDF and GloVe word embeddings and RF classifier that achieves an accuracy of 77% for the RUSA (Roman Urdu Sentimment Analysis) dataset. Nagra et al. [32] perform deep SA for Roman Urdu using the Faster Recurrent Convolutional Neural Network (FRCNN) model on the RUSA-19 dataset. They also use RNN, rule-based, and N-grams models for better results Although RNN takes less time than RCNN, RCNN performs better than the baseline. The proposed RNN and RCNN models achieve 91% and 92% accuracy respectively, outperforming the baseline. Chandio et al. [33] propose a deep recurrent architecture for Roman Urdu using attention-based RU Bidirectional LSTM (RU-BiLSTM). For the complete coverage, they focus on both direction and context-based analysis. Word2Vec, GloVe, and FastText word embedding techniques are used. RUSA-19 and RUECD (Roman Urdu e-commerce) datasets are used which outperform the baseline achieving promising results. Using Word2Vec embedding with BiLSTM achieves an accuracy of 87%.

Li et al. [34] propose a CNN-based approach with transfer learning and attention mechanism to improve the performance of SA for Roman Urdu and Hindi languages. Various word embedding techniques are used including Word2Vec, TF-IDF, GloVe, and FasText. To reduce the number of dimensions in the dataset, an average pooling mechanism is implemented along the hidden layers, achieving an accuracy of 80% with a supervised algorithm for 15,000 user reviews and 92% accuracy with the DL algorithms. Qureshi et al. [35] perform SA of Roman Urdu language using reviews of Pakistan Super League’s (PSL) official song collected from X (formerly Twitter) and Facebook. Five main algorithms including four ML algorithms: NB, RF, LR, KNN, and artificial neural network are used. Each review is either tagged positive or negative. Using NB and LR an accuracy of 97% is achieved. Kabra et al. [36] use CNN-based model and TF-IDF vectorization for word embedding. They stacked different ML models with CNN including SVM, Word2Vec, and LSTM, and CNN+LSTM achieved the highest accuracy of 86.03%. Manzoor et al. [37] use self-attention BiLSTM for the sentence level SA in Roman Urdu that extracts fine-grained information from the sentences. The text representation is done using LSTM. They perform binary classification on a dataset comprising 10,000 instances and achieve 68.4% accuracy using BiLSTM.

Dataset collection

Due to the unavailability of publicly available datasets for ABSA in the Urdu language, for this study, we develop a new dataset comprising tweets related to COVID-19. Initially, a total of 7,230 tweets were extracted from X (formerly Twitter), using the Twint library, using generic keywords related to COVID-19, some of which are shown in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Keywords for Tweets.

https://doi.org/10.1371/journal.pone.0317562.t001

Comprehensive text preprocessing was essential to effectively prepare the dataset for analysis. Our preprocessing pipeline involved several steps. First, all mentions (e.g., @username) and hashtags (e.g., #topic) were removed to eliminate non-relevant text and focus on the main content. Next, we reduced multiple spaces to a single space and removed all non-alphanumeric characters, except necessary punctuation, to standardize the text. Emoticons were extracted and replaced with corresponding sentiment labels (e.g., happy, sad), while URLs were completely removed to prevent any external content bias. Additionally, duplicated tweets were also checked and removed from the dataset. We utilized tools such as UrduHack (https://urduhack.akkefa.com/en/stable/reference/preprocessing.html) and a custom Urdu Text Preprocessing script (https://github.com/MD-Ryhan/Urdu-Text-Preprocessing) to handle language-specific challenges such as normalization of Urdu text and cleaning of various textual anomalies. The final dataset used in this study comprises 6,000 tweets. Since, for the Urdu language, a syntactic rule-based (POS) approach is used to perform ABSA, the POS tags were assigned to the tweets using publicly available CLE (https://www.cle.org.pk/index.htm) tagger that comprises a total of twelve main categories with subdivisions, resulting in 32 tags. This tagset is evaluated on 100k words of CLE Urdu digest corpus resulting in 96.8% accuracy [41].

Since there is no dataset and POS tagger available publicly for ABSA in Roman Urdu, we generate a dataset by translating 6,000 Urdu language tweets, collected from the (https://www.ijunoon.com/) website, through a crawler developed using the selenium library that automatically translates the tweets from Urdu to Roman Urdu. During the translation phase, several issues were encountered due to the linguistic complexities leading to some words not being translated into Roman Urdu. Therefore, the translated dataset is manually verified to correct the spellings of these words to ensure the quality of the dataset. These issues are as follows:

Words with “ء”: Words containing ء were not recognized and translated, e.g., “جانیۓ” (know) which contains “ء” once and “مائیکرولاک ڈاؤن” (micro lockdown) which contains “ء” two times.

Misspelled Words: Correct spellings are often ignored when using social media and many spelling mistakes were identified in the Urdu dataset that the translator could not identify and translate, e.g., the word “کلینیکل” (clinical) was written as “کلینکل” (clincl), “انکار, (reject) was written as “انقار” and “قصائی” (butcher) was written as “کسائی”.

Missing Space: In some sentences, spaces were missing between two words, hence these words were not identified and translated by the translator, e.g., in the sentence “ہزارافرادمتاثرہوۓ” (thousand people got effected), a space is missing between the words “افراد” (people) and “متاثر” (effected). Similarly, “میں مسلسل” (I consistently) was written as “میمسلسل” due to missing space.

English Loan Words: Some words do not have an English translation and are used as such with Urdu characters, e.g., “ایسٹرازینیکا” (AstraZeneca), “ٹروڈو” (Trudo), “لوزکر” (luzker), and “گائیڈلائینز” (guidelines) etc. The translator did not translate these words.

Words with Diacritics: Words containing diacritics were not properly identified by the translator, e.g., “قلؔت” (scarcity), “اللہ” (Allah), and “إنَّا ِلِلَّٰهِ وَإِنَّا إِلَيْهِ رَاجِعُونَ” (we belong to Allah and to Him we shall return).

Abbreviations: Words containing abbreviations were not identified by the translator, e.g., “سی یوآئی” (ICU) and “این سی اوسی” (NCOC).

Words Separated by Underscore: Words separated by an underscore instead of a space were also not identified by the translator, e.g., “مسجدحرام” (Holly Mosque).

Compound Words: Some compound words were not identified by the translator, e.g., “جوش و امنگ” (enthusiasm).

Words with Independent Existence: In the Urdu language, two different independent words often combine to form a single word, e.g., “دوچار” (afflicted) is a word as well as “دو” (two) and “چار” (four). These types of words were not recognized by the translator.

Consecutive Character Usage: Words containing characters appearing consecutively were also difficult for the translator to identify, e.g., in the word ”کووڈ” (COVID) and “نوول” (novel), the character “و” is appearing twice consecutively.

Other Issues: Some words were translated in Roman Urdu but with incorrect spellings, e.g., ”کرونا” (corona) was translated as “Krwana”, “ایس او پیز” (SOPs) was translated as “S O Pays”, and “مثبت” (positive) was translated as “misbet”.

Urdu sentiment lexicon

We classify the tweets into positive, negative, or neutral sentiments using Urdu sentiment lexicon [42] that is developed by translating different lexical and linguistic resources such as SentiWordNet, English opinion word list, modifiers, English-to-Urdu-bilingual dictionary, and novel scoring mechanism. The lexicon development process, proposed by Asghar et al. [42] comprises four modules, (1) collection of English opinion words, (2) translation of opinion words in Urdu, (3) assigning scores to sentiments based on SentiWordNet and manual scoring method, and (4) collection and storage of modifiers. The resulting sentiment lexicon contains 3,651 positive sentiments, 5,730 negative sentiment, and no neutral sentiments words.

Rule-based approach for Urdu language

In this section, we outline our methodology for extracting aspect terms, aspect term polarity, and aspect categories using syntactic rules developed specifically for Urdu. This approach involves a series of eight predefined rules that leverage the morphological and syntactic properties of Urdu, a language rich in inflection and complex sentence structures. These rules systematically identify sentiment expressions and their corresponding aspects based on their linguistic context within the sentence. To enhance clarity, we provide a detailed description of each rule, including examples that demonstrate their application. The details of these rules are as follows:

Rule 1: The polarity of opinion words present in a sentence is identified using the sentiment lexicon and then the closest noun associated with the opinions is extracted as an aspect term. In the example below, the negative opinion term “خطرناک” (Dangerous) is associated with the aspect term “کورونا” (Corona) and its polarity is negative.

VBF—ہے NN—خطرناک PSP—لیے PSP—کے Q—سب NN—کورونا

Corona is dangerous for everyone.

Rule 2: The opinion term is identified in a sentence and its associated polarity is extracted from the lexicon [19]. The associated aspect term is extracted by finding the aspect comprising of left and right nouns separated by the words “کی”, “کا”, “کے”, and “کو”. In the example below, “خطرہ” (Danger) is an opinion term with negative polarity and the multiword aspect term associated with it is “کورونا کی بیماری” (Corona Disease).

VBF—ہے NN—خطرناک NN—بیماری PSP—کی NN—کورونا

Corona Disease is a danger.

Rule 3: Firstly, sentiment words are identified and then the continuous chain of noun phrases are searched which are separated by the words “کی”, “کا”, “کے”, and “کو”. In the example below, the opinion term “تیزی” (accelerate) has a negative sentiment, and the multiword aspect term associated with it is “کورونا وباء کے کیسز” (cases of the corona disease) and the polarity will be negative.

.AUXA—گئی VBF—ہو NN—شروع VBI—آنا NN—تیزی PRP—میں NN—کیسز PSP—کے NN—وباء NN—کورونا

Cases of the Corona disease is beginning to accelerate.

Rule 4: After identifying the opinion term, if no phrases are found separated by the words “کی”, “کا”, “کے”, and “کو”. then search to find the continuous noun phrase chain and extract them as an aspect term. In the example below, two noun phrases i.e., i.e., “محکمہ تعلیم” (Education Department) and “کورونا وائرس” (Corona Virus) are associated with negative opinion term “نقصان” (effect/loss) hence the closest aspects will be considered as the aspect terms. Therefore, the multiword aspect terms will be “محکمہ تعلیم” (Education Department) and “کورونا وائرس” (Corona Virus) and the polarity will be negative.

JJ—صرف NN—وائرس NN—کرونا VBF—بتائیں NN—بات CD—ایک PRP—مجھے

AUXT—ہے VBF—دیتی NN—نقصان PSP—کو NN—تعلیم NN—محکمہ

Tell me one thing why corona virus effects only the education department?

Rule 5: Due to the limitation of the POS tagger, some important aspects of COVID-19 do not get the right tags. For instance, the tagger assigned tags to “ایس او پیز” (SOP’s) as NN—ایس INJ—او NN—پیز. The tagger considered the term “او” as an injective (INJ) although this should be a noun, and it should be a noun phrase consisting of three tokens. To address this issue, in this rule, we manually extract this term as an aspect. Here, “مثبت” (Positive) is the opinion word, “ایس او پیز” (SOP’s) is the aspect term having a positive polarity.

.AUXT—ہیں VBF—ملتے NN—نتائج JJ—مثبت PSP—سے VBI—کرنے NN—عمل PSP—پر NN—پیز INJ—او NN—ایس

Implementing SOPs yields positive results.

Rule 6: If an opinion term is directly associated with an adjective, then consider it as a potential aspect term [19]. In the example below,“بیان” (statement) is an opinion word, and “بڑا” (major) is an adjective term which is directly associated with the opinion term, therefore, the aspect term will be “بڑا بیان” (major statement) having a positive polarity.

RB—متعلق PSP—سے NN—قسم JJ—نئی PSP—کی NN—وائرس NN—کورونا

.NN—بیان JJ—بڑا PSP—کا NN—صحت NN—ادارہ JJ—عالمی

World Health Organization’s major statement on the new strain of corona virus.

Rule 7: If a sentence contains more than one aspect, the aspect closest to the sentiment word will be considered. In other words, the closest aspect directly associated with the opinion will be extracted. In the example below, two aspect terms are associated with the opinion word “آضافہ” (increase). Therefore, the closest aspect “ہلاکتوں کی تعداد” (death rate) will be considered as a potential multiword a aspect term having a negative polarity.

JJ—بھر NN—ملک PSP—سے NN—وجہ PSP—کی NN—وائرس NN—کورونا

.NN—آضافہ PSP—میں NN—تعداد PSP—کی NN—ہلاکتوں PSP—میں

The death rate has increased across the country due to corona virus.

Rule 8: If a sentence does not have a positive or negative sentiment, the polarity of the aspect will be considered neutral. In the example below, “ایس او پیز” (SOP’s) is not associated with a positive or negative sentiment, hence the polarity of the aspect term “کورونا ایس او پیز” (Corona SOP’s) will be neutral.

.NN—آگئ NN—سامنے NN—رپورٹ PSP—سے NN—حوالے PSP—کے NN—پیز INJ—او NN—ایس NN—کورونا

The report came out regarding Corona SOPs.

Once the ATEs are extracted, they are grouped into their respective ACTs that are summarizations of the aspects and are coarse-grained as compared to ATEs. To group the aspects into their respective categories, the extracted ATEs are analyzed, and seven ACTs are defined i.e., education, general, government policies, health, public gatherings, travel, and others by manually analyzing the ATEs. The polarity of the ACTs is considered the same as ATEs. The health category deals with different aspects i.e., deaths, COVID-19 cases, recoveries, hospitals, and ventilators. Similarly, the education category comprises aspects like online classes, online exams, holidays, etc. The public gathering comprises marriage gatherings, social gatherings, etc. The government policies category contains aspects like NCOC, SOP’s, lockdown, and the travel category deals with the ban on travel and its related aspects. If a tweet contains a generic description of COVID-19, it is assigned to the general category and if a tweet discusses none of these categories, it is assigned to the others category. Tables 2 and 3 show the statistics after the proposed syntactic rules are applied to the Urdu dataset.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Statistics of ATE and ATP.

https://doi.org/10.1371/journal.pone.0317562.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Statistics of ACT.

https://doi.org/10.1371/journal.pone.0317562.t003

Unsupervised approach for Roman Urdu

For Roman Urdu, instead of adopting a rule-based approach, aspects are extracted from the data based on collocations because there is no POS tagger available to tag Roman Urdu data. Collocations are used to extract the ATEs from the tweets that mainly refer to the lexical relationship of two or more words that frequently occur together and make a common expression with a meaning that can be derived from at least one of its components [43]. This technique helps in the identification of phrases related to a particular aspect, even if they are not present adjacently in the text. Firstly, bigram collocations are extracted from the translated Roman Urdu dataset. A total of 2,468 collocations are extracted from the data. Then, the strength of association between the extracted collocations is extracted using point-wise mutual information (PMI) which is a good measure to find association among words based on information theory. It works by comparing the probability of two words that occur together with the probability of observing them independently. It estimates whether the two words are genuinely associated with one another or are placed together by chance. The mutual information of two words with probabilities P(word₁) and P(word₂) can be defined using Equation 1. The probability of individual words P(word₁), P(word₂) and their combined probabilities can be observed by counting the number of occurrences of word₁, word₂ and the co-occurrence of word₁ and word₂ in a dataset normalized by the size of the dataset [44].

(1)

Latent Dirichlet Allocation Algorithm (LDA) [45] is used to find ACT which is coarse grain as compared to ATEs. LDA is a probabilistic topic modeling approach that represents the document as an arbitrary blend over a set of random topics. Each topic is characterized by a distribution of vocabulary. LDA is categorized into two processes i.e., generative process and inference process. The generative process generates the documents in which the values of word-topic probability and the proportion of topic for each document are known. Further, the inference process finds the value of latent variables i.e., word-topic probability and topic proportion of the already known documents. The equation to find the probability of word topic and topic proportion is given below.

(2)

(3)

Rule-based approach for Urdu language

The efficiency of the proposed approach is checked by manually annotating the instances of COVID-19 tweets. We manually extract ATEs, ATP, ACT, and ACP from the dataset. Then, ATEs, ATP, ACT, and ACP are extracted from the dataset using the proposed rule-based approach. For the extraction of ATP, the Urdu sentiment lexicon is used comprising 3,651 positive and 5,730 negative sentiment words. The performance of manual annotation of ATEs, ATP, ACT, and ACP is compared with those predicted by the proposed rule-based approach as shown in Table 5.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Results of rule-based approach for Urdu language.

https://doi.org/10.1371/journal.pone.0317562.t005

Although the achieved F1-scores are satisfactory, we encountered several issues that if addressed can further improve the performance of the proposed approach. For instance, the performance of the syntactic rule-based approach is highly dependent on the POS tagger. However, due to the nature of our dataset, some words could not get quality POS tagging. In the example below, the extracted aspect term should be “شہریار آفریدی کا کرونا” (Corona test of Shehryar Afridi) against the opinion term “منفی” (negative) but it is extracted as “بعد شہریار آفریدی کا کرونا”.

NN—شہریار NN—بعد PSP—کے VBI—رہنے PSP—میں NN—قرنطینہ

NN—آگیا JJ—منفی NN—کرونا PSP—کا NN—آفریدی

After living in quarantine, the Corona test of Shahriar Afridi turned out negative.

Numbers are very important in our dataset as they portray the death rates and recoveries of the patients. These numbers are not removed from the data during the preprocessing phase. Some of the information about the aspects is incomplete because they are not incorporated while defining the rules. In the example below, the opinion word is “مثبت” (positive) and the extracted aspect is “فیصد کیسز”(% cases) which is not conveying the quality of information about the aspect.

NN—شدہ NN—رپورٹ PSP—کے NN—کورونا PSP—میں NNP—اٹلی

VBF—ہیں JJ—مثبت NN—کیسز NN—فیصد PSP—96

In Italy 96% of reported cases of corona are positive.

Additionally, our main perception was that the syntactic rule-based approach is domain-independent as it extracts the aspects and their polarities based on syntactic patterns however, in the above example, it is observed that the opinion word “مثبت” (positive) shows the positive opinion. However, after analyzing the context of the sentence it is negative as the patients of COVID-19 are up to 96% in Italy. Depending on the context its polarity should be negative, however, it is assigned as positive by the proposed rule-based approach.

Moreover, it is also observed that the efficiency of the rule-based method is dependent on the coverage of the lexicon and the accuracy of the Urdu text written by the user. Many X (formerly Twitter) users do not follow the language standard while writing on OSNs. Consequently, misspelled words are not correctly identified by the tagger resulting in the wrong assignment of the POS tag and affecting the performance of the aspect extraction process. Similarly, lexicons are mostly built in a standard format of the language, hence the misspelled words usually remain uncategorized. To address this, we found that by applying spell-checking methods before aspect extraction and aspect polarity identification further enhances the efficiency of the rule-based approach.

Comparison of rule-based approach with state-of-the-art

We compare our work with the existing state-of-the-art method on Urdu ABSA. According to the standard provided by SemEval [49], there are four classification tasks to perform ABSA: extraction of Aspect Term/s (ATEs), Aspect Term Polarity (ATP), Aspect Category (ACT), and Aspect Category Polarity (ACP). However, the existing study [19] focuses solely on one task, ATEs. In contrast, our study addresses three of these tasks—ATEs, ATP, and ACT—treating ACP equivalently to ATP. For comparison purposes, we conduct experiments within the context of COVID-19 as well as on the datasets covering a broader range of topics. Our approach outperforms the existing state-of-the-art study by an improved F1-score of 5% for ATEs. This enhanced performance is primarily due to the six syntactic rules we developed for ATE extraction, compared to the two rules employed in the previous study. These results are documented in Table 6 and underscore the adaptability of our methods to diverse sentiment analysis scenarios, demonstrating their potential applicability beyond pandemic-related discourse.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Comparison with state-of-the-art-method (P = Precision, R = Recall, F = F1-score).

https://doi.org/10.1371/journal.pone.0317562.t006

Statistical T-test for significant aspects in Roman Urdu

Aspect terms are identified from the tweets using the collocation technique. For this purpose, stop words are removed from the tweets to avoid filtering them as a significant aspect and to reduce noise from the data. For this, the bigram approach is used and initially, 51,149 collocations are identified from the data. To filter out the significant aspects from the extracted collocation, PMI is used which shows the strength of the association between the two words. The PMI values of extracted collocations from our dataset range from . According to the PMI, it is assumed that the words that are closer to zero are significantly associated with each other and they do not occur by chance. Therefore, lower PMI values show more association between two words and vice versa. To evaluate this and to determine the best PMI value for our dataset, a statistical t-test is used on the extracted collocations. For this purpose, a total of 7 experiments are performed and each experiment contains two samples i.e., significant aspects and non-significant aspects. The details of these experiments are listed in Table 7. To check whether these samples are statistically significant or not, our Null Hypothesis (H₀) and Alternative Hypothesis (H_A) are as follows:

H₀: The selected collocations are not significant.

H_A: The selected collocations are significant.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Ranges of significant and not-significant ATEs for Roman Urdu.

https://doi.org/10.1371/journal.pone.0317562.t007

The significance of the hypothesis is statistically evaluated using a two-tailed t-test (t) and confidence interval (CI) with a significance level of 0.05 on the samples of the aspects. Our null hypothesis is accepted for experiments 2 to 7. For expriment 1, our alternative hypothesis is accepted. Therefore, by considering this experiment we chose the collocations with values ranging from as significant aspect terms in our dataset, and a total of 2,648 collocations are identified as significant aspect terms in our dataset.

LDA for aspect categories in Roman Urdu

To identify the aspect categories, the LDA model is trained on the Roman Urdu dataset comprising COVID-19 related tweets. The performance of LDA is checked by considering two evaluation measures i.e., perplexity and coherence. Perplexity quantifies how well a model generalizes the unseen data. A lower value of perplexity indicates better performance of the model. In this research, we achieve the perplexity value of -7, as shown in Table 8, indicating the strong capability of the model to predict the unseen aspects in the given dataset. It also shows that our model effectively finds the aspect categories for SA and accurately captures the relation among the words and aspect categories within the dataset, thus empowering it to predict unseen aspect categories with high precision.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. Results of LDA model for ACT on Roman Urdu dataset.

https://doi.org/10.1371/journal.pone.0317562.t008

Coherence quantifies how well the topics are semantically related to each other. We assess the coherence of identified aspect categories and our model achieves the coherence value of 41, as shown in Table 8. This indicates that the extracted categories are semantically related and offer a coherent illustration of underlying sentiment-bearing topics in a dataset. It highlights the quality of aspect categories identified by LDA. The high value of coherence indicates that the generated aspect categories capture significant and semantically coherent patterns in the dataset. Additionally, it shows that this model successfully grouped the words that are closely associated in terms of sentiments, assisting improved understanding of aspects in reviews. Additionally, the combination of high coherence and low perplexity score shows the effectiveness and strong performance of the LDA model for the identification of aspect categories demonstrating the ability of the model to represent and uncover the latent sentiment-bearing aspect categories in the dataset which is important for the comprehension of overall sentiment expressed in the dataset. The limitation of this study is that the evaluation of the model is performed on a domain-specific dataset i.e., COVID-19 tweets, therefore it might be possible that the results may vary when this model is applied to other domains. Moreover, the choice of hyperparameters and preprocessing techniques may impact the interpretability and performance of the LDA model.