Contributions

The key contributions of this work are as follows.

• To include manually annotating the reviews based on the appreciation and judgment of components of the appraisal theory and making a categorization of it into “High” or “Low” according to predefined sentiment criteria.
• To ensure the accuracy of the manual annotations, the SHAP is used, giving transparency to make efficient decisions about the proposed model and validating the manual annotations by assigning values to the importance of the features to ensure alignment with human-understandable reasoning.
• To improve feature extraction further, the paper also incorporates SimCLR, a self-supervised learning method better able for generalization across different datasets so that the model improves its ability to adapt to a variety of mobile app reviews.
• In order to do good sentiment classification, MobileBERT is where computational efficiency is the focus, which is especially the case in scenarios with limited computing resources, such as mobile devices.
• To further improve generalization, the approach of knowledge distillation takes place, where a smaller model called a student learns to mimic the predictions of a larger model called a teacher, to improve the overall performance of the model.
• To make the model robust to the linguistic differences, data augmentation by the form of replacing the words with their synonyms using WordNet is injected, so that the model is robust enough to capture diverse expressions of sentiment from user-generated content.
• To optimize the proposed framework, Optuna is used for hyperparameter tuning, which helps in automatically estimating the best configurations for the network, such as rate of learning, batch size, weight decay, size of epochs, etc., in order to ensure the best model performances.
• To process the reviews, the ReviewDataset class is used, in which the reviews are tokenized using MobileBertTokenizer, and augmentation of reviews are prepared using synonyms, and for handling imbalanced dataset, weighted sampling using DataLoader is utilized.
• To validate the robustness of the model, K-fold stratified cross-validation is used, which ensures each fold has the same representation of high as positive and low as negative reviews.

Problem statement

Existing studies have not employed the components of appraisal theory in the manual annotation process which are essential for the proper analysis of the nuanced component of sentiment in user reviews. Traditional forms of SA typically relied on polarity, i.e., positive and negative only, or on some form of simple keyword-based methods. Moreover, in many models there are no guarantees for accuracy in annotations or transparency in the process by which the model arrives at a decision. To compute the performance of mobile apps, there is a shortage of an efficient and computationally inexpensive hybrid DL-based framework. This study addresses these gaps by considering the elements of appraisal theory for manual annotation, and validates the annotations using SHAP (SHapley Additive exPlanations) from XAI for accuracy and interpretability. The framework also utilizes a combination of SimCLR, MobileBERT, knowledge distillation and data augmentation to improve the model’s performance, robustness and scalability and offer a comprehensive solution for SA of mobile apps reviews.

Dataset

The proposed model in this paper is evaluated with 19,278 reviews [45] of 7 well-known mobile applications as shown in Table 2. The reviews were fetched by using the Google Play Reviews API and processed using Jupyter Notebook.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Description of dataset.

https://doi.org/10.1371/journal.pone.0349555.t002

Data annotation

In this research work, the mobile app reviews are annotated by a group of trained researchers who have knowledge of sentiment analysis. In order to make the annotations reliable and consistent, we carried out an inter-rater agreement check. In this study, the Cohen Kappa rule is employed to measure the consistency and accuracy among annotators’ work. This process guaranteed reliability and consistency of the annotations, which gave a good ground on which to train ML models. With this approach, we ensured a high level of reliability of the annotation, which is also essential to the validity and efficiency of the sentiment analysis model.

Examples of data annotation

Table 3 comprises of mobile app reviews and its annotations using two major aspects of appraisal theory – appreciation and judgment. Appreciation is about the good qualities of the app, as in its functionality, aesthetics or reliability while judgment is about app’s general performance and the user’s experience, which may include some areas where they can be improved. The sentiment column classifies each review to either high or low depending on the sentiment expressed in the feedback. These reviews are from popular mobile apps which includes -espn, snapchat, skype, opera mini, mx player, vlc mediaplayer, daraz online shopping etc. These reviews are meant to illustrate the range of user perceptions regarding different features in the apps.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Data annotation examples.

https://doi.org/10.1371/journal.pone.0349555.t003

Data preprocessing

In the proposed model, we performed a number of data preprocessing steps for enhancing the quality of app reviews and for preparing the data for ML analysis. First, we would have performed tokenization to segment each review sentence into smaller tokens, with some delimiters including semicolons, commas, and periods. Next, we changed the text to lower case, so that it is uniform and there are no discrepancies of case sensitivity. We then removed the punctuation, as punctuation doesn’t have significant semantic value, and will add noise into the data. Following this, we removed stop words such as “the,” “is” and “and” that are not meaningful for any SA, so that we can concentrate on more meaningful words. For example, a review like “The app is very slow and crashes often” was tokenized as [“app,” “slow,” “crashes,” “often”] with the help of stop words removed to emphasize what made the app being reviewed so great.

Detailed description of proposed model

The proposed approach is developed in two separate phases, which mainly focus on different components. In the first phase, manual annotation in respect to the perceived performance of apps’ reviews is performed. The predefined performance indicators are based on keywords and phrases that are commonly associated with positive or negative sentiments in user reviews.

Once the reviews are manually annotated, the annotations are validated using SHAP from Explainable AI (XAI), which provides an interpretability layer on top of the sentiment model’s predictions. SHAP assigns feature importance values to the words and phrases present in the reviews to provide transparency, which assists the proposed model in making efficient decisions. This validation through SHAP ensures the accuracy of the manually labeled data. The SHAP values also give insight into the influence of the different features (words or phrases) as to what can influence whether a review is considered “High” or “Low”. In the second phase, we introduce advanced NLP techniques in order to improve the model performance and generalization. The model architecture uses SimCLR to extract important features from the data. SimCLR helps the model generalize better across different distributions of mobile app reviews so that they become more robust to different styles of reviews and content. MobileBERT, a lightweight version of the BERT model, is used to optimize the fine-tuning of a classification head in order to accomplish SA while maintaining a high computational efficiency, especially in mobile environments with limited resources.

In this study, the teacher model is a pre-trained SimCLR MobileBERT model, a powerful feature extractor of textual data specifically designed to deal with the complexities of sentiments of reviewing an app. This model is not a fine-tuned model but is applied to produce soft target probabilities used to train the student model. The student model is a fine-tuned sentiment mobileBERT model, highly tailored to predict sentiment labels (positive or negative) of reviews of mobile apps. The student model is trained with knowledge distillation, where the DistillationLoss of KL divergence is between the soft probabilistic predictions of the teacher model and the hard prediction of the student model as well as the standard cross-entropy loss on the true sentiment labels. The method enables the student model to learn effectively with the labeled data (i.e., app reviews with sentiment labels), as well as with the knowledge coded in the teacher model and thereby improves its accuracy in the sentiment analysis tasks without generalizing to other app reviews.

Data augmentation is also implemented in the form of synonym replacement using WordNet, which boosts the diversity of data used for training and ensures that the model is robust and does not get confused with any linguistic variations in the user-generated content. Optuna method is employed for hyperparameter tuning of parameters like learning rate, etc. Hyperparameters are sampled, like the learning rates from 1e-5 to 1e-4, batch sizes from 16, 32, 64, weight decay from 0.01 to 0.1, and epochs from 3 to 5, which maximize the ROC AUC score of the model. The dataset is processed using the Review Dataset class, which includes tokenization of the reviews using MobileBertTokenizer and augmentation with the help of a synonym. The performance of the model is tested by K-fold stratified cross-validation, and an equivalent amount of positive and negative reviews will be used in each fold. After the model is trained and tested across 5-fold cross-validation validation, the results in the form of mean fold accuracy of 88.63% and the ROC AUC of 90.91% are achieved, which shows the effectiveness of the hybrid model (SimCLR, MobileBert, knowledge distillation, data augmentation, and Optuna for hyperparameter optimization). This approach offers a scalable, efficient idea for SA that can deal with the linguistic diversity of user reviews, with high accuracy. The proposed model, as shown in Fig 1 and its pseudocode, is represented as Algorithm 3.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Proposed model to evaluate the performance of mobile apps.

https://doi.org/10.1371/journal.pone.0349555.g001

Algorithm 1. Performance evaluation framework for mobile application reviews.

Input: Mobile application reviews dataset

Output: Mean classification accuracy and average ROC–AUC

Phase I: Manual Annotation and Explainability Validation

1. For each review : a. If r satisfies predefined performance indicators, assign sentiment label = High. b. Otherwise, assign sentiment label = Low. c. Store the manually annotated review.

2. Apply SHAP to compute feature importance scores.

3. For each review : a. Identify dominant features influencing the sentiment prediction. b. If SHAP-based validation confirms annotation consistency, approve the annotation.

Phase II: Representation Learning and Model Optimization

4. Extract semantic representations using SimCLR.

5. Initialize MobileBERT and perform task-specific fine-tuning.

6. Initialize teacher model (BERT-Large).

7. Train lightweight student model via knowledge distillation.

8. Enhance training data using synonym-based augmentation (WordNet).

9. Conduct hyperparameter optimization using Optuna.

10. Tokenize reviews and construct the training dataset.

Phase III: Model Validation and Performance Assessment

11. Perform 5-fold cross-validation: a. Partition the dataset into training and validation subsets. b. Train the optimized model. c. Evaluate performance using Accuracy and ROC–AUC. d. Record fold-wise results.

12. Compute overall mean Accuracy and average ROC–AUC.

13. Return final performance metrics.

SHAP (SHapley Additive exPlanations) for explainability

SHAP is an explainable AI (XAI) methodology with its roots in the cooperative game theory. It is useful to explain the predictions of a model by assigning the contribution of each feature to the final prediction. In particular, SHAP estimates the role of each word (or feature) in the review in predicting sentiment. The Shapley value for feature iii is calculated using the formula shown as Equation 1.

(1)

1. i) : This is the Shapley value for feature i.
2. ii) : This represents the summation over all subsets S of the feature set N that exclude feature i.
3. iii) : This is the weight applied to each subset, ensuring fair distribution of contributions.
4. iv) : This is the contribution of feature i, calculated by the difference in model predictions when feature i is included and excluded from the subset S.

SimCLR (Self-Supervised Contrastive Learning)

SimCLR is a self-supervised learning model that is used to enhance the extraction of features by training a model to differentiate between similar and dissimilar data points. With respect to our case, the framework assists the model to learn powerful representations of the review text, which increases its capacity to be general to various datasets. The mathematical equation of the contrastive loss function used in SimCLR is shown as Equation 2.

(2)

L_contrastive: This represents the contrastive loss. It is a metric of the ability of the model to learn to differentiate between similar and dissimilar pairs. The smaller the contrastive loss, the better the performance, as the model is successfully finding or capturing similar entities that are near one another and dissimilar entities that are distant from each other.

: This is a similarity operation between two feature vectors, z(i,), and z(j). The value of similarity is logarithmicized, presumably due to the desirability of penalizing very low similarities more than high similarities. T is a temperature parameter, which is a scaling factor. It is used to regulate the similarity. Increased temperature results in a more uniform distribution of similarities, whereas decreased temperature results in the model paying more attention to very similar or very different pairs. This assists in optimization of the contrastive loss function.

MobileBERT

MobileBERT is a thin and efficient transformer model. It employs a self-attention mechanism, which enables the model to attend to words in the review text that are important for prediction, irrespective of their position in the input data, thereby capturing contextual relationships between words. This is especially important for sentiment classification tasks, since words can often have different meanings based on their context. Self Attention Mechanism allows MobileBERT to capture contextual relationships between words in a sentence, regardless of their position. The self-attention mechanism used in transformers is mathematically described as Equation 3.

(3)

Attention(Q, K, V) = Softmax(A) V,

where: i) Q, K, and V are the query, key, and value matrices, respectively.

ii) dk is the dimension of the key vectors.

Distillation loss function

Knowledge distillation is harnessed in this work to provide a higher efficiency for the student model. Knowledge distillation transfers this “soft knowledge” from a larger teacher model to a smaller student model. Specifically, the distillation loss is a linear combination of two parts: 1) the Kullback-Leibler (KL) divergence between the outputs from teacher and student respectively and 2) the cross-entropy loss between predicted labels by student and true labels. The formula for distillation loss is shown as Equation 4.

(4)

where: are the softmax outputs in respect to the teacher and student models, respectively.

L(CE) is the cross-entropy loss and is a hyperparameter that controls the balance between the KL divergence and cross-entropy loss.

In this way, the student model is forced not only to minimize classification error with respect to current labels but also to learn from more confident predictions made by teacher model, resulting in better generalization.

PCA (Principal Component Analysis) for dimensionality reduction

Analogous, PCA is used to reduce the dimensionality of feature space by projecting the data onto the most critical principal components that account for maximum variance in the data. This approach reduces model complexity while maintaining critical information. The representation of the most significant features in a clear and understandable way is the benefit of PCA, which is in line with the purpose of enhancing the efficiency of the model in mobile environments when the resources are limited. PCA is more applicable to generalization in relation to a wide variety of data, including reviews of multiple mobile apps across platforms.

PCA is mathematically expressed as Equation 5

(5)

Here X represents the original data matrix, and W is the concatenation of eigen vectors associated with components. PCA can be useful for visualizing high dimensional data in a lower dimension space, and is important for analyzing the model behavior.

Hyperparameter optimization with optuna

By using Bayesian optimization, Optuna can efficiently and automatically search over the hyperparameter space (also known as hyperparameter tuning), such as learning rate and batch size. This guarantees that the model is trained with the best set of hyperparameters.

The loss valued minimized during optimiztion, is the objective function for Optuna and is represented as Equation 6.

(6)

Where: where represents the hyperparameters and L_val is the validation loss. This optimization technique improves the model’s performance by fine-tuning the hyperparameters.

Evaluation metrics

To evaluate the model’s performance, several metrics are calculated, including accuracy, F1-score, precision, recall, and a confusion matrix. These metrics provide a detailed understanding of the model’s classification performance. The entire process is implemented using PyTorch and PyTorch Geometric, with visualization of the results carried out using matplotlib and seaborn. Accuracy: Accuracy measures the percentage of correctly classified samples out of all samples in the dataset. The mathematical formula to calculate accuracy is shown as Equation 7.

(7)

Precision: Precision evaluates the model’s ability to avoid false positives (FPs). The formula for Precision is represented as Equation 8.

(8)

Recall: Recall measures the model’s ability to capture all the true positives in a class. The mathematical equation to calculate Recall is represented as Equation 9.

(9)

F1-Score: F1-score as shown in Equation 10, is the harmonic mean of precision and recall. It balances the trade-off between the two:

(10)

ROC-AUC: ROC-AUC is just the ability of your model to differentiate between classes. TPR represents the rate of actual positives correctly predicted by the ML model, and FPR shows the rate of actual negatives that are falsely predicted by ML model as positive. ROC-AUC is the bottom section of the ROC curve that is used to measure the capacity of your model being able to differentiate between the classes. The mathematical equation to calculate ROC curves is shown as Equation 11.

Let A = TPR, B = FPR

(11)

The range of the ROC curve is from 0 to 1. The symbol d represents a small change and measures how much the A changes as the B changes.

Experimental setup

The experimental setup for this study aims to fine-tune MobileBERT for SA and to include methods, i.e., knowledge distillation and data augmentation, in order to enhance the model performance. The setup starts with the installation of required libraries such as optuna for hyper parameter optimization, panda for dealing with dataset, torch for model building. A custom dataset class is implemented, Review Dataset, with MobileBERT’s tokenizer for transforming the text input into tokenized input. The text data is further augmented with synonym replacement from the WordNet corpus to introduce variability in the training data to boost the model’s ability to generalize. Random-based selection of synonyms from the WordNet synsets is performed using synonym-augmentation function.

The main model architecture is based on the MobileBertModel from the Hugging Face transformers library. The model is first used as a backbone in the SimCLR-MobileBERT-model for the pretraining using contrastive learning. Subsequently, FineTuned Sentiment Model is fine-tuned with sentiments and knowledge distillation is used for transferring knowledge from a SimCLR-MobileBERT-Model to FineTunedSentimentModel. The distillation loss, which is implemented in the DistillationLoss class, is a integration of cross-entropy-loss and Kullback KL divergence which works to match the soft targets (teacher’s predictions) and hard targets (true labels). AdamW optimizer for the training process, get-linear-schedule-with-warmup using StratifiedKFold from sklearn.model-selection to maintain consistency of classes in each fold has been performed. Evaluations of a model is performed with different evaluation metrics.

Fold 0 Results

In Fold 0 in Figs 2–6, there were high model accuracy in 2352 true negatives and 1448 false negatives, but the number of misclassifications (26 false positives and 30 true-positives) reveals the discrimination of the classes. The PR curve (AP = 0.4432) in Fold 0 exhibits a very steep Precision vs. Recall drop, which is strongly influenced by a high rate of False-Positives attempting to recall more samples of the positions, which is an inefficiency of the model itself at the start of the Fold.

Fold 1 Results

As the model advances to Fold 1, the performance is enhanced; the AP score goes up to 0.9747. The confusion matrix reveals 2350 true negatives and 1109 true positives, with 27 false positives and a significantly high number of false negatives (370). The model remembers more positive samples at the expense of a higher false negative rate.‌‌ Figs 7–11 below shows the PR curve of this fold (AP = 0.9747), demonstrating a defined tradeoff between precision and recall, where accuracy is still comparably high at the start of the recall values but begins to decrease as recall rises. This indicates that the model has not maximized its classification ability, but the point is that it is moving in the right direction of maximizing recall at the expense of precision, which is a familiar attribute in the early stages of learning.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Precision–Recall Curve for Fold 0.

https://doi.org/10.1371/journal.pone.0349555.g007

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. Precision–Recall Curve for Fold 1.

https://doi.org/10.1371/journal.pone.0349555.g008

Download:

• PNG
  larger image
• TIFF
  original image

Fig 9. Precision–Recall Curve for Fold 2.

https://doi.org/10.1371/journal.pone.0349555.g009

Download:

• PNG
  larger image
• TIFF
  original image

Fig 10. Precision–Recall Curve for Fold 3.

https://doi.org/10.1371/journal.pone.0349555.g010

Download:

• PNG
  larger image
• TIFF
  original image

Fig 11. Precision–Recall Curve for Fold 4.

https://doi.org/10.1371/journal.pone.0349555.g011

Fold 2 Results

A more fined learning process is exhibited in Fold 2 where the model is depicted to be very refined. The confusions matrix depicts that 2343 negatives were true and 1381 were true positives, this is much less than before, all those false negatives have been reduced, and this means that the model is now more precise in finding positives without compromising too much precision. The AP score of 0.9878 demonstrates a significant increase of the precision and recall, and the PR curve displays the more fluent decay, which is much more appropriate to regulate the trade-off between the two indicators. These features make the model reasonably learn, improve on the decision boundaries, and also learn to get more discriminative features out of the training set at this stage.

Fold 3 Results

At Fold 3, the model achieves the stability stage of the performance, as did the AP score 0.9958. The confusion matrix reveals that there are 2366 true negatives and 1373 true positives, although the amount of false positives (11) and false negatives (105) is very minor. The PR curve and AP score of 0.9958 indicate there is a near-perfect trade-off between precision and recall, where the model stands at a high level of precision throughout the entire range of recall. This shows that the model has acquired the skills to determine positive and negative instances effectively without compromising a substantial amount of precision.

Fold 4 Results

The highest level of performance of the model is in Fold 4 with a value of 0.9887 on AP. The confusion table depicts the true negatives and true positives are 2347 and 1435, respectively, with very few misclassifications (43 false negatives and 30 false positives). The PR curve and (AP = 0.9887) in this fold depict a close balance in precision and recall, signifying that the model has now attained high levels of generalization, and it can now equally detect a positive or negative instance. The diminishing false positives and false negatives as compared to the previous folds indicate that the model has narrowed its decision boundaries, which results in the reduced number of misclassifications.

Folds 0 to Fold 4 are represented as Figs 2–6 while PR curves of Fold 0 to Fold 4 are shown as Figs 7–11, respectively.

ROC analysis of each fold

The ROC curves, as shown in Figs 12–16 for the different folds, suggest that the model becomes more and more successful at differentiating between the positive and negative classes with increasing Area Under the Curve (AUC) values as we go all the way from Fold 0 to Fold 4. Fold 0 (AUC = 0.5747) has a relatively poor performance, with the curve being a little bit better than the diagonal line, which means that the model does slightly better than random guessing. On the opposite side, Fold 1 (AUC = 0.9852), Fold 2 (AUC = 0.9933), Fold 3 (AUC = 0.9974), and Fold 4 (AUC = 0.9951) exhibit a much stronger result, showing a curve closer to the top-left corner, which is a high positive class with very few false-positives. This sharp increase in the curves indicates that the model’s performance in classifying both positive and negative instances improves as the folds go by, with Fold 3 showing almost perfect classification. These results represent the model improvement process, whereby it is trained over the cross-validation process; the model decision boundaries are iteratively improved, and the misclassifications are minimized. The rising values of every fold are an indication that the model is learning and adapting to the data successfully for classification accuracy and generalization improvement.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 12. ROC Curve for Fold 0.

https://doi.org/10.1371/journal.pone.0349555.g012

Download:

• PNG
  larger image
• TIFF
  original image

Fig 13. ROC Curve for Fold 1.

https://doi.org/10.1371/journal.pone.0349555.g013

Download:

• PNG
  larger image
• TIFF
  original image

Fig 14. ROC Curve for Fold 2.

https://doi.org/10.1371/journal.pone.0349555.g014

Download:

• PNG
  larger image
• TIFF
  original image

Fig 15. ROC Curve for Fold 3.

https://doi.org/10.1371/journal.pone.0349555.g015

Download:

• PNG
  larger image
• TIFF
  original image

Fig 16. ROC Curve for Fold 4.

https://doi.org/10.1371/journal.pone.0349555.g016

PCA of each fold

The PCA analysis across the five folds does clearly show the progressive learning and generalization ability of the proposed model. In PCA Fold 0, the data points are pressed near one dominating direction, meaning that poor variance capture results, and the feature representation in this first stage is more limited. Moving to PCA Fold 1, there’s an increase in the spread along PCA Component 1, indicating an increase in the capture of variance and an increase in sensitivity to inherent differences in the features. In PCA Fold 2 the distribution is more balanced between the components, which would indicate that the model is learning more discriminative and stable feature representations. PCA Fold 3 has a relatively compact but well-structured distribution, which seems to reduce noise as well as provide better, consistent feature extraction. Finally, PCA Fold 4 is the most mature representation with the coherent separation and control of the dispersion in the main principal components, which shows the maturity and strength of the model in describing the essential nature of the data. In this case, the gradual acceleration from Fold 0 to Fold 4 indicates that the proposed model has learned to learn better features, variance, and generalization ability among cross-validation folds. PCA Fold 0, PCA Fold 1, PCA Fold 2, PCA Fold 3, and PCA Fold 4 results are illustrated in Figs 17–21.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 17. PCA Results for Fold 0.

https://doi.org/10.1371/journal.pone.0349555.g017

Download:

• PNG
  larger image
• TIFF
  original image

Fig 18. PCA Results for Fold 1.

https://doi.org/10.1371/journal.pone.0349555.g018

Download:

• PNG
  larger image
• TIFF
  original image

Fig 19. PCA Results for Fold 2.

https://doi.org/10.1371/journal.pone.0349555.g019

Download:

• PNG
  larger image
• TIFF
  original image

Fig 20. PCA Results for Fold 3.

https://doi.org/10.1371/journal.pone.0349555.g020

Download:

• PNG
  larger image
• TIFF
  original image

Fig 21. PCA Results for Fold 4.

https://doi.org/10.1371/journal.pone.0349555.g021

The model evaluation result by each fold is shown as Table 4.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Model evaluation results across five-fold cross-validation.

https://doi.org/10.1371/journal.pone.0349555.t004

Comparative analysis with related literature

In comparison to different models that exist in the literature, our proposed model categorizes several advanced techniques that increase the level of accuracy, semantic understanding, and generalization for different datasets. For instance, in [13], while the precision and recall of AdaBoost, XGBoost, and ANNs were 98% and 85%, respectively, they didn’t have advanced techniques to extract features and detect complex semantic features. Our proposed model goes beyond this by incorporating SimCLR for self-supervised learning for better feature extraction and MobileBERT for ensuring computational efficiency and excellent generalization across different data distributions. Similarly, the work presented in [14] identifies the importance of app store data for research but does not utilize the SA. Our model fills this gap by using knowledge distillation and data augmentation, which helps improve both accuracy and semantic understanding of the classification and better covers user sentiments. In [15], the UX-MAPPER tool, although very useful to comprehend the UX factors, lacks the addressing of SA capabilities. Our proposed model is an improvement over this, and it uses something like MobileBERT and SimCLR that helps in performing better SA and promoting more semantic understanding of users’ feedback, which can provide more actionable insights to app developers. In [16], although SVM outperformed NB on classifying sentiment, there is a big advantage of using MobileBERT and SimCLR on our decision-making model, not only improving the feature extraction but also improving the capability on diverse datasets, making it not only capture the semantic nuances in user reviews. IndoBERT in [17] achieved 87.76% accuracy using a dictionary-based method, but our model goes beyond a dictionary-based approach using MobileBERT for efficiency, SimCLR for self-supervised learning, and SHAP for interpretability, which provides superior semantic analysis and generalization. The fine-tuning of BERT in [18] showed an F1 score of 0.80 for positive sentiment; however, our model, by using MobileBERT and SimCLR, outperforms BERT in extracting better features and providing semantic understanding to achieve better generalization among datasets. In [19], SVM achieved an accuracy of 84% for sentiment classification, whereas our model achieves an accuracy and a ROC AUC of 88.63% and 90.91%, respectively, demonstrating even better generalization and semantic processing capability using MobileBERT, SimCLR, and data augmentation. The study in [20] suggests Wiscom gives valuable suggestions but lacks SA capabilities, whereas our model, through MobileBERT and SimCLR, provides full-fledged SA, providing more accurate and semantically enriched insights into user feedback. In [21], BERT achieved positive sentiment percentages for different categories of apps, but our model improves upon BERT using MobileBERT for efficiency, SimCLR for feature extraction, and knowledge distillation for better generalization, achieving better semantic analysis. In [22], feature selection methods, such as Information Gain, GINI Index, and Correlation Matrix, help SVM to improve its performance. Our proposed model outperforms this in that it incorporates PCA for dimensionality reduction and SimCLR for better feature extraction, which helps in getting better results in both accuracy and semantic understanding. Word2Vec and LSTM in [24] helped recall a high accuracy rate; however, there are modes lacking the flexibility and semantic depth in our model by integrating MobileBERT and SimClar and data augmentation that perform better on diverse datasets. In [27], though SA on mobile app service quality has been conducted, it did not use the advanced method, such as SimCLR and MobileBERT, that our model uses to help the accuracy and robustness of user feeds and understand semantic information. The research study in [29], which focuses on the SA in the context of m-banking, does not use self-supervised learning and advanced transformers, which are the foundation of our model for ensuring better generalization, scalability, and semantic analysis. In [32], ABSA with BERT was used for the sentiment triplet extraction, but our model is improved in terms of SimCLR for feature extraction, MobileBERT for efficiency, and knowledge distillation for better robustness and semantic understanding. SVM in [34] achieved 89.33% accuracy but lacks semantic capabilities as compared to our proposed approach and provides better generalization, accuracy, and semantic knowledge among various review data. Finally, in [42], different ML algorithms, such as RF, were applied, but our model is superior to these techniques in terms of MobileBERT and SimCLR in terms of feature extraction, accuracy, and semantic variations in reviews. This makes our proposed model more effective in extracting insights and capturing the complexity of the semantics of user feedback than traditional ML approaches.

Comparative analysis with baseline approaches

Compared to the baseline approaches, which are mainly based on traditional ML models or lightweight transformer-based frameworks such as DistilBERT, our model builds on the recent state-of-the-art algorithms like SimCLR and MobileBERT, which contribute significantly to the ability of the models to understand the semantics and extract features. Nakamura et al. (2025) [15] were interested in automated UX review analysis, but they did not implement formal sentiment classification, whereas our proposed model uses the SimCLR contrastive learning algorithm and MobileBERT to extract sentiments efficiently, which helps to interpret the emotion on a deeper level. In the same manner, feature selection and topic modeling are used by Salleh et al. (2025) [23] and Susanto and Mauritsius (2025) [24], but they are not sufficient to provide the deep semantic representation, which is solved by our model through combining SimCLR with MobileBERT and PCA, thereby improving the generalization. On the contrary, Jafari et al. (2025) [25] are more concerned with prioritizing reviews to action by developers, but their model is less optimized with regard to sentiment classification and context. Moreover, the methods used by authors in [38,39] are limited to DistilBERT and classical text mining tools that have a smaller semantic capacity and cannot effectively handle complex linguistic phenomena. Amirkhalili and Wong (2025) [40] and Rahayu and Hasibuan (2025) [42] reviewed banking apps but did not employ self-supervised learning or advanced transformers in comparison to our proposed model, which offers better contextual insights, model accuracy, and practical viability by combining SimCLR with MobileBERT. A comparative analysis with the baseline approaches is shown in Table 5.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Comparative analysis with baseline approaches.

https://doi.org/10.1371/journal.pone.0349555.t005

Threats to validity and limitations

This study has several limitations. First, the subjective judgment can still be operationalized in the manual categorization of mobile app reviews with performance parameter-based guidance of appraisal theory, as well as the use of SHAP-based interpretability. Second, the sample of reviews in mobile apps is also narrow and this could limit the extrapolation of the results to other ypes of user-generated content. Third, linguistic variability in synonym-based augmentation with WordNet can also present the problem of linguistic variation that is not necessarily representative of natural user queries. Fourth, despite the fact that MobileBERT and SimCLR are efficient and representation learning models, the performance of the models can fluctuate when used on unseen datasets with a varying linguistic pattern. Lastly, stratified k-fold cross-validation, Optuna optimization, and distillation enhance robustness, but these methods cannot remove the threats of dataset bias, overfitting, or domain dependency.