Semantic orientation (SO)

In the field of (SA), semantic orientation (SO) is an unsupervised learning approach because (SO) does not need preparation of labeled data [16]. Rather, this approach computes how distant a term is towards being negative or positive[17]. By performing unsupervised learning machine, Lexical rules are used in sentiment analysis Rather than basically analysis documents at syntax-level[18]. Kamps, Marx [19] had proposed a model that used lexical relations amid the sentiment analysis task had proposed a model that made utilization of lexical relations amid the sentiment analysis task. In a follow-up study, Qin, Lu [20] outlined a methodology that is utilizations different surveys that are around a similar domain to mine useful printed data. At that point, they formulate semantic comparability measures to recognize a specific sentiment direction. Esuli and Sebastiani [21] proposed a semi-supervised machine learning process, with WordNet in place of a primary vocabulary asset. In their model, a seed set that is imported from WordNet is considered. In their approach, words with the nearby orientation usually have a close label. They implemented a statistical based approach in order to identify the semantic orientation of the seed terms with gloss categorization. In a study conducted by Peng and Shih [22] an unsupervised learning technique for retrieving opinion terms from a document was proposed using parts of speech patterns. In a case of a specific opinion terms are not documented, it was input to a search engine in order to get top total results. After that, with the use of a compiled sentiment lexicon, the sentiments of obscure expressions are computed relying upon the sentiments of close known opinion terms inside the snippets in the retrieved results. Li and Liu [23] designed a k-means clustering model to cluster text blogs toward two categories, which are a positive category and a negative category. The TF-IDF (term frequency—inverse document frequency) weighting is actualized on the content of a text. Next to that, they have implemented a voting majority approach in order to obtain more accurate results. Chaovalit and Zhou [24] had compared between the n-gram machine-learning models against the semantic orientation approach on movie reviews domain. The results show that n-gram machine learning approach outperformed on semantic orientation approach nevertheless require a great amount of computational time in order to train the model. To determine the effects of created an opinion lexicon, Taboada, Brooke [25] found that manually constructing an opinion lexicon is better than constructing a lexicon automatically in order to generate more accurate lexicons. [26] Proposed a sentiment analyses model called SMARTSA which is a lexicon based model. They developed a hybrid lexicon to improves a global lexicon for sentiment analyses based on domain knowledge. [27] had proposed a model that employments the genetic algorithm to generate a sentiment-Lexicon from the training set data. The lexicon-word alongside with its polarities used to implement a manual classification method to obtain the overall polarity test data sentiment. They noted that stop-words may be useful to include in the lexicon.

Machine learning approach (ML)

The more suitable machine—learning approach to the SA often is a supervised approach in particular, and text classification techniques in general [28]. Like text classification techniques, dealing with the problem of SA as a topic-based text classification problem by machine learning approach [29]. Any text classification algorithm can be applied, for instance, Support Vector Machine (SVM), Naïve Bayes (NB), and K-Nearest Neighbour (KNN). Authors like Pang, Lee [5], Abbasi, Chen [30] proposed a model that employs three of the more used machine learning classifiers in the task of text classification, which was Naïve Bayes (NB), K-Nearest Neighbour, Support Vector Machine (SVM). Then they implement this new model on a data set to categories it into a positive and negative group. Khan, proposed a Semi-supervised framework for sentiment analysis called SWIMS. The developing of a general-purpose sentiment lexicon, Min–Max–Normalized-SentiMI, is implemented with using Senti-WordNet to be weighted features. Then SVM machine learning classifier used in sentiment classification.

Combination classifier

Researchers have carried out many studies to compare different types of machine learning (ML) classifiers in addition to making use of machine learning techniques for sentiment classification as pointed out earlier. In the last two decades, extensive studies have been done for classifier combination, which has proved successful in enhancing the performance of a wide range of applications [31, 32].

Polikar [33], Zhou [34] reported that a classifier combination can be considered in the machine learning paradigm in that the combined classifier is used to train multiple learners in resolving one task. Ensemble or classifier combination methods attempt to build a set of hypotheses and combine them for use, which is not the case with ordinary machine learning approaches that try to learn one hypothesis based on the training data [35]. Dasarathy and Sheela [36] conducted one of the earliest studies that deal on ensemble learning, where they employed two or more classifiers and discussed the partitioning of the feature space. In 1990, Hansen and Salamon used an ensemble of similarly configured artificial neural networks (ANN) to enhance the generalisation performance of the ANN [37]. Meanwhile, Wang, Zhang [38] showed that the combination of weak classifiers through Boosting, the predecessor of the suite of AdaBoost algorithms, helped generate a strong classifier in Probably Approximately Correct (PAC) sense. Following these seminal works, many studies relating to ensemble learning have been carried, which frequently appear in the literature under many creative names and ideas [33].

By the rise of deep learning, research in artificial intelligence (AI) has gained new vigor and prominence [39]. In [40], it presented sentiment analysis Utterance-Level model with extrinsic evaluations on deep convolutional neural network algorithem to extract textual features. Likewise, to classify the multimodal heterogeneous fused feature vectors applied Multiple Kernel Learning. [41] designed a first deep learning approach to aspect extraction in sentiment analysis. 7-layer deep convolutional neural network applied to tag each word in opinionated sentences as either aspect or non-aspect word. In this research we combined semantic knowledge and machine learning, in which different different approaches can cover for each other’s flaws[42].

A recent study by [43] involved designing a classifying model that uses a combination of several classifiers as sometimes the lack of quality in a particular classifier can be compensated by the quality of another. It does it by the conclusion the right solution provided by a set of three solutions. The number of the selection algorithms among others are majority approach (simple voting), plural (total) voting. In this work, a voting approach is chosen as the combination classification method due to the simplicity of the application and high performance that obtained when a combined method is used in other text classification fields [44, 45].

Malay sentiment analysis

We could find only a few volumes of research in the literature that applied opinion mining techniques and sentiment analysis over Malay text. There were several techniques involved in our approaches for the Malay language to apply them over Malay text. Some of these approaches implemented traditional ML techniques that were proposed for other languages, while others, despite the rare endeavour, developed new techniques for Malay text.

In Malay SA, Samsudin, Puteh [46] suggested a model which can carry out the task of Malay sentiment analysis. The data used was unstructured and noisy. However, they could not use extra steps, for example lemmatizing, tokenization, stemming and part of speech to solve the problem of the unstructured and noisy data, because these steps require extra tools that were unavailable for the Malay language. They used three ML classifiers SVM, NB, and k-NN. This study was the first to highlight the Malay sentiment classification problem. Samsudin, Hamdan [47] continued their work to improve Malay sentiment analysis. They proposed numerous pre-processing undertakings and a feature selection technique, named FS, which improved the results of Malay opinion mining while using the three classifiers namely Naïve Bayes (NB), Sequential Minimal Optimization (SMO) and k-Nearest Neighbor (KNN). Samsudin, Puteh [48] discussed the use of a feature selection technique FS in opinion mining (OM) using online messages, that were created by the Malaysians. The experiments showed that the technique was better than the traditional ‘filter’ typed feature selection techniques like. Categorical Proportional Difference (CPD), Document Frequency (DF), Information Gain (IG), and CHI Square (CHI). Samsudin, Hamdan [47], Samsudin, Puteh [48], focused on improving the performance of machine learning by using some pre-processing activities and a feature selection. Isa, Puteh [49] considered pre-processing methods for the stemming text in the Malay language using the Reverse Porter Algorithm (RPA) and Backward Forward Algorithm (BFA). After testing, their model’s results showed some enhancement in processing time (when compared to the backward-forward technique), which was an advantage in this model. The performance of the model, during the task of sentiment analysis, was revealed to be similar to using both stemming technique types. Puteh, Isa [50] used an Artificial Immune System (AIS), called Negative Selection Algorithm (NSA), as the individual classifier in sentiment mining for the SAM News Malay newspaper. NSA was able to sentiment mine the newspaper’s data while it was in a standard language. Several problems occurred when major an important detector word when the data did not use a standard language. Additionally, the NSA sentiment-mining model required clean data to operate accurately.

On the contrary, not much of the works were handled using the lexicon approach. According to Chaovalit and Zhou [24], the semantic orientation approach was found to be more efficient but slightly less accurate for use in applications such as Twitter. Furthermore, classifying these data needed no prior training. In general, the semantic orientation approach was found to be practically feasible for automatically mining opinions from unstructured data.

The previous work [15], was more related to our current research. The work dealt with building a hybrid SA model for Malay sentiment analysis. This was achieved by combining both semantic orientation and machine learning techniques through the use of k-NN with a set of features based on the lexicon. Using these features that were based on a polarity lexicon with different classifiers, such as NB, DBN, and SVM, a slight system improvement could be achieved. However, on employing combination classifiers approaches, we could notice significant improvements. The current study is an extension of our earlier research. We used the set of 13 features (as shown in Table 1) that was utilized in [15] to test another traditional classifier for improving the accurate result of our current hybrid model. The main differences compared to our previous work are the use of different machine-learning classifiers and the merging of the hybrid and combination classifiers methods.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Features extracted for each review.

https://doi.org/10.1371/journal.pone.0194852.t001

In recent years, two studies have attempted on Malay sentiment analysis include [51, 52]. Alfred, Yee [51] proposed a model involved three machines learning classifier NB, KNN, and SVM. They discuss the Issues and parameters that affecting Malay sentiment analysis of news headlines using machine learning approaches. Unlike Alfred, Hasbullah, Maynard [52] reported a semantic Role Labeling (SRL) techniques to filter and classify the public sentiment reviews. The dataset collected from official Malaysian government leaders’ social media sites. Meanwhile, they investigated the effects of public sentiment over Malaysian government officials for policy making and the future development in Malaysia. In addition, a Malay SA and other languages had mentioned as a multilingual sentiment analysis task by Chaturvedi, Cambria [53], [54].

There are many limitations in Malay sentiment classifications researches. Most of the previous research in Malay sentiment classification that conducts the machine learning approach focused on the pre-processing phase [47–49]. This is because of the nature of reviews that were usually written in unstructured language or mix language (i.e. Malay and English) by Malay natives on various online communication applications. Conversely, not many works handled with the lexicon approach. According to [24], the semantic orientation (SO) approach is slightly less accurate but is more efficiency to use in applications such as (Twitter) because no prior training is required in order to classify the data. Overall, the semantic orientation approach almost feasible to mine opinions from unstructured data automatically.

Malay Corpus processing

There were no ready-to-use Malaysian reviews data available on the Web. Therefor the performance measurement of this classification model was in agreement with the Malay Reviews Corpus (MRC) that gathered information from numerous online blogs and forums of Malaysian website. The core websites that contributed to this study were ‘http://www.putera.com’ and ‘http://www.mesra.net’. Many websites were considered to ensure that the collected data would represent online reviews by Malaysian communities. The MRC contained almost 2,000 reviews of which 1,000 were deemed as positive, while the remaining 1,000 were considered negative. (Fig 2) presents sample one of the reviews.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. One sample of the reviews.

https://doi.org/10.1371/journal.pone.0194852.g002

Pre-processing

The gathered data were raw in nature. Thus, it was imperative to pre-process the raw data in order to apply classifier. The pre-processing task comprises cleaning, tokenisation, normalization and removal of stop words in a text. The input text should be clean from punctuation marks such as commas and periods. Duplicate words may deviate or modify the overall sentiment of the text and must be removed to present as input text for sentiment analysis. The original meaning of the word deviates due to repeated characters in words such as ‘loooonnng’. Thus, such words with repeated characters need to be brought to their original form. Stop words such as a, an, this, the, and which are very common and need to be removed as these do not determine the text sentiment. A list of ready-to-use stop words is available online for the Malay language. Table 3 presents a sample of the Malay stop words with their corresponding English translation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Sample of the Malay stop words.

https://doi.org/10.1371/journal.pone.0194852.t003

Feature extraction

In feature extraction methods, we represented each review to a set of 13 features by used the lexicon to tag the existing words in the review along with its polarity, an example is shown in (Fig 2) to clarify that. Then we simply calculated all features mentioned in Table 1, for each review as it mentions previously.

Naive Bayes (NB)

The Naive Bayes (NB) classifier is commonly used for the review classification. The algorithm can determine the rear possibilities of the classes to relate the review with the help of a feature vector table. The review is then assigned to the class with the maximum rear possibility. Normally employed two models of the Naïve Bayes approach, multinomial and Bernoulli’s multivariate, for text classification. The Naïve Bayes is a stochastic model for generating documents, which follows the Bayes’ rule. The classification of the potential class, c*, for a new document, d, can be calculated by the following Eq (1): (1)

The NB classifier is used to compute the posterior probability as follows: (2) Where p(c_j|d_i) represents the later probability of the class c_j with a new document d_i as the input and p(c_j) as the probability of the class c_j, which may be computed by the following equation: (3) Where N_i represents the total number of the documents classified in the class, c_j, and N represents the count of documents in all classes. P(d|c_j) is the probability of a document d when a class c_j is the input and P(d) are the probability of document d.

Support vector machines (SVM)

Support vector machines (SVMs) come under the relatively new class of machine learning techniques. In the machine learning community, SVM is a widely popular method for text classification and one of the most efficient techniques for text classification as established by many studies [55, 56].

Based on the concept of structural risk minimisation, a derivative from the computational learning theory, the SVM separates the training data points into two classes by using a decision surface. Decisions are made based on the support vectors that are the sole selected efficient elements in the training set.

SVM can be a solution for two-class problems that deal with the optimisation of the separating hyperplane between the two data sets. Assuming X to be a set of labeled training points (feature vector) (x₁, y₁),…, (x_n, y_n), and each training point x_i ∈ RN is assigned a label y_i ∈ {−1, +1}, where i = 1,…, n. The goal here is to calculate the function f(x) = w.x_i + b and to identify a classifier y(x) = sign(f(x)) that can be solved through the following convex optimisation: (4) Let λ be the regularisation parameter, where x_i: feature vectors, y_i: ∈{-1,+1}, w: normal vector to hyperplane, and b: offset of the hyperplane.

Deep belief network (DBN)

In machine learning, a deep belief network (DBN) is a generative graphical model and is composed of multiple layers of latent variables with connections between the layers but not between units within each layer. In DBN, multiple RBM models are accumulated together and the training process is set from the bottom to the top. However, DBN is different from the multi-layer neural networks. In the multi-layer neural networks, the feature expression performace is robust with the increasing hidden layers. The Backpropagation algorithm may lead to some overfitting problems. In gradient descent method, if the initial value is closer to the optimal solution, the efficient results can be obainted. However, it is difficult to define an optimal initial solution. The DBN can be used to solve this problem efficiently. The training process can be described in Fig 3. The process can be described as the four steps.

1. In the bottom of RBM, the original data is set as training data.
2. The extracted features fromthe bottom of the RBM set as the input training data in the upper layer of RBM.
3. Other higher layer of RBM is repeated by the above two steps.
4. Fine tuning: through these superviseed process, all the parameters are trained supervisedly in the DBN.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. The illustration DBN training processes.

https://doi.org/10.1371/journal.pone.0194852.g003

In Fig 3, Supposed that an observation is the joint distribution of x and l hidden units h₁, h₂, …,h_l

The distribution can be expressed by (5): (5) where x = h⁰

In the proposed algorithm, the DBN supervised learning approach is used as the classifer. Above DBN unsupervised learning algorithm, each layer only can ensure that weights of the self layer achieve the optimal solution of the vector mapping. However, the eigenvector mappingis not optimal in the entire DBN. In the DBN supervised learning approach, in the top of the back-propagation networks are used for spreading errors from top to bottom in each layer of RBM to process fine-tuning in the entire DBN networks. The RBM network training model process can be considered as the network weight initialization in BP netwotks, which overcomes the shortcomings of the BP network because the random initialization of the weight parametersis easy to fall into local optimum and long training time.

Voting classifier combination

The most straightforward voting approach is the majority voting. This technique considers only the most probable class presented by each single classifier to identify the most repeated class label among the output set. In addition, it helps establish whether the overall sentiment of a document is negative or positive. The final classification result is based on the simple majority vote amongst the three classifiers, meaning that two classifiers should agree on the document class out of the three. Weighted majority voting employs a trainable variant of the majority of the voting. In this, every single vote is increased by a weight before the actual voting. The weight for each classifier could be obtained by calculating the accuracies of the classifiers on a validation set. For prediction of an unknown instance, the vote uses each classification model from its sub-process to determine the predicted class to the unknown example with maximum votes.

Baseline model experiments

Initially, SVM, NB and DBN classifiers were applied as a baseline to the entire Unigram feature space. This helped us to assess the overall performance of the classifiers on Malay sentiment analysis without using any features. Table 4 showed the experimental results employing the SVM, NB, and DBN classifiers. Compared with SVM, NB, DBN and classifier combination classifiers performance, the best result was the combination classifier, which was selected as a baseline classifier—in this case, since the classifier combination one achieved the accuracy rate with 80.90%. Furthermore, a comparative evaluation of the sentiment-based features was performed to determine the effectiveness of different feature sets as shown in Table 1. We applied machine-learning classifiers (SVM, NB, DBN and the combination approach) for all features to assess the importance of these features. Furthermore, the impact and the relevance of different sets of features were evaluated for sentiment classification on the MRC.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Performance of the NB, SVM, and DBN classifiers.

https://doi.org/10.1371/journal.pone.0194852.t004

Evaluation of NB, SVM, DBN and combination classifier

A 10-fold cross-validation procedure was utilised to apply DBN, NB, SVM and combination classifiers to the test set. Table 5 presents the data on the accuracy of the DBN, NB, SVM and combination classifier in terms of F-measure of the Malay sentiment analysis. The selected feature was marked by ‘1’ symbol, and consecutively the obtained accuracy is displayed for the combined selected features. The 54 runs testing were set for evaluating the classifiers accuracy. From run (1) to run (9), the full features for every set were selected for evaluating the classifiers accuracy, which was labelled by Group 1. From run (10) to run (53), the parts features of every set or the full features for every set combined with the parts features of every set were selected for evaluating the classifiers accuracy, which was labelled by Group 2. The No.54 run will be analyzed in Fig 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Measure for NB, SVM, DBN and combination (Comb.) classifiers.

https://doi.org/10.1371/journal.pone.0194852.t005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Illustrates highest results of DBN, NB, SVM and classifier combination + 13 sentiment features and Unigram features.

https://doi.org/10.1371/journal.pone.0194852.g004

In Group 1, for run number (2), (5) and (8), the NB classifier was applied to test a set of sentence-level features (F5, F6, F7, and F8). The best three results were obtained, which were 88.81%, 88.70% and 88.88%. In addition, compared with Tables 4 and 5, the quality of Malay sentiment analysis by the NB classification model was affected by the use of feature sets. The results suggested that the use of the feature sets can only outperform the baseline model trained from unigrams. Based on the average F-measures of the Malay sentiment analysis of run number (8), by combining the feature sets of Sentiment words presence-level features, sentence-level features and sentiment words polarity level features. This combination yielded best result in Group 1 of 92.80%.

In Group 1, the SVM classifier was applied by using feature sets to obtain the results in the third column. Table 5 presented the accuracy of the Malay sentiment analysis in terms of F-measure by applying the SVM classifier to different feature sets. The run number (4) that used only the features (F11, F12, and F13) showed a result of 91.55%. This indicates clearly the positive effect on the performance of the SVM classifier. This result was close to the highest value of the SVM result obtained from all performed experimental results. The highest result of 92.17% in run (8) was obtained employing the sentiment words presence-level features, the sentence-level features and the sentiment words polarity level features. Compared with Tables 4, the used feature sets led to increasing performance of the baseline model. It had a clear impact on the quality of Malay sentiment analysis that employed the SVM classifier. As observed from the results of this experiment and the two previous experiments, the SVM classifier led to better results than those obtained by means of the NB classifier. This revealed that the best individual machine learning technique for Malay sentiment analysis was the SVM classifier. In Group 2, run number (10), (16) and (17) displayed the highest three accuracies. Based on the average F-measures of the Malay sentiment analysis of run number (10), the SVM classifier’s performance is greater than for the NB classifier by combining the feature sets of F1(Presence of positive words) F5 (Cumulative frequency of positive words in the first three sentences), F9 (Weighted probabilities of a positive review) and F11 (Average conditional probability of positive subjective words).

In Group 1, as observed from Table 5 of the DBN classifier, run number (4),(8) and (9) displayed the highest three accuracies. Based on the average F-measures of the Malay sentiment analysis, subjective words conditional probability features (F11, F12 and F13) enhanced the DBN classifier’s performance. The best result obtained 93.01% in run number (4). As noticed from these results and from our previous experiments, the performance of DBN classifier was better than the NB and SVM classifiers. This also meant that the effect of the feature set on the DBN classifier’s performance is greater than for NB and SVM classifiers. In Group 2, run number (25), (32) and (35) displayed the highest three accuracies. Based on the average F-measures of the Malay sentiment analysis of run number (25), the DBN classifier’s performance is greater than for NB and SVM classifier by combining the feature sets of F1 presence of positive words F5 Cumulative frequency of positive words in the first three sentences and F9 Weighted probabilities of a positive review.

In Group 1, in the last column of Table 5, it shown shows the accuracy of the Malay sentiment analysis in terms of the F-measure by employing the combination algorithm with different sets of features. The last column of Table 5 empirically evaluated a classifier combination method. In this combination, three classifiers (SVM, NB and DBN) were integrated together for Malay sentiment analysis in order to test the various features set impacts and indicate impacts of the combination classifier approach. Compared with all results in Table 4, it had a clear impact on the quality of Malay sentiment analysis that employed the combination classifier model. As observed from the results of this experiment and the two all previous experiments, the combination classifier led to the best results among the DBN classifier, the NB classifier and the SVM classifier in the baseline model. The use of the feature sets showed noticeable improvement in Table 5. The combination model with applied feature set always showed the highest results. As seen in the last column of Table 5, run number (4), (6) and (8) showed the highest three results. These results performed the highest value obtained from all performed experimental results among the DBN classifier, the NB classifier and the SVM classifier. In Group 2, run number (14), (26) and (41) displayed the highest three accuracies. Based on the average F-measures of the Malay sentiment analysis, the combination classifier’s performance was the best compared with DBN classifier, the NB classifier and the SVM classifier.