OCC emotion model.

The OCC emotion model was the first structural model of emotions, proposed by Ortony, Clore, and Collins in 1988 for use in the field of artificial intelligence [23]. It provides an emotion classification scheme that defines 22 basic emotions and uses a consistent cognitive structure for the expression of emotions. This scheme is widely used because its constituent definitions are easy to implement on a computer. The OCC model expresses emotions by means of several functions and emotional rules. D(p, e, t) is used to indicate the expectation degree of subject p desiring to experience an event e. If the event is expected to generate beneficial results, then the function value is positive; otherwise, the function value is negative. Consider the “angry” emotion, for example; I_g(p, e, t) is used to indicate a combination of the overall intensity variables (e.g., expectation, realization, and approximation), and P_j(p, e, t) represents the probability of generating an “angry” state. Then, the rule for generating “anger” is as follows: where F_j() is the function for expressing anger. Although the above rule does not itself produce an experience of anger or an angry feeling, it can be used to trigger another rule. Thus, if the intensity of “anger” is set to I_j for a given threshold T_j, the following can be derived:

This rule activates the emotion of anger. When the strength exceeds the given threshold, the “angry” emotion is generated, and the strength value obtained can be mapped onto one of several more specific “angry” feelings; for example, “mad” corresponds to a moderate value, whereas “hatred” corresponds to a high value. The makeup of the other emotional rules is similar.

However, although the OCC model considers only a generation mechanism based on the cognitive factors driving emotion, emotion is in fact also affected by personality, mood and other non-cognitive factors. Therefore, inadequacies arise when expressing emotion using only the OCC model.

Personality factors.

Compared with emotion, personality is a relatively constant variable. There is no unified standard for the description of personality. In this study, the FFM model [24] was adopted, which is the model that is most widely used at present in the psychological field to describe personality. This model divides the characterization of personality into 5 categories: openness, conscientiousness, extraversion, agreeableness, and neuroticism. The specific descriptions are as follows:

Openness: imagination, aesthetics, affection, creativity, intelligence, etc.;

Conscientiousness: competence, fairness, orderliness, responsibility, consciousness, restraint, etc.;

Extraversion: enthusiasm, sociality, decisiveness, sense of adventure, optimism, etc.;

Agreeableness: confidence, straightforwardness, altruism, compliance, humility, empathy, etc.;

Neuroticism: anxiety, hospitality, depression, self-awareness, impulsion, fragility, etc.

In this study, a five-element group was defined to describe a person’s personality traits, where −1 ≤ O, C, E, A, N ≤ 1.

Mood factors.

To accurately measure people’s moods, scholars have proposed several measurement theories and methods in the context of psychology. In particular, research shows that the PAD model can effectively measure and explain mood. This model is a three-dimensional measurement model proposed by Mehrabian and Russell in 1974. In this model, mood is divided into the dimensions of Pleasure (P), representing the positive and negative characteristics of an individual’s emotional state; arousability (A), representing the individual’s level of neural physiological activation; and dominance (D), representing the individual’s level of control over the situation and others. The possible values for each dimension lie in the range of [-1, +1]. Based on reference [23], we selected the OCC emotion model, which is partially related to images of natural scenery, as the basis for our model; the mapping relationship between this model and the corresponding values for each dimension of the PAD model is as shown in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Mapping relationship between PAD values and the OCC model.

https://doi.org/10.1371/journal.pone.0191064.t001

In this study, a three-element group M(P, A, D) was defined to measure a person’s mood features, where −1 ≤ P, A, D ≤ 1.

Emotional modelling of scene images.

According to reference [25], the mapping relationship between personality and mood is defined as follows: (1) where P, A and D on the left-hand sides of the equations represent the three affective dimensions in the PAD model and O, C, E, A, and N on the right-hand sides represent the 5 personality traits in the FFM model.

Because personality spans a five-dimensional space and mood spans a three-dimensional space, according to Eq (1), the mapping matrix between personality and mood can be derived as follows: such that (2)

According to the mapping relationship between the OCC emotion model and the PAD values given in Table 1, the mapping matrix between the OCC model and the PAD values can be derived as follows:

Therefore, the improved OCC emotion model integrating personality and mood can ultimately be expressed as follows: (3)

In this way, a 10 × 1 matrix is obtained for quantifying emotion.

Image feature selection and emotional semantic mapping.

The visual features of images that are commonly considered include colour, texture and shape. In particular, through observation of the human visual system and in combination with the characteristics of natural scene images themselves, we find that colour plays a crucial role in people’s understanding of the emotional semantics associated with images. For a natural scene image, on the one hand, the image itself is typically extremely irregular, so the extraction of texture and shape features is difficult. On the other hand, these two types of features do not play an obvious role in determining the image’s emotional semantics. Therefore, in this study, colour features were extracted as visual features to reflect the emotional semantics of natural scene images [26]. The HSV (Hue, Saturation, and Value) colour space can effectively reflect human perception of colour [27]; therefore, we chose the HSV space as the working space in which to calculate the colour histogram of each image, and we selected the three colours that accounted for the largest proportion of each image as the main colours of that image.

In daily life and work, when people observe an object, they will tend to have some association with that object. Similarly, when people see a colour image, they will have some association with the colour that applies the most direct stimulation; in other words, they will understand this image in a way that is informed by unique human emotions. For example, when people see an orange colour, they tend to associate it with the golden autumn and rich fruit. Therefore, it is regarded as an abundant, joyous and happy colour, symbolizing warmth, cheerfulness and liveliness. Meanwhile, green is a very beautiful and elegant colour. It is vibrant, symbolizing hope and life. In this study, a summary of the relationships between colours and the improved OCC emotion model was constructed through an experimental investigation. First, we randomly selected 1,000 images from the SUN Database [28] and designed a website as an open-environment experimental platform, whose architecture is shown in Fig 1. The professional psychology software E-Prime was used to present images to subjects, and the specific experimental settings are shown in Table 2. The subjects who participated in the investigation were limited to college students of different years and employees in certain industries, aged between 18 and 45. Before the experiment, the subjects were required to fill out several electronic documents, such as a basic personal information sheet, a personality evaluation scale and an experimental feedback form. During the experiment, the images were randomly presented, and a prompt interface was displayed approximately 1 s before the presentation of each image. After the subject completed the requested choice, there was a rest interface that lasted approximately 1 s to prevent fatigue and deterioration of mood. During the extraction of subject data, the rationality and validity of the data were preliminarily evaluated, and obviously invalid data were removed before statistical analysis. The vocabulary with the highest frequency and the corresponding relationships were summarized, and the statistical survey results are presented in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Framework of the behavioural experiment platform.

https://doi.org/10.1371/journal.pone.0191064.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Experimental scheme for emotional semantic acquisition.

https://doi.org/10.1371/journal.pone.0191064.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Mapping between colour and emotion.

https://doi.org/10.1371/journal.pone.0191064.t003

SVM-KNN image classification algorithm based on MapReduce.

The SVM algorithm is a machine learning method proposed by Vapnik in 1995, based on statistical learning theory and structural risk minimization theory [29]. It is designed for two-category classification, has a good generalization ability, and is suitable for solving nonlinear problems involving high-dimensional data; consequently, it has been widely used in classification, identification, retrieval, and similar tasks. However, Vapnik demonstrated through analysis that in the classification of data into two categories, the SVM algorithm tends to produce some misclassifications in the boundary or overlap area between the two categories. The KNN algorithm is a distance- and example-based non-parametric method, proposed by Cover and Hart in 1968 [30]. Although this algorithm is simple and effective, for each datum to be classified, it is necessary to first calculate the similarity between it and each sample in the space to derive the K nearest adjacent points before the comparison is performed. Therefore, the computational cost of the KNN algorithm is relatively large.

However, we can view an SVM classifier as a nearest neighbour classifier that has only one representative point for each class and combine the SVM and KNN algorithms to generate a new algorithm, namely, the SVM-KNN algorithm [31]. This algorithm selects an appropriate penalty parameter and kernel function for the SVM algorithm and incorporates weight coefficients for the KNN algorithm, which can be derived based on the ratio of the number of samples in a class to the total number of samples. When calculating the distance between a datum to be classified and a sample representing a class, the distance is multiplied by the weight coefficient for this class, and the results are used as the basis for comparison. A class with a larger sample size is assigned a smaller weight, and a class with a smaller sample size is assigned a larger weight, thereby avoiding the undesirable impact on classification caused by an uneven sample distribution. At the same time, the Euclidean distance similarity used in the KNN algorithm is replaced with the cosine similarity of a vector space, which has a more stable effect on classification. This improved SVM-KNN algorithm combines the merits of the two original algorithms and optimizes their performance. Furthermore, it does not require as many calculations as the original KNN algorithm does and simultaneously achieves an improved classification accuracy. The main steps of the algorithm are as follows:

1. A training set of scene image samples is used to train the SVM algorithm; thus, the coefficient ω* and the constant b* of the decision function f(x) = sgn(ω*x + b*) are derived to obtain the support vector set D_sv and the given threshold ξ. In addition, the test data set D is preprocessed.
2. If D is empty, then execution is stopped; if D is not empty, then the datum x to be classified is extracted from D.
3. The datum x to be classified is used to calculate to derive dis = |g(x)|.
4. If dis > ξ, then a suitable kernel function is selected for classification by the SVM algorithm. Finally, f(x) is exported.
5. If dis < ξ, then the KNN algorithm is used for classification. The value of K is selected, and the datum x to be classified is imported. The support vector set D_sv is used as the sample points. In the distance calculations, the weight coefficients are used to balance the samples, and then, the final results are exported.
6. The datum x, which has now been classified, is removed from the test data set D, and then algorithm returns to Step (2) for continued execution.

With the advent of the era of big data, the amounts of available image data of various types have increased to orders of magnitude of GB, TB, or even PB. The time efficiency and classification accuracy of a classification algorithm with a single-node architecture as described above will decline sharply when applied to such massive amounts of data. Therefore, image classification algorithms with single-node architectures are facing new challenges [32]. To address this issue, the MapReduce parallel programming model was used in this study to adapt the SVM-KNN algorithm to a parallel design.

The basic idea is as follows: First, a large-scale scene image training data set is subjected to segmentation such that its balance is maintained; thus, each segment can be considered to be generally independent, and feature extraction and classification can be performed for the different segments in parallel to reduce the training time of the SVM algorithm. For a small data set that has already been classified, the SMO algorithm [33] is adopted for training to increase the efficiency of deriving the support vector set for each small data set. Each segment is trained to filter out non-support vector points while retaining support vector points. Then, the training results for two segments are combined to form the next input, and this process is iterated until a single support vector set is obtained. Then, this set is judged to determine whether it has reached the desired iterative precision. Once the SVM classifier has been trained, the test data are also subjected to segmentation processing. The data are successively extracted for calculation on their own nodes to derive dis for comparison with the given threshold ξ, during which each node can independently choose whether to use the SVM algorithm or the KNN algorithm for classification. After the classification of all test data has been completed, the classification results for each node are subjected to unified processing and analysis. Thus, the realization of the parallel SVM-KNN (PSVM-KNN) algorithm requires 4 pairs of Map and Reduce functions, as follows: IterationMapper() and IterationReducer() for the generation of the support vector set, the coefficient w* and the constant b* through iterative training; DisMapper() and DisReducer() for the calculation of dis; SVMMapper() and SVMReducer() for the parallel SVM classification functions; and KNNMapper() and KNNReducer() for the parallel KNN classification functions.

Function Iteration ()

{

IterationMapper()

{

split(); //the scene image training set is segmented and sent to each node for processing

while(support vector set>1)

  the support vector set of each segment is calculated;

}

IterationReducer()

{

//the support vector sets output from IterationMapper() in key/value form are paired and processed

if(the final support vector set has reached the desired iterative precision)

return D_sv, w*, b*;

  else

 call Iteration();

}

}

Function Dis()

{

DisMapper()

{

  split_dis(); //the test data set is segmented and sent to each node for processing

   for the test data set D′ in each segment do

    calculate dis in each segment;

}

 DisReducer()

 {

//the support vector sets output by DisMapper() in key/value form are paired and processed

 return dis;

  if(dis > ξ)

   call SVM();

  else

   call KNN();

}

}

Function SVM()

{

 SVMMapper()

 {

  split_SVM(); //the support vector set with dis > ξ is segmented and sent to each node for processing

  for each dis in each segment do

   use the SVM algorithm for classification;

 }

 SVMReducer()

 {

  the results output from SVMMapper() in key/value form are paired and processed;

 }

}

Function KNN()

{

 KNNMapper()

 {

  split_KNN(); //the support vector set with dis ≤ ξ is segmented and sent to each node for processing

  for each dis in each segment do

  {

   add the weight coefficients to recalculate dis;

   apply the KNN algorithm calculation;

  }

 }

 KNNReducer()

 {

  the results output from KNNMapper() in key/value form are paired and processed;

 }

}

For the parallel SVM-KNN classification algorithm, the steps of the semantic extraction and classification process for a large-scale collection of scene images are as follows:

1. In accordance with the FFM model, extract human personality traits and calculate the quantized values of the mood factors according to Eq (2).
2. Extract each image’s colour features and obtain the corresponding OCC emotional words according to Table 2. Assign weights of 0.8, 0.6 and 0.35 to the emotional words corresponding to the first, second and third most predominant colours, respectively, according to the proportion of each colour in the colour histogram, based on the results of repeated experiments.
3. Multiply the P, A, and D values of the corresponding emotional words in Table 1 by the derived weights to update the mapping matrix Y.
4. Obtain the quantized emotional values for each image through calculation according to Eq (3) and select the largest value to classify the emotional semantics of the image.
5. Obtain training and test data as described above and use the MapReduce parallel programming model to construct a parallel implementation of the SVM-KNN classification model on the Hadoop platform to realize the automatic emotional semantic classification of scene images.

Experimental environment.

A Hadoop cluster consisting of five computers was used, with one as the Master node and the remaining four as Slave nodes. Each node had a basic configuration consisting of a 4G quad-core processor, 1 TB of hard disk space, and the Ubuntu operating system.

Data sources.

The experimental data used in this article consisted of images obtained from the SUN Database scene image library (publicly accessible at http://groups.csail.mit.edu/vision/SUN/) combined with images we acquired ourselves. The Sun Database is free to use for researchers in related fields. The library includes 131,067 image samples in 908 scene categories. In total, 20,000 scene images were randomly selected as test samples, and 50,000 scene images were used as training samples. For experimental convenience, all images used in this study were processed to 200 pixels × 200 pixels in size.

Classification accuracy.

To verify the effectiveness of the algorithm, an experimental comparison was first performed in terms of classification accuracy.

Fig 2 shows the classification results for several scene images identified as “joyous” and “hopeful” by users with extraverted personalities. Table 4 lists the classification test accuracies for users of the 5 different personality types for the 20,000-scene-image test sample set in the 10 emotional categories, as obtained using the PSVM-KNN algorithm proposed in this study with a training sample size of 30,000 images.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Classification results for extraverted users for the “joyous” and “hopeful” emotions.

https://doi.org/10.1371/journal.pone.0191064.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. PSVM-KNN classification test accuracy.

https://doi.org/10.1371/journal.pone.0191064.t004

Fig 2 and Table 4 show that the classification performance of the system is satisfactory; the overall classification accuracy is high. Users with different personalities show overall consistency in their understanding of the same image, but the classification accuracy for users with neurotic personalities is lower, indicating that for the same image, the emotional understanding of users with this personality type is more complex and fluctuates significantly. The system’s classification accuracy is also low for the emotions of “lost” and “hate”, indicating that the emotional understanding of visual elements associated with these emotions is rather fuzzy.

To further illustrate the classification performance of the PSVM-KNN algorithm proposed in this study, we compared the experimental results of the PSVM-KNN, SVM, and SVM-KNN algorithms and the parallel support vector machine (PSVM) algorithm proposed in reference [34] for different training sample sizes. Table 5 compares the average test accuracies achieved for users with all different personality types using the different algorithms with different training sample sizes for positive emotions such as “joyous”, “proud” and “hopeful”. Fig 3 compares the average classification accuracies of the different algorithms for different numbers of classification categories.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Comparison of the average classification accuracies of different algorithms for different numbers of classification categories.

https://doi.org/10.1371/journal.pone.0191064.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Comparison of the average classification accuracies of different algorithms for different numbers of training samples.

https://doi.org/10.1371/journal.pone.0191064.t005

The results presented in Table 5 show that the classification accuracy increases with an increase in the number of training samples. However, for the SVM and SVM-KNN algorithms, which have a traditional single-node architecture, even though the accuracy is improved, the training time is also very long when the number of training samples is more than 5,000. For the PSVM and PSVM-KNN algorithms, the classification accuracies are higher than those of the SVM and SVM-KNN algorithms, and when the number of training samples reaches 10,000, the classification accuracies of these two parallel algorithms are both above 95%. Of the two, the accuracy of the PSVM-KNN algorithm proposed in this study is higher. Fig 3 shows that an increase in the number of emotion categories considered will cause the classification accuracy to decrease. However, it is obvious that the decreases for the traditional SVM algorithm and the SVM-KNN algorithm based on a single-node architecture are greater, whereas the decreases for the parallel algorithms implemented in a cluster environment are smaller; in particular, the decreases for the PSVM-KNN algorithm proposed in this study are the smallest in magnitude, thereby fully demonstrating the effectiveness of the proposed algorithm.

The precision rate and recall rate are common criteria used to evaluate image classification performance. The precision rate is the proportion of relevant images that the system retrieves relative to all images it finds during one round of analysis and retrieval. The recall rate is the proportion of relevant images the system retrieves relative to all relevant images in the image library during one round of analysis and retrieval. The accuracy rate and recall rate can be combined to calculate F1 = 2 × precision rate × recall rate/(precision rate + recall rate). The F1 value is used as an overall indicator of classification quality. A higher F1 value indicates a better classification effect and a higher classification accuracy. Fig 4 compares the average precision rates, recall rates and F1 values for users with all different personality types for all 10 emotional categories. Fig 5 shows the relationships between the number of training samples and the F1 value for the different classification algorithms.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Comparison of the average precision rates, recall rates and F1 values for the different classification algorithms.

https://doi.org/10.1371/journal.pone.0191064.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Relationships between the number of training samples and the F1 value for the different classification algorithms.

https://doi.org/10.1371/journal.pone.0191064.g005

The results presented in Fig 4 show that because of the use of the MapReduce distributed parallel programming model in a cluster environment, the PSVM and PSVM-KNN algorithms have higher precision rates, recall rates and F1 values than the SVM and SVM-KNN algorithms, which are based on a single-node architecture. Moreover, because the algorithm proposed in this study uses the KNN algorithm to optimize the SVM algorithm’s performance, it achieves a greater advantage in classification performance. Fig 5 shows that for all of the different classification algorithms, the accuracy of their classification results increases with an increase in the number of training samples. However, their rates of increase eventually decline because once the training sample set reaches a sufficient size for a good classification model to be obtained, a further increase in the number of training samples will have no significant impact on the classification performance. Furthermore, the F1 values of the parallel algorithms are significantly higher than those of the algorithms based on the traditional single-node architecture, clearly demonstrating that the parallel approach is suitable for large-scale data processing. As the amount of data to be processed increases, the performance does not exhibit a linear decline. Instead, it tends to reach an optimum.

To further assess the correctness of the method proposed in this paper, we performed an evaluation using Youden′s index. Youden′s index (also called Youden′s J statistic) is a measure for evaluating the authenticity of screening experiments; it is commonly used in the medical field and captures the total performance of a screening method in terms of the missed diagnosis rate and the misdiagnosis rate [35]. For image classification problems, it is a measure of the balance between the precision rate and the recall rate. A larger value of Youden′s index indicates a better screening effect and higher authenticity. The calculation formula is as follows: Youden′s index = recall rate + precision rate– 1 [36].

Fig 6 compares the experimental results in terms of Youden′s index for the classification results presented in Fig 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Comparison of the Youden′s index values achieved using different classification algorithms.

https://doi.org/10.1371/journal.pone.0191064.g006

The results in Fig 6 show that the algorithm proposed in this paper achieves a Youden’s index value of approximately 0.85, which is much higher than the values achieved by the other classification algorithms. This finding further demonstrates that the classification results reported in this paper achieve a good balance between the precision rate and the recall rate and moreover, that the screening effect and authenticity of classification are excellent.

Training time.

To investigate the time performance of the parallel algorithms in a cluster environment, training time experiments were conducted for the single-node algorithms and the parallel algorithms based on different training sample sizes. However, the training times for the single-node algorithms (SVM and SVM-KNN) were essentially the same, as were the training times for the parallel algorithms (PSVM and PSVM-KNN). Therefore, only the training times for the SVM-KNN and PSVM-KNN algorithms are compared in Fig 7.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Comparison of training times.

https://doi.org/10.1371/journal.pone.0191064.g007

As shown in the figure, when the number of training samples is small, the training time for the PSVM-KNN algorithm is nearly identical to that for the SVM-KNN algorithm. However, as the number of training samples increases, the training time for the SVM-KNN algorithm increases exponentially. For larger training sample sizes, given the same number of training samples, the training time for the SVM-KNN algorithm is much longer than that for the PSVM-KNN algorithm on the Hadoop platform. As the number of training samples increases, the time difference becomes more obvious. These findings also demonstrate the power of distributed parallel computing in processing large amounts of data in a cluster environment.

System speedup.

The speedup is the ratio of the running time of a task in a single-node environment to that of the same task in a multi-node environment [37]. It is an important indicator of the efficiency of a parallel algorithm on the Hadoop platform. To verify the performance of the proposed algorithm on the Hadoop platform, the speedup of the algorithm was tested on three data sets containing 3,000, 16,000 and 50,000 images in different emotion categories. The results are shown in Fig 8.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Speedup comparison.

https://doi.org/10.1371/journal.pone.0191064.g008

Ideally, the speedup of the system should grow linearly with the number of computer nodes. However, due to communication overhead and load balancing, the speedup does not increase linearly in practice. As seen in Fig 8, when the number of images is small, the speedup of the system increases as the number of nodes increases. However, the rate of increase is not high. As the number of images increases, the growth rate also increases. When the number of images reaches 50,000, the speedup of the system grows almost linearly, thereby providing another demonstration of the superior performance that can be achieved by exploiting the Hadoop cluster environment when handling large-scale data sets.