Image-based emotion elicitation

Several studies have analyzed the sentiment and emotions that can be elicited from images [13–16]. Much of this research builds on earlier work in emotional semantic image retrieval [17, 18], which aims to establish links between low-level image features and emotions to enable automatic image retrieval and categorization [6].

Early approaches to visual sentiment analysis predominantly relied on manual feature extraction, focusing on low-level visual attributes such as color, texture, and composition to infer emotional content [13, 19]. In recent years, the field of emotion elicitation from visual images has increasingly adopted AI tools to enhance the accuracy and efficiency of emotion recognition. Convolutional neural networks, for instance, have the ability to automatically learn and extract complex, high-level features directly from raw image data [20–23]. Additionally, reinforcement learning has been adopted to fine-tune pre-trained models on specific datasets to improve performance in domain-specific emotion analysis [24–26].

Recent advancements have enabled Large Language Models (LLMs) to handle multimodal inputs, such as text, images, and video, making them highly general-purpose tools. In [27], the authors compare the performances of deep learning models with LLMs for image-based emotion recognition using a facial expression image dataset. While deep learning models, such as convolutional neural networks and other specialized architectures, were trained specifically for this task, LLMs were evaluated on the same dataset without any specialized fine-tuning. Among the LLMs, the best-performing model achieved an accuracy of 55.8%, outperforming some deep learning models but falling short of the top specialized models. Despite not surpassing the best deep learning models, LLMs demonstrated competitive performance, particularly with smaller datasets, positioning them as a viable option for specific tasks without requiring extensive additional training.

The study in [8] explores the ability of ChatGPT-4 [12] and Google Bard [28] to interpret emotional cues from both visual and textual data. For visual emotion recognition, the authors employed the Reading the Mind in the Eyes Test [29], which includes 36 images of the eye region of human faces. For textual emotion analysis, they used the Levels of Emotional Awareness Scale [30], consisting of 20 open-ended questions designed to evoke emotional responses. The findings show that ChatGPT-4 excelled in recognizing emotional cues from visual data, achieving scores comparable to human benchmarks and demonstrating no biases related to gender or emotion type. In contrast, Google Bard’s visual recognition performance was near random, with significantly lower scores. Both models, however, performed exceptionally well in interpreting emotions from text, exceeding human averages.

The study in [31] provides a quantitative evaluation of GPT-4V’s performance across 21 benchmark datasets for various tasks related to emotion recognition, including visual sentiment analysis, micro-expression recognition, facial emotion recognition, dynamic facial emotion recognition, and multimodal emotion recognition. The findings demonstrate that GPT-4V has strong visual understanding capabilities, effectively integrating multimodal cues and temporal information, both essential for emotion recognition. However, the model struggles with micro-expression recognition (involuntary facial movements revealing people’s hidden feelings), a task requiring specialized knowledge. While GPT-4V outperforms random guessing, it falls short compared to supervised systems.

In [32], the authors explore the potential of Large Language Models (LLMs) for enhancing emotion recognition using facial data by leveraging multiple learning conditions. They investigate three key approaches: fine-tuning on emotion datasets, as well as zero-shot and few-shot learning for scenarios with limited or no labeled data. The authors utilize In-Context Learning and Chain-of-Thought reasoning to enhance the interpretability and accuracy of LLMs in emotion recognition, enabling the models to provide more informed predictions. Extensive experiments demonstrate that LLMs, even without parameter updates, deliver competitive results in emotion recognition tasks, often outperforming traditional methods.

While most efforts in the literature focus on evaluating the capabilities of LLMs in extracting emotions from facial images, our work shifts the focus to emotion recognition from general, non-facial images. Accordingly, we use the GAPED image dataset, a well-established resource for emotion elicitation. Several studies have employed this dataset for a range of emotional research tasks. For example, Moyal et al. [33] used GAPED images to elicit discrete emotions such as fear, disgust, sadness, and happiness. Balsamo et al. [34] used valence and arousal ratings to evaluate the effectiveness of GAPED images in evoking emotional responses. Moreover, the author in [35] used GAPED images to explore how different combinations of valence and arousal influence cognitive processing, particularly in emotionally ambiguous contexts. A recent study [36] used the GAPED dataset to assess valence and arousal ratings within a Malaysian population using a 9-point Likert scale, with the goal of identifying culturally specific patterns in emotional responses.

Text-based emotion elicitation

Numerous studies have explored the extraction of emotions from text, with early work focusing on sentiment analysis and emotion classification using lexical resources and traditional machine learning techniques [37–39]. These approaches often rely on manually designed features, such as word frequency, polarity lexicons, and syntactic structures, to infer emotional content. A prominent example is the use of associations of words with specific emotions like anger, joy, or sadness, allowing for rule-based emotion detection from textual data [39].

More recently, deep learning models have revolutionized the field by automatically learning complex patterns in text. Recurrent neural networks and their variants have been widely adopted to capture the sequential nature of text and detect emotions based on contextual word embeddings [40, 41]. Transformer-based models, particularly BERT and its variations, have significantly advanced emotion elicitation by leveraging attention mechanisms to understand the relationships between words over long sequences. These models, trained using supervised learning techniques, have achieved state-of-the-art results in emotion classification and sentiment analysis tasks [42, 43]. They are typically fine-tuned on specific labeled emotion datasets to enhance their emotion recognition performance in a supervised learning setting [44].

In addition to supervised approaches, unsupervised and semi-supervised techniques have been proposed to handle scenarios where labeled data is limited. These methods use large pre-trained models combined with transfer learning and self-supervised objectives, allowing them to perform well even in domains with little emotion-labeled text [45, 46].

LLMs have demonstrated commendable performance in sentiment analysis tasks [47–51]. Their emergence has significantly advanced emotion elicitation from text, as they possess a deep understanding of natural language semantics and context.

In a recent study, [47] evaluated the alignment between LLMs and human emotions and values using a novel psychometric text-based assessment called Situational Evaluation of Complex Emotional Understanding, specifically tailored for evaluating Emotional Intelligence across both human participants and LLMs. Their findings revealed that most LLMs achieved above-average scores on this assessment, with GPT-4 notably surpassing 89% of human participants, underscoring its potential in understanding and reflecting human emotional and value-based nuances.

The study presented in [50] aims to evaluate the effectiveness of large language models (LLMs), specifically GPT-4, in estimating concreteness, valence, and arousal for multi-word expressions, which are essential for understanding emotional and cognitive responses to language. Using the GPT-4o model, researchers compared its predictions with human ratings on concreteness from a prior study and found high correlation scores, indicating that the model accurately reflected human judgments. The study further extended these assessments to multi-word expressions, where GPT-4o continued to produce reliable estimates, underscoring the model’s potential in psycholinguistic research.

In [51], the potential of GPT-4 for automating emotion annotation in a zero-shot setting is examined. The study used four publicly available emotion recognition datasets: firsthand emotional reports labeled with basic emotions, a Twitter dataset with various emotion classes, a Reddit-based dataset featuring a broad range of emotion categories, and a multi-genre English dataset annotated in the valence-arousal-dominance space. Results from human evaluation experiments consistently indicated a preference for GPT-4’s annotations over those by human annotators across multiple datasets and evaluators.

Image dataset

We utilize the Geneva Affective PicturE Database (GAPED) [4], which contains 730 images, classified into three primary categories: positive, neutral, and negative. The positive category includes images of human and animal infants, as well as nature scenes. Neutral images typically depict inanimate objects. The negative category is further divided into four subclasses: animal mistreatment (featuring scenes of animal cruelty), human concerns (depicting violations of human rights), snakes, and spiders. Table 1 summarizes the number of images in each category along with their average valence and arousal ratings as evaluated by human participants.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Number of images in each category of the dataset along with their average valence and arousal ratings provided by human participants. The table also illustrates the distribution of human ratings for valence and arousal using the Likert scale.

https://doi.org/10.1371/journal.pone.0324127.t001

Each image in the GAPED dataset was rated by human evaluators across two emotional dimensions: valence and arousal. These evaluations were provided by sixty participants from a second-year psychology class, with an average age of 24 years. The participants, though predominantly native French speakers, represented diverse cultural backgrounds.

The ratings, expressed as decimal values from 0 to 100, represent the emotional responses elicited by the images. Valence scores reflect the emotional tone, where 0 corresponds to very negative emotions and 100 to very positive ones, with 50 representing a neutral response. Arousal scores, on the other hand, measure emotional intensity, where 0 indicates calmness and 100 represents high stimulation, with 50 denoting neutrality.

In the GAPED dataset, positive images generally exhibit high valence scores (70 and above) paired with low arousal scores (below 22). Negative images tend to have lower valence scores (under 50) and moderately higher arousal scores (ranging from 53 to 61). Neutral images typically register valence scores around 55 and arousal scores approximately 25. The image categories have some overlap, i.e., certain images in the Negative category were rated Neutral (for valence or arousal, or both) by human raters. Despite this overlap in rating ranges, the positive, neutral, and negative categories display clear emotional distinctions.

We implemented two key updates to the image dataset. First, we added a textual description to each image, generated by GPT-4, by prompting the model with “What’s in this image?" to describe the content of the image. Second, we assigned valence and arousal ratings on a 3-point Likert scale to each image. Valence was categorized as Negative, Neutral, or Positive, while arousal was classified as Calm, Neutral, or Stimulated. These ratings were derived by mapping the original continuous valence and arousal values to the Likert scale based on the ranges established by human raters in [4]. The valence scale was divided as follows: Negative [0,40), Neutral [40,71), and Positive [71,100]. For arousal, Calm [0,23), Neutral [23,45), and Stimulated [45,100].

It is important to note that this new rating system may lead to changes in the original classification of some images. For instance, an image initially categorized as Negative may now be classified as Neutral based on its updated valence score derived from human ratings. Table 1 illustrates the distribution of human ratings for valence and arousal using the Likert scale.

Valence and arousal ratings for images

We used GPT-4 Turbo [12] for all prompts assessing emotional perception from images. The model was administered two types of rating questions: one requiring numeric response ratings and the other using a Likert scale format.

• Numeric response rating: GPT is prompted to generate numeric ratings for valence and arousal on a scale from 0 to 100, emulating human ratings from GAPED. The following prompt was used:

  You are participating in a visual emotion perception study focused on understanding how different visual scenes affect emotions. As you view the uploaded image, consider its emotional impact.

  Valence: Rate the emotional tone of the image on a scale from 0 (very negative) to 100 (very positive), with 50 indicating neutrality. Arousal: Assess the emotional intensity of the image on a scale from 0 (very calm) to 100 (very stimulated), with 50 representing a neutral state.

  Provide only your estimated ratings for Valence and Arousal for the image, formatted as: “Valence: [value], Arousal: [value]”.

• Likert scale rating: GPT is prompted to provide ratings based on a 3-point scale (Negative, Neutral, Positive for valence; Calm, Neutral, Stimulated for arousal). A similar prompt to the one used for numeric ratings was employed, but it included the specific rating options.

  You are participating in a visual emotion perception study focused on understanding how different visual scenes affect emotions. As you view the uploaded image, consider its emotional impact.

  Valence: Rate the emotional tone of the image on a scale Negative, Neutral, Positive. Arousal: Assess the emotional intensity of the image on a scale Calm, Neutral, Stimulated.

  Provide only your estimated ratings for Valence and Arousal for the image, formatted as: “Valence: [value], Arousal: [value]".

For each rating type, two learning conditions were used to evaluate the model’s performance: zero-shot prompting, where the model generates responses based solely on its pre-trained knowledge without any specific examples, and few-shot prompting, where a small set of example images is provided to guide the model’s responses. In the few-shot setting, the prompt included randomly selected images from each category: 3 from Positive, 3 from Neutral, and a total of 8 from Negative (2 from each sub-category), along with their corresponding human ratings. See S1 Fig for prompt templates used in few-shot prompting.

We conducted four assessments: (1) numeric response ratings using zero-shot prompting, (2) numeric response ratings using few-shot prompting, (3) Likert scale ratings using zero-shot prompting, and (4) Likert scale ratings using few-shot prompting. To ensure consistency in numeric ratings, results are based on the average of 10 responses for numeric response ratings. For Likert scale ratings, the most frequent response from 9 prompts was used to determine the final rating, with 9 prompts chosen to avoid the possibility of ties.

Valence and arousal ratings for image descriptions

Similar to the image rating tasks, GPT-4 Turbo was used for all text-based assessments, where the model was given two types of rating questions: numeric response ratings and Likert scale ratings. In these assessments, each text input was a previously GPT-generated description of an image, with the model relying solely on the description rather than the image itself.

• Numeric response rating: GPT is prompted to generate numeric ratings for valence and arousal on a scale from 0 to 100. The following prompt was used:

  You are participating in a visual emotion perception study focused on understanding how different visual scenes affect emotions. As you read the following image description, consider its emotional impact.

  Valence: Rate the emotional tone of the image description on a scale from 0 (very negative) to 100 (very positive), with 50 indicating neutrality. Arousal: Assess the emotional intensity of the image description on a scale from 0 (very calm) to 100 (very stimulated), with 50 representing a neutral state.

  Provide only your estimated ratings for Valence and Arousal for the image description, formatted as: “Valence: [value], Arousal: [value]".

• Likert scale rating: GPT is prompted to provide ratings based on a 3-point scale (Negative, Neutral, Positive for valence; Calm, Neutral, Stimulated for arousal). A similar prompt format was used but adapted to the rating options.

We employed both zero-shot and few-shot prompting, with few-shot prompts including a set of text-based examples similar to those used in the image assessment.

Valence and arousal ratings for images

Numeric response rating.

GPT-4’s performance in numeric response rating tasks for valence and arousal compared to human raters shows strong correlations. In the zero-shot setting, the Pearson correlation is r = 0.87 (p<0.001) for valence and r = 0.72 (p<0.001) for arousal. In the few-shot setting, the correlations are r = 0.86 (p<0.001) for valence and r = 0.80 (p<0.001) for arousal. Tables 2 and 3 provide a detailed breakdown of these results, including the mean, standard deviation, and range of ratings given by human raters in the GAPED dataset, alongside GPT-4’s ratings under both zero-shot and few-shot learning conditions.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Numeric response ratings for valence across images under zero-shot and few-shot learning conditions. SD represents standard deviation; MAE represents mean absolute error.

https://doi.org/10.1371/journal.pone.0324127.t002

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Numeric response ratings for arousal across images under zero-shot and few-shot learning conditions. SD represents standard deviation; MAE represents mean absolute error.

https://doi.org/10.1371/journal.pone.0324127.t003

Overall, our findings demonstrate that GPT-4 closely approximates human ratings for both valence and arousal in both zero-shot and few-shot settings. As shown in Table 2, the comparison between GPT-4’s zero-shot mean and the human mean indicates that GPT-4 performs remarkably well without any prior task-specific training. Moreover, the consistently low Mean Absolute Error (MAE) across all image categories suggests that the model’s predictions align closely with human assessments, with only minor deviations. For instance, the valence MAE for all image ratings indicates that GPT-4’s predictions deviate from human ratings by an average of 10.5 points. Note that the maximum possible deviation between GPT-4 and the human rating for any image is 100, as both valence and arousal were rated on a 0–100 scale. In our results, MAE values ranged from approximately 5 to 15, indicating that the average error across all image categories was relatively small, representing only 5–15% of the maximum possible error.

This performance improves with the inclusion of examples in the few-shot setting for most image categories. For instance, the Mean Absolute Error (MAE) for valence in Positive images decreases from 13.9 in the zero-shot setting to 8.5 in the few-shot setting, as shown in Table 2. Similarly, for Neutral images, the MAE drops from 6.7 to 5.5. However, in some categories of Negative images, the inclusion of examples did not improve performance and, in certain cases, even resulted in slightly worse predictions compared to the zero-shot setting.

Similar trends are observed for arousal, as detailed in Table 3, where GPT-4’s performance improves with the inclusion of examples in the few-shot setting. For all images, the Mean Absolute Error (MAE) decreases from 11.1 in the zero-shot setting to 9.5 in the few-shot setting. Notably, for Positive images, the MAE drops significantly from 13.3 to 7.8. Comparable enhancements are observed across the other categories.

Fig 1 shows the distribution of differences in valence and arousal ratings between GPT-4 and human raters across all images and individual image categories, using kernel density estimates. Positive differences indicate that GPT-4 rated the emotional impact (either valence or arousal) higher than human raters, while negative differences suggest lower ratings by GPT-4. Each peak within the plots represents the most commonly observed difference. For example, the All Images panel in Fig 1 illustrates the close alignment between human and GPT-4 ratings for both valence and arousal, with peaks near zero indicating strong agreement. In the Positive image category, the few-shot model’s ratings show greater alignment with human ratings, as the few-shot peaks for valence and arousal are closer to zero compared to those in the zero-shot setting. In the Animal mistreatment panel of Fig 1, a prominent peak around +5 indicates that humans generally assign a stronger negative valence to these images compared to GPT-4, while a peak near -5 reveals that GPT-4 tends to provide slightly lower arousal ratings for these images. Similar patterns appear in the Human Concerns panel. Conversely, in the Snakes and Spiders panel, GPT-4 shows a tendency to assign stronger negative valence and heightened arousal relative to human ratings. Fig 2 shows examples with close, moderately close, and divergent ratings between human raters and GPT-4, within each image category.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Distributions of rating differences across image categories.

Distributions of valence and arousal rating differences between human participants and GPT-4 predictions across image categories.

https://doi.org/10.1371/journal.pone.0324127.g001

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Examples of images across different categories with textual descriptions.

Each category includes an example image demonstrating close agreement between human and GPT-4 ratings (difference ≤10), along with examples showing moderate agreement (difference between 10 and 20) and low agreement (difference >20). All images are sourced from [4].

https://doi.org/10.1371/journal.pone.0324127.g002

Likert scale rating.

Fig 3 shows the agreement between human and GPT-4 valence and arousal ratings of all images on the 3-point Likert scale under zero-shot and few-shot conditions. Each matrix highlights the frequency of predicted ratings compared to actual human ratings, illustrating areas of alignment and discrepancy in emotional impact perception across various prompts.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Confusion matrix for 3-point Likert scale prompts for images.

Confusion matrix for 3-point Likert scale prompts under zero-shot and few-shot prompting. Values are expressed as percentages.

https://doi.org/10.1371/journal.pone.0324127.g003

For valence, the top left matrix in Fig. 3 reveals a high accuracy in predicting Positive (98%) and Negative (77%) images. However, approximately 27% of Neutral valence ratings were misclassified as Negative and 6% as Positive. The inclusion of examples in few-shot prompting improved the model’s ability to predict Neutral valence, increasing accuracy from 67% to 77%. However, this approach has led to a slight reduction in the prediction accuracy for Positive and Negative images.

Similarly, for arousal, the model performed well in predicting Calm (100%) and Stimulated (69%) states under zero-shot prompting, as shown in Fig. 3. Few-shot prompting further enhanced accuracy for Neutral arousal predictions but had no significant effect on predictions for Calm or Stimulated states.

S2 Fig and S3 Fig provide detailed confusion matrices, showing GPT-4’s performance for each image category under both zero-shot and few-shot conditions. Tables 4 and 5 present the valence and arousal classification performance metrics for zero-shot and few-shot prompting across various image categories. Each metric evaluates the model’s ability to classify emotional valence categories (Negative, Neutral, or Positive) or arousal states (Calm, Neutral, or Stimulated). Performance is assessed using both a two-class classification approach for individual categories and a three-class classification approach for all images combined. The two-class approach is used because, within individual categories (e.g., Negative), images are only ever rated as Negative or Neutral by both human raters and GPT-4, with Positive ratings not observed. The three-class approach is necessary for all images combined, as all three classes (Negative, Neutral, and Positive) are present.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Valence classification performance metrics for zero-shot and few-shot prompting across different image categories.

https://doi.org/10.1371/journal.pone.0324127.t004

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Arousal classification performance metrics for zero-shot and few-shot prompting across different image categories.

https://doi.org/10.1371/journal.pone.0324127.t005

For valence, the accuracy results in Table 4 show that GPT-4, without any prior training (zero-shot), correctly classified over 75% of the images in most categories, with the exception of Snakes and Spiders. Overall, zero-shot prompting consistently outperformed few-shot prompting across most metrics, including accuracy, recall, and F1-score. This indicates that the zero-shot approach was more reliable for classifying valence across a diverse set of images, providing a more consistent performance across different categories. Few-shot prompting, while still effective, showed slightly lower classification performance, suggesting that additional examples did not significantly improve the model’s ability to generalize across the evaluated image categories.

Table 5 shows moderate accuracy for both zero-shot (0.57) and few-shot (0.55), indicating challenges in classifying arousal levels across all image categories. The high recall for Positive images for both zero-shot and few-shot indicate that most positive arousal instances were correctly identified. The zero-shot performance is very poor for the Neutral image category, indicating that the model failed to identify any Neutral cases correctly. Few-shot shows a slight improvement. Both zero-shot and few-shot prompting perform similarly for Negative image categories. Precision is notably high for both Animal Mistreatment and Human Concerns, but the recall is lower, suggesting that the model is more cautious in predicting high arousal but may miss some cases. The relatively higher F1-score in Snakes and Spiders suggest that GPT-4 is more effective in identifying high-arousal cases associated with those images.

Upon examining Negative images (Animal Mistreatment, Human Concerns, Snakes, and Spiders), human ratings fluctuated between Neutral and Negative, indicating that some images did not convey intensely negative emotions. According to the 3-point scale rating, 42% of these images received a Negative valence rating from humans, while the remaining 58% were rated as Neutral. In comparison, GPT-4 classified 41% of the images as Negative, with the rest rated as Neutral and occasionally as Positive. Notably, the ratings do not always align between human raters and GPT-4. Images rated as Positive by GPT-4 within this category (3% of Negative images) were often misinterpreted. For example, GPT-4 failed to recognize that an image depicted an act of killing or that a group of individuals were refugees.

Valence and arousal ratings for image descriptions

Numeric response rating.

GPT-4’s performance on numeric response rating tasks for valence and arousal, based on image descriptions, achieved a Pearson correlation of 0.79 (p<0.001) for valence and 0.65 (p<0.001) for arousal in the zero-shot setting. In the few-shot setting, the Pearson correlations were 0.78 (p<0.001) for valence and 0.71 (p<0.001) for arousal. The detailed results are presented in Tables 6 and 7, respectively.

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. Numeric response ratings for valence across image descriptions under zero-shot and few-shot learning conditions. SD represents standard deviation; MAE represents mean absolute error.

https://doi.org/10.1371/journal.pone.0324127.t006

Download:

• PNG
  larger image
• TIFF
  original image

Table 7. Numeric response ratings for arousal across image descriptions under zero-shot and few-shot learning conditions. SD represents standard deviation; MAE represents mean absolute error.

https://doi.org/10.1371/journal.pone.0324127.t007

Tables 6 and 7 demonstrate that GPT-4 closely approximates human ratings for both valence and arousal across zero-shot and few-shot settings. The MAE in the zero-shot setting shows that GPT-4 performs well in most categories. However, the inclusion of examples in the few-shot setting did not consistently improve valence ratings for negative image categories. For instance, the MAE for valence in Positive images decreases from 9.2 in the zero-shot setting to 6.5 in the few-shot setting, whereas in the Human Concerns category, it increases from 19.8 to 22.4. In contrast, for arousal, the few-shot setting generally led to a reduction in MAE across nearly all image categories. Fig 4 shows the distribution of differences in valence and arousal ratings between GPT-4 and human raters across all images and individual image categories, using kernel density estimates.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Rating differences for image descriptions across image categories.

Distribution of differences in valence and arousal ratings between humans and GPT-4 for image descriptions across image categories.

https://doi.org/10.1371/journal.pone.0324127.g004

Likert scale rating.

Fig 5 shows the agreement between human and GPT-4 valence and arousal ratings of all images on the 3-point Likert scale, under zero-shot and few-shot conditions, respectively. Valence ratings are comparable under zero-shot and few-shot prompting. The model is well predicting the Positive and Neutral images.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Confusion matrix for 3-point Likert scale prompts for image descriptions.

Confusion matrix for 3-point Likert scale prompts under zero-shot and few-shot prompting. Values are expressed as percentages.

https://doi.org/10.1371/journal.pone.0324127.g005

Similar to the valence zero-shot matrix, Fig. 3 shows that GPT-4 performs well in correctly predicting Calm and Stimulated arousal states, achieving 100% and 69% accuracy, respectively, under zero-shot prompting. Providing examples improved the model’s accuracy in predicting Neutral images, but had no significant effect on its performance for Calm or Stimulated arousal states.

Tables 8 and 9 present the valence and arousal classification performance metrics for zero-shot and few-shot prompting based on image descriptions. The results reveal patterns similar to those observed in the image rating task. For instance, the accuracy results in Table 8 show that GPT-4 correctly classified over 70% of the descriptions in most categories, with the exception of Snakes and Spiders. Overall, zero-shot and few-shot prompting exhibit comparable performance across most evaluation metrics. Additionally, Table 9 highlights moderate accuracy for both prompting conditions, suggesting persistent challenges in classifying arousal levels from image descriptions.

Download:

• PNG
  larger image
• TIFF
  original image

Table 8. Valence classification performance metrics for zero-shot and few-shot prompting across different image categories.

https://doi.org/10.1371/journal.pone.0324127.t008

Download:

• PNG
  larger image
• TIFF
  original image

Table 9. Arousal classification performance metrics for zero-shot and few-shot prompting across different image categories.

https://doi.org/10.1371/journal.pone.0324127.t009

Figures S4 Fig and S5 Fig show the performance of GPT-4 against human ratings for each image description across different image categories under zero-shot and few-shot prompting.

Study limitations

It is crucial to acknowledge that the numerical scales for valence and arousal used in this study are not standard in everyday discussions of human emotions. This deviation might have impacted the outcomes, considering GPT-4’s training largely comprises datasets where such specific formats are infrequent.

Our analysis was primarily concentrated on GPT-4 due to its notable performance. Nonetheless, to obtain a comprehensive understanding, it is vital to include a diverse range of models, compare their performances, and delineate their respective strengths and weaknesses.

Additionally, while GPT-4 has shown impressive capabilities in various image emotion rating tasks, its closed-source nature poses limitations on the transparency and interpretability of its outputs. The inaccessibility to the underlying model architecture and training data restricts our ability to fully comprehend the factors influencing its judgments, which is essential for thorough analysis and validation.

Finally, detecting hallucinations in emotion rating tasks can be complex. One potential method to enhance reliability is prompting the model to justify each rating. Analyzing these justifications in relation to their corresponding ratings could uncover inconsistencies or unsupported responses. Future research could explore this approach to enhance the robustness of emotion rating tasks.