1. Introduction

Public health is “the art and science of preventing disease, prolonging life and promoting health through the organized efforts of society” [1]. The overall aim of public health is to sustainably promote the health and wellbeing of the entire population, strengthening public health services and reducing inequalities [2], including surveillance, prevention and control of infectious diseases, promoting healthy lifestyles, researching disease, and preventing injury. Designated populations can vary, depending on context, from small neighbourhoods to global regions [3]. Moreover, some populations may be more vulnerable and at higher risk of disease due to pre-existing chronic conditions, age and developmental stage, gender and sexual identities, having disabilities or being in marginalised groups in the community (e.g., homeless people) [4].

One of the greatest achievements of public health in the 20^th century is vaccination against infectious diseases, which saves lives, improves individual and population health, and reduces health care costs by billions of dollars annually. Despite the success of vaccines, uptake of vaccination is still inadequate, often delayed, and unstable [5]. Understanding the factors that affect vaccine acceptance or refusal is key to increasing vaccination uptake. In October 2014, World Health Organization (WHO) Strategic Advisory Group of Experts on Immunization (SAGE) published a report of the SAGE Working Group on Vaccine Hesitancy outlying several factors that can influence vaccine acceptance or refusal grouped in four categories: populations, subgroups, communities, and individuals [6].

Social listening is “an active process of attending to, observing, interpreting, and responding to a variety of stimuli through mediated, electronic, and social channels” [7]. It helps public health experts to better engage with the population [8]. Social media has proven to be useful in public health, for example, for understanding the public perception of an epidemic [9,10]. In the context of COVID-19 pandemic, a scoping review analysed 81 studies on social media from 2020; from these and using a modified version of the framework Social Media and Public Health Epidemic and Response (SPHERE) [11], 48 studies were categorised in the public attitudes theme [12]. A systematic review performed in 2019 on existing opinion mining approaches using data from social media platforms found that 94% of the 461 included studies use data only from one social media platform and the vast majority of the studies used data from X (formerly known as Twitter), followed by Sina Weibo and Facebook [13]. Furthermore, in another systematic review of techniques for stance detection published in 2023 [14], social media was the data source of 72% of the 96 included articles in the study.

Several studies have used natural language processing algorithms to extract the general sentiment of a text [15–18], and machine learning approaches with labelled data in order to infer public health opinion from social media data. In addition, there are open-source tools available to support such processes, such as Crowdbreaks [19] and epitweetr [20] which use machine learning methods and labelled data to monitor X for extraction of stance towards public health interventions and early detection of public health threats. However, large annotated datasets are required to have enough representation of the specific tasks for ensuring proper learning and to have enough domain-specific examples when applying machine or deep learning techniques to extract or infer the stance from a text, especially in unstructured data such as social media [21]. Furthermore, having non-experts annotators will require a higher number of annotators to equal the results from the expert annotators, and increase costs [22]. These aspects may be underestimated or not properly considered when conceptualising such studies.

In recent years, the emergence of Large Language Models (LLMs) have revolutionised the field of natural language (NLP) processing. The automated identification of a text author’s position towards a specific topic (stance detection) is key in public health and other fields. LLMs, thanks to their vast training data and enormous model size with often billions of parameters, have demonstrated remarkable proficiency in this domain. A study from Brown et al., 2020, illustrated the effectiveness of models like GPT-3 in accurately determining stance from text, outperforming NLP models in both precision and recall [23]. Furthermore, the adaptability of LLMs to diverse datasets and their capacity to understand nuanced expressions have made them invaluable tools for researchers and practitioners alike. For instance, Devlin et al., 2018, highlighted the versatility of the Bidirectional Encoder Representations from Transformers in handling context-dependent stance detection tasks, setting a precedent for the more advanced LLMs [24].

More recently, LLMs have entered the public debate with the launch of ChatGPT, a chatbot originally based on the GPT-3.5 model, and more recently the GPT-4 model. These LLMs by OpenAI have triggered unprecedented public discourse around artificial intelligence across all domains. Shortly after the launch of ChatGPT, the first open LLMs were released, notably the Llama models by Meta (e.g. Llama 2 [25]). During the years of 2023 and 2024, new incumbents such as Mistral AI and Llama 3 released a number of highly performant open LLMs as a free alternative to commercially available models such as the GPT models. Early results have shown the incredible power in text understanding across domains that these models provided out of the box, pre-trained. This opened up the opportunity for the use of such models for large-scale social media analysis in many domains, including public health, without the resource-intensive steps of data annotation and model training for every individual use case.

The objective of this study was to assess how these new methods would perform for extracting the public’s stance towards vaccination using social media posts. The theme of vaccination was chosen for its relevance to public health, as well for its importance in the public discourse and the broad range of views therein. Further, our focus on stance, rather than just sentiment, is based on the notion that stance is a better measure of vaccination intention. However, stance can be more difficult to assess, because stance can be very different from sentiment (for example, the sentence “I am angry that I haven’t been vaccinated yet” expresses a negative sentiment but a positive stance towards vaccination). Finally, our choice of methods was based on practical utility and methods broadly used (or usable in the future) in various domains: annotation by experts, crowds, lexicon- and rule-based sentiment analysis tools, and large language models (LLMs).

2. Materials and methods

2.4. Classifying the vaccine stance using LLMs

2.4.1. Software.

We used Jupyter Notebook 6.5.2 and Python 3.9.12 to classify the same tweets according to the stance towards vaccination using GPT versions 3.5 (GPT-3.5) and 4 (GPT-4), and Ollama 0.1.17 and Python 3.11.6 to classify the same tweets according to the stance towards vaccination using Mistral (7B), Mixtral (8x7B), Llama3 (8B) and Llama3 (70B). The specific scripts are available in the repository [31]. These LLMs were selected based on its popularity of use (GPT-3.5), expected good performance (GPT-4), and usability with low computing power needed and no cost (Mistral (7B) and Llama3 (8B)) and expected good performance (Mixtral (8x7B) and Llama3 (70B)).

2.4.2. Prompt engineering.

We used different prompts to classify the tweets based on instructions and examples provided [S1 File: Prompts used for large language models, Table 1]. Prompts 1 and 2 were zero-shot prompts, i.e. without labelled examples, and thus the simplest instructions provided to these LLM asking for the sentiment regarding vaccination. The remaining prompts 3 to 8 used few-shot classification, i.e., few labelled examples for each class were provided for the model to extract the features for each class and use these learned features to predict the class of new tweets [23,33,34]. Prompt 3 contained the same examples provided to crowd annotators (i.e., two examples per class). Prompts 4 to 7 were built based on the previous prompts using a random sample of 100 tweets. Prompts were tested in this sample and the wrongly classified tweets were used as examples for the following prompt. For example, if a tweet considered by experts to have a negative class was classified otherwise, this tweet was included in the next prompt as an example for the negative class. Furthermore, examples of tweets misclassified as neutral were added as exceptions for this class. All prompts except from prompt 1 requested the LLM model to provide an explanation on the classification.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Main features of the eight prompts.

https://doi.org/10.1371/journal.pdig.0000631.t001

The same parameters were used for all prompts within each LLM. For GPT-3.5, we used the following parameters: text-davinci-003 as model, 0.8 as temperature, and 0 as frequency and presence penalty. For GPT-4, we used the following parameters: gpt-4 as model, “user” as role, 0.8 as temperature, and 0 as frequency and presence penalty. For Mistral (7B), Mixtral (8x7B), Llama3 (8B) and Llama3 (70B), we used 0.8 as temperature.

3. Results

3.2. Extracting the vaccine stance using crowdsourcing

A total of 467 crowd annotators classified the tweets with the majority of tweets (64.5%) being classified by three different crowd annotators, followed by 14.7% and 7.4% of the tweets being classified by 10 and four crowd annotators, respectively.

The majority of the 756 tweets had an agreement: 85.6% had a partial agreement (i.e., 75% of the crowd annotators agreed on the stance per tweet) and, from these, 91.2% had full agreement.

The stance towards vaccination of tweets with partial agreement was distributed as follows: 89.5% as neutral and 10.5% as positive, with no tweet classified as negative. The stance towards vaccination of tweets with full agreement were 98.1% classified as neutral and 1.9% classified as positive, with no tweet classified as negative (Fig 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Distribution of the tweets per stance, prompt and large language model (percentage of total number of tweets).

The dashed lines represent the stance distribution of positive, neutral and negative tweets according to the experts’ full and partial agreement.

https://doi.org/10.1371/journal.pdig.0000631.g001

3.5. Comparison of the extraction methods

3.5.1. Overall performance for tweets with partial and full agreement.

The overall accuracy for all methods was worse than just selecting the majority class for all tweets, except for prompts 2–8 of GPT-4 in tweets with partial agreement among experts and for all prompts of GPT-4, prompt 3, 4 and 6 of GPT-3.5, prompts 4, 6 and 8 of Llama3 (70B) and prompts 5, 7 and 8 of Mixtral (8x7B) in tweets with full agreement among experts (Fig 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Accuracy and 95% confidence interval per prompt, model and experts’ agreement.

https://doi.org/10.1371/journal.pdig.0000631.g002

All prompts of GPT-4 except for prompt 1, half of the Mixtral (8x7B) prompts (4, 5, 7 and 8), prompt 3 of GPT-3.5 and prompts 4 and 8 of Llama3 (70B) performed better than crowd annotators when comparing the results with the experts’ classification in tweets with partial agreement among experts. In addition to those, prompt 1 of GPT-4, prompts 4, 5 and 6 of GPT-3.5 and prompt 6 of Llama3 (70B) performed better than crowd annotators in tweets with full agreement among experts.

Most of the prompts of Llama3 (8B), some of the prompts of GPT-3.5, Mistral (7B) and Mixtral (8x7B), and one prompt of Llama3 (70B) performed worse than extracting the general sentiment of the tweet using VADER.

Overall, Mistral (7B)’s performance was lower than that of crowd annotators. Interestingly, prompts 1 and 2 had the best accuracy with Mistral (7B) in tweets with partial and full agreement among experts, differently to the other models. Likewise, prompt 2 of Llama3 (8B) had the best accuracy among the 8 prompts.

3.5.2. Overall performance per expert.

The accuracy for each prompt and model varied among experts. The experts who labelled a higher proportion of tweets as neutral stance towards vaccination had higher variability in the accuracy of each prompt within each LLM; whereas the experts who labelled a lower proportion of tweets as neutral stance towards vaccination had a more stable accuracy among the prompts within each LLM. The prompts of some LLM of only two of the experts were statistically different than just classifying all tweets neutral.

3.5.3. Overall performance for the 2–2 agreement tweets.

The accuracy ranged from 0.22 (prompt 6 of Llama3 (8B)) to 0.52 (prompt 4 of GPT-4) with a similar relative difference of accuracy among prompts and methods as shown for the full- or partial-agreement tweets.

3.5.4. Performance per class for partial and full agreement.

The performance metrics per class and method showed high variability according to the experts’ agreement on stance towards vaccination with an overall highest performance for neutral tweets based on the F1 score and sensitivity (Fig 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. F1 score, sensitivity and specificity per model, prompt (if applicable), stance and experts’ agreement.

https://doi.org/10.1371/journal.pdig.0000631.g003

The performance of crowd annotators was the lowest for the minority class (negative stance towards vaccination) having the lowest F1 score and sensitivity (0.08) and the highest specificity (0.99). The metrics of the other methods and prompts had more variability when comparing among all of them. Overall, the F1 score was higher for the negative class and lower for the positive class for all methods and prompts. The sensitivity had higher variability among methods and prompts within methods with an overall tendency of higher values for neutral and negative classes. The specificity had an overall higher value for the neutral and negative classes, except for Llama3 which had a higher value for neutral and positive classes.

4. Discussion

Understanding the population’s perception of public health interventions, such as vaccination, and current public health events is critical for public health experts and epidemiologists. This understanding helps them adjust risk communication and community engagement activities. Achieving this is challenging and requires a holistic approach. Our study concentrates on using social media to assess users’ stances towards vaccination. However, our method could be easily extended to cover other public health interventions and objectives, as well as other platforms or sources of information. With the continuous advancement in LLM technology and better understanding of their performance in different studies and contexts, their usage and integration into public health surveillance systems could greatly improve our ability to monitor and respond to changing public health attitudes.

Our study provides an analysis of attitudes towards vaccination, derived from 1,000 randomly selected English-language tweets collected between 2019 and 2022. This analysis is based on a consensus among four experts regarding the vaccination stance expressed in each tweet. Moreover, we extracted vaccination stances using eight alternative methods: crowd annotators, a rule-based sentiment analysis tool, and six large language models (LLMs), specifically focusing on tweets where at least three out of four experts agreed on the tweet’s classification (i.e., partial expert agreement). These eight methods were chosen because each offers unique advantages. Crowd annotators are considered the gold standard for scalable human annotation, but are nonetheless slow and costly. Rule-based sentiment analysis, while not incorporating the latest machine learning technologies, is valued for its simplicity, speed, and the ability to automate and run on local computers.

The six LLMs, which all offer automation and speed, were selected because they represent the leading edge in their field. The two GPT models from OpenAI, known for underpinning ChatGPT, are among the most recognized and utilised. GPT-3.5, the foundation of the free version of ChatGPT, and GPT-4, the most advanced model publicly available via the OpenAI platform and the basis for ChatGPT Plus, are considered state-of-the-art in LLM technology as of this writing. Access to both models is restricted to OpenAI’s platform. Mistral (7B) and Mixtral (8x7B), from Mistral AI, are considered state-of-the-art for open models. The Mistral 7B model, smaller with 7 billion parameters, can operate on a personal computer, such as a laptop. Mixtral (8x7B), a more complex “mixture of experts” model, may not run on most personal computers but can be installed on local server environments. This versatility implies that Mistral AI’s open models provide privacy benefits, as they allow data to stay on-premises. Furthermore, the two new models launched in April 2024 by Meta (Llama3 8B and 70B) are considered, at the time of writing, the state-of-the-art for open models with similar technical requirements as Mistral (7B) and Mixtral (8x7B), respectively.

We included a lexicon and rule-based sentiment analysis tool, specifically designed to extract only the general sentiment of a text, as a baseline for comparison with the more advanced LLMs. Our findings indicated that this method achieved higher accuracy than some LLMs, particularly those utilising zero-shot prompting. This suggests that inadequate prompt engineering can result in poorer outcomes than simply determining the general sentiment with a straightforward tool like VADER. Similar to crowdsourcing, VADER demonstrated high specificity but low sensitivity for minority classes, indicating that while these methods are less effective in identifying minority classes, they benefit from a reduced rate of false positive detections.

We observed a notable class imbalance among the three identified stances or classes (negative, neutral and positive), similar to what has been found in previous studies [35,36]. The majority were neutral, followed by positive, with negative being the least common stance. A high proportion of tweets (76% of the 1,000 annotated tweets) achieved a partial agreement among experts, but full agreement was limited to 37% of these. This pattern was consistent across all the other methods, with the exception of one GPT-3.5 and several Llama3 (8B) prompts showing a more balanced class distribution.

In the analysis of the tweets classified using crowdsourcing, 86% of the tweets showed partial agreement among annotators, similar to the experts pattern. However, a significantly higher 91% of these reached full agreement. This might be due to the homogeneity of Amazon Mturk annotators. There was no partial or full agreement in negative tweets, with all neutrally-classified tweets reaching full agreement. This could be attributed to our methodological choice of not including a category for unclear or irrelevant tweets as in other studies [13,14], and instructing annotators to opt for neutrality in cases of uncertainty.

In the analysis of the tweets classified using LLMs, the predominant prediction was neutral, except in GPT-3.5 zero-shot prompts where a more even class distribution was found, which could be a consequence of the simplicity of these prompts. GPT-4 demonstrated a consistent class distribution across prompts, as did Mistral (7B) and Mixtral (8x7B) for few-shot prompts, suggesting less prompt-dependency. This is a notable improvement over GPT-3.5, where simpler prompts showed a significantly different class distribution compared to more complex ones.

Some LLMs have shown similar results using different prompts, where having more examples or instructions slightly increased the performance. This is the case for GPT, Mixtral (8x7B) and Llama3 (70B). However, Mistral (7B) and Llama3 (8B), not only showed a higher prompt-dependency for their performance but also a worse performance with higher number of examples and instructions. This could be explained by being more sensitive to overfitting, in which the model relies on the type of information included in the examples focusing their learning only on these, which translates in a much worse performance when being exposed to different texts not included in the list of examples. These aspects are expected to be seen also when using LLMs for other tasks, so the results found in this study in terms of prompt-dependency and sensitivity to overfitting could be easily transferred to other studies.

All methods showed lower performance on the tweets with partial experts’ agreement, indicating a common challenge in resolving ambiguity or unclarity. In general, this ambiguity is caused by having single words with multiple meanings, i.e., polysemy, which is more noticeable in short texts as social media posts in which there is not much information on the context [37]. Other factors, such as humour, sarcasm, or general misunderstandings, indicate that a single objective assessment can be challenging, depending as much on the reader’s interpretation as on the content itself. These considerations are important when employing scalable methods and evaluating the potential risks of misclassification.

Furthermore, despite the 2–2 agreement tweets having lower overall accuracy than the partial experts’ agreement tweets, the relative comparison of accuracy among methods and prompts was very similar. This shows a stability on the expected accuracy when using a specific method and prompt, when applicable, for the same tasks, even if the agreement among experts varies.

Generally, the highest performing prompts of GPT-3.5 and GPT-4 surpassed crowdsourcing in the overall accuracy, as supported by previous studies [38–41]. Furthermore, the overall accuracy of the best performing prompts of Mixtral (8x7B) and specific accuracy for the minority class of the best performing prompt of Mistral (7B) also outperformed crowdsourcing. The cost benefit and time effectiveness of LLMs over crowd annotators, despite usage fees, has been previously highlighted [38]. Furthermore, a study has shown that up to 46% of crowd workers resorted to LLMs for task completion [42].

Each method showed varying performance per class. Some prompts of GPT-4 and Mixtral (8x7B) stood out with the highest overall accuracy and statistically significant better performance than just selecting the majority class for all tweets. They also had the highest F1 scores across all classes, indicating a reduced impact from the class imbalance. GPT-3.5 had the greatest variability in performance by class, particularly in the tweets with full agreement, showing higher dependency on the prompt than other LLMs. Crowdsourcing showed its weakest performance in the negative class, being the method with the lowest performance, suggesting a need for more diverse examples in the dataset. These differences in performance per class must be considered to use complementary models depending on their performance and the objective of the annotation as suggested in a previous study [40].

Mixtral (8x7B), an open-source “mixture of experts” model, demonstrated a performance comparable to GPT-4, a strong result given its accessibility and lower computational requirements. It showed consistent performance across different prompts, with simpler prompts often yielding better results. The accuracy of Mixtral (8x7B) in tweets with full experts’ agreement was statistically better than merely selecting the majority class, a trend also observed for many GPT-4 prompts, and some GPT 3.5 prompts. Given the open nature of Mixtral (8x7B) and its ability to be installed in a local computing environment, this makes Mixtral (8x7B) a very interesting model for use in data-sensitive situations.

Llama3 was expected to have a similar performance as Mistral (7B) and Mixtral (8x7B) based on the preliminary assessments and benchmarking of these models for other tasks [43]. On the one hand, our findings show that Llama3 (8B) had a lower performance than the other models, except for prompt 2, but had a similar relative performance among prompts as Mistral (7B), in which zero-shot prompts had better results than the few-shot prompts. On the other hand, Llama3 (70B) showed a similar performance than Mixtral (8x7B) with the same computing requirements allowing the model to be installed locally for data-sensitive situations,

The limitations of this study is that we did not analyse the explanations provided by LLMs, nor collected these from crowd annotators, which could offer insights into the discrepancies with expert classification. A study by Huang et al., 2023, suggests that GPT-3.5 provides statistically clearer explanations than human-written ones [39], indicating a potential area for future research. Additionally, while our study was limited to English tweets, a previous research indicates that GPT’s performance is consistent in non-English languages when prompts remain in English [40], allowing for multilingual studies.

Future studies could investigate the accuracy of using different methods, including LLMs with different prompts, for other public health interventions or using data from other social media platforms. Furthermore, the impact on the accuracy when using non-English tweets and/or using prompts in a different language than the tweets could be investigated. In general, the speed of the emergence of new models requires a continuous assessment of the models for a public health context.

In summary, GPT-4, Llama3 (70B) and Mixtral (8x7B) showed a consistent high performance across prompts, while GPT3.5, Llama3 (8B) and Mistral (7B) showed more variability and prompt dependency, showing a higher relevance of the prompt engineering for these models. In general, LLMs, particularly in the minority negative class, outperformed crowdsourcing, highlighting their efficacy even with limited class-specific data. Overall, the use of best-in-class models like the open Mixtral (8x7B) and Llama3 (70B) models or the closed GPT-4 model with few-shot prompting are the best methods. All current alternatives have significant risks of substantial misclassification.

Supporting information

Acknowledgments

We would like to thank the crowd and experts for their valuable inputs in the annotation of the tweets.