Social listening for non-communicable diseases

Many researchers are working on expanding knowledge on social media listening for Non-Communicable Diseases in the literature, and some key contributions are providing support for identifying user behaviors and situations in various cases worldwide. Some of the essential papers are included in this section. For disease-specific social listening studies, various social media listening tools and data collection and analysis techniques,

A. Rodrigues et al. [5] focused on European social media conversations to understand the experiences of lung cancer patients, caregivers, and healthcare professionals. Using Talkwalker and SocialStudio.

J. Chauhan et al. [6] analyzed the experiences of melanoma patients across 14 European countries using SocialStudio and Talkwalker. They identified significant impacts on patients’ daily lives and emotions.

M. Mazza et al. [7] used social listening on Twitter, patient forums, and blogs to explore metastatic breast cancer patient experiences.

Z. Perić et al. [8] investigated GVHD patient needs and lifestyles across Europe using Talkwalker to collect data from Twitter, Facebook, Instagram, and YouTube. The research employed a combination of quantitative and qualitative methods to analyze the quality of life, treatment efficacy, and unmet needs.

Most insights from these studies focus on variables and result presentation formats, which help shape our tool development, like deciding what data to store and how to display results in graphs or dashboards. However, these studies relied on data from existing tools, such as Talkwalker and SocialStudio, with manual analysis performed by researchers. This sets our work apart: We offer a clear approach to automating data collection and computational analysis. Our research also incorporates the latest large language model techniques to summarize extracted knowledge into concise, human-readable paragraphs.

Applications of social media listening

Different studies have employed various tools and techniques to analyze social media data, with a focus on understanding public sentiment and opinions. For analysis techniques employing NLP,

K. H. Manguri et al. [9] analyzed Twitter data from the COVID-19 pandemic using the hashtags #coronavirus and #COVID-19. They applied NLP and Sentiment Analysis.

J. Burzynska, A. Bartosiewicz, and M. Reka [10] used the SentiOne tool for NLP to analyze social media data on COVID-19 in Poland. Their analysis revealed an increase in public discussions and information sharing as the pandemic progressed. For analysis techniques utilizing Latent Dirichlet Allocation (LDA) and Topic Modeling,

C. C. Shoults et al. [11] employed LDA and t-SNE to analyze social media discussions about telemedicine on Reddit and Twitter,

A. C. Sanders et al. [12] conducted sentiment analysis and clustering of Twitter data to study public opinions on face masks during COVID-19. Additionally, quantitative and qualitative analysis, along with bot detection, were employed for social media data analysis;

G. Spitale, N. Biller-Andorno, and F. Germani [13] analyzed Telegram conversations about the Green Pass in Italy using both quantitative and qualitative methods.

Most research on social media listening leverages technologies such as Python for data aggregation, often utilizing packages like Tweepy, which typically necessitate API access or reliance on social media platform functionalities. These studies commonly employ Natural Language Processing (NLP) techniques, including tokenization, stopword removal, sentiment analysis, and topic modeling. In contrast, our open-source NCD Listener tool utilizes Python packages such as Selenium and Beautiful Soup 4, providing out-of-the-box solutions that require only a URL for data acquisition. We also utilize advanced NLP techniques, encompassing tokenization, lemmatization, normalization, and BERT for topic classification. Nevertheless, the insights derived from this research are primarily conveyed through descriptive statistics and dashboard visualizations as well.

Criteria for formulating analytical questions

Based on existing research, we found that there are various criteria for measuring the performance of dashboards and generative AI. However, we are particularly interested in the dashboard performance criteria proposed by S. Almasi et al. [14]. The researchers examined the criteria used to analyze dashboards across various studies by investigating the questions and response formats. They then summarized the criteria used in the analysis, including factors such as Usefulness, Operability, Learnability, and others.

We are also interested in the generative AI performance criteria proposed by J.-L. Peng et al. [15]. The researchers examined performance measures for generative AI from various studies to evaluate the performance of Large Language Models (LLMs). They discussed how to assess the capabilities of LLMs to determine the tasks for which they should be responsible. They proposed a two-step framework consisting of “core capabilities” and “representative capabilities” to explain how LLMs can be applied according to specific capabilities, along with the evaluation method for each step.

Therefore, we define the dashboard performance criteria in terms of content criteria, which are divided into two components: the Appropriateness of the amount of information for task performance and the Quality of the information provided for dashboard communication effectiveness. For the generative AI performance criteria, we define the core ability criteria, which consist of 3 parts: Reasoning, Societal Impact, and Domain Knowledge, for the selection of generative AI-based summarization approaches.

A previous work [16] focused on the development of a social media listening tool for Non-Communicable Diseases (NCDs). This tool integrated data collection from social media, knowledge extraction via keyword matching and machine learning for comment categorization, and presented results through a dashboard and Generative AI-powered summaries of NCD-related discussions. However, the absence of user group testing meant the developed tool did not directly align with user needs. Consequently, the objective of the current work is to rigorously test and further refine the system to optimize its alignment with user requirements.

Moreover, this research introduces a tool designed to minimize financial costs, time expenditure, and human resources required for the automated collection, analysis, and presentation of Non-Communicable Disease (NCD) information from social media. This addresses a significant gap, as much existing research in Social Listening for NCDs [5–8] typically necessitates considerable resources for data aggregation, analysis, and visualization. While most studies on Applications of Social Media Listening [9–13] have achieved automated data processing and presentation, they often still demand programming expertise for utilization, and some are not directly focused on health content [9,12,13]. This limitation frequently leads to challenges in health data analysis, often yielding only sentiment insights rather than in-depth information such as symptoms or treatment methodologies. Consequently, this tool significantly advances studies in information science, health data science, and related fields, broadening their potential for wider application.

Ethics statement

This study was conducted in strict adherence to ethical guidelines, with informed consent obtained from all participants in written forms before their involvement. The Human Ethics Committee of Khon Kaen University, Thailand, based on the ethics of human specimen experimentation of the National Research Council of Thailand, confirmed that formal ethics approval was not required as the research primarily constituted a quality assurance measure with anonymous survey collection (Application Number: HE682030). Confidentiality and data privacy were rigorously maintained throughout the research process, and all data were analyzed anonymously to protect participants’ privacy and well-being.

Systems design

The NCDs Listener system was developed using Python to collect, extract knowledge, and summarize social media posts related to NCDs. It helps turn these posts into meaningful insights. The system implements a 5-step workflow:

1. Data Scraping Step: When a user provides a URL for a social media post, the NCDs Listener tool uses web scraping to extract relevant data. The system accesses the URL and then scans the content based on a set of predefined HTML tags, as specified by the researcher for the specific platform. This process automatically gathers data (the user does not need to define any HTML tags), including information that may be hidden within the page structure. This data is then stored in a four-column table: name, comments, likes, and replies. The Selenium Python package [18] was used to reveal and access this data, and Beautiful Soup 4 [19] collected information through the specified tags.
2. Data Preprocessing Step: The system cleanses the collected data by removing duplicate comments, filtering out extremely short (less than or equal to 5 words) text entries, and anonymizing personal names by replacing them with Name(n), where “n” represents the comment number. Our privacy-preserving methods involved removing names from both the database and the web interface, preventing users from extracting and reusing this identifying information. The preprocessed data can be exported in CSV format for future analysis. The tool only scrapes data that is shared publicly on the internet.
3. Extracting Knowledge Step: The system employs Natural Language Processing (NLP) techniques, including tokenization, stop word removal, and normalization (or lemmatization), utilizing Python packages NLTK [20] and PythaiNLP [21]. Keyword matching is applied to identify knowledge about Non-Communicable Diseases (NCDs) specifically. Additionally, the application of a machine learning model further aids in categorizing comments.
4. Data Adjustment Step: Users can add diseases and symptoms they wish to include or exclude those they do not want. The system then filters the comments based on the specified diseases and symptoms, tailoring the data to the user’s requirements. Upon obtaining the user’s desired information, the system proceeds to process all relevant text comments (aligned with user specifications). These comments are then converted into text vectors using the Embedding-001 model [22]. This vectorization creates a knowledge base, enabling the Generative AI to produce medically-focused summaries.
5. Data Visualization Step: The enhanced data is summarized and displayed on an interactive dashboard. Additionally, a comprehensive summary is generated by a Google Gemini 2.0 Flash model [23]. To ensure the accuracy of this generated summary, the Retrieval-Augmented Generation (RAG) [24] principle is employed, utilizing the previously vectorized data as its knowledge base. Users can also explore the data from a summary statistics table about the post or export the results to a PDF. The workflow is presented in Fig 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. System design and workflow for social media–based NCD monitoring and summarization.

The workflow consists of five stages: (1) Data Scraping – Social media data are collected using Selenium and BeautifulSoup, extracting relevant elements (names, comments, likes, replies) into .CSV files. (2) Data Preprocessing – Duplicate and extremely short comments are removed, and personal identifiers anonymized. (3) Extracting Knowledge – Cleaned text is tokenized, stop words removed, and normalized or lemmatized; NCD-related content is identified via keyword matching, and comment categories classified using a BERT model. (4) Data Adjustment – Content is refined for NCDs of interest, vectorized (Embedding-001), stored in MongoDB, and processed with the Google Gemini API. (5) Data Visualization – Interactive dashboards present results, with retrieval-augmented generation producing summaries and medical-focused narratives. Final datasets and reports are exportable in .CSV and PDF formats.

https://doi.org/10.1371/journal.pone.0332484.g001

Artificial intelligence models for insight extraction

In this work, we employed 2 AI models in the insight extraction. First, machine learning was used for topic classification. Second, LLMs were used for content summarization.

Machine learning for topic classification.

This section details the methodology for the development and evaluation of the BERT model [24] for topic classification. It outlines the complete model development pipeline and presents a comprehensive performance analysis, including an interpretation of the confusion matrix (presented in Fig 2) to assess classification accuracy.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Confusion matrices of classification performance for the Thai and English models.

The matrices illustrate the performance on the hold-out test sets for the Thai model (left), evaluated on 930 instances (30% of the Thai test data), and the English model (right), evaluated on 1,036 instances. The diagonal cells represent the number of correctly classified instances for each class, while off-diagonal cells indicate misclassifications. Both models demonstrate strong predictive accuracy, particularly for the "Sharing experience" and "Non-informative" classes. A comprehensive breakdown of performance metrics, including precision, recall, and F1-scores, is provided in Table 1 (Thai) and Table 2 (English).

https://doi.org/10.1371/journal.pone.0332484.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Performance metrics for the fine-tuned BERT model on the Thai test set.

Overall accuracy: 96.24%.

https://doi.org/10.1371/journal.pone.0332484.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Performance metrics for the fine-tuned BERT model on the English test set.

Overall accuracy: 80.89%.

https://doi.org/10.1371/journal.pone.0332484.t002

BERT Model Output Categories

The fine-tuned BERT model serves as an automated classifier, designed to categorize social media comments with high efficacy. This classification enables a structured analysis of public discourse surrounding NCDs. The model effectively assigns each comment to one of the following three predefined categories:

Non-informative: Comments that are not related to NCDs or lack meaningful information, such as simply stating the name of a disease or unclear descriptions of symptoms. This category is used to filter out such data, thereby improving the accuracy and relevance of the final results.

Sharing experience: Comments that share personal experiences with diseases, including symptoms (before or after diagnosis), treatments, and how the disease has affected the quality of life.

Inquiring: Comments that inquire about various aspects of NCDs, such as symptoms, treatments, or disease impacts.

The model development process was conducted in 4 sequential stages:

1. Step 1: Dataset Collection and Annotation A dataset comprising over 3,000 comments related to Non-Communicable Diseases (NCDs) was extensively collected from social media platforms. This dataset was then manually categorized into three distinct classes: “Non-informative”, “sharing experience”, and “inquiring”. To ensure high-quality and reliable ground-truth data, the final label for each comment was determined by achieving a consensus among three expert annotators.
2. Step 2: Data Partitioning and Preprocessing For robust model evaluation, the annotated dataset was partitioned into a 70% training set and a 30% held-out test set using the train_test_split function from the scikit-learn library in Python. This division ensures that the model’s final performance is assessed on unseen data, providing an unbiased measure of its generalization capability. To mitigate the effects of class imbalance within the training data, an oversampling technique was applied using the “imbalanced-learn” Python package. Subsequently, the raw text underwent a data preparation pipeline involving two key steps: tokenization and stop-word removal. These procedures were performed using the PyThaiNLP library for Thai text and the Natural Language Toolkit (NLTK) for English text.
3. Step 3: Model Selection via Transfer Learning We employed a transfer learning approach to leverage the extensive linguistic knowledge encapsulated in large pre-trained language models. For the Thai language classifier, the “wangchanberta-base-att-spm-uncased” model was selected as the foundational architecture [26]. For the English language classifier, the “bert-large-uncased” model was utilized [27].
4. Step 4: Model Fine-Tuning and Training Separate models for the Thai and English languages were fine-tuned using identical hyperparameters. Each BERT-based model was trained with a maximum sequence length of 200 characters and a learning rate of . To prevent overfitting and ensure the model generalized well, an Early Stopping mechanism was implemented. This technique effectively concludes the training process when performance on a validation set ceases to improve [28,29]. The performance of the resulting fine-tuned models was then comprehensively evaluated on the 30% held-out test set using Python’s scikit-learn library to calculate key performance metrics including accuracy, precision, recall, and F1-score. These evaluation results are detailed in Tables 1 and Table 2.

Thai Model Performance

As shown in Table 1, the Thai BERT model demonstrated exceptionally high performance for two of the three categories.

The “Sharing experience” class achieved an F1-score of 0.95, with relatively strong Precision (0.97) and strong Recall (0.94), suggesting that the model captures most instances of this class while maintaining reasonable accuracy. The “Non-informative” class performed the best overall, with an F1-score of 0.98, benefiting from near-perfect Precision (0.98) and high Recall (0.99).

Conversely, the “Inquiring” class proved to be the most significant challenge, with a low F1-score of 0.40. This score reveals a substantial weakness in the model’s ability to handle questions.

Low Recall (0.40): This is the most critical issue. It signifies that the model correctly identified only 40% of all actual “Inquiring” comments in the test set. The confusion matrix would show that the remaining 60% (False Negatives) were predominantly misclassified as “Sharing experience” or “Non-informative”. This suggests a significant linguistic ambiguity where questions in Thai social media are often embedded within narrative text, making them difficult for the model to distinguish.

Low Precision (0.40): This indicates that when the model predicted a comment was “Inquiring”, it was correct only 40% of the time. This is due to a high number of False Positives, where comments from the other two classes were incorrectly labeled as “Inquiring”.

English Model Performance

The English BERT model exhibited a similar performance pattern, though the challenge with the “Inquiring” class was less severe (Table 2).

The model performed well on the Sharing experience” and Non-informative” classes, achieving F1-scores of 0.78 and 0.85, respectively. This indicates robust and reliable classification performance for the majority of comments, with particularly strong Precision on the “Non-informative” class (0.88).

The “Inquiring” class remained the weakest, with an F1-score of 0.56.

It’s moderate Recall (0.60): The model successfully identified 60% of all true “Inquiring” comments, a considerable improvement over the Thai model. However, the confusion matrix would still show a notable number of False Negatives, where nearly 40% of questions were missed.

Moderate Precision (0.53): With a Precision of 0.53, about half of the comments that the model labeled as “Inquiring” were indeed questions. The other half were False Positives, indicating that the model still struggles with comments that have question-like features but are not explicit inquiries. The primary source of confusion was with the “Sharing experience” class, where comments with question-like features were often misclassified.

In summary, the fine-tuned BERT model demonstrated strong overall performance across both test sets, achieving high accuracy in classifying “Sharing experience” and “Non-informative” categories, with consistently elevated precision, recall, and F1-scores. However, performance was notably lower for the “Inquiring” category, with comparatively reduced precision, recall, and F1-scores, indicating that this class remains more challenging for the model to predict accurately. These results suggest that while the model is effective in identifying the majority of content types, further refinement is required to improve its ability to recognize inquiring statements.

Dashboard design process

In designing the dashboard, a review of previous research, as discussed [3–13], was conducted. The findings indicated that various studies utilized different types of charts, such as bar charts, pie charts, and line charts, even when presenting the same variables. To ensure that the dashboard design aligns with user needs, an assessment of required variables and the appropriateness of chart types was conducted. This assessment involved a total of 10 participants, comprising 5 general users or students from the Faculty of Science, Department of Statistics, Khon Kaen University who had not yet completed the introductory data science course, and 5 data scientists or students from the same department who had completed the course and had shown a high level of proficiency in the subject matter, along with faculty members from the Department of Statistics, Khon Kaen University for the academic year 2024. The research process was carried out as follows:

1. The researcher presented all extracted variables to the participants, with each variable represented using 4 different chart types.
2. The participants subsequently completed an assessment consisting of 8 items to evaluate the relevance of each variable and the appropriateness of the chart types. Additionally, interviews were conducted to gather further feedback and identify areas for improvement.
3. The researcher analyzed participants’ preferences regarding the variables and chart types using the mean score and frequency of selection.
4. Based on the findings, the researchers developed the NCDs Listener dashboard.
   The dashboard design process is presented in Fig 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. The dashboard design process.

The process began with the selection of 10 participants. Each was presented with a set of extractable variables and four chart types for visualization. For every variable, participants rated its importance on a 1–5 Likert scale and selected the best chart from the four options presented. This was repeated for eight variables. Ratings were analyzed to interpret preferences for desired variables, while chart type preferences were determined by identifying the most frequently selected option. The results of these evaluations were used to guide the presentation and design of the final visualizations. These visualizations were then integrated into the development of the NCDs Listener dashboard, ensuring that both content and presentation reflected participant preferences.

https://doi.org/10.1371/journal.pone.0332484.g003

The study identified a total of 8 variables that could be extracted and visualized:

1. Gender of the patient
2. Source of the opinion (Whose experience is being shared?)
3. Categorized comments (From machine learning models)
4. Mentioned diseases
5. Mentioned symptoms
6. Mentioned treatment
7. Mentioned causes of the disease or patient behaviors
8. Most frequently used words across all opinions

For each variable, participants were asked 3 key questions:

1. Do you consider this variable to be relevant and capable of providing useful information? (Likert scale [31])
2. Which chart type do you find most suitable for representing this variable? (Selection 1 chart from 4 options, including bar charts and pie charts designed by the researchers)
3. Do you have any suggestions or recommendations for improvement? (Open-ended response)

The findings will be utilized to develop charts and dashboards for the NCDs Listener tool, which will subsequently be tested with users in the following section.

Evaluation of dashboard communication effectiveness and selection of generative AI-based summarization approaches

This section presents an experimental study in which target users interact with the results of 6 case studies generated by the NCDs Listener tool. Participants will review both the dashboard visualizations and generative AI-based summaries. The experiment consists of 3 key components: experiment design, question formulation, evaluation process, and data analysis.

Design of the assessment for ability and satisfaction.

This study evaluates user satisfaction and comprehension of the dashboard, as well as the selection of the most effective generative AI-based data summarization method within the NCDs Listener tool. A mixed research design is employed, combining a cross-sectional study and an experimental research approach. The study population is divided into 2 groups:

1. General users, including students from the Department of Statistics, Faculty of Science, Khon Kaen University who have not yet completed the introductory data science course in the academic year 2024.
2. Data scientists, including students from the Department of Statistics, Faculty of Science, Khon Kaen University, who have completed the introductory data science course in the academic year 2024, as well as faculty members from the Department of Statistics in the same academic year.

The study involves a total of 30 volunteers.This cohort is distinct from the 10 volunteers who participated in the initial dashboard design interviews. The 30 participants provided an actual power of 81% (as detailed in Table 3). were selected through a combination of stratified sampling and simple random sampling from the list of students in the Department of Statistics, Faculty of Science, Khon Kaen University. The sample size was determined using the G*Power program, following Cohen’s [32] effect size criteria. The calculation was based on an effect size of 0.80, with a 90% confidence level and an error margin of 0.10. Although increasing sample size generally improves accuracy, the chosen sample size was justified by the actual power of 0.81. This demonstrates that the sample is interpretable and adequately powered for the study’s aims. The results of the sample size calculation are presented in the following Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Parameters for sample size calculation in evaluating dashboard communication and generative AI summaries.

https://doi.org/10.1371/journal.pone.0332484.t003

Participants were exclusively drawn from undergraduate statistics students to ensure sufficient data literacy for meaningful interaction with both dashboard visualizations and generative AI summaries [33]. Restricting the sample to a single department also enabled stratification by prior Data Science coursework, thereby controlling between-subject variability and facilitating direct comparisons across technical proficiency levels [34]. Moreover, the modest cohort size reflects the practical constraints of time, budget, and logistics inherent to early-stage prototype testing [35]. The participant groups for evaluating the dashboard and generative AI’s communication effectiveness were not the same individuals who participated in the prior identification of desired variables and charts.

We obtained written informed consent from all participants before data collection. Sample collection and assessments were conducted between February 27, 2025, and March 10, 2025.

Evaluation process.

Before data collection, we will provide participants with an overview of the research, including its objectives and the results generated by the NCDs Listener web application. This information will be presented through slide presentations and oral explanations. The data collection process consists of the following steps:

A. Assessment of dashboard communication ability

1. Participants will be randomly assigned to view the NCDs Listener tool results presented across 6 dashboards.
2. They will complete 2 assessment tasks designed to evaluate the effectiveness of dashboard communication.
3. Following the assessment, participants will be interviewed to provide additional suggestions or identify areas for improvement.

B. Assessment of generative AI-based opinion summaries

1. Participants will be randomly assigned to review data and summaries generated by generative AI, categorized into 2 summary approaches: more medical focus response, general response
2. Each summary approach consists of 6 types, resulting in a total of 12 summaries. Each participant will review all 12 summaries.
3. Participants will complete 3 evaluation tasks related to generative AI-based summaries.
4. An interview will follow to gather additional suggestions or areas for improvement.

The entire evaluation process, encompassing both the assessment of dashboard communication ability and the assessment of generative AI-based opinion summaries, is illustrated in Fig 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Evaluation strategy illustrates the process for assessing both the communicative effectiveness of the dashboard and the optimal summarization approach for generative AI content.

A probability sample of 30 users was selected and divided into two main assessment paths. The first path involved evaluating six dashboards through two assigned tasks, followed by data analysis using a Likert scale to measure user perceptions, ultimately determining the dashboard’s communication effectiveness. The second path focused on assessing generative AI summaries, where participants reviewed twelve AI-generated summaries and completed three evaluation tasks. Statistical comparisons were conducted using the t-test and Mann–Whitney U test to identify significant differences in performance and preferences. The combined results from both assessments informed the selection of the most suitable summarization approaches and guided improvements in dashboard presentation and AI integration.

https://doi.org/10.1371/journal.pone.0332484.g004

Finally, the collected data will be analyzed using statistical methods, ensuring rigorous evaluation and interpretation of the results.

Data analysis.

The analysis of user needs and satisfaction is primarily conducted using statistical methods to ensure an objective evaluation.

(1) Assessment of dashboard communication ability

The effectiveness of dashboard communication is assessed based on Likert scale ratings, with the following interpretation criteria for the average scores, as shown in Table 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Likert scale interpretation thresholds for user-desired variables and dashboard communication effectiveness.

https://doi.org/10.1371/journal.pone.0332484.t004

These statistical measures will be used to evaluate user feedback and determine the overall effectiveness of the dashboard in conveying relevant information.

(2) Statistical analysis of generative AI-based opinion summaries

To evaluate user satisfaction with generative AI-generated summaries, a Mann–Whitney U test will be conducted using the SPSS statistical software. This test will determine whether there is a significant difference in satisfaction levels between the more medical-focused response and the general response. The results of the hypothesis test will be interpreted based on the P-value. This analysis will provide insights into how different user groups perceive the effectiveness of the 2 summary approaches, supporting further refinement of the NCDs Listener tool.

User-centered dashboard design

The researchers conducted a user needs assessment to guide the development of the dashboard. Participants were asked to select their preferred variables from a predefined set of 8 variables and to identify the most suitable chart type for each selected variable. The findings from this assessment are summarized in the following Table 5.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Preferred variables (mean score 4.21) and chart types for dashboard construction based on user assessment (n=10).

https://doi.org/10.1371/journal.pone.0332484.t005

Based on the results presented in Table 5, the researchers identified variables with an average score exceeding 4.21 as those that users preferred to be displayed on the dashboard. Specifically, the variable “Gender of the patient” will be visualized using a pie chart, while “Mentioned diseases”, “Mentioned symptoms”, “Mentioned treatment”, and “Patient behaviors” will be presented using a bar chart. The remaining variables were categorized by the researchers as hidden variables, which users can access at any time if desired.

Evaluation of dashboard communication effectiveness

The researchers conducted an assessment of satisfaction and understanding regarding the dashboard by having users review the results of the NCDs Listener tool across all 6 cases, as presented in Table 6.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Dashboard communication effectiveness evaluation results for data scientists and the general public (n=30).

https://doi.org/10.1371/journal.pone.0332484.t006

The results presented in Table 6 indicate that both data scientists and general users assessed the dashboard as having good communication ability across all case studies. However, the data scientists group consistently provided lower average scores than the general users group for each evaluation criterion. It can be concluded that the general user group tended to perceive the dashboard as having higher data quantity and quality compared to the data scientist group.

Selection of generative AI-based summarization approaches

In this section, we evaluate the communication capabilities of generative AI to identify the optimal summary approach preferred by both user groups, categorized by each criterion based on the questions used.

The analysis, as presented in Table 7, revealed a statistically significant difference (P<0.001) in the evaluation scores of generative AI summary comprehension. This difference was observed between users who read summaries generated with a medical-focused response and those with a general response across all evaluation criteria: Domain Knowledge, Societal Impact, and Reasoning. Users exposed to the medical-focused response summaries consistently exhibited a significantly higher Mean Rank. This finding suggests that users who read the more medically focused summaries demonstrated a superior understanding of the generative AI summaries developed by the researchers in all evaluated aspects. Consequently, the medical-focused response was adopted for generating summaries within the NCDs Listener tool, effectively addressing users’ specific needs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. User preference for generative AI summarization: Medical-focused vs. general responses (n=30).

https://doi.org/10.1371/journal.pone.0332484.t007

Web application development

In this section, the researcher utilized the results to develop the NCDs Listener tool, ensuring alignment with user requirements. The dashboard was designed to display the variables specified by the user while allowing access to additional variables at any time. Users can modify the data according to their needs, with adjustments dynamically updating the charts and data presented in the table above. Additionally, users can export the table data in CSV format for future use. At the bottom of the dashboard, a summary is generated by generative AI using the more medical-focused response, as illustrated in Fig 5.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. NCDs listener example dashboard interface displaying comprehensive analytical results and data visualization capabilities.

The dashboard features an editable and exportable raw data table at the top, enabling direct data manipulation and export capabilities. The left panel presents a customizable dashboard area with interactive toggle switches displayed in green (active/enabled) and red (inactive/disabled) states for dynamic parameter control. The central visualization area displays statistical charts and graphical representations of the processed NCD data. The bottom of the chart contains a legend and a data display area for a better understanding of the data. At the bottom, an AI-generated summary section provides automated insights from Google Gemini integration, offering intelligent analysis and interpretation of the uploaded content. This integrated interface facilitates comprehensive non-communicable disease data analysis through intuitive visualization, real-time parameter adjustment, and automated analytical summaries for enhanced research workflow efficiency.

https://doi.org/10.1371/journal.pone.0332484.g005

Performance and functional comparison

In this section, we tested our tool with real users, from the data extraction process to image display, using posts containing 30 to 2,000 comments from both Facebook and Reddit. We found that processing on Facebook took an average of 5 minutes, with a maximum of 80 minutes. For Reddit, the average processing time was 1 minute, with a maximum of 30 minutes. This difference in processing time is attributed to the complexity of the websites; Facebook requires the bot to navigate several additional steps for comment access before data collection, whereas Reddit involves only one to two steps. Furthermore, the speed of Thai word segmentation is slower than that of English.

For comparing functional differences between our tool and existing ones, we conducted in-depth research, including free tool trials, reviews, and pricing documentation [42–45]. We observed that most existing tools primarily focus on posts or search terms. While they offer continuous updates, provide demographic data (such as gender, likes, posting time, and hashtags), sentiment analysis, topic analysis, and data insights—functions largely similar to ours—these tools typically require a paid subscription. Our tool highlights methodologies for web data ingestion, utilizing technologies such as Selenium and Beautiful Soup to facilitate the aggregation of post comments without requiring programming. Furthermore, we implemented keyword matching and a BERT Comment classification model for the cleaning and analysis of Non-Communicable Diseases (NCD)-related data. Generative AI technologies are also integrated to enable deeper insights.

Ethical considerations in social media data utilization

Ethical considerations are paramount when using social media data, particularly concerning personal identifiers, comments, and other potentially sensitive information. Legally, Article 9(2)(e) of the General Data Protection Regulation (GDPR) addresses the processing of special categories of personal data that the data owner has publicly made available, which could potentially be used to identify an individual [46].

To protect both users and those whose data are collected, the researchers removed names after cleaning the data. This measure helps prevent the identification of individuals, allowing for valuable research insights from publicly available social media discussions while safeguarding privacy.

Limitations

This research focuses on the development of a social media listening tool for Non-Communicable Diseases (NCDs), designed for the general public and data scientists via a web application. The primary goal was to create a tool that meets user needs and provides understandable insights, prioritizing both speed of development and methodological rigor. The user-centered development process involved multiple phases, including an initial dashboard design phase with 10 participants, followed by a testing phase with 30 participants to evaluate dashboard communication effectiveness and select a generative AI-based summarization approach. This study utilized a relatively small dataset from 30 participants in the evaluation phase, all of whom were statistics students from a single institution. While this homogeneous cohort was beneficial for controlled initial usability testing and yielded statistically significant preliminary findings, with a calculated actual power of 81% (as detailed in Table 3), the dataset is too limited in size and diversity to support broad generalizations. The current results should therefore be interpreted strictly as indicative of initial system performance under controlled conditions, rather than as definitive evidence of its effectiveness across diverse populations.

Currently, our tool supports content in both Thai and English and is designed to extract public comments exclusively from two social media platforms: Reddit and Facebook. The tool demonstrates optimal effectiveness when applied to comment data related to Non-Communicable Diseases (NCDs). Furthermore, as our web application processes comments and chat-like messages, it encounters inherent variations in language, writing style, and terminology. The nature of discussions within a post also influences the results. While this approach allowed us to develop the tool effectively within our limitations, it is important to note that these factors may lead to some variations in the accuracy of the results. Additionally, the input of unreliable data by users can impact the results, even with the assistance of our BERT model designed to filter uninformative content.