Social networks as platforms

According to Carrigan and Jordan [10], the specific characteristics of higher education platforms are as follows:

• Data-generating: In recent years, a substantial body of literature on “big data” has emerged, often characterized by dimensions such as volume, variety, velocity, and veracity [11]. Generated as a byproduct of user interactions with digital platforms, these data accumulate rapidly and are typically processed in real time. Their scale and speed exceed human analytical capacities, making automated methods essential for their organization and analysis.
• Opacity: While data are generated as a byproduct of user activity, platform owners maintain privileged access to them. This creates an asymmetrical structure in which user-facing interfaces coexist with backend systems designed to analyze and influence behavior. As a result, users are primarily treated as objects of prediction, with their actions reduced to data traces feeding real-time analytical processes.
• Lock-in: Platforms tend to retain users through various mechanisms. Their use entails investments of time and attention, creating cumulative sunk costs. At the same time, platforms operate through their own classificatory systems—sets of categories embedded in the interface—that shape how users interpret and respond to interactions, for example by encouraging engagement through metrics such as followers, likes, or shares.

Social network sites, characterized by visibility, accessibility, and widespread use, have become established platforms within the university system, moving from a marginal to a central role in academic communication. Despite this, the relationship between university public communication and the use of social media as institutional channels remains underexplored. Existing research (e.g., [12]) highlights a shift from descriptive accounts of informal use toward the recognition of the need for strategic communication, emphasizing audience targeting and platform-specific dissemination as key challenges.

Social media in public communication

To understand the potential of social media in the public sphere, various scholars have conducted in-depth research. The works of Lovari [13], Lovari and Piredda [14], Ducci [15], Freberg [16], Lovari and Valentini [17], Johann et al., [18], Wigley and Zhang [19], have identified and analyzed the multiple ways of using social network site platforms and their growing importance within communication strategies increasingly integrated into public relations and communication programs. The layered presence of social media has had a significant impact on the institutional sphere, redefining communication and interaction between Public Administration and citizens.

As previously mentioned, social media platforms are particularly adopted and promoted communication channels by universities, aiming for “connectivity” of the academic community, seizing primarily the opportunity for effective promotion of the institution’s identity image and to represent a solid and distinctive reputation [20, 21, 22]. Equally significant aspects of university communication through social media emerge from a summary review of the international literature on the subject [23]: the ability to attract potential students [24], preferences regarding post formats and categories of content published on chosen social media platforms [25], and engagement in relation to factors such as the type of content published or the chosen format for social media posts [22,26,27]; the evaluation of universities’ presence on platforms [28], and the communication resources employed [20].

The breadth and specificity of investigation relevant to our research work make the study by Capriotti and Zeler [23] significant. They aim to analyze the institutional communication of universities on social media by conducting a content analysis of the communication strategies of 70 university entities in the United States, Europe, and Latin America. The study identifies and analyzes three main dimensions (posting, interactivity, and content) of institutional communication on social media in universities, broadening the understanding of social media management. The results reveal different dynamics in the management of institutional communication of universities on social media. Initially, universities were implementing institutional communication on social media characterized by limited dissemination of proprietary content. According to the authors, although with some variations between territories, this reflects trends already emerged in previous studies [20]. Secondly, a clear trend towards the increasing adoption of dialogic resources and interactive strategies emerges. However, universities maintain a predominantly informative approach in their social networks. The priority seems oriented towards institutional content over those related to teaching and research. This raises questions about the authenticity of the dialogue and the effectiveness of online communication with university users.

Public institutional communication

In our study, we will adopt the definition of Ducci and Lovari [9], according to which “public institutional communication” represents a complex, strategic, and integrated activity concerning public goods, rights, and general interest topics by public sector organizations. This activity is based on strategies and tools for information and relationship with citizens, media, and other stakeholders, pursuing impartiality and inclusiveness to promote participation in democratic life and build trust in the collective interest. Public institutional communication, in turn, is divided into different types based on purposes and content (or areas of intervention), articulated in various sectors. The authors specifically identify eight types or sectors of public institutional communication that we will use in the empirical part dedicated to content analysis (see Table 1). It is important to note that this distinction is theoretical and may involve overlaps and hybridizations in the operational activities within the same public institutions (Ducci & Lovari, [9] pp.22–28):

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Description of the areas and contents of Public Institutional Communication.

https://doi.org/10.1371/journal.pone.0351336.t001

Manual and automated post classification

As we will see in more detail in the next section, the data subject of our research were categorized by three researchers (scholars in the field of institutional public communication) and a third categorization was conducted using the davinci-002 model developed by OpenAI (More information on the models available at https://platform.openai.com/docs/models).

During the period between April 2023 and June 2023, the researchers performed a manual categorization of the content (a total of 2487 posts from the two universities), assigning the information in the posts to categories identified according to criteria outlined in the literature [9].

This phase was characterized by a separate process in which the researchers applied their experience in the field of institutional public communication to interpret and assign the information to the appropriate categories. Subsequently, to assess the classification capacity of an automated system, the post texts were subjected to classification using the OpenAI API, employing the davinci-002 model. The combination of the manual approach of the researchers with the classification capabilities of the automatic model may provide interesting insights into the opportunities and criticalities of using artificial languages in social research.

Several studies have explored how LLM-based classification can be embedded into a researcher’s workflow to enhance both speed and rigor. LLMs can automatically classify and filter large text datasets, helping researchers identify relevant content before manual review [29]. LLMs can conduct inductive thematic analysis, generating preliminary topic clusters that guide deeper human interpretation [30]. Researchers can use LLMs in an iterative feedback loop, where initial classifications are manually adjusted, and the refined categories are fed back into the model for improved accuracy [31].

Very few studies have compared classifications generated by artificial intelligence models with human classifications [32,33], our research aims to make a contribution in this direction.

Data and classification task metrics

The data used for the analysis consisted of posts from the official Facebook pages of two Italian universities: the University of Messina and the University of Salerno. Specifically, the dataset included a total of 1,030 posts from the University of Messina and 1,459 posts from the University of Salerno. The posts were collected between March 2020 and March 2023 in compliance with Meta’s data usage policies. The text of the posts was extracted directly from the Facebook pages of the universities by parsing the page content saved through the browser. No data cleaning intervention was conducted on the posts. The texts were analyzed in their original form, including all emojis, links, hashtags, and other elements as they appeared on the Facebook posts. The classification of the posts was based on eight predefined categories. While the raw data cannot be shared due to Meta’s data usage policies, the data produced by the authors — including the annotated datasets — are publicly available on Hugging Face (https://huggingface.co/HuM-HILab/pub_comm_classification).

These categories were described in the preceding section and are outlined in Table 2.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Classification categories.

https://doi.org/10.1371/journal.pone.0351336.t002

The manual classification process involved three researchers (here named: R1, R2, and R3). They are all specialized in Communication Studies and share a common background in the literature on public communication, particularly in the Italian context. Each researcher independently classified the posts without sharing their results with the others, thus avoiding mutual influence among evaluators. Before analyzing the capability of an automatic system to classify the posts, we first evaluated the agreement among individual researchers. For this purpose, to assess the reliability of agreement among a fixed number of human raters, we selected Fleiss’ Kappa ( [34], which is specifically designed for measuring inter-rater reliability in multi-rater settings and for categorical classification tasks. Unlike Cohen’s Kappa, which is limited to the assessment of agreement between two raters, Fleiss’ Kappa allows for the simultaneous evaluation of agreement among more than two evaluators. This measure was chosen because it accounts for agreement occurring by chance and provides a standardized way to compare the consistency of annotations across multiple evaluators.

To calculate Fleiss’ Kappa (, the following formula was used:

where is the observed proportion of agreement and is the proportion of agreement expected by chance.

This metric allows us to quantify inter-rater agreement: a value of 0 indicates agreement equivalent to chance, while a value of 1 indicates complete agreement. Landis and Koch [35] provide interpretations for intermediate values, as shown in Table 3.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Landis and Koch (1977) k value explanation.

https://doi.org/10.1371/journal.pone.0351336.t003

Methodology

Step 1.

The aim of the first step is to investigate the feasibility of using AI-based automated classification in comparison with, or as a complement to, human classification performed by researchers. This step includes two key components:

1. assessing the degree of agreement among the researchers themselves, and
2. evaluating how closely a zero-shot classification model aligns with those human judgments.

The first component is essential, as human classifications are not perfectly consistent—different researchers may apply the same classification schema with subtle variations. Therefore, measuring inter-rater agreement provides a necessary baseline for interpreting the validity of any comparison with AI outputs.

The second component involves introducing a zero-shot classification model, specifically the davinci-002 model accessed via the OpenAI API. This choice was motivated by the model’s high accuracy in text understanding and classification tasks. Its balance between performance and cost makes it suitable for large-scale experimentation. This step aimed to evaluate the reliability of a Large Language Model (LLM) [36] in a zero-shot classification setting [37]. Zero-shot classification allows a model to assign categories to data without having been explicitly trained on labeled examples of those categories.

The full methodological workflow for Step 1 is shown in Fig 1. It includes the manual classification of Facebook posts by three researchers, the measurement of inter-rater agreement, and the comparison of those results with the output of the zero-shot classification performed by the LLM.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Methodological workflow for Step 1: evaluation of the feasibility of zero-shot classification.

The process includes manual classification by researchers, assessment of inter-rater agreement, agreement analysis with a zero-shot approach of a pre-trained model.

https://doi.org/10.1371/journal.pone.0351336.g001

Step 2.

In the second phase of the study, the goal was to improve the accuracy and alignment of automated classification by fine-tuning a language model using human-annotated data. Specifically, a BERT model was fine-tuned on various combinations of labels provided by different researchers, reflecting different interpretations of the classification criteria. These customized models were then used to classify the same set of Facebook posts. Finally, the outputs of the fine-tuned models were compared with the original human classifications to assess their level of agreement and evaluate whether fine-tuning improved consistency with researcher judgments. Fig 2 shows schematically the process.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Methodological workflow for Step 2: improving automated classification through fine-tuning.

The process involves training a BERT model on researcher-provided labels, using the fine-tuned models to classify posts, and evaluating agreement between the model predictions and the original human classifications.

https://doi.org/10.1371/journal.pone.0351336.g002

To enhance the automatic classification and create a model specifically trained with the researchers’ classification data, we decided to apply a Bidirectional Encoder Representations from Transformers (BERT) model pre-trained in Italian (More information about the model is available here: https://huggingface.co/dbmdz/bert-base-italian-cased). This model was fine-tuned using the researchers’ classification results. BERT models are particularly well-suited for text classification tasks for several reasons. Since their introduction [38], and have been proved to be among the most performant Large Language Models (LLMs) in the literature. BERT models can be fine-tuned on task-specific data, enabling adaptation to a specific analytical context and alignment with researcher-defined classification criteria.

Research question1

Despite sharing a common background, the researchers exhibited moderate agreement in their classifications. This observation evidences the inherent subjectivity in human text classification. Below are the detailed Fleiss’ Kappa results, as Fig 3 shows.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Fleiss k values, comparing researchers’ classifications.

R1-R2 combines classification data randomly 50% from R1 and R2, in the same way for R2-R2; R1-R2-R3 combines ⅓ of data from the three researchers.

https://doi.org/10.1371/journal.pone.0351336.g003

These results underscore the variability in agreement among researchers, even within the same domain. While some categories show high levels of agreement, others reveal substantial differences in interpretation. The overall kappa values suggest that while there is a reasonable degree of consensus with R1 and R2, R1-R3 and R1-R2-R3, the couple R2-R3 shows a lower agreement.

This variability points to the challenges inherent in human-made classification and highlights the potential value of developing robust automated systems to support or enhance consistency in classification tasks.

A central challenge in both human and machine-based classification lies in semantic ambiguity. Many social media messages are inherently open to multiple interpretations due to factors such as polysemy, implicit tone, or overlapping communicative functions (e.g., in our categories Communication of Institutional Activities and Normative Communication). This often leads to disagreement among human annotators, even when they are trained experts. In such contexts, classification is not a matter of objectively “correct” labelling, but rather of interpretive judgment.

The same posts were subsequently classified using the OpenAI API, specifically utilizing the davinci-002 model, as described above.

We compared the classification results obtained from OpenAI’s API with those provided by the three researchers. The results are summarized in Fig 4, where Fleiss’ Kappa values are calculated to measure the agreement between the classifications. Although the overall Fleiss’ Kappa values are not particularly high, indicating moderate agreement, the results reveal a certain degree of agreement in classifications across various categories. Some categories show a moderate or substantial level of agreement, even with a zero-shot approach.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Kappa index for overall agreement and for each category.

https://doi.org/10.1371/journal.pone.0351336.g004

This suggests that with further refinement and training, automated classification systems could potentially reach an agreement level that is comparable to that of human researchers, especially in categories where higher Kappa values were observed.

Research question 2

The central question of our research is whether the level of agreement between the classifications produced by the BERT model and those generated by the researchers is comparable to the agreement among the researchers. To analyze the agreement levels in an unbiased manner, we fine-tuned the BERT model with classification samples from individual researchers as well as with combinations of classifications from different researchers.

To train and evaluate the classification models, the dataset was randomly divided using an 80/20 holdout strategy: 80% of the data was used for training, while the remaining 20% was set aside for testing. The split was performed using a fixed random seed to ensure reproducibility and to prevent any data leakage between training and evaluation phases. Each post was assigned exclusively to either the training or test set, ensuring that the model was evaluated only on unseen data.

The training data consisted of manual classifications performed independently by three researchers, each with expertise in communication studies. Separate models were trained using the full set of annotations from each individual researcher, as well as using combined datasets created by merging randomly sampled annotations from multiple researchers (e.g., 50/50 or 1/3 splits). This strategy allowed us to assess not only the effectiveness of individual annotation-based models, but also how combining different human perspectives impacted model performance and generalizability.

Inter-rater disagreement was not resolved through adjudication or consensus; instead, it was treated as an analytical variable. This allowed us to compare AI-generated classifications with human annotations, as well as agreement levels among the human annotators. Table 4 summarizes the various training datasets used, together with training metrics for each model:

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Model performance. F1 Micro measures overall model performance by aggregating true positives, false positives, and false negatives across all classes—giving equal weight to each instance. F1 Macro calculates the F1 score for each class independently and then averages them—treating all classes equally, regardless of their size. F1 Weighted also averages F1 scores across classes, but weights them according to the number of true instances per class—balancing class importance based on their frequency.

https://doi.org/10.1371/journal.pone.0351336.t004