Objectives

This evaluation study has a threefold objective: (1) to evaluate the usability of a demonstrator XR system in providing access to intangible culture heritage experiences for people of different backgrounds, (2) to assess the effects of the demonstrator on its users’ general well-being, and (3) to assess its potential to address societal issues, including ageing societies, cultural tourism, disappearing communities, and immigration and multiculturalism.

Secondary objectives include (a) testing a machine learning (ML) and large language model (LLM)- based application that records and analyses the participants’ voice to determine affect, as a proxy for psychological well-being, and (b) connecting visitors with and improving their knowledge of the four chosen cultural heritage sites.

In terms of addressing societal issues (objective 3), we focus specifically on the issues of how to mitigate the potential negative impacts of cultural tourism, foster inter-generational relationships and belonging within an ageing population, promote the involvement of artists in interpreting and re-inventing intangible cultural heritage for the local populations and visitors alike, foster better communication and understanding between immigrants and local communities, in the context of cultural heritage sharing.

The case studies can be summarised in terms of the PICO framework [28], as shown in Table 1.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Characterisation of studies in terms of the PICO framework.

https://doi.org/10.1371/journal.pone.0336120.t001

This study was registered in the OSF Registries (Registration ID: 9gmpx), on May 28, 2025 [29], where a detailed account of study design, methodology and analysis plan can be found.

Participant characteristics

Participants are healthy adult volunteers living in Portugal (CS1), Scotland (CS2), Finland (CS3) or Greece (CS4) at the time of study and able to give informed consent. For CS1, CS3 and CS4, participants will be over 18 years old at the time of initial consent. For CS2, participants should belong to the older range of population in Scotland, 60 years old or older at the time of initial consent, or, to a younger range, 18-30 years old, at the time of initial consent. Exclusion criteria for participants include suffering from severe hearing or speaking problems—i.e. incompatible with a spoken dialogue interview —, inability to speak fluent Portuguese (CS1), English (CS2), Finnish (CS3) or Greek (CS4), and lack of consent or ability to give it.

Participants will be identified according to age, locality and their involvement with cultural heritage, with support of local stakeholders, namely: Fundação Eugenio de Almeida and Mora Museum of Megalithism (Portugal), Historic Environment Scotland (HES), Koli Culture Society and Nature Centre Ukko (Finland), and Kavala Municipality (Greece). Participants will be informed of the study through a range of platforms including social media, newsletters, email lists, and media calls (e.g. radio), as well as individually. The recruitment period will run from July 2025 to June 2026, with workshops and open activities taking place between summer 2025 and summer 2026.

Sample size

We aim to recruit around 200 participants in total - c.50 participants for each case study - who will be evaluated before and after being exposed to the XR demonstrators, therein conforming two groups for analysis, namely pre- and post- intervention. For well-being assessments, we will compare participants through their self-reports using the Positive and Negative Affect Schedule (PANAS [30]) scale, and a well-being visual analogue scale (VAS). We will also collect physiological measures of stress levels, namely, cortisol and cortisone levels through saliva swabs and physiological signals through wearable biometric devices. For self-reporting and VAS data, we estimate a sample size of 25 participants per group, based on a pre-post mean comparison, assuming a type I error probability and statistical power of 0.80, and aiming for a large effect size d>0.80, according to Cohen’s interpretations of effect sizes [31]. For physiological markers, we calculated the sample size following the analysis of a similar study [3]. In that study, differences in mean cortisol levels () pre- and post-intervention ranged from (morning) to (afternoon), with an overall mean difference of . The standard deviations (s) for the overall means were s_pre = 0.16 and s_pos = 0.07. This indicates a large effect size, . The standard deviations for the afternoon readings (the worst case scenario) are 0.08 and 0.06, pre- and post-intervention respectively, resulting in a smaller effect size of d = 0.98. Based on this, we estimate a sample size between 35 and 46 participants to detect a statistically significant difference at the p < 0.05 level with power of 0.9. Overall, recruiting approximately 50 participants for each case study should provide the power we need for the intended statistical analyses. Power calculations for statistical analyses were performed with R, version 4.3.2 [32], using the pwr package [33].

For machine learning modelling, we aim to recruit the minimal number of participants required for learning algorithms to be able to generalise well-being and emotional factors (stress, anxiety, affect) in their predictions. While there is no clear consensus in the machine learning literature regarding sample size calculations, for relatively simple machine learning algorithms and metrics, such as Euclidean distance and linear discriminant analysis, lower bounds of 1.2 to 1.4 participants per feature (variable) in the model have been proposed, for an expected probability of misclassification at most 50% greater than an asymptotic value of 0.1 [34]. For estimating levels of emotional experience, if we take a previous study that employed the arousal-valence model as a guide [13], assuming a model R = 0.5, a regularisation factor of 0.8, mean arousal and valence scores of 30 (standard deviation of 22) and a reduced features set of 40 features, would require about 300 participants, according to the method proposed by Riley et al. [35]. As this calculation is rather conservative, and as we may be able to augment the data collected in this study with other data sets, if each site recruits 40-50 participants, we should still be able to build effective machine learning models.

Machine learning methods

An effective strategy to address the relatively small size of the data set to be collected in this study is to leverage pre-trained models, which have been trained on large-scale, diverse data sets. These models, often based on architectures such as Transformers [36] (e.g., Whisper [37], BERT [38], and LLMs), capture rich, generalised representations of linguistic, acoustic, and contextual features. In the context of this study, these pre-trained models can be employed in a zero-shot or few-shot manner, directly applying their knowledge to the task or fine-tuning them on the smaller, domain-specific datasets collected within the project. Fine-tuning involves adapting the models to the unique linguistic, acoustic, and cultural characteristics of the INT-ACT data, thus improving their relevance and performance.

Additionally, transfer learning techniques such as domain adaptation and multi-task learning can be applied to make the models more robust to domain shifts and enhance their generalisability across related tasks, such as speech segmentation, diarisation, and sentiment analysis. Few-shot learning paradigms, supported by sophisticated data augmentation techniques like text paraphrasing, audio synthesis, or contrastive data sampling, can further mitigate the effects of limited training data. Finally, leveraging unsupervised or semi-supervised learning approaches, such as self-training or pseudo-labeling, can extract additional value from unlabeled or partially labeled data within the data set. By combining these strategies, we can significantly reduce the dependency on large volumes of labeled data while simultaneously enhancing the accuracy, robustness, and interpretability of the resulting models, ensuring their effectiveness in extracting meaningful insights from complex, interdisciplinary data sources.

Study design

The study involves collecting and analysing recorded speech, well-being ratings, survey answers and saliva samples from participants collected over a two-day period - over three months for the whole evaluation. Participants will complete the study individually. For CS2, an older person (60 or older) will be paired with a younger person (30 or younger). Participants will be randomly assigned to a control group (CG) or an intervention group (IG). The control group will take part in a conventional cultural heritage activity related to each case study’s cultural heritage site, which will be supported by written material and visual resources available to the general public. The intervention group will explore the relevant intangible cultural heritage case study’s site using the INT-ACT XR demonstrator.

The following materials will be employed in the evaluation:

• A VAS, ranging between 0 and 100, to measure nervousness (nervous/calm), happiness (sad/happy), interest (bored/interested), and belonging (connected/isolated), as well as general well-being levels (wellness absent/maximum imaginable wellness). This has been adapted from Johnson et al.’s [39] VAS assessment of happiness, wellness, interestedness, confidence, and optimism in order to understand the effects of museum object handling and art viewing on early dementia and caregiver participants. It also encompasses Grossi et al.’s [3] single VAS of “wellness absent/maximum imaginable wellness” developed to measure the effect of cultural activities on stress reduction. This will allow subjective evaluation of the four pairs of co-related emotions informing the use and impact of the demonstrator— e.g. in fostering attention (bored/interested) and belonging (connected/isolated)—alongside a subjective assessment of general well-being.
• The PANAS scale, which consists of 20 items—i.e. words describing feelings and emotions —, scored in a 5-point Likert scale from "very slightly or not at all" to "extremely". Originally developed by Watson et al. [30], PANAS is a measure of positive and negative affect. It is brief and has been shown to be sensitive to short-term fluctuations, making it suitable for our two-day evaluation study. Besides, there is evidence of convergent validity between the Positive Affect subscale and social activity and diurnal variation in mood, as well as evidence of discriminant validity between the Positive Affect subscale and measures of stress, general distress and depression. The opposite is true for the Negative Affect subscales [40]. As a widely validated psychological scale with these characteristics, including PANAS serves a two-fold purpose. It provides a self-reported baseline in combination with the adapted VAS and contributes to its validation, while complimenting the robustness of the VAS and physiological measurements in evaluating the predictive ability of the machine learning models.
• A System Usability Scale (SUS), adapted to the INT-ACT demonstrator, as a measure of perceived system usability [41,42].

Each case study will be conducted according to the following procedure (Table 2). On Day 1, the activity will start with a familiarisation session (1 hour)—either individually or with an assigned pair (CS2)—to familiarise the participants with the demonstrator (in the case of the IG) or alternative material (CG). Additionally, study participants will be asked to complete the well-being VAS and PANAS, provide a saliva sample, and wear a physiological monitoring device [43] for the remainder of the study. This device will measure participant’s heart rate (HR), heart-rate variability (HRv), respiration, oxygen saturation, blood volume pulse (BVP), electrodermal activity (EDA), galvanic skin response (GSR), movement (acceleration, rotation, cardinal direction), skin temperature (SKT), for affect recognition.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. INT-ACT evaluation study design procedure.

https://doi.org/10.1371/journal.pone.0336120.t002

On Day 2, prior to the activity, participants will be asked to discuss study-relevant topics (sustainable tourism, ageing societies, disappearing communities, and immigration and multiculturalism) according to each CS. These discussions will be audio recorded, using an encrypted audio recorder. A “mood diary”, where participants will be reflecting upon their general mood, will also be audio recorded. This will consist of spontaneous speech in response to the open question ‘how are you feeling?’.

The cultural heritage activity will be supported by the INT-ACT XR demonstrator (IG) or standard site material (CG). The participant (or pair of participants) will be given a single-user tablet to access the XR demonstrator (IG) or printed site materials (CG). Both groups will be provided with audio recorders and clip-on microphones, which they will use to record their impressions of their XR experiences.

Finally, a debriefing session will include a conversation about the visit, its impact on the participants’ well-being, their level of satisfaction with the activity and how aspects of intangible cultural heritage contributed to it, whether the demonstrator helped (or hindered) their engagement with the site, the perceived impact on their well-being, and any other insights relevant to the objectives of the study. This conversation will be audio recorded. Additionally, they will be asked to complete another well-being VAS and PANAS, provide a saliva sample following the above-described procedure, and assess the tool’s usability by completing the SUS.

Data management and storage

Participation in this research study will be based on informed consent. The study will digitally collect the following types of data:

1. Full name/signature and email to consent
2. Age at the time of participation (but not date of birth)
3. Sociodemographic information: sex, gender, ethnicity, location, education, primary language; self-reported rating of mood. This will be fully anonymous data
4. Speech data, as part of the voice recordings of interviews, mood diaries and dialogues around the demonstrator’s use and impact

Name, signature, email, age, sociodemographic information and self-reported rating of mood (VAS and PANAS) will be collected digitally through LimeSurvey, an online survey server hosted at the University of Edinburgh. Audio-recordings will be collected in person by the INT-ACT project’s research teams in each country (Scotland, Portugal, Finland and Greece) using an encrypted recording device.

All participants will be assigned a unique identification code upon entering the study. This number will be used to identify the participant and thus personal details will not be held by members of our study.

Study data will be stored, accessed, and backed up according to a pre-defined common strategy following the INT-ACT project’s Data Management Plan [46]. During the study, data will be stored in encrypted and secure servers in line with the University of Edinburgh’s data protection policy, which abides by the General Data Protection Regulation (GDPR) and the Data Protection Act 2018.

Ethical considerations, declarations

This study has been granted ethical approval by the Edinburgh Medical School Research Ethics Committee (25-EMREC-004, 31 March 2025; and later with approved amendments 25-EMREC-004-A01, 19 May 2025) and complies with advice from its academic sponsor, the Academic and Clinical Central Office for Research and Development (ACCORD), at The University of Edinburgh. There are no expected risks to participants, who are healthy consenting adults informed that this is an non-invasive, exploratory study with no diagnostic aim.

Participants are free to withdraw from all aspects of the study at any point without providing a reason. The project will retain the right of continued use of the data collected up to that point. To safeguard rights, the minimum personally-identifiable information possible will be collected.

Data protection procedures comply with the requirements of the sponsoring institutions’ ethics committees, the EU’s GDPR and the UK’s Data Protection Act, 2018. In compliance with ethic guidelines and open science initiatives, the study was registered with OSF Registries (Registration ID: 9gmpx) [29]. The main ethical consideration for this study relates to data confidentiality, as it involves collection of audio data, deemed to be identifiable. Voice conversion will be employed to anonymise the audio recordings. Science and public communications will only include results on analyses undertaken after pre-processing the voice recordings and well-being survey data, ensuring that audio data or any other identifiable information will never be published or disseminated.

Statistical analysis

• Baseline well-being index responses. Well-being VAS, PANAS and cortisol levels pre- and post-intervention and between CG and IG will be compared through ANCOVA, while correlations between VAS, PANAS and cortisol analyses will be drawn through Pearson correlation tests. Generalised linear mixed models will also be used as appropriate after data exploration.
• ML algorithms will be employed to create models of affect and well-being against a VAS target function, using speech and wearable device data. The predictive accuracy of these models will be assessed in terms of correlation coefficients and root mean square error measures.
• SUS scores will be normalised to a percentile rank for better visualisation and compared among case studies. The statistical inference techniques mentioned above will also be used to approach SUS analyses as appropriate.

Integrative analyses

The study will integrate qualitative analysis with ML and LLMs. Thematic analysis through deductive manual coding will explore themes related to past, present and future, memory, and perspectives pertaining to each case study. Focus will be given to:

• CS1. Potential negative impacts of cultural tourism and alternative solutions through sustainable cultural tourism practice
• CS2. Comparison of improvement in mental well-being for ageing participants and intergenerational aspects
• CS3. Ways in which artists can help to combat the disappearance of rural communities and cultures
• CS4. Comparison of improvement in local-immigrant connection and other aspects of multiculturalism

This will be integrated and complemented with LLMs “theme and sentiment” analyses which can identify both explicit and nuanced themes, analyse sentiment, and provide structured insights from textual data. By generating concise summaries, extracting keywords, and scoring themes based on their relevance, LLMs can significantly enhance the depth and efficiency of the analysis. However, challenges such as bias in the training data of LLMs can influence the accuracy and objectivity of the results. These biases, if not carefully managed, can perpetuate stereotypes or misrepresent certain narratives, potentially impacting the integrity of the findings. Addressing such challenges requires careful evaluation, fine-tuning, and validation of the models to ensure their outputs align with the goals of the study.