https://orcid.org/0000-0002-6374-8644
Figures
Abstract
Virtual Reality (VR) has paved its way into experimental psychology due to its capacity to realistically simulate real-world experiences in a controlled way. Theoretically, this technology opens the possibility to conduct experiments anywhere in the world using consumer hardware (e.g. mobile-VR). This would allow researchers to access large scale, heterogeneous samples and to conduct experiments in the field in cases where social distancing is required – e.g. during the COVID-19 pandemic. Here, we investigate the feasibility of carrying VR experiments in the field using mobile-VR through a stress inductive (public speaking task) and a relaxation (nature) task and contrast them with results in the laboratory (HTC Vive and mobile-VR). The first experiment employed a 2 (device: HTC Vive Pro (HMD) versus Wearality Sky VR smartphone adapter) x 3 (audience: ‘none’, ‘attentive’, ‘inattentive’) between-subjects design. Thirty-four participants took part in the experiment and completed a public speaking task. No significant difference was detected in participants’ sense of presence, cybersickness, or stress levels. In the second experiment, using an inexpensive Google Cardboard smartphone adapter a 3 (between: device setting) x 2 (within: task) mixed-design was employed. Sixty participants joined the experiment, and completed a public speaking and a nature observation task. No significant difference in participants’ sense of presence, cybersickness, perceived stress and relaxation were detected. Taken together, our results provide initial evidence supporting the feasibility and validity of using mobile VR in specific psychological field experiments, such as stress induction and relaxation tasks, conducted in the field. We discuss challenges and concrete recommendations for using VR in field experiments. Future research is needed to evaluate its applicability across a broader range of experimental paradigms.
Citation: Quirós-Ramírez MA, Feineisen A, Streuber S, Reips U-D (2025) Virtual Reality experiments in the field. PLoS ONE 20(4): e0318688. https://doi.org/10.1371/journal.pone.0318688
Editor: Iftikhar Ahmed Khan, University of Lahore - Raiwind Road Campus: The University of Lahore, PAKISTAN
Received: May 14, 2024; Accepted: January 21, 2025; Published: April 8, 2025
Copyright: © 2025 Quirós-Ramírez et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The public speaking VR experience is available at https://osf.io/6y9w4/. The Experimental data are available at https://osf.io/s9bfq/. The Nature task videos are available from YouTube, see the S2 Fig caption for links.
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Behavioral research using Virtual Reality (VR) has expanded exponentially in the last decades [1–3]. VR provides multiple benefits to experimental research, including an increased experimental control and ecological validity [4,5], as well as decreasing experimental costs, increasing portability, and easing the exchange of experimental set-ups between researchers. Research in VR spans a wide variety of psychological fields and topics, from eating disorders [14,15], social anxiety and stress [16], addictions [17], racism [18], all the way to marketing [19]. VR has been widely applied in clinical environments, such as in VR exposure therapy (VRET) for treating fears of heights, flying [20] and social phobias [21]. It has also been utilized for psychological assessment, including body image [22], psychosis [23], and emotions [24], and is considered pivotal for advancing the psychotherapy of eating and weight disorders [25].
There are, however, situations in which it is not possible to access participants for experiments in the laboratory, for example the Covid-19 pandemic and its restrictions. Bringing VR technologies outside of the laboratory would enable scientists to (a) access to populations that would not be accessible due to physical and geographical constraints, (b) generally bring VR experiences to ecologically valid settings, i.e. people’s homes, (c) enable access to a wide and diverse participant pool, and (d) keep conducting studies in circumstances where face-to-face activities are not possible to carry out normally – like the Covid-19 crisis. Even though, there are different resources for running experiments online [6–13], these web options were not made for VR research, thus, they are rarely suitable for carrying out VR experiments.
Is it possible to reliably conduct VR experiments in the field? Here, we investigate whether mobile-VR is a reliable alternative to conducting VR experiments outside of the laboratory. For this purpose we carried out two experiments. Experiment 1 investigates the effect of VR equipment (high-end VR vs. mobile VR) on behavioral and physiological outcomes of a psychological experiment. Experiment 2 investigates the effect of the experimental environment (laboratory vs. field) and device (high-end VR vs. mobile VR) on the behavioral outcomes. The results of these experiments will provide insights into the effectiveness and feasibility of using mobile VR equipment for specific types of psychological experiments conducted in field settings.
Conducting VR experiments outside of the laboratory poses multiple challenges with regards to equipment management and experimental control. The laboratory provides a controlled and safe environment void of possible distractors, such as noises or people entering the room, and typically counts with the physical presence of an experimenter, who monitors the experiment, provides a safety, and serves as a contact point for questions. Another challenge is the availability of VR equipment outside the laboratory. Given the dimensions and costs of the high-end VR equipment typically used in the laboratory, it would be difficult, and perhaps not recommended in most cases, to deliver it to participants outside of the laboratory. To curb these shortcomings, the following experimental decisions were made: (1) an experimenter would be virtually available through a videoconferencing platform to enhance experimental control and (2) using mobile VR with the participants’ own smartphones instead of the laboratory’s high-end VR equipment.
We expect to see similar results in the measures of performance and subjective experience of the participants in the tasks in the laboratory and in the field. However, we expect the laboratory condition with the high-end VR headset to score better in terms of immersion and presence and cause less cybersickness in participants (see the Virtual Reality section for details). If we can determine that (a) virtual environments elicit similar responses in high-end commercial VR equipment and low-cost mobile VR equipment, and that (b) these responses are similar if the experiment is conducted in the laboratory as well as in the field, this would provide promising evidence for the feasibility of using mobile VR in field experiments. Importantly, if none of our results speak against using VR in the field, the combination of low-cost devices and remote support via video calls could expand opportunities for studying behavior in situ under varied conditions.
Virtual reality
Virtual Reality refers to a three-dimensional (3D) computer simulation of an artificial world, also known as virtual environment (VE). This simulation provides a multisensory experience for the user, based on action-perception loops achieved through tracking and display hardware. Based on information obtained from tracking devices placed on the user, such as head position, the computer simulation is updated to provide specific sensorial feedback for the user through different types of displays, such as visual or olfactory displays, giving the user the illusion of being inside this artificial 3D world [26]. Currently, the most widespread VR devices are called head mounted displays (HMD). These can track, at least, the user’s head rotation and can include extra trackers to be placed in different parts of the body, as well as hand controllers for interactivity, while providing visual and auditory feedback to the user.
There is a particular feature that characterizes VR: presence. Presence is a psychological construct that refers to the subjective experience or illusion of ‘being there’ in the virtual environment [26]. A high level of presence leads to a high level of behavioural response realism—users respond similarly in virtual and real environments [27]. Presence can be measured through questionnaires [26]. It is usually utilized as a measure of success and quality of a virtual environment [28].
One downside of VR is the proneness of some users to suffer sickness symptoms similar to those of motion sickness, for example headaches or queasiness. Cybersickness in VR can be caused by multiple reasons, such as a discrepancy or temporal delay between visual and vestibular information [29].
VR research carried out before the 2010s relied exclusively on specialized, very expensive, and bulky technology. The boom of consumer VR around the year 2014 [26] significantly improved accessibility to the technology, enabling a larger number of research laboratories to adopt it. However, there are still multiple drawbacks and constraints preventing the use of VR in a similar manner as psychologists use computers nowadays. Namely, (a) specialized programming knowledge is required to create the VR environments, (b) even though the technology has become less expensive in comparison to its previous versions, a robust VR set-up can be outside of the budget of the average laboratory, (c) a moderate degree of expertise is still necessary to set-up and manage the hardware, ensure user experience, and provide proper guidance.
A plausible solution to some of these constraints may be the use of mobile VR. Mobile VR refers to the use of a consumer-grade smartphone to run VR environments and tasks, together with a special adapter to convert the smartphone into a head mounted display (HMD). There are multiple adapter options, ranging from very sophisticated hardware, like Samsung Gear VR, which visually resembles the high-end HMDs, to the lowest-cost available one: Google Cardboard (Fig 1). Unlike Samsung Gear VR – which is only compatible with a subset of Samsung smartphones – Google Cardboard supports a large variety of smartphones, including the iPhone. It is also possible to build a Cardboard following Google’s specifications with home-owned materials, or to purchase pre-manufactured ones (with bulk prices as low as $0,05 per unit, as of 2024).
Low-cost Virtual Reality smartphone adapter made out of cardboard, which can be purchased as a pre-manufactured viewer or built by users following specifications published by Google.
Up to date, a handful of studies report using VR outside of the laboratory. Steed et al. [30] conducted a Human-Computer Interaction (HCI) experiment aimed at developing design guidelines for future consumer VR applications. The study probes the VR features of presence and embodiment through a simple observation task with avatars. Participants could join if they owned a Google Cardboard or Samsung Gear VR. As the authors mention, their study is uncontrolled, and there is no comparison with laboratory results. Then, Mottelson and Hornæk [31] developed a different Human Computer Interaction (HCI) study to investigate pointing, 3D tracing, and body-illusions in the laboratory (HTC Vive) and in the field (Google Cardboard), with a follow-up task in the laboratory using Google Cardboard. Their results show the potential of empirical HCI studies using VR outside of the laboratory. Finally, Mottelson et al. [32], Radiah et al. [35], Huber and Gajos [33], Saffo et al. [34], and Steed et al. [30] tested different tasks with consumer-owned devices, showcasing the possibility of recruiting VR device owners and replicating previous results from the laboratory. In a recent survey, Ratcliffe et al. [36] outlined the series of possibilities and challenges potentially presented by the use of VR in experiments outside of the lab. However, whether mobile VR is feasible for field experiments in Psychology remains an open question.
Experiment 1
We first investigated the effect of using a low-end mobile VR for experiments as an alternative to the usual high-end VR. For this purpose, we designed and developed a stress inductive, public speaking VE where participants were asked to interact with a virtual crowd by giving an impromptu speech, inspired by existing stress induction paradigms in VR (e.g. the VR Trier Social Stress Test by Zimmer et al. [37]).
Method
Materials.
Virtual Environment. The VE for Experiment 1 is a computer-generated classroom, where the participant acts as a speaker, standing behind a lectern (see Fig 2). Participants do not see their bodies in the VE. The speech topic is “Evolution”. The virtual classroom contains, on top of the lectern, a note with three written key points for the speech that reads: What is evolution?, Which kinds of biological factors exist (e.g., natural selection, mutation)?, and What theories of evolution (e.g., Lamarck, Darwin) and which other perspectives (like creationism) are known?
The participant stands in front of the lectern, which has the talking points for the impromptu speech.
The room has moving ceiling ventilators, which make some noise, an emergency exit behind the speaker, and three rows of seats in front of the speaker. Additionally, there are three single seats placed very close to the lectern.
There is a virtual audience seated in front of the participants that can be either ‘attentive’ (follows the participant’s head with their gaze) or ‘inattentive’ (purposely looks away from the participant’s head). We included a control condition, ‘none’, where the classroom is empty.
Apparatus. As a high-end commercial VR device, we utilized the HTC Vive Pro HMD (Fig 3A) powered by a VR ready PC (VR One 6RE, msi, Taipei Taiwan; CPU: Intel® Core™ i7 processor, 6th generation; Graphics card: GeForce® GTX 1070 with 8 GB GDDR5). The Vive Pro has a resolution of 1440 x 1600 pixels per eye, a refresh rate of 90 Hz, and a 110° FOV. The sound was provided by Hi-Res Certified™ on-ear headphones, which are part of the HMD. Two base stations were placed in the lab to provide a tracking area of approximately 3 x 2 m for the participants.
(A) HTC Vive Pro headset, (B) Wearality Sky, smartphone VR adapter.
As a low-end mobile VR, we utilized the Wearability Sky, a VR smartphone adapter with double Fresnel lenses and 150° FOV (Fig 3B). The VE was presented on a smartphone (ONEPLUS 3, OnePlus, Shenzhen People’s Republic of China) with 6 GB RAM, a resolution of around 540 x 960 pixels per eye, 60 Hz refresh rate and a display size of 5.5’’. Sound was provided with on-ear headphones (E45BT, JBL, Los Angeles United States).
Design.
The experiment was conducted in a 2 (device) x 3 (audience) between-subjects design. The device levels were HTC Vive Pro (HMD) and Wearality Sky (VR smartphone adapter), and the audience levels were ‘none’, ‘attentive’, ‘inattentive’. Participants were randomly assigned to one of the different experimental conditions. Questionnaires were provided in German using paper-pencil questionnaires.
Measures.
Visual Analogue Scale (VAS). Self-reported stress was measured by using the Visual Analogue Scale [6] in a pre-task (from here on: VAS-A) and a post-task (from here on: VAS-B) version. In the VAS-A, participants had to rate two statements: “I feel stressed." and “I feel strained." on a line between “do not agree” and “agree” (0-100, no numbers visible to participants). In the VAS-B, three items were included to evaluate the source of stress, in addition to the VAS-A statements: “Talking was particularly stressful.", “The (in-) attentiveness of the audience was particularly stressful.” and “To be immersed in a virtual environment was particularly stressful.” Ratings were collected at four time points: at the beginning (, baseline; VAS-A), after the task instructions (
, instructions; VAS-A), immediately after the task (
, post-task; VAS-B), and at the end (
, recovery; VAS-B).
Heart Rate (HR). Heart rate (HR), a physiological stress marker, was continuously measured throughout the experiment using a strap-based device (Polar OH1, Polar Electro, Büttelborn, Germany) worn on the left forearm. For the evaluation and comparison, the HR measurement was subdivided into eight distinct intervals for evaluation and comparison, with the arithmetic mean, minimum, and maximum HR calculated for each interval. The HR intervals are shown in Table 1 and in Fig 4.
HR: heart rate, VAS: Visual Analogue Scale.
iGroup Presence Questionnaire (IPQ). The sense of presence was measured using the German version of the iGroup Presence Questionnaire (IPQ) [38]. It is comprised of 13 items assessing three subscales, (a) Spatial Presence, the sense of being physically present in the VE, (b) Involvement, the dedicated attention to the VE as well as the experienced involvement, and (c) Experienced Realism, the subjective experience of realism in the VE. Additionally, IPQ includes one item to assess the overall Sense of Being There in the VE. Each item is rated on a 7-point Likert scale with varying anchors, where higher values indicate a greater sense of presence. The IPQ has demonstrated good internal consistencies, with Cronbach’s α ranging from .85 to 0.87 [38].
Cybersickness. Cybersickness was evaluated using a simplified symptom checklist adapted from the Simulator Sickness Questionnaire (SSQ) by Lane and Kennedy [39]. The checklist included five dichotomous (“yes”/“no”) items for symptoms commonly associated with VR: vertigo, nausea, headache, general discomfort, and fatigue. These symptoms were drawn from the three dimensions of the SSQ (Nausea, Oculomotor, and Disorientation). Participants could also report additional symptoms if experienced. If one or more symptoms were reported, participants answered two follow-up questions to attribute the symptoms to either performing the task or being immersed in VR. Cybersickness was considered present if symptoms were attributed to VR immersion. This simplified assessment was chosen to reduce participant burden and was deemed sufficient given the limited movement and interaction in the VEs, which were designed to minimize temporal delay. As a result, the overall risk of severe cybersickness was expected to be low.
Previous experience with VR. Participants’ previous experience with VR was recorded by asking participants to rate how often they have used VR in the past, selecting between the five possible options: “never", “1 to 5 times", “6 to 10 times", “11 to 15 times", “more than 15 times."
Procedure.
The experiment was conducted in accordance with the ethical standards of the University of Konstanz and with the Helsinki Declaration. The Ethics Committee (institutional review board, IRB) of the University of Konstanz issued a waiver for our experiments (RefNo: IRB24KN05-008/w). All participants gave written informed consent before taking part in the study. Participants were rewarded for their participation and they were explicitly made aware that they were allowed to withdraw from the study at any point in time, without losing their reward nor affecting the study.
Participants arrived at the laboratory and were given information about the experiment (risks and benefits, data acquisition, etc.). They were requested to give written informed consent in order to take part in the experiment. For HR measurement, the strap was placed on the participant’s left arm and instructions on how to complete the questionnaires were given. HR recording was started (= 0 min), and the predetermined schedule began with the baseline rating (t0/: VAS-A). Then (t1), the participants received instructions for the task (see Supporting information for the detailed instructions).
Right after, participants were requested to complete the questionnaires (: VAS-A). Then, the virtual session (t3) started. Participants entered the VE and had the chance to explore it briefly, for 90 seconds. This was the first time the participants learned about the topic of the speech, by reading the key points on the note on top of the lectern.
One and a half minutes after the immersion, a tone signalized the participants to begin their speech (t4) and after five minutes, a second tone chimed, signaling the end of the task. Participants removed the headset and immediately after this, the next rating was completed (t5/: VAS-B). During a recovery period, further questionnaires were provided (t6: IPQ, cybersickness, VR experience, demographics). Last, the stress rating was filled out once again (
: VAS-B). Afterwards, the HR band was removed. In a debriefing, the participants were informed that the objective of the procedure was to elicit a stress response. The chronological procedure and HR intervals are shown in Fig 4.
Participants.
Thirty-four participants were recruited from the University of Konstanz and the Konstanz University of Applied Sciences through the participant management platform SONA Systems® Uni Konstanz. The recruitment took place in February 2020. Each participant was randomly assigned to an experimental condition. There were no significant differences between the conditions for age and previous experience with VR.
Four of the participants were excluded post-hoc due to technical issues during their experimental session (n = 3) or disregard of the instructions (n = 1). This left a sample of 30 participants (21 females and 8 males), mean age 21.5 ± 2.4. All participants were informed about the nature of the experiment and were free to withdraw from it at any time. Participants were all German-speakers, had no neurological or psychiatric diagnosis, and wore no eyeglasses on the experimental session day (contact lenses were allowed).
Results
Device comparison.
Presence, the experience of being actually in a Virtual Environment has been demonstrated to be key for behavioral realism [27]. Thus, a requisite for successfully replicating VR experiments designed for high-end devices in mobile VR is the comparability of experienced presence between devices. On the other hand, it is important to examine whether mobile VR experiments could elicit more cybersickness symptoms than high-end VR devices. This could be problematic in terms of participant dropout rates and behavioral outcomes. Therefore, we evaluated whether there was a difference in sense of presence or cybersickness from the public speaking experience depending on the device.
Presence. The sense of presence, measured using the IPQ, was analyzed by evaluating each of the IPQ subscales (realism, spatial presence, involvement) independently. Considering that previous VR experience could have an effect on the sense of presence, two-tailed Spearman correlations between each IPQ subscale and VR experience (“never", “1 to 5 times", “6 to 10 times", “11 to 15 times", “more than 15 times.") were carried out. There was a significant correlation detected for experienced realism, r(29) = −0.41,p = .028, but not for the other subscales (Spatial presence: r(29) = −0.11, p = .588; Involvement: r(29) = −0.17, p = .377) nor general presence, r(29) = −0.20, p = .289. Taking this correlation into account, we then conducted a ONEWAY ANOVA each with the presence subscales and, because of the detected correlation, experienced realism as dependent variables and the device as factor. No significant difference was detected between devices (general presence: F(1, 27) = 1.19, p = .286; spatial presence: F(1, 27) = 0.95, p = .340; involvement: F(1, 27) = 3.62, p = .068; experienced realism: F(1, 17.55) = 0.67, p = .423). Mean presence values can be seen in Table 2.
Cybersickness. Approximately 17% of all the participants experienced cybersickness symptoms due to VR. To examine device differences in cybersickness, a Fisher’s exact test was calculated for a 2 (yes/no occurrence of symptoms) x 2 (device) crosstab. No significant difference for the devices were found, χ2(1, N = 30) = 2.68, p = .157.
Stress response.
To evaluate the physiological stress response from participants’ HR, an 8 (time: eight repeated baseline adjusted HR measures) x 2 (device) x 3 (audience) mixed factorial ANOVA was conducted to compare HR levels. The course of the mean HR can be observed in Fig 5 and separate for factor device in Fig 6. There was a significant main effect of time, F(2.17, 43.33) = 89.29, p < .001, = .817, f = 2.11. Bonferroni-adjusted posthoc analysis showed a significant difference (p<0.001) of t4tot0 (mean difference = 22.1, p<.001) indicating the HR response was due to the task. The HR score in t4 (M = 98.1 bpm, SD = 17.6) was the highest HR score, significantly different from the other time bins. There was no significant difference between t7 and t0 (mean difference = −1.94, p = .222) indicating a recovery of the HR back to baseline values. No statistical main effect was found for device, F(1, 20) = 0.05, p = .831 nor for audience, F(2, 20) = 0.14, p = .873. No significant interactions were detected (Time*device: F(2.17, 43.33) = 1.76, p = .182, Time*audience: F(4.33, 43.33) = 0.56, p = .706, Time*device*audience: F(4.33, 43.33) = 1.05, p = .395).
Course of the mean HR in beats per minute (bpm) per experimental interval ().
Course of the mean HR in beats per minute (bpm) per experimental interval () and factor device.
We evaluated the subjective post-task VAS rating regarding whether participants found the public speaking task stressful. A one-tailed one-sample t-test, with a reference value of 50 (0 = do not agree, 100 = agree) was conducted. The mean rating (M = 58.8, SD = 26.3) was significantly higher than the reference value, t(29) = 1.83, p = .039, r = .15.
Discussion
The results show that in both the high-end VR device and the low-end mobile VR participants experienced physiological and subjective stress. Given that there was no significant difference detected regarding the type of audience in the virtual environment, following studies will be done utilizing the ‘attentive’ audience, where follows the participant with their gaze. Under the current conditions, no differences in presence or cybersickness were observed between devices. These findings provide preliminary evidence supporting the potential feasibility of using mobile VR for psychological experiments focused on specific tasks, such as stress induction and relaxation.
The next step investigating the use of mobile VR for field studies is to study the effect using mobile VR in different settings: in the laboratory, as it was done in Experiment 1, or in the field.
Experiment 2
In the second experiment, we set out to investigate the effect of setting (laboratory vs. field) on the VR experiments. Further, we tested two different tasks, including the VR public speaking task used in Experiment 1. To select a second VE that was meaningfully different from the public speaking VE, we focused on three dimensions: interaction level between the environment and the participant, the expected effect in the participant, and the medium of the VE (i.e. whether it is computer generated or a video). Because the public speaking task is interactive, stress inductive and computer generated (CG), we decided to select a VE that is passive, inductive of mood improvement and is a 360 degrees video instead of CG. Given that previous studies have demonstrated that nature in VR is conductive of relaxation and mood improvement, in a similar manner than nature in real life [40], we opted for a mood improving nature environment as a second task.
Method
Materials.
Virtual environments. We utilized the public speaking task described in Experiment 1 (using only the ‘attentive’ audience). As a second virtual environment for comparison, we utilized a publicly available 360° video of a natural setting. A still shot of the video can be seen in Fig 7.
According to Wang et al. [41], different types of video forest environments can have different effects on physiological stress and subjective mood states. After evaluating a set of 360° videos in a preliminary study (details in Supporting information), the video of a forest with a stream was selected (Meditation & Relaxation – Music Channel, 2019). The video consists of a forest with conifers, a flowing stream and includes some evidence of human presence in form of a bridge. The scene is accompanied by sounds of birds and water.
Apparatus. As a high-end commercial VR device, we utilized the HTC Vive Pro, as specified in Experiment 1. As low-end commercial VR, we utilized a previously manufactured Google Cardboard with a 90° FOV (Fig 1). Participants used their own compatible smartphone (limited in this case to Android OS) in both the laboratory and field conditions. Sound was provided by the participant’s own headphones. Each Google Cardboard was assembled by the participant. After the experiment, participants were able to keep it or dispose of it at their convenience.
Design.
Experiment 2 was conducted in a 3 (between: device setting) x 2 (within: task) mixed design. The device setting comprised the levels HTC Vive Pro in the laboratory (vive-lab), Google Cardboard in the laboratory (cardboard-lab), and Google Cardboard in the field (cardboard-field). Each participant completed the public speaking and nature tasks in a randomized order. The study was conducted in German. Participants received all the information, instructions, and questionnaires on a computer through WEXTOR® (https://wextor.eu [13,42]). Contact between the participants and the experimenter during the experimental phase was limited to interactions via the online video platform Zoom, regardless of the condition. In the laboratory conditions, the experimenter was in a separate room from the participant.
Tasks.
Public speaking Task. The public speaking VE from Experiment 1 was utilized. Some minor adjustments in the procedure were made: Participants were informed about the speech topic (‘Evolution’) together with the task instructions before entering the virtual environment, and were given 2 min time to prepare before their speech. In the cardboard conditions, the environment was run from the VR-mobile app that participants downloaded on their smartphones. In the HTC Vive Pro condition, the experience ran from the PC.
Nature Task. Participants were instructed to sit in a comfortable position and were informed that they would experience a nature environment. They were immersed in the VR, looked at the 360° nature video for 5 min, after which an alarm signalized the end. The video was played from YouTube in either the Virtual Desktop app for the HTC Vive Pro or the YouTube player in the participant’s smartphone.
Measures.
Visual Analogue Scale (VAS). The VAS was used to repeatedly evaluate stress perception in the public speaking task. Participants were asked to rate the statement “I feel stressed.” or “During the task I felt stressed.” (VAS-B), on a line between “not at all” and “very much” (0-100, numbers not visible to the participant). The rating occurred at three different times during the experiment: at the beginning (t0, baseline), immediately after the task (t1, post-task, VAS-B), and after a recovery period (t2, recovery). Furthermore, for both, the public speaking and the nature task, VAS statements were used as control questions after the tasks (t1, VAS-control). For the public speaking task, the participant rated their satisfaction with their performance during in the speech they gave. For the nature task, the participant rated how much they liked the environment.
Positive Affect Negative Affect Schedule (PANAS). The mood state before and after the nature task was measured using the Positive Affect Negative Affect Schedule (PANAS) originally developed by Watson et al. [43] . It consists of 20 adjectives, ten per primary mood dimension (positive or negative). Participants are asked to rate these items referring to their mood at the present moment. The intensity of feeling is graded between 1 (= very slightly or not at all) and 5 (= extremely) for each of the adjectives. The German version of the PANAS has demonstrated good internal consistency, with Cronbach’s α = .85 for Positive Affect (PA) and α = .86 for Negative Affect (NA) [44]. PANAS is a well-validated measure, frequently used in experimental contexts, and its application here ensures reliable assessment of mood changes induced by VR tasks.
System Usability Scale (SUS). The System Usability Scale (SUS) by Brooke [45] was used to get an impression of the subjectively perceived handling of the devices and experiences. The questionnaire consists of 10 statements about, for example, the complexity of the system (“I found the system unnecessarily complex") and the requirement of prior knowledge for successful handling (“I needed to learn a lot of things before I could get going with this system"). The items are rated on a 5-point Likert scale with values between 0 (= strongly disagree) and 4 (= strongly agree). Previous research by Bangor et al. [46] demonstrated good internal consistency for the SUS, with Cronbach’s α = .91. Furthermore, the SUS has been shown to correlate highly (r = 0.91) with the Usability Metric for User Experience (UMUX) [47], which itself closely aligns with ISO 9241-11 usability standards, showing high sensitivity for slightly differences between systems. Given its robust psychometric properties and applicability to technology evaluations, the SUS was deemed appropriate for assessing the usability of both high-end and low-cost VR setups in this study.
iGroup Presence Questionnaire (IPQ) and Cybersickness were utilized as in Experiment 1.
Procedure.
As in Experiment 1, Experiment 2 was conducted in accordance with the ethical standards of the University of Konstanz and with the Helsinki Declaration. Informed consent was obtained from all patients who took part in the study. Participants were rewarded for their participation and they were explicitly made aware that they were allowed to withdraw from the study at any point in time, without losing their reward nor affecting the study.
For simplicity purposes, from here on we will refer to the public speaking task and its ratings as Session A and to the nature task and its ratings as Session B. As mentioned before, the order of the sessions was randomized per participant. The experiment started for both lab conditions with a quick personal welcome (ca. 2 min) to show the computer where all information would be presented. Participants first received the study information, gave their consent to take part in the experiment, and filled out demographic questions. After that, they were informed about the device they were going to use. Participants from both cardboard conditions were instructed to (a) download the VE using a link provided, and (b) build their cardboard from a package they received (which was shipped to participants from the cardboard-field condition in advance). A self-made video tutorial explaining how to assemble the cardboard was provided. Every participant received a guide with pictures on how to use and get immersed with their respective device.
Session A began with the baseline VAS rating on the subjectively perceived stress (). Afterwards, the participants were instructed on the public speaking task. If there were no questions, after the 2-min preparation time, they started the experience and gave their improvised 5-min speech. They would then remove their headset and return to the computer for further instructions. Participants then rated one more time their subjective stress perception as well as their perceived task performance (
), and filled out questionnaires with regard to their experience with VR (QA: IPQ, sickness questions, SUS). To provide a similar recovery phase for every participant, after filling out the questionnaires, participants were asked to write a story about a topic of their own choice, for example the weather, an experience, or an invented storyline; the story should be exclusively for their own pleasure. They had 5 min to write this story. Lastly, participants completed the stress rating one last time (
) and were informed that the experimental session had finished.
Session B began with a baseline rating of the PANAS (). Participants were instructed to sit down and simply experience the virtual environment. They started the experience and watched the nature video for 5 min. A tone signalized the end; participants took off their headset and returned to the computer for the second rating of the PANAS (
). They also rated on a VAS (VAS-control) to what extent they liked the virtual environment. As in Session A, participants completed questionnaires (QB: IPQ, sickness questions, SUS), followed by writing a story for 5 min and finished the session with the last rating (
).
After these two sessions, participants filled out a question about their previous experience with VR. They received a debriefing as well as their allowance. For an overview about the procedure see S1 Fig.
Participants.
Sixty participants (39 females and 21 males, mean age 22.7 ± 2.7) were recruited from the University of Konstanz through the participant management platform SONA Systems® Uni Konstanz and the German National Research and Education Network (DFN) scheduler. The recruitment period took place between January and April 2021. Each participant was randomly assigned to an experimental condition. The nature task happened first for 31 participants. Due to technical problems one participant could not complete the public speaking task, thus their data was excluded from the analysis of this task (N = 59). There were no significant differences between the conditions for age, sex, previous VR experience, and the control ratings. Participants were all German-speakers, reported no neurological or psychiatric diagnosis, and wore no eyeglasses on the experimental session day (contact lenses were allowed).
Results
Mixed analyses of variance (ANOVA) for main effects over time and planned comparisons for the interaction analyses of specific interest were conducted. From here on, the terms t0 to t2 are used to refer to VAS or PANAS ratings, for the public speaking task and the nature task respectively. To examine the time x condition interaction, the difference scores between the times were calculated ( =
,
=
). The baseline-corrected t1 (t1 - t0) and Δt1 (t1 - t0) are identical.
Presence.
Sense of presence was tested separately for both tasks. One-way ANOVAs were calculated with the presence subscales as dependent variables and the condition as group factor for both nature and speech tasks to examine the planned comparisons. However, it was first examined whether there is a relation between the previous VR experience and presence. Thus, a two-tailed Spearman correlation with previous experience was conducted for each IPQ subscale per task.
Public speaking task.
A mixed ANOVA was calculated with the VAS stress rating as the repeatedly measured dependent variable, time () as within-subjects, and condition as between-subjects, factor. The course of perceived stress per condition can be observed in Fig 8. There was a significant main effect of time, F(2, 112) = 49.54, p < .001,
= .469. Pairwise comparisons with a Bonferroni correction showed a significant difference between
(p < .001) and
(p < .001). No significant interaction between time and condition was detected, F(4, 112) = 1.91, p = .114.
Planned comparisons of the Δ-scores were then analyzed for stepwise interactions. First, a comparison between both lab conditions was conducted. For (change post-task in comparison with baseline) no significant difference was detected, t(56) = −1.31, p = .195. However, there was a difference for
(recovery), t(56) = 2.05, p = .045. In the following step, both cardboard conditions (lab and field) were compared. No significant difference was detected for
, t(56) = −1.07, p = .289 nor for
, t(56) = −0.08, p = .935. The results provide evidence that mobile VR in the field is conducive of stress induction, perhaps even more than in the lab conditions.
Presence. In the public speaking task, the Spearman correlations showed no significance for the presence and VR experience in this task (general presence: r(59) = 0.141, p = 0.144; involvement: r(59) = 0.089, p = 0.250; experienced realism: r(59) = −0.029, p = 0.412; spatial presence: r(59) = 0.076, p = 0.285). The planned comparison between both lab conditions and then between cardboard-lab and cardboard-field showed no significant differences between the presence scores (see Table 3 for details). Mean presence ratings per subscale are shown in Fig 9.
Nature task.
Negative affect. There was a significant main effect of time, F(1.44, 82.36) = 13.58, p < .001, = .192. Bonferroni-adjusted pairwise comparisons revealed significant differences between
(p < .001) and
(p = .001). The negative affect decreased by 2.2 (SD = 3.0) from the baseline to t1 in the vive-lab condition, by 0.6 (SD = 2.3) in the cardboard-lab, and by 2.6 (SD = 4.3) in the cardboard-field condition. No overall significant interaction between time and condition was detected, F(2.89, 82.36) = 2.00, p = .123. The course of perceived negative affect per condition is shown in Fig 10.
The planned comparison between the lab conditions showed no significant differences for , t(57) = −1.58, p = .119, nor for
, t(57) = 1.52, p = .133. This was also the case for the comparisons between the cardboard settings for both Δt1, t(57) = 1.92, p = .060, and Δt2, t(57) = 0.40, p = .690.
Positive affect. No significant main effect of time was detected, F(2, 114) = 1.95, p = .148. On average, the positive affect decreased after the task, there was, however, no significant differences between t1 and t0 (p = .144), t1 and t2 (p > .999), or t2 and t0 (p = .792). There was a significant time x condition interaction (F(4, 114) = 3.01, p = .021, = .095). The course of perceived positive affect per condition is shown in Fig 11.
Planned comparisons did not reveal any significant differences for Δt1 between lab conditions, t(57) = 0.94, p = .351 or cardboard-lab and cardboard-field conditions, t(57) = −1.94, p = .057. There were, however, differences for Δt2 between lab conditions, t(57) = −3.14, p = .003 and cardboard conditions, t(57) 2.07, p = .043.
Presence. In the nature task, the Spearman correlations showed no significant relations for the presence and VR experience in this task as well (general presence: r(60) = −0.080, p = 0.272; involvement: r(60) = −0.215, p = 0.250; experienced realism: r(60) = −0.124, p = 0.173; spatial presence: r(60) = 0.003, p = 0.491). The planned comparison between both lab conditions and then between cardboard-lab and cardboard-field showed no significant differences between the presence scores (see Table 3 for details). Mean presence ratings per subscale are shown in Fig 12.
Cybersickness.
From the 59 participants, the cybersickness questions for both tasks (public speaking and nature) were answered completely by 53 participants. Thirteen participants out of these 53 overall valid cases reported sickness symptoms (six of them in both tasks). In the public speaking task, 11 participants reported an average of 1.6 (SD = 0.8) sickness symptoms; most of them experienced vertigo (n = 8). A Fisher’s exact test for the 2 x 3 crosstab containing the dichotomic variable “cybersickness occurred” (“yes" or “no"), and the different conditions, showed no significant difference of sickness occurring between the device settings χ2(2, n = 55) = 1.45, p = .584. In the nature task, an average of 1.8 (SD = 0.6) symptoms was reported by 11 participants. Most of them experienced vertigo (n = 8) and/or fatigue (n = 6). The Fisher’s exact test revealed no significant difference between the devices, χ2(2, n = 57) = 0.36, p = .917. The results suggest the comparable experience of cybersickness throughout the different device settings.
Usability.
The SUS results were examined descriptively. Mean scores for the usability were in general 81.8% (SD = 13.2), 82.6% (SD = 12.5) for the nature task and 80.5% (SD = 16.7) for the speech task (detailed mean scores and errors are shown in Table 4). The cardboard-lab condition received the lowest usability scores for both tasks. One out of 60 participants in the nature task (cardboard-field condition) and three out of 59 participants in the speech task (cardboard-lab condition) reported usability values below 50%, which is the threshold for non-acceptable usability scores. Participants’ individual SUS scores per task and condition can be seen in Fig 13.
General discussion
Our findings indicate that VR experiments conducted in the field can elicit results comparable to those conducted in the lab under defined experimental conditions, supporting the feasibility of VR psychological field experiments. As expected, we found comparable results in both of our experimental tasks when carried out in the lab and in the field. Participants’ performance was overall comparable throughout the different conditions, as well as their subjective experience, as demonstrated by the usability results. In the particular case of stress induction, the results hint to a higher capacity of mobile VR in the field to induce stress in participants, however, further studies are necessary.
Unlike our original expectations, the laboratory condition did not have better presence outcomes than the field condition, regardless of potential distractors happening in the field condition and the difference in processing power and field of view between devices. One reason for this could be the emotional experience undergone by the participants. Previous research [22,48] has shown a relationship between emotion inductive VR experiences and enhanced presence. Thus, our results suggest that, in the context of psychological experiments, VR experience design and selection may be more relevant than experimental location and device selection.
Participants in the mobile VR conditions did not experience more cybersickness than with the HTC Vive Pro, as we had hypothesized. Overall, only 20% of the participants experienced any cybersickness symptoms. In comparison to previous studies [49], which report cybersickness in more than half of the participants, the occurrence in our study is rather low. This could be due to the lack of locomotion or navigation in our experimental tasks. Cybersickness often occurs when seen and experimented motions differ (sensory conflict) during the VR experience [50].
The widespread availability and use of smartphones, combined with inexpensive cardboard VR devices, offers the potential to broaden access to psychological research by enabling remote participation and expanding the reach of studies beyond traditional laboratory settings. Taken together, our results provide preliminary evidence that VR can be a viable method for conducting psychological studies in field settings within specific experimental paradigms.
General benefits of field VR studies
Beyond the capacity of using VR as a research tool outside of the laboratory, there are other benefits that come with VR field studies. First, mobile VR increases generalizability, given that each participant uses their own equipment, i.e. their own mobile phone. Avoiding limiting participants to a specific set of devices also decreases potential sampling effects [51]. In our study, we did not limit participation to a specific smartphone maker nor provided a specific Android mobile phone. Smartphones running the iOS can also utilize Google Cardboard.
Using mobile VR as the avenue for VR studies can potentially reduce self-selection effects. Unlike the alternative option of recruiting VR headset owners, cardboards can be shipped to a wider variety of participants, and these, at the same time, do not need to own a VR device or be knowledgeable about this technology.
Finally, the use of mobile VR can help increase the participant pool and reduce sample homogeneity, given that a cardboard can be shipped virtually anywhere, increasing the potential of recruiting diverse participant samples in different environments.
Low-cost mobile VR for laboratory studies.
Our results suggest that low-cost Cardboard devices can perform comparably to high-end VR systems on specific measures, such as subjective stress, demonstrating their potential as an accessible option for some VR-based psychological experiments. In our experiments, presence, a key element of VR experience, was experienced to a similar degree with the mobile VR. This finding is promising not just for VR field studies, but also for low-cost VR laboratory studies. Often, the high cost of a full VR set-up can make it difficult for laboratories to employ VR as a research tool. As seen in our results, it is even possible to obtain available materials online and utilize them for experimental purposes with the mobile VR, avoiding a second bottleneck for VR studies, which is the development of a 3D environment. Thus, it is possible to benefit from the special characteristics of VR without incurring in high costs.
Considerations for running field VR studies
Based on our experience with VR experiments in the field, there are initial recommendations we can make for researchers who attempt to use mobile VR for field studies. It is useful to note that when combining field VR data with data from sensors on the smartphones used one has to consider the variations in these devices [52].
Availability through videoconference.
For comparability purposes between conditions, we carried all our experiments through videoconference, where we delivered the different electronic forms that contained instructions and questionnaires to the participants. During the experimental tasks, the participant was invited to turn off sound and camera for privacy purposes, increasing the chances of reducing demand effects. Even though the interaction with the participants was minimal, the experimenter was available for the participant throughout the experiment in case of questions or problems. This facilitated the flow of the experiments in the field and made it possible in our case to have a close-to-zero dropout rate throughout conditions (only one participant’s data had to be excluded due to technical issues). Indeed, the final goal of VR field experiments would be to have a fully unsupervised set-up, it is recommended to utilize the videoconference format for initial trials or in experiments where it is possible and meaningful to have this blended set-up.
Shipping the cardboards.
Unassembled cardboards were shipped to a postal address given by the participant at the time of registration for the experiment (which happened online). In a pilot study, pen and pencil questionnaires were also sent to participants together with a prepaid return envelope to ship the answers back to the experimenters. In cases in which it is not possible for the participants to access online forms, it is also possible to utilize this format. Given that a stable Internet connection and a computer were basic requirements for our experiment (because of the video conference), we made use of WEXTOR (https://wextor.eu [13]) for the questionnaires instead of paper questionnaires and avoided the need for participants to ship back their questionnaires, which we hypothesize may be another step likely to increase the dropout rate.
Assembling the cardboards.
Participants received a video with a visual explanation on how to assemble the cardboard. Despite this, some of the participants struggled with the assembly process and requested help by the experimenter via the video call. Lack of such support can potentially become another dropout cause in unsupervised field experiments. In future studies, it is necessary to experiment and investigate different solutions to improve the cardboard assembly task. It is important to remark that it is also possible to build a cardboard from scratch with a small set of easy-to-find materials. This alternative can also be explored in the future in cases where it may be meaningful.
Limitations
Interactivity with mobile VR.
One pitfall of using Google Cardboard is the lack of basic interaction devices, such as controllers or extra trackers, which are included by default in more sophisticated VR equipment. A typical interaction method for Cardboard is the use of ‘gaze’ measured by head position (since there is in general no eye-tracker directly available in the mainstream smartphones). This method can, however, be straining for the user and disturb interactive tasks.
This lack of readily available interaction methods means that the variety of tasks that can currently be used with mobile VR are somewhat restricted. Several experimental interaction methods have been developed for cardboard, however, they are not readily available yet (e.g. [53]). Thus, in future studies, it will be necessary to explore and test different interaction possibilities and find the most versatile and effective for VR field studies.
Selected tasks.
Even though our study included tasks spanning three dimensions—interaction level, media, and participant effects—it is important to acknowledge that several experimental features remain untested. For example, tasks requiring navigation within the virtual environment or those involving participant embodiment in the VE were not included in our experiments. These untested features represent critical areas for future research, as task-specific requirements, such as locomotion, embodiment, or others, may significantly influence the feasibility and effectiveness of mobile VR in field settings. Consequently, our findings should be interpreted as initial evidence within a limited scope of experimental paradigms, underscoring the need to evaluate mobile VR across a broader range of tasks and dimensions in order to be able to determine its generalizability to various types of psychological experiments.
Population.
In our study, all our participants were young and technologically versed. It is yet to be investigated what are the possible problems and solutions of use of VR in the field for people of different age groups and technological literacy levels. Furthermore, a pre-requirement for mobile VR is for the participant to have access to a smartphone. Potentially, a cheap smartphone could be shipped to the participants but this would increase the complexity of the set-up for the user. Smartphone-based field studies are feasible and cross-checking of quality of data from such devices is possible (e.g. [54]).
Conclusion
Virtual Reality is a promising tool for psychological research, though VR field experiments are still relatively uncommon. In this study, we have provided preliminary evidence supporting the feasibility of using VR as a medium for specific psychological field experiments, such as stress induction and relaxation tasks, and demonstrated that low-cost VR equipment can be effectively employed in both laboratory and field settings. We envision that our findings will help advance the use of VR for field-based assessments and applications, paving the way for broader adoption in psychological research.
Supporting information
Experiment 1
Speech instructions.
The following are the written instructions the participant received before the speech task in Experiment 1. Instructions were provided in German: “Soon, you will go into virtual reality. You will be situated in a conference room. You can first have a look around, if you want to. On a lectern in front of you, there will be a note with three written speech key points on it. Your task is to hold a five-minute speech about the topic of these key points. It is up to your choice if you include all three, two, or just one key point in your speech. Try to complete the task to the best of your ability. Your speech is going to be recorded on audio. Please start your speech when you hear the audio signal (a ’beep’). You are going to hear a second signal when your time, the five minutes, is over. Please try to speak for the whole time. Try to the best of your ability and be willing to get involved in the virtual environment. Do you have any questions?"
Experiment 2
S1 Fig. Procedure in Experiment 2.
Ratings for the stress task: VAS, for the nature task: PANAS, b: VE = virtual environments, c: IPQ, sickness, SUS.
https://doi.org/10.1371/journal.pone.0318688.s001
(TIF)
Preliminary study for the 360° video selection.
To choose an appropriate 360° nature video as second task for Experiment 2, we first pre-selected five publicly available videos that depict a forest including nature sound, varying in including water elements (a stream or waterfall) or signs of human presence (a bridge). The final collection was composed by these five nature videos and three videos with other content to mask the intention of the study (tornado, roller coaster, traffic jam). Still shots of this set of 360 videos can be seen in S2 Fig. A pilot study was conducted to select the 360° video with the highest valence and lowest arousal. Ten participants rated the seven 360° videos in the collection. To measure valence and arousal a Visual Analogue Scale reaching from 0-100 was used for each dimension per video. The video showing a forest and a stream (B, in S2 Fig) was rated to have the highest valence (M = 90.95, Median = 96.25, SD = 11.99) and lowest arousal (M = 14.05, Median = 2.00, SD = 27.38). Furthermore, it showed a lower variance for both valence and arousal in comparison to the second-best rated video, indicating consistency across the participants.
S2 Fig. Still shots of the set of 360 videos used in the preliminary study.
F-H are the videos used in the study. Links to the videos: A: https://www.youtube.com/watch?v= jYLArug2nsU (Forest with wind). B: https://www.youtube.com/watch?v=YFWCsiJjaq0 (At a sparkling stream). C: https://www.youtube.com/watch?v=0Ai-iKQEnQY (Waterfall). D: https://www.youtube.com/watch?v=VP9tyky7SBQ (Bridge with water). E: https://www.youtube.com/watch?v=8-WpK_Lyr_Q (Pine forest). F: https://www.youtube.com/ watch?v=TAZDxjPMxLc (Traffic jam). G: https://www.youtube.com/watch?v=V5rJVPSSoFs (Rollercoaster Canada without screaming). H: https://www.youtube.com/watch?v=b07EL3Mf7eg (Tornado).
https://doi.org/10.1371/journal.pone.0318688.s002
(TIF)
Acknowledgments
Authors would like to acknowledge the help of Jakob Heisenstein with initial tests for shipping the cardboards in Experiment 2 and Artuk Kakhorov for his help porting the public speaking task to mobile VR.
References
- 1.
Foreman N. Virtual reality in psychology. Themes in science and technology education. 2006. p. 225–52.
- 2.
Gaggioli A. Using virtual reality in experimental psychology. In G. Riva & C. Galimberti (Eds.), Towards Cyberpsychology: mind, cognition, and society in the internet age. 2001. IOS Press. p. 157-174.
- 3. Pan X, Hamilton AF de C. Why and how to use virtual reality to study human social interaction: the challenges of exploring a new research landscape. Br J Psychol 2018;109(3):395–417. pmid:29504117
- 4. Loomis JM, Blascovich JJ, Beall AC. Immersive virtual environment technology as a basic research tool in psychology. Behav Res Methods Instrum Comput 1999;31(4):557–64. pmid:10633974
- 5. Parsons TD. Virtual reality for enhanced ecological validity and experimental control in the clinical, affective and social neurosciences. Front Hum Neurosci. 2015;9:660. pmid:26696869
- 6. Reips U-D, Funke F. Interval-level measurement with visual analogue scales in Internet-based research: VAS Generator. Behav Res Methods 2008;40(3):699–704. pmid:18697664
- 7. Balietti S. nodeGame: real-time, synchronous, online experiments in the browser. Behav Res Methods 2017;49(5):1696–715. pmid:27864814
- 8. Crump MJC, McDonnell JV, Gureckis TM. Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PLoS One 2013;8(3):e57410. pmid:23516406
- 9. Garaizar P, Reips U-D. Best practices: two web-browser-based methods for stimulus presentation in behavioral experiments with high-resolution timing requirements. Behav Res Methods 2019;51(3):1441–53. pmid:30276629
- 10. Henninger F, Shevchenko Y, Mertens UK, Kieslich PJ, Hilbig BE. lab.js: a free, open, online study builder. Behav Res Methods. 2022;54(2):556–73. pmid:34322854
- 11. Peer E, Brandimarte L, Samat S, Acquisti A. Beyond the Turk: alternative platforms for crowdsourcing behavioral research. J Exp Soc Psychol 2017;70(1):153–63.
- 12.
Reips U-D. Birnbaum M. Behavioral research and data collection via the internet. In: Vu KPL, Proctor RW, editors. The handbook of human factors in web design, 2nd edn. Mahwah, New Jersey: Erlbaum; 2011. .
- 13. Reips U-D, Neuhaus C. WEXTOR: a Web-based tool for generating and visualizing experimental designs and procedures. Behav Res Methods Instrum Comput 2002;34(2):234–40. pmid:12109018
- 14. Keizer A, van Elburg A, Helms R, Dijkerman HC. A virtual reality full body illusion improves body image disturbance in anorexia nervosa. PLoS One 2016;11(10):e0163921. pmid:27711234
- 15. Riva G. The key to unlocking the virtual body: virtual reality in the treatment of obesity and eating disorders. J Diabetes Sci Technol 2011;5(2):283–92. pmid:21527095
- 16. Shiban Y, Diemer J, Brandl S, Zack R, Mühlberger A, Wüst S. Trier social stress test in vivo and in virtual reality: dissociation of response domains. Int J Psychophysiol. 2016;110:47–55. pmid:27742258
- 17. Saladin ME, Brady KT, Graap K, Rothbaum BO. A preliminary report on the use of virtual reality technology to elicit craving and cue reactivity in cocaine dependent individuals. Addict Behav 2006;31(10):1881–94. pmid:16516397
- 18. Hasler BS, Spanlang B, Slater M. Virtual race transformation reverses racial in-group bias. PLoS One 2017;12(4):e0174965. pmid:28437469
- 19. Van Kerrebroeck H, Brengman M, Willems K. When brands come to life: experimental research on the vividness effect of virtual reality in transformational marketing communications. Virtual Reality 2017;21(1):177–91.
- 20. Krijn M, Emmelkamp PMG, Olafsson RP, Biemond R. Virtual reality exposure therapy of anxiety disorders: a review. Clin Psychol Rev 2004;24(3):259–81. pmid:15245832
- 21. Grillon H, Riquier F, Herbelin B, Thalmann D. Use of virtual reality as therapeutic tool for behavioural exposure in the ambit of social anxiety disorder treatment. In: Proceedings of the 6th International Conference on Disability, Virtual Reality and Associated Technologies. 2006.
- 22. Riva Giuseppe. Virtual reality as assessment tool in psychology. Stud Health Technol Inform. 1997;9–9.
- 23. Rus-Calafell M, Garety P, Sason E, Craig TJK, Valmaggia LR. Virtual reality in the assessment and treatment of psychosis: a systematic review of its utility, acceptability and effectiveness. Psychol Med 2018;48(3):362–91. pmid:28735593
- 24. Herbelin B, Benzaki P, Renault O, Grillon H, Thalmann D, Riquier F. Using physiological measures for emotional assessment: a computer-aided tool for cognitive and behavioral therapy. Int J Disabil Hum Develop 2005;4(4):269–78.
- 25. Behrens SC, Streuber S, Keizer A, Giel KE. How immersive virtual reality can become a key tool to advance research and psychotherapy of eating and weight disorders. Front Psychiatry. 2022;13:1011620. pmid:36405894
- 26.
Jerald J. The VR book: Human-centered design for virtual reality. Morgan & Claypool: 2015.
- 27. Kisker J, Gruber T, Schöne B. Behavioral realism and lifelike psychophysiological responses in virtual reality by the example of a height exposure. Psychol Res 2021;85(1):68–81. pmid:31520144
- 28.
Gaggioli A, Bassi M, Fave AD. Quality of experience in virtual environments. Emerg Commun. 2003, 5:121–36.
- 29. Lombard M, Ditton T. At the heart of it all: the concept of presence. J Comput-Mediat Commun. 1997;3(2).
- 30. Steed A, Frlston S, López MM, Drummond J, Pan Y, Swapp D. An “In the Wild” experiment on presence and embodiment using consumer virtual reality equipment. IEEE Trans Vis Comput Graph 2016;22(4):1406–14. pmid:26780804
- 31.
Mottelson A, Hornbæk K. Virtual reality studies outside the laboratory. In: Proceedings of the ACM Symposium on Virtual Reality Software and Technology. VRST; 2017. .
- 32. Mottelson A, Petersen G, Lilija K, Makransky G. Conducting unsupervised virtual reality user studies online. Front Virt Reality. 2021;2:681482.
- 33. Huber B, Gajos KZ. Conducting online virtual environment experiments with uncompensated, unsupervised samples. PLoS One 2020;15(1):e0227629. pmid:31999696
- 34.
Saffo D, Yildirim C, Di Bartolomeo S, Dunne C. Crowdsourcing virtual reality experiments using VRChat. Conference on Human Factors in Computing Systems - Proceedings. 2020. p. 1–20.
- 35. Radiah R, Mäkelä V, Prange S, Rodriguez SD, Piening R, Zhou Y, et al. Remote VR studies: a framework for running virtual reality studies remotely via participant-owned HMDs. ACM Trans Comput-Hum Interact 2021;28(6):1–36.
- 36.
Ratclife J, Soave F, Bryan-Kinns N, Tokarchuk L, Farkhatdinov I. Extended reality (xr) remote research: a survey of drawbacks and opportunities. In: Proceedings of the Conference on Human Factors in Computing Systems. 2021. p. 1–13.
- 37. Zimmer P, Buttlar B, Halbeisen G, Walther E, Domes G. Virtually stressed? A refined virtual reality adaptation of the Trier Social Stress Test (TSST) induces robust endocrine responses. Psychoneuroendocrinology. 2019;101:186–92. pmid:30469086
- 38. Schubert T, Friedmann F, Regenbrecht H. The experience of presence: factor analytic insights. Presence: Teleoperat Virt Environ 2001;10(3):266–81.
- 39.
Lane NE, Kennedy RS. A new method for quantifying simulator sickness: development and application of the simulator sickness questionnaire (SSQ); 1988. Available from: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citation&list_uids=115546
- 40. Browning MHEM, Mimnaugh KJ, van Riper CJ, Laurent HK, LaValle SM. Can simulated nature support mental health? comparing short, single-doses of 360-degree nature videos in virtual reality with the outdoors. Front Psychol. 2020;10:2667. pmid:32010003
- 41. Wang X, Shi Y, Zhang B, Chiang Y. The influence of forest resting environments on stress using virtual reality. Int J Environ Res Public Health 2019;16(18):3263. pmid:31491931
- 42.
Reips U-D. Shevchenko Y. How to conduct online experiments. Sage Research Methods: Doing Research Online. 2023.
- 43. Watson D, Clark LA, Tellegen A. Development and validation of brief measures of positive and negative affect: the PANAS scales. J Pers Soc Psychol 1988;54(6):1063–70. pmid:3397865
- 44. Krohne H, Egloff B, Kohlmann C, Tausch A. PANAS – positive and negative affect schedule - German version. Diagnostica. 1993;42(2):139–56.
- 45. Brooke J. SUS - A quick and dirty usability scale. Usability and context Usability evaluation in industry. 1996;189(194):4–7.
- 46. Bangor A, Kortum P, Miller J. An empirical evaluation of the system usability scale. Int J Hum-Comput Interact 2008;24(6):574–94.
- 47. Lewis JR. The system usability scale: past, present, and future. Int J Hum–Comput Interact. 2018;34(7):577–90.
- 48. Ban os R, Botella C, Liano V, Guerrero B, Rey B, Alcaniz M. Sense of presence in emotional virtual environments. Presence: Teleoperat Virt Environ. 2004;13(1):156–9.
- 49.
Siess A, Beuck S, Wölfel M. Virtual reality – quo vadis? How to address the complete audience of an emerging technology. In: Collaborative European Research Conference. 2017. p. 315–24.
- 50.
Reason JT, Brand JJ. Motion sickness. Academic Press: 1975.
- 51. Götz FM, Stieger S, Reips U-D. Users of the main smartphone operating systems (iOS, Android) differ only little in personality. PLoS One 2017;12(5):e0176921. pmid:28467473
- 52. Kuhlmann T, Garaizar P, Reips U-D. Smartphone sensor accuracy varies from device to device in mobile research: the case of spatial orientation. Behav Res Methods 2021;53(1):22–33. pmid:32472500
- 53. Chen T, Xu L, Xu X, Zhu K. GestOnHMD: enabling gesture-based interaction on low-cost VR head-mounted display. IEEE Trans Vis Comput Graph 2021;27(5):2597–607. pmid:33750694
- 54. Stieger S, Reips U-D. Well-being, smartphone sensors, and data from open-access databases: a mobile experience sampling study. Field Methods 2019;31(3):277–91.