http://orcid.org/0000-0001-9242-246X
Figures
Abstract
Anecdotal evidence from philanthropic fundraisers shows that virtual reality (VR) technology increases empathy and can influence people toward pro-environmental behavior. Non-profit organizations are increasingly marketing their causes using virtual reality and they report increased donations when VR technology is employed. In VR, users are immersed in situations intended to feel more like the real world through technology, such as 360° video viewed through 3D headsets that block out visual and auditory distractions. The framing of the message as either positive or negative has long shown to have an effect on behavior, although consensus on the impact of framing has not been reached in relation to encouraging contributions to public goods. This paper focuses on field experiments used to investigate the effects of varying degrees of visual immersion and positive versus negative message framing on respondents’ contributions to a conservation charity. Participants were exposed to a five-minute underwater film about coral reefs and the importance of protecting them. We employed a 2x2 experimental design using 3D head-mounted displays comparing 360° film footage vs. unidirectional film and a positive message vs. a negative message. After watching the film, each participant completed a short questionnaire and had the opportunity to donate to a marine conservation charity. In addition, we tested a control treatment where no video was observed. The video was filmed in Indonesia which is host to some of the world’s most biodiverse reefs that are under great threat from human activity. We also conducted the study in Indonesia, sampling a total of 1006 participants from the Bogor city area and tourists on the island of Gili Trawangan—which is popular for scuba diving and snorkeling. We find significant differences in observed behavior and reported emotions between all treatments compared to the control condition. Among the tourist sample, we find significant differences between the 360° film with a negative message which garnered significantly larger average donation amounts compared to the unidirectional film with both positive and negative framing. Overall, we can infer from these studies that virtual reality is an effective way to raise awareness of environmental threats and encourage behavioral action, especially when tailored to target groups. New technology, such as the VR head-mounted display, is highly effective at attracting interest which is an important point to encourage organizations to invest in new technologies.
Citation: Nelson KM, Anggraini E, Schlüter A (2020) Virtual reality as a tool for environmental conservation and fundraising. PLoS ONE 15(4): e0223631. https://doi.org/10.1371/journal.pone.0223631
Editor: Stefano Triberti, University of Milan, ITALY
Received: September 23, 2019; Accepted: February 26, 2020; Published: April 6, 2020
Copyright: © 2020 Nelson et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All excel raw data files are available from the public repository OSF at the following: https://osf.io/kqbc5/.
Funding: AS received funding from the Leibniz Association SAW-2014-ZMT-1 317. https://www.leibniz-gemeinschaft.de/en/research/leibniz-competition/funded-projects.html KN received funding from Germany Federal Ministry of Education and Research (BMBF) university competition for the "Sea and Ocean" science year 2016-2017 https://www.hochschulwettbewerb.net/201617/3d-virtual-reef-prototype/ The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
The Intergovernmental Panel on Climate Change (IPCC) recently issued its special report on the impacts of global climate change on nature and society [1]. The report paints a grim picture of the consequences of climate change if the earth’s temperature rises by even 0.5°C and further states that rising temperatures will result in food shortages, more wildfires, and a mass die-off of coral reefs as soon as 2040. Despite growing attention in the media pointing to the human-induced rate at which the climate is changing, mounting evidence from across the behavioral sciences has found that most people regard climate change as a non-urgent and psychologically distant risk—spatially, temporally, and socially—which has led to deferred action and decision making about mitigation and adaptation responses [2].
Among the greatest challenges of communicating climate science to the public is bridging the knowledge-to-action gap [3]. This gap refers to the general lack of environmental behavior change by individuals or society at large despite increases in communication about environmental problems and heightened public awareness of such issues. According to several studies, communication methods that are successful in bridging this gap involve cognitive and behavioral dimensions[3–6]. For example, people gain an understanding about an issue, experience an emotional response, such as interest or worry, and actively respond by changing climate-relevant behavior or political action. Research has shown that the content of the message [7,8], and level of visual immersion [9–11] can have an impact on the intensity of emotional responses and influence behavior [12]. Recently the nature documentary series Our Planet has received a plethora of media attention for the use of the negative framing of human-induced climate change impacts on nature which the series uses to push viewers toward a conservation website "to discover what we need to do now to protect our jungles" [13]. Beyond framing, virtual reality (VR) technology has received attention for being able to increase immersion and one’s sense of presence, which is thought to affect emotions and connect people better to the subject matter (Note: Herein the terminology “virtual reality” includes 360° films viewed using head-mounted displays which provide an immersive experience) [14].
In this study, we examine virtual reality using a field experiment with real financial decisions. Using a head-mounted display, participants experience the five-minute video, Coral Reefs: Life Below the Surface [15]. Then participants are asked to fill out a short survey which includes a request for donations to a coral reef conservation organization. Aiming to test the effects of visual immersion (low vs. high) and message framing (positive vs. negative), participants are exposed to one of the treatment conditions (see Table 1 for descriptions): positive video (VID_POS), negative video (VID_NEG), positive 360° video (360_POS), negative 360° video (360_NEG), and no video (control).
Literature review
The research concept to experimentally test the impact of different levels of immersion (low vs. high immersive environments) on emotional response and pro-social behavior was born out of the recent hype in popular media articles that virtual reality “can make you a better person” and is “the ultimate empathy machine” [16]. Such articles make claims that virtual reality increases empathy and the technology is successful at influencing behavior, such as increasing donations for charity (e.g. The Wall Street Journal “Charities Use Virtual Reality to Draw in Donors” [17]; AR Post “Using Virtual Reality for Increased Charity Donor Outreach and Funding” [18]; and The Guardian “Can Virtual Reality Emerge as a Tool for Conservation?” [19]). However, at the point of conceptualization of this research, only anecdotal evidence was being used to support these claims. There is currently a lack of scientific research that can back up claims that increasing the level of immersion through the use of virtual reality technology can influence subsequent behavior, such as increased donations to charity (as the articles claim it can) [20].
Immersive visual experiences.
An article by Fortune magazine in 2017 claimed, “the emotional impact of VR has proven to increase awareness, evoke empathy, and elicit action” pointing to examples from a Facebook report that “48% of virtual reality charity content viewers were likely to donate to the causes they experienced” and a report by the United Nations claiming that their VR production Clouds over Sidra which films the life of a 12 year old Syrian refugee helped raise twice the charity’s normal rate” [16]. Although these studies may sound promising, they lack controlled experimental designs and many uncontrolled factors could account for the supposed increase in donations (e.g. increased media attention of the Syrian crisis may have led to higher donations that year regardless of the VR technology used at the fundraiser). The study from the aforementioned Facebook report and one from the market research company Nielson [21] found donation intentions to be higher following exposure to a 360° video compared to other media, but these considered hypothetical donation decisions only.
Several lab studies performed at the Virtual Human Interaction Lab of Stanford University focus on the effects of virtual reality on encouraging prosocial behavior [11] and environmental behavioral intentions [22]. The evidence seems promising that virtual reality interactions increase a person’s feeling of presence which can connect them better to the subject matter and affect one’s behavior. In one study, Ahn, Bailenson, and Park [23] compare the effects of cutting a tree in virtual reality to reading a written description or watching a video depiction of the tree-cutting process to encourage paper conservation. The study shows that the virtual experience results in a 20% decrease in paper use during the experiment as compared to participants who read a print description of tree cutting. In another experiment by Ahn et al. [23], the participants wear a virtual reality headset and can walk around on all fours to experience what it’s like to be a cow that is raised for dairy and for meat. As the participant wanders around the virtual stall with hay, he can see the embodiment of himself as a cow get poked with a cattle-prod while he is also actually experiencing a slight poke in the ribs while he is being led to the slaughterhouse in the virtual world. For a time after the VR experience, participants reported eating less meat [22]. On the contrary, preliminary research carried out by the authors of this paper found no statistical differences among university students exposed to differing levels of immersion in an underwater virtual world in relation to their charitable giving behavior to coral reef conservation [24]. While some scholars believe virtual reality can create empathy with non-human actors by simulating, for example, what it’s like to be a cow or a piece of coral [22], other scholars are highly critical of this concept [25].
A lab study by Gürerk and Kasulke [26] tests 360° videos using VR headsets and finds that participants exposed to VR gave more frequently and donated higher amounts compared to those who received only a written donation request with no video. The research described herein differs from the study by Gürerk and Kasulke [26] in several important ways. We test the effectiveness of 360° films on raising money for charity outside of the lab using a natural field experiment. We control for the presence of the head-mounted technology ensuring that any differences in our results are driven by the immersive experience provided by the 360° video compared to a non-360° video so we know it’s not the new technological gear driving the result. And finally, we compare the combined effects of the level of immersion with a positive or negative audio message about the future of coral reefs based on climate change predictions.
Framing: Positive versus negative message valence. Framing in this context is defined as a semantic restructuring of identical information [27]. Valence framing structures information in either a positive or a negative light. In this manuscript, positive message valence describes action that leads to a favorable outcome (e.g., save coral reefs and species biodiversity) and negative message valence indicates the consequences of inaction which lead to adverse conditions (e.g., loss of coral reefs and species biodiversity). Classic economic theory of rational choice assumes that as long as the same information is presented, regardless of how the information is presented or framed, an individual will always interpret the information in the same way and behave accordingly [28]. Thus, preferences should not be affected by framing. However, a multitude of studies have proven otherwise, beginning with the seminal work of Tversky and Kahneman [7], which eventually led to a Nobel Prize in economics in 2002, and which describes a cognitive bias known as “the framing effect” wherein people’s decisions are affected based on positive or negative semantics (i.e., people make different decisions if the options are presented as gains or losses).
Communicating climate change to encourage environmentally friendly behavior is crucial in transforming knowledge into action. The way climate change messages are framed can significantly alter the impact they have on a recipient, but not much is known about the effects of positive versus negative message valence on environmental attitudes and even less is known about the effect they have on actual behavior [29]. There is a surprising scarcity of empirical research examining the effects of message framing on pro-environmental behavior, wherein the measurement is observed behavior rather than merely behavioral intention [30,31]. Many studies examining the effects of framing have been conducted in the field of health psychology which focus on comparing the relative effectiveness of information that focuses on the positive consequences of performing a particular behavior (gain) or on the negative consequences of inaction (loss) [32–34]. However, a review of the literature on positive and negative message framing proves to be rather inconsistent and highly contextual with some studies showing that positive messages are more effective and other studies showing increased effectiveness with negative messages.
According to a health study by Rothman et al. [35], evidence indicates that negative loss frames are found to be more effective for encouraging detection behavior and positive gain frames are better for encouraging prevention behavior. There is less empirical evidence, however, when communicating climate change risks as to whether insights and theories from other domains are transferable to climate action [36]. Additionally, some studies suggest that framing effectiveness differs based on a person’s decision stage (i.e., early stage of determining whether an issue is problematic based on its personal risk/costs or the later stage of establishing intention to act). When personal risk to an individual has yet to be well-established, fear-based messages seem to be more effective at eliciting emotional responses [35,36]; whereas, once a person shifts from considering risks associated with a potential future situation to considering the potential solutions to avoid said scenarios, positive messages that emphasize a desired outcome and the benefits of adopting a specific behavior are more salient because these messages are now congruent with the actions that could eliminate risk or the fear associated with a specific issue [37,38]. The inconsistency across studies supporting, in some cases, a stronger influence using negative loss frames and, in other cases, a stronger influence with positive gain frames, seems to point to additional influencing factors. According to a review of the literature, these include: the purpose of the information campaign (i.e., raise awareness, change opinions, evoke emotion, call-to-action to avoid or to do something, plan for future behavior); stage of individual decision process (i.e., risk consideration, prevention, detection, goal setting, short-term action, long-term behavior change); the particular behavior in question (i.e., actively doing something or avoiding a particular behavior, personal action, political action, societal action); the relationship between an individual and the behavior in question (i.e., demographics, personality type) [29,36,38].
Most research of valence framing does not consider social dilemmas such as climate change, and the nature of impacts of framing may be different in such situations [28]. A theoretical study by Moser and Dilling [39] proposes shifting the discourse towards a positive motivational-oriented approach offering solutions rather than the negative approach that highlights sacrifices as a more effective strategy to encourage climate-change-related behaviors. Gifford and Comeau [8] empirically test this using a motivational (positive) or sacrificial (negative) priming message in a survey eliciting intentions to engage in mitigative behaviors and they find that the positive message is more effective at encouraging several environmental behaviors compared to the negative message, but these results are not significantly different than the control condition. Several demographic factors such as gender, age, income, and education level also play a role in determining behavior. A study by Avineri and Waygood [28] examines the effect of positive and negative valence framing on people’s perceptions of CO2 emissions based on how a travel mode is framed (i.e., it provides environmental benefit (positive frame) or its potential to reduce environmental loss (negative frame). The study finds that negative framing is more effective at highlighting the differences between CO2 emission amounts of alternative travel modes which they suggest is more likely to influence travel-related decisions. These studies are highly informative and begin to fill the gap in the literature on climate-change-related behavior, although more research is needed that pushes beyond hypothetical responses and intended behavior. Given the dearth of empirical evidence in this field, the research presented in this paper measuring the effects of framing on emotions, intentions, and actual pro-environmental behavior is timely and fills a crucial gap in research.
Methodology
Indonesia is the largest archipelagic country in the world, encompassing more than 17,000 islands, 86,700 square kilometers of coral reefs, and supporting nearly 230 million people. The people of Indonesia are increasingly dependent on marine resources for food and income sources, including tourism. Presently about 70% of the country’s protein sources come from fish (in some poor coastal communities this figure approaches 90%). In 2016, total contributions from tourism in Indonesia amounted to 60 billion USD in revenue (6.2% of GDP) [40]. Indonesia is a global hotspot for coastal marine biodiversity. Blessed with nearly 18% of the world’s coral reefs, Indonesia sits firmly at the center of the “Coral Triangle”, the region with the world’s highest marine biodiversity [41]. Despite the clear importance of coral reefs, they are rapidly declining as a result of a range of local drivers (e.g., overexploitation of fisheries, coastal development and pollution) and global drivers (e.g., global warming and ocean acidification) [42]. Conservation of coral reefs and the resources they provide is a collective responsibility and requires action from people in countries across the globe to reduce greenhouse gas emissions. Locally, efforts to reduce additional pressure from already stressed reefs can improve their resilience to climate change [43].
The study was conducted in two locations in Indonesia. We sampled students from Bogor Agricultural University (Institut Pertanian Bogor—IPB), a general population from a public park in Bogor, and a mostly foreign tourist sample that were visiting the popular scuba diving destination of Gili Trawangan. We chose these locations and sample populations with our research collaboration partners in Indonesia (IPB and Gili Eco Trust) to cover: 1) the typical lab experiment student population, 2) a more generalized public population, and 3) a targeted population.
Bogor has a population of approximately one million people and is located 60km from the capital city Jakarta on the island of Java. Bogor is an important economic, cultural, and scientific center for Indonesia. Bogor is a land-locked city and is situated in the mountains, at a distance, relatively-speaking, from the popular coral reef diving destinations of Indonesia. In contrast, Gili Trawangan, a small island situated between Bali and Lombok, is represented by a remarkably biodiverse coral ecosystem. Over the last decade Gili Trawangan has grown into a major destination for tourists worldwide of all budgets and interests [44]. Gili Trawangan is the second most frequented destination in South East Asia for SCUBA diving certification (second only to Koh Tao in Thailand) [45]. Although only six square miles, the island receives heavy tourist traffic with up to 2000 new visitors per day and over one million tourists annually [46]. Increased tourism results in rapid development, challenging the island’s infrastructure to keep up with the growth and putting pressure on the fragile coral ecosystem surrounding the island [45].
Field experiments were conducted during March and April, 2018. All participants watched a five minute video and then completed a short questionnaire, except for those in the control condition that only completed the survey. The experiment employs a between-subjects 2x2 design examining the effectiveness of increased visual immersion through 360° film technology compared to classic unidirectional film footage and positive or negative valence framing (see Table 1 for treatment descriptions). We measured differences in donation behavior and emotions post-treatment across the four treatments and the control (no video).
In total, 1006 subjects participated in the experiment. 487 subjects were recruited from the IPB campus and 248 people from the general public were recruited from an outdoor public park in the center of Bogor. We then conducted the experiment with 271 participants on the popular diving and tourism island Gili Trawangan. The research was approved by the institutional ethical review committee for Leibniz ZMT, and was conducted in accordance with the ethical standards of the German Data Protection Act, and the 1964 Helsinki Declaration and later amendments. Informed consent was obtained from all individual participants involved in the study. See Table 2 below for details on sample size by treatment and location.
Several field assistants were trained on the protocol and assisted with recruitment, video screening, and survey administration in Bogor and Gili Trawangan. In Bogor, eleven trained enumerators recruited participants on-the-spot from common areas including outdoor meeting spaces, benches/tables, cafes, and dormitories on the university campus over a period of four days during the class week. On the weekend, the same enumerators recruited participants from a busy outdoor public park located in the center of Bogor city where many people come to gather on the weekend to socialize, exercise, and meet friends and family. In Gili Trawangan, we actively recruited participants walking along the main beachfront street and we also had a VR station set up next to the harbor to recruit participants as they waited for the ferry to depart the island (both are public spaces). The enumerators approached people at random and cited a scripted invitation for participation. Willing participants were then assigned into a treatment group based on their randomized selection of an ID number that had a secret coding system known only to the enumerator which corresponded to a specific treatment condition. The enumerator then helped the respondent with the headset and headphones and started the video in the preferred language (the video was available in Bahasa, English, and German). The video lasted five minutes. After the participant watched the video, they filled out the survey questionnaire which included: voluntary consent; a request for a donation to charity; questions about emotional feelings, sense of presence, environmental engagement and personal perspectives, and basic demographics (for full survey, see S1 and S2 Files).
Respondents from Bogor were incentivized through enrollment in a lottery with a ten percent chance to win 100,000 Indonesian Rupiah (IDR) (Exchange rate on April 4, 2018 1USD = 13762.24IDR or 1EUR = 16902.1IDR). Participants were asked if they were to win the 100,000IDR, how much (if any) would they donate to the coral reef charity, Gili Eco Trust. For notifying the respondents in Bogor about the lottery winnings, the field assistant would text the ID code to the head researcher to check if the respondent’s number matched the randomly pre-selected winning ID codes. In the cases that the respondent won the lottery, the donation amount was deducted from their winnings. Lotteries, including raffles, are commonly accepted methods used in experimental economic research and in charity fundraisers [47]. Respondents from Gili Trawangan were not offered the lottery incentive and were not paid for participation. The surveys in Gili Trawangan included an empty envelope with the corresponding ID number where the respondent would put their donation and drop the envelope into a collection box. The purpose of different experimental designs was to create plausible scenarios for each context and to reach sufficient donations in the Bogor sample, to possibly observe a treatment effect. Both designs were pretested.
The authors recognize that people may treat windfall earnings or possible future earnings differently than money in their own pockets. Therefore, we do not compare donation results across the Bogor sample and the Gili Trawangan sample. The donation data are treated and analyzed as two separate studies. We are interested in treatment differences and trends that may emerge across controlled conditions where all respondents were treated the same. To avoid overstating donations in the Bogor case and to make the results broadly applicable, we focus on significant differences in donations between treatments compared to the control condition rather than the actual donation amounts in the respective studies.
In all treatment conditions, we used the Zeiss VR One headset, Nokia 6 smartphone, and Sony MDRZX110 headphones to present the film to participants. We were particularly interested in understanding if the increased sense of immersion provided by 360° film technology influences charitable donation decisions compared to standard unidirectional film. 360° film viewed using a VR headset provides an immersive experience by surrounding one’s field of vision with continuous video imagery. Other studies have compared watching a 360° film on a mobile phone or a tablet to watching the film using a headset [26]. However, we want to know the effect of the increased visual immersion provided by the 360° omnidirectional film technology. In the study by Gürerk and Kasulke [26], the new technology of the headset and the additional piece of equipment which completely obstructs ones vision thereby blocking out any other visual distractions may be enough to drive the differences they observe in donations between treatments. It is not possible to disentangle the effects of the equipment itself versus the level of immersion created by the full field of vision compared to a limited field of vision without testing both the classic unidirectional video and the 360° video with the headset on. Therefore, all participants, except those in the control condition, viewed the video wearing the VR headset.
Visual framing: Unidirectional versus 360° film
Unidirectional film is shot with one video camera facing in only one direction at a time. The viewer has a limited field of view given that they are only able to see in one direction. The viewer is a passive participant in the visual story dictated by the filmmaker. Participants who were exposed to the unidirectional film using the VR headset would see the video projected onto a screen in front of them similar to watching a movie in a theater or on TV. The screen is rectangular and framed on a solid indigo background (Fig 1a). The participant could move their head in any direction but they would only see the indigo background in every direction except forward facing where the video is playing.
Note: The 360° film imagery in 3b is a dynamic 360 degree effect which surrounds the user in every direction with video footage but this cannot be captured in a static image. Source: Schmitt, H. and Schmitt, C. [The Jetlagged]. Coral Reefs: Life Below the Surface. Retrieved from https://www.youtube.com/watch?v=2TPG8lcfeDc&t=64s.
360° film is shot using an omnidirectional camera where a view in every direction is recorded simultaneously. The viewer has the freedom to look around in every direction throughout the video. When a 360° video is viewed using a VR headset, the movement of one’s head changes the direction of the view. This creates an immersive experience given that the video responds to the user’s movement. When the user looks right, they see video footage to the right, when they look down, they see the corresponding video footage showing the sea floor and when they look up, they see the surface of the water (Fig 1b shows a relative wider field of vision but cannot fully capture the 360° experience with a static two-dimensional image).
Valence framing: Positive versus negative
To test the effectiveness of positive and negative valence, two scripts were semantically structured so that environmental gains and losses were distinctly presented. The positive valence narration focuses on the benefits that the reefs provide and will continue to provide in the future if we act now to save them. The negative valence focuses on the losses we will suffer if we fail to act (see Table 3 below for excerpts from the narrative script; see S3 File for full narrative). The imagery between the positive and negative valence framing was the same.
Charitable organization receiving donations
Donations were collected for the coral conservation organization Gili Eco Trust. The Gili Eco Trust is a local community-organized conservation organization on Gili Trawangan that focuses efforts on restoring and regenerating coral reefs using Biorock technology [48]. In addition to implementing and maintaining one of the largest coral restoration projects in the world, the Gili Eco Trust manages several sustainability programs on the island including weekly beach cleans, garbage and compost collection, plastic bottle bank (money for plastic), recycling services (sorting, cleaning, crushing, and shipping off the island to a recycle center in Lombok), sourcing alternative plastic products for island businesses, advocating for sustainable policies and infrastructure, and providing conservation education to the local schools. The Gili Eco Trust operates solely on donations from local businesses, tourists, and volunteers.
Results
The summary statistics of each sample population are described in Table 4 below. The mean age of participants in the study was 23 with the student sample representing a younger average age of 20 years old and the tourist population averaging 29 years old. On average there were slightly more female participants than males. The student sample came from locations that were closer to the sea with an average of 74km relative to the public sample at 200km and the tourist sample at 319km. Income was measured based on the participants’ perception of their family income in relation to the average income level from their country.
Most people across the sample considered themselves to be in the middle-income category with the general public sample perceiving themselves on average to be on the low end of the middle-income category. The average level of education was the same for the student population and the tourist population, while the general public in Bogor were lower on average by one level of education. The level of religiosity was measured based on self-identification on a 7-point Likert scale of ‘not at all religious’ to ‘the most religious’. Participants from Bogor claimed to be much more religious compared to the tourists from Gili Trawangan who overall self-identified as ‘not very religious’.
Bogor, Indonesia
The results of treatment differences based on the average contribution are reported for the student population from IPB and the general public from Bogor in Fig 2. There are no significant differences in the average contribution amounts between the two populations; therefore, the data are combined for the regression analyses.
Treatment conditions with different superscript letters are significantly different at p<0.01; those with the same superscript letters are not statistically different from each other using the one-way ANOVA with Bonferroni correction for multiple comparisons.
Analyzing the results using the one-way ANOVA with Bonferroni correction for multiple comparisons, we found that average contribution amounts were significantly higher in all treatment conditions at p<0.001 when compared to the control condition. The four treatment conditions, however, were not significantly different from each other (see Fig 2). As can be seen from the bar graph, respondents in both the student sample and the general population from Bogor donated more on average in the 360° positive message frame treatment than the average contributions in the other treatments, however, this amount was not significantly higher when compared to the 360° negative treatment or to the classic unidirectional video treatments.
Upon examining donations on the extensive margin (i.e., to give or not to give), we must consider that donations would come out of any lottery winnings. Therefore, the frequency of donation was very high (>90%) in all conditions, including the control, with no significant differences between treatments. As reported in Fig 2, there was a treatment effect on the intensive margin (i.e., average amount given) but only when the treatments are compared to the control. We did not find a significant effect between positive and negative framing (VID_NEG vs. VID_POS and 360_NEG vs. 360_POS) or between high and low visual immersion (VID_NEG vs. 360_NEG and VID_POS vs. 360_POS).
For the linear OLS regression, we combined the data from the student population and the general population in Bogor since there were no statistical differences in their donation behavior (see Table 5). An environmental engagement index (Cronbach’s α = 0.68) was created from the combined mean score of all the environmental engagement indicators including the questions: I prefer buying new things rather than fixing broken things (reverse coded), I prefer to eat meals with meat/fish rather than vegetable only (reverse coded), I bring my own reusable bag to the market, I recycle, I bring my own container for take out, I volunteer time for an environmental group, I use single use plastic (reverse coded), I make travel plans based on the amount of CO2 emissions generated by transportation, I purchase recycled materials, I only buy products from companies that protect the environment.
The average donation amount was used as the dependent variable. Several iterations of the regression were analyzed by first including the demographic variables, then adding the environmental engagement index and personal environmental perspectives variables, and adding the treatment assignments (see S1 & S2 Files for full questionnaire). The results showed that age and income were significant indicators of donation amount and as age and income increase, donation amounts also increase. The behavioral intention indicator ‘sustainable future’ which consisted of ranking oneself on the question, “In the future, where do you see yourself on the spectrum of living a sustainable and environmentally conscious lifestyle?”, was also significantly positively correlated to the amount donated. However, after controlling for the treatment effects, the significance of the behavioral intention decreased. Compared to the left out control treatment, all other treatments were highly significant determinants of the donation amount. The treatment variables were strong explanatory variables as can be seen by the increase in R2 when the treatments are added to the regression model. Other factors, such as gender; education; distance a person lives to the sea; perceived level of personal religiosity / climate-change optimism / generosity / environmental consciousness; and previous experience seeing a real reef did not appear to be significant drivers of donation decisions.
Gili Trawangan
The results presented in Fig 3 show the treatment differences on the extensive margin. The bar graph shows that the frequency of donations was higher in the virtual reality treatments. The rate of donating was significantly higher (p<0.001) in all treatment conditions when compared to the control condition using a binomial probability test. There were no significant differences between positive and negative framing in either the low visual immersion (VID_NEG vs. VID_POS) or high visual immersion (360_NEG vs. 360_POS) treatments. When we compared the effects of level of visual immersion, we found the rate of giving to be significantly higher (p<0.01) in the high visual immersion treatments for both the positive (VID_POS vs. 360_POS) and negative frames (VID_NEG vs. 360_NEG).
Treatment conditions with different superscript letters are significantly different; those with the same superscript letters are not statistically different from each other.
Fig 4 shows the results of the average donation amounts across the treatment conditions. There were significant differences (p<0.05) between both of the high immersion VR treatments and the control condition (360_NEG vs. CONTROL and 360_POS vs. CONTROL), but the low immersion treatments were not significantly different than the control. 360° virtual reality combined with a negative framing significantly increased donations on the intensive margin compared to the classic unidirectional videos with negative framing (VID_NEG vs. 360_NEG). Although the average donations in the positive 360° virtual reality (360_POS) treatment (26,800IDR) appeared to be much higher than those in the classic unidirectional (VID_POS) treatment (14,600IDR), these amounts were not statistically different. We did not observe any statistically significant differences between the positive and negative framing for either the low visual immersion (VID_NEG vs. VID_POS) treatments or the high visual immersion (360_NEG vs. 360_POS) treatments. However, the Bonferroni correction for multiple comparisons is known to be quite conservative. When we relax the restrictions and compare across two treatments only using t-tests, significant differences are revealed between the VID_NEG vs. CONTROL (p<0.01), VID_POS vs. CONTROL (p<0.01), and between 360_POS vs. VID_POS (p<0.05).
Treatment conditions with different superscript letters are significantly different at p<0.05; those with the same superscript letters are not statistically different from each other using the one-way ANOVA with Bonferroni for multiple comparisons.
When we analyzed the donation amount as a dependent variable using an OLS regression (Table 6), the data showed that age and distance that a person lives from the sea in kilometers were both significant indicators of donation behavior. The older a person is and the further they live from the coast was correlated with higher donation amounts. Although just on the border of significance at p = 0.052, a person’s self-perception of their level of environmental consciousness was correlated with donation amount. Interestingly, there was a negative relationship between environmental consciousness and donation amount, but this result was not significant at p<0.05. Confirming the results of the treatment comparison, we saw significant differences between the highly immersive VR treatments (360_NEG and 360_POS) compared to the omitted CONTROL variable. Surprisingly there was not a significant effect of the level of income on donation amount as was seen in the first study. Unlike the Bogor sample, we did not see a significant effect of the level of income on the donation amount.
Emotional responses: Pooled sample
The questions in the survey eliciting emotional responses and sense of presence were provided with a Likert scale presenting seven ordered response levels with labeled poles. The positive poles were coded with 1, and the negative poles with 7, respectively. We compared the mean scores for negative (NEG) versus positive (POS) valence and low immersive virtual experience (VID) versus high immersive virtual experience (360). We have pooled the data across the Bogor and Gili Trawangan samples for this analysis since we used the same treatments and the same survey questionnaires in both sample populations. Referring to Table 7, there were statistically significant differences showing that people in the positive treatment felt less sad compared to those in the negative treatment. People in the positive treatment and those in the VR treatment (360) reported feeling happier than those in the negative treatment and those in the classic video treatment. Respondents in the negative treatments and in the VR treatments felt on average more helpless, but also more hopeful compared to those in the positive framing and video treatments. Participants exposed to the 360° virtual reality film reported feeling significantly more worried about global climate change than their counterparts in the classic video treatments.
As can be expected, those exposed to the 360° virtual reality film felt a significantly higher sense of presence than those in the classic video treatment. Presence was measured by the questions ‘Did you feel like you were part of the underwater world?’; ‘Did you feel like you were surrounded by corals and fish?’; ‘Did it feel like you were really under water?’. Additionally, those that experienced the 360° virtual reality film perceived significantly higher global and personal consequences from global climate change.
When comparing the control condition to the negative and positive message valence and the low and high visual immersion, we saw significant differences on almost every indicator of emotional feeling and sense of presence (see Table 8). The only indicators that did not appear to be significantly different from the control condition were those responses in the positive frame and 360° films to feeling hopeful.
In all other cases, compared to the control, those in the treatment conditions felt overall significantly sadder, less happy, more helpless, less hopeful (only NEG and VID conditions), more reflective, more worried about climate change, and more strongly that climate change would have global and personal consequences.
Discussion and conclusion
In this paper we employ field experiments to examine the effects of virtual reality and message framing on conservation behavior and emotions. We surveyed two populations—an Indonesian sample in Bogor, Indonesia and a tourists sample in Gili Trawangan, Indonesia. We measure real behavior using donations to a conservation charity organization. We find that communicating an issue through the use of virtual reality technology is more effective at increasing donations than written donation requests, but the level of effectiveness is dependent on the target population and the immersive experience. The most effective treatment for the tourist population in Gili Trawangan was the 360° VR film with a negative audio message. This treatment increased donations by more than fifty percent compared to the classic video treatments. Although there was a fifteen percent increase in the donations when compared to the positive message framing, this difference was not statistically significant. The tourists differentiated their behavior between the type of media they experienced. The number of people donating was significantly higher in the classic video treatments compared to the control. And this effect increased significantly when participants were exposed to the 360° virtual reality video compared to those that saw the classic video. In our Bogor sample, however, the 360° VR film with positive message framing garnered the highest average donation amounts by at least a five percent margin.
In Bogor, all forms of media (classic unidirectional video and 360° VR video) led to significant increases in the average contribution amounts compared to no media communication (control). This result differed in the Gili Trawangan population where respondents in all treatments gave more, but only those in the 360° VR treatments gave significantly more on average than those in the control. We try to explain this difference given that we used the VR head mounted displays to screen all video treatments and the personal familiarity with the technology and the experience created with the technology may be different between the populations. The VR head mounted display creates a relatively immersive environment even without the video’s 360° responsive motion capability. It does so by blocking out other visual and auditory distractions and through physical proximity. The Bogor sample may have been more sensitive to the overall immersion created by the VR head mounted display due to the fact that this is a relatively new piece of technology that not many people in Bogor may have experienced before. In fact, we observed that many people froze their gaze locked in to looking in one direction only during the 360° VR treatments, which essentially limits a 360° experience to the experience provided by the classic video treatment. The nature of the technology physically displays the content of the video proximally very close to an individual within just a mere few centimeters away from a person’s eyes. Far fewer participants in the Bogor sample have seen a coral reef in real life compared to the sample in Gili Trawangan (Bogor = 51% and Gili Trawangan = 83%). The realistic and very close up underwater experience may have been the first time some have seen a glimpse into life underwater. Upon donning the headset, one participant in Bogor actually shrieked and was scared because she didn’t know how to swim and felt physically and emotionally like she was in danger because she felt as if she was in the water. We also had many comments from divers in Gili Trawangan that the experience really felt like they were diving and we even saw some attempting to equalize the pressure in the ears as the virtual descent underwater begins. So an increase in familiarity with the VR headset technology and more familiarity with underwater experiences, may explain the differences we see between the two samples. The tourists from Gili Trawangan appear to be more sensitive to the differences between viewing a classic video using a VR headset and viewing a 360° video with a VR headset. This is reflected in the significant differences observed in donations on the extensive and intensive margins in the Gili Trawangan sample but not in the Bogor sample.
We do not find significant differences in donation behavior between positive and negative message framing when paired with visual immersion (i.e., VID_POS vs. VID_NEG or 360_POS vs. 360_NEG) in either of the sample populations. The differences in the framing were purposefully subtle focusing only on a portion of the script towards the end of the video. The majority of research focusing on message framing is limited to text and still visual images [38] with very few exceptions examining video messages [35,49,50]). Therefore, relatively little is known about how people process positively and negatively framed information in the context of moving images, and especially in the context of VR. A negative relationship between virtual reality and memory recall has been documented in other studies which was attributed to underlying mechanisms such as limited cognitive capacity and mediated arousal [51]. Advancing from written text alone to text with still images to moving images with audio already represents a considerable increase on one’s cognitive burden to process information. Adding yet another layer of immersion through the VR headset and increased field of vision to 360° may have resulted in less capacity to process the audio information, and later transform it into action, thereby lowering any effect of framing. The visual attention grabbing capabilities of immersive visual experiences may drain mental resources related to audio attention and mental processing. Human brains have limited cognitive resources to navigate through the world, including perceiving virtual experiences. According to a study by Lang [52] using the Limited Capacity Model of Motivated Mediated Message Processing, our brains are continuously allocating processing resources to encode, store, and retrieve information during mediated message use as a function of the structure, content, motivation, and personal relevance of the message. Put simply, media experiences that may strain cognitive resources (i.e., underwater scenes represent highly vivid and unusual imagery with constant movement in an alien environment) could mean that people hear the audio message, it triggers an emotional response, but they are unable to process the content of the information in such a way that leads to action. Our study shows that emotional reaction does not mean behavioral action. In the experiments presented in this paper, it is clear that participants heard the audio and it affected their feelings given that we do see differences in the emotional responses between the positive and negative framing, but this effect does not carry over into behavioral action in the form of increased donation amounts. This distinction between acknowledging the effect of an emotional appeal on one’s mood and acting upon it provides important evidence that an induced emotional state does not necessarily translate into behavioral action. Therefore, measuring only behavioral intentions is not sufficient to deduce behavior change. Perhaps pairing audio narration with written subtitles to reinforce the salience of the message may lead to behavior change.
Both positive and negative framing, and low and high visual immersion significantly affect emotional responses in comparison to no media communication. In Bogor, the video in all cases (VID and 360) heightened the awareness and concern about climate change and fostered higher donations. The tourist population in Gili Trawangan also reported increased emotional concern in all the treatment cases compared to the control. Increased emotions, however, did not necessarily translate symmetrically to action across the treatments. In all cases, those in the 360° treatments differed significantly in their donation behavior from the control. The framing had more of an effect on one’s emotional feeling but there was no difference between the positive and negative framing on how one perceived global climate change. In line with similarly framed experimental studies, we conclude that positive and negative frames can have systematic impacts upon both emotional responses and actual behavior but these are independent and not mutually exclusive [35,53]. There appear to be confounding factors that determine how framing plays a role and these are not well understood in a climate change context. Our study found mostly no differences in the framing except in one’s reported emotional state. It should be noted that studies that focus only on measuring emotion or intention as a proxy for action, may be overstating the behavioral impact of framing interventions and without additional content, changes in framing alone may not be effective at behavior change. And worse, focusing only on a particular type of framing ignores the nuances in the decision making process and between individuals that may require different types of messages targeted at different audiences to evoke emotion and change behavior. There are rarely ‘one-size fits all’ approaches to communicating complex issues such as climate change and messages should be tailored to fit the needs of the organization and the target audience. As noted by Lorenzoni, Nicholson-Cole, and Whitmarsh [4], considering climate change risk involves judgments about uncertain and complex science, potential future impacts, and personal perceptions and values. Therefore, some context is always necessary when presenting information about climate change [54], but a positive or negative message valence does not seem to make much of a difference in our study. At least in the case of tourists, we can say that the negative framing is most effective when combined with virtual reality technology.
The results of the present study hold a complex set of implications for climate communication—but ones that we believe are broadly in line with the existing wider literature on framing and virtual reality communication. Communicating the crisis of coral reefs using visual information is related to an increased sense of presence and emotion, with the highest scores in the 360° VR treatment. This study provides empirical support that the use of 360° video using VR head-mounted devices can have the potential to attract more people to donate, and increase donations. When trying to communicate important video messages while fundraising in public where there can be many distractions, our research would also support the use of immersive portable viewing devices such as VR headsets even with the use of classic unidirectional videos. The VR devices may attract people and the immersive and individual experience created by the device may be sufficient to drive donations even without the increased immersion created by a 360° viewing experience. Further, our study shows that some observations, but not all, from VR behavioral studies in the lab translate to behavior in the field and the same is true with lessons learned from message framing studies taken from other domains when translated into the context of climate change. Given the relative infancy of this realm of literature, it is necessary to continue to conduct research examining the effects of virtual reality technology on human behavior, as well as the impacts of positive and negative message valence in the context of climate change. This is especially important during a time when information sources on these subjects are dominated by popular media and anecdotal evidence as opposed to evidence-based research.
Supporting information
S3 File. Video script for “Coral Reefs: Life Below the Surface” https://www.youtube.com/watch?v=2TPG8lcfeDc.
https://doi.org/10.1371/journal.pone.0223631.s003
(DOCX)
Acknowledgments
First and foremost, we want to thank Claudia and Hendrik Schmitt from "The Jetlagged" for the outstanding underwater film production. We are grateful to the Institut Pertanian Bogor (IPB) and Dr. Luky Adrianto for helping facilitate this research. This research would not have been possible without the dedicated and hard-working master’s students from IPB and interns from Gili Eco Trust that worked as research assistants throughout this process. Special thanks to Delphine Robbe and Siân Williams of the Gili Eco Trust who provided their expertise and supported our activities to make this research possible.
References
- 1. Masson-Delmotte Valerie, Zhai P, Pörtner HO, Roberts D, Skea J, Shukla PR, Pirani Anna, Moufouma-Okia W, Péan C, and Pidcock R. 2018. "IPCC, 2018: Summary for Policymakers." Global Warming of 1.
- 2. Van der Linden Sander, Maibach Edward, and Leiserowitz Anthony. 2015. "Improving public engagement with climate change: Five “best practice” insights from psychological science." Perspectives on Psychological Science 10 (6):758–763. pmid:26581732
- 3. Moser , Susanne C, and Lisa Dilling. 2011. "Communicating climate change: closing the science-action gap." The Oxford handbook of climate change and society:161–174.
- 4. Lorenzoni Irene, Sophie Nicholson-Cole, and Lorraine Whitmarsh. 2007. "Barriers perceived to engaging with climate change among the UK public and their policy implications." Global environmental change 17 (3–4):445–459.
- 5. Nelson , Katherine M., Schlüter A, and Vance C. 2018a. "Distributional preferences and donation behavior among marine resource users in Wakatobi, Indonesia." Ocean & coastal management 162: 34–45.
- 6. Nelson , Katherine M., Schlüter A, and Vance C. 2018b. "Funding conservation locally: Insights from behavioral experiments in Indonesia." Conservation Letters 11, no. 2: e12378.
- 7.
Tversky Amos, and Kahneman Daniel. 1975. "Judgment under uncertainty: Heuristics and biases." In Utility, probability, and human decision making, 141–162. Springer.
- 8. Gifford Robert, and Louise A Comeau. 2011. "Message framing influences perceived climate change competence, engagement, and behavioral intentions." Global Environmental Change 21 (4):1301–1307.
- 9. Shin Donghee. 2018. "Empathy and embodied experience in virtual environment: To what extent can virtual reality stimulate empathy and embodied experience?" Computers in Human Behavior 78:64–73.
- 10. Innocenti Alessandro. 2017. "Virtual reality experiments in economics." Journal of behavioral and experimental economics 69:71–77.
- 11. Rosenberg , Robin S, Baughman Shawnee L, and Bailenson Jeremy N. 2013. "Virtual superheroes: Using superpowers in virtual reality to encourage prosocial behavior." PloS one 8 (1):e55003. pmid:23383029
- 12. Baberini M., Coleman C. L., Slovic P., & Västfjäll D. (2015). Examining the effects of photographic attributes on sympathy, emotions, and donation behavior. Visual communication quarterly, 22(2), 118–128.
- 13. Lowry B. 2019. 'Our Planet’ is visually stunning and deadly serious. Accessed April 8, 2019.
- 14. Diemer Julia, Alpers Georg W, Peperkorn Henrik M, Youssef Shiban, and Andreas Mühlberger. 2015. "The impact of perception and presence on emotional reactions: a review of research in virtual reality." Frontiers in psychology 6:26. pmid:25688218
- 15.
Schmitt, H., Schmitt, C., Nelson, K. 2017. Coral Reefs: Life Below the Surface. Youtube https://www.youtube.com/watch?v=2TPG8lcfeDc&t=64s:The Jetlagged.
- 16.
Samit, Jay. 2017. "How These Charities Are Using Virtual Reality to Reach Donors This Holiday Season." http://fortune.com/2017/11/21/virtual-reality-charities-donations/.
- 17.
West, Melanie. 2015. "Charities Use Virtual Reality to Draw in Donors ". https://www.wsj.com/articles/charities-use-virtual-reality-to-draw-in-donors-1448663492.
- 18. Chang Patricia. 2018. "Using Virtual Reality for Increased Charity Donor Outreach and Funding."
- 19.
Millar, Heather. 2016. "Can Virtual Reality Emerge as a Tool for Conservation?". https://www.theguardian.com/environment/2016/jun/28/can-virtual-reality-emerge-as-a-tool-for-conservation.
- 20. Kandaurova Maria. and Lee Seung Hwan Mark. 2019. The effects of Virtual Reality (VR) on charitable giving: The role of empathy, guilt, responsibility, and social exclusion. Journal of Business Research, 100, pp.571–580.
- 21.
Nielson. 2017. "Virtual empathy: How 360-Degree video can boost the efforts of non-profits." https://www.nielsen.com/us/en/insights/news/2017/how-360-degree-video-can-boost-the-efforts-of-non-profits.html.
- 22. Ahn Sun Joo Grace, Bostick Joshua, Ogle Elise, Nowak Kristine L, McGillicuddy Kara T, and Bailenson Jeremy N. 2016. "Experiencing nature: Embodying animals in immersive virtual environments increases inclusion of nature in self and involvement with nature." Journal of Computer-Mediated Communication 21 (6):399–419.
- 23. Ahn Sun Joo Grace, Bailenson Jeremy N, and Dooyeon Park. 2014. "Short-and long-term effects of embodied experiences in immersive virtual environments on environmental locus of control and behavior." Computers in Human Behavior 39:235–245.
- 24. Nelson K., Cepok J., Marnholz K., Arzaroli R., Grosse C., Zachmann G. 2019. "Effects of virtual reality on giving to charity." Experimental Economics for the Environment, Münster, Germany, February 25–26, 2019.
- 25. Ramirez Erick. 2017. "Empathy and the limits of thought experiments." Metaphilosophy 48 (4):504–526.
- 26. Gürerk Özgür, and Kasulke Alina. 2018. "Does virtual reality increase charitable giving? an experimental study." An Experimental Study (October 8, 2018).
- 27.
Hallahan, Kirk. 2008. "Strategic framing." The International Encyclopedia of Communication.
- 28. Avineri Erel, and Owen E Waygood D. 2013. "Applying valence framing to enhance the effect of information on transport-related carbon dioxide emissions." Transportation research part A: policy and practice 48:31–38.
- 29. Spence Alexa, and Pidgeon Nick. 2010. "Framing and communicating climate change: The effects of distance and outcome frame manipulations." Global Environmental Change 20 (4):656–667.
- 30. Jacobson Susan K., Morales Nia A., Chen Beida, Soodeen Rebecca, Moulton Michael P., and Jain Eakta. 2019. "Love or Loss: Effective message framing to promote environmental conservation." Applied Environmental Education & Communication 18(3): 252–265.
- 31. Cheng Tania, Woon Danielle Kathryn, and Lynes Jennifer K. 2011. "The use of message framing in the promotion of environmentally sustainable behaviors." Social Marketing Quarterly 17 (2):48–62.
- 32. Pounders, Kathrynn R., Lee Seungae, and Mackert Mike. 2015. "Matching temporal frame, self-view, and message frame valence: Improving persuasiveness in health communications." Journal of Advertising 44(4): 388–402.
- 33. Teng Lefa, Zhao Guangzhi, Wu Yuanyuan, Fu Hongyu, and Wang Jiajing. 2019. "Positive versus Negative Messaging In Discouraging Drunken Driving: Matching Behavior Consequences With Target Groups." Journal of Advertising Research 59(2): 185–195.
- 34. Gong Jingjing, Zhang Yan, Yang Zheng, Huang Yonghua, Feng Jun, and Zhang Weiwei. 2013. "The framing effect in medical decision-making: a review of the literature." Psychology, health & medicine 18 (6):645–653.
- 35. Rothman , Alexander J, Bartels Roger D, Wlaschin Jhon, and Salovey Peter. 2006. "The strategic use of gain-and loss-framed messages to promote healthy behavior: How theory can inform practice." Journal of communication 56 (suppl_1):S202–S220.
- 36. Pelletier , Luc G, and Elizabeth Sharp. 2008. "Persuasive communication and proenvironmental behaviours: How message tailoring and message framing can improve the integration of behaviours through self-determined motivation." Canadian Psychology/Psychologie Canadienne 49 (3):210.
- 37. Hardisty , David J, and Weber Elke U. 2009. "Discounting future green: money versus the environment." Journal of Experimental Psychology: General 138 (3):329.
- 38. O’Keefe Daniel J, and Jensen Jakob D. 2007. "The relative persuasiveness of gain-framed loss-framed messages for encouraging disease prevention behaviors: A meta-analytic review." Journal of health communication 12 (7):623–644. pmid:17934940
- 39. Moser , Susanne C, and Lisa Dilling. 2007. "Toward the social tipping point: Creating a climate for change." Creating a climate for change: Communicating climate change and facilitating social change:491–516.
- 40.
BPS. 2018. "Badan Pusat Statistik (Statistics Indonesia)." accessed January 2018. https://www.bps.go.id/.
- 41. Hughes , Terry P, Bellwood David R, and Connolly Sean R. 2002. "Biodiversity hotspots, centres of endemicity, and the conservation of coral reefs." Ecology Letters 5 (6):775–784.
- 42. Hoegh-Guldberg Ove. 2011. "Coral reef ecosystems and anthropogenic climate change." Regional Environmental Change 11 (1):215–227.
- 43. Hughes , Terry P, Baird Andrew H, Bellwood David R, Margaret Card, Connolly Sean R, Carl Folke, Richard Grosberg, Ove Hoegh-Guldberg, Jackson Jeremy BC, and Janice Kleypas. 2003. "Climate change, human impacts, and the resilience of coral reefs." science 301 (5635):929–933. pmid:12920289
- 44. Nelson , Katherine M., Partelow S, and Schlüter A. 2019. "Nudging tourists to donate for conservation: Experimental evidence on soliciting voluntary contributions for coastal management." Journal of environmental management 237: 30–43. pmid:30780053
- 45. Partelow Stefan, and Nelson Katherine. 2018. "Social networks, collective action and the evolution of governance for sustainable tourism on the Gili Islands, Indonesia." Marine Policy.
- 46. Halim Hengky Sumisto. 2017. "Scrutinizing Coastal Ecotourism in Gili Trawangan, Indonesia." International Journal of Marine Science 7.
- 47. Carpenter Jeffrey, and Matthews Peter Hans. 2017. "Using raffles to fund public goods: Lessons from a field experiment." Journal of Public Economics 150:30–38.
- 48. Goreau Thomas, and Prong Paulus. 2017. "Biorock electric reefs grow back severely eroded beaches in months." Journal of Marine Science and Engineering 5 (4):48.
- 49. Apanovitch , Anne Marie, McCarthy Danielle, and Salovey Peter. 2003. "Using message framing to motivate HIV testing among low-income, ethnic minority women." Health Psychology 22 (1):60. pmid:12558203
- 50. Schneider , Tamera R, Salovey Peter, Pallonen Unto, Mundorf Norbert, Smith Nelson F, and Steward Wayne T. 2001. "Visual and Auditory Message Framing Effects on Tobacco Smoking 1." Journal of Applied Social Psychology 31 (4):667–682.
- 51.
Bailey, Jakki, Jeremy N Bailenson, Andrea Stevenson Won, June Flora, and K Carrie Armel. 2012. "Presence and memory: immersive virtual reality effects on cued recall." Proceedings of the International Society for Presence Research Annual Conference, Philadelphia, PA.
- 52. Lang Annie. 2006. "Using the limited capacity model of motivated mediated message processing to design effective cancer communication messages." Journal of communication 56:S57–S80.
- 53.
Cepok, J., Marnholz, K., Arzaroli, R., Grosse, C., Nelson, K.M., Weller, R., Zachmann, G. (2020). In review. Effects of VR on Emotions and Behavioral Intentions. IEEE Virtual Reality Conference Proceedings.
- 54. Hulme Mike. 2008. "The conquering of climate: discourses of fear and their dissolution." Geographical Journal 174 (1):5–16.