Can’t simply roll it out: Evaluating a real-world virtual reality intervention to reduce driving under the influence

Driving under the influence (DUI) increases the risk of crashes. Emerging technologies, such as virtual reality (VR), represent potentially powerful and attractive tools for the prevention of risky behaviours, such as DUI. Therefore, they are embraced in prevention efforts with VR interventions primed to grow in popularity in near future. However, little is known about the actual effectiveness of such DUI-targeting VR interventions. To help fill the knowledge gap, this study explored the effects of one VR intervention as delivered in the real world. Using pre and post test design, including an intervention group (n = 98) and a control group (n = 39), the intervention evaluation examined young drivers’ (aged 18 to 25, no known history of DUI) intention and self-reported behaviour three months after the intervention as compared to the baseline. The results did not provide evidence for statistically significant effects of the VR intervention on self-reported DUI behaviour during the three months post intervention and DUI intention at three months post intervention. Such results might be due to the fact that the recruited participants generally self-reported little DUI behaviour, i.e. positively changing behaviour that is already positive is inherently challenging. Nevertheless, the results question the utility of funding the roll-out of arguably attractive technologies without a thorough understanding of their effectiveness in particular settings. To improve the potential for future positive outcomes of such interventions, we provide suggestions on how VR software might be further developed and, subsequently, leveraged in future research to improve the likelihood for behavioural change, e.g. by collecting, analysing and presenting objective driving performance data. Alternatively, future endeavours might focus on participants with known DUI history and examine the effects of the VR intervention for this particular higher-risk group.

"Road traffic injuries are the eighth leading cause of death globally, and the leading cause of death for young people aged 15-29." (WHO, 2015). The European Commission (EC) states "The young people, between 15 and 24 years old, make up 11% of the population but 17% of all road fatalities. This means that the young still face the largest risk in traffic." (See Table 1 Drink-driving is identified as a main contributor to 30% of the fatal and 9% of the non-fatal injuries (ATC, 2011). Still research shows that 7.8% of the young driver respondents had driven after drinking alcohol while 20% rode with a driver who had been drinking (CDCP, 2016). Drug-driving is a main behavioral factor in 7% of the fatal and 2% of the non-fatal crashes (ATC, 2011).
These findings lead the project partners to two conclusion: -Current actions play their role and need to stay in place; and -New actions need to be developed to support reaching the EU targets.
Such unacceptably high level of fatalities and injuries caused by completely preventable voluntary risky behaviour (young people's alcohol and drug use when driving) negatively influences many young people's daily lives in relation to health, education, employment and well-being. This is why one of the VAST objectives was achieving a real and measurable behavioural change in the young people's (aged 16-25, incl. ones with less opportunities) attitudes towards drink/drug-driving. The VAST project promoted cooperation between stakeholders with the goal to help young people (incl. ones with fewer opportunities) grow and play an active part in their society by making safer driving an integral part of their life. This is aligned with the objectives of the Europe 2020 strategy and its flagship initiatives. They aim at supporting the effective use of human and social capital by improving its health and providing opportunities for both youth workers and young people to acquire new skills, knowledge and experience through informal education, mobility and cooperation. The project supported the EU Youth Strategy in promoting dialogue between youth and policy makers. VAST increased active citizenship, fostered social integration, and ensured their inclusion in policy development. Through the planned activities, VAST explored synergies between the fields of informal education, training and youth. It provided a platform for exceptional civil society players from Argentina, Belgium, Bulgaria, the Netherlands and Romania to continue their outstanding activities and leading role in helping their countries reach targets in the fields of health, education and employment.

Project partners
VAST involved five project partners, four from the European Union and one from South America

Aims and objectives
VAST aimed to embrace the potential of next generation virtual reality technologies in an effort to address the overrepresentation of young people in road crashes.
An unfolded potential to build on the project partners' massive experience in deploying Information technologies through peer-to-peer communication strategies was identified at the beginning of VAST.

VAST Evaluation Plan
8 VAST used the advancement and availability of virtual reality experience as a leverage. A tailored International campaign specifically designed to engage and appeal to contemporary and future generations of drivers was designed to harvest the power of those technologies for social and public good.
In the broader perspective it is hoped that the project contributed to the participating countries reaching their targets of reducing road fatalities in relation to the European Road Safety Policy Orientations 2011-2020 and the UN "Decade of Action on Road Safety (2011-2020)".
The VAST objectives were to: 1. Achieve a real and measurable behavioral change in the young people's attitudes towards drink/drug-driving (incl. ones with less opportunities). The Joint International Campaign on Road Safety was geared to reach out to young people with social, economic and geographical difficulties. A robust psychological methodology was used to evaluate if actual behavioral change was achieved due to the campaign implementation.

Develop a higher level of cooperation between the partners through capacity building by:
-Going from cross-fertilization of experience to a joint peer-to-peer campaign. A Joint International Campaign on Road Safety was implemented and evaluated in each of the partner countries.
-Involving a substantial number of youth workers (two per partner) in the partners' meetings, the youth workers' mobilities and the European and South American Youth Road Safety Summits.
-Establishing grounds for significant conventional and social media presence. Youth workers and young volunteers (aged 18 to 25) were extensively trained in media relations through three different thematic workshops in each country. PR experts supported campaign implementations.

Directly involve a massive number of young people:
-25 young volunteers in trainings and implementing campaigns; -250 young people in the National Youth Summits on Road Safety; -3000 direct participants (600 per country) in the Joint International Campaign on Road Safety; -35 young people/youth workers participants in the European (24) and South American (11) Road Safety Summits.

Provide basis for increased involvement of road safety stakeholders and high-level lobbying for
better road safety policy development through: -Establishment of five Project Advisory Groups (PAG). Each PAG was expected to consist of 5 to 10 people, representing state or regional government institutions, academia, business and other road safety NGOs. At least two PAG meetings were to be held in each country.
-Involving recognised International speakers in the European and South American Youth Road Safety Summits. Each hosting partner had travel and subsistence budget to support the participation of two International experts (1 from the region and 1 from overseas) outside the project consortium.

Summary of findings
The VAST project succeeded in fulfilling its aims and objectives as described below. All country coordinators reported successful implementation of the Joint International Road Safety Campaign. The project foresaw 3,000 participants, 600 per country, to be directly involved. The target was overpassed with more than 50%, based on the reported 5,301 directly reached people. Additional 202,797 people were reached indirectly. Both numbers reveal that the partners' expected outreach was overpassed, too. Initially, they planned to reach a total of 2,950 people.
The campaigns were implemented by 20 youth workers and 31 young volunteers. The initial plan was to involve 52 youth workers and volunteers so the estimation was very close to the actual number. Thus the partners were able to get enough on the ground which ultimately translated into the better than initially planned outreach. It is an interesting observation that this success is not due to a much higher number of actions than initially expected. The partners planned to implement 89. However, the final number was 93.
It has to be noted that 30 working days were envisaged for tailored actions such as bringing the virtual reality simulator to local events, answering invitations from interested parties, and participation in school/university road safety days or implementing open days at the partners' own premises. However, there was no minimum or a maximum number of working days per action. Only one problem was reported during implementation and it was resolved in a due term.
Overall, the country coordinators reported a rating of 6.20/7.00 on their satisfaction with the implementation and a little lower 6.00/7.00 satisfaction, perceived in the directly reached participants.
Unfortunately, not all youth workers and volunteers, reported by the country coordinators provided their feedback. In total 45 did (18 youth workers and 27 volunteers, 22 females and 23 males, average age 24 years). They reported being directly involved in a total of 295 actions. The number is higher than the one reported by the country coordinators and the likely reason for that is an overlap, e.g. youth workers and volunteers engaging with the same participants.
It is interesting to see that youth workers and volunteers were less critical than the country coordinators and reported a rating of 6.60/7.00 on their satisfaction with the implementation and a little lower 6.47/7.00 satisfaction, perceived in the directly reached participants. Both ratings are overly positive and hint for a job well done by not only the country coordinators but by the youth workers and volunteers themselves. 75% thought the campaign will help make the young people safer traffic participants. 82% thought the campaign could not have been carried out with similar success but without the Virtual reality equipment. The overall rating of the impact of the virtual reality simulation on the involved participants was 6.49/7.00. These very high ratings and confirmation of impact from the directly involved people were confirmed at large by the project external audience during the two implemented International Summits. The VAST objectives were rated as clear with 4.55/5.00 in Buenos Aires and 4.30/5.00 in Brussels. They were also rated as achievable with 4.38/5.00 in Buenos Aires and 4.27/5.00 Brussels. The project working steps were rated as clear with 4.45/5.00 in Buenos Aires, and 4.30/5.00 in Brussels. Conference participants expected it would produce a lasting change in the participating young people with 4.49/5.00 in Buenos Aires and 3.76/5.00 in Brussels. VAST received an overall rating in respect of its design, implementation, impact, sustainability of 4.18/5.00 in Buenos Aires and 3.72/5.00 in Brussels. 92% of the Summits' participants said it was successfully implemented. The other 8% thought the success was to a certain extent.
The project expected results, mentioned in the project proposals, were: 1. Achieve a real and measurable behavioural change in the young people's attitudes towards drink/drug-driving (incl. ones with less opportunities).
It was not possible to test for such an effect due to the lack of quality data, required by the intended statistical tests.

Develop and implement a Joint International Campaign on Road Safety.
Accomplished! The campaign was implemented by the project partners in Argentina, Belgium, Bulgaria, the Netherlands and Romania.

Involving a substantial number of youth workers (two per partner) in the partners' meetings, the youth workers' mobilities and the European and South American Youth Road Safety Summits.
Accomplished! Eight youth workers provided feedback as being involved in the youth workers' mobilities in Belgium, 7 -in Argentina.

Train 40 youth workers and young volunteers (aged 18 to 25) in media relations through three different thematic workshops in each country.
Accomplished! 49 people took part in the workshops (43 in "Media relations", 41 in "Social media management" and 44 in "Public relations and event management").

Directly involve 25 young volunteers in trainings and implementing campaigns.
Accomplished! 31 volunteer were trained in the media workshops. 27 took part in implementing campaigns.

Directly involve 250 young people in the National Youth Summits on Road Safety.
Accomplished! 273 people provided feedback after the National Youth Summits on Road Safety.

Directly involve 3000 direct participants (600 per country) in the Joint International Campaign on Road Safety.
Accomplished! 5,301 people were reached directly through the campaigns.

American (11) Road Safety Summits.
Accomplished! 37 of the delegates provided feedback after the European and 53 after the South American Road Safety Summit.

Involve between 25 and 50 road safety stakeholders in the Project Advisory Groups (PAG).
Accomplished! 48 stakeholders took part in the first PAG meeting, 38 -in the second one.

Involve recognised International speakers in the European and South American Road Safety
Summits.

General description
iRED was subcontracted as an independent external project evaluator for the implementation of an integrated evaluation system that: -analysed the level of implementation of the working programme; -verified the achievement level of activities that could affect the implementation of each subsequent activity; -collected information about the effectiveness of the work in each partner's context.
The current final evaluation report was developed based on the findings. A special focus of the evaluation is the long term effect of the campaign on the direct young participants in regards to assessing any shifts in their attitudes towards the risky behaviour of drink/drug driving. Psychological models were used to measure the success of the International Road Safety Campaign in regards to motivating safer behaviour in the campaign participants. The Theory of Planned Behaviour (TPB) (Ajzen, 1985(Ajzen, , 1987(Ajzen, , 1991 was selected as theoretical underpinning. Measures were taken before and three months post participation. The participants were incentivised to participate in the data collection before their participation by connecting the use of the virtual simulator with completing a survey. The follow-up survey did not offer incentives. A social media campaign was planned to be implemented at the beginning of the project in order to collect data from the general public through a survey, effectively establishing a baseline of a control group. The survey respondents were also contacted at the end of the campaign for second data collection. This was used to account for general shifts in people's attitudes due factors external to VAST.

Project activities
VAST facilitated adoption of innovation in every day prevention activities. It built on the partner consortium previous work, already done in putting together the project application, by adapting a proper implementation methodology for achieving the project objectives (See Table 5.1). The consortium implemented the following activities:

Establishing the project context for the innovation (Activity Leader (AL): RYD NL).
The project partners sent two representative each to elaborate on the needs analysis, already done in preparing the project proposal, going into deeper details. In order to properly build the scope of the VAST Joint International Road Safety Campaign, the partners reviewed the "3D Tripping" Dutch experience during the project kickoff meeting, which was carried out in Amsterdam, the Netherlands.

Media capacities development (AL: NCSD).
Each partner selected a group of high-potential volunteer multiplies to undergo a set of capacity-building workshops at youth workers' level together with the partners youth workers on the topics of "Media relations", "Social media management" and "Public relations and event management".

Virtual simulator assembly (AL: RYD NL).
Instead of buying an off-the-shelf simulator and software, which is very expensive, the project partners bought the needed parts (powerful laptop, driving seat console, G29 steering wheel, Oculus rift goggles and TV) and assembled the machine. RYD NL offered the simulation software for free to the partnership.

Establishment of Project advisory groups (AL: NCSD).
Each partner established a Project advisory group (PAG) of road safety stakeholders, such as representatives of state or regional government institutions, academia, business and other road safety NGOs. The VAST project was presented for discussion and suggestions for implementation on local level to the PAGs' first meetings. The subsequent National Youth Summits on Road Safety and Joint International Road Safety Campaign were introduced to the audience.

National Youth Summits on Road Safety (AL: PIHE).
In an effort to prepare the public for the implementation of the Joint International Road Safety Campaign national large scale youth events were carried out in cooperation with the PAG members in the form of National Youth Summits on Road Safety. They were organized by the project young road safety volunteers and were open for active young people and youth workers who are interested in road safety. During the Summits, the project was presented to the general public and the media as well as relevant road safety issues weree discussed. The subsequent Joint International Road Safety Campaign was introduced together with the opportunities it was providing.

Implementation of the Joint International Road Safety Campaign (AL: RYD BE).
A joint road safety intervention was carried out by the five partners through a series of activities effectively representing local actions with the new virtual reality simulators. The focus was on drink/drug-driving prevention for young people through a powerful education method using virtual reality. In one-to-one peer sessions the campaign offered to the young people to experience "what it's like to drive a car whilst drunk, on Ecstasy, high on Cannabis and tripping on Magic Mushrooms" in a safe environment. Thus non-formal learning through selfexperience, self-reflexion and peer discussions was encouraged. The campaigns were youth-led by the volunteers, trained in the media workshops, and supported by youth workers from the partner organizations.
Each partner developed its own version of the campaign that best met local needs as defined by the PAGs and the National Youth Summits on Road Safety. Information material and give-aways were tailored to the local actions.

Second Project advisory groups' meetings (AL: PIHE).
A second set of PAG meeting were carried out to share the campaign achievements and collect constructive feedback. During the meetings the results of the Joint International Road Safety Campaigns were presented. The subsequent European and South American Youth Summits on Road Safety were introduced.

European Road Safety Summit (AL: RYD BE). A European Road Safety Youth Summit was
implemented to disseminate the project experience from the PAGs meetings, the National Youth Summits on Road Safety and the Joint International Road Safety Campaign in Europe. To enlarge the added value for the project partners, each was represented by two youth workers and their participation took the form of two-week youth workers mobility in support of the European Summit implementation for RYD BE. Young people and youth workers with interest in road safety youth work from European countries were invited to participate. The summit also featured road safety best practices from South America.

South American Road Safety Summit (AL: MiNU). A South American Road Safety Youth Summit was
implemented to disseminate the project experience from the PAGs meetings, the National Youth Summits on Road Safety and the Joint International Road Safety Campaign in South America. To enlarge the added value for the project partners, each was represented by two youth workers and their participation took the form of two-week youth workers mobility in support of the South American Summit implementation for MiNU. Young people and youth workers with interest in road safety youth work from South American countries were invited to participate. The summit also featured road safety best practices from the European Union.

Final project meeting (AL: MiNU). A Final project meeting was held in Buenos Aires after the South
American Road Safety Youth Summit during the youth workers' mobilities. During it besides purely technical tasks, such as preparation of final reports, the partners worked on preparing the project results and findings for international transferability by providing grounds for open access to its achievements and the results from its evaluation.

Project evaluation (AL: External).
iRED was subcontracted as an independent project evaluator.

Tasks and measurable indicators
The following tasks were initially set for the project: -Reach 10 000 people through information materials and 500 000 through conventional and social media.

Overview of VAST evaluation
The evaluation of VAST adopted a formative evaluation approach for the preparation and dissemination activities of the project and a summative approach for the implementation activities as demonstrated in  The formative evaluation set outs a number of objectives to be achieved. Initially information was about the process of developing the Joint International Road Safety Campaign. Subsequently details were collected from each partner country regarding the implementation of the campaign on national level which was followed by an analysis to determine the long-term effect of the campaign on the involved participants.
Following this, the evaluation aimed to verify the variables that affected the implementation and these shall serve to assist future endeavours. Finally, the evaluation set out to collect information about the effectiveness of the intervention in the different countries involved. It should be noted that the geographical location of the partner countries (Europe and South America) provided a wide coverage and hence allowed for a useful comparison to be made between different jurisdictions.
The main tool of the formative evaluation was implementation activities questionnaires which underlined the starting point and achievements of the project in relation to the target groups. The collected data was This final evaluation includes assessing the project impact in each partner country. The evaluation shows the overall project achievements and can be used for dissemination activities on an International level.
The evaluation of VAST is conducted throughout the duration of the project (See Table 5.2) which is scheduled to run from 1 st of July 2017 to 31 st of December 2018. It was carried out consecutively as follows: •   Table 6.1 gives an overview of the questionnaires which were used during the first evaluation phase. returned completed questionnaires (3 from Argentina, 2 from Belgium, 2 from Bulgaria, 2 from the Netherlands and 2 from Romania). 10 were males and 1 was female. The average age was 34 years. Table 6.2 the average score of the questions asked in regards to the clarity of the project is presented: Average score values were skewed to the positive side of the scale. The issues regarding the objectives of the project, the participants` role in the project, envisaged working steps and the timeframe were completely clear.

Evaluation 1: Preparation activities
The last questions explored whether the participants received enough information and experience in regards to the 3D virtual simulator. All participants said definite "Yes" providing the last piece of information that the Kick-off meeting had fulfilled its purpose.

QUE 2: Questionnaire at the end of the Media capacities workshop series
During the preparation phase of the VAST project a series of capacity-building workshops on the topics of "Media relations", "Social media management" and "Public relations and event management" was organised by each partner in their respective country. During the sessions the participating youth workers and volunteers developed their own traffic safety campaign.
The participants of the workshops (youth workers and volunteers) were asked to fill in a questionnaire after the series was completed. A total of 49 people took part in the workshops (43 in "Media relations", 41 in "Social media management" and 44 in "Public relations and event management"). 40 were foreseen in the project application. See Figure 6.1 for a breakdown per country and role in the project. The average age of the participants was 25 years. 30 of them were male, 18 were female and 1 preferred not to say. 27 reported to be currently occupied as employees, 12 as students, 8 as school pupils, 1 as a volunteer and 1 as an NGO Board member. They heard about the project and the workshops through their NGO (42 people), friends/acquaintances/fellow students (21 people), university/school (6 people) and by e-mail (5 people). The numbers add to more than 49 possibly because people received information from multiple sources. Figure 6.2 reveals their motivation to join the project. The second part of the questionnaire helped us learn about the workshops' participants in relation to the broad theme of VAST, i.e. road safety. Figure 6.3 casts light on their modes of transport. The collected data revealed that most of the participants were new to volunteering in a peer-to-peer campaigns. 37% reported as having experience in that. Even less, 27%, were experienced specifically in road safety campaigns. Nevertheless, the participants 1) consider themselves as being safe road users with an average rating of 4.18/5.00 and 2) they reported road safety to be a common topic of conversation. Figure   6.4 shows who the participants discuss risky behaviour in public traffic with. The final set of questions explored the added value of the workshops. 98% reported that they have learned something new during the workshops. The overall satisfaction with the experience was rated at 4.27/5.00.
Expectations were met with a rating of 4.25/5.00. When an opportunity to reflect on which part of the workshops they found most beneficial and why, the participants provided a multitude of answers, revealing the timeliness and added value of the activity. Here are some examples: -How to behave in front of camera and microphone.
-Everything was useful, because i did not work with media before. considered the provided material as sufficient. Table 6.3 presents the clarity of the future work post the workshops participation. Despite the high ratings, the participants anticipated potential problems with most of them being related to overestimating the possible media reach and not reaching project goals. Proposed solutions spanned from setting reasonable goals through constant progress monitoring to increasing efforts.

QUE 3.1: Questionnaire at the first Project advisory groups meetings
As per the project timeline each partner established a Project advisory group (PAG) of road safety stakeholders. 48 PAG members, almost double the planned 25, submitted feedback forms after their first meeting. For breakdown by country see Figure 6.5.  Table 6.4 reveals the average scores on the questions the PAG memebers were asked in regards to the clarity of the project.  The only somewhat problematic area was the role of the PAG members in the project. Nevertheless, 40 PAG members stated that they were interested in following the projects during its full term and that they would be able to contribute to achieving its objectives. 29 of them committed to participating in a second Project advisory group meeting in 2018.

QUE 4: Questionnaire at the National Youth Summits on Road Safety
VAST was introduced to the local communities through National Youth Summits on Road Safety. They were organized by the project young road safety volunteers and were opened for active young people and youth workers who are interested in road safety. A common questionnaire was developed to collect the feedback of the participants in the Summits. It was translated in local languages to increase the likelihood for completion. Table 6.5 provides details on the provided feedback. With 273 collected feedback forms, the objective of involving 250 participants in National Youth Summits on Road Safety was achieved by the project partners. Furthermore, Table 6.5 reveals good gender balance as well as an average age in line with the intention to involve young people. It also shows that the Summits fulfilled their purpose. The project objectives were clear, also achievable, as were the project working steps with ratings above 4 for all countries with the exception of the Netherlands. There is also positive feeling about the added value the project will bring in regards to driving under the influence of drugs and/or alcohol, although the values are somewhat lower in Belgium and the Netherlands. Table 7.1 gives an overview of the questionnaires which were used during the second evaluation phase.

7.1.QUE 5: Questionnaire regarding the national implementation plans of the Joint Campaign
The country coordinators received a questionnaire with the request to complete it online after the planning of the implementation of the Joint International Road Safety Campaign. In the following Table 7.2, a summary of the results based on the questionnaire is presented.

7.2.QUE 6: Questionnaire regarding the national implementation of the Joint Campaign
The country coordinators received a questionnaire with the request to complete it online after the implementation of the campaign. The first part of the questionnaire focused on the campaigns implementation specifically while the second one looked into the dissemination activities that accompanied it. In the following Tables 7.3 and 7.4, a summary of the results is presented. Most of the partners did not report any problems during the implementation of the campaign. Only in Romania lack of internet access which made it difficult to collect online evaluation forms was reported. The implemented solution was to use printed forms.
Dissemination activities were an important facet of VAST which is why the media capacity workshops were implemented at the beginning of the project. All countries were expected to use a variety of dissemination materials which is why the second part of the post-campaign questionnaire to the country coordinators focused on them. The country coordinators were asked to reflect on their dissemination activities with the request to provide data online. In the following Table 7.5, a summary of the results based on questionnaires is presented. All partner countries provided information on their dissemination activities. Dissemination materials were used in all countries with 3,550 people being reached through them. The country coordinators rated the added value of the budget for social media at 5.6/7.0 and reported 12,943 people reached with social media.
The added value of the budget for the external PR experts was rated at 6.0/7.0 by the two partners that reported using it. The following results were reported for traditional media: Overall, their reported satisfaction with the implementation is 6.82/7.00. They also perceived very high satisfaction on behalf of the involved participants, 6.50/7.00. 13 of them believed that the campaign will help make the young people safer traffic participants, 3 thought it may do that. 13 of them believed that the campaign could not have been carried out with similar success but without the Virtual reality equipment, 3 thought it may have and 1 that it could have. The impact of the virtual reality simulation on the involved participants was rated at 6.69/7.00.
One youth worker reported encountered problems as well as implemented solutions: It was difficult to get all the people to fill the questionaries. Solution: We made more implementations.

Belgium
8 people, 4 youth workers and 4 volunteers, provided their feedback in regards to the implementation in Belgium. 4 of them were females, 4 -males. The average age of the youth workers is 25 years, the one of the volunteers -22 years. The youth workers and volunteers reported engaging in 2 actions on average or Overall, their reported satisfaction with the implementation is 6.25/7.00. They also perceived very high satisfaction on behalf of the involved participants, 6.38/7.00. Half of them believed that the campaign will help make the young people safer traffic participants, the other half thought it may do that. None of them believed that the campaign could not have been carried out with similar success but without the Virtual reality equipment. The impact of the virtual reality simulation on the involved participants was rated at 6.63/7.00.
No problems with the implementation were reported. Overall, their reported satisfaction with the implementation is 6.89/7.00. They also perceived very high satisfaction on behalf of the involved participants, 6.63/7.00. All of them believed that the campaign will help make the young people safer traffic participants and that it could not have been carried out with similar success but without the Virtual reality equipment. The impact of the virtual reality simulation on the involved participants was rated at 6.50/7.00.

Bulgaria
No problems with the implementation were reported. Overall, their reported satisfaction with the implementation is 6.13/7.00. They also perceived a high satisfaction on behalf of the involved participants, 6.25/7.00. Half of them believed that the campaign will help make the young people safer traffic participants, the other half thought it may do that. None of them believed that the campaign could not have been carried out with similar success but without the Virtual reality equipment. The impact of the virtual reality simulation on the involved participants was rated at 6.25/7.00.

Netherlands
No problems with the implementation were reported.

Romania
5 people, 2 youth workers and 3 volunteers, provided their feedback in regards to the implementation in Romania. 2 of them were females, 3 -males. The average age of the youth workers was 37 years, the one of the volunteers -22 years. The youth workers and volunteers reported engaging in 5 actions on average or 26 in total.
Overall, their reported satisfaction with the implementation is 6.80/7.00. They also perceived a very high satisfaction on behalf of the involved participants, 6.60/7.00. All of them believed that the campaign will help make the young people safer traffic participants. One of them believed that the campaign could not have been carried out with similar success but without the Virtual reality equipment, two through it could and the other two throught it might. The impact of the virtual reality simulation on the involved participants was rated at 6.00/7.00.
One youth worker and one volunteer reported encountered problems. Both reported the problems with filling in questioners with participants. Solution: We used printed sheets than bought a tablet.

7.4.QUE 8: Questionnaire regarding the national implementation of the Joint Campaign
A special focus of the evaluation is the long term effect of the campaign on the direct young participants in regards to assessing any shifts in their attitudes towards the risky behaviour of drink/drug driving.
Psychological models were used to measure the success of the International Road Safety Campaign in regards to motivating safer behaviour in the campaign participants. The Theory of Planned Behaviour (TPB) (Ajzen, 1985(Ajzen, , 1987(Ajzen, , 1991 was selected as theoretical underpinning.

Theory of the Planned Behaviour
In psychology, the Theory of planned behaviour (See Figure 7.1) is a theory that links one's beliefs and behaviour. TPB states that a person's attitude toward a particular behaviour, their subjective norms, and perceived behavioural control, together shape their behavioural intentions and behaviours. The concept was proposed by Icek Ajzen (1985,1987,1991) to improve on the predictive power of the Theory of reasoned action by including perceived behavioural control. It has been applied to studies of the relations among beliefs, attitudes, behavioural intentions and behaviours in various fields such as advertising, public relations, advertising campaigns, healthcare, sport management and sustainability. This TPB was considered a good fit for VAST as the project targets drink-and drug-driving behaviour of young people and, on a higher level their intention, to perform it. As it is highly unlikely that many young people intend to drive under the influence of drugs or alcohol and even less likely to actually do so, VAST aimed at influencing the participants' attitude towards the behaviour.

Evaluation design
The intervention was designed as a randomised controlled trial (RCT). RTC is a trial in which subjects are randomly assigned to one of two conditions: one (intervention) in our case directly participating in the VAST campaign, and the other (control) which does not experience the virtual simulation. The two groups were to be followed up three months after the VAST campaign to see if there are any differences between them in taken measures. Measures were intended to be taken before and approximately three months post participation (See Table 7.6).

Recruitment of participants
Participants in the intervention group were recruited face-to-face by the project partner during the implemented VAST actions (See Table 7.7). They were incentivised to participate in the data collection before their participation by connecting the use of the virtual simulator with completing a survey. The follow-up survey did not offer incentives. A social media campaign was planned to be implemented at the beginning of the project in order to collect data from the general public through a survey, effectively establishing a baseline of a control group. The survey respondents were also contacted at the end of the campaign for second data collection. The Control group data was expected to be used to account for general shifts in people's attitudes due factors external to VAST. Even the data from Bulgaria had problematic anonymous identifiers which hampered the intended analysis on top of the low numbers.
By all means, a quality set of post-intervention data from both the participants in the interventions and the Control groups was needed to complete the evaluation of the impact the project had on participants as originally intended. The lack of such data made performing the analysis impossible. Table 8.1 gives an overview of the questionnaires which were used during the third evaluation phase.

QUE 3.2: Questionnaire at the second Project advisory groups meetings
The VAST project envisaged a second set of PAG meeting to be carried out upon completion of the Joint International Road Safety Campaign to share its achievements and to collect constructive feedback. The subsequent European and South American Youth Summits on Road Safety would had been introduced.
After gathering for a second time PAG road safety stakeholders were asked to complete an online questionnaire.
A total of 38 PAG members submitted feedback forms after their second meeting. This kept the number of active road safety stakeholders within VAST above the planned 25. For breakdown by country see Figure   8.1.   The provided answers in regards to the project are similar to the ones given during the first PAG meeting and show consistent understanding of VAST. At this stage even a bigger percentage (87% in comparison with 83 during the first meeting) stated that they were interested in following the projects during its full term. It has to be noted though that the number of total people involved is less. 33 PAG members said they would be able to further contribute to achieving the VAST objectives. Probably the most important outcome of the PAGS is that 21 members committed to continue working as part of the Project advisory group beyond the project completion in December 2018.

QUE 9: Questionnaire at youth workers' mobility
Two two-week youth workers' mobilities took place in the VAST framework. To enlarge the added value for the project partners, each was represented by two youth workers and their participation took place in parallel with the European Youth Summit, organised by RYD BE, and the South American Youth Summit, organised by MiNU. The first youth workers' mobility took place in Brussels, Belgium, between 13 and26.09.2018. The second youth workers' mobility took place in Buenos Aires, Argentina, between 12 and 25.11.2018. Only the visiting youth workers' opinions were surveyed because the project does not subsidise host youth workers. They were asked questions in regards to the project in general and the youth workers' mobilities as a tool. 8 out of 8 participants completed the survey for the first mobility, 7 out of 8 for the second. Table 8.3 shows the collected results: It is encouraging to see that the project design remained relevant throughout the project preparatory activities. The project partners' representatives were quite clear and very positive about the project process and the intended results.
The quality of the youth workers' mobilities was reported to be very high in any aspect with average scores very close or equal to the maximum possible 5. All respondents confirmed their appropriateness to be used as a framework for partners' meeting. Nevertheless, two participants shared doubt that the mobilities add value beyond the meetings, i.e. to the hosting partner or to the VAST conferences. This is an issue that would require in future endeavours increased effort on behalf of the project consortium to harvest the potential of delivering added value. However, the overall notion was positive in those aspects as well. A future evaluator should also keep in mind to look for underlying reasons for other-than-positive feedback to be able to provide better recommendations. Despite being given the opportunity to elaborate further, the two participants did not share details.

QUE 10: Questionnaire at the International conferences
The VAST Road Safety Summits took place in: The participants were asked four questions related to the VAST project on a 1 to 5 Likert scale. All ratings were highly positive and above 4 with the exception of the potential for a long-term impact of the project.
The last received a 3.75/5.00 rating which nevertheless is still very positive reflection on the project's added value. The average scores are presented in Table 8.4. Do you think that such projects will produce a lasting change in the participating young people attitudes and/or behaviour on the road in regards to drink/drug-driving?

3.76
Finally, the participants were asked to rate VAST overall in respect of its design, implementation, impact, sustainability. Again, the project received a positive rating of 3.72/5.00 which underlines its inherent qualities, something the project partners may wish to keep in future follow-ups. The participants were also asked to reflect specifically on the Joint International Road Safety Campaign implementation whether it was or was not successfully implemented. All participants pointed out its successful implementation.

Buenos Aires, Argentina
The second VAST Road Safety Summit was planned to take place in South America. Similar to the European one the intention was to disseminate the project experience from the PAGs meetings, the National Youth Summits on Road Safety and the Joint International Road Safety Campaign in South America. To enlarge the added value for the project partners, their representatives participated in the South American Summit through a two-week youth workers mobility, hosted by MiNU. Stakeholders, young people and youth workers with interest in road safety youth work from South American countries were invited to participate. The summit also featured road safety best practices from the European Union.
100+ delegates attended the Summit. All VAST partners were represented. 53 of the delegates (22 male and 31 female) provided feedback. The average reported age was 37 years.
The participants were asked the same four questions related to the VAST project on a 1 to 5 Likert scale as in the case of the European Summit. Again, all ratings were highly positive and above 4, a very positive reflection on the project added value. The average scores are presented in Table 8.5.

Question Result
Are the objectives of the VAST project clear? 4.55 In your opinion, were the objectives achieved? 4.38 Are the working steps of the project clear? 4.45 Do you think that such projects will produce a lasting change in the participating young people attitudes and/or behaviour on the road in regards to drink/drug-driving?

4.49
The participants were asked to rate VAST overall in respect of its design, implementation, impact, sustainability. The project received a very positive rating of 4.18/5.00 underlining its inherent qualities in the case of South America, too. The participants were also asked to reflect specifically on the Joint International Road Safety Campaign implementation whether it was or was not successfully implemented.
46 participants pointed out its successful implementation. 7 said it was successfully implemented to a certain extent.

Conclusions and suggestions
The targets set by the European Road Safety Policy Orientations 2011-2020 and the UN "Decade of Action on Road Safety (2011-2020) of reducing road fatalities seem less and less likely to be reached. Technology gives hope for a higher outreach at a lower cost in prevention activities.
The VAST project partners possess massive experience in deploying Information technologies through peer-to-peer communication strategies. Through the project they put that experience to work and embraced the potential of next generation virtual reality technologies in an effort to address the overrepresentation of young people in road crashes. A tailored International campaign specifically designed to engage and appeal to contemporary and future generations of drivers was implemented to harvest the power of those technologies for social and public good.
The VAST country coordinators reported a successful implementation of the Joint International Road Safety Campaign in 93 separate actions by 20 youth workers and 31 young volunteers. The implementation was rated at 6.20/7.00 by the country coordinators and at 6.60/7.00 by the youth workers and the young volunteers. The perceived satisfaction of the involved participants was 6.00/7.00 and 6.47/7.00 by the country coordinators and the youth workers and the young volunteers respectively. Unfortunately, due to low quality data, the effect on the participants themselves was not evaluated. This is where the main recommendation for continuing that excellent initiative comes. In future projects, if a second set of data is required from the participants, it may be more practical to look into the short-term effect of the campaign rather than the long-term one. Although, investigation of the long-term effect seems to be more valuable, VAST showed that in this specific case it was not achievable. On the other hand, while the participants were still on the campaign implementation site, they could had been asked to complete a second survey immediately after driving the VR simulator. It has to be acknowledged that such an approach was no part of the current evaluation design. Nevertheless, it is recommended to be used by evaluators of future similar initiatives.
Despite this shortcoming, the VAST reached 5,301 people directly and 202,797 people indirectly, numbers much higher than the initially planned ones. The overall rating of the impact of the virtual reality simulation on the involved participants was perceived at 6.49/7.00.
This successes could also be attributed partially to the high involvement of external stakeholders in VAST.
38 road safety stakeholders participated in the Project Advisory Groups from the project beginning through to its completion. Even more were involved in the project through the National Youth Summits on Road Safety (273 people), European Road Safety Summit (37 people) and the South American Road Safety Summit (53 people). VAST received an overall rating in respect of its design, implementation, impact, sustainability of 4.18/5.00 in Buenos Aires and 3.72/5.00 in Brussels with 92% of the International Summits' participants confirming its successful implementation.
Additionally the general public was well informed about this success with 3,550 people receiving dissemination materials in person, 12,943 people reached through social media and 183,098 people reached through conventional media.
It shall be expected that with the experience gained in VAST the project partners will be able to mainsteam the developed campaign into their regular activities. It may also be expected that the participating organizations are ready to build on their new capacity and follow-up VAST through larger joint activities.
Thus, on top of the main recommendation descussed above in respect to collecting quality data for evaluating the impact of the project on the directly involved participants, some specific suggestions for futher work can be made: 1. Despite the implemented youth workers mobilities and their added value in VAST, no similar activities were planned or implemented on International level for the benefit of the young volunteers involved. A youth exchange may well suit the purpose in future projects. meet together, for example in an International road safety forum. This will provide an opportunity for sustainability.
3. A follow-up campaign implementation should be planned nationally if not Internationally not only to establish the VAST campaign but also to try to involve additional stakeholders, e.g. invite active road safety stakeholders from other countries to implement the campaign outside the current VAST geographical outreach. It would be a waste of involvement, knowledge and experience, if the acquired new capacity is not put to work in the future and subsequently expanded upon.
With the above listed suggestions in mind, this evaluation can be concluded with the following statements regarding the VAST project:  VAST achieved its aims and objectives with one exception which we were unable to test for.
 The project methodology was found feasible and capable to contribute to road safety.
 The target group was appropriate and well defined.
 The campaigns effectively reached the target group.
 The project was implemented as planned.