2.1. Sociotechnical imaginaries on autonomous vehicles

Studies agree on the importance of sociotechnical imaginaries – analyzing values and norms as means for better understanding the accessibility to the infrastructure in the smart cities [13]. However, biased images of urban processes lean towards corporate and instrumental visions of the city rather than including and reflecting public values and citizens’ interests [14]. Therefore, it is necessary to better understand the peoples’ perceptions to reveal the citizens’ interests in smart city data-based developments.

This article uses the definition of sociotechnical imaginaries by Jasanoff [2] who takes into consideration the complexities of future vision attained through science and technology, “collectively held, institutionally stabilized, and publicly performed visions of desirable futures, animated by shared understandings of forms of social life and social order attainable through, and supportive of, advances in science and technology” [2, p.6]. Therefore, sociotechnical imaginaries allow us to better understand the interplay of technology and the societal aspects – potential consequences, benefits, and risks through the perceptions of the individuals.

Prior studies on the socio-technical imaginaries have emphasized that algorithms used in automated decisions often tend and are seen to disregard data diversity as social sorting is quite prominent in algorithmic selections. For example, some studies examined biases and inequalities of algorithmic calculations [15] by foremost exploring how vulnerable social groups are neglected by algorithms, like ethnic minorities, older people, and people with disabilities, and the intersections of these [16]. It is argued that technologies that rely on data to make automated decisions can yield biased and discriminatory results [17]. However, the bias in data is not generated by the discrimination that exists in society, as those data can be filtered, but rather from the intrinsic characteristics of statistical algorithms.

There is a wide array of studies about the imaginaries and techno-visions of autonomous vehicles, such as getting information about real responses through simulated worlds [18], or using controversies to help people identify their own issues and imagine alternative possibilities [8]. In literature, it is argued that the imagination of autonomous vehicles is determined by the moral beliefs of one’s society or culture based on what is morally right, wrong, good, bad, etc. [19]. Therefore, prior studies on the imaginaries have mostly focused on the ethical and moral dilemmas, like studying the evolution of moral and ethical choices [20], the ethics of accidents with autonomous vehicles, and the moral choices in designing and implementing such algorithmic solutions [19], and individuals being often powerless in datafication and developing data-based smart cities [1].

Other studies have explored the perceptions of algorithmic threats across socio-demographic variables [21,22], including age – younger people being more tolerant, whereas gender and education did not play a significant role, gender biases and stereotypes [20]. A study conducted by Nagtegaal [23] found that people perceive human decisions as fairer in high-complexity tasks, while favoring algorithmic ones in low-complexity tasks. On the other hand, Araujo et al. [10] found a preference for algorithmic decisions in high-stakes health and justice decisions. Another study found that people assigned less responsibility to self-driving cars at fault compared to human drivers, potentially influenced by media promoting AVs as solutions to current mobility challenges [11,12]. It is argued that such sociotechnical imaginaries create a future-perfect narrative, sidestepping critical questions about the social impacts of disruptive technologies [24].

2.2. Investigating sociotechnical imaginaries on autonomous vehicles

Several research methods have been used to measure people’s perceptions of algorithms, including surveys [10,22], interviews [25], and the Technology Acceptance Model [26], which rely on explicit data —self-reported information that reflects conscious thoughts and attitudes— that may sometimes be insufficient. While these approaches offer advantages, such as capturing subjective views based on life experiences, they can be imprecise because there may be a discrepancy between what people say and what they do (for real-time vs. retrospective, subjective measurements, see [27,28]). Cognitive methods like eye-tracking and EEG offer a significant complement to traditional approaches by providing precise data on visual attention and uncovering hidden, implicit interactions between people and algorithmic systems [29] improvised in a digital form. Our eyes follow what interests us, and our brains are active when we are intrigued or emotionally engaged by what we observe.

Recently, there has been a rising interest in interdisciplinary approaches, like combining anthropological and computational approaches [30], and a need for new methodologies in studying machine behaviour [31], such as the methodological twin-move of making big data thick and thick data big [32]. These studies have explored how autonomous vehicles will “see” to navigate themselves and, more importantly, avoid colliding with people, other vehicles, and objects [18], trust and threats of autonomous machines [33], liability and responsibility of autonomous vehicles [34], and pedestrians’ acceptance [22]. Additionally, the relationship between driver trust and attention during autonomous driving has been shown to play a crucial role in traffic safety and public acceptance. For example, a study by Li et al. [35] found that while latent hazard notifications improve driver attention and safety by increasing focus on potential risks, they also boost driver trust, which paradoxically reduces attention and diminishes 15.12% of the expected safety benefits due to behavioral adaptation.

However, there are methodological difficulties and gaps in measuring implicit beliefs —subconscious attitudes and perceptions that individuals may not be aware of— and stereotypes towards algorithms [36]. Therefore, there is an increasing interest in eye-tracking, VR, and other experimental tools and cognitive research methods [37,38], which allows researchers to analyze visual behaviors to better understand the implicit and hidden understandings of people towards data technologies. Eye-tracking enables exploration of human cognitive capabilities [39] and turning the process on the other way around – using the cognitive tools and human perception to understand the machines, algorithms, and AI, and how to engage the citizens in datafied smart city planning [8].

One of the ways for operationalizing and measuring sociotechnical imaginaries and people’s cognitive perceptions, particularly in the context of technology, involves the exploration of moral decision-making processes. Alan Turing already in the ‘50s suggested the “imitation game” proposing a method to check if a machine is intelligent compared to a human, introducing the controversial topic about how/if a machine’s intelligence can be measured via a method later called the “Turing test” [40]. However, whether autonomous vehicles include explicit moral decision-making is still unknown since coding morals into machines is not an easy task [41]. Nevertheless, autonomous vehicles involve social risk, which constitutes an ethical decision [42]. Any current or near-future autonomous system lacks the wealth of knowledge people use to make moral decisions, such as diverse characteristics of victims or subtle features of the environment. However, a car’s actions in road traffic can have moral consequences, and if an algorithm faces a situation with moral consequences, it is possible that its actions would be considered motivated by morals [41]. But then again, how could ‘morals’ be coded?

Additionally, scenarios are widely used to understand and respond to the impacts of human activities and to describe potential alternative futures [43]. Picture scenarios, in particular, help designers and decision-makers explore human behavior and interactions with different outcomes. This method sparks imagination and provides new perspectives in uncertain futures. Studies have shown that visual discourse analysis reveals how pictures are used as a medium to construct and disseminate sociotechnical imaginaries [12]. For example, Kastner [44] used eye-tracking technology to explore the relationship between people’s moral judgments about morally challenging situations and their visual attention to related images.

As many experimental tasks, including the scenarios, are performed on computers, scholars have conducted online studies instead of having participants come to a lab [45,46]. Although there are obvious limitations, such as reduced experimental control and data quality when experiments are run online, some human-computer interaction research can reliably be conducted online as it enables the possibility to scale up the study and compare data across different participants from different locations and backgrounds [47–49]. Despite the development of in-lab and online experiments, we still do not know about the differences, benefits, obstacles, and disadvantages of such experimental settings.

3.1. Samples’ structure

To design the sample for both online and in-lab experiment, we have used the combination of strategic [51] and representative sampling principles [52]. Besides, we also took into consideration the necessary pragmatic choices [28] usually faced in the online panel and platform studies, such as limited access to panels from different cities, and no monitoring and controlling over the experiment from the researchers’ side. The pragmatic approach was also additionally used in designing the sample in the in-lab study, where the purposeful sample was combined with the convenience sample elements and experimental approach with interviewing to ensure the study of the perception of algorithms as a rather abstract topic (see, e.g., [29]).

A purposeful sampling strategy [51] was used for the in-lab experiment (N = 16). Based on this sampling approach, the sample is homogeneous in regard to their place of residence – Estonia, and therefore having similar experiences with experimenting with the autonomous vehicles in the public space. Besides, the study participants were assumingly aware of using algorithms in designing transport solutions, based on their higher education level. However, their perception towards social diversity and automated technologies might vary. Besides, we purposefully planned the sample to be internally heterogeneous to ensure the expression of various views and understandings. So, the small-scale in-lab experiment was not intended for broad statistical generalization, but rather to develop, refine, and test the developed method prior to conducting our large-scale online eye-tracking study. It was used as a methodological cross-check to verify that the online eye-tracking yielded fixation and reaction-time metrics comparable to those of the in-lab eye tracker. However, when scaled up, the in-lab experiment combined with pre- and post-experiment interviews could serve as an independent study in the future. So, the aim of the in-lab study was to technically and methodologically compare the two eye-tracking approaches.

By employing a purposeful sampling strategy [51], we ensured a diverse range of perspectives, allowing for the identification of potential biases in the research design and the opportunity to address them for our online experiment. Therefore, following the principles of purposeful sampling, the sample consisted of a relatively homogeneous group of students as young experts who were potentially aware of algorithm- and AI-based approaches in transport planning. In addition, to diversify the responses, half of the study participants were social scientists (N = 8), and the other half computer scientists (N = 8). Our in-lab participants included engineers and scientists from the field of Computer Science, many of whom have hands-on experience with autonomous vehicles and algorithms. While other participants have been actively involved in studying and evaluating algorithms, providing further depth to their insights. Their technical expertise in relevant areas allowed us to capture valuable perspectives on both the development and testing processes of autonomous vehicles, which enriched the outcomes of our study. At the same time, the inclusion of Social Scientists was crucial, as they brought essential perspectives on the societal, ethical, and behavioral aspects of autonomous vehicle adoption, ensuring a more comprehensive understanding of the broader implications of the technology.

The online experiment was conducted through RealEye – a web-based eye-tracking platform. The participants were recruited through a crowd-sourced platform, CINT, which includes online panel members from 130 global countries. In this study, we selected participants based on national census data to ensure the territorial representativeness of the study results. Besides, the experiences with diverse transport modes were used as an additional criterion for participant selection. Our large-scale eye-tracking study involves participants from 26 cities around the world (N = 1276). Cities selected for the experiment are Amsterdam (Netherlands), Ankara (Turkey), Berlin (Germany), Buenos Aires (Argentina), Curitiba (Brazil), Guayaquil (Ecuador), Helsinki (Finland), Ho Chi Minh City (Vietnam), Hong Kong (China – special administrative region), Jakarta (Indonesia), Johannesburg (South Africa), London (United Kingdom), Los Angeles (United States), Manila (Philippines), Mexico City (Mexico), Mumbai (India), New Delhi (India), New York (United States), Paris (France), Reykjavík (Iceland), Salvador (Brazil), São Paulo (Brazil), Singapore (Singapore), Stuttgart (Germany), Tokyo (Japan), and Zurich (Switzerland). The experiment was conducted in English. Given that the participants are rather digitally aware and literate, along with the global spread of the CINT platform, where panel members are often engaged in international studies conducted in English, we believe that language did not pose any significant barriers.

In our online experiments, we ensured diversity and representativeness by carefully selecting participants from cities with varying socio-demographic profiles, including those that have hosted autonomous vehicle pilot programs. Specifically, we selected cities based on the Bloomberg Aspen Initiative on Cities and Autonomous Vehicles 2017 [53] list to target cities with exposure to AV technologies, and the CIMI Index 2020 [54], which measures the advancement of the cities in regard to implementing innovative technologies in developing the cities. We included in the study cities with diverse values of the CIMI index, including those with high, middle, and low values. We assumed that individuals from the cities with diverse experiences with smart city technologies might have different experiences with autonomous vehicles as one of the recent innovations, and therefore might have various perceptions of the potential consequences of these solutions. Our initial aim was to recruit participants from 20 cities, with 100 participants from each city (see Appendix 1). However, due to availability constraints and recruitment challenges on the CINT platform, some cities – particularly cities with high CIMI ranks and those listed in the Bloomberg Aspen AV Initiative Cities, which were underperforming – were substituted with other cities from a similar category to maintain the integrity of the sample structure. As a result, we expanded our selection to include an additional 6 cities. The final sample structures of the online and in-lab studies are presented in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Online and in-lab final sample structure.

https://doi.org/10.1371/journal.pone.0335672.t001

To further enhance the representativeness of our sample, we utilized the function offered by the CINT platform, which enabled us to draw participants in proportions reflective of the national census. This ensured that our online sample was representative across key demographic variables such as age, gender, and education level. Nevertheless, territorial representativeness within a county remains elusive in online crowd-sourced studies [55] compared to face-to-face surveys [56], and some regions, particularly rural, may still be under-represented. By leveraging the platform’s capabilities, we were able to ensure a broad and diverse participant pool, providing a strong foundation for analyzing the cognitive engagement and perceptions of autonomous vehicles across different geographical regions.

3.2. Scenario design and stimuli

To study the sociotechnical imaginaries of autonomous vehicles through implementing an eye-tracking approach, we employed a visual narrative scenario method as a stimulus for both in-lab and online experiments [57,58]. We assumed, and as the trial studies proved, the approach enables addressing the uncertainties and complexities of automated technologies in long-term urban mobility challenges and measuring the people’s cognitive engagement with the potential consequences of these technologies.

We used autonomous vehicle as an empirical case study and considered it as a proxy to study the data-based public services offered in the cities, which are assumed to provide equal access to transport and city infrastructure, such as hospitals to all residents despite their social background. Our experiment relies on a previous study by Awad et al. [9] who designed the Moral Machine to explore the moral dilemma of autonomous driving. However, our study strived to develop further this idea to explore sociotechnical imaginaries on AVs and perceptions of social diversity to address a critical gap in the literature: how automated systems account for diverse social identities. Understanding perceptions of diversity is crucial, as these perceptions influence trust, fairness, and the inclusivity of algorithmic systems. In our experiments, we used scenarios to explore one of the ethical concerns related to machine learning, namely, how people perceive social diversity when an autonomous public bus or a driver has to make a decision in case the bus experiences a sudden brake failure.

To design the scenarios, we followed Crenshaw [59] and Vertovec [60] approaches that diversity shall not be analyzed only in terms of ethnicities, race, and countries of origin, but rather in terms of the intersectionality that affects where, how, and with whom people live — super-diversity. It is argued by Crenshaw [59] that theorists need to take both sex and race on board and show how they interact to shape the multiple dimensions of people’s experiences. To avoid the one-dimensional appreciation of contemporary diversity, we depicted passengers, pedestrians, and the driver by using the intersection of sex and race with other social categories that are dominant in our society and affect and shape how people interact with AVs.

Two sets of scenarios were shown to participants in a randomized sequence within each set, and across the two sets, for both studies. Randomization was used to eliminate the selection bias and how confounders potentially impact our results. One set of scenarios was about the autonomous public bus that needs to decide between saving pedestrians in the street or passengers in the bus, so the responsibility relies on the algorithms to make such a decision. While in the other set of scenarios, the only difference was that the responsibility relies on a human being (driver). The scenario categories used in this study to depict social demographic background of pedestrians and passengers are the following: nationality (EU/Asia/US/Africa), income (rich/homeless), micro-mobility (scooter/robot delivery), age (young/adult/old), disability (wheelchaired), COVID-19 (vaccinated/not vaccinated), residency (urban/rural). Some of the categories in this study were visually distinguishable, while others, such as COVID-19 status, were represented using symbols. However, for each scenario, participants had the option to click on ‘Show Description’ for additional details. We first conducted the in-lab experiment (experiment running 27 April-13 May 2022) to test the developed method and collect small-scale data, and based on that, slightly redesigned the online experiment (experiment running from 13 September-28 October 2022). Minor differences in designing the scenarios in the online experiment were needed, due to the time limit (see Table 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Experiment design: in-lab vs. online.

https://doi.org/10.1371/journal.pone.0335672.t002

The scenarios are designed to compare how people perceive social diversity when an autonomous bus or a regular bus faces a life-or-death situation. Participants were asked, “Which group of people will be saved first?” We used will instead of should as proposed by Kastner [44] since we were particularly interested in people’s imaginaries in regard to automated decision-making – what they think that the autonomous bus or the driver will choose to save: two passengers in the bus or two pedestrians crossing the road – instead of what should be the case which implies the moral or normative decision. Also, we used the word save instead of kill following Kastner’s [44] suggestion, since saving the group would be the morally correct choice and reduce the anxiety over choosing to kill. Moreover, we tried to limit participants’ moral reasoning (such as the trolley problem, respecting road signs) as much as possible because we were particularly interested in analyzing participants’ attention to social diversity, limiting other moral reasoning. Nevertheless, saving someone over someone inherently encompasses moral and ethical considerations.

In the experiment, participants were briefed on the procedure and scenarios. In-lab, they could ask researchers for further guidance, unlike in the online version. However, the online experiment included a section for feedback. In both formats, a two-second white screen buffer was used between slides for eye relaxation and focus. Participants began by clicking “start”, then on a split screen, they chose between two options by clicking an image: save two pedestrians (right) or two passengers (left).

3.3. In-lab and online eye-tracking experiments

In-lab eye-tracking experiment occurred in an approximately 30m² room where participants sat at the desk with the head stabilized at the chin rest located in the middle of the monitor and 42–62 cm away from the screen that showed the scenarios in the form of images. The eye-tracking device was positioned on the desk below the screen. Whereas the online experiment was conducted using RealEye’s platform, which predicts a person’s gaze point using only webcams as data collectors. People could take part in the online experiment from their home at a convenient time using a laptop or desktop PC, thus increasing the chances for higher data acquisition [61]. Since the experiment was unsupervised, RealEye used algorithms to assess whether people’s head is centered on the screen and their eyes are properly situated on the screen.

For the in-lab experiment, we used an SR Research Portable Duo eye-tracking device, which produced stable, low-noise, binocular recordings up to 2000 Hz with the head stabilized and an average accuracy down to 0.15°. The blink recovery was around 0.5 milliseconds (ms) with a 2000 Hz sampling rate and was also compatible for participants wearing glasses or lenses. The eye-tracker was focused on the glint in the participants’ eyes, followed by a preliminary 9-point calibration of the device. Since the online experiment was webcam eye-tracking, data were captured at a sampling rate varying between 30 Hz and 60 Hz, depending on the participants’ webcam and internet connection speed.

RealEye uses the computing power of a regular PC/laptop to run AI that analyzes images coming from a webcam. The AI detects a participant’s pupils and predicts a gaze point with an accuracy of ~110px (~ 1.5 cm) and an average viewing angle error of ~4.17 degrees. This allows analyzing users’ interaction on a website with precision reaching the size of a single button. The platform did not record any image or sound to the servers, only storing the gaze point predictions in the form of basic text data similar to “Timestamp: 10, GazePointX: 200, GazePointY: 330”. RealEye and CINT are fully compliant with GDPR and guarantee data protection for each participant. The camera was switched off immediately after each test.

3.4. Data analysis

We analyzed and compared eye-tracking data based on the average fixation duration, cumulative (total) fixation duration, and reaction time of participants, as expressed to the visual stimulus and scenarios presented. These metrics serve as proxies for cognitive processing and attention. Average fixation duration is the average length of time a participant’s eyes stop scanning the scenarios and fixate the vision in one point/place. Cumulative fixation duration is the sum of all fixation points (counts); it indicates the overall time spent fixating within the given stimuli. Reaction time is the time from when the visual stimuli appeared to when an action (i.e., selecting a picture) was taken (for the online experiment, there was a limited exposure time to each scenario).

Specifically, the fixation duration provides insights into the depth of cognitive engagement, while the reaction time sheds light on the decision-making time of participants. In general, longer fixations are associated with deeper cognitive processing and attention, while shorter reaction times indicate quicker decision-making or recognition processes. Therefore, the differences in these metrics between the two methods can provide valuable insights into how participants perceive AV and non-AV scenarios.

4.1. Differences in cognitive engagement across in-lab and online eye-tracking methods

We have compared the cognitive engagement of people when observing autonomous vehicles versus non-autonomous vehicles. We assumed that there is a difference in how people perceive autonomous vehicles versus non-autonomous vehicles, and this is due to many factors such as contextual familiarity, personal relevance and moral deliberation, social desirability, and external influences.

Cognitive engagement refers to the depth and intensity with which participants process information. In eye-tracking studies, fixations serve as a proxy for cognitive engagement, with longer fixation duration indicating greater cognitive processing and attention paid to specific aspects of the presented stimuli. The cognitive intricacies tied to AVs mirror collective imaginaries of a technology-driven landscape. In this study, two sessions of scenarios were almost identical, with the only difference being the type of vehicle. We were interested in seeing whether there was a difference in how people perceive autonomous vehicles in a life-death situation compared to manual vehicles. It is important to highlight that there are some differences in the experiment design and the sample between the online and in-lab study (see Table 2).

Online and in-lab eye-tracking findings revealed a tendency for higher fixation during scenarios involving algorithmic decisions in the case of AVs. Specifically, in the online study, the average fixation duration in AV scenarios was about 5.8% more compared to non-AV scenarios. In the in-lab experiment, the average fixation duration of AV scenarios was slightly higher than non-AV scenarios (about 1.71% more compared to non-AV scenarios). The longer fixation duration (see Table 3–4) indicates that people were processing more information and thinking more deeply about the decision the algorithm, or the driver, was making.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Fixation duration per category from the online experiment (average, st. deviation 𝛔), N = 1276.

https://doi.org/10.1371/journal.pone.0335672.t003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Fixation duration per category from the in-lab experiment (average, st. deviation 𝛔), N = 16.

https://doi.org/10.1371/journal.pone.0335672.t004

The contrast between the online and in-lab eye-tracking experiments offers intriguing insights into how participants perceive and interact with AV and non-AV scenarios. In the online experiment (Table 3), all scenarios—ranging from “Nationality” to “Urban/Rural”—displayed significant differences (p < 0.001) in fixation durations between AV and non-AV settings. This denotes a clear distinction in how participants engaged with the two types of scenarios. Specifically, they exhibited longer fixation durations for AV scenarios, indicating deeper scrutiny or increased processing when considering autonomous vehicles.

Conversely, the in-lab experiment data (Table 4) painted a different picture. The comparative analysis of fixation durations across distinct scenario categories in the in-lab experiment, despite revealing observable disparities between AV and non-AV visual stimuli, the small sample size did not allow to make strong conclusions. The in-lab experiment engaged in purposeful sampling, curating a participant pool with specific characteristics and presumable predispositions towards the stimuli, which could potentially moderate the perceptual variances across scenarios. This requires delving deeper into potential participant-specific factors that might elucidate these perceptual tendencies.

4.2. Exploring cognitive engagement: Viewing behaviors within in-lab and online eye-tracking methods

There are several aspects within the method that might explain the observed differences in the results across the two methods. Viewing behaviors were analyzed within the in-lab eye-tracking study and the online experiment. A particularly intriguing result emerged in the viewing behaviors between the Data&Algorithm-Informed (DAI) group which is potentially more informed about data and algorithms, and the Data&Algorithm-Uninformed (DAU) who may lack technical knowledge and experience with data and algorithms.

In both experiments, DAU and DAI demonstrated a higher average fixation duration toward AV scenarios (Fig 1). However, the cumulative fixation duration provided a different picture in-lab – DAI and DAU tend to fixate longer towards non-AV.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Average and cumulative fixation duration differences between DAI and DAU.

Average fixation duration (left boxplot) and cumulative fixation duration (right boxplot) difference between DAI and DAU in online experiment (A), and in-lab experiment (B).

https://doi.org/10.1371/journal.pone.0335672.g001

As we dig deeper into the specifics in the sections below, the intricacies of these differences and their implications for societal imaginaries concerning autonomous vehicles become more evident.

4.2.1. Fixation duration variations within online eye-tracking.

The study’s analysis of online viewing behaviors provides insights into how different groups of people perceive and react to autonomous vehicle scenarios. The findings suggest that DAI, who are aware of data and algorithms, tend to have shorter average fixation duration and take less time to make decisions compared to DAU (Fig 1A). Additionally, both groups showed a tendency to fixate longer on AV than on non-AV.

Online viewing behaviors were analyzed between and within 1) Data&Algorithm-Informed (n = 508) – people who are aware of data and algorithms and participated at least once in similar studies, and 2) Data&Algorithm-Uninformed (n = 768) – those who never participated before in similar studies and assumingly might not be aware of the data and algorithms. In addition, viewing behaviors were analyzed across two sets of scenarios – autonomous vehicle and non-autonomous vehicle. For DAI participants, classified as such due to their prior participation in similar studies and presumed familiarity with algorithmic systems, the average fixation duration (the depth of fixating on one point) and total fixation duration showed a trend towards being higher in AV scenarios. However, the t-test revealed that this difference was not statistically significant (t(7080) = 1.95, p = 0.051). For DAU participants, total fixation duration was statistically significantly longer in AV scenarios compared to non-AV scenarios, as indicated by the t-test results (t(10748) = 2.11, p = 0.034).

To examine the effects of group (DAI vs. DAU) and scenario setting (AV vs. non-AV) on fixation time, a Two-Way ANOVA (Table 5) was conducted because it (i) allowed simultaneous estimation of two main effects plus their interaction and (ii) is known to be robust when data sizes are large and reasonably balanced [62]. The analysis revealed a significant main effect of group, F(1, 17860) = 171.26, p < 0.001, partial η² = 0.009, indicating that fixation times were significantly different between the DAI and DAU groups. There was also a significant main effect of scenario setting, F(1, 17860) = 8.11, p = 0.004, partial η² = 0.0005, suggesting a significant difference in fixation times between AV and non-AV scenarios. However, the interaction between the groups and scenario setting was not significant, F(1, 17860) = 0.00085, p = 0.976, indicating that the effect of the scenarios on fixation time did not differ between the DAI and DAU groups.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Results of two-way ANOVA for fixation time comparing DAI vs. DAU groups and AV vs. non-AV Setting*.

https://doi.org/10.1371/journal.pone.0335672.t005

The assumptions of normality and homogeneity of variances were assessed using Shapiro-Wilk and Levene’s tests. Shapiro-Wilk on a random 5,000-residual subset (upper practical limit for the test [63]) indicated right-tail skew, W = 0.663, p < 0.001, likely due to the inherent skewness of the data derived from 26 cities. Levene’s test indicated unequal variances across the four experimental cells (DAI-AV, DAI-non-AV, DAU-AV, and DAU-non-AV), F(3, 17860) = 24.72, p < 0.001. To guard against bias from these violations, we replicated the analysis using (a) a Welch heteroscedastic two-way ANOVA (HC3 estimator) and (b) a generalized linear model with a Gamma error distribution and log link – both appropriate for positively skewed duration data. The Welch test revealed group F = 182.1, p < 0.001; scenario F = 8.25, p = 0.004; interaction F = 0.00091, p = 0.98. The Gamma-log GLM showed +31% for DAU relative to the reference category DAI for AV scenarios (p < 0.001), −6% for non-AV relative to AV within DAI group (p = 0.036), and no statistically significant interaction (p = 0.71). Thus, both tests replicate the group and scenario main effects and the null interaction observed in the classical ANOVA, with overlapping 95% confidence intervals.

To validate the findings, a Wilcoxon Signed-Rank test was conducted to compare fixation times between AV and non-AV settings within each group. Wilcoxon Signed-Rank tests conducted at the fixation-event level showed a significant AV–non-AV difference for DAI (p = 0.029) and a highly significant difference for DAU (p = 0.0001). When the test is run on participant-level means, the DAU effect remains significant (p = 0.044) while the DAI effect becomes a non-significant trend (p = 0.105); thus, the overall scenario effect reported in the ANOVA is robust, although its strength within the expert group is modest.

Furthermore, an interesting observation was the heavy-tailed boxplot (Fig 1A, left image) and the presence of outliers with notably prolonged fixation durations (Fig 1A, right image). A significant factor to consider is the context [64], such as the geographical diversity of our participant sample, with individuals coming from various cities, each carrying its unique set of cultural, socio-economic, and educational backgrounds. Such diversity can explain variations in how participants process, interpret, and engage with the stimuli. For instance, visual content that might be easily understood in one city could be novel or require deeper contemplation for participants from another. The association between outliers in fixation times and participants’ countries was assessed using a chi-square test. which test yielded χ2 = 1284.06, df = 60 and p < 0.001. The low p-value suggests a statistically significant association, indicating that the distribution of fixation times varies significantly across different countries.

Fig 2 represents heatmaps aggregating fixation data from DAI and DAU participants in a specific scenario and is intended as an illustrative example to show the differences in attention patterns between the two groups for AV and non-AV scenarios (to see the difference between DAI and DAU by means of average fixation time across all scenarios, see Fig 1). The heatmaps’ warmer colors indicate regions of intensified fixations. DAI, with their overall shorter fixation durations than DAU, appears to distribute their attention more in specific areas across the scenarios. This suggests an analytical perspective on some elements of stimuli, possibly due to their data and algorithmic expertise. Conversely, DAU, with their longer fixation durations, seem to focus more intently on AV scenarios, possibly indicating areas of intrigue or uncertainty.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Online eye-tracking heatmaps of DAI and DU.

Warmer colors (e.g., red and orange) indicate areas with higher attention or fixation durations, and cooler colors (e.g., green and blue) represent areas with lower attention.

https://doi.org/10.1371/journal.pone.0335672.g002

4.2.2. Fixation duration variations within in-lab eye-tracking.

Similarly to online results, the in-lab experiment showed that both DAU and DAI fixated longer on a single point in the AV scenarios. However, unlike the online experiment, the cumulative fixation duration in-lab showed different results – the overall time spent fixating on the given stimuli on average was higher in non-AV (Fig 1B). Specifically, DAI participants (background in computer science) demonstrated higher total fixation duration in non-AV in comparison with DAI (background in social science), who showed slightly higher total fixation duration in non-AV than AV.

The heatmaps (Fig 3) represent aggregated data across all scenarios presented in the experiment. However, the chosen image is only an illustrative one to provide an overview of the differences in gaze patterns between the DAI and DAU groups across the two sets of experimental conditions. The heatmaps make it clear that their attention was drawn significantly to the “additional information” in the non-AV scenarios provided alongside the images, which described the passengers and pedestrians. This behavior indicates a shared interest in understanding the human context and social dimensions of the scenarios. However, how they engaged with the whole content showcased differences in their analytical approaches.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. In-lab eye-tracking heatmaps of DAI and DU.

https://doi.org/10.1371/journal.pone.0335672.g003

On one hand, DAU participants engaged more holistically with the scenarios. Their gaze was spread evenly, oscillating between the visual stimuli and the supplementary text, suggesting an integrated approach to understanding both the technological and social facets of the scenarios. This was also reflected during the interviews, where DAU intended to not only understand but also position AVs within a broader societal and ethical frame, and critically evaluated biases in human-driven vehicles where the social demographics of the driver have been depicted in non-AV scenarios.

On the other hand, DAI participants, with their technical background, demonstrated a more segmented approach. In AV scenarios, their primary focus was on the images themselves, possibly trying to discern technological or mechanical attributes. In contrast, for non-AV scenarios, their attention shifted predominantly to the contextual text. This implies that while they were keen on understanding the technological nuances in AV scenarios, they leaned towards grasping the socio-contextual narrative in non-AV situations. Based on the interview data, DAI showed a critical evaluation of AV scenarios, as they noted that the mere presence of diverse data in algorithmic systems does not automatically guarantee equitable or fair outcomes in AV decision-making processes. They argue that while diverse data is a critical element in developing robust and inclusive algorithms, how this data is processed, weighted, and applied within the algorithm is equally important. However, longer fixation duration in non-AV scenarios shows their attempt to critically evaluate, or perhaps even challenge, the constructs and decisions embedded within human-driven vehicle representations. According to them, humans share their own values and beliefs, and therefore, this might lead to biased decisions, whereas the algorithm decides based on the type and availability of data – detached from human prejudice and biases.

Determining the precise differences behind such attention distribution requires more granular analysis in further studies. This involves breaking down the scenarios into specific Areas of Interest (AOI) and examining cognitive engagements within these AOI.

4.2.3. Exploring decision-making: Reaction time within in-lab and online eye-tracking.

Reaction time, when evaluated alongside fixation durations, offers a deeper understanding of participants’ decision-making processes during the experiment. The reaction time is the period from the presentation of the stimuli to the moment of the decision made by the participants. The analysis of reaction time complements our understanding of the fixation duration variations observed in the previous sections, offering valuable insights into how these variations are indicative of participants’ cognitive engagement and internal deliberations. Faster reactions could indicate instinctual responses, possibly influenced by prevalent societal narratives or personal beliefs about AVs. Conversely, extended reaction times might suggest participants grappling with their internal biases, beliefs, or the contemplation of potential outcomes in the presented scenario.

The reaction time data (Table 6) showed that in-lab DAI and DAU average reaction time was longer in non-AV, whereas online DAI and DAU showed slightly longer reaction time in AV scenarios. In addition, when comparing the differences of in-lab DAI versus DAU, the average reaction time for both AV and non-AV scenarios is longer for DAI – they took more time to make a decision than DAU. This confirms the findings shown above that overall DAI fixated longer than DAU. Nevertheless, the online DAI and DAU average reaction time was almost identical.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Average reaction time in seconds comparison between DAI and DAU in online and in-lab experiment.

https://doi.org/10.1371/journal.pone.0335672.t006

In addition, in-lab participants were people who showed particular interest in joining the experiment, and especially a group of participants who were engaged directly with AV algorithms, which might affect the reaction time because of their interest in the topic.

4.3. Evaluation of data quality in online and in-lab eye-tracking experiment

The differences and similarities in the method could be explained by the fact that, due to technical aspects (Table 7), the quality of measurement differs across methods.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Technical comparison between in-lab and online eye-tracking.

https://doi.org/10.1371/journal.pone.0335672.t007

The experiment monitoring during the in-lab experiment was much higher in comparison with the online experiment. This explains the difference in data quality between the experiments. The in-lab experiment was supervised by the experimenter, who sat at one end of the desk with a laptop computer that monitored the experiment and calibrated the eye-tracker (if needed) for proper gaze recording. However, the online experiment was unsupervised, so there was no control during the online experiment. During the in-lab experiment, after the first fixation has been accepted, the remaining calibration targets are displayed in sequence, and fixation values are collected for each scenario. The calibration system presents these targets in a random order, which discourages participants from anticipating the location of the next dot and cascading away from the current target before it disappears. However, it was important to remind the participant to look at each calibration dot until the next target appears. After the calibration, the validation process started. In each validation phase, 9 dots appeared one after the other, in random order, on each of the four edge positions of the calibration and in the center.

Validation showed the experimenter the gaze position accuracy achieved by the current calibration model. When the participant fixates these dots, the calibration model is used to estimate the gaze position of the participant, and the error (difference between the actual target position and the computed gaze position) is calculated. If a position was not successfully validated, the participant was sent back to the instructional page and had to recalibrate. If the eye-tracker was recording accurately, then the experimenter pressed the key to continue the experiment, and in case there were deviations, then the calibration was performed until the eye-tracker was showing high accuracy of eye movements. To ensure that the accuracy of the calibration parameters is maintained during the whole experiment time, a drift check was performed every 5^th scenario by showing a dot in the center to reorient the participants’ eyes to the center of the screen. This resulted in acquiring only high-quality data, which was not the case with the online experiment.

In the online settings, the system of calibration and validation was all automated. A standard 40-point calibration on RealEye software was shown to each participant in 3 backgrounds – grey, black, and white- to minimize the influence of the monitor’s light intensity during the eye-tracking experiment. The calibration is then validated on 9 points. After successful calibration and validation, participants completed two sets of scenarios shown in random order (same as in-lab) – autonomous scenarios and non-autonomous scenarios. The task was self-paced, meaning that the next scenario started whenever participants clicked on one of the images (left image or right image). Since the experiment was all automated, in case participants were facing any technical issues, such as a poor internet connection or stimulius loading, then the skip URL was sent to participants to end the experiment.

Also, different from the in-lab experiment, there was no manual drift check feature to monitor the accuracy of the webcam eye tracker throughout the experiment. In case participants were moving their head outside the standard centered frame, then an automated pop-up message was shown to participants to fixate their head in the center of the screen. Or, if participants had many head movements, then an automated message was shown on the screen, i.e., “Many head movements,” and sent back to calibration. Only after the participant fixated the head and the webcam was able to locate the eyes, then the experiment resumed. Not being able to control the calibration and validation, and the lack of a drift check feature has significantly influenced the data quality. Each result has been assigned one of the data quality grades: Perfect, Very Good, Good, Average, Low, or Very Low. Almost 90% of the in-lab eye-tracking data are high-quality data, while the average of the online eye-tracking data is Good.

The quality of data obtained from an eye-tracking study is significantly influenced by the sampling rate of the eye-tracker. Higher sampling rates allow for more precise capture of eye movements, thus enhancing the overall quality of the data collected. The online attrition rate is 39% from which 2010 granted camera access, but only 1276 participants have completed the test and provided results, whereas all in-lab participants filled in a pre-registration form, and there was no dropout. The online experiment was opened by 13036 participants, but only 10% of them completed it. The exact number of people invited to pre-register for the in-lab experiment is not known as it was sent to the student email list by different department representatives, and no data is saved of those who opened the link.

5.1. Limitations and further studies

We, as authors, are aware of the potential limitations and biases that might arise in this research. While we made every effort to ensure scenarios used in this study were as natural and unbiased as possible, we acknowledge that there might still be inherent biases that could have influenced participants’ viewing behaviors. Therefore, we are aware that visual behavior might be altered since being eye-tracked induces social-norm-based looking behavior [73]. Additionally, the small in-lab sample size did not allow us to generalize the findings, even though the main aim was a technical and methodological comparison of online and in-lab tracking methods. However, we suggest further studies to scale up the in-lab experiment and complement it with other cognitive tools (e.g., electroencephalography (EEG), see [74]) to triangulate and enrich insights into sociotechnical imaginaries. Complementary cognitive tools, alongside eye-tracking, help to capture brain activity, subconscious cognitive and emotional responses to context-sensitive stimuli, which may enable to uncover the hidden mechanisms that shape trust and acceptance and inform socially aligned AV design.

Another limitation was related to the potential sample biases in the online study. While our aim was to get online participants based on their respective city consensus data, it is important to highlight that participants of the online platforms often belong to younger age groups, have higher educational backgrounds, and are more active in digital environments. This sample skewness might lead to an over-representation of views from educated individuals and neglect the perspectives of less educated, and therefore assumingly less aware of the use of algorithmic approaches in planning the transport in the cities. The sample might lead to an underrepresentation of older people, people from rural areas, and vulnerable groups who may have different experiences and viewpoints on such technologies. Moreover, certain cities exhibited a lower response rate compared to others; therefore, we had to add additional cities to maintain the integrity of the sample structure by substituting the cities based on the categories we selected.

Furthermore, we aimed to enhance data accuracy by requiring participants to use laptops rather than mobile phones, as laptops encourage more stationary behavior, improving eye-tracking precision. However, conducting the experiments online without direct supervision made it difficult to control environmental factors like distractions, lighting, and participant focus, which we acknowledge as a potential limitation.

We suggest future studies to dig deeper into specific AOI within the scenarios, and variances across different countries and demographics. Considering the potential influence of culture and country context transportation infrastructure in the cities, knowledge and experiences with technologies, and comparing perceptions across different cities or countries could provide valuable perspectives on how these factors play in shaping the perception towards AVs. Moreover, understanding how individuals with different personal experiences with AVs (e.g., people who use or do not use AVs) engage with specific elements of scenarios, such as images or text can provide richer insights into their reasoning processes and concerns.