2.1 Ethics statement

The federally registered Institutional Review Board (IRB) of the sponsoring institution (American Institute for Behavioral Research and Technology) approved this study with exempt status under HHS rules because (a) the anonymity of participants was preserved and (b) the risk to participants was minimal. The IRB is registered with OHRP under number IRB00009303, and the Federalwide Assurance number for the IRB is FWA00021545. Informed written consent was obtained as specified in the Procedure section below.

2.2 Participants

Participants were recruited online from the Amazon Mechanical Turk (MTurk) subject pool between May 28th and June 12th, 2024. Participants were screened by CloudResearch to prevent bots and suspect users from entering our subject pool. Participants were required to be at least 18 years old, and they were allowed to participate only if they answered Yes to the question, “Are you eligible to vote in the United States?” and No to the question, “Do you know a lot about politics in Australia?” We also asked participants to rank how familiar they were with both Bill Shorten and Scott Morrison – two men who ran for Prime Minister of Australia in 2019 – on a scale from 1 to 10, where 1 was labeled “Not at all” and 10 was labeled “Quite familiar”; we eliminated participants who answered above a 3 on either scale. We screened our participants in these ways to try and ensure that they would be “undecided” about whom they might vote for [cf. 1]. After screening, we had data from 536 participants to analyze.

The participants ranged in age from 18 to 82 (M = 37.9, median = 36, SD = 11.1). 62.1% (n = 333) of the participants identified themselves as female, 36.4% (n = 195) as male, and 1.5% (n = 8) chose not to identify their gender. 75.0% (n = 402) of the participants identified themselves as White, 10.8% (n = 58) as Black, 5.6% (n = 30) as mixed, 4.9% (n = 26) as Asian, and 3.7% (n = 20) as other. Regarding the level of education people reported having completed, 0.9% (n = 5) said none, 5.6% (n = 30) said primary, 33.2% (n = 178) said secondary, 44.2% (n = 227) said bachelor’s degree, 15.1% (n = 81) said master’s degree, and 2.8% (n = 15) said doctorate. 39.6% (n = 212) of the participants identified themselves as liberal, 30.8% (n = 165) as moderate, 18.8% (n = 101) as conservative, 7.6% (n = 41) as none, and 3.2% (n = 17) as other. The mean familiarity score for Scott Morrison was 1.05 (SD = 0.27), and the mean familiarity score for Bill Shorten was 1.02 (SD = 0.16).

2.3 Procedure

Participants were given brief instructions and were asked for their consent to participate in the study (S1 Text). They were then asked a series of basic demographic questions, after which they were given brief, roughly equivalent, descriptions of the two political candidates (S2 Text). Then we asked them eight questions about the candidates – three opinion questions about each candidate (regarding liking, trusting, and overall impression) (S1 Fig) and two questions about their voting preferences. In the first, participants indicated their voting preference on an 11-point scale from 5 to 0 to 5, with the names of the candidates counterbalanced at each end. Finally, they were asked a forced-choice question: “If you had to vote right now, who would you vote for?”.

They were then assigned at random to either Group 1 (pro-Morrison), Group 2 (pro-Shorten), or a control group; this was done without the participant’s knowledge. At this point, one of the three platforms (mentioned above) loaded into place for each participant, and he or she was given brief instructions on how to use that platform to conduct further research on the candidates (S2 Fig).

We will now briefly describe the manipulation performed on each platform; bear in mind, however, that the order of platform presentation was randomized for each participant. The order in which we describe them here is not necessarily the order in which each participant saw them. (We belabor this point in order to avoid subjecting our readers to an order effect).

On Kadoodle, our simulation of Google search, participants were instructed to conduct research on the two candidates by clicking on one or more search results and then reading the content on the pages linked to those search results (S3 Text). Each participant had access to five pages of search results, with six results per page (Fig 1). They could switch between pages by clicking on a link at the bottom of each page, just as one can on Google search (Fig 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Example of Kadoodle search results page.

Reprinted from [3] under a CC BY license, with permission from the American Institute from Behavioral Research and Technology, original copyright 2024. This figure is similar but not identical to the original image and is therefore for illustrative purposes only.

https://doi.org/10.1371/journal.pone.0327209.g001

All the search results were real, sourced from Google.com. The webpages were real, as well, although we removed all active links from each page. The only difference between what the three different groups saw in this experiment was in the ordering of the search results. In Group 1, the ordering favored Scott Morrison (in other words, the higher the search result in the list, the more the content in the linked webpage favored Morrison); in Group 2, the ordering favored Bill Shorten; and in Group 3, the ordering was mixed, favoring neither candidate (Fig 2). The content in the webpages favored one particular candidate by showing positive information about that candidate, negative information about his opponent, or both. The ranking of the webpages was based on bias ratings provided by five independent observers.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Position and content of Kadoodle search results for each bias group.

Participants in Group 1 saw search results favoring Scott Morrison (dark pink indicates stronger bias). Participants in Group 2 saw search results favoring Bill Shorten (dark blue indicates stronger bias). Participants in Group 3 saw no bias in search results.

https://doi.org/10.1371/journal.pone.0327209.g002

After participants clicked on at least one search result, a button appeared in the upper-left part of the screen that allowed participants to end the search. If a full 5 minutes passed, participants were notified that the search period was over. When the search ended, participants were again asked those eight questions we described earlier (six opinion questions, and two voting-preference questions). After these questions were answered, participants were informed that they could now conduct further research about the candidates on another platform.

On Twiddler, our X simulation, participants were told to scroll from the beginning to the end of a list of 30 tweets (S4 Text). They were also told, “Your task is to use this Twiddler feed to try to further clarify your views on each candidate so that you are better able to decide which one deserves your vote.” 29 of the tweets in the feed of 30 appeared to come from regular users and were neutral in their content, meaning that they were not biased to favor either of the candidates. These 29 tweets were the same for all participants. The manipulation involved only the special tweet in the 2nd position (Fig 3). That tweet was apparently sent by the company (rather than by a user), and it was labeled, “Twiddler alert.” It contained a brief description of a negative news story about the opponent to the candidate favored in all the manipulations seen by that participant. The same negative news story was seen by every participant in the two bias groups. The only difference was the name at the beginning of the headline. So if the content in each manipulation always favored Bill Shorten, the news alert read, “Scott Morrison caught spending taxpayer money on lush vacation in Mexico.” If Scott Morrison was the favored candidate, the news alert read “Bill Shorten caught spending taxpayer money on lush vacation in Mexico.” Participants could move on to the next page by clicking the “Continue” button below and to the left of the phone image (Fig 4). Participants were then presented with those eight questions again, and then given brief instructions for how to continue their research on a third platform.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Position and content of Twiddler alert for each bias group.

In Group 1, users saw one targeted message in the second position which was biased toward Scott Morrison (dark pink). In Group 2, users saw one targeted message in the second position which was biased toward Bill Shorten (dark blue). In Group 3, users saw one targeted message which was randomly chosen to be either pro-Morrison or pro-Shorten.

https://doi.org/10.1371/journal.pone.0327209.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Example of Twiddler content.

Reprinted from [11] under a CC BY license, with permission from the American Institute from Behavioral Research and Technology, original copyright 2023. This figure is similar but not identical to the original image and is therefore for illustrative purposes only.

https://doi.org/10.1371/journal.pone.0327209.g004

On Dyslexa, our Alexa simulator, participants were told how to use this IPA (S5 Text), and they were then shown a list of 10 questions that were phrased to ask Dyslexa which candidate had stronger policies on 10 separate political issues (see Fig 5 and S6 Text). They could ask one question by clicking it, after which Dyslexa, using the original Alexa voice (Amazon Polly), responded orally with an answer supportive of Morrison, Shorten, or, in the control group, either candidate (depending on which of these three groups the participant had been assigned to at the beginning of the experiment). After Dyslexa finished answering the question, the remaining questions would be greyed out, and participants could proceed by clicking the “Done Asking Questions” button that appeared directly below the list of 10 questions (Fig 6). When research on the third platform was completed, participants were again asked those eight questions – six opinion questions and two voting-preference questions.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Dyslexa questions and answers for each bias group.

In Group 1, participants chose one question out of the 10, and the answer was biased toward Scott Morrison (dark pink). In Group 2, participants chose one question, and the answer was biased toward Bill Shorten (dark blue). In Group 3, participants chose one question, and the answer was randomly chosen to be either pro-Scott or pro-Morrison.

https://doi.org/10.1371/journal.pone.0327209.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Example of Dyslexa content.

Reprinted from [10] under a CC BY license, with permission from the American Institute from Behavioral Research and Technology, original copyright 2022. This figure is similar but not identical to the original image and is therefore for illustrative purposes only.

https://doi.org/10.1371/journal.pone.0327209.g006

Finally, participants were asked whether anything “bothered” them about the content that had been displayed during the searches. We asked this question to try to determine whether the participants in the two bias groups perceived any bias in the content. We could not ask directly about the perception of bias, because leading questions of this sort have long been known to inflate answers artificially [29].

4.1 Summary and conclusions

This study is the 13th in a series of studies that have been published or that are currently under review; these studies measure the impact of new forms of influence made possible by the internet. The series began with a report about the “search engine manipulation effect” (SEME) which was published in the Proceedings of the National Academy of Sciences in 2015 [1]. That study, which was based on data from five randomized, controlled experiments with 4,556 participants from the US and India, showed that bias in search results can easily shift the voting preferences of between 20% and 80% of undecided voters without their knowledge. That study has been partially or fully replicated multiple times [2–8]. In another SEME study, we showed that bias in search results can – for people who have not yet made up their minds – shift people’s opinions about non-political topics – perhaps, we speculated, about any topic at all [3]. In another SEME study, we presented evidence suggesting that SEME is an unusually large list effect because it is maintained by a daily regimen of operant conditioning. Because most searches are for simple facts (“What is the capital of Kentucky?”), and because correct answers invariably occur in the highest-ranking search result, people learn to trust high-ranking search results over lower ones. When, at some point, someone asks an open-ended question (“What is the best guitar brand?”), people continue to trust, and hence to click, high-ranking results [4].

Our most recent studies, published or submitted since 2023, have reported findings on the targeted messaging effect (TME) [11], the search suggestion effect (SSE) [9], the video manipulation effect (VME) [32], the opinion matching effect (OME) [38], the digital personalization effect (DPE) [39], and the multiple exposure effect (MEE) [5]. Using simulations of three different popular online platforms – Google search, X, and Alexa – the MEE study showed that when people were exposed repeatedly to similarly biased content on the same platform, the impact of the manipulation increased with each exposure; in other words, MEE appears to be an additive effect.

The present study examines the impact of exposure to similarly biased content generated by different means on simulations of multiple online platforms – once again, Google search, X, and Alexa. We gave participants access to three different platforms, with those platforms presented in a random order. The results were similar to those we found in the MEE study: The number of undecided voters voting for the favored candidate increased with each successive platform exposure by 42.4%, then 56.5%, and then 66.7% over the pre-exposure level of voting preference; each of the three increments was statistically significant.

If tech platforms were like news organizations – with thousands of such organizations, each with its own viewpoint, competing for our attention every day – the increments we found in our experiment would be relatively harmless. The problem we all face is that the internet is currently dominated by a small number of large monopolies [see 40] that share similar cultural and political values [41]. That overlap in values could change, of course, but our MPE findings remind us about a possible downside to homogeneity on the internet. Repeated exposures to similarly biased content generated by different mechanisms on different platforms might be impacting people additively. The overlap in bias might occur because of shared values at large companies, but it might also occur when people get trapped in filter bubbles [42, cf. 43], where they are being shifted from one small platform to another, each platform promoting similar values or political viewpoints.

The bottom line here, which we see as a matter of some concern, is that platforms that seem to serve very different purposes (say, TikTok versus Facebook) might generate content that is similarly biased. When this occurs, the impact on users might be additive.

4.2 Limitations and future research

Our study suffers from a number of limitations, some of which we are attempting to understand and remedy in ongoing research. The most obvious limitations result from our procedure. We are presenting three different platforms one right after the other; how, if at all, would our results change if we spaced these presentations out over time? In the real world, people might sometimes move quickly from one platform to another and, in so doing, might be exposed to similarly biased content. With spaced exposures – depending on the magnitude of the time intervals between exposures – one could argue that the impact might be larger or smaller than the impact of immediately consecutive exposures [44]. We also presented only three platforms; would the impact of more exposures produce diminishing returns? We also did not attempt to measure how long the impact of our manipulations would last. If you are trying to influence voters, that is an important issue.

We also did not examine the possible impact of inconsistent content. What if two platforms presented content favoring Candidate A and a third platform presented content favoring Candidate B? Might inconsistent content produce only small or no effects, just as content in control groups often does [45]? Bear in mind that a single platform has a high degree of control over the content it presents to its users; whereas, content impacting users on different platforms might vary in ways that undermines the combined effects of such content.

As always, we worry about ways in which participants we obtain from MTurk might differ from the population of eligible voters in the US [46, 47]. On this issue, we can note only that we have used a third party quality assurance company (CloudResearch) to screen out bots and suspicious users, and that we ended up with a diverse sample of 536 people who claimed to be registered US voters (see S1–S5 Tables). Given the pattern and size of the effect we found, we have no reason to believe that it occurred because of defects in our sample. We acknowledge that because we only had US voters in our sample, further research should be conducted to determine whether our results are applicable to voters in other countries.

Perhaps more important is the fact that our design selected for a particular type of participant. The sample was composed not only of undecided voters – an important criteria for inclusion in most of our experiments – but almost exclusively of a particular type of undecided voter – namely, the “low-information” voter. Such voters are known to be different in some respects from high-information voters [48], but, again, we have no reason to believe that conducting our experiment with low-information voters casts doubt on our findings.

We note also that we only examined MPE in the present study using simulations of three platforms. Additional research is needed to determine the extent to which our findings are applicable to platforms such as YouTube, Facebook, Instagram, TikTok, and so on. We recently completed the building of a simulated Facebook environment so that we can begin to assess how various kinds of content might be affecting Facebook users.

Finally, we note that we detected post-manipulation changes in our participants by asking eight questions that measured opinions and voting preferences. How might other political behavior be influenced? In particular, to what extent might the shifts we found in voting preferences be predictive of shifts in actual votes? That, too, is a matter to be explored in future research.