Characterizing misinformation susceptibility: Signal detection theory

In this work, we used Signal Detection Theory (SDT) [9] to characterize the different facets of misinformation susceptibility. SDT is a mathematical framework originally developed in the context of sensory psychology to disentangle how different factors influence people’s ability to discriminate between true information (signal) and misinformation (noise). Applied to the study of misinformation susceptibility, SDT distinguishes two distinct aspects in the identification of fake news: (1) the ability to distinguish between real news and fake news, and (2) response biases to judge news as real or fake regardless of veracity [10]. From this point of view, understanding why people are susceptible to believing false information involves analyzing how they categorize true and false statements. SDT identifies four possibilities: correctly identifying true information (hit), correctly identifying false information (correct rejection), incorrectly identifying true information as false (miss), and incorrectly identifying false information as true (false alarm). From this perspective, the problem of misinformation susceptibility is equivalent to identifying the causes of false alarms [11–13].

As stated, SDT suggests two main reasons for false alarms. First, people might struggle to distinguish accurate from inaccurate information, leading to more false alarms and more misses. This difficulty would manifest as a low discrimination sensitivity index, , which measures the ability to discriminate between true and false information. Discrimination sensitivity is defined as:

(1)

Where H is the proportion of hits and FA is the proportion of false alarms. In this equation, both H and FA are transformed into z-scores using the inverse cumulative distribution function such that a proportion of 0.5 is converted to a z-score of 0 [14]. Thus, a of zero indicates chance-level performance, and higher scores reflect greater accuracy.

Second, people may tend to believe information is true, regardless of its veracity. This would result in a high number of both hits and false alarms, reflected in a low response bias index c, which indicates the threshold for judging information as true or false. Response bias is defined as:

(2)

A c of zero signifies no bias, a positive c indicates a bias towards judging information as false, and a negative c indicates a bias towards judging information as true. Although misinformation susceptibility might be driven by a general tendency to accept all information as true, it seems more likely that this depends on the congruence of the new information with respect to prior beliefs [12]. According to SDT, this would likely be reflected in a lower c index (indicating a stronger bias toward belief) for belief-congruent information relative to belief-incongruent information [10].

Taking an SDT approach allows us to investigate how several psychological factors affect people’s tendency to accept false information as true, as well as characterize this influence. In particular, we seek to identify whether each psychological factor in our study that influences people’s false alarm rates also

Psychological drivers of misinformation

What makes some people more susceptible to belief in health-related misinformation? One influential view suggests that misinformation susceptibility is the result of a lack of engagement in analytic reasoning [15]. For example, according to dual-process theories, human reasoning is composed of two distinct types of processes: Type 1 and Type 2 [16–18]. On the one hand, Type 1 processing is characterized as automatic, with intuitive answers quickly coming to mind in response to a given stimulus. Importantly, judgments based on Type 1 processing are accurate most of the time, but can lead to mistakes in certain cases. In contrast, Type 2 processing is characterized as deliberate, which tends to result in a more rigorous evaluation of information —but this system can also lead to inaccurate responding [16]. Research in the dual-process tradition has shown that, in general, misinformation acceptance is most likely to occur when intuitive and autonomous processes are elicited [19]. For example, when people are exposed to a large volume of information in a short period, Type 1 processing is more likely to be activated, which could make them more prone to trusting misinformation [20]. Conversely, when individuals have the opportunity to think analytically and carefully about a piece of information (i.e., when Type 2 processing is active), they tend to be more successful at identifying false information [21]. Accordingly, in this work, we investigated the effect of cognitive load on susceptibility to health-related misinformation. We hypothesize that loading participants’ working memory will increase the false alarm rate.

In addition to external influences on analytic reasoning, the individual-difference dimension of cognitive reflection also plays a critical role in susceptibility to misinformation. This refers to the capacity to resist intuitive responding and instead engage in deliberate thought when evaluating information [22]. A substantial body of research has demonstrated that individuals who score higher in cognitive reflection are significantly more accurate at distinguishing real news from fake news [15,23–26]. Notably, some studies suggest that encouraging conscious deliberation can reduce susceptibility to fake news even when controlling for individuals’ partisan affiliations [21]. Based on these findings, we hypothesize that cognitive reflection will be negatively associated with false-alarm rates and positively associated with discrimination sensitivity.

Another influential account, known as motivated reasoning, is based on the idea that people’s susceptibility to misinformation is not due to a faulty process of careful consideration, but rather to starting their reasoning with a pre-set goal, which leads them to interpret new information in service of reaching that goal [1,27,28]. For example, some people wrongly believe that the deployment of 5G technology is linked to the spread of the coronavirus, and they frame new information in line with this belief [29]. Some beliefs could be considered especially effective “sources” for motivated reasoning. In particular, pseudoscience, conspiracy theories, and paranormal beliefs are collectively referred to as epistemically unwarranted beliefs [30,31]. The interconnectedness of these beliefs highlights the shared attributes among pseudo-scientific, conspiratorial, and paranormal thoughts [30]. For example, individuals with a predominant disposition to believe in epistemically unwarranted beliefs may be highly motivated to accept misinformation, such as claims that the SARS-CoV-2 virus was a bioweapon artificially created by the Chinese government [32] or intentionally created by powerful people [33]. Based on these considerations, we hypothesized that pseudoscience, conspiracy theories, and paranormal beliefs are positively correlated with false alarm rates and negatively associated with discrimination sensitivity.

Beyond investigating psychological factors related to the analytic reasoning and motivated reasoning accounts, we also included two additional individual-difference measures. The first one is naïve skepticism. This mindset is characterized by individuals’ tendency to disregard information without applying critical thinking or evidence-based reasoning, rejecting claims simply to be skeptical [34,35]. Naïve skeptics not only tend to question official sources of information (e.g., mainstream media outlets, scientific and governmental organizations) but are also particularly vulnerable to believing misinformation [35,36]. It has been pointed out that naïve skeptics tend to disregard information that is inconsistent with their idiosyncratic and group identities [37]. From an SDT point of view, this tendency entails a low veracity threshold for belief-congruent information, which should lead to more false alarms. At the same time, this tendency entails a high veracity threshold for belief-incongruent information which should lead to more misses. Notably, these two effects can only happen simultaneously if the discrimination sensitivity is low. Based on these considerations, we hypothesized that naïve skepticism is negatively associated with discrimination sensitivity.

The second one is bullshit receptivity, which refers to the tendency to perceive statements as deep or significant, even when they are in fact nonsensical or meaningless [38]. Concretely, people with higher levels of bullshit sensitivity might interpret abstract language as coherent, even when it lacks any depth or clarity. For example, consider the following abstract phrase: “The invisible dance of the universe whispers truths beyond reality”. People with higher levels of bullshit sensitivity may perceive the statement as conveying a deeper metaphysical insight, even though the phrase itself is ambiguous and lacks clear meaning. Previous research on political misinformation has found negative associations of bullshit receptivity with analytical thinking [25] and response bias [39]. Based on this evidence, we hypothesize that bullshit sensitivity should be positively associated with false alarm rate and negatively with response bias.

The current research

The present study had two main goals. In the first place, we sought to gauge the influence of working memory load on people’s susceptibility to misinformation. While previous research has used time pressure to manipulate cognitive elaboration [11,12,21,40], we induced working-memory load more directly through a secondary digit recall task. Similarly to previous research examining the effects of working memory load on reasoning processing and decision making [41,42], the current work compared the credibility that people assign to health-related misinformation in two scenarios: one taxing working memory (i.e., inducing Type 1 processing) and the other involving open or free processing (i.e., facilitating Type 2 processing). More concretely, participants rated 32 COVID-19 headlines as true or false. In the working memory load phase, participants received a 5-digit number to remember before rating each COVID-19 statement, and were then asked to recall it as accurately as possible after evaluating the statement. In the no working memory load phase, participants received and recalled the 5-digit number before evaluating each COVID-19 statement.

In the second place, the current study investigated whether the individual-difference dimensions cognitive reflection, epistemically unwarranted beliefs, naïve skepticism, and bullshit receptivity were associated with misinformation susceptibility. Specifically, we analyzed the relationships between these individual-difference dimensions and false-alarm rate, discrimination sensitivity, and response bias. To summarize, our hypotheses are as follows:

1. Loading working memory increases false alarm rates on the fake news identification task.
2. Cognitive reflection is negatively associated with false alarm rates and positively associated with discrimination sensitivity.
3. Epistemically unwarranted beliefs are positively associated with false alarm rates and negatively associated with discrimination sensitivity.
4. Naïve skepticism is negatively associated with discrimination sensitivity.
5. Bullshit receptivity is positively associated with false alarm rates and negatively with response bias.

Ethics

Ethical approval was obtained by the Ethics Committee of the University of Tarapacá (reference number N°14/2022, internal codes CEC C11-2022 and CEC C12-2023) prior to the start of the study. Written informed consent was obtained from all individual participants prior to their participation in the study. All procedures performed with human participants were in accordance with the ethical standards as laid down in the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards.

Participants

We calculated the initial sample size using the sofware Gpower 3.1 [43]. Assuming a small effect size of 0.15, 2 repeated measures, 2 covariates (naïve skepticism and bullshit receptivity), a type I error of , and a statistical power of , we estimated a sample of a sample of 598 participants (299 for each of the 2 starting conditions). Additionally, we planed to recruit 10% more participants to compensate for potential sample losses. Our actual sample consisted of a total of 634 people working or studying at one of 5 universities in Chile (Universidad de Tarapacá, Universidad de Talca, Universidad de Católica del Norte, Universidad de Chile, and Universidad de Magallanes) participated in return for approximately 8 dollars. The mean age was 21.9 years (SD 4.42). Gender wise, 57.2% (n = 363) identified as female, 39.4% (n = 250) as male, and 3.4% (n = 22) as non-binary or other. Education-wise, 93.9% (n = 596) were university students. Regarding religion status 46.8% (n = 289) declared to be Catholic or Christian, and 53,2% (n = 329) identified as agnostic, atheist or without religion. Regarding political tendency, 49.6% (n = 289) identified as left lining, 8.2% (n = 48) as center lining, 15.6% (n = 91) as right lining, and 26.6% (n = 155) as apolitical. Data collection took place between 28 August 2023 and 15 December 2023 and this study received ethical approval from the University of Tarapacá Ethics Committee.

Materials

Procedure

Participants started by completing the COVID-19 headlines evaluation task and the 5-Digit recall task. As explained earlier, there were two within-participant conditions: working memory load and no working memory load. The COVID-19 headlines were randomly assigned to both conditions (i.e., 16 per condition, half true, half false) for each participant. The order of these conditions was counterbalanced across participants. After completing the main experimental task, participants answered the questionnaires. Our study was implemented in Psychopy^® [49] and is available in: https://anonymous.4open.science/r/health_missinfo_exp-7CDC.

SDT indexes calculation

For each participant, we calculated hit rates (H) as the proportion of true headlines judged as true and false-alarm rates (FA) as the proportion of false headlines judged as true. When a participant gave no true responses (a proportion of 0), we followed the recommendations of [50] and converted the values to , where N is the number of headlines. Similarly, if the proportion of true responses was 1, we converted values to . Discrimination sensitivity scores were calculated using Eq (1) and response bias was calculated according to Eq (2).

Data analysis

To characterize the study variables, we conducted descriptive analysis of central tendency and dispersion. We then analyzed participants’ performance using a generalized linear mixed-effects model. Subsequently, conducted a Welch’s t-test to evaluate the effect of the working memory load task on participants’ digit recall performance and reaction times to the COVID-19 headlines. The measurement models of the individual differences instruments were tested using exploratory structural equation modeling (ESEM) with GEOMIN rotation [51] and, for CRT, which is unidimensional, confirmatory factor analysis (CFA). These models were estimated from the polychoric correlation matrix using the weighted least squares WLSMV estimation method, which is robust to non-normal discrete variables and performs adequately for ordinal variables [52]. After establishing the measurement models’ goodness of fit, we calculated latent-variable correlations between our individual-difference dimensions and the false-alarm rate, discrimination sensitivity, and response bias, while restricting covariances among latent variables to zero. We used Cohen’s [53] recommended r values of 0.10, 0.30, and 0.50 to demarcate small, medium, and large effects, respectively. Finally, to evaluate the joint influence of our individual-difference dimensions on the false-alarm rate, discrimination sensitivity, and response bias, we used structural equation modeling. These models were estimated with the weighted least squares WLSMV estimation method. The fit of these models was assessed following the cut-point recommendations proposed by [54] for the comparative fit index (CFI), Tucker–Lewis index (TLI), and root mean squared error of approximation (RMSEA): CFI and TLI > 0.90 is acceptable and >0.95 is satisfactory; RMSEA < 0.08 is acceptable and <0.06 is satisfactory.

5-digit recall task

To assess whether our working memory load manipulation influenced participants’ performance, we compared the number of correctly recalled digits and the time to answer to the COVID-19 headlines across the working memory load and no working memory load conditions. First, a paired-samples t-test indicated that the average number of correctly recalled digits was lower in the working memory load (M = 4.21, SD = 0.658) condition than in the no working memory load (M = 4.91, SD = 0.155) condition (, p < 0.001, d = 1.117). Second, another paired-samples t-test indicated that participants took longer to answer the COVID-19 headlines in the working memory load (M = 8.68, SD = 3.430) condition than in the no working memory load (M = 7.98, SD = 2.960) condition (t(634) = 5.466, p < 0.001, d = 0.217). These two results indicate that our manipulation of working memory load indeed affected participants’ performance.

Linear mixed effects modeling

To analyze participants’ performance accounting for the nested structure of our working memory load experiment, in which each participant responded to 16 COVID-19 headlines within each load condition, we employed a generalized mixed-effects regression model with the logit link function. This model was fitted using the glmer function in the lme4 package (Version 2.0.1) in R. In this model, the dependent variable was whether each response was correct or not. The model included working memory load (with vs. without load) as a fixed effect. We incorporated random intercepts and slopes for participant ID and item ID to capture variations across participants and problems. According to this analysis increasing working memory load did not significantly affect the likelihood of a correct response (, SE = 0.05, , p = .256).

Effects of working memory load on false-alarm rates and SDT indices

Regarding our main COVID-19 headlines judgment task, a paired-samples t-test showed that there was no difference in the proportion of false alarms between the working memory load (M = 0.092, SD = 0.121) and the no working memory load (M = 0.084, SD = 0.111) conditions (, p = .110, ) (Fig 1). A second paired-samples t-test showed that there was no difference in discrimination sensitivity between the working memory load (M = 2.76, SD = 0.858) and the no working memory load (M = 2.78, SD = 0.799) conditions (t(634) = 0.512, p = .609, d = 0.02. Finally, a third paired-samples t-test showed that there was no difference in response bias between the working memory load (M = 0.05, SD = 0.262) and the no working memory load (M = 0.07, SD = 0.264) conditions (t(634) = 1.45, p = .147, d = 0.06. Overall, these results indicated that despite its effects on the number of correctly recalled digits and the time to answer the COVID-19 headlines, working memory load did not affect participants’ susceptibility to judge false COVID-19 information as true.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Effects of working memory load on false alarms rate, discrimination sensitivity, and response bias.

https://doi.org/10.1371/journal.pone.0356642.g001

Relations between false-alarm rates and SDT indices

Following previous research that used SDT to model misinformation susceptibility [11,13], we sought to replicate the effects of truth sensitivity and acceptance threshold on the acceptance of misinformation. Consequently, we conducted multiple-regression analyses using SDT indices and c to predict individuals’ false-alarm rates. As in previous research, to ensure the mathematical independence of predictors and the outcome, we calculated all scores separately for responses to odd-numbered and even-numbered headlines. We built two regression models, using responses to odd-numbered items as predictor variables and responses to even-numbered items as criterion variables, and vice versa. Our results indicated that acceptance of health-related misinformation showed significant negative associations with and c for both models (odd-to-even: r² = .182, F(2,632)=70.5, p < .001, , , , ; even-to-odd: r² = .173, F(2,632)=65.9, p < .001, , , , ). It is worth noting that the odd-even reliability of the SDT measures is r = .401 for and r = .207 for c. It is likely that the small variance of the proportion of hits and false alarms (see Table 1) and the number of COVID-19 headlines affected the reliability of the SDT measures.

Relations between false-alarm rates and individual-difference dimensions

To establish the conditions for conducting latent variable analysis and structural equation modeling later, we first assessed the fit of the measurement models. As shown in Table 2, all measurement models exhibited adequate fit. Therefore, to identify the univariate effects of the 7 individual-difference measures on false-alarm rates, we computed the correlations among latent variables, restricting their covariances to zero. As can be seen in Table 3, this analysis showed medium direct effects in the case of pseudoscience and conspiracy theories, as well as small direct effects in the case of paranormal beliefs, naïve skepticism-science, naïve skepticism-government, and bullshit receptivity. Furthermore, we observed a small inverse effect on cognitive reflection.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Measurement models fit.

https://doi.org/10.1371/journal.pone.0356642.t002

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Latent variable correlations between individual difference dimensions and false-alarm rate, discrimination sensitivity, response bias.

https://doi.org/10.1371/journal.pone.0356642.t003

Relations between discrimination sensitivity and individual-differences dimensions

Similarly, we calculated a correlation of latent variables between the 7 individual-difference measures and truth sensitivity (see Table 3). As can be seen, we found a medium inverse effect in the case of pseudoscience and small inverse effects in the case of conspiracy theories, paranormal beliefs, naïve skepticism-science, naïve skepticism-government, and bullshit receptivity. Furthermore, we observed a small direct effect in the case of cognitive reflection.

Relations between response bias and individual-Differences dimensions

We also calculated a correlation of latent variables between the 7 individual-difference measures acceptance threshold (see Table 3). This analysis showed only small inverse effects in the case of pseudoscience, conspiracy theories, naïve skepticism-science, and bullshit receptivity.

Joint effects of individual differences dimensions on false-alarm rates and discrimination sensitivity

Finally, to establish an explicative model of the joint effects, we estimated two structural equation models with correlated latent variables: one with the false alarm-rate as the dependent variable (Fig 2) and another with discrimination sensitivity as the dependent variable (Fig 3). In the case of the first model, the individual differences dimensions explain jointly a 25.8% of the variability in the false alarm-rate, showing adequate levels of fit (; CFI = 0.941; TLI = 0.934; RMSEA = 0.038 [90%CI, 0.035–0.041]; and SRMR = 0.056). In the case of the second model, the individual differences dimensions explain jointly a 22.1% of the variability in the false alarm-rate, showing adequate levels of fit (; CFI = 0.940; TLI = 0.934; RMSEA = 0.039 [90%CI, 0.036–0.042]; and SRMR = 0.057). We note that in both covariate models, we found suppression effects, in particular, negative suppression. This type of suppression occurs when the independent variables are highly correlated, which can cancel or reverse their effects on the dependent variable [55,56]. This prevented us from interpreting the specific effects of the predictors of both covariate models. Instead, we refer the reader to Table 3 for the univariate effects (see previous three paragraphs).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Covariate model: False-alarm rate.

FA = False-alarm rate.

https://doi.org/10.1371/journal.pone.0356642.g002

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Covariate model: Discrimination sensitivity.

= Discrimination Sensitivity.

https://doi.org/10.1371/journal.pone.0356642.g003