Impact of COVID-19 vaccination on preventive behavior: The importance of confounder adjustment in observational studies

Laura Sità; Marta Caserotti; Manuel Zamparini; Lorella Lotto; Giovanni de Girolamo; Paolo Girardi

doi:10.1371/journal.pone.0313117

Abstract

The COVID-19 pandemic has underscored the critical role of observational studies in evaluating the effectiveness of public health strategies. However, although many studies have explored the true impact of vaccination on preventive behavior, their results may be skewed by potential biases and confounding variables. This study examines the application of covariate adjustment and propensity score (PS) estimation, particularly through inverse probability treatment weighting (IPTW), to assess their performance in reducing bias in a framework featuring ordinal outcomes and cumulative logistic regression models, as commonly used in observational studies related to social sciences and psychology. Before applying these methods to the case study, we conducted a simulation study that accounted for the presence or absence of model misspecification in an observational scenario with ordinal outcomes, binary treatment, and a continuous confounder. Our findings demonstrate the effectiveness of combining covariate adjustment with PS methods in reducing bias and improving causal inference. These methods were subsequently applied to an Italian observational study on COVID-19 vaccine hesitancy conducted during the initial phase of the vaccination campaign (April-May 2021). Our analysis revealed that vaccination status had a limited short-term impact on the adoption of preventive measures. This study highlights the importance of employing appropriate adjustment techniques in observational research, particularly when evaluating complex behavioral outcomes. The results support the combined use of covariate adjustment and PS methods to enhance the reliability of findings, ultimately contributing to more informed public health decision-making.

Citation: Sità L, Caserotti M, Zamparini M, Lotto L, de Girolamo G, Girardi P (2024) Impact of COVID-19 vaccination on preventive behavior: The importance of confounder adjustment in observational studies. PLoS ONE 19(11): e0313117. https://doi.org/10.1371/journal.pone.0313117

Editor: Srebrenka Letina, University of Glasgow, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND

Received: January 25, 2024; Accepted: October 18, 2024; Published: November 25, 2024

Copyright: © 2024 Sità et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: The data and the R code for reproducing simulations and results are available for download at https://osf.io/68mj5/.

Funding: This work was supported by Fondazione Cariplo (grant n° 2020–5195), awarded to Dr. Giovanni de Girolamo, the Italian Ministry of Health (Ricerca Corrente), awarded to Dr. Giovanni de Girolamo, and IRCCS Centro San Giovanni di Dio Fatebenefratelli Institutional resource, awarded to Dr. Giovanni de Girolamo. This work was partially supported by DAIS - Ca’ Foscari University of Venice within the IRIDE program, awarded to Paolo Girardi. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: All authors declare that they have no competing interests to be disclosed.

Introduction

During the Coronavirus Disease 2019 (COVID-19) pandemic, the urgent need to develop strategies to contain the spread of the virus led many researchers to conduct observational studies. Indeed, these kinds of studies have played a crucial role in understanding the factors related to the pandemic’s spread [1], assessing the effects on people’s health [2], and evaluating the effectiveness of policies related to restrictive measures [3]. Specifically, the analysis of observational data has been the most commonly employed tool to evaluate the efficacy of public health strategies such as social distancing, mask-wearing, and COVID-19 vaccination campaign. However, the most appropriate study design for estimating the true extent of the treatment effect, according to a causal inference framework, is the randomized controlled trial (RCT), which ensures comparability of outcomes between treated and untreated participants. In particular, the assumption of ignorability [4] guaranteed by the RCT is one of the main necessary conditions for obtaining unbiased results by comparing experimental outcomes between treated and untreated participants [4]. Instead of focusing on individual outcomes, the potential outcome framework [5] allows comparison of a measure related to an outcome of interest between two groups. The Average Treatment Effect (ATE) is a measure used in causal inference to quantify the average effect of a treatment or intervention on an outcome across a population. It represents the average difference in outcomes between the group that receives the treatment and the group that does not as follows where Y₁ and Y_o are the potential outcomes when the treatment is applied and not applied, while E[Y₁] and E[Y₀] are the averages in case of treatment or no treatment, respectively.

In observational studies estimation of the ATE is likely to be biased because the researcher has little or no control over the independent variable(s), but observes and analyzes data as they naturally occur, due to ethical implications, lack of randomization, and time or cost constraints [6, 7]. In fact, the lack of controlled experimental conditions can alter the observed relationships between treatment and the outcome due to the presence of confounders. Confounders are characteristics that influence both the likelihood of treatment or exposure and the outcome. These include sociodemographic factors such as age, gender, education, and socioeconomic status, which are general confounding variables in observational studies according to the American Psychological Association (APA) guidelines. However, confounders should be carefully selected based on theoretical considerations, including the use of causal diagrams [8].

Implementing adjustment methods in observational studies is crucial for several reasons. First, it helps mitigate confounding, where extraneous variables can skew results. Second, it enhances the accuracy and reliability of findings, thus increasing their generalizability. Third, proper adjustment methods allow researchers to approximate the rigor of randomized controlled trials, thereby strengthening causal inferences. Ultimately, these adjustments are essential for producing valid, credible, and actionable insights from observational data. Covariate adjustment is one of the most popular methods because it allows researchers to quantify effects related to the dependent variable while controlling for a selected set of covariates [9]. However, there are circumstances in which the results of regression models may be affected by issues such as collinearity among covariates/confounders, over-adjustment bias and data sparsity. In such cases, alternative methods should be considered. A useful solution is provided by propensity score (PS) techniques [10], which are defined as the probability of receiving a particular treatment given a set of observed covariates [11]. The estimation and use of propensity scores can be beneficial for balancing covariates, reducing confounding, and handling high-dimensional covariates [12]. This method is particularly advantageous in studies that collect a multitude of potential confounding variables, as it summarizes their effects into a single score [12]. It is also highly effective when the treated and untreated groups have a similar covariate distribution [13], and the confounding effects are limited [14]. Depending on the study design, a common approach involves matching treated (or exposed) and untreated (or unexposed) individuals to have similar estimates of propensity scores. An alternative approach, called Inverse Probability of Treatment Weighting (IPTW), estimates weights equal to the inverse of the probability of receiving the treatment they received, thereby balancing the distribution of covariates between treated and untreated groups.

In the initial phase of the pandemic, the adopting adjustment procedures was essential to assess, through observational studies, how psychosocial factors, such as risk perception and coping, affected people’s adherence to protective behaviors like wearing mask, practicing physical distancing, and frequent hand washing [15, 16]. Similarly, when the vaccine became available, researchers examined how vaccination influenced people’s adherence to these behaviors as the pandemic progressed. While vaccination is an effective measure for reducing disease severity and preventing its spread, achieving effective pandemic control requires a comprehensive approach that includes going adherence to preventive measures and addressing vaccine hesitancy. To this end, the aim of this paper is to assess whether vaccination status (received vs. refused) affected adherence to preventive measures against the spread of COVID-19 in Italy during the initial stage of the vaccination campaign (April-May 2021) by using different techniques for confounders adjustment. For this purpose, we analyzed a dataset derived from an Italian observational study on COVID-19 vaccine hesitancy, sponsored by the World Health Organization (WHO) Regional Office for Europe [17]. The structure of this paper is as follows. In the Materials and Methods section, we present the dataset and the adjusting methods used, which include a regression model framework, covariate adjustment, propensity score based on IPTW, and their combination [18]. In the third section, we report the settings and results of a simulation study using a framework commonly employed in social sciences and psychology, which is based on ordinal outcomes and cumulative logistic regression models. The aim of the simulation is to demonstrate how bias can be reduced using two adjustment techniques, and their combination, in two different observational scenarios. In the Results section, we present the results of our study with the scope to estimate the effect of COVID-19 vaccination on the adherence of preventive measures. Finally, the last section offers a commentary on the primary results, with references to the open issues in causal inference and methodological recommendations.

Materials and methods

Study setting and participants

The dataset was obtained from the COVID-19 Monitoring run in Italy (COMIT) survey. The survey was conducted in four waves from January to May 2021, and we focused on the third and fourth waves, as the vaccination campaign began during this period in Italy. The questionnaire has been described in detail in previous publications [17, 19]. Out of 5,007 respondents, we selected 1,468 individuals who had been offered the COVID-19 vaccine by local health authorities. Of these, 88.7% reported having received the vaccine. The questionnaire collected various sociodemographic variables (e.g., age group, gender, employment status, education level, and presence of chronic diseases), as well as information on past COVID-19 infection (e.g., yes, no, and don’t know). In addition, a panel of validated questions was asked to generate individual factor scores on perceived COVID-19 risk, trust in health institutions, trust in health information sources (e.g., Ministry of Health and WHO), trust in media information sources (e.g., traditional and social media), frequency of use of health information sources, and frequency of use of media information sources [31]. Interviews were conducted by Doxa SPA. Sampling weights were produced comparing the study sample and the Italian population on stratification variables (gender, age, area of residence, size of living centers, education level, and employment status). This study was approved by the Ethics Committee of the coordinating institution of the national survey (Ethics Committee of the IRCCS San John of God Fatebenefratelli of Brescia, Italy, protocol 286/2020, registration ISRCTN 39724), and all participants provided informed consent. The outcome variables of interest, related to adherence to protective behaviors, were measured by the following questions/statements: 1) I adhered to physical distancing in public (physical distancing); 2) I wore a face mask in public (mask use); 3) I used hand sanitizer when soap and water were not available (hand disinfection). All questions were rated on a 7-point response scale (from 1 = Never to 7 = Always). Fig 1 shows the theoretical relationships in terms of Directed Acyclic Graph (DAG) between outcome (adherence to protective measures), treatment (vaccination status), and a set of confounders.

Download:

Fig 1. DAG between adherence to protective measures, vaccination status, and confounders.

https://doi.org/10.1371/journal.pone.0313117.g001

Adjustment procedures

In all studies that control for confounders, the analytical strategy used to estimate the causal effect involves the identification, measurement, and appropriate adjustment for a sufficient set of variables or related proxies [20]. Causal diagrams, including DAGs, can be used to support causal inference, connecting all relevant variables with arrows that represent the relationships, supported by background knowledge, theoretical assumptions, and constraints. Since confounding arises from variables that influence both exposure and outcome, strategies to reduce confounding involve either breaking the association of confounders with outcome (e.g., regression adjustment) or disrupting their association with exposure (e.g., matching, adjustment, or weighting based on PS).

Covariate adjustment in regression analysis.

The primary objective is to estimate the impact of the treatment effect on the dependent variable while adjusting for other relevant variables that may affect the results as confounders. The application of multiple linear regression requires the specification of several assumptions, including linearity, independence among observations, and homoscedasticity, to be correctly identified [19]. Although heteroscedasticity can result in an overestimation of the standard error and the significance of the estimates, misrepresenting the model by assuming a linear relationship when it is not can pose a significant issue in covariate adjustment methods. Multiple linear regression with covariate adjustment can be described as follows: where Y_i is the outcome, β is the effect of the treatment variable X (in the simplest case: treatment X = 1, no treatment X = 0), while other coefficients γ₁,…,γ_k account for the influence of confounders on the outcome Z₁,…,Z_k. Finally, the random error term ϵ_i is generally assumed to be a white noise. In this setting, the estimated value for β, , is the ATE estimate. The settings described above can be extended to generalized linear models (GLM). As previously mentioned, Foster noted that although regression has several positive features, its functional form can be misspecified and a few outliers can affect the estimates [21]. In addition, the presence of too many covariates for a given sample size may render the estimate unreliable [22]. One way to potentially address these and other issues is to use alternative methods, such as propensity score (PS) procedures [23].

Propensity score methods.

The primary goal of propensity score (PS) is to estimate the probability of receiving or not receiving treatment based on confounders. The usual assumptions in causal framework, such as exchangeability, consistency assumption, and positivity [10, 22], are necessary for the validity of results obtained through PS methods. In addition, the model used to estimate the propensity scores should adequately capture the relationship between the covariates and the treatment assignment, as its misspecification can lead to biased estimates [23]. For an illustration of PS application and a more comprehensive explanation, one may consult the work of Harder and colleagues [24]. PS can be integrated into statistical analysis in multiple ways, the most prevalent of which are matching on PS, stratifying on PS, introducing covariate adjustment using PS, and calculating IPTW using PS [25]. For the simulation and the real data application, we selected IPTW, as weighting each individual (treated and control) allows us to consider the full sample population without losing any subjects (e.g., through PS stratification). All statistical analyses and simulations were performed using R Statistical Program, version 4.3 [26], with the Weightit R package. All the simulation code, statistical analyses, figures, and tables can be reproduced using the materials available in the online OSF repository https://osf.io/68mj5/.

Regression models

Given the ordinal nature of the response variables, we employed weighted cumulative logistic models (CLM) assuming proportional odds and using the sampling weights provided by the survey company. For the model with covariate adjustment we selected a set of variables that included sociodemographic characteristics (e.g., age, gender, education level, and employment status), past COVID-19 contagion, and standardized factor scores related to COVID-19 perceived risk, conspiracy, trust in health institutions, trust in health information sources, trust in media information sources, frequency of use of health information sources, and frequency of use of media information sources. More details on the estimation of the factor scores can be found in the study by de Girolamo et al. [17]. The inclusion of the latter set of variables is essential, as they generally affect both adherence to protective behaviors and COVID-19 vaccination. In addition we included in the PS and CLM models a series of interaction terms between gender and some key variables (age, educational level, COVID-19 perceived risk, and conspiracy) [27, 28] The results of the CLM regression models are presented in terms of Odds Ratios (ORs). A 95% Wald confidence interval (CI) was used for ORs estimated with the covariate adjustment, while for the PS method, we accounted for the uncertainty of weights creation by calculating the relative 95% CI using 500 bootstrap replications.

Simulations

The theoretical model

The proposed simulation aims to serve as an illustrative example of the application of the two previously described approaches, covariate adjustment, PS weighting, and their combination, in the context of bias reduction. We considered the same classical theoretical model, represented by the DAG in Fig 1, in which the treatment exerts a direct effect on the outcome, while we considered for simplicity the presence of only one confounder (i.e., age) affecting both the outcome and the treatment.

In behavioral and social science, it is common to consider measurements based on a Likert scale as dependent variables, as reported in the dataset under study. Accordingly, we considered a generative cumulative logistic regression model based on the assumption of proportional odds. In a first simulative scenario, the generative model is denoted as follows for i = 1, … n, where n is the sample size, α_j is an intercept that varies with j = 2,…, J = 5 with Y_i is an ordinal outcome with 5 categories, X is a binary predictor denoting the presence of a treatment (1 = treatment; 0 = no treatment), while the function g() is a logit link function. The variable Z_i is a confounder while the values β and γ regulate the strength of the treatment and of the covariate-to-outcome confounding, respectively.

In this formulation, X is expressed as a binary variable. The confounder variable Z_i is assumed to follow a uniform distribution between 0 and 1. The values related to the coefficients α_j are set to have a symmetric baseline probability of 0.15, 0.20, 0.30, 0.20, and 0.15, for each category from 1 to 5, respectively.

To include a covariate-to-treatment confounding effect, the term Z_i modifies the probability of an individual being under-treatment as follows in which the individual probability of treatment assignment τ_i varies with Z_i. Since Z_i is distributed according to a uniform random variable (Z_i ~ U(0,1)), the average probability to be assigned to the treatment is 50%.

In addition to this first scenario in which we generated and estimated the coefficients with same model, we considered a second scenario modifying the generative cumulative model and the relationship between the treatment and confounders keeping fixed all the other conditions. Compared to the first simulation settings, the second scenario proposed unfavorable conditions for procedures based on covariate adjustment since the estimating model is misspecified.

Simulation settings and bias estimation

In our simulation, for each scenario we varied the size of the sample (N = 200, 500, 1,000, and 2,000), while for the effect size we accounted for low, medium, and high effects (β = 0.25, 0.5, and 1.0 that result in Odds Ratio = exp (β) = 1.24, 1.65, and 2.72, respectively), varying also the strength of the confounders to treatment parameter (γ = 1 and 2). Considering the conditions of the first scenario with γ = 1 and β = 0.5, an example of the relationship among these three variables is shown in Fig 2 in which the pairwise comparison among X, Y, and Z is made by graphical representations (boxplot, histogram, bar plot, and mosaic plot). A discrete correlation was observed between the confounder (Z) and the outcome (Y) (Spearman’s correlation ~ 0.23) with higher values of the continuous confounder at increasing of the outcome category. The distribution of the confounder variable visually differs considering the treated (X = 1) and untreated group (X = 0). Lastly, the distribution of the outcome (Y) reports higher frequency for modalities 4–5 for treated individuals compared to non-treated ones. The data were simulated using the function genOrdCat of the package simstudy under R environment.

Download:

Fig 2. Bivariate and marginal distributions of a simulated dataset of the first scenario with n = 1000, β = 0.5, and γ = 1.

https://doi.org/10.1371/journal.pone.0313117.g002

Four different approaches were used to estimate β: method A) cumulative logistic regression model without covariate (age) adjustment (Y_i = βX_i); method B) cumulative logistic regression model with covariate (age) adjustment (Y_i = βX_i + γZ_i); method C) cumulative logistic regression model PS weights without covariate (age) adjustment; and method D) cumulative logistic regression model with PS weights and covariate (age) adjustment. The cumulative regression models of methods A and B were estimated using the function clm of the R package ordinal, while for models C and D we used the function ordinal_weightit of the R package Weighit.

The first step of a PS weighting method can be achieved using a logistic regression model in which treatment status (treatment X_i = 1, or control X_i = 0) is regressed on the confounding covariates. In recent years, the literature has reported a wide range of procedures spacing from adaptive binary regression, machine learning, and predictive models with strategies of variable selection and regularization to predict the propensity scores. Adopting a simple and flexible approach, we estimated each individual weight (, and ) using a Generalized Boosted Model (GBM) which provides more precise estimates than classical logistic model, especially when the relationship between X and Z is complex and non-linear. GBM is based on decision trees and creates a predictive procedure by combining multiple simple models using an iterative algorithm. We set the number of trees to 200 and a learning rate of 0.1, since the relationship between X and Z was simple and to speedup simulations. For more details about the choice of GBM parameters see Elith et al. [29]. Then, the stabilized weights (WS) were adopted to improve precision of the causal effect estimates, and they were calculated as where p_x is the proportion of treated individuals ().

The estimation bias of β is evaluated using the Average Relative bias (ARebias) as follows where the is the estimation of the k-th simulation. As a measure of variability of the estimates we also calculated the Mean Squared Error (MSE) as follows

We performed K = 1000 simulations for each combination of β, γ, and n.

Simulation results

As reported in Table 1, when considering the simulation settings of the first scenario, method A (no adjustment) a positive and high degree of bias was found, with a higher average relative bias (ARebias) observed for higher values of γ. A decrease in ARebias values was observed with increasing effect size β (Table 1). Bias was consistently reduced with all adjustment procedures. The reduction in bias with effect size and sample size was generally monotonic, while an increase was observed when the confounder on the outcome parameter (γ) moved from 1 to 2. The performance in terms of ARebias of method D (PS weighting with covariate) was comparable to that of method B (covariate adjustment), while only PS showed a slightly worse performance. For large sample sizes (N = 2000), the bias was always less than 5%. Taking into account the variability of the estimates, method B showed the lowest MSE compared to the other adjustment methods. It is crucial to contextualize the interpretation of the results within the specific simulation setting in which the model was estimated, considering the same form of the generative one. This first scenario created a favorable condition for effective bias reduction by covariate adjustment. The results changed when the second scenario was considered. As shown in Table 2, the presence of model misspecification implied a strong increase in the relative bias for all methods. Method A (no adjustment) showed a strong and positive bias, which was higher when the confounder-to-treatment coefficients were equal to 2 (compared to γ = 1); a decrease in ARebias values was observed with increasing sample size and the treatment to outcome coefficient β (Table 2). Bias was partially reduced with all adjustment procedures. Bias reduction for covariate adjustment (method B) was only partially achieved, with ARebias values above 5% in most cases. ARebias values were also higher for models adjusted by PS weighting (method C), but generally only for smaller effect sizes (β ≤ 1). Method D (PS weighting with covariate) obtained the best performance in terms of bias reduction (≤5%), except for smaller effect and sample sizes. As reported in the previous scenario, the MSE was lower for method B than for the other three methods.

Download:

Table 1. First scenario.

ARebias (standard error) and MSE by combination of effect size (β),strength of the confounder-to-outcome parameter (γ) and population size (n) for the model without (A) and with adjustment method (B = covariate, C = PS, D = PS+covariate). In bold |ARebias| higher ≥ 5%.

https://doi.org/10.1371/journal.pone.0313117.t001

Download:

Table 2. Second scenario.

ARebias (standard error) and MSE by combination of effect size (β), strength of the confounder-to-outcome parameter (γ) and population size (n) for the model without (A) and with adjustment method (B = covariate, C = PS, D = PS+covariate). In bold |ARebias| higher ≥ 5%.

https://doi.org/10.1371/journal.pone.0313117.t002

Results

The main characteristics of participants by vaccination status are shown in Table 3: COVID-19 vaccinated individuals had higher levels of education (Standardized Mean Difference, SMD = 0.300) and were more likely to work in the health sector (SMD = 0.272) compared to those who refused vaccination. The difference in age and gender by vaccination status was limited (SMD = 0.033 and 0.158, respectively). Compared to those who refused the vaccine, vaccinated individuals were more likely to have no history of COVID-19 infection (SMD = 0.366) and to report no chronic diseases (SMD = 0.210). Respondents who received the vaccine reported higher scores for all protective measures compared to those who were not vaccinated, suggesting slightly higher compliance with these measures among vaccinated individuals. In addition, respondents who refused the vaccine perceived less risk from COVID-19, reported higher conspiracy scores, lower trust in health institutions and information sources (both health and media), and less use of media information (all SMDs > 0.3). To avoid numerical instability of the estimating algorithm due to low frequencies on the first 2 categories (1 and 2), we recoded the outcome variables (physical distancing, mask use, and hand disinfection) to be a 5-point Likert scale collapsing the first two categories into the third one.

Download:

Table 3. Main characteristics of respondents by vaccination status.

https://doi.org/10.1371/journal.pone.0313117.t003

Due to the observational nature of this study, we cannot directly assess the effect of vaccination status on adherence to protective behaviors because differences in covariates (e.g., perceived COVID-19 risk, conspiracy, etc…) could have influenced both the variables of interest and vaccination status. To compare the effect of vaccination on adherence to protective behaviors, we used covariate and PS weighting adjustments. Since we are interested specifically in the effect of the vaccination on those who received it, we estimated the PS weights in order to estimate the Average Treatment effect of the Treated (ATT); at this end, we set the scores for vaccinated individuals to 1, estimating the probability of refusal among those who declined the vaccine [30].

Given the PS model, we estimated the probability of inclusion using Generalized Boosted Model based on the same set of covariates used for covariate adjustment and weighted by sampling weights. We adopted the default options for GBM parameters (number of trees:10000; learning rate: 0.01 for binary) since the model complexity was high due to the presence of several covariates. The loveplot diagram shown in Fig 3 suggests an acceptable covariate balancing provided by PS procedure, with almost all SMDs below 0.2 and most of them below 0.1. After applying propensity score weighting, among who refused the vaccine the effective sample size (ESS) was reduced from the original sample size of 159.8 to 69.8. This ESS reduction is primarily due to the evident differences between the treated and control groups in terms of baseline covariates. While this reduction implies a decrease in the precision of our estimates, the weighting procedure is essential to correct for potential confounding and to ensure that the treated and untreated groups are comparable.

Download:

Fig 3. Loveplot diagram.

Estimated standardized mean difference (smd) and effective sample size (ESS) before and after PS weighting procedure.

https://doi.org/10.1371/journal.pone.0313117.g003

The resulting distribution of weights is shown in S1 Fig, indicating a wide range of weights distribution for the untreated group (vaccination refused). Model results in terms of OR and 95% CI are reported in Table 4.

Download:

Table 4. Non-adjusted (A) and adjusted ORs (B, C, and D) for social distancing, mask use, and hand disinfection by vaccination status (done vs. refused) for the four models (B = covariate adjustment, C = PS, D = PS with covariate adjustment).

https://doi.org/10.1371/journal.pone.0313117.t004

Without any adjustment (method A), adherence to physical distancing increased among the vaccinated respondents (vs. refused; OR = 1.45, 95% CI = 1.07–1.96). However, all adjusted models showed no effect of vaccination status on this protective measure (OR ≈ 1). A similar result was obtained for adherence to mask use, where the unadjusted model showed a statistically significant increase in this behavior among those who chose to be vaccinated (OR = 1.64; 95% CI = 1.20–2.23). In contrast, all adjusted models estimated an OR close to 1, but PS with covariate adjustment method suggested a decrease in adherence to mask use (OR ~ 0.75), although the result was not statistically significant. A different result was reported for the adherence to hand disinfection: the unadjusted model showed a strong positive association with vaccination status (received vs. refused; OR = 2.21, 95% CI = 1.63–3.01), while the models with covariate and PS adjustment showed a weak positive, though not statistically significant, association between vaccination and hand disinfection.

Discussion

The present study demonstrates the practical value of the adjustment techniques in assessing the influence of COVID-19 vaccine uptake on COVID-19-related behaviors. We analyzed a subset of respondents who were offered the vaccine at the beginning of the Italian vaccination campaign. The results illustrate how vaccination status and the inclusion of adjustment procedures can provide more reliable evidence on the impact of vaccination on key preventive measures against COVID-19. We also evaluated the adjustment capabilities of the two most commonly used confounder adjustment techniques, namely covariate adjustment, propensity score, as well as their combination, through a simulation study, considering a typical framework in observational studies related to social sciences and psychology, where the outcome takes an ordinal form (i.e., Likert scale). In our simulation study, the use of a propensity score procedure with inverse probability weighting resulted in a slightly higher relative bias compared to covariate adjustment, especially when the confounding variable had a strong effect on the outcome. When the model was misspecified, IPTW methods showed a reduced degree of bias compared to covariate adjustment, especially when IPTW was combined with covariate adjustment.

The PS methods have become increasingly popular for adjusting for confounding in observational studies [25, 31]. The results of our simulation study refer to two specific scenarios. In the first scenario, the estimation model was correctly specified. In this context, no significant differences were found between the three adjustment methods, although covariate adjustment showed some superiority in terms of reduction and variability of the estimate. The superiority of covariate adjustment over PS has previously been demonstrated in the context of logistic regression models [32], particularly in the presence of strong confounder-to-outcome effects [33]. In the second scenario, we introduced model misspecification for both the confounder-to-outcome and the treatment-to-outcome relationships. All methods showed some degree of relative bias, with PS methods demonstrating superior performance due to the flexibility of GBM-based techniques [34]. However, simulations considering a larger number of scenarios and adjustment techniques have shown that each adjustment method has its own advantages and disadvantages, and no single method is consistently superior [5, 35]. Furthermore, in the absence of a strong theoretical model, model misspecification cannot be excluded. Our simulation showed that using PS weighting with covariate adjustment can be an effective solution, as reported by Nguyen et al [18] and other related proposals [34]. A method that performs well under model misspecification is more desirable than one that is optimized for an ideal scenario [36]. In addition, PS expands the ability to examine outcomes in observational studies across a wide range of scenarios, partially addressing confounding effects and improving the internal validity of study results [37].

The utility and relevance of considering different confounder adjustment techniques are also evident when the literature shows mixed results. On one side, Yamamura et al. [38] using panel data collected from March 2020 to September 2021 (N = 54,007) found that vaccinated individuals were more likely to stay home, wash their hands frequently and wear masks than unvaccinated individuals, consistently from the early stages of COVID-19 until after vaccine distribution. Continuing data collection until September 2022 (N = 70,908) comparing preventive behaviors before and after vaccination, the same working group [39] showed that after vaccination people increased preventive behaviors, such as mask use and hand hygiene. These authors also showed that this behaviour was predicted by pro-social motivation. On the other side, several recent studies have observed that COVID-19 vaccination leads to a decrease in participants’ compliance with public health measures [40, 41]. Another study that surveyed participants before the first dose and after their second dose reported a decreased willingness to use a face mask and to maintain physical distancing [42]. However, these authors observed that hand hygiene was preferred over physical distancing [42]. In addition, other studies conducted in Italy reported a decline over time in behaviors or attitudes related to compliance with protective behaviors toward COVID-19 [43, 44]. A large observational study (12 countries and 80,305 respondents) reported no differences in physical distancing between those who had received one vaccine dose and those who had not, but differences were observed between the latter and those who had completed the vaccination cycle [45]. Interestingly, using PS adjustment, Wright et al. [46] reported no change in adherence to protective measures between October 2020 and March 2021 after COVID-19 vaccination, although a gradual decline in overall adherence was observed from the first year of the pandemic.

In our study, unadjusted analyses showed that those who chose to be vaccinated were more likely to engage in protective behaviors compared to those who did not. In contrast, adjusted methods showed that vaccinated individuals did not change their short-term adherence to preventive measures, with an apparent reduction in mask use in the PS models, though not statistically significant due to large confidence intervals. The results are consistent with recent evidence: COVID-19 vaccination had a limited effect on adherence to protective behaviors. The possible reduction in adherence to protective behaviors may be attributed to the perceived lower probability of infection due to vaccine protection. Most people tend to be selective in how they evaluate data in everyday life; they are more likely to accept explanatory models that reduce risk and increase perceived benefit in their own experience [47]. Another possible interpretation of the results can be related to the moral hazard theory [48], according to which people underestimate or ignore the collective consequences of their actions, assuming that the efforts of others (in this case others adopting protective behaviours) will continue to protect them, resulting, however, in public health, economic and social costs to the community [49].

It is important to emphasize that adjustment methods are particularly relevant in applied psychology research: causal inference is an essential tool for designing interventions for target populations, as it represents a unique method for identifying interventions a priori [21]. Failure to identify underlying assumptions or relying on implicit and opaque inferences exacerbates the replicability crisis in psychological science [50]. This issue can hinder the growth of cumulative research and may lead researchers or readers to mistakenly interpret ambiguous findings as causal [51].

Finally, in the specific context of the considered study, we emphasize that several assumptions must be verified in order to derive causal inference from observational studies using adjustment methods. Regarding PS methods, the plausibility of the exchangeability assumption can be strengthened by carefully selecting covariates, particularly those related to trust in health institutions, health information sources, and conspiracy beliefs, as these are key drivers of vaccine hesitancy and adherence to preventive measures [52]. Consistency can be ensured through the extensive efforts made during the COVID-19 vaccination campaign to monitor and verify vaccine administration. Finally, it is crucial to consider the potential violation of the no-interference assumption, especially in the context of vaccination and adherence to protective behaviors [53]. In particular, the vaccination of an individual may result in the phenomenon of "free riding," whereby those who have refused the vaccine may still benefit from the protection provided by vaccinated individuals due to concerns about potential herd immunity. However, the study was conducted during the early phase of the vaccination campaign and we can assume that the interference should be limited. Another crucial point concerns the COVID-19 vaccination campaign: in the initial phase, vaccination was primarily offered to vulnerable populations (e.g., elderly, immunosuppressed, with comorbidities, etc…) and to essential workers (e.g., health workers, civil servants, etc…). The specific selection of the population invited for vaccination may limit the external validity of the results and, consequently, the generalizability of the findings, despite the incorporation of sampling weights. Lastly, the WHO dataset was based on self-reported questions, which may raise concerns about item validity.

Conclusions

This study convincingly demonstrates the critical importance of adjustment techniques in observational research, particularly when evaluating the impact of COVID-19 vaccination on preventive behaviors. Analysis of data from the initial Italian vaccination campaign shows that the uptake of the vaccine uptake had only a marginal effect on adherence to protective behaviors, with a slight decrease in mask use. This is likely attributable to a perceived reduction in the risk of infection due to the vaccine’s protective properties. It is important to note that this study demonstrated the significant enhancement in results reliability that can be achieved through the use of covariate adjustment, propensity score (PS) methods and their combination. In scenarios of model misspecification, PS methods demonstrated superior performance, consistent with existing research, highlighting their flexibility in handling confounders.

Supporting information

S1 Fig. Distribution of the estimated stabilized weights by vaccination status.

https://doi.org/10.1371/journal.pone.0313117.s001

(TIF)

References

1. Cartenì A, Di Francesco L, Martino M. How mobility habits influenced the spread of the COVID-19 pandemic: Results from the Italian case study. Sci Total Environ. 2020;741:140489. pmid:32599395
- View Article
- PubMed/NCBI
- Google Scholar
2. Parohan M, Yaghoubi S, Seraji A, Javanbakht MH, Sarraf P, Djalali M. Risk factors for mortality in patients with Coronavirus disease 2019 (COVID-19) infection: a systematic review and meta-analysis of observational studies. Aging Male. 2020;23:1416–1424. pmid:32508193
- View Article
- PubMed/NCBI
- Google Scholar
3. Dergiades T, Milas C, Mossialos E, Panagiotidis T. Effectiveness of government policies in response to the first COVID-19 outbreak. PLOS Glob Public Health, 2022;2:e0000242. pmid:36962226
- View Article
- PubMed/NCBI
- Google Scholar
4. Greenland S, Mansournia MA. Limitations of individual causal models, causal graphs, and ignorability assumptions, as illustrated by random confounding and design unfaithfulness. Eur J Epidemiol. 2011;30:1101–1110. pmid:25687168
- View Article
- PubMed/NCBI
- Google Scholar
5. Rubin DB. Causal inference using potential outcomes: Design, modeling, decisions. J Am Stat Assoc. 2005;100:322–331.
- View Article
- Google Scholar
6. Colnet B, Mayer I, Chen G, Dieng A, Li R, Varoquaux G, et al. Causal inference methods for combining randomized trials and observational studies: a review. Stat Sci. 2024; 39:165–191.
- View Article
- Google Scholar
7. Mattei A, De Stavola BL, Mealli F. Preface to the papers on “causal inference from non-experimental studies: Challenges, developments and applications.”. J R Stat Soc Ser A. 2020;183:1329–1332.
- View Article
- Google Scholar
8. Williamson EJ, Aitken Z, Lawrie J, Dharmage SC, Burgess JA, Forbes AB. Introduction to causal diagrams for confounder selection. Respirology. 2014;19:303–311. pmid:24447391
- View Article
- PubMed/NCBI
- Google Scholar
9. Li L, Kleinman K, Gillman MW. A comparison of confounding adjustment methods with an application to early life determinants of childhood obesity. J Dev Orig Health Dis. 2014;5:435–447. pmid:25171142
- View Article
- PubMed/NCBI
- Google Scholar
10. D’Onofrio BM, Sjölander A, Lahey BB, Lichtenstein P, Öberg AS: Accounting for Confounding in Observational Studies. Annu Rev Clin Psychol. 2020;16:25–48. pmid:32384000
- View Article
- PubMed/NCBI
- Google Scholar
11. Rosenbaum PR and Rubin D. The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika. 1983;70:41–55.
- View Article
- Google Scholar
12. Guo S, Fraser MW. Propensity Score Analysis: Statistical Methods and Applications. Sage Publications. 2015.
13. Rubin DB. Estimating causal effects from large data sets using propensity scores. Ann Intern Med, 1997;127:757–763. pmid:9382394
- View Article
- PubMed/NCBI
- Google Scholar
14. Hade EM, Lu B. Bias associated with using the estimated propensity score as a regression covariate. Stat Med, 2014;33: 74–87. pmid:23787715
- View Article
- PubMed/NCBI
- Google Scholar
15. Rubaltelli E, Tedaldi E, Orabona N, Scrimin S. Environmental and psychological variables influencing reactions to the COVID-19 outbreak. Br J Health Psychol. 2020;25:1020–1038. pmid:32951244
- View Article
- PubMed/NCBI
- Google Scholar
16. Caserotti M, Girardi P, Rubaltelli E, Tasso A, Lotto L, Garavuzzi T. Associations of COVID-19 risk perception with vaccine hesitancy over time for Italian residents. Soc Sci Med. 2021;272:113688. pmid:33485215
- View Article
- PubMed/NCBI
- Google Scholar
17. de Girolamo G, Ferrari C, Candini V, Buizza C, Calamandrei G, Caserotti M, et al. Psychological well-being during the COVID-19 pandemic in Italy assessed in a four-waves survey. Sci Rep. 2022;12:17945. pmid:36289273
- View Article
- PubMed/NCBI
- Google Scholar
18. Nguyen TL, Collins GS, Spence J, Daurès JP, Devereaux PJ, Landais P et al. Double-adjustment in propensity score matching analysis: choosing a threshold for considering residual imbalance. BMC Med Res Methodol. 2017;17:1–8. pmid:28454568
- View Article
- PubMed/NCBI
- Google Scholar
19. Candini V, Brescianini S, Chiarotti F, Zarbo C, Zamparini M, Caserotti M, et al. Conspiracy mentality and health-related behavior during the COVID-19 pandemic: a multiwave survey in Italy. Public Health. 2023;214:124–132. pmid:36549021
- View Article
- PubMed/NCBI
- Google Scholar
20. Matthay EC, Hagan E, Gottlieb LM, Tan ML, Vlahov D, Adlet NE, et al. Alternative causal inference methods in population health research: Evaluating tradeoffs and triangulating evidence. SSM Popul Health. 2020;10:100526. pmid:31890846
- View Article
- PubMed/NCBI
- Google Scholar
21. Foster EM. Causal inference and developmental psychology. Dev Psychol. 2020;46:1454–1480. pmid:20677855
- View Article
- PubMed/NCBI
- Google Scholar
22. Shiba K, Kawahara T. Using propensity scores for causal inference: Pitfalls and tips. J Epidemiol. 2021;31:457–463. pmid:34121051
- View Article
- PubMed/NCBI
- Google Scholar
23. Pan W, Bai H. Propensity score methods for causal inference: an overview. Behaviormetrika, 2018;45:317–334.
- View Article
- Google Scholar
24. Harder VS, Stuart EA, Anthony JC. Propensity score techniques and the assessment of measured covariate balance to test causal associations in psychological research. Psychol Methods. 2010;15:234–249. pmid:20822250
- View Article
- PubMed/NCBI
- Google Scholar
25. Austin PC, Yu AYX, Vyas MV, Kapral MK. Applying propensity score methods in clinical research in neurology. Neurology. 2021;97:856–863. pmid:34504033
- View Article
- PubMed/NCBI
- Google Scholar
26. R Core Team. (2021). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.
27. Walcherberger C, Eberl JM, Partheymüller J, Paul KT, Stamm TA. COVID-19 vaccine hesitancy and gender. Eur J Pol Gender. 2022;5:270–274.
- View Article
- Google Scholar
28. Morales DX, Beltran TF, Morales SA. Gender, socioeconomic status, and COVID-19 vaccine hesitancy in the US: an intersectionality approach. Sociol Health Illn. 2022;44:953–971. pmid:35500003
- View Article
- PubMed/NCBI
- Google Scholar
29. Elith J, Leathwick JR, Hastie T. A working guide to boosted regression trees. J Anim Ecol. 2008;77:802–813. pmid:18397250
- View Article
- PubMed/NCBI
- Google Scholar
30. Wang A, Nianogo RA, Arah OA. G-computation of average treatment effects on the treated and the untreated. BMC Med Res Methodol. 2017;17:1–5. pmid:28068905
- View Article
- PubMed/NCBI
- Google Scholar
31. Leisman DE. Ten Pearls and Pitfalls of Propensity Scores in Critical Care Research: A Guide for Clinicians and Researchers. Crit Care Med. 2019;47:176–185. pmid:30543566
- View Article
- PubMed/NCBI
- Google Scholar
32. Wilkinson JD, Mamas MA, Kontopantelis E. Logistic regression frequently outperformed propensity score methods, especially for large datasets: a simulation study. J Clin Epidemiol, 2022;152:176–184. pmid:36126791
- View Article
- PubMed/NCBI
- Google Scholar
33. Ukoumunne OC, Williamson E, Forbes AB, Gulliford MC, Carlin JB. Confounder-adjusted estimates of the risk difference using propensity score-based weighting. Stat Med, 2010;29: 3126–3136. pmid:21170907
- View Article
- PubMed/NCBI
- Google Scholar
34. Vansteelandt S, Bekaert M, Claeskens G. On model selection and model misspecification in causal inference. Stat Methods Med Res, 2012;21:7–30. pmid:21075803
- View Article
- PubMed/NCBI
- Google Scholar
35. Zanutto EL. A comparison of propensity score and linear regression analysis of complex survey data. J Data Sci. 2006;4:67–91.
- View Article
- Google Scholar
36. Schafer JL, Kang J. Average causal effects from nonrandomized studies: a practical guide and simulated example. Psychol Methods. 2008;13:279–313. pmid:19071996
- View Article
- PubMed/NCBI
- Google Scholar
37. Lalani N, Jimenez RB, Yeap B. Understanding propensity score analyses. Int J Radiat Oncol Biol Phys. 2020;107:404–407. pmid:32531385
- View Article
- PubMed/NCBI
- Google Scholar
38. Yamamura E, Kohsaka Y, Tsutsui Y, Ohtake F. Association between the COVID-19 Vaccine and preventive behaviors: panel data analysis from Japan. Vaccines. 2023;11:810. pmid:37112722
- View Article
- PubMed/NCBI
- Google Scholar
39. Yamamura E, Tsutsui Y,Ohtake F. COVID-19 vaccination, preventive behaviours and pro-social motivation: panel data analysis from Japan. Humanit Soc Sci Commun. 2024;311, 476 (2024).
- View Article
- Google Scholar
40. Si R, Yao Y, Zhang X, Lu Q, Aziz N. Investigating the links between vaccination against COVID-19 and public attitudes toward protective countermeasures: Implications for public health. Front Public Health. 2021;9:702699. pmid:34368065
- View Article
- PubMed/NCBI
- Google Scholar
41. Zhang N, Liu X, Jin T, Zhao P, Miao D, Lei H, et al. Weakening personal protective behavior by Chinese university students after COVID-19 vaccination. Build Environ. 2021;206:108367. pmid:34566244
- View Article
- PubMed/NCBI
- Google Scholar
42. Corea F, Folcarelli L, Napoli A, Del Giudice GM, Angelillo IF. The impact of COVID-19 vaccination in changing the adherence to preventive measures: Evidence from Italy. Vaccines. 2022;10:777. pmid:35632533
- View Article
- PubMed/NCBI
- Google Scholar
43. Della Polla G, Pelullo CP, Di Giuseppe G, Angelillo IF. Changes in behaviors and attitudes in response to COVID-19 pandemic and vaccination in healthcare workers and university students in Italy. Vaccines. 2021;9:1276. pmid:34835207
- View Article
- PubMed/NCBI
- Google Scholar
44. Grano C, Singh Solorzano C, Di Pucchio A. Predictors of protective behaviors during the Italian COVID-19 pandemic: An application of protection motivation theory. Psychol Health. 2022;37:1584–1604. pmid:35459428
- View Article
- PubMed/NCBI
- Google Scholar
45. Goldszmidt R, Petherick A, Andrade EB, Hale T, Furst R, Phillips T, et al. Protective behaviors against COVID-19 by individual vaccination status in 12 countries during the pandemic. JAMA Netw Open. 2021;4:e2131137–e2131137. pmid:34698850
- View Article
- PubMed/NCBI
- Google Scholar
46. Wright L, Steptoe A, Mak HW, Fancourt D. Do people reduce compliance with COVID-19 guidelines following vaccination? A longitudinal analysis of matched UK adults. J Epidemiol Community Health. 2022;76:109–115. pmid:34244309
- View Article
- PubMed/NCBI
- Google Scholar
47. Botterill L, Mazur N. Risk and risk perception: A literature review. Kingstrom, ACT: Aust Govt Rural Ind Res & Dev Corp. 2004.
48. Peltzman S. The effects of automobile safety regulation. J Political Econ. 1975;83:677–725.
- View Article
- Google Scholar
49. Lee K. Moral hazard in public health policies. Appl Econ. 2023;1–19.
- View Article
- Google Scholar
50. Collaboration OS. Estimating the reproducibility of psychological science. Science. 2015;349:aac4716. pmid:26315443
- View Article
- PubMed/NCBI
- Google Scholar
51. Grosz MP, Rohrer JM, Thoemmes F. The taboo against explicit causal inference in nonexperimental psychology. Perspect Psychol Sci. 2020;15:1243–1255. pmid:32727292
- View Article
- PubMed/NCBI
- Google Scholar
52. Jennings W, Stoker G, Bunting H, Valgarðsson VO, Gaskell J, Devine D, et al. Lack of trust, conspiracy beliefs, and social media use predict COVID-19 vaccine hesitancy. Vaccines. 2021;9:593. pmid:34204971
- View Article
- PubMed/NCBI
- Google Scholar
53. Tchetgen EJT, VanderWeele TJ. On causal inference in the presence of interference. Stat Methods Med Res. 2012;2:55–75. pmid:21068053
- View Article
- PubMed/NCBI
- Google Scholar

[ref1] 1. Cartenì A, Di Francesco L, Martino M. How mobility habits influenced the spread of the COVID-19 pandemic: Results from the Italian case study. Sci Total Environ. 2020;741:140489. pmid:32599395
View Article
PubMed/NCBI
Google Scholar

[2] View Article

[3] PubMed/NCBI

[4] Google Scholar

[ref2] 2. Parohan M, Yaghoubi S, Seraji A, Javanbakht MH, Sarraf P, Djalali M. Risk factors for mortality in patients with Coronavirus disease 2019 (COVID-19) infection: a systematic review and meta-analysis of observational studies. Aging Male. 2020;23:1416–1424. pmid:32508193
View Article
PubMed/NCBI
Google Scholar

[6] View Article

[7] PubMed/NCBI

[8] Google Scholar

[ref3] 3. Dergiades T, Milas C, Mossialos E, Panagiotidis T. Effectiveness of government policies in response to the first COVID-19 outbreak. PLOS Glob Public Health, 2022;2:e0000242. pmid:36962226
View Article
PubMed/NCBI
Google Scholar

[10] View Article

[11] PubMed/NCBI

[12] Google Scholar

[ref4] 4. Greenland S, Mansournia MA. Limitations of individual causal models, causal graphs, and ignorability assumptions, as illustrated by random confounding and design unfaithfulness. Eur J Epidemiol. 2011;30:1101–1110. pmid:25687168
View Article
PubMed/NCBI
Google Scholar

[14] View Article

[15] PubMed/NCBI

[16] Google Scholar

[ref5] 5. Rubin DB. Causal inference using potential outcomes: Design, modeling, decisions. J Am Stat Assoc. 2005;100:322–331.
View Article
Google Scholar

[18] View Article

[19] Google Scholar

[ref6] 6. Colnet B, Mayer I, Chen G, Dieng A, Li R, Varoquaux G, et al. Causal inference methods for combining randomized trials and observational studies: a review. Stat Sci. 2024; 39:165–191.
View Article
Google Scholar

[21] View Article

[22] Google Scholar

[ref7] 7. Mattei A, De Stavola BL, Mealli F. Preface to the papers on “causal inference from non-experimental studies: Challenges, developments and applications.”. J R Stat Soc Ser A. 2020;183:1329–1332.
View Article
Google Scholar

[24] View Article

[25] Google Scholar

[ref8] 8. Williamson EJ, Aitken Z, Lawrie J, Dharmage SC, Burgess JA, Forbes AB. Introduction to causal diagrams for confounder selection. Respirology. 2014;19:303–311. pmid:24447391
View Article
PubMed/NCBI
Google Scholar

[27] View Article

[28] PubMed/NCBI

[29] Google Scholar

[ref9] 9. Li L, Kleinman K, Gillman MW. A comparison of confounding adjustment methods with an application to early life determinants of childhood obesity. J Dev Orig Health Dis. 2014;5:435–447. pmid:25171142
View Article
PubMed/NCBI
Google Scholar

[31] View Article

[32] PubMed/NCBI

[33] Google Scholar

[ref10] 10. D’Onofrio BM, Sjölander A, Lahey BB, Lichtenstein P, Öberg AS: Accounting for Confounding in Observational Studies. Annu Rev Clin Psychol. 2020;16:25–48. pmid:32384000
View Article
PubMed/NCBI
Google Scholar

[35] View Article

[36] PubMed/NCBI

[37] Google Scholar

[ref11] 11. Rosenbaum PR and Rubin D. The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika. 1983;70:41–55.
View Article
Google Scholar

[39] View Article

[40] Google Scholar

[ref12] 12. Guo S, Fraser MW. Propensity Score Analysis: Statistical Methods and Applications. Sage Publications. 2015.

[ref13] 13. Rubin DB. Estimating causal effects from large data sets using propensity scores. Ann Intern Med, 1997;127:757–763. pmid:9382394
View Article
PubMed/NCBI
Google Scholar

[43] View Article

[44] PubMed/NCBI

[45] Google Scholar

[ref14] 14. Hade EM, Lu B. Bias associated with using the estimated propensity score as a regression covariate. Stat Med, 2014;33: 74–87. pmid:23787715
View Article
PubMed/NCBI
Google Scholar

[47] View Article

[48] PubMed/NCBI

[49] Google Scholar

[ref15] 15. Rubaltelli E, Tedaldi E, Orabona N, Scrimin S. Environmental and psychological variables influencing reactions to the COVID-19 outbreak. Br J Health Psychol. 2020;25:1020–1038. pmid:32951244
View Article
PubMed/NCBI
Google Scholar

[51] View Article

[52] PubMed/NCBI

[53] Google Scholar

[ref16] 16. Caserotti M, Girardi P, Rubaltelli E, Tasso A, Lotto L, Garavuzzi T. Associations of COVID-19 risk perception with vaccine hesitancy over time for Italian residents. Soc Sci Med. 2021;272:113688. pmid:33485215
View Article
PubMed/NCBI
Google Scholar

[55] View Article

[56] PubMed/NCBI

[57] Google Scholar

[ref17] 17. de Girolamo G, Ferrari C, Candini V, Buizza C, Calamandrei G, Caserotti M, et al. Psychological well-being during the COVID-19 pandemic in Italy assessed in a four-waves survey. Sci Rep. 2022;12:17945. pmid:36289273
View Article
PubMed/NCBI
Google Scholar

[59] View Article

[60] PubMed/NCBI

[61] Google Scholar

[ref18] 18. Nguyen TL, Collins GS, Spence J, Daurès JP, Devereaux PJ, Landais P et al. Double-adjustment in propensity score matching analysis: choosing a threshold for considering residual imbalance. BMC Med Res Methodol. 2017;17:1–8. pmid:28454568
View Article
PubMed/NCBI
Google Scholar

[63] View Article

[64] PubMed/NCBI

[65] Google Scholar

[ref19] 19. Candini V, Brescianini S, Chiarotti F, Zarbo C, Zamparini M, Caserotti M, et al. Conspiracy mentality and health-related behavior during the COVID-19 pandemic: a multiwave survey in Italy. Public Health. 2023;214:124–132. pmid:36549021
View Article
PubMed/NCBI
Google Scholar

[67] View Article

[68] PubMed/NCBI

[69] Google Scholar

[ref20] 20. Matthay EC, Hagan E, Gottlieb LM, Tan ML, Vlahov D, Adlet NE, et al. Alternative causal inference methods in population health research: Evaluating tradeoffs and triangulating evidence. SSM Popul Health. 2020;10:100526. pmid:31890846
View Article
PubMed/NCBI
Google Scholar

[71] View Article

[72] PubMed/NCBI

[73] Google Scholar

[ref21] 21. Foster EM. Causal inference and developmental psychology. Dev Psychol. 2020;46:1454–1480. pmid:20677855
View Article
PubMed/NCBI
Google Scholar

[75] View Article

[76] PubMed/NCBI

[77] Google Scholar

[ref22] 22. Shiba K, Kawahara T. Using propensity scores for causal inference: Pitfalls and tips. J Epidemiol. 2021;31:457–463. pmid:34121051
View Article
PubMed/NCBI
Google Scholar

[79] View Article

[80] PubMed/NCBI

[81] Google Scholar

[ref23] 23. Pan W, Bai H. Propensity score methods for causal inference: an overview. Behaviormetrika, 2018;45:317–334.
View Article
Google Scholar

[83] View Article

[84] Google Scholar

[ref24] 24. Harder VS, Stuart EA, Anthony JC. Propensity score techniques and the assessment of measured covariate balance to test causal associations in psychological research. Psychol Methods. 2010;15:234–249. pmid:20822250
View Article
PubMed/NCBI
Google Scholar

[86] View Article

[87] PubMed/NCBI

[88] Google Scholar

[ref25] 25. Austin PC, Yu AYX, Vyas MV, Kapral MK. Applying propensity score methods in clinical research in neurology. Neurology. 2021;97:856–863. pmid:34504033
View Article
PubMed/NCBI
Google Scholar

[90] View Article

[91] PubMed/NCBI

[92] Google Scholar

[ref26] 26. R Core Team. (2021). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.

[ref27] 27. Walcherberger C, Eberl JM, Partheymüller J, Paul KT, Stamm TA. COVID-19 vaccine hesitancy and gender. Eur J Pol Gender. 2022;5:270–274.
View Article
Google Scholar

[95] View Article

[96] Google Scholar

[ref28] 28. Morales DX, Beltran TF, Morales SA. Gender, socioeconomic status, and COVID-19 vaccine hesitancy in the US: an intersectionality approach. Sociol Health Illn. 2022;44:953–971. pmid:35500003
View Article
PubMed/NCBI
Google Scholar

[98] View Article

[99] PubMed/NCBI

[100] Google Scholar

[ref29] 29. Elith J, Leathwick JR, Hastie T. A working guide to boosted regression trees. J Anim Ecol. 2008;77:802–813. pmid:18397250
View Article
PubMed/NCBI
Google Scholar

[102] View Article

[103] PubMed/NCBI

[104] Google Scholar

[ref30] 30. Wang A, Nianogo RA, Arah OA. G-computation of average treatment effects on the treated and the untreated. BMC Med Res Methodol. 2017;17:1–5. pmid:28068905
View Article
PubMed/NCBI
Google Scholar

[106] View Article

[107] PubMed/NCBI

[108] Google Scholar

[ref31] 31. Leisman DE. Ten Pearls and Pitfalls of Propensity Scores in Critical Care Research: A Guide for Clinicians and Researchers. Crit Care Med. 2019;47:176–185. pmid:30543566
View Article
PubMed/NCBI
Google Scholar

[110] View Article

[111] PubMed/NCBI

[112] Google Scholar

[ref32] 32. Wilkinson JD, Mamas MA, Kontopantelis E. Logistic regression frequently outperformed propensity score methods, especially for large datasets: a simulation study. J Clin Epidemiol, 2022;152:176–184. pmid:36126791
View Article
PubMed/NCBI
Google Scholar

[114] View Article

[115] PubMed/NCBI

[116] Google Scholar

[ref33] 33. Ukoumunne OC, Williamson E, Forbes AB, Gulliford MC, Carlin JB. Confounder-adjusted estimates of the risk difference using propensity score-based weighting. Stat Med, 2010;29: 3126–3136. pmid:21170907
View Article
PubMed/NCBI
Google Scholar

[118] View Article

[119] PubMed/NCBI

[120] Google Scholar

[ref34] 34. Vansteelandt S, Bekaert M, Claeskens G. On model selection and model misspecification in causal inference. Stat Methods Med Res, 2012;21:7–30. pmid:21075803
View Article
PubMed/NCBI
Google Scholar

[122] View Article

[123] PubMed/NCBI

[124] Google Scholar

[ref35] 35. Zanutto EL. A comparison of propensity score and linear regression analysis of complex survey data. J Data Sci. 2006;4:67–91.
View Article
Google Scholar

[126] View Article

[127] Google Scholar

[ref36] 36. Schafer JL, Kang J. Average causal effects from nonrandomized studies: a practical guide and simulated example. Psychol Methods. 2008;13:279–313. pmid:19071996
View Article
PubMed/NCBI
Google Scholar

[129] View Article

[130] PubMed/NCBI

[131] Google Scholar

[ref37] 37. Lalani N, Jimenez RB, Yeap B. Understanding propensity score analyses. Int J Radiat Oncol Biol Phys. 2020;107:404–407. pmid:32531385
View Article
PubMed/NCBI
Google Scholar

[133] View Article

[134] PubMed/NCBI

[135] Google Scholar

[ref38] 38. Yamamura E, Kohsaka Y, Tsutsui Y, Ohtake F. Association between the COVID-19 Vaccine and preventive behaviors: panel data analysis from Japan. Vaccines. 2023;11:810. pmid:37112722
View Article
PubMed/NCBI
Google Scholar

[137] View Article

[138] PubMed/NCBI

[139] Google Scholar

[ref39] 39. Yamamura E, Tsutsui Y,Ohtake F. COVID-19 vaccination, preventive behaviours and pro-social motivation: panel data analysis from Japan. Humanit Soc Sci Commun. 2024;311, 476 (2024).
View Article
Google Scholar

[141] View Article

[142] Google Scholar

[ref40] 40. Si R, Yao Y, Zhang X, Lu Q, Aziz N. Investigating the links between vaccination against COVID-19 and public attitudes toward protective countermeasures: Implications for public health. Front Public Health. 2021;9:702699. pmid:34368065
View Article
PubMed/NCBI
Google Scholar

[144] View Article

[145] PubMed/NCBI

[146] Google Scholar

[ref41] 41. Zhang N, Liu X, Jin T, Zhao P, Miao D, Lei H, et al. Weakening personal protective behavior by Chinese university students after COVID-19 vaccination. Build Environ. 2021;206:108367. pmid:34566244
View Article
PubMed/NCBI
Google Scholar

[148] View Article

[149] PubMed/NCBI

[150] Google Scholar

[ref42] 42. Corea F, Folcarelli L, Napoli A, Del Giudice GM, Angelillo IF. The impact of COVID-19 vaccination in changing the adherence to preventive measures: Evidence from Italy. Vaccines. 2022;10:777. pmid:35632533
View Article
PubMed/NCBI
Google Scholar

[152] View Article

[153] PubMed/NCBI

[154] Google Scholar

[ref43] 43. Della Polla G, Pelullo CP, Di Giuseppe G, Angelillo IF. Changes in behaviors and attitudes in response to COVID-19 pandemic and vaccination in healthcare workers and university students in Italy. Vaccines. 2021;9:1276. pmid:34835207
View Article
PubMed/NCBI
Google Scholar

[156] View Article

[157] PubMed/NCBI

[158] Google Scholar

[ref44] 44. Grano C, Singh Solorzano C, Di Pucchio A. Predictors of protective behaviors during the Italian COVID-19 pandemic: An application of protection motivation theory. Psychol Health. 2022;37:1584–1604. pmid:35459428
View Article
PubMed/NCBI
Google Scholar

[160] View Article

[161] PubMed/NCBI

[162] Google Scholar

[ref45] 45. Goldszmidt R, Petherick A, Andrade EB, Hale T, Furst R, Phillips T, et al. Protective behaviors against COVID-19 by individual vaccination status in 12 countries during the pandemic. JAMA Netw Open. 2021;4:e2131137–e2131137. pmid:34698850
View Article
PubMed/NCBI
Google Scholar

[164] View Article

[165] PubMed/NCBI

[166] Google Scholar

[ref46] 46. Wright L, Steptoe A, Mak HW, Fancourt D. Do people reduce compliance with COVID-19 guidelines following vaccination? A longitudinal analysis of matched UK adults. J Epidemiol Community Health. 2022;76:109–115. pmid:34244309
View Article
PubMed/NCBI
Google Scholar

[168] View Article

[169] PubMed/NCBI

[170] Google Scholar

[ref47] 47. Botterill L, Mazur N. Risk and risk perception: A literature review. Kingstrom, ACT: Aust Govt Rural Ind Res & Dev Corp. 2004.

[ref48] 48. Peltzman S. The effects of automobile safety regulation. J Political Econ. 1975;83:677–725.
View Article
Google Scholar

[173] View Article

[174] Google Scholar

[ref49] 49. Lee K. Moral hazard in public health policies. Appl Econ. 2023;1–19.
View Article
Google Scholar

[176] View Article

[177] Google Scholar

[ref50] 50. Collaboration OS. Estimating the reproducibility of psychological science. Science. 2015;349:aac4716. pmid:26315443
View Article
PubMed/NCBI
Google Scholar

[179] View Article

[180] PubMed/NCBI

[181] Google Scholar

[ref51] 51. Grosz MP, Rohrer JM, Thoemmes F. The taboo against explicit causal inference in nonexperimental psychology. Perspect Psychol Sci. 2020;15:1243–1255. pmid:32727292
View Article
PubMed/NCBI
Google Scholar

[183] View Article

[184] PubMed/NCBI

[185] Google Scholar

[ref52] 52. Jennings W, Stoker G, Bunting H, Valgarðsson VO, Gaskell J, Devine D, et al. Lack of trust, conspiracy beliefs, and social media use predict COVID-19 vaccine hesitancy. Vaccines. 2021;9:593. pmid:34204971
View Article
PubMed/NCBI
Google Scholar

[187] View Article

[188] PubMed/NCBI

[189] Google Scholar

[ref53] 53. Tchetgen EJT, VanderWeele TJ. On causal inference in the presence of interference. Stat Methods Med Res. 2012;2:55–75. pmid:21068053
View Article
PubMed/NCBI
Google Scholar

[191] View Article

[192] PubMed/NCBI

[193] Google Scholar

Figures

Abstract

Introduction

Materials and methods

Study setting and participants

Adjustment procedures

Covariate adjustment in regression analysis.

Propensity score methods.

Regression models

Simulations

The theoretical model

Simulation settings and bias estimation

Simulation results

Results

Discussion

Conclusions

Supporting information

S1 Fig. Distribution of the estimated stabilized weights by vaccination status.

References