A randomized controlled trial of a video intervention shows evidence of increasing COVID-19 vaccination intention

Increasing acceptance of COVID-19 vaccines is imperative for public health. Previous research on educational interventions to overcome vaccine hesitancy have shown mixed effects in increasing vaccination intention, although much of this work has focused on parental attitudes toward childhood vaccination. In this study, we conducted a randomized controlled trial to investigate whether vaccination intention changes after viewing an animated YouTube video explaining how COVID-19 mRNA vaccines work. We exposed participants to one of four interventions–watching the video with a male narrator, watching the same video with a female narrator, reading the text of the transcript of the video, or receiving no information (control group). We found that participants who watched the version of the video with a male narrator expressed statistically significant increased vaccination intention compared to the control group. The video with a female narrator had more variation in results. As a whole, there was a non-significant increased vaccination intention when analyzing all participants who saw the video with a female narrator; however, for politically conservative participants there was decreased vaccination intention for this intervention compared to the control group at a threshold between being currently undecided and expressing probable interest. These results are encouraging for the ability of interventions as simple as YouTube videos to increase vaccination propensity, although the inconsistent response to the video with a female narrator demonstrates the potential for bias to affect how certain groups respond to different messengers.

combination with any other race or ethnicity) reduces the sample to 1,013. Doing so did not result in any coefficients from the models in the main text becoming statistically insignificant. There was some attenuation in the statistical significance for the Male-Narr-Video coefficient (e.g., in Model 2 the coefficient was 0.379, p = 0.026). Notably, the main effect term for Female-Narr-Video had a larger difference from the Male-Narr-Video in Model 2 and was further from statistical significance (coefficient = 0.272, p = 0.192).
In two alternate specifications, the autofit option results in a single coefficient for the interaction term between Female-Narr-Video and conservative respondents (negative, but not statistically significant), while producing different coefficients for each level of the outcome for the main effect of Female-Narr-Video. In these cases, the coefficient for the main effect Female-Narr-Video is statistically significant for responses greater than "probably yes" (the fourth and final level reported in the model). For both specifications, we also estimated a model imposing parallel lines (a single coefficient across all levels of vaccine intent) on the main effect variable. Since we are testing whether the earlier finding that some of the difference in response to the Female-Narr-Video associates with how conservative respondents may react with increased uncertainty, theory guides the imposition of the parallel line on the main effect variable (1). In both instances, when the parallel line is imposed on the main effect variable, the interaction term follows the pattern reported in Model 2 in the main text and AIC/BIC tests show support for the non-autofit model in which the main effect is subject to the parallel line imposition and the interaction term allowed to vary across levels of vaccine intent. Finally, predicted probability calculations show a similar pattern for both autofit and non-autofit models, particularly when comparing conservatives exposed to the two different video treatments.
Restricting the reference group for political ideology to only those who identified as moderate by dropping those who identified as something else results in the replicating the finding of a significant association for the Male-Narr-Video treatment variable and vaccine intent for both the autofit (coefficient = 0.472, p = .003, all variables reported for Model 2) and parallel lines imposed on the Female-Narr-Video models (coefficient = 0.471, p = 0.003). Under an autofit model, the Female-Narr-Video x conservative interaction has a single coefficient for all levels of the outcome (coefficient = -0.430, p = 0.150), while the main effect coefficient for Female-Narr-Video has a statistically significant coefficient for a response above "probably yes" (coefficient = 0.472, p = 0.003). Maintaining the parallel line specification in Model 2 for the main effect for Female-Narr-Video (coefficient = 0.352, p = 0.067) replicates the pattern in the model in the main text for the interaction term having a statistically significant association for the third level of the dependent variable (coefficient = -0.935, p = .003). Comparing predicted probabilities for either a "probably yes" or "definitely yes" on vaccine intent shows that both these models reproduce the pattern reported for predicted probabilities for Model 2 in the main text. Using the same parameters as in the main text (a respondent under 55, white, male, B.A. or higher), we compare the predicted probabilities for conservatives exposed to both videos. Under the autofit model, a conservative exposed to the Male-Narr-Video has an 82.0% predicted probability of having a definitely or probably yes response, while one exposed to the Female-Narr-Video has a predicted probability of 64.6%. For the model with the parallel lines imposed, the comparison is 81.6% to 60.7%. Despite some difference in the magnitude of the difference, the substantive pattern and direction is similar. Finally, Akaike and Bayesian Information Criteria show a slight preference for the model for the imposition of parallel lines on the main effect (2955.824 for the non-autofit model compared to 2947.535 for the autofit model).
We find a similar pattern in broadening the sample to exclude only those who did not pass the attention check item, regardless of time spent with the video or text. Doing so increases the analytic sample to 1256 cases. Other than the pattern with the Female-Narr-Video interaction and main term in autofit models, this specification does not change the significance or direction of any coefficients, although the text intervention coefficient increases (coefficient = 0.219, p = 0.147) relative to the Male-Narr-Video coefficient (coefficient = 0.440, p = 0.004). The pattern in the Female-Narr-Video and interaction term follows the pattern discussed in the previous paragraph: in the autofit model, the main effect variable has a significant positive association (coefficient = 0.399, p = 0.036) for the highest level outcome of the dependent variable. Following the pattern of Model 2 in the main text, in the model with parallel lines imposed on Female-Narr-Video, the interaction term has a significant negative association at the third level of the outcome (coefficient = -0.754, p = 0.011). Using the same method of comparing predicted probabilities as the previous paragraph for an under 55, white, male conservative with a B.A. or higher exposed to the Male-Narr-Video has an 81.2% predicted probability of having a definitely or probably yes response under the autofit model, compared to a predicted probability for exposure to the Female-Narr-Video has of 67.1%. The estimates from the model with parallel lines imposed are 80.9% and 64.0%, respectively. And, as above, the information criteria suggest the non-autofit model with parallel lines imposed is slightly better (3171.610 for the parallel lines model, compared to 3171.909 for the autofit model).
Analyses that restricted the sample based on level of education suggested that the associations reported in Models 1 and 2 in the main text may reflect somewhat distinct processes based on respondents' level of education. When Model 1 was restricted only to respondents with an education level lower than a B.A., the Male-Narr-Video group (coefficient = 0.744, p = 0.004) and Female-Narr-Video group (coefficient = 0.078, p = 0.770) both had coefficients with symmetrical associations across the different levels of vaccine intent. More notably, the magnitude of difference between the two coefficients is substantially increased. In Model 2, the coefficients for both video treatments, political conservatives, and the interaction between conservative and Female-Narr-Video all had symmetrical associations with vaccine intent. While the coefficients for both political conservatives (coefficient = -0.382 p = 0.153) and the interaction term (coefficient = -0.356, p = 0.491) were negative, neither is statistically significant, nor is the coefficient for Female-Narr-Video (coefficient = 0.180, p = 0.556). It should be noted, however, that the statistical power for this model is substantially attenuated, since it includes only 395 cases.
When restricted only to respondents with an education level of a B.A. or higher, neither video treatment coefficient is statistically significant in Model 1 and the two coefficients are of a similar magnitude (Male-Narr-Video coefficient = 0.244, p = 0.216; Female-Narr-Video coefficient = 0.249, p = 0.207). In Model 2, however, the coefficient for interaction term for Female-Narr-Video and conservative is of a notable magnitude for a response higher than "undecided as of now" (coefficient = -1.150, p = 0.003), reflecting the pattern in the main text model. Additionally, the interaction term's coefficient for at the second level (higher than "probably not") is negative and modestly above the cut off for statistical significance (coefficient = -0.788, p = 0.064). Including these interaction terms in the model affects the coefficient for the main term for Female-Narr-Video (for respondents other than conservatives). It is increased in magnitude and statistically significant (coefficient = 0.517, p = 0.038). The coefficient for conservative is essentially zero (coefficient = 0.018, p = 0.930). As above, this model has reduced statistical power due in part to the reduced sample size and in part due to the lower level of variation in vaccine intent among those with a B.A. or higher in the sample.
The findings from these sample restrictions on the basis of education may, therefore, reflect a specification pattern, finding a greater difference between the two video treatments across respondents with lower levels of education and, at higher levels of education, particularly strong evidence of the conditioning of the effect of the gender of the narrator based on political ideology. In other words, the impact of the gender of the video narrator on vaccination intention might reflect a pattern of the Female-Narr-Video being less persuasive to those with a lower level of education and a pattern of more educated conservatives reacting negatively to the Female-Narr-Video.
Post-treatment bias analysis. As discussed in the text, we excluded respondents from our main analysis on the basis of time devoted to treatment and an attention check. Since both of these bases for exclusion were after exposure to the treatment, it is possible that this exclusion could create biased estimates of the treatment conditions. To check for evidence and possible extent of such bias, we undertook two additional analyses. First, we compared models for which no cases are excluded (full sample) with models based on the exclusion criteria (analytic sample). To facilitate easier comparison, we use OLS models for these comparisons, since coefficients can be compared across models without additional transformations. (We also use the OLS model as a basis of comparison to the reported partial proportional odds model in a brief note below.) Second, we used a logistic regression analysis to examine which measured characteristics, if any, associate with increased or decreased likelihood of exclusion.
The OLS model comparisons are reported in Supplemental Table 3. If there were post-treatment bias, we would anticipate that OLS coefficients would differ meaningfully between the analytic and full sample. This difference would result because respondents with some characteristic (such as general distrust of scientific authority) might be more likely to fail an attention check or devote time to the task and to have a lower intent to vaccinate. We would also anticipate that any post-treatment bias would result in similar changes in coefficients across treatment conditions, particularly for those treatment conditions that are most similar (the two video treatment conditions). These models, however, show: 1. A statistically significant association with increased vaccination intent for the video narrated by the man in both the analytic and full samples. There is an increase in effect size of .023 for the analytic sample compared to the full sample (.231 v. 208), which is around 10% of the reported effect size, but substantively a small difference. 2. No association between vaccination intent and the video narrated by the woman, consistent across both samples. There is a small decrease of .004 in the reported coefficient in the analytic sample compared to the full sample (around 6% of the coefficient)-that is, to the extent that there is a difference in Female-Narr-Video estimate between the two samples, it is in the opposite direction as the male-narrated video.

No association between the text intervention and vaccination intent, consistent across both
samples. There is a small increase (.002) in the reported coefficient in the analytic sample compared to the full sample (4%). 4. In the analytic sample, the interaction term between the female narrated video and conservative is negative (and approaching significance). Inclusion of this term increases the coefficient for the main effect of the female narrated video (although it is still not statistically significant). If there are nonproportional effects across different levels of vaccination intent, these patterns could be expected. This interaction term, however, is near zero in the full sample and there is a much smaller change in the main effect coefficient for the female narrated video. We note that partial proportional odds models of the full sample are similar to the patterns above observed in the OLS full sample (significant Male-Narr-Video association and insignificant associations for Female-Narr-Video, the text intervention, and the interaction term).
The finding that the Male-Narr-Video intervention associates with increased vaccination intent is robust to either sample. The inconsistency in main effect coefficient change casts doubt on post-treatment bias as an account for the small differences in the coefficients for the Male-Narr-Video intervention between the analytic and full samples. An alternative, as discussed in the text, is that the difference is due to noise caused by non-compliant respondents who are in the full sample.
The finding of difference between the two samples with the interaction term for Female-Narr-Video and conservative respondents poses a further puzzle for an argument that the results are driven by a posttreatment bias. Given the random assignment to the treatments, we would not anticipate a post-treatment bias emerging in just one of the three experimental conditions. Yet, if the finding in the main text were the result of a post-treatment bias, we would have to conclude that it was the result of a post-treatment bias unique to the one treatment condition (female narration of the video). If true, that fact lends support to the main text's claim that responses to the video with female narration experienced unique variation among respondents. Furthermore, to produce the empirical pattern observed in the analytic sample, a posttreatment bias would need to result in both (a) excluding non-conservative respondents who were more hesitant and (b) excluding conservative respondents who were more intent uniquely happening in the female-narrated video condition.
Logistic regression analysis of the full sample exposed to one of the three treatment conditions (Male-Narr-Video, Female-Narr-Video, and text) can provide further insight about possible post-treatment bias. We summarize the results of this analysis here, reporting odds ratios and p-values. Three control variables significantly associate with an increased likelihood of exclusion: having a B.A. or higher degree (OR 2.170, p<.001), identifying as politically conservative (OR 2.085, p<.001), and identifying as Black (OR 3.099, p<.001). Notably, these predictors include one that consistently associates in our models with one of increased vaccine intent (B.A. or higher) and two that associate with decreased intent (conservative and Black), suggesting that exclusion may not correspond just with characteristics that associate with decreased propensity to vaccinate. When compared to Male-Narr-Video as a reference category, neither of the other treatment conditions has a significant association with exclusion: Female-Narr-Video (OR 1.169, p=.356) and text (OR 0.942, p=.701).
In summary, while exclusion risks post-treatment bias, it does not always produce post-treatment bias. Our examination does not uncover evidence of obvious post-treatment bias and, even if there is unmeasured post-treatment bias, the results of the effects of the Male-Narr-Video are robust to the full sample. Additionally, if the analytic finding of the interaction term between Female-Narr-Video and conservative is the result of a post-treatment bias, it would be one unique to the Female-Narr-Video condition, which is consistent with our interpretation that responses to the the Female-Narr-Video experienced unique variation.

Comparison of Partial Proportional Odds and OLS Models
The use of OLS models for the post treatment bias analysis raises the question of whether OLS models or the partial proportional odds models that we report are better fits for the data. We compared Model 2 of the analytic sample of both OLS and the models reported in the main text using AIC and BIC criteria, both of which provide strong support for the partial proportional odds models. Specifically, the information criteria for the partial proportional odds model (AIC 2861.918, BIC 2983.757, df 24) suggest that these models are a substantial improvement over the OLS model (AIC 3818.797, BIC 3879.717, df 12).

F e m a l e -N a r r -V i d e o
Very Moderately Only a little Not at all

Plot shows median with interquartile range
The narrator was….

Code used for data analysis
We imported the data from Qualtrics to SPSS for initial processing and defining all variables. We then exported the SPSS file into Stata for analysis. We, therefore, include SPSS codes for the data processing and Stata code for the data analysis.

Start of Block: Introductory text
Q1 What: We are conducting this survey to examine how different methods of communication affect how people understand the COVID19 mRNA vaccines. We will share information with you in one randomly assigned format and ask you a series of short questions. The entire process will take less than 20 minutes to complete.

Who: This survey is being conducted by Macalester College Professors Leah Witus and Erik
Larson. If you wish to contact us, you may do so at VaccineCommunicationStudy@gmail.com.
Data Use and Confidentiality: We are not collecting any names or email addresses as part of this survey and will not know your identity. Your responses will be only be seen by Leah Witus and Erik Larson, who will analyze the results. Aggregate results may be included in a publication in a scholarly journal. Data will be stored securely and managed by Leah Witus and Erik Larson.
Participants: This survey is being administered via Amazon Mechanical Turk. Responding to this survey is voluntary and even if you decide to participate, you may decide to withdraw at any point. You will receive payment via Amazon Mechanical Turk by entering the code you receive after the completion of this survey. There is a way to train your immune system to make the right virus-fighting antibodies without getting sick first. And that's what a vaccine is.

End of
Vaccines boosts your immune system so that if you are exposed to a virus you already have the right antibodies ready to stop the virus from replicating and prevent you from getting sick. 1. Vaccines have been used for over 100 years. Back in the day, the vaccines were weakened versions of the actual virus. 2. This initial form of vaccination was admittedly tricky, because the weakened virus given as a vaccine had to be close enough to the actual virus so that the antibodies made by the immune system would work to prevent infection if the person was exposed to the real virus. 3. But the weakened virus vaccine had to be different enough from the real virus so that the vaccine itself would not infect people and make them sick. That was vaccine technology 100 years ago. But today, thanks to decades of biomedical research, we have more options for making safe vaccines.
So how does the COVID-19 vaccine work? The first two vaccines developed for COVID-19, from the companies Pfizer and Moderna, are mRNA vaccines instead of weakened virus vaccines. This is part of the reason they were able to be developed so quickly: instead of needing time to find the delicate balance between being close enough to the actual virus to produce the right antibodies, but weakened enough to not cause infection, mRNA vaccines are clever, because they're not viruses at all. They're just mRNA with the instructions for one part of the virus. And do they work?
your immune system is working hard to produce antibodies after you get the vaccine. A very small number of people have had other side effects.
It may be reassuring to know that the safety standards for vaccines are even higher than for other medicines.
Getting a vaccine is truly like gaining a superpower -it trains your own immune system to fight viruses and protects you from getting sick.
One more note about vaccines. You may hear talk of vaccination rates. Some people call this herd immunity. 1. The idea is that there are some people in any society that cannot get a vaccine for a particular virus. This includes small babies, and people with immune disorders. 2. So how can we protect these people that are not able to get a vaccine? Well if the vast majority of people in a population do get vaccinated, the virus will not be able to spread, because it can't travel from person to person, since most people have antibodies. This protects the people that can't get vaccinated. 3. That's why, even if you are not particularly worried about getting sick from a certain virus, you can still do a societal good deed if you get vaccinated.
So to sum it up, mRNA vaccines are a new type of safe, and highly effective vaccine, getting vaccinated for COVID-19, and other diseases, is a way to train and boost your body's natural immune system and it could save another person's life. Not bad.  (1) Only a little (2) Moderately (3) Very (4) The text was enjoyable (1) o o o o The text was informative (2)  Once you have copied this code, be sure to click the next arrow to submit your survey.