https://orcid.org/0000-0002-1816-1676
Figures
Abstract
Aim
When developing public health measures in a pandemic, it is important to examine attitudes and beliefs relating to vaccination uptake. We report the discrimination of a single-item vaccination intention scale and derive cutpoints in terms of sensitivity (true positives) and specificity (true negatives) in relation to subsequent vaccination status.
Subject and Methods
In a sample of UK adults (n=1119) recruited through an online survey platform, vaccination intention was measured on a 0–10 numerical rating scale (0=very unlikely, 10=very likely) at the beginning of the UK COVID-19 vaccination rollout (January 2021), and self-reported vaccination status was gathered after vaccination had been offered to all adults (October 2021). Discrimination of the scale was measured by the area under the receiver operating characteristic (ROC) curve.
Results
The responders reporting being vaccinated or unvaccinated were 1034 (92.4%) and 85 (7.6%), respectively. The area under the ROC curve was.956 (95% CI.943,.967), indicating a high degree of discrimination. The combined value of sensitivity and specificity was greatest at a cutpoint of 8 on the scale (sensitivity =.821, specificity =.988). If, however, the individual values of sensitivity and specificity are required to be simultaneously optimized, this occurs at point 6 (sensitivity =.886, specificity =.871).
Citation: Sim J, Smith LE, Amlôt R, Rubin GJ, Sevdalis N, Sherman SM (2025) Discrimination of a single-item scale to measure intention to have a COVID-19 vaccine. PLoS One 20(5): e0322503. https://doi.org/10.1371/journal.pone.0322503
Editor: Cristiana Abbafati, Sapienza University of Rome, ITALY
Received: January 28, 2025; Accepted: March 23, 2025; Published: May 5, 2025
Copyright: © 2025 Sim et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data are available on the OSF https://osf.io/2jm8p/
Funding: Financial support for this study was provided by a Keele University Faculty of Natural Sciences Research Development award to SS, JS and NS, and a King’s COVID Appeal Fund award granted jointly to LS, GJR, RA, NS, SS and JS. LS, RA and GJR are funded by the National Institute for Health and Care Research Health Protection Research Unit (NIHR HPRU) in Emergency Preparedness and Response, a partnership between the UK Health Security Agency, King's College London and the University of East Anglia. The views expressed are those of the authors and not necessarily those of the NIHR, UK Health Security Agency, the charities or the Department of Health and Social Care.
Competing interests: No authors have competing interests.
Introduction
Effective public health measures are crucial in countering the impact of a pandemic such as COVID-19. It is important, therefore, to understand attitudes and beliefs in the general public towards outbreaks of infectious diseases, and intentions regarding the uptake of specific preventive initiatives in particular, as the basis for an appropriate and effective public health strategy [1]. A key element in such a strategy in relation to COVID-19 has been vaccination [2,3]. A number of recent studies have focused on sociodemographic, clinical, behavioural and attitudinal factors associated with willingness to be vaccinated against COVID-19 and have included a measure of vaccination intention [4–8]. These studies attempt to identify psychological factors that could be modified and socio-demographic variables that could be targeted to improve vaccination uptake. However, intention does not always lead to behaviour [9].
Multi-item measures of vaccination intention potentially have good predictive properties by virtue of their coverage of multiple predictors, and the use of multiple items increases the reliability (internal consistency) of a scale [10]. There is, however, an abundance of multi-item scales investigating vaccine- or theory-specific determinants of vaccination through self-report. The sheer volume and variability of those scales means that a single-item measure might be helpful not only in offering an efficient (if imperfect) measure, but also in allowing some comparability if used alongside and/or correlated with multi-item scales in subsequent research. Additionally, single-item measures may be more practical in routine clinical or public health practice, where the time and resources required to collect and analyse data from a wide range of variables may be lacking. Similarly, single-item measures may be more suitable for inclusion in population surveys that seek to cover a broad range of attitudes, beliefs or reported behaviours, where parsimony may be needed in terms of the items included in the survey instrument and the amount of data generated [11]. Single-item measures may also be more intuitive to respondents, thereby increasing their face validity; this, and the speed with which a single item can be completed, may minimize missing data and improve the response rate [12]. Furthermore, whilst the determinants of a person’s intention to be vaccinate are likely to be multiple, requiring separate indicators of different constructs, the strength of the resulting intention is unidimensional and thus in principle suited to measurement by a single item.
Attitudes and related psychological variables are typically measured on some type of numerical rating scale, so that their intensity can be quantified. If, however, a rating scale is used to indicate the intended performance or non-performance of a future action, an appropriate cutpoint has to be identified on the scale in order to derive this binary classification. Such a cutpoint should demonstrate good discrimination (classificatory accuracy). Using data from a broader study, we report on the discrimination of a simple, pragmatic single-item rating scale used to measure intention to have a COVID-19 vaccination in a UK sample of adults and derive an optimal cutpoint on the scale.
The aim of this study was to test the discrimination of a single-item vaccination intention scale, and to derive cutpoints, in terms of sensitivity and specificity, that could act as practical indicators of future vaccination status.
Statistical considerations
In clinical epidemiology, important statistical features of a test intending to produce a diagnostic, prognostic or other binary classification are sensitivity and specificity. These statistics indicate the relationship between a positive or negative outcome on the test and the true positive or negative status of those tested. Sensitivity indicates the proportion of those who test positive that are truly positive and thereby indicates the probability of a positive test given that disease is present, while specificity indicates the proportion of those who test negative that are truly negative and thereby indicates the probability of a negative test given that disease is absent [13]. They are estimated as [14]:
There is a trade-off between sensitivity and specificity – improving the sensitivity of a test will reduce its specificity, and vice versa. Sensitivity and specificity are not influenced by the prevalence of the outcome of interest.
It is also possible to calculate positive and negative predictive value. Whereas sensitivity is the probability of a positive test when disease is present, positive predictive value is the probability of disease when a test is positive. Correspondingly, whereas specificity is the probability of a negative test when disease is absent, negative predictive value is the probability of the absence of disease when a test is negative. Unlike sensitivity and specificity, positive and negative predictive value are influenced by the prevalence of the disease [15].
Translating this into the current case, being classified as positive or negative equates to being deemed likely or unlikely to be vaccinated, respectively, and being a true positive or a true negative equates to subsequently being vaccinated or unvaccinated, respectively (Table 1). The sensitivity and specificity of a specific cutpoint on an attitude scale used to predict vaccination status can be calculated accordingly.
Materials and methods
Sample
Data were obtained from two waves of the COVID-19 Vaccine Acceptability Study (CoVAccS), which investigated attitudes and beliefs relating to COVID-19 and COVID-19 vaccination in the UK; measurement of vaccination intention was part of this study. The survey recruited a representative sample of respondents through an online survey research platform (Qualtrics); detailed methods are reported elsewhere [16,17]. One wave of data collection was completed during 13–15 January 2021, at the very beginning of the UK COVID-19 vaccination rollout (T1), with a further longitudinal follow-up wave completed during 4–15 October 2021, after the vaccination had been offered to all adults (T2). The T1 questionnaire included a question on vaccination intention: ‘Now that a coronavirus vaccination is available, how likely is it that you will have one?’ Participants were asked to respond on an eleven-point numerical rating scale, anchored 0 = ‘extremely unlikely’ and 10 = ‘extremely likely’ (S1 Fig). A continuous, rather than a categorical, intention scale had been used in the main study in order to facilitate the analyses used in a linear regression model.
Of the 1148 respondents to both waves of the survey, five had missing data and those who had already received one or more doses of the vaccine at T1 (n=24) were excluded from the analysis. This left 1119 participants (97.5%) who had completed the intention scale at T1 and subsequently reported their vaccination status at T2. The mean (standard deviation) age of participants at T1 was 48.2 (15.1) years. There were 594 (53.1%) female and 522 (46.6%) male respondents; 3 (0.3%) other gender identities were reported.
Statistics
The discrimination of the intention scale and an appropriate predictive cutpoint within the scale were identified through nonparametric receiver operating characteristic (ROC) analysis. This produces a ROC curve that plots sensitivity (true positive rate) on the vertical axis against 1 – specificity (false positive rate) on the horizontal axis. A curve that assumes a straight line on an upward diagonal from the bottom left-hand to the top right-hand corner on this plot represents a process of classification that is no better than chance, or guessing. Curves that depart from the diagonal and approach the upper left-hand corner represent progressively greater discrimination. Movement of the curve upwards on the vertical axis denotes increasing sensitivity and movement to the left on the scale denotes increasing specificity, i.e., lower values of (1 – specificity). The area under the ROC curve (AUC) was estimated and an exact binomial 95% confidence interval (CI) calculated. The AUC quantifies the classificatory accuracy of the scale, taken across the full range of cutpoints, from a minimum of 0 to a maximum of 1 [13]. The points on the scale can be identified at different places on the ROC curve and their respective discrimination can thereby be determined.
A cutpoint was defined as the point on the scale at or above which an individual would be predicted to be vaccinated. We examined two methods that have been proposed to identify an optimal cutpoint. The first of these, the Youden index [18], represents the point on the scale at which sensitivity and specificity are maximized, without according any differential weight to either sensitivity or specificity – it seeks the overall maximum of the two. This index is calculated for a specific cutpoint (c) as:
For example, a cutpoint with a sensitivity.610 of and a specificity of.720 would yield a Youden index of.610 + (.720–1) =.330. The point on the scale with the largest Youden index is the optimal cutpoint on this criterion.
The second method is the index of union [19], which differs from the Youden index in that it seeks to maximize sensitivity and specificity at the same time; it seeks the point on the scale at which both of these are simultaneously as high as possible. This statistic takes into account the AUC from the ROC curve and is calculated for a specific cutpoint (c) as:
For example, if the AUC were.789, a cutpoint with a sensitivity.610 of and a specificity of.720 would yield an index of union of (│610 –.789│ + │.720 –.789│) =.248. The point on the scale with the smallest index of union index is the optimal cutpoint on this criterion.
Finally, we calculated positive likelihood ratios (LR +) and negative likelihood ratios (LR –) for points on the scale [20]:
The positive likelihood ratio is the probability of a positive classification among vaccinated individuals divided by the probability of a positive classification among unvaccinated individuals. A ratio greater than 1 indicates how much more likely a vaccinated individual is to be classified as positive than an unvaccinated individual; higher values are better. The negative likelihood ratio is the probability of a negative classification for a vaccinated individual divided by the probability of a negative classification for an unvaccinated individual. A ratio less than 1 indicates how much less likely a vaccinated individual is to be classified as negative than an unvaccinated individual; lower values are therefore better. Analyses were conducted in Stata 17.
Results
Table 2 indicates, for each point on the intention scale, the number of participants who reported being either vaccinated or unvaccinated. Of the 1034 (92.4%) participants who reported being vaccinated, 25 reported having received one dose and 1009 reported having received two doses. The vaccine received was reported as follows: Astra-Zeneca, 597 (57.7%); Pfizer-BioNtech, 395 (38.2%); Moderna, 36 (3.5%); Janssen (Johnson & Johnson), 1 (0.1%); another vaccine or don’t know, 5 (0.5%). Only 85 (7.6%) out of 1119 respondents remained unvaccinated; none of these had scored either 9 or 10 on the intention scale. Fig 1 shows the ROC curve. The AUC was.956 (95% CI .943,.967; n=1119), indicating a high degree of discrimination.
Table 3 shows the sensitivity and specificity of each point on the scale, as well as the likelihood ratios, the Youden index and the index of union. Values of sensitivity, specificity, and sensitivity-plus-specificity are also displayed in Fig 2. The maximum combined sensitivity and specificity was for point 8 on the scale, as it here that the Youden index was highest (.809). Here, sensitivity is .821 (95% CI .797, .843) and specificity is .988 (95% CI .936, .998). The positive likelihood ratio for this point on the scale is 69.792, indicating that a person ultimately vaccinated is nearly 70 times more likely to be classified as positive than a person ultimately unvaccinated, at this cutpoint. The negative likelihood ratio of 0.181 for point 8 indicates that a person ultimately vaccinated is only 18% as likely to be classed as negative than a person ultimately unvaccinated. At point 8, positive predictive value is .999 (95% CI .993, 1.000) and negative predictive value is .312 (95% CI .260, .370).The lowest index of union (.154) is for scale point 6, indicating that sensitivity and specificity are simultaneously at their maximum at this point; sensitivity is .886 (95% CI .865, .904) and specificity is .871 (95% CI .783, .926). Here, positive predictive value is .988 (95% CI .979, .993), and negative predictive value is .387 (95% CI .321, .458).
One might wish to use the intention scale to predict unvaccinated, rather than vaccinated, status, defined as a scale point at or below which an individual is predicted to remain unvaccinated. The overall discrimination of the scale and the optimal cutpoints remain unchanged, but values of sensitivity and specificity are transposed. These are shown in in Table 4.
Discussion
Our analysis suggests that an optimal cutpoint on this intention scale in relation to a judgment of subsequent vaccination is 8, in that the combination of sensitivity and specificity is greater here than at any other point on the scale. This is the a priori cutpoint that had been used in previous analysis of data from the CoVAccS study [16,17,21] and our analysis here validates this choice. If, however, it is desired to maximize both sensitivity and specificity simultaneously, the optimal cutpoint would be 6, as both sensitivity and specificity are at their maximum at this point on the scale.
The Youden index and the index union serve to maximize sensitivity and specificity, in different ways. However, there may be reasons to opt for cutpoints that prioritize one of the other of these statistics. Accordingly, one can choose other cutpoints on the scale with a view to achieving a high level of either sensitivity or specificity, based upon the relative costs of misclassification (i.e., false positives or false negatives). For example, it might be thought most important to avoid a high false negative rate, which would mean that the future vaccination rate would be underestimated and efforts to promote vaccination might therefore be disproportionate to their benefit. In such a scenario, the emphasis would be placed on sensitivity and a cutpoint lower than 8 might be favoured (Table 3). However, this requires caution in view of the inverse relationship of sensitivity and specificity; privileging sensitivity in this way will be at the cost of specificity. Conversely, the concern might be about a high false positive rate, whereby the future vaccination rate would be overestimated and insufficient effort might be put into a vaccination campaign as a result. Here, the emphasis would switch to specificity and a cutpoint of 8, at which specificity is virtually at its maximum, would remain a reasonable choice, though at the cost of some degree of sensitivity.
There are differing opinions on what constitutes a poor, moderate or good value of the AUC, but according to de Hond et al [22], most commentators would classify the value of.956 that we obtained under the labels ‘very good’, ‘high’ or ‘excellent’. Our estimate compares favourably with the findings of a study examining vaccination willingness, as predicted by a measure of fear of COVID-19; this yielded AUC values of.57 and.69, at two different timepoints [23]. Other studies of prediction tools related to COVID-19 have yielded AUC values of.79 [24],.88 [25] and.92 [26], with which our estimate again compares favourably.
A strength of our analysis is that participants’ reporting of vaccination status is highly unlikely to have been influenced by recall of their prior vaccination intention as this occurred more than eight months previously. The sample size was large and therefore provided a precise estimate of the AUC, as indicated by the width of the associated confidence interval. However, the small number of unvaccinated participants means that our estimates of specificity in Table 3 are less precise than those of sensitivity (and, correspondingly, our estimates of sensitivity in Table 4 are less precise than those of specificity). It should be noted that our analysis is based on reported vaccination status rather than actual vaccination status, on which we had no information. It is unlikely that participants wrongly recalled their vaccination status [27–30], but they might have reported it incorrectly through the influence of social desirability or cognitive dissonance. As the study was completed anonymously, the influence of social desirability should be lessened. It is also possible that some respondents confused vaccination for COVID-19 with vaccination for influenza, though as we asked respondents separately about influenza vaccination this may be unlikely. The use of projective techniques [31] or other indirect measures of intention may offset a social desirability bias, but may be less intuitive than a question directly focused on vaccination intention.
Positive and negative predictive value depend upon prevalence, and in a population with a lower rate of vaccination, positive predictive value would be expected to be lower, and negative predictive value to be higher, than reported here [15]. The estimates of these statistics that we have reported should not therefore be generalized to populations in which the rate of vaccination differs. In contrast, sensitivity and specificity do not depend upon prevalence, so our findings may apply to other countries with different rates of vaccine uptake. However, further testing and validation would be required to determine whether they are transferable to other populations, or to other types of vaccination than COVID-19. Differences in the perceived risk of a different type of infection and of the effectiveness of the vaccine available, together with public health messaging and the availability of relevant healthcare interventions, might affect the measurement of vaccination intention. Finally, our findings relate to vaccination status nine months after vaccination intention was measured, as dictated by the broader study in which this analysis is based; those who were unvaccinated at this time may well have subsequently received a vaccine.
Conclusion
A single-item measure of vaccination intention is quick to administer and does not require complex or lengthy analysis in order to predict subsequent (non-)uptake of vaccination. In view of the good discrimination of the 0–10 vaccination intention scale (with anchors of ‘extremely unlikely’ and ‘extremely likely’) that we have evaluated, its use represents an effective yet pragmatic option within routine public health practice to identify individuals who will subsequently have themselves vaccinated. Unless a differential weight is to be placed on either sensitivity or specificity, we recommend a cutpoint of 8 when seeking to identify those who will subsequently receive vaccination.
References
- 1. Adu P, Poopola T, Medvedev ON, Collings S, Mbinta J, Aspin C, et al. Implications for COVID-19 vaccine uptake: A systematic review. J Infect Public Health. 2023;16(3):441–66. pmid:36738689
- 2. Suthar AB, Wang J, Seffren V, Wiegand RE, Griffing S, Zell E. Public health impact of covid-19 vaccines in the US: observational study. BMJ. 2022;377:e069317. pmid:35477670
- 3. Haas EJ, Angulo FJ, McLaughlin JM, Anis E, Singer SR, Khan F, et al. Impact and effectiveness of mRNA BNT162b2 vaccine against SARS-CoV-2 infections and COVID-19 cases, hospitalisations, and deaths following a nationwide vaccination campaign in Israel: an observational study using national surveillance data. Lancet. 2021;397(10287):1819–29. pmid:33964222
- 4. Shiloh S, Peleg S, Nudelman G. Vaccination against COVID-19: A longitudinal trans-theoretical study to determine factors that predict intentions and behavior. Ann Behav Med. 2022;56(4):357–67. pmid:34864833
- 5. Enticott J, Gill JS, Bacon SL, Lavoie KL, Epstein DS, Dawadi S, et al. Attitudes towards vaccines and intention to vaccinate against COVID-19: a cross-sectional analysis-implications for public health communications in Australia. BMJ Open. 2022;12(1):e057127. pmid:34980631
- 6. Ogilvie GS, Gordon S, Smith LW, Albert A, Racey CS, Booth A, et al. Intention to receive a COVID-19 vaccine: results from a population-based survey in Canada. BMC Public Health. 2021;21(1):1017. pmid:34051770
- 7. Sherman SM, Smith LE, Sim J, Amlôt R, Cutts M, Dasch H, et al. COVID-19 vaccination intention in the UK: results from the COVID-19 vaccination acceptability study (CoVAccS), a nationally representative cross-sectional survey. Hum Vaccin Immunother. 2021;17(6):1612–21. pmid:33242386
- 8. Stead M, Jessop C, Angus K, Bedford H, Ussher M, Ford A, et al. National survey of attitudes towards and intentions to vaccinate against COVID-19: implications for communications. BMJ Open. 2021;11(10):e055085. pmid:34711602
- 9. Sniehotta FF, Scholz U, Schwarzer R. Bridging the intention–behaviour gap: Planning, self-efficacy, and action control in the adoption and maintenance of physical exercise. Psychology & Health. 2005;20(2):143–60.
- 10.
Nunnally J, Bernstein I. Psychometric theory, 3rd edn. New York: McGraw-Hill; 1994.
- 11. Allen MS, Iliescu D, Greiff S. Single item measures in psychological science. Eur JPsychol Assess. 2022;38(1):1–5.
- 12. Wanous JP, Reichers AE, Hudy MJ. Overall job satisfaction: how good are single-item measures?. J Appl Psychol. 1997;82(2):247–52. pmid:9109282
- 13.
Pepe M. The statistical evaluation of medical tests for classification and prediction. Oxford: Oxford University Press; 2003.
- 14.
Woodward M. Epidemiology: study design and data analysis, 2nd edn. Boca Raton: Chapman & Hall; 2004.
- 15.
Streiner DL, Norman GR. PDQ epidemiology, 3rd edn. Hamilton: People’s Medical Publishing House; 2012.
- 16. Sherman SM, Sim J, Cutts M, Dasch H, Amlôt R, Rubin GJ, et al. COVID-19 vaccination acceptability in the UK at the start of the vaccination programme: a nationally representative cross-sectional survey (CoVAccS - wave 2). Public Health. 2022;202:1–9. pmid:34856520
- 17. Smith LE, Sim J, Cutts M, Dasch H, Amlôt R, Sevdalis N, et al. Psychosocial factors affecting COVID-19 vaccine uptake in the UK: A prospective cohort study (CoVAccS - Wave 3). Vaccine X. 2023;13:100276. pmid:36819214
- 18. Youden WJ. Index for rating diagnostic tests. Cancer. 1950;3(1):32–5.
- 19. Unal I. Defining an optimal cut-point value in ROC analysis: An alternative approach. Comput Math Methods Med. 2017;2017:3762651. pmid:28642804
- 20. Choi BC. Slopes of a receiver operating characteristic curve and likelihood ratios for a diagnostic test. Am J Epidemiol. 1998;148(11):1127–32. pmid:9850136
- 21. Smith LE, Sim J, Amlôt R, Cutts M, Dasch H, Sevdalis N, et al. Side-effect expectations from COVID-19 vaccination: Findings from a nationally representative cross-sectional survey (CoVAccS - wave 2). J Psychosom Res. 2021;152:110679. pmid:34823113
- 22. de Hond AAH, Steyerberg EW, van Calster B. Interpreting area under the receiver operating characteristic curve. Lancet Digit Health. 2022;4(12):e853–5. pmid:36270955
- 23. Mertens G, Lodder P, Smeets T, Duijndam S. Fear of COVID-19 predicts vaccination willingness 14 months later. J Anxiety Disord. 2022;88:102574. pmid:35512598
- 24. Gordon AJ, Govindarajan P, Bennett CL, Matheson L, Kohn MA, Camargo C, et al. External validation of the 4C Mortality Score for hospitalised patients with COVID-19 in the RECOVER network. BMJ Open. 2022;12(4):e054700. pmid:35450898
- 25. Liang W, Liang H, Ou L, Chen B, Chen A, Li C, et al. Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19. JAMA Intern Med. 2020;180(8):1081–9. pmid:32396163
- 26. Dong Y-M, Sun J, Li Y-X, Chen Q, Liu Q-Q, Sun Z, et al. Development and validation of a nomogram for assessing survival in patients with COVID-19 pneumonia. Clin Infect Dis. 2021;72(4):652–60. pmid:32649738
- 27. Stephenson M, Olson SM, Self WH, Ginde AA, Mohr NM, Gaglani M, et al. Ascertainment of vaccination status by self-report versus source documentation: Impact on measuring COVID-19 vaccine effectiveness. Influenza Other Respir Viruses. 2022;16(6):1101–11. pmid:35818721
- 28. Wielgosz K, Dawood FS, Stockwell MS, Varner M, Newes-Adeyi G, Ellington S, et al. Agreement between pregnant individuals’ self-report of coronavirus disease 2019 (COVID-19) vaccination and medical record documentation. Obstet Gynecol. 2022;140(6):989–92. pmid:36357976
- 29. Surie D, Bonnell LN, DeCuir J, Gaglani M, McNeal T, Ghamande S, et al. Comparison of mRNA vaccine effectiveness against COVID-19-associated hospitalization by vaccination source: Immunization information systems, electronic medical records, and self-report-IVY Network, February 1-August 31, 2022. Vaccine. 2023;41(29):4249–56. pmid:37301704
- 30. Archambault PM, Rosychuk RJ, Audet M, Bola R, Vatanpour S, Brooks SC, et al. Accuracy of self-reported COVID-19 vaccination status compared with a public health vaccination registry in Québec: Observational diagnostic study. JMIR Public Health Surveill. 2023;9:e44465. pmid:37327046
- 31.
Kaufman MB, Bornstein MH. Projective tests. In: Frey BB, editor. The SAGE encyclopedia of educational research, measurement and evaluation. Vol 3. Thousand Oaks: Sage; 2018. p. 1322–5.