Figures
Abstract
Background
Financial incentives may be important for improving response rates to data collection activities and for retaining participants in longitudinal studies. However, for large studies, this introduces significant additional costs. We sought to determine whether an opt-in or an opt-out option for receiving financial incentives when completing questionnaires offers any cost saving measures.
Methods
The Avon Longitudinal Study of Parents and Children has been ongoing for more than 30 years. It has offered a £10 incentive for returning a partly or fully completed annual questionnaire for >10 years, this is provided by default unless a participant chooses to opt out. For questionnaires completed in 2020 by the original parents recruited to the study and by their offspring, we randomised eligible participants to either opt-out or to opt-in to receiving their vouchers. Logistic regressions determined whether opt-out or opt-in made any difference to the proportion of respondents receiving their vouchers.
Results
Respondents are less likely to choose to receive a thank you for their time in the form of a £10 shopping voucher if they are asked to opt in compared to if they are asked to opt out. The odds ratio, adjusted for baseline characteristics was 3.94 (95% Confidence Interval: 3.49, 4.45). There was no difference in response rates according to whether respondents were randomised to the opt-in or opt-out group.
Citation: Northstone K, Bowring C (2025) Does opting in or out affect the take up of incentives in a long running population-based cohort study: A nested randomised trial in ALSPAC. PLoS ONE 20(1): e0316534. https://doi.org/10.1371/journal.pone.0316534
Editor: Adnan Ahmad Khan, Research and Development Solutions, PAKISTAN
Received: July 29, 2024; Accepted: December 13, 2024; Published: January 28, 2025
Copyright: © 2025 Northstone, Bowring. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The informed consent obtained from ALSPAC participants does not allow the data to be made freely available through any third party maintained public repository. However, data used for this submission can be made available on request to the ALSPAC Executive. The ALSPAC data management plan describes in detail the policy regarding data sharing, which is through a system of managed open access. Full instructions for applying for data access can be found here: http://www.bristol.ac.uk/alspac/researchers/access/. The ALSPAC study website contains details of all the data that are available (http://www.bristol.ac.uk/alspac/researchers/our-data/).
Funding: The UK Medical Research Council and Wellcome (Grant ref: 217065/Z/19/Z) and the University of Bristol provide core support for ALSPAC. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. This publication is the work of the authors and KN will serve as guarantor for the contents of this paper. A comprehensive list of grants funding is available on the ALSPAC website (http://www.bristol.ac.uk/alspac/external/documents/grant-acknowledgements.pdf).
Competing interests: The authors have declared that no competing interests exist.
Abbreviations: ALEC, ALSPAC Law and Ethics Committee; ALSPAC, Avon Longitudinal Study of Parents and Children; G0, Generation 0 (original parents enrolled in the study); G1, Generation 1 (originally offspring of the parents enrolled in the study)
Background
Effective strategies are critical to engage and retain participants taking part in long term studies. A recent systematic review identified that the most common retention strategy employed was a cash/voucher incentive (59/95 studies reviewed) [1]. However, the authors acknowledged high levels of heterogeneity in this review due to the range of methods, study types and hypotheses examined in the reviewer papers [1]. Nevertheless, there is also evidence to suggest that monetary incentives can improve selection bias in long term research studies [2, 3].
The Avon Longitudinal Study of Parents and Children has been collecting data from its participants via a number of methods for over 30 years. Participants in the study (parents—generation 0 (G0), and their offspring—generation 1 (G1) now aged ~31 years) complete questionnaires annually and they are offered a small incentive of a £10 high street shopping voucher for returning a partly or fully completed a questionnaire. These vouchers can be used in many UK shops and online stores including supermarkets, general household, health and gift stores. This has received full ethical approval from the ALSPAC Law and Ethics Committee (ALEC). Across both generations we would anticipate upwards of 10,000 completed questionnaires. This comes at a substantial financial burden of over £100,000 in incentives alone.
Anecdotally, we are aware that some participants do not use their voucher once it has been issued or that they would be happy to complete their questionnaires regardless of being incentivised. However, we have evidence to suggest that providing an incentive does offer improvement to response rates: We did not offer incentives for our 21-year questionnaire, to G1 respondents, and the response rate dropped by 5.5% compared to the previous questionnaire, completed at 20 years of age. It recovered to previous rates in the next questionnaire (age 22 years) when incentives were offered again. When incentives have been offered in recent years, the study has provided an opt-out option for these respondents to indicate that they do not wish to receive their voucher, however, we were interested to see whether an opt-in option could introduce cost saving, without impacting response rates, given the large cost associated with providing incentives in ALSPAC as noted above. This change in strategy was informed by the “default effect”. This theory suggests that individuals are more likely to accept a default option rather than indicate a different course of action [4]. A frequently cited example from Johnson & Goldstein [5] demonstrates this effect very clearly: They showed that using opting in to organ donation resulted in a much lower sign up of organ donors compared to using opting out. Indeed, Jachimowicz et al reported in their meta-analysis of 58 studies examining default effects that opt-out defaults led to greater uptakes compared to opt-in [6]. However, this effect was moderated by different characteristics of the study types, with substantial heterogeneity seen in effect sizes according to study type. Nevertheless, we hypothesised that changing the option for receiving a voucher from opt-out to opt-in would reduce the uptake of vouchers and thus provide some cost savings.
We therefore randomised eligible participants who received invitations to complete an annual questionnaire sent out to both generations in late 2019/early 2020 to opt-out of receiving their voucher or to opt-in to receiving it.
Methods
The primary aim of the study was to determine whether an opt-in or an opt-out option for receiving financial incentives for returning a partly or fully completed questionnaire offers any cost saving measures for the study. Our secondary aim was to determine the characteristics of those respondents who chose to opt-out versus those who chose not to opt-in and to see whether response rate was affected according to whether respondents were in the opt-in or opt-out arm.
Study population
The Avon Longitudinal Study of Parents and Children (ALSPAC) is an ongoing population-based, birth cohort based in Bristol in Southwest England. Pregnant women resident in the area with expected dates of delivery between 1st April 1991 and 31st December 1992 were invited to take part. These women, their partners (collectively known as G0) and their resulting children (known as G1) [7, 8] have been followed up ever since. 14,541 pregnancies were originally enrolled, however, when the oldest children were approximately 7 years of age, an attempt was made to bolster the initial sample with eligible cases who had failed to join the study originally. This resulted in an additional 913 children being enrolled. 14,901 children were alive at 1 year of age [9] and a total of 14,833 unique women (G0 mothers) enrolled in ALSPAC as of September 2021 [10]. 12,113 G0 partners have been in contact with the study at some point over the last 30 years by providing data and/or formally enrolling from 2010. 3,807 G0 partners are currently enrolled [11].
Process
Throughout the life of the cohort, the primary method of data collection has been via self-completion questionnaires. These cover a wide range of topics and have been administered roughly annually over recent years to both generations. As part of questionnaires planned for distribution in the winter of 2020 to both generations (when G1 were around 27 years of age), we chose to test the value of opting in versus opting out.
All G0 and G1 participants with an active email address were sent an invitation to complete the online questionnaire. Participants were not contacted if our administrative database record indicated that they were deceased, had withdrawn from the study, had declined further contact, had declined to complete questionnaires or we did not have up to date contact details for them.
Non-responders were sent a reminder email to complete the online questionnaire. As part of our general questionnaire reminder strategy, a paper version of the questionnaire was then sent to those who had still not complete online or those participants for whom we did not have a valid email address. Respondents were randomised to either opt-out or opt-in with respect to receiving a £10 voucher as a thank you for completing the questionnaire.
At the end of the questionnaire, respondents were reminded that they could receive a voucher as a thankyou for returning a partly or fully completed questionnaire. Those respondents in the opt-out group were asked “If you don’t wish to receive your thank you voucher, please, check/cross this box” for the paper/online versions respectively. This compared to the text provided to those in the opt-in group: “If you would like to receive a thank you voucher for completing your questionnaire, please check/cross this box.” Given the change in process, those who were randomised to the opt-in process were given the following warning in their invitations: “This year we have changed the way in which we process the vouchers sent out for completing your questionnaire. If you would like to receive a thank-you voucher, please make sure that you check the box at the end of the questionnaire.”Vouchers were received if respondents in the opt-in group ticked the box, or if respondents in the opt-out group did not tick the box. We present the results in terms of respondents willing to receive their voucher.
Please note, the study website contains details of all the data that is available through a fully searchable data dictionary and variable search tool. The online version of the questionnaire was developed and deployed using REDCap (Research Electronic Data CAPture tools [12]; a secure web application for building and managing online data collection exercises, hosted at the University of Bristol. Paper questionnaires were designed, scanned and verified using Teleform data capture software.
The questionnaire for G0 was open between 23rd January and 31st October 2020 and for G1 between 15th November 2019 and 31st July 2020.
Ethics approval and consent to participate
Ethical approval for the study was obtained from the ALSPAC Ethics and Law Committee and the Local Research Ethics Committees. Informed consent for the use of data collected via questionnaires was obtained from all participants following the recommendations of the ALSPAC Ethics and Law Committee at the time. The submission of a completed questionnaire, either on paper or online, was considered to be written consent from participants to use their data for research purposes. Study participants have the right to withdraw their consent for elements of the study or from the study entirely at any time. Full details of the ALSPAC consent procedures are available on the study website (http://www.bristol.ac.uk/alspac/researchers/research-ethics/).
Baseline characteristics
We considered a number of baseline characteristics including mode of questionnaire completion (online or paper) and participant type (G0 mother, G0 partner, G1 male or female), highest level of education (< O level, O level or > O level, where O levels are compulsory exams taken at the age of 16 years in the UK) and Ethnicity (white, other than white).
Statistical analyses
Baseline characteristics (education, ethnicity and participant type) of the eligible participants were compared to determine whether randomisation was balanced. We explored whether different groups of eligible participants were more or less likely to complete the questionnaire in the first place, according to whether they were randomised to opt-in or to opt-out using chi-squared tests to examine any differences. Next we compared differences in baseline characteristics and participant type (G0 or G1, male or female) according to whether respondents received a voucher or not. Logistic regressions were used to investigate the characteristics associated with receiving or not receiving a voucher. Unadjusted odds ratios (ORs) and 95% confidence intervals (CIs) were calculated for each characteristic. Adjusted ORs and 95% CIs were calculated, mutually adjusting for all other characteristics. We calculated the total cost of incentives provided in the opt-in and opt-out groups according to respondent’s baseline characteristics to determine whether there were any differences in savings when moving to opt-in versus opt-out. Here, we employed chi-squared tests to compare the actual savings with the expected savings, assuming that there were no differences between different groups. Finally, we present the proportion of responders opting in to receive their incentive, who completed questionnaires that were sent out after the COVID-19 pandemic and estimated the savings that were made as a result of moving from an opt-out to an opt-in approach.
Results
Response rate
In total, 20,532 eligible participants were invited to complete a questionnaire with 10,737 completing at least one section (response rate of 52.3%; i.e. the number of respondents divided by the number of eligible participants). The questionnaires were completed between January and October 2020 by G0 and between November 2019 and July 2020 by G1.
Overall, 49.7% of all eligible participants were randomised to the opt-in option with 50.3% randomised to the opt-out option. Randomisation was balanced according to eligible participant type, education level and ethnicity (all p>0.05; Table 1).
There were no differences in the proportions of respondents completing the questionnaire who were in the opt-in or opt-out arms (Table 1). Of those who responded, similar proportions of opt-ins and opt-outs were seen according to respondent type or their baseline characteristics (all p>0.10; Table 2).
Overall, 17.8% of respondents chose not to receive their thank you voucher (Table 3). A greater proportion of respondents received a voucher if they were in the opt-out arm (90.7%, compared to 73.7% of those who had to opt-in; unadjusted OR 3.49 (95% CI: 3.12, 3.89).
A greater proportion of G0 mothers than G0 partners received their voucher (77.5% vs 70.2%; Table 3). However, a greater proportion of G1 respondents received their voucher compared to their parents. A slightly higher proportion of G1 females received their voucher compared to G1 males (92.4% vs 89.4%). In terms of education, the lowest proportion of respondents receiving their voucher were in highest category of attainment compared to the other two groups (79.5% vs 84.0% and 84.5%). A greater proportion of respondents of other than white ethnicity received their voucher (87.2%) compared to whites (81.7%).
After adjustment for all other characteristics, the OR for receiving a voucher increased from 3.49 (95% CI: 3.12, 3.89) to 3.94 (95% CI: 3.49, 4.45) for those in the opt-out group compared to the opt-in group. All other characteristics remained strongly associated with voucher receipt after adjustment (Table 3).
It can be seen in Table 4 presents that differential cost savings were made in the opt-in group versus the opt-out group. Overall the total cost of incentives was £9,740 in the opt-in group compared to the opt-out group. Observed differences were larger than expected in those respondents completing their questionnaires on paper, in the G0 respondents, in the highest educational attainment group and in respondents of white ethnicity (all p<0.001).
Discussion
By undertaking a nested randomised trial within an existing longitudinal population study, we have shown that respondents are less likely to choose to receive a thankyou for their time in the form of a £10 shopping voucher if they are asked to opt-in compared to if they are asked to opt-out. Larger reductions in the observed cost of incentives were seen in the older generation, those of who higher education, completing on paper and of white ethnicity,
There is already convincing evidence to show that providing monetary incentives improve response rates to randomised controlled trials [2] and retention rates in cohort studies [3], with other birth cohort studies in the UK considering their use to maintain participation rates [13]. However, to our knowledge, this is the first study to formally test the effect of offering an opt-in process versus an opt-out process for receiving those incentives. This is important for ongoing studies which routinely offer incentives and offers a potential way to reduce expenditure when administering data collection activities. Whilst these savings are likely to be small over time this could built up into a significant amount. For example, in ALSPAC, if we continue to provide an annual questionnaire, we could save almost £200,000 (Table 5) over the next 10 years by switching to an opt-in rather than an opt-out strategy. We will evaluate the ongoing impact of opt-in over time when we have completed a number of additional data sweeps. It is important to note that financial incentives are one of a range of engagement techniques and here we have not explored their use for re-engaging with participants that might be lost to the study.
Strengths of this study include the longitudinal nature of the wider study and its size. ALSPAC has a core set of engaged participants and over 10,000 responded to the questionnaires that were randomised. We included two generations of participants enabling evidence to be gathered in two distinct age groups and we were able to investigate associations with baseline characteristics. Limitations include the possible timing of these questionnaires; although they were sent out well before the COVID-19 pandemic, our reminder process did crossover into the first lockdown in the UK. G0. However, 94% of G1 respondents and 82% of G0 partners completed their questionnaires before lockdown. 60% of G0 mothers completed before the pandemic, however this group had the lowest proportion of respondents receiving vouchers and it is unlikely that the lockdown will have affected their choice to opt-in or opt-out. We have not seen any evidence that changing our protocol to opt-in has affected overall response rates: Table 5 shows that in both generations, response rates were actually higher in the subsequent questionnaire sweep but are then subject to what we would expect given natural attrition. A further limitation is that our results may not be generalisable to a) other types of studies such as cross-sectional studies, household panels or b) those using different monetary structures, types of incentives, such as cash or amounts (e.g. rewards >£10). Finally, there is potential that the differences in wording to describe the voucher used in the opt-in (“a thankyou voucher for completing your questionnaire”) versus the opt-out (“your thank you voucher”) may have affected whether respondents chose to receive their vouchers. Given participants have been with the study for many years and would be used to receiving a voucher as a thank you, it is unlikely that a change in wording would have an effect but it cannot be ruled out.
Conclusions
The introduction of an opt-in option for receiving a financial incentive has reduced the number of questionnaire respondents who received a voucher in both generations of the ALSPAC study compared to an opt-out option. We recommend that studies which rely on volunteers or members of the public undertaking acts of altruism, but may offer financial incentives may want to consider changing their protocols to opt-in rather than opt-out in order to make small savings.
Acknowledgments
We are extremely grateful to all the families who took part in this study, the midwives for their help in recruiting them, and the whole ALSPAC team, which includes interviewers, computer and laboratory technicians, clerical workers, research scientists, volunteers, managers, receptionists and nurses.
References
- 1. Teague S, et al. Retention strategies in longitudinal cohort studies: a systematic review and meta-analysis. BMC Medical Research Methodology 2018. pmid:30477443
- 2. Abdelazeem B, Abbas KS, Amin MA, El-Shahat NA, Malik B, et al. (2022) The effectiveness of incentives for research participation: A systematic review and meta-analysis of randomized controlled trials. PLOS ONE 17(4): e0267534. pmid:35452488
- 3. Booker CL, Harding S, Benzeval M. A systematic review of the effect of retention methods in population-based cohort studies. BMC Pub Health 2011; 11: 249. pmid:21504610
- 4.
Sunstein R, Thaler RH, Thaler R, Nudge: Improving decisions about health, wealth and happiness. 2008: Yale University Press.
- 5. Johnson EJ and Goldstein D. Do defaults save live? Science 2003; 302: 1338–1339.
- 6. Jackimowicz JM, Duncan S, Weber EU, Johnson EJ. When and why defaults influence decisions: a meta-analysis of default effects. Behavioural Public Policy 2019; 3: 159–186.
- 7. Boyd A, Golding J, Macleod J, Lawlor DA, Fraser A, Henderson J, et al. Cohort Profile: The ‘Children of the 90s’; the index offspring of The Avon Longitudinal Study of Parents and Children (ALSPAC). International Journal of Epidemiology 2013; 42: 111–127.
- 8. Fraser A, Macdonald-Wallis C, Tilling K, Boyd A, Golding J, Davey Smith G, et al. Cohort Profile: The Avon Longitudinal Study of Parents and Children: ALSPAC mothers cohort. International Journal of Epidemiology 2013; 42:97–110. pmid:22507742
- 9. Northstone K, Lewcock M, Groom A, Boyd A, Macleod J, Timpson NJ, et al. The Avon Longitudinal Study of Parents and Children (ALSPAC): an updated on the enrolled sample of index children in 2019. Wellcome Open research 2019; 4:51 () pmid:31020050
- 10. Major-Smith D, Heron J, Fraser A et al. The Avon Longitudinal Study of Parents and Children (ALSPAC): a 2022 update on the enrolled sample of mothers and the associated baseline data. Wellcome Open research 2022, 7:283 () pmid:37664415
- 11. Northstone K, Ben Shlomo Y, Teyhan A et al. The Avon Longitudinal Study of Parents and children ALSPAC G0 Partners: A cohort profile [version 1; peer review: awaiting peer review]. Wellcome Open Res 2023, 8:37 ().
- 12. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG, Research electronic data capture (REDCap)–A metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009 Apr;42(2):377–81. pmid:18929686
- 13.
Calderwood L, Brown M, Goodman A, Fitzsomons E, Peycheva D, Ploubidis GB, et al. Survey methods research on mode and incentives. Centre for Longitudinal Studies. 2022. Accessed: https://www.ukri.org/wp-content/uploads/2022/03/ESRC-220311-SurveyMethodsResearchModeIncentives-191009.pdf