Differences between offline and online panel studies

While researchers have been collecting offline panel data for decades, it is only in recent years that the collection of high-frequency online longitudinal data has begun, made possible primarily by the high penetration of the Internet and recent advances in Internet technology. As a consequence, the majority of prior studies analyzing nonresponse and attrition outcomes have focused on offline longitudinal studies [33–35], with limited comparable research using nonresponse data from probability-based online longitudinal studies [12,29,36].

Offline longitudinal studies differ from their online counterparts in many design aspects, and these differences can have different implications for nonresponse and attrition outcomes and the underlying mechanisms influencing them. Table 1 highlights some of the key differences. In the following discussion, we examine these differences in detail.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Key differences between probability-based online panel vs. offline panel studies.

https://doi.org/10.1371/journal.pone.0332902.t001

The mode of data collection is the first design aspect that distinguishes online panels from offline panel studies. In most online panel studies, the sole mode of data collection used is the Internet, although some Internet panels offer alternative modes in addition to the online option. For instance, the AmeriSpeak Panel in the U.S. uses a mixed-mode design by providing a telephone option to panelists without Internet access and to panelists with Internet access who are unwilling to provide an email address [37]. Similarly, the GESIS panel in Germany provides paper-and-pencil questionnaires sent via postal mail to participants unable or unwilling to complete surveys online [38]. Providing a second mode option to online panel respondents undermines the cost and efficiency advantage of the online mode but has the potential to improve sample representation.

A few offline panel studies have switched to or begun experimenting with push-to-web sequential mixed modes [39–48]. This shift from a single mode to a mixed-mode design is mainly necessitated by the need to address the rising costs associated with face-to-face and telephone interviews and to take advantage of increasing Internet penetration. In this mixed-mode design, offline panel respondents are encouraged to complete interviews online first, followed by face-to-face or telephone interviews if necessary. This approach substantially reduces costs through decreased fieldwork and fewer field interviews. Couper and McGonagle (2019) [49] provide a comprehensive review of recent developments in the use of web surveys in mixed-mode data collection for conventional offline longitudinal studies.

The two study approaches also differ in terms of data collection frequency. In online panel studies, data collection can be very frequent. Participants might be asked to provide data daily (as in 7-day wellbeing and diary surveys), weekly, monthly, or bimonthly. Participants in Ecological Momentary Assessment (EMA) studies may be interviewed multiple times a day over several days [50,51]. The high frequency is facilitated by the convenience and accessibility of online surveys. In EMA studies, surveys may also be delivered via push notifications through smartphone apps, where timely response is critical and email alone may not suffice. In contrast, offline panel studies generally have less frequent data collection schedules. Data might be collected annually, biannually, or even less frequently. The differences in data collection frequency have significant implications for the type of data gathered and the responsiveness of a study to new developments such as pandemics and financial crises.

Online panel interviews are typically shorter, ranging from 15 to 30 minutes. This brevity is designed to maintain participant engagement and reduce the likelihood of break-off and survey fatigue. In contrast, aided by the presence of interviewers and long between-wave gaps, offline panel interviews tend to be much longer, lasting between 80 and 120 minutes.

Online and offline studies use different questionnaire designs across survey waves. Most online panels field both longitudinal and cross-sectional surveys, with survey topics and questionnaires varying significantly across different waves. This flexibility allows online studies to adapt to changing research needs and emerging topics of interest such as the COVID-19 pandemic. In contrast, offline studies primarily field longitudinal questionnaires with mostly fixed topics across waves, providing consistent data over time. Occasionally, offline panel studies may field supplemental questionnaires between waves to address specific research questions or capture additional data points.

Different correlates of attrition and response mechanisms in offline vs. online panel studies

The aforementioned design differences can differentially impact nonresponse and attrition outcomes, and the correlates influencing these outcomes may vary between the two study approaches. In any follow-up survey in an offline study, interviewers must first locate respondents and then establish contact with them, conditional on being located. Once contact is made, interviewers attempt to gain cooperation from respondents. Failure at any of these three steps could result in nonresponse. As such, most offline studies have modelled nonresponse and attrition outcomes as a function of whether: sample respondents moved between survey waves, contacts with sample respondents were established conditional on respondents being located, and sample respondents cooperated when approached by interviewers, conditional on contact being made. Non-location occurs when participants move and cannot be reached, while noncontact refers to the difficulty in reaching participants physically despite repeated attempts. These three processes can be modeled separately [35,52–54] or cumulatively without making any distinction among the three processes [28,55].

Given the three aforementioned processes through which nonresponse could occur in offline studies, many prior studies have focused on identifying correlates that predict non-location, non-contact, or the cumulative final outcome [33–35,52–54,56]. Household-, individual-, and interviewer-level variables that have been found to predict these outcomes include, among others, housing arrangement (e.g., renting vs. owning), housing characteristics (e.g., gated entry, single home, communal housing), housing tenure (i.e., years at current residence as proxy for stability), household size, number of children, employment status (e.g., retired, currently working), marital status (as a proxy for geographical stability), changes in events between waves such as changes in employment and household composition [57], level of interviewer experience, interviewer training, and use of the same interviewers across survey waves. However, these household-, individual-, and interviewer-level variables primarily affect attrition through contactability and securing initial cooperation with respondents, making them less relevant for predicting nonresponse patterns in online panel contexts, where surveys are delivered via email and self-administered.

The use of different data collection modes between the two study approaches can impose varying levels of survey burden, which can, in turn, lead to different nonresponse patterns. For instance, since most offline panels use face-to-face and/or telephone modes administered by interviewers, the cognitive effort required to answer difficult or unclear questions may be lower for offline panel respondents as interviewers can clarify questions. Conversely, when answering sensitive questions, the presence of interviewers can create psychological stress for offline panel respondents [58,59].

In this study, we analyze the independent effects of non-demographic variables – respondent personality traits, panel tenure, and survey topics and context – that are likely to be important for predicting nonresponse patterns in high-frequency online longitudinal studies, as detailed below.

Personality traits as predictors of nonresponse

Online surveys are typically delivered via email and self-administered, which reduces the impact of traditional predictors such as contactability and interviewer interactions. We may hypothesize that participants might decline to continue if the frequency of surveys is considered too high, or due to a lack of interest, or disruption in their survey-taking routine. Based on this expectation, we hypothesize that personality traits are important predictors of a respondent’s likelihood to participate consistently in online panel surveys and may have more explanatory power than conventional demographic and household variables commonly found in offline panel studies.

Building on existing theoretical foundations such as the Social Isolation Hypothesis [60], Saßenroth (2013) [61] emphasizes the predictive power of personality traits over traditional socio-demographic factors, particularly highlighting how subjective experiences influence survey participation decisions. This perspective underscores the role of personality traits as critical determinants of nonresponse patterns in high-frequency online longitudinal studies.

The Big Five personality traits – conscientiousness, openness, agreeableness, neuroticism, and extraversion – are widely used in psychological research and have been found to be stable throughout a person’s lifespan and across cultures and languages [62–65]. In this study, they were measured using a 44-item standardized self-report questionnaire administered to respondents [65–67]. Respondents with certain personality profiles may be more inclined to engage regularly and provide complete data, while others may be prone to drop out or respond sporadically. As such, there is a need to look beyond conventional demographic information in analyzing nonresponse patterns in high-frequency online longitudinal surveys.

This argument is supported by drawing a parallel to the literature on medication adherence, where certain personality traits, particularly conscientiousness, have been shown to play a significant role. The conscientiousness trait is defined as “socially prescribed impulse control that facilitates task- and goal-directed behavior, such as thinking before acting, delaying gratification, following norms and rules, and planning, organizing, and prioritizing tasks” [68]. Individuals high in conscientiousness exhibit qualities such as diligence, perseverance, tenacity, orderliness, and industriousness. These traits are crucial for adhering to medication schedules, as they involve remembering to take medications, planning around dosage times, and maintaining a consistent routine. The effect of conscientiousness in predicting medication adherence has been found to be robust across various disease conditions, population subgroups, and cultural settings [69–75].

Similarly, this characteristic is likely relevant for survey completion in high-frequency online longitudinal studies. Completing online surveys regularly requires respondents to pay attention to survey invitations sent via emails, remember due dates, and respond to survey reminders. Conscientious individuals are likely to excel in these areas due to their propensity for planning, organizing, and prioritizing tasks. Therefore, just as conscientiousness is a strong predictor of medication adherence, it should also serve as a strong predictor of nonresponse patterns characterized by low wave nonresponses in high-frequency online longitudinal studies.

In contrast to conscientiousness, a higher score in openness may reflect individuals’ tendency to embrace new ideas and experiences, but this quality may not translate into a strong commitment to complete high-frequency and longitudinal surveys. More agreeable individuals are trusting, altruistic, sympathetic, and cooperative, which may predict participation in a new one-time study but may not be as strong a predictor of sustained participation in a long-term, continuous study, such as a high-frequency longitudinal survey. Neurotic individuals, being susceptible to anxiety and depression, could be more prone to panel fatigue or study dropout. Extraversion is mainly associated with being talkative, sociable, energetic, and friendly, but high extraversion may not translate to long-term commitment needed for participation in an online longitudinal study. Consequently, we expect openness, agreeableness, neuroticism, and extraversion to be positively associated with higher wave nonresponses in a high-frequency online longitudinal study.

Several studies in the economic and psychological literature have extensively documented personality traits as independent predictors (net of other important factors) and causes of significant life outcomes, including educational attainment, employment outcomes, wealth, mental and physical health status, well-being, marriage, divorce, criminal activity, volunteering, and productivity, among others [76–82]. If personality traits are found to predict nonresponse patterns in high-frequency online longitudinal studies, even after accounting for conventional respondent- and household-level demographic variables, this could have practical implications for how nonresponse adjustment weights are calculated.

In offline panel studies, a few investigations have found that personality traits independently predicted survey completion. Satherley et al. (2015) [83] analyzed the first four waves of response data from the New Zealand Attitudes and Values Study, an offline longitudinal national panel where respondents were interviewed annually using self-administered paper questionnaires, with a web option provided in waves 3 and 4. They found that conscientiousness was positively associated with consistent participation in the panel study. Similarly, Hansson et al. (2018) [26] analyzed response data from the first three waves of the Swedish longitudinal population-based Health, Ageing, and Retirement Transitions in Sweden (HEARTS) study, an offline panel study with annual follow-ups using both paper-pencil questionnaires and online modes. They found that higher extraversion and neuroticism, and lower agreeableness, independently predicted attrition. However, given that online and offline panel studies differ in many design aspects (see Table 1), it is unclear to what extent these findings can be generalized to the context of high-frequency online longitudinal data collection.

Within online panel contexts, Cheng et al. (2020) [29] analyzed wave-to-wave response data from panel surveys fielded in the UAS and found that conscientiousness and openness to experience predicted the incidence of unit nonresponse in subsequent survey waves, even after controlling for cognitive ability and demographic characteristics. Additionally, Lugtig (2014) [18] analyzed response data from surveys conducted in the LISS panel in the Netherlands and found that respondents who dropped out were less likely to be conscientious and more likely to be extraverted and agreeable.

Recent studies in the public opinion literature have begun to explore the role of personality in survey participation. Hibbing et al. (2019) [84] show that traits like openness and conscientiousness are linked to response styles that may bias survey estimates. Valentino et al. (2020) [85] find that openness and extraversion are negatively associated with participation in online surveys compared to face-to-face modes, suggesting potential personality-driven selection biases. While these studies rely on cross-sectional or dual-mode data, our study extends this work by using a richer 44-item personality battery and panel data to examine how personality traits predict actual nonresponse patterns over time in high-frequency online longitudinal surveys.

Panel tenure and nonresponse patterns

Panel tenure, defined as the length of time a respondent has been part of an online panel, is a crucial factor in understanding and predicting nonresponse patterns in online data collection contexts. Over time, respondents who have been part of a panel for longer periods may develop a stronger sense of commitment and familiarity with the survey process, which can lead to higher response rates and lower attrition rates. Likely, attrition is selective, whereby individuals who are more inclined to participate may be more likely to remain in the panel. By including personality traits—particularly conscientiousness—as predictors in our analysis, we are able to partially account for the heterogeneity in the propensity to respond to surveys, acknowledging that we cannot cleanly separate the roles of selectivity and panel tenure. In what follows, panel tenure therefore is taken to reflect both possible habituation in panel response and the difference in unobserved characteristics between respondents with different tenure lengths.

New panel respondents may not have yet developed the routine of regularly taking surveys in an online environment. Previous studies have shown that prior survey experiences are predictive of cooperation in subsequent surveys, indicating that the early survey experiences of fresh respondents are crucial for their continued participation [28,86]. Positive initial experiences can foster a sense of engagement and commitment, leading to longer panel tenure and reducing the likelihood of nonresponse and attrition.

Most importantly, studies have found that individuals who drop out during the first membership year have distinct demographic, psychological, and psychosocial characteristics compared to those who leave in subsequent years [22,26].

Fig 1 illustrates how attrition decreases over time. It shows the cumulative attrition for the first of respondents recruited into the UAS in 2014. One observes that in the first year, 13% of the new respondents dropped out. In the second year, an additional 8% dropped out, and an additional 5% in the third year. By the end of the observation period annual attrition had fallen to just a few percent per year. Batches that were recruited in later years show similar patterns, although attrition levels have gone up over time, in line with general experience in panel studies.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Attrition over time of the first UAS batch recruited in 2014.

https://doi.org/10.1371/journal.pone.0332902.g001

The pattern suggests that the most productive effort in maximizing retention may be early on in a respondent’s tenure. By analyzing panel tenure effects, researchers can identify factors that contribute to quick dropout and apply these insights to newer panel members. Understanding the impact of panel tenure also can help inform the design of nonresponse adjustment weights. In our panel, respondents are invited to participate in a sequence of 22 core surveys, administered biennially, and are also invited to topical surveys such as the COVID-19 series. Although there may be slight variation in the number of ad hoc survey invitations, the overall exposure to survey requests is broadly similar across respondents. For this reason, we consider panel tenure (i.e., time in the panel) to be a reasonable proxy for survey experience.

Survey topics and context effects

Leverage-salience theory, which incorporates both survey features and respondent characteristics, suggests that different survey features (e.g., survey topics) have varying effects (“leverages”) on participation decisions based on respondents’ socioeconomic characteristics, lifestyles, and preferences [87]. The relevance and interest of survey topics in influencing survey participation decisions, as predicted by leverage-salience theory, have mainly been examined in cross-sectional studies [88,89].

Understanding the role of survey topics and context in online longitudinal studies is important for several reasons. First, longitudinal data collection requires sustained participation over time, and varying survey topics may differentially impact respondents’ willingness to continue participating. Second, as online panels become an increasingly popular method for data collection, identifying factors that influence long-term engagement and reducing nonresponse is essential for maintaining data quality and representativeness. The COVID-19 pandemic has provided a unique opportunity to examine how a highly topical and relevant issue may influence nonresponse patterns.

In this study, we analyze two novel datasets generated from two recently completed online longitudinal studies. These studies primarily differ in their survey topics and contextual factors. Specifically, one study focused on well-being topics both before and during the COVID-19 pandemic, while the other solely addressed COVID-19-related topics and was fielded when COVID-19 was declared a public health emergency in March 2020. The importance and relevance of the COVID-19 study were clearly emphasized to potential participants in the consent survey by including the following text:

“We are launching a new survey series to help us track how U.S. residents are coping with the coronavirus pandemic. Participants will be asked to fill out a survey every two weeks. We are interested in learning how life changes or stays the same over time for U.S. residents and their family and friends. For that reason, we hope you will join the project so that we have input from every part of the country and from every walk of life. The survey will focus on what is happening in your life, and what you may be doing the same or differently now. We want to learn about the impacts of the pandemic on your family, your job or income, and how you are coping with the changes that have been and continue to be made to help contain the virus.”

We leverage the timing and relevance of the COVID-19 study, along with the disproportionate effects the pandemic had, to conduct a comparative analysis of survey nonresponse patterns and their correlates between the two studies. On the one hand, we expect the COVID-19 survey to attract a large number of participants, as it was very topical during the pandemic. On the other hand, nonresponse rates in the COVID-19 survey may be higher among subgroups of respondents whose lives and livelihoods were disproportionately affected by the pandemic.

While we did not begin with strong directional hypotheses regarding potential moderating effects of panel topic, the COVID-19 context provided a unique opportunity to explore whether broader societal conditions and topic salience could moderate the relationship between individual characteristics and survey participation. Our analyses of topic effects are primarily exploratory, but the observed differences between the two studies suggest that contextual factors may meaningfully influence nonresponse patterns.

In summary, this study makes several contributions to the emerging literature on nonresponse and attrition in high-frequency online longitudinal data collections. First, it contributes to our understanding of how design differences between offline and online panels differentially influence nonresponse, attrition outcomes, and underlying mechanisms. Second, it offers novel insights into the critical role of respondent personality traits, particularly conscientiousness, in predicting nonresponse patterns in these contexts, even when accounting for other predictors, which could have implications for the construction of nonresponse adjustment weights. Third, by leveraging data from two recent and unique datasets, the study provides new evidence on the independent effect of panel tenure and the potential moderating effect of survey topics on nonresponse patterns.

Data and study population

Data are from two recently completed online longitudinal studies conducted in the Understanding America Study (UAS). Established at the University of Southern California (USC) in 2014, the UAS is a nationally representative, probability-based internet panel of US-households of approximately 15,000 respondents aged 18 and above. Respondents answer surveys on a computer, tablet, or smart phone, wherever they are and whenever they wish to participate. Surveys are conducted in English and Spanish. Panel members are recruited through address-based sampling. Prior access to internet is not a pre-requisite to be in the panel; respondents without prior internet access are provided with a computer tablet and broadband internet subscription. Respondents answer surveys once or twice a month covering various substantive topics. Partly as a result of this, the UAS comprises a vast amount of background information on its respondents, including extensive measures of physical and mental health, income, labor force participation, cognitive functioning, and demographics. For more information about the UAS cohort profile, see Kapteyn et al. (2024) [12].

UAS recruitment procedures follow the Tailored Design Method [90], starting with a postcard, next a short recruiting survey with $5 included and $15 promised for return of the recruiting questionnaire. At the end of the questionnaire, respondents are asked for their interest in participating in a longer run study. If interested, but without Internet, they are offered an Internet-enabled tablet. The process includes several reminders and an offer of additional incentives for consenting respondents to start the first Internet survey, if they have not done so within a given time period. Panel members are paid $20 for a thirty-minute survey. The UAS has instituted a “Sleeper Protocol”, which involves several follow-ups with panel members who have not responded to a survey invitation during a specified number of months. The UAS website (https://uasdata.usc.edu/index.php) provides complete information about UAS methodologies, including response and retention rates for every recruitment batch.

Understanding coronavirus in America study (UCA)

The first data source is from the Understanding Coronavirus in America (UCA) Study [5]. UCA commenced on March 10, 2020, immediately following the World Health Organization’s declaration of COVID-19 as a global pandemic. All active members of the UAS panel were invited to participate in the UCA. About 7,000 UAS participants signed up for the study. The number of active UAS panel members was approximately 10,000 when the UCA study was launched in early 2020. The number has since grown to 15,000. UCA participants received a survey every two weeks. The bi-weekly surveys elicited information on a broad array of socio-economic, behavioral, and health outcomes associated with the COVID-19 pandemic. A total of 28 biweekly survey waves were conducted, spanning the period March 2020 to July 2021. While the UCA continuously added new participants to the study after the first wave, our analysis specifically focuses on the 5,891 participants recruited in the first wave. This approach ensures that we possess complete response data for the entire duration of the study, from the initial wave to the study’s conclusion across all 28 waves. The UCA website (https://uasdata.usc.edu/page/Covid-19±Home) provides additional information about the study, including download information on available micro data files and documentation.

The UAS monthly events panel surveys

The second data source is from the UAS monthly events surveys conducted among UAS respondents aged 50 and above. The data collection began in July 2019 and concluded in September 2023, spanning 52 waves. Each individual survey was administered at the beginning of the month, with respondents reflecting on their experiences in the preceding calendar month. These surveys collected information about various life events, including personal and health-related experiences such as health shocks, financial events, as well as personal encounters with sickness or the death of close relatives or friends during the previous month, and solicited self-assessments of their health, life satisfaction, pain levels, hours worked, earnings, employment changes, and medical expenditures during that period. Although the study continuously added new participants after the first wave, our analysis focuses on 3,597 participants recruited in Wave 1. This is to ensure that we have complete response data for the full period of the study across 52 survey waves. More details about the monthly events study including download information on available micro data files and documentation, are presented at the study website (https://uasdata.usc.edu/page/UAS±Monthly±Surveys). All data used in this study were accessed on 15/12/2023.

We acknowledge that this 50 + age restriction overlaps with panel topic, which limits our ability to disentangle topic effects from age-related sample composition. However, to address this concern, we conducted a sensitivity analysis by restricting the COVID-19 sample to respondents aged 50 and above. This allowed for a more balanced age comparison between the two samples. The results—presented in the Supporting Information section in S7 Table —indicate that several previously significant effects, especially those related to personality traits, diminished or disappeared in the age-restricted sample. As such, comparisons between the two studies should be interpreted with caution. But the robustness check provides additional clarity about the role of age composition. Additionally, both studies were implemented within the same probability-based online panel, which enhances comparability and internal consistency across the analyses.

Analytical approaches

Predicted posterior probabilities and latent class membership

After fitting the LCA model with the chosen number of classes, we proceed to generate predicted posterior probabilities for each class for every respondent and assign class membership to the class with the highest predicted probability for each respondent. We present summary statistics of predicted posterior probabilities grouped by predicted class membership to initially assess the accuracy of class classification. Subsequently, we plot observed response probabilities by class membership across survey waves. Alternatively, we can also plot the predicted posterior probabilities. Both approaches however yield similar graphical representations of nonresponse patterns across survey waves.

Predictor variables

Panel tenure variable

Official membership in the UAS begins upon completion of the first demographic survey, also referred to as the My Household survey. To calculate panel tenure, we measured the duration in days between the date the Wave 1 survey was conducted in either of the two longitudinal studies and the date a respondent became an official member of the UAS. For the monthly events and UCA studies, the first waves were fielded on June 3, 2019, and April 1, 2020, respectively. We then grouped respondents into two categories: those with panel tenure less than 1 year (fresh or new panelists) and those with panel tenure of 1 year or more (mature or experienced panelists). This classification enables a comparative analysis of respondent engagement based on their tenure within the panel. The one-year cutoff is meaningful and practical in terms of panel management practices and implementation of targeted interventions.

Other covariates

Other covariates used in the regression analyses include the following: (1) Hispanic ethnicity (yes, no); (2) Race (white only, black only, others); (3) Gender (male, female); (4) Age groups (18–44, 45–64, 65 and above in the UCA study; 50–64, 65 and above in the monthly event study); (5) Highest educational attainment (up to GED or high school, some college, college and above); (6) Household income (below $50k, $50-$75K, $75K and above); (7) Employment status (currently working, currently not working); (8) Household size (1, 2, 3 and above); (9) Self-reported health (rated on a scale from 1 = excellent to 5 = poor).

Several existing theories justify the use of demographic factors as predictors of survey participation. For example, the Opportunity Cost Hypothesis [100,101] explains that individuals weigh the costs and benefits of survey participation, making socio-demographic factors important in shaping their participation decisions. The Social Exchange Theory [90,102] highlights that the norm of reciprocity and trust in exchanges—often influenced by demographic characteristics such as education or socio-economic status—enhances willingness to participate. Additionally, demographic variables, such as education, can serve as proxies for cognitive burden, where individuals with higher educational attainment or higher socioeconomic status (SES) may find survey tasks less burdensome and are thus more likely to complete them [103–105]. A recent study by Earp et al. (2022) [106] provides additional empirical evidence showing that demographic variables serve as good proxies for respondents’ subjective burden levels that predict survey responses in a national survey.

UAS updates respondents’ demographic and household information every three months. All demographic and household information used in this analysis are from the first wave of data collection in the two respective studies. Data on health outcomes are from the latest available UAS wave as this information is collected every two years on the same respondents.

Multinomial logistic regression

We estimate MLogit models to predict class membership for each of the two studies. For each multinomial logistic regression estimated, we report average marginal effects (AMEs), 95% CIs, and the level of statistical significance. AMEs represent the average change in the predicted probability of belonging to an attrition class associated with a one-unit change in a predictor variable, holding all other variables constant. All reported p values were adjusted for multiple hypothesis tests using Holm’s method [107]. To account for uncertainty in class membership classification in the latent class models, one could use probabilities predicting class membership for the chosen class as weights in the regression analysis, which effectively assigns higher weights to chosen classes predicted with higher probabilities. We compared both unweighted and weighted regression results and they are similar. This is because all chosen classes in the latent class models were predicted with high probabilities and accuracy, as discussed in the Results section. We present and discuss unweighted regression results in the main text but include weighted results, accounting for uncertainties in classification, in S1 and S2 Tables in the Supporting Information Section for completeness. We then explore how the underlying mechanisms influencing nonresponse patterns, as well as the individual- and household-level variables that predict them, may differ between the two approaches.

Additionally, for each survey wave, the UAS provides survey weights that correct for unequal probabilities of sampling UAS members and align the sample with the U.S. adult population in terms of gender, race/ethnicity, age, education, and geographic location (Census regions) [108]. We decided not to use the UAS survey weights in our main analyses because the focus of this study is to understand attrition dynamics within the panel—specifically, how nonresponse evolves from the original sample over time. This would appear to be most relevant for other similar longitudinal surveys. Using survey weights would make the results most relevant for the unlikely case where one is able to recruit a sample with identical inclusion probabilities, i.e., where the likelihood that an invited panel member agrees to join the panel is identical across individuals. However, for most estimated average marginal effects, we found that the unweighted and weighted regression results are both quantitatively and qualitatively similar. For comparison and completeness, we present the weighted regression results using UAS survey weights in the Supporting Information section (see S8 and S9 Tables).

Statistical packages

All analyses were conducted using Mplus for Windows Version 8.10. Mplus was mainly utilized for estimating latent class models, generating posterior response probabilities, and classifying class membership [109]. Additionally, Stata for Windows Version 18.0 was used primarily for data management, estimating Multinomial Logistic Regression models for predicting class membership, and generating graphs [110].

Consent and IRB

Oversight of UAS data collection and dissemination was ceded by University of Southern California IRB (UP-14–00148) to BRANY (Biomedical Research Alliance of New York) IRB (22-030-1044). Participants gave informed consent to participate in the UAS before taking part.

Sample summary statistics

Table 2 presents summary statistics for the two samples. We provide mean values for the Big-5 personality traits and self-reported health, and percentages for categorical variables.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Sample summary statistics by study sample.

https://doi.org/10.1371/journal.pone.0332902.t002

With respect to Big-5 personality traits, there are no discernible differences between the two studies. Relative to the monthly events study, the UCA study attracted a significantly higher percentage of fresh panelists (24.26% vs. 13.23% in the monthly events study) and Hispanics (15.99% vs. 6.14% in the monthly event sample). These differences can be explained by the growth of the UAS and the different time periods covered by the two studies. In the period before March 2020, the sample size of the UAS grew considerably, resulting in more panel members with shorter tenure at the start of the UCA study.

Racial and ethnic compositions are comparable between the two studies. Participation among females was higher in the UCA study compared to the monthly events survey (57.82% vs. 52.32% in the monthly events study). Both studies are comparable in terms of highest educational attainment and household income. The UCA sample had a higher percentage of working individuals (56.13% vs. 41.90% in the monthly events sample), reflecting the age differences between the two studies. We observe differences in household composition between the two samples, with 2-person households accounting for half of the monthly events sample and 3-person households and above making up approximately half of the UCA sample; this difference mainly reflects age differences between the two samples. The mean values for self-reported health are similar between the two samples.

Latent class model selection

In Table 3, model fit statistics are presented by the number of latent classes separately for the two study samples. Across both samples, the log-likelihood values, AIC, and BIC consistently improve with an increase in the number of classes. However, in the monthly events study sample, the LMRT and BLRT statistics yield conflicting results. The LMRT suggests that a model with 4 classes best fits the data, while the BLRT indicates that a model with 8 classes performs better. Entropy values exceed 0.8 for all latent class models in both samples, indicating high accuracy in class membership assignment. Taken together, these fit statistics suggest that the models with more classes fit the data better. However, considering the substantive interpretability and practicality of implementing targeted interventions, we select latent class models with 5 classes in the monthly events sample and 6 classes in the UCA sample as our final models, highlighted in bold text.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Model fit statistics by number of response pattern classes.

https://doi.org/10.1371/journal.pone.0332902.t003

Additionally, for sensitivity analysis and completeness, we provide in S1, S2, S3, and S4 Figs in the Supporting Information Section showing observed attrition patterns and in S3, S4, S5, S6 Tables in the Supporting Information Section presenting average marginal effects from multinomial logistic regression results predicting class membership: for the latent class models that are one class smaller and one class larger than the chosen number of latent classes for each of the two study samples, leading to similar substantive conclusions.

Class membership prediction

Table 4 presents the mean posterior probabilities categorized by predicted class membership. Rows represent class membership, and columns represent the corresponding predicted probabilities. For instance, for the first row in the monthly events data, conditional on being in class 1, the predicted probability of being in classes 1,2,3,4, and 5 are 0.958, 0.000, 0.002, 0.013, and 0.027, respectively. Diagonal values close to 1, and off-diagonal values close to 0, indicate a high level of accuracy in classification. For instance, the mean predicted probabilities for being in classes 1, 2, 3, 4, and 5 are 0.958, 0.988, 0.979, 0.957, and 0.991, respectively, in the monthly events study sample. The mean predicted posterior probabilities across all assigned classes are 0.983 (SD = 0.071, Min = 0.48, Max = 1) for the monthly events study sample and 0.954 (SD = 0.107, Min = 0.352, Max = 1) for the UCA study sample. Overall, the results indicate that classes are predicted with high accuracy, which is consistent with the high entropy values presented in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Classification matrix.

https://doi.org/10.1371/journal.pone.0332902.t004

Response patterns in the monthly events study

Fig 2 shows observed posterior response probabilities by class membership across survey waves for the monthly events study. Notably, there were sudden drops in response rates in waves 4 and 6 that were mainly due to a glitch in the case management system, which resulted in failure to send out reminders. This issue was addressed in subsequent waves.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Posterior response probabilities of the 5-class LCA model for the monthly events panel study (n = 3,597).

https://doi.org/10.1371/journal.pone.0332902.g002

The first group of respondents consists of individuals with response propensities close to 1 across all 52 monthly survey waves. We refer to this group as the “Stayers” group, which comprises approximately 54% of the sample. Participants in this group completed almost all of the 52 monthly survey waves. Similar to the “Stayers” group, the “Good Responders” group, constituting about 11% of the sample, remained until the end of the study but missed a few surveys, with an average response rate of approximately 0.8 across all survey waves.

The “Mid-wave Attritors” group, constituting 8% of the sample, exhibits a response pattern similar to the “Stayers” group until Wave 15, after which nonresponse gradually increased in the Mid-waves, and they did not return. Similar to the “Mid-wave Attritors” group are the “Wave-10 Attritors” group, who make up a significant 12% of the sample and began to leave earlier than the “Mid-wave Attritors” in Wave 10, dropping linearly from an average response rate of about 85% in Wave 8 to about 5% in Wave 31 and then taper off until the end of the study period.

At the lower end of the nonresponse spectrum are a group of respondents with a response probability close to zero right from the beginning. We refer to this group as the “Non-responders” group, making up about 15% of the sample.

In S1 and S2 Figs in the Supporting Information Section, we present nonresponse patterns for the two latent class models: one with a reduced class (i.e., 4 classes) and one with an additional class (i.e., 6 classes) compared to the selected class for comparison and completeness. These nonresponse patterns demonstrate similarities with the patterns in the chosen class, particularly among the “Stayers” and “Non-responders” groups.

Response patterns in the UCA study

Fig 3 presents observed posterior response probabilities across survey waves by class membership for the UCA study.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Posterior response probabilities of the 6-class LCA model for the COVID-19 panel study (n = 5,891).

https://doi.org/10.1371/journal.pone.0332902.g003

There are both similarities and differences in nonresponse patterns and their sample compositions between the two studies. For instance, both studies have a “Stayers” group but their share in the UCA sample is larger than they are in the monthly events sample (62% vs. 54% in the monthly events study). The “Mid-wave Attritors” group in the UCA sample differs from the “Mid-wave Attritors” group in the monthly events sample. Mid-wave leaving respondents in the UCA study responded to all surveys until mid-wave (i.e., Wave 14) and then began to drop out slowly until wave 23, reaching a response rate of about 0.8, with a steep decline afterward until the end of the study period. Their sample share, however, is the same in both samples, accounting for 8% of respondents. The response pattern for “Good Responders” and their sample share (11%) is similar to a large extent in both studies.

The “Gradual Attritors” group, making up 8% of the UCA sample, completed the first survey with a response rate close to 1 and then began to leave gradually and linearly after Wave 1 until the end of the study, eventually dropping to a response rate of 25%. Notably, the response patterns for the “Gradual Attritors” group in the UCA study is similar to that for the “Mid-wave Attritors” group in the monthly events study but they differ in the timing where the gradual attrition begins (Wave 2 vs. Wave 23 in the monthly events study).

The “Early Attritors” group, making up 7% of the UCA study respondents, completed the first two surveys with a response rate of 0.8. Then, their response rate began to decline steeply until Wave 8, where it reached 0.2, tapering off afterward until the end of the study period. Their response pattern is similar to the Wave-10 attrition pattern found in the monthly events study but differs mainly in the timing of the steep decline (Wave 2 vs. Wave 8 in the monthly events study).

Looking at the lowest end of the nonresponse spectrum, the “Non-responders” group with a response rate close to 0 across all waves make up a smaller share of the sample in the UCA study compared to the monthly events study (4% vs. 15% in the monthly events study).

For comparison, in S3 and S4 Figs in the Supporting Information Section, we present response patterns for the latent class model with a reduced class (i.e., 5 classes) and the latent class model with an additional class (i.e., 7 classes), compared to the selected model. These patterns demonstrate similarities with the chosen latent class model.

Multinomial logistic regression results

Tables 5 and 6 present average marginal effects (AMEs) from multinomial regression models predicting membership classes characterized by various nonresponse patterns in the monthly events and UCA datasets, respectively. As previously mentioned, AMEs represent the average change in the predicted probability of belonging to an attrition class associated with a one-unit change in a given predictor, holding all other variables constant. For example, if the AME for a given predictor is 0.012 for a particular attrition class, this means that a one-unit increase in that predictor is associated with a 1.2 percentage point increase in the probability of being classified into that attrition class, holding all other variables constant. Below, we highlight key differences in the estimated coefficients between the two studies. Significance levels have been adjusted for multiple hypotheses testing for each latent class using Holm’s method. By considering every estimated coefficient as representing a separate hypothesis we are conservative. However, as we are not just interested in the overall effect of for example personality or demographics, but rather in their individual effects, this seems appropriate. The 95% confidence intervals have not been adjusted. The correction for multiple hypotheses testing renders very few of the estimated coefficients significantly different from one in Table 5 except for personality traits (the monthly events survey). In Table 6 (the UCA study), more coefficients (i.e., AMEs), including some demographic variables, differ significantly from zero. In the Discussion and Conclusion section, we provide additional contexts and detailed explanations about these differences.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Multinomial logistic regressions predicting class membership in the monthly events panel study. Reported coefficients are average marginal effects (AMEs), representing the average change in the predicted probability of each outcome category associated with a one-unit change in a given predictor, holding all other variables constant.

https://doi.org/10.1371/journal.pone.0332902.t005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Multinomial logistic regressions predicting class membership in the COVID-19 panel study. Reported coefficients are average marginal effects (AMEs), representing the average change in the predicted probability of each outcome category associated with a one-unit change in a given predictor, holding all other variables constant.

https://doi.org/10.1371/journal.pone.0332902.t006

Differences by respondent personality traits

Our results in both Tables 5 and 6 show consistent effects of conscientiousness in predicting nonresponse patterns. In the monthly events study, the regression results in Table 5 reveal that respondents with higher conscientiousness scores are less likely to be in the ”Non-responders” and “Mid-wave Attritors” classes and are more likely to be in the “Stayers” class. For instance, our results reveal that a one-point increase in Conscientiousness is associated with a 0.7 percentage point increase in the predicted probability of being classified as a Stayer in the monthly events study, holding all other variables constant. This difference is statistically significant even after adjusting for multiple hypotheses testing. Although the estimated average marginal effects of conscientiousness are negative in the regression models predicting membership in the “Non-respondents”, “Wave 10 Attritors”, “Mid-wave attritors” and “Good Responders” attrition classes in the monthly events study, they are not statistically significant after applying adjustments for multiple hypotheses tests.

Similarly, in the UCA study, results from Table 6 show that respondents with higher conscientiousness scores are more likely to be associated with the “Stayers” class: A one-point increase in conscientiousness score is associated with a 0.8 percentage point increase in the probability of being classified as a Stayer, holding all other variables constant. In contrast, a one-point increase in conscientiousness is associated with a 0.3 percentage point decrease in the predicted probability of being classified as either a “Gradual Attritor” or a “Good Responder”. These differences remain significant even after adjustments for multiple hypothesis tests. For the remaining attrition classes in the UCA study, the estimated average marginal effects for the Conscientiousness trait are negative but are not statistically significant.

It is worth mentioning that without adjustments for multiple hypotheses tests, as done in prior studies such as Lugtig (2014) [18] and Cheng et al. (2020) [29], all estimated average marginal effects for conscientiousness across the attrition classes would have reached statistical significance levels in both study samples. Despite the more stringent and conservative approach we used in handling the estimated regression coefficients, our results still emphasize the importance, robustness, and significance of conscientiousness in predicting nonresponse patterns in high-frequency online longitudinal studies, while minimizing the risk of false conclusions.

In contrast to conscientiousness, individuals with high openness scores are more likely to be associated with the attrition classes than with the “Stayers” class in both studies. This difference is more pronounced in the UCA sample than in the monthly events data. For instance, a one-point increase in the Openness score is associated with a 0.5 percentage point decrease in the probability of being classified as a “Stayer” and a 0.2 percentage point increase in the probability of being classified as an “Early Attritor” in the UCA study (p-value<0.01). Without adjustments for multiple hypothesis tests, the estimated average marginal effects for openness would have been statistically significant across all the attrition classes. However, as previously mentioned, the adjusted results provide a more robust conclusion.

Similar to openness, more extroverted, neurotic, and agreeable respondents are less likely to be associated with the “Stayer” class and more likely to fall into one of the attrition classes. For instance, in the monthly events study, a one-point increase in Extroversion, Neuroticism, and Agreeableness scores is associated with a 0.5 to 0.6 percentage point decrease in the predicted probability of being classified as a “Stayer” (p < 0.05). Comparable effect sizes for these traits were also observed in the UCA study.

Taken together, our results underscore the robustness, importance, and significance of respondent personality traits, particularly conscientiousness, in predicting nonresponse patterns in high-frequency online longitudinal studies. The relationships we found are consistent with our previously stated hypotheses, as well as with the general findings in the medication adherence literature.

Differences by panel tenure status

The results in Tables 5 and 6 show striking differences in the panel tenure effect between the two studies. The monthly events attrition regression results in Table 5 show no significant differences in average marginal effects by panel tenure across all membership classes. In contrast, the results in Table 6 reveals that, compared to fresh respondents, experienced panelists are approximately 4.4 percentage point more likely to be classified as “Non-responders” and 3.3 percentage points less likely to be classified as “Good responders” (p-value<0.05). No significant panel tenure effects were found for the remaining attrition classes in the UCA study, indicating that experienced respondents who decided to leave the UCA study did so from the beginning.

Differences among demographic subgroups

Hispanics. The regression results in Tables 5 indicate no significant difference between Hispanic and non-Hispanic individuals across all attrition groups in the monthly events study, except that Hispanic respondents are less likely to be classified as “Mid-wave Attritors.” However, the results from Table 6 reveal a different pattern: compared to non-Hispanic individuals, Hispanic respondents are 8.8 percentage points more likely to be classified as “Non-Responders” (p-value<0.01), 3.7 percentage points more likely to be classified as “Good Responders” (p-value<0.05), and 11.1 percentage points less likely to be classified as “Stayers” (p-value<0.01).

Race. In both studies, we did not find significant differences in average marginal effects by respondent race across all membership classes.

Gender. No significance differences in average marginal effects by respondent gender are found in either of the two studies.

Age Groups. In the monthly events study, age has no significant impact. Note however that the age range is restricted to 50 + , a plausible explanation for the absence of age effects. In the UCA study, we observe a large, significant, and negative association between age and nonresponse patterns, with younger participants displaying significantly higher dropout rates compared to their older counterparts across all attrition classes and these differences widen with age. For instance, compared to the 18–44 age group, individuals aged 45–64 and 65 and older are 17.7 and 22.2 percentage points more likely, respectively, to be classified as “Stayers” in the UCA study (p-value<0.01). However, in the sensitivity analysis where we restricted the UCA sample to respondents aged 50 and above, the large and significant age effects disappeared. Detailed results from the age-restricted supplementary analysis are included in the Supporting Information section (S7 Table).

Education. In the monthly events study, we find no association between education and membership of any of the attrition classes. In contrast, in the UCA study, individuals with at least a college education are 3.4 percentage points less likely to be classified as Gradual Attritors (p-value<0.05) and 8.2 percentage points more likely to be classified as Stayers, compared to those with at most a GED or high school diploma (p-value<0.01).

Household Income, Household Size, and Employment Status. Household income has no effect on nonresponse patterns in either study after controlling for other factors. Household size seems to better predict nonresponse patterns in the UCA study than in the monthly events study. Notably, in the UCA study, respondents from households with at least three people (relative to those in single-person households) are approximately 6 percentage points less likely to be classified as “Stayers.”

Differences by health status

In the monthly events study, worsening health is associated with an increased likelihood of dropout and is a significant predictor of membership in the “Mid-wave Attritors” group (p-value<0.01), and to a lesser extent, in the “Stayers” group (p-value<0.1) (Note that the rating scale for the self-reported health variable ranges from 1 = Excellent to 5 = Poor). Our analysis of the interaction effects between age and health status show that the health effects are largest among older respondents aged 65 and above (results not shown). This finding aligns with existing literature, which suggests that individuals, particularly in the older age group, with poor health often prioritize healthcare needs over survey participation, leading to higher dropout rates [53,54,111,112].

However, the UCA study presents a contrasting picture, as no discernible association between health status and any of the attrition classes is observed. Additionally, we did not find any interaction effects between age and health status in the UCA study (results not shown).

It is worth reiterating that the unweighted regression results (i.e., AMEs) presented in Tables 5 and 6 of the main text are both quantitatively and qualitatively similar to their weighted counterparts based on UAS survey weights, as shown in S8 and S9 Tables in the Supporting Information section.

Practical implications related to personality traits

Our findings have several important implications for the design, implementation, and management of online panel surveys. For instance, survey organizations could implement targeted interventions to keep respondents with lower conscientiousness engaged, such as offering frequent reminders via text messages or additional monetary incentives. Additionally, personality traits could be utilized as predictive tools to identify participants at higher risk of attrition, allowing for early intervention before dropout occurs. Providing additional support to individuals high in neuroticism or agreeableness, who may be more prone to attrit, can also enhance their commitment to the study.

In terms of nonresponse adjustment strategies, online survey researchers could consider incorporating personality measures into weighting schemes to improve the accuracy of bias adjustments, given that personality traits like conscientiousness are not only strong predictors of survey participation but also causally linked to significant life outcomes. While population-level benchmarks for personality traits do not exist—making traditional post-stratification infeasible—researchers can still leverage internal benchmarks by collecting personality data from all respondents at baseline. Ideally, this should occur immediately after respondents become panel members or immediately after respondents complete the baseline mandatory demographic survey that is prerequisite for becoming panel members. Having personality data for the full initial sample enables researchers to incorporate these variables into longitudinal weights and track potential attrition-related bias over time.

Strength and limitations

A key strength of this study is the use of data from two recently completed studies, both conducted within the same probability-based online panel dedicated to social science and health-related research. The inclusion of rich background information for all respondents enhances the precision of statistical inferences, and the availability of Big Five personality traits data allowed us to emphasize the significance of personality factors, particularly in relation to attrition. However, a limitation is that the study does not assess the extent to which selective attrition may result in biases in some key substantive variables. Another limitation is that our results may have limited generalizability beyond the context we examined. Despite this, our findings provide valuable insights into the complexities and unique patterns of nonresponse and attrition, as well as their correlates and underlying mechanisms, in the context of high-frequency online longitudinal data collection—a topic that has received limited attention thus far.