Nonresponse bias

Unit nonresponse is a major component of total survey error, which itself is currently the default framework for conducting survey methodology research [6–8]. The other types of error are not a subject of this study, nor will I delve into other issues connected to cross-national comparisons. Measurement error or problems with survey research costs will only be mentioned. As for nonresponse itself, if not connected to systematic error, it will pose a threat only to the precision of the estimates. On the other hand, “nonresponse bias results from the contrast between those responding and not responding to the surveys” [9]. Nonresponse bias threatens the representativeness of a survey. For a formal presentation: [e.g., 3 p29–32].

There are many possible ways of estimating the size of nonresponse bias. The most common in survey practice—but highly criticized [10–13] for establishing only a higher bound of the possible error and having a limited relation with other measures—is the RR. As for the other most feasible, possible measures of survey representativeness, they can be divided into two categories. The first category requires individual-level auxiliary data about a whole drawn sample. The most popular members of this category are so-called R-indicators [14–16]. Unfortunately, such required auxiliary data are not available for this study. The second category, which does not require such data, can be divided into the two most popular approaches. The first one, sometimes called “external criteria of representativeness,” requires access to reliable population statistics [9, 17–19]; they can be obtained both from official statistics and other survey data. This approach would not be optimal for an assessment concerning survey data covering such a wide geographical area as the ISSP. Official data or other survey data can and do strongly differ in quality between countries (not to speak of continents). Therefore, differentiation between the effects of nonresponse bias and bias coming from faulty population statistics would be problematic.

As for the second approach, Sodeur [20] proposed a way of estimating the quality of a net sample based on the so-called “internal criteria of representativeness.” It was then developed by Kohler [21] and used in several subsequent studies [e.g., 17, 22–24]. This method is based on finding the proportion in the sample, which is known ex definitione. Souder’s example is heterosexual couples living together in two-person households; the gender balance in this situation should be 50–50. “Internal criteria assessments probe the integrity of the sample by exploiting this a priori knowledge and point to potential irregularities in the survey process” [25]. Kohler [21] argues that this measure would not be affected by coverage and observational errors nor by item-nonresponse (unfortunately only for face-to-face surveys, where gender is in most cases assessed by the interviewer), making it a good candidate for the role of assessing survey representativeness. Ideally, it would be possible to control for same-sex couples, yet this is rarely the case (surely not for the ISSP data, crucial for this study). However, the working assumption could be made that there are no more or no fewer gay couples than lesbian couples in the population. For ESS rounds 1 to 8 and European Quality of Life Survey rounds 1 to 4, where controlling for same-sex couples is possible, the correlation between indicators using strict (with control) and lenient (as in this paper) approaches to assess nonresponse bias was 0.96 and 0.93, respectively [25]. The impossibility of using a strict approach is an important limitation of this study.

Additionally, counting only official marriages would make a big difference between countries with different legal landscapes regarding this matter, so both formal and informal partnerships should be included. In the following parts of this paper, I will refer to Sodeur’s internal criteria of nonresponse bias assessment as a nonresponse bias indicator (NBI).

For an extensive discussion on the usability of the different nonresponse bias measures and indicators in similar research, see the work of Jabkowski, Cichocki, and Kołczyńska [25]. Additionally, it must be stressed that, in reality, nonresponse bias is a property of a specific variable. It can have different sizes for different survey questions. For this reason, the measure used here must be treated only as a proxy. There are some additional possibilities for assessing the quality of the survey in terms of the nonresponse bias. One could use the gross sample information if it consists of individuals with known characteristics, but this is possible only in the case of individual samples [26]. Additionally, nonrespondents could be targeted by very short follow-up questionnaires, as in the ESS, but such follow-up may also be vulnerable to nonresponse. A similar vulnerability is shared by any method that compare early and late respondents, or individuals who needed the conversion procedures with the rest of the respondents [19]. Such assessment methods are only possible for the secondary data analysis, when required information is easily accessible to external researchers, which is rarely the case.

Survey modes

The face-to-face mode has usually, especially in the past, been considered the best mode for achieving high RRs [27, 28] or considered the “gold standard,” which is reflected in the mode most commonly used in the most important comparative survey programs. For example, the Eurobarometer, ESS, and European Quality of Life Survey waves conducted in pre-COVID-19 times (i.e., before 2020) were based solely on the face-to-face mode. Moreover, in the case of the World Values Survey and the European Values Study, this mode was clearly dominant. The rationale for considering this mode the best may be the possibility it provides for the interviewer to use many motivational and conversion procedures and to gather paradata. It can also be driven by the mode preference of potential respondents. Due to very limited research, preference for face-to-face surveys among respondents may be the most widespread, at least in some countries [29, 30]; but apparently, declared preference also depends on the mode of the question administration if the preference is studied through declarative, not behavioral, data [first observed by 31]. I will not discuss the possible causes of these preferences, whether are they psychological [32], cognitive [33], or of another type. The widespread usage of this mode in scientific surveys may also come from the lack of the sampling frame necessary to use any other mode, but this is mostly the case in low- and middle-income countries. On the other hand, the main downside of this survey mode is its cost—usually the highest of all other single-mode surveys [34]. Moreover, the high RRs it generates may stem, if proper supervision and backchecking is not implemented, from fraudulent activity on the part of the interviewer.

Mail surveys have commonly been considered as generating lower RR than face-to-face surveys [28, 35–37]. On the other hand, they are treated as a way to overcome the social desirability bias present mainly in the face-to-face context [38, 39]; however, this type of bias is not a focus of this study. The downside of the mail mode is that it becomes more problematic if no individual frame is available. Random within-household selection made by household members may be of lower quality than one made by a skilled interviewer. However, the RR of mail surveys may also be higher than those of current web surveys [40, 41]. Nevertheless, there are multiple examples of well-designed mail surveys that achieved similar or even higher RRs than one would expect from face-to-face surveys in a similar context [42 p351–352].

While mixed-mode surveys in a very broad sense were part of survey practice from the early years (when self-administered questionnaires started to be used for asking sensitive questions in face-to-face surveys) and started to be more widely discussed in the 1960s, they became very popular only in the 21st century [43]. The idea of mixing different modes of data collection is motivated mainly by the desire to reduce cost, social desirability bias, or nonresponse (but not all at once). It may also involve modes of contact and a follow-up phase. Mix-mode surveys could possibly be cheaper than face-to-face surveys while achieving higher RR if carefully designed [44]. I will not discuss the complex differences between various types of mixed-mode [42 p398–449, 45–47] implementation: sequential and parallel design, mode choice [48], mode preference [29, 49–51], adaptive or responsive design [52]. It is because the mixed-mode surveys I will analyze in the forthcoming parts of this paper are heterogeneous in implementation specifics, as are single-mode surveys. In a study consisting of 795 cases, it would be impossible to address these differences substantially.

As stated previously, a body of literature concerning relations between survey mode and nonresponse bias exists but is not very large; nevertheless, any endeavor aimed at finding clear conclusions in it may be fruitless. This paper is a systematic review, not a narrative one. Therefore, the list of findings is more of an illustration. I will start with the single-mode comparisons. De Leeuw [28] in her experiment found that only face-to-face surveys (in comparison to mail and computer-assisted telephone interviewing, CATI) generated significant differences between gender proportion in the sample and the register data. Laaksonen and Heiskanen [53] found in face-to-face, phone, and web parallel surveys in Finland that the face-to face mode was the least representative as measured by R-indicators. Klausch, Hox, and Schouten [54] found in their mode experiment comparing face-to-face, CATI, web, and mail modes that face-to-face and web modes achieved the best representativeness measured by R-indicators. There are also some comparisons between single-mode and mixed-mode surveys. Fowler et al. [55] found results from a mail and CATI mixed-mode survey more representative than from a single-mode mail survey. Dillman et al. [56] showed that using a mixed-mode survey design can potentially raise RR compared to a single-mode survey, but only non-face-to-face modes were included in this study; the authors did not find any significant differences in representativeness between single-mode and mixed-mode approaches. Luiten and Schouten [57] compared a single-mode CATI survey to a mixed-mode web, mail, and CATI survey; while both had similar RR and cost, the mixed-mode approach was more representative as measured by R-indicators. Klausch, Hox, and Schouten [54] compared single-mode CATI, mail, and web surveys with their respective mixed-mode approaches with added face-to-face follow-up; in most cases, the bias measured by R-indicators for mixed-mode surveys was lower. Bianchi et al. [58] found that switching the mode in the panel survey from a face-to-face to a mixed-mode approach resulted in similar RR and sample composition while lowering the cost. Villard and Fitzgerald [34], after analyzing a series of mode experiments conducted under the auspices of the ESS, concluded that “mixed-mode data collection designs are unlikely to increase RR or reduce nonresponse bias in survey estimates compared to data collected face to face. In fact, overall, they suggest that RR would decrease and nonresponse bias would probably increase.” Cornesse and Bosnjak [59], after conducting a meta-analysis of 96 surveys, found that using mixed-mode surveys resulted in nonresponse bias reduction measured by R-indicators compared to single-mode surveys. Sztabiński [60], in an ESS mode experiment in Poland, found that using a mail, web, and face-to-face mixed-mode survey resulted in lower RR, higher costs, and a similar sample composition as a parallel single-mode survey. Wolf et al. [44], after an analysis of the 2017 round of the European Values Study, found that face-to-face surveys were more representative than some mixed-mode surveys but that the latter were much cheaper and had higher RR.

The preceding section offered an extensive overview of the current state of knowledge about the relationship between survey modes and various propositions of measures of survey representativeness. While an important body of research exists in this field, the results can be difficult to reconcile due to the limitations of traditional forms of narrative review or even such approaches as simple vote counting of significant results. These limitations can lead readers to gain a false impression of conflicting results in many fields [61 p20]. To address these issues, this study endeavors to shed more light on this subject by harnessing the potential of meta-analysis, which is a “statistically defensible approach to synthesizing empirical findings” [62 p8]. Therefore, this study aims to answer the following research questions: What is the relation between survey mode and survey representativeness, and how is this relation affected by other survey characteristics and broader context?

Meta-analysis

Most commonly, meta-analysis is used to create proper statistical synthesis of results taken from many articles or clinical trials. In that situation, some typically used dependent variables, called effect sizes or treatment effects [77 p17], are usually computed to either statistically standardize the different measures used by different authors or comply with standard settings of experimental studies in medicine. Therefore, readers probably encountered two types of effect sizes: “those based on differences of two means and those based on differences of two proportions” [78 p13]. However, these examples do not exhaust the usefulness of meta-analysis. On the contrary, “effect size is simply the size of anything that may be of interest” [79 p34], whether it is measured in exactly the same way for every study or recalculated to make it so. For example, in epidemiological studies, the true population mean, or proportion, may be of interest [80]. Card proposed that the working definition of effect size could be “an index of the direction and magnitude of association between two variables” [62 p87], a definition that would include the measure used in this paper, if interpreted sensu largo.

An important question while doing a meta-analysis is whether to use a fixed- or random-effects model [77 p61–86]. Fixed-effects models are justified only when the studies included are nearly identical and one is not interested in treating them as part of a bigger population. Therefore, random-effects models, being more robust and allowing for some generalizations, should be used in most cases. The simplest meta-analytical random-effects model looks like the equation below:

“Parameter k is the number of included studies and estimator stands for single-study effect size, i.e., the difference between the observed and the true proportion of females in two-person heterosexual households. For each study, weights are computed as reciprocals of the variance of , i.e., = V_i + τ², where V_i is within-study variance and τ² denotes between-study variance” [23, 77].

When enough heterogeneity between studies is found, meta-regression is the usual method of choice to analyze it [81]. To avoid spurious findings while conducting meta-regression, methods such as using the so-called Knapp-Hartung test [82] or permutation test were proposed [83–85]. Nevertheless, these typical two-level models, where the “hidden” first levels are usually individuals and the second levels are study summaries are sometimes not enough [86–89]. When one could expect that studies are nested in some higher-level cluster, one should use the three-level meta-regression model to deal with such dependencies. For discussion on why using more common hierarchical models is not enough, please see [90]. Studies can and often are nested in more than one set of clusters at a time—that is, random factors may be crossed; to address that situation, cross-classified models can be used [91]. Many types of meta-analytical models have already been used to conduct research in the area of survey methodology, all of which either synthesize results from different papers or are based on self-calculated measures for selected survey programs [e.g., 10, 37, 40, 41, 48, 59, 92–98].

The effect size I used for the meta-analysis was, as stated before, NBI. The choice of moderators for the models was based on both state-of-the-art and statistical evaluation. Survey mode and frame constitute probably the most basic set of survey characteristics. The subject of mode was extensively described in the introductory parts of this paper. As for the sampling frame, the frame of individuals is usually considered better in terms of representativeness than other types of frames, primarily because of the burden connected to the process of within-household selection of the respondent at the last sampling stage [22, 99, 100]. Therefore, I used the binary variable individual/nonindividual (household/address, area probability sampling). To not overcomplicate the model, I ignored the possible impact of the within-household selection method on nonresponse bias [96]. Exploratory subgroup analysis and previous knowledge about possible issues connected with the aforementioned respondent selection not being supervised by the interviewer (as in, for example, mail mode) led to the conclusion that the interaction between those two moderators may also be an important factor in explaining differences in NBI values (Readers can conduct all analyses mentioned in the article, and others, using the R script available at [101].) Therefore, mode, frame, and interaction between them were selected for the simplest model specification. The analyzed dataset consists of studies from two survey programs that differ significantly in terms of funding and organization [102, 103], hence the inclusion of the binary variable ESS/ISSP in the extended model. Institutional experience can influence outcome rates of a survey [3 p102–113] and thus at least the upper boundary of possible nonresponse bias. I chose to include a moderator that for each observation simply counts how many surveys were done previously in this country and program. This study, as stated previously, is based on observational data—like many meta-analyses in survey methodology [98]. If one, however, would like to apply the categories of variables usually included in the causal inference models, then it should be noted that this study is focused chiefly on the effect of survey mode on nonresponse bias and that every candidate for the meta-regression moderator listed above should be treated as a confounder. The sampling frame, institutional experience, and survey program can usually have an impact on the choice of mode(s) and separately on the nonresponse bias. As for the last candidate for a moderator—the RR (RR6)—its inclusion can be used to estimate the partial effects of survey modes holding RR constant. In the causal inference model, it would be a mediator. RR6 was included only in the last of the models presented in the results.

After testing if the inclusion of stated moderators is not only theoretically but also statistically sound (addressed in the results section), identification and exclusion of outliers were performed based on the influence diagnostics of outliers [104] and on quantile-quantile plots [105, 106]. The four identified highly problematic observations included ESS surveys in Slovakia conducted in rounds 4, 5, and 6 and an ESS survey in Lithuania from round 9. All of them were face-to-face surveys with unusually high value of NBI. Lithuanian cases were already identified by Jabkowski, Cichocki, and Kołczyńska [25]; the sampling design switched from one based on the individual register frame in round 3 to an area sample with house enumeration in round 4. “This change dramatically increased interviewer discretion in regard to respondent selection,” which resulted in oversampling women. As for the case of Lithuania in round 9, it is a bit more problematic. Between rounds 8 and 9, the data collector changed, the mode switched from PAPI to CAPI, and the Kish grid replaced the last birthday method. Moreover, the DEFFp rose from 1.34 to 1.64 and the number of completed interviews dropped from 2,122 to 1,835 (References to documentation are available at [70].) Nevertheless, none of these, unfortunately, explains such a rise of NBI. The final models after the exclusion of outliers are based on 791 observations.

To answer the research questions, I created random-effects meta-regression (alternatively called mixed-effects) models using the restricted maximum likelihood method to estimate τ² [77 p114–117]. Every model was created with the R package metafor [107]. Software citations are available at [108] and the R script for replication at [101]. To address the issue of surveys probably being nested in some third-level clusters, which may result in observation dependency, I introduced cross-classified models [91], where surveys were nested simultaneously in program waves and countries. Surveys conducted on one country’s population at different points in time can be treated as repeated observations by some, therefore the need for nesting. Countries can also differ in survey climate, but it is hard to measure, and such measure was not included. Instead, the random intercepts for countries may, to some degree, serve the role of estimating it. The author considered including a variable measuring cultural trust at this level, but the most important, comprehensive source of such were the World Value Survey results, and the idea of basing survey evaluation on data from a methodologically less “diligent” survey was rejected. Surveys being part of specific comparative program waves (counted here separately for both programs) can share many characteristics, such as timeframe, methodological standards, and especially questionnaire topics. Topic saliency can have significant impact on unit nonresponse [109–111], but it is not uniform for the whole population. Therefore, it was accounted for only in the random intercept of waves. The moderator that explained part of wave variance only is the survey program because it does not differ inside the cluster. It is usually considered that there should be at least 20 third-level clusters [112 p46–48]. However, it is a matter of discussion [113] and is not always followed by researchers [86, 114]. In this case, surveys were nested in 32 waves and 53 countries, which is a clearly sufficient number. As the ICC calculation and the checking of I² distribution between model levels [115] are extremely problematic in cross-classified models, such values were gathered for surveys nested in countries and in waves separately. Cross-nested models with three sets of moderators are presented in the results part. The final two-level model and the three-level model with RR6 as an effect size can be consulted in the S1 Table.

The raw, nontransformed proportions [for transformation proposal, see, e.g., 80] were used to calculate NBI. It is because the standard error of the NBI is based on the known “true” proportion of 0.5; therefore, there was no identified risk for the variance calculation. Moreover, subsamples and proportions were calculated without using weights. As weights are not standardized between countries and in the ISSP most of them are poststratification weights, it is better for comparisons to use raw data [For extensive discussion on this subject, please see 25.] If design weights were available for all analyzed cases, it would be the best choice to use them in calculations.

Meta-regression

Meta-regression based on the full set of observations, with the basic set of moderators (mode*frame), was compared with models including additional variables: program, repetitions, and a set of both. The likelihood ratio test suggests that, statistically, the full set is the best: χ²(2) = 13.09;p = 0.001. The model using this set of moderators, restricted maximum likelihood, and the Knapp-Hartung method, with p-values adjusted according to the permutation method, found mail mode, mixed-mode (no face-to-face), frame, program, repetitions, and mail-nonindividual frame interaction moderators to be significant. This model was used for quantile-quantile plot and influence diagnostics, which both have resulted in flagging four observations as problematic outliers. These outliers were already described in the data and methods section and were excluded from any further analysis, which lowered the number of observations to 791. The two-level model designed the same as in the previous step, but with outliers excluded, differs in the frame moderator not being statistically significant and the mixed-mode (with face-to-face) starting to be statistically significant, having a pseudo-R² of 77.52%, and a significant test of moderators: F(df1 = 9, df2 = 781) = 29.0204;p = 0.001. Multimodel inference [117] confirmed that the final set of moderators is optimal. This model is available in S1 Table.

As calculating the ICC and I² distribution is problematic for cross-nested models, I calculated them for models with country and wave as the third level separately. Every three-level model in this paper uses t- and F-distributions for making inference, and uses “an improved method for approximating the degrees of freedom” [118 p128] for these distributions, as the full Knapp-Hartung method is unavailable, and the models also exclude the four outliers. According to a likelihood ratio test (intercept only), the model with surveys nested in waves (and no moderators) is statistically better than the two-level model: χ²(1) = 5.2;p = 0.02 and has an ICC value of only 0.1, 28.6% of the total variance at level 2, and only 3.28% at level 3. The model with surveys nested in countries (and no moderators) is statistically better than the two-level model: χ²(1) = 97.64;p<0.0001 and has an ICC value of 0.65, 13.1% of the total variance at level 2, and 24.1% at level 3. The cross-classified model (surveys nested in countries and waves) has all three variance components identifiable according to likelihood profile plots [118 p171]. Table 2 presents a set of three cross-classified models with different sets of moderators. The first model uses the most basic set—mode, frame, and their interaction—and has a pseudo-R² of 58%. The second model uses the optimal set of available moderators with added program and repetition (“institutional experience”) variables and has a pseudo-R² of 67%. The third model, aimed at estimating the partial effect of the mode holding outcome rates constant, includes an additional moderator: RR (RR6). It has a pseudo-R² of 69%. For context, readers can find a similar cross-classified model using RR6 as an effect size in S1 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Cross-classified meta-regression models estimating the relationship between survey characteristics and NBI.

https://doi.org/10.1371/journal.pone.0283092.t002

The three presented models agree that when the frame is held at the “individual” level, mail mode and mixed-mode without face-to-face in the mix have significantly lower NBI than single-mode face-to-face. According to Model II, when the frame is held at the “individual” level, every mode is better in terms of nonresponse bias than pure face-to-face. When one holds the frame at the “nonindividual” level, the only mode bearing significantly different results than face-to-face is the mail mode; that mode used with a nonindividual frame is connected to a massive and significant increase in the NBI (This is the strongest effect observed in this study.) This problematic combination of mail and address frame in the survey design was most used in France (16 out of 28 cases). One should bear in mind that there are far fewer cases of mail and mixed-mode (no face-to-face) surveys connected to the household/address frame than to the frame of individuals: 28 to 73 for mail and 3 to 49 for mixed-mode. The ESS has significantly lower NBI than the ISSP and also more experience possibly gathered by the institution results in slightly lower bias. Model III, estimating partial effects of the mode, bears very similar results. Additionally, in this model, a higher RR is connected to lower bias.