Confidence Intervals

We most often think of sampling error as the precision of a point estimate, represented by 95% confidence intervals. Confidence intervals are useful for two purposes. First, they characterize the precision of the estimate. Second, they provide context about whether estimated parameters are likely to be equal between two populations or time-points. In the absence of non-sampling error, if a large number of repeated samples is taken from the target population using the same sampling design, 95% of the resultant 95% confidence intervals about each sample point estimate will contain the true, albeit unknown, intervention coverage in the population [3]. In practice, the interpretation of confidence intervals is slightly different. Suppose, for example, we have a point estimate of intervention coverage of 70%, with a 95% confidence interval of 65%–75% in a program for improving MNCH. This confidence interval is typically taken to mean that we are 95% certain that the true intervention coverage in the population lies somewhere between 65% and 75%. Consider if the program had set a target of achieving 70% population coverage. In this case, one could not say with a high degree of confidence that the program had achieved its target coverage, even with a point estimate for coverage at 70% since fully half of the confidence limit lies below 70%.

Sample Size and Sampling Design

The level of sampling error around a point estimate depends on the number of observations selected from the target population (sample size), the underlying prevalence and variance of the characteristic of interest in the target population, and the sampling design used in selecting the sample. In general, for a given sampling design, the precision of a point estimate improves with increasing sample size. Thus, a larger sample size yields a tighter confidence interval. However, given a specified sample size, the sampling design also directly affects the precision of an intervention coverage point estimate. In practice, data from nationally representative surveys nearly always come from a cluster survey rather than a simple random sample of households. In cluster surveys, the sampling error is affected by the number of respondents per group as well as the intracluster (or intraclass) correlation coefficient, which provides a measure of how similar the elements being analyzed in each cluster are with respect to a particular characteristic of interest [2]. In general, for a given sample size, the precision of a point estimate ascertained from a cluster sample design decreases as the sample size per cluster increases, and as the intracluster correlation coefficient increases.

Reporting Sampling Errors and Confidence Intervals

Although the sample size and sampling design are frequently reported in survey methods, sampling errors represented by confidence intervals are rarely reported in the main body of reports of national surveys. DHS survey reports and reports for AIS and MIS surveys conducted through the DHS program include standard errors and accompanying confidence intervals of ±2 standard errors (slightly wider than the 95% confidence interval) for most key indicators in Appendix B only. In MICS survey reports, standard errors and 95% confidence intervals appear in Appendix S. Almost no MIS surveys conducted outside of the DHS program include sampling errors in the survey reports or appendices.

Comparing Survey Results

One oft-cited advantage of large well-conducted surveys is that their results can be compared meaningfully both over time and between locations. In the absence of a formal hypothesis test, informal conclusions about the statistical significance of a difference are sometimes drawn by examining whether two confidence intervals overlap. Indeed, if two 95% confidence intervals do not overlap, then a formal hypothesis test would reject the null hypothesis of equality with a p-value below 5%. Unfortunately the converse is not true. If 95% confidence intervals overlap, it is still possible for the p-value from the hypothesis test to fall below 5% [4],[5]. An example from the 2011 Ethiopia DHS report (presented in Box 1 and Figure 1) illustrates how difficult it can be to correctly interpret MNCH intervention coverage estimates without considering confidence intervals [6].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. Two ways of looking at coverage of full immunization in Ethiopia—with and without confidence intervals.

Ethiopia DHS surveys 2000, 2005, 2011 [33].

https://doi.org/10.1371/journal.pmed.1001386.g001

Sampling Error and Data Disaggregation

When analyzing survey data for MNCH coverage indicators, sampling errors and the accompanying confidence intervals are influenced, at times substantially, by changes in sample size owing to disaggregating the data by socio-behavioural or demographic characteristics of interest, such as where the respondents live or their household socioeconomic status. While DHS and MICS surveys typically report the confidence intervals around many key indicators in their appendices, they do not report confidence intervals of disaggregated point estimates. To illustrate why it is important to consider the sampling error of such estimates, we present summary sampling characteristics of established malaria control coverage indicators from the 2007 Zambia DHS survey (Table 1) [7]. Point estimates were weighted to account for unequal probability sampling and any differential non-response. Standard errors were estimated to account for correlated data at the primary sampling unit level using the Huber White Sandwich estimator, which accounts for the loss of precision due to increased intracluster correlation coefficient.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Sampling characteristics of selected point estimates from the 2007 Zambia DHS Survey [7].

https://doi.org/10.1371/journal.pmed.1001386.t001

For assessing progress of malaria control efforts in Zambia, the standard indicator for household insecticide-treated mosquito net (ITN) coverage shows that 52% of households possess at least one ITN, with a 95% confidence interval of ±2.5 percentage points (Table 1). The interpretation for programmatic purposes, based on information from the sample of 7,164 households, would be that one has 95% confidence that the true (albeit unknown) proportion of households in Zambia with at least one ITN lies somewhere between 50% and 55%, in the absence of non-sampling error. By contrast, the indicator used for assessing progress in the coverage of access to antimalarial treatment of fevers in children (38%) has a 95% confidence interval of ±3.8 percentage points, because this sample is limited to only those children with fever in the past 2 weeks (5,844 children). When disaggregated by household socioeconomic status, the 95% confidence intervals increase considerably. For example, based on a subsample of 106 children, the proportion with a fever in the past 2 weeks in the wealthiest households who received an antimalarial was estimated to be 40%, with a 95% confidence interval of ±10 percentage points, which would be interpreted to mean that one can only be 95% confident that the true antimalarial coverage lies somewhere between 30% and 50% in this population in Zambia. This level of uncertainty clearly poses challenges in the interpretation of the point estimate for decision making.

Information Error and Information Bias

Information error and information bias (also referred to as measurement error and bias) are common in household survey estimates of MNCH intervention [8]–[13]. Information error arises from errors in measuring MNCH intervention coverage, which often occur when the respondents do not know the exact answer to the survey questions, yet provide answers anyway. Information bias arises from systematic (i.e., non-random) errors in measuring MNCH intervention coverage and includes recall bias and social-desirability bias. Although information error typically results in higher variance, and as a result decreased precision of the point estimate, information bias results in an overestimate or underestimate of the population point estimate. The real problem is that one does not know which way this type of error is biasing the results.

The measurement of vaccination coverage provides a good example of how information error and bias can affect the ability of decision makers to use information from surveys to evaluate the trends in coverage over time. As described by Cutts and colleagues elsewhere in this collection [8], information on vaccination coverage is usually kept in home-based vaccination records but once a child is past infancy, parents may not retain these records. If a vaccination card is not available, DHS and MICS surveys request information from each mother on the vaccinations received by children born in the past 5 years. Because home-based vaccination records are less likely to be available for children born 4 years ago than for those born 1 year ago, recall error or bias may be more likely for older than for younger children, which reduces the ability of decision makers to draw strong conclusions about trends.

Even with the best designed surveys and questionnaires, recall error and bias can affect MNCH intervention coverage estimates [14]–[16]. For example, for estimating ITN coverage, a key indicator for malaria control programs [17], surveys must ascertain the type of net used and, for nets that are not long-lasting insecticidal nets, when the net was procured and treated with insecticide to distinguish nets that are ITNs from those that are not. Data from Eritrea show substantial date heaping (rounding off of time since an event by respondents) at 12 months for both net procurement and insecticide re-treatment. Date heaping affects whether nets meet the definition of an ITN and therefore has the potential to bias coverage estimates [18].

Measurement of ITN use by children <5 years old also provides a good example of how social desirability bias can impact MNCH coverage estimates. Suppose a national survey obtained an estimate of ITN use among children of 60%, with a 95% confidence interval of 55%–65% but that respondents in selected households tended to tell household interviewers that their ITNs were being used more often than they really were because the respondents feel socially obligated to respond positively to questions about recent net use, a situation that is thought to occur [19]. In this scenario, social desirability bias results in systematic over-reporting of the use of ITNs. Even in a well-designed and implemented survey with a sampling error ±10%, we cannot be 95% certain that the true proportion of child ITN use falls within the calculated 95% confidence interval of 55%–65%. In fact, we would need to use expert but subjective judgment as to the direction and the magnitude of any bias when interpreting this coverage estimate and the degree to which bias may affect results.

Selection Bias

Another common form of non-sampling error is selection bias, which occurs when potential respondents do not all have a known and non-zero probability of selection. In such a case, no matter how large a sample is taken, survey point estimates will always differ from the true, yet unknown, population characteristic under study [3]. Common sources of selection bias in surveys that measure MNCH intervention coverage include: (1) failure to include some eligible respondents in the sampling frame, (2) failure to account properly for the sampling design in analyzing survey data because the sampling design parameters (e.g., cluster size) have not been reported, and (3) the use of non-probability sampling designs.

Coverage bias results when the sampling frame excludes some members of the target population or includes ineligible persons who are not part of the target population. It can occur when certain districts of a country are excluded from a survey because of security considerations. It can also occur when members of the target population are homeless, nomadic, or institutionalized and therefore not identified with one of the physical homes used in the multistage household sampling designs common in low- and middle-income countries.

The household sampling methods designed by the World Health Organization for measuring child immunization coverage under the Expanded Programme on Immunization (EPI) provide a good example of the importance of considering sampling design when interpreting survey data. In this program, 30 clusters are selected and seven children within each cluster are selected for assessment of their immunization status [20],[21]. Variations of this protocol are widely used to measure sub-national intervention coverage in low- and middle-income countries as they require considerably fewer resources to implement than the well-established protocols employed in DHS and MICS surveys [8],[22], which enumerate all eligible households in the selected cluster and then make random selections from that updated list. While improvements have been made over the years, the EPI survey method does not typically include the development of an accurate sampling frame within each of the selected 30 clusters for selecting the seven children, and thus the survey design does not ensure that each child sampled has a known non-zero probability of selection [23]. Moreover, sampling weights cannot be used when analyzing these data to account for unequal probabilities of child selection, which requires accurate counts of the households in each cluster. As a result, selection bias cannot be ruled out, and may actually be quite likely when using traditional EPI 30×7 cluster survey methods [24]. Box 2 presents a hypothetical example of these issues.

MNCH parameters are also sometimes assessed using “rapid” survey protocols that are designed to use minimal samples sizes to obtain local estimates of intervention coverage in comparison with a desired value for programmatic purposes. The small sample size survey technique called Lot Quality Assurance Sampling (LQAS) is a good example [25]. There are several varieties of this survey technique with varying levels of rigor [26]–[31], all of which seek to classify districts as having either adequate or inadequate intervention coverage. LQAS surveys are usually short, consisting of a few questions focused on a small number of binary outcomes. They are quick to administer and analyze, sometimes taking only 1–2 days per district. When implemented using best practices, study teams select households randomly from an up-to-date sampling frame so the results are representative of the population. At other times, however, respondents are identified using a convenience sample approach in which only one person is selected in a truly random fashion and subsequent respondents are selected from nearby households. Additionally, in some LQAS protocols field staff stop collecting data when it becomes clear that the final few respondents will not affect the district's adequate/inadequate classification. In all cases where an LQAS design does not allow each respondent to have a known and non-zero probability of selection, the sampling is not truly random and the validity of results is threatened by selection bias.

Non-response bias is also a common source of selection bias. Non-response bias results when the answers to survey questions differ between selected respondents who participate in the survey and selected respondents who choose not to participate, or are unable to participate, in the survey. The response rate is measured as the proportion of individuals selected to take part in a survey who actually complete the questionnaire. The potential for non-response bias goes up as the response rate goes down. This type of bias is common in phone and internet-based surveys—interview methods more commonly used in high-income countries where populations have access to such technology. In such surveys, it is not uncommon to have response rates as low as 50%, which must be corrected for by using survey weighting and other techniques. In low- and middle-income countries where survey interviews are conducted in person with individuals in selected households, non-response bias is usually less of an issue. The protocol used by DHS, MICS, AIS, and MIS surveys for limiting non-response bias requires interviewers to return to selected households at least three times if selected individuals in the household are unavailable for the interview [1]. This practice greatly limits household and individual non-response, with response rates among eligible women of reproductive age in these surveys typically exceeding 95%. However, non-response bias can be a serious issue for estimates of MNCH intervention coverage obtained from surveys that do not follow this protocol. For example, EPI surveys do not require multiple visits to selected households and do not document how many potentially eligible households were skipped because the respondent was absent. [23]. Both DHS and MICS surveys report the response rates of households and of eligible women and men who were selected to participate in the survey. This practice should be followed in all survey reports, and where response rates are below 85%–90%, surveys results should be interpreted with caution.