^{1}

^{2}

^{*}

^{1}

LA is a member of the technical advisory group of the UN IGME. JRN is a consultant for UNICEF. The views expressed in this paper are those of the authors and do not necessarily reflect the views of the United Nations Children's Fund. The authors declare that no other competing interests exist.

Conceived and designed the experiments: LA JRN. Performed the experiments: LA JRN. Analyzed the data: LA JRN. Contributed reagents/materials/analysis tools: LA JRN. Wrote the first draft of the manuscript: LA. Contributed to the writing of the manuscript: LA JRN.

Leontine Alkema and colleagues use a bootstrap procedure to assess the uncertainty around the estimates of the under-five mortality rate produced by the United Nations Inter-Agency Group for Child Mortality Estimation.

Millennium Development Goal 4 calls for an annual rate of reduction (ARR) of the under-five mortality rate (U5MR) of 4.4% between 1990 and 2015. Progress is measured through the point estimates of the United Nations Inter-agency Group for Child Mortality Estimation (UN IGME). To facilitate evidence-based conclusions about progress toward the goal, we assessed the uncertainty in the estimates arising from sampling errors and biases in data series and the inferior quality of specific data series.

We implemented a bootstrap procedure to construct 90% uncertainty intervals (UIs) for the U5MR and ARR to complement the UN IGME estimates. We constructed the bounds for all countries without a generalized HIV epidemic, where a standard estimation approach is carried out (174 countries). In the bootstrap procedure, potential biases in levels and trends of data series of different source types were accounted for. There is considerable uncertainty about the U5MR, particularly for high mortality countries and in recent years. Among 86 countries with a U5MR of at least 40 deaths per 1,000 live births in 1990, the median width of the UI, relative to the U5MR level, was 19% for 1990 and 48% for 2011, with the increase in uncertainty due to more limited data availability. The median absolute width of the 90% UI for the ARR from 1990 to 2011 was 2.2%. Although the ARR point estimate for all high mortality countries was greater than zero, for eight of them uncertainty included the possibility of no improvement between 1990 and 2011. For 13 countries, it is deemed likely that the ARR from 1990 to 2011 exceeded 4.4%.

In light of the upcoming evaluation of Millennium Development Goal 4 in 2015, uncertainty assessments need to be taken into account to avoid unwarranted conclusions about countries' progress based on limited data.

In September 2000, world leaders adopted the United Nations Millennium Declaration, committing member states (countries) to a new global partnership to reduce extreme poverty and improve global health by setting out a series of time-bound targets with a deadline of 2015—the Millennium Development Goals (MDGs). There are eight MDGs and the fourth, MDG 4, focuses on reducing the number of deaths in children aged under five years by two-thirds from the 1990 level. Monitoring progress towards meeting all of the MDG targets is of vital importance to measure the effectiveness of interventions and to prioritize slow progress areas. MDG 4 has three specific indicators, and every year, the United Nations Inter-agency Group for Child Mortality Estimation (the UN IGME, which includes the key agencies the United Nations Children's Fund, the World Health Organization, the World Bank, and the United Nations Population Division) produces and publishes estimates of child death rates for all countries.

Many poorer countries do not have the infrastructure and the functioning vital registration systems in place to record the number of child deaths. Therefore, it is difficult to accurately assess levels and trends in the rate of child deaths because there is limited information (data) or because the data that exists may be inaccurate or of poor quality. In order to deal with this situation, analyzing trends in under-five child death rates (to show progress towards MDG 4) currently focuses on the “best” estimates from countries, a process that relies on “point” estimates. But this practice can lead to inaccurate results and comparisons. It is therefore important to identify a framework for calculating the uncertainty surrounding these estimates. In this study, the researchers use a statistical method to calculate plausible uncertainty intervals for the estimates of death rates in children aged under five years and the yearly reduction in those rates.

The researchers used the publicly available information from the UN IGME 2012 database, which collates data from a variety of sources, and a statistical method called bootstrapping to construct uncertainty levels for 174 countries out of 195 countries for which the UN IGME published estimates in 2012. This new method improves current practice for estimating the extent of data errors, as it takes into account the structure and (potentially poor) quality of the data. The researchers used 90% as the uncertainty level and categorized countries according to the likelihood of meeting the MDG 4 target.

Using these methods, the researchers found that in countries with high child mortality rates (40 or more deaths per 1,000 children in 1990), there was a lot of uncertainty (wide uncertainty intervals) about the levels and trends of death rates in children aged under five years, especially more recently, because of the limited availability of data. Overall, in 2011 the median width of the uncertainty interval for the child death rate was 48% among the 86 countries with high death rates, compared to 19% in 1990. Using their new method, the researchers found that for eight countries, it is not clear whether any progress had been made in reducing child mortality, but for 13 countries, it is deemed likely that progress exceeded the MDG 4 target.

These findings suggest that new uncertainty assessments constructed by a statistical method called bootstrapping can provide more insights into countries' progress in reducing child mortality and meeting the MDG 4 target. As demonstrated in this study, when data are limited, uncertainty intervals should to be taken into account when estimating progress towards MDG 4 in order to give more accurate assessments on a country' progress, thus allowing for more realistic comparisons and conclusions.

Please access these websites via the online version of this summary at

The UN website has more information about the

More information is available from UNICEF's ChildInfo website about the

All UN IGME child mortality estimates and data are available via

The assessment of levels and trends in child mortality is of key importance for measuring progress toward Millennium Development Goal (MDG) 4, which calls for a reduction of two-thirds in mortality in children under five (the under-five mortality rate [U5MR]), and for measuring the impact of interventions. For countries without well-functioning vital registration (VR) systems (the great majority of developing countries), assessing levels and trends in U5MR is challenging because of limited data availability and/or issues with data quality. The United Nations Inter-agency Group for Child Mortality Estimation (UN IGME, including the United Nations Children's Fund, the World Health Organization, the World Bank, and the United Nations Population Division) produces and publishes estimates of child mortality annually for all UN member states. The most recent set of estimates was published in 2012

When analyzing trends in U5MR, the focus is generally on the “best” estimates, i.e., the point estimates. This can lead to inaccurate conclusions about countries' progress in reducing the U5MR. For example, the annual rate of reduction (ARR) in Benin between 1990 and 2005 was estimated to be 2.4% in the most recent UN IGME estimates, based on all data available in 2012. Based on the data assumed to have been available in 2006 (i.e., data series from fieldwork conducted prior to 2006), the estimate of the ARR for the same period is only 1.8% (authors' calculation, using the current UN IGME U5MR estimation method); additional data increased the point estimate of the ARR by 33%. Such changes in point estimates of past ARRs are to be expected in many developing countries without well-functioning VR systems because data in such countries are mostly collected retrospectively and are subject to sampling and non-sampling errors. That is, a recent survey can provide data points for the 1990s and 2000s and can potentially further change the point estimate of the ARR for past periods. For an evidence-based analysis of progress in reducing U5MR, an uncertainty assessment of the estimates for U5MR levels and trends, in the form of uncertainty intervals (UIs), is thus required.

UIs are also required for more informed comparisons across countries. For example, even though the UN IGME point estimates for the ARR from 1990 to 2011 are very similar for Ghana and Equatorial Guinea (2.1% and 2.3%, respectively), the estimate for Equatorial Guinea is based on five data points in total and thus highly uncertain. Lower bounds of the UIs for the ARRs that quantify the minimum progress that most likely has been made would allow for a more informative ranking of countries in terms of proven progress in reducing U5MR.

In short, to avoid inaccurate conclusions and comparisons about countries' progress in reducing the U5MR, uncertainty in the estimation of U5MR and its rate of reduction needs to be assessed and taken into account when analyzing trends. The objective of this article is to propose a method to construct plausible UIs for the U5MR as well as for the ARR. In the previous UN IGME publication in 2011

The UN IGME 2012 database is publicly available on CME Info (

Observations of child mortality vary within and between data series because of sampling and non-sampling errors. This is illustrated in

Connected dots represent series of indirect estimates of U5MR from birth histories. The source and source date for each series is given in the legend.

In 2011, UIs were constructed for the UN IGME estimates for countries where the default method was used, based on the uncertainty in (the parameters of) the local log-linear fits. The resulting UIs were deemed implausible (too narrow) by experts for many countries, and therefore not published. One reason why uncertainty was likely to be underestimated in that approach is the ignorance of the structure of the U5MR data, specifically, potential biases in levels and trends of U5MR data series (explained in more detail in the next section). A second reason is the omission of the possibility that a specific data series is of inferior quality, and a third reason is the lack of inclusion of model uncertainty (i.e., the choice of span parameter α in the loess fit and expert adjustments, and, more generally, the choice of a loess smoother instead of a different curve fitting procedure).

We used a bootstrap method

Because of errors in observed data series, true national child mortality rates are unknown. If there are many data series with small errors available in a given country, the UN IGME point estimates are likely to be very close to the true rates; the fewer the number of data series or the larger the errors in the data series, the further away the UN IGME estimates could be from the truth. Bootstrapped uncertainty bounds are based on the following assessment. Supposing that the UN IGME point estimates for a given country were equal to the truth, which data series could have been obtained in that country (instead of the data series included in the UN IGME database), and, from those, what UN IGME point estimates could have been constructed? For countries with a small number of data series that are deemed to have large errors, the range of data series that could have been obtained will range widely, and, consequently, a wide range of UN IGME point estimates could have been obtained. On the other hand, in countries with many data series of high quality, estimates for a given year will tend to be more similar. Bootstrapped uncertainty bounds are based on the set of estimates that could have been obtained for each country, based on the scenario that the UN IGME estimates are equal to the truth, and reflect the data availability in the country, as well as the likely errors in the available data series, based on an assessment of biases in data series.

Bootstrapping refers to creating a large number of “new” datasets (that could have been observed instead of the dataset at hand), and then repeating the curve fitting procedure to obtain a large number of point estimates in the form of U5MR trajectories. The set of “bootstrapped” trajectories illustrates the uncertainty associated with the original estimates. Associated 90% uncertainty bounds for the U5MR are obtained by selecting the 5th and 95th percentiles of the bootstrapped trajectories. Similarly, the UI for the ARR for a country in a given period is obtained by selecting the 5th and 95th percentiles of the bootstrapped ARR estimates, where each bootstrapped ARR estimate is calculated from one bootstrapped U5MR trajectory.

The difficulty in the bootstrap procedure is the first step: how to generate “new” datasets that lead to a similar variability in point estimates as the dataset at hand. Non-parametric and parametric bootstrapping procedures incorporate different approaches to the sampling step. In the non-parametric bootstrap procedure, data are resampled from the original dataset with replacement. This procedure is not easily applicable in the U5MR estimation context because U5MR observations are organized in series, or distinct sampling units. That is, surveys and censuses that collect birth histories provide a number of U5MR data points, as illustrated in

In this study, we used a parametric bootstrap procedure to generate new datasets

Potential biases in trends and levels of U5MR data series need to be accounted for in the data model for the U5MR, as illustrated in _{i}_{i}_{i}_{i}_{0,s} and slope β_{1,s} for each data series by source type, and for

For countries with data from VR systems, the log-differences δ_{i}

Based on the estimates of mean bias in levels and trends and error variance by source type, as well as the variability of biases across data series, “new” data series were sampled around the current UN IGME estimates in the first step of the bootstrap procedure, after which the loess smoother was fitted to the bootstrapped dataset. Instead of resampling all observed non-VR data series in the country in each bootstrap, one randomly selected data series was left out for countries with at least three data series. The leave-one-out step was motivated by the issue that an included data series could have been of low quality. By leaving out one series at a time at random, the influence of any one series on the resulting curve fit is reduced.

The dataset consisted of 867 data series and 8,336 observations for 174 countries where the UN IGME loess estimation procedure was used. An overview of the number of data series and observations is given in

Source Type | Number of Data Series (Number of Observations) |

VR | 96 (3,209) |

DHS Direct (with reported sampling errors) | 185 (2,580) |

DHS Direct (without reported sampling errors) | 29 (94) |

Others Direct (including MICS and Census Direct) | 118 (355) |

DHS Indirect | 10 (50) |

MICS Indirect | 66 (318) |

Census Indirect | 178 (887) |

Others Indirect | 127 (623) |

Other Source Types | 58 (220) |

Model performance for the default method and the proposed bootstrap method of constructing UIs was assessed based on the UN IGME 2011 dataset

The goal of constructing UIs is to enable and promote evidence-based assessments of progress toward reducing U5MR. We propose to categorize countries based on the UI for their ARR for 1990 to 2011 into five categories, as summarized in

Category | ARR 1990–2011 | Evidence of Progress? | ||

Lower Bound (L) | Upper Bound (U) | Progress in Reducing U5MR? | Progress at ARR of 4.4% or Above? | |

1 | L≤0% | U≥4.4% | Not clear (estimate of ARR is highly uncertain) | |

2 | L≤0% | 0%≤U<4.4% | Not clear | Unlikely |

3 | 0%<L<4.4% | U<4.4% | Likely | Unlikely |

4 | 0%<L<4.4% | U≥4.4% | Likely | Not clear |

5 | L≥4.4% | U≥4.4% | Likely | Likely |

The analysis was carried out in open source software R

Mean biases in U5MR levels and trends, as well as 90% prediction intervals for the expected range of U5MR values for a “new” U5MR data series, given a “true” U5MR of 100 deaths per 1,000 live births, are shown in

For a “true” U5MR of 100 deaths per 1,000 live births (represented by the black line), the 90% prediction interval for a U5MR observation is shown in light blue (for DHS direct series, this excludes the sampling variability), and the predicted mean observed U5MR is represented by the blue vertical line (the difference between the mean U5MR and 100 represents the mean bias). The blue horizontal line represents the 90% prediction interval for an observation based on uncertainty in the bias parameters only (excluding sampling and non-sampling variability). SE, sampling error.

The UIs for all 174 countries are given in

Point estimates and UIs for the U5MR in 1990 (red) and 2011 (black). Within regions, countries are ordered by the point estimate of the U5MR for 2011. Lao PDR, Lao People's Democratic Republic; Korea Rep., Republic of Korea; OPT, Occupied Palestinian Territory; St Vincent & the Gren., Saint Vincent and the Grenadines.

Year | UI Width (U5MR) | UI Width Relative to U5MR (Percent) | ||

Mean | Median | Mean | Median | |

1990 | 21 | 16 | 20 | 19 |

2000 | 21 | 16 | 25 | 24 |

2010 | 31 | 23 | 52 | 48 |

The increase in uncertainty from 1990 to 2011 in the U5MR can be explained by more limited data availability for recent years. For the 86 high mortality countries, out of a total of 4,528 observations used for the UN IGME loess estimation procedure, 1,283 observations have reference dates in the 1980s, and 1,148 observations have reference dates in the 1990s. In contrast, only 483 observations were available since the year 2000. The median extrapolation period (number of years between the last available observation and the year 2011.5) for these countries is 5.2 y.

Within regions, countries are ordered by the lower bound of the UI. Lao PDR, Lao People's Democratic Republic; Korea Rep., Republic of Korea; OPT, Occupied Palestinian Territory; St Vincent & the Gren., Saint Vincent and the Grenadines.

Point estimates (lines), UIs (shaded areas), and data series (connected dots) for Ghana (green) and Equatorial Guinea (red).

The categorization of

Countries are grouped based on the categories from

Based on the upper bounds of the UIs, the countries in categories 2 and 3 are not considered to be on track for meeting MDG 4. For the 31 countries in category 4 (36%), it is not clear whether they are on track for meeting the MDG 4 target or not. This category includes countries such as Suriname, with a point estimate for the ARR of 2.7%, as well as Lao People's Democratic Republic, with a much higher point estimate of 6.0%. The great uncertainty in the ARR in these two countries around the point estimate is explained by the scarcity of the data; in both countries, there are only two observations since 2000. Finally, for the 13 countries in category 5 (15%), there is evidence that the ARR has already exceeded the MDG 4 target of 4.4%. Maldives, the country with the highest point estimate for the ARR, 10.9%, is also the country with the highest lower bound for the ARR, 9.9%. The UI for the Maldives is relatively narrow (compared to other high mortality countries) because of greater data availability; the Maldives dataset includes a number of data series in the past decades, going back as far as the 1960s, and VR data from 2006 to 2011.

Validation measures for the U5MR and ARR (from 1990 to 2005) based on the UN IGME 2011 dataset are summarized in

Indicator | Bootstrap Method | Default Method | ||

Below | Above | Below | Above | |

U5MR 1990 | 3 | 3 | 14 | 10 |

U5MR 2000 | 10 | 6 | 31 | 17 |

U5MR 2005 | 17 | 7 | 37 | 14 |

ARR 1990–2005 | 6 | 16 | 11 | 29 |

We used a bootstrap procedure to construct uncertainty bounds for the U5MR and the ARR for all countries, to provide more information about the evidence (or lack thereof) of countries' progress in reducing the U5MR. Our analysis was based on estimates of biases in levels and trends for different source types, and the variability therein. We found that there is substantial across-survey variation in biases and that half-widths of 90% prediction intervals for “new observations” tend to be at least 20% of the U5MR level for most source types. Ignoring these biases can lead to an underestimate of uncertainty in U5MR. We found that substantial uncertainty exists about U5MR levels and trends for high mortality countries, especially in more recent years; the median relative width of the UI compared to the U5MR level is 48% among 86 high mortality countries for 2011, compared to 19% in 1990. The greater uncertainty for recent years is explained by more limited data availability.

The validation of the UIs for the UN IGME 2011 dataset suggested that bounds that would have been constructed in 2006 based on all available data at that time are a substantial improvement upon the default method. In particular, the default bounds for the ARR that would have been constructed in 2006 would have been problematic (i.e., in categorizing countries as in

We acknowledge that the bootstrapped bounds do not include the uncertainty associated with various steps in the UN IGME fitting procedure and are therefore likely to represent an underestimate of the uncertainty associated with the UN IGME estimates. A full uncertainty assessment is complicated because of set rules and expert adjustments in the UN IGME fitting procedure (the exclusion of outlying data, the rule for setting the span parameter α, and the adjustment of α for selected countries if the fit is deemed inappropriate). Instead of focusing attention and resources on trying to incorporate this uncertainty associated with various steps in the UN IGME fitting procedure into the uncertainty assessment, we favor exploration of alternative methods for estimating child mortality that incorporate an appropriate data model to reduce potential biases in the point estimates as well as the UIs.

The main objective of our proposed method was to assess the uncertainty in countries with high U5MR, where point estimates can be volatile because of great uncertainty. We focused on countries where a standard U5MR estimation method was used. The UN IGME 2012 estimates of child mortality levels and trends

Alternative estimation methods and estimates of levels and trends in U5MR exist. In 2011, the Institute for Health Metrics and Evaluation (IHME) published national and global estimates of U5MR _{10}–transformed U5MR over time are assumed to be realizations of a Gaussian process. These trajectories are constructed based on the assumption that observations from non-VR data series are conditionally independent (loosely interpreted, that the observations are randomly scattered around the U5MR trend line), which is different from our approach, in which we take account of potential biases in levels and trends in data series. A validation exercise of the uncertainty bounds that IHME constructed in 2010 (based on similar fitting methods and similar assumptions about data series) showed that while the IHME's point estimates were not biased, the IHME uncertainty bounds may be too narrow, which is potentially explained by their approach, which does not account for substantial across-survey variation in biases

Country studies have been carried out using estimates of U5MR or ARR based on a single survey

Point estimates on child mortality based on limited information may substantially under- or overestimate the truth. Uncertainty assessments can and should be used to complement point estimates to avoid unwarranted conclusions about levels or trends in child mortality and to reduce confusion about differences in estimates, such as estimates from different groups such as the UN IGME and IHME, or after updating point estimates in light of new data. The new uncertainty assessments provide more insights into countries' progress in reducing child mortality. In particular, a comparison of the lower bound of the UI for the ARR across countries is more informative to pinpoint countries where we are confident that U5MR has declined since 1990, compared to ranking countries by their point estimates of the ARR (which can be highly uncertain, and thus arbitrarily high or low, for countries with limited data). In the coming years, including in 2015, UIs can be used to assess countries' progress toward MDG 4, as illustrated here (

(PDF)

(DOCX)

(PDF)

We thank the members of the UN IGME, its Technical Advisory Group, and Joy Lawn and Mikkel Oestergaard for very helpful discussions, suggestions, and comments. We also thank Danzhen You for assistance with the data and Jennifer Chunn and John Wilmoth for insightful comments and suggestions related to the manuscript.

annual rate of reduction

Demographic and Health Surveys

Institute for Health Metrics and Evaluation

Millennium Development Goal

Multiple Indicator Cluster Surveys

under-five mortality rate

uncertainty interval

United Nations Inter-agency Group for Child Mortality Estimation

vital registration