Citation: Murray CJL, Lopez AD (2010) Production and Analysis of Health Indicators: The Role of Academia. PLoS Med 7(11): e1001004. https://doi.org/10.1371/journal.pmed.1001004
Published: November 30, 2010
Copyright: © 2010 Murray, Lopez. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: No specific funding was received for this study; however, the Institute for Health Metrics and Evaluation is funded by a grant from the Bill & Melinda Gates Foundation. No funding bodies had any role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: CJLM works in academia generating health indicator estimates; is the Director of the Institute for Health Metrics and Evaluation; is a Professor of Global Health at the University of Washington; and from 1998 to 2003 worked at the World Health Organization in Geneva, Switzerland. ADL works in academia generating health indicator estimates; is a member of the editorial board for PLoS Medicine; is Head of the School of Population Health and Professor of Global Health at the University of Queensland; is an Affiliate Professor of Global Health at the Institute for Health Metrics and Evaluation; and from 1980 to 2002 worked at the World Health Organisation in Geneva, Switzerland in a series of technical and senior managerial posts.
Abbreviations: WHO, World Health Organization
Provenance: Commissioned; not externally peer reviewed. The idea for a commissioned cluster of articles on global health estimates came from Ties Boerma and Colin Mathers at the World Health Organization.
- Progress on the timely, valid, reliable, and comparable measurement of health indicators requires tools and instrument innovation; enhanced national capacity to collect, process, archive, and share data; norms and standards for indicator definitions and computation; multiple independent analyses of data; and effective translation of results into policy.
- Academia and the associated process of peer-reviewed publication and open debate should play the central role in the creation of new tools and instruments and in the rigorous analysis of existing data to yield the best possible assessments of key health indicators.
- The United Nations and World Bank have important roles to play in helping countries build capacity for data collection and analysis, setting norms and standards, and helping national decision-makers translate evidence into new policy directions.
Evolving Roles for the State, Supra-National Bodies, and Science
The collection, computation, and publication of health indicators was for much of the 19th century and the first half of the 20th century the business of states—reflecting the etymology of the word “statistics” (from Statistik, or “of the state”). With the spread of independence in the developing world in the second half of the 20th century, the United Nations (UN) began to play a supra-national role in the production of health indicators to bolster weak capacity in some developing countries. Yet, over the past five or six decades, the expectations for the field have expanded considerably. New constructs have been introduced such as cause-specific mortality, quality of care, and functional health status. Measurements must often deal with the simultaneous problems of synthesizing conflicting sources, missing data, instrument bias, substantial non-sampling error in surveys, and missing denominators, particularly in administrative data. Reflecting these complexities, in the last 30 years, the role of the academy has expanded from tools and methods innovation to the production of comparable measurements for key health indicators in various countries  and globally ,. Academic demographers have been involved in the production of comparable population measurements for nearly 50 years . In the last two decades, epidemiology and other health-related disciplines have also been actively engaged in population-level health measurement .
Reflecting the progressive diversification of actors in the production of health indicators, alternative statistics are now available from national authorities, from various UN agencies, and from the peer-reviewed scientific literature –. Alternative measurements have engendered considerable concern from health advocacy groups and practitioners who rely on sound measurement to guide programs, and from donor agencies. With the 2015 target looming, interest in reliably monitoring the progress of countries in attaining (or not) the UN Millennium Development Goals has intensified. In this commentary, we consider the potential roles of different actors and highlight four key issues that are widely debated.
Key Ingredients for Sound Measurement and Policy Translation
There are five distinct areas that contribute to the ultimate production of timely, comparable, valid, and reliable health indicator measurements and their translation into national policy dialogue.
Tools and Instrument Innovation
The set of tools available for data collection and analysis is rapidly changing. Advances in computing mean that innovations such as electronic medical records or Bayesian measurement models are now feasible even in low-resource settings. Most of the innovation in this area will continue to come from the private sector (including non-governmental organizations) and academia. The role for national statistical authorities or UN and World Bank institutions will likely be restricted to agenda promotion, advocacy for application, and financing.
Capacity to Collect, Process, Archive, and Share Data
Whether in a developed or developing country, good health measurement requires capacity in national statistical offices and Ministries of Health to collect administrative data on case reporting, service delivery and expenditures, and household data through surveys and censuses. Capacity to collect, process, and archive is both human and physical. Our view is that academia has an important role in providing appropriate training and education; national governments a critical role in nurturing this capacity at national, regional, and local levels; donor agencies a role in helping low-income countries finance capacity development; and the UN system a role in advocating for capacity development and helping broker financial and technical assistance. The success of the United Nations Children's Fund (UNICEF) in raising funding for the Multiple Indicator Cluster Surveys is a clear example where this role has been extremely beneficial. Harnessing the capability of scientists within countries and around the world for data analysis requires archiving and sharing data in a way that protects confidentiality but stimulates innovation.
Norms and Standards for Health Indicators
Given the rapid innovation in tools and instruments and the expanding set of constructs for measurement, comparable measurement requires a periodically revised set of nomenclatures, definitions, and data standards. Non-governmental organizations or public–private partnerships have been effective in promulgating some standards such as Health Level Seven (HL7) or Systematized Nomenclature of Medicine—Clinical Terms (SNOMED CT). Nevertheless, there is a key role for supra-national bodies such as the World Health Organization (WHO), given its governance mandate, in setting universal standards. WHO has successfully stewarded the International Classification of Diseases for over 60 years and should continue to be a leader in setting a wide range of health norms and standards.
Analysis of Data
It is the nature of health data that estimation of health indicators often requires more than data processing. Measurements from different sources may conflict, data for some localities or years may be missing, survey instruments may be biased, administrative data systems may exclude groups such as the poor, and so on. After processing, data need to be analyzed with the best available scientific methods. Analysis is required both for the production of comparable, valid, and reliable measurements as well as for the more difficult task of interpretation or causal inference. National statistical authorities, academics, and UN institutions all undertake data analysis. Given the myriad data challenges outlined above, there is considerable scope for reasonable scientists to disagree on the interpretation of data, whether for descriptive or causal reasons. Given that the task is fundamentally a scientific challenge, we believe that it should primarily be the role of independent scientists in academic institutions, independent of the agencies or programs that collect or collate the data. Additionally, the discipline of peer review, although imperfect, brings a transparency and objectivity to the analysis of data.
Sound measurement of health indicators should serve as input to a more informed national health policy debate. The gap between science and policy formulation can be large. Through their trusted relationship with governments, UN and World Bank institutions can play a key role in policy translation. In some countries, independent think tanks, academics, and national academies of science can also play a key role in the translation of empirical findings into policy options.
Four Issues in the Debate
1. Estimates or Country Reporting?
The debate over the best measurements for child and maternal mortality is sometimes framed as a debate on estimates versus country reports . Proponents for the latter appear to believe that “estimates” are inherently undesirable and fraught with uncertainty, while country reports are inherently valid, reflecting the reality on the ground. This debate masks two different issues. First, in 2010, all sound measurements are in some way “estimates” in that appropriate statistical methods have been applied to the primary processed data to yield a measurement. For example, in a high-income country like the United States, measurement of the maternal mortality rate must deal with misclassification of maternal deaths to other causes, census counts need to be adjusted for under-enumeration, and smoking rates from the Behavioral Risk Factor Surveillance System must be adjusted for nonresponse bias by age, sex, and race.
The second issue is one of political provenance. For some health indicators, it is awkward for some UN agencies to disagree with national authorities. This debate is code speak for avoiding disagreeing with Ministries of Health or other government authorities by publishing government figures regardless of the quality of data or the rigor of the analysis applied to the data. The real issue is whether we want science-based measurement or political measurement. The world needs the former; there really is no role for the latter.
2. Multiple Figures Will Confuse Donor Country Agencies and Legislators and Decision-Makers in Low-Income Countries
Some advocacy groups believe that public debate over the magnitude and trend of key health indicators can undermine support for health programs among donors and will confuse leaders in developing countries. For example, in the maternal mortality debate, one advocate argued that scientists should be locked in a room until they agree on one set of numbers. We believe the drive to artificial consensus is fundamentally misguided. Health leaders in developing countries have the sophistication and experience to understand that measurement is challenging and that there will be uncertainty in the assessment of levels and trends. The recent unfolding of “Climategate” illustrates how forced consensus can undermine public confidence in an important agenda . Rather than forcing a false consensus among scientists, we believe that it is the responsibility of the scientific community to encourage robust public discourse on critical health indicators; such dialogue will lead to both a better understanding of the health challenges nations face and an appreciation of the realities of uncertainty in measurement. It ought also to accelerate efforts in countries to improve critical health information systems.
3. The UN as Player and Referee in the Scientific Arena
Some excellent scientists work for the UN and its technical agencies. In their role as scientists they sometimes analyze data and produce measurements for health indicators. Occasionally, this work is subject to the normal process of peer review, but more often it is published under the imprimatur of the UN. The global public health and development communities need the UN, especially WHO, to assume the leading role in setting norms and standards for indicators. It is difficult to see how the UN can be the trusted developer of norms and standards and a respected neutral broker when it is also trying to compete in the health measurement arena . Inevitably, this creates a conflict of interest, often leading to UN agencies promoting their own estimates using their imprimatur, depriving countries and other users of the advantages of scientific exchange and debate. In our view, these efforts progressively undermine the UN's capacity to set standards in the long term. The UN can, and we believe should, use their effective relationships with governments to promote the translation of scientific findings into more effective national policy formulation. Can one clearly distinguish data analysis to produce timely, valid, and reliable health indicator measurements from analyses to support policy translation in a given country? One useful guide to the dividing line is that when there is a need for innovation in methods or application of methods, the role for academia is more compelling. Timely application of existing analytical tools to inform policy as part of an ongoing institutional dialogue may be more effectively delivered by the UN institutions, national think tanks, or consulting organizations.
4. Good Practice Guidelines
The rise of low-cost computational power has already had a far-reaching impact on the analysis of data to produce comparable, valid, and reliable measurements of health indicators. An impressive array of computationally intensive methods has now been applied in health , with further innovation and methodological advances on the horizon. Given the methods innovation and the increasingly diverse set of scientists drawn to the challenge of health measurement, it is becoming increasingly important to create a set of guidelines for health indicator studies. These guidelines should cover key areas such as the definition of a systematic analysis using all available data, requirements for transparency on data and methods, necessary tests of predictive validity, calculation of uncertainty intervals based on sound scientific principles, and attempts to adjust data for known bias.
We believe that the diversification of actors involved in health indicator measurement over the last 30 years or so has been tremendously positive. The expanding role of academics in health measurement, whether at the national or global level, is helping to shift practice away from a political state-based model of health measurement to a science-based model. As analytical capacity in academic institutions in developing countries continues to expand, more national and regional comparative studies will emerge from scientists at these institutions. This is a most welcome development, and efforts to strengthen analytical capacity must remain a priority . There are important roles for states, inter-governmental organizations, donor agencies, the private sector, and academia. By harnessing the extraordinary power of innovation in the scientific world, decision-makers in all countries will benefit from more timely, comparable, valid, and reliable measurements of the health constructs that they seek to influence.
ICMJE criteria for authorship read and met: CJLM ADL. Agree with the manuscript's results and conclusions: CJLM ADL. Wrote the first draft of the paper: CJLM ADL. Contributed to the writing of the paper: CJLM ADL.
- 1. Masanja H, de Savigny D, Smithson P, Schellenberg J, John T, et al. (2008) Child survival gains in Tanzania: analysis of data from demographic and health surveys. Lancet 371: 1276–1283.H. MasanjaD. de SavignyP. SmithsonJ. SchellenbergT. John2008Child survival gains in Tanzania: analysis of data from demographic and health surveys.Lancet37112761283
- 2. Noor AM, Mutheu JJ, Tatem AJ, Hay SI, Snow RW (2009) Insecticide-treated net coverage in Africa: mapping progress in 2000–07. Lancet 373: 58–67.AM NoorJJ MutheuAJ TatemSI HayRW Snow2009Insecticide-treated net coverage in Africa: mapping progress in 2000–07.Lancet3735867
- 3. Murray CJ, Lopez AD (1997) Global mortality, disability, and the contribution of risk factors: Global Burden of Disease Study. Lancet 349: 1436–1442.CJ MurrayAD Lopez1997Global mortality, disability, and the contribution of risk factors: Global Burden of Disease Study.Lancet34914361442
- 4. Brass W, Princeton University Office of Population Research (1968) The demography of tropical Africa. Princeton (New Jersey): Princeton University Press. W. BrassPrinceton University Office of Population Research1968The demography of tropical AfricaPrinceton (New Jersey)Princeton University Press
- 5. Ezzati M, Vander Hoorn S, Lawes CMM, Leach R, James WPT, et al. (2005) Rethinking the “diseases of affluence” paradigm: global patterns of nutritional risks in relation to economic development. PLoS Med 2: e133.M. EzzatiS. Vander HoornCMM LawesR. LeachWPT James2005Rethinking the “diseases of affluence” paradigm: global patterns of nutritional risks in relation to economic development.PLoS Med2e133
- 6. Hogan MC, Foreman KJ, Naghavi M, Ahn SY, Wang M, et al. (2010) Maternal mortality for 181 countries, 1980–2008: a systematic analysis of progress towards Millennium Development Goal 5. Lancet 375: 1609–1623.MC HoganKJ ForemanM. NaghaviSY AhnM. Wang2010Maternal mortality for 181 countries, 1980–2008: a systematic analysis of progress towards Millennium Development Goal 5.Lancet37516091623
- 7. Rajaratnam JK, Marcus JR, Levin-Rector A, Chalupka AN, Wang H, et al. (2010) Worldwide mortality in men and women aged 15–59 years from 1970 to 2010: a systematic analysis. Lancet 375: 1704–1720.JK RajaratnamJR MarcusA. Levin-RectorAN ChalupkaH. Wang2010Worldwide mortality in men and women aged 15–59 years from 1970 to 2010: a systematic analysis.Lancet37517041720
- 8. Rajaratnam JK, Marcus JR, Flaxman AD, Wang H, Levin-Rector A, et al. (2010) Neonatal, postneonatal, childhood, and under-5 mortality for 187 countries, 1970–2010: a systematic analysis of progress towards Millennium Development Goal 4. Lancet 375: 1988–2008.JK RajaratnamJR MarcusAD FlaxmanH. WangA. Levin-Rector2010Neonatal, postneonatal, childhood, and under-5 mortality for 187 countries, 1970–2010: a systematic analysis of progress towards Millennium Development Goal 4.Lancet37519882008
- 9. Flaxman AD, Fullman N, Otten MW, Menon M, Cibulskis RE, et al. (2010) Rapid scaling up of insecticide-treated bed net coverage in Africa and its relationship with development assistance for health: a systematic synthesis of supply, distribution, and household survey data. PLoS Med 7: e1000328.AD FlaxmanN. FullmanMW OttenM. MenonRE Cibulskis2010Rapid scaling up of insecticide-treated bed net coverage in Africa and its relationship with development assistance for health: a systematic synthesis of supply, distribution, and household survey data.PLoS Med7e1000328
- 10. Lu C, Schneider MT, Gubbins P, Leach-Kemon K, Jamison D, et al. (2010) Public financing of health in developing countries: a cross-national systematic analysis. Lancet 375: 1375–1387.C. LuMT SchneiderP. GubbinsK. Leach-KemonD. Jamison2010Public financing of health in developing countries: a cross-national systematic analysis.Lancet37513751387
- 11. Horton R (2010) Maternal mortality: surprise, hope, and urgent action. Lancet 375: 1581–1582.R. Horton2010Maternal mortality: surprise, hope, and urgent action.Lancet37515811582
- 12. Committee to Review the IPCC (2010) Climate change assessments: review of the processes and procedures of the IPCC [prepublication copy—subject to further editorial revision]. Committee to Review the IPCC2010Climate change assessments: review of the processes and procedures of the IPCC [prepublication copy—subject to further editorial revision].Available: http://reviewipcc.interacademycouncil.net/report.html. Accessed 20 September 2010. Available: http://reviewipcc.interacademycouncil.net/report.html. Accessed 20 September 2010.
- 13. Murray CJL, Lopez AD, Wibulpolprasert S (2004) Monitoring global health: time for new solutions. BMJ 329: 1096–1100.CJL MurrayAD LopezS. Wibulpolprasert2004Monitoring global health: time for new solutions.BMJ32910961100
- 14. Guerra CA, Gikandi PW, Tatem AJ, Noor AM, Smith DL, et al. (2008) The limits and intensity of Plasmodium falciparum transmission: implications for malaria control and elimination worldwide. PLoS Med 5: e38.CA GuerraPW GikandiAJ TatemAM NoorDL Smith2008The limits and intensity of Plasmodium falciparum transmission: implications for malaria control and elimination worldwide.PLoS Med5e38
- 15. Lopez AD (2010) Sharing data for public health: where is the vision? Bull World Health Organ 88: 467–467.AD Lopez2010Sharing data for public health: where is the vision?Bull World Health Organ88467467