Advertisement
  • Loading metrics

Guidelines for Accurate and Transparent Health Estimates Reporting: the GATHER statement

  • Gretchen A. Stevens ,

    stevensg@who.int

    Affiliation Department of Information, Evidence and Research, World Health Organization, Geneva, Switzerland

  • Leontine Alkema ,

    Authors listed alphabetically.

    Affiliation Department of Biostatistics and Epidemiology, University of Massachusetts, Amherst, Massachusetts, United States of America

  • Robert E. Black ,

    Authors listed alphabetically.

    Affiliation Department of International Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, United States of America

  • J. Ties Boerma ,

    Authors listed alphabetically.

    Affiliation Department of Information, Evidence and Research, World Health Organization, Geneva, Switzerland

  • Gary S. Collins ,

    Authors listed alphabetically.

    Affiliation Nuffield Department of Orthopaedics, Rheumatology & Musculoskeletal Sciences (NDORMS), University of Oxford, Oxford, United Kingdom

  • Majid Ezzati ,

    Authors listed alphabetically.

    Affiliation Department of Epidemiology and Biostatistics, School of Public Health, Imperial College London, London, United Kingdom

  • John T. Grove ,

    Authors listed alphabetically.

    Affiliation Global Development Program, Bill & Melinda Gates Foundation, Seattle, Washington, United States of America

  • Daniel R. Hogan ,

    Authors listed alphabetically.

    Affiliation Department of Information, Evidence and Research, World Health Organization, Geneva, Switzerland

  • Margaret C. Hogan ,

    Authors listed alphabetically.

    Affiliation Independent consultant, Seattle, Washington, United States of America

  • Richard Horton ,

    Authors listed alphabetically.

    Affiliation The Lancet, London, United Kingdom

  • Joy E. Lawn ,

    Authors listed alphabetically.

    Affiliation Department of Infectious Disease Epidemiology, London School of Hygiene & Tropical Medicine, London, United Kingdom

  • Ana Marušić ,

    Authors listed alphabetically.

    Affiliation Department of Research in Biomedicine and Health and Cochrane Croatia, University of Split School of Medicine, Split, Croatia

  • Colin D. Mathers ,

    Authors listed alphabetically.

    Affiliation Department of Information, Evidence and Research, World Health Organization, Geneva, Switzerland

  • Christopher J. L. Murray ,

    Authors listed alphabetically.

    Affiliation Institute for Health Metrics and Evaluation, University of Washington, Seattle, Washington, United States of America

  • Igor Rudan ,

    Authors listed alphabetically.

    Affiliation Centre for Global Health Research, Usher Institute for Population Health Sciences and Informatics, The University of Edinburgh, Edinburgh, Scotland, United Kingdom

  • Joshua A. Salomon ,

    Authors listed alphabetically.

    Affiliation Department of Global Health and Population, Harvard T.H. Chan School of Public Health, Boston, Massachusetts, United States of America

  • Paul J. Simpson ,

    Authors listed alphabetically.

    Affiliation PLOS Medicine, Cambridge, United Kingdom

  • Theo Vos ,

    Authors listed alphabetically.

    Affiliation Centre for Global Health Research, Usher Institute for Population Health Sciences and Informatics, The University of Edinburgh, Edinburgh, Scotland, United Kingdom

  • Vivian Welch ,

    Authors listed alphabetically.

    Affiliations Bruyére Research Institute, Bruyére Continuing Care, Ottawa, Ontario, Canada, Centre for Global Health, University of Ottawa, Ottawa, Ontario, Canada

  •  [ ... ],
  • The GATHER Working Group

    All authors are members of The GATHER Working Group.

  • [ view all ]
  • [ view less ]

Guidelines for Accurate and Transparent Health Estimates Reporting: the GATHER statement

  • Gretchen A. Stevens, 
  • Leontine Alkema, 
  • Robert E. Black, 
  • J. Ties Boerma, 
  • Gary S. Collins, 
  • Majid Ezzati, 
  • John T. Grove, 
  • Daniel R. Hogan, 
  • Margaret C. Hogan, 
  • Richard Horton
PLOS
x

Correction

9 Aug 2016: Stevens GA, Alkema L, Black RE, Boerma JT, Collins GS, et al. (2016) Correction: Guidelines for Accurate and Transparent Health Estimates Reporting: the GATHER statement. PLOS Medicine 13(8): e1002116. https://doi.org/10.1371/journal.pmed.1002116 View correction

Measurements of health indicators are rarely available for every population and period of interest, and available data may not be comparable. The Guidelines for Accurate and Transparent Health Estimates Reporting (GATHER) define best reporting practices for studies that calculate health estimates for multiple populations (in time or space) using multiple information sources. Health estimates that fall within the scope of GATHER include all quantitative population-level estimates (including global, regional, national, or subnational estimates) of health indicators, including indicators of health status, incidence and prevalence of diseases, injuries, and disability and functioning; and indicators of health determinants, including health behaviours and health exposures. GATHER comprises a checklist of 18 items that are essential for best reporting practice. A more detailed explanation and elaboration document, describing the interpretation and rationale of each reporting item along with examples of good reporting, is available on the GATHER website (gather-statement.org).

Introduction

Global, regional, national, and subnational data for population health indicators are needed to monitor health and to guide resource allocation. However, health data are rarely available for every population and year, and in some cases there are discrepancies in available measurements. Additionally, differences in measurement methods mean that data might not be comparable over time or across populations.

Because of these data gaps and measurement challenges, incomplete data together with statistical or mathematical models are often used to calculate estimates of health indicators. These estimates are used by government officials, non-governmental organizations and funding agencies to make comparisons among populations, to track changes over time—e.g., to monitor progress toward targets such as the Sustainable Development Goals—and to obtain a comprehensive picture of causes of death, burden of disease, or risks to health [1,2]. The available data and analysis methods used to produce estimates often have features or assumptions that affect their interpretation. In recent years, diverse data sources and statistical models of increasing flexibility and sophistication have been used to calculate health estimates. Some have raised questions about whether the data search, access, and inclusion process is sufficiently rigorous [3], and whether users understand the complex methods often used to derive estimates [4]. Others have argued that discrepancies in estimates can lead to confusion—e.g., whether changes were a result of true epidemiological change or of a new method of analysis—and might lead to rejection of estimates [1].

Accurate interpretation and responsible use of health estimates requires understanding of the input data on which estimates were based, including their quality, and of the methods used to derive the estimates from the input data [47]. The need for guidelines for reporting of health estimates was a key conclusion of World Health Organization (WHO) expert meetings in February and December, 2013, which were the impetus for the present set of guidelines [8].

Development of the Guidelines for Accurate and Transparent Health Estimates Reporting

To meet this challenge, the GATHER working group was convened by WHO in 2014, with the aim to define and promote good practice in reporting of global health estimates. The working group’s approach was based on published guidance for developing reporting guidelines [9]. All members of WHO’s Reference Group on Global Health Statistics were invited to join the working group; other experts and journal editors with complementary expertise were sought and invited to join. The working group consists of practitioners, including statisticians, from academia and WHO, journal editors, representatives of the EQUATOR network [10], and members of existing guideline steering groups. The working group reviewed existing reporting guidelines for relevance to global health estimates and sought guidance from experts who had previously developed reporting guidelines. The group determined that existing reporting guidelines would not ensure adequate reporting of global health estimates.

On the basis of the review of existing guidance and reporting guidelines [1116] and of input from working group members, we generated a comprehensive list of potential reporting items. We subsequently sought feedback from a broader community of researchers and users of estimates through an online survey between January and February, 2015. Working group members distributed the survey to their respective networks. We received 118 responses (further details are available on the GATHER website: gather-statement.org). The responses were compiled, summarized and presented at a 2-day consensus meeting held in London, UK, in February, 2015.

The primary objective of the working group consensus meeting was to agree on the list of items that should be reported whenever health estimates are published. During the meeting, reporting items were evaluated in light of the responses to the online survey, and working group members agreed to retain, omit, or combine items to generate the checklist in Table 1.

thumbnail
Table 1. GATHER checklist of information that should be included in reports of global health estimates.

https://doi.org/10.1371/journal.pmed.1002056.t001

The GATHER working group and the responses to the online survey, both drawn from our networks of collaborators, were dominated by residents of high-income countries. We therefore sought additional feedback from a geographically diverse group of stakeholders—including 130 country focal points for WHO mortality estimates—by sharing an earlier version of this statement before publication. We revised this statement based on the feedback received.

Aim of GATHER

GATHER aims to define and promote good practice in reporting of global health estimates. Reporting of estimates should serve the needs of their two primary audiences: decision makers and researchers. Decision makers include planners, policy makers, and monitoring staff in governments, as well as global, regional, and national public health experts, funding agencies, and civil society organizations. These users need information about data sources and analysis methods, including key assumptions and limitations, in a way that is accessible without advanced training in statistics. They also need an explanation of how new estimates compare to previously published estimates, including why they differ. Researchers require a higher degree of detail about methods, so that they can fully understand and potentially reproduce studies and advance methods. The GATHER checklist includes only the minimum essential reporting items to serve these audiences; other good practices in reporting are recommended in the accompanying explanation and elaboration document available on the GATHER website.

Compliance with GATHER is not an indicator of a study’s quality [12,17,18]. Rather, it ensures that key information is available so that an informed researcher can judge the study’s quality and increases the chance that the study results will be used appropriately by decision makers. Improvements in reporting may incidentally improve quality, because the reporting required for compliance with GATHER could assist analysts in identifying errors or improving methods.

Scope of GATHER

GATHER defines best practices for documenting studies that report global health estimates. Global health estimates include all quantitative population-level estimates (including global, regional, national, or subnational estimates) of health indicators, including indicators of health status such as estimates of total and cause-specific mortality, incidence and prevalence of diseases, injuries, and disability and functioning; and indicators of health determinants, including health behaviours and health exposures (Box 1).

Box 1: Definitions of technical terms.

Health indicator: A measureable quantity that can be used to describe a population’s health or its determinants. Indicators can be categorized into four domains: health status (e.g., life expectancy, HIV prevalence), risk factors (e.g., childhood stunting, prevalence of smoking), service coverage (e.g., immunization coverage rate), or health systems (e.g., hospital bed density, death registration coverage). [19]

Health estimates: Quantitative population-level estimates (including global, regional, national, or subnational estimates) of health indicators, including indicators of health status such as estimates of total and cause-specific mortality, incidence and prevalence of diseases, injuries, and disability and functioning; and indicators of health determinants, including health behaviours and health exposures. Examples of health indicators that fall within the scope of GATHER include life expectancy, disability-adjusted life-years by cause, under-five mortality rate, maternal mortality ratio, mortality rate from road traffic injuries, HIV incidence, prevalence of stunting in children younger than 5 years, prevalence of current tobacco use, prevalence of obesity in adults, and condom use among sex workers.

Data inputs: All numerical inputs to mathematical or statistical models that are used to generate global health estimates. Model inputs may include raw health data, processed health data, covariates, and other parameters. Raw health data are measures derived from primary data collection with no adjustments or corrections. Processed health data are health statistics that have been calculated from raw health data, but which are not the result of synthesizing multiple data sources. Examples of processing raw health data include cleaning data by removing implausible values, calculating an indicator with an algorithm, or adjusting a statistic for bias.

Covariates: Data, including non-health data, which are used in a statistical model to improve the estimation of the health indicator of interest. These data are population-specific and are available for every population included in the analysis. A common covariate is gross domestic product per capita.

GATHER aims to define best practices for reporting of studies that synthesize information from multiple sources to quantitatively describe past and current population health and its determinants. These studies include comparisons among multiple populations, over time or by place of residence. GATHER covers reporting of studies that disaggregate disease and injuries by underlying cause as defined by a classification system such as the International Classification of Disease (ICD) as well as those that attribute disease and injury to their determinants, e.g., the number of deaths attributable to tobacco smoking. These reporting guidelines were not designed for reports of a health indicator from a single study or data source, such as a health survey or health service records for a single period.

Health determinants can range from proximal determinants of health, such as behaviours like tobacco smoking that have a direct effect on incidence of disease and mortality, to intermediate determinants of health, such as availability of essential medicines, to distal determinants of population health, such as wealth inequality. Of the universe of health determinants, these reporting guidelines were developed for estimates of health behaviours and health exposures [19]. They were not designed for service coverage indicators, nor were they designed for health systems indicators, such as those related to health financing or health workforce. The guidelines were also not designed for estimates of distal determinants of health, such as average educational attainment or wealth inequality. Nevertheless, researchers preparing health estimates that do not fall in the scope of GATHER might find GATHER useful when documenting their study. In particular, a commitment to documenting all data inputs and analysis methods should be a universal feature of published reports providing estimates designed for policy planning.

Overview of GATHER

GATHER comprises a checklist of 18 items that are essential for best reporting practice (Table 1). An electronic version of the checklist and a more detailed explanation and elaboration document, describing the interpretation and rationale of each reporting item along with examples of good reporting, are available on the GATHER website.

Global health estimates are regularly published in scientific journals and in reports of intergovernmental agencies and non-governmental organizations. The GATHER checklist is designed to be flexible enough to be used for both types of publication. The items in the checklist are organized into four sections: (1) objectives and funding, (2) data inputs, (3) data analysis, and (4) results and discussion. Data inputs are further disaggregated into two groups: data inputs that were synthesized as part of the study (usually the health indicator being estimated), and data inputs from another source or study that contributed to the analysis, but were used without modification (if any; common data inputs of this type are population data or covariates such as average educational attainment or per capita gross domestic product). Methods of data analysis range from a simple averaging of available data to computationally intensive multi-step processes that cannot be run on a standard desktop computer. The reporting items described here are appropriate for all data analysis methods, regardless of their complexity. Importantly, any method of synthesizing available data to make estimates for a population relies on a model and should be reported accordingly.

In most cases, full reporting of a new set of estimates will not be possible in the main text of an article or a report. Rather, authors will have to make use of online appendices to ensure complete reporting as prescribed by the GATHER Statement. Whether the required materials appear in the main text or in an appendix will depend on the purpose and audience of the report, and we therefore leave this decision to the authors’ and editors’ discretion.

Implications and limitations

We propose the GATHER checklist as a tool to be used by authors, reviewers, and journal editors, in order to promote best practices in reporting global health estimates. In this statement, we have presented the development, aim, and an overview of the guidelines. Users of the guidelines should refer to the GATHER website for further explanation and examples of good reporting for specific items.

GATHER considers open access to data inputs and access to analytical or statistical source code to be best practice in reporting. Recent reports on waste in research have highlighted that full documentation of research, including protocols for sharing data and code, increase the value of research that is undertaken [20,21]. Funding agencies [22,23] and journals [24,25] are increasingly requiring that researchers make input data and, in some cases, source code available. In line with these requirements, GATHER considers that data underlying health estimates should be accessible online, except in situations, such as third party ownership, when this is not possible. We nonetheless acknowledge that requiring open access to data inputs might require additional resources for documentation and archiving of data resources.

Sharing source code also involves an investment of resources, especially if the code is fully documented and available online for off-the-shelf use. Sharing code often leads to requests for technical assistance from users, which are time-consuming and typically unfunded. Despite these challenges, in view of the use of global health estimates for policy prioritization and funding allocation, we consider availability of code to be essential. Given that researchers are not necessarily resourced for sharing code, we consider that a minimum would be for researchers to share key segments of code and that they should not be held responsible for providing user support. Moving forwards, we hope that funding agencies and researchers will consider open access to data and code to be an integral part of any project, and that future studies will be planned and funded accordingly.

GATHER also requires that authors report a quantitative measure of the uncertainty associated with global health estimates, such as uncertainty intervals. Global health estimates are usually affected by multiple sources of error, such as measurement error during data collection, inability to register all cases or obtain a truly random sample, errors in adjusting input data for sources of bias, and the use of a model to calculate estimates [26,27]. Users of these estimates should be informed about their overall uncertainty. Best practices for calculation of uncertainty intervals, and especially for combining multiple sources of uncertainty, are an area of active research. By requiring that researchers report a quantitative measure of uncertainty, and that they state which sources of uncertainty are accounted for, we aim to advance science in this area.

The field of global health estimates is rapidly evolving because of increasing availability of health data and innovation in statistical methods. The reporting guidelines presented here are designed to be flexible enough to guide reporting of estimates regardless of the underlying data availability and the complexity of the statistical methods. We anticipate that, as experience with these guidelines accumulates, methods and data evolve, and suggestions for improvements are made, GATHER will evolve as well. The explanation and elaboration document, available on the GATHER website, will be a living document that will be updated and clarified as needed, based on accumulated experience using the guidelines. We encourage submission of users’ comments via our website.

Acknowledgments

We acknowledge the thoughtful comments provided by 118 anonymous survey respondents; by the participants at WHO’s meeting on country use of health estimates, in June, 2015, Glion sur Montreux; and by Monika Bloessner, Mia Cokljat, George Davey-Smith, Shah Ebrahim, Simon I. Hay, Susan Norris, Shefali Oza, Christopher Paciorek, Juan Pablo Peña Rosas, and Iveta Simera.

Author Contributions

Set the process for developing the guideline: GAS LA REB JTB GSC ME JTG DRH MCH RH JEL AM CDM CJLM IR JS PJS TV VW. Developed the comprehensive list of reporting items and conducted the online survey: GAS, with inputs from all authors. Compiled the survey results, prepared the background materials for the 2015 meeting in London, and planned the meeting: GAS DRH. Attended the meeting: LA REB GSC ME JTG DRH RH JEL AM CDM IR PJS GAS TV VW. Wrote the first draft of the manuscript: GAS DRH. Wrote the first draft of the Explanation and Elaboration document: MCH GAS. Contributed to the selection and wording of the reporting items, the writing of the manuscript, and the writing of the Explanation and Elaboration document: GAS LA REB JTB GSC ME JTG DRH MCH RH JEL AM CDM CJLM IR JS PJS TV VW. Conducted the second round of consultation: GAS. Agree with the manuscript’s results and conclusions: GAS LA REB JTB GSC ME JTG DRH MCH RH JEL AM CDM CJLM IR JS PJS TV VW. All authors have read, and confirm that they meet, ICMJE criteria for authorship.

References

  1. 1. Graham WJ, Adjei S (2010) A call for responsible estimation of global health. PLoS Med 7: e1001003. pmid:21151346
  2. 2. Boerma T, Mathers CD (2015) The World Health Organization and global health estimates: improving collaboration and capacity. BMC Med 13: 50. pmid:25858025
  3. 3. Hambleton IR, Howitt C, Rose AM, Samuels TA, Unwin N (2015) Global trends in dietary quality. Lancet Glob Health 3: e593. pmid:26385296
  4. 4. Byass P (2010) The imperfect world of global health estimates. PLoS Med 7: e1001006. pmid:21152416
  5. 5. Walker N, Bryce J, Black RE (2007) Interpreting health statistics for policymaking: the story behind the headlines. Lancet 369: 956–963. pmid:17368157
  6. 6. Murray CJ (2007) Towards good practice for health statistics: lessons from the Millennium Development Goal health indicators. Lancet 369: 862–873. pmid:17350457
  7. 7. Atun R (2014) Time for a revolution in reporting of global health data. Lancet. 384: 937–938. pmid:25059946
  8. 8. World Health Organization (2013) Global Health Estimates: Proposals on the way forward. Summary of a Technical Meeting, WHO, Geneva, 13–14 February 2013.: World Health Organization.
  9. 9. Moher D, Schulz KF, Simera I, Altman DG (2010) Guidance for developers of health research reporting guidelines. PLoS Med 7: e1000217. pmid:20169112
  10. 10. Altman DG, Simera I, Hoey J, Moher D, Schulz K (2008) EQUATOR: reporting guidelines for health research. Lancet 371: 1149–1150. pmid:18395566
  11. 11. Garnett GP, Cousens S, Hallett TB, Steketee R, Walker N (2011) Mathematical models in the evaluation of health programmes. Lancet 378: 515–525. pmid:21481448
  12. 12. Moher D, Liberati A, Tetzlaff J, Altman DG, Group P (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 6: e1000097. pmid:19621072
  13. 13. Lang TA, Altman DG (2013) Basic statistical reporting for articles published in clinical medical journals: the SAMPL Guidelines. In: Smart P, Maisonneuve H, editors. Science Editors' Handbook: European Association of Science Editors.
  14. 14. Eddy DM, Hollingworth W, Caro JJ, Tsevat J, McDonald KM, et al. (2012) Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—7. Value Health 15: 843–850. pmid:22999134
  15. 15. Husereau D, Drummond M, Petrou S, Carswell C, Moher D, et al. (2013) Consolidated Health Economic Evaluation Reporting Standards (CHEERS)—explanation and elaboration: a report of the ISPOR Health Economic Evaluation Publication Guidelines Good Reporting Practices Task Force. Value Health 16: 231–250. pmid:23538175
  16. 16. Boerma JT, Mathers C, Abou-Zahr C (2010) WHO and global health monitoring: the way forward. PLoS Med 7: e1000373. pmid:21151348
  17. 17. von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, et al. (2007) The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Bull World Health Organ 85: 867–872. pmid:18038077
  18. 18. Schulz KF, Altman DG, Moher D, Group C (2010) CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. PLoS Med 7: e1000251. pmid:20352064
  19. 19. World Health Organization (2014) Global Reference List of 100 Core Health Indicators: Working Version 5. Geneva: World Health Organization. 20 p.
  20. 20. Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, et al. (2014) Reducing waste from incomplete or unusable reports of biomedical research. Lancet 383: 267–276. pmid:24411647
  21. 21. Chan AW, Song F, Vickers A, Jefferson T, Dickersin K, et al. (2014) Increasing value and reducing waste: addressing inaccessible research. Lancet 383: 257–266. pmid:24411650
  22. 22. Walport M, Brest P. (2011) Sharing research data to improve public health. Lancet 377: 537–539. pmid:21216456
  23. 23. Bill & Melinda Gates Foundation (2015) Bill & Melinda Gates Foundation Open Access Policy. Bill & Melinda Gates Foundation. http://www.gatesfoundation.org/how-wework/general-information/open-access-policy. Accessed March 18, 2015.
  24. 24. Bloom T, Ganley E, Winker M (2013) Data access for the open access literature: PLOS’s data policy. Dec 12, 2013. http://www.plos.org/data-access-for-the-open-access-literature-ploss-data-policy/. Accessed March 18, 2015.
  25. 25. Nature Publishing Group (2014) Availability of data, materials and methods. Nature Publishing Group. http://www.nature.com/authors/policies/availability.html. Accessed March 18, 2015.
  26. 26. Murray CJ, Frenk J (2008) Health metrics and evaluation: strengthening the science. Lancet 371: 1191–1199. pmid:18395581
  27. 27. Mathers CD, Salomon JA, Ezzati M, Begg S, Vander Hoorn S, et al. (2006) Sensitivity and uncertainty analyses for burden of disease and risk factor estimates. Global burden of disease and risk factors. New York: Oxford University Press. pp. 399–426.