Skip to main content
  • Loading metrics

Different scientific approaches are needed to generate stronger evidence for population health improvement

The challenge of evaluating low-agency population interventions

Over the last 30 years we have witnessed a growing interest in population approaches to prevention of non-communicable diseases (NCDs) among both policymakers and researchers. The population approach to prevention, where interventions are delivered to all, irrespective of baseline risk, is generally considered more effective than the ‘high-risk’ approach, where interventions are targeted at those at high baseline risk of diseases [1,2]. Furthermore, low-agency population interventions, which place low demands on recipients’ personal resources, are likely to be the most effective and equitable approach to prevention [2]. Examples of low-agency population interventions include taxes on unhealthy commodities (e.g., tobacco, alcohol, gambling, and some foods), regulation of the marketing of these commodities, and structural interventions to make healthier choices easier (e.g., cycle lanes and mandatory installation of seat belts in cars).

The evaluation of low-agency population approaches to prevention presents particular challenges. The nature of these interventions means that they are generally designed and implemented by large organisations such as governments, rather than by researchers. This means that issues of evaluation are often not paramount. By definition, population interventions are delivered to whole populations, meaning that a concurrent control group may not always be obvious. For example, when new national restrictions on food marketing to children are introduced, the only possible concurrent control group is another country—which may differ substantially from the intervention country in ways that undermine its utility as a control group. The lack of an appropriate control also precludes randomisation, even at the group level. In these cases, observational designs making best use of routinely available data and treating the intervention as a ‘natural experiment’ are often the best available evaluative approaches [3].

The challenge of multiple possibilities

Two examples of evaluations of low-agency population interventions recently published in PLOS Medicine, both focussing on the same fiscal policy, illustrate some of the challenges of this approach. Ryota Nakamura [4] and Juan Carlos Caro [5] and their respective colleagues both evaluated a revision to the taxation of sugar-sweetened beverages (SSBs) in Chile, introduced in October 2014, using retrospective, natural experimental designs. The tax reform resulted in an increase in the taxation of SSBs containing at least 6.25 mg/ml of sugar (from 13% to 18%) and a reduction in taxation of SSBs containing less than 6.25 mg/ml of sugar (from 13% to 10%). Both studies used a similar approach to evaluation; both approached the evaluation from an economic perspective, underpinned by theory relating to price elasticities of demand. Both used the same main data source (from a consumer panel representing a stratified random sample of households in urban areas of Chile). But each made different choices in how the data were analysed. Key similarities and differences between the two studies are summarised in Table 1. The two studies came to similar conclusions concerning the overall effectiveness of the policy (weak) and its distributional impacts (more effective in higher socio-economic groups), but the details of their findings differed widely in magnitude and, in some cases, direction. For example, Nakamura et al. found an overall 5.8% decrease in volume (millilitres) of SSBs purchased following introduction of the tax change, whereas Caro et al. found a 1.9% decrease. The decrease was 21.6% for high tax/sugar drinks in Nakamura et al., but only 3.4% in Caro et al.

Table 1. Key differences in study design, methods, and main findings between Nakamura et al. [4] and Caro et al. [5].

It is not obviously the case that any of the design choices made by either Nakamura et al. [4] or Caro et al. [5] were wrong. They are just different. This illustrates well the challenge of synthesising findings from across evaluations of low-agency population interventions. Alongside researchers’ justifiably choosing a variety of different evaluative designs and analytical strategies, interventions that appear similar may also vary in detail. For example, a wide range of different approaches to SSB taxation have now been implemented internationally [6,7]. In addition, contextual factors are likely to lead to similar interventions achieving different effects in different settings. In these situations, meta-analysis is likely to be challenging; only outcomes common to studies can be analysed and important factors that may influence outcomes will be overlooked. Other, more conceptual, approaches to synthesis may be more appropriate [8].

Disconnected efforts to generate evidence on public health challenges can also be inefficient. In the case of the Chilean SSB tax evaluations, two teams with funding from two different countries led studies in parallel. With greater coordination (and, without doubt, no small measure of imagination and political will), the funding could have been pooled, enabling a wider range of questions to be answered instead of the same set twice. Whilst pragmatic evidence on the effect of SSB taxes on price and sales is now beginning to accumulate [914], fewer attempts appear to have been made to investigate wider questions on, for example, reformulation, total diet, health outcomes over different time frames, and the political processes that facilitate and hinder SSB taxes being implemented. Greater coordination of efforts internationally could facilitate more efficient and more strategic approaches to accumulating evidence on SSB taxes, as well as a range of other low-agency population health interventions for NCD prevention.

The challenge of complexity

Both of the evaluations described above took a relatively simple approach to evaluating a somewhat complex problem. Neither paper presented a theory of change [15] for the intervention they were evaluating. However, it is clear that in both cases the implicit theory of how the intervention was thought to achieve its likely impacts was relatively simplistic: a linear pathway from price change to changes in consumption [16]. Yet both sets of authors also hinted at more complicated mechanisms to explain unexpected findings, such as the socio-economic patterning of outcomes. For example, they suggested that other purchasing behaviours (e.g., taking advantage of promotions, buying in bulk), commercial behaviours (e.g., flexing or cross-subsidising prices across products or brands), or the effects of co-occurring education, regulation of product labelling, advocacy campaigns, and media coverage may have influenced their observed outcomes [4,5].

Public health scientists have increasingly highlighted the importance of taking context into account in evaluating population interventions [17]. It has also been suggested that such interventions may be better considered as events in complex, adaptive systems, rather than within simpler linear causal models [18]. Systems thinking prompts us to ask different questions, which in turn leads to different kinds of evaluations. For example, in the case of the Chilean SSB tax, instead of asking ‘What are its impacts on price and sales?’ we would ask ‘What are the diverse impacts of announcing and introducing the tax, how do these come about, and how do they interact over time?’ While still considering impacts on price and sales, this would also prompt us to think more broadly about the wide-ranging potential intended and unanticipated consequences of such a tax at all levels of society, in all sectors, and for all stakeholders. It also prompts us to consider the potential for change in outcomes as a continuous process, rather than a single event, consequent upon the continuous adaptation that occurs within food systems in response to external stimuli. While such approaches to evaluation remain in their infancy, some valuable examples are emerging, such as a recent study evaluating the impacts of a voluntary scheme in which retailers were challenged to reduce availability of low cost, high strength beers and ciders [19,20]. In this study, the researchers engaged with the complexity of the context and the potential for wide-ranging consequences of the intervention, assessing these through multiple methods and diverse sampling strategies, as well as through analyses informed by complexity theory and systems thinking.

Meeting the challenges

Broader, more nuanced, and more informative research questions cannot be answered using traditional, more simplistic approaches to evaluation. The present evidence base is significantly limited by a failure to embrace different ways of thinking and working, including addressing questions of context and those related to complex adaptive systems [17,18]. However, a consequence of taking a broader approach to evaluation that embraces complexity will be that it challenges prevailing methodological orthodoxies. Researchers, funders, and journals may all be reluctant to relinquish existing hierarchies of evidence and ‘traditional’ methods of evidence synthesis. With so many intractable global public health challenges associated with NCDs in need of robustly researched solutions, this seems short-sighted. Greater ambition and leadership are needed among researchers, funders, and policymakers to enable smarter approaches to the development and evaluation of low-agency population interventions, including taxes on unhealthy commodities, regulation of marketing, and structural interventions to make active living easier. Ambition and leadership coupled with greater international collaboration to identify opportunities for—and to coordinate efforts to fund, implement, and build capacity for—quasi-natural and natural experimental evaluations of these interventions could more rapidly advance science on NCD prevention.


We thank our colleagues and collaborators at the Centre for Diet and Activity Research and beyond who have helped us to develop our ideas in relation to population health intervention research.

Opinions expressed in this article are those of the authors and do not necessarily represent those of our funders.


  1. 1. Rose G. The strategy of preventative medicine. Oxford: Oxford University Press; 1993.
  2. 2. Adams J, Mytton O, White M, Monsivais P. Why are some population interventions for diet and obesity more equitable and effective than others? The role of individual agency. PLoS Med. 2016;13(4):e1001990. pmid:27046234
  3. 3. Craig P, Cooper C, Gunnell D, Haw S, Lawson K, Macintyre S, et al. Using natural experiments to evaluate population health interventions: new Medical Research Council guidance. J Epidemiol Community Health. 2012;66(12):1182–6. pmid:22577181
  4. 4. Nakamura R, Mirelman AJ, Cuadrado C, Silva-Illanes N, Dunstan J, Suhrcke M. Evaluating the 2014 sugar-sweetened beverage tax in Chile: an observational study in urban areas. PLoS Med. 2018;15(7):e1002596. pmid:29969456
  5. 5. Caro JC, Corvalán C, Reyes M, Siva A, Popkin B, Smith Taillie L. Chile’s 2014 sugar-sweetened beverage tax and changes in prices and purchases of sugar-sweetened beverages: an observational study in an urban environment. PLoS Med. 2018;15(7):e1002597. pmid:29969444
  6. 6. Chriqui JF, Chaloupka FJ, Powell LM, Eidson SS. A typology of beverage taxation: multiple approaches for obesity prevention and obesity prevention-related revenue generation. J Public Health Policy. 2013;34(3):403–23. pmid:23698157
  7. 7. Backholer K, Blake M, Vandevijvere S. Sugar-sweetened beverage taxation: an update on the year that was 2017. Public Health Nutr. 2017;20(18):3219–24. pmid:29160766
  8. 8. Panter J, Guell C, Prins R, Ogilvie D. Physical activity and the environment: conceptual review and framework for intervention research. Int J Behav Nutr Phys Act. 2017;14(1):156. pmid:29141646
  9. 9. Colchero MA, Popkin BM, Rivera JA, Ng SW. Beverage purchases from stores in Mexico under the excise tax on sugar sweetened beverages: observational study. BMJ. 2016;352:h6704. pmid:26738745
  10. 10. Colchero MA, Rivera-Dommarco J, Popkin BM, Ng SW. In Mexico, evidence of sustained consumer response two years after implementing a sugar-sweetened beverage tax. Health Aff (Millwood). 2017;36(3):564–71. pmid:28228484
  11. 11. Falbe J, Rojas N, Grummon AH, Madsen KA. Higher retail prices of sugar-sweetened beverages 3 months after implementation of an excise tax in Berkeley, California. Am J Public Health. 2015;105(11):2194–201. pmid:26444622
  12. 12. Silver LD, Ng SW, Ryan-Ibarra S, Taillie LS, Induni M, Miles DR, et al. Changes in prices, sales, consumer spending, and beverage consumption one year after a tax on sugar-sweetened beverages in Berkeley, California, US: a before-and-after study. PLoS Med. 2017;14(4):e1002283. pmid:28419108
  13. 13. Alvarado M, Kostova D, Suhrcke M, Hambleton I, Hassell T, Samuels TA, et al. Trends in beverage prices following the introduction of a tax on sugar-sweetened beverages in Barbados. Prev Med. 2017;105S:S23–5. pmid:28716655
  14. 14. Cawley J, Willage B, Frisvold D. Pass-through of a tax on sugar-sweetened beverages at the Philadelphia International Airport. JAMA. 2018;319(3):305–6. pmid:29071342
  15. 15. De Silva MJ, Breuer E, Lee L, Asher L, Chowdhary N, Lund C, et al. Theory of change: a theory-driven approach to enhance the Medical Research Council’s framework for complex interventions. Trials. 2014;15:267. pmid:24996765
  16. 16. Mytton OT, Eyles H, Ogilvie D. Evaluating the health impacts of food and beverage taxes. Curr Obes Rep. 2014;3(4):432–9. pmid:26626920
  17. 17. Craig P, Di Ruggiero E, Frohlich KL, Mykhalovskiy E, White M, Canadian Institutes of Health Research (CIHR)–National Institute for Health Research (NIHR) Context Guidance Authors Group. Taking account of context in population health intervention research: guidance for producers, users and funders of research. Southampton: NIHR Evaluation, Trials and Studies Coordinating Centre; 2018 [cited 2018 Jul 13].
  18. 18. Rutter H, Savona N, Glonti K, Bibby J, Cummins S, Finegood DT, et al. The need for a complex systems model of evidence for public health. Lancet. 2017;390(10112):2602–4. pmid:28622953
  19. 19. McGill E, Marks D, Sumpter C, Egan E. Consequences of removing cheap, super-strength beer and cider: a qualitative study of a UK local alcohol availability intervention. BMJ Open. 2016;6(9):e010759. pmid:27687895
  20. 20. Sumpter C, McGill E, Dickie E, Champo E, Romeri E, Egan E. Reducing the strength: a mixed methods evaluation of alcohol retailers’ willingness to voluntarily reduce the availability of low cost, high strength beers and ciders in two UK local authorities. BMC Public Health. 2016;16:448. pmid:27230466