Introduction

Regulatory measures that aim to improve population nutrition have become an increasingly popular public health strategy against obesity. As a growing number of approaches are being field tested, a new dimension of evidence has become available to inform future policy-making more realistically [1] than modeling exercises and researcher-manipulated studies in controlled settings [2, 3]. However, evaluations of early policy efforts have not been systematically and comprehensively examined. Although one recent systematic review [4] analyzed natural experiments in the areas of physical activity and nutrition, it relied on a search of PubMed only and excluded outcomes measured directly in the food environment. It reported mostly null results across the categories of interest to this study and did not identify any studies on fiscal policies or food supply measures [4].

Evaluations of policy interventions are methodologically challenging as they are necessarily observational and involve long and often indirect cause-and-effect chains that occur in parallel with a myriad of other changes in the population and environment [4–6]. Preventive interventions that target environments rather than individual behaviors present the additional difficulty that the desired impact might emerge only gradually or cumulatively in conjunction with other interventions [7]. These considerations suggest that only measuring ultimate outcomes of interest, such as changes to nutritional patterns or body weight, is not an adequate indication of policy success or failure. Instead, the impact of real-life public health interventions may be more appropriately assessed by substantiating a logical pathway connecting intervention and outcome, and by demonstrating realization of immediate program goals or the presence of more distal jurisdiction-wide trends in average weight or nutritional intake. [6–8].

In this paper, we reviewed current research evaluating real-life policy interventions addressing obesity. We used a realist review approach [9] which focused on program mechanisms to provide a more nuanced assessment of policy success or failure. Specifically, we investigated the effect of statutory provisions of a regulatory nature that aim to reduce the consumption of energy-dense foods and beverages in the general population. Outcomes included both direct (e.g. BMI) and indirect (e.g. calorie count of food choices) measures.

Methods

Realist approaches review interventions along a continuum of indicators of successful implementation [9]. Based on a theory of the mechanisms by which an intervention is intended to bring about an intended outcome, the aim of a realist review is to provide a nuanced assessment of the extent to which interventions work and at which point of the implementation there is a failure to produce the desired end result [9]. This approach stands in contrast to the traditional review that focuses on one cumulative outcome as a maker of intervention success or failure; for instance the ability of a regulation providing information to consumers to result in changes in average BMI. Following the realist approach, the outcomes of interest for this systematic review align with the assumed program logic of interventions designed to reduce the consumption of unhealthy food and beverages and ultimately curb the prevalence of overweight and obesity. To this end, we collected data regarding (i) the effect of these interventions on average BMI or weight and on calorie intake and related proxy measures and (ii) indicators measuring parameters on the assumed causal pathway to changed consumption patterns, including measures of the degree of program implementation and non-behavioral consumer responses such as awareness and knowledge. In recognition that new policies may be evaluated on the basis of process indicators alone, we allowed all methodological approaches that included some measure of comparison, including studies of implementation progress with an assumed baseline of zero. A review protocol was developed and registered on the International Prospective Register of Systematic Reviews (PROSPERO) prior to commencement of this study (registration number CRD42015025276). As summarized in Fig 1 and in line with a realist review approach, our search and selection methods were informed by the likely program logic of interventions in the principal areas identified in the literature as possible regulatory levers [10,11].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Assumed pathways from interventions to health outcomes.

https://doi.org/10.1371/journal.pone.0182581.g001

Study selection

We considered all studies published between 2004, the year WHO member states first acknowledged a role for market-related regulatory interventions for obesity prevention in the Global Strategy on Diet, Physical Activity [12] and October 31^st, 2015 for inclusion in this review, with no initial restriction on the language of publication.

We included all full-scale policy interventions designed to improve population nutrition, regardless of whether the outcome(s) reported was related to the food environment or to behavioral patterns. Eligible studies (i) examined an enacted statutory intervention (ii) applied to the entire population of its jurisdiction and (iii) targeted the consumption of energy-dense foods and beverages. All interventions that are not part of a full-scale, jurisdiction-wide policy were excluded, such as pilot programs and private sector or NGO actions without a change of primary or secondary legislation. Differential sales taxes and low-level soda taxes, usually enacted solely as means to raise revenue [13], were excluded due to the lack of public health policy intent. Accordingly, eight studies reporting the effects of differential sales taxes were excluded (also see Fig 2), regardless of the presence or direction of any effect on sales levels, consumption patterns, or weight and health outcomes. In addition, we excluded all interventions aimed only at children or other defined or implicit sub-groups (e.g. school-based programs or the US Special Supplemental Nutrition Program for Women, Infants, and Children), but retained those that provide a social safety net open to anybody in demonstrated need (e.g. the US Supplemental Nutrition Assistance Program (SNAP)/ food stamp program).

After removal of duplicates, we screened 25,323 items for relevance according to the inclusion criteria. The first reviewer (JS) initially assessed each title and, where available, abstract. A subset of 10% of the initial search results was again reviewed for eligibility according to the inclusion criteria by a second reviewer (JMS). Where study eligibility was disputed, the co-authors reached a consensus decision. The first reviewer then retrieved and assessed the full text of 302 articles that had been determined to possibly meet the inclusion criteria in the first round of screening. In addition to studies reporting on the evaluation of one specific intervention matching the inclusion criteria, we also retained eleven systematic reviews whose inclusion criteria overlapped at least partially with ours. We reviewed the reference lists of these reviews for additional eligible studies before excluding the reviews themselves from further analysis. Together with the hand-searching of the reference lists of all included studies, this process yielded an additional seven eligible articles. The same two reviewers independently assessed the 48 selected studies for methodological quality prior to inclusion in the review using the appraisal tools for Quality Assessment of Before-After (Pre-Post) Studies With No Control Group [14] and for Quality Assessment of Observational Cohort and Cross-Sectional Studies [15] developed by the National Heart, Lung, and Blood Institute. At this stage, we excluded a further three studies which reported evaluation outcomes, but did not detail or reference the underlying methodology. An additional two studies assessing health department-led initiatives aimed at improving the availability of healthier food in corner stores in New York City and Philadelphia were excluded due to these interventions being programmatic in nature without a change in legislation or regulation. By contrast, studies reporting on the effect of local subsidies for healthy purchases made under the US Supplemental Nutrition Assistance Program (SNAP) are considered within scope of the inclusion criteria given that incentive programs targeting SNAP recipients are subject to the laws and US Department of Agriculture (USDA) regulations governing the SNAP scheme and, at the time the interventions under evaluation were implemented, required jurisdictions to obtain formal approval from the USDA Food and Nutrition Service. The flow diagram in Fig 2 below summarizes the database search and study selection process. Also see Prisma Checklist, S3 File)

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. PRISMA flow diagram—Summary of search and selection process.

https://doi.org/10.1371/journal.pone.0182581.g002

Results

The 36 studies [16–53] span six different types of interventions: a majority (n = 19) report on calorie posting on chain restaurant menus, followed by changes to food infrastructure (n = 5), subsidies for healthy food purchases (n = 5), taxation of unhealthy foods and beverages (n = 5), government food standards (n = 1), and nutrition labeling of products (n = 1). Approximately 80% of included studies (n = 30) assessed interventions implemented in the United States. Evaluation strategies varied and resulted in different endpoints, often with multiple strategies used in one report to quantify the success of program implementation and the effects on behavior. Methodological quality against the Quality Assessment of Before-After (Pre-Post) Studies With No Control Group [14] and the Quality Assessment of Observational Cohort and Cross-Sectional Studies tools varied between studies. The majority of studies were judged to be of medium quality. Recurring limitations were related to limited sampling frames and overly descriptive approaches. Methodological limitations are considered as part of the narrative synthesis below and further discussed in the discussion and conclusion section. The results table in the annex (S1 File) provides a detailed overview of the included studies, including quality appraisal. In the following, we summarize the results in a narrative meta-synthesis.

Discussion and conclusion

These findings indicate that isolated regulatory interventions frequently result in improvements of the most proximal outcomes, measured in the food environment and situated at the very beginning of the logic model. However, the interventions assessed here fail to achieve an effect on consumption that could plausibly be considered as clinically significant, i.e. as having an effect on individuals’ nutritional intake to the extent that it would reduce the incidence of overweight, obesity, and related chronic diseases. This is a differentiation between different levels of policy success and failure that have not been highlighted previously [4].

When compared to just a few examples of effect estimates put forward during policy development and decision-making processes, it is clear that current interventions are falling short of the public health impact hoped for by policy-makers and predicted by many researchers. For example, in New York City, the Department of Health estimated that the new calorie posting rule would lead to “at least 150,000 fewer New Yorkers [becoming] obese, [and] at least 30,000 fewer cases of diabetes” [52] over five years. However, with the exception of the study focused on Starbucks, the New York City-based studies profiled here failed to find statistically and clinically significant calorie reductions [21–24,33]. Moreover, a recent look at the sustained impact of the intervention, published just after the cutoff date for this review, concluded that even minimal improvements in consumer awareness appear to have diminished over time [53]. Meanwhile, the Danish forecast that a fat tax would eventually add 5.5 days to the average Dane’s lifespan [54] will remain unrefuted given the quick abolition of the measure, but appears tenuous given the early evaluation results. Similarly, one of the evaluations of New York City’s Green Cart program reported that the city originally estimated that the generated increased intake of fruits and vegetables would measurably improve the health status of 75,000 individuals and avert loss of at least 50 lives a year [35]. However, program administrators concluded that the direct impact of the intervention on morbidity and mortality would be too difficult to quantify and the program evaluators observed that direct health-enhancing arguments for Green Carts subsequently faded [35].

This is not to say that these interventions may not deliver cumulative behavioral and health effects in the long-term, especially where they act in parallel with complementary interventions and change social and political perceptions of nutrition. In this context, it is notable that recent studies at national level and in hotspots of obesity prevention activities such as New York City have found both a shift in the attitudes of consumers towards sugary drinks and an actual reduction in average soft drink consumption [55–57].

Some of the interventions discussed above may also be inconsequential and not have a meaningful impact on consumption: a recent review of regulation targeting sugar-sweetened beverages argued that policies are squandering the potential for more pronounced behavioral impact because of their restrained design, possibly to appease industry and political opponents [58]. Indeed, very few of the above interventions match the designs identified in the literature as the more effective public health approaches, be it displays of physical activity equivalents instead of plain calorie counts [59] or excise taxes amounting to price increases of at least 15–25%, equivalent to the long advocated penny per ounce tax [60–62]. In 2015, Berkeley, California, passed a tax on sugar-sweetened beverages that matched the magnitude suggested by public health experts and in 2014, Mexico implemented a tax of one peso per liter which, if passed on to consumers, comes close to the recommended level with a 10% increase in price. Two early evaluations, published just after the cut-off date for this review, reported that in both locations, the taxes were generally passed on, with higher price increases relative to taxation levels reported for Mexico compared to Berkeley [63,64]. The study in Berkeley employed comparison cities for control of pre-post trends and reported pass-through rates of 69% for soda and 47% for all taxed products [63]. Neither jurisdiction reported deleterious effects on the prices of non-taxed beverages such as bottled water, with the exception of slight price increases for diet soda in Berkeley [63,64]. A third evaluation of the Mexican tax published in 2017 indicated that purchases of taxed beverages fell by 7.6% in the two years after implementation of the tax compared with a 2.1% drop in untaxed beverage purchases over the same period [65]. These two fiscal interventions warrant close attention from the experts and policy-makers as they represent rare examples of current policy recommendations being put into practice. Most recently, additional jurisdictions have approved soda taxes, among these decisions by the legislative bodies of Philadelphia City and Cook Country, which includes Chicago, as well as popular votes in San Francisco, Oakland, and Albany in California and in Boulder in Colorado [66]. Both the British and Irish governments have announced plans for the introduction of soda taxes in 2018 [66].

Some limitations must be taken into account when interpreting the results of this systematic review. Firstly, despite an expansive search of a variety of databases and broad inclusion criteria, it is likely that some evaluations of real-life interventions are not available through academic and gray literature repositories. It is reasonable to suppose that in many instances, especially in lower level jurisdictions and in middle and low-income countries, no formal evaluation of relevant policies would have been undertaken and/or reported. As a result, only a small number of studies from outside the OECD were identified, with several articles describing interventions in Ghana [67] and the Pacific island region [68,69] excluded at the appraisal stage due to the unavailability of detailed evaluation processes and results. Similarly, it is possible that unsuccessful policy interventions remain underreported and those that are published put a greater emphasis on intermediate program outcomes that show greater progress than the actual policy end goal. These limitations underline the need to make methodically sound evaluations a routine component of policy implementation and highlight the usefulness of some form of centralized repository for comprehensive evaluation reporting that is accessible globally.

Secondly, our study purposely used appraisal tools tailored to observational study designs that are amenable to evaluating real-life policies. As outlined in the introduction, real-life policy experiments do not always fully align with the methodological expectations of evidence-based health sciences, particularly when compared to targeted prevention delivered in health care settings. This issue was raised in relation to the study by Dumanovsky and colleagues who employed a simple pre-post study design to quantify the effect of menu labeling in New York City [21]. Criticized for the chosen study design [70], the authors responded that the methodology needs to match both the reality of a policy in progress and the limited resources of a public agency carrying out its own evaluation while stymied by the refusal of industry to share its sales data [71]. Despite some reservations about these study designs expressed in the literature, studies using conventional cross-sectional designs or simple pre-post designs posed little difficulty for appraisal and data extraction. Conversely, studies that evaluated implementation processes were problematic to appraise and summarize. Aspects of design, common in the evaluation of food infrastructure interventions such as the SNAP subsidies in Philadelphia and NYC and NYC’s Green Cart program, complicated the assessment of studies examining food infrastructure improvements and food subsidization programs. These aspects included the use of descriptive approaches and a mix of different study designs of varying quality within single reports. The difficulties that we encountered suggest that scholarly assessment of study quality and the reality of policy-making in perennially resource-constrained health departments occasionally collide. As a result, even more differentiated appraisal tools need to be used for evaluation in recognition that studies with descriptive approaches can be useful for charting implementation progress by ensuring that program logic is in place.

To conclude, our review underlines that the immediate expectations associated with the examined types of regulatory interventions need tempering. At this point in time, the policy examples discussed above primarily deliver proof of feasibility: the fact that they survived the policy-making process and have been mostly successful in reaching immediate program goals should enhance the political palatability of such approaches even if, at the time of examination, there has been little demonstrated impact on risk factors and health outcomes. Policy-makers should therefore not dismiss such recent policy experiments as failures, but pursue the example of these jurisdictions as necessary building blocks for more stringent and comprehensive nutrition policy and obesity prevention regimes.

Supporting information

Acknowledgments

The authors gratefully acknowledge the support of the HealthyLaws research team and advisory committee. This project was conducted as part of a study funded by an Australian National Preventive Health Agency Grant, project ID: 182BRA2011. JMS was also supported by an NHMRC Capacity Building Grant (565501) and an Australian National Preventive Health Agency Fellowship (20STR2013F).