Rationale

Given that Healthy Cities entails various theoretical elements, including: “proximal and distal determinants of health, proximal and distal interventions for health, and ‘known impact’”; this complicates evaluation [8, pp. i23]. O’Neill and Simard [3] have suggested that limited recognition of the political nature of evaluation is one factor in explaining why so few evaluations have been done for Healthy Cities projects; the required stakeholder support can be extensive, as with Europe’s Phase V evaluation, which engaged Healthy Cities representatives, the WHO, and academics [11]. For instance, different perspectives on indicators are evident when looking at an early evaluation of the Healthy Cities project in Chittagong, Bangladesh from 1994. This evaluation highlighted the limited utility in impact indicators as substantial time is required to collect these measures, whereas process evaluations can work to quickly ameliorate issues and keep morale high [2]. And further, considered quantitatively, with Europe’s Phase I evaluation having 53 indicators and decreasing to 32 indicators in Phase V—and nevertheless, facing contestation in cities [11]. Evidently, there are various ways an evaluation of Healthy Cities can be undertaken. In fact, it has been noted that creativity is needed to assess the worth of a Healthy Cities Project [12]. In light of these contentions, we present a protocol to synthesize evidence on how process evaluations of Healthy Cities have been conducted and their findings.

Objective

We are interested understanding approaches taken to evaluate the approach itself across various settings (i.e., process evaluation). We are not interested in evaluating the relationship between the Healthy City projects on individual-level outcomes associated with behaviour change or similar (e.g., tobacco use, alcohol use, unhealthy diet, physical inactivity). In other words, we are interested in assessing impacts (and/or ways of working to achieve such impacts, such as processes or structures) rather than outcomes. An impact is “a result or change that has come about as the result of complex interactions between a range of factors, including Healthy City designation, but potentially much wider (multi-causality, such as socio-political change, economic up/downturns, extending or limiting the range of stakeholders, etc.) and is driven by priorities determined by the Healthy Cities value system” (11); whereas an outcome is “is a result or change that can be directly attributed to such an activity or intervention within a Healthy City […] Or more succinctly: outcome evaluation assesses the result in terms of achievement of objectives set” (11). Because ‘The’ Healthy City does not exist, as each city operates within its own unique context [10], we are not expecting to collate a universal list of indicators [3]. Instead, our aim is to spark discussions and inform future approaches through collating information on how evaluations have been conducted to-date.

In addition to this primary aim, we are also interested in understanding if consideration is afforded to the context in which the city operates, such as cities of the global south, and if so, how this consideration is built into evaluation. This is supported by the aforementioned evaluation from Chittagong, which determined that Healthy Cities projects established in developing countries are fundamentally different than those in industrialized nations, calling for both the need to adapt projects and conduct comparative analysis of Healthy Cities projects in developing countries [2].

While Healthy Cities seems to have an inherent tension between research and practice [13], evaluating Healthy Cities is important for several reasons. These reasons include: assessing any changes to political processes, community health status, or others; sustaining the political legitimacy of Healthy Cities; affording comparison across cities; drawing on successes to maintain community mobilization and participation; and advancing scientific knowledge [3].

(i) Identifying the research question

This study protocol can be applied to synthesize evidence on how evaluations of Healthy Cities (including healthy municipalities, healthy communities, healthy islands, and others) have been conducted through drawing on both peer-reviewed and grey literature. Thus, asking: how has the Healthy Cities project or approach been evaluated?

Considering that “the real success of any Healthy City project is when it ceases to be a project, because the system it set up to ensure that health issues are given priority, to involve all stakeholders and to ensure that all sectors recognize that their role in healthy cities becomes part of the structure of local governance” [16, pp. 27], the inclusion and exclusion criteria for this study have been developed to not only better understand how Healthy Cities have been evaluated, but how evaluations also assess such institutionalization structures.

Inclusion criteria

As such, the inclusion criteria eligibility is as follows: (i) must discuss the healthy cities project/approach (including healthy municipalities, healthy communities, healthy islands, and others), (ii) describe an evaluation undertaken or report results of an evaluation of the approach itself applied within a city, cities, community, etc., and (iii) available in English due to resource constraints.

Exclusion criteria

References are excluded if they are either (i) not evaluating the Healthy City approach itself, (ii) evaluating health outcome indicators (e.g., physical activity rates) rather than an evaluation of the approach itself—and cases where the Healthy City approach was evaluated using various indicators that may include health outcomes will be included, (iii) evaluating a program that is not a Healthy City project, despite seeking to improve health (i.e., uses language of “healthy city” or “healthy community” without aligning with or referencing “Healthy Cities”), (iv) commentary, reflection, or interview discussing evaluation issues (i.e., not an evaluation with described methods), or (v) the full text is not available.

(ii) Identifying relevant studies

The search strategy was developed with an academic librarian possessing relevant expertise of appropriate databases and comprehensive search strategies for knowledge synthesis projects. The strategy was designed to cast a wide net, searching a multitude of databases and grey literature from the WHO, namely the: Ovid MEDLINE, Ovid EMBASE, Web of Science Core Collection, Scopus, and WHO Institutional Repository for Information Sharing (IRIS) databases. These numerous databases will be searched to minimize any potential gaps, as the Ovid databases are focused on biomedicine, whereas the Web of Science Core Collection and Scopus draw from a range of disciplines, thus ensuring articles that may be missed through searching biomedical databases will be retrieved in interdisciplinary databases, and vice versa. In addition, through searching the WHO’s IRIS, grey literature hits, specifically WHO texts, will be included that would not be retrieved through these four aforementioned databases. The search strategy will include relevant keywords such as healthy city/healthy cities, healthy municipality/municipalities, healthy community/communities, healthy island(s), evaluation, assessment, effectiveness, efficacy, and indicator(s), and the subject heading of program evaluation. No publication date limit will be applied, but results will be filtered so that only English language results are included. The databases to be searched and associated search strategies are provided in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Databases, rationale, and associated search strings.

https://doi.org/10.1371/journal.pone.0276179.t001

(iii) Study selection

Covidence software [17] will be used to remove duplicates and undertake the first screen of references by two independent reviewers. This entails two reviewers independently reading the titles and abstracts of all references and screening in any that may be relevant to the research question for full-text review. In the case of the grey literature where abstracts are often unavailable, executive summaries, table of contents, or similar will be screened, following the grey literature protocol utilized by Godin et al. [18]. In the case where these may be unavailable, the grey literature will be advanced to full-text screen. Any conflicts arising from the first screen will be resolved through discussion by the two independent reviewers. In the second stage, the list of references marked for full-text review will be read by two independent reviewers to assess alignment with the inclusion criteria and determine the final references marked for inclusion in the study. Discrepancies will be discussed and resolved by the review team.

Studies marked for inclusion will be further assessed through ‘citation chaining’ (i.e., circling back to stage iii to ensure a comprehensive identification of studies). This entails both reviewing sources cited in the included references and checking resources that cite these included studies to determine if there are any potentially missed references that meet the inclusion criteria. This will be undertaken with Web of Science automatically when this stage is reached.

(iv) Charting the data

Data extracted from the full texts will be charted across: title, author(s), journal or book title, location of evaluation, method of evaluation (e.g., case study, interviews, focus groups), who conducted the evaluation, indicators assessed/what was evaluated, rationale for the evaluation, and consideration for city context. These dimensions were informed by the work of O’Neill and Simard [3] and de Leeuw [19], and may be adjusted during the review process. For example, additional dimensions deemed worthy of consideration during the review process may be included. Or similarly, dimension(s) that yield minimal information may be excluded.

(v) Collating, summarizing, and reporting the results

The PRISMA diagram will be used to illustrate the review process and delineate stages where studies were eliminated with the associated reason. The charted data will be narratively synthesized, drawing on the dimensions to thematically group findings in the discussion of the manuscript. We hypothesize being able to point to both commonalities and differences across evaluations and raise important considerations for those undertaking an evaluation of a Healthy Cities project.