Database search.

Two database searches were conducted, one in August 2016 with no date restrictions, and repeated in July 2020 for the time period September 2016 to July 2020 with the following search criteria. English-language articles were searched in CINAHL, ProQuest Central, SCOPUS, Informit (all databases) and Embase. Five categories of search terms were combined (sample search strategy provided in S1 Appendix): 1. Child, parent, family; 2. Place-based/level, area-based/level, community-based/level, neighborhood-based/level, complex community, collective impact; 3. Disadvantage, poverty, vulnerable, socio-economic, inequality, well-being; 4. Intervention, initiative, program, trial; and 5. Outcome, impact, efficacy, evaluate, feasibility, protocol, pilot. Additional papers were retrieved by examining reference lists of identified papers and by separate searches using the titles of identified place-based initiatives.

Grey literature search.

Many evaluations of public policy driven place-based initiatives are commissioned to consultants, independent research groups, research consortiums or university departments and are presented in report form. Inclusion of material not controlled by commercial publishers (“grey literature”) in evidence reviews reduces publication bias and provides a more complete and balanced picture of the evidence [32]. We used three approaches to identify grey literature relevant to this review: 1. A Google search of known initiatives and initiatives identified via secondary sources, with the terms ‘evaluation’, ‘report’ or ‘pdf’ entered in an attempt to source evaluation reports; 2. Searching known databases containing research and evaluation reports (e.g., www.childtrends.org, www.researchconnections.com and Child Family Community Australia Information Exchange); and 3. Searching websites established specifically for initiatives and/or the initiative’s evaluation (e.g., National Evaluation of Sure Start website and Toronto First Duty website).

Types of studies.

We included initiatives if an impact evaluation study had been conducted. All types of impact study designs were considered eligible for inclusion (e.g., randomised controlled trials (RCTs), quasi-experimental, non-experimental, cohort, cross-sectional, pre- and post-), if at least one child outcome had been reported.

Types of place-based initiatives.

Inclusion criteria. Literature pertaining to a place-based initiative was initially included if the initiative met the following criteria:

• Population: targeted (but not limited to) children (infancy to 4 years) and pregnant women who live in socioeconomically disadvantaged areas.
• Place-based. Showed evidence of a place-based approach, with a focus on people and place [33].
• Location: high income countries (as defined in NationMaster) [34].
• Sponsoring organisation: government administered program. Showed evidence of federal or state government initiating, leading and/or managing the initiative.
• Size/scale of initiative: implemented at a national, state or regional level, or was a multi-site demonstration project.
• Outcomes: goal of improving multiple outcomes for children and their families.

Exclusion criteria. Initiatives were excluded if the primary goal was improving a single child outcome domain (e.g., obesity prevention, prevention of child abuse/neglect), targeted a specific adult or child clinical population, or if the primary aim was broad social, health, economic, or physical regeneration or improvement (e.g., the physical quality of homes or public spaces), even though a subsidiary benefit may have been improved outcomes for children.

Selection of sources of evidence.

Inclusion criteria. Article title and abstract screening was initially conducted by Author 1 (FB) with potentially eligible studies included for full text review. Author 1 and Author 5 (JN) conducted the full text review, with disagreements resolved through consensus. In this review, multiple results from the same initiative are reported together. Therefore, once initiatives were selected for inclusion, publications that presented results from the same initiative were collated and assessed as ‘primary’ or ‘secondary’ studies. Primary studies were those that provided the principal source of outcomes evaluation information for each initiative for completion of the evidence appraisal. Secondary studies were those that provided detail about process evaluation and contextual information about how the intervention changed over time, and were included in the review only where this information was not available in the primary source. Many of the initiatives reported impact evaluations conducted at multiple time points. In these cases, the most recent was used as the primary source, and supplemented with the earlier reports as required. For some initiatives, evaluations were reported in both peer reviewed and grey literature. Peer reviewed papers were prioritised for inclusion over grey literature where they were reporting on the same data.

Exclusion criteria. Articles were excluded if they reported no original data or evaluated only a single component of a broader place-based initiative, including local evaluations.

Types of outcome measures and other data items of interest.

Place-based initiatives by definition aim to improve multiple and interrelated outcomes across pregnancy and birth, child, parent, family and community domains [35]. Rather than approaching this scoping review with a pre-determined set of outcomes, we examined the included initiatives to determine what outcomes were measured and collated and coded them as per the domains and categories in Table 1. In determining whether the place-based initiatives were effective at improving outcomes (Aim 3), significance was set at P ≤ 0.05.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Data items applied to initiatives.

https://doi.org/10.1371/journal.pone.0261643.t001

Other data items of interest were broadly informed by our research questions and are summarised in Table 1. Where appropriate, the beginning of each sub-section briefly defines and justifies the inclusion of the item of interest. We collected overview data to enable the characteristics of the initiatives to be described (location, size/scope, year of commencement), along with initiatives’ aim and service model, funding and delivery structure, the size and selection process for local delivery areas, and theories of change. These were summarised and combined with outcome data to help shed light on what aspects may contribute to effectiveness. As our first aim was to examine the study designs and methods for evaluating place-based initiatives, we identified the following data items of interest: quality, overall evaluation design, length and timing, process evaluation, local evaluation, and impact study design. For impact study design we documented a range of design features including the study sample, comparison group (if relevant), and method of data collection. We were interested in the context in which initiatives were implemented and evaluated (Aim 2), therefore we initially summarised these findings in a free text field then specifically coded a range of items where the contextual environment directly affected the initiative (e.g., change in scope or funding).

Aims and service model.

A brief description of each initiative, including the aim and service model was extracted and is summarised in Table 3. There was considerable diversity in the aims of the initiatives and thus in the range of programs and services provided. Some focused primarily on strengthening universal services through ‘joined-up working’ and service integration (ABC Programme [40], Sure Start [8, 41–46], Best Start [52–54], Toronto First Duty [60–63], Flying Start [48–51], Starting Well [65, 66]), or on improving childcare and kindergarten quality (First Steps [55, 56], Smart Start [57, 58], NEYAI [64]). Others focussed more on addressing gaps in current service delivery (Communities for Children [10, 37–39], Georgia Family Connection [59], Neighbourhood Nurseries Initiative [47]). Models of service delivery also varied. Some initiatives provided centre-based delivery via children’s centres (Neighbourhood Nurseries Initiative [47], Toronto First Duty [60–63]), others had a more diffuse model of service delivery in the community (ABC Programme [40], Communities for Children [10, 37–39], Georgia Family Connection [59], Starting Well [65, 66]), and some provided a mix of both.

Funding and delivery structures.

Funding and delivery structures for all included initiatives were also extracted (not reported in tables for brevity). Some initiatives were wholly funded and implemented by government organisations (Sure Start [8, 41–46], Best Start [52–54], Flying Start [48–51], Starting Well [65, 66]). Others were funded by the government but contracted non-government organisations to deliver at the community level (Communities for Children [10, 37–39]). For Neighbourhood Nurseries Initiative [47], funding was available to both non-government and privately operated childcare centres. In Ireland, Canada and the USA it was more common for the government to work in partnership with philanthropic and corporate partners with shared responsibilities for funding, governance and implementation (ABC Programme [40], NEYAI [64], First Steps [55, 56], Smart Start [57, 58], Toronto First Duty [60–63], Georgia Family Connection [59]).

Size and selection of delivery areas.

Previous research has highlighted the importance of geographic scale and the concept of ‘place’ as potential influences on the effectiveness of place-based initiatives [7, 23]. We extracted the size of local delivery areas and how they were selected, as summarised in Table 3. These varied considerably between initiatives and indeed was not uniform within initiatives. ‘Place’ in USA state-based initiatives (First Steps [55, 56], Smart Start [58, 67], Georgia Family Connection [59]) was defined at county level, and usually started as demonstration projects in a defined number of counties before expanding to cover the whole state. For the majority of the UK initiatives, areas were much smaller. Sure Start areas, for example, averaged around 13,000 people with around 700 children aged 0–3 and were targeted to 20% of the most deprived areas in England [8]. Flying Start targeted highly concentrated pockets of disadvantage within already deprived Local Authority areas, and used school catchment areas to define their delivery boundaries [48]. Toronto First Duty in Canada also based their delivery areas around schools, in keeping with their school hub service model [60]. The ABC Programme selected bounded areas in which resident populations identified with each other as a community [40]. Neighbourhood Nurseries Initiatives aimed to increase nursery ‘places’ in disadvantaged neighbourhoods, and expected any new nurseries be located near major roads [47]. Communities for Children sites were chosen based on criteria for multiple aspects of disadvantage and each site was defined differently, from a collection of postcodes to one or more defined Statistical Local Areas [37]. Similarly, Best Start sites ranged from whole municipalities to a small collection of rural towns or areas with a high Aboriginal population [52]. In the smaller demonstration projects Starting Well and NEYAI, the target delivery areas were described as a collection of suburbs [64, 66].

Theories of change.

A theory of change (or program logic model) explains how and why an initiative is intended to work [68]. From an evaluation perspective, the value of articulating a theory of change for complex initiatives is that it helps evaluators understand not just whether and how an initiative works, but which parts of an initiative have the greatest impact on outcomes [68]. We appraised all included initiatives to determine whether a theory of change had been developed. We found all initiatives had articulated a theory of change, either in text or figure form, as summarised in Table 3. All but one initiative (Neighbourhood Nurseries Initiative) had collaboration/partnership as a component of their theory of change, with this considered a ‘key ingredient’ to success for many. For example, Georgia Family Connection [59] theorised that its collaboration model was the primary difference between it and the comparison group. All but one initiative (Communities for Children) included modified universal services as part of their logic model, with three initiatives (Georgia Family Connection [59], First Steps [55, 56], Starting Well [65, 66]) also including the development of additional targeted services in their model. Communities for Children [38] theorised that plugging unmet service gaps would improve outcomes. Ten initiatives (Communities for Children [38], ABC Programme [40], Sure Start [8], Flying Start [48], Best Start [52], First Steps [55, 56], Smart Start [58, 67], Georgia Family Connection [59], NEYAI [64], Starting Well [65, 66]) theorised that involving the local community in decision-making would be beneficial; and all twelve initiatives included some degree of local area autonomy in their model.

Quality.

The evaluations of two initiatives were classified as high quality (Communities for Children [10, 37–39], Flying Start [48–51]), six as medium quality (Sure Start [8, 41–46], Neighbourhood Nurseries Initiative [47], Best Start [52–54], First Steps [55, 56], Smart Start [58, 67], Georgia Family Connection [59]), and four as low quality (ABC Programme [40], Toronto First Duty [60–63], NEYAI [64], Starting Well [65, 66]) (Table 3 and S2 Appendix).

Evaluation design overview.

Five initiatives (Sure Start [8, 41–46], Neighbourhood Nurseries Initiative [47], Flying Start [48–51], Communities for Children [10, 37–39], Toronto First Duty [60–63]) had a comprehensive evaluation design that combined the impact evaluation with process evaluation, local evaluation, and/or some cost-benefit or cost-effectiveness analysis. Comprehensive designs were a particular feature of the large national initiatives in the UK and Australia. Within these broad elements, evaluation designs took a range of forms. For the large, national initiatives like Sure Start [8], Communities for Children [37, 38] and Flying Start [48, 49], evaluation designs aligned with the structure outlined in Fig 1. Some initiatives applied a specific evaluation model to their evaluation (Best Start [52]), while others used more generic evaluation terms to describe their evaluation approach, e.g., ‘formative’ and ‘summative’ (Toronto First Duty [60]).

For all initiatives, the evaluation was commissioned to independent external evaluators. Nine appeared to have their evaluations commissioned and designed after implementation had commenced resulting in a lack of pre-intervention baseline data (Flying Start [49], Starting Well [65, 66]), delays in the commencement of data collection (Flying Start [49]) and the use of less-than-ideal datasets. An example of this is the NEYAI evaluation, which was based on children who participated in a year of free pre-school and received the NEYAI intervention, and compared them to children who attended another type of free pre-school [64]. The evaluation report focussed more on the benefits of pre-school than on the benefits of NEYAI. Two initiatives received funding for an impact evaluation a long time after the initiative had been implemented (Georgia Family Connection [59], Smart Start [67]). For example, evaluation funding for Smart Start ceased after 10 years [58] without a whole initiative evaluation having been conducted. Philanthropic funding was made available some years later to evaluate longer term outcomes of the program using routinely collected data [57].

Process evaluation.

Process evaluation seeks to understand the explanatory elements that may influence the outcome of an intervention [69]. It helps to determine whether an intervention’s failure to show any positive effects is due to the design of intervention itself or due to poor implementation [69]. Traditional process evaluation includes an assessment of quality, reach, dosage, satisfaction and fidelity [70]. For place-based initiatives, additional process evaluation considerations may include how to measure whether organisations are working in a ‘joined-up’ way and the level of community involvement in decision-making, if these were part of the theory of change [6]. None of the initiatives comprehensively evaluated all the expected elements of process evaluation with a whole-of-initiative synthesis. There was considerable diversity in the approaches that were taken to process evaluation, although some commonalities were apparent.

Of the ten process evaluations that were conducted (Communities for Children [71], ABC Programme [40], Sure Start [43], Neighbourhood Nurseries Initiative [47], Flying Start [48], Best Start [52], First Steps [55, 56], Smart Start [58], Toronto First Duty [60, 61, 63], Starting Well [66]), there was broad alignment between the aims of the initiatives and the process evaluation designs. For example, initiatives that aimed to improve service quality strongly focussed on measuring service quality indicators such as kindergarten or childcare quality (First Steps [55, 56], Neighbourhood Nurseries Initiative [47]), while initiatives that aimed to improve access to services measured reach (Communities for Children [71], First Steps [55, 56], Neighbourhood Nurseries Initiative [47]). Two initiatives that had a specific focus on joined-up working and partnerships as a means for improving service coordination, conducted assessments of the difference in this pre- and post-implementation (Communities for Children [71], Best Start [52]). Initiatives that aimed to build service capacity developed service profiles and looked at the difference in the number of services available pre- and post- (Communities for Children [71], Neighbourhood Nurseries Initiative [47]). The ABC Programme [40] was the only initiative to include a specific aim to increase the use of evidence and data in decision-making, and their process evaluation assessed reported changes in the use of evidence and data in local planning and service delivery. Other features typical of process evaluation designs included the collection of ‘performance monitoring indicators’, and number and type of services provided.

Fidelity was not commonly examined by the initiatives. First Steps was a notable exception, and undertook an examination of fidelity of their programs against pre-defined Program Accountability Standards [56]. They found an improvement in the fidelity of implementation over a two-year period, with a particularly high degree of fidelity for mature evidence-based programs.

Sure Start’s process evaluation framework was comprehensive and the findings span multiple reports, not all of which could be included in this review. A key finding was that due to the rapid scale-up of the program, and the variation in the number and type of programs being implemented, the quality of programs being delivered varied widely [8]. Moreover, they found a relationship between well implemented programs and better outcomes for children [43].

Local evaluation.

Local evaluation is where each geographic area (e.g., community or neighbourhood) evaluates its own activity. Collecting and synthesising local evaluation learnings provides valuable explanatory evidence about how and why initiatives may or may not be working as intended. Previous research has highlighted the challenges in collecting local evaluative data in a format that is both meaningful for local management and that enables whole-of-initiative synthesis [16, 17]. We identified and briefly appraised any findings that were collated in whole-of-initiative evaluation studies. Eight initiatives included local evaluation as part of their evaluation design (Communities for Children [71], Sure Start [8], Neighbourhood Nurseries Initiative [47], Flying Start [48], Best Start [52], First Steps [56], Smart Start [58], Toronto First Duty [60, 61, 63], NEYAI [64]). These primarily examined process elements that took into account the local geographic context. Evaluators noted that local variation in existing infrastructure, community capacity, networks and rurality impacted on implementation. Others observed that arbitrary administrative boundaries conflicted with the local place boundaries set by the initiative.

Impact study designs.

Impact (or outcome) evaluations examine the positive and negative effects of an intervention, using a set of pre-specified outcome measures [72]. An inclusion criteria for this review was that an impact study had been conducted. We examined the design of each impact study, the dataset(s) used, length of study, and the number and range of outcomes assessed (Table 1). Table 3 contains an overview of the findings for each initiative.

Impact evaluation studies varied considerably in design. Some initiatives used a combination of designs and data sources to assess impact. The ABC Programme [40] is described last in the following summary, as it was the only initiative that did not include a quasi-experimental design in their evaluation.

For the quasi-experimental impact evaluations, broadly, three types of sampling approaches were employed. Six initiatives (Communities for Children [10, 39], Sure Start [41, 42, 44, 45], Neighbourhood Nurseries Initiative [47], Flying Start [49–51], Smart Start [67], Toronto First Duty [62]) used a general population sample from geographic areas where the initiative was conducted, irrespective of which elements of the possible initiative had been delivered and irrespective of whether or not the sample had actually received any form of intervention. This approach sought to determine the whole-of-community, population level impact of the initiative. In a more tailored approach, three initiatives (Best Start [52–54], Georgia Family Connection [59], Neighbourhood Nurseries Initiative [47]) used an ‘intervention area’ or ‘targeted’ population sample. Again population level data were examined, but only included geographic areas where it was known that interventions designed to improve specific outcomes of interest had been implemented (for example, in Best Start, examination of breastfeeding rates only in the communities where a breastfeeding program had been provided [53]). Five initiatives (Neighbourhood Nurseries Initiative [47], Best Start [52], First Steps [55], NEYAI [64], Starting Well [65, 66]) assessed individual-level impact, using the less optimal approach of intervention samples comprising only participants known to have received some form of the intervention. Several initiatives used more than one type of design, using population-level data where available, and supplementing this with individual-level data for some outcomes of interest.

Seven initiatives used the stronger design of a cohort sample (Communities for Children [10, 39], Sure Start, Flying Start [49–51], Smart Start [67], NEYAI [64], Starting Well [65, 66]), while six used a cross-sectional sample (Sure Start [41], Neighbourhood Nurseries Initiative [47], Best Start [52–54], First Steps [55], Georgia Family Connection, Toronto First Duty [62]). Sure Start used both, reflecting a change in their study design part-way through the evaluation. Two initiatives used only their own collected data to assess impact (Communities for Children [10, 39], NEYAI [64]), four used only secondary datasets (Smart Start [67], Georgia Family Connection [59], Toronto First Duty [62], Starting Well [65, 66]), while five used a mix of both (Sure Start [41, 42, 44, 45], Flying Start [49–51], Best Start [52–54], Neighbourhood Nurseries Initiative [47], First Steps [55]). Initiatives using secondary datasets were more likely to have a cross-sectional impact study design.

The ABC Programme [40] used a pre- and post- evaluation design, comparing outcomes for parents and children who participated in the initiative (i.e., intervention sample). The initiative collected its own data using a set of core measures.

Four initiatives (ABC Programme [40], Best Start [52], NEYAI [64], Starting Well [66]) were most recently evaluated within three years of implementation. This was more common in demonstration projects. The longest time participants were followed up after implementation ranged between two years and 16 years, with a four to five year timeframe being the most common.

Synthesis of results.

Table 5 draws together information about the design of initiatives, their impact study design, theories of change and positive pregnancy & birth/child outcomes at population level to assist in drawing conclusions about effectiveness. It is difficult to draw definitive conclusions given the mixed quality, with three studies that did not measure outcomes at the population level, only four studies that measured whether outcomes were sustained over time, and one study that used a non-experimental design. Nevertheless, some inferences can be made. For the eight initiatives that used a population level sample, all found evidence of impact. For the four initiatives that measured population level impact over time (best design), three found evidence of sustained impact, but for one measure only. Given place-based initiatives are expected to improve outcomes across a range of measures, this is a somewhat disappointing result. Initiatives that used a targeted population sample were most likely to report positive results. For example, Best Start only measured the impact of the initiative on breastfeeding rates with communities where it was known that breastfeeding was specifically targeted, and found a positive effect [53]. Similarly, Georgia Family Connection identified the communities that targeted low birth weight and only included these communities in their study design. They too found a positive effect [59]. Initiatives that used routinely collected datasets to measure outcomes over longer time periods (Georgia Family Connection [59], Smart Start [57]) were more likely to demonstrate positive outcomes compared to purposely designed studies, yet were able to measure fewer outcomes due to the limitations of data availability. Initiatives that used a general population sample and a purposely designed study sample for their impact study and used a broader range of measures were less likely to find sustained positive effects (Communities for Children, Sure Start), although Communities for Children and Sure Start found positive effects in the early years that were not sustained over time [39, 45]. The ABC Programme [40] found positive effects across all outcomes it measured, however its pre- and post- evaluation design is considered a lower level of evidence compared to the more robust quasi-experimental design employed by the other initiatives examined.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Synthesis of results.

https://doi.org/10.1371/journal.pone.0261643.t005

Some initiatives used multiple designs within their evaluation framework. For example, the Neighbourhood Nurseries Initiative [47] used three different samples to assess for impact. In a general population sample (all parents living in a Neighbourhood Nursery Initiative ‘rich’ area) there was no evidence of impact on work status and childcare uptake. Similarly, in a targeted population sample (parents who were identified as being ‘work ready’ and living in a Neighbourhood Nursery Initiative ‘rich’ area) there was no evidence of impact. However in an intervention sample (participants who were known to have used the intervention) there was positive impact on work status and childcare uptake. Their examination of reach found that the initiative only reached 10% of the eligible population.

There is no clear relationship between the size of the local delivery area and initiative effectiveness, with initiatives implementing ‘local’ solutions at a large (e.g., county) and small (e.g., school neighbourhood) sized area demonstrating impact. Nor is there a clear relationship between the mechanisms by which the intervention was theorised to improve outcomes and effectiveness, although the inclusion of universal services (maternal and child health services, childcare, pre-school) in the service model of initiatives appeared to be mostly beneficial in demonstrating positive results.