What is a Theory of Change?

A Theory of Change (ToC) comprises a sequence of causal steps that describe how an intervention is expected to produce certain outputs and outcomes. These steps often include what inputs will be used, which activities will be conducted, what outputs will be produced, and, finally, what outcomes are expected to be achieved through this process, i.e, what “change” is expected to happen [1, 2]. According to the Aspen Institute [3], Theory of Change (ToC) can be understood as a tool to facilitate the development of solutions to complex social problems. A ToC is generally presented in graphic form, as a diagram, showing the connections between interventions and outcomes (causal pathways) while explicitly stating assumptions and related evidence [4]. This diagram usually represents a working model containing preconditions, expected results, rationales, assumptions, and indicators.

The concept of ToC was preceded, during the 50s and 60s, by an approach called Program Theory, which proposed a way of understanding and planning interventions with focus on the roles of context, input, processes, and products for intervention design [5]. Over time, increasingly more attention was given to explaining how an intervention was expected to work in terms of its underlying causal links [6]. In 1995, The Roundtable on Comprehensive Community Development, supported by the Aspen Institute [7], advocated for the importance of making it clear for stakeholders the assumptions of how change is to be achieved by a program, popularizing the term Theory of Change, described as “a theory of how and why an initiative works” [8].

The concept of ToC is related to, and often overlaps with, several other tools and frameworks, such as Logical Model, Program Theory, Action Theory or Logical Framework. A Logical Model, for instance, is defined as a graphical representation of a program theory, showing the relevant causal connections up to the results [5]. A Program Theory is an explicit theory of how an intervention is expected to achieve its goals [5]. An Action Theory, on the other hand, describes the pathways for developing an intervention. Finally, a Logical Framework, or Logframe, is a method for planning and evaluation, usually specifying inputs, activities, outputs, outcomes, and impacts in a matrix, and possibly also including indicators and assumptions [9, 10]. Logframes are considered less flexible than ToC diagrams, and more limited in representing multiple, interacting, and nonlinear causal pathways. Importantly, ToCs should make assumptions and contextual factors explicit to explain the causal steps, i.e. why and how an intervention is expected to lead to the anticipated changes [9]. Furthermore, a core feature that distinguishes ToC from other similar approaches is the emphasis on it being a living process, constantly updated based on lessons learned [11, 12].

These steps often include what inputs will be used, what activities will be conducted, what outputs will be produced, and, finally, what outcomes are expected to be achieved through this process. Fig 1 below presents a model of how a ToC can be depicted, including how to include assumptions and indicators. The following Fig 2 illustrates a concrete ToC for interventions addressing Domestic Violence (DV) against pregnant women:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Example of Theory of Change framework.

Source: De Silva et al., 2014, p.4 [13].

https://doi.org/10.1371/journal.pone.0282808.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Example of theory of change for an intervention.

Source: Sapkota et al., 2019, p.6 [14].

https://doi.org/10.1371/journal.pone.0282808.g002

In this review, Theory of Change is defined, according to De Silva et al. (2014b, p.2) [15], as follows:

"ToC is ‘a theory of how and why an initiative works’ which can be empirically tested by measuring indicators for every expected step on the hypothesized causal pathway to impact. It is developed in collaboration with stakeholders and modified throughout the intervention development and evaluation process through an ‘ongoing process of reflection to explore change and how it happens’. It is visually represented in a ToC map which is a graphic representation of the causal pathways through which an intervention is expected to achieve its impact within the constraints of the setting in which it is implemented".

Visually, each intervention is connected to an expected result, in a causal structure, in order to reveal the necessary activities to promote change [16] and without losing the implicit complexity of the actions. Moreover, the connections between inputs, outputs, and outcomes are explained to demonstrate how and why certain actions are expected to lead to change [16].

ToC can be used for many purposes. It serves as a tool to plan an intervention since it makes all steps and assumptions explicit and visible. It can also be used as a tool for communicating and discussing the rationale of the intervention with staff members, stakeholders, the target population, and other audiences. Therefore, a ToC can be useful for providing transparency about why certain actions have been taken and allows for the intervention to be better replicated in the future, facilitating coordination and collaboration. Finally, given its ability to depict the causal process through which an intervention is meant to work and what change it is expected to produce, ToC can also inform monitoring and evaluation processes. In depicting the key steps and expected deliverables, it facilitates the identification and development of key indicators [9, 16, 17].

While the ToC approach is most commonly used as an intervention planning and design framework, it can also be used after an intervention has been implemented as a way of synthesizing empirical evidence of a pilot study [18], a single intervention [18], or a collection of studies [14, 19–22].

By articulating long-term results, pre-conditions, and interventions, the ToC provides a basis for implementing and documenting policies, strategic plans, and monitoring and evaluation processes [16].

ToC has been widely used in the health sector for planning, depicting, implementing, and evaluating interventions [11, 12, 23, 24]. For instance, Barnhart et al. (2020) [18] developed a ToC based on the experience of three different implementation phases of a maternal and child health program, the Better Birth program [18]. Barriers, lessons learned and ways of applying these lessons were identified, and later integrated into a renewed ToC. In another example, Aggarwal (2021) [19] reviewed the literature for potential interventions and delivery mechanisms for reducing the self-harm of individuals in countries of low and middle income [19]. Based on the findings of the review, the authors propose a ToC for a model of delivery of psychosocial interventions. Furthermore, in the field of Implementation Science, ToC has been a recurring choice to describe models, plan solutions to complex problems and evaluate programs in the health area, as described by Breuer et al. (2016) [23]. Recent reviews suggest there is a growing number of published evaluations in the health area that use ToC, as identified by Breuer et al. (2016) [23] and Lam et al. (2021) [24]. However, both reviews highlight that studies, overall, lack an appropriate level of detail regarding how the ToCs were developed and used.

The use of evidence in Theory of Change frameworks

Evidence-informed decision-making has been increasingly used as a method in clinical policy and practice. The use of evidence makes the decision-making processes more explicit, favoring transparency, reproducibility, credibility, and reliability [25]. We argue that the development and use of ToC frameworks should, therefore, also include explicit and systematic considerations on how to incorporate evidence.

In this review, we will follow the Lomas et al. (2005) [26] conceptualization of evidence as “facts (actual or asserted) intended for use in support of a conclusion” (p.1). The authors [26] distinguish colloquial evidence (or tacit knowledge), provided by stakeholders’ opinions and viewpoints, from scientific evidence (or explicit knowledge), gathered by a prescribed set of procedures in order to produce knowledge recognized as explicit, systemic, and replicable, and further distinguishes two scientific views on evidence depending on whether they emphasize context-free general truths (more aligned with evidence-based medicine), or context-sensitive facts (more aligned with the applied social sciences). The authors also highlight that combining and interpreting these forms of evidence requires a deliberative process, often involving the participation of stakeholders to judge the relative weight of different pieces of evidence.

In theory, a ToC would be based on the analysis of available scientific evidence (explicit knowledge) and the consultation and deliberation with stakeholders (tacit or colloquial knowledge) [2] developed through rigorous and participatory processes. ToC approaches, however, usually emphasize stakeholder input (stakeholder consultation in the development of a plan to articulate the initial conditions, objectives, and means to achieve the desired results in a causal structure [16]) while making limited use of explicit or academic knowledge such as program evaluations and published research [23].

Two literature reviews have previously investigated the methods used to incorporate evidence into ToCs in the health sector. In a systematic review, Breuer et al. (2016) [23] investigated how ToCs are used to design and evaluate public health interventions. The authors identified the following methods used to incorporate evidence into ToC frameworks: “workshops and working groups, document reviews, interviews and discussions, surveys, program observation, literature reviews, and existing conceptual frameworks or theory, […] consultations or interviews [with stakeholders]” (p.4). Moreover, in a scoping review, Lam et al. (2021) [24] investigated how ToCs are used in food security contexts. The authors find that the methods applied for using evidence in ToCs were primarily participatory approaches, such as workshops or interviews, i. e., favoring tacit knowledge. They also report that most studies do not provide details on how the ToCs were developed, which is “consistent with the sentiments in the literature, which highlighted the limited guidance, training, and agreement on how to develop a ToC” (p.7) [24].

In this review, we expand the existing literature by focusing on how to systematically use research evidence to inform the development of ToCs in the health sector. Consistent with the previous reviews, our findings below show that almost none of the recent studies have systematically investigated the use of evidence to inform ToCs in the health sector. Furthermore, we did not encounter any study that provides a comprehensive and rigorous account on what methods can or should be used to incorporate evidence into ToCs. Most of the primary studies that address how a ToC was developed only give brief descriptions of the methods applied to incorporate evidence into this process. These methods include, for instance, structured stakeholder consultation, literature searches and the use of institutional data, which are applied as sources of information to draft or improve ToCs. However, each study seemed to choose specific versions of these methods (e.g. stakeholder consultations) without a clear rationale for why this choice was made over the alternatives (e.g. a literature review).

This project was commissioned by the World Health Organization, which identified the need for more explicit use of evidence in ToC development. This rapid literature review specifically aims to advance our knowledge on how to enhance the systematic and transparent use of the best available evidence in the ToC development for decision-making in the health sector. The rapid review methodology was chosen to ensure agility and efficiency in applying the available resources while keeping methodological rigor.

This rapid review adds to the existing literature in two ways. First, it provides an up-to-date and comprehensive review of the existing methods for systematically incorporating evidence into ToC frameworks in the health sector. Second, it creates a typology to classify the findings from the literature. In this typology, we propose a sequence of 7 stages relevant to the process of evidence incorporation into ToCs. We also indicate which research methods the included studies applied for each of the proposed stages. These 7 stages were developed through a qualitative analysis of the data extracted from the included studies (further details provided in the “Data Synthesis” section below). Each stage corresponds to a specific set of objectives. The stages are: 1) "Define the problem", 2) "Define expected outcomes", 3) "Define interventions", 4) "Define change mechanisms", 5) "Model ToC", 6) "Validate ToC" and 7) "Revise ToC".

Research questions

To that end, we will address the following key questions:

1. What are the available tools, methods, and recommendations to systematically incorporate research evidence into the development and adaptation of Theories of Change in the health sector?
2. What are the main principles, processes, and practical steps recommended by the literature on how to systematically incorporate research evidence into the development and adaptation of theories of change in the health sector?

To answer these questions, the acronym PCCS was used, as follows (Table 1):

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. PCCS acronym for the research question.

https://doi.org/10.1371/journal.pone.0282808.t001

Study design

This project applied a rapid review methodology. Rapid reviews can be defined as a type of literature review in which “the steps of the systematic review are streamlined or accelerated to produce evidence in a shortened timeframe." (p. XIII) [27]. In a rapid review, evidence is synthesized and its validity is assessed using an abbreviated systematic review method, so to identify results in a shorter timeframe. Regardless of the abbreviated approach, this endeavor followed the principles of the scientific method and the key principles of evidence synthesis, including rigor, transparency, and reproducibility. This rapid review was conducted and reported following the process outlined by Tricco et al. (2017) [27]. We used a rapid review approach to be able to provide high quality evidence for decision making, in a timely way. Given the time and budget limitations of this project, some adaptations were made to adjust to the needs of the rapid review design. These adaptations are described in the section “Shortcuts taken for the rapid response” below.

The study protocol was published in the Open Science Framework (OSF) platform (register: osf.io/t53sm) [28]. After the publication of the protocol, we made an addition in the research design by expanding the inclusion criteria also to include scoping reviews.

Eligibility criteria

The inclusion criteria applied in this review were:

• Topic: studies that investigated how research evidence was used in the development and adaptation of ToC in health policies; or studies that explicitly described how evidence was or should be used to inform the development or adaptation of the ToC in the health sector;
• Study type: systematic reviews, scoping reviews, guides, or methodological studies. The methodological studies could be case studies, qualitative or quantitative studies, as long as their primary focus was how to develop or update a ToC. We excluded empirical primary studies, overviews, integrative reviews, evidence syntheses, technology assessments, and economic assessments;
• Language: published in English, Spanish, or Portuguese;
• Date: published on any date;
• Place: any country or region.

Shortcuts taken for the rapid review

In this rapid review, only the screening process was done by two reviewers independently. Data extraction and methodological quality assessment of the studies were done by one researcher and reviewed by a second.

Data sources and searches

Published articles were searched on November 8, 2021, in the following databases: PubMed, BVSalud, Cochrane Library, Health Systems Evidence (HSE), Health Evidence, and Epistemonikos.

Gray literature was retrieved from OpenGrey and Google Scholar search engines limited to the first ten pages. A list of references obtained from a WHO researcher that had been previously investigating the topic was also screened. Due to time constraints, no additional search for gray literature was conducted.

Search strategies were developed based on the combination of keywords structured by the acronym PCCS, using the MeSH search terms on the PubMed database. The strategy was adapted to the other databases.

Additional studies were retrieved from a list provided by a topic expert and the WHO library.

The implemented searches can be found in S1 Appendix.

Study selection and data screening

Study screening was preceded by calibration of the inclusion and exclusion criteria among all reviewers using a sample of the search results. After calibration, two reviewers selected the studies through title and abstract screening independently. Disagreements were decided by a third reviewer.

After title and abstract screening, two researchers independently screened the included studies in full-text. Conflicts were resolved through consensus or by a third reviewer.

The screening was conducted in the reference management software, Rayann QCRI [29].

Missing data

Only reports published in Portuguese, Spanish, and English were included, and potentially relevant information in other languages was not considered. In addition, primary studies that addressed the use of ToC to apply evidence in the health sector but that were not included in systematic reviews or scoping reviews were also not covered by this study. Finally, relevant literature from other fields than health might also have had relevant information to answer this review’s question, but they were not included.

Data extraction

Data extraction was conducted according to the research questions. A data extraction spreadsheet was made, observing the main elements to inform this rapid review, and covered the following categories: first author, published year, study objective, population and sample characteristics, method, main findings (divided into Definition of ToC, Participants in the ToC development, Steps for the development of a ToC, Steps for revising a Theory of Change, Use of research evidence in the development of a ToC, and Use of research evidence for revising a ToC), study limitations, conclusions, conflicts of interest and last year of search. In the subcategories “Use of research evidence in the development of a ToC” and “Use of research evidence for revising a ToC”, we captured all relevant information from the primary studies regarding their reported strategies for evidence incorporation into ToCs. Data for the other subcategories were collected to identify and contextualize each study and provide further information on how ToCs were used and how evidence was considered. The fields “Definition of ToC”, “Steps for the development of a ToC” and “Steps for revising a Theory of Change” were added to the data extraction template after a round of initial piloting of the data extraction process to facilitate the interpretation of the research findings.

Quality assessment

Quality assessment was done by one researcher and reviewed by another. Systematic reviews were assessed using the AMSTAR 2 tool [30]. Systematic reviews were classified according to their confidence degrees as high, moderate, low, or critically low. This classification followed the AMSTAR 2 critical domains that check for the presence of a search protocol, search strategy, inclusion and exclusion criteria, assessment of the risk of bias, and interpretation of results. To assess the quality of the scoping review, we used the Joanna Briggs Institute’s Checklist for Systematic Reviews [31]. To evaluate the studies self-described as methodological, we used the Joanna Briggs Institute’s Checklist for Qualitative Studies [31]. For the other included studies, that we also considered methodological, we applied quality assessment tools compatible with each of their specific designs, they were: Joanna Briggs Institute’s Checklist for Qualitative Studies [31] (4 studies), Joanna Briggs Institute’s Checklist for Text and Opinion [31] (2 studies), Scale for the Assessment of Narrative Review Articles (SANRA) [32] (1 study), and Critical Appraisal of a Case Study tool from the Center for Evidence-Based Management [33] (1 study). The quality assessment results can be found in S2 Appendix.

Data synthesis

After data extraction, studies were compared, and the findings were summarized qualitatively into themes, in the form of key principles, steps, and procedures for the systematic incorporation of research evidence into the development and adaptation of a ToC. A qualitative narrative synthesis was conducted regarding “Definitions of ToC”, “Participants in the ToC development”, “Methods for the development of the ToC” and “Use of research evidence in the development and revision of a ToC”.

To further contextualize how and when evidence can be integrated into the ToC development process, we also identified, based on the available literature, the main stages relevant for the process of evidence incorporation into ToCs. The basis for this process was the extracted data regarding the steps for developing and revising a ToC and the strategies for evidence use (see S6 Appendix for the relevant extracted data). Through an inductive qualitative data analysis process, we created a typology of stages and coded the extracted information accordingly. When grouping the steps based on content similarity, we realized they could be classified with labels similar to the phases often included in the “policy cycle”, such as “Policy Formulation”, “Implementation” and “Evaluation” (p.33) [34]. Thus we regrouped the data following this framework, adapting some of these phases as labels (i.e. "Define the problem", "Define interventions"), and creating new ones when necessary (i.e. "Define change mechanisms", "Model ToC", "Validate ToC", "Revise ToC") based on our content similarity analysis. Here, we present these labels as stages for the development of ToCs. The different stages reflect the objectives and functions of each step, as shown in section “Stages for the development of a ToC” below and in S3 Appendix. Thereafter, we conducted a frequency count to assess how many of the included studies reported each stage. For ease of understanding, these stages were organized in a logical sequence, according to our interpretation.

In addition to categorizing the stages for developing a ToC, we classified the extracted data in terms of “types of activities” conducted. This typology was also created through an inductive qualitative data analysis process, based on the original extracted material. Within each stage, we grouped the included methods according to the specific actions that were conducted. While the classification in stages reflects the goal or purpose of each step taken, the classification into the types of activities summarizes the concrete actions undertaken to achieve this goal (see section “Stages for the development of a ToC” below).

Study inclusion

The search retrieved 880 records, of which 141 were removed as duplicates. 698 articles were excluded at the title and abstract screening as they did not meet the eligibility criteria. 41 documents were selected for full-text screening, of which 23 were excluded for not discussing ToC, the use of evidence, or not matching the inclusion criteria. In addition, we screened 23 articles recommended by an expert on the topic. After excluding duplicates, the remaining studies were screened in full-text. Based on the exclusion criteria, all of these studies were excluded. The excluded articles and the reasons for exclusion are listed in S4 Appendix, separated between articles found in the databases and those recommended by the expert (Fig 3). Finally, 18 remaining articles were included in this rapid review.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. PRISMA flowchart.

Source: authors’ elaboration, adapted from PRISMA 2020 [35].

https://doi.org/10.1371/journal.pone.0282808.g003

Despite there being no limitation on the year of publication as an inclusion criterion, all included studies (n = 18) were published between 2013 and 2021 [1, 13, 14, 18–24, 36–43], and among those, the mode (n = 5) was the year 2021 [19, 22, 24, 39, 42]. For more details on the characteristics of each study and the data extracted, see S5 and S6 Appendices.

Characteristics of included studies

The studies presented different designs with seven systematic reviews [19, 21–23], one scoping review [24], and ten studies with other designs that were considered “methodological” since their primary focus was on how to develop or update a ToC. The included methodological studies had a variety of study methods, as reported by their respective authors: case study (n = 1) [18], mixed methods (qualitative and quantitative) (n = 1) [42], methodology studies (n = 2) [36, 37], report (n = 1) [40], formative research (n = 1) [13]. Four publications [1, 20, 38, 43] did not name the study design. They were classified by the authors of this rapid review as methodological studies based on the information described in the method section, for the purpose of quality assessment.

Regarding the countries conducting the studies, five studies were carried out in countries that, according to The World Bank (2021) [44], are classified as high-income economies (HIE) [21, 23, 36, 37, 42], one in Lower-middle-income economies (LMIE) [14], and one in Upper-middle-income economies (UMIE) [43]. Two studies included both HIE and LMIE [18, 19]. The remaining studies did not report this information or it did not apply to the study design [13, 20, 22, 38–41]. Except for De Buck et al. (2018) [1], which had authors from both HIE and LMIE, the authors of the other 17 included studies were affiliated to institutions from HIE [13, 14, 18, 19, 22, 24, 36–43].

Sixteen studies reported no conflict of interest [1, 13, 14, 18–20, 24, 36, 39]. One study included information about funding but not on competing interests [40]. One study did not have information on either [20].

The included studies focused on the following themes and sectors: public health [13, 21, 23, 40], nutrition [24, 38], healthy eating and physical exercise [20], mental health [1, 22, 36], health promotion [1], violence prevention [4], obstetrics [18], immunization data [39], research-policy gap [43], realist RCTs [37], and reducing sedentary behavior [42].

Methodological quality of included studies

The methodological quality of seven systematic reviews was assessed using AMSTAR-2 [30]. Five of these were considered critically low [19, 21–23, 39] and two considered low [14, 41]. None of the publications provided a list of excluded studies with justifications [14, 19, 21–23, 39, 41]. In addition, the five critically low did not account for biases [10, 12, 13, 17, 19, 21–23], or did not refer to methods being established prior to the review [39]. Six studies were assessed with the JBI Critical Appraisal Checklist for Qualitative Research [31]. These studies contained two important weaknesses: no statement locating the researcher culturally or theoretically, nor any mention of approval by the ethics committee [1, 20, 36, 37, 42, 43]. One study [24] was assessed with the JBI Critical Appraisal Checklist for systematic reviews and research syntheses [31]. However, methods to mitigate potential errors in the findings in this study were not found. Two studies were assessed with the JBI Critical Appraisal Checklist for text and opinion papers [31], and the reports were considered satisfactory [13, 40]. One study [38] was assessed with SANRA [32] and had a final score of 10/12. The two limitations of the study were that the electronic search report was not detailed enough and the selection of studies was not well described. One case study was assessed with the CEBMA tool [33]. In this study, there is no mention of whether the data analysis was repeated by another researcher and it is not clear whether the researcher’s perspective was taken into account [18]. More details are found in S2 Appendix.

Definitions of ToC

Nine studies [1, 13, 14,18, 22, 23, 38, 40, 43] presented a definition of ToC, with some differences between them, as shown in Table 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Definitions of ToC from the included studies.

https://doi.org/10.1371/journal.pone.0282808.t002

Participants in the ToC development

Four studies reported that their respective authors participated in the development of the ToC [22, 36, 39, 40]. Other studies included multidisciplinary groups composed of professionals [42], service users [20], program evaluators and implementers, community members [18], methodological experts, and experts on the topic [1]. Ten studies did not report any information about the participants in the development of the ToC [13, 14, 19, 21, 23, 24, 37, 38, 41, 43].

Methods for the development of the ToC

Some of the studies [14, 19, 20] followed existing methods for intervention design in their efforts to build a ToC. The methods used were De Silva (2014) [13], the Six steps in quality intervention development (6SQuID) [45] framework [36, 42], and the guidelines [43] offered by the World Bank [46] and Morra and Rist [47].

Three studies [21, 22, 41] synthesized ToC evidence from previously published ToCs. Tancred et al. (2018) [41], for instance, conducted a systematic review to synthesize relevant elements for a ToC for school-based interventions integrating health and academic education to prevent violence and substance use among students. This review searched for other ToCs that were synthesized through a meta-ethnographic analysis.

Stages for the development of a ToC

The included studies generally did not provide a detailed account of what steps were taken to develop their respective ToCs. However, from the available information, it is possible to see that the development of the ToC was rarely a linear or hierarchical process. In reality, the development process is continuous, involving reflection and adaptation as new evidence comes to light, requiring the causal pathways to be changed [15].

All studies included in this review presented at least one stage adopted for the construction of the ToC. In total, seven development stages were identified. The most frequent type was “Define interventions” [1, 19, 22, 36, 39, 41, 42] and “Model ToC” [13, 18–20, 22, 23, 38, 42, 43], mentioned in 55.5% (n = 10) of the included studies. All identified stages and their respective frequencies can be found in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Frequency of each stage of ToC development.

https://doi.org/10.1371/journal.pone.0282808.t003

Table 5, below, presents these stages with further details. In addition to categorizing the stages for developing a ToC, we also classified the extracted data in terms of the “activities” conducted. While the classification in stages reflects the goal or purpose of each step taken, the classification into the activities summarizes the concrete actions undertaken to achieve this goal. These activities are depicted in the second column of Table 4. In the third column of this table, “Activities’ details”, we paraphrase the steps described in the included studies to present guidance on how to undertake each activity. The original material can be found in S3 Appendix.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Synthesis of Stages in ToC development and their respective activities.

https://doi.org/10.1371/journal.pone.0282808.t004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Stages for ToC development.

https://doi.org/10.1371/journal.pone.0282808.t005

We emphasize that among the “Activities”, the lines “Identify and synthesize evidence base” refer to the processes of searching, selecting, producing, and synthesizing evidence for the corresponding stages of the ToC development. In these cases, the use of the term evidence is encompassing both tacit or colloquial evidence and explicit or published evidence, as further detailed in Table 5.

Use of research evidence in the development and revision of a ToC

All included studies mentioned at least one source of evidence used to inform the development of the ToC. However, generally, the studies did not provide a structured and detailed discussion of how the incorporation of evidence was undertaken. Evidence sources included: literature reviews [1, 19, 20, 23, 36, 37, 42, 43], either systematic reviews or narrative reviews; review of reports and normative documents; surveys, interviews, meetings, workshops and focus groups with key actors, including community members and stakeholders [1, 13, 18, 20, 23, 24, 36, 38–40]; previous empirical studies; existing related psychological and sociological theories [1, 21, 37]; consultation with experts [18–20]; and previous theories of change [21, 22, 41]. Many studies [18, 20, 23, 36, 37, 39] combined more than one evidence source for the development of the ToC; such as a combination of literature review, consultations with stakeholders, and previous empirical studies.

The ToC diagram can be developed in a meeting with stakeholders, often in a workshop format with the help of a trained facilitator [13]. However, it is common to find the implementation or evaluation team making a first draft based on the literature review and then validating the draft in a stakeholder meeting. Mayne and Johnson (2015) [38] note that while bringing a pre-drafted ToC may decrease the level of buy-in, it makes it easier to bring relevant research findings into it.

De Silva et al. (2014) [13] report that in their study, the development of the ToC was supported by a ToC expert. They also conducted brainstorming activities with key actors to gather information about expected outcomes. Afterward, a survey with stakeholders was carried out, a workshop was set up to foster debate, and, lastly, the ToC was further refined through another group session.

Few studies reported on how to use evidence to revise a ToC. The ToC was revised both in the initial development process, by experts [1, 19, 20] or peer review [1], and after the intervention was initiated, in response to evidence gathered by a discussion with practitioners [40], hypothesis testing (mechanisms and moderators evaluations) [37] and program evaluation [36].

Summary of main findings

This rapid review identified 18 studies that addressed the use of evidence in ToC or in the revise of a ToC in the health sector. This study adds to the existing literature in two ways. First, it provides an up-to-date and comprehensive review of what are the existing methods and recommendations for incorporating evidence into ToC frameworks in the health sector. Second, it creates a typology to classify the findings from the literature. In this typology, we propose a sequence of seven stages relevant to the process of evidence incorporation into ToCs.

An initial challenge for this review was the scarcity of data. In line with the conclusion from Breuer et al. (2016) [23] and Lam et al. (2021) [24], we find that authors rarely detail how the ToCs were developed and even more rarely include explicit considerations about evidence incorporation in the process. Future research projects should consider using the Breuer et al. (2016) [23] proposed checklist for reporting ToC in Public Health Interventions. It has five criteria, assessing the ToC approach, the ToC development process, presentation of the ToC, description of the intervention process, and use of the ToC for evaluating a policy. This checklist, however, has not yet been validated and might not be appropriate for all possible uses of a ToC. As such, it may require adaptations depending on the context in which it is used.

The studies included in this review, nevertheless, provide a rich portrayal of the field. The findings corroborate the definition of ToC provided by De Silva et al. (2014) [13], according to which a ToC graphically presents how and why an intervention is expected to work and, in general, is developed by a participatory process [4].

We observed significant diversity across topics and areas within the health sector in which ToC approaches can be applied. It was also noted that there is no single way to develop a ToC that suits all purposes, but that adaptations are made according to the context of its application.

Stages for ToC development

The stages for developing a ToC were observed to be different in the included studies. The process of developing a ToC has multiple possibilities regarding its form and content, being essentially dynamic and nonlinear. However, for didactic and visual purposes, we classified and organized these stages in a logical order (See Table 5, below).

It is important to highlight that this logical sequence is our interpretation, based on a qualitative analysis of the extracted data. As we mentioned before, the authors often do not provide a structured presentation of how the ToCs were developed. Moreover the stages are rarely followed in this strict logical sequence.

Prior to the stages, it is also important to discuss the purpose and goal of the ToC. As we mentioned above, ToCs can have many purposes, such as supporting the implementation of an intervention, supporting monitoring and evaluation efforts, sharing and aligning knowledge and expectations about outcomes among stakeholders, synthesizing knowledge and communicating to a specific audience. Given the time, work, and coordination requirements to gather stakeholders and research evidence to develop a ToC, it is paramount that its goals are well defined beforehand.

Methods for evidence incorporation into ToC development

From Table 4 above, it is possible to observe how different sources of evidence are used in the development of ToCs. There were three main sources of evidence: literature search, consultation with key actors, and use of institutional data. More specifically, there were a variety of ways to use evidence in ToC, such as searching for systematic reviews, conducting systematic and narrative literature reviews, using primary studies, using theoretical frameworks, using institutional and epidemiological data, conducting meetings, interviews, surveys, and workshops with stakeholders, or a combination of these components. Therefore, both colloquial and scientific evidence were commonly used sources of knowledge to support ToCs. Four studies reported the use of evidence for revising a ToC. The findings showed that revising a ToC can be done by evaluating programs, mechanisms, and moderators, and by consultation with experts and other key actors.

In Table 6, below, we synthesize the methods for evidence identification and synthesis presented in the Results section (see Table 4). These methods are organized by ToC development stage:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Methods used by the included studies for evidence incorporation.

https://doi.org/10.1371/journal.pone.0282808.t006

Table 6 describes what methods for evidence incorporation the included studies applied for the different stages of ToC development, with the stage “Define expected outcomes” supported, mostly, by the use of scientific evidence, both at a global and local level. As for the “Validate ToC” stage, we identified the use of evidence only at the local level, but, in a way, incorporating both scientific and colloquial evidence, through experts and users. We also observe that for the “Define the problem”, “Define intervention” and “Revise ToC” stages, authors explore evidence both at a global and local level, including scientific and colloquial knowledge. This demonstrates the ability of a ToC to be built from different sources of knowledge, considering colloquial evidence without giving up on scientific rigor.

Table 6, however, should not be understood as a recommendation of which methods should be applied for evidence incorporation into ToCs, nor as a conceptually informed exhaustive list of the possibilities in that regard. The included studies did not present justification as to why a specific method was chosen, or why other methods were left out. Moreover, no study provided a normative account on how to incorporate evidence into ToC development, nor provided specific guidance for that purpose.

Therefore, we identify a gap in the existing literature. The fact that only 18 studies explicitly discuss how to incorporate evidence into ToC development in the health sector, and none offers guidance or a framework for this shows that it is a neglected area. This stands in sharp contrast with the increasing recognition of evidence as an indispensable element for policymaking in the health sector.

This literature gap is in line with the findings from Lam et al. (2021) [24], which report a fragmentation in the field and a lack of guidance and conceptual consistency regarding how to develop a ToC. There is, therefore, an existing need for a new theoretical framework, offering clear advice on how to incorporate evidence into ToC development. This framework should draw on the existing practice, as discussed in this rapid review, while also integrating methodological and conceptual considerations, to guide the many ways through which evidence can be used to inform ToCs.

Strengths and limitations of the present study

This project applied a rapid review methodology, following the guidance provided by Tricco et al. (2017) [27]. Regardless of the abbreviated approach, this endeavor respected the scientific method and the key principles of evidence synthesis, including rigor, transparency, and reproducibility. Furthermore, the shortcuts adopted in this review allowed us to focus on the most critical and relevant literature to our research questions.

A few adaptations from a complete systematic review were made. First, only studies involving the health sector were included in the eligibility criteria, which may have limited relevant information about ToCs. Second, the inclusion criteria for systematic reviews, scoping reviews, and methodological studies and guidelines focusing on the development of theories of change in the health sector may not have covered the entire literature on the use of evidence in the theory of change. Third, only studies in English, Spanish, and Portuguese were included, due to the language skills of the reviewers. Fourth, data extraction and the methodological quality assessment of the studies were conducted by one reviewer and checked by another. Fifth, some of the included studies did not explicitly state their research design, which required the reviewers to classify these studies as a specific type to choose and apply the methodological assessment tools that were the most adequate.