Theory of Change (ToC) has become an established approach to design and evaluate interventions. While ToC should—in line with the growing international focus on evidence-informed health decision-making–consider explicit approaches to incorporate evidence, there is limited guidance on how this should be done. This rapid review aims to identify and synthesize the available literature on how to systematically use research evidence when developing or adapting ToCs in the health sector.
A rapid review methodology using a systematic approach, was designed. Eight electronic databases were consulted to search for peer-reviewed and gray publications detailing tools, methods, and recommendations promoting the systematic integration of research evidence in ToCs. The included studies were compared, and the findings summarized qualitatively into themes to identify key principles, stages, and procedures, guiding the systematic integration of research evidence when developing or revising a ToC.
This review included 18 studies. The main sources from which evidence was retrieved in the ToC development process were institutional data, literature searches, and stakeholder consultation. There was a variety of ways of finding and using evidence in ToC. Firstly, the review provided an overview of existing definitions of ToC, methods applied in ToC development and the related ToC stages. Secondly, a typology of 7 stages relevant for evidence integration into ToCs was developed, outlining the types of evidence and research methods the included studies applied for each of the proposed stages.
This rapid review adds to the existing literature in two ways. First, it provides an up-to-date and comprehensive review of the existing methods for incorporating evidence into ToC development in the health sector. Second, it offers a new typology guiding any future endeavors of incorporating evidence into ToCs.
Citation: Romão DMM, Setti C, Arruda LHM, de Melo RC, de Araujo BC, Tan AR, et al. (2023) Integration of evidence into Theory of Change frameworks in the healthcare sector: A rapid systematic review. PLoS ONE 18(3): e0282808. https://doi.org/10.1371/journal.pone.0282808
Editor: Muhammad Shahzad Aslam, Xiamen University - Malaysia Campus: Xiamen University - Malaysia, MALAYSIA
Received: August 26, 2022; Accepted: February 22, 2023; Published: March 9, 2023
Copyright: © 2023 Romão et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting information files.
Funding: This project was funded by the World Health Organization (https://www.who.int). The authors declare no known financial conflicts of interest. TK is employed full-time with WHO, working in the fields of research, evidence and policy. PND had consultancy contracts with WHO when the work on the Theory of Change was commissioned. The authors alone are responsible for the views expressed in this paper and they do not necessarily represent the views, decisions or policies of the institutions with which they are affiliated. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
What is a Theory of Change?
A Theory of Change (ToC) comprises a sequence of causal steps that describe how an intervention is expected to produce certain outputs and outcomes. These steps often include what inputs will be used, which activities will be conducted, what outputs will be produced, and, finally, what outcomes are expected to be achieved through this process, i.e, what “change” is expected to happen [1, 2]. According to the Aspen Institute , Theory of Change (ToC) can be understood as a tool to facilitate the development of solutions to complex social problems. A ToC is generally presented in graphic form, as a diagram, showing the connections between interventions and outcomes (causal pathways) while explicitly stating assumptions and related evidence . This diagram usually represents a working model containing preconditions, expected results, rationales, assumptions, and indicators.
The concept of ToC was preceded, during the 50s and 60s, by an approach called Program Theory, which proposed a way of understanding and planning interventions with focus on the roles of context, input, processes, and products for intervention design . Over time, increasingly more attention was given to explaining how an intervention was expected to work in terms of its underlying causal links . In 1995, The Roundtable on Comprehensive Community Development, supported by the Aspen Institute , advocated for the importance of making it clear for stakeholders the assumptions of how change is to be achieved by a program, popularizing the term Theory of Change, described as “a theory of how and why an initiative works” .
The concept of ToC is related to, and often overlaps with, several other tools and frameworks, such as Logical Model, Program Theory, Action Theory or Logical Framework. A Logical Model, for instance, is defined as a graphical representation of a program theory, showing the relevant causal connections up to the results . A Program Theory is an explicit theory of how an intervention is expected to achieve its goals . An Action Theory, on the other hand, describes the pathways for developing an intervention. Finally, a Logical Framework, or Logframe, is a method for planning and evaluation, usually specifying inputs, activities, outputs, outcomes, and impacts in a matrix, and possibly also including indicators and assumptions [9, 10]. Logframes are considered less flexible than ToC diagrams, and more limited in representing multiple, interacting, and nonlinear causal pathways. Importantly, ToCs should make assumptions and contextual factors explicit to explain the causal steps, i.e. why and how an intervention is expected to lead to the anticipated changes . Furthermore, a core feature that distinguishes ToC from other similar approaches is the emphasis on it being a living process, constantly updated based on lessons learned [11, 12].
These steps often include what inputs will be used, what activities will be conducted, what outputs will be produced, and, finally, what outcomes are expected to be achieved through this process. Fig 1 below presents a model of how a ToC can be depicted, including how to include assumptions and indicators. The following Fig 2 illustrates a concrete ToC for interventions addressing Domestic Violence (DV) against pregnant women:
Source: De Silva et al., 2014, p.4 .
Source: Sapkota et al., 2019, p.6 .
In this review, Theory of Change is defined, according to De Silva et al. (2014b, p.2) , as follows:
"ToC is ‘a theory of how and why an initiative works’ which can be empirically tested by measuring indicators for every expected step on the hypothesized causal pathway to impact. It is developed in collaboration with stakeholders and modified throughout the intervention development and evaluation process through an ‘ongoing process of reflection to explore change and how it happens’. It is visually represented in a ToC map which is a graphic representation of the causal pathways through which an intervention is expected to achieve its impact within the constraints of the setting in which it is implemented".
Visually, each intervention is connected to an expected result, in a causal structure, in order to reveal the necessary activities to promote change  and without losing the implicit complexity of the actions. Moreover, the connections between inputs, outputs, and outcomes are explained to demonstrate how and why certain actions are expected to lead to change .
ToC can be used for many purposes. It serves as a tool to plan an intervention since it makes all steps and assumptions explicit and visible. It can also be used as a tool for communicating and discussing the rationale of the intervention with staff members, stakeholders, the target population, and other audiences. Therefore, a ToC can be useful for providing transparency about why certain actions have been taken and allows for the intervention to be better replicated in the future, facilitating coordination and collaboration. Finally, given its ability to depict the causal process through which an intervention is meant to work and what change it is expected to produce, ToC can also inform monitoring and evaluation processes. In depicting the key steps and expected deliverables, it facilitates the identification and development of key indicators [9, 16, 17].
While the ToC approach is most commonly used as an intervention planning and design framework, it can also be used after an intervention has been implemented as a way of synthesizing empirical evidence of a pilot study , a single intervention , or a collection of studies [14, 19–22].
By articulating long-term results, pre-conditions, and interventions, the ToC provides a basis for implementing and documenting policies, strategic plans, and monitoring and evaluation processes .
ToC has been widely used in the health sector for planning, depicting, implementing, and evaluating interventions [11, 12, 23, 24]. For instance, Barnhart et al. (2020)  developed a ToC based on the experience of three different implementation phases of a maternal and child health program, the Better Birth program . Barriers, lessons learned and ways of applying these lessons were identified, and later integrated into a renewed ToC. In another example, Aggarwal (2021)  reviewed the literature for potential interventions and delivery mechanisms for reducing the self-harm of individuals in countries of low and middle income . Based on the findings of the review, the authors propose a ToC for a model of delivery of psychosocial interventions. Furthermore, in the field of Implementation Science, ToC has been a recurring choice to describe models, plan solutions to complex problems and evaluate programs in the health area, as described by Breuer et al. (2016) . Recent reviews suggest there is a growing number of published evaluations in the health area that use ToC, as identified by Breuer et al. (2016)  and Lam et al. (2021) . However, both reviews highlight that studies, overall, lack an appropriate level of detail regarding how the ToCs were developed and used.
The use of evidence in Theory of Change frameworks
Evidence-informed decision-making has been increasingly used as a method in clinical policy and practice. The use of evidence makes the decision-making processes more explicit, favoring transparency, reproducibility, credibility, and reliability . We argue that the development and use of ToC frameworks should, therefore, also include explicit and systematic considerations on how to incorporate evidence.
In this review, we will follow the Lomas et al. (2005)  conceptualization of evidence as “facts (actual or asserted) intended for use in support of a conclusion” (p.1). The authors  distinguish colloquial evidence (or tacit knowledge), provided by stakeholders’ opinions and viewpoints, from scientific evidence (or explicit knowledge), gathered by a prescribed set of procedures in order to produce knowledge recognized as explicit, systemic, and replicable, and further distinguishes two scientific views on evidence depending on whether they emphasize context-free general truths (more aligned with evidence-based medicine), or context-sensitive facts (more aligned with the applied social sciences). The authors also highlight that combining and interpreting these forms of evidence requires a deliberative process, often involving the participation of stakeholders to judge the relative weight of different pieces of evidence.
In theory, a ToC would be based on the analysis of available scientific evidence (explicit knowledge) and the consultation and deliberation with stakeholders (tacit or colloquial knowledge)  developed through rigorous and participatory processes. ToC approaches, however, usually emphasize stakeholder input (stakeholder consultation in the development of a plan to articulate the initial conditions, objectives, and means to achieve the desired results in a causal structure ) while making limited use of explicit or academic knowledge such as program evaluations and published research .
Two literature reviews have previously investigated the methods used to incorporate evidence into ToCs in the health sector. In a systematic review, Breuer et al. (2016)  investigated how ToCs are used to design and evaluate public health interventions. The authors identified the following methods used to incorporate evidence into ToC frameworks: “workshops and working groups, document reviews, interviews and discussions, surveys, program observation, literature reviews, and existing conceptual frameworks or theory, […] consultations or interviews [with stakeholders]” (p.4). Moreover, in a scoping review, Lam et al. (2021)  investigated how ToCs are used in food security contexts. The authors find that the methods applied for using evidence in ToCs were primarily participatory approaches, such as workshops or interviews, i. e., favoring tacit knowledge. They also report that most studies do not provide details on how the ToCs were developed, which is “consistent with the sentiments in the literature, which highlighted the limited guidance, training, and agreement on how to develop a ToC” (p.7) .
In this review, we expand the existing literature by focusing on how to systematically use research evidence to inform the development of ToCs in the health sector. Consistent with the previous reviews, our findings below show that almost none of the recent studies have systematically investigated the use of evidence to inform ToCs in the health sector. Furthermore, we did not encounter any study that provides a comprehensive and rigorous account on what methods can or should be used to incorporate evidence into ToCs. Most of the primary studies that address how a ToC was developed only give brief descriptions of the methods applied to incorporate evidence into this process. These methods include, for instance, structured stakeholder consultation, literature searches and the use of institutional data, which are applied as sources of information to draft or improve ToCs. However, each study seemed to choose specific versions of these methods (e.g. stakeholder consultations) without a clear rationale for why this choice was made over the alternatives (e.g. a literature review).
This project was commissioned by the World Health Organization, which identified the need for more explicit use of evidence in ToC development. This rapid literature review specifically aims to advance our knowledge on how to enhance the systematic and transparent use of the best available evidence in the ToC development for decision-making in the health sector. The rapid review methodology was chosen to ensure agility and efficiency in applying the available resources while keeping methodological rigor.
This rapid review adds to the existing literature in two ways. First, it provides an up-to-date and comprehensive review of the existing methods for systematically incorporating evidence into ToC frameworks in the health sector. Second, it creates a typology to classify the findings from the literature. In this typology, we propose a sequence of 7 stages relevant to the process of evidence incorporation into ToCs. We also indicate which research methods the included studies applied for each of the proposed stages. These 7 stages were developed through a qualitative analysis of the data extracted from the included studies (further details provided in the “Data Synthesis” section below). Each stage corresponds to a specific set of objectives. The stages are: 1) "Define the problem", 2) "Define expected outcomes", 3) "Define interventions", 4) "Define change mechanisms", 5) "Model ToC", 6) "Validate ToC" and 7) "Revise ToC".
This rapid review aims to identify and synthesize the available literature on how to systematically use research evidence to inform the development and revision of Theories of Change in the health sector.
To that end, we will address the following key questions:
- What are the available tools, methods, and recommendations to systematically incorporate research evidence into the development and adaptation of Theories of Change in the health sector?
- What are the main principles, processes, and practical steps recommended by the literature on how to systematically incorporate research evidence into the development and adaptation of theories of change in the health sector?
To answer these questions, the acronym PCCS was used, as follows (Table 1):
Materials and methods
This project applied a rapid review methodology. Rapid reviews can be defined as a type of literature review in which “the steps of the systematic review are streamlined or accelerated to produce evidence in a shortened timeframe." (p. XIII) . In a rapid review, evidence is synthesized and its validity is assessed using an abbreviated systematic review method, so to identify results in a shorter timeframe. Regardless of the abbreviated approach, this endeavor followed the principles of the scientific method and the key principles of evidence synthesis, including rigor, transparency, and reproducibility. This rapid review was conducted and reported following the process outlined by Tricco et al. (2017) . We used a rapid review approach to be able to provide high quality evidence for decision making, in a timely way. Given the time and budget limitations of this project, some adaptations were made to adjust to the needs of the rapid review design. These adaptations are described in the section “Shortcuts taken for the rapid response” below.
The study protocol was published in the Open Science Framework (OSF) platform (register: osf.io/t53sm) . After the publication of the protocol, we made an addition in the research design by expanding the inclusion criteria also to include scoping reviews.
The inclusion criteria applied in this review were:
- Topic: studies that investigated how research evidence was used in the development and adaptation of ToC in health policies; or studies that explicitly described how evidence was or should be used to inform the development or adaptation of the ToC in the health sector;
- Study type: systematic reviews, scoping reviews, guides, or methodological studies. The methodological studies could be case studies, qualitative or quantitative studies, as long as their primary focus was how to develop or update a ToC. We excluded empirical primary studies, overviews, integrative reviews, evidence syntheses, technology assessments, and economic assessments;
- Language: published in English, Spanish, or Portuguese;
- Date: published on any date;
- Place: any country or region.
Shortcuts taken for the rapid review
In this rapid review, only the screening process was done by two reviewers independently. Data extraction and methodological quality assessment of the studies were done by one researcher and reviewed by a second.
Data sources and searches
Published articles were searched on November 8, 2021, in the following databases: PubMed, BVSalud, Cochrane Library, Health Systems Evidence (HSE), Health Evidence, and Epistemonikos.
Gray literature was retrieved from OpenGrey and Google Scholar search engines limited to the first ten pages. A list of references obtained from a WHO researcher that had been previously investigating the topic was also screened. Due to time constraints, no additional search for gray literature was conducted.
Search strategies were developed based on the combination of keywords structured by the acronym PCCS, using the MeSH search terms on the PubMed database. The strategy was adapted to the other databases.
Additional studies were retrieved from a list provided by a topic expert and the WHO library.
The implemented searches can be found in S1 Appendix.
Study selection and data screening
Study screening was preceded by calibration of the inclusion and exclusion criteria among all reviewers using a sample of the search results. After calibration, two reviewers selected the studies through title and abstract screening independently. Disagreements were decided by a third reviewer.
After title and abstract screening, two researchers independently screened the included studies in full-text. Conflicts were resolved through consensus or by a third reviewer.
The screening was conducted in the reference management software, Rayann QCRI .
Only reports published in Portuguese, Spanish, and English were included, and potentially relevant information in other languages was not considered. In addition, primary studies that addressed the use of ToC to apply evidence in the health sector but that were not included in systematic reviews or scoping reviews were also not covered by this study. Finally, relevant literature from other fields than health might also have had relevant information to answer this review’s question, but they were not included.
Data extraction was conducted according to the research questions. A data extraction spreadsheet was made, observing the main elements to inform this rapid review, and covered the following categories: first author, published year, study objective, population and sample characteristics, method, main findings (divided into Definition of ToC, Participants in the ToC development, Steps for the development of a ToC, Steps for revising a Theory of Change, Use of research evidence in the development of a ToC, and Use of research evidence for revising a ToC), study limitations, conclusions, conflicts of interest and last year of search. In the subcategories “Use of research evidence in the development of a ToC” and “Use of research evidence for revising a ToC”, we captured all relevant information from the primary studies regarding their reported strategies for evidence incorporation into ToCs. Data for the other subcategories were collected to identify and contextualize each study and provide further information on how ToCs were used and how evidence was considered. The fields “Definition of ToC”, “Steps for the development of a ToC” and “Steps for revising a Theory of Change” were added to the data extraction template after a round of initial piloting of the data extraction process to facilitate the interpretation of the research findings.
Quality assessment was done by one researcher and reviewed by another. Systematic reviews were assessed using the AMSTAR 2 tool . Systematic reviews were classified according to their confidence degrees as high, moderate, low, or critically low. This classification followed the AMSTAR 2 critical domains that check for the presence of a search protocol, search strategy, inclusion and exclusion criteria, assessment of the risk of bias, and interpretation of results. To assess the quality of the scoping review, we used the Joanna Briggs Institute’s Checklist for Systematic Reviews . To evaluate the studies self-described as methodological, we used the Joanna Briggs Institute’s Checklist for Qualitative Studies . For the other included studies, that we also considered methodological, we applied quality assessment tools compatible with each of their specific designs, they were: Joanna Briggs Institute’s Checklist for Qualitative Studies  (4 studies), Joanna Briggs Institute’s Checklist for Text and Opinion  (2 studies), Scale for the Assessment of Narrative Review Articles (SANRA)  (1 study), and Critical Appraisal of a Case Study tool from the Center for Evidence-Based Management  (1 study). The quality assessment results can be found in S2 Appendix.
After data extraction, studies were compared, and the findings were summarized qualitatively into themes, in the form of key principles, steps, and procedures for the systematic incorporation of research evidence into the development and adaptation of a ToC. A qualitative narrative synthesis was conducted regarding “Definitions of ToC”, “Participants in the ToC development”, “Methods for the development of the ToC” and “Use of research evidence in the development and revision of a ToC”.
To further contextualize how and when evidence can be integrated into the ToC development process, we also identified, based on the available literature, the main stages relevant for the process of evidence incorporation into ToCs. The basis for this process was the extracted data regarding the steps for developing and revising a ToC and the strategies for evidence use (see S6 Appendix for the relevant extracted data). Through an inductive qualitative data analysis process, we created a typology of stages and coded the extracted information accordingly. When grouping the steps based on content similarity, we realized they could be classified with labels similar to the phases often included in the “policy cycle”, such as “Policy Formulation”, “Implementation” and “Evaluation” (p.33) . Thus we regrouped the data following this framework, adapting some of these phases as labels (i.e. "Define the problem", "Define interventions"), and creating new ones when necessary (i.e. "Define change mechanisms", "Model ToC", "Validate ToC", "Revise ToC") based on our content similarity analysis. Here, we present these labels as stages for the development of ToCs. The different stages reflect the objectives and functions of each step, as shown in section “Stages for the development of a ToC” below and in S3 Appendix. Thereafter, we conducted a frequency count to assess how many of the included studies reported each stage. For ease of understanding, these stages were organized in a logical sequence, according to our interpretation.
In addition to categorizing the stages for developing a ToC, we classified the extracted data in terms of “types of activities” conducted. This typology was also created through an inductive qualitative data analysis process, based on the original extracted material. Within each stage, we grouped the included methods according to the specific actions that were conducted. While the classification in stages reflects the goal or purpose of each step taken, the classification into the types of activities summarizes the concrete actions undertaken to achieve this goal (see section “Stages for the development of a ToC” below).
The search retrieved 880 records, of which 141 were removed as duplicates. 698 articles were excluded at the title and abstract screening as they did not meet the eligibility criteria. 41 documents were selected for full-text screening, of which 23 were excluded for not discussing ToC, the use of evidence, or not matching the inclusion criteria. In addition, we screened 23 articles recommended by an expert on the topic. After excluding duplicates, the remaining studies were screened in full-text. Based on the exclusion criteria, all of these studies were excluded. The excluded articles and the reasons for exclusion are listed in S4 Appendix, separated between articles found in the databases and those recommended by the expert (Fig 3). Finally, 18 remaining articles were included in this rapid review.
Source: authors’ elaboration, adapted from PRISMA 2020 .
Despite there being no limitation on the year of publication as an inclusion criterion, all included studies (n = 18) were published between 2013 and 2021 [1, 13, 14, 18–24, 36–43], and among those, the mode (n = 5) was the year 2021 [19, 22, 24, 39, 42]. For more details on the characteristics of each study and the data extracted, see S5 and S6 Appendices.
Characteristics of included studies
The studies presented different designs with seven systematic reviews [19, 21–23], one scoping review , and ten studies with other designs that were considered “methodological” since their primary focus was on how to develop or update a ToC. The included methodological studies had a variety of study methods, as reported by their respective authors: case study (n = 1) , mixed methods (qualitative and quantitative) (n = 1) , methodology studies (n = 2) [36, 37], report (n = 1) , formative research (n = 1) . Four publications [1, 20, 38, 43] did not name the study design. They were classified by the authors of this rapid review as methodological studies based on the information described in the method section, for the purpose of quality assessment.
Regarding the countries conducting the studies, five studies were carried out in countries that, according to The World Bank (2021) , are classified as high-income economies (HIE) [21, 23, 36, 37, 42], one in Lower-middle-income economies (LMIE) , and one in Upper-middle-income economies (UMIE) . Two studies included both HIE and LMIE [18, 19]. The remaining studies did not report this information or it did not apply to the study design [13, 20, 22, 38–41]. Except for De Buck et al. (2018) , which had authors from both HIE and LMIE, the authors of the other 17 included studies were affiliated to institutions from HIE [13, 14, 18, 19, 22, 24, 36–43].
Sixteen studies reported no conflict of interest [1, 13, 14, 18–20, 24, 36, 39]. One study included information about funding but not on competing interests . One study did not have information on either .
The included studies focused on the following themes and sectors: public health [13, 21, 23, 40], nutrition [24, 38], healthy eating and physical exercise , mental health [1, 22, 36], health promotion , violence prevention , obstetrics , immunization data , research-policy gap , realist RCTs , and reducing sedentary behavior .
Methodological quality of included studies
The methodological quality of seven systematic reviews was assessed using AMSTAR-2 . Five of these were considered critically low [19, 21–23, 39] and two considered low [14, 41]. None of the publications provided a list of excluded studies with justifications [14, 19, 21–23, 39, 41]. In addition, the five critically low did not account for biases [10, 12, 13, 17, 19, 21–23], or did not refer to methods being established prior to the review . Six studies were assessed with the JBI Critical Appraisal Checklist for Qualitative Research . These studies contained two important weaknesses: no statement locating the researcher culturally or theoretically, nor any mention of approval by the ethics committee [1, 20, 36, 37, 42, 43]. One study  was assessed with the JBI Critical Appraisal Checklist for systematic reviews and research syntheses . However, methods to mitigate potential errors in the findings in this study were not found. Two studies were assessed with the JBI Critical Appraisal Checklist for text and opinion papers , and the reports were considered satisfactory [13, 40]. One study  was assessed with SANRA  and had a final score of 10/12. The two limitations of the study were that the electronic search report was not detailed enough and the selection of studies was not well described. One case study was assessed with the CEBMA tool . In this study, there is no mention of whether the data analysis was repeated by another researcher and it is not clear whether the researcher’s perspective was taken into account . More details are found in S2 Appendix.
Definitions of ToC
Nine studies [1, 13, 14,18, 22, 23, 38, 40, 43] presented a definition of ToC, with some differences between them, as shown in Table 2.
Participants in the ToC development
Four studies reported that their respective authors participated in the development of the ToC [22, 36, 39, 40]. Other studies included multidisciplinary groups composed of professionals , service users , program evaluators and implementers, community members , methodological experts, and experts on the topic . Ten studies did not report any information about the participants in the development of the ToC [13, 14, 19, 21, 23, 24, 37, 38, 41, 43].
Methods for the development of the ToC
Some of the studies [14, 19, 20] followed existing methods for intervention design in their efforts to build a ToC. The methods used were De Silva (2014) , the Six steps in quality intervention development (6SQuID)  framework [36, 42], and the guidelines  offered by the World Bank  and Morra and Rist .
Three studies [21, 22, 41] synthesized ToC evidence from previously published ToCs. Tancred et al. (2018) , for instance, conducted a systematic review to synthesize relevant elements for a ToC for school-based interventions integrating health and academic education to prevent violence and substance use among students. This review searched for other ToCs that were synthesized through a meta-ethnographic analysis.
Stages for the development of a ToC
The included studies generally did not provide a detailed account of what steps were taken to develop their respective ToCs. However, from the available information, it is possible to see that the development of the ToC was rarely a linear or hierarchical process. In reality, the development process is continuous, involving reflection and adaptation as new evidence comes to light, requiring the causal pathways to be changed .
All studies included in this review presented at least one stage adopted for the construction of the ToC. In total, seven development stages were identified. The most frequent type was “Define interventions” [1, 19, 22, 36, 39, 41, 42] and “Model ToC” [13, 18–20, 22, 23, 38, 42, 43], mentioned in 55.5% (n = 10) of the included studies. All identified stages and their respective frequencies can be found in Table 3.
Table 5, below, presents these stages with further details. In addition to categorizing the stages for developing a ToC, we also classified the extracted data in terms of the “activities” conducted. While the classification in stages reflects the goal or purpose of each step taken, the classification into the activities summarizes the concrete actions undertaken to achieve this goal. These activities are depicted in the second column of Table 4. In the third column of this table, “Activities’ details”, we paraphrase the steps described in the included studies to present guidance on how to undertake each activity. The original material can be found in S3 Appendix.
We emphasize that among the “Activities”, the lines “Identify and synthesize evidence base” refer to the processes of searching, selecting, producing, and synthesizing evidence for the corresponding stages of the ToC development. In these cases, the use of the term evidence is encompassing both tacit or colloquial evidence and explicit or published evidence, as further detailed in Table 5.
Use of research evidence in the development and revision of a ToC
All included studies mentioned at least one source of evidence used to inform the development of the ToC. However, generally, the studies did not provide a structured and detailed discussion of how the incorporation of evidence was undertaken. Evidence sources included: literature reviews [1, 19, 20, 23, 36, 37, 42, 43], either systematic reviews or narrative reviews; review of reports and normative documents; surveys, interviews, meetings, workshops and focus groups with key actors, including community members and stakeholders [1, 13, 18, 20, 23, 24, 36, 38–40]; previous empirical studies; existing related psychological and sociological theories [1, 21, 37]; consultation with experts [18–20]; and previous theories of change [21, 22, 41]. Many studies [18, 20, 23, 36, 37, 39] combined more than one evidence source for the development of the ToC; such as a combination of literature review, consultations with stakeholders, and previous empirical studies.
The ToC diagram can be developed in a meeting with stakeholders, often in a workshop format with the help of a trained facilitator . However, it is common to find the implementation or evaluation team making a first draft based on the literature review and then validating the draft in a stakeholder meeting. Mayne and Johnson (2015)  note that while bringing a pre-drafted ToC may decrease the level of buy-in, it makes it easier to bring relevant research findings into it.
De Silva et al. (2014)  report that in their study, the development of the ToC was supported by a ToC expert. They also conducted brainstorming activities with key actors to gather information about expected outcomes. Afterward, a survey with stakeholders was carried out, a workshop was set up to foster debate, and, lastly, the ToC was further refined through another group session.
Few studies reported on how to use evidence to revise a ToC. The ToC was revised both in the initial development process, by experts [1, 19, 20] or peer review , and after the intervention was initiated, in response to evidence gathered by a discussion with practitioners , hypothesis testing (mechanisms and moderators evaluations)  and program evaluation .
Challenges and barriers reported by the included studies.
The limitations reported in the included publications largely focused on the ToC development process and use of the ToC, the interventions performed, and the methods used in the studies.
Regarding the ToC, the authors reported that the process of using a ToC framework is time-intensive and it can lead to the oversimplification of complex relationships . Moreover, ToCs are specific for each program and it is not simple to develop a ToC that is generalizable to other settings [41, 42]. The authors also reported that stakeholders’ difficulty understanding the scientific method was a limitation to the evaluation of interventions using a ToC approach . Finally, one scoping review highlighted that many studies do not report how the ToC used in the paper was developed .
Limitations related to the interventions performed in the studies were also mentioned, such as lack of time for a follow-up evaluation of the interventions , and limited availability and lack of evidence of culturally adapted interventions for LMICs . In addition, some studies did not address adverse structural conditions that affect participants , or the effectiveness of interventions in the real world . In another study, the authors reported problems in the intervention development process because it was not validated and relevant parameters were not included in the theory .
Regarding the methodological aspects of the studies, limitations reported by the systematic reviews included methodological flaws in the primary studies of relevant interventions, which may have affected statistical significance . Insufficient quality assessment of the included studies ; absence of dual data extraction, and inability to effectively assess the quality of included articles ; and inclusion of studies in English only  were also reported as key limitations. In the only scoping review included, authors reported difficulties during the gray literature search, which was attributed to limitations of the Google search platform (insufficient database to capture gray literature studies) . Regarding the other studies included, Yearwood et al. (2018)  highlighted that it was not possible to systematically evaluate the results achieved by the initiative, making it difficult to make precise conclusions about the effectiveness of the intervention.
Gaps reported by the included studies.
The gaps reported in the included studies were the need for better methodological and reporting quality to assess the reliability of results [21, 22, 39, 40], the need to conduct evaluations of the proposed ToC models , and the need to synthesize previous theoretical frameworks to avoid possible biases . In addition, studies suggest that ToC reports should be more detailed, especially regarding how the ToC was developed. The inclusion of a visual representation of the ToC (diagrams) would also better facilitate the reader’s understanding and ease reproducibility [23, 24].
Summary of main findings
This rapid review identified 18 studies that addressed the use of evidence in ToC or in the revise of a ToC in the health sector. This study adds to the existing literature in two ways. First, it provides an up-to-date and comprehensive review of what are the existing methods and recommendations for incorporating evidence into ToC frameworks in the health sector. Second, it creates a typology to classify the findings from the literature. In this typology, we propose a sequence of seven stages relevant to the process of evidence incorporation into ToCs.
An initial challenge for this review was the scarcity of data. In line with the conclusion from Breuer et al. (2016)  and Lam et al. (2021) , we find that authors rarely detail how the ToCs were developed and even more rarely include explicit considerations about evidence incorporation in the process. Future research projects should consider using the Breuer et al. (2016)  proposed checklist for reporting ToC in Public Health Interventions. It has five criteria, assessing the ToC approach, the ToC development process, presentation of the ToC, description of the intervention process, and use of the ToC for evaluating a policy. This checklist, however, has not yet been validated and might not be appropriate for all possible uses of a ToC. As such, it may require adaptations depending on the context in which it is used.
The studies included in this review, nevertheless, provide a rich portrayal of the field. The findings corroborate the definition of ToC provided by De Silva et al. (2014) , according to which a ToC graphically presents how and why an intervention is expected to work and, in general, is developed by a participatory process .
We observed significant diversity across topics and areas within the health sector in which ToC approaches can be applied. It was also noted that there is no single way to develop a ToC that suits all purposes, but that adaptations are made according to the context of its application.
Stages for ToC development
The stages for developing a ToC were observed to be different in the included studies. The process of developing a ToC has multiple possibilities regarding its form and content, being essentially dynamic and nonlinear. However, for didactic and visual purposes, we classified and organized these stages in a logical order (See Table 5, below).
It is important to highlight that this logical sequence is our interpretation, based on a qualitative analysis of the extracted data. As we mentioned before, the authors often do not provide a structured presentation of how the ToCs were developed. Moreover the stages are rarely followed in this strict logical sequence.
Prior to the stages, it is also important to discuss the purpose and goal of the ToC. As we mentioned above, ToCs can have many purposes, such as supporting the implementation of an intervention, supporting monitoring and evaluation efforts, sharing and aligning knowledge and expectations about outcomes among stakeholders, synthesizing knowledge and communicating to a specific audience. Given the time, work, and coordination requirements to gather stakeholders and research evidence to develop a ToC, it is paramount that its goals are well defined beforehand.
Methods for evidence incorporation into ToC development
From Table 4 above, it is possible to observe how different sources of evidence are used in the development of ToCs. There were three main sources of evidence: literature search, consultation with key actors, and use of institutional data. More specifically, there were a variety of ways to use evidence in ToC, such as searching for systematic reviews, conducting systematic and narrative literature reviews, using primary studies, using theoretical frameworks, using institutional and epidemiological data, conducting meetings, interviews, surveys, and workshops with stakeholders, or a combination of these components. Therefore, both colloquial and scientific evidence were commonly used sources of knowledge to support ToCs. Four studies reported the use of evidence for revising a ToC. The findings showed that revising a ToC can be done by evaluating programs, mechanisms, and moderators, and by consultation with experts and other key actors.
In Table 6, below, we synthesize the methods for evidence identification and synthesis presented in the Results section (see Table 4). These methods are organized by ToC development stage:
Table 6 describes what methods for evidence incorporation the included studies applied for the different stages of ToC development, with the stage “Define expected outcomes” supported, mostly, by the use of scientific evidence, both at a global and local level. As for the “Validate ToC” stage, we identified the use of evidence only at the local level, but, in a way, incorporating both scientific and colloquial evidence, through experts and users. We also observe that for the “Define the problem”, “Define intervention” and “Revise ToC” stages, authors explore evidence both at a global and local level, including scientific and colloquial knowledge. This demonstrates the ability of a ToC to be built from different sources of knowledge, considering colloquial evidence without giving up on scientific rigor.
Table 6, however, should not be understood as a recommendation of which methods should be applied for evidence incorporation into ToCs, nor as a conceptually informed exhaustive list of the possibilities in that regard. The included studies did not present justification as to why a specific method was chosen, or why other methods were left out. Moreover, no study provided a normative account on how to incorporate evidence into ToC development, nor provided specific guidance for that purpose.
Therefore, we identify a gap in the existing literature. The fact that only 18 studies explicitly discuss how to incorporate evidence into ToC development in the health sector, and none offers guidance or a framework for this shows that it is a neglected area. This stands in sharp contrast with the increasing recognition of evidence as an indispensable element for policymaking in the health sector.
This literature gap is in line with the findings from Lam et al. (2021) , which report a fragmentation in the field and a lack of guidance and conceptual consistency regarding how to develop a ToC. There is, therefore, an existing need for a new theoretical framework, offering clear advice on how to incorporate evidence into ToC development. This framework should draw on the existing practice, as discussed in this rapid review, while also integrating methodological and conceptual considerations, to guide the many ways through which evidence can be used to inform ToCs.
Strengths and limitations of the present study
This project applied a rapid review methodology, following the guidance provided by Tricco et al. (2017) . Regardless of the abbreviated approach, this endeavor respected the scientific method and the key principles of evidence synthesis, including rigor, transparency, and reproducibility. Furthermore, the shortcuts adopted in this review allowed us to focus on the most critical and relevant literature to our research questions.
A few adaptations from a complete systematic review were made. First, only studies involving the health sector were included in the eligibility criteria, which may have limited relevant information about ToCs. Second, the inclusion criteria for systematic reviews, scoping reviews, and methodological studies and guidelines focusing on the development of theories of change in the health sector may not have covered the entire literature on the use of evidence in the theory of change. Third, only studies in English, Spanish, and Portuguese were included, due to the language skills of the reviewers. Fourth, data extraction and the methodological quality assessment of the studies were conducted by one reviewer and checked by another. Fifth, some of the included studies did not explicitly state their research design, which required the reviewers to classify these studies as a specific type to choose and apply the methodological assessment tools that were the most adequate.
This rapid review identified studies that addressed the use of evidence in ToC, as well as strategies for developing a ToC in healthcare contexts. Several strategies for using evidence in the development of a ToC were reported, such as retrieving evidence from the scientific literature, using theoretical frameworks, and consultations with stakeholders. These strategies were applied in isolation or combined. Little evidence was found about the use of evidence for the revision and updating of a ToC. No study provided a normative account on how to incorporate evidence into ToC development, nor provided specific guidance for that purpose.
Furthermore, we identified a significant gap between the available literature on evidence integration for ToCs and the state-of-the-art debates around Evidence-Informed Decision-making (EIDM). While the EIDM debate has been advocating for increased use of evidence to support all stages of the policy cycle , it seems that ToC development has yet to be fully addressed. Similarly, the literature specializing in ToCs does not seem to incorporate many of the tools developed within the EIDM area, such as Rapid Reviews, Scoping Reviews, Evidence Gap Maps, among others. It also does not provide a structured account of how to bridge the techniques for ToC development with the need to systematically and transparently incorporate evidence into all stages of the policymaking process. A possible way of solving this issue could be the establishment of a protocol that ensures that different approaches share fundamental elements concerning the use of evidence in the development of a ToC. Therefore, it is crucial that future research addresses this gap.
Based on the findings, we conclude that there is a need for studies to explore in a clearer and more detailed way how evidence can be used in the development and updating of ToCs in the health sector. It is also important that studies that report a ToC should explicitly outline the process through which it was developed.
S1 Appendix. Search strategies.
S2 Appendix. Methodological quality of the included studies.
*Critical domains; CL: Critically low.
S3 Appendix. Stages of ToC development: Qualitative analysis.
S4 Appendix. Full-text articles excluded, with reasons.
S5 Appendix. Characteristics of the included studies.
*Study type: When the authors did not inform the study design, the reviewers attributed a classification based on the provided description of the methods. HIC: High-income economies; LMIC: Lower-middle-income economies; UMIC: Upper-middle-income economies. ** The systematic review had a ToC as one of its outputs.
S6 Appendix. Results of the included studies.
S1 Checklist. PRISMA 2020 checklist.
We thank the following individuals who assisted the implementation of this project: Laura dos Santos Boeira and Marcel Henrique de Carvalho.
- 1. De Buck E et al. Engagement of stakeholders in the development of a Theory of Change for handwashing and sanitation behaviour change. Int J Env. Health Res 28, 8–22 (2018) [Internet]. [cited 2021 Dec 7] https://www.tandfonline.com/doi/abs/10.1080/09603123.2017.1415306?journalCode=cije20 pmid:29260884
UND (Group). Theory of change: UNDAF companion guidance. (2017) [Internet]. [cited 2021 Dec 7]. https://unsdg.un.org/resources/theory-change-undaf-companion-guidance.
Anderson AA. The Community Builder’s Approach to Theory of Change. A practical guide to theory development. New York: The Aspen Institute Roundtable on Community Change. (2006) [Internet]. [cited 2021 Dec 7]. https://www.aspeninstitute.org/wp-content/uploads/files/content/docs/rcc/rcccommbuildersapproach.pdf.
UK Aid Connect. Guidance Note: Developing a Theory of Change (2018). [Internet]. [cited 2021 Dec 7]. https://assets.publishing.service.gov.uk/media/5964b5dd40f0b60a4000015b/UK-Aid-Connect-Theory-of-Change-Guidance.pdf.
Funnell SC & Rogers PJ. Purposeful program theory: effective use of theories of change and logic models. San Francisco (2011) [Internet]. [cited 2021 Dec 7]. http://au.wiley.com/WileyCDA/WileyTitle/productCd-0470478578.html.
Weiss CH. Evaluation Research: Methods of Assessing Program Effectiveness. Upper Saddle River, N.J., Prentice Hall, 1972. 160p., ISBN 100132921936.
Weiss CH. Nothing as Practical as Good Theory: Exploring Theory-Based Evaluation for Comprehensive Community Initiatives for Children and Families. In Connell J. et al. (eds.). New Approaches to Evaluating Community Initiatives. Washington: Aspen Institute, 1995. 1, 65–92.
Stein D & Valters C. Understanding ‘theory of change’in international development: A review of existing knowledge. London: London School of Economics (2012) [Internert]. [cited 2021 Dec 7]. http://eprints.lse.ac.uk/56359/.
- 9. Reinholz DL & Andrews TC. Change theory and theory of change: what’s the difference anyway? International Journal of STEM Education (2020);7(1):1–2. [Internet]. [cited 2021 Dec 7]. Available from https://danielreinholz.com/wp-content/uploads/2020/07/2020-ijSTEM-Change-Theories.pdf.
United Nations Environment Programme (UNEP). Chemicals and Waste Management Programme. Monitoring, Evaluation and Learning Strategy and Action Plan. UNEP(2020) [Internet]. [cited 2022 Dec 5]. https://wedocs.unep.org/bitstream/handle/20.500.11822/35798/MELSAP.pdf.
Van Es M, Guijt I, & Vogel I. Theory of change thinking in practice: a stepwise approach. Den Haag: Hivos (2015) [Internet]. [cited 22 Dec 5]. https://themimu.info/sites/themimu.info/files/gis-seminar-2019/resources/Theory_of_Change_Guidelines.pdf
Vogel I. ESPA guide to working with Theory of Change for research projects. Ecosystem Services for Alleviation of Poverty. (2012) [Internet]. [cited 2022 Dec 5]. http://www.espa.ac.uk/files/espa/ESPA-Theory-of-Change-Manual-FINAL.
De Silva M, Lee L & Ryan G. Using Theory of Change in the development, implementation and evaluation of complex health interventions. in London: The Centre for Global Mental Health at the London School of Hygiene & Tropical Medicine and The Mental Health Innovation Network (2014). [Internet]. [cited 2021 Dec 7]. https://www.mhinnovation.net/sites/default/files/downloads/resource/ToC_A%20Practical%20Guide.pdf.
- 14. Sapkota D, Baird K., Saito A. & Anderson D. Interventions for reducing and/or controlling domestic violence among pregnant women in low- and middle-income countries: a systematic review. Syst Rev 8, (2019). [Internet]. [cited 2021 Dec 7]. https://europepmc.org/article/pmc/6889323. pmid:30940204
- 15. De Silva M. et al. Theory of change: a theory-driven approach to enhance the Medical Research Council’s framework for complex interventions. Trials (2014). [Internet]. [cited 2021 Dec 7]. https://trialsjournal.biomedcentral.com/articles/10.1186/1745-6215-15-267
Taplin DH & Clark H. Theory of change basics: A primer on theory of change. (2012).). [Internet]. [cited 2021 Dec 7]. https://www.theoryofchange.org/wp-content/uploads/toco_library/pdf/ToCBasics.pdf.
- 17. Astbury B & Leeuw FL. Unpacking black boxes: mechanisms and theory building in evaluation. Am. J. Eval. (2010). [Internet]. [cited 2021 Dec 7]. Available from https://journals.sagepub.com/doi/abs/10.1177/1098214010371972.
- 18. Barnhart DA et al. Optimizing the development and evaluation of complex interventions: lessons learned from the BetterBirth Program and associated trial. Implement Sci Commun 1, (2020) [Internet]. [cited 2021 Dec 7] https://implementationsciencecomms.biomedcentral.com/articles/10.1186/s43058-020-00014-8. pmid:32885188
- 19. Aggarwal S, Patton G, Berk M. & Patel V. Psychosocial interventions for self-harm in low-income and middle-income countries: systematic review and theory of change. Soc Psychiatry Psychiatr Epidemiol 56, 1729–50 (2021) [Internet]. [cited 2021 Dec 7]. Available from https://link.springer.com/article/10.1007/s00127-020-02005-5. pmid:33394070
- 20. Aromatario O. et al. Using theory of change to develop an intervention theory for designing and evaluating behavior change SDApps for healthy eating and physical exercise: the OCAPREV theory. BMC Public Health 19, (2019) [Internet]. [cited 2021 Dec 7]. Available from https://bmcpublichealth.biomedcentral.com/articles/10.1186/s12889-019-7828-4. pmid:31675951
- 21. Bonell CP et al. Theories of how the school environment impacts on student health: systematic review and synthesis. Health Place 24, 242–9 (2013) [Internet]. [cited 2021 Dec 7]. Available from https://europepmc.org/article/MED/24177419. pmid:24177419
- 22. Meiksin R et al. Theories of change for e-health interventions targeting HIV/STIs and sexual risk, substance use and mental ill health amongst men who have sex with men: systematic review and synthesis. Syst Rev 10, (2021) [Internet]. [cited 2021 Dec 7]. Available from https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-020-01523-2. pmid:33423693
- 23. Breuer E, Lee L, De Silva M & Lund C. Using theory of change to design and evaluate public health interventions: a systematic review. Implement Sci 11 (2016) [Internet]. [cited 2021 Dec 7] https://implementationscience.biomedcentral.com/articles/10.1186/s13012-016-0422-6. pmid:27153985
- 24. Lam S et al. How and why are Theory of Change and Realist Evaluation used in food security contexts? A scoping review. Eval. Program Plann [Internet]. [cited 2021 Dec 7] https://www.sciencedirect.com/science/article/abs/pii/S0149718921001038.
- 25. Dobrow MJ, Goel V & Upshur REG. Evidence-based health policy: context and utilisation. Soc. Sci. Med. 58, 207–217 (2004) [Internet]. [cited 2021 Dec 7]. Available from https://www.sciencedirect.com/science/article/abs/pii/S0277953603001667?via%3Dihub pmid:14572932
Lomas J, Culyer T, McCutcheon C, McAuley L & Law S. Conceptualizing and combining evidence for health system guidance: Final report Ottawa. (2005) [Internet]. [cited 2021 Dec 7]. https://savoir-sante.ca/en/content_page/download/116/222/21?method=view.
Tricco AC, Langlois E, Straus SE & Organization WH. Rapid reviews to strengthen health policy and systems: a practical guide. (World Health Organization, 2017) [Internet]. [cited 2021 Dec 7] https://apps.who.int/iris/handle/10665/258698.
De Araujo B et al. Integration of evidence into Theory of Change frameworks in the healthcare sector: a rapid review protocol. (2021).
- 29. Ouzzani M, Hammady H, Fedorowicz Z & Elmagarmid A. Rayyan—a web and mobile app for systematic reviews. Syst. Rev. 5, 1–10 (2016) [Internet]. [cited 2021 Dec 7] https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-016-0384-4.
Shea BJ et al. AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. (2017) [Internet]. [cited 2021 Dec 7].
Joanna Briggs Institute (JBI). Critical appraisal tools. The University of Adelaide (2018) [Internet]. [cited 2021 Dec 7]. https://jbi.global/critical-appraisal-tools.
- 32. Baethge C, Goldbeck-Wood S & Mertens S. SANRA—a scale for the quality assessment of narrative review articles. Res Integr Peer Rev Internet (2019) pmid:30962953
Critical Appraisal Checklist for a Case Study. Oxford (2014) [Internet]. [cited 2021 Dec 7]. https://casp-uk.net/casp-tools-checklists/.
Cairney P. Understanding public policy: theories and Issues. 2011 edition. Houndmills. Palgrave, Basingstoke, Hampshire; New York. [Internet]. [cited 2023 Jan 30]. https://bit.ly/3RkvpfQ.
- 35. Page MJ, McKenzie JE & Bossuyt PM. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372, (2021) [Internet]. [cited 2021 Dec 7] Avaliable from: https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1003583.
- 36. Hartley JE, McAteer J, Doi L & Jepson R. CARE: The development of an intervention for kinship carers with teenage children. Qual Soc Work 18, 926–43 (2019) [Internet]. [cited 2021 Dec 7] https://journals.sagepub.com/doi/full/10.1177/1473325018783823 pmid:32903309
- 37. Jamal F et al. The three stages of building and testing mid-level theories in a realist RCT: a theoretical and methodological case-example. Trials 16, (2015) [Internet]. [cited 2021 Dec 7] https://trialsjournal.biomedcentral.com/articles/10.1186/s13063-015-0980-y.
- 38. Mayne J & Johnson N. Using theories of change in the CGIAR Research Program on Agriculture for Nutrition and Health (draft). Can. J. Program Eval. 30, (2015) [Internet]. [cited 2021 Dec 7] https://journals.sagepub.com/doi/pdf/10.1177/1356389015605198.
- 39. Osterman AL, Shearer JC & Salisbury NA. A realist systematic review of evidence from low- and middle-income countries of interventions to improve immunization data use. BMC Health Serv. Res. 21, (2021) [Internet]. [cited 2021 Dec 7] https://bmchealthservres.biomedcentral.com/track/pdf/10.1186/s12913-021-06633-8.pdf. pmid:34238291
Rippon S & South J. Promoting asset based approaches for health and wellbeing: exploring a theory of change and challenges in evaluation. (2017) [Internet]. [cited 2021 Dec 7] https://eprints.leedsbeckett.ac.uk/id/eprint/4497/.
- 41. Tancred T. et al. A systematic review and synthesis of theories of change of school-based interventions integrating health and academic education as a novel means of preventing violence and substance use among students. Syst. Rev. 7, (2018) [Internet]. [cited 2021 Dec 7] https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-018-0862-y. pmid:30424812
- 42. Tirman L. et al. Stand Up for Health: Programme theory for an intervention to reduce sedentary behaviour in contact centres. Eval Program Plann 89, (2021) [Internet]. [cited 2021 Dec 7] https://www.sciencedirect.com/science/article/abs/pii/S0149718921000975?via%3Dihub pmid:34634757
- 43. Yearwood AC. Applying a logical theory of change for strengthening research uptake in policy: a case study of the Evidence Informed Decision Making Network of the Caribbean. Rev Panam Salud Publica 42:e91, (2018) [Internet]. [cited 2021 Dec 7]. Avaliable from https://www.scielosp.org/pdf/rpsp/2018.v42/e91. pmid:31093119
World Bank Country and Lending Groups–World Bank Data Help Desk (2021) [Internet]. [cited 2021 Dec 7]. https://datahelpdesk.worldbank.org/knowledgebase/topics/19280-country-classification.
- 45. Wight D, Wimbush E, Jepson R & Doi L. Six steps in quality intervention development (6SQuID). J Epidemiol Community Health 70, 520–525 (2016) [Internet]. [cited 2021 Dec 7]. Avaliable from https://jech.bmj.com/content/70/5/520.long. pmid:26573236
Roberts D & Khattri N. Designing a Results Framework for Achieving Results. (2012) [Internet]. [cited 2021 Dec 7]. https://openknowledge.worldbank.org/handle/10986/32158.
Morra-Imas LG, Morra LG & Rist RC. The road to results: Designing and conducting effective development evaluations. World Bank Publications (2009) [Internet]. [cited 2021 Dec 7]. https://openknowledge.worldbank.org/handle/10986/2699.
World Health Organization. Evidence, policy, impact: WHO guide for evidence-informed decision-making (2021) [Internet]. [cited 2021 Dec 7]. https://www.who.int/publications/i/item/9789240039872.