https://orcid.org/0000-0001-5615-6918
Figures
Abstract
Background
Group model building is a process of engaging stakeholders in a participatory modeling process to elicit their perceptions of a problem and explore concepts regarding the origin, contributing factors, and potential solutions or interventions to a complex issue. Recently, it has emerged as a novel method for tackling complex, long-standing public health issues that traditional intervention models and frameworks cannot fully address. However, the extent to which group model building has resulted in the adoption of evidence-based practices, interventions, and policies for public health remains largely unstudied. The goal of this systematic review was to examine the public health and healthcare applications of GMB in the literature and outline how it has been used to foster implementation and dissemination of evidence-based interventions.
Methods
We searched PubMed, Web of Science, and other databases through August 2022 for studies related to public health or health care where GMB was cited as a main methodology. We did not eliminate studies based on language, location, or date of publication. Three reviewers independently extracted data on GMB session characteristics, model attributes, and dissemination formats and content.
Results
Seventy-two studies were included in the final review. Majority of GMB activities were in the fields of nutrition (n = 19, 26.4%), health care administration (n = 15, 20.8%), and environmental health (n = 12, 16.7%), and were conducted in the United States (n = 29, 40.3%) and Australia (n = 7, 9.7%). Twenty-three (31.9%) studies reported that GMB influenced implementation through policy change, intervention development, and community action plans; less than a third reported dissemination of the model outside journal publication. GMB was reported to have increased insight, facilitated consensus, and fostered communication among stakeholders.
Citation: Estrada-Magbanua WM, Huang TT-K, Lounsbury DW, Zito P, Iftikhar P, El-Bassel N, et al. (2023) Application of group model building in implementation research: A systematic review of the public health and healthcare literature. PLoS ONE 18(8): e0284765. https://doi.org/10.1371/journal.pone.0284765
Editor: Jai K. Das, Aga Khan University Medical College Pakistan, PAKISTAN
Received: June 15, 2021; Accepted: April 9, 2023; Published: August 17, 2023
Copyright: © 2023 Estrada-Magbanua et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All data files are available from the Github repository (https://github.com/weannemyrrh/Group-Model-Building-Data).
Funding: This study was funded through a grant awarded to NE by the NIH National Institute on Drug Abuse, Grant Number 450 UM1DA049415 (https://www.drugabuse.gov/funding/nida-funding-opportunities) and a grant awarded to TH by the Centers for Disease Control and Prevention through the NYU-CUNY Prevention Research Center, Grant Number 451 U48DP00639 (https://www.cdc.gov/grants/index.html). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Group model building (GMB) is a process of engaging stakeholders in a participatory modeling process to elicit their perceptions of a problem and explore concepts regarding the origin, contributing factors, and potential solutions or interventions to a complex issue [1, 2]. For decades, GMB had been widely used in fields such as business, public policy, criminal justice, and environmental resource planning, among others [3]. More recently, however, it has emerged as a novel method for tackling complex, long-standing public health issues that traditional intervention models and frameworks cannot fully address.
In the past, conventional public health approaches have tended to lean towards more downstream, compartmentalized interventions that do not take into account the whole-system perspective. However, these approaches have been insufficient, as many public health issues meet the key criteria defining a dynamic complex system as defined by Sterman [4]: (a) strong interactions between the various actors of the system, (b) constant fluctuations and changes in trends and behaviors over time, (c) an internal complex causal structure subject to feedbacks from the different actors within the system, and (d) long delays between actions and effects. When not considered, these characteristics can lead to counterintuitive, unintended consequences that may be difficult to predict. For example, past attempts to address syndemic issues such as substance abuse through well-intentioned but single-perspective policies (e.g. increased policing or increased dispensing of medications for opioid use disorders) have instead, in the long term, led to a reinforcing cycle of stigma and compassion burnout [5, 6]. Therefore, it is evident that addressing these dynamically complex public health problems more effectively requires a shift from the traditional linear ways of thinking to a more holistic manner of examining the context, relationships, and systems within which they exist. GMB offers an organized, collaborative approach towards achieving that goal.
GMB adopts system dynamics (SD) methodology, which is a set of principles for framing and understanding the nonlinear behavior of complex systems as they change over time using feedback loops and stocks and flows [7]. SD involves a sequence of five stages: articulation of the problem that needs to be solved, formulation of a hypothesis that conceptualizes the dynamics of the system, formulation of a simulation model representing the system, model testing, and policy design and evaluation. In GMB, a purposefully selected, ideally diverse range of stakeholders, such as policy makers, community leaders, and persons with lived experience who can inform the project’s focus are engaged in these stages through a sequence of scripted group exercises carried out under the guidance of a skilled modeling team [8]. Scripts are used to direct facilitated structured small group exercises that ultimately inform the design, development and application of a systems dynamics model. Depending on the purpose of the GMB, facilitators select and carry out two or more scripts typically built around the following order of tasks (a) presentation (i.e. orienting or explaining SD concepts to participants) (b) divergent thinking (i.e. brainstorming), (c) convergent thinking (i.e. building consensus, shared understanding, or shared decision-making), and (d) evaluation (i.e. group assessment/group ranking of ideas). Table 1 presents examples of classic GMB scripts for each task. The end products of these activities are presented as causal loop diagrams (CLD) and/or stock and flow diagrams which can be simulated to explore the problem of interest (Fig 1) [9].
(A) Graphs over time: graphs drawn by participants depicting the behavior of variables over time. (B) Variable elicitation: variables affecting opioid use elicited from participants. (C) Creating a CLD: a CLD indicating connections and feedback behaviors between variables developed by participants (D) Dots exercise to address priorities: Output of a dots exercise demonstrating priority actions ranked by participants.
As the GMB process builds on an iterative cycle of collective deliberation, stakeholder consultation and verification, and collaborative modeling, the resultant models integrate scientific knowledge and practical experience [11], leading to the identification of systems strategies for intervention that addresses the target problem [12, 13]. Moreover, the added value of stakeholder engagement fosters group ownership of the model and engenders a shared insight of the system behavior surrounding the problem. Overall, the success of GMB lies not just in the construction of the model itself but in the process it takes to get there [14].
Applying GMB as a method to optimize processes for implementing interventions is fairly recent in public health and healthcare. Nonetheless, studies that document its application in these fields have begun to emerge in the past decade, providing insight into the extent to which it may be applicable across health issues. What remains largely unstudied is the extent to which GMB has resulted in the adoption and integration of evidence-based practices, interventions, and policies for public health. Further, despite the increase of GMB in social and behavioral research, no systematic review of its application and effectiveness in public health and healthcare has yet been conducted.
This paper is designed to examine the literature on GMB and how it has been used and disseminated to foster implementation of evidence-based interventions (EBIs) in public health and healthcare. The following research questions frame our systematic review: (a) where is the application of GMB most prevalent in public health and healthcare; (b) how have GMB activities and strategies been disseminated and applied to improve EBI implementation and systems thinking capacities; and (c) what individual, group/organizational, or community level outcomes have resulted from GMB activities.
Methods
Search strategy
This is a systematic review of GMB applications in public health and implementation science. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Guidelines were used to delineate each step of the review process (Fig 2).
We consulted seven electronic databases (CINAHL, Embase, Web of Science, PubMed, ProQuest, Scopus and EBSCO) to ensure an exhaustive coverage of the literature from any year up to August 2022 using the following keywords: ("system dynamics modeling" OR "group model building" OR "participatory system dynamics" OR "participatory model building" OR "community based system dynamics" OR "mediated modeling" OR "collaborative modeling" OR "heterogenous problem solving" OR "participatory stakeholder engagement" OR "collaborative model building”) AND (“implementation” OR “dissemination”). After studies were selected through our inclusion criteria for the final review, the reference list of selected studies and systematic reviews were also checked as a secondary method of identifying relevant publications.
Inclusion and exclusion criteria
We included studies where research questions were related to public health or health care and GMB was cited as one of the main methodologies. We did not eliminate studies from our search based on language, geographic location, or date of publication. Systematic reviews, theoretical papers, and articles missing explicit mentions of GMB or its related terms were excluded from this study.
We used Covidence, an online systematic review software, to conduct study screening. Three authors (WE, PZ, and PI) independently screened all titles, abstracts, and full-text articles and noted reasons for excluding studies during full-text review. Differences in screening decisions were resolved by the corresponding author (NS) who served as a third independent reviewer. The decisions were also supported by a group discussion on the article’s eligibility until consensus was reached.
After screening, three authors (WE, NS, DL) conducted a critical appraisal on the final articles to ensure the quality of studies included in this systematic review. Studies were considered good quality if they met a minimum set of criteria defined by the study authors as consistent with the standards of GMB and SD, namely, (a) the study explicitly carried out activities or scripts that support collaborative model building, and (b) the study correctly described SD concepts (e.g. polarity of the causal links are indicated, the type of feedback loops are discussed, stocks and flows are well specified, and/or resulting simulation results are presented) in the results or discussion sections.
Data extraction
Three authors (WE, PZ, and PI) independently piloted a structured abstraction form on ten studies; the final abstraction form was approved by three co-authors (TH, DL, and NS). One of two authors (PZ and PI) independently extracted data such as participant type, GMB team composition, GMB session characteristics, model attributes, model dissemination, implementation effects, and outcomes from the included studies onto a shared spreadsheet. A third author (WE) reviewed each of the abstractions against the original articles to ensure accuracy and consistency.
The models resulting from the GMB sessions were coded by the reviewers into three developmental phases or stages of validation. Studies were considered to be in the demonstration phase, the first phase, if they involved developing a high-generality, low-resolution scoping and consensus-building model generated by the stakeholder groups involved in GMB. They were considered to be in the research modeling phase if they were characterized by the use of historical data to calibrate and test the model developed during the demonstration phase. Finally, they were in the management phase if they involved running different scenarios across the model to identify the most effective intervention and policy options [15].
Each of the GMB activities mentioned in the studies were coded as a GMB script based on the definitions provided on the free Scriptapedia website [10]. For example, GMB activities mentioned in the studies that asked participants to write down a list of key variables affecting a phenomenon or public health issue were coded as “Variable Elicitation,” as they corresponded with the definition for that script on Scriptapedia.
The studies were also reviewed to identify whether or not the outcomes proposed by Rouwette et al. [16] occurred as a result of the GMB sessions. These outcomes include reaction, insight, commitment, behavior, communication, consensus, shared language, system changes, results, further use method, and efficiency and are classified under four levels (individual, group, organization, and project). Three new variables, namely systems thinking skills, confidence in the model, and improved understanding of role and responsibility were also added as outcomes to reflect common GMB goals and findings reported in the literature [17–19]. These variables were coded positively if a study reported outcomes that were consistent with the definitions provided in Table 2. Outcomes that were not found or reported in the articles were coded as “not mentioned”.
We calculated percentages for categorical variables and means and medians for continuous variables. We did not conduct a meta-analysis of the study findings due to the heterogeneity in how the outcomes were reported and presented across all the articles.
Results
The search yielded 2,032 unique articles for title and abstract screening, of which 420 articles were deemed potentially relevant and selected for full text review. A total of 324 studies was then excluded as further in-depth review revealed that they were not related to public health or health care, did not involve GMB methods, or were not published in a peer-reviewed journal. The critical appraisal of the remaining studies led to the additional exclusion of 24 studies as they did not meet the minimum criteria for quality. In these studies, modeling activities were not explicitly described or conducted, or the feedback loops, polarities, or stock and flow diagrams developed were not consistent with the standards of SD. In the end, 72 studies published from 2002 to 2022 were included in the final review. Fig 2 shows the selection of studies for review based on PRISMA while Table 3 outlines the list of all included studies and their outcomes.
Study information
More than half of the documented studies on public health applications of GMB were published in the last five years from 2017–2022 (n = 42, 58.3%). Most of the studies were conducted in high-income countries, particularly the United States (n = 29, 40.3%) and Australia (n = 7, 9.7%), although studies were found across six continents and across the different country-level income classifications as defined by the World Bank [94]. GMB methods were most heavily utilized in the domains of nutrition (n = 19, 26.4%), health care/hospital administration (n = 15, 20.8%), and environmental health (n = 12, 16.7%). GMB was conducted for the following purposes: to build a valid model (n = 59, 81.9%), to design/improve implementation (n = 34, 47.2%), to foster decision-making (n = 29, 40.3%), and to build rapport among stakeholders (n = 16, 22.2%).
Session characteristics
The characteristics of the GMB sessions varied widely across studies. The number of participants in the sessions ranged from as few as five people to as many as 50 people in one session, with 15 as the median number of participants. Most of the studies required two to three sessions; however, some only had one session while one study required 22. Each session lasted about 1.5 hours to a full day, with a mean duration of four hours.
GMB relies on participants who represent relevant, diverse stakeholder groups and experts [9]. Among all the participant types in this review, community coalition members/advocacy groups (n = 39, 54.2%) were the most often engaged in GMB projects, followed by government officials (n = 38, 52.8%) and domain experts (n = 34, 47.2%). The correctional or justice system (n = 4, 5.6%) and funding agencies (n = 2, 2.8%) were the least commonly represented.
GMB sessions were typically directed by a school- or university-based team with training in SD modeling. Actual facilitation of the sessions was conducted by a GMB team of, on average, seven persons who took on roles such as conveners, community facilitators, modelers, recorders, note-takers, and observers [8]. The most common GMB scripts mentioned in the literature were causal loop diagramming (n = 46, 63.9%; convergent thinking), variable elicitation (n = 39, 54.2%; divergent thinking), concept models (n = 29, 40.3%; presentation), graphs over time (n = 21, 29.2%; divergent thinking), and action ideas (n = 21, 29.2%; divergent thinking, evaluation).
Process evaluations determine whether workshops are implemented as intended and resulted in meaningful outputs. Only 11 of the 72 studies (15.3%) reported conducting a process evaluation to assess the implementation of the workshops and their fidelity to stated objectives of the GMB sessions.
Model attributes
The majority of the modelers and system dynamicists reported utilizing Vensim (Ventana Systems, Harvard, MA) (n = 36, 50.0%) and Stella Architect (ISEE Systems) (n = 7, 9.7%) to create and refine the models that emerged from the GMB workshops. The models mostly included qualitative (CLDs and stock-and-flow figures) in nature (n = 45, 62.5%), but 26 studies (36.1%) involved quantitative or simulation modeling. Most of the models and simulations modeled the overall dynamics of a community (n = 54, 75.0%), 19 (26.4%) modeled the dynamics of a group or organization, while three (4.2%) centered only on individual factors and behaviors. Majority of the models created by stakeholder groups reached only the demonstration phase of modeling (n = 44, 61.1%), while 18 (25.0%) studies reached the management modeling phase.
GMB outcomes
Twenty-three (31.9%) studies reported that the findings and models from the GMB workshops influenced implementation through policy change, intervention development, and/or community action plans. These models were used to develop local action plans, revise intervention approaches in grant applications, guide strategic planning and agendas, publish guidelines and communication messages, develop new partnerships, and activate policy leverage points. The remaining studies (n = 49, 68.1%) did not report whether GMB influenced or contributed to implementation.
Insight, consensus, and communication were the most commonly reported outcomes of the papers reviewed (Fig 3). Insight is defined by Rouwette et al. as individual-level learning [16]. We documented that all studies (n = 72, 100%) in our review generated useful insights among participants. In studies that apply GMB, greater insights among participants are produced if the problem to be modeled aligns with their interests and priorities, and if modeling efforts are sufficient to support the aims of the study [16].
Consensus or mental model alignment is a group-level outcome defined as a shared view of the problem or actions among GMB participants. It is considered a prerequisite for shared action [16]. Forty-three (59.7%) of the articles in this review indicated that consensus was achieved after the workshops. A common strategy mentioned in the literature to achieve consensus was the use of CLDs as boundary objects [36]. Boundary objects are representations, such as a diagram, sketch, or prototype, that help individuals collaborate effectively across a boundary such as a difference in knowledge, training, or objective [95]. Other strategies to build consensus that were mentioned in the literature were voting [60, 64] and continued discussion until all concerns were addressed [86].
A change in the quality of communication among GMB participants is necessary for consensus to emerge [16]. Thirty-nine (54.2%) of the articles in this review explicitly mentioned that an improvement in communication among stakeholders occurred during the GMB workshops. Participatory methods such as GMB generally imply the presence of communication; it is possible that this outcome actually occurred more than was reported and was just not explicitly documented as it was thought to be obvious.
Model dissemination
Model dissemination was high, with 70 (97.2%) of the studies publishing the CLDs or the full models developed through GMB. However, while the workshop activities and resultant were disseminated actively to the academic community through peer-reviewed publications in open-source journals (n = 43, 59.7%), information on whether the models were shared back to stakeholders or the non-academic community was limited. Studies that reported doing so (n = 20, 27.8%) utilized formats such as oral presentations to stakeholders and planning committees or written reports. Four recent studies published from 2016–2022 also reported utilizing cloud-based interfaces such as Forio, Dropbox or Runthemodel.com to disseminate the model, with one study uploading the model onto a web-based, user-friendly interface that allowed stakeholders to easily experiment with the model parameters online and generate corresponding charts and graphs.
Discussion
To our knowledge, this is the first comprehensive review of public health or health care applications of GMB using well established systems science frameworks to assess the studies. Our review found that the application of GMB in public health and health care settings has increased steadily in recent years, with more than half of the papers published from 2017 to 2022. The application of GMB was most prevalent in the domains of nutrition, health care and hospital administration, and environmental health. In particular, our findings showed relatively robust work in obesity research, health care service planning, and air and water quality research. Vensim® was most commonly referenced as the software of choice, which may be attributed to its policy of making available a free version to the public and its inclusion of a feature that facilitates quick and easy causal loop diagramming, which is a core activity in most GMB projects.
Of the 72 papers in this review, 23 (31.9%) reported that the resultant models created via GMB had an eventual influence on the design and implementation of policies and programs, resulting in revisions of local action plans and strategic planning, development of new partnerships, and activation of policy leverage points [75]. The studies we reviewed revealed how SD models developed through GMB asserted their impact on policy through various mechanisms, such as by challenging pre-conceived notions of policy-makers and protecting against overconfidence by exposing potential weaknesses and unintended consequences in the system [96]. For example, a GMB session in Fiji involving officials from two government ministries led to a shared endorsement of a data sharing policy when the discussion uncovered that lack of access to evidence was a cause of poor food-related policymaking (Table 4). Another study reported how the identification of clear causal relationships between their non-profit organization’s performance indicators led to a direct revision of their budget [41]. As GMB offers a process for making these causal influences explicit while allowing for a nuanced discussion of their impacts, useful insights and opportunities for organizational or community action are able to emerge.
In addition to insight which was identified as an outcome in all the studies, consensus and communication were the most reported outcomes of GMB. A unique aspect of GMB is its flexibility in accommodating multiple worldviews and experiences from a wide range of stakeholders. The studies in this review described how GMB created an environment for diverse participants to express individual experiences and then build a collective model and action plan to address the shared problem. While disagreements regarding the relationships between variables were common in the initial phases of the process as participants made their mental models explicit, the participatory nature of the method allowed for productive conversations that eventually negotiated and reconciled viewpoints [36, 48, 60]. For example, a graphs-over-time exercise in one GMB session revealed differing answers among parents and youth regarding factors that affect familial relationships [37]. However, the subsequent CLD exercise allowed the two groups to bring those different variables together and explore how they interact in the system, resulting in the development of an intervention that addressed both groups’ concerns (Table 4).
Our study also underlines how GMB can be applied as a strategy for community participation and engagement [97]. Not only does the collaborative process foster a greater sense of ownership over the findings, GMB scripts such as Dots and Graphs Over Time [45, 49] also offer a unique method of eliciting information from stakeholders that traditional approaches such as interviews or surveys may not be able to draw out. Further, the outputs created such as graphs, CLDs, and simulation models can be used as springboards for future engagement with the community, with one study mentioning a plan to follow-up their GMB sessions with a community co-design workshop to develop an intervention for the identified issues [98].
Our review also showed that systems change (n = 14, 19.4%), behavior change (n = 11, 15.3%), and positive results of system change (n = 9, 12.5%), were the least observed outcomes of GMB. As these three outcomes usually take time and may require considerable resources to realize, they may have just not yet been evident in the rest of the studies at the time of publication. This has been documented as a continuing challenge in the community-based SD field, as the delay between participation in GMB and the development of institutional initiatives has been found to reach up to several years or more [3]. Indeed, the papers in this review that reported the positive occurrence of these outcomes were those that had considerable project duration and follow-up periods of up to five years, supporting suggestions in the field to consider the use of longitudinal designs when evaluating participatory modeling activities.
Given our findings, our review has identified several recommendations for future applications of GMB in public health and health care settings. First, transdisciplinary stakeholder involvement is a key contributor to success [36, 56, 76, 99]. A closer evaluation of the 23 papers that reported an influence on implementation revealed that bringing in multiple perspectives was crucial to creating rich discussions and achieving acceptance of the model, with one study noting that diverse stakeholder participation led to lower rates of model rejection [99]. Studies that were unable to schedule single sessions with their desired stakeholders maximized participation by implementing strategies such as creating multiple expert panel sessions or conducting lunch-and-learn sessions during convenient times [79, 100]. It should be noted that in these papers, program implementers (n = 14, 60.9%), coalition members and advocacy groups (n = 11, 47.8%), and government officials (n = 10, 43.5%) were the most widely represented stakeholder groups in the sessions.
Second, while 43 of the 72 papers (59.7%) were published in open-source journals, only a third reported dissemination of the findings back to the stakeholders and the community through other means. As with any method involving the community and the practice of translational, participatory research, the dissemination of findings beyond scientific publication should be a fundamental activity, as journal audiences do not usually include non-academic stakeholders and the general public [101]. Dissemination of GMB findings to participants will provide a tangible deliverable, facilitate decision-making, and sustain implementation impact [36]. In addition to more traditional means of dissemination such as presentations to stakeholders and distribution of reports [36, 56, 85], the more recent published literature showed an increasing trend towards cloud-based model publication [52, 102]. Using cloud-based software expands a model’s reach, as it will allow the public, not just the modeling team, to interact with the product and run simulations from anywhere. Other software such as Stella Architect’s ISEE Exchange also allows modelers to present the structure and behavior of their models in a piece-by-piece, sequential manner, making the narrative more understandable and interesting even to those with limited modeling background.
Third, there is both a gap and inconsistency in how outcomes of GMB are reported in the public health literature. Reporting is focused on the developed CLDs, models, and simulations, but other effectiveness outcomes such as those proposed by Rouwette et al. [16] are not explicitly or consistently documented. Furthermore, only 11 (15.3%) papers indicated that they had conducted process evaluations of the sessions. More rigorous and standardized reporting on the characteristics and effectiveness of GMB are needed to advance the field scientifically and refine its application in public health and health care. For this, we propose using the guidelines and scoresheet created by Rouwette et al., which outlines the recommended context, mechanism, and outcome variables that can be used as standard measurements for assessing GMB activities [16].
Fourth, previous studies have pointed out GMB as a resource-intensive approach that requires extensive technical knowledge, suggesting that its application in public health might be restricted in settings with limited access to modeling tools or training [14, 103]. However, our review of GMB applications across multiple contexts illustrates how the methodology can be tailored to varying resource constraints and participant educational levels. For instance, a participatory modeling exercise in Zambia with small-scale farmers demonstrated how GMB could be adapted to participants with low or no formal educational background and in settings without computers or electricity. The workshop used white boards and images of harvested maize and coins to illustrate variables in the model. Regular drinking glasses filled with water were used to illustrate stock and flow and feedback loops. Evaluations of these sessions a year later showed that participants found these tools to be salient in allowing them to see the interconnected nature of the system and its behavior [61]. Given the growing expansion of GMB across various settings, more research is needed on how the methodology can further be adapted for different population groups and for low resource environments. We recommend that modelers and facilitators in these settings take on an implementation science lens and formally document and evaluate any adaptations in their approach.
Finally, system dynamics practitioners have documented that the reliability, validity and the utility of system dynamics models are associated with the quality of stakeholder engagement, which is, in turn, related to the quality of GMB session facilitation [9, 104]. Although our review describes outcomes of GMB, further evaluation on fostering and sustaining high quality participation among an appropriate mix of stakeholders is warranted. Additionally, evaluation that examines how GMB can establish and strengthen multi-stakeholder research teams, working groups, and/or coalitions should be conducted.
Our study has several limitations. First, our search may not have captured the whole breadth of the public health literature on GMB as GMB may not be properly coded as a keyword in all publications. We mitigated this by consulting seven databases and checking the reference lists of eligible articles to identify any missing studies. We also utilized different variations of GMB terminology in our keywords and tested them across the different databases to ensure that we were capturing core articles in our search. Second, our findings were dependent on the availability of the data reported in the papers. As our selected outcomes were not consistently reported across all papers, it is possible that some outcome characteristics had occurred in real life but were not documented in these publications. Finally, the application of GMB in public health may be underreported in the literature, as some GMB activities may not have been documented, especially those that were conducted by public health practitioners rather than researchers.
Conclusion
GMB is used in system dynamics projects to convene diverse stakeholders in a facilitated, deliberative scientific process that aims to specify, test and validate models that aim to further understanding of, and action to address, complex systems problems [1, 2, 11–13]. This paper demonstrates the growing breadth of GMB activities in public health and health care, with an emphasis on informing interventions for complex health challenges and enhancing the implementation of such interventions. Our findings suggest that effective GMB is associated with higher impact models as well as tangible benefits to participants, including opportunities to learn and apply new systems thinking. While the quality of the studies varies, reflecting a longstanding need to provide more training and capacity building in this method, it is evident that GMB will become increasingly more prevalent and potent as a tool for facilitating stakeholder involvement, advancing implementation science, and ultimately, developing systems solutions in the future.
References
- 1. Andersen D, Vennix J, Richardson G, Rouwette E. Group model building: problem structing, policy simulation and decision support. The Journals of the Operational Research Society. 2007;58(5):691–4.
- 2. Andersen DF, Richardson GP, Vennix JAM. Group model building: Adding more science to the craft. System Dynamics Review. 1997;13(2):187–201.
- 3. Hovmand PS. Community based system dynamics [Internet]. Vol. 9781461487630. 2014. 1 p. Available from: https://www.scopus.com/inward/record.uri?eid=2-s2.0-84929907458&doi=10.1007%2f978-1-4614-8763-0&partnerID=40&md5=3ed840c2546d53297dd69b9fc46f1c43
- 4.
Sterman J. System dynamics: systems thinking and modeling for a complex world. In: ESD Working Papers. Cambridge, Massachusetts: Masachusetts Institute of Technology; 2002.
- 5. Kalk NJ, Robertson JR, Kidd B, Day E, Kelleher MJ, Gilvarry E, et al. Treatment and Intervention for Opiate Dependence in the United Kingdom: Lessons from Triumph and Failure. European Journal on Criminal Policy and Research. 2018 Jun 1;24(2):183–200.
- 6. Fischer B, Rehm J, Tyndall M. Effective Canadian policy to reduce harms from prescription opioids: learning from past failures. CMAJ. 2016 Dec 6;188(17–18):1240. pmid:27821465
- 7. Forrester JW. System dynamics—the next fifty years. System Dynamics Review. 2007 Jun 1;23(2‐3):359–70.
- 8. Andersen D, Calhoun A, Hovmand P, Hower T, Rouwette E, Steins E, et al. Scriptapedia. 2016. Roles in Group Model Building. Available from: https://en.wikibooks.org/wiki/Scriptapedia/Roles_in_Group_Model_Building
- 9. Hovmand PS, Andersen DF, Rouwette E, Richardson GP, Rux K, Calhoun A. Group Model-Building Scripts’ as a Collaborative Planning Tool. Systems Research and Behavioral Science. 2012;29(2):179–93.
- 10. Andersen D, Calhoun A, Hovmand P, Hower T, Rouwette E, Steins E, et al. Scriptapedia. 2020. Scriptapedia. Available from: https://en.wikibooks.org/wiki/Scriptapedia
- 11.
Klaus H, Ravera F, Tarrason D, Prell C. Participatory Modeling. In: International Encyclopedia fo Geography: People, the Earth, Environment, and Technology. Hoboken, NJ: John Wiley & Sons; 2017.
- 12. Richardson G, Andersen D. Teamwork in group model building. System Dynamics Review. 1994;11(2):113–37.
- 13. Andersen DF, Richardson GP. Scripts for group model building. System Dynamics Review. 1997;13(2):107–29.
- 14. Siokou C, Morgan R, Shiell A. Group model building: a participatory approach to understanding and acting on systems. Public Health Research and Practice. 2014;25(1). pmid:25828443
- 15. Costanza R, Ruth M. Using Dynamic Modeling to Scope Environmental Problems and Build Consensus. Environmental Management. 1998;22(2):183–95. pmid:9465128
- 16. Rouwette E, Vennix J, Van Mullekom T. Group model building effectiveness: a review of assessment studies. System Dynamics Review. 2002;18(1):5–45.
- 17. Scott RJ. Explaining how group model building supports enduring agreement. Journal of Management & Organization. 2019;25(6):783–806.
- 18. Scott RJ. Outcomes of Group Model Building. In: Group Model Building Using System Dynamics to Achive Enduring Agreement. 2018. p. 29–44.
- 19. Rouwette E, Vennix J. System Dynamics and Organizational Intervention. Systems Research and Behavioral Science. 2006;23:451–66.
- 20. Waqa G, Moodie M, Snowdon W, Latu C, Coriakula J, Allender S, et al. Exploring the dynamics of food-related policymaking processes and evidence use in Fiji using systems thinking. Health Res Policy Syst. 2017 Aug 29;15(1):74. pmid:28851398
- 21. Gullett HL, Brown GL, Collins D, Halko M, Gotler RS, Stange KC, et al. Using Community-Based System Dynamics to Address Structural Racism in Community Health Improvement. Journal of Public Health Management and Practice [Internet]. 2022;28(Supplement 4). Available from: https://journals.lww.com/jphmp/Fulltext/2022/07001/Using_Community_Based_System_Dynamics_to_Address.6.aspx pmid:35616557
- 22. Hendricks G, Savona N, Aguiar A, Alaba O, Booley S, Malczyk S, et al. Adolescents’ Perspectives on the Drivers of Obesity Using a Group Model Building Approach: A South African Perspective. Int J Environ Res Public Health. 2022 Feb 14;19(4). pmid:35206348
- 23. Broekhuizen H, Ifeanyichi M, Cheelo M, Drury G, Pittalis C, Rouwette E, et al. Policy options for surgical mentoring: Lessons from Zambia based on stakeholder consultation and systems science. PLOS ONE. 2021 Sep 29;16(9):e0257597. pmid:34587196
- 24. Broekhuizen H, Ifeanyichi M, Mwapasa G, Pittalis C, Noah P, Mkandawire N, et al. Improving Access to Surgery Through Surgical Team Mentoring–Policy Lessons From Group Model Building With Local Stakeholders in Malawi. International Journal of Health Policy and Management [Internet]. 2021; Available from: https://www.ijhpm.com/article_4088.html pmid:34380202
- 25. Sušnik J, Masia S, Indriksone D, Brēmere I, Vamvakeridou-Lydroudia L. System dynamics modelling to explore the impacts of policies on the water-energy-food-land-climate nexus in Latvia. Science of The Total Environment. 2021 Jun 25;775:145827. pmid:33611179
- 26. Ibrahim Shire M, Jun GT, Robinson S. Healthcare workers’ perspectives on participatory system dynamics modelling and simulation: designing safe and efficient hospital pharmacy dispensing systems together. Ergonomics. 2020 Aug;63(8):1044–56. pmid:32546060
- 27. Guariguata L, Rouwette EA, Murphy MM, Saint Ville A, Dunn LL, Hickey GM, et al. Using Group Model Building to Describe the System Driving Unhealthy Eating and Identify Intervention Points: A Participatory, Stakeholder Engagement Approach in the Caribbean. Nutrients. 2020 Jan 31;12(2).
- 28. Sarmiento OL, Higuera-Mendieta D, Wilches-Mogollon MA, Guzman LA, Rodríguez DA, Morales R, et al. Urban Transformations and Health: Methods for TrUST—a Natural Experiment Evaluating the Impacts of a Mass Transit Cable Car in Bogotá, Colombia. Frontiers in Public Health [Internet]. 2020;8. Available from: https://www.frontiersin.org/articles/10.3389/fpubh.2020.00064
- 29. Weeks MR, Li J, Lounsbury D, Green HD, Abbott M, Berman M, et al. Using Participatory System Dynamics Modeling to Examine the Local HIV Test and Treatment Care Continuum in Order to Reduce Community Viral Load. Am J Community Psychol. 2017 Dec;60(3–4):584–98. pmid:29154393
- 30. Weeks MR, Lounsbury DW, Li J, Hirsch G, Berman M, Green HD, et al. Simulating system dynamics of the HIV care continuum to achieve treatment as prevention. PLOS ONE. 2020 Mar 19;15(3):e0230568. pmid:32191771
- 31. Appel JM, Fullerton K, Hennessy E, Korn AR, Tovar A, Allender S, et al. Design and methods of Shape Up Under 5: Integration of systems science and community-engaged research to prevent early childhood obesity. PLoS One. 2019;14(8):e0220169. pmid:31369570
- 32. Hennessy E, Economos CD, Hammond RA. Integrating Complex Systems Methods to Advance Obesity Prevention Intervention Research. Health Educ Behav. 2020 Apr;47(2):213–23. pmid:32090653
- 33. Naumann RB, Kuhlberg J, Sandt L, Heiny S, Kumfer W, Marshall SW, et al. Integrating complex systems science into road safety research and practice, Part 2: applying systems tools to the problem of increasing pedestrian death rates. Inj Prev. 2020 Oct;26(5):424–31. pmid:31848213
- 34. Allender S, Owen B, Kuhlberg J, Lowe J, Nagorcka-Smith P, Whelan J, et al. A Community Based Systems Diagram of Obesity Causes. PLOS ONE. 2015 Jul 8;10(7):e0129683. pmid:26153893
- 35. McGlashan J, Hayward J, Brown A, Owen B, Millar L, Johnstone M, et al. Comparing complex perspectives on obesity drivers: action-driven communities and evidence-oriented experts. Obesity Science & Practice. 2018 Dec 1;4(6):575–81. pmid:30574350
- 36. Langellier B.A., Kuhlberg J.A., Ballard E.A., Slesinski S.C., Stankov I., Gouveia N., et al. Using community-based system dynamics modeling to understand the complex systems that influence health in cities: The SALURBAL study. Health and Place [Internet]. 2019;60(()). Available from: http://www.embase.com/search/results?subaction=viewrecord&from=export&id=L2003134782 pmid:31586769
- 37. Escobedo P, Gonzalez KD, Kuhlberg J, Calanche M, Baezconde-Garbanati L, Contreras R, et al. Community Needs Assessment among Latino Families in an Urban Public Housing Development. Hispanic Journal of Behavioral Sciences. 2019;41(3):344–62.
- 38. Fowler PJ, Wright K, Marcal KE, Ballard E, Hovmand PS. Capability Traps Impeding Homeless Services: A Community-Based System Dynamics Evaluation. J Soc Serv Res. 2019;45(3):348–59. pmid:31680707
- 39. Dianati K, Zimmermann N, Milner J, Muindi K, Ezeh A, Chege M, et al. Household air pollution in Nairobi’s slums: A long-term policy evaluation using participatory system dynamics. Sci Total Environ. 2019 Apr 10;660:1108–34. pmid:30743908
- 40. Ansah JP, Islam AM, Koh V, Ly V, Kol H, Matchar DB, et al. Systems modelling as an approach for understanding and building consensus on non-communicable diseases (NCD) management in Cambodia. BMC Health Services Research. 2019 Jan 3;19(1):2. pmid:30606199
- 41. Gamble E, Thorsen A, Black L. Expanding Strategic Opportunities in Nonprofits: Mapping the Interdependencies of Critical Performance Variables. Nonprofit Volunt Sect Q. 2019 Jun;48(3):616–32. pmid:31564768
- 42. Qayoom A H.W. Hadikusumo B. Multilevel safety culture affecting organization safety performance: a system dynamic approach. Engineering, Construction and Architectural Management. 2019 Jan 1;26(10):2326–46.
- 43. Hosseinichimeh N, Kim H, Ebrahimvandi A, Iams J, Andersen D. Using a Stakeholder Analysis to Improve Systems Modelling of Health Issues: The Impact of Progesterone Therapy on Infant Mortality in Ohio. Systems Research and Behavioral Science. 2019 Jul 1;36(4):476–93.
- 44. Baker P, Brown AD, Wingrove K, Allender S, Walls H, Cullerton K, et al. Generating political commitment for ending malnutrition in all its forms: A system dynamics approach for strengthening nutrition actor networks. Obes Rev. 2019 Nov;20 Suppl 2:30–44. pmid:31245905
- 45. Gerritsen S, Renker-Darby A, Harré S, Rees D, Raroa DA, Eickstaedt M, et al. Improving low fruit and vegetable intake in children: Findings from a system dynamics, community group model building study. PLoS One. 2019;14(8):e0221107. pmid:31415644
- 46. Urwannachotima N, Hanvoravongchai P, Ansah JP. Sugar-sweetened Beverage Tax and Potential Impact on Dental Caries in Thai Adults: An Evaluation Using the Group Model Building Approach. Systems Research and Behavioral Science. 2019 Feb 1;36(1):87–99.
- 47. Purwanto A, Sušnik J, Suryadi FX, de Fraiture C. Using group model building to develop a causal loop mapping of the water-energy-food security nexus in Karawang Regency, Indonesia. Journal of Cleaner Production. 2019 Dec 10;240:118170.
- 48. Roberts N, Li V, Atkinson JA, Heffernan M, McDonnell G, Prodan A, et al. Can the Target Set for Reducing Childhood Overweight and Obesity Be Met? A System Dynamics Modelling Study in New South Wales, Australia. Systems Research and Behavioral Science. 2019 Feb 1;36(1):36–52.
- 49. Mui Y, Ballard E, Lopatin E, Thornton RLJ, Pollack Porter KM, Gittelsohn J. A community-based system dynamics approach suggests solutions for improving healthy food access in a low-income urban environment. PLOS ONE. 2019 May 14;14(5):e0216985. pmid:31086409
- 50. Pagano A, Pluchinotta I, Pengal P, Cokan B, Giordano R. Engaging stakeholders in the assessment of NBS effectiveness in flood risk reduction: A participatory System Dynamics Model for benefits and co-benefits evaluation. The Science of the total environment. 2019;690:543–55. pmid:31301495
- 51. Malard J, JF Adamowski, M Rojas D, Carrera J, J Gálvez, Monadres H, et al. Use of participatory system dynamics modelling to assess the sustainability of smallholder agriculture. In St. Joseph, MI: ASABE; 2015. p. 1. (ASABE Paper No. 152189490). Available from: https://elibrary.asabe.org/abstract.asp?aid=46322&t=5
- 52. Koh K, Reno R, Hyder A. Designing an Agent-Based Model Using Group Model Building: Application to Food Insecurity Patterns in a U.S. Midwestern Metropolitan City. J Urban Health. 2018 Apr;95(2):278–89. pmid:29536416
- 53. Ansah JP, Matchar DB, Koh V, Schoenenberger L. Mapping the Dynamic Complexity of Chronic Disease Care in Singapore: Using Group Model Building in Knowledge Elicitation. Systems Research and Behavioral Science. 2018 Nov 1;35(6):759–75.
- 54. Eker S, Zimmermann N, Carnohan S, Davies M. Participatory system dynamics modelling for housing, energy and wellbeing interactions. Building Research & Information. 2018 Oct 3;46(7):738–54.
- 55. Chalise N, Kumar P, Priyadarshini P, Yadama GN. Dynamics of sustained use and abandonment of clean cooking systems: lessons from rural India. Environmental Research Letters. 2018 Mar;13(3):035010.
- 56. Lembani M, de Pinho H, Delobelle P, Zarowsky C, Mathole T, Ager A. Understanding key drivers of performance in the provision of maternal health services in eastern cape, South Africa: a systems analysis using group model building. BMC Health Services Research. 2018 Nov 29;18(1):912. pmid:30497460
- 57. Reno R. Using group model building to develop a culturally grounded model of breastfeeding for low-income African American women in the USA. J Clin Nurs. 2018 Sep;27(17–18):3363–76. pmid:28252834
- 58. Brown H, Proust K, Newell B, Spickett J, Capon T, Bartholomew L. Cool Communities-Urban Density, Trees, and Health. Int J Environ Res Public Health. 2018 Jul 22;15(7). pmid:30037129
- 59. Atkinson JA, O’Donnell E, Wiggers J, McDonnell G, Mitchell J, Freebairn L, et al. Dynamic simulation modelling of policy responses to reduce alcohol-related harms: rationale and procedure for a participatory approach. Public Health Research & Practice [Internet]. Available from: https://www.phrp.com.au/issues/february-2017-volume-27-issue-1-2/dynamic-simulation-modelling-of-policy-responses-to-reduce-alcohol-related-harms-rationale-and-procedure-for-a-participatory-approach/ pmid:28243673
- 60. Freebairn L, Rychetnik L, Atkinson JA, Kelly P, McDonnell G, Roberts N, et al. Knowledge mobilisation for policy development: implementing systems approaches through participatory dynamic simulation modelling. Health Res Policy Syst. 2017 Oct 2;15(1):83. pmid:28969642
- 61. Kopainsky B, Hager G, Herrera H, Nyanga PH. Transforming food systems at local levels: Using participatory system dynamics in an interactive manner to refine small-scale farmers’ mental models. Ecological Modelling. 2017;362:101–10.
- 62. Morrow-Howell N, Halvorsen CJ, Hovmand P, Lee C, Ballard E. Conceptualizing Productive Engagement in a System Dynamics Framework. Innov Aging. 2017 Mar 1;1(1):igx018. pmid:30480112
- 63. Macmillan A, Woodcock J. Understanding bicycling in cities using system dynamics modelling. Journal of Transport & Health. 2017 Dec 1;7:269–79. pmid:29276678
- 64. Mumba C, Skjerve E, Rich M, Rich KM. Application of system dynamics and participatory spatial group model building in animal health: A case study of East Coast Fever interventions in Lundazi and Monze districts of Zambia. PLoS One. 2017;12(12):e0189878. pmid:29244862
- 65. Jetha A, Kernan L, Kurowski A. Conceptualizing the dynamics of workplace stress: a systems-based study of nursing aides. BMC Health Serv Res. 2017 Jan 5;17(1):12. pmid:28056973
- 66. Trani JF, Ballard E, Bakhshi P, Hovmand P. Community based system dynamic as an approach for understanding and acting on messy problems: a case study for global mental health intervention in Afghanistan. Confl Health. 2016;10:25. pmid:27822297
- 67. Jetha A, Pransky G, Fish J, Hettinger LJ. Return-to-Work Within a Complex and Dynamic Organizational Work Disability System. J Occup Rehabil. 2016 Sep;26(3):276–85. pmid:26547909
- 68. Frerichs L, Lich KH, Funchess M, Burrell M, Cerulli C, Bedell P, et al. Applying Critical Race Theory to Group Model Building Methods to Address Community Violence. Prog Community Health Partnersh. 2016;10(3):443–59. pmid:28230552
- 69. Zimmerman L, Lounsbury DW, Rosen CS, Kimerling R, Trafton JA, Lindley SE. Participatory System Dynamics Modeling: Increasing Stakeholder Engagement and Precision to Improve Implementation Planning in Systems. Adm Policy Ment Health. 2016 Nov;43(6):834–49. pmid:27480546
- 70. Macmillan A, Davies M, Shrubsole C, Luxford N, May N, Chiu LF, et al. Integrated decision-making about housing, energy and wellbeing: a qualitative system dynamics model. Environ Health. 2016 Mar 8;15 Suppl 1(Suppl 1):37. pmid:26961081
- 71. Brownson RC, Kemner AL, Brennan LK. Applying a mixed-methods evaluation to Healthy Kids, Healthy Communities. J Public Health Manag Pract. 2015 Jun;21 Suppl 3(Suppl 3):S16–26. pmid:25828217
- 72. Homa L, Rose J, Hovmand PS, Cherng ST, Riolo RL, Kraus A, et al. A participatory model of the paradox of primary care. Ann Fam Med. 2015 Sep;13(5):456–65. pmid:26371267
- 73. Thomas IM, Reilly SR. Group model building: a framework for organizing healthy community program and policy initiatives in Columbia, Missouri. J Public Health Manag Pract. 2015 Jun;21 Suppl 3:S79–83. pmid:25828226
- 74. Brennan LK, Sabounchi NS, Kemner AL, Hovmand P. Systems thinking in 49 communities related to healthy eating, active living, and childhood obesity. J Public Health Manag Pract. 2015 Jun;21 Suppl 3:S55–69. pmid:25828223
- 75. Munar W, Hovmand PS, Fleming C, Darmstadt GL. Scaling-up impact in perinatology through systems science: Bridging the collaboration and translational divides in cross-disciplinary research and public policy. Semin Perinatol. 2015 Aug;39(5):416–23. pmid:26184341
- 76. Keane P, Ortega A, Linville J. Healthy Kids, Healthy Cuba: findings from a group model building process in the rural Southwest. J Public Health Manag Pract. 2015 Jun;21 Suppl 3:S70–73. pmid:25828224
- 77. Moreland JW. Improving Park Space Access for the Healthy Kids, Healthy Communities Partnership in Denver, Colorado. Journal of Public Health Management and Practice. 2015;21:S84–7.
- 78. Skouteris H, Huang T, Millar L, Kuhlberg J, Dodd J, Callaway L, et al. A systems approach to reducing maternal obesity: The Health in Preconception, Pregnancy and Postbirth (HIPPP) Collaborative. Aust N Z J Obstet Gynaecol. 2015 Aug;55(4):397–400. pmid:26121995
- 79. Esensoy AV, Carter MW. Health system modelling for policy development and evaluation: Using qualitative methods to capture the whole-system perspective. Operations Research for Health Care. 2015;4:15–26.
- 80. Ager AK, Lembani M, Mohammed A, Mohammed Ashir G, Abdulwahab A, de Pinho H, et al. Health service resilience in Yobe state, Nigeria in the context of the Boko Haram insurgency: a systems dynamics analysis using group model building. Conflict and Health. 2015 Oct 5;9(1):30. pmid:26442129
- 81. van Nistelrooij LPJ, Rouwette EAJA, Verstijnen IM, Vennix JAM. The Eye of the Beholder: A Case Example of Changing Clients’ Perspectives Through Involvement in the Model Validation Process. Systems Research and Behavioral Science. 2015 Jul 1;32(4):437–49.
- 82. Macmillan A, Connor J, Witten K, Kearns R, Rees D, Woodward A. The societal costs and benefits of commuter bicycling: simulating the effects of specific policies using system dynamics modeling. Environ Health Perspect. 2014 Apr;122(4):335–44. pmid:24496244
- 83. Narayana S A., Elias A A., Pati R K. Reverse logistics in the pharmaceuticals industry: a systemic analysis. The International Journal of Logistics Management. 2014 Jan 1;25(2):379–98.
- 84. Biroscak BJ, Schneider T, Panzera AD, Bryant CA, McDermott RJ, Mayer AB, et al. Applying Systems Science to Evaluate a Community-Based Social Marketing Innovation: A Case Study. Social Marketing Quarterly. 2014 Dec 1;20(4):247–67.
- 85. Rouwette E, Bleijenbergh I, Vennix J. Group Model-Building to Support Public Policy: Addressing a Conflicted Situation in a Problem Neighbourhood. Systems Research and Behavioral Science. 2014 Aug 1;33.
- 86. Merrill JA, Deegan M, Wilson RV, Kaushal R, Fredericks K. A system dynamics evaluation model: implementation of health information exchange for public health reporting. Journal of the American Medical Informatics Association. 2013 Jan;20(e1):e131–8. pmid:23292910
- 87. Hernantes J, Labaka L, Lauge A, Sarriegi J, Gonzalez J. Group model building: A collaborative modelling methodology applied to critical infrastructure protection. Int J of Organisational Design and Engineering. 2012 Jan 1;2:41–60.
- 88. Goh YM, Love PED, Stagbouer G, Annesley C. Dynamics of safety performance and culture: A group model building approach. Accident Analysis & Prevention. 2012 Sep 1;48:118–25. pmid:22664675
- 89. Bridgewater K, Peterson S, McDevitt J, Hemenway D, Bass J, Bothwell P, et al. A Community-Based Systems Learning Approach to Understanding Youth Violence in Boston. Progress in community health partnerships: research, education, and action. 2011 Mar 1;5:67–75. pmid:21441670
- 90. Thompson J, Forster C, Werner C, Peterson T. Mediated Modeling: Using Collaborative Processes to Integrate Scientist and Stakeholder Knowledge about Greenhouse Gas Emissions in an Urban Ecosystem. Society & Natural Resources. 2010 Aug 1;23:742−757.
- 91. Stave K. Participatory system dynamics modeling for sustainable environmental management: Observations from four cases. Sustainability. 2010;2:2762–84.
- 92. Cavana RY, Clifford LV. Demonstrating the utility of system dynamics for public policy analysis in New Zealand: the case of excise tax policy on tobacco. System Dynamics Review. 2006 Dec 1;22(4):321–48.
- 93. Stave KA. Using system dynamics to improve public participation in environmental decisions. System Dynamics Review. 2002 Jun 1;18(2):139–67.
- 94. World Bank. Data. 2021. World Bank Country and Lending Groups. Available from: https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups
- 95. Black LJ. When visuals are boundary objects in system dynamics work. System Dynamics Review. 2013 Apr 1;29(2):70–86.
- 96. Ghaffarzadegan N, Lyneis J, Richardson G. How Small System Dynamics Models Can Help the Public Policy Process. System Dynamics Review. 2011;271(1):22–44.
- 97. Hovmand P. Group Model Building and Community-Based System Dynamics Process. In 2014.
- 98. Gerritsen S, Harré S, Rees D, Renker-Darby A, Bartos AE, Waterlander WE, et al. Community Group Model Building as a Method for Engaging Participants and Mobilising Action in Public Health. Int J Environ Res Public Health. 2020 May 15;17(10):3457. pmid:32429183
- 99. Cassel MA, Hinsberger M. Flood partnerships: a participatory approach to develop and implement the Flood Risk Management Plans. Journal of Flood Risk Management. 2017;10(2):164–72.
- 100. Iturriza M, Hernantes J, Labaka L. Coming to Action: Operationalizing City Resilience. Sustainability. 2019;11(11):1–18.
- 101. Chen PG, Diaz N, Lucas G, Rosenthal MS. Dissemination of results in community-based participatory research. Am J Prev Med. 2010 Oct;39(4):372–8. pmid:20837290
- 102. Hosseinichimeh N, MacDonald R, Hyder A, Ebrahimvandi A, Porter L, Reno R, et al. Group Model Building Techniques for Rapid Elicitation of Parameter Values, Effect Sizes, and Data Sources. System Dynamics Review. 2017;33(1):71–84.
- 103. Vennix JAM. Group model-building: tackling messy problems. System Dynamics Review. 1999 Dec 1;15(4):379–401.
- 104. Bendoly E. System Dynamics Understanding in Projects: Information Sharing, Psychological Safety, and Performance Effects. Production and Operations Management. 2014 Aug 1;23(8):1352–69.