https://orcid.org/0000-0002-2507-3789
Figures
Abstract
Public health scholarship has increasingly called for the use of system science approaches to understand complex problems, including the use of participatory engagement to inform the modeling process. Some system science traditions, specifically system dynamics modeling, have an established participatory practice tradition. Yet, there remains limited guidance on engagement strategies using other modeling approaches like agent-based models. Our objective is to describe how we engaged adolescent youth in co-building an agent-based model about physical activity. Specifically, we aim to describe how we communicated technical aspects of agent-based models, the participatory activities we developed, and the resulting visual diagrams that were produced. We implemented six sessions with nine adolescent participants. To make technical aspects more accessible, we used an analogy that linked core components of agent-based models to elements of storytelling. We also implemented novel, facilitated activities that engaged youth in the development, annotation, and review of graphs over time, geographical maps, and state charts. The process was well-received by the participants and helped inform the basic structure of an agent-based model. The resulting visual diagrams created space for deeper discussion among participants about patterns of daily activity, important places for physical activity, and interactions between social and built environments. This work lays a foundation to develop and refine engagement strategies, especially for translating qualitative insights into quantitative model specifications such as ‘decision rules’.
Citation: Frerichs L, Smith N, Kuhlberg JA, Mason G, Jackson-Diop D, Stith D, et al. (2020) Novel participatory methods for co-building an agent-based model of physical activity with youth. PLoS ONE 15(11): e0241108. https://doi.org/10.1371/journal.pone.0241108
Editor: Emma Louise Giles, Teesside University, UNITED KINGDOM
Received: April 29, 2020; Accepted: October 9, 2020; Published: November 10, 2020
Copyright: © 2020 Frerichs et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: This project was supported by the National Heart Lung and Blood Institute (Grant Number 5K01HL138159; PI Frerichs). It was also supported in part by the National Institutes of Health under award numbers R01AG047869 and K24HL105493. NRS is grateful to the Carolina Population Center for training support (T32 HD091058) and for general support (P2C HD050924). JK and DJD are with System Stars, LLC and an Independent Consultant, respectively. No funding bodies had a role in the design, collection, analysis, interpretation, nor writing of the study. Its contents are the authors’ sole responsibility and do not necessarily represent official NIH views. The funders only provided support in the form of salaries for authors and research materials, but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the ‘author contributions’ section.
Competing interests: JK is employed by System Stars, LLC and Damie Jackson-Diop is an independent consultant. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
Introduction
Understanding how to improve behaviors like physical activity is complex. Many levels of influence, including individual motivations, interpersonal social networks, and built environments, interact to shape behavior over time. Scholars have called for the use of systems science and simulation modeling to help manage the complexity of this type of public health issue [1–4]. Systems science is an interdisciplinary field and approach to inquiry that focuses on understanding interrelated and interacting entities that form a unified whole. Simulation modeling, the process of creating and analyzing a digital prototype that emulates a real-life system, is a method often used in systems science. Furthermore, it is recommended that the simulation models are developed with stakeholders [1, 3, 5, 6], which has potential to produce better models, increase the social capital of communities, and improve the chance that a model successfully influences decision making [7].
To our knowledge, there has been limited work to develop and formalize methods that facilitate stakeholder engagement in building agent-based models for public health issues. Once an agent-based model is created, the simulated animations that result can be useful tools for engaging participants. For example, one description of engagement with an agent-based model related to primary care outcomes focused heavily on simulation results from the ultimate model produced [8]. Yet, there is less information about processes for initial structuring and development of agent-based models with stakeholders. One recent study that engaged stakeholders to build an agent-based model around the issue of food insecurity applied scripts from system dynamics modeling [9]. The authors concluded that the scripts were useful in creating conceptual models, but that future work was needed to establish scripts specific to agent-based models due to the vastly different scale of analysis between agent-based and system dynamics models.
In public health, there has been a relatively strong emergence of scholarly literature focused on participatory approaches to building system dynamics models [10–20]. The popularity of participatory system dynamics is likely due, in part, to the field’s historical participatory approach to model building [7], which has included the development, documentation, and sharing of best practices for facilitators [21] of group model building activities called “scripts” [22, 23]. While the detailed protocols are an important tool to support group model building in system dynamics, the choice and adaptation of a given script (or set of scripts) for use with certain communities or groups relies on managing the modeling process outputs as boundary objects. Boundary objects are the visual representations of the system dynamics model or model elements that result from the engagement process. In system dynamics, boundary objects such as behavior-over-time graphs, causal loop diagrams, and stock and flow diagrams have been considered critical to the process and valuable for their ability to be transformable by all and represent dependencies across stakeholders [24]. The close correspondence between the visual diagramming conventions of system dynamics models and their underlying mathematics has facilitated the management of boundary objects. In contrast, it is less clear how agent-based models can be managed as boundary objects because agent-based models are often defined in lines of software code that can be difficult to directly visualize.
We sought to develop and pilot test new scripted activities specifically for developing agent-based models. Our aim was to develop an approach that would aid participants’ understanding of elements of agent-based models that are difficult to visually represent and, where possible, develop new scripts that would result in boundary objects relevant for agent-based models. In this paper, we describe our process and outcomes from piloting a participatory agent-based model building approach alongside adolescent youth and with a focus on physical activity. We engaged youth as our stakeholders for their unique perspective as well as for the potential additional benefits including improvement in youth’s own leadership, personal agency, and collective empowerment.
Method
The study was reviewed and approved by the institutional review board at the University of North Carolina at Chapel Hill.
Setting
We conducted this project in one small town (population of 10,844) in eastern North Carolina. African Americans are 48.3%, Non-Hispanic Whites are 44.8%, and Hispanics are 5.1% of the population [25]. The town is situated in a county that has one of the highest unemployment rates in North Carolina and 24.6% of residents live in households with incomes below the federal poverty level [25].
Modeling team
In addition to three researchers experienced with participatory modeling, our team included community health leaders. One of the community health leaders and collaborator (DS) has been involved in public health research for over 20 years. Her work has included the formation of a county council for adolescent health and she had been involved in a prior system dynamics participatory modeling project [26]. Another community health leader has experience facilitating youth initiatives (DJ) and has designed and implemented youth curricula with an emphasis on mental and behavioral health. Our team also included public health students with a range of experience areas including the development and facilitation of asset mapping and socio-emotional learning curriculum for high school students. Finally, we worked with the youth participants to develop a list of roles and responsibilities that they could assume on a rotating basis (e.g., ice breaker expert–develops ice breakers for each session, feedback queen/king–communicates what is and is not working during the sessions). This was done to provide leadership opportunities for all youth involved as well as foster ownership and cohesiveness among the group.
Session development
We developed and implemented content for six sessions that were targeted to last between 1–2 hours each. During the sessions, we sought to elicit information from the students on: 1) important physical activity topics to consider, 2) where physical activity takes place, 3) how youths’ location and physical activity changes throughout the day, 4) with whom physical activity takes place, and 5) other factors that influence changes in locations and physical activity levels. Although, the majority of the session content was focused on activities to inform an agent-based model, we also introduced youth to concepts of community-engaged research, implemented team and leadership building activities, and initiated discussions about collecting data for a future quantified model.
When building the model structure, we recognized that agent-based models can be difficult to conceptualize [9]. To make the model structure more accessible, we used the concept of storytelling. We linked the core components of agent-based models; 1) the topic of study, 2) agents, 3) properties and rules, 4) environment, and 5) simulation results, to core components of storytelling: 1) the conflict, 2) characters, 3) character development, 4) setting, and 5) the plot. Table 1 highlights each of these linked concepts plus two examples of the storytelling links in application to agent-based models of physical activity. The first example applies the link to an agent-based model from the extant literature and the second illustrates the application from our pilot study with youth.
The storytelling analogy was introduced in the first session and integrated into all following session materials. More specifically, we began our sessions with the concept of a storytelling ‘conflict’ and we elicited responses from the participants about what most ‘got in the way’ or ‘helped’ them to be physical active, which was used to refine the research questions. We also highlighted that agents in the model were like the characters of a story and used activities throughout the sessions to elicit more information about important qualities and factors that influenced (or were influenced by) their physical activity levels, akin to character development. The model environment was described as a story’s setting and participants were led through activities to identify and describe important locations in their lives and for physical activity. Finally, the model simulation was likened to a story’s plot. An existing simulation model was used to illustrate how simulation models produce outcomes over time and specifically highlight emergent dynamics of agent-based models.
Participant recruitment
Our community partner contacted the school district’s superintendent and held several meetings also including the high school guidance counselors and principal. These meetings focused on communicating the scope of project, outlining the resources needed to carry out the project (e.g., meeting space in the school), and how recruitment would take place. The high school distributed information about the project via an email to all students with a survey link they could respond to indicating their interest. In addition, our community partner recruited additional youth using convenience sampling strategies. Specifically, she contacted individuals who had been involved with her organizations’ other programs and received recommendations from the guidance counselor. To be eligible participants had to be in grades 9–11 and could not have significant cognitive impairments. After students expressed interest, we held a preliminary meeting to explain the scope of the project. The students completed assent forms and their parents/guardians completed and signed informed consent forms as well. Students were provided a $20 cash incentive for each session that they attended.
Session evaluation
After each session we asked students to respond to a written survey with 4–6 short-answer questions. The questions asked the students to recall what they learned during each session, points of confusion, as well as general feedback and feedback related to a specific scripted activities (e.g., “What, if anything, did you learn about by participating in the physical activity mapping activity?”). After the final session, we also implemented a low-risk, not time intensive debriefing technique called “keep, start, stop” [28, 29]. In sequence, we asked the participants: “what should we keep/start/stop doing?” for future sessions.
Overview of session scripts and boundary objects
Fig 1 provides a high-level overview of each session and evaluation. Table 2 provides a detailed summary of each session, description of how elements of storytelling were incorporated, and the specific scripts developed and used. Following, we describe four new scripts we created and implemented specifically for agent-based modeling.
Graphs Over Time script.
The Graphs Over Time script is a well-established component of group model building in system dynamics that we adapted to understand the typical daily patterns of physical activity of youth [33]. The script engages participants in developing plots of one or more variables that capture how the issue of interest and other relevant factors change over time. This activity is typically done in the early stages of a model building effort in order to help better understand and frame the problem. The standard process for this script involves four steps: 1) a facilitator provides an example plot where the x-axis is labeled as “time” and the y-axis is labeled with variable name(s), 2) participants alone, or in small groups, develop their own plots, 3) the participants share their resulting plots with the group, and 4) the facilitator and participants identify themes that emerge from the graphs.
Our goal was to use a similar process to generate and understand plausible patterns of behavior for an agent-based model that could help guide early model calibrations. We followed these same steps with additional annotation exercises (S1 Appendix). A facilitator asked the participants to chart their physical activity level (y-axis) throughout a typical day (x-axis). However, because agent-based models have a focus on social interactions and the environment, we asked participants to further annotate their plots. Specifically, we asked participants to write who they were with (social interactions) and where they were (environment) on sticky notes and overlay these annotations on their plots (Fig 2A). This process created individual plots as boundary objects. Following the session, we created digital graphs that aggregated the groups’ plots. Photos of their individual graphs and the aggregated plots (Fig 2B) were presented and discussed at a subsequent session. Showcasing the aggregated results helped the group identify common themes such as sedentary and active times.
Mapping Important Locations script.
One of the major advantages of agent-based models is the ability to incorporate geographic information such as road networks, locations of specific exposures of interest, and attributes of spatial features. We developed a new script focused on identifying geographic model boundaries, important locations with potential to affect the outcome of interest, and anticipated movement of individuals between various locations (S1 Appendix). For this scripted activity, we obtained a poster-sized map (5x3.5 feet) of the general location of interest (i.e., where the participants lived, socialized, went to school/work) and smaller maps (8.5x11 inch) for each participant. Prior to the session, participants were asked to identify and take pictures of roughly five important places we should include in the model, including places where physical activity happens and places where it does not in their community. Working in small groups, participants were asked to use stickers to locate where their pictures were taken on the map, discuss the locations and their importance, and identify 3–6 locations that the small group felt were most important to their physical activity. Additionally, the participants were guided to annotate their individual maps with examples of how they typically moved from one location to another, highlighting specific routes and modalities of travel (e.g., walk, car). The facilitator also convened the whole group to locate and discuss their prioritized places, travel routes, and modes of travel on the larger map.
This process created individual maps and an annotated group map as boundary objects. Further, following the session, we used the information to create a simple agent-based model that showed the participants’ mobility patterns within their geography. The agent-based model was shown to participants and discussed at a subsequent session. This scripted activity helped identify major patterns of mobility among the population of interest and other important concepts that further refined the modeling problem. For example, from this exercise we identified car transportation as the primary mode of travel to and from school.
State Chart Review script.
The facilitator team used the Graphs Over Time and Mapping Important Locations outputs to develop state charts to include in the model. States and transitions between them are abstractions used in agent-based models to model different conditions that individuals are in at different times (e.g., healthy, infected, dead) and the potential pathways between the states (i.e., an individual can move from healthy to dead or healthy to infected but not dead to healthy). State charts are visual depictions of these conditions. For this scripted activity, we created state charts using AnyLogic (Version 8.4.0 Personal Learning Addition, 2019) and created print-outs of the diagrams. We created states mostly based on the participants’ major locations that they identified as part of their typical daily routines (i.e., home, school, and after school/evening places); wherein each location could correspond to a general likelihood of engaging in different levels of sedentary or physical activity. During the session, we presented the state chart and provided print-outs to each participant. The facilitator provided an overview of the state charts and described how they were informed by the prior sessions. Participants were then asked to use highlighters to trace their potential pathways through the states. The group then discussed the comprehensiveness of the states and whether any states or transitions were missing. The state chart was a useful boundary object for discussion and led to the discovery of missing states (i.e., church, family/friend residences) and refinements to the state chart (Fig 3).
Interviewing to Understand Decision Rules script.
In an agent-based model, decisions and rules need to be formulated to guide each individual agent’s transitions between states (e.g., an agent’s number of friends who are exercising regularly could influence the probability that they will also exercise). Although the “State Chart Review” activity provided useful information about model states, deeper information surrounding what creates transitions between states was largely missing. Thus, we developed a script to help identify the factors that were perceived as important to creating shifts between states (S1 Appendix).
For this activity, we used a two-phased process implemented across two sessions. During the first session, we guided participants through an activity to interview each other with the goal to stimulate deeper thinking about the issue of interest, physical activity. We introduced them to the basics of interviewing for research (e.g., use of open-ended and probing questions, active listening, reflecting and summarizing answers) and two facilitators role-played a mock interview. Next, we divided the participants into pairs. We provided them with a guiding set of questions and approximately 30 minutes to interview each other. Afterwards, the group convened and each individual shared what they learned from their respective partner about influences on physical activity. A facilitator listened and summarized major themes that emerged from the discussion.
During the second session, the facilitators recapped themes from the prior session’s interviews and communicated the session’s goal of further annotating state charts. The facilitator projected simplified state charts that focused the participants on the transitions between specific states (i.e., Home not Active, to Home but Active; Home not Active, to Parks/Active Places). Participants were asked to think about what would make them stay versus make the various transitions and annotate their individual state charts. Next the facilitator asked each participant, in a round-robin style, to provide factors that influence their transition decisions. This process helped establish additional key relationships and model elements previously missing such as how having family members of similar age influenced activity. This activity resulted in annotated state charts as boundary objects (Fig 4)
Results
The artifacts from each of the activities provided important insights for different aspects of the model building process. For example, Graphs over Time artifacts revealed insights with implications for the model time scale while Mapping Important Locations provided insights with implications for environmental boundaries and agent-environment interactions within the model. More specifically, the Graphs Over Time artifacts revealed time periods with more (after school) and less (during school) variation in daily activity and Mapping Important Locations artifacts indicated a few key locations (school, home) most relevant to sedentary and physical activity. Furthermore, all artifacts provided insights relevant to data collection and analysis for future model parameterization. For example, the Decision Rules artifacts indicated that social interactions were likely influential in physical activity choices and more information was needed to operationalize the pathways and processes by which friends and family impact activity decisions on a day-to-day basis.
Results from the post-session surveys indicated the participants found the storytelling format and scripted activities useful and rarely had areas of confusion. From the open-ended responses, we found four major themes about what participants learned, specifically participants described: 1) social connections: learning more about and from each other, 2) leadership growth: improving listening and critical thinking skills, 3) spatial knowledge: gaining new skills in reading and understanding maps, 4) physical activity: increasing awareness about their personal physical activity. Specifically, regarding social connections, participants reported learning more about their peers’ experiences outside of the study. For example, one participant noted learning that someone in the group had gone surfing and hiking and others learned that most people enjoyed physical activity with friends and family. In relation to leadership skills, beyond gaining tangible skills in active listening and thinking deeply about themselves, students were also exposed to youth leadership trainings in the area they could attend beyond the lifespan on the project.
From the ‘keep, stop, start’ activity, the participants indicated that they enjoyed the team-building and leadership activities and suggested they be kept in future sessions. Participants also desired more activities aimed at increasing group cohesion. Suggestions included several outdoor activities (i.e. a picnic and swimming) as well as a trip elsewhere (i.e. to an amusement park or other destination). These findings suggested that the group was continuing to learn more about one another even at the final session and that participants had a desire to form community outside of the traditional meeting space. Participants appreciated the value and insights gained from the model building activities and indicated they would not ‘stop’ any of them. Suggestions about new things to “start” focused primarily on group logistics such as the meeting room, food, and incentives.
In summary, beyond the tangible spatial and mapping skills as well as critical thinking and listening skills students gained, there was also an appreciation for fostering community both environmentally and on a personal level. What may appear as small details that were not obvious to the success to the study at onset, i.e. food suggestions, building in small celebrations outside of formal meeting spaces, and taking time to get to know one another on a personal level, were crucial elements in the eyes of participants.
Discussion
We developed and implemented several innovative strategies for partnering with community members to co-develop the basic structure of an agent-based model. Overall, our process was well-received by the participants. Our team found storytelling to be a helpful analogy in communicating the technical aspects of agent-based models. Additionally, the newly scripted activities resulted in visual representations of the model and boundary objects that the participants could view and discuss together.
We found that the storytelling analogy as well as allowing the individual participants to tell their own stories through the “Interviewing for Decision Rules” activity both improved the level of engagement. The use of imagery and narrative is a well-established method in participatory approaches as it relies less on the written word and provides a variety of media that leverages community expertise. A recent study that engaged participants in developing an agent-based model similarly described using aspects of storytelling [34]. The recent study presented clinical case histories, “life stories”, to help walk participants through the model structure and logic. In contrast to this method, our team drew on our participant’s first-hand knowledge and lived experiences to review state charts, identify pathways, and discuss factors that influence their transitions between places. Our strategy is likely more useful when engaging individuals who have directly experienced the issue at hand; whereas, case histories may be more useful when engaging stakeholders who have second hand experience from individuals they care for whom are affected (e.g., healthcare providers, teachers).
Our process resulted in artifacts that deepened discussion and insights surrounding social interactions and spatial locations, but we also uncovered areas where more work is necessary. Specifically, we found that developing quantifiable decision rules from our resulting artifacts and activities was a challenge. Others have noted that there is limited guidance on the process of translating the visual, qualitative models to quantified simulation models [34]. A recent case study described high-level elements of a participatory approach to develop a dynamic simulation model, but noted that modeling decisions were highly interactive and iterative. Further research is needed to develop more precise strategies for translating qualitative insights into formulaic and mathematical specifications for agent-based models. We recommend that public health draws from the environmental science discipline, which has a more substantial history of participatory engagement using agent-based models. For example, the field has developed and used “companion modeling” for agent-based modeling that emphasizes the use of role-playing sessions [35–37]. The role-playing sessions use information about potential factors that influence decisions to develop games where participants use spreadsheets or board games to ‘simulate’ decisions. The role playing sessions can help refine models by identifying irrelevant factors or clarifying relative weights placed on different factors in the decision making process [35, 36]. Alternatively, additional empirical, quantitative research may be necessary for such formulations. As such, one of our project's next steps is to employ primary data collection with the youth and their peers that will aid in model quantification.
In sum, while participatory engagement to develop system dynamics models has been emerging in the public health literature, there have been far fewer examples for agent-based models. Our work adds to the limited guidance for researchers interested in co-developing agent-based models alongside community stakeholders. We found our methods were promising and produced visual elements relevant for agent-based models that participants can equitably own and discuss in the process.
Supporting information
S1 Appendix. Agent-based model building scripts.
https://doi.org/10.1371/journal.pone.0241108.s001
(DOCX)
Acknowledgments
The authors would like to thank the Tarboro High School for their support and assistance with this project.
References
- 1. Gittelsohn J, Mui Y, Adam A, Lin S, Kharmats A, Igusa T, et al. Incorporating systems science principles into the development of obesity prevention interventions: principles, benefits, and challenges. Current Obesity Reports. 2015;4(2):174–81. pmid:26069864
- 2. Lich KH, Ginexi EM, Osgood ND, Mabry PL. A call to address complexity in prevention science research. Prevention Science. 2013;14(3):279–89. pmid:22983746
- 3. Frerichs L, Lich KH, Dave G, Corbie-Smith G. Integrating systems science and community-based participatory research to achieve health equity. American Journal of Public Health. 2016;106(2):215–22. pmid:26691110
- 4. Diez Roux AV. Complex systems thinking and current impasses in health disparities research. American Journal of Public Health. 2011;101(9):1627–34. pmid:21778505
- 5. Van Voorn G, Verburg R, Kunseler E-M, Vader J, Janssen PH. A checklist for model credibility, salience, and legitimacy to improve information transfer in environmental policy assessments. Environmental Modelling & Software. 2016;83:224–36.
- 6. Voinov A, Bousquet F. Modelling with stakeholders. Environmental Modelling & Software. 2010;25(11):1268–81.
- 7. Király G, Miskolczi P. Dynamics of participation: System dynamics and participation—An empirical review. Systems Research and Behavioral Science. 2019;36:199–210.
- 8. Homa L, Rose J, Hovmand PS, Cherng ST, Riolo RL, Kraus A, et al. A Participatory Model of the Paradox of Primary Care. Annals of Family Medicine. 2015;13(5):456–65. pmid:26371267
- 9. Koh K, Reno R, Hyder A. Designing an Agent-Based Model Using Group Model Building: Application to Food Insecurity Patterns in a U.S. Midwestern Metropolitan City. Journal of Urban Health. 2018;95(2):278–89. pmid:29536416
- 10. Apostolopoulos Y, Lemke MK, Barry AE, Lich KH. Moving alcohol prevention research forward-Part II: new directions grounded in community-based system dynamics modeling. Addiction (Abingdon, England). 2018;113(2):363–71. pmid:28745013
- 11. Baker P, Brown AD, Wingrove K, Allender S, Walls H, Cullerton K, et al. Generating political commitment for ending malnutrition in all its forms: A system dynamics approach for strengthening nutrition actor networks. Obesity Reviews: an official journal of the International Association for the Study of Obesity. 2019;20 Suppl 2:30–44.
- 12. Bridgewater K, Peterson S, McDevitt J, Hemenway D, Bass J, Bothwell P, et al. A community-based systems learning approach to understanding youth violence in Boston. Prog Community Health Partnersh. 2011;5(1):67–75. pmid:21441670
- 13. Dianati K, Zimmermann N, Milner J, Muindi K, Ezeh A, Chege M, et al. Household air pollution in Nairobi's slums: A long-term policy evaluation using participatory system dynamics. The Science of the Total Environment. 2019;660:1108–34. pmid:30743908
- 14. Frerichs L, Lich KH, Funchess M, Burrell M, Cerulli C, Bedell P, et al. Applying Critical Race Theory to Group Model Building Methods to Address Community Violence. Prog Community Health Partnersh. 2016;10(3):443–59.
- 15. Gerritsen S, Renker-Darby A, Harre S, Rees D, Raroa DA, Eickstaedt M, et al. Improving low fruit and vegetable intake in children: Findings from a system dynamics, community group model building study. PloS one. 2019;14(8):e0221107. pmid:31415644
- 16. Kumar P, Chalise N, Yadama GN. Dynamics of sustained use and abandonment of clean cooking systems: study protocol for community-based system dynamics modeling. Int J Equity Health. 2016;15:70. pmid:27113743
- 17. Langellier BA, Kuhlberg JA, Ballard EA, Slesinski SC, Stankov I, Gouveia N, et al. Using community-based system dynamics modeling to understand the complex systems that influence health in cities: The SALURBAL study. Health & Place. 2019;60:102215. pmid:31586769
- 18. Trani J-F, Ballard E, Bakhshi P, Hovmand P. Community based system dynamic as an approach for understanding and acting on messy problems: a case study for global mental health intervention in Afghanistan. Confl Health. 2016;10:25.
- 19. Weeks MR, Li J, Lounsbury D, Green HD, Abbott M, Berman M, et al. Using Participatory System Dynamics Modeling to Examine the Local HIV Test and Treatment Care Continuum in Order to Reduce Community Viral Load. American Journal of Community Psychology. 2017;60(3–4):584–98. pmid:29154393
- 20. Williams F, Colditz GA, Hovmand P, Gehlert S. Combining Community-Engaged Research with Group Model Building to Address Racial Disparities in Breast Cancer Mortality and Treatment. J Health Dispar Res Pract. 2018;11(1):160–78. pmid:30701128
- 21. Richardson GP, Andersen DF. Teamwork in group model building. System Dynamics Review. 1995;11(2):113–37.
- 22. Andersen DF, Richardson GP. Scripts for group model building. System Dynamics Review. 1997;13(2):107–29.
- 23. Hovmand PS, Andersen DF, Rouwette E, Richardson GP, Rux K, Calhoun A. Group Model-Building ‘Scripts’ as a Collaborative Planning Tool. Systems Research and Behavioral Science. 2012;29(2):179–93.
- 24. Black LJ, Andersen DF. Using visual representations as boundary objects to resolve conflict in collaborative model‐building approaches. Systems Research and Behavioral Science. 2012;29(2):194–208.
- 25. Bureau UC. QuickFacts: Tarboro, North Carolina 2019. Available from: https://www.census.gov/quickfacts/tarborotownnorthcarolina.
- 26. Frerichs L, Hassmiller Lich K, Young TL, Dave G, Stith D, Corbie-Smith G. Development of a Systems Science Curriculum to Engage Rural African American Teens in Understanding and Addressing Childhood Obesity Prevention. Health Education & Behavior. 2018;45(3):423–34. pmid:28863266
- 27. Powell-Wiley TM, Wong MS, Adu-Brimpong J, Brown ST, Hertenstein DL, Zenkov E, et al. Simulating the Impact of Crime on African American Women's Physical Activity and Obesity. Obesity. 2017;25(12):2149–55. pmid:29086471
- 28. DeLong TJ. Three Questions for Effective Feedback: Harvard Business Review; 2011. Available from: https://hbr.org/2011/08/three-questions-for-effective-feedback.
- 29. Wikipedia. SKS process. 2016. Available from: https://en.wikipedia.org/wiki/SKS_process.
- 30.
Hovmand PS, Rouwette E, Andersen DF, Richardson GP, Kraus A. Scriptapedia 4.0.62013. Available from: https://en.wikibooks.org/wiki/Scriptapedia.
- 31.
Anyon Y, Brink K., Crawford M., Fernandez M., Hofstedt M., Osberg J., et al. (Eds.) Youth Engaged in Leadership and Learning: a Handbook for Program Staff, Teachers, and Community Leaders. Stanford, CA: John W. Gardner Center for Youth and Their Communities; 2007.
- 32.
Rogers J. Challenging the Process with the Marshmallow Challenge: The Leadership Challenge; n.d. Available from: https://www.leadershipchallenge.com/resource/challenging-the-process-with-the-marshmallow-challenge.aspx.
- 33. Calancie L, Anderson S, Branscomb J, Apostolico AA, Lich KH. Using Behavior Over Time Graphs to Spur Systems Thinking Among Public Health Practitioners. Preventing Chronic Disease. 2018; 15:170254. pmid:29389313
- 34. Freebairn L, Atkinson J-A, Osgood ND, Kelly PM, McDonnell G, Rychetnik L. Turning conceptual systems maps into dynamic simulation models: An Australian case study for diabetes in pregnancy. PloS one. 2019;14(6):e0218875. pmid:31247006
- 35.
Bousquet F. Companion modeling and multi-agent systems for integrated natural resource management in Asia. Manila, Philippines: International Rice Research Institute; 2005.
- 36. Campo P, Bousquet F, Villanueva T. Modelling with stakeholders within a development project. Environmental Modelling & Software. 2010;25(11):1302–21.
- 37. Barreteau O, Bousquet F, Attonaty J-M. Role-playing games for opening the black box of multi-agent systems: method and lessons of its application to Senegal River Valley irrigated systems. Journal of Artificial Societies and Social Simulation. 2001;4(2): http://jasss.soc.surrey.ac.uk/4/2/5.html.