Traditional versus participatory modelling

Systems modelling and simulation, also known as dynamic simulation modelling, is a term given to complex systems science analytic methods–such as system dynamics, Bayesian networks, and agent-based models–that is utilized in many countries and across diverse sectors to support evidence-informed decision making and to drive policy reform [1, 2]. By taking a complex systems view, significant challenges in society including population health crises, climate change, poverty, and civil strife can be better understood and managed through computer simulation models that capture the causal structure underlying the dynamics of these systems [1, 3–8]. Various systems modelling and simulation techniques have traditionally been applied across a range of disciplines including engineering, business, and environmental sciences for decades [9], but is now increasingly utilized in other fields including in public health [10–12]. This is largely attributed to the utility of systems modelling and simulation providing decision makers with both immediate and long-term support in understanding the prospective impacts of alternative strategic actions, where traditional statistical methods may be limited [13–16].

Systems modelling and simulation can provide insights at different levels of scale, including macro, meso, and micro; providing national, state, and local governments with tools that support strategic planning and decision making [4, 17–20]. By recognizing the interdependencies within complex systems, diverse stakeholder groups including those who are a part of the system being modelled are viewed as important communication agents. The involvement of stakeholders is necessary not only for their knowledge contributions but also their key role in coordinating the implementation of strategic system improvements–hence the value in shifting scientists away from working in isolation to develop systems models [21].

Participatory modelling (PM), or stakeholder-based systems modelling, brings together diverse knowledge and interests to engage in a joint learning and planning process to better understand complex systems, as well as possible implications of decision making to manage system challenges. Advances in technology and software have facilitated accessibility of modelling by a broader group of participants, allowing more diverse stakeholders working across a complex system to engage with and contribute to the development of these models [11, 22], as well as to inform decision making and further actions [23]. For example, graphical model interfaces allow stakeholders to better visualize and understand the logic and assumptions of a model than earlier software that required articulation of a model using mathematical equations or computer coding. Such accessibility has also facilitated the participation of those most impacted by policy changes (such as consumer representatives)–helping to work towards all stakeholders sharing a common understanding of a complex problem or issue, inform and enhance collective action, assist collective decision making processes, enhance both individual and social learning, as well as precipitate changes in stakeholder behaviors [9, 21, 23–29].

Evaluating participatory modelling programs—Challenges and opportunities

The inclusion of stakeholders during the PM process can facilitate learning, consensus, and transparent decision making [21]. However, PM evaluation is frequently disregarded or not based on transparent systematic methodological approaches [30]. A recent review paper of 60 randomly selected case studies on environmental PM programs reported that most studies (>60%) did not include evaluation [31]. The studies that did include evaluation were poorly reported, lacking detailed description and justifications on assessment criteria, methods of data collection, and analysis [31].

At the most basic level, evaluations provide systematic comparisons of program objectives and outcomes to understand how well something is working for the purpose of policy, planning, or implementation [32, 33]. According to the Cambridge Dictionary, evaluation is defined as the “process of judging the quality, importance, amount, or value of something” [34]. Applying this definition to the context of this paper, there is opportunity to better understand the quality, importance, and value of PMs [35]. This shifts the focus from solely one aspect of PM to a more holistic consideration of the whole PM process (e.g., knowledge integration and learning, technical systems model development, participatory and integrated planning, etc), providing opportunity for further knowledge on which aspects of PM are particularly important for policy decisions and community learning, as well as the ongoing improvement of PM methods [36, 37].

Evaluators are relied upon to address questions on the effectiveness of investments in local, state, and national programs, as well as to better explain if observed outcomes were (or were not) as planned, and how unintended consequences can be addressed [38–41]. There may be various motivations for conducting an evaluation of PM programs including the desire to improve and share knowledge on good practice for PM, quantitatively and qualitatively report on project impacts, as well as to assess the value of PM for future work [24, 35]. Evaluations also keep the modellers, funders, and other stakeholders of interest held accountable for demonstrating outcomes, as well as to provide merit to the work being evaluated [24]. Thus, PM program evaluations can also support policy makers to make evidence-informed decisions in determining how much weight to give the program or model outputs [24].

Evaluations that comprehensively capture the complex (e.g., uncertain, dynamic) nature of PM can be difficult [42], as embedding participatory approaches in systems modelling and simulation creates several challenges [35]. For instance, the focus of PM outcomes is often still on the knowledge integration and learning process rather than the multi-value perspectives integrated within the participatory process used to develop the models [31]. This can lead to evaluation practices to over- or under-represent certain stakeholder groups’ experiences [31]. Additionally, previous studies that have attempted to evaluate the benefits of PM have reported difficulty in the design of the evaluation process, as complex systems rarely have comparative controls that allow for feasible experimental design (i.e., ‘with modelling’ intervention vs ‘without modelling’ control), making the measure of PM effectiveness challenging [35]. Thus, it is important to understand the distinction between the evaluation in detecting the effectiveness of the model development process, compared to the actual success or failure of the engagement with the model itself.

PM evaluations are also typically constrained by contextual factors including limited time and program budget. PM programs are often funded to the point of delivering the final model, rather than evaluating the process and benefits of PM including the extent to which the final model informed decision making or built consensus. A lack of investment in the evaluation of PM leads to decreased motivation to conduct thorough evaluations and may also risk evaluation efforts to be overly simplified when measuring the impact of PM [24], missing the opportunity to assess the performance of PM in different contexts to inform the adaption and improvement of processes [43].

Objectives

Therefore, this scoping review aims to:

1. Identify published PM evaluation frameworks, criteria, and/or processes irrespective of the modelling method, or the discipline in which they are designed and/or implemented.
2. Assess the identified evaluation frameworks, criteria, and/or processes to understand their applicability to different PM program objectives and contexts.
3. Synthesize the findings to develop a novel evaluation framework that can be adapted and executed broadly across diverse PM programs, regardless of the discipline or modelling method. A flexible framework is necessary as PM itself requires flexibility to the potentially changing priorities and needs of participants.

A scoping review has been deemed the most appropriate approach, compared to a systematic review, as the purpose of this paper is to focus on the broad collection and discussion of available literature, and to present a comprehensive multi-scale evaluation framework for PM programs [44, 45]. Scoping reviews are better suited than systematic reviews when the aim of evidence synthesis is to provide an overview of literature and identify broad knowledge gaps in a topic that has not been extensively reviewed (as opposed to seeking to answer focused questions as is done in systematic reviews) [44, 45]. Scoping reviews also differ from non-systematic literature searches as they are routinely informed by an a priori protocol, and are conducted via a rigorous and transparent approach to minimize error as well as to ensure reproducibility [44]. The development and application of the presented evaluation framework is supported by a participatory systems modelling program for youth mental health (described below in the Discussion). To our knowledge, this is the first multidisciplinary scoping review of evaluation frameworks for PM programs.

Search strategy

A focused search was conducted via Scopus in May 2021 in consultation with an academic librarian at The University of Sydney, utilizing a combination of Boolean operators, wildcards, and truncations to develop the final search strategy (Table 1). Scopus is a meta-database, and includes records from various databases across disciplines including environmental sciences, engineering, mathematics, social sciences and medicine [36]. Additional searches were conducted through hand searching, co-author recommendations, and citation chaining.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Search strategy.

https://doi.org/10.1371/journal.pone.0266125.t001

Inclusion and exclusion criteria

The criteria for inclusion were defined a priori by the authors (GYL, LF) in a Population, Concept, Context format [46], and applied to all yielded records. As detailed in Table 2, this scoping review included sources that described an evaluation framework, criteria, and/or process for PMs. Though there are varying definitions that exist, for the purpose of this review, we have defined an evaluation framework as a tool that presents an overview of the evaluation theory, topics or themes, questions, and/or data sources; evaluation criteria as a performance metric or indicator that further breaks down the evaluation framework, and; evaluation process as a defined procedure guided by theory of how the authors recommend PM evaluation [50].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Inclusion and exclusion criteria in a population, concept, context format, as recommended by JBI.

https://doi.org/10.1371/journal.pone.0266125.t002

Records that presented a standalone theoretical framework (or applied via a case study example) were also included. In contrast, records that only described the methodological tools (e.g., interviews, etc) used to evaluate the implementation of PM programs without describing an evaluation framework, criteria, and/or process were excluded. Records that only described the evaluation of a technical model (i.e., not PM) were excluded, as were records that described PM implementation programs without any consideration of evaluations. Date limits were not set, but studies not published in English were excluded from the review.

Data extraction and synthesis

Using a pro forma approach, the first author (GYL) independently reviewed the titles and abstracts of all yielded records. Uncertainty regarding whether records met the inclusion criteria were resolved through two-weekly discussions with the senior author (LF). To verify the data extraction, a random sample of 10 records were independently checked by LF. Following this verification process, full text review and data extraction was conducted independently by GYL.

To address the first and second objective, a data extraction template was developed by the authors (GYL, LF). Data extraction templates are used in scoping reviews to provide a structured and detailed summary of each record [46], and were used to collate information on yielded records that underwent full text review. The four-dimensional framework (4P) developed by Gray et al. formed the basis of the data extraction template. The 4P framework was selected as it was developed specifically to standardize the reporting of PM programs and therefore provided a useful structure to guide data extraction [37]. This framework has since been adapted to include two additional dimensions by Freebairn–imPact and Prioritizing [13]. The definitions of the resulting six dimensions (6P) were slightly adapted to fit the evaluation objectives of this scoping review. The revised definitions are: Purpose (why PM approaches should be evaluated); Process (the method utilized to execute evaluation framework/criteria/process); Partnerships (which stakeholders were involved in the development of the evaluation framework/criteria/process); Products (evaluation approach–e.g., theoretical, conceptual, and/or implementation); imPact (what were the outcomes/strengths of the evaluation framework/criteria/process), and; Prioritizing (what were the barriers/future opportunities of the evaluation framework/criteria/process) [13]. To ensure an all-inclusive synthesis of records, the JBI template for data extraction [46], as well as additional elements included by the authors were also incorporated into the final data extraction template (Table 3). Once the author (GYL) completed full text review, the senior author (LF) reviewed and verified the final list of records to include for synthesis.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Data extraction template.

https://doi.org/10.1371/journal.pone.0266125.t003

To address the third objective, a narrative synthesis of the findings was conducted and utilized to develop an evaluation framework that can be applied across diverse disciplines and modelling methods. A narrative synthesis allows for the in-depth exploration beyond the description of the included records to understand relationships (e.g., differences, similarities) between the studies [51].

As part of the narrative synthesis, a word cloud was generated to analyze the heterogeneity in terminology identified during the data extraction process for the studies included in the scoping review [52]. Word clouds visually display the most frequently used words in a body of text with the bigger font size of a word illustrating that this word is used more frequently [53]. To ensure that a focused word cloud was generated specific to evaluation, the full text of all 11 studies included for synthesis were uploaded, which was followed by a process of elimination whereby words that were not related to evaluation–such as university, platform, and various stop words–were deleted. Following this process, the authors went through the remaining list of words and merged synonyms as well as the same words presented in its singular or plural form or with tense variation.

Though word clouds can support the identification of commonly utilized terms, there are limitations. For example, it is not clear from just the word cloud exercise alone how many terms appeared in each individual study. Additionally, some words may have different meanings depending on the field that the paper was published in (e.g., the word ‘sustainability’ may have a different meaning when used in environmental PM vs in a public health PM program). Thus, the whole author group engaged in an iterative process whereby the word cloud was utilized as a discussion tool to provide feedback, refine, and finalize the presented evaluation framework. Discussions with the authors were facilitated by GYL and LF from April to July 2021 via informal and formal meetings.

Part I: Scoping review

The initial Scopus search yielded 465 results, and an additional 10 records were identified through hand searching, co-author recommendations, and citation chaining. Most articles were excluded from review based on their titles and abstracts (n = 451), as the majority described the evaluation methods or outcomes of the implementation of a PM program without any reference to an evaluation framework, criteria, and/or process. After screening 24 full-text records, 11 studies were included for synthesis. Though it was not intentional, all included records were from academic journals, as opposed to grey literature and conference papers. The PRISMA flow diagram is presented in Fig 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. PRISMA flow diagram.

https://doi.org/10.1371/journal.pone.0266125.g001

Characteristics of studies

Table 4 summarizes the characteristics of the studies. All but one (10/11, 90.9%) were published in an environmental sciences journal. The remaining one study was published in the International Journal of Environmental Research and Public Health. Approximately half (7/11, 63.6%) of the included studies were published in the past decade (2011–2021). Overall, three quarters (346/475, 72.8%) of the 475 total yielded records reviewed by their abstracts and titles were published in the last decade.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Characteristics of included studies.

https://doi.org/10.1371/journal.pone.0266125.t004

Characteristics of evaluation frameworks, criteria, and/or processes.

The papers included for synthesis either described a theoretical evaluation framework and/or criteria with no application to a case study [24, 54]; described a theoretical evaluation framework and/or criteria applied to a case study [25, 35, 55, 58, 59], or; described an evaluation process applied to a case study [56, 57, 60, 61]. The majority of the evaluation frameworks, criteria and/or processes were developed by building upon already existing work [24, 25, 35, 55–61]. For example, Cash et al’s paper–which described how policy makers were more likely to use scientific evidence if three criteria are met (i.e., credible, salient, legitimate) [62]–was utilized in Hamilton et al. and Falconi et al’s framework [24, 35]. Rowe and Frewer’s work was cited by two different studies [63], with Maskrey et al. referencing this work to include as an evaluation criteria to assess accessibility (e.g., language) during the participatory process, whereas Zorilla et al. referenced it as an evaluation criteria to assess how the participatory process enabled stakeholder values, assumptions, and preferences to be incorporated into decision making [55, 58]. Two of the evaluation frameworks described an empirical process of how their frameworks were developed, supplemented with literature reviews [24, 54].

Key benefits and areas of future research were identified for each paper through the data extraction process (i.e., imPact and Prioritizing categories of Freebairn’s adapted 6P communications framework on reporting PM outcomes). For example, Lynam et al.’s paper was one of the first academic papers that attempted to address the gap in research evidence to support improved evaluation practices in PM [54]. It was also one of the first to identify the need to address power relations when working with communities, as well as the PM process as distinct from the technical model (e.g., encouraging co-learning/communication vs level of accuracy/precision) [54]. As such, Lynam et al. was referenced by various other papers [25, 35, 55, 58, 59], and was used by Zorilla et al. to inform the development of their own evaluation framework [55]. While Lynam et al.’s work was ground-breaking, being one of the first in this field, it was limited in providing a comprehensive description of the theoretical underpinnings of their framework.

The evaluation frameworks, criteria, and/or processes focused on different design features and levels of examination (e.g., project vs system level, short vs long-term observation, etc). For example, Jones et al., Hamilton et al., Zare et al., and Smajgl & Ward noted the importance of evaluation both ex ante and ex post to ensure all involved–including modellers and stakeholders–have a better understanding of what the PM process is aiming to achieve, and to keep everyone accountable to the defined objectives [24, 25, 57, 61]. Jones et al. and Hamilton et al. also differentiated between the levels of stakeholder participation [24, 25]; and although this was not embedded into the evaluation criteria presented by Zorilla et al., there was recognition that future work should distinguish evaluation amongst stakeholder groups from policy makers to farmers [55]. There were two papers that did not explicitly consider the multi-value perspectives integrated within PM, such as consideration of the multiple levels of examination (e.g., diverse stakeholder perspectives, project vs system level, short vs long-term observation, etc) [54, 56]. Maskrey et al., Falconi et al., and Hamilton et al. recognized that evaluations should also consider both the immediate and long-term outcomes [24, 35, 58]. Hedelin et al. and Waterlander et al. focused on the organizational level–for example, focusing on the various elements of the system to understand organizational learning, change, and action [59, 60]. This information is summarized in Table 5.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Characteristics of evaluation frameworks, criteria, and/or processes reported through individually included studies.

https://doi.org/10.1371/journal.pone.0266125.t005

A strength of some of the identified evaluation frameworks, criteria, and/or processes was that they were not only applicable to the specific PM program it was designed for, but adaptable to other PM programs [24, 25, 59]. However, for some this came at the cost of oversimplifying the evaluation framework [25, 55, 56, 61]. The strengths and limitations of each individual study are presented in Table 5. Recurring themes were synthesized from across the papers utilizing Freebairn’s adapted 6P communications framework for reporting PM programs [13]. These themes are presented in Table 6.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Recurring themes synthesized via Freebairn’s adapted six-dimensional (6P) communications framework that standardize the reporting of PM studies.

https://doi.org/10.1371/journal.pone.0266125.t006

Part II: Development of a comprehensive multi-scale evaluation framework

The strengths and limitations of each individual study presented in Table 5 as well as the recurring themes synthesized in Table 6 informed the development of a comprehensive multi-scale evaluation framework. This information was supplemented through the development of a word cloud, which allowed all authors to engage in an iterative process to refine and finalize the presented framework (Fig 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Word cloud generated by uploading the full text of all studies included for synthesis.

https://doi.org/10.1371/journal.pone.0266125.g002

The synthesis process of the scoping review revealed opportunities to develop an evaluation framework that builds off empirical evidence of existing literature. For example, to develop a framework that can be flexibly adapted for diverse PM programs regardless of the discipline or modelling method [24, 54, 59, 61], considers the whole participatory process [24, 25, 35, 54, 55], and evaluates the engagement, learning as well as the integration of stakeholders in the PM process in both the short- and long-term [24, 35, 55–59]. A flexible framework allows for a mixed-methods approach to support real-world implementation [24, 25, 35, 55–58, 60, 93], which is described in further detail below in the Discussion.

It was evident from the synthesis process that differing terminologies, approaches, and assumptions were used, which led to challenges, including PM evaluation efforts being siloed from one another. This poses risk that evaluation processes will not reach their full potential, with associated implications for funders, participating stakeholders, as well as modellers [94]. Additionally, it was evident during the synthesis process that studies either described a comprehensive evaluation framework, criteria, and/or process, or they focused on the actual evaluation methodologies; the two rarely coincided [95, 96]. Therefore, a comprehensive evaluation framework and criteria are needed for PM programs that have theoretical and empirical underpinnings but are also accompanied by practical evaluation tools and methods.

A total of 40 unique evaluation terms were identified as the most utilized across the studies included for synthesis. Process appeared the most frequently (623 times), whereas inclusive appeared the least (six times). There are limitations to word clouds, for example, they arguably provide only a superficial snapshot of themes across studies. Therefore, all authors engaged in an iterative discussion process, utilizing the word cloud presented in Fig 2 as a discussion tool to enable more in-depth exploration of the final list of terms to further analyze and incorporate into the presented PM evaluation framework. Four broad evaluation framework categories were identified, based on the key themes and limitations described above as part of the scoping review synthesis process–feasibility, value, change & action (impact) and sustainability (highlighted in yellow, Table 7). As an evaluation concept, the feasibility or plausibility of PM allows for questions to be asked regarding whether it was possible for all participants to engage and contribute throughout the PM process. Consideration of the value of the PM process allows for the exploration of questions regarding what was gained due to engaging participants in PM (e.g., learning, confidence, trust). Change & action facilitates observations of impact, including ex ante and ex post comparisons of stakeholder relationships, knowledge, and behaviors as a result of the PM process; sustainability allows for the observations of PM outcomes over time (Fig 3). For clarity, it is acknowledged that PM is conducted in a dynamic (changing) environment, and sustainability of impacts may not always be desired. Thus, by sustainability, we refer to the observation of longer-term (prolonged) outcomes of the feasibility, value, and impacts, and not necessarily that these outcomes must remain stagnant.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Comprehensive multi-scale evaluation framework for PM programs.

https://doi.org/10.1371/journal.pone.0266125.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Words extracted for development of a comprehensive multi-scale evaluation framework.

https://doi.org/10.1371/journal.pone.0266125.t007

The remaining 35 terms (highlighted in grey, Table 7) identified during the development of the word cloud have been incorporated as evaluation criteria, presented as evaluation questions in Fig 3. The word level was neither incorporated as an evaluation framework category nor criteria; but as the various levels of evaluation (e.g., multi-value perspectives) were noteworthy across studies, this term was included as a separate component in our evaluation framework–specifically, consideration of project-, individual-, group-, and system-level impacts (Fig 3).

As recognized by Jones et al. [25], Hamilton et al. [24], and Zorilla et al. [55], the consideration of multiple evaluation perspectives, or multiple levels of impact from the project-, individual-, group- to system-level is important as PM processes are increasingly becoming more inclusive to involve stakeholders from diverse backgrounds (e.g., client vs decision makers). Our framework further explores consideration of multiple evaluation perspectives, by recognizing that sublevels can exist within the individual- and group-levels (Fig 3). Consideration of the sublevels of participation enables the recognition of, for example, potential power relations and dynamics amongst the stakeholders, to be able to improve PM design and appropriately measure outcomes [24, 25, 54–56, 59]. This has been reflected in our evaluation framework, presented in Fig 3, with the individual- and group-levels further stratified to include community participants (e.g., consumer representatives) and professional participants (e.g., policy makers). As the differentiation of community and professional stakeholders is not always possible, the evaluation framework and criteria has been developed to be adapted for different PM program contexts.

Application of the presented evaluation framework

The studies included for synthesis (Tables 5 and 6) used a variety of methods to collect evaluation data. It is recommended that the presented evaluation framework adopts a mixed methods approach to align with the PM process. Examples of the potential methods include semi-structured interviews, surveys, journey maps and social network analysis (Fig 5).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Potential methods of data collection to adopt the comprehensive multi-scale framework into evaluation practice.

https://doi.org/10.1371/journal.pone.0266125.g005

As the presented evaluation framework was developed to have broader international applications across disciplines and diverse participatory modelling programs, a more thorough description of how the authors plan to deploy the framework through a mixed-methods approach is described elsewhere. This description includes the tools developed as well as the suggested evaluation time points (ex ante and ex post) in the context of national multi-site youth mental health participatory systems modelling program (Right care, first time, where you live research Program). Further details of the research Program, including our participatory modelling approach, are also described elsewhere [100–102].

In summary, this PM program will develop system dynamics models for youth mental health across eight diverse regions across Australia. The evaluation framework has been translated into online survey and semi-structured interview questions, and will be underpinned by PAR principles, as well as formative, summative, process, and impact evaluation techniques. Novel research methods such as the gamification of online surveys, will enable unique data analysis (e.g., social network analysis), supporting the exploration of diverse stakeholder experiences to improve the PM process.

Strengths, limitations and opportunities for future research

Evaluations have the potential to measure change at the project, individual, group, and system (policy) level to improve understanding of the PM process and the elements that facilitate certain outcomes (e.g., support decision making) [24, 103, 104]. In other words, knowledge that is acquired from evaluation outcomes can be applied prospectively throughout the PM process, rather than retrospectively reflecting on before-after measurements. Through principles of PAR, the proposed evaluation framework embeds continuous cycles of reflection to facilitate shared learning and iterative refinement of processes throughout PM implementation. Careful thought on design aspects are needed to ensure that evaluations are worthwhile as they require additional time, resources, and funding [105]. The presented evaluation framework considers the contributions of all participants involved in the PM process, not only the perspectives of the modellers or funder [105, 106].

The presented evaluation framework is also designed to be adaptive, flexible and iterative, to ensure that the framework remains relevant in the evolving fields and contexts in which PM are being applied [9]. PM by nature is a subjective process that is largely dependent on social interactions, human beliefs, biases and values. To ensure rigor throughout the PM evaluation process, the proposed evaluation framework builds on principles of PAR to empower stakeholders from various backgrounds (e.g., community participants to professional participants), and embeds ongoing reflection and learning so that the PM process can respond to the changing needs of complex systems to ensure that the aims of PM are met–on collaboration, learning, communication, as well as be applied across disciplines and diverse modelling methods [21]. The presented evaluation framework supports the application of a mixed-methods approach with an emphasis on approaching PM evaluations holistically [107].

There are limitations to this scoping review that should be acknowledged. The heterogeneity in terminology was a challenge during the screening, data extraction, and full text review process. Thus, with the process described in which the first author and senior author worked closely to resolve any ambiguity, a robust method was followed to ensure that the studies included are most relevant for the purposes of this scoping review. Additionally, though the described search strategy was broad in that it did not set any limits to the field of study, it was narrow so that only the studies that disclosed an evaluation framework, criteria and/or process in a PM context were included. The choice of database and search terms will also have its limitations. For instance, as our initial search was conducted in May 2021, it is possible that additional literature has been published since then that may meet the inclusion criteria for our scoping review. However, the presented evaluation framework provides a comprehensive synthesis of and builds on the expertise of existing work, adding valuable contributions in the field of PM evaluation.

Additionally, the authors acknowledge that implementation of this framework in different contexts may mean that some aspects of it may be emphasized while other aspects de-emphasized. Our framework provides a uniquely comprehensive lens and a necessary contribution to the PM evaluation literature as an attempt to encourage researchers to consider evaluations in PM programs as a standard process of scientific inquiry.