Search strategy.

We aimed to search for literature providing guidance for evaluation and evaluators (justified based on empirical evidence, other literature, or authors’ own opinions and experience) in contexts of transforming social-ecological systems towards regenerative futures, whilst also recognizing that in these contexts evaluation needs to fundamentally shift compared to how evaluation is frequently undertaken today and past evaluation conventions.

Literature searches were performed in Scopus, Google Scholar and Google (i.e., google.com) (Fig 1). Scopus was chosen over similar databases due to its combination of high overlap with other databases, high coverage of social sciences literature, and support of sophisticated Boolean and other search operators [73]. Although Google Scholar has more limited support of search operators, making it less suited to reproducible query-based searches [73], it was searched because it is the most comprehensive literature database known [73] and could therefore identify academic documents missed by Scopus, as well as moderate amounts of gray literature [74]. Google was searched mainly to obtain gray literature, of which it is an important source [75]. The search language was English, the dominant language of academia [76]. Searches in Google and Google Scholar were performed incognito in Google Chrome Version 110 (Official Build, 64-bit) with all cookies rejected, to avoid bias of search personalization. The global – i.e., no country redirect (NCR) – version of the search engines were used (https://scholar.google.com/ and https://www.google.com/). For explaining the relevance of particular evaluation approaches to transformation-focused evaluation, where information in the main literature sample was too scant, external texts located by hand-searching in Google were consulted.

Search strings (S1 Table) were created both inductively and deductively. Trial searches with initial search strings were tested, and additional terms then added based on terms used by documents found in the searches. All search strings included a term for evaluation (evaluat*). We focused on the term ‘evaluation’ for our searches because of its importance in describing the discipline of evaluation, and also its explicit reference to values, which are fundamental to reaching evaluative decisions [21,77]. We recognize that other terms may be used to describe evaluation (such as ‘assessment’, ‘monitoring’, or ‘appraisal’), and that our review may therefore omit knowledge from sources referring to evaluation only in this way (potentially more prevalent outside the main evaluation discipline). The strings also contained terms related to transformation or regenerative social-ecological systems (based on a recent review [15]).

As recommended for scoping reviews [71], we also included a ‘consultation exercise’ [71], in our case with a group of experienced evaluation and transformation researchers and practitioners, to supplement the findings of the literature review. This ‘advisory group’ was invited to suggest additional documents (academic or gray literature) not found in the database searches that offered substantively different perspectives on transformation-focused evaluation to those already identified, or presented other methodologies relevant to transformation-focused evaluation (Fig 1). They were also invited to draw on their own expertise to suggest additional guidance for transformation-focused evaluation, if this was not already evident from the literature sample and deemed vital for the effectiveness of transformation-focused evaluation. Any guidance based on the opinions of the advisory group were cited as such [78]. In practice the findings of the review were dominated by guidance from the literature sample, with only a handful of supplementary suggestions from the advisory group. Finally, the advisory group was asked to suggest case studies of real-world evaluations that illustrated the application of transformation-focused evaluation, two of which were chosen for the final paper. The advisory group also acted as co-authors in reviewing and editing the manuscript of this paper.

We acknowledge that while the search strategy includes diverse perspectives, it will also omit many voices. Academic knowledge is dominated by Western and Global North perspectives [79,80], and the cultural make-up of our advisory group means that we are unable to fully do justice to the principles and approaches of more marginalized groups, including a wide array of Indigenous and Eastern philosophies and methodologies [34,81], although some members of the advisory group had experience in working with such groups and knowledge.

Document screening for inclusion/exclusion.

Documents were then screened by S.J.B. for inclusion or exclusion in the review (Fig 1; S4 Table). Thus, no co-authors of this paper screened documents that they had themselves authored. Documents were first screened based on title (see S2 Table for full details of screening criteria) and excluded if they were clearly unrelated to desired transformation of social-ecological systems or regenerative social-ecological systems. In Google and Google Scholar, pages were screened until over half of the (ten) results on a page were excludable based on title, since this was typically an indication that relevance of results would decline significantly in subsequent pages; no further pages were screened past this point.

Remaining documents were screened based on abstract and keywords (or an equivalent section, such as a book overview or executive summary in a report). If these clearly did not suggest evaluation would be discussed in the context of desired transformation of social-ecological systems or regenerative social-ecological systems, the document was excluded.

Remaining documents had their full text screened. Reasons for exclusion at this stage included insufficient detail about evaluation (e.g., mentioned only in passing), evaluation insufficiently in the context of desired transformation of social-ecological systems or regenerative social-ecological systems, insufficient focus on how evaluation needs to fundamentally change, and documents neutrally observational of, rather than advocating, fundamental change in evaluation. Accepted documents advocated evaluation for contexts of desired transformation of social-ecological systems or regenerative social-ecological systems, and recognized that fundamental shifts were required in evaluation for it to be fit for these contexts. If a document lacked an abstract and keywords or equivalent section, it was immediately screened based on full text after the title screening. In books, if chapters had abstracts, all chapters were screened as if they were separate documents. Otherwise, the full book was screened.

Acceptable document types included journal articles, reports, books, book chapters, blogs, and formal statements. Editorials that were standalone articles were included, whereas those that primarily summarized articles in a special issue were excluded, given that these articles should in theory appear elsewhere in the literature sample if they are relevant. Documents in a language other than English were translated into English using Google Translate [82]. Documents whose full text could not be accessed were either requested from the University of York Library or the documents’ authors. Some documents in Google or Google Scholar were inaccessible because of a non-functional URL (Fig 1).

Data extraction, coding, analysis and synthesis.

From documents accepted post-screening, guidance for evaluators (and explanation of its importance) was extracted from the text (Fig 2; S4 Table). All data extraction, coding and analysis was undertaken by S.J.B., with the results critiqued by the advisory group. Inductive open-coding was used to identify initial distinct semantic themes [83], with quotes selected from the document to evidence any theme assigned (Fig 2). An iterative approach was used, whereby themes were identified as coding progressed, with earlier themes refined in light of later documents. In Mural (https://www.mural.co/), these first-level themes were then inductively grouped into second-level themes with a similar (either semantic or latent) topic (Fig 2) (for details see mural) [83]. In some cases, one first-level theme was incorporated into more than one second-level theme. A final round of thematic analysis investigated whether the second-level themes, when examined together, revealed any latent ‘meta-themes’ (Fig 2). Key components of the first-level themes were summarized based on the selected quotes, and these fed into the final text of this paper (Fig 2). The descriptions aimed to capture all unique substantive ideas associated with a theme. A saturating approach was used, such that the only themes and ideas recorded in a given document and incorporated into the analysis were those substantively different from ones already identified. This effectively treats each unique idea as equally valid regardless of how many documents mention it, rather than elevating the most commonly mentioned ideas. This was considered to help mitigate biases in the literature (e.g., towards Western perspectives).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Example of the thematic analysis used to identify principles for transformation-focused evaluation.

The example focuses on the Purpose Principles. Image created by S.J.B. in Mural (https://www.mural.co/).

https://doi.org/10.1371/journal.pstr.0000164.g002

We referred to the second-level themes as ‘principles’ because they provide broad advice and guidance and are hypotheses rather than truths: they may or may not be followed and lead to desired outcomes in a particular context [84]. The fidelity to, or effectiveness of, principles is the concern of Principles-focused Evaluation [84]. Principles can be useful ‘anchors’ in the dynamic, uncertain and complex environments that characterize transformations in social-ecological systems, and are sufficiently flexible to guide action even when contexts vary across space and time whereas more rigid, prescriptive criteria would risk losing relevance [84,85]. We maintained the advocative voice of the literature analyzed, e.g., wording the principles as ‘evaluators should…’ or ‘evaluators need to…’. The advisory group was asked to identify complementarities and tensions between the principles.

To identify methodologies for transformation-focused evaluation, we identified any specific named methodologies advocated or applied in accepted documents, with a clear link to transformation-focused evaluation. Under the broad umbrella of ‘methodologies’ we included any overarching evaluation approaches, complex system concepts and frameworks, theories of transformational change, regenerative system frameworks and tools, visioning methods, indicator frameworks, frameworks of evaluation criteria, and analysis methods (many of which imply a method of data collection). We interpreted overarching evaluation approaches as typically not prescribing specific evaluation methodologies, but rather as conceptual analytical perspectives based on principles which then have implications for the kinds of specific methodologies that might be relevant.

As with the principles, our interest was in uncovering the range of different methodologies advocated rather than understanding how many documents advocated a particular approach, so as coding progressed only methodologies not previously advocated were recorded. The relevance of any methodology to transformation-focused evaluation, and to particular principles of transformation-focused evaluation, was determined based on quotes from the document in the main literature sample, other literature, and/or opinions from the advisory group. We wish to emphasize that this is a quick and rudimentary form of Principles-focused Evaluation rather than an in-depth analysis.

1. Complex systems principle: Treat evaluands as nested systems with complex interactions.

Evaluators should treat an evaluated system as having complex, interdependent relations with the surrounding systems it is nested within and interacts with during transformation processes [18,57].

This principle brings evaluation closer to the physical reality of systemic transformations [18]. As highlighted by global crises such as the COVID-19 pandemic, we live in an increasingly globalized and interconnected world, underscoring the need to remain aware of the interlinkages of system changes [56]. There are several key implications of this worldview for evaluation practice. Evaluators must be able to think and work integratively across spatial and temporal scales [113], taking into account ‘macro-issues’ rather than focusing solely on ‘micro-issues’ [57]. For instance, evaluators might adopt a ‘multi-level perspective’ that looks at niche, regime and landscape levels [134], a ‘glocal’ perspective that considers linkages between local and global systems [27], and consider how specific programs and projects contribute to broader strategies and policy [57]. Evaluators must also recognize that while some changes take place rapidly, others (particularly those at wider systemic scales) may take decades [78,113]. The Developmental and Foresight Principles detail further implications for evaluators regarding futures and timescales.

Evaluators must also understand how initiatives and interventions, and their effects, are influenced via complex interactions with their contexts [18]. This requires a more theory-driven approach of understanding mechanisms behind observed outcomes, taking context into account and mapping relationships and interdependencies between actors [43,123]. Examining a pre-existing system before intervening would include understanding its parts and their interconnections, the system as a whole, and the system’s dynamics over time, including its response to other interventions [102]. In contrast, evaluations with a less complexity-aware ‘project mentality’ may treat evaluated interventions ‘as if they existed in a vacuum’ [135], work from a positivist worldview grounded in Newtonian assumptions that favor predictability, replicability, and linear causal relationships between initiatives and measured outcomes [123,136], and adopt the general Western tendency for reductionism rather than appreciating interconnections [121]. Transformations can rarely be commanded and controlled in such linear and predictable ways [99], and innovation is messy and iterative rather than a smooth orderly process [137]. Moreover, many evaluations still focus on planned activities and intended outcomes and impacts rather than more emergent results [105], and although many give due consideration to side-effects on human systems, they neglect impacts on environmental systems, which means that evaluation can inadvertently contribute to the ongoing destruction of natural systems rather than support their protection and recovery [105].

Evaluators should instead bring into view the full range of social, economic and ecological impacts of an initiative beyond its immediate sphere, their interactions, and their modification by external forces [4,134,135,138] – looking not only at how interventions affect targeted feedback loops, but also how these then influence other feedback loops, subsystems, and the system as a whole [102], considering costs, benefits, spillover effects, trade-offs, synergies and dissergies [4,139]. System transformation is often influenced by contextual forces, such as the price of oil influencing renewable energy investment [113], or an initiative might start out as radical but because of environmental changes becomes more mainstream [18]. Other signs of complex system interactions that evaluators should be prepared for include outcomes that are emergent and unanticipated [18,67,99,139], self-organization [139], chaotic processes (where system outcomes are sensitive to initial conditions) [140], feedbacks [138], attractors (patterns that a system is ‘drawn’ to being in) [140], stigmergy (coordinated behavior of independent actors because of the shared environmental context) [140], rebound effects [113], path dependency [141], lock-in effects and system inertia [141,142], and tipping points [89]. A complex systems perspective of power relations may reveal apparently negative outcomes, like backlash from the establishment, to in fact be signs of program success [100]. The uncertainty inherent in complex systems also highlights the importance of mixed methods (see Detective Principle), and the need for constant adaptation and rapid feedback in evaluations [136] (see Developmental Principle).

The Complex Systems Principle has implications for chosen indicators and indicator frameworks. These should be holistic, assessing the multiple dimensions of sustainability in an integrated rather than siloed way [56,143]. For example: food system evaluators should recognize the cultural and social dimensions of how food links to health alongside nutritional dimensions [144]; the evaluation of regenerative buildings should explicitly recognize interconnections between human and ecological systems rather than just listing a limited set of performance requirements [145]; and more broadly, evaluations should assess how well integrated transformative initiatives are across sectors and silos, and the interactions between different transformative initiatives [27] (see Partnerships Principle). Reductionist and fragmented approaches to indicators, e.g., traditional pillars of sustainability (environmental, social, economic) – especially if they focus on easily quantified metrics – may omit important (especially qualitative) aspects, like justice, values, paradigms, worldviews, and human-nature connections [146–148]. Evaluators may need to incorporate harder-to-measure compound indicators to capture systemic impacts [149]. Indicators should in any case be developed and altered dynamically, especially post-intervention when the system is better understood [102] (see also Developmental Principle).

Operationalizing this principle means moving away from reliance on linear logic models and key progress indicators, SMART goals, randomized controlled designs, and standardized, replicable, scalable models that are geared towards evaluating isolated projects and programs in stable environments rather than complex systems [27,123,150]. Evaluators should instead become fluent in systems thinking and complexity science [64,140]. Many concepts and frameworks can encourage systemic perspectives, including boundary critique [106], complexity theory [18], critical systems thinking/heuristics [106], living systems theory [147], relational systems thinking [106], complex adaptive systems [102,147], socio-technical systems [113], social-ecological systems [89], value-creating systems [78,151], and structuration theory [152]. Indicator frameworks from regenerative design and development fields could also help to consider system contexts in a more holistic way [153] (see Transformation Principle). Various analysis methods can then help to visualize, explore or otherwise experience the particular systems in question, such as causal loop diagramming [154], Participatory Impact Pathways Analysis [155], social network analysis [154], agent-based modeling [18], soft systems methodology [154], the World Mandala [78,156,157], Digital Twins [139], digital bioregional story maps [106], systems sensing [26] and systemic constellation [26]. These system concepts and tools can be used to develop more complexity-appropriate theories of change [26], although more traditional positivist approaches can also help in clarifying assumptions in theories of change [36]. Evaluators should also familiarize themselves with methods for conceptualizing and analyzing transformations and transitions in complex systems (see Transformation Principle).

Numerous overarching evaluation approaches are based on complexity and systems thinking principles, including Blue Marble, Design-driven, Developmental, Dynamic, Feminist, Footprint, Systemic, and Visionary Evaluation, amongst many others [67,113,123,91,92,94,107] (Table 2). Footprint Evaluation (aka Sustainability-inclusive Evaluation) provides guidance for evaluations to embed criteria of environmental sustainability and contribute to the protection and recovery of natural systems, encouraging appreciation of the interconnections between human and natural systems [105]. Reflexive Evaluation builds on the multi-level perspective that explains how interactions between niches, socio-technological regimes and landscape may lead to system change [53,125]. Indigenous Evaluation approaches are commonly based on more holistic and relational worldviews [33], where ‘worlds are made up of extended networks of kin to whom we have reciprocal responsibilities’ [109], whilst many Eastern philosophies still in use today also carry a deep appreciation of complexity [34]. The African Evaluation Principles are one of the first and few evaluation association principle sets to so explicitly embed a complex systems approach [88]. Other evaluation approaches are not necessarily specified for complex systems but are nonetheless valuable in such contexts; for instance, Theory-based Evaluation [158] (e.g., Realist Evaluation [18,41]) works from endogenously generated theories of change to unpack the mechanisms behind expected or observed impacts [158] (Table 2).

Capturing all outcomes (both positive and negative) of system interventions – the general principle of ‘true cost accounting’ [56,159] – would be supported by methods such as outcome harvesting/mapping and ripple effects mapping [65] (Box 1). These encourage brainstorming of observed outcomes and impacts regardless of whether they were expected. A form of Adversary Evaluation has additionally been advocated for Dynamic Evaluations in weighing up both positive and negative impacts of system interventions [57].

Part of an evaluator’s role is also to help evaluation users to unpack and navigate complexity, to facilitate understanding and agency [4]. Frameworks such as Three Horizons and the prioritization process in the Most Significant Change method, which are underpinned by (inductive) thematic analysis, can assist in this task [160–162]; other recommendations include for evaluators to report specific impact stories alongside big-picture considerations [4], and orient to principles, as in Principles-focused Evaluation [163] (see Developmental Principle).

2. Context principle: Recognize that evaluands are unique and need context-adapted evaluation.

Evaluators should recognize that every evaluand and its context, culture and transformation have unique characteristics, necessitating context-adapted rather than one-size-fits-all evaluation approaches [4,57,136,140].

Transformation is inevitably context-specific [136]. This is a shift away from Newtonian assumptions of replicability that have characterized many evaluations [136]. As such, evaluators should be wary of using standardized methods and criteria without thought for their appropriateness to the situation [4]. They should instead be ‘bricoleurs’ [56], choosing among the diverse array of available approaches and tools (potentially exploring new and unfamiliar techniques) to select and creatively adapt those best-suited to their particular context [57,136,140] (Box 2). The urgency of societal transformation (see Transformation Principle) means that evaluators may have to rapidly develop credible methods for the situation in hand in limited timeframes (as well as with limited budget and expertise) [67]. Ideally, these methods would be co-developed with the evaluation’s intended users, treating them as authorities on whether and how the methods should be applied (see Power Shift Principle) [4]. This principle is important for ensuring that the effectiveness of evaluations is optimized for particular situations of transformation, and for honoring and celebrating evaluands’ unique qualities [78]. This is likely to make the evaluation more useful to its intended users [131].

This principle requires evaluators and stakeholders to build up a rich understanding of their particular locale, culture and context [123,165]. Evaluators may need to investigate the cultural, political and economic history of a place (e.g., visualized on a timeline of key events), as well as its norms and values [123,165] (see Values Principle). The shifting and contextual nature of LGBTQ+ identity is a good example of how evaluation should not rely on rigid standardized methods but remain more flexible and responsive [111].

The Context Principle has implications for selecting indicators of success. Transformation-focused evaluation has been contrasted against evaluations with market-driven, homogenized approaches that impose uniformity on a multicultural reality [166], and use prescriptive, one-size-fits-all indicators [167,168]. For instance, there has been criticism of education evaluations with a technical-positivist focus on students attaining set goals, and relatively superficial psychometric measurement of learning [169]. Recognizing evaluands as unique instead suggests that indicators should be collaboratively developed and tailored to particular places so that they are meaningful and culturally appropriate [167,168,170], or that evaluators should identify ‘best principles in practice’ rather than looking for adherence to predefined specific ‘best practice’ [123], and give a ‘tentative best judgment’ rather than recommending ‘best practice’ when identifying or extrapolating insights from evaluation results [171]. In education contexts, evaluation would encourage the development of more reflexive, conscientious, autonomous and democratically active citizens, appreciating that students cannot be slotted into predefined boxes [166,169].

Operationalizing this principle requires awareness of the many approaches, frameworks and tools outlined in the other principles. Furthermore, Blue Marble Evaluation makes bricolage explicit as one of its operating principles, whilst many approaches emphasize the importance of designing evaluations to fit their particular contexts (Table 2). Various methods have been advocated for deepening understanding of specific contexts, including examination of publicly available data (e.g., on demographics), media and social media analysis, interviews of people with long-term experience, and design labs [123]. To support stakeholders in exploring their context, building on strengths and focusing on what people already love and are motivated to protect – using approaches including asset-based community development, appreciative inquiry and positive psychology – could be more successful in engaging people in dialogue and encouraging sustained collective action than problem-focused approaches [172].

3. Detective principle: Be a mixed-methods detective in an uncertain, multi-reality and hard-to-control world.

Given the uncertainty and pluralist lived realities in complex systems change, evaluators should draw on multiple, mixed methods to improve understanding about transformations and how interventions contribute to them [102], accepting that there are many equally valid ways of knowing [102], clear-cut conclusions about causality are usually unattainable [62,123,135,136] and most large-scale change is outside the control of individual initiatives [78].

Evaluators should be prepared to seek narratives, correlations, patterns and explanations rather than well-defined causal connections in complex systems change [18,136], and maximize opportunities to understand the system and enhance its learning capacity [123] through using a wide variety of tools, drawing on both qualitative and quantitative data sources [62,135], and improving access to information [123]. Evaluators may even have to be guided by hunches and principles in place of hard data about impact in systems [67]. In this way, conducting a transformation-focused evaluation is similar to being a detective or a court judge weighing up multiple sources of evidence and possible scenarios of change, and using abductive reasoning to identify the most likely chain of events [78].

This principle is important because available information about a complex system is never perfect or ideal [62], with clear causal relationships (e.g., between an intervention in a complex system and the system’s outcomes) and experimentally controlled counterfactuals rarely possible to establish [18,67]. Changes in one system may trigger changes in other systems, leading to transformations that seem distant in space and time from original interventions [174]. Moreover, a system or transformation is never a single reality, given that all actors experience it differently, highlighting the need for epistemological pluralism in evaluations [102]. Mixed methods are also important for engaging appropriately with diverse cultural groups in evaluations [175] (see Justice Principle).

A key shift in this principle is from the language of ‘attribution’ to ‘contribution’, which is more transparent about uncertainty [146]. Evaluators will need to consider multiple contributions from different initiatives and forces rather than attributions to single interventions [64], and assess whether a given contribution is aligning or cooperating with (or at least not obstructing) other efforts, which is important for enabling wider system change [78] (see Partnerships and Transformation Principles). Rather than attempting to causally link an initiative to specific outputs and outcomes, the initiative may also have to shift focus to its influences on wider system conditions and the directionality or trajectory of system change [64,86]. The difficulty of linking interventions to impacts also highlights the importance of examining the designs of strategies, policies, etc. prior to implementation to assess their likelihood of achieving desired impacts (e.g., whether they are aligned to transformation principles) [78] (Box 3).

Adversary Evaluation is the clearest embodiment of the ‘detective’ or ‘judicial’ approach to evaluations [87]. However, this principle has implications for data collection and analysis in evaluations more generally. Real-time data sources (e.g., Big Data) may be key, especially for more developmental evaluation approaches with rapid feedback [139] (see Developmental Principle), in addition to literature reviews, surveys, stakeholder opinions, modeling results, digital storytelling, blogs, social media/network analysis, etc. [4,123]. Combining more interpretivist epistemology and associated qualitative and relatively subjective data (e.g., from focus groups and open-ended interviews), with more positivist epistemology and associated quantitative and relatively objective data (e.g., quantitative indicators and systems modeling), could help to gain a more comprehensive picture of transitions and the diverse perspectives, needs and values of people in the system [102,142,176], whilst information from futures and foresight methods (e.g., scenarios) can complement information from the past and present [177] (see Foresight Principle). It is also important for evaluators to ‘catch the wave’ of recent advances in Artificial Intelligence, Machine Learning and Digital Twins for enriching system understanding [139]. In general, while quantitative modeling may be valuable, evaluators should bear in mind that its role is not so much to find ‘truths’, but more to facilitate the ‘probing’ of the system [134].

Various analysis methods enable structured exploration of the influence of interventions on systemic change in an abductive way, including contribution analysis [18] (Box 4), general elimination method [65], ripple effects mapping [164] (Box 1), and process tracing [18,65]. Comparative case studies and qualitative comparative analysis provide additional analysis methods for assessing complex forms of causation in situations with limited and imperfect data [65]. The Most Significant Change method [65], meanwhile, adopts the approach of collecting personal stories about the changes people have experienced, which can enhance impact evaluations [161] (Box 5). In general, evaluators may have to rely more on grounded-theory or inductive analysis rather than hypothesis-deductive approaches [78].

4. Developmental principle: Remain agile and developmental.

Evaluators should avoid treating evaluands as static, and instead recognize the dynamic, unsettled nature of transformations and therefore the need for continual reflection, feedback, experimentation, adaptation and development [99,141,101].

This principle is in contrast to evaluations based on a linear process of setting predetermined objectives, implementing them, and then summatively evaluating the impact of an evaluand (a project, program, initiative, etc.) in light of those objectives [178]. Indeed, the principle represents a fundamentally different perspective on sustainability that recognizes the need for resilience and continual adaptation, rather than the linear continuation of positive results [27]. It is also a shift from aiming towards fixed, ideal states (which may not exist), instead focusing more on evaluating an ongoing process with emergent outcomes [26,147].

Rather, evaluations should be rapid, agile and adaptive [146], use a monitoring system with real-time feedback [102], and avoid setting evaluation logic (e.g., criteria, approaches and theories of change) in stone from the outset, but rather allow it to evolve over time [18,99]. For instance, theories of change may need frequent revision [18]. Evaluations may also need to expand their time horizons to start seeing meaningful effects and system ‘tipping points’ resulting from interventions [134,102], and ensure capacity for post-implementation evaluation when designing initiatives [179]; in many evaluations, questions about longer-term impact and sustainability are only briefly explored or dismissed as outside the evaluation’s timeframe [24]. Similarly, a shift may be required from assessing outputs, cost-effectiveness, efficacy and impact, to assessing an initiative’s capacity to deal with ongoing global challenges, including reflection and learning processes that can contribute to longer-lasting changes [94] (see Values and Learning Principles). These shifts clash with what many people expect from an evaluation, which could otherwise result in forced premature adoption of models not because they are appropriate, but because evaluators, funders or other stakeholders demand them to fit their preconceived notions of what good evaluation is [180]. Funders will need to recognize the need for more developmental and agile evaluation, with potentially longer funding cycles, rather than requiring identifiable outcomes and concrete strategies from the outset [136,163].

In developmental evaluations it is more empowering to evaluands to have a hopeful and solutions-oriented approach [181] (e.g., appreciative inquiry [162]) that focuses on their positives, strengths, competence and potential, and how these can be developed further, rather than problems, weaknesses, and punishment for poor practice, which can reinforce power inequalities and marginalization [101,182] (see Context Principle). While risks, disappointments, failures and weaknesses should not be ignored, they should be acknowledged as valuable opportunities for learning [62,182] (see Learning Principle).

This principle is important for enabling adaptation to dynamically changing conditions [183]. In reality there is rarely a position of stability from which to retrospectively evaluate a project [18], and systems may revert back to old patterns after an intervention, highlighting the need for ongoing monitoring, trajectory analysis, and a focus on long-term results [56,67,136]. Experimental approaches that require more fixed conditions are rarely feasible in complex systems [18]. Demanding specificity and measurability up front, as in linear logic models, can actually be damaging in complex system contexts because it constrains exploration, experimentation and adaptation [180]. Furthermore, the urgency of addressing global crises (see Transformation Principle) means that there is insufficient time to go through traditional sequences of ‘in-depth situation analysis, comprehensive needs assessment, planning, design, implementation and evaluation’ [56]. Instead, these activities must all happen simultaneously, continually and rapidly [56].

Evaluation approaches such as Developmental Evaluation and Reflexive Evaluation take this principle to heart, supporting more rapid, continual feedback for innovation initiatives adapting to dynamic conditions and moving away from the traditional cycles of formative and summative evaluation that are typically ill-suited to such instability [53,85,89] (Table 2). Methods such as rapid cycle testing [65], snapshot surveys [123], the What? So What? Now What? framework [184], actual-ideal comparison [184], Most Significant Change [184] (Box 5), outcome mapping [184] and Williams’ pattern-spotting framework for qualitative data analysis [185] may also assist rapid, dynamic, developmental evaluation approaches. These methods may be complemented by data science technologies such as Big Data to support more real-time and quasi-real-time evaluations [57]. Principles, which underpin Principles-focused Evaluation, can also be vital ‘anchors’ in environments of dynamism, uncertainty and complexity, and are sufficiently flexible to guide action even when contexts vary across space and time [84] (Box 3). The GUIDE framework offers recommendations for formulating high-quality principles that are Guiding, Useful, Inspiring, Developmental and Evaluable [84]. Rapid feedback or learning memos, debriefs, critical incident reviews, and after-action reviews, meanwhile, are useful methods for sharing emerging insights [123], which may require shorter, more user-friendly report formats [179]. In general, a ‘learning by doing’ approach is advocated, using pilot projects, living labs, policy labs and hackathons to encourage experimentation [141,94].

5. Foresight principle: Include foresight alongside retrospective evidence.

Evaluators should become future-sensitive, using and evaluating information from futures and foresight methods alongside evidence from the past and present to guide transformative action [177].

Many evaluations use a retrospective approach – investigating past events to generate recommendations for future action – with the assumption that what works (or not) in the past will work (or not) in the future [177]. However, in VUCA contexts (characterized by volatility, uncertainty, complexity and ambiguity), and to support transformational change, evaluations also need to draw on futures and foresight methods, integrate future-sensitivity into their terms of reference, and judge interventions against the future [177], rather than relying on generalizations from hindsight [4]. For example, a more future-oriented formulation of the evaluation question ‘To what extent did the intervention respond to beneficiaries’ needs as defined in the program documents?’ would be ‘To what extent is the program geared to ensuring that it meets the needs of the stakeholders in the next ten years, even if circumstances change?’ [177]. Without such future-sensitivity, there is a danger that initiatives will be continually caught out by changing contexts, fail to sustain desirable outcomes, and fall back on marginal, incremental change rather than systemic transformation [113,177].

An important role for evaluators is then to expand foresight evaluation capacity [4], extend attention to mid- and long-term futures [177], and support stakeholders in: envisioning desired futures and innovations [113,177]; exploring possible future scenarios, global shifts and megatrends, including how current paths of action will respond to wider system dynamics [78]; and questioning assumptions about the future (see also Values Principle) [177]. At the least, a simple exercise would be to identify existing strategic plans of relevant organizations that interact with the intervention at hand, or major milestones that are due to occur, such as Agenda 2030 expiring in 2030 [177]. To encourage transformational change, it is important for people to envision radically different desired futures [167,170,177]. These futures can then inform theories of change (including expected times for outcomes to occur), and generate rich, contextually adapted and collaboratively developed indicators of change [170,177]. Evaluators might also evaluate the quality (e.g., transformation-orientation) of envisioned futures [63] (see Transformation Principle), and initiatives’ ‘futures literacy’, future preparedness and resilience, making recommendations for how these could be improved [177].

Future-sensitivity is embedded in a number of overarching evaluation approaches, including Culturally Responsive Indigenous Evaluation (CRIE), Gates et al.‘s ethics of evaluation for socio-ecological transformation, Mickwitz et al.’s theory-based framework for evaluating transitions, the Orders of Outcomes framework, Patton’s Evaluation Criteria for Evaluating Transformation, Patton & Felcis’ Principles for Evaluating Transformation, Reflexive Evaluation and Reflexive Monitoring in Action (RMA), Sitra’s Evaluation Framework, and Blue Marble, Design-driven, Feminist, Indigenous Feminist, and Visionary Evaluation (Table 2). This principle also has an affinity to Theory-based Evaluation, in the sense of futures methods results being used to enrich theories of change [113,177]. Various methods and tools help to structure the envisioning or exploration of desired or possible futures. Methods include horizon scanning, scenario building (exemplified by the work of the Intergovernmental Panel on Climate Change [186]), Delphi analysis, backcasting (working backwards from an envisioned future to understand the necessary intermediate actions), wind-tunneling (testing how existing strategies would fare in different futures), and methods for describing the future as if it exists today (e.g., imagining future newspaper headlines) [177]. Risk analysis, Bayesian statistics, and power-law mathematics can also assist prospective assessment of the effectiveness of alternative strategies prior to any implementation [134,187]. Other futures tools and frameworks include Three Horizons [160], Seeds of Good Anthropocenes [188], the Regenerative Lens [15], anticipatory action learning [189], the Value Proposition Toolkit [64], the ParEvo tool [177], and Futures Wheel [177].

6. Justice principle: Promote justice in power relations, rights, and use of knowledge.

Evaluators should promote justice as a key component of transformations, including through addressing power inequalities, upholding human and non-human rights, decolonizing evaluation, and honoring diverse and marginalized perspectives [136,89,100,106,108,190].

Evaluators can and should be ‘activists’ for democracy and justice [100,108], including: overcoming structural injustices in knowledge systems [191]; enabling self-determination of Indigenous peoples and upholding territorial rights [106]; supporting LGBTQ+ health and equity [111], intersectional justice [111] and human rights in general [175]; and promoting the rights of Nature and other sentient beings, avoiding anthropocentrism [89,192]. As well as applying this principle in stakeholder engagement, evaluators should see evaluation results as opportunities to lobby for justice [136], and local voluntary organizations for professional evaluators could support advocacy and social action [171]. This is a fundamental shift from the more neutral, independent stance traditionally adopted by many evaluators [4].

This principle requires that evaluators become aware of power relations, cultural differences, histories and underlying ontologies within the communities and societies of interest, including how these uphold the status quo and whether unjust power structures have been disrupted [136,190]. It also requires evaluator reflexivity (see Values Principle) to recognize the power implications of any evaluation approaches advocated and what forms of knowledge they treat as valid [111]. This is particularly pertinent to the Western evaluation field, its paradigms and its history. For instance, conventional Western evaluation paradigms are founded on historical violence against Indigenous communities, with evaluations performed in extractive ways without participants’ knowledge or consent, and often used to validate Western scientific methods whilst denigrating Indigenous ways of knowing [109]. Even supposedly neutral evaluation criteria may actually impose White values and knowledge [193], and have often been used as an excuse to ignore and perpetuate violence against Indigenous peoples [109]. Modern algorithms and technologies may be inherently racist in their formulation [194]. Western evaluation has also been seen to grow from White patriarchal logics [136].

Conversely, an important aspect of justice-oriented evaluations is to honor a diversity of perspectives and ways of knowing and evaluating [106,190], giving a voice to the most discriminated and marginalized groups of people (e.g., considering diversity of race, gender and ability) [113,190] and learning particularly from Indigenous, Global South and Eastern philosophies [106,194]. This is a particular shift from the domination of evaluation approaches from the West and Global North [194], and acknowledges, contrary to many Western beliefs, that there is no absolute, objective scientific truth that defines what appropriate evaluation is [111]. For instance, it may be important to: elevate unspoken felt-knowledge and embodied experience as a valid form of knowledge and truth [109]; include place-based ecological knowledge and ancestral wisdom communicated via dreams or intergenerational traditions [106]; ensure meaningful participation of youth [195], LGBTQ+ people [111] and those in poverty [191]; and include views of laypeople or those with on-the-ground experience of real-world problems alongside other ‘expert’ opinion [172,191]. This principle therefore requires evaluators to work in multi-actor, multi-disciplinary teams [113,187], remaining open to new ideas, willing to learn and continuously identifying opportunities for enhancing their own learning [111,101] (see Values and Learning Principles). There have additionally been calls for more South-South cooperation in evaluations to resist the dominance of Global North evaluation approaches [34].

Ethical dilemmas are likely to arise in justice-oriented evaluations, and evaluators should be transparent about these and how they were addressed [136], potentially using Dilemma Resolution techniques [78,196]. More appropriate evaluation questions than ‘Does the intervention work?’ would accordingly be ‘What works for whom, in what ways, under what conditions and with what results?’, because these questions recognize that one intervention will not work for everybody [56]. Indeed, dilemmas and contradictions can become valuable sources of transformative insight and learning if approached with the right attitude [165]. By remaining sensitive, reflexive and transparent in these ways helps evaluators to develop trusting relationships with stakeholders to unearth their perspectives and experiences [129] (see Values Principle).

This principle is important because failing to attend to local context and its power relations, and imposing externally developed evaluation approaches, can simply reproduce problematic power inequalities that hinder transformation [136]. Embracing this principle instead deepens appreciation of systemic complexity and interdependence [135,181], enables more meaningful judgements of the outcomes and impacts of transformation [67,102], encourages greater resilience through the ‘requisite variety’ in evaluation design and processes [134,106], and can generate more powerful learning between people with different perspectives [182]. As a particular example, including voices of young people is important because they tend to have more transformative values (e.g., fighting for social and environmental justice, peace, etc.), are good at building networks and alliances, have strong communication and technological savviness and connectivity, and offer fresh perspectives [195]. The Justice Principle is also an ethical imperative in itself [78].

Operationalizing this principle means applying evaluation approaches that are sensitive to cultural context, inequalities (including those related to race, gender, socio-economic status, sexual orientation, species, etc.) and conflict [34,89,111,122]. Transformative Evaluation has an explicit focus on promoting social justice, challenging power hierarchies in society and working in an inclusive and empowering way with minority and marginalized communities [54]. The Navigating the Intersection of Culture and Evaluation (NICE) Framework, Gates et al.’s ethics of evaluation for socio-ecological transformation, Prague Declaration, and Transformative Empowerment Evaluation, Transformative Participatory Evaluation, Culturally Responsive Evaluation, Equitable Evaluation, Equity-focused Developmental Evaluation, and Inclusive Evaluation, adopt similar stances (Table 2).

Approaches addressing more specific arenas of (in)justice include: Indigenous Evaluation and the Nation-to-Nation (N2N) Systems Evaluation Framework, which focus on elevating and empowering the knowledge and cultures of Indigenous peoples and First/Tribal Nations (Box 6); Feminist Evaluation, focused on advancing gender-based justice; LGBTQ+ Evaluation, which promotes LGBTQ+ inclusion and liberation; Aotearoa/New Zealand’s evaluation guidelines, which explicitly reference care toward the environment; and Made in Africa Evaluation, which aims to decolonize evaluation in Africa and support locally developed African evaluation approaches that promote African livelihoods and values (Table 2). The African Evaluation Principles similarly call on evaluations to empower Africans, address inequalities and power asymmetries, and embrace a diversity of perspectives particularly from the Global South and Indigenous cultures [88].

7. Power Shift Principle: Shift power towards evaluation users and blur the evaluator-evaluand dichotomy.

Evaluators should aim to be guides rather than directors or controllers of transformation-focused evaluations: evaluations should thus be genuinely participatory and empowering for evaluation users [101], allowing them to take greater ownership, leadership and responsibility of designing and delivering evaluations, and acting on their findings [136,121,130].

This principle sees participation as going beyond tokenistic consultation and instead involving a genuine transfer of power towards users of evaluations [101,130]. These stakeholders are not simply sources of data [136] and evaluators not mere data collectors [162]. Rather, users should be involved in deliberative discussions throughout the process of designing, conducting and drawing conclusions and recommendations from evaluations [122,197], and trusted as ultimate authorities on what is appropriate for their context, including the purpose of evaluation [122], evaluation questions [122], which data to collect [136], and what criteria and indicators to evaluate against [89]. Moreover, evaluation outputs (e.g., reports) should be disseminated to stakeholders with the agency to act on the findings, and not only to funders [35]. Indeed, evaluation data should be treated as autonomously owned by evaluated communities [111].

This principle signifies a fundamental shift in the role evaluators play [106], and blurs the traditional dichotomy between evaluator and evaluand [78]. Transformation-focused evaluators might be better described as participant observers [130], facilitators [136], change agents [4], ‘active design team member[s], fully participating in decisions’ [179], critical friends [179], or partners with other stakeholders [121] – rather than primary experts imposing their own view of reality and evaluation approaches on communities without consultation [101,130], and conducting top-down evaluations that externally judge evaluands without providing opportunities for development and reflection [198]. Rather, they are guided by critique and deliberation with evaluation users [106], and listen to voices that disagree with the perspectives of the evaluator or funder [179]. Evaluators may not even be the primary evaluators, but rather supporting communities to critically evaluate their own progress [121], building capacity for evaluative thinking and evaluation utilization in evaluated systems [171]. Evaluators may require training in developing these kinds of facilitation, communication and interpersonal skills (including conflict resolution), with which evaluators can help communities access their untapped collective wisdom and creativity, and nurture conditions of trust and inclusivity where insights can be safely and willingly shared [172,176]. Overall, however, evaluators may have to move fluidly between more ‘internal’ roles (invested in the system) and ‘external’ roles (independent from the system), and remain able to adopt the stance of a more external, independent, critical expert if this is considered most appropriate for the case in hand [100], such as where there is a risk of organizations cherry-picking results or greenwashing [78] (see Autonomous Evaluation and Values Principles).

Examples of this role reconfiguration include genuine youth-participatory evaluation, in which evaluation is not something ‘done to’ young people but ‘with or by’ young people, potentially with young people as co-evaluators or directors of evaluation and adult evaluators as coaches or facilitators [195]. In education, there have been calls for greater participation of students in self-assessment, self-reflection and peer assessment rather than didactic teacher-led approaches, disrupting the traditional power asymmetry [169].

This principle is important for a number of reasons. Overall, it embodies the kinds of ethically imperative power shifts sought in the transformation of wider social-ecological systems [190] (see Justice Principle). Shifting power towards the evaluand can deepen participants’ understanding of evaluation [136], and by building evaluations around dialogue in community settings, people can learn from each other, become more aware of social forces influencing their beliefs and behavior, and learn the importance of giving others feedback [182] (see Learning Principle). More technocratic evaluations – in which the methodology, questions and analysis are ultimately designed and imposed by the expert evaluator research team – may lead evaluators to ask the wrong questions and overlook important aspects, such as political dimensions [191]. Adopting this principle would also encourage commitment to act on evaluation results [136] and increase the likelihood that transformation-focused evaluation would become an integral, self-sustaining part of working cultures rather than a periodic externally applied exercise [101] (see Learning Principle).

Operationalizing this principle requires evaluators to build trust-based relationships with stakeholders in culturally appropriate ways that are context-aware [136], acknowledge and challenge power imbalances [190] (e.g., incorporating voices of the most marginalized [199] – see Justice Principle), and encourage deliberation about different values [89] (see Values Principle). A wide variety of evaluation approaches are built around greater participation and empowerment: as well as all those listed under the Justice Principle, they include Adversary, Collaborative, Deliberative, Democratic, Design-driven, Developmental, Participatory, Real-time and Reflexive Evaluation (Table 2). Empowerment Evaluation aims to enhance communities’ capacity to self-evaluate [101,200]. Utilization-focused Evaluation, meanwhile, encourages working with evaluation users in participatory ways that foster a sense of ownership by the users as they shape the evaluation approach [131]. Broadly speaking, participatory action research is often an appropriate method or framing for these kinds of participatory evaluations [197], and some analysis methods lend themselves well to dialogic and participatory contexts, such as Most Significant Change [162] (Box 5).

8. Autonomous evaluation principle: Support autonomous evaluation practice.

Evaluators should resist evaluation contracts and designs dictated principally by markets and individuals or organizations with vested interests and power in maintaining the unsustainable status quo [25,135]. Instead, evaluators should support stronger professionalization and autonomy of evaluation practice, which includes adopting clearer ethical and critical stances and permission to speak frankly in support of transformational change [25,146].

This principle is important because evaluations led by those in positions of power may be insufficiently critical (e.g., policy evaluations led by the associated government ministry) [19]. If evaluations are driven by globalized markets and vested interests in traditional kinds of economic growth, meanwhile, economic impacts (e.g., ‘value for money’ or ‘social return on investment’) will typically dominate the focus of evaluations, with less attention given to social and environmental considerations [25,135,139], and the evaluation may impose more homogenizing criteria that ignore individual uniqueness [166] (see Context Principle). For example, evaluation criteria for development are often based on industrial and technological paradigms focused on economic efficiency [181], and education evaluations underpinned by neoliberal thinking are product-focused, assessing student merit based on their ability to compete, whilst ignoring sociocultural diversity [166]. By supporting more autonomous and professionalized evaluation practice, however, evaluators could be more selective in choosing to conduct evaluations that align with ethical stances of supporting transformational change [25].

Operationalizing this principle will require collective action from evaluators to develop the ‘features of professionalism’, emphasized in the Prague Declaration [122], including ‘an ethical charter, expert knowledge, proven competencies and self-management’ [25]. As for particular evaluation approaches supporting this principle, Democratic Evaluation is notable for purposely avoiding excessive manipulation of evaluations by government agencies and the ‘information monopoly’ of program managers [98]. Adversary Evaluation has a goal of resisting neoliberal domination of evaluation and contributing to public good [87], whilst the African Evaluation Principles urge evaluators to guard against vested interests [88].

9. Partnerships principle: Cultivate mutualistic partnerships of knowledge and action.

Evaluators should cultivate mutualistic partnerships of knowledge generation and sharing when working together towards transformations, built upon strong ethical and professional codes of conduct [135,122]. They should also encourage and evaluate alignment and cooperation of different transformation efforts [78].

Partnerships of knowledge generation and sharing should promote mutual trust, shared values, transparency, honesty (including about mistakes and limitations), fairness, inclusivity, and shared responsibility for evaluation results [122]. The principle of ukama or ‘relatedness’ from African Ubuntu philosophy has been suggested as a foundation for nurturing solidarity and complementarity between Global North and South [165]. Partnerships should respect actors’ different responsibilities and role boundaries, and avoid letting egos and personal agendas dominate [163]. They should exist not only within evaluation teams, or with commissioners and users of evaluations, and other local stakeholders (including Indigenous peoples) – but also with other evaluators and national or international associations of evaluators, researchers, junior evaluators, students, interns, and other young people [135,122]. To develop strong, lasting relationships, it may be important to engage with stakeholders in person ‘in the field’ rather than only online engagement [122]. Another important aspect of this principle is that evaluators should encourage and assess alignment and cooperation of different transformation efforts [78]. This principle means taking care to avoid: developing only short-term, superficial relationships with partners [122]; failure to maximize accessibility of evaluation results [122]; and repetition, redundancy or incongruence of separate transformation initiatives [78]. Evaluation data are often considered proprietary, resulting in inadequate sharing across the field and therefore lack of coordination and integration [179].

Transparent and respectful knowledge exchange is a key aspect of this principle. Evaluators should devote intentional time and resources for sharing evaluation findings [179], and doing so in an honest and transparent way [146]. International fora and conferences provide suitable spaces for evaluators to share evaluation results and experiences with other evaluators, researchers and policy professionals [146], as well as writing articles [201]. Importantly, evaluators should make a particular effort to share their knowledge with actors working to transform systems [202], and with experts across different disciplines [78]. Evaluation reports should aim to be open-source, engaging, attractive and easily accessible, with clear, transparent and context-sensitive language that avoids unnecessary jargon [179,122]. Transparency means being honest about mistakes, failures and unintended consequences, and how theories of change evolved over time, for instance [179]. At the same time, evaluators should respect the unique protocols around knowledge-sharing and caretaking responsibilities around knowledge in particular (e.g., Indigenous) communities [109]. Overall, evaluators should reflect on whether they are being as effective as possible in helping to disseminate findings, influence others, and ultimately assist transformation [106,179,197].

This principle is important for numerous reasons. Inter- or transdisciplinary, multi-sector and multi-stakeholder approaches and partnerships are necessary for working with wicked problems [64,191,203]. Partnerships can provide a greater diversity of ideas and skills and encourage shared learning: for instance, evaluators could work productively with futurists and modelers to develop more powerful scenario planning (e.g., discussing what trends to pay attention to for the future) [56], and senior evaluators could act as mentors for junior evaluators, students and interns [122]. Indeed, the necessary shift to multiple and bricolaged methods (see Context and Detective Principles) means that having mixed teams of evaluators with different types of expertise – including newcomers, ‘outsiders’ and amateurs – may be critical (because one evaluator is unlikely to know every method) [57,122]. Greater knowledge-sharing by evaluators could furthermore enhance learning and innovation in the evaluation field as a whole [179]. Stronger coordination between different evaluations and ‘cross-case learning’ would enable them to complement each other in providing a more holistic understanding of a system [113,141], provide insights about how initiatives could be deepened, broadened and scaled up [204], and support more alignment and cooperation of initiatives to facilitate system change [78]. Evaluators should bear in mind that evaluation results may be used by new or previously unestablished actors in transitions, such as new NGOs or cooperatives, emphasizing the importance of making results transparent and accessible [113]. Finally, adopting these partnership values could encourage the implementation of evaluation recommendations [122].

While there are no specific evaluation methods to guide these forms of wider partnership-building per se, many of the evaluation approaches that promote decolonization, justice, and genuine participation (see Justice and Power Shift Principles), such as Transformative Evaluation, embed the building of trusting relationships with evaluation users (Table 2). Furthermore, the ‘Yin Yang Principle’ of Blue Marble Evaluation encourages engaging with different perspectives, harmonizing conceptual polarities and divisions, and transcending boundaries [90]. The African Evaluation Principles encourage a focus on developing relationships of reciprocity and mutual accountability [88], whilst the Prague Declaration also emphasizes the importance of mutualistic partnerships [122]. Utilization-focused Evaluation makes a point of encouraging wider dissemination of significant evaluation findings to expand influence [131].

10. Values principle: Surface, question and uphold values.

Evaluators should work with evaluands to surface and reflexively question the underlying values at play in transformation processes [89,101,205,206], and ensure that transformational values are upheld [136].

There are several dimensions to this principle. The first is that evaluators should surface, evaluate, and facilitate discussions about subjective aspects of evaluands, including their underlying values, worldviews and norms that shape behavior [89,101,205]. Evaluation does not take place in a value-free vacuum – there are always political, economic, cultural, and other value-laden forces at play [136]. For instance, different stakeholders will have different perspectives on where a system’s boundary lies [205], there will be plural values held for nature [89], desired system outcomes and evaluation results are likely to be contested [136,106], and political influence or vested interests may sway project outcomes [113]. As in the Context and Justice Principles, evaluators are therefore obliged to take into account the memories, heritage, and associated values of particular places, events, rituals, languages, arts and culture [132], and integrate the intellectual and emotive aspects of evaluation [168]. Indeed, evaluators should recognize the importance of people’s emotional and spiritual development during an evaluation as well as more objective dimensions, like gaining knowledge [182]. This requires a warm, nurturing social environment to encourage respect for other people’s beliefs and values [182].

The second dimension is the importance of reflexivity and questioning underlying values, assumptions, beliefs and worldviews. Evaluators should encourage critical reflection amongst stakeholders and help them to understand their own positionality [206], asking important and difficult reflexive questions even if the answers raise uncomfortable discussions [176], and potentially playing a ‘brokering’ role in mediating conflicts of values [89]. This may include questioning the acceptance of contextual aspects of evaluations (e.g., the policy strategies behind the policy impacts being evaluated, or the fundamental goals of a program or initiative more generally) as fixed and appropriate, whereas in reality they may have flawed assumptions that affect downstream actions [113,137]. Similarly, evaluators should beware blindly applying existing indicators to local contexts without reflecting on whether they truly are measures of success [136] (see Context Principle).

Equally, evaluators should recognize that they themselves are not value-free [25] and never perfectly neutral or objective [4], and reflect on how their own worldviews and values shape evaluations [106] (see Justice Principle). The worldview evaluators carry influences ontology, epistemology and axiology, and thus inevitably the way they evaluate [136]; evaluators see ‘through the eyes of [their] own history and environment’ such that their ‘view of reality is always partial and slanted’ [136]. Similarly, evaluators are never fully ‘external’ to and independent from evaluands – a second-order approach that recognizes evaluators’ embeddedness within the system being evaluated may be more appropriate [106,101]. A ‘good internal praxis’ [165] of reflexivity in evaluation teams would include articulating the assumptions guiding an evaluation and whose they are [132], transparency about power relations, learning from their own stories of success and failure, openness about their fears and vulnerabilities, and sharing this learning with others [165].

The third dimension is that transformation-focused evaluators should uphold and remain true to their own transformational values, whilst appreciating the many different perspectives that people have on what transformation means [78]. They should take an ethical stance of supporting positive transformation (see Transformation Principle), justice (see Justice Principle), and the public good [136], and integrate their ethical codes into evaluation activities, relations, organizations and training [171,122] (see Autonomous Evaluation Principle).

This principle involves a fundamental shift from evaluations built on a positivist worldview that assumes objectivity and neutrality of evaluators [136], leading to evaluations that typically focus on quantitative facts [101], omit important value-laden political and cultural dimensions [166], and lack reflexivity about their own values and approaches [106]. For example, much evaluation of education tends not to be reflexive about why something is learned [169]. While they may be well-suited to some contexts (e.g., evaluating medical interventions), more positivist, quantitative evaluations are often insufficient for the social contexts of transformation [78]. The principle is based on a more constructivist view of people as ‘socio-historical subject[s]’ with strong political-ethical dimensions [169].

There are various reasons why this principle is important to adopt. For one, it brings evaluations closer to the social reality of transformations [78]. Values are inextricably linked to more objective or quantitative aspects examined in evaluations [101]. Ignoring the underlying values and beliefs of participants – the ‘elephants in the room’ – could perpetuate a ‘culture of silence’ that simply reproduces a problematic status quo [101]. Shifts in social norms, values, beliefs, assumptions, and purposes are also powerful indicators of the depth of transformation [121]. Self-reflection by evaluators, meanwhile, requires humility, which disrupts the conventional power dynamic of the evaluator as directive expert [122] (see Power Shift Principle). Lastly, understanding what inhabitants value about their locale has been noted as encouraging regenerative mindsets and action, because these values can then be incorporated into manifesting the place’s potential [147].

Numerous evaluation approaches have an explicit focus on surfacing and reflexivity around people’s values, including: Adversary Evaluation; Deliberative Democratic Evaluation; Developmental Evaluation and especially Equity-focused Developmental Evaluation; Made in Africa Evaluation; the NICE Framework; and Principles-focused, Systemic, Transformative, Value(s)-based, and Visionary Evaluation (Table 2). In Reflexive Evaluation or Reflexive Monitoring in Action, evaluators act as ‘sparring partners’ [53] who continuously encourage innovation teams and networks to challenge underlying assumptions and systemic patterns in designing and managing transition initiatives [53,125]. Empowerment Evaluation and Feminist Evaluation encourage critical thinking particularly about power relations and prescribed roles [100,101]. Other tools and methods that help to elicit values include the Three Horizons framework, the Value Proposition Toolkit and Causal Layered Analysis (e.g., in encouraging participants to imagine radically different desired futures and the contrasting values underpinning them [64,160,170] – see also Foresight Principle), the Theory U framework [67], the Four Levels of Conversation framework [67], and the Most Significant Change method [207] (Box 5).

11. Learning principle: Develop evaluation cultures centered around ongoing learning.

Evaluators should develop evaluation cultures centered around ongoing learning during transformation processes [100,201], rather than driven by accountability requirements for showing positive results [139,162,191,114], and fear of failure [78].

Many evaluation contracts and designs, both today and historically, have been driven by requirements of accountability for the achievement of specific predetermined objectives, compliance with regulation, and the need to show positive results for funders and governments, rather than accountability for learning [139,162,191,114]. In international cooperation, for example, Global South countries are often required to present proof of a well-implemented project to donors, which satisfies donors’ accountability needs but not necessarily the learning needs of the country being evaluated [24]. This focus has given evaluation negative connotations of program problems, punishment and control [136], and created a fear of failure [78]. It also encourages simplistic, linear, ‘value-free’ designs and framings of interventions leading to particular outcomes, with limited timescales tied to funding cycles [139,208].

Transformation-focused evaluators should try to support a cultural shift in the meaning of accountability and purpose of evaluation. Accountability should instead be focused on things like alignment to principles and values, or ensuring rigorous and systematic learning processes and consequent adaptation rather than having particular impacts in complex systems [67,114]. Indeed, strategic learning may be the only outcome in system change that social innovators and evaluators can truly control [67]. A future-sensitive approach (see Foresight Principle) additionally suggests that initiatives should be accountable to future as well as present or past generations [78,177]. This then potentially shifts evaluation from being seen by evaluands as a punitive exercise and primarily for the benefit of funders and regulators, toward seeing the value of open, honest reporting and learning, and evaluation’s benefits for the people being evaluated, and beyond [100,162]. Organizations will need to be able to review negative feedback without defensiveness [201], be prepared to admit that original objectives were not met [179], and confront contradictions or tensions (e.g., between Global North and South perspectives and positionality), recognizing these outcomes as powerful opportunities for learning [165].

Overall, this shift means that adaptive learning for facilitating positive transformations becomes the central purpose of evaluation [139] (see also Developmental and Transformation Principles). Evaluators should advocate the value of evaluation to this end, especially persuading people in power [19], and helping others overcome associations of fear and blame around evaluations [201]. Avoiding the term ‘evaluation’ altogether may be useful in the initial stages of attracting participants in some evaluations [162] (instead using language like ‘review’ or ‘consultation’ [209]). Having stakeholder ‘champions’, who believe in the value of evaluation and are enthusiastic to take results to those with the power to implement change, is also useful [136]. Creating an evaluation culture centered around learning requires further steps to embed this mode of evaluation. For instance, evaluation should ideally become a core activity of all teams, rather than performed by just one person or unit [201]; moreover, longer time horizons are needed, since transformation should never be seen as ‘done’ and instead be understood as a continual learning process [121] (see Developmental Principle).

The real result of an evaluation, after all, is the learning from its process and findings and how this steers action [78,122]. Learning takes many forms, and takes place at many levels and at all stages of an initiative or transition (not just at the start or end) [210]. In transitions there is also a need to ‘unlearn’ old patterns and ways of doing things as well as learn new ones [210]. Some have distinguished between single-, double- and triple-loop learning [211]. ‘Single-loop learning’ (i.e., problem-detection-correction) is limited to asking whether intended outcomes have been achieved, and monitoring predetermined performance indicators [152] – in other words, ‘are we doing things right?’ [211,212]. Double-loop learning, meanwhile, involves examining the causal mechanisms underlying observed change, asking ‘are we doing the right things?’ [211,212]. Triple-loop learning goes even further, exploring and questioning underlying assumptions, values and beliefs, and asking ‘how do we decide what’s right?’ [104,211,212]. Reflexivity (see Values Principle) is therefore key for deeper learning to occur. To support learning towards system innovation, it is important not just to encourage reflexivity but also to monitor and record reflexivity changes and learning journeys (as in Reflective Monitoring in Action) – i.e., collective reflection on an initiative’s reflexivity by the actors involved will in turn help to increase the initiative’s reflexivity [213].

Many different frameworks, in addition to single-, double- and triple-loop learning [211,104], help to conceptualize and facilitate more transformational forms of learning. In Freirean pedagogy and the related Telessala Methodology, students ‘learn by doing’ and become empowered, reflective citizens, rather than unquestioningly absorbing information delivered didactically [101,182]. The concepts of experiential learning [171,206] and design-based learning [94] also emphasize a learning-by-doing approach. In generative learning, learning is based on integration of new and pre-existing knowledge in creative ways [162]. There are also ideas about learning being shaped by social and contextual interaction, such as social learning [191], the Critically Reflective Work Behavior framework (used particularly for social learning in businesses) [94], and the socio-cultural perspective on learning, whereby learning is understood to originate from social and material interaction and is shaped by cultural, historical and relational factors, such that learning involves the ‘whole person’ [43]. ‘Transformative learning’ shares features of triple-loop learning, experiential and social learning; it is action-oriented and includes critical reflection of underlying assumptions and beliefs and how these are shaped by context [171]. In futures research, ‘anticipatory action learning’ is used to reflexively interrogate possible and desired futures and how they could be brought about [189] (see Foresight Principle). While learning traditions including collaborative learning (from educational sciences), organizational learning (from management studies), and interactive learning in the learning economy (from institutional economics) have value for understanding learning in system change processes, they are not yet sufficient for working with complexity [210]. Hence, novel theories about complexity-informed learning have emerged from transformation-focused evaluation practices, such as system learning, whereby actors learn to redefine system barriers as opportunities and are thus inspired to design activities contributing to systemic change [213].

Many overarching evaluation approaches assimilate these ideas about learning, including Blue Marble, Design-driven, Developmental, Dynamic, Empowerment, Equitable, Participatory, Real-time, Reflexive, Transformative and Visionary Evaluation (Table 2). For instance: Transformative Evaluation draws on transformative learning theory [199]; Reflexive Monitoring in Action incorporates ideas about system learning [213]; and Empowerment Evaluation aligns with Freirean principles [101]. Several evaluation frameworks for evaluating complex systems change (e.g., those of Cabaj, Hargreaves, and Preskill et al.) also aim to create cultures of learning (Table 2). A number of other data collection and analysis methods are also geared towards supporting transformational learning, such as authentic dialogue [101], After Action Reviews [67], and appreciative inquiry [64]. Failure Reports and Intelligent Failure Learning Loops are structured processes to encourage admission and understanding of ‘failures’ in interventions and learn from them [67]. More generally, interviews and focus groups are important data collection methods for enabling reflection [123].

12. Transformation principle: Focus on evaluating and supporting systemic transformations towards regenerative futures.

Evaluators should facilitate systemic transformation towards regenerative futures, remembering the urgency for global transformation in a ‘time-is-running-out world’ [4,26], and therefore focus on evaluating transformations and integrate transformational principles across different facets of evaluations [63].

Whilst the nature of transformation is inevitably context-dependent, transformation should be understood as getting to deeper causes of problems, rather than incremental, superficial or reformist change that perpetuates problematic current systems [4,136]. Evaluators should aim to assist transformations to regenerative, resilient futures, which feature ‘adaptive sustainability’ [4], social justice [190] (see Justice Principle), and the flourishing of life [106]. Evaluation can support these kinds of transformations in multiple ways, including by understanding the implications and impacts of ongoing transformations, as well as providing developmental feedback for purposeful interventions in systems change [19]. This principle is vital, and ethically imperative, because the urgency of current crises means that fundamental change is needed across societies to safeguard the survival and thriving of life [26,113].

This principle shifts evaluation criteria. Many past evaluations have been geared towards increasing our understanding about current systems and measuring problems rather than evaluating transformative principles, visions and action [4,172,214]; while these have produced much useful information, they have tended to support only incremental improvement [113] which at best slows the ‘incremental destruction of life’ [172]. Such evaluations have tended to focus on criteria of economic growth and productivity [215], effectiveness in achieving micro-scale outcomes [24], and whether money, time and effort were spent efficiently [24], and give insufficient attention to environmental impacts [135]. Transformation-focused evaluation criteria, on the other hand, would aim to encourage transformative action [214] and regenerative thinking [216]. They would need to link interventions to system-level impacts [86] (such as how underlying structures, patterns and mindsets of systems are shifting [64,122]), developing and continually testing theories of system change [64]. Transformation-focused evaluation criteria would assess: visions of futures and transformation trajectories, and whether these are meaningful and genuinely oriented towards system change rather than more isolated or incremental change [63,113] (see Foresight Principle); experimentation [113]; the alignment and momentum amongst different actors, networks, institutions, ideas, initiatives and movements towards transformation [4] (see Partnerships Principle); and whether transformation rhetoric is backed up by appropriate action rather than greenwashing [4]. They would move away from assessing isolated pillars of sustainability (e.g., environmental, social, economic) towards assessing interactions between different dimensions, including co-evolutionary relationships between human systems and wider nature [217]. Transformation-focused evaluation would also assess the health of social-ecological systems over multiple generations into the future [109,215], and measure how conditions conducive to life are being created or maintained in a place, so people can see their role in increasing a place’s vitality or quality of life [172].

This principle also underscores the imperative to fundamentally redesign evaluation practice, as laid out in all the other principles, and integrate these transformational principles across different facets of evaluations [19,63]. This is a major paradigmatic shift, and not merely tweaking evaluation, like a greater reliance on technology as the evaluation field adapted to the COVID-19 pandemic [19]. The principle particularly highlights a shift from the traditional neutrality of evaluation, as it recognizes that evaluators are not independent of the system being evaluated (see Values Principle); rather, they have a personal interest in successful transformation (‘skin in the game’) [4]. There have been calls for evaluators to instead become more vocal communicators and advocates for transformation [146]. Evaluations are thus seen as part of transformation processes, not removed from them – i.e., seeing evaluation as system change, not just for or about system change [4,64]. Evaluators may have to be pickier about the contracts they choose to accept, refusing those that are not sufficiently geared towards transformation or restrict the evaluator’s ability to encourage more transformational approaches [78] (see Autonomous Evaluation Principle).

Numerous overarching evaluation approaches explicitly support transformational change as an ethical imperative. These include Blue Marble, Dynamic, Feminist, Footprint, Transformative, and Visionary Evaluation, Transformative Empowerment Evaluation, Transformative Participatory Evaluation, the African Evaluation Principles, Gates et al.‘s ethics of evaluation for socio-ecological transformation, and the Prague Declaration (Table 2). Many other approaches and frameworks, while not necessarily taking such a strong ethical stance, nonetheless focus on assessing transformational, systemic change, including Cabaj’s Inquiry Framework for Evaluating Systems Change, Hargreaves’ framework for evaluating system change, Latham’s Systems Change Evaluation framework, Mickwitz et al.’s theory-based framework for evaluating transitions, the Orders of Outcomes framework, Patton’s Evaluation Criteria for Evaluating Transformation, Patton & Felcis’ Principles for Evaluating Transformation, and the TIPC methodology (Table 2). The Systems Change Evaluation Canvas tool can aid reflection when designing a systems change evaluation [218].

Various theories and frameworks help to conceptualize transformations and other forms of change in complex social-ecological or socio-technical systems. These include transition theory and regime shifts [113,152,219], panarchy and adaptive cycles [152], Meadows’ leverage points and the related Intervention Level Framework [32,67,102], the Action Scales Model [102], the World Mandala [78,156,157] and Ambition Loops [78,220,221] (particularly for identifying synergistic actions), actor network theory [113], innovation systems theory [113], the multi-level perspective [113], the Stewards’ Pathway (a developmental theory of system change) [64], Kania et al.’s system change framework [64], and ‘directionality’ of system change [138,152]. The Signals of Transformational Change and related Four Dimensions of Transformational Change frameworks make an important shift from only tracking limited, specific quantitative metrics and indicators to tracking broader signals of systemic shifts at different stages of progress [62,222]. It is recommended to present evidence of contributions to system changes (e.g., changes in an initiative’s system maps over time and/or more narrative-based ‘contribution stories’) to different stakeholders to obtain their interpretations, because these will vary depending on whose perspective is taken into account [102]. This is supported by tools such as SenseMaker, based on the Cynefin framework from management science, which helps to obtain micro-narratives from the system context, drawn from people’s experiences, reflections, perceptions and attitudes, and make sense of emergent patterns from those narratives [102,223]. The Most Significant Change method can also help to identify anecdotal evidence of transformative impacts [206] (Box 5). Other frameworks such as Three Horizons, the Six Pillars framework, Causal Layered Analysis and the Theory U framework also help to convene conversations about transformational versus reformist change, and structure exercises of visioning radically different futures [67,78,170] (see Foresight Principle).

Various disciplines, frameworks and tools exist for focusing evaluation on regenerative systems and futures. Fields such as regenerative design, regenerative development and regenerative sustainability, as well as many Indigenous frameworks and philosophies, already embrace many of the principles laid out in this paper in support of the flourishing of life – such as considering change as more unpredictable and emergent, recognizing unique locales, embracing reflexivity and co-creation, and seeing the evaluand as part of a wider social-ecological system [15,172,224]. Regenerative design and development tools, such as the Regenerative Community Development Evaluation Tool [216], Regenerative Capacity Index [225], and LENSES framework [167], emphasize a shift from focusing on evaluating outcomes (e.g., of sustainable buildings) to evaluating processes of design and building community regenerative capacity [172,224], avoiding generic top-down forms of assessment [224]. Various other tools encourage ambition, imagination and regenerative thinking in exercises of envisioning desired and possible futures, such as the Regenerative Lens [15] or Seeds of Good Anthropocenes [188].